The University of Texas at Austin Computer Science doctoral student Soumyajit Gupta, Texas iSchool alumni Vivek Khetan, former postdoctoral researcher Mucahid Kutlu, and Associate Professor Matthew Lease were recently awarded Best Student Paper Award at the 41st European Conference on Information Retrieval (ECIR 2019). Their paper titled, “Correlation, Prediction, and Ranking of Evaluation Metrics in Information Retrieval,” was presented this April in Cologne, Germany.
According to Lease, “search is now critical to 21st century information access, yet ensuring search algorithms work well is challenging given the vast scale at which algorithms must be evaluated.” Evaluation metrics are ways researchers and algorithm designers assess how well search results ultimately satisfy information needs of the searcher.
In their paper, Lease and his collaborators explored strategies for optimizing the choice of evaluation metrics to measure, and assessed 23 popular evaluation metrics. “Search algorithm developers cannot possibly consider every evaluation metric in assessing how well their systems perform, so it is critical that they are judicious about focusing their effort on evaluation metrics that are most informative to improving the search experience for the end-user.”
Another important aspect of the work is that the metrics considered by the team are language-neutral, meaning that these metrics can also be used to assess search algorithms running in non-English languages, such as Modern Standard Arabic. The research team proposed two methods for algorithmic selection of evaluation metrics, and these methods provided both lower time and space complexity than prior work. These methods also provided a theoretically justified, practical approach to automatically select the most informative and distinctive evaluation metrics to measure.
Lease was especially happy for his student collaborators, noting that “this was a total team effort, with each of us making distinct contributions to the development and analysis of methods.” He also described it as a “slam dunk” for Ph.D. student Soumyajit Gupta, given that this is his first research publication on search engine research, and their first research collaboration together.
Lease also describes the work as “a fantastic example of international research collaboration,” funded by the government of Qatar in a program founded to foster such international partnerships. “The potential value for advancing evaluation of search algorithms for Arabic also helps ensure technological innovation and advances extend to the diversity of the world and its many languages.”
Soumyajit Gupta, an advisee of Lease, is a Texas Computer Science PhD student, Vivek Khetan is an alumnus of the Texas iSchool, and former iSchool postdoctoral fellow, Mucahid Kutlu, is now a faculty member at TOBB University of Economics and Technology in Turkey.
This research was funded by a National Priorities Research Program (NPRP) grant # 7-1313-1-245 from the Qatar National Research Fund (a member of the Qatar Foundation), whose objective is to “competitively select research projects that will address national priorities through supporting basic and applied research as well as translational research/experimental development.”
Lease received the three-year grant in 2015 in collaboration with Qatar University Associate Professor of Computer Science Tamer Elsayed, to improve current search engine technology for the Arabic-language Web. Elsayed is also an iSchool graduate, receiving his Ph.D. from Maryland’s program.