Best Student Paper Awarded to iSchool Associate Professor and Student CollaboratorsSandlin, Anu  | Apr 30, 2019
The University of Texas at Austin Computer Science doctoral student Soumyajit Gupta, Texas iSchool alumni Vivek Khetan, former postdoctoral researcher Mucahid Kutlu, and Associate Professor Matthew Lease were recently awarded Best Student Paper Award at the 41st European Conference on Information Retrieval (ECIR 2019). Their paper titled, “Correlation, Prediction, and Ranking of Evaluation Metrics in Information Retrieval,” was presented this April in Cologne, Germany.
According to Lease, “search is now critical to 21st century information access, yet ensuring search algorithms work well is challenging given the vast scale at which algorithms must be evaluated.” Evaluation metrics are ways researchers and algorithm designers assess how well search results ultimately satisfy information needs of the searcher.
In their paper, Lease and his collaborators explored strategies for optimizing the choice of evaluation metrics to measure, and assessed 23 popular evaluation metrics. “Search algorithm developers cannot possibly consider every evaluation metric in assessing how well their systems perform, so it is critical that they are judicious about focusing their effort on evaluation metrics that are most informative to improving the search experience for the end-user.”
Another important aspect of the work is that the metrics considered by the team are language-neutral, meaning that these metrics can also be used to assess search algorithms running in non-English languages, such as Modern Standard Arabic. The research team proposed two methods for algorithmic selection of evaluation metrics, and these methods provided both lower time and space complexity than prior work. These methods also provided a theoretically justified, practical approach to automatically select the most informative and distinctive evaluation metrics to measure.
Search is now critical to 21st century information access, yet ensuring search algorithms work well is challenging given the vast scale at which algorithms must be evaluated.
Lease was especially happy for his student collaborators, noting that “this was a total team effort, with each of us making distinct contributions to the development and analysis of methods.” He also described it as a “slam dunk” for Ph.D. student Soumyajit Gupta, given that this is his first research publication on search engine research, and their first research collaboration together.
Lease also describes the work as “a fantastic example of international research collaboration,” funded by the government of Qatar in a program founded to foster such international partnerships. “The potential value for advancing evaluation of search algorithms for Arabic also helps ensure technological innovation and advances extend to the diversity of the world and its many languages.”
Soumyajit Gupta, an advisee of Lease, is a Texas Computer Science PhD student, Vivek Khetan is an alumnus of the Texas iSchool, and former iSchool postdoctoral fellow, Mucahid Kutlu, is now a faculty member at TOBB University of Economics and Technology in Turkey.
This research was funded by a National Priorities Research Program (NPRP) grant # 7-1313-1-245 from the Qatar National Research Fund (a member of the Qatar Foundation), whose objective is to “competitively select research projects that will address national priorities through supporting basic and applied research as well as translational research/experimental development.”
Lease received the three-year grant in 2015 in collaboration with Qatar University Associate Professor of Computer Science Tamer Elsayed, to improve current search engine technology for the Arabic-language Web. Elsayed is also an iSchool graduate, receiving his Ph.D. from Maryland’s program.
Matt Lease receives grant from QNRF to improve Arabic language search engine technologyZhang, Yang  | Nov 04, 2015
While search engines have become incredibly accurate for navigating through websites written in English, finding relevant webpages in other languages is often more difficult.
UT iSchool Associate Professor Matthew Lease and Qatar University Assistant Professor of Computer Science Tamer Elsayed are collaborating together to improve current search engine technology for the Arabic-language Web. Lease and Elsayed received an $884,000 grant from the Qatar National Research Fund for three years for their project “Efficient and Scalable Evaluation for Searching Massive Arabic Social Media and Web Collections.”
“In addition to significantly less research and development investment having been made, the non-English Web is smaller in size for many languages, making it harder to find a relevant needle in a haystack,” Lease said. “Linguistic differences from English can further require tuning search algorithms for each language of interest, and some human populations are inherently polylingual. For example, Arabic is not a single language, but rather a collection of closely-related languages, from Modern Standard Arabic - used for formal writing - to several regional dialects - used in conversation and informal writing.”
To create a controlled environment for search engine experimentation, the professors will crawl the Arabic Web to collect a massive dataset “snapshot”. They will use crowdsourcing to reach Arabic speakers around the world and collect diverse search queries to evaluate the effectiveness of search algorithms developed.
The project also includes significant funding for Lease to fully support doctorate student research assistants as part of his Information Retrieval and Crowdsourcing Research Lab.
“Student research is essential to scientific progress, and I look forward to seeing the amazing things my future research assistants will accomplish on this project,” he said. “It’s been a great pleasure getting to know and help mentor Tamer’s students at Qatar University, and vice versa for him helping mentor students working on the project here at UT-Austin.”
The two professors met at the University of Maryland School of Information while Lease was interviewing for a post-doctorate opportunity and Elsayed was finishing his doctoral degree.
“We were excited to reconnect and renew our iSchool ties across our separate continents,” Lease said. “This project idea provided the perfect opportunity to work together on a problem of mutual interest which is of great practical importance to society and presents us with plenty of tough technical challenges to make it all work.”
Matt Lease on the Information Retrieval and Crowdsourcing LabDec 31, 2013
How would you characterize the purpose and goals of the Information Retrieval and Crowdsourcing Lab?
To advance the state-of-the-art methodologies for search (i.e., how we both build effective search engines and measure that effectiveness, across a diverse range of search tasks) and human computation / crowdsourcing (i.e., how we effectively mobilize and organize people online to accurately perform information processing tasks, particularly difficult tasks which remain beyond what today's best intelligent systems can achieve automatically).
What attributes (e.g. skills, interests, background) make a student an ideal candidate to work with you in the IR & Crowdsourcing Lab?
My funded research assistants (RAs) typically have a computer science or equivalent background, with strong backgrounds in both computing and math. Beyond my RAs, I have also advised many other students from other backgrounds who bring other diverse skills to bear on these problem areas.
For example, I recently advised published research and a Master's Thesis on legal issues in crowdsourcing. This research anticipated subsequent litigation that has occurred regarding the question of whether "microwork contributors" on crowdsourcing platforms should be classified as employees rather than independent contractors. Given how thoroughly such crowdsourcing has become ingrained in how we build intelligent systems today, I was particularly concerned that our technical house of cards could come crashing down if the legal foundation proved faulty. I mention this just as one example of how crowdsourcing is such a fascinating socio-technical area which offers such a rich diversity of interesting research questions which students from different backgrounds could pursue.
The number one need by far a student needs to succeed is the passion, drive, and imagination to do good work which will change the world. We are not standing by the sidelines to wait to see what tomorrow's world will look like. Instead, we are the ones leading the charge to build technology and make discoveries that will impact the world we live in today and make dreams for the future become a reality. This is what means to be at world-class research university and lead the charge at the forefront of science. There's no better place to be.
How many departments on campus are currently represented in the IR & Crowdsourcing Lab and what possible collaborations do you foresee in the future?
We regularly work with faculty and students from computer science (CS), electrical and computer engineering (ECE), and linguistics. We also interact with others from Mathematics, Statistics and Scientific Computing (to be renamed "Statistics and Data Science"), and McCombs' Information, Risk and Operations Management. Currently we have two pending projects with others units: one with ECE which uses search engine technology to find bugs in software, and one with CS which integrates AI and crowdsourcing to create an intelligent building, a form of "ubiquitous computing".
What are some of the resources the lab has to offer?
Google has kindly donated a pool of Android Phones and Google TV devices, and we have some fast computers and cool datasets. The main resource is the awesome students that are there to work with, along with lots of free caffeine!
Where can people learn more about the Information Retrieval and Crowdsourcing Lab?
My crowdsourcing webpage has become the defacto place on the Internet to track important research events (conferences, journals, tutorials and talks, etc.). I created it just to track these things for myself, but it has turned out to prove useful to many others as well.
I've been fortunate to be part of two significant research initiatives charting future research.
- In terms of search engine technology, SWIRL'12: The Second Strategic Workshop on Information Retrieval in Lorne, brought together 45 of the top researchers in the field to chart a roadmap of long-term challenges and opportunities for the field. Our report is online at: http://sigir.org/forum/2012J/2012j_sigirforum_A_allanSWIRL2012Report.pdf
- In terms of crowdsourcing, I worked with leading researchers from seven other universities to envision the future of crowdsourcing and important research challenges and opportunities to be tackled. The paper appeared at ACM CSCW 2013 and can be found online at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2190946.