INF385T.3/CS395T: Human Computation and Crowdsourcing (Fall 2017)

THIS COURSE IS CROSS-LISTED; IF ONE SECTION IS FULL, PLEASE ENROLL IN THE OTHER. All students will receive the same credit toward graduation requirements regardless of which section they enroll in.

ON THE WAITLIST? I will do my best to ensure that any graduate student who wants to be in the class can enroll. Show up the first day of class and I will probably be able to get you in.

Registration notes specific to Computer Science (CS) students:

• Masters students: Want Minor Coursework credit? Email gradoffice@cs.utexas.edu to obtain prior approval.
• Masters students: Want Major Coursework credit? Email gradoffice@cs.utexas.edu if enrolled in the INF listing (to count it as a CS395T).
• PhD students may count this class toward the Depth requirement.

Instructor: Matt Lease (see "About the instructor" at bottom · publication list)
Day and Time: Thursdays 12-3pm
Location: UTA 1.212
Unique IDs: 28350 (INF) and 51885 (CS)

Syllabus
Weekly Schedule

Past offerings: Fall 2015 · Spring 2014 · Spring 2012 · Spring 2011

Textbook: none required, all readings online

Prerequisites: No prior knowledge is required; all interested and motivated students are invited to attend. This course typically attracts significant student participation across a wide variety of disciplines: information science, computer science, linguistics, electrical engineering, and design studies. Course activities are intended to serve the needs of both (1) those studying to work professionally in the area or conduct research in IR, and (2) non-specialists interested in gaining broader exposure and understanding of human computation and crowdsourcing methods and systems.

Course summary. This graduate seminar will review the latest research in human computation and crowdsourcing by reading peer-reviewed conference and journal papers. Students will work individually or in pairs on a self-selected, semester-long course project. The course culminates in a public poster session where course projects are presented

Global growth in Internet connectivity and participation is driving a renaissance in human computation: use of people rather than machines to perform certain computations for which human competency continues to exceed that of state-of-the-art algorithms (e.g. AI-hard tasks such as interpreting text or images). Just as cloud computing now enables us to harness vast Internet computing resources on demand, crowdsourcing lets us similarly call upon the online crowd to manually perform human computation tasks on-demand. As crowd computing expands traditional accuracy-time-cost tradeoffs associated with purely-automated approaches, the potential to achieve these enhanced capabilities has begun to change how we design and implement intelligent systems.

While early work in crowd computing focused simply on collecting more data from crowds to train automated systems, we are increasingly seeing a new form of hybrid, socio-computational system emerge which harnesses collective intelligence of the crowd in combination with automated AI at run-time in order to better tackle difficult processing tasks. As such, we find ourselves today in an exciting new design space, where the potential capabilities of tomorrow.s computing systems is seemingly limited only by our imagination and creativity in designing algorithms to compute with crowds as well as silicon.

Examples of human computation systems: DuoLingo · EyeWire · FoldIt · GalaxyZoo · MonoTrans · Legion:Scribe · Mechanical Turk · PlateMate · ReCaptcha · Soylent · Ushahidi · VizWiz

Introductions to Human Computation and Crowdsourcing:

• Alexander Quinn and Benjamin B. Bederson. Human computation: a survey and taxonomy of a growing field. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2011.
• Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matthew Lease, and John Horton. The Future of Crowd Work. In Proceedings of the 2013 conference on Computer supported cooperative work, pp. 1301-1318. ACM, 2013.
• Edith Law and Luis von Ahn. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 5, no. 3 (2011): 1-121.
• Matthew Lease and Omar Alonso. Crowdsourcing and Human Computation, Introduction. Encyclopedia of Social Network Analysis and Mining (ESNAM), pages 304-315, September 2014.
• See also: AAAI Crowdsourcing and Human Computation (HCOMP) conference proceedings

Advances in research have also translated into a thriving private sector, with many existing startups and opportunities for more.

Want to publish original research?

In previous offerings of the course, several of the best, most innovative course projects have been extended beyond the semester until the work was in publishable form. If you have a great idea and are willing to work hard to get it published, the course project provides a great opportunity to refine the idea and get started developing the project with regular feedback and advising from the instructor. Examples of past course projects that were subsequently published include (see publications for links):

• Akash Mankar, Riddhi J. Shah, and Matthew Lease. Design Activism for Minimum Wage Crowd Work. In 5th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2017. See extended technical report: arXiv 1706.10097. [ bib | pdf | tech-report ]
• Yalin Sun, Pengxiang Cheng, Shengwei Wang, Hao Lyu, Matthew Lease, Iain Marshall, and Byron C. Wallace. Crowdsourcing Information Extraction for Biomedical Systematic Reviews. In 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track, 2016. 3 pages. arXiv:1609.01017. [ bib | pdf ]
• Xiaoyu Zeng and Ruohan Zhang. Participatory Art Museum: Collecting and Modeling Crowd Opinions. AAAI 2017, pages 5017-5018. [pdf]
• James Cheng, Monisha Manoharan, Matthew Lease, and Yan Zhang. Is there a Doctor in the Crowd? Diagnosis Needed! (for less than $5). In Proceedings of the iConference, 2015.
• Yinglong Zhang, Jin Zhang, Matthew Lease, and Jacek Gwizdka. Multidimensional Relevance Modeling via Psychometrics and Crowdsourcing. In Proceedings of the 37th international ACM SIGIR conference on Research and Development in Information Retrieval, pages 435-444, 2014.
• Di Liu, Ranolph Bias, Matthew Lease, and Rebecca Kuipers. Crowdsourcing for Usability Testing. In Proceedings of the 75th Annual Meeting of the American Society for Information Science and Technology (ASIS&T), October 28-31 2012.
• Stephen Wolfson and Matthew Lease. Look Before You Leap: Legal Pitfalls of Crowdsourcing. In Proceedings of the 74th Annual Meeting of the American Society for Information Science and Technology (ASIS&T), 2011
• Adriana Kovashka and Matthew Lease. Human and Machine Detection of Stylistic Similarity in Art. In Proceedings of the 1st Annual Conference on the Future of Distributed Work (CrowdConf), San Francisco, September 2010

How to post your course paper online as a technical report? See an example from a previous semester.

• arXiv (Create new submission)
• Social Science Research Network (SSRN)
• UT CS Technical Reports (Create new submission)

Looking for a funded Research Assistant (RA) position? I typically do not offer RA positions until a student has taken a course with me and demonstrated their abilities and drive to succeed. While the availability of an RA position depends on available funding, I am often looking for new RAs to help me advance the current state-of-the-art in research.