The Benefits of a Rationale

When collecting human ratings or labels for items, it can be difficult to measure and ensure data quality control, due to both task subjectivity (i.e., lack of a gold standard against which answers can be easily checked), as well as lack of transparency into how individual judges arrive at their submitted ratings. Using a paid microtask platform like Mechanical Turk to collect data introduces further challenges: crowd annotators may be inexpert, minimally trained, and effectively anonymous, and only rudimentary channels may be provided to interact with workers and observe work as it is performed. To address this challenge, I demonstrate how a very simple methodology can provide new traction on these problems: all we do is require judges to provide a rationale supporting their decisions.