Crowdsourcing Using Mechanical Turk: Quality Management and Scalability — Harvard School of Engineering and Applied Sciences – 11/10
ø
I will discuss the acquisition of “labels” for data items when the labeling is imperfect. Labels are values provided by humans for specified variables on data items, such as “PG-13” for “Adult Content Rating on this Web Page.” With the increasing popularity of micro-outsourcing systems, such as Amazon’s Mechanical Turk, it often is possible to obtain less-than-expert labeling at low cost. I will present strategies of managing quality in a crowdsourcing environment, showing in parallel how to integrate data acquisition with the process of learning machine learning models. I illustrate the results using real-life applications from on-line advertising: leveraging Mechanical Turk to help classify web pages as being objectionable to advertisers. Time permitting, I will also discuss our latest results showing that mice and Mechanical Turk workers are not that different after all.
