Research Notes

Traditional workplaces spend a fair amount of effort mentoring and training their workforce as a way to increase the quality of their work and their job satisfaction. Does mentoring crowd workers also increase the quality of their work? How can one mentor the crowd workforce? These were the questions we tried to tackle this weekend at the Crowd Camp Workshop at CHI.

First we approached these questions by setting up a task that we thought people could improve through mentoring: slide design. We asked Mechnical Turkers to help us improve the design of a set of three slides (which we purposely created to look really ugly). We provided Turkers with a set of guidelines for well-designed slides that included tips on color, graphics, text, etc. We then gave each Turker a slide to improve.

The control group received the guidelines but no mentorship. The group with mentorship in addition of receiving the guidelines also received an offer to chat with an “expert” at any given point. The expert was accessible through a chat window next to the slide. We then asked a different group of Turkers to evaluate the quality of the slides. We then compared the ratings of the baseline slides we provided them with those created by the Turkers in each of the two groups. As we expected, the mentorship condition led to higher quality slides. However, we also realized how time-consuming the mentorship process was. More work needs to be done to assess if the cost of mentorship is worth it from a purely economical perspective.

This was my first time playing with Mechanical Turk, but I got to learn from people with a lot of experience with these platforms. The team included: Bjoern Hartmann, Edith Law, Kurt Luther, Kurtis Heimerl, Lixiu Yu, Philipp Gutheim, and Sanjay Kairam.

The idea of training a workforce that often treated as machines was an interesting one. It raises all sorts of interesting issues with regards to the ethics of crowd work and the responsibilities of crowd employers. This was actually part of the topic that another team tackled during the workshop.

This entry was posted on Monday, May 7th, 2012 at 1:03 am and is filed under meta.