Research Notes

Write up by Andrés Monroy-Hernández and Elena Agapie, building on the work of J. Nathan Matias

Motivated by the disappearance of local newspapers, this past summer, we started to explore new ways of supporting community news production through collaborative writing tools. The first incarnation of this is NewsPad, a system for neighborhood communities to collaboratively to report on local events such as festivals and town hall meetings.

One of the first challenges we encountered when testing NewsPad in the wild, was the difficulty of bootstrapping these collective action efforts to produce even lightweight articles in the form of lists, also referred to as listicles.

We decided to explore this challenge using on-demand, location-based labor through TaskRabbit. We were able to produce articles about the events in under an hour, and for less than $100. Here we we share some of initial reflections after running a few experiments.

The challenge with event reporting: locative, time-sensitive tasks

From our interviews with neighborhood bloggers and our initial experiments, it seems like the task of putting together an event report, was a big enough task that people had a hard time committing  and following through. Also, creating an article about an event in the physical world, often requires attending in person. Although there are many other challenges worth exploring, such as finding ways of intrinsically motivate participation through volunteers, we decided to first focus on these issues of task-size, location, and time. In particular we were interested in these two questions:

1. How can we break the task of producing an event report into smaller, simpler tasks that anyone is capable of performing?

2. How can we coordinate these participants within the tight time and location constraints of an event? Given that once the event is over, there’s not much that can be done to collect information, we set ourselves the constraint of being able to produce the article in under an hour.

We have run five experiments reporting small local events in Seattle relying on on-demand paid labor from TaskRabbit, an online platform that enables the outsourcing of offline and online tasks.

Using this technique, we have been able to report events such as a town hall local transit meeting, a local art show, and a book presentation of the founder of Reddit.

Six-step process for crowdsourcing local stories

One of our goals with these experiments was to identify the menial organizational parts of the process so they could be offloaded to an automated system. For now, we had an online TaskRabbit worker act as a manager of other TaskRabbits and make sure they follow this script.

1. Choosing the event to cover. This is something that we did manually by traversing local event listings, and social media. In the future, we are planning to automate it more by, for example, using event detection techniques from Twitter data that might trigger the process automatically. Also, we are also hoping that the organizations behind the events might be the ones to start the process.

2. Identifying tasks. In the stories that we covered, we had the TaskRabbit manager identify which of the following tasks were needed and then recruit other workers to execute them. Tasks included:

• Taking photos at the event.
• Interviewing event attendees producing either text, video, or audios.
• Collecting organically-produced content from social media.
• Curating all the content collected by the workers and put it together using NewsPad in the form of a listicle.

3. Recruiting workers and assigning them roles. Again, we relied on online crowdworker for this. This person would select and give people pre-defined instructions on what they would do at the event. We hope to automate most of this through the use of TaskRabbit’s API.

4. Worker coordination. We relied on Twitter and phone for all the communication among crowdworkers. We think that this can also be automated for the most part as most of the interactions involved canned notifications and well-crafted guidelines. In some cases, a human would be needed though, especially when things are going awry, for example if the workers don’t produce any content for a long time (~20 min) a reshuffling of tasks need to happen and a human might have to make a judgment call on who to assign the tasks or even hire someone new.

6. Dissemination. Once the articles were written, we advertised them on Twitter and to the organizers of the events. The posts were received positively by the event organizers who responded with messages: “Wow!  Awesome pics and writing!!… I’d love to meet you and chat in person!!”, or “it’s so great to see the artists featured in such a personal way!”

However, it is unclear if the posts reach all the people they could reach. Newspapers were able to build audiences by being present at people’s doorsteps and in the streets, what would be the equivalent of these hyperlocal crowdsourced community news? Would it make sense to print out these stories and ask offline crowdworkers to post them around the neighborhoods? These are questions we’ll be exploring in the future.

Challenges and Opportunities with Offline Crowdwork

Physical crowdsourcing opens new opportunities for distributed work beyond the screen. In this experiments we used TaskRabbit, and to some extent Craigslist (although we eventually just redirected those individuals to TaskRabbit for easier billing). We found TaskRabbit to be effective at what we were trying to accomplish, but here are some of the key features that we found particularly useful:

• Worker verification. TaskRabbit does background checks for all workers, guaranteeing some degree of reliability.
• Reputation system. Workers are ranked by employers, their expertise and history is reflected in their profiles.
• Private tasks. Workers can be individually be targeted to perform a task, this is especially useful with workers with whom we had a positive interaction.
• Integrated payment and recruitment. This stands in contrast with alternatives like Craigslist where one could find local workers but where payment and recruitment requires a lot more back and forth.

On the other hand, some of the challenges we encountered with this approach were:

• Sparsity of worker population. We couldn’t always find a worker, especially when posting the requests with short notice (less than 2 days before the event).
• Task cancellation takes up to 24 hours – not enough time to notify someone about this
• Variability in  availability. We found that for the same task one can get 14 bids one day and 3 bids the next day. It is a bit unpredictable.
• Time-consuming participants selection. Unless the task is set with a specific price, the tasks go through a bidding phase that is time consuming.
• Last minute cancellations are disruptive. Perhaps the only solution for this in the future might be to hire multiple workers for the same task as a backup, but this might increase the price significantly.
• Cost. Averaging around $35 for the presence of a worker at a physical event for one hour, this approach might be hard to scale. This, however, should be compared with the cost of producing an article elsewhere, for example, @whopayswriters reports that the cost of a Buzzfeed article is $100, and The Atlantic is $200.

We are currently working on automating this process to the extent possible as well as doing a demographic analysis of the TaskRabbit workforce. Stay tuned for a follow up.

NewsPad was created by our summer intern J. Nathan Matias and is soon to be released under an Apache license.

This entry was posted on Tuesday, November 12th, 2013 at 8:14 pm and is filed under civic computing, cooperation, social creativity.