About

CrowdFlower cleans up messy and incomplete data using an online workforce. Typical users of CrowdFlower are data scientists who use the software to create training data to build models and train machine learning algorithms.

Once data is uploaded, the system automatically allocates the work to contributors and tests them against known answers hidden within the task (what CrowdFlower refers to as a “job” ^[5]). The way in which contributors perform on these hidden test questions calibrates how much the system trusts them on an individual level. As long as contributors remain trusted they’re allowed to continue working on a given job. If they become untrusted, they’re removed from the job and all of their work is disregarded. Multiple contributor judgments are collected and an aggregate answer with an associated confidence score (agreement of the contributors weighted by the trust of each contributor) is provided as a result – effectively returning the “most trusted judgment,” for a given unit of data.

• Researchers at the Harvard Tuberculosis Lab used it to identify drug-resistant TB cells.^[6]
• After the 2010 Haiti earthquake, the company helped to route text messages to the proper aid workers, to get them translated, and to ensure that the people sending the texts had a chance of getting what they needed.^[7]
• Similar relief efforts were handled after the 2010 Pakistan floods.^[8]
• In 2009, the company worked with Samasource to provide work for refugees in Kenya who completed microtasks; iPhone users donated their time by checking for accuracy through Give Work, an app.^[9]