{"id":604,"date":"2017-07-27T02:33:54","date_gmt":"2017-07-27T02:33:54","guid":{"rendered":"http:\/\/uscictdialdev.wpenginepowered.com\/?page_id=604"},"modified":"2017-07-27T02:33:54","modified_gmt":"2017-07-27T02:33:54","slug":"crowdflower","status":"publish","type":"page","link":"https:\/\/dialport.ict.usc.edu\/index.php\/resources\/collecting-annotationsdata\/crowdflower\/","title":{"rendered":"Crowdflower"},"content":{"rendered":"<p><a href=\"https:\/\/www.crowdflower.com\/\">About<\/a><\/p>\n<p>&nbsp;<\/p>\n<p>CrowdFlower cleans up messy and incomplete data using an online workforce. Typical users of CrowdFlower are\u00a0<a class=\"mw-redirect\" title=\"Data scientist\" href=\"https:\/\/en.wikipedia.org\/wiki\/Data_scientist\">data scientists<\/a>\u00a0who use the software to create training data to build\u00a0<a title=\"Predictive modelling\" href=\"https:\/\/en.wikipedia.org\/wiki\/Predictive_modelling\">models<\/a>\u00a0and train\u00a0<a title=\"Machine learning\" href=\"https:\/\/en.wikipedia.org\/wiki\/Machine_learning\">machine learning<\/a>\u00a0<a title=\"Algorithm\" href=\"https:\/\/en.wikipedia.org\/wiki\/Algorithm\">algorithms<\/a>.<\/p>\n<p>Once data is uploaded, the system automatically allocates the work to contributors and tests them against known answers hidden within the task (what CrowdFlower refers to as a &#8220;job&#8221;\u00a0<sup id=\"cite_ref-5\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/CrowdFlower#cite_note-5\">[5]<\/a><\/sup>). The way in which contributors perform on these hidden test questions calibrates how much the system trusts them on an individual level. As long as contributors remain trusted they&#8217;re allowed to continue working on a given job. If they become untrusted, they&#8217;re removed from the job and all of their work is disregarded. Multiple contributor judgments are collected and an aggregate answer with an associated confidence score (agreement of the contributors weighted by the trust of each contributor) is provided as a result &#8211; effectively returning the &#8220;most trusted judgment,&#8221; for a given unit of data.<\/p>\n<ul>\n<li>Researchers at the\u00a0<a title=\"Harvard University\" href=\"https:\/\/en.wikipedia.org\/wiki\/Harvard_University\">Harvard<\/a>\u00a0<a title=\"Tuberculosis\" href=\"https:\/\/en.wikipedia.org\/wiki\/Tuberculosis\">Tuberculosis<\/a>\u00a0Lab used it to identify\u00a0<a title=\"Drug resistance\" href=\"https:\/\/en.wikipedia.org\/wiki\/Drug_resistance\">drug-resistant<\/a>\u00a0TB cells.<sup id=\"cite_ref-6\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/CrowdFlower#cite_note-6\">[6]<\/a><\/sup><\/li>\n<li>After the\u00a0<a title=\"2010 Haiti earthquake\" href=\"https:\/\/en.wikipedia.org\/wiki\/2010_Haiti_earthquake\">2010 Haiti earthquake<\/a>, the company helped to route text messages to the proper aid workers, to get them translated, and to ensure that the people sending the texts had a chance of getting what they needed.<sup id=\"cite_ref-7\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/CrowdFlower#cite_note-7\">[7]<\/a><\/sup><\/li>\n<li>Similar relief efforts were handled after the\u00a0<a title=\"2010 Pakistan floods\" href=\"https:\/\/en.wikipedia.org\/wiki\/2010_Pakistan_floods\">2010 Pakistan floods<\/a>.<sup id=\"cite_ref-8\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/CrowdFlower#cite_note-8\">[8]<\/a><\/sup><\/li>\n<li>In 2009, the company worked with\u00a0<a title=\"Samasource\" href=\"https:\/\/en.wikipedia.org\/wiki\/Samasource\">Samasource<\/a>\u00a0to provide work for\u00a0<a title=\"Refugee\" href=\"https:\/\/en.wikipedia.org\/wiki\/Refugee\">refugees<\/a>\u00a0in\u00a0<a title=\"Kenya\" href=\"https:\/\/en.wikipedia.org\/wiki\/Kenya\">Kenya<\/a>\u00a0who completed microtasks;\u00a0<a title=\"IPhone\" href=\"https:\/\/en.wikipedia.org\/wiki\/IPhone\">iPhone<\/a>\u00a0users donated their time by checking for accuracy through Give Work, an app.<sup id=\"cite_ref-9\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/CrowdFlower#cite_note-9\">[9]<\/a><\/sup><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>About &nbsp; CrowdFlower cleans up messy and incomplete data using an online workforce. Typical users of CrowdFlower are\u00a0data scientists\u00a0who use the software to create training data to build\u00a0models\u00a0and train\u00a0machine learning\u00a0algorithms. Once data is uploaded, the system automatically allocates the work to contributors and tests them against known answers hidden within the task (what CrowdFlower refers [&hellip;]<\/p>\n","protected":false},"author":18,"featured_media":0,"parent":599,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-604","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/pages\/604","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/comments?post=604"}],"version-history":[{"count":0,"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/pages\/604\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/pages\/599"}],"wp:attachment":[{"href":"https:\/\/dialport.ict.usc.edu\/index.php\/wp-json\/wp\/v2\/media?parent=604"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}