How to Perfect Real-Time Crowdsourcing
The new techniques behind instant crowdsourcing makes human intelligence available on demand for the first time.
One of the great goals of computer science is to embed human-like intelligence in common applications like image processing, robotic control and so on. Until recently the focus has been to develop an artificial intelligence that can do these jobs.
But there's another option: using real humans via some kind of crowdsourcing process. One well known example involves the CAPTCHA test which can identify humans from machines by asking them to identify words so badly distorted that automated systems cannot read them.
However, spammers are known to farm out these tasks to humans via crowdsourcing systems that pay in the region of 0.5 cents per 1000 words solved.
Might not a similar process work for legitimate tasks such as building human intelligence into real world applications?
The problem, of course, is latency. Nobody wants to sit around for 20 minutes while a worker with the skills to steer your robotic waiter is crowdsourced from the other side of the world.
So how quickly can a crowd be put into action.?That's the question tackled today by Michael Bernstein at the Massachusetts Institute of Technology in Cambridge and a few pals.
In the past, these guys have found ways to bring a crowd to bear in about two seconds. That's quick. But the reaction time is limited to how quickly a worker responds to an alert.
Now these guys say they've find a way to reduce the reaction time to 500 milliseconds–that's effectively realtime. A system with a half second latency could turn crowdsourcing into a very different kind of resource.
The idea that Bernstein and co have come up with is straightforward. These guys simply “precruit” a crowd and keep them on standby until a task becomes available. Effectively, they're paying workers a retainer so that they are available immediately when needed
The difficulty is in the messy details of precruitment. How many workers do you need to keep on retainer, how do cope with drop outs and how do you keep people interested so that they are available to work at a fraction of a second's notice?
Bernstein and co have used an idea called queuing theory to work out how to optimise the process of precruitment according to how often the task comes up, how long it takes and so on.
They've also developed an interesting psychological trick to keep workers ready for action. When workers are precruited, a screen opens up on their computer which downloads the task. The download occurs extremely quickly but if no task is to hand, the screen shows a “loading” bar.
It turns out that the loading bar keeps workers focused on the forthcoming task for up to ten seconds, at which point their attention begins to wander. At that point, if no task materialises, the worker can be paid off.
Bernstein and co have even tested how well this works using a whack-a-mole type task which appears on workers screens after a randomly chosen period between 0 and 20 seconds. They recruited 50 workers to carry out 373 whacks and found the median length of time between the mole's appearance and the worker moving the mouse toward the mole to click on it was 0.50 seconds.
“Our model suggests…that crowds could be recruited effectively instantaneously,” they say.
That could change the nature of crowdsourcing. Bernstein suggest that real time crowdsourcing could be used to point cameras, control robots and produce instant opinion polls.
But first crowdsourcers will have to change the way they do business. Bernstein and co suggest that they could build a retainer into their system design so that they have a pool of ready-to-go workers available at any instant.
Workers could even be assessed at how good they are for this kind of task allowing them to build up a reputation for the work. Bernstein and co suggest two new reputation statistics–the percentage of time workers respond to a precruitment request and how quickly they respond on these occasions.
Shouldn't be too hard to set up. An interesting new business model for the likes of Mechanical Turk and others or perhaps for an enterprising new start up.
Reference: Analytic Methods for Optimising Real time Crowdsourcing
$250 Device Global Maps & Text Messages
$169 SPOT Connect: Any Cell Phone SatComm Capable
2012 PREPRINT FOR COMMENT: The Craft of Intelligence
2012 Integrity, Reflexivity, & Open Everything
2012 THE OPEN SOURCE EVERYTHING MANIFESTO: Transparency, Truth & Trust
2010 INTELLIGENCE FOR EARTH: Clarity, Diversity, Integrity, & Sustainability
2010 The Ultimate Hack Re-Inventing Intelligence to Re-Engineer Earth (Chapter for Counter-Terrorism Book Out of Denmark)
2010: Human Intelligence (HUMINT) Trilogy Updated
Crowdsourcing Mapped Water Data via SMS/txt msg
Future of Multinational Intelligence & Operations
Graphic: Twitter as an Intelligence Tool
Howard Rheingold: Crap Detection & Critical Thinking
Howard Rheingold: Infotention Skills + Citizen Intel RECAP
Howard Rheingold: News Filters for the Future – Technical Services or Human Networks?
Howard Rheingold: Open Source Intelligence Meets Real-Time News and Data Curation – SwiftRiver
Journal: DARPA & MIT Discover “Share the Wealth”
Journal: Haiti Public Intelligence Emergent
Journal: IC on Twitter, Still Not Making Sense
Journal: Neither Twitter nor Facebook Lead Revolution
Journal: Twitter Aggregation Way Cool
Journal: Twitter Breaks the News
Koko: Crowd-Sourcing Weather Forecasting
Letter to Secretary General of the United Nations
Letter to Sir Richard Branson “The Virgin Truth”
Letter to Microsoft CTO Operation Cloudburst
M4IS2 (Multinational, Multiagency, Multidisciplinary, Multidomain Information-Sharing and Sense-Making)
Letter to President Obama ON INTELLIGENCE
Mini-Me: When All Else Fails, Try Crowd-Sourcing
Patrick Meier: Crisis Mapping Syria – Automated Data Mining and Crowdsourced Human Intelligence
Peter Meier: Crowdsouring Syrian Imagery & Crisis Reports
Patrick Meier: Filtering Tweets Real-Time for Crisis Response
Patrick Meier: How to Crowdsource Better Governance in Authoritarian States
Patrick Meier: Mobile Technologies, Crisis Mapping, & Disaster Response
Patrick Meier: The Digital Operations Center of the American Red Cross
Patrick Meier: Trails of Digital Trustworthness
Patrick Meier: Ushahidi Emergent as Democracy in Being
Paul Fernhout: Open Letter to the Intelligence Advanced Programs Research Agency (IARPA)
Reference: Lee Felsenstein & Dave Warner Converse
Reference: Strategic Analytic Model for Creating a Prosperous World at Peace
Review: A Democratic Approach to Sustainable Futures — A Workbook for Addressing the Global Problematique
Review: The World Is Open–How Web Technology Is Revolutionizing Education
Review: The World Sensorium — The Social Embryology of World Federation 1946
Search: global brain human brain + RECAP
Search: holistic cybernetic integral
Steele on World Brain to Amazon Developers Conference (VIDEO 2007)
Steele on Open Everything to Gnomedex (VIDEO 2007)
Steele on Open Source Future at Echo Chamber (AUDIO 2006)
Tom Atlee: New Developments in Co-Intelligence Work
Whitehouse Needing Online and Mobile Crowdsourcing & Collective Intelligence for Decision-Support
World Brain & Global Game 101-104
Worth a Look: Momentum Crowdsourcing for Open Source Intelligence
Yoda: Big Data Tough Love, Everyone Fails