I recently had the distinct honor of being on the opening plenary of the 2012 Skoll World Forum in Oxford. The panel, “Innovation in Times of Flux: Opportunities on the Heels of Crisis” was moderated by Judith Rodin, CEO of the Rockefeller Foundation. I’ve spent the past six years creating linkages between the humanitarian space and technology community, so the conversations we began during the panel prompted me to think more deeply about innovation in the humanitarian space. Clearly, humanitarian crises have catalyzed a number of important innovations in recent years. At the same time, however, these crises extend the cracks that ultimately reveal the inadequacies of existing humanita-rian organizations, particularly those resistant to change; and “any organization that is not changing is a battle-field monument” (While 1992).
These cracks, or gaps, are increasingly filled by disaster-affected communities themselves thanks in part to the rapid commercialization of communication technology. Question is: will the multi-billion dollar humanitarian industry change rapidly enough to avoid being left in the dustbin of history?
Crises often reveal that “existing routines are inadequate or even counter-productive [since] response will necessarily operate beyond the boundary of planned and resourced capabilities” (Leonard and Howitt 2007). More formally, “the ‘symmetry-breaking’ effects of disasters undermine linearly designed and centralized administrative activities” (Corbacioglu 2006). This may explain why “increasing attention is now paid to the capacity of disaster-affected communities to ‘bounce back’ or to recover with little or no external assistance following a disaster” (Manyena 2006).
Okay. You got me. I can’t really tell you everything you need to know about big data. The one thing I discovered last week – as I joined more than 2,500 data junkies from around the world for the O’Reilly Strata conference in rainy Santa Clara California—is that nobody can, not Google, not Intel, not even IBM. All I can guarantee you is that you’ll be hearing a lot more about it.
What is big data? Roughly defined, it refers to massive data sets that can be used to predict or model future events. That can include everything from the online purchase history of millions of Americans (to predict what they’re about to buy) to where people in San Francisco are most likely to jog (according to GPS) to Facebook posts and Twitter trends and 100 year storm records.
Phi Beta Iota: Big data is most important for what it can tell you about true cost and whole system cause and effect, inclusive of political corruption and organizational fraud. These are past and present issues, not future issues. We design the future based on the integrity present today. This is why “open everything” matters.
With that in mind, here’s the three most important things you need to know about big data right now:
A new program can find and compare relationships in complicated data without having to be asked specific queries
Are there subtle patterns lurking in data that can foretell of a coming financial-system crash? What can explain the variations in sports-star salaries? How about the complex relationship between genes and certain diseases? Scientists in various fields have been searching for better ways to analyze large piles of data for such patterns, but the difficulty has always been that they need to know what they’re looking for in order to find. A new software program, described in the latest issue of Science, is designed to find the patterns in data that scientists don’t know to look for.
David Reshef, one of the scientists behind MINE, as the program is called, explains, “Standard methods will see one pattern as signal and others as noise. There can potentially be a variety of different types of relationships in a given data set. What’s exciting about our method is that it looks for any type of clear structure within the data, attempting to find all of them. … This ability to search for patterns in an equitable way offers tremendous exploratory potential in terms of searching for patterns without having to know ahead of time what to search for.” MINE compares different possible relationships (including linear, exponential, and periodic) and returns those that are strongest.