I recently had the distinct honor of being on the opening plenary of the 2012 Skoll World Forum in Oxford. The panel, “Innovation in Times of Flux: Opportunities on the Heels of Crisis” was moderated by Judith Rodin, CEO of the Rockefeller Foundation. I’ve spent the past six years creating linkages between the humanitarian space and technology community, so the conversations we began during the panel prompted me to think more deeply about innovation in the humanitarian space. Clearly, humanitarian crises have catalyzed a number of important innovations in recent years. At the same time, however, these crises extend the cracks that ultimately reveal the inadequacies of existing humanita-rian organizations, particularly those resistant to change; and “any organization that is not changing is a battle-field monument” (While 1992).
These cracks, or gaps, are increasingly filled by disaster-affected communities themselves thanks in part to the rapid commercialization of communication technology. Question is: will the multi-billion dollar humanitarian industry change rapidly enough to avoid being left in the dustbin of history?
Crises often reveal that “existing routines are inadequate or even counter-productive [since] response will necessarily operate beyond the boundary of planned and resourced capabilities” (Leonard and Howitt 2007). More formally, “the ‘symmetry-breaking’ effects of disasters undermine linearly designed and centralized administrative activities” (Corbacioglu 2006). This may explain why “increasing attention is now paid to the capacity of disaster-affected communities to ‘bounce back’ or to recover with little or no external assistance following a disaster” (Manyena 2006).
Okay. You got me. I can’t really tell you everything you need to know about big data. The one thing I discovered last week – as I joined more than 2,500 data junkies from around the world for the O’Reilly Strata conference in rainy Santa Clara California—is that nobody can, not Google, not Intel, not even IBM. All I can guarantee you is that you’ll be hearing a lot more about it.
What is big data? Roughly defined, it refers to massive data sets that can be used to predict or model future events. That can include everything from the online purchase history of millions of Americans (to predict what they’re about to buy) to where people in San Francisco are most likely to jog (according to GPS) to Facebook posts and Twitter trends and 100 year storm records.
Phi Beta Iota: Big data is most important for what it can tell you about true cost and whole system cause and effect, inclusive of political corruption and organizational fraud. These are past and present issues, not future issues. We design the future based on the integrity present today. This is why “open everything” matters.
With that in mind, here’s the three most important things you need to know about big data right now:
A new program can find and compare relationships in complicated data without having to be asked specific queries
Are there subtle patterns lurking in data that can foretell of a coming financial-system crash? What can explain the variations in sports-star salaries? How about the complex relationship between genes and certain diseases? Scientists in various fields have been searching for better ways to analyze large piles of data for such patterns, but the difficulty has always been that they need to know what they’re looking for in order to find. A new software program, described in the latest issue of Science, is designed to find the patterns in data that scientists don’t know to look for.
David Reshef, one of the scientists behind MINE, as the program is called, explains, “Standard methods will see one pattern as signal and others as noise. There can potentially be a variety of different types of relationships in a given data set. What’s exciting about our method is that it looks for any type of clear structure within the data, attempting to find all of them. … This ability to search for patterns in an equitable way offers tremendous exploratory potential in terms of searching for patterns without having to know ahead of time what to search for.” MINE compares different possible relationships (including linear, exponential, and periodic) and returns those that are strongest.
How does public transportation affect education? What impact does population density have on public health? Is there a connection between CO2 levels and obesity?
Officials in the City of Portland, Ore., have collaborated with IBM to find answers to those and other questions, developing an interactive model that connects the relationships between the city’s core systems that handle the economy, housing, education, public safety, transportation and health care.
Is there a new anti-intellectualism? I mean one that is advocated by Internet geeks and some of the digerati. I think so: more and more mavens of the Internet are coming out firmly against academic knowledge in all its forms. This might sound outrageous to say, but it is sadly true.
Robert David STEELE Vivas: Digerati are not geeks. They are adept at social media, a process, rather than the substance of any discipline. Their scorn for the mandarins of knowledge would not be possible if academia had not lost its soul, sanctioned massive intellectual corruption, and fragmented itself to the point of irrelevance. The serious educational literature (not something the digerati read) is clear: inspiration and innovation
emerge faster, better, and cheaper from minds that are prepared, to include a foundation of memorization and a deep familiarity with the thinking of those who have come before. The digerati point of view half-right and is embodied in Smart Mobs, Wisdom of the Crowd, Everything is Miscellaneous, and Maria Popova’s latest thought, that “information curation is the new authorship.” The digerati approach splits the roles of originator of an idea and connector of an idea down, and assumes that “the collective” can replicate and even surpass the individual human brain, without recognizing that the whole is only as good as the sum of the part foundation plus whatever the collective adds. My own finding re Wikipedia is that the mob destroys intellectuals. My own efforts to enhance the Open Source Intelligence page there were destroyed by idiots that “assumed” that because I pointed to oss.net so much (to many of the 800 people whose work is there including the 144 that received Golden Candle Awards) I was “self-promoting.” The digerati are fragile and very shallow, and by Larry Sanger’s very interesting account, a new form of neo-Luddite. The academy is corrupt and fragmented–we are in an era where all forms of organization have lost their soul and whatever semblance of philosophical context they may once have possessed. We are suffering from the Paradigms of Failure that I discussed in the pre-amble to ELECTION 2008: Lipstick on the Pig (EIN, 2008). There is only one option leading to stabilization & reconstruction: INTEGRITY. The digerati aren’t–as a general rule–very appreciative of holistic thinking or in-depth expertise–they are a spoiled generation badly in need of some personal suffering and exposure to global reality–IMHO.