The National Geospatial-Intelligence Agency (NGA) Disparate Data Challenge encourages participants to offer solutions that can demonstrate effective capabilities that enable access to data that is wildly disparate in its formats, schemas, interfaces and locations, so that it may be available for search, business metrics and data and information analytics.
Here is our submittal to NGA that was selected within Stage 1 of the competition.
This is the author’s preliminary draft of the second of three monographs focused on the future of the US Army as an expeditionary force in a complex world that is rapidly decentralizing while also facing major development challenges. This second monograph (the first presented a notional Grand Strategy for discussion) presents the holistic analytic model and the resulting strategic generalizations from the Marine Corps’ original study, Overview of Planning and Programming Factors for Expeditionary Operations in the Third World (Marine Corps Combat Development Command, March 1990). The model is neither complete nor current – it is a starting point for reflection. A new comprehensive model is needed that supports Grand Strategy not only across the D3 – Defense, Diplomacy, and Development – planning and programming domains, but across Whole of Government (WoG) as well, and ideally, also into the multinational and “eight tribe” conceptual space as well – future operations demand the full integration of both estimative intelligence and operational inclusion of all elements of society, not just government – military.
Flash floods in Beijing have killed over 70 people and forced the evacuation of more than 50,000 after destroying over 8,000 homes and causing $1.6 billion in damages. In total, some 1.5 million people have been affected by the floods after Beijing recorded the heaviest rainfall the city has seen in more than 60 years.
The heavy rains began on July 21. Within hours, users of the Guokr.com social network launched a campaign to create a live crisis map of the flood’s impact using Google Maps. According to TechPresident, “the result was not only more accurate than the government output—it was available almost a day earlier. According to People’s Daily Online, these crowd-sourced maps were widely circulated on Weibo [China’s version of Twitter] the Monday and Tuesday after the flooding.” The crowdsourced, citizen-generated flood map of Beijing is available here and looks like this:
I just had the pleasure of speaking with my new colleague Jakob Rogstadius from Madeira Interactive Technologies Institute (Madeira-TTI). Jakob is working on CrisisTracker, a very interesting platform designed to facilitate collaborative social media analysis for disaster response. The rationale for CrisisTracker is the same one behind Ushahidi’s SwiftRiver project and could be hugely helpful for crisis mapping projects carried out by the Standby Volunteer Task Force (SBTF).
One of the principal Research and Development (R&D) projects I’m spearheading with colleagues at the Qatar Computing Research Institute (QCRI) has been getting a great response from several key contacts at the UN’s Office for the Coordination of Humanitarian Affairs (OCHA). In fact, their input has been instrumental in laying the foundations for our early R&D efforts. I therefore highlighted the initiative during my recent talk at the UN’s ECOSOC panel in New York, which was moderated by OCHA Under-Secretary General Valerie Amos. The response there was also very positive. So what’s the idea? To develop the foundations for a Twitter Dashboard for the Humanitarian Cluster System.
So that’s what he’s been up to! My good friend and mentor Ken Banks of Kiwanja fame has just launched a very interesting initiative entitled “Means of Exchange“. Ken wants to democratize opportunities for radical economic self-sufficiency and thus render local communities more resilient to exogenous shocks. “I’ve been taking an increasing interest in economic resilience,” writes Ken, “particularly how technology might help buffer local communities from global economic down-turns. Ironically, since I started my research the world has entered a period of growing economic uncertainty. The causes–although fascinating–don’t so much interest me, more the response at local, grassroots level.”
To say that Ken’s ideas directly resonate with my ideals would be a huge understatement. My iRevolution blog is currently in its fifth year of production and as the About page explains, “This blog features short thought pieces on how innovation and technology are revolutionizing the power of the individual through radical self-sufficiency, self-determination, independence, survival and resilience.” I’m incredibly excited by Ken’s new initiative. He quotes this excellent comment by Calvin Coolidge, which really hits home:
“We pay too little attention to the reserve power of the people to take care of themselves. We are too solicitous for government intervention, on the theory, first, that the people themselves are helpless, and second, that the government has superior capacity for action. Often times both of these conclusions are wrong.”
I have written many a blog post on this very people-centered notion as applied to crisis early warning and humanitarian response. Hence my pitch two years ago for a Match.com applied to humanitarian relief. Take this blog post, for example: “The Crowd is Always There: A Marketplace for Crowdsourcing Crisis Response.” But Ken is not just advocating for a “Match.com for Economic Resilience,” he is also building the infrastructure to make it happen: “A number of apps to support this work are planned for rollout during the year, with the first due for release in summer 2012.”
My colleague Robert Kirkpatrick from Global Pulse has been actively promoting the concept of “data philanthropy” within the context of development. Data philanthropy involves companies sharing proprietary datasets for social good. I believe we urgently need big (social) data philanthropy for humanitarian response as well. Disaster-affected communities are increasingly the source of big data, which they generate and share via social media platforms like twitter. Processing this data manually, however, is very time consuming and resource intensive. Indeed, large numbers of digital humanitarian volunteers are often needed to monitor and process user-generated content from disaster-affected communities in near real-time.
Meanwhile, companies like Crimson Hexagon, Geofeedia, NetBase, Netvibes, RecordedFuture and Social Flow are defining the cutting edge of automated methods for media monitoring and analysis. So why not set up a Big Data Philanthropy group for humanitarian response in partnership with the Digital Humanitarian Network? Call it Corporate Social Responsibility (CRS) for digital humanitarian response. These companies would benefit from the publicity of supporting such positive and highly visible efforts. They would also receive expert feedback on their tools.
My colleague Jeannine Lemaire from the Core Team of the Standby Volunteer Task Force (SBTF) recently pointed me to Geofeedia, which may very well be the next generation in crisis mapping technology. So I spent over an hour talking with GeoFeedia’s CEO, Phil Harris, to learn more about the platform and discuss potential applications for humanitarian response. The short version: I’m impressed; not just with the technology itself and potential, but also by Phil’s deep intuition and genuine interest in building a platform that enables others to scale positive social impact.
Situational awareness is absolutely key to emergency response, hence the rise of crisis mapping. The challenge? Processing and geo-referencing Big Data from social media sources to produce live maps has largely been a manual (and arduous) task for many in the humanitarian space. In fact, a number of humanitarian colleagues I’ve spoken to recently have complained that the manual labor required to create (and maintain) live maps is precisely why they aren’t able to launch their own crisis maps. I know this is also true of several international media organizations.
There have been several attempts at creating automated live maps. Take Havaria and Global Incidents Map, for example. But neither of these provide the customi-zability necessary for users to apply the platforms in meaningful ways. Enter Geofeedia. Lets take the recent earthquake and 800 aftershocks in Emilia, Italy. Simply type in the place name (or an exact address) and hit enter. Geofeedia automatically parses Twitter, YouTube, Flickr, Picasa and Instagram for the latest updates in that area and populates the map with this content. The algorithm pulls in data that is already geo-tagged and designated as public.
The United Nations Committee of Experts on Global Information Management (GGIM) recently organized a meeting of thought-leaders and visionaries in the geo-spatial world to identify the future of this space over the next 5-10 years. These experts came up with some 80+ individual predictions. I’ve included some of the more interesting ones below.
The use of Unmanned Aerial Vehicles (UAVs) as a tool for rapid geospatial data collection will increase.
3D and even 4D geospatial information, incorporating time as the fourth dimension, will increase.
Technology will move faster than legal and governance structures.
The link between geospatial information and social media, plus other actor networks, will become more and more important.
Real-time info will enable more dynamic modeling & response to disasters.
Free and open source software will continue to grow as viable alternatives both in terms of software, and potentially in analysis and processing.
I recently had the distinct honor of being on the opening plenary of the 2012 Skoll World Forum in Oxford. The panel, “Innovation in Times of Flux: Opportunities on the Heels of Crisis” was moderated by Judith Rodin, CEO of the Rockefeller Foundation. I’ve spent the past six years creating linkages between the humanitarian space and technology community, so the conversations we began during the panel prompted me to think more deeply about innovation in the humanitarian space. Clearly, humanitarian crises have catalyzed a number of important innovations in recent years. At the same time, however, these crises extend the cracks that ultimately reveal the inadequacies of existing humanita-rian organizations, particularly those resistant to change; and “any organization that is not changing is a battle-field monument” (While 1992).
These cracks, or gaps, are increasingly filled by disaster-affected communities themselves thanks in part to the rapid commercialization of communication technology. Question is: will the multi-billion dollar humanitarian industry change rapidly enough to avoid being left in the dustbin of history?
Crises often reveal that “existing routines are inadequate or even counter-productive [since] response will necessarily operate beyond the boundary of planned and resourced capabilities” (Leonard and Howitt 2007). More formally, “the ‘symmetry-breaking’ effects of disasters undermine linearly designed and centralized administrative activities” (Corbacioglu 2006). This may explain why “increasing attention is now paid to the capacity of disaster-affected communities to ‘bounce back’ or to recover with little or no external assistance following a disaster” (Manyena 2006).
The Syria Tracker Crisis Map is without doubt one of the most impressive crisis mapping projects yet. Launched just a few weeks after the protests began one year ago, the crisis map is spearheaded by a just handful of US-based Syrian activists have meticulously and systematically documented 1,529 reports of human rights violations including a total of 11,147 killings. As recently reported in this NewScientist article, “Mapping the Human Cost of Syria’s Uprising,” the crisis map “could be the most accurate estimate yet of the death toll in Syria’s uprising […].” Their approach? “A combination of automated data mining and crowdsourced human intelligence,” which “could provide a powerful means to assess the human cost of wars and disasters.”
On the data-mining side, Syria Tracker has repurposed the HealthMap platform, which mines thousands of online sources for the purposes of disease detection and then maps the results, “giving public-health officials an easy way to monitor local disease conditions.” The customized version of this platform for Syria Tracker (ST), known as HealthMap Crisis, mines English information sources for evidence of human rights violations, such as killings, torture and detainment. As the ST Team notes, their data mining platform “draws from a broad range of sources to reduce reporting biases.” Between June 2011 and January 2012, for example, the platform collected over 43,o00 news articles and blog posts from almost 2,000 English-based sources from around the world (including some pro-regime sources).
Syria Tracker combines the results of this sophisticated data mining approach with crowdsourced human intelligence, i.e., field-based eye-witness reports shared via webform, email, Twitter, Facebook, YouTube and voicemail. This naturally presents several important security issues, which explains why the main ST website includes an instructions page detailing security precautions that need to be taken while sub-mitting reports from within Syria. They also link to this practical guide on how to protect your identity and security online and when using mobile phones. The guide is available in both English and Arabic.
Eye-witness reports are subsequently translated, geo-referenced, coded and verified by a group of volunteers who triangulate the information with other sources such as those provided by the HealthMap Crisis platform. They also filter the reports and remove dupli-cates. Reports that have a low con-fidence level vis-a-vis veracity are also removed. Volunteers use a dig-up or vote-up/vote-down feature to “score” the veracity of eye-witness reports. Using this approach, the ST Team and their volunteers have been able to verify almost 90% of the documented killings mapped on their platform thanks to video and/or photographic evidence. They have also been able to associate specific names to about 88% of those reported killed by Syrian forces since the uprising began.
Depending on the levels of violence in Syria, the turn-around time for a report to be mapped on Syria Tracker is between 1-3 days. The team also produces weekly situation reports based on the data they’ve collected along with detailed graphical analysis. KML files that can be uploaded and viewed using Google Earth are also made available on a regular basis. These provide “a more precisely geo-located tally of deaths per location.”
In sum, Syria Tracker is very much breaking new ground vis-a-vis crisis mapping. They’re combining automated data mining technology with crowdsourced eye-witness reports from Syria. In addition, they’ve been doing this for a year, which makes the project the longest running crisis maps I’ve seen in a hostile environ-ment. Moreover, they’ve been able to sustain these import efforts with just a small team of volunteers. As for the veracity of the collected information, I know of no other public effort that has taken such a meticulous and rigorous approach to documenting the killings in Syria in near real-time. On February 24th, Al-Jazeera posted the following estimates:
Syrian Revolution Coordination Union: 9,073 deaths
Local Coordination Committees: 8,551 deaths
Syrian Observatory for Human Rights: 5,581 deaths
At the time, Syria Tracker had a total of 7,901 documented killings associated with specific names, dates and locations. While some duplicate reports may remain, the team argues that “missing records are a much bigger source of error.” Indeed, They believe that “the higher estimates are more likely, even if one chooses to disregard those reports that came in on some of the most violent days where names were not always recorded.”
The Syria Crisis Map itself has been viewed by visitors from 136 countries around the world and 2,018 cities—with the top 3 cities being Damascus, Washington DC and, interestingly, Riyadh, Saudia Arabia. The witnessing has thus been truly global and collective. When the Syrian regime falls, “the data may help sub-sequent governments hold him and other senior leaders to account,” writes the New Scientist. This was one of the principle motivations behind the launch of the Ushahidi platform in Kenya over four years ago. Syria Tracker is powered by Ushahidi’s cloud-based platform, Crowdmap. Finally, we know for a fact that the International Criminal Court (ICC) and Amnesty International (AI) closely followed the Libya Crisis Map last year.
I recently gave a guest lecture at the University of Texas, Austin, and finally had the opportunity to catch up with my colleague Josh Busby who has been working on a promising crisis mapping project as part of the university’s Climate Change and African Political Stability Program (CCAPS).
Josh and team just released the pilot version of its dynamic mapping tool, which aims to provide the most comprehensive view yet of climate change and security in Africa. The platform, developed in partnership with AidData, enables users to “visualize data on climate change vulnerability, conflict, and aid, and to analyze how these issues intersect in Africa.” The tool is powered by ESRI technology and allows researchers as well as policymakers to “select and layer any combination of CCAPS data onto one map to assess how myriad climate change impacts and responses intersect. For example, mapping conflict data over climate vulnera-bility data can assess how local conflict patterns could exacerbate climate-induced insecurity in a region. It also shows how conflict dynamics are changing over time and space.”
The platform provides hyper-local data on climate change and aid-funded interventions, which can provide important insights on how development assistance might (or might not) be reducing vulnerability. For example, aid projects funded by 27 donors in Malawi (i.e., aid flows) can be layered on top of the climate change vulnerability data to “discern whether adaptation aid is effectively targeting the regions where climate change poses the most significant risk to the sustainable development and political stability of a country.”