Out of Band: Robert David Steele on OSINT
Hal Berghel, University of Nevada, Las Vegas sent questions, here are my answers.
Hal Burghel: Tell us about open source generally and open source intelligence specifically.
Robert David Steele: It’s vital to remember that the whole point of connectivity is to move and make sense of content, securely, in service to humans. For this readership, let’s jump right to the open source ecology I’ve devised [see Figure 1], and make the point that open source is the only form of engineering that’s affordable, interoperable, scalable, and therefore sustainable. At a higher level of thinking, our task is to reinvent intelligence (decision support) so as to reengineer Earth. My briefing to engineers at the University of British Columbia [in 2009] became a book chapter after a second delivery of the briefing in 2010.1
OSINT is the art and science of creating ethical, evidence-based decision support using only open sources and methods—legal and ethical in every respect. From an engineering point of view it requires access to digital sources (surface Web 5 percent, deep Web 15 percent), analog sources (no more than 30 percent), and human sources (most of whom must be face to face, 50 percent). OSINT requires processing of two types: big data at petabyte toward exoscale magnitudes; and desktop analytic toolkits able to detect patterns and anomalies in smaller datasets. Beyond that, OSINT is all about humans—analysts who can think, and deciders who can listen.
I started the modern OSINT initiative in 1988, after five years as a Marine Corps infantry officer and nine years as a spy for the Central Intelligence Agency. Asked to return to the Marine Corps as a civilian, I resigned from the CIA and spent our first $20 million on a top secret/sensitive compartmented information (TS/SCI) computer system with our own top-secret communications system. In one corner I put a PC with access to the Internet, at the time consisting largely of “The Source,” an early “one-stop” offering for accessing various streams of information. Within two weeks, the analysts started lining up for the PC. When I asked why, they said “there’s nothing in the secret databases about Burundi, Haiti, Somalia, or other places where the Marine Corps is likely to go.”
Hal Berghel: You’ve said that 80 percent of the mission-critical information that the intelligence community needs is publicly available.
Steele: I’m not the first to say this. Two former Directors of Central Intelligence, Allen Dulles and William Colby, have said the same thing. This has two engineering implications: first, that 80 percent of the information is outside the secret world and needs external connectivity; and second, that the 80 percent, if ever accessed, can’t be controlled and doesn’t need to be “protected” by very expensive secret hardware and software.
The reality is that the secret world produces “at best”—in the words of Marine Corps General Tony Zinni—4 percent of what the top national security leaders need to know, and I’ve written about this in CounterPunch.2 Agriculture, energy, health and human services, veterans affairs—everyone—needs decision support for strategy, policy, procurement, and day-to-day operations, but this is not what the secret world does.
What we have today is a very secret intelligence community that spends 70 percent of its money on contractors and technical collection that’s not processed, and less than 1 percent on accessing and exploiting the 80 percent of the information that’s publicly available in 183 languages we don’t speak. The flip side of this pathology is that the secret world doesn’t provide decision support to most of the cabinet departments, any of the assistant secretaries or office directors, or any of the front-line action officers across all the mission areas far removed from “national security.”
Berghel: You speak of the inherent intelligence in the “seven intelligence tribes”3 and explain how collective tribal information can be used in modern intelligence gathering.
Steele: Bill Gates once told a US president that the US government is 3 percent of his client base and therefore only gets 3 percent of his time. I mention this to emphasize that the government is the least intelligent, least informed “tribe” that processes information. The
other tribes—I split one in two to now total eight, are academia, civil society, commerce, law enforcement, media, the military, and nongovernmental/nonprofit organizations. On any given issue, at least 80 percent of the information the government needs to access if
it wants to be serious, is known by or controlled by someone who in general is not responsive to the government and more often than not does not speak or write in English.
Crowdsourcing and the wealth of networks are terms that are in vogue. What the government generally, and the secret world particularly, refuse to acknowledge is that information is a team sport and nature bats last. The government is only as good as its ability to do outreach, and if it relies on lies, nature—reality—will always reveal the truth at some future date.
In the 21st century, national intelligence is about public intelligence, not secret intelligence. We need engineers who can build open source collection and processing tools that anyone can use. We need to close down the mass not-so-secret surveillance capability that produces few tangible benefits for the public at large and instead invest in open cloud, open data, open spectrum—open source everything. We still need a few spies and secrets, but only a tiny fraction of what we have now.
Berghel: Former CIA and National Intelligence Council officer Paul Pillar remarks in his latest book4 that “Policy has shaped intelligence more than vice versa,” and this has led to significant corruption of intelligence.
Steele: I gave Paul’s book a rave review. What he’s really saying is that policy doesn’t care what intelligence thinks, for two reasons. First, the secret intelligence world generally does not have much to offer any serious decision. Second, between the lack of integrity of political appointees (such as CIA directors) who misrepresent what the CIA does know (for example, no weapons of mass destruction in Iraq), and elected politicians … the intelligence community is more a patsy than a player.
The title of my next book is Intelligence with Integrity: Enabling Hybrid Governance with Open Source Decision Support.5 As I briefed an energetic young group at Yale University in February 2014, we do not lack information or (public) intelligence—we lack integrity across the board.
Berghel: You have argued for your own brand of pareto optimality for the intelligence community: 80 percent of the most important information is free but ignored, and 95 percent of the budget is spent on the 20 percent that’s of less value.
Steele: US Army Major General Bob Scales (retired) is on record as saying that the infantry is 4 percent of the US military force, takes 80 percent of the casualties, and receives 1 percent of the budget. The same is true in the intelligence world, with suicide, alcoholism, divorce, and adultery being the casualties, and the spies and analysts being the force. Secret technical collection takes most of the money because it is the most expensive part of the process. We don’t process what we collect, really, the National Security Agency processes no more than 5 percent against its top targets and more typically 1 percent—because we aren’t held accountable for eating what we kill.
I can’t overstate what William Binney, the NSA engineer and whistleblower, has said. The NSA and the secret world are in the business of keeping the problem alive and keeping the money moving. To this I would add that they are not in the business of informing every mission area across the government. Even the president receives mediocre intelligence, and on the counterintelligence side, we have no bench. We have religious, ideological, and financial traitors across the board, and no one challenges them. The only optimality in the secret world is fraud, waste, and abuse.
Berghel: You believe that intelligence should start with what we can learn from open sources and then move to the hidden, rather than the other way around. But what accounts for the intelligence community’s ardent commitment to stealth?
Steele: Secrecy is a cult. When badly managed—and it’s very badly managed today and has been for a quarter century—secrecy is a means of avoiding accountability. I testified to Senator Daniel Moynihan’s Secrecy Commission in 1996, and among the points I made was
that secrecy is used to enable lies to Congress and the White House—80 percent of what the CIA claims is clandestine intelligence is actually provided by foreign intelligence services as a handout, or collected in legal traveler debriefings here in the USA. Ninety-nine percent of what the NSA collects isn’t processed. The National Reconnaissance Office is incapable of delivering the bandwidth, and the National Geospatial-Intelligence Agency has no clue why that matters.
Here’s an operational example. I did the first Somali piracy analysis for the US Central Command in 2005. We had commercial imagery, and had identified the specific boats, docks, families, and villages. We provided this information to the US Special Operations Command and to the US Navy’s Irregular Warfare specialists. In 2008 I had a chance to ask them both why they did nothing, and the similarity of their answers stunned me. Almost to the word: “It wasn’t an expensive enough problem.”
I cannot help but remember the [1974] book Zen and the Art of Motorcycle Maintenance, where the engineer used a piece of a beer can as a shim. For me that is a brilliant piece of engineering. For the secret world, only a million-dollar custom-made shim will do, and they won’t notice if the beltway bandit sells them a piece of a beer can claiming it is the custom shim. I cannot overstate the ignorance and inattentiveness of today’s contracting officers and contracting officer technical representatives in the secret world.
Berghel: Where did the government go wrong with the Open Source Center?
Steele: I tried very hard, along with several others, to create a separate open source program that would not be controlled by one of the secret agencies. The fact is that the secret world lacks both the intelligence and the integrity to make the most of open sources.
In the case of the Open Source Center [OSC], let me start with specific areas where its longest serving director Doug Naquin and I agree—this quote is from my recent letter to the International Journal of Intelligence and Counterintelligence, responding to a published interview with Naquin, by Hamilton Bean:
“Most startling to me were the areas of agreement between Naquin and myself. We agree on the intelligence community not altering its thinking. We agree that the IC leadership (both strategic and operational) is incapable of making serious tradeoffs from unprocessed secret collection to fully developed OSINT. We agree that the secret disciplines are stovepiped and out of touch with holistic decision support needs for Whole of Government PPBES [Planning, Programming, Budgeting, and Execution System]. We agree that OSINT tradecraft exists and matters very much. We agree that a more distributed business model for OSINT is essential for supporting national security, to which I would add national competitiveness.”6
Now here are the three specific areas where the OSC has failed badly for lack of integrity at the highest levels of the US secret world—which is to say, despite their good intentions, they have been designed to fail.
The OSC is not allowed to talk to subject-matter experts on anything. They can only do translations of published information. This is by dictate of the clandestine service, which claims to be in charge of human intelligence but is incompetent at all 15 slices of human intelligence as I have defined them in a seminal monograph for the US Army.7
The OSC has failed—as I also failed during my time as a for-profit CEO—to capitalize on the urgent need for unclassified decision-support across every mission area in the US government and within all of the oversight committees of Congress.
The OSC has been blocked from substantive collaboration with other governments to create regional information-sharing and sensemaking partnerships, and also with other tribes to create collaborative consortiums where the participants don’t need to have security clearances in order to contribute, so that the OSC could “harness the distributed intelligence of the whole nation,” a phrase of mine used by Al Gore in 1994. All three of these would have been excellent engineering challenges if properly managed within a commitment to go all in on open source.
I recently communicated all this to Vice President Joe Biden (but the IC may have intercepted and destroyed the communication); as a service to your readers I am making that package available online at www.phibetaiota.net/2014/05/2014-robert-steele-open-letter-to-vice-president-joe-biden.
Berghel: A neoconservative mantra for the past 40 years has been that the CIA has a lousy track record on assessing global threats to US interests. Examples frequently include the failure to anticipate the fall of the Shah of Iran in 1979 and the collapse of the Soviet Union in 1991. However, it appears to me that the CIA has been pretty accurate overall. The 2002 CIA report “Principle Challenges in Post-Saddam Iraq,”8 for example, predicted the following:
• achieving a stable government would be “long, difficult and probably turbulent,”
• indigenous factions would engage in violent conflict,
• reconstruction would require enormous outside assistance and
• a destabilized Iraq would provide a fertile home for Al-Qaeda and other terrorist groups.
From a casual observer’s point of view it looks as if they pretty much nailed it.
Steele: One report isn’t very impressive. In fact it’s vital to distinguish between what selected individuals with high integrity within the CIA know from what selected individuals with low integrity misrepresent the CIA as knowing, with what the CIA actually publishes. The CIA also had it right, at the individual level, on Vietnam—the clandestine service knew that Ho Chi Minh was a nationalist, not a communist, and [CIA analyst] Sam Adams nailed the corrupt US military process with the immortal statement, “Here we are in a guerilla war and we are not allowed to count the guerillas.” The most recent disappointment from the CIA is the “Global Trends 2030: Alternative Worlds” report in which they endorse fracking as a solution to our energy shortfalls. … there is no energy shortfall in the US—the sought-after excess is for export, to keep legacy refineries alive.
My last two books are particularly relevant to this question. The first, Intelligence for Earth: Clarity, Diversity, Integrity, & Sustainability,9 emphasizes the need for a holistic analytic model that integrates true-cost economics. The secret world doesn’t do either. The second book
goes far beyond OSINT. The Open Source Everything Manifesto: Transparency, Truth, & Trust10 explores why we need an ecology of open source everything—open cloud, open data, open hardware, open software, open spectrum, open standards, and more—or each individual “open” will fail alone. This book was actually written for the IT world, but because it was published by a New Age publisher, it wasn’t made known to the engineers. I consider it an ethical and technical manifesto.
Berghel: How will the US extricate itself from this orgy of delusion? Everything the intelligence agencies do seem to be miscalculated for effect.
Steele: I was reading the internal New York Times study on innovation this Saturday morning [17 May 2014], recommended to me by my virtual CTO Stephen E. Arnold of ArnoldIT. On page 16 it lists the following “Hallmarks of Disruptive Innovators:”11
• introduced by an “outsider,”
• less expensive than existing products,
• targeting underserved or new markets,
• initially inferior to existing products, and
• advanced by an enabling technology.
For a quarter century I’ve focused on trying to help governments reform intelligence, and I now realize that Buckminster Fuller (don’t reform, displace) and Russell Ackoff (stop doing the wrong things righter, do the right thing instead) got it right. It’s time we bury the secret world. GoogleEarth and the Keyhole Markup Language (KML) are a great start that needs to be extended to allow tiling to handle nongeospatial information provided by a mix of world content vendors and crowdsourcing. The secret world lost sight of its constitutional duties and betrayed the public—it’s time for the public to do public intelligence in the public interest.
Berghel: OSI is of keen interest from the standpoint of “survivable journalism”—reporters are besieged these days by National Security Letters threatening prosecution under the Espionage Act of 1917. How might we create an OSINT infrastructure that the few remaining investigative journalists might draw on to add timeliness and accuracy to their reporting without fearing prosecution?
Steele: I share your concern about the prosecution of journalists who get close to doing damage to the deep state and the financial powers behind the deep state. However, of greater concern to me is the absence of responsible holistic analytics with true cost economics. From agriculture to education and energy to family, health, the military, the prison complex, and water, I see gag laws, I see self-censorship, and I see a dumbed-down readership.
For me the engineering challenge of the 21st century is to reboot education, intelligence (decision support), and research simultaneously. How can we create a global “World Brain” and autonomous Internet that can’t be censored or shut down, that puts into any handheld device anywhere the truth about any topic? My 1996 article, “Creating a Smart Nation”12 is still the standard-setter on this topic.
Berghel: So, what might the technical computing community do to support the OSINT movement?
Steele: Embrace the Open Source Everything manifesto. Even if you are working on a proprietary system, code against the day when it will be made open source. Think about the open source ecology. All the stovepipes and proprietary fiefdoms are dying. I wrote an Open Letter to Microsoft that I am happy to share,13 it has most of my technical points and a number of graphics. [Figure 2] is one graphic not in that letter, my technical vision for creating a smart nation.
References
1. R.D. Steele, “The Ultimate Hack: Re-Inventing Intelligence to Re-Engineer Earth,” blog, 1
Nov. 2010; www.phibetaiota.net/2010/11/2010-the-ultimate-hack-re-inventing-intelligence
-to-re-engineer-earth-chapter-for-counter-terrorism-book-out-of-denmark.
2. R.D. Steele, “Intelligence for the President and Everyone Else,” CounterPunch, 1 Mar.
2009; www.counterpunch.org/2009/03/01/intelligence-for-the-president-and-everyone-else.
3. R.D. Steele, “Search: Seven Tribes (now Eight Tribes),” blog, 25 Nov. 2009; www.phibetaiota.
net/2009/11/search-seven-tribes-now-eight-tribes.
4. P.R. Pillar, Intelligence and U.S. Foreign Policy: Iraq, 9/11, and Misguided Reform, Columbia
Univ. Press, 2014.
5. R.D. Steele, “Intelligence with Integrity: Enabling Hybrid Public Governance with Open-
Source Decision-Support,” blog, 18 Jan. 2013; www.phibetaiota.net/2013/01/2013-robert-steele-intelligence-with-integrity-decision-support-in-the-public-interest-book-seeks-donations-for-each-chapter.
6. R.D. Steele, “Hamilton Bean: The Paradox of Open Source: An Interview with Douglas J.
Naquin with Letter from Robert Steele,” blog, 23 Dec. 2013; www.phibetaiota.net/2013/12/
hamilton-bean-the-paradox-of-open-source-an-interview-with-douglas-j-naquin-with-letter
-from-robert-steele.
7. R.D. Steele, “Reference (2010): Human Intelligence (HUMINT)—All Humans, All Minds, All
the Time [Full Text Online for Google Translate],” blog, 18 Nov. 2011; www.phibetaiota.
net/2011/11/reference-human-intelligence-humint-all-humans-all-minds-all-the-time-full-text-online-for-google-translate.
8. “Principle Challenges in Post-Saddam Iraq,” declassified report, CIA Nat’l Intelligence
Council, Jan. 2003; www.foia.cia.gov/sites/default/files/document_conversions/89801/
DOC_0005674817.pdf.
9. R.D. Steele, Intelligence for Earth: Clarity, Diversity, Integrity, and Sustainabilty, Earth Intelligence Network, 2010.
10. R.D. Steele, The Open Source Everything Manifesto: Transparency, Truth, and Trust,
Evolver Editions, 2012.
11. R.D. Steele, “Stephen E. Arnold: NYT Innovation Report—and They STILL Don’t Get It,” blog, 17 May 2014; www.phibetaiota.net/2014/05/stephen-e-arnold-nyt-innovation-report-and-they-still-dont-get-it.
12. R.D. Steele, “1996 GIQ 13/2 Creating a Smart Nation: Strategy, Policy, Intelligence,
and Information,” blog, 19 Jul. 1995; www.phibetaiota.net/1995/07/1995-giq-132-creating-a-smart-nation-strategy-policy-intelligence-and-information.
13. R.D. Steele, “2014 Robert Steele—an Open Letter,” blog, 3. Feb. 2014, www.phibetaiota.
net/2014/02/2014-robert-steele-an-open-letter.
Hal Berghel is an IEEE and ACM Fellow and a professor of computer science at the University of Nevada, Las Vegas. Contact him at hlb@computer.org.
IEEE Computer, Vol.47, No. 7, pp. 76-81
PDF (6 pages): IEEE Computer Steele Interview As Published
