Robert Steele Discusses Open Source and Secret Intelligence
HighGainBlog, 4 April 2013
Robert Steele has been a prescient thinker in the fields of search and content processing for decades. Best known for his work in open source intelligence, Mr. Steele has published widely on what I call “politico-info issues.”
One April 2, 2013 Mr. Steele and I continued our discussion of online information which which appeared in Beyond Search in May of 2008. Most recently, The full text of my discussion with him appears below:
What is your background in intelligence?
I first started dealing with Top Secret Sensitive Compartmented information (TS/SCI) in 1976. I was selected to be the S-1/Adjutant for Battalion Landing Team 3/4 (3rd Battalion, 4th Marines, heavily reinforced) and found myself as a first-tour lieutenant in a Chief Warrant Officer job. In this position, I was responsible for the personnel management and security, including the security of all classified materials, for over 1,500 Marine embarked on six ships visiting six countries over six months.
In 1979 I was competitively selected to join the Central Intelligence Agency’s clandestine (spy) service, and was released from the Marine Corps to accept that appointment. I spent nine years with the CIA, doing three tours overseas as a case officer recruiting and handling agents, and then returned to Washington to do three desk tours, one in counterintelligence, one in advanced information technology, and one in futures planning and programming for technical systems (satellites).
In 1986, due to a lack of intelligence support from the national or other intelligence services, the Marine Corps invited me to help write the Marine Corps Master Intelligence Plan (MCMIP) as well as a plan for a Marine Corps Intelligence Center (MCIC). In 1988 I resigned from the CIA to become the second-ranking civilian in Marine Corps intelligence, and the senior civilian responsible for creating the MCIC from scratch, Planning and Programming Factors for Expeditionary Operations in the Third World. Published in 1990, this included the creation of an original analytic model with 144 factors across military, civilian, and natural-geographic domains, each with three to five degrees of difficulty defined by the warfighters themselves. This had never been done before and I do not believe it has been done since. This is precisely what the United Nations (UN) and the North Atlantic Treaty Organization (NATO) each need today. My Foreword to the first book just published by NATO on public intelligence, Internet-Based Intelligence in Public Health Emergencies – Early Detection and Response in Disease Outbreak Crises is a concise articulation of the challenges and the benefits to NATO.
In 1992 I became the champion for Open Source Intelligence (OSINT), and against some considerable opposition from the CIA, I received permission from the Marine Corps to organize an international conference. To my astonishment, we had over 620 attendees. This scared both CIA and the Marine Corps lawyers, so I was forbidden to run a second conference. I resigned after 18 years of service, giving up any possibility of a pension, in order to pursue what I believed to be the essential foundation for the future of intelligence. Today, 31 March 2013, marks precisely 20 years of “walk-about” as the CEO of Open Source Solutions Network, Inc. and more recently the non-profit, Earth Intelligence Network. Everything – over 30,000 pages – from all these years is free online at Phi Beta Iota Public Intelligence Blog.
It certainly sounds like through your almost 40 years of employment you have experienced virtually every aspect of the intelligence world from public to private. What three lessons remain with you from those 40 years?
My first lesson was that, for all the money we spend on it, the secret world is not really providing the return on investment taxpayers should expect. Intelligence – decision support – is simply not being provided to everyone that needs it. From my first tour handling secrets, I remember most clearly shredding it all each morning – nothing being sent to my Commanding Officer (CO) was the slightest bit relevant to our mission and our next location. In 1988 the Marine Corps spent 20 million dollars on an information system that could receive all secrets from all agencies, and we were stunned to find out that there was nothing in the secret databases relevant to our needs. Years later, General Tony Zinni, then the Commanding General of the US Central Command (USCENTCOM), would go on record saying that he got, “at best” 4% of what he needed each day from secret sources and methods. Everything else had to be found from human, print, and online open sources.
My second lesson, is that 95% or more of what we need to create ethical evidence-based decision support is available inexpensively and openly from academics, civil society, commerce, governments, law enforcement organizations, the media, all militaries, and non-governmental/non-profit organizations. It has been a real sadness for me to see the US secret world grow from $20 billion a year to $80 billion a year over the past 20 years, and still not produce anything useful. For just $3 billion a year, an Open Source Agency (OSA) could meet 90% of our needs with the added advantage of being able to meet the needs at all levels of government–including Congress, and also public needs across society from school rooms to board rooms.
My third lesson, is that all of our checks and balances are broken. I was invited to be the keynote speaker at a forthcoming conference in Wales on the past, present, and future of intelligence. In the course of preparing my remarks, I had the opportunity to plot all of the reform articles, books, and testimony, with spikes from 1992-1994, and then after 9/11, from 2002-2004. What really jumped out at me is the lack of congressional oversight. Traditional academics have not done broad research and ignore all sources beyond their limited “approved” circle. Additionally, the media simply parrots the party line and does no investigative journalism in the intelligence arena. Everything I would want to do today if I were the Director of National Intelligence (DNI) was clearly articulated between 1989-1994, but ignored. The Aspin-Brown Commission recommendations in 1996 were quite well-developed and none have been implemented. The 9/11 Commission produced little of value because it did not ask the tough questions. 9/11 was a policy and integrity failure at the political level, not an intelligence failure at the professional level. However, the 9/11 Commission was helpful in recommending the creation of an Open Source Agency (OSA) that is co-equal to the CIA. Unfortunately, as with all the other commissions over history, the bureaucracy, the industrial complex, and Congress, all remain static.
Where do you believe open source intelligence fits into all of this?
In order to appreciate the importance of open source intelligence it helps to emphasize that intelligence is synonymous with decision-support – the output of a very robust process of requirements definition, collection management, source discovery and validation, multi-source fusion, historically- and culturally-informed analytics, and the sharp visualization that answers an important question for a particular decision-making considering a particular decision challenge.
Few realize that most of what is produced by the secret world is not intelligence at all. Rather, it is secret information that is generic in nature and often not useful to decision-makers. We are at the very end of the Industrial Era and, due to grand legalized fraud by the banks, our economies are imploding. We are at the end of the Era where 50% of every government dollar could be documented waste, and the federal government could get away with borrowing one trillion dollars a year to feed all the earmarks that yield a 5% kickback to Congressional reelection campaigns. We need to begin making data-driven decisions, decisions that are both whole systems in nature, understanding cause and effect, and also deeply rooted in true cost economics, with a full appreciation for how much water, fuel, toxins, and child labor as well as tax avoidance are represented by any given product, service or behavior.
After I realized that the secret world had little substance to offer in support of Marine Corps strategy, policy, acquisition, or most operations, I moved into the open source world. I also inventoried open sources, open softwares, and open services around the world – sources and methods that were active and accessible in 183 languages, in comparison to the six or seven that are common in the secret world, where most analysts work in English and have very little foreign, historical, or cultural experience.
When Lt Gen Dr Brent Scowcroft, USAF (Ret) and the other members of the UN High-Level Panel on Threats Challenges, and Change identified and prioritized the ten high level threats to humanity, in 2004, I was struck by both their coherence, and the degree to which open sources rather than secret sources were the primary means of addressing each. I was so inspired by this that I funded the creation of the Earth Intelligence Network, a 501c3, and pulled together 23 others to design the World Brain and Global Game, with the intent of enabling participatory democracy and consumer-driven commerce. I did a study of the utility of open source against these ten threats, my findings were published in 2007, and the bottom line is that open sources are 95-99% of the answer for the non-military, non-terrorist, non-criminal threats, and no less than 75-85% of the answer for war, proliferation, genocide, and crime. For decades we have been looking through the wrong end of the telescope, trying to drill down to a few secrets, and ignoring the wealth of insight available in open sources.
In the 1990’s I served on the Information Handling Committee (IHC) and also the Advanced Information Processing and Analysis Steering Group (AIPASG) for the US IC. In retrospect I realize that all of the different secret compartments fragmented how we spent $1.5 trillion over 20 years, and that is why we lack a comprehensive Whole of Government cloud today, a multi-lingual automated warning system, or an all-source fusion analytic workstation such as we knew we needed in 1986 – it was called CATALYST for Computer Aided Tools for the Analysis of Science & Technology. Today, at the US Special Operations Command (USSOCOM), for many years the top OSINT producer (unlike the CIA’s Open Source Center (OSC), with produces Open Source Information (OSIF) that they then classify secret), they are still using over 25 softwares none of which share data effectively, all non-interoperable, all requiring fat-fingering.
Excellent points, how does open as a concept match up with the need for secrecy?
This is a very important question, because most people do not understand that open source software is vastly more secure than proprietary software. In my experience, secrecy is used by governments and corporations primarily to avoid accountability. Transparency, truth, and trust are the common currency for the 21st Century, in my view, and I am quite certain that going “all in” on Open Source Everything (OSE), a story I tell in THE OPEN SOURCE EVERYTHING MANIFESTO: Transparency, Truth, & Trust (North Atlantic Books / Evolver Editions, 2012) is the only affordable, scalable means of creating a prosperous world at peace that works for all.
What I have found is that in the secret world, secrecy has become a cult that justifies everything else and also justifies a failure to actually produce intelligence – decision-support. I testified to the Secrecy Commission and was further honored with a private office call on Senator Daniel Patrick Moynihan (D-NY). I was deeply moved by his taking the trouble to mention open source in his cover letter to the President and Congress. He got it. Most do not.
We will always needs spies and secrecy—I said this in 2000 in my first book, ON INTELLIGENCE: Spies and Secrecy in an Open World. What I have realized in the past decade is that counterintelligence – the sniffing out of the traitors among us, of the corrupt among us – is where the emphasis on secrecy should be placed. With respect to decision-support, Michael Herman, the dean of the intelligence academics in the United Kingdom, has concluded, as have I, that secrecy is an impediment to decision-support, and that the best decision-support is not secret and thus can be shared with Congress, the media, and the public.
It was the absence of public open intelligence – and the complete failure of all checks and balances – that allowed Dick Cheney and the neo-cons to take America into a $4 trillion war on the basis of 935 now-documented lies. In the current chaos and with the various looming catastrophes that we face, from climate change to earthquakes triggered by fracking to nuclear population from Fukushima and so on, I believe the 21st Century and the third era of intelligence will be centered on Open Source Everything (OSE). I have published extensively on this topic, and a collective of relevant items online at 21st Century Public Intelligence 3.2.
On balance, we can do a better job focusing on what needs to be secret, if we first focus on making the most of open sources.
Most systems are incompatible even if labeled “open”. What are your suggestions in addressing this issue?
Proprietary software is unsafe, does not scale, and is unaffordable. I have been unhappy with all vendors for the past 40 years because not a single one of them is committed to helping people make sense of information – they focus on trapping customers into using them as a core system, make promises they cannot keep, and then over-charge for configuration management and data conversion. I am also very concerned about Google’s computational mathematics and programmable search engines – I have a very high regard for Google’s expertise, and a very low regard for the government’s ability to understand now Google can manipulate search outcomes and other forms.
My solution – and I first saw this among Paul Strassman’s works – is to make data the equivalent of money and mandate that all data adhere to standards for file transfer among systems. As you have pointed out, many people espouse this requirement but do not enforce it. The time has come to not only enforce open data and standards, but to also go “all in” on open cloud, open hardware, open software, and open spectrum.
In my experience, most of the legacy data is not worth a great deal. It is full of holes, dated, and to the extent there is anywhere there worth harvesting, it would be cheaper to have one workstation and one or two people with legacy knowledge, for as needed retrieval. If I were allowed to help create an Open Source Agency (OSA), it would be committed to going “all in” across all the opens, refusing to license any software or hardware, and following in the path of the Open Source Ecology movement that has done so well with farming technologies.
Military and intelligence entities operate in a closed environment. How do your notions of open fit into this often slow moving and traditional approach to high value information?
I am the professional intelligence officer who started the OSINT fight in 1988. More recently, Admiral James Stavrides, USN, outgoing commander of NATO, did a marvelous TED presentation in which he called for Open Source Security. The military and intelligence bureaucracies are corrupt to the bone. They fight for budget share, promotions, and retirement jobs, rather than for the public interest. We keep going into wars based on lies, and then losing those wars, and no one – flag officers especially – gets fired. At the same time, a private can be dishonorably discharged for losing his weapon, or in the case of Private Bradley Manning, for exposing massive hypocrisy on the part of the government he is fighting for.
Changes to the Earth that used to take 10,000 years now take three or less. We can no longer afford Industrial Era bureaucracies, top-down command and control, or rule by secrecy. Epoch B, which started in 2012, is about bottom-up crowd-sourcing, multinational consensus-building, and a commitment to transparency, truth, and trust.
Recently I published a graphic to guide all those – 90% of the government in the case of the USA – that do not get adequate secret intelligence support but lack the budget for open sources and methods. The key is to formulate your requirements in detail, document the failure of the secret world to meet them, and then provide a planning, programming, and budgeting systems (PPBS) request for a redirection of funding from the secret world to the open world. OMB is the adjudicator. I believe the time has come to cut the secret intelligence budget back to under $20 billion a year, return $30 billion toward savings, and redirect the remaining $30 billion to creating a Smart Nation through a nation-wide network that integrates education, open intelligence, and multi-disciplinary as well as multi-national research. My graphic on the old versus the new intelligence process is also online.
What is the role of humans (subject matter experts) in a more open intelligence environment?
Put as simply as possible, humans are essential, computers are not. Right now we are wasting over $70 billion a year on complex technical collection systems that do not work as advertised, they are simply a form of corporate welfare, and we process less than 5% of what we collect. Meanwhile, we hire children as analysts to facilitate the clearance process, and refuse to talk to anyone who is not a US citizen with a secret clearance. Jim Bamford, in Body of Secrets: Anatomy of the Ultra-Secret National Security Agency, ended the book with the following sentence:
Eventually NSA may secretly achieve the ultimate in quickness, compatibility, and efficiency—a computer with petaflop and higher speeds shrunk into a container about a liter in size, and powered by only ten watts of power—the human brain.
I was interviewed for the top job in defense human intelligence, and while I was not selected, it was an opportunity to reflect on human intelligence and human resources. I came to two conclusions:
First, there are at least fifteen “slices” of human intelligence that need to be managed as a whole, but are not. I wrote an entire monograph on this, Human Intelligence: All Humans, All Minds, All the Time, and also put together a graphic that has been popular online.
Second, and this stems from my four years of accessing hundreds of job descriptions across the public and private sector, I believe the Human Resource (HR) function has become lazy, stupid, and largely useless. I am stunned by the manner in which applicants are forced to submit individual tailored applications for hundreds of jobs, rather than being evaluated as a whole person and then matched to any number of jobs. I am particularly scornful of computer keyword matching overseen by very low ranking clerks with no substantive knowledge.
Human intelligence and innovation is the one inexhaustible resource we have on this planet. We should be striving to provide free cell phones and free access to the Internet and free education to the five billion poor, as I recently suggested to Sir Richard Branson with my concept for “The Virgin Truth,” rather than continuing our disgraceful collapse as a hollow civilization.
What is the opportunity for open everything in our unstable and volatile world?
What a wonderful question. I am not the first to see the value of open, but I may be the first to call for Open Source Everything (OSE) and to write a book outlining the specifics of how the OSE approach can help us all. I have also posted extracts online.
I have read broadly, in 98 categories, and in passing became the #1 Amazon reviewer who focuses exclusively on non-fiction. What I have seen in my reading across thousands of books is that the Industrial Era has been characterized by the abuse of power and secrecy, and by the use of multiple “information pathologies,” to concentrate wealth and make major national decisions that consistently benefit the few at the expense of the manner.
What inspires me about the future, is that the Internet has changed everything. We are moving very quickly toward an Autonomous Internet that cannot be censored or shut down. I see Bio-Economics, Capitalism Reincarnated, Collective Intelligence, Integral Consciousness, Education for Freedom & Innovation, Evolutionary Dynamics, non-violent Self-Determination and Secession, and ultimately, a World Brain and Global Game – all in the next 20 years.
The reason open everything matters is that it is the only affordable, inter-operable, scalable means of creating a prosperous world at peace that works for all. All this time we have been fragmenting our efforts, fragmenting our wealth, and very foolishly allowing banks to leverage debt and compound interest to impoverish rather than empower. Open Money, BitCoin, the Sharing Economy, time hours, I just see so much positive, and in every single instance, it is OPEN, not closed. I’ve posted lists of my reviews on the above themes online.
What is the major benefit of adopting your open everything approach?
I have to give credit to everyone else that has pioneered one or more opens, and especially to Michel Bauwens and the Peer to Peer movement, and more recently to Richard Falkvinge and the Pirate Party. In my view, open everything is what our indigenous forefathers did – the native tradition of sitting in a circle and not stopping the conversation until everyone, without exception, had agreed to a course of action that explicitly took into account the “Seventh Generation. Unilateral militarism based on lies, virtual colonialism at the point of a gun, predatory capitalism that Al Capone correctly called legalized crime – these all thrive in closed environments where you can fool most of the people most of the time. The open everything approach liberates the public from its bondage to out-moded forms of organization that are neither affordable nor agile. The open everything approach liberates the human imagination and creates infinite wealth. The open everything approach creates heaven on Earth.
What must be done to overcome the prejudice against “out of the box” thinking in US and Europe?
Prevail. I had a variation of this conversation with my middle son one day, asking him how he thought his generation would overcome all of the waste we have left them, and his answer stunned me. He said, “we’re waiting for all of you to die – after that it won’t be an issue.” Personally I have not given up on America the Beautiful and the dream that was the Republic, but we have all been battered, looted, abused, and genocided by the concentration of wealth and power. The iconoclasts like myself have been deliberately marginalized, and yet now, I sense from the traffic to my website and the various NATO officers visiting my LinkedIn profile page, that the tide has turned. Everything I suggested in 1992-1994, everything that caused my former boss as CIA to say that it confirmed my place on the lunatic fringe, has come to pass. It appears that I had to wait for 20 years for the younger generation of leaders to rise up, now they are ready to listen, and it is very possible that within five years NATO will be a force for peace, creating a Multinational Decision Support Centre that can practice Information Peacekeeping and Peacekeeping Intelligence, both concepts I developed with a few others in the 1990’s. So, if you live long enough, eventually a majority will agree with you – be strong, be persistent, as Winston Churchill said, “NEVER GIVE UP.”
What contributions can you make to “open everything” with regard to intelligence and war fighting?
I have done all one man can do by himself. I have submitted proposals to the Secretary General of the United Nations (2009, 2012), to the President of the United States, to the Pope, to the CTO of Microsoft, and most recently, to Sir Richard Branson. At some point someone somewhere is going to realize that we need an Open Source Agency (OSA) in order to overcome the residual waste and toxic inheritance of the Industrial Era, and in order to nurture all the opens so as to create infinite wealth, a world that works for all. I have never been about rank or money. It would make me very happy if I could be part of a team, perhaps one sponsored by NATO, creating a prototype Multinational Decision Support Centre on the Mediterranean, one that focuses on North Africa, the Middle East, and the Caucasus. I would like to see a similar endeavor in Chile or Argentina focused on the Southern Hemisphere from Indonesia to South Africa, and a third in Singapore or Indonesia. What distresses me is the continued waste of the public wealth by adults who should know better. My book in progress is entitled INTELLIGENCE with INTEGRITY: Enabling Hybrid Public Governance with Open-Source Decision-Support. I know how to move the world. I just need a place to plant my lever – the truth at any cost lowers all other costs.
As for war-fighting, war is a racket funded by banks and enabled by corrupt politicians. Sun Tzu had it right – the acme of skill is to defeat the enemy without fighting, and that is precisely what China is doing with its soft power. I would go a step further and suggest that We are One, and that true divinity comes from respecting the limits of the Earth and the dignity of every person. Open everything includes open hearts and open minds, and therein we can find the grace of God.
ArnoldIT Comment
Mr. Steele’s observations are thought provoking. His ideas can add significantly value to many “intelligence related” programs. For more information, navigate to his Public Intelligence information service.
Jasmine Ashton, April 4, 2013
Sponsored by Augmentext
