There is a need for hard data and statistics on OSINT — why it has been relegated to the basement and distributed rather than consolidated as with the other disciplines, what it's successes and failures have been.
In how many cases did it provide Intelligence that was critical to mission success ? NEEDS DATA
Defining mission critical is important. If you mean, mission critical for political decisions, then neither classified nor OSINT matter. It is well established, from Kissinger to Pillar and others, that intelligence plays no substantive role when ideology rules.
Henry Kissinger: Intelligence as Useless as a Booger — and Less Tasty
Review: Intelligence and U.S. Foreign Policy – Iraq, 9/11, and Misguided Reform
If you mean mission critical at the strategic, acquisition, or operational levels, then news flash, we still lack Whole of Government strategic intelligence, intelligence support to joint acquisition, and 1:50,000 combat charts for 90% of the world (no real change since 1988 when I and SOCOM J-2 rang the bell on this one). We are also incapable of HUMINT in denied areas including the Middle East, or of providing responsive intelligence for fast-moving tactical maneuvers and targeting, something General Bob Scales addresses in his still relevant book, Firepower in Limited War. This is where you need to read Boyd Sutton on Global Coverage and my reviews of the books by Bean and Olcott (links further down). In time sensitive situations when my team has supported SOCOM, over and over and over again, OSINT comes through in 24 hours for a pittance of cost, while classified spins its wheels and lies about its abilities. I won the Burundi Exercise in front of the Aspin-Brown Commission with OSINT, beating the entire IC overnight (Thu to Mon).
Reference: Open Source Burundi Exercise
The single most successful end to end program — until it was destroyed — was that created by Ben Harrison at SOCOM after I briefed General Peter Schoomaker, then CINCSOC.
1997 CINCSOC 10-Minute Briefing That Created SOF OSINT
Ben was a genius about demanding that all SOF requirements for OSINT be entered into COLISEUM so that within the year he could demonstrate, going head to head against the classified disciplines, that he was answering 40% of all global SOF demands for intelligence with OSINT. Ben would no doubt observe that part of his success lay with the fact that often what was asked for was open source information (OSIF) rather than OSINT (tailored decision support) but I know of many extraordinary successes at SOCOM in merging open sources in multiple languages from multiple mediums and locations to achieve an intelligence picture that neither DIA nor CIA could match.
Most people have no clue that OSINT is acquisition, not collection, and it is not normally subject to the collection constraints prevailing for the classified disciplines. This was established by SOCOM in dialog with the DoD Assistant Secretary for intelligence permissions in 1997, and has been forgotten.
Most people have no clue how easily OSINT cover support plans can completely obscure everything essential about the origin, nature, and urgency of the requirement. I remain stunned by how inept both the US IC and its contractors are in relation to non-official cover and multi-layer multinational cut-outs….ineptitude is either compounded from or rooted in sheer laziness and people not being held accountable for doing it right — this problem also infects the classified disciplines — for example, CIA “case officers” live immunity rather than cover, and I want to puke every time I think about a CIA “case officer” going to a meeting surrounded by guys with ink (bodyguard) in black SUVs.
As a general statement, from Allen Dulles to LTG Mike Flynn to myself, it has been clearly established and stated that open sources can answer 80% of all requirements, and that is low-balling it. In many cases it can answer up to 95% but in some cases its utility might be much lower. Hereto the side is the graphic that was central to my seminal chapter in Loch Johnson's five volume series on strategic intelligence (link below).
Something you have not asked is why OSINT is not even more useful. I would cite three reasons:
01 The IC is deathly afraid of OSINT being validated as faster, better, cheaper. They need not fear this as long as we have a corrupt political process that could care less about ethical evidence-based decision-support. If we ever have an honest White House and honest Congress, a $3B a year Open Source Agency will within 3 years justify a reduction of the secret intelligence budget to under $25B a year. I would observe that Boyd Sutton documented the need for a major open source investment — Keith Hall then D/NRO provided the number, $10M a year times each of 150 lower tier targets not covered by classified for a total of $1.5B a year — I and Congressman Simmons doubled that number to provide a matching amount for a domestic network of educational citizen intelligence centers — and George Tenet, then DCI, immediately locked the study up. He concealed the findings from Congress and refused to address the need.
2006 Sutton (US) Global Coverage, Looking Backward, Looking Forward
1997 Sutton (US) The Challenge of Global Coverage
02 The IC, DoJ, and WH have retarded timid lawyers whose ignorance is compounded by the lack of a competent champion for OSINT (CIA's Open Source Center is the proverbial runt, a red-headed step-child locked in the closet with a sock in its mouth — they are not even allowed to leverage human subject matter experts by dictat from the CIA's clandestine service). Social media is hugely important and cuts across all functional areas, but the emphasis on political correctness, NSA abuses, and a general lack of deep professionalism in both government and contractor circles are severely constraining our ability to leverage foreign and domestic social media for legitimate purposes.
03 OSINT is not “expensive enough.” No kidding. The intelligence-industrial complex exists to spend as much money without accountability as it can, and OSINT is disappointing. Perhaps this is the reason it has been mistakenly classified as TECHINT (witless US citizen butts in seats having clearances but no foreign language, cultural, or historical knowledge). “Return on investment” is not a concept the DNI cares to consider or he would be out of a job. Nor is it a concept OMB cares to consider or its politically appointed “leaders” would be impeachable for betrayal of the public trust.
There are two major benefits of an Open Source Agency that everyone appears reluctant to acknowledge. This was the central thrust of my arguments to Jim Clapper when he was Undersecretary of Defense for Intelligence, but he refused to listen.
2009 DoD OSINT Leadership and Staff Briefings
01 OSINT establishes the baseline for what can be known ethically and inexpensively. It sets the bar for determining the value-added of the classified discipines.
02 OSINT establishes the baseline for M4IS2 (Multinational, Multiagency, Multidisciplinary, Multidomain Information-Sharing and Sense-Making) across the “eight tribes” and among nations and across all academic and policy and acquisition domains, to include the establishment of open source software and hardware for processing, analysis, and protected dissemination “without frontiers.” What we have today is a travesty. We have made no major gains since the requirements for an all-source analysis tool-kit were defined in the late 1980's [the demo “caves” are a waste of resources and cannot be replicated across town, much less in the field]. I cannot stress this enough. The US IC is retarded both intellectually and technically in relation to the integration of holistic analytics, true cost economics, and open source everything engineering.
2015 Robert Steele – Foreword to Stephen E. Arnold’s CyberOSINT: Next Generation Information Access
2014 Robert Steele: Appraisal of Analytic Foundations – Email Provided, Feedback Solicited – UPDATED
1976+ Intelligence Models 2.1 . 1957+ Decision Support Story
BOOK: The Open Source Everything Manifesto: Transparency, Truth, & Trust
BOOK : Intelligence for Earth: Clarity, Diversity, Integrity, & Sustainability
Lowenthal says “For no other collection discipline do we have separate programs in separate agencies. Indeed, it would be ludicrous to suggest CIA, DIA and INR each have their own imagery or signals programs. For OSINT, however, this sort of duplicative frittering of resources was deemed acceptable. Moreover, with each agency left to its own open source devices, there has been little accountability for failing to collect, process and exploit open sources”
Lowenthal, who worked for me in the late 1990's, got this right, I am the champion of an Open Source Agency, and you need to study some of the references at Phi Beta Iota, including the DoD briefings. Lowenthal's one noteworthy historical observation from his time on the HPSCI staff is “we don't pay (the IC) to get information in the most expensive dangerous way possible.” 95% of the time OSINT is faster, better, cheaper AND it can be shared with media, Congress, public, and foreign governments.
2013 Open Source Agency: Executive Access Point 2.1
How many domestic terrorist attacks has OSINT prevented?
Wrong question in part because the classified sources and methods have not prevented any terrorist attacks either. OSINT would shape national strategy, policy, acquisition, and operations publicly, such that we have sound foreign policy, do not support 42 of 44 dictators, and stop terrorizing the rest of the world, creating tens of thousands of “terrorists” (freedom fighters) with our indiscriminate unilateral militarism, predatory capitalism, and virtual colonialism. 90% of secret “intelligence” is secret collected information that is never processed and does no one any good at all. Today we ignore legitimate grievances, assassinate with drones (a 98% collateral damage ratio) and wage elective wars on borrowed money “because we can.” No one is held accountable by facts, holistic analytics, or true cost economics. 111 citizens were killed in the USA in the month of March by law enforcement professionals. Terrorists are not the threat (the FBI is behind most terrorist plots in the USA) – WE are the threat. See the preconditions of revolution that exist in the USA today here to the side. An honest government would understand that it should not be abusing the 99% at home or abroad.
BLOWBACK, Legitimate Grievances, & Integrity Lost
Election 2008 Chapter: Legitimate Grievances
Journal: Why they hate us (II): How many Muslims has the U.S. killed in the past 30 years?
Owl: Visualization of Drone Deaths — 98% “Collateral Damage
Worth a Look: Book Reviews on Empire as Cancer Including Betrayal & Deceit
Richelson Study: HUMINT statistically provided more critical intelligence to mission success in more cases than any other INT.
This is true, I agree with him on this, and would point out with the attached graphic and my seminal monograph on HUMINT that OSINT is HUMINT, not TECHINT, this is where Clapper and Vickers are over the cliff and in free fall — and where Brennan has completely missed the boat as did his predecessors. My vision of OSINT got corrupted by the military-industrial complex, we were doing OSINT for SOUTHCOM SOCOM and CENTCOM with great precision (for example, $60,000 in 1997 to identify and evaluate 396 terrorist, insurgent, and opposition web sites in 29 languages in 60 days); or later, identifying the financial assets of the top Islamic families around the world — spies don't do real estate or analog tax and yacht records). 90% of what we need to know is openly available, and within that 90%, it is faster, better and cheaper at least half the time to get the right subject matter experts to create tailored knowledge or distill their grasp of broad knowledge into what we need to know, than to scour for ourselves. OSINT is HUMINT. Knowing who to ask, how, is what produces sensationally timely and useless analytics from top indigenous practitioners that no combination of “experts” in the US IC can match — often for $250 to $2500 and within 48 hours.
What has OSINT success been in the following: Syria, Ukraine, China, ISIL, Iran, North Korea, Any county in Africa, the Saudis crashing the US energy sector?
OSINT in hands of the US public stopped the elective invasion against Syria and has revealed the misadventures of the USA in relation to Ukraine. OSINT has made it clear Saudi Arabia is funding ISIS and CIA is training them, they are a false flag cosmetic operation. OSINT penetrates North Korea, illuminates all of Africa, etcetera. Most skeptics of OSINT have the problem that they do not think in terms of multinational multilingual sources and methods, and they is not aware that the secret agencies refuse to give prominence to OSINT (despite claims that it is often featured in the President's Daily Brief) because they are deathly afraid that the rise of OSINT will spell the demise of secret sources and methods that General Tony Zinni is on record as stating were useful to him “at best” 4% of the time.
By the by, the US IC is not serious about China, their idea of “doubling” resources is to go from four analysts to eight at CIA, they are inept when it comes to covering all critical technologies in Chinese publications and online, and Saudi Arabia is not “crashing” the US energy sector, the corruption of the US Government and the energy industry is doing that without any foreign help. I am still aghast that a National Intelligence Council report would recommend fracking as part of the answer to our national energy needs (fracking causes earthquakes, uses up scarce fresh water, and contaminates what remains in the acquifers) but this is what they did in Global Trends 2030. By now a properly managed US intelligence community would have placed before the public the many incontrovertible paths toward free energy and what that means for desalinating water (which among other things keeps the sea at its present level) and nurturing hydroponics industries that eliminate all forms of chemicals from the agricultural process.
Energy as Category @ Phi Beta Iota
When you are ready I can introduce you to some folks and you can get some direct quotes from them. One of the things you need to understand is that everyone you talk to in the policy and intelligence worlds as well as the media and think tank worlds is going to lie to you at one level or another.
By the by, I always look for opportunities to point out that most — not all — academics are largely worthless as researchers, they are lazy, monolingual, have citation cabals, and do not go outside what they can get to from their university library or the comfort of their air-conditioned cubicle. They also refuse to take seriously practitioner experience such as I amassed from 800 speakers across 66 countries. Not a single “academic” article on OSINT today is properly sourced — the exceptions are Hamilton Bean's book and to a lesser extent Olcott's book. Add to this the fact that of all published paper roughly 1% get published, and you have a major problem. Here to the side is the fragmentation of the scientific world, the social science world is even worse.
Review: No More Secrets – Open Source Information and the Reshaping of U.S. Intelligence
Review: Open Source Intelligence in a Networked World
Here are three starting points for thinking about next steps.
2007 Open Source Intelligence (Strategic) 2.0
OSINT Literature Review, Name Association, Lessons Learned
1997 Sutton (US) The Challenge of Global Coverage
Your quest for the truth on OSINT reminds me of then Maj Rob Simmons, USAR, ultimately a Colonel and a Congressman. Here is his original 1995 study done at DIA a twenty years ago.
Open Source Intelligence: An Examination of Its Exploitation in the Defense Intelligence Community
Our secret intelligence community is an abject failure today because it lacks integrity across the board and is simply not serious about creating ethical evidence-based decision support at the strategic, operational, tactical, and technical levels.
One final comment: I believe that ethics and leadership going into the future are going to demand that we get intelligence (decision-support) right. Someone, possible a CWO in a paper out of DIA said in the 1990's that intelligence (decision-support) is an inherent responsibility of command that cannot — should not — be delegated and I regret to this day not noting down his name and the specific citation. I believe that. I also believe the education, intelligence, and research are going to have to come under one Secretary-General at the national level in the not too distant future — those managing each of these three domains in isolation from one another are well-intentioned but incompetent and out of touch with both reality and the possibilities of a human break-out.
We don't have leaders in the IC or the USG, we have “kiss up kick down” bosses who have been promoted on the basis of going along to get along and keeping the money moving, not on the basis of actually being competent constantly evolving maturing intelligence professionals. Here to the side is the single most popular graphic on this website.
2014 Robert Steele Answers to Richard Olivier on 21st Leadership with Annotated Bibliography & Links
Robert Steele: Online Review Books on Education, Intelligence, Research
A few contextual pieces of mine and two of my favorite graphics going forward into the future:
2014 Steele on Intelligence in Ecuador (English)
2014 Robert Steele Applied Collective Intelligence
2014 Robert Steele On Defense Intelligence – Seven Strikes
2013: The Evolving Craft of Intelligence 3.6 As Published
2009 Intelligence for the President–AND Everyone Else
INTERVIEW: Open Everything – with Robert David STEELE Vivas UPDATED to Add Parts V and VI
