2007 Open Source Intelligence (Strategic) 2.0

Now that everyone is paying attention, this is being posted in full text online in support of a larger M4IS2 / OSE dialog.

Document:  Strategic OSINT (Chapter 6 in Strataegic Intelligence Vol 2) 10 MB

Links added below throughout, updated where appropriate.

See Also:

21st Century Intelligence Core References 2.0

Open Source Intelligence (OSINT)

Robert David Steele

Introduction
OSINT and Intelligence Reform
– History
– Requirements
– Collection
– Processing
– Analysis
– Covert Action
– Counterintelligence
– Accountability, Civil Liberties, and Oversight
– Strategic Warning
– Strategic Sharing
– Emerging Prospects
— Digitization
— Visualization
—  Peer-to-Peer (P2P)
OSINT and Electoral Reform
OSINT and Governance Reform
OSINT and Strategic Budgetary Reform
Notes 1-44

Executive Summary

            Open Source Intelligence (OSINT) is the only discipline that is both a necessary foundation for effective classified intelligence collection and analysis, and a full multi-media discipline in its own right, combining overt human intelligence from open sources, commercial imagery, foreign broadcast monitoring, and numerous other direct and localized information sources and methods not now properly exploited by the secret intelligence community.  OSINT is uniquely important to the development of strategic intelligence, not only for the government, but for the military, law enforcement, business, academia, non-governmental organizations, the media, and civil societies including citizen advocacy groups, labor unions, and religions for the simple reason that its reliance on strictly legal and open sources and methods allows OSINT to be shared with anyone anywhere, and helps create broader communities of interest through structured information sharing.

It can be said that at the strategic level in particular, but at all four levels of analysis (strategic, operational, tactical, and technical) generally, the secret intelligence communities of the world are inside-out and upside-down.  They are inside-out because they persist in trying to answer important questions with unilaterally collected secrets, rather than beginning with what they can learn from the outside-in: from the seven tribes[1] and the 90+ nations that form the Coalition.  They are upside down, at least in the case of the United States of America (USA) and selected other major powers, because they rely too much on expensive overhead satellite systems, instead of bottom-up ground truth networks of humans with deep historical, cultural, and localized knowledge.

In the long-run, I anticipate that OSINT will displace 80% of the current manpower and dollars devoted to secret sources & methods, and that this will offer the taxpayers of the respective nations a Return on Investment (RoI) at least one thousand times better than what is obtained now through secret sources and methods.  A proper focus on OSINT will alter the definition of “national” intelligence to embrace all that can be known from the seven tribes across both the home nation and the coalition nations, and will dramatically reform intelligence, electoral processes, governance, and the application of the national, state, and local budgets in support of the public interest.

Strategically, OSINT will restore informed engaged democracy and moral capitalism, a new form of communal capitalism, in America and around the world.  OSINT is, at root, the foundation for the emergence of the World Brain, and the empowerment of the public.

The bulk of this chapter will focus on OSINT and intelligence reform at the strategic level, but it is essential that the reader appreciate the implications of OSINT for electoral, governance, and budgetary reform so as to better realize the enormous implications of the revolution in intelligence affairs[2] for which OSINT is the catalyst.

 

Figure 1:  Four Strategic Domains for Reform Catalyzed by OSINT[3]

The impetus for reform across all four strategic domains could emerge from within any one of the four.  If the economy collapses and the war on Iraq combined with an attack on Iran cause a clear and present danger to emerge in the form of global Islamic counter-attacks that are asymmetric and indiscriminate as well as widespread, we can anticipate not just the ejection of the extremist Republicans, but also of the complacent and equally corrupt and ignorant Democrats.[4]

There is a growing awareness within the public, described by some as “smart mobs,” or “wisdom of the crowds,” or—our preferred term—Collective Intelligence, that it is now possible for individuals to have better intelligence based on open sources and methods, that is being made available to—or acknowledged by—the President.[5]  We will see, within the next four years, a dramatic increase in both historical accountability,[6] and current accountability for actions impacting on future generations and other communities.

Electoral reform will be inspired by citizens realizing that both the Republican and Democratic parties have been come corrupt as well as inept at representing the public interest.

Governance reform will be inspired by citizens realizing that in today’s world, we need a networked model of governance that elevates intelligence to the forefront—decisions must be made in the public interest and be sustainable by consensus and conformance to reality, not purchased by bribery from special interests who seek to loot the commonwealth and/or abuse their public power to pursue the ideological fantasies of an extremist minority.

Budgetary reform will be inspired by citizens who understand that we still need to be able to defend ourselves, but that waging peace worldwide is a much more cost effective means of both deterring attacks and of stimulating sustainable indigenous wealth that is inherently stabilizing.

OSINT and Intelligence Reform

Open Source Intelligence (OSINT) should be, but is not, the foundation for all of the secret collection disciplines, and it could be, but is not, the foundation for a total reformation of both the governmental function of intelligence, and the larger concept of national and global intelligence, what some call Collective Intelligence or the World Brain.[7]

Secret intelligence, inclusive of covert action and counterintelligence, has failed in all substantive respects since the end of World War II and through the Cold War.  In failing to meet the mandate to inform policy, acquisition, operations, and logistics, secret intelligence has contributed to the “50 Year Wound”[8] and failed to stimulate a redirection of national investments from military capabilities to what General Al Gray, then Commandant of the Marine Corps, called “peaceful preventive measures.”[9]

Secret intelligence became synonymous with clandestine and secret technical collection, with very little funding applied to either sense-making information technologies, or to deep and distributed human expertise.  The end result at the strategic level can be described by the following two observations, the first a quote and the second a recollected paraphrase:

Daniel Ellsberg speaking to Henry Kissinger:

The danger is, you’ll become like a moron.  You’ll become incapable of learning from most people in the world, no matter how much experience they have in their particular areas that may be much greater than yours” [because of your blind faith in the value of your narrow and often incorrect secret information]. [10]

Tony Zinni speaking to a senior national security manager:

“80% of what I needed to know as CINCENT I got from open sources rather than classified reporting.  And within the remaining 20%, if I knew what to look for, I found another 16%.  At the end of it all, classified intelligence provided me, at best, with 4% of my command knowledge.”[11]

Secret intelligence may legitimately claim some extraordinary successes, and we do not disagree with Richard Helms when he says that some of those successes more than justified the entire secret intelligence budget, for example, in relation to Soviet military capabilities and our counter-measures.[12]  However, in the larger scheme of things, secret intelligence failed to render a strategic value to the nation, in part because it failed to establish a domestic constituency, and could be so easily ignored by Democratic presidents and both ignored and manipulated by Republican presidents.[13]

In this first section, we will briefly review both the failings of each aspect of the secret intelligence world, and summarize how OSINT can improve that specific aspect.

History

The history of secret intelligence may be concisely summarized in relation to three periods:

• Secret War.  For centuries intelligence, like war, was seen to be the prerogative of kings and states, and it was used as a form of “war by other means,” with spies and counter-spies, covert actions and plausible deniability.[13]
• Strategic Analysis.  During and following World War II, Sherman Kent led a movement to emphasize strategic analysis.  Despite his appreciation for open sources of information, and academic as well as other experts, the clandestine and covert action elements of the Office of Strategic Services (OSS) and the follow-on Central Intelligence Group (CIG) and then Central Intelligence Agency (CIA), grew out of control, well beyond what President Harry Truman had envisioned when he sponsored the National Security Act of 1947.[14]
• Smart Nation.  Since 1988 there has been an emergent movement, not yet successful, but increasingly taking on a life of its own in the private sector outside of government.  Originally conceptualized as an adjunct to secret intelligence, a corrective focus on open sources long neglected, it was soon joined by the Collective Intelligence movement that has also been referred to as “smart mobs” or “wisdom of the crowds,” or “world brain.”  H. G. Wells conceptualized a world brain in the 1930’s.  Qunicy Wright conceptualized a world intelligence center in the 1950’s.  Others have written about smart nations, collective intelligence, global brain, and the seven tribes of intelligence.[15]

Although the U.S. Intelligence Community has individuals that respect the value of open sources of information, and every major commission since the 1940’s has in some form or another called for improved access to foreign language information that is openly available,  the reality is that today, in 2006, the United States of America (USA) continues to spend between $50 billion and $70 billion a year on secret collection, almost nothing on all-source sense-making or world-class analysis, and just over $250 million a year on OSINT.  This is nothing less than institutionalized lunacy.

The future history of secret intelligence is likely to feature its demise, but only after a citizen’s intelligence network is able to apply OSINT to achieve electoral, governance, and budgetary reform, with the result that secret intelligence waste and defense acquisition waste will be converted into “waging peace” with peaceful preventive measures and a massive focus on eliminating poverty, disease, and corruption, while enabling clean water, alternative energy, and collaborative behavior across all cultural boundaries.[16]

Requirements

Requirements, or Requirements Definition, is the single most important aspect of the all-source intelligence cycle, and the most neglected.  Today, and going back into history, policymakers and commanders tend to ignore intelligence, ask the wrong questions, or ask questions in such a way as to prejudice the answers.  There are three major problems that must be addressed if we are to improve all-source decision support to all relevant clients for intelligence:

1.  Scope.  We must acknowledge that all levels of all organizations need intelligence.  We cannot limit ourselves to “secrets for the President.”  If we fail to acknowledge the needs of lower-level policy makers, including all Cabinet members and their Assistant Secretaries; all acquisition managers, all operational commanders down to civil affairs and military police units; all logisticians; and all allied coalition elements including non-governmental organizations, then we are not being professional about applying the proven process of intelligence to the decision-support needs of key individuals responsible for national security and national prosperity.

2.  Competition.  We must acknowledge that open sources of information are vastly more influential in the domestic politics of all nations, and that it is not possible to be effective at defining requirements for secret intelligence decision-support in the absence of a complete grasp of what is impacting on the policy makers, managers, and commanders from the open sources world.  Consider the figure below.

Figure 2: Competing Influences on the Intelligence Consumer[17]

3.  Focus.  Third, and finally, we must acknowledge that at the strategic level, our focus must of necessity be on long-term threats and opportunities that are global, complex, inter-related, and desperately in need of public education, public recognition, and public policy that is sustainable, which is to say, non-partisan or bi-partisan.  Consider, for example, the following findings from the Report of the High-level Panel on Threats, Challenges and Change, A more secure world: Our shared responsibility[18]

Figure 3:  OSINT Relevance to Global Security Threats

            The average utility and relevant of OSINT to these global threats is—on the basis of my informed estimate—82.5%, which comes very close to the generic “80-20” rule.  We must conclude that any nation that persists in spending 99.9 percent of its intelligence funds on collecting secrets,[19] and less than one half of one percent of its intelligence funds on OSINT, is quite literally, clinically insane (or insanely corrupt) at the highest levels.

In all three of the above cases, only OSINT can deliver a solution that is affordable, practical, and infinitely shareable with all stake-holders both in and out of government.

Collection

Secret collection has made three fundamental mistakes across several generations of management:[20]

1.  Denigrated OSINT.  It chose to ignore open sources of information, assuming that the consumers of intelligence were responsible for their own OSINT, and that OSINT would not impact on secret collection.  In fact, OSINT can dramatically reduce the cost and the risk, and increase the return on investment in secret sources & methods, simply by helping with targeting, spotting, assessment, validation, and the over-all strategic context of what needs to be collected “by other means.”  It merits very strong emphasis that this failure to respect open sources of information falls into three distinct forms:

• Complete disrespect for history in all languages.  There is no place within the U.S. Government where one can “see” all Chinese statements on the Spratley Islands, or all Iranian statements on the competing Caliphate concept, or all Brazilian statements on alternative energy sources.  We simply do not compute history, and consequently what little we know about current events and threats is known is isolated ignorance of history.
• Complete abdication of any responsibility for monitoring, understanding, and engaging sub-state or transnational entities as major factors in both international affairs, and as threats or potential allies in domestic security and prosperity.
• Finally, almost complete abdication for more nuanced topics other than standard political-military calculations, with very important sustained failures to collect information on socio-economic, ideo-cultural, techno-demographic, or natural-geographic matters.  This has been compounded by an extraordinary laziness or ignorance is relying almost exclusively on what can be stolen or obtained readily in English—the USA simply does not “do” the key 31 languages,[21] much less the totality of 185 languages necessary to understand the sub-state threat and the global network of cause and effect.

2.  Official Cover.  We have relied almost exclusively, at least in the USA, on “official cover” for our spies, and known trajectories for our satellites.  Non-Official Cover (NOC), which does not offer any form of diplomatic or other official immunity from incarceration or eviction, has been treated as too expensive, too complicated, and not worthy of full development.  The result has been the almost total compromise of all U.S. secret agents and case officers overseas, as well as their varied not-so-secret thefts of the codebooks of other nations.  We not only don’t know what we don’t know, we are in denial about the basic fact that what we do know has been compromised.

3.  Failure to Process.  Finally, and this applies to both clandestine human collection and secret technical collection, we have failed, with deliberate ignorance at the management level, to devote any resources of significance to processing—to sense-making.  Today, eighteen years after the needed functionalities for an all-source analytic desktop toolkit were published, we still do not have a desktop analytic toolkit.  Today, despite major advances in the private sector with respect to machine-speed translation, and machine-speed statistical, pattern, and predictive analysis, the large majority of our classified intelligence analysis is still done the old-fashioned way: reading at human speed, cutting and pasting, attempting to make sense of vast volumes of secret information while lacking equivalent access to vast volumes of open source information (and especially open source information in any language other than English), limited by the physics of the 24-hour day.

OSINT combines the proven process of intelligence with the ability to collect, process, and analyze all information in all languages all the time.  We collect, at best, 20% of what we need to collect, at 99% of the cost, and we spill most of that for lack of processing capabilities.  It can be said, as an informed judgment, that Washington is operating on 2% of the relevant strategic information necessary to devise, implement, and adjust national strategy.[22]  We should not be sending spies where schoolboys can go, nor should we be ignoring scholarship in all languages.

There will still be a need for selected clandestine human operations, especially against organized crime and translation terrorist groups, but they will need to shift toward non-official cover, and multinational task forces.  Secret technical collection will need to emphasize commercial collection first, dramatically refocus secret collection, and shift the bulk of the future resources toward processing—making sense of what we do collect—and toward close-in technical collection inclusive of beacons for tracking bad guys and bad things.

Collection Management will require draconian reform.  Instead of defaulting to the tasking of secret collection capabilities, an enlightened collection manager will first determine if they can FIND the information for free in their existing stores of knowledge; then determine if they can GET  the information for free from allied government or any of the seven tribes; and then determine if they can BUY the information from a commercial provider, ideally a localized provider with direct indigenous access, in the time and with the operational security (e.g. cover support plans) appropriate to the need.  Only if the first three options are unsuited to the need should the collection manager be tasking secret sources and methods, and even that will have to change to accommodate new possibilities from multinational secret task forces able to leverage the collection capabilities of varied countries, many of them vastly superior to the USA when it comes to both deep cover clandestine human penetrations, and the related ability to place close-in secret technical collection devises.[23]

OSINT is, without question, the catalyst for a revolution in how we collect intelligence.

Processing

Apart from our failure to actually invest in processing (known within the US Intelligence Community as Tasking, Processing, Exploitation, and Dissemination, or TPED), we have made three consistent mistakes over time that have made it virtually impossible, and now unaffordable, to actually do automated all-source analysis:

1.  No Standards.  We failed to establish data standards that could be used at the point of entry for both secret and open sources of information.  This applies to both information sources, and information software.  Not only was the intelligence community much to slow to adopt commons standards such as eXtended Markup Language (XML) Resource Description Framework (RDF), Web Ontology Language (OWL), Simple Object Access Protocol (SOAP), today it is either ignorant of or reluctant to move ahead aggressively with Open Hypertextdocument System (OHS)[24] and eXtended Markup Language Geospatial (XML Geo).  The obsession with security, and the pathology of limiting contracts to the established firms in the military-industrial complex who profit from proprietary softwares and human “butts in seats” rather than real-world low-cost answers, can be blamed for the chasm between the secret intelligence world and the real world of open sources and standards.

2.  No Geospatial Attributes.  In the fall of 1988 it was made known to the U.S. Intelligence Community and clearly articulated by the author to a meeting of the General Defense Intelligence Program (GDIP), that in the absence of geospatial attributes for every datum entering the all-source processing system (actually an archipelago of private databases), that machine-speed all-source analysis and fusion would be an impossibility.  Despite this, the individual secret collection disciplines of clandestine human intelligence (HUMINT), signals intelligence (SIGINT), and imagery intelligence (IMINT) refuse to do anything other than persist with their human analytic reporting that provides date-time-group (DTG), and geographic place names where known, but no standard geospatial attributes for relating information to a map.  Today Google Earth is being used in extraordinary ways to visualize relationship databases of real estate, shipping, and other important topics, and the individual citizen is light years ahead of the average “cut and paste” analyst at the federal, state, and local levels.

3.  No Integration.  There is no one single place where all known information comes together.  Despite critical concerns raised by every Congressional and Presidential commission since the 1940’s, the U.S. Intelligence Community has continued to be “flawed by design,”[25] and persisted in the turf wars between the Central Intelligence Agency (CIA) and the Federal Bureau of Investigation (FBI), between the FBI and the Drug Enforcement Administration (DEA), between FBI and the Department of Justice (DoJ), and between the Departments of State and Defense.  Within the Department of Defense (DoD), the services have not only competed with one another, but actively conspired to fabricate and manipulate intelligence to exaggerate the threats relevant to their budget share.  A corollary of this abysmal situation is that processing within the stovepipes has been focused on the delivery of documents rather than on making sense of all of the information in the aggregate.  With the exception of selected efforts at the National Security Agency, the Army’s Intelligence and Security Command, and the U.S. Special Operations Command, virtually all civilian and military analysts are still in cut and paste mode, and do not have the tools for pattern or trend analysis or anomaly detection, much less predictive analysis.

In processing, it is machine speed translation and statistical analysis, based on standards and global distributed information integration, that permits early warning, anomaly detection, and structured analysis that can be completed in a timely—that is to say, relevant—manner.  OSINT is where the real innovation is occurring, and I anticipate that within ten years, the secret world will be sharply restricted to no more than 20% of its present cost and size, while the balance of the funding is re-directed to a mix of OSINT that can be shared with anyone, and peaceful preventive measures in lieu of a heavy metal military.

Among the corrective measures required in secret processing, which OSINT will facilitate, are a shift toward the Internet as the common operating environment; the adoption of open source software to provide a generic access and collaborative sharing environment for all seven tribes;[26] the development of 24/7 “plots” at every level of governance in which all information can be seen in time and geospatial context;[27] and the creation of a national skunkworks with an anti-trust waiver for the public testing and certification of all open sources, softwares, and services.  Rapid promulgation of free wireless within urban areas and in the Third World will help accelerate both sharing in the North and West, and uploading of useful information from the East and South.

Analysis

In evaluating the failure of  analysis, it is important to understand that most U.S. analysts are too young, too inexperienced in the real world, and too isolated from foreign or even U.S. private sector experts, to realize that the secret information they are receiving is out of context, often wrong, and largely irrelevant to strategic analysis.  Their managers are too busy trying to be promoted or to win bonuses or please the White House (or the representative of the White House, the Director of Central Intelligence (DCI)).  As a result, the strategic analysis vision of Sherman Kent has been dishonored and largely set aside.  There have been three major failures in analysis over time.

1.  Hire Young.  The intelligence management philosophy in both the national civilian hires and at the military theater and service center levels has combined “hire to payroll” with obsessive lazy security parameters that have resulted in an analytic population that is largely young, white, and mostly bereft of overseas experience and especially long-term residency in foreign countries.  On the one hand, budgets have been used to hire low and promote over time, treating analysis as an entry-level hiring challenge rather than a mid-career sabbatical challenge; this has been deeply and pathologically influenced by a low-rent security philosophy that has combined paranoia over foreign contacts (and relatives), with an unwillingness to spend the time and thoughtfulness necessary to clear complicated individuals that have led complicated lives.  This personnel management failure stems from the larger philosophical management failure, which confuses secrets with intelligence, and thus demeans expertise from the open source world while assuming that young analysts will succeed because they have access to secrets, rather than because of any application of analytic tradecraft such as might take twenty years to refine.

2.  Hard Target Focus.  In keeping with the military-industrial complex and its desire to profit from the Cold War, the national and military intelligence communities devoted virtually their entire budgets and most of their manpower to the “hard targets” (generally, Russia, China, Iran, India, Pakistan, Libya, and—hard to believe but true—Cuba).  They ignored all of the “lower tier” issues and Third World countries,[28] and also focused only on very big threats, not on very big opportunities for peaceful preventive measures where a few dollars invested in the 1970’s might have eradicated Anti-Immune Deficiency Systems (AIDS) or dependency on Middle Eastern oil.  This was of course in keeping with policy preferences, and even when the CIA did excellent work (for example, accurately forecasting the global AIDS epidemic), it could safely be ignored because its work was not available to the public or even to most Members of Congress.  A very important consequence of this narrow focus was the complete failure to ensure that all of the sources of national power—diplomatic, informational, military, economic (DIME)—were funded, acquired, fielded, and applied in a coherent and timely manner.  The entire military-industrial-intelligence complex has been skewed toward a heavy metal military—a few big platforms or big organizations—that are only relevant 10% of the time.  We are completely unprepared for, we are not trained, equipped, or organized, for small wars, waging peace, or homeland defense.  This is still true—truer that ever—in the aftermath of 9-11 and the invasions and occupations of Afghanistan and Iraq.

3.  Local Now.  Finally, U.S. Intelligence (and many foreign intelligence communities) focused on the local now instead of the global future.  “Current Intelligence” dominated the President’s Daily Briefing (PDB), and over time longer-term research fell by the wayside.   This problem was aggravated by a draconian editing process in both the national civilian and theater or service level military, where a twelve month research project could be subject to eighteen month editing cycles, such that the work was out of date or thoroughly corrupted by the time it was finally released to a relatively limited number of policymakers.  With most of the intelligence products being released in hard-copy, or messages that were printed out and not saved electronically, the overall impact of U.S. intelligence production, and especially Codeword production, must be judged as marginal.[29]

OSINT is “the rival store.”[30]  Whereas I spent the first eighteen years of my campaign to foster an appreciation on OSINT and focusing on the urgency of integrating OSINT into secret sources toward improved all-source analysis, I plan to spend the next eighteen years burying 80% of the classified world.  They are too expensive, too irrelevant, and pathologically anti-thetical to the new and correct Swedish concept of Multinational, Multiagency, Multidisciplinary, Multidomain Information Sharing (M4IS).[31]  OSINT analysis will in the future be the benchmark by which classified sources and methods are judged to be relevant and cost-effective, or not.  The Director of National Intelligence (DNI) has chosen to remain focused on secrets for the President.  So be it.  OSINT, from a private sector and non-governmental foundation, will capture all the other consumers of intelligence.  The day will come when “clearances” are severely devalued and open source access—international open source access in all languages all the time—is ascendant.  The DCI must serve all levels of the government, all seven tribes, and must balance between open and closed sources so as to inform decision-makers—and their publics—in order to preserve and enhance the long-term national security and prosperity of the USA.  Secret sources and methods—and the existing military—have demonstrably failed in both regards.

Analytic tradecraft notes are available online and should be consulted.[32]  All-source analysts should not be hired until they have first proven themselves as masters of all open sources in all languages relevant to their domain and not be considered for mid-career hire unless they are one of the top 25 cited authorities in the field.  They must know how to leverage their historian, their librarian, and the Internet.  They must know how to identify and interact with the top 100 people in the world on their topic, regardless of citizenship or clearances.  Finally, they must understand that they are—and must be trained to be—managers of customer relations and requirements definition; of open sources; of external experts; and of classified collection management.  Analysts must know and practice the “new rules” for the new craft of intelligence, with specific reference to being able to actually do forecasting, establish strategic generalizations, and drill down to the neighborhood and tribal levels, not simply hover at the nation-state level.[33]

Covert Action

Covert action consists of agents of influence, media placement, and paramilitary operations.  Covert action assumes two things that may once have been true but are no longer true: that an operation can be carried out without its being traced back to the USA as the sponsor; and that the fruits of the operation will be beneficial to the USA.  In each of these three areas, the USA has acted with great disdain for the normal conventions of legitimacy, accountability, morality, and practicality, and today the USA is suffering from what is known as “blow-back”—it is reaping the dividends from decades of unethical behavior justified in the name of national security, but unfounded upon any substantive grasp of long-term reality.

1.  Agents of influence are individuals bribed covertly who are charged with getting their governments or organizations to pursue a course of action that the USA deems to be necessary, but which may not be in the best interests of the indigenous public or its government.  Regardless of what one may think of the local country and its government and public, what this really means is that agents of influence are responsible for disconnecting local policies from local realities, and imposing instead a reality or choice selected by the US Government.  This is inherently pathological.  There are certainly some success stories—support to Solidarity in Poland, for example, but this was a capitalization on the fall of communism, not the cause.

2.  Media placement uses individuals, generally foreign journalists, who are bribed covertly to create and publish stories that communicate an alternative view of reality, one sanctioned by the US Government but generally at odds with the actual facts of the matter.  There is a constructive side to media placement, for example the promulgation of information about atrocities committed by dictators or Soviet forces, but generally the US Government supports most of the dictators it deals with, and reserves this tool for deposing individuals that dare to oppose predatory immoral capitalism or virtual colonialism.  Consequently, most media placement activities consist of propaganda seeking to manipulate rather than deliver the truth.  Media placement by spies should not be confused with Public Diplomacy by diplomats or Strategic Communication by the military—the latter two are overt truth-telling missions, although mis-guided practitioners may occasionally stray into propaganda and the manipulation of the truth.[34]

3.  Paramilitary operations are not only direct assaults on the sovereignty of other nations, but they tend to bring with them black markets, drug running, money laundering, corruption, and the proliferation of a culture of violence and the small arms with which to do indiscriminate violence.  The Phoenix program of assassinations in Viet-Nam, the support to the contras and the mining of the Nicaraguan harbors (an act condemned by the World Court), the arming of the Islamic fundamentalists for jihad in Afghanistan, join the planned overthrows of the governments of Chile, Guatemala, Iran, as causes of long-term and costly “blow-back.”  Of all of these, Iran is the most interesting.  Had we allowed the nationalization of the oil in Iran and the fall of the Shah, we might today have both a non-fundamentalist Iran as a bulwark against the radicals from Saudi Arabia, but we might also be less dependent on oil, and less subject to the whims of the extraordinarily corrupt Saudi regime and its US energy company allies.

OSINT is the anti-thesis of all three forms of covert action.  As David Ignatius noted so wisely in the 1980’s, overt action rather than covert action delivers the best value in both the short and the long run.  Promulgating the tools for truth—cell phones, wireless access, access to the Internet—is a means of fostering informed democracy and responsible opposition.  It is also a means of creating stabilizing indigenous wealth.  OSINT provides a historical and cultural foundation for achieving multi-cultural consensus that is sustainable precisely because it is consensual.  As Jonathan Schell documents so well in The Unconquerable World: Power, Nonviolence, and the Will of the People, there are not enough guns in the world for force our way or protect our borders.[35]  Only by fostering legitimacy, morality, charity, and full participation of all, can we stabilize the world to the mutual benefit of the USA and the rest of the world.

OSINT, in addition to being vastly superior to covert action as a means for establishing reasonable goals that are sustainable over time, is also very well suited to documenting the extraordinary costs of historical covert actions.  Only now is the public beginning to understand the lasting damage caused by the US sponsorship of assassination attempts against Fidel Castro, capabilities that were ultimately turned against the unwitting President, John F. Kennedy, and his brother Robert.  We have sacrificed our national values, and our international credibility at the alter of covert action, and we are long overdue for a deep “truth and reconciliation” commission that evaluates the true costs of covert action, and that then defines much more narrowly the conditions and protocols for engaging in covert action in the future.

Counterintelligence

Strategic counterintelligence is completely distinct from tactical counterintelligence.[36]  In strategic counterintelligence, one is looking for emerging threats at the strategic level, not individual penetrations of specific organizations.  This is an area where OSINT should, but does not, shine.  The US Intelligence Community—and consequently the US policy community—have completely missed the end of cheap oil, the end of free water, the rise of Bin Laden and the rise of pandemic disease, even global warming, precisely because national counterintelligence was focused obsessively on penetrating foreign security services, and not on the strategic environment where natural and other threats of omission and commission were to be found.  There are three areas where strategic counterintelligence can benefit considerably from comprehensive OSINT, inclusive of the digitization and statistical analysis of all available historical information.

1.  National education.  Thomas Jefferson said that “A Nation’s best defense is an informed citizenry.”  This is absolutely correct, and even more so today, when central bureaucracies are no match for agile networked transnational groups.  The USA has failed to understand the strategic implications of its lack of border control, its mediocre educational system designed to create docile factory workers, and the trends toward obesity, insularity, and indifference that characterize the bulk of the population today.  We have gone hollow for lack of focus.

2.  Environment.  The Singapore military was stunned by the emergence of Severe Acute Respiratory Syndrome (SARS), but unlike the U.S. military—they got it.  They realized they were responsible for defending Singapore against all threats, not just man-made or man-guided threats, and added national health and border security against airborne, waterborne, and human or animal borne diseases, to their charter.  Similarly, the Singapore police have an extraordinarily nuanced and enlightened understanding of their global and regional information needs and responsibilities in relation to deterring and resolving all forms of crime impacting on Singapore.  In the USA, and globally with dire consequences for the USA, there are threats associated with the environment and how it changes (including water, energy, and raw material resources) that are simply not understood, not acknowledged, and not being acted upon responsibly by any US Administration, be it Democratic or Republican.[37]

3.  Ideology.  There are two ideological threats to US security today, one external, the other internal.  The two together are very troubling.  Externally, the radical and violent fundamentalist stream of Islam has been armed and energized by jihad in Afghanistan, in Chechnya, and in Iraq.  Other small jihads in Indonesia, the Philippines, and southern Thailand, as well as selected locations in Muslim Africa, add to this threat.  Internally, US Christian fundamentalists—these are the people that graduated from rote reading of the Bible to the Left Behind fiction series—have assumed a terribly excessive importance in extremist Republican circles, in part because the Texas corporate energy interests chose to make common cause with them.  The Middle East, oil, and the almost cult-like extreme religious right, have hijacked American democracy.  The American left, nominally but not intelligently led by the Democratic party (which is as corrupt as the Republican party, but more inept), meanwhile, abandoned faith and God and the sensible calming effect of religion as a foundation for community and ethics.[38]  The American ideology of capitalism has also been corrupted.  Immoral predatory capitalism, and pathologically inept formulas for “developmental economics” as imposed on failed states by the International Monetary Fund (IMF) and the World Bank, have given rise to populism and other forms of indigenous resistance now witting of the collusion between their corrupt elite and immoral foreign capitalism that are in combination looting the commonwealth of many peoples.[39]

In all three of these cases, OSINT has an extraordinary role to play.  Under the leadership of Congressman Rob Simmons (R-CT-02), a moderate Republican with an extraordinarily deep background in both intelligence and on the Hill, the campaign continues for a national Open Source Agency funded at $3 billion per year, under the auspices of the Department of State (as a sister agency to the Board of Governors that controls the Voice of America and other public diplomacy outlets).  However, fully half the budget is intended to fund fifty Community Intelligence Centers and networks across the country (each receiving $30 million at Full Operating Capability or FOC).  These centers are needed for two reasons: first, to provide 119 and 114 numbers for citizen mobilization (119 alerts all cell phones within a 5 kilometer radius) and citizen neighborhood watch inputs (114 receives cell phone photos, text messages, any form of information, all with geospatial and time tags); and second, to serve as dissemination nodes for transmitting to all schools, chambers of commerce, churches and synagogues and mosques, labor unions, civil advocacy groups, and so on, the wealth of “real world” information to be collected, processed, and shared, free via the Internet, by the Open Source Agency.  This will impact very favorably on the environment, as these centers will help citizens at the county, state, and regional levels understand, with precision, where each of them stands with respect to access to clean water, alternative energies and related lifestyle choices, and global threats to their children and grandchildren based on easy access  to the actual U.S. federal budget in relation to real world threats and needs.  Militarism will be reduced, poverty and disease will be eliminated, and the USA can rejoin the community of nations as force for good.  Finally, all competing ideologies can be subject to scrutiny and understanding, and the majority of Americans who are not part of the nutty right can come together consensually to limit the damage these people can do to the Republic, while also holding their political and corporate allies accountable for serving America as a whole rather than a lunatic fringe element.

Dramatically redirecting national intelligence toward OSINT will substantially reduce the cost of secrecy, estimated by the Moynihan Commission as being on the order of $6 billion a year (probably closer to $15 billion a year today),[40] and will also eliminate perhaps 70% of the costs associated with establishing the trustworthiness of individuals being considered for clearances.  The security and clearance system of the U.S. Government is broken beyond repair.  Not only does it take over two years from most investigations to be completed, but they are generally sub-standard investigations that go through the motions and generally do not detect basic aberrations, such as a fascination with child pornography and online molestation of children, as was the case recently with a senior manager in the Department of Homeland Security.  The fact is that most sheriffs and other state and local officials are not “clearable” for a variety of reasons, and we may as well recognize that not only is OSINT better suited for most national intelligence information sharing, but we really do not need most of the grotesquely expensive and dysfunctional Top Secret“ compartments” (over 400 of them, half in the civilian world and half in the military world) and all the attendant costs, including the costs of ignorance stemming from compartmented information not being shared.  At least at the strategic level, we need a national intelligence system where we are less concerned about betrayal from within, and more focused on emerging strategic threats to our long-term security and prosperity, threats that must not be limited to man-made capabilities, but include animal borne diseases and other environmental conditions that tend to be shut out from national security decision processes.

Accountability,  Civil Liberties, and Oversight

As all of the preceding sections should have made clear, OSINT is the essential contributing factor to dramatically improving the accountability and oversight of the U.S. Intelligence Community and the policymakers, acquisition managers, and operational commanders that respond to White House Direction.  OSINT is also a means of dramatically enhancing not just civil liberties, but civic engagement in the practice of democracy.  By providing citizens at every level with structured OSINT on any issue for any zip code or other geographic grouping, and by making it possible for citizens to immediately connect with other like-minded citizens and with accountable officials, OSINT in practice is an enabler of a new form of constant engaged informed democracy.  Civil liberty infractions will be broadcast or podcast, rapidly aggregated, and civil pressure brought to bear.  By harnessing citizens are part of the “home guard” and empowering them with immediate and understandable access to indications and warning information, we will dramatically improve the reporting of relevant information, and—through the Community Intelligence Centers—be able to process, make sense of, and act on—or discount—the “bottom up” dots that I am convinced will comprise at least 50% of the relevant dots needed to prevent the next 9-11.

It is also important to emphasize that at the strategic level, we need to be concerned not just with accountability and oversight of secret intelligence, but with the much larger issue of whether Congress and the Executive are being responsible in representing the public interest.  For this reason we include very brief but vital sections at the end of this chapter on OSINT and electoral reform, governance reform, and budgetary reform.  OSINT is the ultimate resources for citizens to hold their government accountable, and to protect their civil liberties over time.

Strategic Warning

Although CIA has done some fine work on global threats, and I particular like the work done under John Gannon as Assistant Director of Central Intelligence for Analysis & Production (Global Trends 2015, which led to Global Trends 2020),[41] on balance the U.S. Intelligence Community has failed abysmally at strategic warning because of some fundamental operational and philosophical failures.[42]

Operationally, despite fifty years of extraordinarily generous funding for multi-billion dollar satellite systems, the U.S. Intelligence Community still cannot do wide area surveillance, real-time change detection, or “the last mile” inclusive of seeing into urban area, under jungle canopy, and into the deep ravines of mountainous terrain.

Philosophically U.S. Intelligence has been a disaster in strategic terms.  The cult of secrecy limited “intelligence” to “secrets for the President” and left everyone else, from Cabinet-level leaders to military acquisition manager and operational commanders, to Governors and Mayors, completely without “decision-support.”  Perhaps worse, the U.S. Intelligence Community has refused to recognize the seven tribes of intelligence, shutting out, for the most part, state and local officials with overseas knowledge, business travelers, academics, non-governmental observers, journalists, labor union leaders, religious travelers, and so on.  The obsession with government secrecy over public sharing has cost this Nation fifty years of time—the one strategic factor that can not be bought nor replaced[43]—and at least 3 billion souls of good will.  U.S. Intelligence is a small part of the overall federal government, and it merits comment that most of our problems today cannot be blamed on U.S. Intelligence as much as on a corrupt Congressional and Executive all too eager to ignore, for example, the Peak Oil warnings of 1974-1979 in order to keep the bribes going and the public docile.  This is not, however, to excuse the U.S. Intelligence Community, in as much a focus on OSINT from 1988 onwards would have done much to illuminate and correct the policy errors that benefited from secrecy, obscurity, and public inattentiveness.

Strategic Sharing

The U.S. Intelligence Community is incapable today, five years after 9-11, of creating a single consolidated watch list of suspected terrorists.  The U.S. Government as a whole is incapable of sharing everything that it knows for lack of collaborative mindsets, willing management, interoperable systems, and coherent data sets.  There are three primary impediments to the U.S. Intelligence Community ever being able to share readily:

1.  High Side Security.  The obsession with security is occasioned in part by the fact that the secret intelligence world, even though it has “compartments,” has never learned to disaggregate secret from non-secret information.  Everything is stored at the “high side,” at the highest possible level of security, meaning that nothing can be shared with anyone who is not cleared for the highest level of security, however unclassified the information might be.

2.  Third Party Rule.  The secret world has for decades operated under a “third party rule” that prohibits the sharing of any information received from one party, with another party.  This rule is extremely detrimental to multi-lateral sharing, and imposes enormous time, manpower, and dollar costs when something needs to be shared and the sharing must be coordinated.  The default condition of the secret world is “do not share.”

3.  Legacy Systems.  As John Perry Barlow noted in an article in Forbes,[44] if you want to see the last remnants of the Soviet Empire, go visit the CIA and look at their computer systems.  The U.S. Intelligence Community as a whole is still mired in 1970’s technology managed by 1950’s mind-sets, totally out of touch with 21^st Century information networks, both machine and human.

OSINT is going to be the catalyst for M4IS and strategic sharing.  OSINT is the only discipline that can easily distribute the collection, processing, and analysis burden across all coalition nations (e.g. the ninety nations comprising the U.S. Central Command coalition), and also the only discipline whose products can easily be shared with non-governmental organizations as well as state and local authorities all over the world who will never qualify for “clearances.”  It will be our challenge in the next eighteen years to develop an alternative global intelligence community that relies almost exclusively on “good enough” open sources, and that consequently forces the secret world into proving its “added value” in relation to cost, risk, and time, on every topic, every day.

Emerging Prospects

Apart from increased public access to the Internet, inclusive of electronic mail, the deep web, and the dramatically increased availability of free multi-media communications and information sharing capabilities, several factors are supportive of a displacement of secret sources and methods by open sources and methods:

1.  Digitization.  It is a mistake to believe that all relevant information is being digitized today.  Tribal histories (e.g. those from Iraq) and vast quantities of important information are still being produced in Industrial Era mediums, and Friday sermons by Islamic imams as well as the sermons by all the other faiths, are not part of the digital revolution.  In strategic terms, however, digitization is extremely important for three reasons:

• Most current information from mainstream and niche media as well as individual publishers and bloggers, in all languages, is now available digitally.
• Historical information, including policy and financial statements of great importance to specific nations, industries, organizations, and tribes, can now be affordably and effectively digitized.
• Hand-held devices are rapidly becoming a primary means of collecting and sharing information, with imminent prospects of being able to harness, selectively, all that any group of individuals can see and hear and think, and is willing to upload as needed.

2.  Visualization.  Digital information, including historical information, can now be visualized, not only in relation to content analysis and links between paragraphs and among individuals, but in relation to a geospatial foundation such as Google Earth provides in rudimentary but quite compelling terms.  This is moving OSINT well beyond secret sources and methods because it can draw on a much greater body of information and expertise in real time, and apply all modern machine analytic tools with fewer security, legal, and policy constraints.  The centralized unilateral secret bureaucracies are losing ground—rapidly—to distributed open multinational networks.

3.  Peer-to-Peer (P2P).  “Ground truth” is taking on a whole new meaning as individuals exercise the power to share complex information directly with one another, eliminating the intermediary journals, web sites, and government or media offices that in the past have played the role of editor, judge, and broker of meaning and value.

The power of OSINT at the strategic level can simply not be exaggerated nor underestimated for the simply reason that it harnessed the distributed intelligence of the Whole Earth, in real time as well as in historical memory time, across all languages and cultures.  There isn’t a bureaucracy in the world that can match its networked power.  To drive that point home, consider the game of baseball.  In today’s secret environment, government bureaucrats accustomed to unlimited budgets and secret methods continue to try to win the game by bribing a player (Clandestine Intelligence), putting a “bug” in the dug-out (Signals Intelligence), trying to “sniff” the direction and speed of the ball (Measurements & Signatures Intelligence), or taking a satellite picture of the field every three days (Imagery Intelligence). The new craft of intelligence requires integrates the audience. It uses the collective wisdom of all the participants. It encourages the crowd to participate. Open source intelligence harnesses what everyone sees and knows.  It changes the rules of the game.  Any catch in the stands is an out.  That is how we win against asymmetric opponents who know our Achilles’ heels all too well.

OSINT and Electoral Reform

The USA is a Republic.  An extraordinary characteristic of Republics is that voters have the power to dissolve the government should it become so ineffective or destructive as to warrant its termination.  The Constitution, and the voters, are the foundation of the American democracy, not the three branches of government.  If the Executive is mendacious, the Congress is corrupt, and the Judiciary is so unrepresentative of the values of the people as to be a mockery of Justice, then the public has the power to change the rules of the game for elections.  It is OSINT that can be used by citizens to break away from the Republican and Democratic parties, and develop new networked means of demanding minimalist changes such as suggested by Ralph Nader and enhanced by the author: voting on week-ends so the poor don’t lose work; restoring the League of Women Voters as the arbiters of multi-party debates; demanding that Presidential candidates announce their Cabinets in advance of the election, and including at least the Secretaries of Defense and State, and the Attorney General, in Cabinet-level debates; applying the instant run-off concept to ensure a true majority election; and of course ending gerrymandering and corporate funding for any elected official.

OSINT and Governance Reform

            Government at the Federal level has become incompetent, and is wasteful of the taxpayer dollar for two reasons: special interest corruption both in Congress (bribery) and in the Executive (revolving-door favoritism); and an industrial era structure that is largely disconnected from reality to the point that ideological fantasy can supplant a reasoned policy process.  At a minimum, the Republic needs a Coalition Cabinet and some means of assuring the citizenry that Presidents will not be able to simply appoint cronies from their own party; the Executive needs to be restructured to provide for integrated policy development, not just national security policy development; strategic planning focused out seven generations (over 200 years) must be demanded and be publicly transparent and accountable; and the fundamentals of national power must be mandated: quality education for all, health care for all, and an end to poverty at home.  Presidents and their teams must be elected for their ability to govern rather than campaign.  OSINT will make all of this possible, sooner than later if a national Open Source Agency is created as a new fourth branch of government, independent of Congress and the Executive, with a lifetime appointment for its Director, and a Board of Directors comprised of former Presidents, Leaders of the Senate and House, and retired Supreme Court Justices.

OSINT and Strategic Budgetary Reform

Finally, we come to budgetary reform.  OSINT has already made it clear that we have a Department of Defense costing $500 billion a year (not counting the cost of the war in Iraq) that is relevant to only 10% of the threat (state on state warfare), that is largely incompetent at small wars and homeland defense, and that we are, as a Republic, not investing properly in peaceful preventive measures inclusive of the spread of participatory democracy and moral capitalism.  The RoI on our “Big War” military is not only not there, the existence of that Big War force leads ignorant Presidents and their mendacious Vice Presidents to seek out wars as an option for capturing “cheap” oil (never mind the cost in blood, spirit, and treasure).  The American Republic specifically, and all other countries, are long over-due for what I call “reality-based budgeting.”  OSINT will restore sanity and sensibility to the public treasury and how it is applied.

There is, in the immortal words of Arnie Donahue[45] in 1992, “plenty of money for OSINT.”  There is also plenty of money for participatory democracy and moral capitalism.  Our problem has been that we have allowed the mandarins of secrecy to pretend to be informing the President, rather narrowly and very expensively, while failing to demand that the Republic develop a public intelligence capability suitable for directing public policy and public spending in an intelligent sustainable manner.

9-11, the Iraq War, and the varied accomplishments—or crimes—of the Bush Administration may stand in history as a bright turning point in the history of the Republic.  One doubts that anything less might have awakened the somnolent public.

ENDNOTES