Several intelligence community initiatives to develop improved tools for data search, analysis and fusion were described in the latest report to Congress (pdf) from the Office of the Director of National Intelligence on data mining.
A new program called DataSphere is intended “to aid in the discovery of unknown terrorism relationships and the identification of previously undetected terrorist and terrorism information” through analysis of communication networks and travel patterns.
A continuing program called Catalyst seems to be a glorified search engine that “will enable data fusion/analytic programs to share disparate repositories with each other, to disambiguate and cross-correlate the different agencies' holdings, and to discover and visualize relationship/network links, geospatial patterns, temporal patterns and related correlations.”
Although these and other initiatives do not yet constitute or engage in “data mining,” they were described in the new report “in the interest of transparency,” ODNI said. See “2010 Data Mining Report,” Office of the Director of National Intelligence, April 2011.
Comment and Two Complex Links Below the Line
Phi Beta Iota: There are four people in the IC with a clue on this topic that we know of, two still active (Andy Shepard and Carol Dumaine) and two retired (Charlie Allen and Joe Markowitz). Sadly, the DNI has no authority and hence no ability to actually scorch this together. The National Counterterrorism Center (NCTC) is a global joke, one of the most pathetic kludges of the unwitting and witless we've ever encountered. Over 80 databases, each with its own password and security protocols, each taking up to three days to reinstate access if a password is lost. Put that on top of the fact that for $80 billion a year (actually much more) we collect less than 20% of what is relevant and produce “at best” 4% of what a major commander needs and nothing for everyone else. All of this was known to Andy Shepard, Diane Webb, Dennis McCormick, Gordon Oehler, Bill Donnelly, and Robert Steele in 1989. The founding members of the Advanced Information Processing and Analysis Steering Group (AIPASG) under Terry Kees found over 20 such projects across the IC, each about $10 million, each with different statements of work and each with different “sweetheart” contractors, none of whom talked to each other. We still do not have it because the US IC is corrupt across the board, and lacks both maturity and integrity. Deja vu.