Stephen E. Arnold: Information Black Holes + Findability

Commerce, Corruption
Stephen E. Arnold
Stephen E. Arnold

Information Black Holes: Autonomy and Its Value Proposition

I follow two or three LinkedIn groups. Believe me. The process is painful. On the plus side, LinkedIn’s discussions of “enterprise search” reveal the broken ribs in the body of information retrieval. On the surface, enterprise search and content processing appear to be fit and trim. The LinkedIn discussion X-ray reveals some painful and potentially life-threatening injuries. Whether it is marketing professionals at search vendors or individuals with zero background in information retrieval, the discussions often give me a piercing headache.

One of the “gaps” in the enterprise search sector is a lack of historical perspective. Moderators and participants see only the “now” of their search work. When looking down the information highway, the LinkedIn search group participants strain to see bright white lines. Anyone who has driven on the roads in Kentucky knows that lines are neither bright nor white. Most are faded, mere suggestions of where the traffic should flow.

In 1999, I picked up a printed document called “Information Black Holes.” The subtitle was this question, “Will the Evolution of EIPs Save British Business £17 Billion per Year?” The author of the report was an azure chip consulting firm doing business as “Continental Research.” The company sponsoring the research was Autonomy. Autonomy as a concept relates to “automatic”, “automation,” and “autonomous.” This connotation is a powerful one. Think “automation” and the mind accepts an initial investment followed by significant cost reductions. Autonomy had a name and brand advantage from its inception. Who remembers Cambridge Neurodynamics? Not many of the 20 something flogging search and content processing systems in 2014 I would wager.

As you may know, Hewlett Packard purchased Autonomy in 2011. I doubt that HP has a copy of this document, and I know that most of the LinkedIn enterprise search group members have not read the report. I understand because 15 year old marketing collateral (unlike Kentucky bourbon) does not often improve with age. But “Information Black Holes” is an important document. Unwittingly today’s enterprise search vendors are addressing many of the topics set forth in the 1999 Autonomy publication.

What I find interesting is that in the 15 years between my seeing the document and yesterday’s scan of LinkedIn posts about the “value” of enterprise search, the arguments have not advanced. Most of those commenting about the value of enterprise search are less focused and less informed than Autonomy’s management was years ago.

I have a copy of the document, and I want to summarize some of its key points. The reason I am writing about “ancient” information for today’s 20-somethings is to fill a gap. If a search vendor wants to sell a search product, why not learn from the companies that have already explored the digital Yellowstone. “Content intelligence” is not a new concept, and it did not make it easier for Fulcrum or iPhrase to close deals. “Predictive analytics” is an important part of the Autonomy IDOL black box, and those numerical algorithms were not new at the end of the 1980s when Dr. Michael Lynch was working with signal processing to whip up a music synthesizer. The same may be said about “natural language processing” and most of the other so-called innovations search vendors and those seeking a new search engine perceive as revolutionary breakthroughs.

The Black Holes report is one of the touchstones for the assertion that information access delivers significant value to an organization. The assertions from 1999 are the same as those vendors make today. The lack of verifiable data continues. Autonomy’s Black Holes report makes clear how a search vendor has to dance around the barn to answer to a very fundamental question, “Does a search engine make the licensee more money?”

Let’s look at how Continental Research tackled the problem.

First, 1999 was the year of the portal. Portal would make an organization’s information assets available to system users. The portal was the 1999 precursor of Big Data or sentiment analysis. Software that aggregated an organization’s information assets would work just like a sorcerer’s wand, a silver bullet, and a magical habichuela. Autonomy had packaged its Digital Reasoning Engine (the precursor of the Autonomy Integrated Data Operating Layer) as the Autonomy Portal-in-a-Box. Autonomy did not state that its technology would yield £17 billion per year. Autonomy just asked the question, “Will the evolution of enterprise information portals save £17k billion per year?” Big difference. Today individuals hunger for something more concrete than a question. Hard evidence of the value of an investment over time in an enterprise search system is needed. The reason is that many organizations have spent significant sums on enterprise search systems and cannot tie that money to a specific payoff. The uncertainty flows from the chief financial officer to other employees in the organization. In most organizations with experience licensing five, six, and seven figure systems, search is just fuzzy, expensive, and a source of complaints. My thought is that today’s marketers should pay attention to what Autonomy did to deflect the “value” bullet.

Second, the 1999 Black Holes report nailed the problem of “exponential increases in the amount of digital data and information.” Isn’t that Big Data expressed in 1999 terminology? In fact, the report identified the digital data that was a challenge: Transactional data, documents, digital files, e-mail, video, and audio files. The Black Holes report stated:

There is concern over the volume of data that is buried deep within “information black holes”—company databases, file servers, and desktops….The crux of the issue for most businesses is whether they can organize, access, and use relevant information that should be “at their fingertips. These factors point conclusively to a need for greater order in, and better control of, information and data in organizations.”

I read this passage as a group of sentences that with minor editing could be used by most of the vendors selling enterprise search solutions today, January 4, 2014. Black Holes nailed Big Data, access, federation of disparate data sources, structured and unstructured information, and non-text content. Today’s marketers are recycling concepts that have been around a long time. Why are these marketers beating a 15 year old drum? The answer is easy, “Enterprise search technology has not delivered a solution.” A problem remains. The problem is growing. Like the technology used by most vendors, not much has changed. Progress has been difficult to discern.

This is the Autonomy Black Hole illustration. Notice that the diagram includes archives, employee knowledge, and real-time sources of information from outside the organization. This diagram is as timely in 2014 as it was in 1999. © Autonomy, 1999

The Black Holes document then dives into the cost of running a business without comprehensive information. The consulting firm apparently conducted an old-fashioned survey, not one of those Survey Monkey-Web polls. The data, like today’s surveys from azure chip consulting firms, have to be viewed with some caution.

I want to ignore the data, which point out that those in the sample, wasted time looking for information. This, I believe, is the genesis of the often-cited IDC “study” that quantifies the cost of information. Unfortunately, IDC, unlike Autonomy, did not bring the Dr. Michael Lynch brainpower to bear on the obvious—Wasting time is not good for a business. The big point, in my opinion, is that without an Agentware/DRE/IDOL system, employees duplicate work. The figure Autonomy’s researchers slapped on this key point was 36 percent. If the number was spot on, the op0erating costs of a business were inflated by one third. Remember. This was in the late 1990s. What consulting firm has updated the Black Hole figure? If you, gentle reader, please, use the comments section of the blog to provide some color for this outline. I quite like the Autonomy illustration for this point. Not many search vendors have hit this level since the Black Hole document.

The Black Hole document does a very good job of explaining the problem of increasing amounts of digital information. Here’s a quote from the survey:

Basically, I think that the information has become more sophisticated than the brains
looking at it.” — Respondent, Autonomy research June 199

I think this is timely, fresh, and highly pertinent to today’s 2014 market. In 1999, Autonomy pegged the increase in digital information as doubling for UK firms every 24 months. At the time of the Black Hole document, Autonomy was marketing mostly in the UK with only tentative probes into the much larger US market. The Black Hole document revealed that law firms were struggling, reporting, “Efforts to archive information are failing.” Today Recommind, a remarkably Autonomy-like system, is selling to law firms. The question, obviously, is, “Why didn’t Autonomy dominate the market?” and “Why hasn’t Recommind become the de facto standard for legal information access?” I have some theories, but it is clear from the Black Hole document that search and content processing were needed by law firms. What’s interesting is that the problem seems to remain unresolved. Perhaps no search system delivers the combination of factors necessary to create a Google-style dominant position in one market niche?

The Black Hole document then nails causes of the findability problem. Flash back to 1999. Now fast forward to 2014. Has this problem abated: email logjams.

The Black Hole document concludes with a question, “The end of information black holes?” The telling point is that as early as 1999, Autonomy had developed a positioning that put other vendors in one bucket—basic keyword search. Autonomy was defining a new bucket—information infrastructure. In one rhetorical master stroke, Autonomy set itself apart because only Autonomy can deal with “information black holes.” The report states:

Autonomy believes that enterprise information portals will save companies millio0ns by enabling them to create a single gateway to access and use information from hundreds of sources internally and externally. They [portals]  will enable users to access relevant content in an appropriate context immediately.

The Black Holes document pitches the “Portal-in-a-Box” solution. Autonomy would implement a solution that works like the Web, federates disparate content, and deliver personalized business information.

With some minor editing, this 1999 Autonomy document would be usable today.

Each time I review the collateral, I realize that Autonomy only 30 months after opening for business, had hit upon some themes that would allow the company to collect $11 billion from Hewlett Packard in 2011.

  1. Autonomy defined the argument for the value of search and content processing
  2. The Portal-in-a-Box solution eliminated the incredible complexity that other competitors did little to simplify for ultimate decision makers. Sure, Autonomy had complex diagrams. Autonomy addressed senior management in Black Holes, not the systems professionals.
  3. The rhetoric allowed Autonomy to define itself as delivering a solution that eliminated wasted time. Time is money. Therefore, less wasted time automatically allows the investment in Autonomy to pay for itself. Nifty, unlike the IDC-type analysis that chases the phantom of value in looking up information as an act in itself.

I don’t want to argue whether Autnomy Agentware, i3, DRE, and IDOL are better than other solutions. What is clear in this early document is that Autonomy had a better grasp of marketing search and content processing systems. Like it or not, Endeca was only one-tenth the size of Autonomy when HP inked the $11 billion deal. Endeca sold to Oracle for about $1 billion. Yep, that’s one tenth the price HP paid. The value of Autonomy was dictated by a market transaction.

My premise is that Autonomy was the pre-eminent marketer of search and content processing. The difference between Bayesian-Laplace, Monte Carlo, and Markov Chain methods in IDOL and the brute force, academic linguistic efforts of other vendors was secondary to marketing.

Autonomy was and until the 2011 sell out to HP, the champion of search and content processing marketing. If I have the energy, I will dig out one of my old reports about Autonomy and post it on the search history Web site. Today’s vendors and marketers may benefit from a deeper understanding of what Autonomy wrought.

Oh, the problem of finding information remains. But there’s one big difference between 1999 and 2014. Organizations have experienced the thrills and spills of enterprise search. Better marketing does not translate to better findability.

Stephen E Arnold, January 6, 2014

See Also:

Findability @ Phi Beta Iota