Stephen E. Arnold: Disappearance of Scientific Big Data — and a Solution

Collective Intelligence, Commercial Intelligence
Stephen E. Arnold
Stephen E. Arnold

Scientific Data Going, Going, Gone

Years ago I did a report for a sci-tech database publisher. I wrote up the results of a number of on site visits at research universities. I reported that there was no mechanism to preserve researchers’ data. The reason was pretty obvious: Research facilities at universities are less important than sports teams, business activities, and fund raising. When the researcher moved on, the data just sat somewhere until there was a housecleaning or a hard drive wiped. If financial support disappeared, none of the facilities I visited had an old school records management system in place. If a researcher took the data with him or her, those data may or may not have been managed in a thoughtful way. On to the next grant, tally ho.


I was not surprised to read “Researchers Say Valuable Scientific Data Disappearing at Alarming Rate.” (If this link 404s you are on your own, you Google search experts, you.) Here’s the passage I noted:


While all data sets were available two years after publication, the odds of obtaining the underlying data dropped by 17 per cent per year after that, they reported.


So when did I do my report for the grand viziers at the sci-tech outfit? It was 1998. How about that data half life?


Why didn’t the database publishing company fund a program to archive research data? In my experience, the notion of capturing raw data was too far from the familiar path of summarizing articles and selling access to those summaries to libraries.


Too late now? Maybe. This is one more example of “if it is not available, how can one search for information and data”? Will the sci-tech database company remember my report and diagram? Nah. The familiar way is the better way I suppose.


Stephen E Arnold, December 20, 2013

Click on Image to Enlarge
Click on Image to Enlarge

Phi Beta Iota: Ever since the concept of a “Smart Nation” was coined in the early 1990’s we have had a benchmark concept for creating, sharing, and preserving big data. It can safely be said that the US Government has not just been ignorant on this matter, it has been downright stupid, corrupt, and dangerous. An Open Source Agency (OSA) would solve this problem, among many others.

See Also:

Big Data @ Phi Beta Iota

Open Source Agency @ Phi Beta Iota

Smart Nation @ Phi Beta Iota

Opt in for free daily update from this free blog. Separately The Steele Report ($11/mo) offers weekly text report and live webinar exclusive to paid subscribers, who can also ask questions of Robert. Or donate to ask questions directly of Robert.