30 Resources to Find the Data You Need
Let's say you have this idea for a visualization or application, or you're just curious about some trend. But you have a problem. You can't find the data, and without the data, you can't even start. This is a guide and a list of sources for where you can find that data you're looking for. There's a lot out there.
Universities
Being a graduate student, I always look to the library for books and resources. Many libraries are amping up their technology and have some expansive data archives. Many statistics departments also tend to keep a list of data somewhere.
DATA SOURCES:
- Data and Story Library – An online library of datafiles and stories that illustrate the use of basic statistics methods, from Carnegie Mellon
- Berkeley Data Lab – Part of the UC Berkeley library system. Hey, they've even on Twitter.
- UCLA Statistics Data Sets – Some of the data that UCLA stat uses in their labs and assignments.
News Sources
I'm sure you've seen a graphic in the paper or I guess more likely, on a news site, and wondered about another aspect of the data. Major news organizations always put their sources somewhere on the graphic or are mentioned in the accompanying article. It's usually not a direct link, but a quick online search will get you to the right place. Sometimes, you'll have to email someone to get the same data, but those people are usually happy that you're interested in their data or analysis.
DATA SOURCES:
- The New York Times – They also have several data-rich APIs
- Wall Street Journal
- Guardian Datablog – Provides a lot of free-to-use data via Google spreadsheets.
Geographic Data
Got some mapping software, but no geographic data? You're in luck. There are plenty of shapefiles, etc. at your disposal.
DATA SOURCES:
- TIGER – From the US Census Bureau, detailed data about roads, railroads, rivers, and zipcodes. Probably the most extensive you're going to find.
- OpenStreetMap – One of the best examples of data and community effort.
- Geocommons – Both data and a map maker.
- Flickr Shapefiles – Boundaries as defined by Flickr users.
Sports
America loves its sports, and thus, has decades of sports data. You'll find it on Sports Illustrated or the sports organizations' sites, but you'll also find more on sites dedicated to the data.
DATA SOURCES:
World
There are several noteworthy international organizations that keep data about the world, mainly health and development indicators. It does take some sifting though, because a lot of the data sets are pretty sparse. It's not easy to get standardized data across countries with varied methods.
DATA SOURCES:
- Global Health Facts
- UNdata – Most of the data I used for Progress came from this data search engine from the United Nations.
- World Health Organization
- OECD Statistics
Government and Politics
With the new administration, there's been a fresh emphasis on data and transparency, so there are lot of government organizations that supply data. They've been doing this for a while, but with the launch of data.gov, much of the data is finding itself in one place. There are also plenty of non-governmental sites that aim to make politicians more accountable.
DATA SOURCES:
- Census Bureau – Incredibly important data about the country with more effect on your life than you probably know
- Data.gov
- DataSF – San Francisco recently launched their own data site. Hopefully, other cities follow suit. Check out the showcase.
- Follow the Money
- OpenSecrets – Interesting site MAPlight is powered by data from OpenSecrets.
General Sources
You're usually going to find the best data straight from the source, but there are lots of applications and sites that try to make all data easier to find or easier to access.
DATA SOURCES:
- Freebase – Free data and a community effort. For some types, the data are kind of sparse, but it continues to get better.
- Numbrary
- Many Eyes – More of a visualization and exploratory site than for data, but they do have a data section.
- Infochimps – Did you get your invite?
- Swivel
- Amazon Public Data Sets
- DBpedia – Allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data.
- Wikipedia – Lots of HTML tables. Copy and paste in Excel.
Get it From an API
Plenty of sites and applications make their data freely available via APIs. Twitter has an API (duh). Google has lots of APIs. Yahoo does too. So on and so forth. Visit Programmable Web for a detailed catalog of what's available.
Scrape the Data
When all else fails, you can always find a site that serves the data through HTML pages, and then scrape the data with Python, javascript, or whatever language you're comfortable with. I use Python with the Beautiful Soup library that makes parsing pretty easy.
For example, I scraped weather data from Weather Underground a while back (although I don't think the script works anymore). I also used it gather television sizes from CNet.
I'm still figuring out how to scrape AJAX-based sites though. I'd be happy to hear any tips from anyone who has experience with that.
Did I miss anything? Where do you get your data from?
Data: Where can I get large datasets open to the public?
Quora
Here are many of the links mentioned so far:
http://archive.ics.uci.ed
http://aws.amazon.com/dat
http://crawdad.org/
http://data.austintexas.g
http://data.cityofchicago
http://data.govloop.com
http://data.gov.uk/
http://data.medicare.gov
http://data.seattle.gov
http://data.sfgov.org
http://data.sunlightlabs.
https://datamarket.azure.
http://developer.yahoo.co
http://econ.worldbank.org
http://en.wikipedia.org/w
http://factfinder.census.
http://ftp.ncbi.nih.gov/
http://gettingpastgo.socr
http://googleresearch.blo
http://books.google.com/n
http://linkeddata.org/
http://medihal.archives-o
http://public.resource.or
http://rechercheisidore.f
http://reddit.com/r/datas
http://snap.stanford.edu/
http://timetric.com/publi
https://wist.echo.nasa.go
http://www2.jpl.nasa.gov/
http://www.archives.gov/r
http://www.bls.gov/
http://www.crunchbase.com
http://www.dartmouthatlas
http://www.data.gov/
http://www.datakc.org
http://www.delicious.com/
http://www.factual.com/
http://www.freebase.com/
http://www.google.com/pub
http://www.guardian.co.uk
http://www.infochimps.com
http://www.kaggle.com/
http://www.nationalarchiv
http://www.nyc.gov/html/d
http://www.ordnancesurvey
http://www.philwhln.com/h
http://build.kiva.org/
http://www.imdb.com/inter
http://imat-relpred.yande
http://www.dados.gov.pt/p
http://dbpedia.org
