Enterprise Open Source Intelligence Gathering – Part 1 Social Networks
Posted by Tom on October 26, 2009 – 3:49 pm
Note: the direct web version is nearly unreadable. A full copy is therefore provided below.
UPDATE: You can now download my slide deck from SlideShare.
Next week I will be speaking at the 7th Annual Ohio Information Security Summit on “Enterprise Open Source Intelligence Gathering”. Here is the talk abstract:
What does the Internet say about your company? Do you know what is being posted by your employees, customers, or your competition? We all know information or intelligence gathering is one of the most important phases of a penetration test. However, gathering information and intelligence about your own company is even more valuable and can help an organization proactively determine the information that may damage your brand, reputation and help mitigate leakage of confidential information.
This presentation will cover what the risks are to an organization regarding publicly available open source intelligence. How can your enterprise put an open source intelligence gathering program in place without additional resources or money. What free tools are available for gathering intelligence including how to find your company information on social networks and how metadata can expose potential vulnerabilities about your company and applications. Next, we will explore how to get information you may not want posted about your company removed and how sensitive metadata information you may not be aware of can be removed or limited. Finally, we will discuss how to build a Internet posting policy for your company and why this is more important then ever.
Leading up to my talk at the summit this series of posts will focus on several of the main topics of my presentation. I plan on referencing these posts during the presentation so attendees can find out more information about a specific topic that will be discussed. I will touch on the following main points in this series: Part 1 – Gathering intelligence on social networks, Part 2 – Gathering intelligence from blogs/message boards/document repositories, Part 3 – Putting together a simple monitoring program/toolkit and creating a Internet postings (social media) policy for your company.
This first post in the series will focus on gathering intelligence on social networks. The topic of gathering intelligence from social networks will be looked at in two ways. First, through the eyes of the penetration tester or attacker. Second, from a monitoring perspective relative to the enterprise and business.
What is OSINT?
Open Source Intelligence (OSINT) is basically finding publicly available information, analyzing it and then using this information for something. That something can be extremely valuable from the eyes of an attacker. For a fantastic overview of how OSINT is used specifically from a penetration testing perspective I suggest you check out the presentation that Chris Gates recently did at BruCON. Chris goes into detail and provides good examples on how OSINT can be used in gathering intelligence on a network infrastructure as well as how to profile company employees. All of the techniques Chris talks about should be used in a penetration testing methodology.
Why look for OSINT about your company?
I have found that OSINT is surprisingly often overlooked by most businesses from a security monitoring perspective. If a company does any monitoring of public information at all it is usually found in your public relations and/or marketing groups. These groups traditionally don’t look for things that could be used to target or profile an organization. The same information that is being viewed by your PR/Marketing department needs to be looked at by your in house information security professionals. Specifically, I suggest people in your information security department with an “attacker mindset” look at this OSINT. This could be people on an internal penetration testing team or someone involved with the security assessments in your organization. You should really ask yourself: If you don’t know what information is publicly available about your company…how can you properly defend yourself from attack?
OSINT and Social Networks
Social networks have recently become the 4th most popular method for online communication (even ahead of email) today. If you are not looking for OSINT on social networks you are potentially missing major and vital pieces of information. Having said that, searching for OSINT on social networks brings its own challenges and needs to be looked at slightly different then looking at other forms of OSINT. For example, you might find that searching for information on social networks like Facebook different because there is both private and public information. Facebook as an example has a built in search feature “behind” a valid login id and password. Searching Facebook in this manner can yield better results then just going to Google or using a specific social network search engine (I’ll talk more about Facebook below).
1. Social Network Search Engines
There are lots of different search engines that specifically look for “public” information on some of the major social networks. The disadvantage about these types of search engines is that they only pull public information that can be easily indexed. Private information like the Facebook example above cannot be indexed without violating the TOS (Terms of Service) even though there are tools like Maltego that can have transforms written to “page scrape” this information (more on that in the Maltego section below). Here is a list of social network search engines that I recommend you check out to search for this type of public information (there are more…this is just the list I use). While there are other ways to search specific social networks (like search.twitter.com or FriendFeed) the list below just mentions search engines that search multiple networks:
Wink http://wink.com/
Spock http://spock.com (has a search for “private” profile info but is a pay service…haven’t checked that feature out)
Social Mention http://socialmention.com/
WhosTalkin http://www.whostalkin.com/ (this is one of my favorites! Lots of socnets included!)
Samepoint http://www.samepoint.com/
OneRiot http://www.oneriot.com/
Kosmix http://www.kosmix.com/
YackTrack http://www.yacktrack.com
Keotag http://www.keotag.com/
Twoogle http://twoogel.com/ (Google/Twitter search combined)
KnowEm Username Check http://knowem.com/
Firefox Super Search Add-On https://addons.mozilla.org/en-US/firefox/addon/13308 (over 160 search engines built in)
Don’t forget about photo/video social networks and social bookmarking sites:
Pixsy http://www.pixsy.com/
Flickr Photo Search http://www.flickr.com/search/?s=rec&w=all&q=”comapny name”&m=text
YouTube/Google Video Search http://video.google.com/videosearch?q=”company name”
Junoba Social Bookmark Search http://www.junoba.com/ (Digg, Delicious, Reddit, etc..)
Pay Services (might be worth checking out):
Filtrbox http://www.filtrbox.com/
Vocus http://www.vocus.com/
2. Maltego
Maltego goes without saying…it’s probably the best tool to “visually” show you information found on some of the social networks and the relationships that information has connected to it. I have found that Maltego works well for Twitter, Facebook, LinkedIn and MySpace profiles (publicly available). The Twitter transforms are probably the highlight since you can dig into conversations as well. There is also a Facebook transform that was specifically written to search within the Facebook network using a real user account. However, this transform doesn’t work anymore due to recent structural changes to the way Facebook HTML was coded. Note that this transform violates Facebook TOS since it did screen scraping but when it did work it was a great way to search status and group updates not available to public search engines! If anyone wants to help get this transform working again there is a thread on the Maltego forum about it.
Lastly, if you want more information on Maltego and how to use it I suggest checking out the work Chris Gates has done in his Maltego tutorials here and here to learn more. Keep in mind. Maltego works great for finding information if you need it for a specific scope, like a pentest. Maltego even works great if you need to dig a little deeper into something you find on a social network. In terms of automating a monitoring process, I suggest using Google dorks, Yahoo Pipes!, and other techniques which we will talk about here and in future posts.
3. Google Dorks (Facebook, MySpace, LinkedIn)
While you can just simply type in your company name into Google and see what comes up…It’s way easier to use a little Google dork action to search for information on specific social networks. As I stated before, this will only pull publicly available information but you might be surprised what you find about your company just in these searches! Simply paste these into the Google search bar/window. Note: change “bank of america” to whatever you like…not picking on bofa but there is a ton of information about them on social networks! 
Facebook Dorks
Group Search: site:facebook.com inurl:group (bofa | “bank of america”)
Group Wall Posts Search: site:facebook.com inurl:wall (bofa | “bank of america”)
Pages Search: site:facebook.com inurl:pages (bofa | “bank of america”)
Public Profiles: allinurl: people “John Doe” site:facebook.com
*To search personal profile status updates (unless they were made public wall posts via pages or groups) you need to be logged into Facebook and use the internal Facebook search engine. Setting your status updates privacy settings to “Everyone” is actually everyone in Facebook. Rumor has it that next year “Everyone” will mean everyone on the Internet! FTW!
MySpace Dorks
Profiles: site:myspace.com inurl:profile (bofa | “bank of america”)
Blogs: site:myspace.com inurl:blogs (bofa | “bank of america”)
Videos: site:myspace.com inurl:vids (bofa | “bank of america”)
Jobs: site:myspace.com inurl:jobs (bofa | “bank of america”)
LinkedIn Dorks
Public Profiles: site:linkedin.com inurl:pub (bofa | “bank of america”)
Updated Profiles: site:linkedin.com inurl:updates (bofa | “bank of america”)
Company Profiles: site:linkedin.com inurl:companies (bofa | “bank of america”)
While these are Google dorks from the top three social networks (Twitter actually has a really good search engine, search.twitter.com, which I don’t think needs explaining), you can easily modify these for your own use and even include more advanced search operators to include or exclude additional queries. The point of using Google dorks is to make it easier to quickly search for information on social networks without going to each site individually. Still, with most social networks if you want to find private information you either need to login as a user or use some social engineering get the information you want.
What’s next?
In part three of this series I will talk about how to use Google dorks and various search queries for monitoring purposes. Once you have the dorks you want to query, it’s trivial to plug these into Google Alerts to create RSS feeds. Take your feeds and plug them into your favorite RSS reader and you have a simple monitoring tool. More on this in part 3 including a section on aggregating this type of into and customizing it via Yahoo! Pipes which I like to think as the preferred and most customizable method for monitoring social networks.
Next up…in part 2 I will talk about how to find company information on blogs, message boards and document repositories. Oh, and sprinkle a little bit of metadata into the mix as well.
