Twitter Can Predict the Stock Market
- By Lisa Grossman
- October 19, 2010
The emotional roller coaster captured on Twitter can predict the ups and downs of the stock market, a new study finds. Measuring how calm the Twitterverse is on a given day can foretell the direction of changes to the Dow Jones Industrial Average three days later with an accuracy of 86.7 percent.
“We were pretty astonished that this actually worked,” said computational social scientist Johan Bollen of Indiana University-Bloomington. The new results appear in a paper on the arXiv.org preprint server.
Bollen and grad student Huina Mao stumbled on this computational crystal ball almost by accident. Earlier studies had found that blogs can be used to gauge public mood, and that tweets about movies can predict box office sales. An open source mood-tracking tool called OpenFinder sorts tweets into positive and negative bins based on emotionally charged words.
But Bollen wanted to build a more nuanced emotional barometer. He used a standard psychology tool called the Profile of Mood States, a quick questionnaire that is used frequently in pharmaceutical research or sports medicine.
The original questionnaire asks people to rate how closely their feelings match 72 different adjectives, including “friendly,” “peeved,” “active,” “on edge” and “panicky,” and uses the responses to measure mood along six dimensions: calmness, alertness, sureness, vitality, kindness and happiness.
Bollen and colleagues checked a huge Google database to see what other words are commonly used in conjunction with the original 72 adjectives, and added those words to their lexicon. Then the researchers took 9.8 million tweets from 2.7 million tweeters between February and December 2008, selected the tweets that indicated a confession of emotion (tweets that included the words “I feel” or “I’m feeling,” for instance), and ran the test on the entire data set.
“We’re using Twitter like a psychiatric patient,” Bollen said. “This allows us to measure the mood of the public over these six different mood states.”
As a sanity check, the researchers looked at the public mood on some easily-predictable days, like Election Day 2008 and Thanksgiving. The results were as expected: Twitter was anxious the day before the election, and much calmer, happier and kinder on Election Day itself, though all returned to normal by Nov. 5. On Thanksgiving, Twitter’s “Happy” score spiked.
Then, just to see what would happen, Mao compared the national mood to the Dow Jones Industrial Average. She found that one emotion, calmness, lined up surprisingly well with the rises and falls of the stock market — but three or four days in advance.
“I sank into my chair. That’s a pretty big result,” Bollen said. “It was one of those ‘Eureka!’ moments.”
But this surprising correlation said nothing about whether Twitter could be used to tell the future. To test that idea, the researchers trained a machine-learning algorithm to predict whether the stock market would go up or down, first using only the Dow Jones Industrial Average from the past three days, then including emotional data.
The algorithm did pretty well using stock market data alone, predicting the shape of the stock market with 73.3 percent accuracy. But it did even better when the emotional information was added, reaching up to 86.7 percent accuracy.
“Including this mood information leads to higher accuracy,” Bollen said. He stressed that their algorithm is highly simplified, and not the best stock market predictor anyone could come up with. But “we’re presuming on the basis of what we found, if you have some kind of super-duper algorithm and you add our time series, its accuracy will go up, as well.”
The fact that Twitter mood could predict the stock market’s movements even in the middle of 2008 is also significant, Bollen added.
“This was probably one of the most difficult periods to predict,” he said. “We had a presidential election, we had what looked to be financial Armageddon, we had the start of what has been the deepest and greatest recession since the 1930s… If our algorithm was able to predict Dow Jones Industrial Average in that period, we figured that may establish some kind of lower baseline. It could do a lot better in other periods of time.”
But why does it work? “The short answer is, we don’t know,” Bollen said. It’s reasonable to assume that people’s moods will have some effect on their investments, he says, but more research is needed to figure out exactly how.
“It’s a pretty interesting result,” commented computer scientist Sitaram Asur of HP Labs. But even though the correlation is there, Asur is reluctant to believe that the moods captured on Twitter can cause the stock market to change. Not everyone on Twitter plays the stock market, he notes, or even lives in the United States. And he would like to see the algorithm used on tweets from a wider span of time.
“If it is true, if we can actually find this correlation to be consistent, that will be a very important result,” he said. “But right now, I would be cautious about saying how important this is.”
Bollen agrees that the result has some shortcomings. “We need to expand this,” he said. The next step, he said, is to “put some of our money where our mouths are, and try to do this in real time.”
Comment: This reminds me of a term I came up with in 2005, “geo-emotional supercomputer,” which can be seen as related. – Jason Liszkiewicz
