Data Mining And The Financial Markets

For Data Mining

Data Mining

In an article in the Financial Times dated 20th May 2013. High Frequency Trading contributed to the fall in the Dow Jones Industrial Average in May 2010, according to US Regulators. However HFT of today is very different from that of three years ago.

This is because of “Big Data”. Financial Markets are big producers of big data, Trades, Quotes, Earnings Statements, Consumer research reports.

About 2 years ago, it became common for hedge funds to extract market sentiment from social media. The idea came from Zhang et al, to develop algorithms based on millions of messages posted by users of twitter and to detect public sentiment and trends, in relation to individual companies. Within the past couple of years it has become popular to develop algorithms that fire up orders as soon as unscheduled information is published, such as natural disasters or terrorist attacks. This is hardly a crazy concept; the stock market is fuelled largely by the perceptions of investors and how those investors react to news.

When it goes wrong was evidence by the so called hash crash of 23rd April 2013, the market dropped by 143 points caused by a hacked bogus tweet about a terrorist attack on Barrack Obama sent from the much respected sources Associated Press twitter feed.

Unlike the crash that happened in 2010 when high level sales caused further sales. It was not a speed crash: it was a “big data” crash. The panic however brief, demonstrates how tightly intertwined Wall Street has become with Twitter, a site that acts both a chat room and news service, where Journalists and publications regulary send out breaking news. There was also concerns over what many suggested was the lurking menace of trading algorithms that scan the news and trade quickly, causing flash crashes.

Text MiningNatural language processing Image

Data analysis of Natural Language (Articles, Books) using text as a form of data. It is often joined with data mining, the numerical analysis of data works, and referred to as “text and Data Mining”. This method TDM involves using advanced software that allows computers to read and digest digital information far more quickly than a human can. TDM software breaks down digital information into raw data and text, and analyses it, and comes up with patterns in Stock Market and Commodities.

Regression is the main statistical technique used to quantify the relationship between two or more variables.

According to a recent Gartner report the business intelligence market is growing 9% per year, will exceed 80 billion by 2015, with about 50% from predictive analytics by that time. Despite all this, the best opportunity for you is still the one you feel most passionate about and know the best, that no one else has recognised. The possibilities are endless.

Recommended Further Research

God and Norman Bloom

The Signal and The Noise Book By Nate Silver, The Art And Science of Prediction.

Fooled By Randomness: The Hidden Role Of Chance in Life and in the Markets, By Nassim Nicholas Taleb.

Pi Movie, 1998, Written and Directed By Darren Aronofsky.

References:

http://www.thestreet.com/story/13044694/1/how-traders-are-using-text-and-data-mining-to-beat-the-market.html

http://searchsqlserver.techtarget.com/definition/data-mining

http://nerdsonwallstreet.com/stupid-data-miner-tricks-quantitative-finance-85/

www.gartner.com/it-glossary/masterdatamanagement-mdm

Leave a Reply

Your email address will not be published. Required fields are marked *