Even before we speak about the book – let’s talk about the title of the book. There is a long list of book titles with two opposing or complimenting words: Crime & Punishment, Pride & Prejudice, the Beauty & the Beast and my personal favourite- the Ghost & the Darkness. Nate Silver’s non-fiction effort will never rival the popularity of these fiction classics, but the Signal and the Noise is a brilliant title and it is what attracted me to the book.
The book is about the art and science of prediction. It’s central premise is that data based prediction in the real world is extremely difficult and in most fields the track record of such predictions is very poor.
He cites prediction examples from a wide variety of fields: elections, sports, weather, economics, gambling and geology. He singles out economics as one area where the quality of predictions has consistently been bad – often worse than a coin flip. Similarly, the track record in the field of earthquake prediction has been abysmal. On the other hand, weather forecasting has been consistently improving with the advent of faster computers and better models.
He sites multiple reasons for this failure of prediction in spite of every increasing computing power. Some of these are: over simplification of inherently complex systems, confusing correlation for causation, lack of understanding of Bayesian statistics, poor data quality and over-confidence in models. so how does one identify the signal and ignore the noise – so as to arrive at a better understanding?
Before suggesting a solution, the writer gives a start warning: for all its hype, Big Data is only likely to make the situation worse. There is a lot more information now, but also a lot of this information is unlikely to be useful. The ratio of the signal to the noise is getting worse. In spite of more data or rather because of more data, our predictive ability is likely to get worse.
The solution is to think probabilistically in line with Bayesian principles, that is to have a starting position, make a prediction using available data but make continuous adjustments based on actual observations. He warns us against over-confidence in basic predictability of events and our models.
The book is well written with lots of examples. It makes for engrossing reading; albeit the recommendations at the end are not necessarily very conclusive.
The Books On Our Shelf 