Paul Ormerod’s „Why most things fail“ was published in 2005. Before the subprime crisis, but nevertheless it is a crisis book, that deals with information, lack of information and „bounded rationality“ (Stieglitz / Akerlof). The question is the very basic „why do companies fail?“ – a welcome antonym to all sorts of overconfident management literature. And it seems to me it is he last book written before Big Data came to life. Ormerod leaves some barriers to decision-making and predictive analysis – these are exactly the barriers companies – such as Google or Amazon – try to break.
„The capacity of firms to deal with market situations in a cognitive sense, their capacity to process information and turn it into knowledge, is small compared to the sheer scale oft he problems which confront them. Companies can never deal completely with the complexity of the real world.“
Ormerod points out, that individual and corporations have to deal with limited knowledge and limited capacities to make sense of the knowledge that is available to them. This is the only reason, why such powerful institutions such as market research and advertising came to life. Individuals don’t know what they want, neither do the companies, but they invest billions to find out. Big Data basically is just the next step after market research and advertisement.
In economics there is a contrasting movement between the amount of information companies try to achieve and economists evaluation of the power of rationality. The latest concept (I leave out Nate Silver’s take on Bayes here, because „The Signal and the Noise“ is more about reasoning) is Stieglitz’ and Akerlof nobelprize-winning concept of bounded rationality: Agents „lack complete information“ and some agents even have more information than others – information is „asymmetrical“.
And just in case, all information was available to the individual, would be willing to make sense of it. As Daniel Kahnemann in „Thinking, fast and slow“ argues, we rather live our life in an easy way before we bother to start thinking. Humans are just no analytical beings, they are intuitive. We would simply go crazy, if we would estimate what is happening around us all the time, we decide on the basis of our experiences.
Without explicitely mentioning Big Data Ormerod still opens wide the curtail for Large-Scale Analysis: „Modern mathematical techniques can identify the proportion of the sequence that contained recognizable patterns (…). The existence of a relatively high ratio of ,signal’ to ‚noise’ is a necessary condition for reasonable forecasts of the relevant data series to be made.“ Nevertheless it there will always remain an „inherent lack of predictability“, „it is simply not possible to make accurate forecasts about the future“, writes Ormerod.
Nevertheless, it has to be acknowledged, that Ormerod speaks about „historical data“ exclusively. Projects such as Google Now deal a lot with real-time data correlated with historical data – it is yet unknown, what this will mean at least for the immediate future.
„In other words, even when we have all the data and all the information that exists in a particular context, uncertainty can still prevail“, writes Ormerod. Why? Because of the different concepts of risk and uncertainty. Risk is quantitive (“Risk refers to situations in which the outcome cannot be known with certainty, but the probability of any outcome is understood perfectly.”), whereas uncertainty “refers to situations in which the probability of the various outcomes is itself unknown”.
To Ormerod the quantitative processing of social data only makes sense „if the economy and society are like giant machines, and all we need to do to predict and control them is to assemble enough data and information and analyze it sufficiently cleverly“. And this is exactly what Joseph Weizenbaum was already refering to in 1976 mentioning the „Computermetapher“. It is exactly what transhumanists like Ray Kurzweil are dreaming about. It is basically where all our paranoia about identity and data privacy comes from. But the good news is: as long as we humans make failures, we are still humans.
„Why most Things fail – Evolution, Extinction and Economics“ from Paul Ormerod. Published in 2005 by faber and faber. £8.99.