IAN MANN REVIEWS | Lies, damn lies, and statistics: Understanding data better

sanne bl
sanne bl

The Number Bias: How Numbers Lead and Mislead Us, by Sanne Blauw

Quick! How many numbers have you seen today that report on what is going on in the world?

Many! The beauty of numbers is that they (supposedly) don't lie. Either there are 28 boxes on the shelf or there aren't. Either there is R36.67 in your account or there isn't.

If only all numbers were that simple. The most important are not, and therein lies the value of this easily accessible and entertaining book that spent weeks on the bestseller lists in the Netherlands. Author Sanne Blauw has a PhD in econometrics and wrote this book to educate and reveal the limitations and dangers of what we take far too lightly.

Numbers, it emerges, are not always that reliable.

The opening chapter of the book deals with IQ scores. During WWI, 1.75 million American recruits completed IQ tests – segregated by race – under Robert Yerkes, which have since been described by some critics as among the "racist beginnings of standardised testing". The tests, which have since been the subject of numerous studies on bias in IQ testing, for reasons ranging from the type of questions asked to the method of testing to alleged manipulation of results. They were also criticised for a cultural bias towards recruits who were familiar with America. 

At the time of testing, some of the WW1 recruits given the IQ tests could not read or write, and some had never held a pencil. The test results revealed little about intelligence, but a great deal about power, privilege, access to education, poverty, deprivation, and language proficiency.

More generally, the very notion of intelligence is not necessarily something that can be measured numerically. As the saying goes, ask a fish to climb a tree and it will spend its life believing it is stupid. Intelligence is not a fact like the number of boxes or the money in your bank. It is a 'made up' abstract construct: contextual, cultural, and vague.

Another number with potentially serious connotations – and consequences – is a credit score. It is used in assessing whether you should be granted credit to buy a house or car, based on more than just your ability to repay the loan. It is based on something much more difficult to assess – your trustworthiness in the future.

The algorithms that are relied on to make such a crucial decision don't only look at your past, but at what people similar to you, as well as your friends, have done in similar circumstances. These reports are provided by agencies to lenders, who aren't privy to how the data firm arrived at these scores. But they make life-altering decisions nevertheless.

How reliable is the big data credit-scoring that agencies use? "A staggering quarter of all people found errors in their credit reports from one of the three big bureaus." Mistakes creep into other data sets too. Between 2009 and 2010 there appeared to be 17 000 pregnant men in the UK.

A common number that readers of this column rely on is GDP – the monetary value of final goods and services produced in a country in a specific period of time. This, too, is an abstract: it is a construct that we have chosen, and one that excludes and include items that could or could not have been included.

Is this important? Well, it, too, has important ramifications. When GDP falls, we are in a recession and we must hold back on expenditure because you may lose your job or have to pay higher taxes.

The idea itself was conceived in the US just before WWII to measure the 'national income', made up of the income of households and companies. With the war looming, the government needed to spend significant amounts of money on arms rather than people.

A drop in national income might dampen support for the war effort, so the war expenditure had to be disguised. That is why, still today, money spent on armaments produced for governments is included among the goods and services of the GDP.

But again, it's a construction.

All data is gleaned from somewhere. How it was gathered, from whom, under what circumstances, and in what quantities, affects quality. In conservative states such as Mississippi people surveyed report being gay less often than in progressive states such as New York. This may be because there are fewer gay people there; however, there are relatively as many searches for gay porn in conservative states as in progressive ones.

While big data should be more reliable given the sampling size, it is not necessarily so.

As Cathy O'Neil, the author of Weapons of Math Destruction, explains, the eerie fact that "because of the programme's self-learning capabilities, algorithms can become so complex that no one, not even the programmers, understands which steps the software is taking."

Numbers can be misleading for other reasons too. Amy Cuddy described the results of an experiment she conducted in a TED talk viewed by a jaw-dropping 58 million people. Her experiment describes how adopting 'power poses' causes hormonal changes that radically increase confidence and reduce stress in otherwise trying settings, such as a job interview.

How many of the 58 million viewers do think had their opinions shaped by her report?

Her experiments were performed on 42 people, but when later re-tested on 200 people, no difference in hormonal levels were detected. How many of the 58 million do you think know that? I didn't.

Some 69 years ago, in his book How to Lie with Statistics, Huff showed that one can make a good estimate of the number of children in a house by counting the number of storks that nest on its roof. As you know storks don't cause babies, even though the number of storks may correlate with the number of babies.

As shown in the example of the immigrants and black Americans in the WWI IQ test, correlation doesn't mean causation. People with larger families have bigger houses, with have bigger roofs, on which more storks land.

Why do people get the importance of numbers so wrong? This is hardly surprising when you consider that one in four adults in developed countries perform at or below the lowest level of numeracy, and that mathematics anxiety is found in about 30% of fifteen-year-olds.

How can we improve our understanding of facts represented in numbers? Blauw suggests taking a step back and asking six questions.

1. Who is the messenger? If a study sponsored by a tobacco company shows that smoking might correlate to poor health, but does not cause it, there is reason to believe the result is not reliable.

2. What do I feel about the number? Feelings are an important part of being human and that makes you susceptible to bias. Being aware of your feelings, look for sources with a different perspective.

3. How has the research summary number been standardised? Does the number deal with an invented concept, such as economic growth or intelligence? Has the number been blown up into something it is not? Try to find research that measures the concept in a different way.

4. How has the data been collected? Are the circumstances such that you would rather not have told the truth? Consider the conservative environment and being gay, described above. The number only ever applies for the specific group that was studied.

5. How has the data been analysed? Does the number relate to an alleged causal link? Could the link have come about by chance? Were any other factors involved? Could the causal link work in the opposite direction?  

6. How have the numbers been presented? It is not valuable to know you have a 5% greater chance that you may get a particular disease, if you do not know what the percentage is of. If the numbers are presented in a graph, are the axes stretched, or squashed together? This could mislead.

We cannot leave important decisions to numbers and calculation methods if we haven't given them further thought. You will find this book immensely helpful in this quest.

Readability         Light --+-- Serious

Insights              High -+--- Low

Practical             High --+-- Low

*Ian Mann of Gateways consults internationally on strategy and implementation, is the author of 'Strategy that Works' and a public speaker. Views expressed are his own.

Brent Crude
All Share
Top 40
Financial 15
Industrial 25
Resource 10
All JSE data delayed by at least 15 minutes morningstar logo
Company Snapshot
Voting Booth
Do you think it was a good idea for the government to approach the IMF for a $4.3 billion loan to fight Covid-19?
Please select an option Oops! Something went wrong, please try again later.
Yes. We need the money.
11% - 1005 votes
It depends on how the funds are used.
74% - 6751 votes
No. We should have gotten the loan elsewhere.
15% - 1417 votes