Household surveys systematically exclude or undercount vulnerable populations and these exclusions and underestimations will materially affect estimates of need such as access to clean water or adequate sanitation, writes Ronelle Burger.
Machine learning has promised a wide range of improvements, ranging from lower crime due to more effective police patrolling to longer lives with precision medicine. Captivated by data science’s giant leaps ahead, it is easy to forget those left behind.
The uncounted, the unsurveyed, the undocumented and unresearched have many different faces, but often include materially poor communities, socially marginalised individuals such as migrants and the homeless, and residents in wartorn areas. Over the coming years, this divide between the seen and unseen may grow even deeper and wider as societies become increasingly reliant on data for decision-making.
Such a dynamic can have at least three undesirable pathways: Firstly, if all goes according to the machine learning masterplan, countries that have good information systems will earn a large social dividend, becoming more efficient and well run, which would work in the direction of further increasing global inequality.
Secondly, if the customisation of systems and policies is increasingly reliant on big data sets, then these systems and policies would reflect the characteristics and preferences of those who are surveyed and documented, not taking account of the characteristics and preferences of those missing from the data sets. But even more concerning perhaps, is the risk that a greater reliance on data will erase many uncounted at-risk countries and vulnerable groups from global policy dialogues, advocacy campaigns and aid prioritisation decisions. In this funding environment, it has become very difficult to attract funding for your cause without a headline-grabbing statistic, or at least a clear sense of the parameters and size of the problem.
Less money available
Following the economic crisis, there is less money to go around, but that may pass. However, the focus on transparency and improved monitoring and evaluation is likely to sharpen over the next few years and this could effectively deprioritise many sectors where impact is harder to measure over a short period, and communities where the quantification of impact is challenging due to for instance their remote location, language difficulties or lack of local measurement and data gathering skills. Because you need money and resources to generate reliable data, there is also a bias against informality and smallness.
In economics, we often invoke the streetlight effect to illustrate the problem by focusing on the problems that are easiest to solve. As the story goes, a policeman encounters a drunk man searching for his car keys under a street light and helps him with his search. Then, after a while of searching without any luck, the policeman asks the drunk man if he is sure that he lost keys here. He then explains that he lost the keys in a park far from there, but he is searching for the keys under the street light because this is where the light is and he can see clearly.
Sifting through all 27 000 economics articles about Africa published over the past two decades, Obie Porteous from Middlebury College in the US found most researchers were looking for their keys in the patch of sidewalk lit by the streetlight.
The research was not evenly spread across the continent’s 54 countries.
Instead, it was dominated by five countries - namely South Africa, Ghana, Kenya, Uganda and Malawi. The 2020 study found that five countries accounted for 45% of the articles on African countries in economics journals and 65% of the articles on African countries in the top five economics journals, even though these countries represented only 16% of the overall population of all African countries.
By contrast, seven countries – Democratic Republic of Congo, Angola, Chad, Sudan, South Sudan, Somalia and Guinea – also represent 16% of the African continent’s population, but were the focus of only 3.5% of all articles on African countries in economics journals.
When exploring the reason for this bias, the author cited factors related to country-specific cost and benefits related to data gathering – including the number of annual tourist visits to the country, violence and safety and having English as an official language – as the main explanation.
Similarly, Kalle Hirvonen from the International Food Policy Research Institute in Addis Ababa’s 2020 study, reports that there were more articles published in the top two health economics journals over the past decade about the Nordic countries, with a combined population of 27 million people, than about African or Asian countries, which represent 2.9 billion of the world’s population.
More generally, it is known that household surveys systematically exclude or undercount vulnerable populations – including homeless people, prisoners, hospital patients, nomadic groups and informal housing dwellers.
Admittedly, the omissions and undercounting are not intentional and these are difficult groups to survey.
Roy Car-Hill from York University estimated that these missed individuals amount to 4% of the world’s total population, but as much as one-fifth of the world's poorest quintile. Because of the overrepresentation of the poor and the vulnerable, these exclusions and underestimates will materially affect estimates of need such as access to clean water or adequate sanitation.
Data availability tends to mirror the existing inequalities in our global system.
Therefore, a greater reliance on data could further widen these inequalities if not accompanied by enhanced vigilance. Innovations such as image processing of satellite images or aerial photographs have helped generate knowledge and research about under-surveyed areas. And researchers have become more imaginative and savvy about gathering data with new approaches such as citizen science or with alternative modalities such as phone interviews or WhatsApp when it is unsafe or prohibitively expensive or too time-consuming to use fieldworkers to do in-person surveys.
But in the long-term, more coordinated effort will be needed to encourage the research community to pay attention to this problem, including dedicated funding mechanisms which should include funding for data sets but also more broadly support for data literacy and survey skills. The United Nations has started this process with its 2030 Sustainable Development Goals, calling for a different type of Data Revolution.
Societies cannot allow their tools to determine their destiny. Measurement matters for how societies evolve and thus it is crucial to acknowledge, recognise and measure what we value and care about.
- Ronelle Burger is a professor in the Department of Economics and a researcher at the Research on Socioeconomic Policy group (RESEP) at Stellenbosch University.
Disclaimer: News24 encourages freedom of speech and the expression of diverse views. The views of columnists published on News24 are therefore their own and do not necessarily represent the views of News24.