An open society does not hide data

2011-01-15 14:16

Here’s a story that always leaves South African journalists with their jaws on the ground.

A reporter walks into a city hall, approaches the ­information desk and asks to be given copies of the mayor’s correspondence on her official email address for the last couple of weeks.

The reporter is offered a cup of coffee and asked to wait.

A short while later the official returns and hands over a wad of paper with all the emails he has requested.

The correspondence reveals all kinds of ­details, including some political horse­trading between the mayor and a political ­rival.

This is a not a joke.

The journalist happens to be a former colleague who was given this ­access-to-information experience while on a trip to the United States.

Oh, how we laughed.

The thought of doing something similar in South Africa seemed so ludicrous.

But this little story reminds us what transparency really means and the ­value of the government being freely available for public scrutiny.

In the United States, the government launched a website called, an astounding collection of official datasets.

 A year and a half later there are more than 300 000 datasets available on the site.

Now 10 other countries are developing similar “open data” initiatives including Canada, the United Kingdom, Norway, Australia, New Zealand and even little Seychelles.

South Africa, despite moves toward an “e-government” concept is not among them.

Here there is little official data available online except for that offered by StatsSA.

For example, it would appear that there is no central database of awarded state tenders.

The only consolidated tender database that I could locate is maintained by the Treasury and only deals with tenders that are national in nature, and most tenders are handled at provincial and local government level.

Can you imagine the potential for citizens monitoring corruption if details of every awarded tender, the person or entity they were awarded to and how much they were paid were freely and easily available?

This “open data movement” has breathed new life into a growing field called data journalism which many predict to be one of the future paths of our profession.

At the heart of this is the idea that journalists gather and process vast collections of raw data, seeking deeper understanding and insight and developing, not just information, but also tools which empower their audiences.

Recently, the Media24 Investigations team tackled a project which could broadly be described as data journalism although in an infant form.

But it nevertheless illustrates the consequences of official data being ­released into the public domain.

For several years the Department of Mineral Resources has been under pressure to make available online the records of all new prospecting and mining right applications being filed in South Africa.

Several months ago the department finally put this information online, on the face of it a bonanza of ­information for those interested.

The dataset included at least a million individual records but it was made available in a format which made it extremely difficult to search.

We spent many, many hours converting the records into a format that made the information practically searchable and writing a simple computer script to help us rapidly sift and analyse the data.

Thanks to these relatively simple (although for us, ambitious) tools we were able to discover the extent of the ANC’s Chancellor House Mineral Resources’ burgeoning ­interests in mining in South Africa, the similar interests of many other prominent figures and expose very directly a number of serious shortcomings in the application of the ­government’s new mining policies.

All of this emerged from one set of data. Can you imagine the potential if tens of thousands of datasets and record collections were available to the public?

Recently journalists I was working with tried to gain access to data for a reporting project which could have produced powerful results.

We were working in the Eastern Cape trying to investigate whether there was a correlation between the quality of municipal water supplies and disease outbreaks, and whether state health and infrastructure spending was aligned to these hotspot areas.

Eventually we had to abandon the project as it proved impossible to gain access to the data we needed – or even to determine whether such data actually existed in government.

But such a project could have provided the government with a useful, fact-based insight into whether they were spending our tax rands in the right places or not.

Recent moves epitomised by the Protection of Information Bill and the media tribunal threats don’t augur well for increased transparency and the release of state data.

Join the conversation! encourages commentary submitted via MyNews24. Contributions of 200 words or more will be considered for publication.

We reserve editorial discretion to decide what will be published.
Read our comments policy for guidelines on contributions. publishes all comments posted on articles provided that they adhere to our Comments Policy. Should you wish to report a comment for editorial review, please do so by clicking the 'Report Comment' button to the right of each comment.

Comment on this story
Comments have been closed for this article.

Inside News24

Traffic Alerts
There are new stories on the homepage. Click here to see them.


Create Profile

Creating your profile will enable you to submit photos and stories to get published on News24.

Please provide a username for your profile page:

This username must be unique, cannot be edited and will be used in the URL to your profile page across the entire network.


Location Settings

News24 allows you to edit the display of certain components based on a location. If you wish to personalise the page based on your preferences, please select a location for each component and click "Submit" in order for the changes to take affect.

Facebook Sign-In

Hi News addict,

Join the News24 Community to be involved in breaking the news.

Log in with Facebook to comment and personalise news, weather and listings.