Designing an index to measure data values in action

The Data Values Manifesto calls for people to have a say in how their data is collected, managed, and used. Credit: Oleg Elkov via Shutterstock.

Connected by Data is a UK-based campaign for people to have a powerful voice in data governance. We launched last year, just a few months before the Data Values Manifesto was published. In our mission to change narratives, policy, and practice around data governance, we spend a lot of time calling for communities to be able to shape the data that affects them along an agenda that tightly tracks the five Data Values Manifesto principles. 

In my first few months as Research Director, I started to build a collection of examples of participatory data governance, but two challenges quickly became clear. First, it would be hard to identify all the different instances where people were gaining power in defining and governing data. And, second, getting a clear picture of where principles of participatory data governance were being put into practice around the world would be incredibly difficult. As we started to think about common global advocacy agendas for collective data governance, we realized that we were facing significant data gaps. 

Although normative arguments for data policy to be shaped by public participation are increasingly put forward, there is a dearth of comparative evidence on how far these norms are making it into policy and practice around the world. And while we can celebrate ad-hoc bright spots of good practice, we have few tools that tell us whether these are exceptions and outliers or signs of new approaches to data governance becoming embedded. 

Thanks for reading The Data Values Digest! Subscribe for free to receive new posts and support our work.

From Manifesto to mainstream

It’s a similar problem to the one I faced over a decade ago in the early years of the open data movement which led me to co-create the Open Data Barometer (ODB), a multi-dimensional index measuring policies and practices around open data. And it’s a similar problem to the one that led me to catalyze the Global Data Barometer project in 2019, building on the expert-survey methodology of the ODB to create a powerful new evidence-collection exercise on data governance, capabilities, availability, and use for the public good. By establishing a global network of country researchers, and gathering primary quantitative and qualitative evidence through a detailed multi-part survey, the Barometer’s studies provide eyes-on-the-ground insights from the field, while also building up a comparative and longitudinal picture of how policies and practices are changing (or not). Studies like the Global Data Barometer have their limitations, but they are also invaluable in filling data gaps and offering a foundation for cross-country research and action. 

Given my history with the Global Data Barometer, and how aligned my current research and campaigning work is to the Data Values Manifesto, you can imagine then how happy I was to discover a project, backed by International Development Research Centre, to look at ‘Measuring Data Values in Action’ using the Global Data Barometer. 

Over the first six months of this year I had the opportunity with this project to explore how far evidence from the Global Data Barometers’ first pilot edition could be used to benchmark country performance on Data Values or to inform action and advocacy to strengthen country-level practices. And I was able to look at how future editions of the Global Data Barometer could be updated to provide critical metrics to support both advocacy and practice on translating Data Values from Manifesto to manifested and mainstream. 

You can find detailed notes from that exploration over on the Connected by Data blog, and a summary in this policy brief. In short, we found that, while the existing Global Data Barometer does not cover all the points of the Data Values Manifesto, the contextual insight and comparative scores it provides can be used by a range of actors. For example:

  • Data Values advocates can compare indicators across countries to identify relative strengths and weaknesses and identify focus areas for advocacy, as well as to track implementation of country commitments and hold key stakeholders to account;

  • International organizations can draw on contextual evidence to gather background information to support engagement with government officials; 

  • Governments can use indicators to benchmark their activity, set ambition for reforms, and track progress over time;

  • Practitioners can explore and learn from examples in the data to identify bright spots and good practice in comparable countries;

  • Donors can look at patterns in the data to shape programming decisions, targeting areas of common challenges across clusters of countries; 

  • Researchers can use data across multiple years to explore ‘what works’ and provide long-term evidence on the impact of reforms.

Unsurprisingly, all the data from the Barometer is provided as open data, and if you want to explore it to inform a project you can:

  • Access the full open dataset and filter to indicators or sub-questions of interest.

  • Visit an individual country results page on the website, and select a relevant indicator under the ‘Supporting Evidence’ heading. This will show scores, justifications, and URLs related to the indicator and its sub-questions. 

The appendix of our policy brief contains a mapping between Data Values themes and relevant Barometer indicators. 

However, perhaps the most exciting opportunity presented by the Global Data Barometer, if funders are able to back future editions, is the opportunity to gather further insight and indicators through the next round of the expert survey. In particular, we identified four candidate questions for the second edition (and questions that, in the meantime, you might find it useful to ask of the settings you are working in): 

  • To what extent are there established mechanisms to enable broad and inclusive public input into the design of key public datasets?

  • Is there evidence that changes have been made to the design (or use) of key public datasets as a result of public engagement?

  • To what extent is there broadly accessible critical data literacy education in the country?

  • To what extent are civil society groups equipped to engage with governments on issues of data governance?

Better primary data on these four points, in addition to better secondary data on funding for responsive and inclusive data systems, would allow us to move from approaching data values through aspiration and anecdote, to a focus on agenda, evidence, and impact. 

You can read the full policy brief here.

You can find Global Data Barometer evidence here.

Related Articles


Your email address will not be published. Required fields are marked *