This text shows a real case of how the Open Data Editor (ODE) impacted the workflow of an organisation working to serve the public good.
Picture of a rally in 2021 at the Place de l’Hôtel de Ville, Paris, around the ‘Abolition of Nuclear Weapons’ airship.
Organisation: Observatoire des armements / CDRPC
Location: Lyon, France 🇫🇷
Knowledge Area: Peace and conflicts
Type of Data: Defence spending
Founded in 1984, the Observatoire des armements is the only independent centre of expertise and resources specialising in defence and security in France. Currently located in Saint-Just, Lyon, the documentation centre is open to the public by appointment. The archives contain unique items that are available to students, researchers, journalists and citizens seeking information.
The Observatoire operates on two main axes: the strengthening of democratic control of the arms industry and transfers, and the elimination of nuclear weapons as well as the recognition of their health and environmental consequences for populations. They do it by regularly disseminating independent information through the media, publishing books and reports, and promoting seminars, training and debates to encourage citizen engagement.
The Challenge
Problem
The Observatoire uses several different databases as sources for its reports, which follow different frameworks and criteria. They operate in a field of knowledge where there are no defined standards for data handling, so the data obtained from, for example, the United Nations Register of Conventional Arms (UNROCA), the SIPRI Arms Transfers Database, or the Open Security Data Europe, are different from each other. The Observatoire also works with data collected by researchers linked to the CDRPC, for example, this map built with Amnesty International France, powered by OpenStreetMap.
They need to standardise the data into a single spreadsheet to feed the information available at the Weapon Watch Open Data Environment platform, which is currently in conception.
They also face a lack of specialised human resources. The Observatoire currently has 15 active members and a network of 45 contributors (citizens, activists, researchers…) who actively contribute to the project through their research. Only three people have technological knowledge, and only one of them manages the databases, and even then in a self-taught way, without any specific training in data science or data management.
Impact
Compiling data from multiple sources into a single spreadsheet with hundreds of rows and dozens of columns is a manual, time-consuming task. It’s often days of work checking that each cell follows the same pattern as the others, or that there are no typos, different formatting or blank cells.
As they can’t rely on a permanent technical team, this can affect the equilibrium of groups already engaged in investigations and other data research.
Examples of some of the different databases the Observatoire works with.
The Solution
The Observatoire uses the Open Data Editor (ODE) at two different stages in the data collection and production process. The first is to load the spreadsheets from different sources so that they can see in seconds how many and what kind of errors the spreadsheet contains – either empty cells or inconsistencies in the pattern of information. They are then able to correct the missing or incorrect data.
The second stage comes after all the data has been compiled into a single spreadsheet. Before publication, the file is prepared with as much information as possible in the metadata. This is necessary to understand the context of the shared information and increases the quality of the dataset.
According to the team, ODE is also very useful for understanding how data works in general and the importance of standards.
In this spreadsheet, ODE flagged 48 inconsistencies in seconds.
Here, two cells used an incorrect pattern for the number that identifies the company (SIREN) that sells arms to the French government. In other words, ODE has solved a problem that would have jeopardised the accuracy of the content.
The ODE metadata panel proved to be a key resource for the team to learn more about how data works.
The Results
- Reduced error resolution time from days to seconds
- Eliminated 95% of manual review work
- Enabled the team to focus on what matters
- Using the app also helped with data literacy
The Observatoire des armements continue to develop the Weapon Watch Open Data Environment platform and can much more easily arrive at impressive and preliminary results with quality-assured data, such as the map below, entitled La guerre se fabrique près de chez nous (‘War is produced near us’), a tool for the surveillance network of military material production companies.
Working with the ODE will help them propose, for the first time, a standardised model to track the local industries’ activities.
Quote
Sayat Topuzogullari, Coordinator
“When I did the Open Data Editor course, it was very useful to understand that some datasets that we put on our websites are not really ready to be shared, for example because of all the metadata and therefore contextualization. Our center is taking on this task of publishing databases, with 40 years of experience in research and investigation. I’ve shared my experience with our researchers, and ODE meets our quality requirements. We know that it is not enough to put some points on the map or share some files. We really need to turn this research data into reusable and truly available data”.
About the Open Data Editor
The Open Data Editor (ODE) is Open Knowledge’s new open source desktop application for nonprofits, data journalists, activists, and public servants, aiming at helping them detect errors in their datasets. It’s a free, open-source tool designed for people working with tabular data (Excel, Google Sheets, CSV) who don’t know how to code or don’t have the programming skills to automatise the data exploration process.
Simple, lightweight, privacy-friendly, and built for real-world challenges like offline work and low-resource settings, ODE is part of Open Knowledge’s initiative The Tech We Want — our ambitious effort to reimagine how technology is built and used.
And there’s more! ODE comes with a free online course that can help you improve the quality of your datasets, therefore making your life/work easier.
↪️ Take the course: Learn how to use ODE
All of Open Knowledge’s work with the Open Data Editor is made possible thanks to a charitable grant from the Patrick J. McGovern Foundation. Learn more about its funding programmes here.
