Continuing in their mission to make spending data more accessible and comprehensible, the Spending Stories team and the team of Data.Gov.Uk are releasing a reporting tool today that will help journalists and analysts to pick the freshest and best departmental spending data to work with when exploring the UK central government expenditure.
Spending data is juicy for journalists – why does it get neglected?
Many reasons. One key one is that the shelf-life of a spending dataset is pretty short from a journalist’s point of view, if they have to wait 6 months or even a year for spending data they need for a story to be released, then chances are – the sniff of the story they were wanting to write will probably have gone stale.
Journalists, campaigners and activists need access to well-structured, machine readable and timely data from national as well as sub-national administrations. At OpenSpending, we’re often contacted by journalists with story ideas, or they approach us with a lead. The stumbling stone for them is either lack of information, or worse data that they can’t use because they are not sure of its completeness. The problem is thus the one of trees falling in a wood: If a transaction is missing from a list – does that mean there was no transaction for that amount on that date, or does it mean that the transaction simply was not reported?
These distinctions are important for anyone trying to understand the data – and up to now they have been pretty tricky to answer. As an attempt to make this a little easier, today, we announce the availability of an automatic reporting tool for spending data (available both on data.gov.uk and on OpenSpending), the result of a collaboration between data.gov.uk and us in order to increase the visibility of the spend data and to increase the ease of browsing the substantial volume of datasets that make up the reporting of Government expenditure in data.gov.uk.
The tool lists departments registered as data publishers on data.gov.uk and details how precisely they have followed the HM Treasury reporting guidelines. It will also make the whole of the reported data available for search and analysis both on data.gov.uk and on the OpenSpending site.
The tool is useful to those both using the data, and those within government in assuring that departments are reporting on time. It helps to check:
- Quality of the data (i.e. adherence to HMT reporting guidelines, well-structured data)
- Status of reporting (i.e. how complete the reports are or if there is a reporting period missing)
Why was this possible?
Having all of these datasets organised under a single catalogue at Data.Gov.UK in simple spreadsheet format combined with the Data.gov.uk team’s work in making the necessary metadata available enabled the OpenSpending team to create an extraction system to be set up to clean the data on a regular basis. The team then cleaned over 6000 column names to add compliance with HMT guidance.
How does it work?
The report generator then highlights in red departments who are registered as a publisher on Data.gov.uk but have failed to publish any information on their spending, in yellow those who have published data which cannot be interpreted as spending data (e.g. PDF format or not complying with the template provided by HMT) and green those departments whose records have been updated as regularly as demanded as per the publication requirements (latest data must have been published as recently as a month ago).
The first stage of this release deals with central departments, who are obliged to report all spending over 25,000 GBP. Subsequent stages to follow soon after will monitor local councils and other governmental bodies, which have different reporting requirements. The interface will be useful both inside and out of government, to ensure transparency regulations are met and to better understand where gaps in data may alter the completeness of the picture offered by government data.
- Reporting tool on data.gov.uk
- More extensive list of entities at OpenSpending
- Our briefing with all the details and link to code.
Interested in more regular updates from the Spending Stories team? Join the discussion via the OpenSpending mailing list.
Lucy is a free range "tech-translator", blogging about her work at http://techtohuman.com/.
Formerly, Lucy worked for Open Knowledge leading School of Data, co-editing the Data Journalism Handbook and coordinating the OpenSpending community.
Very impressive tool for cleaning up some messy data. However I wonder if there are any more recent insights into the evidence that some government departments actually release data that is well formed but where the data is actually wrong — missing transactions or transactions where the amount is incorrect. Clearly it would be very difficult to detect this in the general case since it would require cross-checking with the suppliers’ records, but the question remains of how useful this data is if it there are good reasons to believe that it contains well-formed but incorrect entries.