Support Us

You are browsing the archive for spending stories.

UK Departmental Government Spending – Improving the Quality of Reporting

Lucy Chambers - September 13, 2012 in Featured, Open Spending, Releases

Continuing in their mission to make spending data more accessible and comprehensible, the Spending Stories team and the team of Data.Gov.Uk are releasing a reporting tool today that will help journalists and analysts to pick the freshest and best departmental spending data to work with when exploring the UK central government expenditure.

Spending data is juicy for journalists – why does it get neglected?

Many reasons. One key one is that the shelf-life of a spending dataset is pretty short from a journalist’s point of view, if they have to wait 6 months or even a year for spending data they need for a story to be released, then chances are – the sniff of the story they were wanting to write will probably have gone stale.

Journalists, campaigners and activists need access to well-structured, machine readable and timely data from national as well as sub-national administrations. At OpenSpending, we’re often contacted by journalists with story ideas, or they approach us with a lead. The stumbling stone for them is either lack of information, or worse data that they can’t use because they are not sure of its completeness. The problem is thus the one of trees falling in a wood: If a transaction is missing from a list – does that mean there was no transaction for that amount on that date, or does it mean that the transaction simply was not reported?

These distinctions are important for anyone trying to understand the data – and up to now they have been pretty tricky to answer. As an attempt to make this a little easier, today, we announce the availability of an automatic reporting tool for spending data (available both on and on OpenSpending), the result of a collaboration between and us in order to increase the visibility of the spend data and to increase the ease of browsing the substantial volume of datasets that make up the reporting of Government expenditure in

The tool lists departments registered as data publishers on and details how precisely they have followed the HM Treasury reporting guidelines. It will also make the whole of the reported data available for search and analysis both on and on the OpenSpending site.

The tool is useful to those both using the data, and those within government in assuring that departments are reporting on time. It helps to check:

  1. Quality of the data (i.e. adherence to HMT reporting guidelines, well-structured data)
  2. Status of reporting (i.e. how complete the reports are or if there is a reporting period missing)

Why was this possible?

Having all of these datasets organised under a single catalogue at Data.Gov.UK  in simple spreadsheet format combined with the team’s work in making the necessary metadata available enabled the OpenSpending team to create an extraction system to be set up to clean the data on a regular basis. The team then cleaned over 6000 column names to add compliance with HMT guidance.

How does it work?

The report generator then highlights in red departments who are registered as a publisher on but have failed to publish any information on their spending, in yellow those who have published data which cannot be interpreted as spending data (e.g. PDF format or not complying with the template provided by HMT) and green those departments whose records have been updated as regularly as demanded as per the publication requirements (latest data must have been published as recently as a month ago).

The first stage of this release deals with central departments, who are obliged to report all spending over 25,000 GBP. Subsequent stages to follow soon after will monitor local councils and other governmental bodies, which have different reporting requirements. The interface will be useful both inside and out of government, to ensure transparency regulations are met and to better understand where gaps in data may alter the completeness of the picture offered by government data.

Interested in more regular updates from the Spending Stories team? Join the discussion via the OpenSpending mailing list.

How Spending Stories Fact Checks Big Brother, the Wiretappers’ Ball

Lucy Chambers - February 27, 2012 in Open Spending, Spending Stories

This piece was co-written with Eric King of Privacy International and comes as Privacy International launches a huge new data release about companies selling surveillance technologies. It is cross-posted on the MediaShift PBS IDEA LAB and the OpenSpending blog.

Today, the global surveillance industry is estimated at around $5 billion a year. But which companies are selling? Which governments are buying? And why should we care?

We show how the OpenSpending platform can be used to speed up fact checking, showing which of these companies have government contracts, and, most interestingly, with which departments…

The Background

Big Brother is now indisputably big business, yet until recently the international trade in surveillance technologies remained largely under the radar of regulators and civil society. Buyers and suppliers meet, mingle and transact at secretive trade conferences around the world, and the details of their dealings are often shielded from public scrutiny by the ubiquitous defence of ‘national security’. Perhaps unsurprisingly, this environment has bred a widespread disregard for ethics and a culture in which the single-minded pursuit of profit is commonplace.

For years, European and American companies have been quietly selling surveillance equipment and software to dictatorships across the Middle East and North Africa – products that have allowed these regimes to maintain a stranglehold over free expression, smother the flames of political dissent and target individuals for arrest, torture and execution.

They include devices that intercept mobile phone calls and text messages in real time on a mass scale, malware and spyware that gives the purchaser complete control over a target’s computer and trojans that allow the camera and microphone on a laptop or mobile phone to be remotely switched on and operated. These technologies are also being bought by Western law enforcement, including small police departments in which the ability of officers to understand the legal parameters, levels of accuracy and limits of acceptability is highly questionable.

The data that has just been released on the Privacy International Website included the following:

  1. An updated list of companies selling surveillance technology, and
  2. Naming all the government agencies attending an international surveillance trade show known as the wiretappers ball.

Some names are predictable enough: the FBI, the US Drug Enforcement Administration, the UK Serious Organized Crime Agency and Interpol, for example. The presence of others is deeply disturbing: the national security agencies of Bahrain and Yemen, the embassies of Belarus and the Democratic Republic of Congo and the Kenyan intelligence agency, to name but a few. A few are downright baffling, like the US department of Commerce or the US Fish & Wildlife Service and Clark County School District Police Department.

Now, with the aid of OpenSpending, anyone can cross reference which contracts these companies hold with governments around the world. The investigation continues…

Using OpenSpending to speed up fact-checking

Privacy International approached the Spending Stories team to ask for a search widget to be able to search across all of the government spending datasets for contracts held between governments and these companies (until this point, it had only been possible to search one database at a time).

The Spending Browser is now live at And, as the URLs correspond to the queries, individual searches can be passed on for further examination and, importantly, embedded in articles directly. Try it yourself against the list of companies listed in the Surveillance Section of the Privacy International Site (Just enter a company e.g. ‘Endace Accelerated’ into the search bar).

The Spending Browser will become increasingly more powerful as ever more data is loaded into the system.

Want to help make this tool even more powerful? Get involved and help to build up the data bank.


You can read more about the background to these stories on the Privacy International Site and recent coverage by the International Media:

Get Updates