The following is a guest post from Chris Taggart, co-founder of OpenCorporates.com and member of the Open Knowledge Foundation’s Working Group on Open Government Data.
When the amazing Emer Coleman first approached me a year and a half to get feedback on the plans for the London datastore, I told her that the gold standard for such datastores was that run by the District of Columbia, in the US. It wasn’t just the breadth of the data; it was that DC seemed to have integrated the principles of open data right into its very DNA.
And we had this commitment in mind that when we were thinking which were the US jurisdictions we’d scrape first for OpenCorporates, whose simple (but huge) goal is to build an open global database of every registered company in the world.
While there were no doubt many things that the DC company registry could be criticised for (perhaps it was difficult for the IT department to manage, or problematic for the company registry staff), for the visitors who wanted to get the information it worked pretty well.
What do I mean by worked well? Despite or perhaps because it was quite basic, it meant you could use any browser (or screenreader, for those with accessibility issues) to search for a company and to get the information about it.
It also had a simple, plain structure, with permanent URLs for each company, meaning search engines could easily find the data, so that if you search for a company name on Google there’s a pretty good chance you’ll get a link to the right page. This also means other websites can ‘deep-link’ to the specific company, and that links could be shared by people, in social networking, emails, whatever.
Finally, it meant that it was easy to get the information out of the register, by browsing or by scraping (we even used the scraper we wrote on ScraperWiki as an example of how to scrape a straightforward company register as part of our innovative bounty program).
It was, for the most part, what a public register should be, with the exception of providing a daily dump of the data under an open licence.
So it was a surprise a couple of weeks ago to find that they had redone the website, and taken a massive step back, essentially closing the site down to half the users of the web, and to those search engines and scrapers that wanted to get the information in order to make it more widely available.
In short it went from being pretty much open, to downright closed. How did they do this? First they introduced a registration system. Now, admittedly, it’s a pretty simple registration process, and doesn’t require you to submit any personal details. I registered as ‘Bob’ with a password of ‘password’ just fine. But as well as adding friction to the user experience, it also puts everything behind the signup out of the reach of search engines. Really dumb. Here’s the Google search you get now (a few weeks ago there were hundreds of thousands of results):
The other key point about adding a registration system is that the sole reason is to be able to restrict access to certain users. Let me repeat that, because it goes to the heart of the issue about openness and transparency, and why this is a step back from both by the District of Columbia: it allows them to say who can and can’t see the information.
If open data and transparency is about anything, it’s about giving equal access to information no matter who you are.
The second thing they did was build a site that doesn’t work for those who don’t use Internet Explorer 7 and above, including those who used screenreaders. That’s right. In the year 2011, when even Microsoft are embracing web standards, they decided to ditch them, and with them nearly half the web’s users, and all those who used screenreaders (Is this even allowed? Not covered by Americans With Disabilities Act?).
In the past couple of weeks, I’ve been in an email dialogue with the people in the District of Columbia behind the site, to try to get to the bottom of this, and the bottom seems to be, that the accessibility of the site, the ability for search engines to index it, and for people to reuse the data isn’t a priority.
In particular it isn’t a priority compared with satisfying the needs of their ‘customers’, meaning those companies that file their information (and perhaps more subtly those companies whose business models depend on the data being closed). Apparently some of the companies had complained that they were being listed, contacted and or solicited without their approval.
That’s right, the companies on the public register were complaining that their details were public. Presumably they’d really rather nobody had this information. We’re talking about companies here, remember, who are supposed to thrive or fail in the brutal world of the free market, not vulnerable individuals.
It’s worth mentioning here that this tendency to think that the stakeholders (hate that word) are those you deal with day-to-day is a pervasive problem in government in all countries, and is one of the reasons why they are failing to benefit from open data the way they should and failing too to retool and restructure for the modern world.
Sure, we can work around these restrictions and probably figure out a way to scrape the data, but it’s a sad day to see one of the pioneers of openness and transparency take such a regressive step. What’s next? Will the DC datastore take down its list of business licence holders, or maybe the DC purchase order data, all of which could be used for making unsoliticited requests to these oversensitive and easily upset businesses?
p.s. Apparently this change was in response to an audit report, which I’ve asked for a copy of but which hasn’t yet been sent to me. Any sleuthing or FOI requests gratefully received.
p.p.s. I also understand there’s also new DC legislation that’s been recently been passed that require further changes to the website, although again the details weren’t given to me, and I haven’t had time to search the DC website for them
This post is by a guest poster. If you would like to write something for the Open Knowledge Foundation blog, please see the submissions page.
1 thought on “When Washington DC took a step back from open data & transparency”
Comments are closed.