Blog

6 lessons from sharing humanitarian data

Cross-posted from scraperwiki.com This post is a write-up of the talk I gave at Strata London in May 2015 called “Sharing humanitarian data at the United Nations”. You can find the slides on that page. The Humanitarian Data Exchange (HDX) is an unusual data hub. It’s made by the UN, and is successfully used by […]

The best data opens itself on UK Gov’s Performance Platform

This is a guest post by Francis Irving (@frabcus), CEO of ScraperWiki, who has made several of the world’s first civic websites such as TheyWorkForYou and WhatDoTheyKnow. This is the third in a series of posts about the UK Government’s Performance Platform. Part 1 introduced why the platform is exciting, and part 2 described how […]

9 models to scale open data – past, present and future

The possibilities of open data have been enthralling us for 10 years. I came to it through wanting to make Government really usable, to build sites like TheyWorkForYou. But that excitement isn’t what matters in the end. What matters is scale – which organisational structures will make this movement explode? Whether by creating self-growing volunteer […]

World’s first REAL commercial open data curation project!

The following post is by Francis Irving, CEO of ScraperWiki. Can you think of an open data curation project where the people who work on it come from multiple commercial companies? In the mid 1990s, as open source code began to boom, the equivalent was commonplace. Geeks working at ISPs would together patch the Apache webserver into shape. Startups like […]

From CMS to DMS: C is for Content, D is for Data

This is a joint blog post by Francis Irving, CEO of ScraperWiki, and Rufus Pollock, Founder of the Open Knowledge Foundation. It’s being cross-posted to both blogs. Content Management Systems, remember those? It’s 1994. You haven’t heard of the World Wide Web yet. Your brother goes to a top university. He once overheard some geeks […]

And so corporations begin to open data…

The following post is by Francis Irving, CEO of ScraperWiki. Now it seems almost normal that red in tooth and claw competitors, like Microsoft and Google, are both major contributors to the latest version of a popular open source operating system kernel. Businesses are gradually realising they can share the costs of anything based on […]

“Should Britain flog off the family silver to cut our national debt?”

The following post is from Francis Irving, CEO of ScraperWiki. ‘Should Britain flog off the family silver to cut our national debt?‘ — that’s the question the UK current affairs documentary Dispatches tackled last Monday. ScraperWiki worked with Channel 4 News and Dispatches to make two supporting data visualisations, to help viewers understand what assets […]

Election data!

If you’d asked me back in 2005, I’d have told you that the 2010 election would be the first online election. It turned out not to be. For example, the YouTube and Facebook leaders debate was much less important than the Television debates. However, there are a few places relating to data where the Internet […]

Sources of data on data.gov.uk

When data.gov.uk was launched, I had a quick browse around the data, to get a feel for what was in it. Most data sets that I randomly looked at were from statistics.gov.uk (from the Office for National Statistics). Today, I decided to investigate, and work out some basic statistics about the source of the data. […]

Open organisations, need for two more definitions!

If starting a new, public interest, organisation, there are three obvious principles you might like to have. Finance – have all bank transactions automatically public in real time. Plus accounts. Software – all software made by the organisation to be open source. Information – voluntarily subscribe to some sort of FOI law. The software one […]

Clearer Climate Code

GISTEMP is a crucial open data set, because it contains the historical global temperature record. Not very important right now, but in the medium term absolutely vital for the continuing functioning of our society given the likelihood of adverse climate change. Stations that measure temperature naturally do so at specific points in space, and the […]

A Wikipedia of English law

Writing in Times Online in April 2006 the eminent Professor Richard Susskind, legal tech guru and adviser to the great and good, spelt out his vision for a “Wikipedia of English law”: This online resource could be established and maintained collectively by the legal profession; by practitioners, judges, academics and voluntary workers. If leaders in […]

Free our Bills!

Free Our Bills! is a campaign led by a cheeky platypus, just escaping from the portcullis of Parliament. Sign up now, or read on… Sometimes data being free isn’t good enough – it needs to be released in a properly structured format. If you want to reproduce the text of Bills (proposed new laws in […]

Review of economics of trading funds published by UK Government

Hot off the press – the UK’s Department for Business, Enterprise and Regulatory Reform has published a review of the economics of trading funds. The review follows (I think) recommendation 9 of the Power of Information review: Recommendation 9. By Budget 2008, government should commission and publish an independent review of the costs and benefits […]

David Cameron for open knowledge?

In a speech today, the Leader of the Opposition in the UK said some interesting things about freeing up Government information. He begins with some general open information philosophy: But look at our Government at home. It’s still bureaucratic, still top-down and still old-world. It still thinks it knows best and that it should keep […]

When is my bus?

Sometimes you find some data whose lack of freedom is totally mysterious from a commercial point of view. At mySociety, we recently released made some travel time maps which help you work out where you should live that would have the quickest commute to your place work. Interactive flash sliders to balance this delicate equation […]

Open social data progress

My last post here, Google vs. Facebook, was about how our own personal knowledge, that you’ve put into social networks like Facebook or MySpace, should also be open. By this I mean that you should have control of it, and it should be encoded in open formats with open protocols. The last week there’s been […]

Google vs Facebook

Facebook has striken fear not only into the hearts of incumbent dot-com billionaires, but also into the hearts of open data freaks. It’s terrifying me – not least, because I use it. I give the most personal, sensitive information that I have to a private US corporation. I have no way of getting the data […]

We need an Open Service Definition

There’s a buzz at GUADEC, an open source computer desktop conference in Birmingham right now, about the idea of the Online Desktop. Increasingly we all use web services rather than local applications, and store our own personal knowledge in other people’s proprietary formats and software. GMail rather than Outlook, Flickr rather than iPhoto. Just as […]

Giving us our own information

Yesterday, the UK’s Cabinet Office published an important new review of Public Sector Information. It’s a fat document to read through, but it oozes with ideas and stories (many no doubt from readers of this blog) about the importance of open data. The obvious question is, what will the Government do with it at all […]

A great day for the law and for the people

Today the Department for Constitutional Affairs’ long awaited Statute Law Database project has launched, free at point of use for anyone. It’s super. Last week, access to consolidated versions of the law of the UK wasn’t possible without paying lots of money. Now it is free. There are some down sides – 40 acts are […]

Parliamentary data, in a can

Public Whip and TheyWorkForYou have been around for a few years now, grabbing data from the UK Parliament website and making it more useful, and accessible with better navigation, search and email alerts. It’s little known, but we also make all that open data available in a separate screen scraping project called parlparse. Rufus first […]

Free access to the laws that bind us?

In many countries, such as the United States, laws are published as open data, which anybody can copy and reproduce. In the United Kingdom, only the changes to the law (patches, in computer science terms) are published freely. You can find them on the Office of Public Sector Information (OPSI) website. They are known as […]