Measuring the success of your project on Open Data

The following guest post is by Armand Brahaj, who works on Open Data Albania. He’s also a member of the OKF’s Working Group on Open Government Data.

I had the chance to visit the Transparency Camp in Washington DC this year. The event was organized by the Sunlight Foundation, who invited international activists that were involved in transparency projects all around the world. For two days I followed many interesting open talks and on the third day I decided that I might have something to share as well. I presented a modest topic that dealt with measuring the success of open data projects.

Before I go any further I should let you know a little bit about our project. It all started while I was watching a wonderful speech from Hans Rossling on TED.com a few years ago. I got inspired by the idea that we need more data, and knowledge derived from data, in my country. We have many journals, many TV stations, and a lot of radio stations, but the information which is transmitted by these media to the masses is barely information. The media in my country are often protecting the interests of private businesses, and the news supports or attacks their oponents in the political arena. News focuses on individuals in politics, sometimes describing them as immoral, sometimes as corrupt, sometimes as criminals. But rarely do we get facts, or data, or analyses based on numbers. This was not just my perception: a study showed that 76% of media focused on individuals, with only about 3% on issues. The remaining 21% is a mix of individuals and issues. If you have not guessed by now, my country is in the Balkans and that day I started imagining a project on Open Data Albania.

While trying to raise some funding for the project I managed to chat with a couple of really useful people. The first one was Tony Bowden from mysociety.org. We met in train station, while he was transiting near the city I temporarily live. He mentioned two things that we should improve in our initial project proposal – (good) PR, and risk assessments. Basically he was telling me that before we start a project, we should think about how to avoid failure! Next I spoke with Janet Haven from the Open Society Institute office in Budapest, whose advice was vital to our project. Janet explained to me that the project was good, but that we needed to consider who would make use of the data we publish. Ideally, we should collaborate with NGOs involved in advocacy that might need data we produce, and sometimes dig in the direction they need to bring the facts to light.

As much as this advice looks logical and meaningful, to a guy with a technical backgroud like me these things hadn’t seemed that crucial. Technical people tend to see in front of them a couple of thousand RDF triples stored in repository, with web services that will allow other technical people to reuse them, with mashups, nice API interactivity and anything they can challenge themselves. As much as I wanted our project to “help people”, this was my first conception of it. Many RDFs, some visualisation via Google API and let the community have it. Let them work on it, let them use it, and spread the knowledge!

I will not go into every detail of what changed after the first proposal. In brief, we focused on journalists and NGOs. If you want to spread information, use channels that already exist, rather than attempt to immediately create new ones. We tried to figure out how could we reach more people with data, and bring more analytical thinking to people. Of course, the medium is the media. Journalists in Albania and in almost every country are required to deliver news fast, and they have little time for investigations and data digging. They lack training in statistics as well. Instead of aiming to achieve a large number of visitors for our website, we decided that we should reach a large number of journalists and NGOs. Will they use our data? They will, of course – if we deliver it in the right form. We started by gathering data and creating brief articles. These articles were given to journalists who can connect the information we provide with socio-economic changes in the society and political reforms linked to our indicators. We did lots of PR and training work as well, and I consider the project successful.

###How do we measure the success of a project!?

Different project types have different metrics. Our project was about spreading data and knowledge while offering the chance to reuse our data based on the Open Linked Data principles.

Most of the people that have a project on the web will instantly ask about page-visitors. We decided that this was irrelevant for our project. We needed to measure with a different metric.

1.Influence

The first question we asked was: Are we influencing critical thinking? Are we having an impact?
In the 6 months of our project we had a median of 1.5 media reports quoting Open Data Albania in their articles. This is what were able to monitor. We were indeed having some influence in the media. Our data was being used. Media was bringing more data and knowledge to their audience.

2.Authority

How far do people believe what we say; trust and rely on our information?
We had a couple of NGOs use our data in advocacy. Some newspapers quoted and used our data to prove politicians wrong. And our data were being quoted by media supporting different political camps.

3.Dependency
In an Open Data project it is quite important how much data you put in the stream, but also how many organizations depend on your work. How far people are relying on your REST service to get data for their websites, and how your work affects operations of other organizations is an important indicator of success.

4.Branding

I‘m not an expert on the power of branding, but every project should aim to create branding that people can easy recognize. Measuring it is also easy in the www era. Enter your project name in Google, for example “Open Data Albania”[ http://www.google.com/search?q=%22Open+Data+Albania%22] and compare the result (the About xxx) with a great project that you know for example “SunLight Foundation”[ http://www.google.com/search?q=%22SunLight+Foundation%22]. You should be able to see and value how well you are doing.

The above metric for measuring success is based on open data projects but I think it can be adapted in other projects as well. It is not all about how much we do, but how much we influence society. Any feedback is welcomed.

I would like to thank Tony Bowden, Janet Haven and Andi Dobrushi for their support for Open Data Albania.