We’re very pleased that the diagram’s maintainers, Anja Jentzsch, Richard Cyganiak, and Chris Bizer, have decided to use CKAN to maintain a registry of information about the datasets, from which the diagram will be automatically updated. They have put out a call for up to date information about datasets included in the diagram until next Wednesday 8th September.

From their announcement:

We are in the process of drawing the next version of the LOD cloud diagram. This time it is likely to contain around 180 datasets altogether having a size of around 20 billion RDF triples.

For drawing the next version of the LOD cloud, we have started to collect meta-information about the datasets to be included on CKAN, a registry of open data and content packages provided by the Open Knowledge Foundation.

The list of datasets about which we have already collected information is be found here:

• http://www4.wiwiss.fu-berlin.de/lodcloud/

In addition to basic meta-information about a dataset such as its size and the number of links pointing at other datasets, we also collect additional meta-information about the license of the dataset, alternative access options like SPARQL endpoints or dataset dumps, and whether there exist a voiD description of the dataset or a Semantic Web Sitemap.

So if your dataset is not listed yet and you want to have it included into the next version of the LOD cloud, please add it to CKAN until next Wednesday (September 8th, 2010).

Also, if we have collected wrong information about your dataset or if your dataset is only partially described up till now, it would be great if you could add the missing information.

Guidelines about how to add datasets to CKAN as well as about the tags that we are using to annotate the datasets are found here:

• http://esw.w3.org/TaskForces/CommunityProjects/LinkingOpenData/DataSets/CKANmetainformation

We thank all contributors in advance for their input and help, which hopefully will allow us to draw the next version of the LOD cloud as accurate as possible.

Share This

Related posts:

1. New Linking Open Data group on CKAN
2. CKAN 0.9 Released
3. CKAN 0.11 Released

From the blog post:

As part of the government’s ongoing work around transparency, today we are releasing some of the custom software code we’ve developed – a CKAN module for Drupal. This is available for anyone to review, use, or modify. We’re excited to see how developers and colleagues across the world put this work to good use in their own applications and projects.

The code itself is attached to this blog post as a tar.gz file and contains one main package with two sub-packages within. This code release allows content to be synched from CKAN into Drupal. CKAN is the system we use as our “back end” to store information about all the data government has released. Drupal is a system to publish web content, and serves as our “front end” through which people can use to find our datasets and comment on them.

The main CKANPackage code creates a Drupal custom content type to represent data in the same way as CKAN. The first sub-package is the CKANImporter which imports packages from CKAN into Drupal and allows this to take place as a one-off batch import or as an update to the latest changes since a specified time. The second sub-package is CKANDatagovuk which correlates fields in CKAN with Drupal hooks.

The code release includes comments in the files to assist users with the functionality. You can of course contact us should you have any questions.

Share This

Related posts:

1. Canadian citizen-driven data catalogue datadotgc.ca is powered by CKAN
2. Data.gov.uk goes public - and its using CKAN!
3. Data.gov.uk Launched - and it’s Using CKAN

• When? 1st September 2010
• Where? Fjord Office, Friedrichstrasse 210, Berlin
• Register? You can register here!

Speakers will include:

• Martin Belam, The Guardian
• Jonathan Gray, The Open Knowledge Foundation
• Christian Heise, ZEIT Online
• Gerd Kamp, Deutsche Presse Agentur
• Georgi Kobilarov, Uberblic Labs
• John O’Donovan, BBC News
• Tom Scott, BBC Earth
• Ole Wintermann, Bertelsmann Foundation

From the blurb:

Data Journalism and the new and exciting possibilities that the Web of Data opens up for creators and consumers of news and media online will be the topic of this first meetup.

We have a brilliant lineup of speakers from media organisations like the BBC, The Guardian, the Deutsche Presse Agentur, the Bertelsmann Foundation coming to Berlin and talking about data journalism and the latest developments and projects in this field, and our friends from ZEIT Online will join the discussion.

The event takes place at the office of our friends at Fjord in the heart of Berlin. Starting at 2pm, you’ll hear talks followed by a panel discussion and an open space for working groups, and when the official programme ends at 7pm we’ll of course have drinks with all of you.

Language of all talks at the event will be English, but don’t be surprised to hear a bit of German here and there in conversations.

Share This

Related posts:

1. Data Driven Journalism, Amsterdam, 24th August 2010
2. Open Everything Berlin + CC Salon Berlin
3. Slides and notes from Data Driven Journalism event

• http://opengovernmentdata.org/camp2010/

You can register at:

• http://opengovdatacamp.eventbrite.com/

From the announcement:

What is it?

Basic details are as follows:

• What? A two day workshop for people interested in open government data.
• When? 18-19th November 2010
• Where? University of London Union, London, UK
• How much? Tickets cost £10 to help cover costs. You can sign up here!
• Hashtag? ##ogdcamp2010

Tell me more…

Its been a big year for open government data. Around the world governments and public bodies have been opening up official datasets for the public to reuse. There has been an explosion of new applications, competitions, hackdays and other initiatives from local authorities, central government departments, international bodies and others. This event will bring together movers and shakers from the world of open government data — including government representatives, policymakers, lawyers, technologists, academics, advocates, citizens, journalists and reusers.

What will happen?

There will be two days of discussions, drafting, planning and hacking. Crucially we hope to:

• Build consensus around key legal, technical and policy issues related to opening up government information.
• Strengthen the community of people working on different aspects of opening up official data around the world — from both inside and outside government. (Many people working on this area will not have met in person!)
• Encourage the exchange of experiences, expertise and ideas between those involved in leading open government data initiatives in different countries.
• Make things! We hope there will be plenty of space for developers to hack on things — from refining core bits and pieces of technology to rapid prototyping of new ideas.

What will the format be?

Presentations will be kept to a minimum. Each day will begin with a sprinkling of short talks followed by plenty of time to talk, plan and work on things.

Can I submit a presentation?

We are going to put out a call for short presentations (around 30 x 10 minute slots) shortly. Details/links will be posted on the open-government discussion list.

Can I propose a session?

Yes please! Again, we’re going to brainstorm, plan and schedule sessions on the open-government discussion list — so head there if you have any cunning ideas!

What kinds of topics will be covered?

Possible sessions include:

• How can we encourage other countries to open up official information?
• Open government data in law and policy: obstacles and opportunities
• Promoting reuse: competitions, community engagement, the role of the media
• Finding open government data: catalogues, registries and metadata
• Raw Data Now! Technical aspects of opening up government data
• The role and value of linked data
• Open government data and data journalism

What kinds of outputs will there be?

Projected outputs include things like:

• First draft of an international ‘open data manual’ (organised as a ‘Book Sprint’)
• A set of key open government data principles
• A timeline of key developments for open government data around the world
• A fairly comprehensive list of official initiatives — including data catalogues and competitions
• A list of key examples of the reuse of open government data
• Launch of RawDataNow.com — illustrating what we mean by ‘raw data’ aimed at those who publish official information
• Brainstorming about projects which would make it easier for citizens to find, analyse and visually represent the data they are looking for

Who’s behind the event?

Open Government Data Camp was conceived and is being primarily organised by the Open Knowledge Foundation. The event is also supported by:

• Cabinet Office, UK
• EU LAPSI project, Turin, Italy
• EU LOD2 project, Leipzig, Germany
• Guardian, UK
• Sunlight Foundation, USA

Who is coming?

You can find a list of participants at:

• http://opengovernmentdata.okfnpad.org/camp2010

If you add your name to the list, please don’t forget to register! (And vice versa: if you’ve registered, please also add your name to the pad page above…)

Can I sponsor the event?

Yes please! We are still actively seeking sponsorship for lunches, coffee, travel and accommodation for international participants and so on. If you think you might be interested, please contact jonathan dot gray at okfn dot org.

What countries will be represented?

We are currently expecting representation from:

• Argentina
• Australia
• Austria
• Belgium
• Brazil
• Canada
• Denmark
• Finland
• France
• Germany
• Hungary
• Iceland
• India
• Ireland
• Italy
• Luxembourg
• Netherlands
• New Zealand
• Norway
• Russia
• Spain
• Sweden
• Taiwan
• United Kingdom
• United States

Why do I have to pay?

The £10 ticket price is to help cover costs. If the ticket price is a problem, don’t hesitate to let us know. We won’t turn anyone away because they can’t afford to come!

Share This

Related posts:

1. Open Government Data in Austria
2. Mapping open government data initiatives around the world
3. Data Hunt events for Open Government Data?

The New Zealand Government Open Access and Licensing framework (NZGOAL) was approved by Cabinet on 5 July 2010 as government guidance for State Services agencies to follow when releasing copyright works and non-copyright material for re-use by third parties. It standardises the licensing of government copyright works for re-use using Creative Commons licences and recommends the use of ‘no-known rights’ statements for non-copyright material. It is widely recognised that re-use of this material by individuals and organisations may have significant creative and economic benefit for New Zealand.

NZGOAL does not apply to information or works containing personal or other sensitive information, except for its guidance on anonymising datasets which, once stripped of personal information, might be licensed or released.

State Services agencies should make their copyright works which are or may be of interest or use to people available online for re-use on the most open of licensing terms within NZGOAL – the Creative Commons Attribution (BY) licence – unless a restriction applies. Likewise they should provide public online access to non-copyright material that is or may be of interest to people, using a ‘no-known rights’ statement.

NZGOAL provides a series of open licensing and open access principles for copyright works and non-copyright material. These principles address, among other things, issues relating to open licensing, open access, creativity, authenticity, non-discrimination, open format and charging. It also guides agencies through the questions they need to consider before releasing material for re-use.

One of the key points is the adoption of the 100% open Creative Commons Attribution license as the default for government information:

Open access to copyright works with Creative Commons Attribution (BY) licence as default

26 Unless a restriction in paragraph 29 applies, State Services agencies should make their copyright works which are or may be of interest or use to people available for re-use on the most open of licensing terms available within NZGOAL (the Open Licensing Principle).6 To the greatest extent practicable, such works should be made available online. The most open of licensing terms available within NZGOAL is the Creative Commons Attribution (BY) licence.

They also explicitly discourage noncommercial licenses to make sure that open government data can be used (and contributed to!) by commercial users — a point which is also made in the Open Knowledge Definition:

Non-discrimination

33 Except where necessary to protect their own or others’ commercial or other interests, agencies should not discriminate, when selecting an NZGOAL licence, between individual, not-for-profit and commercial uses of the relevant copyright works (the Non-Discrimination Principle).

Finally they urge departments to publish in formats which make the data easy to reuse (machine readable ones?):

Formats

49 When licensing copyright works and releasing non-copyright material for re-use, agencies should:

a. consider the formats in which they ought to be released, taking into account, where relevant, the wishes of those who will or are likely to re-use the works or material;

b. release them in the formats they know or believe are best suited for interoperability and re-use and are searchable and indexable by search engines; and

c. in the case of datasets, add their details into data.govt.nz.

50 When releasing works or material in proprietary formats, agencies should also release the works or material in open, non-proprietary formats (the Open Format Principle).

You can read the full document online here:

• http://www.e.govt.nz/policy/nzgoal/html-version

Amazing work — and we hope that other governments will take note and follow suit!

If you’re interested in open government data, you can join in discussion on our open-government list which includes representatives from key initiatives around the world, both within and outside government.

Share This

Related posts:

1. Eduserv study on open content licensing in cultural heritage sector published
2. New report on access to information and open government data
3. Australian government releases open data for MashupAustralia competition

Russ has written back to us with some notes and questions from a session on this at OSCON:

Okay, so, as promised, here is my report on the “Open Data Definition” BOF held on Wednesday, July 21, at 7PM. There were about ten people present, which is a reasonable attendance, particularly when set against the Google Android Hands-on session at which they gave out free Nexus One phones.

Didn’t seem wise to me to start from scratch, especially given the good work done by the Open Knowledge Foundation on their Open Knowledge Definition: http://www.opendefinition.org/okd/. So we read through it section by section, by way of review. Here are the questions we arrived at (thanks to Skud aka Kirrily Robert for taking notes):

1. What happens with data that’s not copyrightable? 1a. What about data that consists of facts about the world and thus even a collection of it cannot be copyrighted, but the exact file format can be copyrighted? Many sub-federal-level governments in the US have to publish facts on demand but claim a copyright on the formatting.
2. What about data that’s not accessible as a whole, but only through an API?
3. We’re thinking that OKD #9 should read “execution of an additional agreement” rather than “additional license”.
4. Does OKD #4 apply to works distributed in a particular file format? Is a movie not open data if it’s encoded in a patent-encumbered codec? Does it become open data if it’s re-encoded?
5. What constitutes onerous attribution in OKD #5? If you get open data from somebody, and they have an attribution page, is it sufficient for you to comply with the attribution requirement if you point to the attribution page?

This serves as an invitation to discuss these issues on the new list open-data@opensource.org . Send subscription requests to open-data-subscribe@opensource.org . Unsubscribe by sending a request to open-data-unsubscribe@opensource.org .

If these issues are successfully resolved, then this committee will recommend to the OSI board that the OKD should be adopted as OSI approved. If they can’t be resolved by, say, the end of 2010, then we will give up on trying. Either way, the intent is to lay down the list by the end of this year unless the participants desire otherwise.

So if you’d like to join the conversation, please join the list! We’ve also created an Etherpad to gather responses to some of these issues:

• http://okfnpad.org/okd-questions

Share This

Related posts:

1. Should the Open Source Initiative adopt the Open Knowledge Definition?
2. Belarusian translation of the Open Knowledge Definition (OKD)
3. Russian translation of the Open Knowledge Definition (OKD)

Here at the National Council for Voluntary Organisations (NCVO) we’ve recently started taking an interest in open data, and its implication for charities and the voluntary sector.

We know that some voluntary organisations which specialise in open data have been leading the charge - the Open Knowledge Foundation is a not-for-profit company, mySociety is a registered charity - and often the most exciting and innovative uses of open data are made by volunteers in their spare time. But we know that many voluntary organisations find it difficult to find the time and skills to develop their ICT capabilities, and can find the challenge of implementing new technologies in their organisation daunting. This is daunting not just because of the time and resources required, but also because it requires a change in organisational culture.

Our interest in open data culminated recently when the Coalition government published its paper Building the Big Society, which included five themes. The NCVO research team have looked at the evidence behind each of these themes - you can see the results on our website.

The first four themes relate well to NCVO’s usual work - they talk about participation, transferring power, communities and supporting organisations - but the fifth (publish government data) was less familiar, and is a new concept to much of the voluntary and community sector.

Looking at this fifth theme, I came up with a number of opportunities and challenges that open data presents for charities and voluntary organisations:

1. Open data will give charities new ways to find and share information on the need of their beneficiaries - who needs their services most and where they are located.

The sharing of information will be key to this - it’s not just about using data that the government has opened up, but also opening your own data. Organisations like New Philanthropy Capital and NCVO’s own Strategy and Impact unit stress the need for charities to demonstrate the impact that their services have - opening their data can help to do this.

This can create a more joined up service for users, provide cost savings and mean that organisations can meet unmet needs. But organisations need to think about how to access and manipulate this information - will it need specialist staff or volunteers? Some organisations might need outside help to be able to do this.

2. Charities will be able to use the evidence found in open data to boost their campaigns and lobby government. Voluntary sector organisations have been at the forefront of opening up data.

Providing services directly to those in need isn’t the only way that charities help the most vulnerable - their campaigning and lobbying also has a very important role in this. Open data can help charities speak truth to power, whether it’s challenging government spending or using their own data to lobby for better targeting of services. Data that charities gather themselves through their beneficiaries and communities can make the case even more forcefully, but again charities need to have the skills to be able to do this.

3. Many of the skills needed to create, access and use open data are not yet widespread in the voluntary sector. There is a cost to effectively creating and using this data, while sharing commercially sensitive data could reduce competitiveness.

This is a really important point - the uneven spread of the skills, knowledge and resources needed means that some organisations risk being left behind while others use open data to its full potential. At the moment much of work being done is by interested and passionate individuals, but there may not be enough of these to go around.

4. As open data becomes embedded in government, voluntary organisations which contract with government may be compelled to produce and share data as part of those contracts.

This is a bit hypothetical at the moment - I’ve seen no evidence of this happening in government contracts yet. But it seems possible to me that as a culture of open data becomes embedded in government, this culture informs their contracting arrangements. If this does happen, charities will need to be ready.

Conclusion

So how does the voluntary sector keep up with the open data revolution? Well it needs to make sure that staff and volunteers have the skills and knowledge needed to create and use open data. Charities need to learn from each other too, particularly by talking to organisations that are ahead of the game. Perhaps most importantly, examples of the power of open data will show charities how important this is.

Share This

Related posts:

1. Opening up European public sector information: two recommendations
2. Meeting on UK Public Sector Information Re-use Request Service
3. New report published: The Socioeconomic Effects of Public Sector Information on Digital Networks

In recent years the practice and philosophy of making data freely available for use and re-use has been taken up by many different institutions, from national governments to international organizations such as the World Bank. Journalists too have started to tap into the potential of free accessible data.

Now, with major news providers such as The Guardian beginning to open up the data they work with, the opportunities for producing visualisations or web mash-ups based on this information readily exist.

Increasing online open data availability puts the processing power into journalists’ hands; rather than relying on outside specialists such as policy makers to provide the insights, raw data can now be analysed and interpreted in newsrooms.

This is the emerging field of data-driven journalism, in which journalists gather, analyse and visualise ‘big’ data and combine it with compelling, credible storytelling. 

European Journalism Centre (EJC) questioned Jonathan Gray (see right), community coordinator at the Open Knowledge Foundation, a UK non-profit organisation dedicated to promoting open knowledge internationally, about how journalists can exploit the potential of open data. Jonathan is also the founder of WhereDoesMyMoneyGo.org, a web application which allows users to explore and visually represent UK public spending.

EJC: Where and how can journalists access free and open data sources on the Internet?

Jonathan Gray: There is currently a huge wealth of freely accessible data on the Internet, scattered over web real estate belonging to government departments, academics, NGOs, news organisations, technologists and others. Yet despite the fact that there is so much information out there, it is not always easy to find the exact information you are looking for; get hold of raw data sources (as opposed to reports, articles and other material about these sources), and find out whether or not you can freely reuse the information (just because something is accessible online, this does not always mean you’re allowed to reuse or republish it).

To address these issues we started the CKAN project, which aims to make it easy to find collections of documents and datasets which anyone can freely re-use for any purpose.

CKAN is now being used by the UK government in its data.gov.uk project, and we are helping open government data advocates around the world set up new instances to track sources of open data in over a dozen countries. We’re also working hard to make sure that other data catalogues in other countries are interoperable with CKAN, and to promote uptake of the project in different communities creating and using public datasets - from geospatial data analysts to civic hackers, from climate modellers to semantic web technologists. We hope that CKAN will become an international, multilingual, distributed one-stop shop for open data.

If you are interested in using this technology to set up a data registry in your country (whether you’re an advocate or a civil servant), we’d love to hear from you.

JG: Having some programming skills is no doubt useful for journalists whose work may depend on extracting, analysing, and understanding information in large databases. This may be particularly valuable for investigative journalists trawling public sources to build up a richer picture of complex chains of events or states of affairs. However new digital technologies are making it increasingly easy for journalists without programming skills to explore and analyse datasets.

Social web services such as Many Eyes or Google Chart Tools mean that anyone can visually represent data sets ‘on the fly’. Free and open source desktop applications also enable people to drill down into databases in increasingly sophisticated ways.

While these kinds of tools can go a long way, there are obviously still limits as to what non-technical journalists can do. A good example of this is the recent release of the COINS data in the UK – a huge release of information on public spending, unprecedented in its scope and detail. We had journalists from many different national newspapers and news outlets calling us to ask what was in the data, and in particular whether we had found any good stories. At the time several people expressed disappointment with the release, basically complaining that getting useful or interesting information from the database was like getting blood from a stone. The good news is that since then we’ve seen at least half a dozen new projects which let people sort, search, explore and comment on the data, and no doubt we will see many more in the next few months and years.

To quote Peter Murray-Rust, OKF Advisory Board member and tireless advocate of open data in chemistry, “Data is difficult”. Whether we’re talking about statistical data, environmental data or spending data, the chain from production to presentation can often be long and complicated. The more people involved in the process of cleaning up, checking, interpreting, and visually representing datasets the better. Hence at the OKF we are strong advocates of a community-driven approach, involving experts from across the board, as well as interested and motivated citizens.

We hope to move to a situation where rather than a single official point of contact for datasets (whether from national governments, international bodies or NGOs), we have an ecosystem of open data with lots of datasets connected together, accessible via many different interfaces and with plenty of tools to help people understand and interact with the data.

Rather than the traditional treasure hunt, for example looking for data buried deep on an official website or PDF document, working out how to use the shiny front end interface, etc., we hope there will be more of a two way relationship with information around us, i.e. delivery on demand according to interests, read/write access, commenting, telling stories with data, enabling people to embed dynamic visual representations which link back to source, and so on. By explicitly opening up datasets for others to freely reuse without restriction we allow a thousand flowers to bloom.

While the core task of journalists will presumably continue to be much the same, i.e. interpreting, communicating and framing the information in meaningful narratives, with comment and analysis, I think the precise division of labour between journalists and others remains to be seen. Hopefully we’ll see some boundaries begin to blur.

EJC: What are the common challenges associated with data usage, production and presentation? And what is there to learn for journalists who want to go in this direction from your perspective?

JG: As I alluded to above, piecing together and interpreting data can be hard. (Getting data in the first place can also be hard—but that’s another story!). There are lots of great examples of this from Ben Goldacre’s analysis of a recent Guardian article take on NHS death rates, to dodgy newspaper graphs (see this presentation from Simon Field, CTO of the UK’s Office for National Statistics).

We’ve experienced a number of difficulties when trying to make sense of UK spending data as part of the Where Does My Money Go? project, including (but by no means limited to) missing data, different figures from different government sources, retired schema, absent keys, changing categories, multiple different codes, data delivered on thousands of sheets of paper, and so on.

Before this project I never knew working out where public funds are spent could be so much like theology, with some intangible, ineffable base layer (the actual spending) and many different denominations of interpreting this for different purposes, with different rules, different explanations, different calendar dates and different terminology. Luckily it’s all falling into place, but not without talking to lots of helpful government folks and outside experts.

To understand a dataset is often not just to understand a bunch of facts about the world, but to understand all kinds of assumptions and processes involved in the production of that data - which can be critical, for example, in understanding whether the books have been cooked. Another good reason why it’s better to have many pairs of eyes and hands!

In terms of things for journalists to learn, I guess I’d like to see us move from a situation where we consult data on a case-by-case basis in order to provide facts for stories, to one where we allow stories to emerge from the data (i.e. data-driven journalism!). For example, more systematically following transcripts and drilling down into datasets from national parliaments and international bodies such as the EU or the UN. Also being able to give events a broader sense of context, for example by being able to link directly to documents and datasets and letting people discuss and comment on these.

I think visualisation tools may turn out to be very powerful with respect to giving people a ‘bigger picture’. We are currently taking baby steps, but I’m sure things will pick up pace as the technologies begin to get more robust, and we start to build a visual language for interfacing with datasets, similar to the kinds of graphical user interfaces we are all now used to with our operating systems, or on web browsers and web applications.

EJC: Do you think journalists are capable of producing offerings similar to your Where does my money go? project or to CKAN?

JG: Yes of course! There is no compelling reason why journalists shouldn’t continue to contribute to these kinds of projects. I guess it’s more a question of people’s time and priorities.
Starting to have more provision for learning to code on journalism courses might be a good way to systematically encourage journalists to start working on and engaging with these kinds of projects. But I think it’s often also to do with motivation. Many really good programmers I know had no formal training, but learned by having a project they wanted to do and teaching themselves with input from various free/open source software communities. Formal training is often neither necessary nor sufficient to code on projects like Where Does My Money Go, They Work For You, and so on - its far more important to set aside the time, and stick at it.

Having more widespread acceptance of this sort of thing as something which is ‘legitimate’ for journalists to undertake may well help people to justify spending their time in this way (to themselves, to employers, etc). There already seems to be growing interest here - e.g. from BBC, Guardian, Telegraph, New York Times, and so on – so hopefully it won’t be long before programming comes to be accepted as a valuable part of a journalist’s repertoire.

EJC: Your work provides a great example, though we think one aspect is missing: storytelling. Would you consider presenting short stories, interviews, videos on top of your data in the future?

JG:Yes definitely. I think one of the main things that sets Hans Rosling’s Gapminder apart from other similar projects is his wonderfully entertaining set of lectures where he talks about an issue (population growth, development, health, quality of life, etc) with reference to Trendalyzer, the Gapminder visualisation tool. We’re really proud to have Hans on the Open Knowledge Foundation’s advisory board, and I think he’s someone who really understands the relationship between stories and interactive visualisation tools. On the one hand his narrative makes the colourful bubbles meaningful, on the other hand the colourful bubbles are indispensable in his lectures! A good illustration of why a picture is worth a thousand words is to watch any of his videos where he narrates hundreds of bubbles moving around the screen—highlighting a significant move like a football commentator highlighting something in a game. While the words may aid understanding and capture key aspects, they are no substitute for the image. They are also clever because after you’ve watched a few, you have a pretty good idea how to use the tool to ask your own questions and build your own visualisations.

These kinds of tools are new, and can often not be particularly intuitive to use, partly because we have no ‘accepted’ way of doing things yet. Tutorials and help menus can be boring, and the video lectures are a great way of teaching people to use the tool without them noticing! We’d definitely like to do more of this kind of thing with our own projects. In general I think that storytelling will play a very important part in new visualisation tools—from creating narratives within the visual design and the user’s experience, through to being able to seamless embed and integrate visualisations within external articles, texts and other media.

Share This

Related posts:

1. Data Driven Journalism, Amsterdam, 24th August 2010
2. Slides and notes from Data Driven Journalism event
3. Data Journalism Meetup, Berlin, 1st September 2010

The European Journalism Centre in collaboration with the University of Amsterdam organises the first round table on data-driven journalism on 24 August in Amsterdam. The one day event brings together specialists in fields which intersect with data-driven journalism: data mining, data visualisation and multimedia storytelling to discuss the possibilities of this emerging field, examine and understand the needed tools and workflows, and spread the know-how for data-driven journalism. What can we learn from the existing projects? How can we integrate the existing tools in the journalistic workflows? What skills are needed to enter this field? These are just a few of the issues which will be addressed in this event.

In particular I’m keen to talk about with the other participants about open data, and the role journalists can play in helping to open up official information and to help present it to the public in new ways. They asked me for a quote to use for the event:

Opening up content and data produced by public bodies will enable new forms of reportage as well as a new generation of services enabling the public to participate in the news making process. New tools to analyse, represent, deliver and give context to public data are beginning to revolutionise the way we understand large and complex issues, from Hans Rosling’s analysis of flu statistics, to the Guardian MP expenses crowdsourcing tool, and to the Afghanistan Election Data project. An ecosystem of open data that anyone can reuse or contribute to will be critical for a new generation of data driven journalism to flourish.

You can find out more at:

• http://datadrivenjournalism.net/

If you’re going to be in Amsterdam, participation is free and you can register here.

Share This

Related posts:

1. Slides and notes from Data Driven Journalism event
2. Interview with European Journalism Centre on Data Driven Journalism
3. Data Journalism Meetup, Berlin, 1st September 2010

As interfaces for open datasets from political and societal institutions become more and more available, the possibilities for easy and uncomplicated data visualization are expanding in very promising ways. With a little programming knowledge, or a bit of support, journalists and bloggers are able to back up the conclusions in their articles with facts in a illustrative way, using diagrams or maps. Even further, they can create, demonstrate, or underline interrelations through the integration of different datasets, using programming interfaces.

Very active in offering such programming interfaces (APIs) is the Worldbank, which provides an API for querying indicators relevant for describing the world’s development status, like birth rates, CO2 emission levels, and education expenditure. The aim of this article is to show how such data can be used, particularly, as an example, how it could be visualized on a map with the help of Google Maps. The following map shows the income level in different countries through colored pins. Clicking on a pin brings up additional information about the capital of a country and the meaning of the corresponding colour. The explanation is (hopefully) not too technical, so that it should be comprehensible to non-programmers as well, at least in its essentials. Some programming skills will be necessary for a realization, but it shouldn’t take more than 2-3 hours.

Show bigger map

The following three steps are necessary on the way to your own Worldbank open data mashup:

1. Worldbank API - Select indicators and formulate query

The API from the Worldbank can be used directly via a URL in the browser. You can choose which indicator or which country you wish to find results for by specifying the parameters in the URL. For example, the following query:

• http://open.worldbank.org/countries?per_page=100&incomeLevel=LIC

returns a list with all countries with a low income level (LIC). (You can get a more structured view of the result by selecting the source view in your browser.) A more detailed explanation about the usage of the API can be found on the website of the Worldbank. The fact that you can use the API directly through the browser also gives you the chance to play around a bit with the different parameters to get a better feeling for what the API can do. Once you have created a useful API URL (in our example: http://open.worldbank.org/countries?format=json&per_page=500), the URL can be queried through the corresponding programming function (e.g. cURL for the programming language PHP used in our example).

2. Convert the result to a format readable by Google Maps

Now the queried data has to be converted in a format readable by Google Maps. A good way to go here is KML, which is a descriptive language for geodata, used for example to locate places on a map or annotate them with additional information. There is also the alternative possibility of using the Google Maps-API directly to visualize the data. The advantage of the KML option is that at the end of the process it comes out with a code snippet which can be copied straight into weblogs and content management systems. The datasets from the Worldbank API are returned in XML or JSON, both structured data formats used to represent several datasets of different kinds and corresponding properties. Generally there are standard programming functions to process these formats in the different languages, for example in PHP the function “json_decode()” is used read datasets given in the JSON format. Now you can loop through the single datasets and write the properties, which should be presented on a map, in a KML string. A list of the possible KML properties which can be used can be found in the KML documentation hosted by Google. In our example the main properties are the name of the country and the income level, which should be shown when selecting a pin on the Google map, and the longitude and latitude of the capital of the corresponding country (see illustration below). In the process of this transformation it is also possible to carry out some graphical formatting, for example representing every country with a low income level through a red pin. The created file now has to be saved somewhere on a web server as filename.kml, so that it is accessible through the web.

3. Integrate into blog/article

Phew! Maybe that last section really was a bit technical! But the good part is, now you are more or less ready! Google Maps can process KML files directly, so that you can copy the corresponding URL straight into the search field of Google Maps. If you have done everything correctly, the datasets taken from the Worldbank API should be shown on the resulting map. Anyone who wants to try this can take the KML file URL used for this example:

• http://www.opendata-network.org/wp-content/uploads/2010/07/Worldbank_Data-API_GoogleMaps_IncomeLevel.kml

Copy the link into the search field and look what happens. Via “Link” -> “Customize and preview embedded map” the desired clipping and zoom level of the map can be selected, and: ready! The HTML code which you have thus created can now be copied into your own website, and the map with the data overlay will automatically be loaded via Google!

Conclusion

Hopefully this article shows how easy it is - even by today’s standards - to integrate data from openly available data sources into your own website. With a little imagination and some programming skills, much more can be realized than shown in this example. Comparisons can be made by overlaying different datasets, or through the use of timelines. Maps can be complemented by your own datasets, or by data from other open programming interfaces. So, grab your keyboard! And anyone who has experimented a bit and has created interesting visualizations: it would be great if you added a comment below!

Share This

Related posts:

1. Google vs Facebook
2. Striking confirmation from Google of the problems with ‘open’ APIs
3. Canadian citizen-driven data catalogue datadotgc.ca is powered by CKAN