We’re very pleased that the diagram’s maintainers, Anja Jentzsch, Richard Cyganiak, and Chris Bizer, have decided to use CKAN to maintain a registry of information about the datasets, from which the diagram will be automatically updated. They have put out a call for up to date information about datasets included in the diagram until next Wednesday 8th September.

From their announcement:

We are in the process of drawing the next version of the LOD cloud diagram. This time it is likely to contain around 180 datasets altogether having a size of around 20 billion RDF triples.

For drawing the next version of the LOD cloud, we have started to collect meta-information about the datasets to be included on CKAN, a registry of open data and content packages provided by the Open Knowledge Foundation.

The list of datasets about which we have already collected information is be found here:

• http://www4.wiwiss.fu-berlin.de/lodcloud/

In addition to basic meta-information about a dataset such as its size and the number of links pointing at other datasets, we also collect additional meta-information about the license of the dataset, alternative access options like SPARQL endpoints or dataset dumps, and whether there exist a voiD description of the dataset or a Semantic Web Sitemap.

So if your dataset is not listed yet and you want to have it included into the next version of the LOD cloud, please add it to CKAN until next Wednesday (September 8th, 2010).

Also, if we have collected wrong information about your dataset or if your dataset is only partially described up till now, it would be great if you could add the missing information.

Guidelines about how to add datasets to CKAN as well as about the tags that we are using to annotate the datasets are found here:

• http://esw.w3.org/TaskForces/CommunityProjects/LinkingOpenData/DataSets/CKANmetainformation

We thank all contributors in advance for their input and help, which hopefully will allow us to draw the next version of the LOD cloud as accurate as possible.

Share This

Related posts:

1. New Linking Open Data group on CKAN
2. CKAN 0.9 Released
3. CKAN 0.11 Released

From the blog post:

As part of the government’s ongoing work around transparency, today we are releasing some of the custom software code we’ve developed – a CKAN module for Drupal. This is available for anyone to review, use, or modify. We’re excited to see how developers and colleagues across the world put this work to good use in their own applications and projects.

The code itself is attached to this blog post as a tar.gz file and contains one main package with two sub-packages within. This code release allows content to be synched from CKAN into Drupal. CKAN is the system we use as our “back end” to store information about all the data government has released. Drupal is a system to publish web content, and serves as our “front end” through which people can use to find our datasets and comment on them.

The main CKANPackage code creates a Drupal custom content type to represent data in the same way as CKAN. The first sub-package is the CKANImporter which imports packages from CKAN into Drupal and allows this to take place as a one-off batch import or as an update to the latest changes since a specified time. The second sub-package is CKANDatagovuk which correlates fields in CKAN with Drupal hooks.

The code release includes comments in the files to assist users with the functionality. You can of course contact us should you have any questions.

Share This

Related posts:

1. Canadian citizen-driven data catalogue datadotgc.ca is powered by CKAN
2. Data.gov.uk goes public - and its using CKAN!
3. Data.gov.uk Launched - and it’s Using CKAN

We didn’t build libraries for a literate citizenry. We built libraries to help citizens become literate. Today we build open data portals not because we have public policy literate citizens, we build them so that citizens may become literate in public policy.

In a brilliant article on The Guardian website, Charles Arthur argued that a global flood of government data is being opened up to the public (sadly, not in Canada) and that we are going to need an army of people to make it understandable.

I agree. We need a data-literate citizenry, not just a small elite of hackers and policy wonks. And the best way to cultivate that broad-based literacy is not to release in small or measured quantities, but to flood us with data. To provide thousands of niches that will interest people in learning, playing and working with open data. But more than this we also need to think about cultivating communities where citizens can exchange ideas as well as involve educators to help provide support and increase people’s ability to move up the learning curve.

Interestingly, this is not new territory. We have a model for how to make this happen – one from which we can draw lessons or foresee problems. What model? Consider a process similar in scale and scope that happened just over a century ago: the library revolution.

In the late 19th and early 20th century, governments and philanthropists across the western world suddenly became obsessed with building libraries – lots of them. Everything from large ones like the New York Main Library to small ones like the thousands of tiny, one-room county libraries that dot the countryside. Big or small, these institutions quickly became treasured and important parts of any city or town. At the core of this project was that literate citizens would be both more productive and more effective citizens.

But like open data, this project was not without controversy. It is worth noting that at the time some people argued libraries were dangerous. Libraries could spread subversive ideas – especially about sexuality and politics – and that giving citizens access to knowledge out of context would render them dangerous to themselves and society at large. Remember, ideas are a dangerous thing. And libraries are full of them.

Cora McAndrews Moellendick, a Masters of Library Studies student who draws on the work of Geller sums up the challenge beautifully:

…for a period of time, censorship was a key responsibility of the librarian, along with trying to persuade the public that reading was not frivolous or harmful… many were concerned that this money could have been used elsewhere to better serve people. Lord Rodenberry claimed that “reading would destroy independent thinking.” Librarians were also coming under attack because they could not prove that libraries were having any impact on reducing crime, improving happiness, or assisting economic growth, areas of keen importance during this period… (Geller, 1984)

Today when I talk to public servants, think tank leaders and others, most grasp the benefit of “open data” – of having the government sharing the data it collects. A few however, talk about the problem of just handing data over to the public. Some questions whether the activity is “frivolous or harmful.” They ask “what will people do with the data?” “They might misunderstand it” or “They might misuse it.” Ultimately they argue we can only release this data “in context”. Data after all, is a dangerous thing. And governments produce a lot of it.

As in the 19th century, these arguments must not prevail. Indeed, we must do the exact opposite. Charges of “frivolousness” or a desire to ensure data is only released “in context” are code to obstruct or shape data portals to ensure that they only support what public institutions or politicians deem “acceptable”. Again, we need a flood of data, not only because it is good for democracy and government, but because it increases the likelihood of more people taking interest and becoming literate.

It is worth remembering: We didn’t build libraries for an already literate citizenry. We built libraries to help citizens become literate. Today we build open data portals not because we have a data or public policy literate citizenry, we build them so that citizens may become literate in data, visualization, coding and public policy.

This is why coders in cities like Vancouver and Ottawa come together for open data hackathons, to share ideas and skills on how to use and engage with open data.

But smart governments should not only rely on small groups of developers to make use of open data. Forward-looking governments – those that want an engaged citizenry, a 21st-century workforce and a creative, knowledge-based economy in their jurisdiction – will reach out to universities, colleges and schools and encourage them to get their students using, visualizing, writing about and generally engaging with open data. Not only to help others understand its significance, but to foster a sense of empowerment and sense of opportunity among a generation that could create the public policy hacks that will save lives, make public resources more efficient and effective and make communities more livable and fun. The recent paper published by the University of British Columbia students who used open data to analyze graffiti trends in Vancouver is a perfect early example of this phenomenon.

When we think of libraries, we often just think of a building with books. But 19th century mattered not only because they had books, but because they offered literacy programs, books clubs, and other resources to help citizens become literate and thus, more engaged and productive. Open data catalogs need to learn the same lesson. While they won’t require the same centralized and costly approach as the 19th century, governments that help foster communities around open data, that encourage their school system to use it as a basis for teaching, and then support their citizens\’ efforts to write and suggest their own public policy ideas will, I suspect, benefit from happier and more engaged citizens, along with better services and stronger economies.

So what is your government/university/community doing to create its citizen army of open data analysts?

Other posts by David that you might find of interest include:

• Open Data as an Old Idea - Gov 2.0 Ignite Talk
• Case Study: How Open data saved Canada $3.2 Billion
• Three laws of Open Data
• 3 Ways Open Data are making Vancouver better

Share This

Related posts:

1. Libraries in Cologne open up bibliographic data!
2. Launch of it.ckan.net for open data in Italy!
3. Open Government Data Camp 2010, 18-19th November 2010

We are delighted to announce that an Italian instance of CKAN is now live! You can see this at:

• http://it.ckan.net/

The translation was undertaken by Stefano Costa, who is coordinator of the OKF working group on open data in archaeology.

There are currently 67 packages available — thanks to the Extracting Value from Public Sector Information (EVPSI) project. In particular, the NEXA Center contributed material generated as part of the EVPSI project, which is funded by the Piedmont Region and coordinated by the University of Turin.

The site was launched on Sunday by OKF Director Rufus Pollock and NEXA Center co-director Juan Carlos De Martin at the 2010 Festival of Economics in Trento and is a collaboration between the Open Knowledge Foundation, the EVPSI project and the NEXA Center for Internet & Society.

The datasets that are currently available on the Italian instance of CKAN come from a first mapping of some of the main silos of public sector information (PSI) in Italy. Many more packages will be provided soon by EVPSI and the NEXA Center, as a product of a much more detailed mapping of PSI holding entities in the Italian Region of Piedmont.

Open data in Italy

Is Italy behind other countries with respect to open data? Judging from the data of the EVPSI project (and from the infringement procedure the the EU started against Italy), the answer to this question is ‘yes’, but things are changing. The Italian CKAN will hopefully help accelerate this change - providing a way for open data users and distributors to find datasets and see whether or not they can reuse them!

The new datasets on it.ckan.net include many which aren’t open, to help people get a ‘big picture’ about what datasets are out there, who holds them, how to download them and how open they are.

There are several bodies that produce data for their own institutional purposes, but most of the databases with clear commercial interest are only available by paying. And even when data are made available on the web they are distributed under restrictive terms of use or under unclear or no terms of use at all. That, considering the default status of potentially copyright and/or database right protected material (i.e. “All rights reserved”) implicitly means that no re-use is possible. This attitude is caused by a combination of factors, including:

• lack of knowledge about the open data initiative and the benefits of open data for citizens and society at large
• complex sub-licensing of datasets among many different public and private bodies, so that nobody can be considered the actual owner of data
• a general fear of situations implying a loss of control over the re-use of data (coupled with a lack of internal guidelines about the access and re-use of data)
• a difficult financial situation of PSI holders, pushing them to maximize their short run monetary income, without appropriately taking into account positive spillovers for the rest of society and in the medium/long run

For example ISTAT, the national institute of statistics, put their data online for free use, but unfortunately commercial reuse is not allowed - which may inhibit the development of innovative applications and services. See an overview of ISTAT datasets at CKAN.

A notable exception to this mindset is Regione Piemonte, that has recently launched a portal for open data at:

• http://dati.regione.piemonte.it/

That result has been facilitated by the existence of common regional guidelines about the re-use of public data. What is more, all their currently available data are released under the CC0 license, enabling unrestricted re-use and dissemination by anyone, even for commercial purposes.

There are other regional governments offering some of their data (for example geospatial data) for free, but Piemonte is the only one explicitly adopting an open license. In all other cases, one has to ask for each case, and usually the answer is “free for non-commercial use” only.

The key point is that national and regional governments own large datasets that would be quite easily made available to the public. This process would however require 3 distinct actors, as outlined in the Open Data study by Becky Hogge:

• government heads
• civil servants (acting as the “middle layer”)
• a small but determined group of citizens (or “civic hackers”)

Minister Brunetta promised “data.gov.it” in 6 months, but in the meantime we would like to get a more detailed picture of how open Italian public information is. In particular it will be interesting to see if any local authorities besides Regione Piemonte will consider following in the footsteps of many other local and national bodies around the world - and open up their data!

Interested in starting a new CKAN instance in your country?

If you’re interested in starting a new instance of CKAN for open data in your country, the Open Knowledge Foundation would be delighted to help! If you are able to help coordinate the translation and liaise with other local folks interested in open data — we can set up, host, and maintain the instance on our servers. Just pop us a line on the ckan-discuss list:

• http://lists.okfn.org/mailman/listinfo/ckan-discuss

Share This

Related posts:

1. Launch of Norwegian instance of CKAN!
2. Canadian citizen-driven data catalogue datadotgc.ca is powered by CKAN
3. Italian Translation of the Open Knowledge Definition (OKD)

A new report released late last month charts the history of open government data in the UK and the US, and it’s a fascinating read. Written by OKF board member Becky Hogge for a consortium of grant-giving organisations including the Hewlett Foundation, the Ford Foundation, the Omidyar Network, the Open Society Institute and DfID, the Open Data Study:

“…explores the feasibility of advocating for open government data catalogues in middle income and developing countries. Its aim is to identify the advocacy strategies used in the US and UK data.gov and data.gov.uk initiatives, with a view to building a set of criteria that predict the success of similar initiatives in other countries and provide a template strategy to opening government data.”

I was interviewed for the report, as were John Wonderlich from the Sunlight Foundation, Tom Steinberg from mySociety and Ory Okollah from Ushahidi. Other interviewees include experts like Ethan Zuckerman and Toby Mendel, and - of course - Sir Tim Berners Lee.

The report draws some new and surprising conclusions. As well as recognising the role of organisations like the OKF and mySociety in bringing about data.gov.uk, it emphasises how crucial engagement with civil servants was to the success of the open data project in the UK. It raises interesting questions about what motivates politicians to embrace open data strategies, and even posits that the long battle to open up geospatial data in the UK worked in a positive way: “the barrier [opening geospatial data] imposed in the UK may have served as a common call to action among both civil society and the middle layer government administrators, which in turn served to strengthen the crucial communication between these two groups in the trajectory towards data.gov.uk, and ultimately enrich the final proposition when compared to data.gov.”

The report contains mixed findings about the prospects of similar projects in developing and middle income countries, providing a useful and very detailed checklist for advocates working within those countries to consult, and pointing to the potential role of international donors in this context. In short, I’d recommend reading this report to anyone interested in open government data, or indeed, in advocacy generally. Because, as Becky notes in her blog post introducing the report:

“I’d be hard pressed to think of an idea that has permeated as quickly as open data has from the fringe to the centre.”

Share This

Related posts:

1. Open government data in Germany: what next?
2. Open government data in Norway: mounting interest but no breakthrough yet
3. Data.gov.uk goes public - and its using CKAN!

A series of promising new initiatives gives reasons to be a lot more optimistic about government data reuse in Norway today than anyone could have been a year ago. The right tools will hopefully soon be available. Now convincing examples of reuse are needed.

When the application deadline expired at midnight we had received 135 applications to Nettskap! We are very pleased!

This exuberant tweet was published on the Ministry of Government Administration’s official account on May 10. The idea competition Nettskap 2.0, a Norwegian version of the Apps for Democracy or Show Us a Better Way contests, had been announced just a month earlier, which makes the quantity of applications quite impressive. Nettskap applicants were not required to place government data reuse at the heart of their projects, but the Ministry had encouraged this. It worked - 90 projects, two out of three, are based on the reuse of data.

Winners of the competition are to be announced in June, and will receive funding totaling 2,5 Million Kroner (around 320.000 Euro).

Nettskap is just one of several pieces of good news for government data enthusiasts coming out of Norway lately. In April, the same Ministry unveiled plans for a Norwegian datastore. The URL has been chosen - data.norge.no - and work is now underway on the project details and tender. In the meantime, a blog is set up at data.norge.no where government data questions are discussed.

Two reports on public data reuse have been published already this year, helping to raise interest in the topic. The first was written by a group at the University of Bergen (led by me). The second was produced by an expert group put together by the Norwegian Board of Technology, a consultative office to Parliament.

Norway has long had the potential to become a very interesting place for government data reuse. The public sector is, relatively speaking, considered well-run and efficient, and lots of quality data are collected. There is a deep-rooted tradition of transparency in government, supported by a Freedom of Information Act (FOIA). And the population is more tech-savvy than most, always quick to adopt new communication technology and services.

But despite these favourable conditions, Norway has not been in the data reuse vanguard so far. The accelerating activity and increased attention given to government data issues during the last months might change that. In this article I will consider three factors which I believe will decide the outcome:

• Infrastructure: The coming datastore needs to be supplemented by an open license for data reuse and practical initiatives to encourage data publishing.
• Principles: The PSI directive has been implemented in the FOIA. However, the fundamental principle of division of labour between the public and private sector (and civil society) in data reuse is still not generally acknowledged.
• Reuse in practice: Some public sector agencies are already doing an excellent job making data available, while others haven’t even started. Convincing examples of good reuse are quickly needed for recent positive developments to take hold.

Building the infrastructure The “data.govs” in the US, UK and other datastores give much-needed ammunition to government data advocates everywhere. In a list of recommendations that concluded our report at the University of Bergen, the swift creation of a Norwegian data.gov was in first place, and many others have voiced the same demand. After first indicating that it was under consideration, the Minister of Government Administration, Rigmor Aasrud, announced the URL data.norge.no in early April.

Not content with waiting for a datastore, several private initiatives were taken during last autumn to start a crowdsourced collection of public sector data sources. This was an important part of our own Bergen project, where we started with an open Googledoc spreadsheet. Some 130 data sources were registered there in a couple of months. And one IT engineer started a wiki called datakilder.no, with essentially the same goal.

The experiences from these experiments are available to the Ministry as it now plans its official datastore, and I believe they could be valuable. Moreover, through a cooperation with the Open Knowledge Foundation, we recently exported the Googledoc data to a Norwegian language instance of the foundation’s Comprehensive Knowledge Archive Network registry (the same software that is used in data.gov.uk).

We are currently considering how to continue this work now that data.norge.no has been announced. Personally, I think an independent datastore run by the open data community has an important role to play in pressing government agencies to release data (here we can be inspired by our German and Canadian colleagues. We have no guarantee that the official datastore will be filled with interesting data instantly - probably it will not. Also, an independent datastore can be used for experiments - such as including international and private sector data sources, and promoting and testing the principles behind linked open data.

Besides the datastore question, the need to define standard licenses for data reuse has been a key request from the open data community. At a conference held by the Ministry in Oslo last December, Opera Software CTO Håkon Wium Lie said that coming up with licenses should be priority number one for the Ministry. Lie, who helped organize a citizen campaign for opening up government data back in 1994, said that in principle there were only two reasons for not publishing data: To protect personal data and national security.

Licenses are certainly sorely needed. Today, practice varies greatly between government agencies. Some attach explicit conditions to their data, others do not inform at all. Some only allow non-commercial reuse. Typically, these differing conditions are confusing to potential re-users. Last December, the Norwegian Mapping Authority released map “tiles” for non-commercial reuse (but not the underlying vector data). It soon emerged that the definition of non-commercial was flawed. For example, would embedding map tiles in a for-profit news website count as commercial use? The Mapping Authority had to edit its terms and conditions.

It remains to be seen how open, or how restrictive, the licenses will be. This is certainly one of the crucial questions now.

During interviews and conversations with civil servants for the University of Bergen report, we noticed great discrepancies in knowledge and interest. We found agencies that had been working professionally with their data sources for years, while others hadn’t given the issue much thought at all. The need is clearly there not only to define licenses, but also to produce a comprehensive and practical guide to opening up data for reuse. We have tried to contribute by translating Ton Zijlstra and James Burke’s government data flowchart into Norwegian.

Adding to the recent positive developments, the Norwegian Ministry has made it clear that it is working on such a guide. I believe the open data community outside government should be involved here as well. This could be done in a number of ways. One possibility is to have everyone comment on a draft of the guide, but the community should also contribute with its own initiatives. Setting up a wiki collecting examples of good practices is one option.

Implementing the principles Government agencies should make data available for reuse in relevant formats, then leave it to the private sector and civil society groups to present the data in creative and interesting ways (for-profit or not). This assumption of a division of labour is central to the whole idea of government data reuse.

In practice, government agencies will present data as well, and it is hard to see that it would make sense to stop them doing that - as long as the data are released for reuse at the same time. However, the government data debates in Norway show that the fundamental principle is not accepted by everyone. Now and again you hear the argument that it is “unfair” that private companies should earn money on reusing the data collected by a government agency.

Another typical argument against publishing data is that they can be misinterpreted. In a survey of state agencies for the University of Bergen report, 43 percent of respondents agreed that “private businesses and individuals can misunderstand data and disseminate misleading information”. They are probably right - it is impossible to guarantee against misinterprations. However, today it is a very difficult argument to make that information should be kept from the public sphere because of such concerns (how much information would we not have to withdraw if that were a guiding principle?).

An easily discernible tendency that is noted with frustration by those seeking access to government data, is the public sector agency that spends resources on producing its own presentation of data, without releasing the “raw” data. In the worst case, this is the spin doctor’s data presentation.

The good news is that the open data community - from entrepreneurs to journalists to activists - have been handed a new and potentially powerful tool to persuade agencies to make data available.

The EU’s PSI directive was implemented in Norwegian law through the changes in the Freedom of Information Act which came into force January 1, 2009. This is the new section 9:

Right to request access to a collation of information from databases: Any person may request access to a collation of information that is electronically stored in the databases of an administrative agency insofar as the collation can be done using simple procedures.

In the regulations, the Norwegian Mapping Authority has been permitted to continue its policy of charging for access to map data. Given the importance of map data for so many types of applications, the Mapping Authority’s pricing regime has been heavily criticized for years, dominating the limited discourse on government data.

That debate will predictably continue. Less well known, but now beginning to seep through to the open data community, are the possibilities created by the preparatory documents to the revised FOIA. In the Ministry of Justice’s guide to the Act, the right to copies of digitally stored data is extended to copies in “all existing formats”. It remains to be seen how this will be interpreted in concrete cases. But it is already clear that the unpopular practice in many government agencies of publishing data sets in the pdf format will have to change. At the least, agencies will have to make spreadsheet versions of those data available.

Having data reuse principles “merged” with the FOIA will probably have positive consequences for those of us who work for better access to government data. The FOIA carries a lot of weight in Norway. It gives the open data community a much stronger foundation for seeking access to data than arguments about innovation and economic growth can.

Adding to this is the well-established routines among journalists of using the FOIA as a tool in their daily work. For over a decade, media organizations have had online access to the correspondence register of ministries, and agencies on other government levels have implemented their own public registers. On basis of the information in the register (sender, receiver, letter headline etc), journalists claim to see copies of the correspondence. The press associations follow up by lobbying the ministries and agencies, trying to influence them to adopt liberal practices in releasing documents.

In this environment, claiming access to data just becomes an extension of an existing practice and tradition, not something entirely new and unproven. Civil servants are used to handling FOIA requests. In theory at least, this should work in the open data community’s favour.

Gaining real access In our research for the Bergen project, we found that only one third of state agencies inform about data sources on their website homepage. Moreover, it is difficult to say if the information is complete (in most cases it probably isn’t). FOIA or not, with no information about data sources it is exceedingly difficult for potential reusers to locate the data. Hence, even with infrastructure and principles in place, there will still be ways to “hide” data.

This means that the open data community will have to continue lobbying and struggling for access to data. Hopefully the government will come up with incentives for agencies to make data available. But pressure from the community will be essential.

That pressure will have much greater chance for success if there are convincing examples of creative reuse to point to. Here contests such as Nettskap are important - they will speed up the production of good reuse cases. The community can do a lot in this area as well, by highlighting intelligent reuse of data at home and abroad.

In the best case scenario, user groups build “coalitions” with government agencies. There is a potential for this organized around location - for example, journalists, IT consultants, scientists and activists can team up with the city government with the common aim to improve a region’s competitiveness and transparency. Or such coalitions can be organized around interests - such as environment/climate data, where there is both commercial and civil society interest.

A government data champion? Norway and the other Scandinavian countries are potential open data champions. Progress has been made in several key areas during the last year, as I have tried to show in this article - more progress than could have been anticipated a year ago.

The Norwegian open data community that I have mentioned repeatedly, is in fact still emerging and very fragmented. One reason is that the interest for open data unites people with very different backgrounds - in the IT sector (business development and programming), in universities, in the media, in the public sector agencies.

Hence, an important challenge will be to bring people closer together and start building more efficient coalitions around the common interest of reusing data for different purposes. To accomplish this, the power of the convincing example cannot be overestimated.

Even if this succeeds, the community will still need help. Political leadership is essential, as only a clear sign that the government gives the question of data reuse priority will give the great majority of agencies the incentive to act. Unlike in the US and UK, we are still waiting for open data to be promoted from the highest political level.

Share This

Related posts:

1. Open government data in Norway: mounting interest but no breakthrough yet
2. Launch of Norwegian instance of CKAN!
3. What do you think about open government data in Australia?

As well as being used to power http://ckan.net and http://data.gov.uk CKAN is now helping run 7 data catalogues around the world including ones in Canada (http://datadotgc.ca / http://ca.ckan.net), Germany (http://de.ckan.net/) and Norway (http://no.ckan.net).

CKAN.net has also continued to grow steadily and now has over 940 registered packages:

Changelog

This is our largest release so far (56 tickets) with lots of new features and improvements. Main highlights (for a full listing of tickets please see the trac milestone):

• Package edit form: new pluggable architecture for custom forms (#281, #286)
• Package revisions: diffs now include tag, license and resource changes (#303)
• Web interface: visual overhaul (#182, #206, #214-#227, #260) including a tag cloud (#89)
• i18n: completion in Web UI - now covers package edit form (#248)
• API extended: revisions (#251, #265), feeds per package (#266)
• Developer documentation expanded (#289, #290)
• Performance improved and CKAN stress-tested (#201)
• Package relationships (Read-Write in API, Read-Only in Web UI) (#253-257)
• Statistics page (#184)
• Group edit: add multiple packages at once (#295)
• Package view: RDF and JSON formatted metadata linked to from package page (#247)

Bugfixes:

• Resources revision history (#292)
• Extra fields now work with spaces in the name (#278, #280) and international characters (#288)
• Updating resources in the REST API (#293)

Infrastructural:

• Licenses: now uses external License Service (’licenses’ Python module)
• Changesets introduced to support distributed revisioning of CKAN data - see doc/distributed.rst for more information.

Thanks

Lastly a big thank-you to everyone who has contributed to this release and especially to the folks at data.gov.uk!

Share This

Related posts:

1. CKAN 0.11 Released
2. CKAN 0.6 Released
3. CKAN 0.9 Released

At the Open Knowledge Foundation we have been working hard to support the development of open data catalogues around the world. The main purpose of these is to make official datasets easy to find and to reuse — whether by researchers, journalists, or web developers building new applications or services for citizens.

Work is underway in Austria, Canada, Finland, France, Germany, Hungary, Italy, Norway, Spain, and, of course, in the UK with data.gov.uk. Most of these data catalogues will be built and maintained by the public in the first instance — thanks to the wiki-like nature of the software behind CKAN, our open source registry for open data.

Once a data catalogue is up and running, the next step is to populate it with information about government datasets. Recently we’ve been thinking of ways to encourage people to start adding this kind of information. While in the past we’ve had data discovery workshops and package parties, in recent weeks we’ve been talking with a number of open data advocates in different countries about putting on a series of ‘Data Hunt’ events. These would turn the process of finding and documenting interesting (and hopefully open!) government datasets into a game.

In particular we’d like to record details of the following for official documents and datasets:

• What is it? (description)
• Where can I find it? (URL/download URLs)
• Can I reuse it? (legal/licensing information)
• Has anyone else created a version of it that is machine-readable / in format X / etc?

We’re still in the early stages of planning for these events — but thought we’d put out a call for input at the outset. Below are some of the things that we’re currently thinking about:

• How can we structure the events to be as fun as possible? E.g. should we have teams? Points? Prizes? Different categories (the most datasets added, the most open datasets added, the most interesting datasets added, …)?
• Would it be useful to have a how-to guide for Data Hunt participants — e.g. with ideas on where to find datasets, what kind of information to look for, how to tell if a dataset is openly licensed or not, and so on?
• Should we try and coordinate events internationally — e.g. to have several around the world at the same time?

If you have any cunning thoughts or suggestions about any of these points - please drop us a comment on this post! also if you’d like to help out — e.g. by organising an event in your country, by designing a nice logo, etc — we’d love to hear from you!

Share This

Related posts:

1. Open Government Data Camp 2010, 18-19th November 2010
2. Launch of it.ckan.net for open data in Italy!
3. Open government data in the UK, US and further afield: new report

• http://datadotgc.ca

The site uses the CKAN software under the hood with a Drupal front end - and they’ve added some nice features like a custom input form (with Canadian government departments, license options and suchlike), and a ‘league table’ showing which departments share most datasets:

Colin Calnan, who helped work on the site, has just posted details of the technical work involved in getting an instance up and running, Drupal integration and so on. Following are a few excerpts from his post:

For the benefit of the programming and Drupal community, I’m going to run through, with the aid of code samples, the development of the Drupal module to communicate with the CKAN API (which is where the data is stored). I’ll also walk through Theming, integration with Google Charts, Tag Clouds and most importantly, caching.

What is CKAN?

CKAN is a registry or catalogue system for datasets or other “knowledge” resources. CKAN aims to make it easy to find, share and reuse open content and data, especially in ways that are machine automatable.

CKAN is a nice big database that is built to accept user input of the type of data we’re trying to collect for datadotgc.ca. It has a slick front and back end that allows administrative access to the collected data. You can find out more on their website.

CKAN’s API

In order to utilize the power of CKAN I needed to link it up to Drupal. CKAN has a powerful and flexible API that I used extensively in the module.

The Foundation

Early on in the project I got in touch with the wonderful team at CKAN and they then put me in touch with Sean Burlingford from the data.gov.uk development team. They had also built their site in Drupal and Sean had lots of information on how they tweaked their CKAN site to work with Drupal. He worked hard to open source some of the work that they had done, and released it just in time for us to get started. Sean’s module provided the basic API connectivity we needed to get started and was the foundation for our module.

The Build

How do you integrate Drupal with the CKAN API? Let’s start with the basics: [...]

There’s lots more of the technical nitty gritty (including code snippets) on Colin’s post (and more to come soon, he assures us!).

If you are interested in starting an instance of CKAN for open (government) data in your country, drop us a line on our ckan-discuss list where there is a growing group of developers and translators who can help you to get something up and running!

Share This

Related posts:

1. Data.gov.uk releases CKAN Drupal Module
2. Interested in making an open data catalogue? Virtual meeting on 11th February 2010
3. What features should be included in a catalogue of open government data?

London’s authorities have opened up lots of their data this year, kicking off in January with the launch of the London Data Store (which we blogged about here). You can download, mash-up and analyse stats about crime, housing, education, planning and so on, but the biggie that many developers are desperate to get their hands is transport information. The GLA and 4ip held a half-day conference on Monday at City Hall on this public data, and transport data got top billing.

Dr David Mountain at Placr described how he’s been screen scraping the London Underground departure boards. Although his results were somewhat predictable - that waiting times increased during peak hours, or when the line had a problem - there was obvious interest from the audience in knowing what to expect on unfamiliar lines, and comparison between particular stations.

It is incredible to think that bus and train companies don’t make their timetables open. Chris Osbourne from ITO World has an interesting way to automatically produce extremely helpful bus maps, yet only two small British areas allow it. He made the point that councils subsidise a massive number of bus services, yet the ‘bus barons’ hug their data and stifle the innovation that might actually increase their custom whilst improving their usefulness to society.

Titled “Possibilities of Real Time Data”, the conference was positioned to highlight the value of open data, but I was surprised that there was not a discussion to work through the problems faced. The room seemed to be a healthy mix of technologists, representatives from London boroughs and other public data holders including Transport For London. Only briefly did we glean an idea of the road blocks TFL and other providers see preventing them releasing bus, tube and rail information. Data Protection was mentioned and clearly there are many more areas to be discussed, but surely this needs to be discussed in public to move it forward?

The success of using more general data to closely monitor and improve performance was the theme of a CTO of a council in Washington DC. Bryan Sivak described how they have been publishing lots of public data for about 5 years now on their Data Catalog.

He had stories about improved snow clearing with residents reacting to the map of snow-ploughed areas. They’ve had a couple of years’ worth of application competitions, the widely copied ‘Apps For Democracy’, but they also felt a clear need to develop some themselves. He mentioned a couple of measurable successes of using data: increased rehousing of homeless and spotting trends in overtime use.

What was most interesting regards my involvement as a CKAN developer were Sivak’s future plans. Firstly he mooted “Data Catalog in a Box”, where they open-source all their code so other cities/countries can reap similar benefits. It is great to see another example of publicly funded software being given back to the public, rather than locked away for private use. And although there is sure to be plenty of overlap with CKAN, and competition is healthy, no doubt we can both learn from each other and it creates more options for organisations wanting to catalog their open data.

The other future plan for Washington DC that piqued my interest was their “OpenCity API”. They have linked up with 7 of the top cities in the US to agree on the same web API for ‘government service requests’. i.e. for filling pot holes, requesting snow clearing etc. All this openness of data, source and sharing of APIs is very promising - other governments take note!

Share This

Related posts:

1. Launch of it.ckan.net for open data in Italy!
2. Consuming the Transport for London Data
3. Canadian citizen-driven data catalogue datadotgc.ca is powered by CKAN