Support Us

Introducing Datapkg: A Tool for Distributing, Discovering and Installing Data “Packages”

February 23, 2010 in CKAN, datapkg, News

Datapkg 0.5 has been released! This is the first release deemed suitable for public consumption (though we are still in alpha)! This announce therefore serves as both introduction and release announcement.

Introduction

From the docs:

datapkg is an user tool for distributing, discovering and installing data (and content) ‘packages’.

datapkg is a simple way to ‘package’ data building on existing packaging tools developed for code (e.g. Debian apt, PyPI, CRAN, Gems, CPAN). datapkg is designed to integrate closely with the CKAN (Comprehensive Knowledge Archive Network).

In terms of the big picture, datapkg is the “apt-get/aptitude/dpkg” part of the vision for a ‘Debian of Data’ (i.e. scalable, distributed, open data infrastructures! — for more see this post or these recent slides):

debian of data

Datapkg is a key part of making data sharing automatable. As an end-user tool it allows automated (command-line or scripted) discovery, installation and sharing of data “packages” either standalone or via interaction with a registry like CKAN.

Trying It Out

If you’re interested in giving it a spin here are installation instructions. Once you’ve got it running you can then do things like (see the manual for more):

Search for a package in an Index e.g. on CKAN.net::

# let's search for iso country/language codes data (iso 3166 ...)
$ datapkg search ckan:// iso
...
iso-3166-2-data -- Linked ISO 3166-2 Data
...

Get some information about one of them (in this case 2-digit ISO country codes in RDF)::

$ datapkg info ckan://iso-3166-2-data
....
....

Let’s install it (to the current directory)::

$ datapkg install ckan://iso-3166-2-data .

This will download the Package ‘iso-3166-2-data’ together with its “Resources” and unpack it into a directory named ‘iso-3166-2-data’.

Extending

datapkg is intended to be a generic tool for data packaging. As such, we want it to deal with as many “distribution” formats and as many different registries as possible. We’ve therefore designed datapkg to be extensible so that it can easily be adapted to talk with other systems. What kinds of plugins might one write?

  • A plugin to discover data “packages” from RDFa information in web-pages, especially those in Government data catalogues (suggested by Ed Summers
  • A plugin to Ensembl http://www.ensembl.org/
  • A plugin to extract download urls or SPARQL endpoints from VoID descriptions (suggested by Richard Cynganiak)

We’re looking for more such suggestions as well as for people who’d like to implement plugins. If you’re interested please get in touch: http://www.okfn.org/contact/

Get Updates