Changes include:

• feature to list and search tags
• feature to make data available in machine-usable form via sql dump
• feature to purge a revision and associated changes
• support for reserved html characters in urls
• upgrade to Pylons 0.9.6
• new spam management utilities including (partial) blacklist support

The CKAN code is available from:

• http://pypi.python.org/pypi/ckan/0.5

The data is available from:

• http://ckan.net/license/

We’ve currently got 135 packages. If you come across a large dataset or substantial collection, please consider registering it on CKAN!

Share This

We already have some fairly good working processes for collaborative development of unstructured text: the two most prominent examples being source code of computer programs and wikis for general purpose content (encyclopedias etc). However these tools perform poorly (or not at all) when we come to structured data.

The purpose of this short essay is to pose the question: how do we collaboratively develop knowledge when it is in the form of structured data (as opposed to unstructured text)?

There are two aspects of structured data that distinguish it from plain text:

1. Referential integrity (objects point to other objects)
2. Labelling to enable machine processing (and the addition of ’semantics’)

To illustrate what I mean consider the following use case which comes from our own public domain works project. Here we are storing data about cultural works. In the simplest possible setup we have two types of object: a work and a creator. A given work may have many creators (authors) and a given creator may have created many works. Furthermore each work and each creator have various attributes. For the purposes of this discussion let us focus on only two:

1. name (creator) and title (work)
2. date of death (creator) and date of creation (work)

If we were to adopt a wiki setup (a la wikipedia) we would create a web page for each creator and each work. There would be a url pointing to any associated objects with some kind of human-processable (but likely not machine-processable) indicator of the nature of the link. Attributes would also be included as plain text perhaps with some simple markup to indicate their nature perhaps not. The unique identifier for a given object would come in the form of a url.

This is a not unattractive approach as it is very easy to implement — at least initially — because wikis for plain text are so well developed (and in fact it is the approach taken by the current v0.1 of public domain works). The problem arise when once one goes beyond simple data entry. For example

1. Searching, particularly structured searching (e.g. find more all creators who died more than seventy years ago and whose works are more than 100 years old), is slow and cumbersome compared to working with a database. Referential integrity isn’t enforced and the unique identifiers (url names) aren’t

2. Programmatic insertion and querying of the data is very limited. For example suppose we obtain a library catalogue and wish to merge it into the existing data. To do this we need to query the existing db repeatedly to try and identify matches between existing objects and objects in the catalogue.

3. No support for ACID, in particular no way:

   1. To have (and enforce) referential integrity in your data structures [^wiktionaryz]
   2. To do atomic commits which preserve referential integrity (even in a simple wiki this is a problem in that renaming a page and changing references to it have to be separate operations rather than one atomic commit)

4. ‘Data loss’/No data structure: when data structure isn’t “enforced” it may be extremely (or impossible) to extra relevant information (e.g. date of death in above example). In such circumstances, at least from a programmer’s point of view, the data is now ‘lost’. It also makes it much harder to enforce data constraints when data is entered or to check data validity once entered.

Thus we really want an approach that supports:

1. Versioning at the model level (i.e. not just of individual attributes)
2. Other data types than plain text
3. Associated tools:
   • No off-the-shelf tools that will version
   • No off-the-shelf tools to do visualization (e.g. showing diffs)[^1]
   • Web interface to provide for direct editing (and integration of associated tools such as diffs, changelogs etc)
   • Programmatic API to access data

The obvious way to proceed with this is to develop ‘versioned domain models’. That is to develop traditional software-based or database-backed ‘domain model’ which can then be versioned. This would be very similar to the way that subversion first models a filesystem and then adds versioning of that filesystem[^3][^4].

Footnotes

[^wiktionaryz]: The wikitionaryz project (now renamed OmegaWiki) have been working on integrating referential intergrity into a wiki-like interface.

[^1]: there are a bunch of (pre 1.0 AFAICT) tools for doing diffs on xml data. See e.g. http://www.logilab.org/projects/xmldiff

[^2]: see http://www.martinfowler.com/ap2/timeNarrative.html for software patterns for objects that change with time. There is also a very extensive book on the topic on time-oriented db applications in sql by the father of the temporal parts of sql3: http://www.cs.arizona.edu/people/rts/tdbbook.pdf

[^3]: the subversion model can best be gleaned from its API. A pythonic version of that API can be seen in: http://www.rufuspollock.org/code/svnrepo/svnrepo.py

[^4]: http://www.musicbrainz.org/ already go some way towards having a versioned domain model in relation to music and its creators.

Share This

1. Addressing and atomisation: Are annotations specific to particular parts of the resource. If so how do we store this address (relatedly: how is the resource ‘atomised’ and how to we address these atoms, or range of atoms). For example, do we address by word, by character, by paragraph or by section? Do we wish to store ranges rather than a single address? Do we wish to allow a given annotation to be associated with multiple ranges/atoms?

2. Permissions: Are there restrictions on the creation (deletion/updating etc) of annotations.

3. Will the underlying resource change and if so are annotations intended to be robust to those changes.

Let’s concentrate on the first issue for the time being as it is the most immediately important. Furthermore, defining the ‘atoms’ of the resource sharply narrows the implementation options.

The Simple Case: Mod a Blog

If one is happy to have fairly large atoms (pages, or even sections of some piece of text) then implementing an annotation system can be reduced to grabbing your favourite CMS or blogging software and feeding the text in in appropriate chunks. This is often satisfactory and is a simple, low tech solution that will pretty much work out of the box. A classic example of this approach is http://www.pepysdiary.com/ which works so well because the subject matter (Samuel Pepy’s diary) has a very obvious atomisation (namely the daily diary entries) suited perfectly suited to blog software (in this case movable type).

You can even start doing a bit of modding, for example to present recent annotations (http://www.pepysdiary.com/recent/) or to present the text plus annotations all in one piece. (Given that commentonpower seems to fall neatly into this category with most commentable atoms of the right size for ‘blog’ entries I wonder why they didn’t just implement it as a plugin for wordpress — perhaps it was such a simple app that it easier to ‘roll their own’. Update: since this was written we’ve had a chat with the developers and apparently commentonpower does use wordpress though it isn’t a plugin).

Getting More Atomic

Once you want to have atoms below a size comfortable for individual html pages/blog entries, wish to allow people to comment on chunks too large for an individual page, or to comment on ranges one starts to have problems with this approach. The main challenge at this point is to find some way to extract the addressing information from the client doing the annotation. Confining ourselves to the web the challenge becomes way to structure the interface and the text so that one can determine range start and end points. This is a non-trivial matter. Possible options include:

• Javascript: in theory the selection/range objects should help us out here unfortunately cross-browser support is patch (firefox as usual is excellent and IE pretty bad). If one does not want to be as precise as to get ranges javascript could also be used to extract e.g. element ids.
• Copy and paste of the quote to annotate with some backend algorithm to determine the actual range. Nice and simple but not clear that one can ‘invert’ (i.e. find a unique range from a given selection) unless the selection is large.
• If addressing fairly large atoms (e.g. a paragraph or large) one could just insert a unique piece of user interface equipment (e.g. a button or link) with each atom. Note however that this prevents support for ranges.

Separating Data and Presentation

Whatever one chooses to do it does seem sensible to clearly separate data and presentation. This is particularly important when there is so much uncertainty over the user interface. In particular, it would be good to clearly specify the annotation format and implement a programmatic interface to it independent of the standard (human) user interface. That way is easy to switch interfaces (or have multiple ones). Given that annotations are essentially just a comment it would seem sensible to try and reuse an existing format such as Atom (or RSS) for the machine interface to the comment store. [marginalia] already had such a format based on atom. I’ve recently reimplemented a stripped down version of this format for the annotation store backend in python in preparation for adding annotation support to openshakespeare web interface, see:

http://project.knowledgeforge.net/shakespeare/svn/annotater/trunk/

Of course as discussed above this isn’t quite as simple as it looks as your user interface can constrain what you can and can’t store (using a blog approach you can’t store ranges and from what I have read getting reliable character offsets is problematic). Nevertheless it seems the best place to start.

Share This

• Get data out there — that’s why we’re developing KnowledgeForge
• Make sure that data is open data (i.e. is properly licensed) — that’s why we’re developing open knowledge definition
• Make sure that data can be found — that’s why we’re developing CKAN

But as Francis Irving recently pointed out to me you need to give people a reason to put data out there in a form that is discoverable. In particular, he suggested that giving people tools to do something with their data, such as visualize it, would be one of the best ways to encourage data sharing. To this end we’ve been putting together a very simple demonstration of a data store with some visualization capabilities at:

http://econ.dev.okfn.org/store/ http://www.openeconomics.net/store/

The data itself (along with any associated with metadata) is stored in a subversion repository here:

http://project.knowledgeforge.net/econ/svn/trunk/data/

At present to add data to the store you just need to upload it to subversion but in future we hope to implement a web interface to do this.

Share This