Open Knowledge Foundation Blog » Uncategorized

Beginnings of an Object Description Mapper

Rufus Pollock — Sat, 21 Aug 2010 14:42:07 +0000

The analogue to an Object-Relational Mapper for RDF. Helping to make OWL Description Logic accessible from Python in a way that will seem familiar to people who are accustomed to things like SQLAlchemy and Django.

http://packages.python.org/ordf/odm.html

Share This

Cataloguing Bibliographic Data with Natural Language and RDF

William Waites — Mon, 09 Aug 2010 17:19:01 +0000

In the grand tradition of W3C IRC bots, I’ve started some speculative work on a robot that tries to understand natural language descriptions of works and their authors and generates RDF. It is written in Python and uses ORDF, the NLTK and FuXi.

Before going into implementation details, here’s an example of a session:

12:41 < ww> biblio forget
12:41 < biblio> ww: ok
12:41 < ww> Solzhenitsyn's name is "Aleksander Isayevitch Solzhenitsyn"
12:42 < ww> He was born on December 11th 1918
12:42 < ww> He died on August 3rd 2008
12:42 < ww> He wrote TFC in 1968
12:42 < ww> TFC's title is "The First Circle"
12:42 < ww> "YMCA"'s name is "YMCA Press"
12:42 < ww> They published TFC in 1978
12:42 < ww> biblio think
12:42 < biblio> ww: I learned 25 things in 0:00:00.218296
12:42 < ww> biblio paste
12:42 < biblio> ww: http://pastebin.ca/1913826

The natural language parsing is somewhat simplistic, the kinds of grammatical constructions it can understand are limited (but growing), the resolution of pronouns (e.g. he, they) only looks at the previous named subject and it will get confused if there is more than one pronoun referring to a different thing in the same sentence but all of these things can be improved.

Broadly, the process follows the following steps:

(NLTK) Tokenise the sentence and classify for parts of speech
Create references for named entities (capitalised words, URIs and phrases enclosed in double quotes)
(NLTK) Create a lexicon, the part of a grammar that grounds it to individual words and append it to the canned grammar that describes the structure of sentences. This is a feature grammar not a context-free grammar
(NLTK) Parse the input sentences creating a syntax tree with the root at the main verb in the sentence
The syntax tree is annotated with the logical structure of the sentence (see Analysing the meaning of sentences). This logical representation is cunningly constructed so as to also be runnable Python code (with eval). Running it transforms the syntax tree into an RDF representation.
(FuXi) the “biblio think” command causes the RDF of the current session to be run through a number of inference rules that encode higher level meaning. That if “X wrote Y” then X must be a person, Y must be a work and X must have contributed to Y.

The neat bit is really the way it generates RDF, translating a logical structure that looks like,

statement(
  predicate(
    bnode(
      rdf_type(umbel("Verb")),
      label("is"),
      racine("be"),
      tense(nlp("Present")) 
    ), 
    named("aHLIkuXm14335") # "The First Circle" 
  ), 
  posessive(
    bnode(
      rdf_type(umbel("Noun")), label("title"), racine("title")
    ), 
   named("aHLIkuXm14333") # "TFC"
  ) 
)

and the constituent parts bubble up and return an RDF Graph that looks like this:

 entity:aHLIkuXm14333 a nlp:NamedEntity;
      rdfs:label "TFC".

 entity:aHLIkuXm14335 a nlp:NamedEntity;
      rdfs:label "The First Circle".

 [ a umbel:Verb;
      rdfs:label "is";
      lvo:nearlySameAs lve:be;
      nlp:directObject entity:aHLIkuXm14335;
      nlp:subject [ a umbel:Noun;
                    rdfs:label "title";
                    lvo:nearlySameAs lve:title;
                    nlp:owner entity:aHLIkuXm14333];
      nlp:tense nlp:Present].

And this sort of structure is the basis for the reasoning step. Provenance information, using OPMV is also kept, pointing back to the original IRC message that was parsed so the entire process should be repeatable.

I suppose since IRC is not necessarily the most accessible of media — though I can’t really see why — the same engine could be easily glued to a web server with a simple chat-like interface. Perhaps this is easier or more natural than web forms. Perhaps not. More research is needed.

In any case, I’m working on improving the natural language parsing and the inference rules as time permits so hopefully the robot will become more and more clever.

Source code for the IRC bot is available at: http://bitbucket.org/ww/sembot

You can play with a live version of the bot by joining irc://irc.oftc.net/ and joining #okfn or engaging in a private chat with biblio. It understands the command “sembot help” and I’ll try not to break it too badly while anyone’s playing with it.

Share This

About Inferencing

Rufus Pollock — Mon, 02 Aug 2010 13:17:31 +0000

Inferencing, or machine reasoning has a slightly unsavoury reputation perhaps stemming from the failure of Strong AI and its association with science fiction. This is a bit unfortunate and it could be argued that it has led Semantic Web technologies to be underdeveloped.

With the Semantic Web and RDF we are concerned with simple statements, or assertions. When humans make statements they generally rely on a large amount of background knowledge and contextual information to make their meaning clear without it being explicit. For example, if I say, “Mary had a little lamb,” it is unnecessary to explain that Mary is a person, Mary is female, a lamb is a young sheep, a sheep is a kind of quadrupedal animal or to digress in a discussion of what it means to “have” something, to what extent notions of ownership can extend to animals, or even the idea of time, past, present and future.

If we transcribe this first line of the nursery rhyme into the Notation 3 language so that it can be ingested by a computer, we might get,

Mary had [ a Lamb; size little ].

And because this transcription has to be done manually by a human unless we can invent some very good Natural Language Processing software, this is probably the most we can expect someone to do before it begins to get very tedious. We certainly don’t want to have to teach the computer about background facts more than once, but we can imagine that we have some set of background information at our disposal, indeed OpenCyc and SUMO can help here. Indeed OpenCyc can teach us that a lamb is a young sheep, a domesticated animal, a ruminant, a quadruped, a terrestrial organism, etc..

It is natural to want to be able to ask simple questions such as, “what type of animal did Mary have” which are actually quite easy to express in a query language like SPARQL,

SELECT ?animal ?type         WHERE {                 Mary had ?animal .                 ?animal a Animal .                 ?animal a ?type         }

however if this query were to be evaluated only against the facts transcribed from the nursery rhyme it would return no results. Mary had a lamb not an animal. To get the right answer we need to introduce rules in addition to our background facts. In this case the rule we need is very simple,

{ ?x a ?class . ?class subClassOf ?superclass } => { ?x a ?superclass }

This simply says that, for all things (x), if they have a class, and that class is a subclass of some superclass, then the thing is also whatever the superclass is. So if x is a lamb and lamb is a subclass of sheep then x is a sheep. Likewise, since sheep is a subclass of animal then x is an animal.

It is precisely this kind of situation where machine reasoning is helpful, to evaluate this type of simple rule. It is nothing spectacular, just following chains of statements made by humans to answer questions that would be obvious to a two-year old. That said, this is just a toy example, the same principle can be used with facts and questions that are not quite so obvious. However, if the rules get much more complicated or numerous, it becomes quite a lot more computationally expensive to evaluate them.

We have a very good reasoning engine, called FuXi, that is supported in ORDF for implementing these sorts of rules. Behind the scenes it is used in searching for specific types of things in the Bibliographica, one can search for publications, or articles, or books or, in some cases, chapters, but a search at a higher level of granularity will return all types of results and a search at a lower level will return only the types sought.

Share This

Cataloguing Bibliographic Data with Natural Language and RDF

Bad Science on Open Data

David Jones — Fri, 02 Jul 2010 11:35:57 +0000

The following article is from Guardian columnist Dr Ben Goldacre and was originally published on his blog as “Nullius in verba. In verba? Nullius!”. He kindly allowed us to reprint it here. It discusses the pros and cons of publishing data in the context of investigative medical journalism.

Ben Goldacre, Not In The Guardian, Saturday 26 June 2010

Here is some pedantry: I worry about data being published in newspapers rather than academic journals, even when I agree with its conclusions. Much like Bruce Forsyth, the Royal Society has a catchphrase: nullius in verba, or “on the word of nobody”. Science isn’t about assertions on what is right, handed down from authority figures. It’s about clear descriptions of studies, and the results that came from them, followed by an explanation of why they support or refute a given idea.

Last week the Guardian ran a major series of articles on the mortality rates after planned abdominal aortic aneurysm repair in different hospitals. Like many previously published academic studies on the same question, they again discovered that hospitals which perform the operation less frequently have poorer outcomes. I think this is a valid finding.

The Guardian pieces aimed to provide new information, in that they did not use the Hospital Episodes Statistics, which have been used for much previous work on the topic (and on the NHS Choices website to rate hospitals for the public). Instead they approached each hospital with a Freedom of Information Act request, asking the surgeons themselves for the figures of how many operations they did, and how many people died.

Many straightforward academic papers are built out of this kind of investigative journalism work, from early epidemiology research into occupational hazards, through to the famous recent study hunting down all the missing trials of SSRI antidepressants that companies had hidden away. It’s not clear whether this FOI data will be more reliable than the Hospital Episodes numbers – “discuss the strengths and weaknesses of the HES dataset” is a standard public health exam question – and reliability will probably vary from hospital to hospital. One unit, for example, reported a single death after 95 emergency AAA operations on FOI request, when on average about one in 3 people in the UK die during this procedure, and that suggests to me that there may be problems in the data. But there’s no doubt this was a useful thing to do, and there’s no doubt that hospitals should be helpful and share this information.

So what’s the problem? It’s not the trivial errors in the piece, although they were there. The article says there are ten hospitals with over 10% mortality, but in the data there are only 7. It says 23 hospitals do over 50 operations a year, but looking at the data there are only 21.

But here’s what I think is interesting. This analysis was published in the Guardian, not an academic journal. Alongside the articles, the Guardian published their data, and as a longstanding campaigner for open access to data, I think this is exemplary. I downloaded it, as the Guardian webpage invited, did a quick scatter plot, and a few other things: I couldn’t see the pattern for greater mortality in hospitals that did the procedure infrequently. It wasn’t barn door. Others had the same problem. I received a trickle of emails from readers who also couldn’t find the claimed patterns (including a professor of stats, if that matters to you). Jon Appleby, chief economist on health policy at the King’s Fund, posted on Guardian CommentIsFree explaining that he couldn’t find the pattern either.

The journalists were also unable to tell me how to find the pattern. They referred me instead to Peter Holt, an academic surgeon who’d analysed the data for them. Eventually I was able to piece together a rough picture of what was done, and after a few days, more details were posted online. It was a pretty complicated analysis, with safety plots and forest plots. I think I buy it as fair.

So why does it matter, if the conclusion is probably valid? Because science is not a black box. There is a reason why people generally publish results in academic journals instead of newspapers, and it’s got little to do with “peer review” and a lot to do with detail about methods, which tell us how you know if something is true. It’s worrying if a new data analysis is published only in a newspaper, because the details of how the conclusions were reached are inaccessible. This is especially true if the analysis is so complicated that the journalists themselves did not know about it, and could not explain it, and this transparency is especially important if you’re seeking to influence policy. The information needs to be somewhere. Open data – people posting their data freely for all to re-analyse – is the big hip new zeitgeist, and a vitally important new idea. But I was surprised to find that the thing I’ve advocated for wasn’t enough: open data is sometimes no use unless we also have open methods.

Share This

Understanding COINS

Lisa Evans — Thu, 17 Jun 2010 08:24:25 +0000

Something amazing has happened since the government spending recorded in the COINS database was made openly available to everyone. I’m talking about the impressive range of free, and in many cases open source, products to display the COINS data.

So far there are COINS search engines from The Guardian and The Open Knowledge Foundation, graphs from Rapid Gate Way and Alpine Interactive and bloggers like Martin Budden have been powering away on their own projects to describe the COINS data. What a triumph for publishing government data. It beats the alternative of using public funds to pay for these tools when the skills and enthusiasm are clearly out there in the community.

That’s not to say that the products to display the data are complete right now, or that we have understood the COINS data completely. We had a few clues about the structure of the data from previous research, but there is no substitute for having the data itself, and we are still building up our knowledge. But given it’s been just over a week since we first laid eyes on the data, I think it’s fair to say that we are making good progress by most IT project standards.

In this post I want to address two questions that drive our thinking at the Open Knowledge Foundation, since the COINS publication. They are: ‘what’s important in COINS?’ and ‘how do we get meaningful results out of it?’

It has taken some discussion with the exceptionally helpful staff at HM Treasury and reading the COINS Guidance(PDF) and other related materials that make more sense now we can see the data — but finally I feel we have more accurate answers to both of these questions.

What’s important in COINS?

The COINS Guidance lists every field in the version of COINS that was released. One of the big challenges with a big complicated data set, like COINS, is knowing which of these fields are important.

To determine this I’ve spoken with the Treasury team about the fields they consider most useful, and the combination of fields they use most frequently.

The answers I got focused mainly on the central government spending and income data.

The spending and income is described for each central government department which you can see in the ‘Department description‘ field. Each department has a number of programmes that will either require or generate money. The department’s programmes are in the ‘programmes object group description‘ part of COINS, and more detail still is in the ‘programme objects description‘, and yet more detail still is in the ‘account codes‘ which are all listed in Annex B.

The ‘Value‘ field tells the actual spending or income in thousands of pounds. If the number is positive it refers to the departments spending, if negative it refers to the department’s income. It should also be able to check if the amount is spending or income from the ‘account code’.

In addition to the spending programme and ‘account code’ information, there are two further categories in COINS that describe the data very usefully, those are:

‘budget boundary‘. There are three choices for ‘budget boundary’: 1) DEL which stands for Departmental Expenditure Limits. These are items that have been budgeted for 3 years, it is estimated that DEL makes up about 80% of the items in COINS. 2) AME which stands for Annually Managed Expenditure. These are the budget items that are difficult to predict accurately and the risk for these is taken by the Exchequer as a whole. We are ignoring everything in AME where the ‘Programme /admin’ is not set to ‘Other’. 3) ‘not DEL/AME’ is budgeting for arm.s length bodies — we are not too concerned about these budget items.
the ‘resource capital‘. There are two options that are both useful for .resource capital. which are 1) ‘capital’ which is investment and capital assets. 2) ‘resource’ which includes all wages, salaries and operating costs.

There are some parts of COINS that we are less concerned with at the moment.

Other than the expenditure and income data, there are plans and estimates in COINS. You can see plans and estimates that should roughly correspond to the supplementary budget information and the supply estimates, respectively. We have been less concerned with plans and estimates as, by their nature, they will be less detailed than the outturn.

There is a CPID code in COINS which is there for a special project within the Treasury called the Whole of Government Accounts (WGA). This project will ensure that there is no double counting of the money when a transaction occurs between government departments. As I understand it, if body A gives money to body B then WGA would be responsible for subtracting the amount body B received from body A’s total. There are scripts in COINS to ‘best guess’ these subtractions using the CPID code, along with the WGA staff performing lots of checks too, but once this matching has been successful the CPID code is largely redundant.

The Whole of Government Accounts also collects information about spending by local authorities and records this spending in COINS, but this is not in a publishable state. However it is possible to view central government grants for local authorities with the field called ‘Local Government Use only‘.

How do I get meaningful results out of COINS?

On the advice of the Treasury guidance we are focusing on the Fact Table more than the Adjustment Table in COINS. In the fact table the field that defines actual spending and income is the ‘Data_type‘ being set to ‘Outturn’ and ‘Data_subtype‘ being set to ‘approved’ or = submitted_outturn (both of these conditions required).

In addition we can set Budget_Boundary to either DEL or if we require the shorter term budget spending then we set AME and then set programme/admin to ‘Other’.

For the 2009-2010 COINS data we can also set the Resource_capital2: set to Resource (on 2010-11 budgeting basis).

With the COINS data defined this way it is then possible look at the spending programmes and associated account codes certain that the results are actual spending and actual income for the time frame, rather than estimated or planned spending or income.

It is wonderful that the publication of COINS has brought so much innovation in the open software community. It will be even more wonderful if we can continue to develop to make public spending data easier to understand, particularly when so many important decisions are being made that will affect our lives.

Share This

Dig the new breed, Part III - wrapping it all up

jwalsh — Fri, 11 Jun 2010 21:51:19 +0000

This is the third in the amazing series of guest blogs from Ant Beck on the impact of linked open data for archaeology.

Part 1: New approaches to archaeological data analysis, as seen in the DART and STAR projects Part 2: Considering the ethics of sharing archaeological knowledge

OK, to recap we have:

A scientific movement that advocates open approaches to data, theory and practice
Emerging foundational interoperability using semantic web technology
The potential to remove a barrier and facilitate the submission of primary data

These three powerful factors could prove to be highly disruptive. In combination they have the potential to turn archaeological data and data repositories from static siloed islands (containing data that is increasingly stale) into an interlinked network of data nodes that reflect changes dynamically.

The linch-pin is the use of triplestores (RDF databases) that provide persistent identifiers. Persistent identifiers allow us to refer to a digital object (a statement, a file or set of files) in perpetuity, even if the underlying storage location moves. This means links between objects are persistent: therefore, when an observation or interpretation changes its effects are propagated through to all the data/events that link to it. I see organisations such as the ADS, Talis (an innovating semantic web technology provider which provide the Talis Platform which includes a free RDF hosting service for open data) and national heritage bodies providing such services.

Some open science projects are likely to adopt RDF as their de-facto data sharing format. RDF triples (subject, predicate, object) provide a schema transparent mechanism for data storage. They are not ideal for all data types (raster data structures for example) but when used with Ontology and SKOS, as demonstrated by STAR, they are powerful analytical, search and inference tools.

So, what is the importance of storing heritage data in RDF? Well, it depends which point of view you take. From a data management perspective there is no longer any need to migrate data formats. However, to facilitate re-use, different “views” of the RDF model can be generated and incorporated into traditional analytical software, such as GIS. Importantly, analysis stops being a “knowledge backwater”: new knowledge can be appended back into the triplestore.

Linked Data concepts in archaeology

From a data curation, re-use and analysis perspective the quality of the data has the potential to be dramatically improved. Deposition is no longer the final act of the excavation process: rather it is where the dataset can be integrated with other digital resources and analysed as part of the complex tapestry of heritage data. The data does not have to go stale: as the source data is re-interpreted and interpretation frameworks change these are dynamically linked through to the archives, hence, the data sets retain their integrity in light of changes in the surrounding and supporting knowledge system.

An example is probably useful at this juncture: In addition to many other things pottery provides essential dating evidence for archaeological contexts. However, pottery sequences are developed on a local basis by individuals with imperfect knowledge of the global situation. This means there is overlap, duplication and conflict between different pottery sequences which are periodically reconciled (your Type IIb sherd is the same as my Type IVd sherd and we can refine the dating range…… Hurrah… now let’s have another beer). This is the perennial process of lumping and splitting inherent in any classification system. Updated classifications and probable dates allow us to re-examine our existing classifications. One can reason over the data to find out which contexts, relationships and groups are impacted by a change in the dating sequences either by proxy or by logical inference (a change in the date of a context produces a logical inconsistency with a stratigraphically related group) While we’re on the topic of stratigraphy, an area of notorious tedium and poor quality data (often with conflicting relationships), RDF allows rapid logical consistency checking as stratigraphic relationships are basically a graph and RDF triples are a graph database. Publically deposited RDF data should be linked data: this means that all the primary data archives are linked to their supporting knowledge frameworks (such as a pottery sequence). When a knowledge framework changes the implications are propagated through to the related data dynamically. This means that policy, development control and research decisions are based upon data that reflects the most-up-to date information and knowledge….. cool huh.

Incorporating excavation data into RDF means that ontology and SKOS can be used to dynamically repurpose the data for policy formulation, planning impact, regional heritage control and mitigation purposes in conjunction with the data in the Sites and Monuments Record (SMR). Raw data can be integrated from multiple different sources with different degrees of spatial and attribute granularity and, where appropriate, generalised so that the data is fit for the end users’ purpose. From a policy perspective curatorial officers no longer have to battle to stop datasets becoming stale and add new datasets to the local SMR. The SMR will remain an essential dataset: even though it is a generalised resource it is the only location of a digital record for resources that are unlikely to be digitised in the future (unless there is a very unlikely reverse in funding patterns). Thus the curatorial officer can develop more effective regional research agendas based upon up-to-date and accurate data.

This has the potential to change the way Historic Environment Information Resources (HEIRs) are managed by curatorial officers and transform how developers (property and software), policy makers and the general public engage with and consume any data. They will be able to support innovative access to primary linked data resources by researchers, planners and most importantly the public. This is a significant and important change in role. In addition the heritage data can be mashed up with other data resources to produce tailor made resources for different end-user communities – following the model successfully employed by data.gov.uk.

Data re-use and mashups are also important for those undertaking research and analysis. The big difference will be for those who undertake research or collect data that transcends different traditional analytical scales. For example, the National Mapping Programme which aims to “enhance the understanding of past human settlement, by providing primary information and synthesis for all archaeological sites and landscapes visible on aerial photographs or other airborne remote sensed data” will provider deeper insights when it is integrated with other data. However, this integration can occur in real time and add tangible interpretative depth. If an interpreter is digitising data from an aerial photograph and they see two ditches cutting one another they are unlikely to be able to tell the relative stratigraphic sequence of the two features. Direct access to excavation or other data will allow the full relationships and their interpretative relevance to be deduced during data collection.

In the longer term consumers of archaeological data will be more used to dealing with primary data, will become more aware of its potential and demand more of the resource. This should produce a ground up re-appraisal of recording systems and a better understanding of archaeological hermeneutics. The interpretative interplay between theory, practice and data as part of a dynamic knowledge system is essential. Although this has been recognised, in reality theory, practice and data have never really been joined up. We don’t have to use a one size fits all approach to conducting excavations, but we can tailor bespoke systems that address local, regional and national research challenges. We can generate interesting and provocative data that can be used to test theory and inform practice and move away from recording systems mired in the theoretical and intellectual paradigms of the mid 70’s.

The virtuous circle is re-established; theory will influence practice, which will change the nature of the data, which will impact on interpretative frameworks, which will provide a body of knowledge against which theory can be tested.

Final comments

There is a new breed: there are people and organisations who don’t want to do what’s always been done. People who are empowered and don’t believe that established institutions and hierarchies are the gatekeepers of progress: organisations that can, and want to, change the way we ‘play the game’, people who want to collaborate. Organisations that want to share. Open approaches can help to make all this happen. This is all facilitated by disruptive technology which is increasingly mature, broadly available for free (or at a low cost) and with low barriers of use and re-use. In the nearly twenty years of studying and working in the heritage sector I’ve seen it change dramatically. I feel we are on the cusp of changing the way we engage with our data which could profoundly alter the way we understand the past, how we can communicate this in the present and how we can sustainably manage a complex resource for the future.

Share This

COINS: A Users Guide

Lisa Evans — Fri, 04 Jun 2010 09:02:51 +0000

At 0930 BST today the UK government released the COINS database, one of the biggest sources of information on UK public spending. Open Knowledge Foundation Director Rufus Pollock says:

The release of this data marks another milestone in the opening up of public data - in which the UK leads the way. While this is by no means the end of the line, this material is substantially more detailed than anything previously available and is a major advance for transparency of public finances.With our Where Does My Money Go? project we’ve already been working to make spending understandable to the general public and this new data is essential to realizing the project’s goal of showing exactly where each pound of your taxes goes.

Update: for latest info see http://www.wheredoesmymoneygo.org/data/coins/

Lets be honest — it’s basically mystery how our tax money is spent. Like all good mysteries it’s compelling to find the truth behind it.

The publication of the COINS database today will a big step forward in resolving this mystery. COINS, which stands for the Combined Online Information System, is the main database used by HM Treasury for budgeting — and reconciling what actually happened against those budget plans.

Public bodies have a requirement to report their spending to COINS. Each local government body, and this includes all councils (except parish), all local police, local fire, local transport and park authorities, report all items of spending over £1million once a year. The record of this spending is gathered by Communities and Local Government (CLG) and audited before it is entered into COINS as spending from CLG.

Similarly, each central government department has to report spending on all items over £1 million and agreements over £5 million and that they define this spending use their own spending codes for this. Some of these items are well defined in COINS — others less so.

Each of these bodies provide not only their spending once a year, but also estimates of their spending for the year ahead, once a month for every item of spending.

With the publication of COINS we can now see, for the first time all in one place, the spending and estimates for all of these public bodies.

But bringing this all together has a slight problem — there’s lots of accounting jargon that we can cut through here, to understand the great significance and value of this publication.

COINS: A User Guide

Permanent url: http://www.wheredoesmymoneygo.org/data/coins/

COINS is a big listing of estimated or actual entries of money.

Each entry in the listing involves a named goverment department’s money.

Some of the entries show a department has bought something like a service or a product. Other entries show a department has recieved some money.

Key features:

Programme objects and Programme object groups: each department creates Programme Objects to which spendings is assigned.
Account types (SCOA = Standard Chart of Government Accounts): standard “accounting-like” classifications of spending. Details of how the money is recieved or spent, so you can choose all spending on Wages & salaries or Current Grants to private sector.
CPID: If money is exchanged between government departments we have a record of which departments were involved. The Counter-party Identifier (CPID) in the entry line is the description of the other department.
Data type: Each of the monthly and yearly budgeting exercises can be identified with the Data Type category. Examples of these are Forecast Outturn March, Forecast Outturn April etc.

The release of the COINs data is a huge step forward for transparency in the UK. We hope that the release of the data will lead to much better public understanding of how public funds are being spent. We’ve been very keen to get hold of the COINS data for our Where Does My Money Go? project and our team are already on the case, working to create intuitive visual representations of the data. If you’d like to follow our progress, you can find us at wheredoesmymoneygo.org or on Twitter at @wdmmg!

Share This

UK Government commits to open up new spending data!

Lisa Evans — Wed, 02 Jun 2010 12:29:46 +0000

It’s exciting times right now for people in the UK interested in how public funds are being used. The new government has proposed to publish unprecedented amounts of spending data in unprecedented detail. In the new Coalition Programme for Government (PDF), the PM has committed to the following, which is very similar to the Conservative pre-election promises but with more detail and — crucially — a schedule!

Historic COINS spending data to be published online in June 2010.

All new central government ICT (information and communication technologies) contracts to be published online from July 2010.

All new central government lender documents for contracts over £10,000 to be published on a single website from September 2010, with this information to be made available to the public free of charge.

New items of central government spending over £25,000 to be published online from November 2010.

All new central government contracts to be published in full from January 2011.

Full information on all DFID international development projects over £500 to be published online from January 2011, including financial information and project documentation.

Local government spending transparency

New items of local government spending over £500 to be published on a council-by-council basis from January 2011.

New local government contracts and tender documents for expenditure over £500 to be published in full from January 2011.

Other key government datasets

Crime data to be published at a level that allows the public to see what is happening on their streets from January 2011.

Names, grades, job titles and annual pay rates for most Senior Civil Servants with salaries above £150,000 to be published in June 2010.

Names, grades, job titles and annual pay rates for most Senior Civil Servants and NDPB officials with salaries higher than the lowest permissible in Pay Band 1 of the Senior Civil Service pay scale to be published from September 2010.

Organograms for central government departments and agencies that include all staff positions to be published in a common format from October 2010.

This is all great news for the Open Knowledge Foundation’s Where Does My Money Go? project. In particular we have been researching the COINS database as a rich source of data to visualise. In addition, it is noted that the current standard for reporting central government spending(PDF) is items above £20m in any year by region, so the £25,000 standard seems like a big improvement, hopefully this will also be spending by region.

Share This

Great News for Open Government Data in the UK

Rufus Pollock — Mon, 22 Mar 2010 16:55:04 +0000

Today, in a speech on “Building Britain’s Digital Future”, UK Prime Minister Gordon Brown announced wide-ranging plans to open up UK government data. In addition to a general promise to extend the existing commitments to “make public data public” the PM announced:

The opening up of a large and important set of transport data (the NaPTAN dataset) — this was one of the government datasets listed as ‘important but unavailable’ at our Workshop on Finding and Re-using Public Information back in November 2008!
A commitment to open up a significant amount of Ordnance Survey data from the 1st April (though details of which datasets not yet specified)
By the Autumn an online e-”domesday” book giving “an inventory of all non-personal datasets held by departments and arms-length bodies
A new “institute” for web science headed by Tim Berners-Lee and Nigel Shadbolt and with an initial £30m in funding

This is great news and it is wonderful to see support at such a key political level for this kind of open data approach — an approach that the Open Knowledge Foundation has long been advocating (most recently here and here).

More detailed excerpts (with emphasis added)

Opening up data

In January we launched data.gov.uk, a single, easy-to-use website to access public data [ed: data.gov.uk catalogue is powered by the Open Knowledge Foundation's CKAN software]. And even in the short space of time since then, the interest this initiative has attracted - globally - has been very striking. The site already has more than three thousand data sets available - and more are being added all the time. And in the past month the Office for National Statistics has opened up access for web developers to over two billion data items right down to local neighbourhood level.

The Department for Transport and the transport industry are today making available the core reference datasets that contain the precise names and co-ordinates of all 350 thousand bus stops, railway stations and airports in Britain.

Public transport timetables and real-time running information is currently owned by the operating companies. But we will work to free it up - and from today we will make it a condition of future franchises that this data will be made freely available.

And following the strong support in our recent consultation, I can confirm that from 1st April, we will be making a substantial package of information held by ordnance survey freely available to the public, without restrictions on re-use. Further details on the package and government’s response to the consultation will be published by the end of March.

e-Domesday Book

And I can also tell you today that in the autumn the Government will publish online an inventory of all non-personal datasets held by departments and arms-length bodies - a “domesday book” for the 21st century.

The programme will be managed by the National Archives and it will be overseen by a new open data board which will report on the first edition of the new domesday book by April next year. The Government will then produce its detailed proposals including how this work can be extended to the wider public sector.

To inform the continuing development of making public data public, the National Archives will produce a consultation paper on a definition of the “public task” for public data, to be published later this year.

The new domesday book will for the first time allow the public to access in one place information on each set of data including its size, source, format, content, timeliness, cost and quality. And there will be an expectation that departments will release each of these datasets, or account publicly for why they are not doing so.

Any business or individual will be free to embed this public data in their own websites, and to use it in creative ways within their own applications.

Mygov

…

So our goal is to replace this first generation of e-government with a much more interactive second generation form of digital engagement which we are calling Mygov.

Companies that use technology to interact with their users are positioning themselves for the future, and government must do likewise. Mygov marks the end of the one-size-fits-all, man-from-the-ministry-knows-best approach to public services.

Mygov will constitute a radical new model for how public services will be delivered and for how citizens engage with government - making interaction with government as easy as internet banking or online shopping. This open, personalised platform will allow us to deliver universal services that are also tailored to the needs of each individual; to move from top-down, monolithic websites broadcasting public service information in the hope that the people who need help will find it - to government on demand.

And rather than civil servants being the sole authors and editors, we will unleash data and content to the community to turn into applications that meet genuine needs. This does not require large-scale government IT Infrastructure; the ‘open source’ technology that will make it happen is freely available. All that is required is the will and willingness of the centre to give up control.

Share This

Talk at Chaos Computer Congress on “CKAN: apt-get for the Debian of Data”

Rufus Pollock — Tue, 05 Jan 2010 13:39:08 +0000

Last week OKFNers including myself attended the 26th Chaos Computer Congress (CCC). As previously announced I was giving a talk entitled “CKAN: apt-get for the Debian of Data”.

Slides from the talk are now up here: http://m.okfn.org/files/talks/ccc_20091228/

CCC was an amazing experience — a great venue, a great bunch of people and lots of interesting conversations. It’s also the first event I’ve attended where talks start at 11am finish at 1am and 8pm in the evening is a prime-time talking slot!

Share This