A Traffic Data Odyssey

Recently, partly as an experiment regarding access to government data, partly out of genuine interest in the material itself, I looked into getting hold of some UK traffic count data — useful for, among other things, doing traffic analysis which is key to much road planning and policy (see e.g. this work by R J Gibbens and Y Saatchi at the University of Cambridge).

The results were rather disappointing and provide an interesting illustration of the kind of obstacles that can arise when trying to get access to Government data.

The Odyssey

From previous experience I knew count data was collected by UK’s Department for Transport in the form of MIDAS (motorway incident detection and automatic signalling).

My journey then began with some simple searching which led me to here: . That page provided me with a clear link to “Traffic Count Data and Logs” (in nice bulk data form it appeared) but also informed me:

The access of items marked with a padlock [the link to the data!] is restricted by username and password. If you don’t have access to a username or password, contact the Mott MacDonald Helpdesk. Documents without a padlock icon are publicly available for download.

Following their instructions I contacted the helpdesk as follows (2007-11-21):

I’m a UK citizen interested in getting access to the Traffic Count Data and Logs dataset linked to from:

http://www.midas-data.org.uk/

It appears that a username and password is required from yourselves in order to do this and so I wondered if you could therefore be kind enough to provide me with such a username and password.

I received a prompt response from Mott MacDonald:

Can you please provide me with a short description on what you require access to the data for.

I am required to pass all third party data access requests to the Highways Agency for approval.

To which I responded:

I’d be interesting in looking at the data to see whether I could use it for research. In addition I’m interested in making it available to others in the UK for their own research and reuse purposes.

Once again Mott MacDonald got back promptly and let me know they’d pass my request on (29th Nov 2007). A week of so later (7th of Dec) I received the following letter (as a scanned tif!) from someone in the Highways Agency (all emphasis added):

…

Following your email to the ITG Helpdesk … the Traffic and Travel Systems Group of the Highways Agency is in a position to supply you with the Traffic Data you requested. I need your acceptance of the conditions stated below and some information regarding the research project you are undertaking before we allow you access to the data. The conditions and information I have requested will allow the Group to justify the costs associated with supplying this data [what costs, it’s already in a bzip file on a website?] and to ensure the data is being used appropriately [why is such paternalism needed?].

Note:- if the project is being undertaken jointly with another organisation then that organisation will also be required to supply the information requested. Please ensure all grant and contract holders, staff and students associated with the grant and project are made aware of the conditions contained within this letter.

Conditions

The data may not be copied to any other persons or organisations without the prior approval of the Highways Agency. The data may only be copied to another person or organisation after that person or organisation has confirmed with the HA the purpose for which the data is required and accepted the conditions laid down in this letter.

The data may not be used for any other purpose within your organisation without the prior written approval of the Highways agency.

The data must not be sold or used for commercial gain.

The data will not be used to contradict or challenge any research project, works or statement made by the Government, the Department of Transport of the The Highways Agency as a result of analysis of the data by them or their agents.

the Highways Agency will be provided, upon publication and free of charge, with: annual progress reports; any interim reports describing significant findings; a complete copy of the final report; and any technical papers resulting from the research.

…

Requested Information

I am aware that at this moment in time you may not be able to given an accurate response to all the requests below. At this stage the Agency only requires indication of how large the project is and what its aims are, ….

i) The title of the project that will be using the data …

ii) An outline and brief summary of the objectives of the project …

iii) Who is funding the project …

iv) An estimate of the level of funding for this research …

v) A project schedule …

…

If you have any questions regarding the conditions or the request for information I will be happy to discuss them with you on the above number …

What a lot of restrictions (and hassle):

All “commercial” usage is out (why? the Highways Authority isn’t selling the stuff)
(Rather more surprising) They use data access permission to engage in censorship by making you promise never to criticize the Government, the DfT or the Highways Agency.
Reuse and redistribution are out because you can’t pass on the dataset (or derived ones) without everyone getting explicit permission (and explaining exactly what they plan to do, sending emails and agreements back and forth etc).

I therefore replied as follows (12th December):

Thank you for your letter. I should point out that in my email of the 29th of November to Mr Lyman (see below) I explicitly stated that I was interested in making this material freely available to others for use and reuse, yet the terms of the letter would seem to preclude this. Could you clarify if this is the case, and if so why the Highways Agency needs to restrict what people do with this data in this way.

On the 19th I received the following reply which really just repeated what was in the letter.

The MIDAS data is freely available for your use providing you accept the
conditions outlined in the letter and supply details of your project.
The same is true for others who wish to access the data. If you would
like to highlight the availability of this information to others, can I
suggest you provide the web address to those who show an interest and my
contact details to obtain a password.

The reason for the terms and conditions as described in the letter
ensure that any potential benefits which may come from the use of this
data can be returned to the Highways Agency. It also ensures that the
data is not used to challenge the work of the Department for Transport
or for commercial gain.

…

Intrigued by their justifications for these restrictions responded as follows (15th of January by now!):

Thank you for your response though I still have several questions which I lay out below.

First what are ‘potential benefits’ the HA is concerned won’t be returned to it? Second, even if this were a concern why couldn’t you simply make the data available but with an associated license (or at most just ask people to fill in a web form and then supply them with the data)?

I am also not clear why the HA should be using access control to ensure the data is “not used to challenge the work of the Department of Transport”. After all surely the Government should welcome open debate and analysis rather than using proprietary control of data for something that seems rather like ‘censorship’.

Finally I cannot understand why one would explicitly wish to prevent people using the data to make ‘financial gain’. I assume the HA or the Dept of Transport are not using the data to ‘make money’ so why not let others do so — after all this would surely be beneficial for society both in terms of making new services available to citizens and generating jobs and income for business.

Thank you again for your time so far and I look forward to hearing further from you in relation to the points above.

As was to be expected I didn’t get much of a response:

These are our standard terms and conditions which all recipients of the
MIDAS data are asked to comply with.

If you would like to provide details of your project and organisation
and agreement to the conditions I shouldn’t imagine there will be any
problem in us providing you with the data as we regularly approve the
use of this data for external parties, university students etc.

I look forward to hearing from you with your details.

At pressed for more explanation:

Thank you for your response but I wonder if you would actually be able to comment on the issues I raised in my previous email. It does seem odd that you would only provide the data under these rather restrictive conditions and it obviously raises the ‘transaction’ costs for all those involved in the process.

But was clearly rebuffed:

I would prefer not to enter into a debate about the terms and conditions of the letter.

If you would like to gain access to the data to assist you with an
individual project please provide details as I have previously outlined.
Until you have provided this information I can provide no further
assistance.

And there the matter rested with me none the better off either in terms of data or understanding of the reasons for the existence of these restrictions.