Last week I went to the first COMMUNIA workshop on Technology and the Public Domain in Turin.

COMMUNIA coordinator Juan Carlos De Martin and Rishab Ghosh of MERIT, University of Maastricht gave opening talks. I was on a panel with Kaitlin Thaney of Science Commons, Nathan Yergler of Creative Commons and Keith Jeffery of euroCRIS. My slides are available at:

• http://m.okfn.org/files/talks/communia_20080118/

Other talks directly related to open knowledge included:

• Xavier Serra (Music Technology Group, Pompeu Fabra University) - on automated metadata production for sound, and the freesound project (content is currently licensed under CC Sampling Plus, but there are plans to introduce other licensing options)
• Simone Brunozzi - on BeeSeek, a peer to peer open-source search engine
• Séverine Dusollier (FUNDP - Centre de recherche informatique et droit) - on setting up a positive legal definition and regime for the public domain

Seeing as though there look to be many areas of common interest between the OKF and the COMMUNIA network, I suggested in my talk and thoughout the day that:

• We should work together to create a set of ‘public domain calculators’ - or algorithms that can help to determine whether or not a given work is out of copyright in a given jurisdiction (such as we’ve been working on with Public Domain Works and the Open Library);
• We should work together to pool open metadata - whether this be bibliographic metadata, or metadata for databases or large collections of knowledge resources (such as are listed in CKAN).

This is a great opportunity to strengthen the community of individuals and organisations with an interest in open knowledge and the public domain across Europe. I look forward to seeing the launch of the Working Groups!

Share This

On Saturday I attended a ‘BarCamp’ on the Power of Information Review Recommendation 8 - which suggests there should be a re-use request service for UK Public Sector Information (we blogged about this in October).

The event was organised by John Sheridan of the Office of Public Sector Information and was attended by representatives from government, the private sector, the media, and nonprofits - including mySociety’s Tom Steinberg, who co-wrote the review in June 2007.

The meeting went well - and quite a bit of time was spent planning what the service will look like and what it will do. Below are some jottings for those that are interested. (These are rough and uncomprehensive, please don’t hesitate to get in touch if there’s anything missing or incorrect!)

Notes

John Sheridan: Government have limited expertise in developing certain kinds of web services. What are barriers to being able to re-use UK PSI?

Brainstorming session: all participants were invited to suggest the kinds of things they’d like to see and discuss throughout the BarCamp.

Rob McKinnon, mySociety NZ

• suggested a kind of e-democracy existed in the 19th c. with suffragists?
• parliament = paper?
• insert vote output legislation
• more paper inside parliament
• parliament is about data
• creating data rather than paper
• he was inspired by Public Whip and They Work For You
• screenscraping HTML from government website
• NZ theyworkforyou site
• (aside, Francis Irving: we used data, then the click use license came along)
• (aside, John Sheridan: several hundred thousand pounds of potential revenue lost through switch to click license)
• no copyright exists on NZ parliamentary debates
• (aside, Richard Quarrell: distinction between local government and central government?)
• screenscraping parliament.nz
• getting metadata from HTML SPAN tags
• trying on a small sample, testing on a larger sample
• doesn’t have to be structured/semantic data, rdf
• making ‘’’source”’ information available - people will make use of it
• politics about politicians or people?
• networked democracy
• making information transparent, facilitate social collabortation, participation
• make data discoverable
  • 80% of TheyWorkForYou NZ’s visitors come via Google’s search
  • use canonical, reliable and readable URLs
• make data linkable
• let people mark content
  • dopplr, upcoming
• more participation through transparency of requests
• (aside, Michael Cross: legal basis for requesting information remained available? National Archives. famous case of documents from East India Company smelling of a certain oil - which turned out to be source of information about health)
• (aside, Richard Quarrel: its an interesting question whether metadata/html tags count as ‘information’ under, e.g. FOI requests…)

Discussion

• John Sheridan: NZ gov effectively developed own microformats. One thing he [John] does is convince government metadata working group to develop microformat. Microformats for licensing like Creative Commons. Adding licensing information to URI.
• John Sheridan: Heaps of work on microformats already exist.
• Tom Steinberg: Respond in a flexible manner to demand for formats rather than blanket mandate for all material to be in format X.
• Richard Quarrell: government sharing - egovernment standards?
• Tom Steinberg: different tags for different government departments

Francis Irving, mySociety

• Freedom of Information requests
• discussed mySociety’s Freedom of Information Filer and Archive service which is under development
• database dumps
• (aside, Glyn Wintle: sometimes government find it easier to give whole database rather than answer a particular query)
• encouraging real names on requests rather than pseudonyms
• (aside, John Sheridan: snag that not all data gov has it owns. can’t re-publish third party information without investigation… local authority can serve requests, but cannot give permission to republish.)
• (aside: rights in, e.g. address data. 3 different bodies own rights: Post Office, …)
• make copyright clear on data
• (aside, John Sheridan: suggest we move naturally on to licensing psi, rights, etc.)

Michael Cross, Free Our Data

• APSI looking at bigger economic picture
• raw data should be made available for free
• (aside, John Sheridan: government encrypting all in sight, if think no-one wants it. whilst policy framework encourages re-use…)
• (aside, Stephan Carlyle: Deal with 40,000 FOI requests. 900,000 environmental information regulation requests. Value added requests. Point to info already published. Publishing often costs less than production. Balanced approach to access. Most requests are members of public wanting to out certain things in their locality. Danger of having too much of a focus on boundaries and exceptions. 97% of requests easy to respond to within 20 days.)
• (Tom Steinberg demonstrated Department of Health Information Asset Register)
• (aside, Richard Quarrell: plan to publish comprehensive list of IARNs - information asset register numbers)
• (aside, Tom Steinberg: do we design a request service to put pressure on government FOI policy, or one that works within the bounds of existing legislation?)
• (aside, Rob McKinnon: refinement with proprietry content?)
• (aside, John Sheridan: Cross Cutting Review [of Knowledge] says material is available for re-use by default)
• Discussion of point 9.7 of click use license, about using PSI in misleading context.

(Lunch)

• Demoing maptasm

Brainstorming for request service

• What are the goals?
• Who are the users?
• What sort of things are people going to request?
• How will people find out that they can request info?
• How will the service help PSIH’s (Public Sector Information Holders) to be more responsive?
• How do we make sure the data is released in internet time?
• Relation to FOI + distinction?
• The process - how should it work?

Aims

1. Government information provision is driven by what people want. Provision should become driven by demand.
2. Not enough to make culture change documents. Need sticks. Quite strong incentives. Shame and money. Pressure through revealing failure to serve. Threat of budget cuts if information isn’t served.
3. Increase knowledge of what is available and what has value (if not published).
4. What ‘raw’ info is available and consistent way of gaining access. Expressing this.
5. Complement other initiatives
6. To be safety net for all other information provision.

Types of request

1. Too expensive to get under other rights of access or re-use
2. Clarification of licenses
3. Change of licensing terms
4. Can’t find
5. Change of law relating to info publication
6. data that doesn’t exist but should
7. Change cost of information
8. Different formats of information
9. Change purchasing or obtaining

Who are the users

1. Civic society
2. Academics
3. Private sector (data products, open models)
4. Public servants/departments
5. Individuals

How will people find out

1. In every copyright statement
2. Within click use license pages
3. OPSI front page
4. Google
5. IFTS site
6. Anywhere you can buy information from government
7. PSIH how to get info pages
8. FOI officer training
9. Information Commissioner’s office (ico)
10. Success stories?
11. Business office
12. Its own blog

• Distinction between procedural and policy complaints?

How does it work

• Category of change requested
• Details of user
• PSIH(s) concerned (and ‘don’t know’)
• Dataset requested
• Problem is wrong policy
• Problem is execution
• Nature of problem
• What I could do if I had it
• How it should work in future
• Story/history

The process - what happens?

1. Published on a page
2. Published on a new items page
3. Problems email alterts + rss feeds
4. Mail named contact?
5. Provides tips and tricks + explanation for obtaining what they want
6. Endorse function inc. status of endorses + use cases
7. Discussion thread inc. authentication for PSIH responses
8. Write to creator of report

• Canonical list of bodies which pulbish PSI? See civil service handbook for lists.

Endorsing page

• Name
• Email
• Allow me to be emailed by (opsi/poster/anyone)

Share This

A couple of weeks back we blogged about the ‘Future of Bibliographic Control’ draft report from a working group at the Library of Congress. Since then, we’ve submitted to the group a brief, collaboratively edited response to the draft and an appendix with some additional detailed comments.

The response was drafted by the Open Knowledge Foundation and Aaron Swartz of the Open Library and was co-signed by over 150 groups and individuals, including:

• Lawrence Lessig, Founder, Creative Commons
• Brewster Kahle, Founder, Internet Archive
• Tim O’Reilly, Founder and CEO O’Reilly Media
• Tim Spalding, Founder, LibraryThing.com
• Peter Suber, Senior Researcher, The Scholarly Publishing and Academic Resources Coalition
• John Wonderlich, Program Director and John Brothers, CTO, Sunlight Foundation
• Paul Miller, Rob Styles, Terry Willan, Talis
• Rick and Megan Prelinger, Prelinger Library & Archives
• … and librarians, system librarians, catalogers, assistant librarians, library support staff, library users, library school lecturers and students, consultants, academics and software developers from Australia, Belgium, Canada, Germany, India, Italy, the Netherlands, Norway, Portugal, Spain, the Ukraine, the UK and the US.

Many, many thanks to all of those who helped to publicise this, and to those who co-signed the response! We hope that the working group consider amending the draft in light of our comments in January.

Share This

Nick Holmes of Binary Law wrote to tell us about the Office of Public Sector Information’s forum for a new PSI re-use request service. His blog post about the forum is here.

As noted in the forum description, the service is being built in response to recommendation 8 in the Power of Information Review by Ed Mayo and Tom Steinberg, published in June:

Recommendation 8. To improve government’s responsiveness to demand for public sector information, by July 2008 OPSI should create a web-based channel to gather and assess requests for publication of public sector information.

In the inaugral post from 5th September, John Sheridan (Head of e-Services at OPSI) says:

The government has accepted this recommendation and OPSI would like to engage with the re-user community with setting this new web-channel up. This is why we have set up this discussion forum, to hear your views and discuss our plans.

We hope those with an interest in re-using PSI let their thoughts be known!

Share This

John Gray has recently been conducting some rather interesting research on the degree of re-use actually being achieved at local government level in the UK pursuant to the Euro Directive on the Re-Use of Public Sector Information (Statutory Instrument SI 2005/1515) which came into force in July 2005. The aim of the directive was to encourage the re-use of public sector information for both commercial and social purposes as part of a wider programme of stimulating the ‘knowledge economy’.

In order to investigate what it is actually happening at the local authority level John has heroically set about writing to all 426 local authorities. His summary of his results make interesting reading:

…

The replies received from the Local Authorities are disturbing in that:

• Over 40% have failed to offer the simple courtesy of a reply.
• 75% of those who have replied - have FAILED to comply with the Euro Directive.
• On the part of Local Authorities there seems to be a very high degree of confusion between the management of FOI (Freedom of Information) and the re-use of PSI (Public Sector Information).

…

The Survey of all 426 Local Authorities clearly indicates that although Central Government resources have been allocated to promoting the needs of the Euro Directive - through OPSI (Office of Public Sector Information); APPSI (Advisory Panel on Public Sector Information) and the Office of the Information Commissioner, acceptance at an operational level with UK Local Authorities is at best poor and at worst - non existent.

Clearly much remains to be done if the directive is going to deliver on its promise to unlock the value of public sector information. In my view, one of the big problems is the current obsession with IP which means that local government is frequently both:

• Uncertain about what they can do with their data (who else has rights in it).
• Anxious not to ‘give away’ anything that could generate revenue.

It would make a big start on the problem if a straightforward default of openness was imposed for all non-personal data — via blanket application of the basic click-use license, for example. Whatever the case the current regime is unsatisfactory, as John put it in his email to me (I couldn’t have put it better):

Local Government forms a bedrock of Public Administration in the UK and is one of the main conduits by which Governance impacts upon the citizen.

In view of such, it is rather disconcerting to note how the Local Government sector seem to have almost totally ignored the open information rights that were and are a fundamental precept of the Euro Directive.

Additional Materials

For convenience we have posted the full text of John Gray’s summary, as well as all additional materials, online at:

http://m.okfn.org/files/misc/gray_psi_2007/

Share This

I’ll be speaking on the Copyright Users panel at the Copyright and Research in the Humanities and Social Sciences conference which takes place this Friday (30th March 2007) in Edinburgh. The event is being jointly organised by The British Academy and the AHRC Research Centre for Studies in Intellectual Property and Technology Law at the University of Edinburgh and aims to examine the recommendations that came out of the British Academy’s review of Copyright and Research.

The conference is an all day event and takes place at the Playfair Library Hall, Old College, University of Edinburgh, South Bridge, Edinburgh EH8. I’ll be travelling up to Edinburgh Thursday so if there is anyone based in the area who’d like to meet to talk about copyright, open knowledge or anything related please drop me a line on rufus [dot] pollock [at] okfn [dot] org.

According to the conference announcement (which has further details including the schedule and how to book a place):

The conference follows the Academy’s Review of Copyright and Research and its subsequent publication of a set of Guidelines by Professor Hector MacQueen of the AHRC Research Centre.

Recent developments in technology, legislation and practice have meant that the various copyright exemptions, which enable creative and scholarly work to advance, are not always achieving the intended purpose. A British Academy Review, Copyright and Research in the Humanities and Social Sciences, published in September 2006, drew attention to the problems that were occurring and recommended ways in which they should be addressed. The Gowers Review of Intellectual Property (published December 2006) is a step in the right direction, as it seeks to strike a balance between copyright owners on the one hand, and copyright users (“follow-on innovators”) on the other hand. However, the main concerns raised by the Academy and others in the academic research community still need to be addressed. The conference aims to consider the findings and recommendations of the Academy Review, and place them in context post-Gowers. It will followed by a dinner to celebrate the Centre’s achievements.

Share This

The Open Knowledge Foundation made a joint response in association with the Open Rights Group) to OfCom’s Public Service Publisher consultation (officially titled: A new approach to public service content in the digital media age).

The Response

The founding of a Public Service Publisher (PSP) is an opportunity to make a significant ongoing investment in the vast landscape of public, ‘open knowledge’ infrastructures already developing on the Net.

We, the undersigned, feel that the PSP could play a vital role in addressing the strategic concerns of the Net as a global and national infrastructure; exploring and protecting the educational, commercial and societal possibilities of what ‘public service’ might mean in this new context.

We are greatly encouraged by the direction expressed by OfCom’s “new approach to public service content in the digital media age.” Our response aims to steer the development of this project in the direction our combined experience and practice suggests would be of most value to the UK public.

Firstly, we commend the suggested investment in open content and open data. In particular we urge that, where the PSP funds the generation of new content, such content should always be made available under a license such that others are free to enjoy, redistribute and, most importantly, reuse and refashion that content.

Secondly, we ask that OfCom pay special attention to the ability of the PSP to invest in architectures of participation, both by supporting the development of Free, Libre and Open Source Software (FLOSS) and Open Content technologies and projects and by investing in the creation of content to encourage the growth of networks around these technologies.

Thirdly, we ask that OfCom recognise the transnational nature of the networked communications environment, and refrain from sanctioning measures designed to limit the benefits derived from the PSP to UK residents alone. The PSP and the projects and content it funds should be viewed as nodes on a global network. It should be assumed that the exchange of information and content across such a global network will be to the net benefit of the UK public.

The success of an endeavour like the PSP will rely upon these details of its founding principles, and we urge OfCom to pay significant attention to those details now. For example, the PSP may commission a website for people to post and discuss short films, investing in the “architectures of participation” suggested above. But unless the use of Free/Open Source software is specified, and the resulting website platform is ‘open’, allowing re-use and modification by other interested parties, the PSP will not be fully meeting its public service remit. Similarly, The PSP might commission a set of short films to be placed on the website, to seed its growth as a network. But unless the PSP commission explicitly requires that the resulting work be ‘open’ so that others are free to use, reuse and redistribute the work, the PSP’s audience will remain ‘consumers’ of content, and the PSP will have failed to maximise the opportunities of the digital age.

Finally, the PSP should engage in advocacy and educational initiatives to enable people, organisations and companies to publish their material using open licenses, formats and technologies. It is our sincere hope that the PSP can become a strong, public voice in favour of open knowledge structures.

We welcome this and any future opportunities to respond to OfCom’s plans for the PSP.

• The Open Knowledge Foundation
• The Open Rights Group
• Free Culture UK

Share This

At Rufus Pollock of the Open Knowledge Foundation’s command (’It’s your civic duty!’) I decided to accept an invitation to the riverside HQ of OFCOM, the UK’s independent regulatory body for television, radio, telcoms and wireless, to participate in a discussion about what the UK’s putative ‘Public Service Publisher’ (PSP) should be.

It seems that OFCOM recently noticed the Internet and decided that some kind of public service intervention was necessary beyond BBC online’s existing offering. Projecting a budget of 100M, they embarked on a consultation process led by Andrew Chitty of ‘convergent media’ production company Illumina Ltd.

The room at OFCOM’s London Bridge offices was populated with execs from Yahoo, Google, and various Internet Service Providers (ISPs) as well as institutional players like the British Film Institute and the BBC. I think I was the only person there not representing a large corporation of some sort. I worked out what my civic duty was going to be when the ‘creative’ director at Wanadoo suggested that the PSP’s 100M budget should be given to the telcos and ISPs for their wonderful PSP-like job of carrying peer to peer network traffic, and nobody batted an eyelid. I spent the rest of the day desperately clawing the discussion back to what the ‘public service’ bit could mean.

While reading through the consultation website and skimming the full consultation document, I was pleasantly surprised to see that heavily watered-down mention was made of non-restrictive IP models:

“…it is unlikely that restrictive IP models will maximise public value in a way which is consistent with the overarching thesis of the paper, namely that new forms of public value can be found in the participatory media environment which are distinct from those in the traditional world of linear broadcasting.”

Whew! For the first few pages I really wasn’t sure we’d even get that far.

Reading through the wordy reiterations of the BBC and OFCOM’s mission statements in relation to one another and the Internet, I was also pleased to see a mention (however vague) of the Creative Commons concept.

Unfortunately, this is diluted in a load of projects bunched together in the ‘already-out-there’ section: . The strange groupings of web sites there set my alarm bells ringing, headlining narrowly UK-focused, mostly government-funded sites of dubious popularity and quality while relegating Wikipedia and Flickr to footnotes in the ‘other links’ section of the ‘User Generated Content’ category alongside (bizarrely) Ebay and the Human Genome Project.

In this blurry segue into a list of web sites, it became clear that the decision had already been made to turn the PSP into a funding agency that gives money to people to make British ‘new media projects’ - presumably with the overarching aims of ‘educating’ and ‘entertaining’ the ‘public’.

What I was really hoping for was a bit of strategic thinking: thinking that might actually recognise that the Net and the emerging universe of electronic devices that people use to communicate, create and use networks, and on which people build their own platforms is an infrastructure, not a fairground.

I was glad to see that one comment I’d made about these sites had made it into the report:

“What we see now are the equivalents of the 19th century end-of-the-pier zoetropes and nickelodeons, but somewhere in there is the new cinema”.

What a pity that it hadn’t been understood at all.

Deep breath.

‘Cinema’ is not a project. It is a complex and interlinked infrastructure, that was only allowed to develop because of the difficulty Edison Laboratories would have had in patenting the Kinetoscope in Europe. This was partly because Edison had borrowed from prior British inventions. In fact, it was two British inventors: Birt Acres and Robert Paul who extrapolated the Kinetoscope into the first 35mm camera - which they never managed to patent effectively. This didn’t stop a war raging over patents - led by the Pathe Freres company in Europe and Edison’s Motion Picture Patent Company (a.k.a. the ‘First Oligopoly’) in the US, patenting and controlling technological development, owning cinemas and developing monopolies throughout the industry. The judiciary of the US - through public interest patent-busting and anti-trust suits - finally broke the First Oligopoly in the early 1910’s, only for others to form, consolidating the power of the Film and global mass media industries in Hollywood as the Independent Studios and their star system emerged in the 30’s, leading to intense vertical integration of the whole film industry.

The British Government’s attempted intervention in this consolidation process was the 1927 Cinematograph Films Act, which put a quota on British Films being shown in UK Cinemas - leading to overproduction of low-quality low budget ‘quota-quickies’ in the run up to WWII, which put the final nail in the coffin of the British Film Industry. It’s been interestingly pathetic since then.

So the question is not which of these ‘projects’ is the next cinema? The question is - what underlies these projects? Who are the Edison Labs and Pathé Frères, MGMs, Paramounts, Foxes, RGOs and Loews of the Net? Who is defining and owning and shaping how the Net is used, understood and extended?

These days, it looks like the search engines. The Googles, the Yahoos, the information associators who have a semantic stranglehold on the Web and increasingly on other parts of the Net. This is not to mention the infrastructure owners: the DNS demagogues, the backbone bonapartes, the people who can hit the ‘off’ switch or start metering access to their network territories.

But what could a Public Service Publisher do about this? Surely it’s in the public interest to address the fact that the infrastructure we’re all using to do business, publish, and socialise online is dangerously similar to Cinema’s vertically integrated Hollywood-centric oligopolies?

Clearly, the PSP is going to do absolutely nothing:

“A further key role for the PSP would be in ensuring that search mechanisms for its content - and conceivably for all public service media content - become as efficient as possible. This would never extend to the development of a search engine, but it would involve working with search engine specialists and the major global and local players in search to establish tagging and discovery mechanisms to facilitate this.”

It sounds like we’re going to help them tighten the stranglehold they already have.

I’m not suggesting for a moment that we should be developing some kind of national search engine like the disastrous French ‘Quaero’ project. As online publishing and metadata usage becomes more sophisticated and widespread, the roles of the search engines will change - their original role of keyword indexing and ’scoring’ websites will be made less necessary by the improved semantic coherence of data on the Net - a.k.a. the ‘Semantic Web’.

But that technical development won’t necessarily loosen their grip. Using their existing market positions the search engines are working hard to consolidate their indexes, page-association ranks, user profiles and acquiring as much high quality data (scanned books, geodata etc.) as they can to insure their long-term market centrality. In an industry where innovative companies could once become giant killers overnight, they know that their ownership of what you could call the ‘means of association’ will have to be complete and coherent if they are going to protect their advertising revenues.

My response to the PSP consultation, emailed to the organisers soon afterwards doesn’t yet appear on the empty ‘responses’ section of the site. For the record, this is what I thought the PSP could do about this at the time:

• Researching and advising on best practice in metadata, exchange and archiving standards.
• Researching and advising on best practice in legal preservation and maintenance of publically funded IPR.
• Producing and maintaining high quality free educational materials for groups and individuals in how to publish their video/audio/text online and archive it well enough for it not to contribute to the uncatalogued backlog.
• Investing in open source software and shared IPR projects that are consistent with and facilitate the above goals.
• Research and develop systems for traversing, searching and making inferences from data generated by the aggregation of all this published material, and make that data, and those queries available via open APIs.

The last point is the crucial one: public interest in maintaining a lively and innovative environment on the Net would be served best by helping to build less centralised search and discovery systems, and making the data underlying those systems, and the systems themselves available to all (irrespective of nationality) using Public Domain licenses.

This would not be a project the PSP would have to start from scratch. There is already an inspiring and powerful world-wide movement to which they could add welcome legitimation and support. This was the final plea I made to the consultation group:

“Please, please *please*, don’t lets reinvent any wheels. There are some great projects and initiatives out there, mostly organised along very ad-hoc and non-institutional lines. If this PSP idea can be kept human-scale at the edges, can be smart and careful in how it invests money and time in things, it could become part of an existing international ecology of open source publishing platforms, advisory organisations and citizen-publishing initiatives.”

However, I’m sorry to say it looks to me like the PSP outlined in the OFCOM report isn’t just going to reinvent the wheel, it’s going to be a tax-payer funded factory for reinvented wheels. What I didn’t understand until yesterday, 25th January 2007, was who would be running the factory.

After fuming over the newly published PSP report, I went to see a presentation by the author Andrew Chitty from Illumina Ltd. at an ‘InSync’ event called ‘Rights will make you Rich?’ organised by Frank Boyd at 01zero-one in Soho: the centre of the UK Film Industry, such as it is.

Andrew Chitty’s presentation was about how the recent Communications Act (2003), which grants IPR rights to independent TV production companies, rather than to the commissioning broadcasters could be mirrored in agreements he, as a Vice Chair of PACT (the UK’s media industry lobby group) is negotiating with the publically funded BBC.

He also talked about how if a similar arrangement could be made with the putative PSP, it’s 100M jackpot could be used to part-fund projects to which independent production companies like Illumina Ltd. would then own international IP rights.

The last thing I heard him say, nodding complicitly to his BBC comissioner in the front row was ‘maybe this one will land us all on that private Greek island’. Then the red mist came down and I vaguely remember lashing out verbally in the ensuing debate before leaving to spare myself total apoplexy.

It is hardly surprising that the PSP report would be so skewed to the interests of the media industry lobby groups. After all, with the UK advertising and media industry in a recession - that structural change and viewer-group fragmentation onto the US-dominated Internet may make permanent - the public purse must look increasingly tempting.

What is so infuriating about this stitch-up is that it completely misses the real commercial opportunities in public service models on the Net.

The infrastructure of the Net as an offshoot of US federally-funded and therefore Public Domain defence research became a common carrier on which millions of businesses, supported by the universe of Free and Open Source software have been built. The disproportionate reach that creative entrepreneurs could have using this common infrastructure gave birth to the Yahoos and the Googles that are now beginning to enclose parts of it.

Sadly, it seems the PSP outlined in Andrew Chitty’s document will be producing a remake of the 1927 Cinematograph Films Act, funding the struggling UK film and TV industry to produce a quota of parochial ‘new media projects’, the IPR to which they may then exploit world-wide.

The challenge for the PSP, totally missed by this consultation, lies in addressing the strategic concerns of the Net as a global and national infrastructure; exploring and protecting the educational, commercial and societal possibilities of what ‘public service publishing’ might mean in this new context.

Saul Albert 26/01/2007

---

OFCOM’s PSP consultation closes on the 23rd March 2007 - so if you want to see a useful PSP, please make sure you get in your response before then!

The opinions expressed above are not necessarily representative of the OKFN.

Many thanks to Rufus Pollock, Paula le Dieu, Gordon Joly, Nick Fry and Becky Hogge for great feedback, corrections and suggestions on drafts of this response.

Share This

The inspiration for writing this, as well as much of the information contained herein, came from the search Roundtable which took place at the IDEI Toulouse ‘Conference on the Software and Internet Industries’ on January 20th 2007. An earlier version of this essay as well as notes from the Roundtable can be found in this post on my personal site.

Introduction

In many parts of the world today we already have a situation in which a single search engine has a dominant market share (that is over 70%). At the same time there is, at least for the present, substantial competition — though entry costs are significant and growing. In such circumstances one must be concerned that a monopoly (or small oligopoly) will develop. Such an outcome would not only creates significant pricing/welfare issues but would raise very serious ‘cultural/social’ issues — any entity, corporate or otherwise, that operated the sole (or only significant) search engine on the planet would wield an absolutely immense, and likely unacceptable, degree of power over our political, cultural and social lives.

This concern has already begun to generate a growing discussion of if, and how, we should regulate online search. There are, however, several unattractive features of direct regulation of search: the cost of regulation itself, the inefficiencies arising from poor or inappropriate regulation (arising from the inevitable information asymmetries between the regulator and the regulated) etc.

Open Search: A Proposal

As a result most economists (and technologists) would prefer a situation where competition did the job of regulation. This brings me the key proposal of this article: there should be a concerted effort (government assisted or otherwise) to develop an open-source enterprise search system along with an associated not-for-profit open web search service built on top of this system.

The second part of this proposal is what differentiates it from previous discussions. It is also crucial for several reasons:

1. Learning-by-doing (’kaizen’ effect) is the single most important route to improving search quality (it is assumed that the service would also be run in an ‘open’ manner with all data and research diffused openly — at least as far as that is consistent with privacy etc). Hence, to develop good software you need to be actively used by a significant (and diverse) user population.
2. Because of the ad-funded model it is perfectly possible for the search service to earn significant income (less, probably, than a closed service would do but the first mover and open-source ‘brand’ advantages should be sizable — look at Firefox). As a result running a complementary service could be a substantial source of development funds.
3. From a societal standpoint the service is what you really care about. While it is true that other people could run a web service with the same code (exactly the point!) it would be nice to guarantee the existence of a service, particularly one closed associated (name/reputation) with the underlying software.

These are all important points from an implementation point of view but they say little about why this proposal is good from a viewpoint of social welfare. The answer though is obvious:

1. We get de facto competition from a service which prices at ‘marginal cost’ (no rent extraction)
2. the algorithms, software and accumulated data (by no means the least important of the 3!) are all open and therefore there are general efficiency gains
3. As everything is open we have a guarantee of/commitment to future competition
4. Extra competition from entry of other service providers using the same codebase (NB: license would want to be of a GPL+/Affero type so that service providers are obliged to contribute back code improvements)
5. Greater transparency (we know why some sites come top, why some don’t).

Some remarks on funding: while in the long-term this ‘open-search’ system could well be self-supporting financially it seems likely that it would need a decent injection of up-front funding to get it going. Where would this money come from? Three possibilities: a) venture capital b) government or private donor c) the general web community. (a) seems improbable given the not-for-profit nature of the endeavour and the complete openness of both code and service (might be possible to modify not-for-profit to allow pay-back to backers but this is more like a straight loan and less like a VC setup). (c) is possible but also seems difficult — one would be looking for significant community support for development, testing, use and promotion but harder to see it as a reliable source of funds, at least initially. That leaves us with (b) up front funding from government or private charity. Given the large technological spillovers, the clear social benefits in promoting competition and transparency, and the ‘utility-like’ nature of search this would seem to be a promising route to take.

The Current Search Model: Existing Problems

Even without the development of a monopoly (or oligopoly) the current search situtation gives cause for concern, as Kamal Jain (Microsoft) detailed in his presentation on the ‘ugly side’ of the ad-funded search model at the Toulouse Roundtable (see below for more info on the Roundtable). In particular:

• Search engines are currently getting to extract all the rents from selling my attention (equivalently: we are selling our attention, and click-stream, for free). This situation does not appear to be sustainable. It is already being addressed in some ways — Firefox, for example, is getting money from Google for clicks — and there are plenty of other methods, for example, one could start a not-for-profit web proxy that auctions queries to search engines and then routes results to users passing money back to those users. Storing and selling your clickstream would be another method (this has been around a while but never really taken off — perhaps because privacy stuff becomes too obvious and not really clear what benefits will be).
• The funding model of search (in which search is funded by advertisers and free for users) may be societally inefficient. Search engines are in a monopolistic position vis-a-vis advertisers and therefore may a) overcharge advertisting (with those charges eventually passed back to users) b) over-provide search quality.
• Little transparency over search engine rankings which are of ever-increasing commercial and social importance.

Share This

The OKF has been very actively involved in the publicgeodata’s campaign on the INSPIRE directive. Now that it appears compromise between all of the parties — the European Commission, Council and Parliament — has been reached it is natural to ask ourselves both: Where next? and How did we do?

Where Next

The immediate point to make here is that on the issues we care about that the compromise allows for national law makers to exercise a lot of discretion on how they implement the Directive. From our point of view this means there’s plenty to fight for at the national level as INSPIRE will need to be ‘transposed’ into each national law. Any optionality is another chance to obtain more ‘open’ legislation as well as an opportunity to make the case for the social and commercial benefits of open geodata.

How Did We Do

All in all I think the campaign has been a tremendous success. Ok, so we didn’t manage to achieve a total u-turn in European geodata policy but

(a) We can do that next time

(b) Though not perfect, INSPIRE is an improvement over the status quo (non-open/unfree geodata is currently the norm across Europe)

What we did achieve with a campaign that was zero-budget, entirely dependent on spare bits volunteers’ time, and only started when the directive was already at second reading was:

1. A petition that was signed by over 7000 citizens from across the EU

2. Letters to MEPs and national ministries making the case for open geodata along with personal contact with many of the parties involved (MEPs, civil servants etc). This will stand us in good stead in the future and likely had some impact on the compromise that was eventually reached.

3. The dissemination and analysis of a large amount of information about what was happening (particular credit to you here Benjamin Henrion)

4. Link ups with other campaiging groups such as the UK’s freeourdata

So, all in all, I think there plenty to be proud of which should give us heart as we prepare ourselves to take the campaign on to the national level.

Share This