Open Knowledge Foundation Weblog

The inspiration for writing this, as well as much of the information contained herein, came from the search Roundtable which took place at the IDEI Toulouse ‘Conference on the Software and Internet Industries’ on January 20th 2007. An earlier version of this essay as well as notes from the Roundtable can be found in this post on my personal site.

Introduction

In many parts of the world today we already have a situation in which a single search engine has a dominant market share (that is over 70%). At the same time there is, at least for the present, substantial competition — though entry costs are significant and growing. In such circumstances one must be concerned that a monopoly (or small oligopoly) will develop. Such an outcome would not only creates significant pricing/welfare issues but would raise very serious ‘cultural/social’ issues — any entity, corporate or otherwise, that operated the sole (or only significant) search engine on the planet would wield an absolutely immense, and likely unacceptable, degree of power over our political, cultural and social lives.

This concern has already begun to generate a growing discussion of if, and how, we should regulate online search. There are, however, several unattractive features of direct regulation of search: the cost of regulation itself, the inefficiencies arising from poor or inappropriate regulation (arising from the inevitable information asymmetries between the regulator and the regulated) etc.

Open Search: A Proposal

As a result most economists (and technologists) would prefer a situation where competition did the job of regulation. This brings me the key proposal of this article: there should be a concerted effort (government assisted or otherwise) to develop an open-source enterprise search system along with an associated not-for-profit open web search service built on top of this system.

The second part of this proposal is what differentiates it from previous discussions. It is also crucial for several reasons:

1. Learning-by-doing (’kaizen’ effect) is the single most important route to improving search quality (it is assumed that the service would also be run in an ‘open’ manner with all data and research diffused openly — at least as far as that is consistent with privacy etc). Hence, to develop good software you need to be actively used by a significant (and diverse) user population.
2. Because of the ad-funded model it is perfectly possible for the search service to earn significant income (less, probably, than a closed service would do but the first mover and open-source ‘brand’ advantages should be sizable — look at Firefox). As a result running a complementary service could be a substantial source of development funds.
3. From a societal standpoint the service is what you really care about. While it is true that other people could run a web service with the same code (exactly the point!) it would be nice to guarantee the existence of a service, particularly one closed associated (name/reputation) with the underlying software.

These are all important points from an implementation point of view but they say little about why this proposal is good from a viewpoint of social welfare. The answer though is obvious:

1. We get de facto competition from a service which prices at ‘marginal cost’ (no rent extraction)
2. the algorithms, software and accumulated data (by no means the least important of the 3!) are all open and therefore there are general efficiency gains
3. As everything is open we have a guarantee of/commitment to future competition
4. Extra competition from entry of other service providers using the same codebase (NB: license would want to be of a GPL+/Affero type so that service providers are obliged to contribute back code improvements)
5. Greater transparency (we know why some sites come top, why some don’t).

Some remarks on funding: while in the long-term this ‘open-search’ system could well be self-supporting financially it seems likely that it would need a decent injection of up-front funding to get it going. Where would this money come from? Three possibilities: a) venture capital b) government or private donor c) the general web community. (a) seems improbable given the not-for-profit nature of the endeavour and the complete openness of both code and service (might be possible to modify not-for-profit to allow pay-back to backers but this is more like a straight loan and less like a VC setup). (c) is possible but also seems difficult — one would be looking for significant community support for development, testing, use and promotion but harder to see it as a reliable source of funds, at least initially. That leaves us with (b) up front funding from government or private charity. Given the large technological spillovers, the clear social benefits in promoting competition and transparency, and the ‘utility-like’ nature of search this would seem to be a promising route to take.

The Current Search Model: Existing Problems

Even without the development of a monopoly (or oligopoly) the current search situtation gives cause for concern, as Kamal Jain (Microsoft) detailed in his presentation on the ‘ugly side’ of the ad-funded search model at the Toulouse Roundtable (see below for more info on the Roundtable). In particular:

• Search engines are currently getting to extract all the rents from selling my attention (equivalently: we are selling our attention, and click-stream, for free). This situation does not appear to be sustainable. It is already being addressed in some ways — Firefox, for example, is getting money from Google for clicks — and there are plenty of other methods, for example, one could start a not-for-profit web proxy that auctions queries to search engines and then routes results to users passing money back to those users. Storing and selling your clickstream would be another method (this has been around a while but never really taken off — perhaps because privacy stuff becomes too obvious and not really clear what benefits will be).
• The funding model of search (in which search is funded by advertisers and free for users) may be societally inefficient. Search engines are in a monopolistic position vis-a-vis advertisers and therefore may a) overcharge advertisting (with those charges eventually passed back to users) b) over-provide search quality.
• Little transparency over search engine rankings which are of ever-increasing commercial and social importance.

Share This