The inspiration for writing this, as well as much of the information contained
herein, came from the search Roundtable which took place at the IDEI Toulouse
‘Conference on the Software and Internet Industries’ on January 20th 2007.
An earlier version of this essay as well as notes from the Roundtable can be
found in this post on my personal
site.
Introduction
In many parts of the world today we already have a situation in which a single
search engine has a dominant market share (that is over 70%). At the same time
there is, at least for the present, substantial competition — though entry
costs are significant and growing. In such circumstances one must be concerned
that a monopoly (or small oligopoly) will develop. Such an outcome would not
only creates significant pricing/welfare issues but would raise very serious
‘cultural/social’ issues — any entity, corporate or otherwise, that operated
the sole (or only significant) search engine on the planet would wield an
absolutely immense, and likely unacceptable, degree of power over our
political, cultural and social lives.
This concern has already begun to generate a growing discussion of if, and how,
we should regulate online search. There are, however, several unattractive
features of direct regulation of search: the cost of regulation itself, the
inefficiencies arising from poor or inappropriate regulation (arising from the
inevitable information asymmetries between the regulator and the regulated)
etc.
Open Search: A Proposal
As a result most economists (and technologists) would prefer a situation where
competition did the job of regulation. This brings me the key proposal of this
article: there should be a concerted effort (government assisted or
otherwise) to develop an open-source enterprise search system along with an
associated not-for-profit open web search service built on top of this system.
The second part of this proposal is what differentiates it from previous
discussions. It is also crucial for several reasons:
- Learning-by-doing (‘kaizen’ effect) is the single most important route
to improving search quality (it is assumed that the service would also be
run in an ‘open’ manner with all data and research diffused openly — at
least as far as that is consistent with privacy etc). Hence, to develop
good software you need to be actively used by a significant (and diverse)
user population. - Because of the ad-funded model it is perfectly possible for the search
service to earn significant income (less, probably, than a closed service
would do but the first mover and open-source ‘brand’ advantages should be
sizable — look at Firefox). As a result running a complementary service
could be a substantial source of development funds. - From a societal standpoint the service is what you really care about.
While it is true that other people could run a web service with the same
code (exactly the point!) it would be nice to guarantee the existence of a
service, particularly one closed associated (name/reputation) with the
underlying software.
These are all important points from an implementation point of view but
they say little about why this proposal is good from a viewpoint of social
welfare. The answer though is obvious:
- We get de facto competition from a service which prices at ‘marginal cost’
(no rent extraction) - the algorithms, software and accumulated data (by no means the least
important of the 3!) are all open and therefore there are general
efficiency gains - As everything is open we have a guarantee of/commitment to future
competition - Extra competition from entry of other service providers using the same
codebase (NB: license would want to be of a GPL+/Affero type so that
service providers are obliged to contribute back code improvements) - Greater transparency (we know why some sites come top, why some don’t).
Some remarks on funding: while in the long-term this ‘open-search’ system
could well be self-supporting financially it seems likely that it would need a
decent injection of up-front funding to get it going. Where would this money
come from? Three possibilities: a) venture capital b) government or private
donor c) the general web community. (a) seems improbable given the
not-for-profit nature of the endeavour and the complete openness of both code
and service (might be possible to modify not-for-profit to allow pay-back to
backers but this is more like a straight loan and less like a VC setup). (c) is
possible but also seems difficult — one would be looking for significant
community support for development, testing, use and promotion but harder to see
it as a reliable source of funds, at least initially. That leaves us with (b)
up front funding from government or private charity. Given the large
technological spillovers, the clear social benefits in promoting competition
and transparency, and the ‘utility-like’ nature of search this would seem to be
a promising route to take.
The Current Search Model: Existing Problems
Even without the development of a monopoly (or oligopoly) the current search
situtation gives cause for concern, as Kamal Jain (Microsoft) detailed in his
presentation on the ‘ugly side’ of the ad-funded search model at the Toulouse
Roundtable (see below for more info on the Roundtable). In particular:
- Search engines are currently getting to extract all the rents from selling
my attention (equivalently: we are selling our attention, and click-stream,
for free). This situation does not appear to be sustainable. It is already
being addressed in some ways — Firefox, for example, is getting money from
Google for clicks — and there are plenty of other methods, for example,
one could start a not-for-profit web proxy that auctions queries to search
engines and then routes results to users passing money back to those users.
Storing and selling your clickstream would be another method (this has been
around a while but never really taken off — perhaps because privacy stuff
becomes too obvious and not really clear what benefits will be). - The funding model of search (in which search is funded by advertisers and
free for users) may be societally inefficient. Search engines are in a
monopolistic position vis-a-vis advertisers and therefore may a) overcharge
advertisting (with those charges eventually passed back to users) b)
over-provide search quality. - Little transparency over search engine rankings which are of
ever-increasing commercial and social importance.
Rufus Pollock is Founder and President of Open Knowledge.
There is a Government funded search engine being developed in Europe, presumably for the reasons you give above. It is called Quaero.
http://en.wikipedia.org/wiki/Quaero
Was French/German, but Germany has recently pulled out
http://www.iht.com/articles/2007/01/02/business/search.php
Thanks for the pointer. I must say though that neither the Quaero project nor the German one look that promising from the point of view of the proposal. Neither system looks like it will be open either as software or as service — the two aspects that which are central to the Open Search proposal set out above.