Science has always recognized the power of sharing in developing new knowledge. But in the search for treatments and cures for diseases like Alzheimer’s and Parkinson’s, the sprawling bodies of highly diverse research data are not easily shared. Either they are considered proprietary resources for making money, or they are hidden in academic databases that others may not know about, often inaccessible because of incompatible software formats. No single researcher really has the resources or incentive to develop an overarching regime to enable cooperation and sharing. And so dozens of academics, nonprofits and pharmaceutical companies have continued their research in relative isolation.

“Companies were caught in a prisoner’s dilemma,” a research at the University of Pennsylvania recently told the New York Times. “They all wanted to move the field forward, but no one wanted to take the risks of doing it.”

But ten years ago, Dr. Neil S. Buckholtz, who oversees dementia research at the National Institutes of Health, realized that the sharing of research data was a collective action problem that might be solved through concerted leadership. He helped instigate a plan by which the NIH stepped up to serve as an “honest broker” between the pharmaceutical industry and academics. The goal was to ensure that all research would be shared openly and freely, and published on the Internet immediately, so that anyone could use it. You could publish a research paper and you could develop new treatments, but no one would own the data. Researchers would even be free to make mistakes or misguided interpretations — because who is to say at the outset that something is necessarily incorrect?

Seven years ago, the NIH persuaded scientists from the FDA, the drug industries, medical-imaging companies, academia and nonprofit groups to cooperate in an ambitious scheme to affirmatively share their findings with each other. As the Times reports (August 13, 2010), the sharing of data is now starting to show results. Scientists studying Alzheimer’s disease routinely share their findings about “biological markers” that indicate the progression of the disease. This has led to recent scientific papers suggesting the value of PET scans and tests of spinal fluids as ways to make early diagnoses of Alzheimer’s.

The conventional business response to such radical ideas of “sharing” is that no company would have adequate incentive to invest in risky research unless they could be assured exclusive ownership of the results, in order to create a revenue-generating “product” (i.e., medical treatment or drug). But the drug industry has had to concede that diseases such as Alzheimer’s are just too scientifically complicated for any single research entity to tackle; the most fruitful way forward is to pursue an “open source” approach that places the basic building-blocks of knowledge into the commons – while sanctioning the private patenting of more refined medical innovations that build on the fruits of the commons.

It’s so common-sensical that it seems faintly ridiculous that a story of this sort should merit lead-story treatment in the New York Times.

Share This

Related posts:

1. The Medical Innovation Convention: A New Global Framework for Healthcare Research and Development
2. Articles in CTWatch Quarterly
3. BioMed Central Open Data Award: winner to be announced this week!

The first is called Open Data: why I need the Open Knowledge Foundation, and in it he introduces some of the issues he wishes to discuss and gives his vision for the role he hopes the OKF community will play in relation to open data. He writes:

After a period of silence on this blog (but not on the Open Knowledge Foundation lists) I hope to publish a flurry of ideas on Open Data. There is no doubt that “Open Data” has arrived and there is enormous interest. (By contrast when I started to investigate it 5 years ago there was nothing). It’s desperately important, more complex than I ever imagined, and it’s critical to address it immediately, responsibly, dispassionately and inclusively. If we manage to set out the concerns now, we may manage to avoid the worst problems that were encountered by the Open Source and later Open Access movements. [They have made enormous progress and without their footsteps Open Data would fall into many of the same pitfalls. But Open Data is Difficult – a phrase I shall repeat frequently.]

I am putting my faith and energy into the Open Knowledge Foundation – its people and its infrastructure. This is because it’s an organisation which is wideranging (it deals with open content of all sorts, open metadata, services, etc.). It has great expertise in legal problems and solutions (where these are necessary) and also how to find alternative approaches. It’s neutral (apart from urging Openness and developing the infrastructure). It’s very professional, and realises that ideas without implementation have less weight. So there is an impressive range of software and information skills. I am reminded of my favourite motto (from the IETF) – “rough consensus and running code”, one the greatest productive mantras of our time.

The enthusiasm is palpable. [Today I had a breakfast Skype session with Jonathan Gray (coordinator of OKF) and it's all about how we can make things happen fast and responsibly.] The OKF works through Working Groups and discussion lists, and so when I had a concern about Open Data I brought it to the OKF and – after a great deal of work – we emerged with the Panton Principles which have now been translated into several languages by OKF members.

Simply, the OKF amplifies the visions of individuals from the almost-impossible to the attainable.

So I am putting some ideas into the OKF melting pot to see what emerges.

In the next post, titled Open Data: The concept of Panton Papers, he lays out his ideas for the Panton Papers:

The current theme is “Panton Papers”. The idea is that part of the value of the Panton Principles is that the whole document is short and the key points are simply made. But the “Principles” can therefore only address the motivation and the procedures for Open data in a general manner, and many of the problems are in the details. I believe that many of the problems in Open Access (which is simpler than Open Data) arose because not enough communal effort was given to the practice of Open Access and I want to avoid as many OD problems as possible before they occur.

Over the last 2 years (when Open Data has started to become important and discussed) I have seen several potentially difficult areas. I’ll simply list the ones I have thought of here and then outline the idea of the Panton Papers. This discussion is mirrored in part by the OKF open-science discussion list and you may wish to subscribe. There’s also a regular working group on open-science. (Almost everything in OKF is Open, but it may take a little while to find out where you want to be!). The issues that I currently have are:

• What is data? Images? Graphs? Tables? Equations? Accounts of experiments? This is a major problem and almost completely unexplored. Without solving this we are held back 10 year or more in our ability to re-use the primary scientific literature (e.g. by closed-access publishers who claim that factual graphs belong to them).
• Why should data be open? (and when should it not be?). I’ve put forward ideas here and here . They range from moral, to legal/quasi-legal to utilitarian.
• Who owns data? This is one of the trickiest areas – there is legal and contractual ownership and there is moral ownership. Generally there is far far too much “ownership” of data.
• When should data be released? This is a key question (see here for an example). Some communities have solved it – most haven’t addressed it and will have to go through the rigour of working out release protocols.
• How and where should data be exposed? I am strongly of the opinion that we need domain-specific repositories (which could be national or international) and the Institutional Repositories are almost never the best place to expose data (I expect and welcome alternative opinions). The “how” depends on understanding what the data and metadata are and is increasingly dependent on specialist software and information standards. “Archival” is often the wrong word to use.
• Datamining and textmining. Most authors, publishers, repository owners are unaware of the enormous power of automated analysis of the literature. Some closed access publishers expressly forbid these activities. We have to liberate the right of the scientific community to do this enthusiastically and efficiently.
• Reproducibility. Science is based on reproducibility – we expect to be able to replicate the “materials and methods” of an experiment and to try to falsify its claims. Physical materials are beyond the immediate discussion (though this may change) but much science is now based on computing. It should be possible to replicate simulations, data cleaning, data analysis, model fitting etc. This is a tricky area. It is difficult (though with virtualization and the cloud is becoming easier) to reproduce the computing environment. Large or complex data sets are a major problem but must be addressed. This is not without monetary cost.

I may add more.

The idea is that each of these is a “Panton Paper”. It may or may not be crafted in Pantonia (the hectare of the Chemistry Department, The OKF headquarters, and the Panton Arms in Cambridge UK). Everything I now write is mutable.

Each paper will have a top level document of similar form to the Panton Principles, i.e. 3-8 ideas, with short explanatory paragraph(s). This document will be crafted by the OKF in public view on a wiki or Ether/Piratepad. Anyone can take part. We shall welcome contributions from a wide range of disciplines (in fact this is essential). At some stage version 1.0 of the paper will be frozen and will be formally published. We have an offer from a major publisher to do this and I am hoping we can announce this at Open Science Summit.

The Paper should carry a wider range of links to other essays in Open Data and should carry examples from different disciplines. For example there is a well tried and accepted process in many areas of bioscience and astronomy as to what when and how data get published.

Peter has started drafting ideas for the first two of these at:

• What is Scientific Data?
• Why Scientific Data should be Open
• Who owns Scientific Data?

If you’d like to get stuck in, please head on over to the open-science list and say hi!

Share This

Related posts:

1. Launch of the Panton Principles for Open Data in Science and ‘Is It Open Data?’ Web Service
2. Panton Principle authors win SPARC Innovator prize!
3. Comments on the Panton Principles and Data Licensing

Since the 19th century, the study of archaeobotanical remains has been very important for combining “strict” archaeological knowledge with environmental data. Pollen data enable assessing the introduction of certain domesticated species of plants, or the presence of other species that grow typically where humans dwell. Not all pollen data come from archaeological fieldwork, and pollen analysis is often done by ecologists without a particular focus on human-associated plants. However, from an archaeologist’s perspective the relationship among the two sets is strong enough to take an interested look at pollen data worldwide, their availability and most importantly their openness, for which we follow the Open Knowledge Definition.

We found that there is a serious misunderstanding by universities and research centers of their role in society as places of research, innovation that is available for everyone. As for dendrochronological data, academia is a closed system producing data (at very high costs for society) that are only available inside its walls, but it’s all done with public money.

Finding pollen data

The starting point for finding pollen data is the NOAA website.

The Global Pollen Database hosted by the NOAA is a good starting point, but apparently its coverage is quite limited outside the US. Furthermore, data from 2005 onwards aren’t available via FTP in simple documented formats, but are instead downloadable as Access databases from another external website. Defining Access databases as a Bad Choice™ for data exchange is perhaps an euphemism.

Unfortunately, a large number of databases covering single continents or smaller regions is growing, and the approaches to data dissemination show marked differences.

Americas

For both North and South America, you can get data from more than one thousand sites directly via FTP. There are no explicit terms of use. Usually, data retrieved from federal agencies are public domain data.

The README document only states NOTE: PLEASE CITE ORIGINAL REFERENCES WHEN USING THIS DATA!!!!!. Fair enough, the requirement for attribution is certainly compatible with the Open Knowledge Definition.

• North American Pollen Database
• Latin American Pollen Database
• Global Pollen Database

Europe

From the GPD website we can easily reach the European Pollen Database, that is found at another website tough (and things can be even more confusing, provided that the NOAA website has some dead links).

You can download EPD data in PostgreSQL dump format (one file for each table, with a separate SQL script create_epd_db.sql). Data in the EPD can be restricted or unrestricted. That’s fine, let’s see how many unrestricted datasets there are. Following the database documentation, the P_ENTITY table contains the use status of each dataset:

steko@gibreel:~/epd-postgres-distribution-20100531$ cat p_entity.dump |
awk -F "\t" {' print $5 '} | sort | uniq -c
    154 R
   1092 U

which is pretty good because almost 88% of them are unrestricted (NB I write most of my programs in Python but I love one liners that involve awk, sort and uniq). We could easily create an “unrestricted” subset and make it available for easy download to all those who don’t want to mess up with restricted data.

But what do “unrestricted” mean for EPD data? Let’s take a more careful look (emphasis mine):

1. Data will be classified as restricted or unrestricted. All data will be available in the EPD, although restricted data can be used only as provided below.
2. Unrestricted data are available for all uses, and are included in the EPD on various electronic sites.
3. Restricted data may be used only by permission of the data originator. Appropriate and ethical use of restricted data is the responsibility of the data user.
4. Restrictions on data will expire three years after they are submitted to the EPD. Just prior to the time of expiration, the data originator will be contacted by the EPD database manager with a reminder of the pending change. The originator may extend restricted status for further periods of three years by so informing the EPD each time a three-year period expires.

Sounds quite good, doesn’t it? “for all uses” is reassuring and the short time limit is a good trade off. The horror comes a few paragraphs below with the following scary details:

1. The data are available only to non-profit-making organizations and for research.

Profit-making organizations may use the data, even for legitimate uses, only with the written consent of the EPD Board, who will determine or negotiate the payment of any fee required.

Here the false assumption that only academia is entitled to perform research is taken for granted. And there are even more rules about the “normal ethics”: basically if you use EPD data in a publication the original data author should be listed among the authors of the work. I always thought citation and attribution were invented just for that exact purpose, but it looks like they have distinctly different approach to attribution. The EPD is even deciding what are “legitimate” uses of pollen data (I can hardly think of any possible unlegitimate use).

• European Pollen Database

Africa

For “Africa” read “Europe” again, because most research projects are from French and English universities. For this reason, the situation is almost the same. What is even worst is that in developing countries there are far less people or organizations that can afford buying those data, notwithstanding the fact that in regions under rapid development the study and preservation of environmental resources are of major importance.

Data are downloadable for individual sites using a search engine, in Tilia format (not ASCII unfortunately). The problems come out with the license:

The wording is almost exactly the same as for the EPD seen above:

Normal ethics pertaining to co-authorship of publications applies. The contributor should be invited to be a co-author if a user makes significant use of a single contributor’s site, or if a single contributor’s data comprise a substantial portion of a larger data set analysed, or if a contributor makes a significant contribution to the analysis of the data or to the interpretation of the results. The data will be available only to non-profit-making organisations and for research. Profit-making organisations may use the data for legitimate purposes, only with the written consent of the majority of the members of the Advisory board, who will determine or negotiate the payment of any fee required. Such payment will be credited to the APD.

• African Pollen Database at NOAA
• Banque Africaine de Données Polliniques/African Pollen Database

Conclusions

The only positive bit of the story, if any, is that these datasets are nevertheless available on the web, and their terms of use are clearly stated, no matter how restrictive. It would be just impossible to write a similar article about archaeological pottery, or zooarchaeological finds.

Appendix: Using pollen data

Pollen data are usually presented in forms of synthetic charts where both stratigraphic data and quantitative pollen data are easily readable. Each “column” of the chart stands for a species or genus. You can create this kind of visualization with free software tools.

The stratigraph package for R can be used for

plotting and analyzing paleontological and geological data distributed through through time in stratigraphic cores or sections. Includes some miscellaneous functions for handling other kinds of palaeontological and paleoecological data.

See the chart for an example of how they look like.

Share This

Related posts:

1. World Bank Opens Up Development Data
2. Biblios - “world’s largest database” of open bibliographic data goes beta!
3. Open Data in Archaeology

The principles are currently maintained by the Open Knowledge Foundation’s Working Group on Open Data in Science.

From the announcement:

Science is based on building on, reusing, and openly criticizing the published body of scientific knowledge. For science to effectively function, and for society to reap the full benefits from scientific endeavors, it is crucial that science data be made open.

That’s the belief of four leaders who have put forth a groundbreaking set of recommendations for scientists to more easily share their data – The Panton Principles – and who have been named the latest SPARC Innovators for their work. [...]

The authors advocate making data freely available on the Internet for anyone to download, copy, analyze, reprocess, pass to software or use for any purpose without financial, legal or technical barriers. Through the Principles, the group aimed to develop clear language that explicitly defines how a scientist’s rights to his own data could be structured so others can freely reuse or build on it. The goal was to craft language simple enough that a scientist could easily follow it, and then focus on doing science rather than law.

The Panton Principles were publicly launched in February of 2010, with a Web site at www.pantonprinciples.org to spread the word and an invitation to endorse. About 100 individuals and organizations have endorsed the Principles so far.

“This is the first time we’re seeing diverse viewpoints crystallize around the pragmatic idea that we have to start somewhere, agree on the basics, and set the tone,” says Heather Joseph, Executive Director of SPARC (the Scholarly Publishing and Academic Resources Coalition). “The authors are all leading thinkers in this area – as well as producers and consumers of data. They each approached the idea of open data from different directions, yet with the same drive to open up science, and ended up on common ground.”

According to Pollock, “It’s commonplace that we advance by building on the work of colleagues and predecessors – standing on the shoulders of giants. In a digital age, to build on the work of others we need something very concrete: access to the data of others and the freedom to use and reuse it. That’s what the Panton Principles are about.”

Further details, including more background information and comments from other leading voices in the open science community, are available at:

• http://www.arl.org/sparc/innovator/panton.shtml

Alma Swan comments, quite rightly, that:

Coming up with the Principles is not going to cut the mustard by itself. They will need to be advocated and promoted so that scientists are interested in debating them.

If you’re interested in helping to promote the principles to scientists in different domains, to research funding bodies, or to the general public, please introduce yourself on our open-science mailing list.

Share This

Related posts:

1. Launch of the Panton Principles for Open Data in Science and ‘Is It Open Data?’ Web Service
2. Introducing the Panton Papers
3. SPARC Europe Seal for Open Access Journals

Dear Champions of Open Science,

Please join us in gathering stakeholders seeking to liberate our scientific and technological commons to enable a new era of decentralized, distributed innovation.

• http://opensciencesummit.com/

While there are many great organizations and talented people thinking about these problems, the discussion is usually spread out across email lists, blogs, friend feed, etc. Experts in one domain (Open Data, Open Access publishing) are not always engaged with those working on policy issues in another (patent reform, funding mechanisms to incentivize openness). To remedy this, I ambitiously (perhaps too ambitiously) decided to attempt to bring everyone together in an event that I hope will help build cohesion between the many groups and individuals striving to transform science.

Over two and a half days, from July 29-31st at Berkeley, we will consider the major challenges and policy questions that must be addressed to successfully update the institutions, practices, and worldviews that comprise our global scientific governance system. Topics include personal genomics, gene patents, open data, the future of scientific publication, peer review, and reputation, citizen science in biology, and many more. We ask you to join us in imagining a new paradigm, one in which a virtuous circle of mutually reinforcing shifts toward transparency and collaboration could unleash hitherto untapped reserves of human ingenuity. It will take tremendous coordinated effort to achieve this vision.

This event is for everyone who cares deeply about making science work more effectively to benefit all humanity. There is a distinct focus on biology and life sciences at the conference because these fields are experiencing rapid change and are essential to our health, environment, and food security. In future years we will try to include greater coverage from other disciplines, for example the exciting “collective science” data sharing efforts emerging in astronomy, math, and physics (arxiv.org being a leading example of open science principles in action).

We ask you to help us in making this first conference a success so that we can build to the next one.

Thank you,

Joseph and OSS volunteers.

Share This

Related posts:

1. Interview with Science Commons for their Voices from the Future of Science
2. Hacks and Hackers, Birmingham, 23rd July 2010
3. Launch of the Panton Principles for Open Data in Science and ‘Is It Open Data?’ Web Service

Having already been scooped by some encouraging and eloquent activity in the open data blogosphere here, here, here and here (perhaps there are more), a few final comments and future thoughts on the 2009 BioMed Central Open Data award.

Our internal scientific, and external academic, editors were vital in the candidate selection process this year, but next year we hope to involve more readers. Any article published in a BioMed Central journal (including those on physics and chemistry portals) in 2010 can be considered for the next award so we encourage all our readers – and editors – to contact us if they read or, even better, re-use any useful data published in 2010.

Thanks again to the judging panel – Lee Dirks, John Wilbanks, Peter Murray-Rust, Rufus Pollock and Cameron Neylon – and to Microsoft Research for supporting the award. Without their contributions the award simply would not have been possible.

As Cameron Neylon acknowledges in his blog about the awards licensing of open data is an important issue, and heterogeneity in approaches to data licensing was identified in the examples of original data published in BioMed Central’s journals identified during the selection process. We’re keen to develop a consensus and code of practice on licensing and re-use that best serves the needs of authors, editors and the open data community, in the many different fields incorporated in BioMed Central’s journals.

Another important issue we hope will be addressed by future nominees is standardization of data. Not only do we want articles that include raw data, but we are also looking for data in fully harvestable, machine readable and re-usable formats where possible, and appropriate. Over the coming weeks and months BMC Research Notes will be ratcheting up its involvement in helping to define domain-specific data file standards, including a publication of guidance and best practice. Any potential contributors are, again, encouraged to get in touch.

Share This

Related posts:

1. BioMed Central Open Data Award: winner to be announced this week!
2. Thoughts from the GLA’s Possibilities of Real Time Data conference
3. Response to ‘The Future of Bibliographic Control’ draft from the Library of Congress

All followers of this blog will know that science publishing now goes much beyond the traditional, IMRaD, journal article. As collective consciences are raised about data sharing it becomes an increasingly important part of the research and publication process, but great heterogeneity in the availability of data underlying peer-reviewed articles remains – arguably even more so in the availability of truly open data.

So in recognition of the challenges overcome by scientists who have shown leadership in openly sharing scientific data, particularly when sharing goes against cultural or community norms, the open access publisher BioMed Central last month launched its Open Data Award.

BioMed Central teamed up with Microsoft Research and Panton Principles to introduce the award, which will be presented to the winning author – of an article published by BioMed Central in 2009 – at BioMed Central’s 4th Annual Research Awards on Thursday 10th June. The support of Microsoft Research and Panton Principles for data-intensive science, and BioMed Central’s commitment to data via journals such as BMC Research Notes is a logical, and exciting, collaboration.

The focus of the shortlisted articles, which were announced last week, is primary research articles that include potentially re-usable data as either additional files with the journal article or via a link to publicly available data hosted elsewhere. As well as many examples from fields that have collectively begun to recognize the benefits of openness, such as genomics, nominations were received across the full spectrum of BioMed Central’s 200+ journals. A number of articles presenting tools, policies or software to encourage data sharing were also published by BioMed Central in 2009, and this is reflected in the make-up of the shortlist.

A further update will follow after the winner is announced, at London’s Gherkin landmark, later this week.

Share This

Related posts:

1. Some final - and future - thoughts on the BioMed Central Open Data award
2. ‘Voyages into publisher copyright’
3. SPARC Europe Seal for Open Access Journals

Take a tour of some of the additions we’ve made to the climate data group at OKF’s CKAN.

The Mauna Loa observatory, Hawaii, has the longest period of continual recording of the amount of CO₂ (carbon dioxide) in the air, the airborne fraction. The data are available in CKAN, and here they are in chart form:

CO₂ is a relatively well mixed gas in the atmosphere, but even so, it would be unwise to rely on a single location for measurements. The Carbon Dioxide Analysis Center maintain a global network of stations collecting CO₂. In fact Mauna Loa is not used for the global average because its height gives it a CO₂ fraction that is lower than the surface average by 1 to 2 ppm.

What about reconstructing historical CO₂ levels? One source is ice cores. On the Antarctic ice sheet snow falls every year and never melts. New snow falls on the snow from the previous year, building up in layers. Eventually the snow builds up to a thickness where it compresses the snow beneath it into solid ice, ice that is impermeable to gas. At that point, air at the surface becomes trapped in little bubbles in the ice.

By drilling down through the ice we can reach older and older ice. Vostok Station sits on the Antarctic ice sheet, above Lake Vostok. Researchers have drilled down through the ice to a depth of 3623m and reached ice that is about 400,000 years old. Drilling was stopped, tantalisingly close to the lake surface, because the Scientific Committee on Antarctic Research (SCAR) raised concerns that life in Lake Vostok, potentially forming a unique biome, may be contaminated.

By measuring the CO₂ content of the gas trapped in the ice core, we can reconstruct the historical levels. Of course, the data are in CKAN. Here’s a chart:

(Other data from the Vostok ice core are also available)

Vostok is well known for being the coldest place on Earth. Vostok Station was established in 1957, and since that time weather records have been kept by researchers working there. The temperature record for Vostok Station is just one of the many thousands of records made available in the Global Historical Climate Network (GHCN). Here’s Vostok’s temperature record for the last three decades (more data are available, but three decades fits nicely):

The different colours are because for a particular station the whole series can be comprised of individual records that cover only part of the range (due to different equipment, different reporting procedures, and so on); each record gets a different colour (unfortunately the records often overlap, confusing the colours).

The station records in GHCN, sometimes augmented by other similar datasets such as SCAR’s READER, are used to reconstruct global temperature anomalies, like the Japan Meteorological Agency Global Surface Temperature Anomaly, HadCRUT3 from the UK’s Met Office and the University of East Anglia’s Climate Research Unit, and, perhaps most famously, GISTEMP from NASA:

The seasonal cycle is evident in the Mauna Loa CO₂ (it’s caused by photosynthesis of plants, mostly in the Northern Hemisphere, drawing down more CO₂ from the atmosphere), and also in the Vostok temperature record. In some cases the seasonal signal and the long term trend are easily visible, in others in takes effort to recover the long term trend. The GISTEMP graph can be thought of as recovering the long term trend from many thousands of individual station records.

Another well known climate data series with both seasonal and long term trends is the National Snow and Ice Data Centre’s Arctic Sea Ice Extent:

The seasonal cycle in the Arctic sea ice is of course due to summer melt and winter freeze.

Another data set available as a CKAN package is the Colorado Sea Level data. This is a measurement of global mean sea level obtained by a series of satellites: TOPEX, then JASON-1 and JASON-2. The next satellite in this series is JASON-3, and it has just secured funding from a European consortium. There is a seasonal cycle in this data too:

Now I think the seasonal cycle in mean sea level is due to the thermal expansion of the oceans. In the summer the ocean warms and expands; both the northern hemisphere and the southern hemisphere are affected but the effects don’t balance so there is a seasonal cycle. Please contact your tour guide (leave a comment!) if you can find a reliable explanation (I looked and was unable to find a good source).

The satellite era has been tremendously useful for earth observation and climate science, but of course the records from satellites are short. For example, the satellite data for sea level only goes back to 1993. Since climate is often a matter of looking at events on long timescales we often have to find longer series from other measurements.

The UK’s Natural Environment Research Council maintains the Permanent Service for Mean Sea Level at the Proudman Oceanographic Laboratory. Using a global network of about 2000 tide gauges they can reconstruct a global mean sea level record that documents sea level rise since 1880. Here’s a chart of the data available from the CKAN package:

The tour is coming to an end now. The data that I’ve shown here are just a selection of the data available, both generally and in CKAN. Often there is much more detailed data (and more detailed science) behind each of these datasets, but one of the reasons I’ve selected many of these datasets is that they are key indicators. They are the headline figures that show increased CO₂ emissions, rising sea levels, decreasing Arctic sea ice. These are the data that a curious member of the public will want to engage with, and that reason makes it important that the data are accessible and freely accessible.

If you’d like to contribute to the climate data group then please drop us an e-mail. If you’d like to continue the tour on your own you might want to try the Red Sea Sea Level records and the Paleo Tree Ring records which are just around the corner in the Open Archeology wing.

If you’re interested in promoting open data in climate science, you may wish to endorse the Panton Principles, which were launched last week.

References and Credits

• The Mauna Loa data comes from NOAA/ESRL.
• Vostok Ice Core data comes from NOAA paleoclimatology.
• Vostok Station temperature data comes from NOAA’s GHCN data.
• The GISTEMP global temperature anomaly comes from NASA GISS.
• Arctic Sea Ice data comes from the National Snow and Ice Data Center.
• Satellite Sea Level data comes from the University of Colorado at Boulder.
• Tide Gauge Sea Level data comes from the Proudman Oceanographic Laboratory.
• Charts were plotted using Google Chart Tools and programs written in Python; the programs for Vostok Station and the GISTEMP anomaly were borrowed from Clear Climate Code.

Share This

Related posts:

1. Clearer Climate Code
2. Clear Climate Code, and Data
3. Climate Change, Climate Sceptics and Open Data

• http://www.pantonprinciples.org/

From the preamble:

Science is based on building on, reusing and openly criticising the published body of scientific knowledge. For science to effectively function, and for society to reap the full benefits from scientific endeavours, it is crucial that science data be made open.

By open data in science we mean that it is freely available on the public internet permitting any user to download, copy, analyse, re-process, pass them to software or use them for any other purpose without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. To this end data related to published science should be explicitly placed in the public domain.

The principles themselves state that legal status of scientific datasets should be made explicit and that content licenses are not appropriate for data. They strongly discourage non-commercial licenses and strongly encourage dedication to the public domain. You can endorse the principles at:

• http://pantonprinciples.org/endorse/

We’d greatly appreciate any help in disseminating the principles - including blogging, microblogging and forwarding to relevant people!

The first draft of Panton Principles was written in July 2009 by Peter Murray-Rust, Cameron Neylon, Rufus Pollock and John Wilbanks at the Panton Arms on Panton Street in Cambridge, UK, just down from the Chemistry Faculty where Peter works. They were then refined with the help of the members of the Open Knowledge Foundation Working Group on Open Data in Science.

The Is It Open Data? web service is also now live. This allows anyone to make and publicly record an enquiry into the openness of a given (scientific) dataset. It is also an initiative of the Working Group on Open Data in Science.

• http://www.isitopendata.org/

Share This

Related posts:

1. Panton Principle authors win SPARC Innovator prize!
2. Comments on the Panton Principles and Data Licensing
3. Virtual meeting for Working Group on Open Data in Science

Clear Climate Code have been working on ccc-gistemp, a project to reimplement in clear Python NASA’s GISTEMP. GISTEMP is a global historical temperature analysis, it produces, amongst other things, graphs like this, that tell you whether the Earth is getting warmer or cooler:

Because this graph is important for studying the world’s climate (and determining the signature of global warming), there is a lot of public discussion about where this data comes from. The raw data underlying the graph is surface weather station temperature records. The raw data is processed to produce the data for the graph:

The box in the middle, labelled “GISTEMP”, is a process that converts the raw station records into the data for the graph on the right, which is the global temperature anomaly. There are descriptions of this process available, for example Hansen and Lebedeff, 1987. A description is one thing, but it might not tell you everything you need to know. Perhaps the description is sufficiently clear and accurate for you to reproduce the process, perhaps not. The ultimate authority on the process is the source code that implements it, because It’s the source code that is executed in order to produce the processed data. So if you want to know exactly what the process involves, you have to get hold of the source code.

In effect it is the source code that adds value to the raw data to produce processed data. So in a sense, the value of the processed data is embodied in the source code. That’s what makes the source code important.

The source code for GISTEMP is written mostly in Fortran by scientists at NASA, and is available from them. This source code is the working code used by the NASA scientists, it is not necessarily the best source code for explaining how the process works (to an interested and competent member of the general public). There is the question of whether NASA, a publicly funded body, should be paying someone to write code that makes a better tool for communicating with the public (for example by writing better documentation, or writing it in a more exemplary style). I am not going to address that question. The source code NASA use is the source code we have right now.

Our goal at Clear Climate Code is to take this code and produce a new version that is clearer, but does the same thing. We have taken great steps forward towards this goal: We have recently released a version which is all in Python and which reproduces NASA’s results exactly. We think much of this code is already a great deal clearer than the starting material, but we continue to make it clearer. Of course we would welcome your support. If you want to help, please join our mailing list, or you can follow our progress at our blog and on twitter.

The reasons Clear Climate Code chose Python as the implementation language for ccc-gistemp are: accessibility, clarity, and familiarity. By accessible I mean that there is a large community of Python programmers, but also there are several tutorials and other materials for learning Python should you be motivated. Python is used to teach undergraduates programming. Python is relatively clear; it’s deliberately designed to be free of the clutter that imperils other programming languages. It’s certainly possible for people who are not professional programmers to create small programs in Python, and examine and modify existing Python programs. And lastly, it’s familiar; Nick Barnes and I already knew Python when we started the project. This seems like a trivial consideration, but in fact Clear Climate Code is an unpaid project and it’s pretty easy to come up with reasons to do something else instead, so the fact that we already knew Python was important.

Hopefully Clear Climate Code illustrates how both code and data are central to the public understanding of science. For an issue like global warming it is absolutely crucial that public are involved. CKAN’s climate data group is a place where non-specialists can access scientist’s data more easily, and hopefully use it to innovate, do their own hobby science, or create visualisations to better communicate with the public. I’m hoping to add more data sources to the climate data group in the near future, if you’re interested in adding more data to this group, please get in touch.

Share This

Related posts:

1. Clearer Climate Code
2. A tour of climate data at CKAN
3. Climate Change, Climate Sceptics and Open Data