Irina Bolychevsky: ‘CKAN has spread because it makes maintaining data more cost-effective and stimulates the economy through data reuse’

Lucas Pretti: You shared these pictures with me. What’s happening here?

Irina Bolychevsky: The first is October 2013. People were in London for the Open Government Partnership Summit and an Open Knowledge Foundation side event, and we met for breakfast before co-working.

Everyone in the picture had contributed to CKAN in some way. David Read and David Raznick were on the core team with me. James Gardner led the team when I joined. James McKinney from Open North used CKAN, and Friedrich Lindenberg, who was working on School of Data and OpenSpending, had to interface with it.

What was magical about this time was that open data was a small, intimate community. It was a movement, and most people knew each other. People were working in this space because they believed transparency could drive accountability and change. So there was a huge amount of enthusiasm, energy and mutual respect, so we liked to hang out!

The second photo is just the team out for a drink.

Lucas Pretti: CKAN turns 20 this year, which takes us back to 2006, well before this picture. But CKAN really started to scale around this period, right? I want to explore that collective process of building what became a standard.

Irina Bolychevsky: A lot of what Open Knowledge did was provide a gathering space for different movements and people. It was an umbrella. It was brilliant at creating networks and places where people working on different things could come together and make things happen.

Irina Bolychevsky: When Rufus Pollock first had the idea for CKAN, it was probably one of a hundred projects. He wanted a registry for open data, which became the DataHub. It was essentially the free-software mentality applied to data: the idea that, by being open, things could be better maintained through collaboration. That’s why CKAN was named after CPAN, the Comprehensive Perl Archive Network, with Perl changed to Knowledge. Not the catchiest of names, to be sure, and we rarely used the full name.

The DataHub was the first experiment. Communities such as the Linked Open Data cloud used it as free hosting, and CKAN later powered the registry for IATI, the International Aid Transparency Initiative. But it was hard to use, and had no sustainability model. Then government changed things.

Lucas Pretti: So this is where CKAN starts becoming something more than a small project?

Irina Bolychevsky: Yes. Rufus convinced the team building data.gov.uk to use CKAN for the back end, because it had a really good API. The site launched publicly in January 2010 with CKAN behind it. The contract brought in some money, and OKFN started hiring.

David Read was one of the first hires. Then Rufus hired James Gardner, who had co-created Pylons, the web framework CKAN was built on. James took it seriously and started operationalising it: making it more robust and sorting out the API. He became the CKAN team lead and began building a team.

Lucas Pretti: And then you came in as the first product owner?

Irina Bolychevsky: Yes, in June 2011. Rufus had got my CV from a mutual acquaintance. I had just left a startup in London and was looking for something more meaningful. He called me out of the blue, and ten minutes later I was on a video call with James and Rufus, and I met James in person the next day. Things moved quickly, and I was invited to meet the team and broader community at OKCon, the Open Knowledge Conference, in Berlin. 

When I joined, there was a team, and there was money from the data.gov.uk contract. But there wasn’t much of a website, or a clear vision for what CKAN should be. The first description I got was “a package manager with a Solr index” and a Postgres database. I thought: brilliant, but who is it for, and what does it actually do for people?

Lucas Pretti: What did you see in it that wasn’t being said?

Irina Bolychevsky: Everyone was getting more interested in data. But how can you use data if you can’t find it, or know its context? Data is always about something. It’s a lens on reality, never a full picture of it.

And data is more useful when you can link it to other data. Knowing how a school performs is useful. Knowing it alongside how much the area receives for civic infrastructure, education, health and mental health is far more useful: you might see that when investment goes up, so does school performance. When you can find relevant data, understand its limits, and link it to other data, you can make inferences that inform decisions, in an organisation, in government or in society.

That needs very good metadata, and search, filtering and faceting. CKAN could do all of that. But it looked and was described in a very developer-oriented way, when the value was really at a policy level: for students, academics, think tanks, organisations and government itself. So one of my first priorities was making CKAN much more approachable for those audiences.

Lucas Pretti: You were the first non-developer on a team of engineers. How did you earn their trust?

Irina Bolychevsky: I’d worked with developers before, in QA, testing and product, and I was already interested in ‘servant leadership’, having read a brilliant essay by Aaron Swartz on it. So I prioritised providing clarity, context, priorities, and specifications that are clear and achievable while leaving room for problem-solving and creativity. Perhaps that also came from studying mathematics and philosophy and having lots of developer friends.

It’s a balance. You give enough structure: this is what we’re trying to do, and this is roughly how we get there. Then you trust people to do it, and you recognise and value what they’ve done.

Irina Bolychevsky: However, there was a lot of developer speak, and I knew that needed some work. Who’s the audience? Who’s going to buy it? So I narrowed CKAN’s scope and simplified the message. We started with “CKAN is for governments doing open data”, because if you’re doing open data, you should be doing it with open source.

Lucas Pretti: Narrowing to governments meant saying no to other things. What did you give up?

Irina Bolychevsky: We gave up prioritising communities. No one really pushed back, because everyone knew CKAN needed to be sustainable. The rest of the organisation ran on grants, so it was valuable to have one part that could support itself, and potentially others, through commercial work. Once the idea was articulated, people agreed.

It did mean we weren’t actively nurturing grassroots, volunteer and civic groups. Happily, they still used CKAN and benefited from it. One of the things I’m proudest of is how many organisations and communities used it, because it was free and open source, to collect data for their own causes, for advocacy or for transparency. Our investment in the product meant they had a good tool to use.

Lucas Pretti: What was happening in the wider public discourse at the time? Today the main issues are extractivism, surveillance, Big Tech and so on. What were people talking about then?

Irina Bolychevsky: The discourse was actively in flux at the time. I remember arriving at OKCon in Berlin, where everyone was talking about openness: open data, open science, open knowledge, open education. Government events were very different, at least at first. Governments had little in-house technical capacity, and a lot of work was outsourced to big corporations. There was confusion and scepticism about open data, and especially about open source, which was mistrusted and assumed to be less secure. People would say, “Sure, transparency is nice.” But the argument that really worked was economic growth.

Every government department produces enormous amounts of data as part of its normal work. That data could be the foundation for businesses, from finding your nearest bench or public toilet to helping people choose a school. If you did a little more work to make the data usable beyond its original purpose, you could unlock enormous value.

That was when Neelie Kroes at the European Commission was saying “data is the new gold”, in the era of “data is the new oil”.

Lucas Pretti: Which you clearly don’t like.

Irina Bolychevsky: I’ve never liked the phrase, because data is nothing like oil. But it was very much in people’s minds. It was just before the big-data rush, and governments were asking: what can we do with data? How valuable is it?

In civil society, we wanted open data for accountability. But we knew that, to sell it, we also had to talk about economic value.

Lucas Pretti: And that connects to your decision to make CKAN specifically for governments.

Irina Bolychevsky: Exactly. Governments were trying to become more digital. So we could say: you have an open-data requirement, in the UK because the Transparency Board recommended it. But you can also show that you’re driving innovation and economic value by publishing all kinds of data, not just spending data but the statistics you already produce.

To make that data accessible, you need a catalogue with search, filters and facets. And it would go against the values of open data to lock it in a proprietary, closed-source system. You need an open-source platform that can grow over time and that you can manage yourselves. That was my pitch.

Lucas Pretti: You’ve described a third CKAN user, alongside publishers and re-users: the person who signs off the budget. What did you tell them?

Irina Bolychevsky: Most governments I spoke to already had a legal or policy requirement to publish some open data. My pitch was: with CKAN, publishing and maintaining that data becomes much easier and more cost-effective, and the data is far more likely to be used, which fuels innovation and the economy. It also looks better, because people can find the data and understand its context, instead of it sitting on a static website where it’s hard to find and goes nowhere.

Irina Bolychevsky: I invested significant time in ckan.org, explaining what we could offer and what CKAN’s features did, so that our small remote team and open-source project looked much more professional. Then we convinced data.gov.uk to renew and expand its contract, including using CKAN on the front end, and the site relaunched in June 2012. At the same time, we bid for the European Commission’s open data portal as part of a consortium led by TenForce, and won. With those contracts coming in, I felt we could really invest.

Lucas Pretti: That investment became a complete redesign, released as CKAN 2.0 in 2013. What did it involve?

Irina Bolychevsky: It was a small team: me as product lead, a designer (Sam Smith), a front-end developer (John Martin) and a back-end developer (Toby Dacre), over two or three months. We rethought the information architecture, made the interface consistent and built a demo site. We also re-templated all of CKAN, so that clients, communities and anyone using the open-source code could customise it far more easily.

I was surprised by how much a nicer interface and a demo site changed people’s relationship to CKAN. Products that are usable and pleasant to use are simply more successful. And it still amuses me that the same CKAN 2.0 design is in use more than a decade later. It was an investment I fought for, and I was very proud of it.

Lucas Pretti: At that point you were effectively the team lead and a decision-maker inside CKAN?

Irina Bolychevsky: I was still the product owner, so I made the product decisions, but I was also hiring and managing the team, doing all of our business development, and a lot of community outreach in person at conferences.

Irina Bolychevsky: Then the US came into the picture. It was politically tricky, but it was hugely influential for CKAN, and it became our main source of funding.

The US had committed to open source and was planning a new data portal. Jonathan Gray put us in touch with the person leading it, and I offered calls to help them think it through, drawing on our experience. Our strategy was to ask the questions that would show them they needed us.

Lucas Pretti: So you weren’t really pitching or selling. You were making yourselves useful.

Irina Bolychevsky: Exactly. We’d ask, “How will you handle this volume of requests? What are you going to use? Have you thought about this?” And we’d explain how we’d solved those problems. James Gardner, who hadn’t yet left, was instrumental in this.

That led to a contract to build a demo and, eventually, a contract for data.gov through a US contractor. It was a big deal, because they had a complicated migration ahead of them. I had weekly or fortnightly calls with around 30 stakeholders, from senior directors to representatives of national organisations, each with their own views and concerns. Politically, it was very demanding.

Lucas Pretti: How do you get a project like that over the line?

Irina Bolychevsky: Data.gov is the front door to a huge number of datasets, and many organisations care about how it works. So the project was about listening, building trust and delivering something that worked well. CKAN was so much more capable than what they had before that it made the case for itself. But it took a lot of work. David Raznick and Adrià Mercader worked incredibly hard to deliver everything data.gov specified, on time and on budget, and we did.

Lucas Pretti: Who were you up against?

Irina Bolychevsky: In general, our main competitor was Socrata, a US company with a lot of venture capital behind it. They would give governments a hosted portal free for the first year, which we couldn’t afford to do. But they owned the infrastructure, and once a government was invested, prices often went up. I believed that open source, which governments could own, host, adapt and change, was the better choice, especially when spending taxpayers’ money.

Lucas Pretti: And that was an opportunity to fund the roadmap, rather than just build what the US wanted.

Irina Bolychevsky: Our model was that each government contract would fund new product development that everyone using CKAN, especially other governments, could benefit from.

I was keen to build a community of governments using CKAN together, talking openly about what they were funding and what they’d gain. The US work, for example, gave CKAN geospatial search and map previews, built as general extensions that everyone could use. We also built the permissions system: organisations, with admin, editor and member roles, and the architecture for managing who could publish and review data.

That was the beginning of what became the CKAN Association in 2014. It was government-led at first, and eventually lots of suppliers joined.

Lucas Pretti: And this is the point where CKAN really takes off?

Irina Bolychevsky: Yes. The open data movement had hit a particular tempo. Over a really short period, maybe a year or two, CKAN went from relatively unknown to the default. Not just for governments: for small organisations, nonprofits, anyone working with data, CKAN was the starting point. By the time I left, there were hundreds of live CKAN sites, and it powered the official national data portals of 25 countries.

People started copying it. There was a Drupal version called DKAN, a Jekyll one called JKAN, and other imitators. It succeeded because it did one thing well: it let organisations publish open data easily.

Lucas Pretti: You were doing all the sales and pricing. Why did governments come to CKAN, rather than you chasing them?

Irina Bolychevsky: I took advocacy for open source and open data seriously, including the role a portal can play in the wider mission. As well as business development, I ran workshops for governments and gave talks and keynotes at open data and open government events.

That’s where CKAN earned its reputation for quality and for values that were authentic. The team and I were passionate and knowledgeable, and it was clear we weren’t a company cashing in on a trend. We were part of the movement driving it. People meet you, talk to you, and come to trust you and the product you’re building.

The second reason was open source. I spent a lot of time explaining its value: avoiding lock-in, and keeping control, autonomy and sustainability. Our government clients took that seriously. It meant they could write CKAN into their requests for proposals, because several providers could build or maintain it.

Lucas Pretti: Jumping to today, what could have been done differently? What is a learning you take from CKAN’s peak?

Irina Bolychevsky: One of the main issues is the trap of publishing data and saying “done”, without anyone doing the real work of achieving accountability. Where’s the follow-up? I thought about that a lot. Open Knowledge’s School of Data tries to close the loop by training journalists to write stories from open data. It’s a fantastic project, but we need more.

The other problem is a familiar one in open source: maintenance. People were happy to write a cheque to launch something, but long-term investment was much harder. Even with our hackathons and events, a flourishing community with real feedback loops between publishing data and using it meaningfully mostly didn’t happen.

Lucas Pretti: If a government asked you today how to build public data infrastructure for the AI era, what would you tell them?

Irina Bolychevsky: I’d put more emphasis on standards and interoperability, and on understanding that data has limits. Data is always to some extent biased and contextual. It’s a snapshot, shaped by how, why and when it was collected. It’s wonderful to use and reuse, but not as the truth. When we only look at data, we can miss deeper meaning.

AI that is well trained, that takes those nuances into account and actively seeks human input and wisdom, could be really powerful. But I’d caution against any public digital infrastructure being led by AI’s interpretation of data. Decisions should be evidence-based and evidence-informed, but that means looking at data in its context, being ready to challenge it, and building in feedback loops and ways to self-correct. You can’t build infrastructure for a goal and walk away. It needs to be monitored, maintained and adjusted as things change and as you learn.

Lucas Pretti: Let’s finish going back to that breakfast photo. What made that team work?

Irina Bolychevsky: We respected each other, we had fun together, and we believed in what we were doing.

In the end, what mattered was context and team. The timing has to be right; there has to be space for the thing to succeed. And it’s very hard to make anything happen alone. With a team you trust, who trust you, who know what they’re doing and work well together, you can do something amazing.