In the last couple of months we’ve had several threads on the okfn-discuss list about distributed storage for open data (see here and here).
Last month we started a distributed storage project, aiming to provide distributed storage infrastructure for OKF and other open knowledge projects.
After researching various technical options, we’ve launched an Open Data Grid based on Allmydata’s open-source “Tahoe” system at:
Anyone can store open data on the grid, or start running a storage node. For more details see the readme. If you’d like to comment on the service feel free to post on the okfn-discuss list!
Dr. Jonathan Gray is Lecturer in Critical Infrastructure Studies at the Department of Digital Humanities, King’s College London, where he is currently writing a book on data worlds. He is also Cofounder of the Public Data Lab; and Research Associate at the Digital Methods Initiative (University of Amsterdam) and the médialab (Sciences Po, Paris). More about his work can be found at jonathangray.org and he tweets at @jwyg.
Congrats on this Jonathan. Very timely and I like the distributed nature of it.
What are some of the intended uses of this? Can files also be deleted/moved? Are there any security features? The ODG page doesn’t say much about this.
Its intended usage is as distributed storage for open (see http://opendefinition.org/) data and content. In particular, here at the Open Knowledge Foundation, we plan to use it to store datasets associated with packages on http://www.ckan.net/ and projects on http://knowledgeforge.net/
Yes files can be deleted and moved depending on their permissions (as encapsulated in the “capabilities” url — see the Tahoe documentation). Usually a file uploader will have full capabilities while others may just have read-only access. We want to to more work here to provide a better, more flexible, permissioning model, as well as a simple to use uploader front-end and we’d love to collaborate with others on this.
What kind of security features?
Greetings;
I am excited about this but alas I have read the ‘readme’ text and I am unsure what to do. I do have ubuntu running but I am not sure how to follow your instructions. Can someone help?
Cheers
t
Hi Tracey,
Thanks for the feedback: installing and setting up Tahoe isn’t as easy as it could be!
Anyway, we’d love to help: just get in touch at info@okfn.org.
In the mean time if you’re having trouble actually installing tahoe (as opposed to setting it up), the easiest way is to use the prebuilt ubuntu/debian packages as listed on http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages.