The following post is by Sam Leon, who’s just joined the OKF as a coommunity coordinator! Read more about Sam here.
Back in July of this year a crowd of coders, scientists and new media artists gathered in Berlin for the Open Science Workshop at OKCon. One of the projects to come out of this gathering was the Data Digitizer, a tool for transcribing documents and tables that are not currently machine-readable. Suggested applications for this tool ranged from the transcription of Brazilian census data to input of tables from economics articles to allow comparisons across multiple articles that examine the same variables.
The project is still ongoing with the code up on github. You can also find an Etherpad that details what was proposed and achieved in the first session of work on the Data Digitizer. Check out how far it’s got with a little demo that is up and running here.
Sam is a data trainer and wrangler at Open Knowledge. He Tweets from @Noel_Mas
When the data digitizer project is completed successfully it will allow massive amounts of data to become available for research and development of applications. It is truly a great project. I am currently trying to digitize Greek government monthly data from 1970 for 10 variables and trust me, my life is miserable. The data is in scanned pdf files of the monthly statistical bulletins and unfortunately there is no program to digitize scanned data currently. That’s why it will stand out. Unique. Well thought.
P.S: wish i had a slight programming ability so as to assist if i could.
George
Great to hear from you George. You don’t need to be a coder to contribute to these things — having a good problem or lots of enthusiasm is equally valuable :-)
By the way, from the sound of your problem you may be interested in joining the economics working group mailing list: http://lists.okfn.org/mailman/listinfo/open-economics. It was a bunch of (non-programmer) people from that working group who were driving the data digitizer idea and it sounds like what they wanted is very similar to what you want to do.
This is awesome and would really help with the art market data I’m currently digitizing.
It’s sad that the backend is the proprietary Google Docs service though.
Hey Rob: we just used google docs as backend DB because this was a hackday and it would make it very easy for people to add new tasks (it would be trivial to replace this with an SQL db) … and … I’m pleased to say we’re already working on this:
Last weekend (at the Africa@Home hackfect we started work on generalizing the datadigitizer into a general-purpose crowd-sourcing engine called “PyBossa”: https://github.com/citizen-cyberscience-centre/pybossa. This already has a substantial portion of the core functionality needed so if you’re interested please jump in and help us improve it.