Repository navigation
seed-data generation script #1
Description
We need a script that generates a seed data file for the Library of Congress Maphub instance.
Each "map record" in the seed data file includes:
- pointers to the map image file URIs
- selected metadata fields (title, description, subject, creator)
We already have the maps in place and scripts to download metadata from the LoC's GMD collection (see scripts directory). The script has to read these map identifiers, iterate over the harvested metadata records, identify matching records (based on the map identifier), and output a maphub map record for each match.
The challenging part of this script is to select the appropriate metadata fields from the OAI-PMH records. We want only those that carry "relevant" semantics about the map. Also some data cleansing (whitespace, special chars, etc.) steps might be necessary. At the end the metadata need to be indexed by Apache Solr / Lucene.
The results should be a script generate-loc-seeddata which takes the directory of map image files and a directory of XML files (= the metadata records) and a set of identifiers (probably a TXT file) as input and generates an outputfile loc-seeddata.yaml
Possible execution:
generate-loc-seeddata maps/ metadata/*.xml
generate-loc-seeddata -n 10 maps/ metadata/*.xml for only 10 maps
Activity
shionguha commentedon Mar 6, 2012
working on harvesting DC metadata using fetch-metadata script and tag analysis right now.
shionguha commentedon Mar 8, 2012
Possible issue with ruby on the server (aka am I missing something?). As I try to work with the ruby interpreter, it gives me the following message:
The program 'ruby' can be found in the following packages:
Ask your administrator to install one of them
Currently, I am working on my machine and then sftp-ing to the server.
behas commentedon Mar 8, 2012
RVM (https://rvm.beginrescueend.com/) is now installed and the problem is fixed:
maphub@samos:~$ ruby -v
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-linux]
In general, I recommend to develop on your local machine (using the same ruby version) and use the server only for running longer jobs
Best,
Bernhard
On Mar 7, 2012, at 10:07 PM, Shion wrote:
Possible issue with ruby on the server (aka am I missing something?). As I try to work with the ruby interpreter, it gives me the following message:
The program 'ruby' can be found in the following packages:
Ask your administrator to install one of them
Currently, I am working on my machine and then sftp-ing to the server.
Reply to this email directly or view it on GitHub:
https://github.com/maphub/maphub-seeddata/issues/1#issuecomment-4383943
Bernhard Haslhofer
Postdoc Associate
Cornell Information Science
301 College Ave.
Ithaca, NY 14850
WWW: http://www.cs.cornell.edu/~bh392/
Skype: bernhard.haslhofer
shionguha commentedon Apr 9, 2012
Basic script has been uploaded along with a screenshot showing terminal run and the sample yaml output file. Waiting for bug fixing + server running...
shionguha commentedon Apr 9, 2012
The script is run as follows:
ruby generate-loc-seeddata.rb -i mapdir -m metadatadir -n numofsamples
i.e. ruby generate-loc-seeddata.rb -i maps -m mods_metadata_8800 -n 170
behas commentedon Apr 9, 2012
Cool, thank you! I will have a look at it later today and get back to you.
Bernhard Haslhofer
Postdoc Associate
Cornell Information Science
301 College Ave.
Ithaca, NY 14850
WWW: http://www.cs.cornell.edu/~bh392/
Skype: bernhard.haslhofer
On Monday, April 9, 2012 at 12:49 AM, Shion wrote:
behas commentedon Apr 11, 2012
Hi Shion,
the code looks good. There is quite a bit of IO overhead, but on Friday I will give you some tips how you can fix this and speed up the script.
When I tried to execute the script on the server I had the following problem:
maphub@samos:~/maphub-seeddata/scripts$ ruby generate-loc-seeddata.rb -i ../../data/maps/ -m ../../data/metadata/ -n 5
Finding all the map files...
Finding all the metadata files ...
Creating output YAML File
generate-loc-seeddata.rb:177:in
<main>': undefined methodparent' for nil:NilClass (NoMethodError)I will add this to the issues list, just to track progress.
It would also be great if you could summarize the usage of the three scripts in the README file. Please keep it short, just explain what the scripts do and how to use them.
Ad Github: avoid checking in non-source files (output, screenshots, etc.). Github is really just for code…
Thanks again and talk to you on Friday,
Bernhard
Bernhard Haslhofer
Postdoc Associate
Cornell Information Science
301 College Ave.
Ithaca, NY 14850
WWW: http://www.cs.cornell.edu/~bh392/
Skype: bernhard.haslhofer
On Monday, April 9, 2012 at 12:49 AM, Shion wrote:
shionguha commentedon Apr 11, 2012
Thanks. I just updated the README. Is this fine or were you thinking about something else?
behas commentedon Apr 13, 2012
I made some changes to the README file and added some comments to the script code...
Bernhard Haslhofer
Postdoc Associate
Cornell Information Science
301 College Ave.
Ithaca, NY 14850
WWW: http://www.cs.cornell.edu/~bh392/
Skype: bernhard.haslhofer
On Wednesday, April 11, 2012 at 4:29 PM, Shion wrote: