Skip to content

ID3A metadata records available for testing injector #1

Open

Description

@keara-soloway

Here is a filename pattern that matches ~170 ID3A records to be injected:

/nfs/chess/aux/metadata/cycles/2026-1/id3a/*/*/*.json

Activity

  1. self-assigned this
    on Feb 26, 2026
  2. vkuznet commented on Feb 26, 2026

    @vkuznet
    Contributor

    @keara-soloway do you know which files out of this list should be used? I see the following:

    # from your sub-areas
    ls /nfs/chess/aux/metadata/cycles/2026-1/id3a/*/*/*.json  | wc -l
    172
    
    # from id3a root
    find /nfs/chess/aux/metadata/cycles/2026-1/id3a/ -name "*.json" | wc -l
    4345
    
    # here are examples of files from your sub-area
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiCr-1/TiCr-1.json
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/metatest-9/metatest-9.json
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/meta-test-4/meta-test-4.json
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/d-beamsize-atten-test-1/d-beamsize-atten-test-1.json
    ...
    

    As you can see this sub-area contains lots of different files and I don't want to crawl and inject all of them. Do we have a specific file name pattern to search for?

    In other words, if people do not organize their data it would be hard to process just generic list as it may contain only (small) portion of actual data we need. The tool I developed will take pattern, and try to inject the data. The FOXDEN server will reject records if they do not satisfies the schema, but it means that after injection we need to do proper bookkeeping of what was injected/rejected. For this reason we either need to have proper definition of data placement, i.e. I remember we discussed that metadata record should be presented in metadata directories, or use specific file naming convention, e.g. metadata-id3a-<some_name>.json.

  3. vkuznet commented on Feb 26, 2026

    @vkuznet
    Contributor

    For instance look at these two files:

    • /nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/sam-c-ff-S0/sam-c-ff-S0.json which does not have all meta-data attributes
    • /nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiV-7/TiV-7.json which seems to have full set of metadata attributes
    • /nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json which most likely contains garbage, e.g. "sample_geometry": "faiwergfjkgjhadsg",
  4. keara-soloway commented on Feb 27, 2026

    @keara-soloway
    ContributorAuthor

    Can we use this pattern, which matches 36 records for just one of those BTRs:
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json
    ?

    All these files contain complete, real metadata records (at least 28 key-value pairs in every one, and no "nonsense" values, as far as I can tell), and they exactly follow the rule for metadata file naming currently in place with automation.

    The current rule for metadata record file naming with automation at 3a is: /nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/<sample_name>.json. Since this rule creates a naming convention issue, I can change the file naming rules to be:
    /nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/metadata_<sample_name>.json
    to disambiguate if / when there are other non-record .json files in that directory. Does this fix the problem?

  5. vkuznet commented on Feb 27, 2026

    @vkuznet
    Contributor

    ok, I adjusted injector script to take an input file list, i.e. a file which contains list of files to be injected. Therefore I can save output of /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json into input_files.txt and use it in injector script as following:

    # save input file list
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt
    
    # use input_list.txt with injector script
    ./injector -path input_list.txt -dry-run -url http://localhost:8300 -token $t
    

    That said, I would advise to use a common naming convention to metadata JSON files ready for FOXDEN to distinguish them with any other JSON file. A prefix metadata_ for such files seems very reasonable to me. This will allow to use injector script using patter like this:

    ./injector -path /nfs/path -pattern "metadata_*.json ...
    

    This way, an injector script will crawl from given path and only pick up files according to provided pattern.

  6. vkuznet commented on Feb 27, 2026

    @vkuznet
    Contributor

    @keara-soloway , I tried your list of files and only 2 records were injected properly to foxden-dev. Here is commands I used:

    # get new kerberos ticket
    kinit
    
    # get new foxden token
    foxden token create write
    
    # get input list files
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt
    
    # inject records to foxden-dev
    foxden_injector -path ./input_list.txt -token /home/vk/.foxden.write.token -url https://foxden-dev.classe.cornell.edu:8300
    

    After inspection of few files I found that only these two files:

    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-19-nf/ti7-19-nf.json
    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-19-tomo/ti7-19-tomo.json
    

    were injected properly because all of their attributes are correct. All other files do not have did value. For instance have a look at:

    /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/ti7-8-tomo/ti7-8-tomo.json
    

    It seems fine but does not have did in it.

    Therefore, at this point I don't know what to do with these files and you should look at procedure how they were created to figure out why did value is missing.

  7. keara-soloway commented on Mar 3, 2026

    @keara-soloway
    ContributorAuthor

    The records were originally made to be injected with foxden CLI which creates and adds "did" based on --did-attrs. I can adjust the automation code to create these records with "did" already in place.

    For now, I copied the records to scratch space and manually added "did". Here is a pattern that matches the adjusted records: /nfs/chess/scratch/user/kls286/injector_records.

  8. vkuznet commented on Mar 5, 2026

    @vkuznet
    Contributor

    @keara-soloway , I think we should still adjust did's, here is what I observe:

    grep did /nfs/chess/scratch/user/kls286/injector_records/*.json
    /nfs/chess/scratch/user/kls286/injector_records/abort_seq_test-1.json:  "did": "/beamline=['3A']/btr=shanks-4838-a/cycle=2026-1/sample_name=abort_seq_test-1"
    /nfs/chess/scratch/user/kls286/injector_records/ceo2-0128-1.json:  "did": "/beamline=['3A']/btr=shanks-4838-a/cycle=2026-1/sample_name=ceo2-0128-1"
    ...
    

    Almost all dids contains brackets and single quotes for beamline values, e.g. ['3A']. This will be taken as is and I don't think it is a good idea to have brackets and additional quotes in beamline value.

  9. keara-soloway commented on Mar 5, 2026

    @keara-soloway
    ContributorAuthor

    Sorry about that, the dids should be fixed now -- no brackets, no quotes, and all lowercase.

  10. vkuznet commented on Mar 6, 2026

    @vkuznet
    Contributor

    @keara-soloway , I also found that all records except two do not contain schema key, see /nfs/chess/scratch/user/kls286/injector_records/ti7-19-tomo.json vs /nfs/chess/scratch/user/kls286/injector_records/ti7-8-test.json, etc. I can manually add schema but I rather wanted to check with you since you may auto-generate these records and something needs to be fixed in spec macros. The foxden CLI has input parameters for schema, while for injector I didn't add it since I assumed that we may crawl metadata records from different beamlines at once. Therefore, each record should have schema key in it.

  11. vkuznet commented on Mar 18, 2026

    @vkuznet
    Contributor

    @keara-soloway , you may missed my previous comment, could you please reply how to proceed, i.e. add manually schema value to records or you'll fix json records.

  12. keara-soloway commented on Apr 30, 2026

    @keara-soloway
    ContributorAuthor

    ID1A3 is now currently generating metadata records that follow this naming pattern:
    /nfs/chess/aux/cycles/2026-2/id1a3/nygren-4918-b/metadata/*/metadata_*.json

    However, these (currently 8) records will probably fail validation since the beamline scientists are intentionally excluding some fields that they no longer want in their schema, but they have not made any changes to the actual ID1A3.json schema yet.

  13. vkuznet commented on May 8, 2026

    @vkuznet
    Contributor

    @keara-soloway as we expected the injection fails due to mismatch of records with their schema:

    foxden_injector -path /nfs/chess/aux/cycles/2026-2/id1a3/nygren-4918-b/metadata/ -patter
    n "metadata_*.json" -token /home/vk/.foxden.write.token -url https://foxden-dev.classe.cornell.edu:8300
    status:500 file:/nfs/chess/aux/cycles/2026-2/id1a3/nygren-4918-b/metadata/ceo2-test-1/metadata_ceo2-test-1.json error:[MetaDat
    a.main.insertData] validateData error: [MetaData.main.validateData] schema.Validate error: invalid data value for key=sample_s
    tate, type=list_str, multiple=false, value=[powder] valuetype=[]interface {}
    ...
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions