You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ID3A metadata records available for testing injector #1
@keara-soloway do you know which files out of this list should be used? I see the following:
# from your sub-areas
ls /nfs/chess/aux/metadata/cycles/2026-1/id3a/*/*/*.json | wc -l
172
# from id3a root
find /nfs/chess/aux/metadata/cycles/2026-1/id3a/ -name "*.json" | wc -l
4345
# here are examples of files from your sub-area
/nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiCr-1/TiCr-1.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/metatest-9/metatest-9.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/meta-test-4/meta-test-4.json
/nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/d-beamsize-atten-test-1/d-beamsize-atten-test-1.json
...
As you can see this sub-area contains lots of different files and I don't want to crawl and inject all of them. Do we have a specific file name pattern to search for?
In other words, if people do not organize their data it would be hard to process just generic list as it may contain only (small) portion of actual data we need. The tool I developed will take pattern, and try to inject the data. The FOXDEN server will reject records if they do not satisfies the schema, but it means that after injection we need to do proper bookkeeping of what was injected/rejected. For this reason we either need to have proper definition of data placement, i.e. I remember we discussed that metadata record should be presented in metadata directories, or use specific file naming convention, e.g. metadata-id3a-<some_name>.json.
/nfs/chess/aux/metadata/cycles/2026-1/id3a/sangid-4554-b/sam-c-ff-S0/sam-c-ff-S0.json which does not have all meta-data attributes
/nfs/chess/aux/metadata/cycles/2026-1/id3a/voisin-4783-a/TiV-7/TiV-7.json which seems to have full set of metadata attributes
/nfs/chess/aux/metadata/cycles/2026-1/id3a/test-123-a/test-9/test-9.json which most likely contains garbage, e.g. "sample_geometry": "faiwergfjkgjhadsg",
Can we use this pattern, which matches 36 records for just one of those BTRs: /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json
?
All these files contain complete, real metadata records (at least 28 key-value pairs in every one, and no "nonsense" values, as far as I can tell), and they exactly follow the rule for metadata file naming currently in place with automation.
The current rule for metadata record file naming with automation at 3a is: /nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/<sample_name>.json. Since this rule creates a naming convention issue, I can change the file naming rules to be: /nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/metadata_<sample_name>.json
to disambiguate if / when there are other non-record .json files in that directory. Does this fix the problem?
ok, I adjusted injector script to take an input file list, i.e. a file which contains list of files to be injected. Therefore I can save output of /nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json into input_files.txt and use it in injector script as following:
# save input file list
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt
# use input_list.txt with injector script
./injector -path input_list.txt -dry-run -url http://localhost:8300 -token $t
That said, I would advise to use a common naming convention to metadata JSON files ready for FOXDEN to distinguish them with any other JSON file. A prefix metadata_ for such files seems very reasonable to me. This will allow to use injector script using patter like this:
@keara-soloway , I tried your list of files and only 2 records were injected properly to foxden-dev. Here is commands I used:
# get new kerberos ticket
kinit
# get new foxden token
foxden token create write
# get input list files
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json > input_list.txt
# inject records to foxden-dev
foxden_injector -path ./input_list.txt -token /home/vk/.foxden.write.token -url https://foxden-dev.classe.cornell.edu:8300
After inspection of few files I found that only these two files:
Therefore, at this point I don't know what to do with these files and you should look at procedure how they were created to figure out why did value is missing.
The records were originally made to be injected with foxden CLI which creates and adds "did" based on --did-attrs. I can adjust the automation code to create these records with "did" already in place.
For now, I copied the records to scratch space and manually added "did". Here is a pattern that matches the adjusted records: /nfs/chess/scratch/user/kls286/injector_records.
@keara-soloway , I think we should still adjust did's, here is what I observe:
grep did /nfs/chess/scratch/user/kls286/injector_records/*.json
/nfs/chess/scratch/user/kls286/injector_records/abort_seq_test-1.json: "did": "/beamline=['3A']/btr=shanks-4838-a/cycle=2026-1/sample_name=abort_seq_test-1"
/nfs/chess/scratch/user/kls286/injector_records/ceo2-0128-1.json: "did": "/beamline=['3A']/btr=shanks-4838-a/cycle=2026-1/sample_name=ceo2-0128-1"
...
Almost all dids contains brackets and single quotes for beamline values, e.g. ['3A']. This will be taken as is and I don't think it is a good idea to have brackets and additional quotes in beamline value.
@keara-soloway , I also found that all records except two do not contain schema key, see /nfs/chess/scratch/user/kls286/injector_records/ti7-19-tomo.json vs /nfs/chess/scratch/user/kls286/injector_records/ti7-8-test.json, etc. I can manually add schema but I rather wanted to check with you since you may auto-generate these records and something needs to be fixed in spec macros. The foxden CLI has input parameters for schema, while for injector I didn't add it since I assumed that we may crawl metadata records from different beamlines at once. Therefore, each record should have schema key in it.
@keara-soloway , you may missed my previous comment, could you please reply how to proceed, i.e. add manually schema value to records or you'll fix json records.
ID1A3 is now currently generating metadata records that follow this naming pattern: /nfs/chess/aux/cycles/2026-2/id1a3/nygren-4918-b/metadata/*/metadata_*.json
However, these (currently 8) records will probably fail validation since the beamline scientists are intentionally excluding some fields that they no longer want in their schema, but they have not made any changes to the actual ID1A3.json schema yet.
Activity
vkuznet commentedon Feb 26, 2026
@keara-soloway do you know which files out of this list should be used? I see the following:
As you can see this sub-area contains lots of different files and I don't want to crawl and inject all of them. Do we have a specific file name pattern to search for?
In other words, if people do not organize their data it would be hard to process just generic list as it may contain only (small) portion of actual data we need. The tool I developed will take pattern, and try to inject the data. The FOXDEN server will reject records if they do not satisfies the schema, but it means that after injection we need to do proper bookkeeping of what was injected/rejected. For this reason we either need to have proper definition of data placement, i.e. I remember we discussed that metadata record should be presented in metadata directories, or use specific file naming convention, e.g.
metadata-id3a-<some_name>.json.vkuznet commentedon Feb 26, 2026
For instance look at these two files:
"sample_geometry": "faiwergfjkgjhadsg",keara-soloway commentedon Feb 27, 2026
Can we use this pattern, which matches 36 records for just one of those BTRs:
/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.json?
All these files contain complete, real metadata records (at least 28 key-value pairs in every one, and no "nonsense" values, as far as I can tell), and they exactly follow the rule for metadata file naming currently in place with automation.
The current rule for metadata record file naming with automation at 3a is:
/nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/<sample_name>.json. Since this rule creates a naming convention issue, I can change the file naming rules to be:/nfs/chess/aux/metadata/cycles/<cycle>/id3a/<btr>/<sample_name>/metadata_<sample_name>.jsonto disambiguate if / when there are other non-record .json files in that directory. Does this fix the problem?
vkuznet commentedon Feb 27, 2026
ok, I adjusted
injectorscript to take an input file list, i.e. a file which contains list of files to be injected. Therefore I can save output of/nfs/chess/aux/metadata/cycles/2026-1/id3a/shanks-4838-a/*/*.jsoninto input_files.txt and use it in injector script as following:That said, I would advise to use a common naming convention to metadata JSON files ready for FOXDEN to distinguish them with any other JSON file. A prefix
metadata_for such files seems very reasonable to me. This will allow to use injector script using patter like this:This way, an injector script will crawl from given path and only pick up files according to provided pattern.
vkuznet commentedon Feb 27, 2026
@keara-soloway , I tried your list of files and only 2 records were injected properly to foxden-dev. Here is commands I used:
After inspection of few files I found that only these two files:
were injected properly because all of their attributes are correct. All other files do not have
didvalue. For instance have a look at:It seems fine but does not have did in it.
Therefore, at this point I don't know what to do with these files and you should look at procedure how they were created to figure out why
didvalue is missing.keara-soloway commentedon Mar 3, 2026
The records were originally made to be injected with foxden CLI which creates and adds "did" based on --did-attrs. I can adjust the automation code to create these records with "did" already in place.
For now, I copied the records to scratch space and manually added "did". Here is a pattern that matches the adjusted records: /nfs/chess/scratch/user/kls286/injector_records.
vkuznet commentedon Mar 5, 2026
@keara-soloway , I think we should still adjust did's, here is what I observe:
Almost all dids contains brackets and single quotes for beamline values, e.g.
['3A']. This will be taken as is and I don't think it is a good idea to have brackets and additional quotes in beamline value.keara-soloway commentedon Mar 5, 2026
Sorry about that, the dids should be fixed now -- no brackets, no quotes, and all lowercase.
vkuznet commentedon Mar 6, 2026
@keara-soloway , I also found that all records except two do not contain
schemakey, see/nfs/chess/scratch/user/kls286/injector_records/ti7-19-tomo.jsonvs/nfs/chess/scratch/user/kls286/injector_records/ti7-8-test.json, etc. I can manually add schema but I rather wanted to check with you since you may auto-generate these records and something needs to be fixed in spec macros. The foxden CLI has input parameters for schema, while for injector I didn't add it since I assumed that we may crawl metadata records from different beamlines at once. Therefore, each record should have schema key in it.vkuznet commentedon Mar 18, 2026
@keara-soloway , you may missed my previous comment, could you please reply how to proceed, i.e. add manually schema value to records or you'll fix json records.
keara-soloway commentedon Apr 30, 2026
ID1A3 is now currently generating metadata records that follow this naming pattern:
/nfs/chess/aux/cycles/2026-2/id1a3/nygren-4918-b/metadata/*/metadata_*.jsonHowever, these (currently 8) records will probably fail validation since the beamline scientists are intentionally excluding some fields that they no longer want in their schema, but they have not made any changes to the actual ID1A3.json schema yet.
vkuznet commentedon May 8, 2026
@keara-soloway as we expected the injection fails due to mismatch of records with their schema: