Repository navigation
filewriter: dynamic metadata #88
Description
The filewriter must be able to write the 'dynamic' metadata for a run.
These are pieces of metadata which require knowledge from the _events (and/or other) streams to compute. They are 'dynamic' in the sense that they cannot be known at run start.
This includes:
raw_data_1/collection_time(becausecollection_timeexcludes vetoed/paused time so needs per-frame veto flags)raw_data_1/framelog/events_lograw_data_1/framelog/frame_lograw_data_1/framelog/good_frame_lograw_data_1/framelog/period_lograw_data_1/framelog/proton_chargeraw_data_1/framelog/raw_events_lograw_data_1/good_framesraw_data_1/proton_chargeraw_data_1/proton_charge_rawraw_data_1/raw_framesraw_data_1/runlog/count_rateraw_data_1/runlog/dae_beam_currentraw_data_1/runlog/good_framesraw_data_1/runlog/good_uah_lograw_data_1/runlog/is_runningraw_data_1/runlog/is_waitingraw_data_1/runlog/monitor_sum_1raw_data_1/runlog/np_ratioraw_data_1/runlog/periodraw_data_1/runlog/period_change_lograw_data_1/runlog/raw_framesraw_data_1/runlog/raw_uah_lograw_data_1/runlog/run_statusraw_data_1/runlog/total_countsraw_data_1/runlog/veto_lograw_data_1/total_countsraw_data_1/total_uncounted_counts
Potential differences from existing files
raw_data_1/total_uncounted_counts- "uncounted count" may not be a valid concept in datastreaming world. But we can write it as a constant zero instead.
Activity
mattclarke commentedon Sep 3, 2026
Not entirely sure how to do this in the ESS filewriter at the moment.
It is almost like doing a post-stopping writing "data" action. i.e.: start writing -> writing streamed data -> stop writing streamed data -> append metadata
Tom-Willemsen commentedon Sep 3, 2026
@mattclarke - thanks - I've written up how we think we'd implement these requirements in #110 - I think I had concluded this would need a number of new writer modules which listen to ev44s and other relevant schemas...