Repository navigation
filewriter: intermediate files & autosave files #96
Description
The filewriter must support intermediate files - i.e. a request to write all of the data so far in the run to a .nxs file, without ending the run.
It must also similarly support autosave files - as above but happens automatically at intervals.
This is especially important as HRPDX plan to run their analysis in histogram mode, without these intermediate files a detector failure 11 hours into a 12 hour run would trash the 11 hours worth of data, with these files at least some data should be recoverable (this is not an issue for event mode, only histogram mode).
Activity
FreddieAkeroyd commentedon Aug 9, 2026
This needs the software to be able to create a copy of the nxs file it has written so far that a consumer can read, and then continue from where it is currently. If it has to close a file and start replaying events to a new file that could cause an unacceptable delay with the next file. There is probably something in the SWMR support that allows such a snapshot to be made and an intermediate file written from that while leaving the original file to continue growing.
Tom-Willemsen commentedon Aug 10, 2026
The ESS filewriter does look to use SWMR by default, so that probably is viable. I was originally worried that SWMR might complicate the implementation too much with extra restrictions around what datasets you are allowed to write/modify.