file_writer: improve run_stop logic #119

Activity

1. Tom-Willemsen

   changed the title [-]file_writer: improve run_stop logic[/-] [+]`file_writer`: improve run_stop logic[/+] on Oct 8, 2026
2. Tom-Willemsen

   added

   filewriter

   on Oct 8, 2026
3. Tom-Willemsen

   added this to the Spring 2027 milestone on Oct 8, 2026
4. Tom-Willemsen

   added this to Data streaming for HRPD-X & SANDALS-IIon Oct 8, 2026
5. Tom-Willemsen

   self-assigned this

   on Oct 9, 2026
6. Tom-Willemsen

   moved this from Ready to In progress in Data streaming for HRPD-X & SANDALS-IIon Oct 9, 2026
7. Tom-Willemsen commented on Oct 9, 2026

   

   Tom-Willemsen

   on Oct 9, 2026

   Last edited by Tom-Willemsen

   MemberAuthor

   More actions

   Thoughts about clocks and the filewriter:

   • The GPS clock is not necessarily in sync with the streaming-machine clock (on SuperMuSR it has frequently been desynced by tens of years due to dodgy GPS data).
   • The NDX clock is not necessarily in sync with either of those clocks (and is frequently desynced by ~minutes as it's on a VM with poor clock accuracy)
   • Computer clocks may jump or go backwards as a result of NTP corrections
   • SANDALS2 detectors which stream their own data to Kafka will each have their own, local independent clock
   • GPS clock may jump or go backwards if it starts getting dodgy data
   • If we use Kafka message timestamps (LogAppendTime on the topics), that is it's own clock
   • The filewriter may be lagging behind the stream, so stopping "now" doesn't necessarily mean all relevant data has been read from Kafka yet.
   • We want to be able to support sending run starts/stops with timestamps other than "now" (e.g. when rewriting a file later if needed)

   Currently:

   • Run start/stop are using computer clock
   • ev44/pu00 use GPS clock
   • Forwarder updates use the NDX clock (EPICS timestamps)

   Options

   A. Run start/stop use GPS time

   • Would need mods to SCB to allow for retrieving run start / run stop / current time according to GPS clock
   • Makes run_start comparable to ev44 timestamps
   • Doesn't help with EPICS f144s
   • GPS clock is nominally the highest accuracy but probably the least reliable in practice

   B. Configure all Kafka topics to use LogAppendTime

   • Kafka timestamp is not necessarily close to the timestamp inside a message like an ev44/pu00 etc
   • Filewriter does it's seeking/runstart/runstop logic with the kafka timestamps
   • If we're on a multi-node kafka then we're back to multiple clocks - but if they all share an NTP server, they should be reasonably well-synced in principle (~milliseconds?)
   • The run_start / run_stop inside a pl72/6s4t would still presumably be streaming-server wall clock. So we'd still be comparing that clock to Kafka clock somewhere.
     • But that means that event timestamps recorded in nexus might be outside start_time - stop_time if the GPS time is rubbish.
     • Still depend on the streaming server clock being reasonably in agreement with the Kafka clock.
   • May be some lag between an ev44 getting produced, going through aggregator (~few seconds), getting to the _events stream. So LogAppendTime may then exclude events that should otherwise have been part of a run

   C. Ignore all of this and assume all the clocks will always be in sync

   Pros: it's easy
   Cons: it's blatantly incorrect

   ---

   I think whatever solution this ticket comes up with, we're going to need to carefully write down what our assumptions are...

   ---

   As far as I can tell in ESS' filewriter, they just assume all the clocks are the same and freely compare them. I guess they've managed to implement a reliable timing system.
8. Tom-Willemsen

   moved this from In progress to In review in Data streaming for HRPD-X & SANDALS-IIon Oct 16, 2026
9. github-project-automation

   moved this from In review to In progress in Data streaming for HRPD-X & SANDALS-IIon Oct 16, 2026
10. Tom-Willemsen

    moved this from In progress to In review in Data streaming for HRPD-X & SANDALS-IIon Oct 16, 2026