Repository navigation
Data streaming: write journals #11
Description
Acceptance criteria
- Python or rust process implemented as a standalone python package (e.g. see packaging for
ibex_bluesky_core/genie/lewisfor how to do this) or rust project- e.g. named
kafka_dae_journals, to sit alongsidekafka_dae_controlandkafka_dae_diagnostics
- e.g. named
- Listens to a kafka stream for run stop messages
- Update a
journal.xmlfile with metadata from the run when awrdnis published - this then needs to read the file and pull out the journal metadata. - Format of written
journal.xmlshould match existingjournal.xmlas closely as possible. - Create a follow up ticket to test against Journal Viewer (currently some missing data)
Notes
- Some metadata won't currently be available from kafka. Create new ticket(s) for these cases. For this ticket, write the data which is currently available from kafka run start/stop messages and corresponds to a field in the
journal.xml. Create follow up ticket(s) for data not currently available.
Discussed Planning 08/01/2026 14:30
Activity
[-]How do we write journals[/-][+]Data streaming: write journals[/+]Tom-Willemsen commentedon Jan 18, 2026
Journal contains some entries like total events, total proton charge, good frames, raw frames that need to be accumulated from the main
_eventsstream (not justrunInfopl72/6s4t). Which means that this process would need some awareness of e.g. vetoes so that it can distinguish good and raw frames, etc - and would also need to subscribe to the high-rate_eventsstream.A better option may be for this process to trigger from a
wrdnmessage, where thewrdn's metadata would contain these sorts of accumulated quantities (or potentially everything that needs to be written to the journal?). This means:.nxsis identical to journal (since the same process calculated both - so in the case of dropped frames etc they will remain consistent)kafka_journal_writerdoesn't need to subscribe to the high-rate_eventsstream, only low-rate metadata topics, which reduces network load and makes this process less performance-sensitive.