Repository navigation
filewriter: deployment topology #98
Description
We need to define how the filewriter and other cooperating processes will run.
- One-per-instrument: each FW process incrementally writing out the events from a single instrument
- Pooled: some number, perhaps dynamic, of filewriters which may pick up a writing job from any instrument. This is the architecture that ESS use.
Things to consider:
- Pooled architecture is a requirement if we might ever collect data faster than it can be written to disk. This seems unlikely at present.
- Consider interaction with
filewriter: intermediate files & autosave files #96 ; this likely makes a pooled architecture more desirable - Needs to work for HRPD-X/SANDALS2/WISH/VESUVIO etc before MNeuData is available. We do not need to solve the MNeuData case in this issue, but we should pick an architecture which makes it possible to extend in likely MNeuData directions later.
- Consider interaction with
filewriter: file archiving #85
Activity
Tom-Willemsen commentedon Jul 27, 2026
@Pasarus your input is welcome, especially r.e. making this extensible towards likely MNeuData approaches.
Pasarus commentedon Jul 27, 2026
An architecture that takes configuration values in either a config file or set of environment variables, with a preference for the latter, would be ideal for deploying en-mass in a typical operator pattern, allowing trivial horizontal scaling based on usage.
One thing to discuss is the requirement for a message queue, and likely how/where that will be hosted (can be inside the MNeuData compute cluster). I foresee a requirement for the message queue, due to the likely desire to queue up requests/provide a pool of compute for filewriters, instead of having 1 filewriter waiting for each instrument and needing a heck of a lot of more hardware, a generic filewriter that can handle any instrument as needed makes the most sense to me.
Another technical caveat will be ensuring that this is not a windows only, un-containerisable piece of software, whilst ensuring that the software should be able to redo a job if it is interrupted part way through (hence the message queue), that will get picked up by another in the queue if the filewriter disappears for some reason.
Tom-Willemsen commentedon Jul 27, 2026
If we choose a pooled architecture, then I think the ESS schemas & kafka topics for FW control would be the starting point for this discussion. Doesn't mean we can't diverge if we deem it necessary.
It will run on Linux (likely in a container) for HRPDX/SANDALS/WISH/VESUVIO etc. As it just has Kafka as input and filesystem as output, I see no special reason why it would be particularly difficult to containerise.
FreddieAkeroyd commentedon Aug 9, 2026
I assume you mean faster than an individual file writer may be able to write a NeXus files from the broker events as opposed to faster than the kafka broker can write to disk?
Tom-Willemsen commentedon Aug 10, 2026
Yes. The Kafka broker side scales at frame-level granularity by splitting the messages for different frames between different Kafka partitions.
event-aggregatoralready sets a per-frame key to do this.However, a pooled filewriter could only scale by splitting
.nxs-writing jobs between servers at file-level granularity, not frame-level. A per-instrument architecture would obviously not scale horizontally at all.The fundamental limits, assuming we're writing events, would most likely be one of:
If we had a filewriter-per-instrument architecture then exceeding those limits means file-writing tasks start backing up and will eventually fail.
With a pooled architecture:
.nxswriting job for twonxsfiles can be on two different servers, potentially overlapping in time..nxs-writing servers ultimately share the same network uplink to the downstream archive servers.