Repository navigation
consider moving metadata ie. TITLE, USERS to a soft ioc #62
Description
this decouples the control code so if we wanted to "plug" something else in instead of the current kdaectrl we can do so without having to implement all of the metadata stuff.
consider using a soft ioc for this (needs autosave though) something like https://github.com/JeffersonLab/softioc may work. we will need to make this containerised.
Metadata
Metadata
Assignees
Labels
No labels
Activity
Tom-Willemsen commentedon May 28, 2026
On further thought - maybe we don't need
kafka_dae_controlto know about these at all?I'm thinking:
pl72run_titlefield.I think this might also neatly bypass issues with changing title/users/rb number mid-run, if they're not in the
RunStartdocument. The only remaining part of that issue would be changing vetos mid run, which we'd still need.My only concern is e.g. race conditions in the case where someone does
g.end();g.set_title()- need to make sure we don't accidentally pick up the next run's title update in filewriter even if it arrives in Kafka "first".Thoughts?
I think INSTETC was suggested as a home for these rather than a new IOC.
rerpha commentedon May 28, 2026
could do - though no real issue if we did need to get the title/users for some reason in kdaectrl - we have the EPICS_CA_ADDR_LIST set already for the blocks.
Think I am convinced this is a good idea though!
FreddieAkeroyd commentedon May 28, 2026
As I was discussing yesterday, I think this is a good point to consider what metadata lives where and not immediately copy the existing systtem. The isisicp was very monolithic and current isisdae ioc carries too many non acquisition parameters, this made it more fiddly when you wan to use ibex with a different acquisition system e.g. replace DAE with hexagons. Also if somebody wanted to runs two different DAEs on the same system, extra dae for testing etc. you suddenly find you might have two titles. So considering:
Tom-Willemsen commentedon May 28, 2026
Looking through
isisdae.db, I think the main candidates to think about then (alongside title/users) are:config.tomlinkdaecontrol- feels like that's fine/flexible enough? If we need to run two "data pipelines" concurrently they'd have differentconfig.tomls - which can either have the same instrument name or something likeemma-avsemma-b- it is flexible enough for either case.kdaecontrol) don't need to handle run number logic themselves.I think broadly we haven't made any big assumptions that would prevent us running two pipelines - e.g. PV prefixes are/should be configured by a
config.tomlso we can spawn two of them, one withIN:INSTRUMENT:DAE1andIN:INSTRUMENT:DAE2(and different kafka topic configurations too).rerpha commentedon May 28, 2026
This all sounds good to me. You might be able to get away with not actually needing to save any state in
kdaectrlin that case which is nice.rerpha commentedon Jun 2, 2026
ADR for this is ISISComputingGroup/ibex_developers_manual#213
ChrisM-S commentedon Jun 2, 2026
Just experimenting here with the options in terms of relations which (although not specifically in an RDB!) are fundamental to keying/finding data and identifying multiple elements which can be indexed subsequently in a possibly more distributed ICAT and also used in data file location. If anything else pops out of this it might be useful, and help in generating a way of identifying what must be in the list mentioned at different points (e.g. at run time on the instrument(s)) at any time after data has started collection.
Given some candidate keys:
Primary key: RB Number, Run Number, Inst Name, Period?
Secondary keys: Detector Element, SEnv Element, Sample, User Data Record (keyed by? user/title), Period?
and where there can be multiple secondary keys (e.g. more than one can be indexed by the primary key), is there anything fundamental missing from the primary key in particular. I think periods are probably secondary?