consider moving metadata ie. TITLE, USERS to a soft ioc

[rerpha]

this decouples the control code so if we wanted to "plug" something else in instead of the current kdaectrl we can do so without having to implement all of the metadata stuff.

consider using a soft ioc for this (needs autosave though) something like https://github.com/JeffersonLab/softioc may work. we will need to make this containerised.

[Tom-Willemsen]

On further thought - maybe we don't need kafka_dae_control to know about these at all?

I'm thinking:

• We autogenerate a run name (e.g. just string version of run number or similar) for the pl72 run_title field.
• Forwarder forwards TITLE and USERS from soft ioc (e.g. INSTETC) as per any other PV (once we fix it to work with strings properly...)
• We were already going to add something to filewriter to "always overwrite with single most recent value" rather than timeseries, so that's how we make filewriter write the most recent title/users/rb at time run is written.

I think this might also neatly bypass issues with changing title/users/rb number mid-run, if they're not in the RunStart document. The only remaining part of that issue would be changing vetos mid run, which we'd still need.

My only concern is e.g. race conditions in the case where someone does g.end();g.set_title() - need to make sure we don't accidentally pick up the next run's title update in filewriter even if it arrives in Kafka "first".

Thoughts?

---

I think INSTETC was suggested as a home for these rather than a new IOC.

[rerpha]

could do - though no real issue if we did need to get the title/users for some reason in kdaectrl - we have the EPICS_CA_ADDR_LIST set already for the blocks.

Think I am convinced this is a good idea though!

[FreddieAkeroyd]

As I was discussing yesterday, I think this is a good point to consider what metadata lives where and not immediately copy the existing systtem. The isisicp was very monolithic and current isisdae ioc carries too many non acquisition parameters, this made it more fiddly when you wan to use ibex with a different acquisition system e.g. replace DAE with hexagons. Also if somebody wanted to runs two different DAEs on the same system, extra dae for testing etc. you suddenly find you might have two titles. So considering:

• what metadata really needs to live where, is it relevant for the functionality of the process
• would you ever need to create a second of these processes, or a different one with an equivalent interface; do you find you are suddenly serving two of the same non acquistion relevant value. This would be for example having an ISIS and some test DAE system present, or just some other DAE present, and not needing to run one acquisition system in "simulation mode" just to get the metadata and a data file

[Tom-Willemsen]

Looking through isisdae.db, I think the main candidates to think about then (alongside title/users) are:

• RB number / ISIS_CYCLE
  • Probably should be treated in same way as title/users, whatever approach that ends up being. Though ISIS_CYCLE needs to be read from somewhere, not settable by users.
• Instrument name / ISMUON
  • This is entirely static and read from a config.toml in kdaecontrol - feels like that's fine/flexible enough? If we need to run two "data pipelines" concurrently they'd have different config.tomls - which can either have the same instrument name or something like emma-a vs emma-b - it is flexible enough for either case.
• Run number
  • For comparison, DLS have a specific "service" for this - https://github.com/DiamondLightSource/numtracker - which multiple detectors can query and it "arbitrates" which detector gets which run or scan number and where those detectors should write their data. That's probably the most flexible solution and means individual detector programs (e.g. hexagons or kdaecontrol) don't need to handle run number logic themselves.
• ERRMSGS/ALLMSGS - should probably go via our IOC logging infrastructure instead.

---

I think broadly we haven't made any big assumptions that would prevent us running two pipelines - e.g. PV prefixes are/should be configured by a config.toml so we can spawn two of them, one with IN:INSTRUMENT:DAE1 and IN:INSTRUMENT:DAE2 (and different kafka topic configurations too).

[rerpha]

This all sounds good to me. You might be able to get away with not actually needing to save any state in kdaectrl in that case which is nice.

[rerpha]

ADR for this is ISISComputingGroup/ibex_developers_manual#213

[ChrisM-S]

Just experimenting here with the options in terms of relations which (although not specifically in an RDB!) are fundamental to keying/finding data and identifying multiple elements which can be indexed subsequently in a possibly more distributed ICAT and also used in data file location. If anything else pops out of this it might be useful, and help in generating a way of identifying what must be in the list mentioned at different points (e.g. at run time on the instrument(s)) at any time after data has started collection.

Given some candidate keys:
Primary key: RB Number, Run Number, Inst Name, Period?
Secondary keys: Detector Element, SEnv Element, Sample, User Data Record (keyed by? user/title), Period?

and where there can be multiple secondary keys (e.g. more than one can be indexed by the primary key), is there anything fundamental missing from the primary key in particular. I think periods are probably secondary?