Initial filewriter options analysis - #110
Conversation
There was a problem hiding this comment.
@mattclarke - sorry to tag you directly on a PR like this, it would great if you had time to look through this ess.md file and tell us where there are areas we're:
- Understating or overstating the amount of work required, or where we'd run strongly against architectural assumptions
- Missing mechanisms in the ESS filewriter which you think would be a better fit than what has been described here
- Overestimating or underestimating the extent of the existing support
Or anything else you feel we are misrepresenting about the ESS filewriter.
Tom-Willemsen
changed the title
| - **Unsupported**; it is unclear how this would be implemented. | ||
| - The ESS filewriter *does* use HDF5 SWMR support, so a separate/cooperating process may be able to take a 'snapshot' of an in-progress file, fix-up the metadata, and then emit that as an intermediate file. | ||
| - We could emit a new runStart/runStop pair, which would be picked up by another filewriter in the job-pool. This is an option because the ESS filewriter is inherently pooled and parallelizable by adding more writer processes. However, this would cause an expensive re-read of all of the data in the run from Kafka, and may cause our intermediate files to then be emitted so slowly that they are not useful to scientists. | ||
| - We could add full 'snapshotting' support to the filewriter, via a dedicated Kafka message, but this may be a rather invasive change. |
There was a problem hiding this comment.
this is a genuine question rather than a comment. What is the logic used to prompt the creation of an autosave file? is it a set number of frames, a time interval or something else?
There was a problem hiding this comment.
An autosave can be triggered after a number of frames, events, uAh, or a number of "dashboard polls" currently.
Triggering the autosaves could be built directly into the filewriter, or some external process would have to look at the counters from kafka_dae_diagnostics and decide to send this new dedicated "autosave now" Kafka message.
There was a problem hiding this comment.
currently we have "autosave" and "updatestore" that are basically doing the same thing, but slightly different use cases.
Autosave is used only on muons, the instrument control program creates an intermediate file automatically and regularly after a specified number of frames, filenames are re-used in a cycle (it writes e.g. .savA, then .savB up to .savE then goes back to .savA) and they are not archived long term. The reason for creating them is pseudo live data for analysis programs, a separate autosave.txt is updated with the latest filename and analysis programs can watch for a change here and then read/copy latest data file.
intermediate updatestore files (.s01, .n001) are explicitly user initiated and currently archived, sometimes a script creates one regularly e.g. waitfor, updatestore, in a loop or sequence. Often they are created by a user pushing the GUI button so they can analyse what has happened so far.
much of the use case for creating these files is based on analysis programs being unable to process live data directly and requiring a file; there was also historically a "backup" case for histogram mode of periodically updatestore in case of a DAE problems.
There was a problem hiding this comment.
there was also historically a "backup" case for histogram mode of periodically updatestore in case of a DAE problems.
I think this will be wanted for HRPD-X; they intend to do long runs relying on histograms for their analysis, so I think this will be needed to mitigate the impact of detector problems 11h into a 12h run.
|
|
||
| This filewriter option proposes a brand-new development, specific to ISIS. | ||
|
|
||
| This would be implemented in Rust, as per other performance-sensitive components of the datastreaming pipeline. |
There was a problem hiding this comment.
i feel like we wrote this in an ADR somewhere, but now i can't find it....
| * Add support for the small number of missing datasets (for example total_counts). | ||
| - [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) | ||
| * **Unsupported** in the filewriter itself | ||
| * The SuperMuSR pipeline generates histograms as a separate step from event-mode Nexus file writing. This is instead done by [MNeuEventLib](https://github.com/ISISMuon/MNeuEventLib) for SuperMuSR. |
There was a problem hiding this comment.
is this something we can use/adapt regardless of the choice we make for a filewriter?
There was a problem hiding this comment.
When I spoke to that team, they did say it "should work for Neutrons" - though I haven't done a detailed assessment.
I think the direction I'm leaning is that we probably need to support histogramming in the FW directly though - as some future instruments may have prohibitively large event files, and if their only reason to exist is to later be histogrammed anyway, then it seems much more efficient to histogram incrementally in the FW as events come in from Kafka.
Having histogramming in the FW directly also makes cases like "event + histogram data in the same file" easier to achieve.
There was a problem hiding this comment.
That project is now being more widely accepted into the MNeuData umbrella, so may be a suitable candidate for this. Anthony L or Alex R are the best people to ask about it.
| * Some of the correctness risks of using C++ are mitigated by the good test coverage and [routine use of sanitizers](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/conanfile.py#L34) | ||
| - The codebase is in a standalone repository that is reasonably well decoupled from other ESS-specific infrastructure. | ||
| - Historic collaboration with ESS has been somewhat patchy - we may struggle to reliably upstream all changes we want to make in a timely manner for HRPD-X, given our relatively tight implementation deadline. Therefore, a local fork, upstreaming changes *when possible*, seems like the most likely option. This creates the risk of significant later divergence unless we make a strong attempt to reconcile after HRPD-X implementation. | ||
| * [`CONTRIBUTING.MD` suggests that all changes would need to go through an ESS steering board](https://github.com/ess-dmsc/kafka-to-nexus/blob/main/CONTRIBUTING.md#the-project); it is highly unlikely that this would be a viable approach in time for HRPD-X commissioning, which strengthens the argument that we would be developing a local fork rather than upstream-first. |
There was a problem hiding this comment.
This is certainly a comment more for the long term but I wonder if there is space to talk about a more formal collaboration in this space? There are other collaborations, e.g. the user office, being entered into so there may be room for this type of discussion.
The flip side of me thinks that a facility solely owning its own file writing code is no bad thing either...
5 participants
Initial analysis of three filewriter-implementation options: ESS, SuperMuSR, greenfield.
No opinion is formed yet on which option is most suitable; these documents just aim to describe what each option would look like.