Skip to content

Data streaming: stack test #101

Open
Open
Data streaming: stack test#101
Assignees
rerphaTom-Willemsen
Milestone
Autumn 2026

Description

@rerpha
rerpha
opened on Jul 29, 2026

This test aims to find issues with the work we have done towards the September 2026 Milestone so far.

We should do a full stack test of our software so far.

This should be conducted on the new streaming server or NDW1836 if unavailable along with the streaming control board and detector modules set up on EMMA (or another test instrument)

We should test:
kdaectrl begins a run -> streaming hardware streams udp as expected -> event_udp_to_kafka forms ev44/pu00 correctly -> kafka_event_aggregator matches these up -> kdaediag reads them correctly.

Top-level workflow to test:

  • A python command writes to a begin PV (and also a GUI if available). This does not have to be genie_python for this test - could manually use p4p.
  • kdaectrl begins a run
  • streaming hardware streams udp as expected
  • event_udp_to_kafka forms ev44/pu00 correctly
  • kafka_event_aggregator matches these up and forwards to main _events stream
  • kdaediag reads them correctly and forms correct diagnostics e.g. spectra

Other things to test:

  • Pause/resume
  • Runcontrol
  • Hard/soft vetoes
  • Multi-period run
  • Timing modes
  • Wiring and spectra mapping are correct

note that this does not need to follow any sort of deployment approach - if containers work for certain processes ie. redpanda and things that aren't IOCs that's great, but if we have to run things manually that's not the end of the world. Infrastructure/deployment work will decide on this stuff for production, this is just a test.

This will require input from DSG.

People to invite to see the test once it's working: HRPD scientists, SANDALS scientists (?), report outcome of test to MN/MG?

Activity

  1. rerpha
    added this to the September 2026 milestone on Jul 29, 2026
  2. Tom-Willemsen
    added this to Data streaming for HRPD-X & SANDALS-IIon Jul 29, 2026
  3. Tom-Willemsen
    moved this to Ready in Data streaming for HRPD-X & SANDALS-IIon Jul 29, 2026
  4. Tom-Willemsen
    assigned
    Tom-Willemsen
    and
    rerpha
    on Jul 29, 2026

  5. Tom-Willemsen commented on Oct 1, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Oct 1, 2026
    Member

    After problems on EMMA at startup, should be doable on ALF soon(tm)

  6. Tom-Willemsen commented on Oct 12, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Oct 12, 2026
    Member

    Packages needing to be installed if locally compiling on ds machine:

    • dnf install perl-FindBin
    • dnf install perl-IPC-Cmd
    • dnf install perl-File-Compare
    • dnf install perl-File-Copy
    • dnf install perl-Time-Piece
    • dnf install cmake
    • dnf install g++

  7. rerpha commented on Oct 12, 2026

    @rerpha
    rerpha
    on Oct 12, 2026 · Hidden as low-quality
    Author

  8. Tom-Willemsen commented on Oct 12, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Oct 12, 2026
    Member

    Positive points

    • The stack seems to work at event rates close around 5-10gbps on the DS server (with simulated - i.e. saluki - data). This is well beyond what HRPDX will need, and still 5-10x what we're expecting on SANDALS2.
    • Starting and stopping runs and basic DAE control functionality works (as far as getting a RUNNING readback from the control board)

    Areas we need to improve

    • PVA from GUI is very unreliable if a PVA PV is ever disconnected
      • Especially easy to make it happen as kdaediag only serves spectra PVs for spectra that exist
    • kdaectrl is streaming run-starts with n_periods=1 even when more than one period is configured (Set number of periods when sending run start kafka_dae_control#80)
    • Spectrum mapping is unimplemented and a mess - det_spec_map not populated, kdaediag only knows about one spectrum, generally we just haven't done the work in this area - Detector spectrum mapping: decide on mechanism and file formats #124
    • Various bits of the GUI don't look great - probably was hard to get working readbacks without hardware or emulator yet
      • Some readbacks formatted as float when they should be int e.g. period/max period
      • Some readbacks visually cut off e.g. period type on experiment setup screen
      • H/W status should be an enum not just an int
      • Some information pointing at wrong PVs e.g. instrument on run information screen points at ...:GOODFRAMES
    • Need better feedback about why it's failing if you press begin and kdaediag goes into "processing" (no frames arrived). If you know the diagnostics to look for it makes sense, but needs to be more user friendly.
    • Pause/resume not setting the pause/resume veto bits correctly
    • Run numbers not formatted to 8 digits in streamed run name or in GUI
    • GUI needs controls to enable/disable vetoes
    • "UKNOWN" shouldn't appear as a setpoint option for DAE timing source or period type
    • In computer control mode GUI should have an option to change the current period manually
    • Every frame seems to have an unconditional TS2 pulse veto
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions