Skip to content

filewriter: specify file layout in detail #76

Open
Open
filewriter: specify file layout in detail#76
Assignees
FreddieAkeroydrerphaTom-Willemsen
Labels
filewriter
Milestone
Spring 2027

Description

@Tom-Willemsen
Tom-Willemsen
opened on Jul 8, 2026

We need to formally document what we expect any 'new' .nxs files to look like, and pay particular attention to differences with the old system.

The goal is to be compatible where possible / where it makes sense, but some information may fundamentally differ between ICP and streaming stack, e.g.:

  • Much of the information in the isis_vms_compat group
  • How we write per-frame veto state and enablement alongside the event-mode data
  • Various metadata fields
  • Some pieces of information which only make sense in ICP (or in the newer streaming system)

This ticket must include discussion with stakeholders such as DSG and Mantid (and other relevant groups)

Acceptance criteria

  • Documented structure for what a 'new' nexus file will look like
  • Documentation describing differences where they exist (effectively a 'porting guide' between ICP and streaming files)
  • An example file in the new format (does not need to have been written by 'new' filewriter; e.g. a simple python program can be used to generate this example file / manipulate an existing file)

Activity

  1. Tom-Willemsen
    assigned
    Tom-Willemsen
    ,
    rerpha
    and
    FreddieAkeroyd
    on Jul 8, 2026
  2. rerpha
    moved this to research/needs discussion in Data streaming for HRPD-X & SANDALS-IIon Jul 14, 2026
  3. rerpha
    added this to Data streaming for HRPD-X & SANDALS-IIon Jul 14, 2026
  4. Tom-Willemsen
    added this to the March 2027 milestone on Jul 14, 2026

  5. Tom-Willemsen commented on Jul 14, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Jul 14, 2026
    MemberAuthor

    For the data that currently exists in the isis_vms_compat group, the meanings of the existing parameters are detailed in a document in \\isis\shares\ISIS_Experiment_Controls\data for tickets\ds_76\libget.txt.

    Mantid can read some of these, but possibly only for backwards-compatibility. As part of this ticket we should decide which of the datasets in isis_vms_compat, if any, we will continue to write under the new filewriter.

  6. Tom-Willemsen
    changed the title [-]Filewriter: specify file layout in detail[/-] [+]`filewriter`: specify file layout in detail[/+] on Jul 25, 2026

  7. Tom-Willemsen commented on Jul 27, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Jul 27, 2026
    MemberAuthor

    Have added a number of issues with the filewriter label, which start to enumerate potential differences in more detail.

    This ticket impeded until those questions/file differences are resolved.

  8. Tom-Willemsen
    added
    filewriter
    on Jul 27, 2026
  9. Tom-Willemsen
    moved this from research/needs discussion to In progress in Data streaming for HRPD-X & SANDALS-IIon Jul 27, 2026

  10. Tom-Willemsen commented on Aug 28, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Aug 28, 2026
    MemberAuthor

    When developing the new filewriter, we would then run some form of nexus-diff as a system test, comparing the filewriter output against the 'reference' file from this ticket, failing if the structure does not match.

    We may need multiple of these reference files, for example:

    • event-only
    • histogram-only
    • event + histogram
    • detectors in event mode, monitors in histogram mode

  11. Tom-Willemsen commented on Sep 7, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Sep 7, 2026
    MemberAuthor

    The following are things that are invalid in our current NeXus files - excluding the items which are just extra things we write that aren't defined in NeXus (e.g. our IX* classes and extra datasets). We should decide whether to fix any of these things to align with the NeXus standard, or whether we will try to stay exactly tied to what the ICP generated.

    NXroot: /
    NXentry: /raw_data_1
        Field: /raw_data_1/definition_local
            This field is now deprecated. see same field in :ref:`NXsubentry` for preferred use
        NXdata: /raw_data_1/detector_1
            "@signal" is not present in the group
            "@axes" is not present in the group
        Field: /raw_data_1/duration
            This field is not a valid NX_INT
        NXinstrument: /raw_data_1/instrument
            NXdetector: /raw_data_1/instrument/detector_1
                Field: /raw_data_1/instrument/detector_1/azimuthal_angle
                    The field has rank 1, should be 3
                    The field rank is 1, but the dimension index of "i" = 2
                    The field rank is 1, but the dimension index of "j" = 3
                Field: /raw_data_1/instrument/detector_1/distance
                    The field has rank 1, should be 3
                    The field rank is 1, but the dimension index of "i" = 2
                    The field rank is 1, but the dimension index of "j" = 3
                Field: /raw_data_1/instrument/detector_1/polar_angle
                    The field has rank 1, should be 3
                    The field rank is 1, but the dimension index of "i" = 2
                    The field rank is 1, but the dimension index of "j" = 3
                Link: /raw_data_1/instrument/detector_1/time_of_flight
                    The field has size (101,), should be tof+1
                    Using "axis" as a field attribute is no longer valid. Use the group attribute "axes"
            NXsource: /raw_data_1/instrument/source
                Field: /raw_data_1/instrument/source/probe
                    The field value is not a member of the enumerated list
                Field: /raw_data_1/instrument/source/type
                    The field value is not a member of the enumerated list
        NXmonitor: /raw_data_1/monitor_1
            Field: /raw_data_1/monitor_1/data
                Using "signal" as a field attribute is no longer valid. Use the group attribute "signal"
            Link: /raw_data_1/monitor_1/time_of_flight
                Using "axis" as a field attribute is no longer valid. Use the group attribute "axes"
        NXmonitor: /raw_data_1/monitor_2
            Field: /raw_data_1/monitor_2/data
                Using "signal" as a field attribute is no longer valid. Use the group attribute "signal"
            Link: /raw_data_1/monitor_2/time_of_flight
                Using "axis" as a field attribute is no longer valid. Use the group attribute "axes"
        NXsample: /raw_data_1/sample
            Field: /raw_data_1/sample/distance
                Units of NX_LENGTH not specified
            Field: /raw_data_1/sample/thickness
                Units of NX_LENGTH not specified
            Field: /raw_data_1/sample/type
                The field value is not a member of the enumerated list

Total number of warnings: 66
Total number of errors: 14

  12. Tom-Willemsen commented on Oct 20, 2026

    @Tom-Willemsen
    Tom-Willemsen
    on Oct 20, 2026
    MemberAuthor

    There is documentation on our current files in https://isiscomputinggroup.github.io/ibex_developers_manual/Nexus.html now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

filewriter

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions