Skip to content

Run plan by default, lean replicate outputs and a persistent Fortran qdyn engine - #143

Draft
David-Araripe wants to merge 272 commits into
devfrom
feature/persistent-qdyn
Draft

David-Araripe wants to merge 272 commits into
devfrom
feature/persistent-qdyn

Conversation

@David-Araripe

@David-Araripe David-Araripe commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Draft — depends on #139 and #142. This branch is built on Shen's Python API (feature/qgpu_support_python_api, merged in as 33546770), because FortranEngine implements its Engine/Config API. Until feature/qgpu reaches dev, this PR also shows Shen's GPU commits. Once #142 is in main and main is in dev, the diff shrinks to the ~25 commits listed below. Please review only those for now.

What this changes

Windowed FEP replicates run from one run plan, leave a handful of files instead of ~530, and run every stage in one qdynp process. Results stay byte-identical to the conventional route.

Configuration: profiles and protocols (490dda51 … 963aaa30)

  • YAML execution profiles (cluster, launcher, cores) and protocols (stages, lambdas, inp: overrides) drive the FEP setup and the generated job scripts. Documented in CONFIGURATION.md.
  • No Slurm account is set unless requested.

Part 0: the run plan is the default (072ce680, c9f86cf8, 0a3a6b2d)

  • qligfep writes run_plan.yaml. One Python runner (QligFEP.run_plan) runs all stages of a replicate through qgpu.FortranEngine.
  • --inp-files selects the old .inp route. --run-plan is accepted as a hidden no-op. NEQ FEP still uses .inp files.
  • qdyn keywords without a Config field travel in Config.extra (1558fbf8).

Parts 1–2: lean outputs and energies in memory (fcd793a0, 7cfbd4ee, 2a5dc5d1)

  • FortranEngine.get_energies() returns a run's energies in memory.
  • A replicate folder holds energies.parquet (all windows, written once at the end and atomically), qfep.out, energies.csv, replicate.log, and trajectories if requested. No .inp, .re, .en or per-stage logs.
  • qligfep_energies exports energies.parquet to .en files, byte for byte.
  • The runner runs qfep itself (--qfep). --production keeps 4 files per replicate.

Part 3: one persistent qdyn process per replicate (ee5de273 Fortran, d04dcbb9 Python)

  • qdyn[p] --serve <folder> reads stages one after another. Between stages it resets module state and keeps coordinates, velocities and polarisation in memory, then repeats the whole setup. Every stage therefore takes the same code path as a fresh process.
  • src/q6/tools/reset_state.py generates the <module>_reset_state routines. --check (tested) verifies they are current.
  • die during a stage writes recovery.re, so the engine can restart from the last good state.
  • FortranEngine(persistent="auto"|True|False). One-process-per-stage stays available as the reference and fallback.

RunResult from FortranEngine.run() (fe6a9848)

  • Same return type as the GPU engine (Feature/qgpu support python api #139). q_energy is exact (from the .en records). upot/ukin/utot/final_* are parsed from qdyn's log (2 decimals, NaN for frames without a printed summary).

Verification on Habrok

Tyk2 ejm_31→ejm_45, both legs, set up without route flags, compared with the conventional replicates in /scratch/p289890/python-api-fortran-tyk2/:

Water Protein
101 .en exported from energies.parquet identical identical
106 .dcd, qfep.out, energies.csv identical identical
Normal terminations / nodes 106 / 8 106 / 8
Files in replicate folder 110 (~480 MB) vs 530 110 vs 530
Wall, parts 0–2 1h15 1h25
Wall, persistent (part 3) 1h10 1h56*
qdynp starts / restart files read 1 / 0 1 / 0

*Slowed down by other jobs sharing the node, not by the engine. To check, one Slurm job ran the same short protein replicate (14 stages) four times in a row on the same 8 cores, alternating two ways of running it:

  • per-stage: the conventional way. Python starts a new qdynp for every stage. It reads the topology and the previous stage's restart file, simulates, writes a restart file and exits. That is 14 program starts.
  • persistent: the new part 3. Python starts qdynp once and sends it the 14 stages. Coordinates and velocities stay in memory, with no restart files. That is 1 program start.

Runs 1 and 2 of each kind are repeats. The order per-stage-1, persistent-1, persistent-2, per-stage-2 means a change in node load during the job affects both kinds about equally. The two per-stage repeats differing by 6 s shows the timing noise.

Columns:

  • Wall: total time for the replicate.
  • qdyn MD loop: time qdyn spends simulating (summed over stages).
  • Overhead: wall minus MD loop. This is starting mpirun/qdynp, reading inputs, restart files, and Python.
Run Wall (s) qdyn MD loop (s) Overhead (s) qdyn starts
per-stage-1 83.4 66.5 16.9 14
persistent-1 69.9 65.8 4.1 1
persistent-2 70.2 66.0 4.2 1
per-stage-2 77.9 66.8 11.1 14

All four runs gave byte-identical energies and trajectories. The MD loop takes the same time in both modes, so the persistent engine does not slow the simulation. Skipping 13 program starts saves ~10 s per replicate (~0.7 s per stage). The node was shared, not exclusive.

Also checked:

  • A protocol with explicit lambdas and an inp: override (force_rms via Config.extra) gives byte-identical results on the plan route and the .inp route.
  • A two-stage qdynp --serve run is identical to two processes.

Testing

export PYTHONPATH=$PWD/src:$PWD/src/core/python/src
python -m pytest src/core/python/test/test_fortran_engine.py test/q6/test_lambda_count.py \
  test/q6/test_persistent_qdyn.py test/qligfep/test_run_plan.py -q          # 90 passed
python -m pytest test/qligfep -m "not slow" -q \
  --deselect test/qligfep/test_resfep_setup.py                               # 790 passed

The engine tests run in both modes (per-stage and persistent) through an autouse fixture.

Not verified

  • Serial qdyn --serve on Habrok (only locally).
  • --serve with an srun launcher (mpi_command). Only mpirun was tested.
  • RunResult (fe6a9848) and the merge of Feature/qgpu support python api #139 on Habrok (tested locally only).

Still written temporarily (in temporary folders, removed on success)

  • stage.inp, the input qdyn --serve reads, overwritten every stage.
  • One run_NNNN.en per window: read into memory and deleted right away.
  • Per-stage logs, deleted after being copied into replicate.log.
  • recovery.re, only when a stage fails.
  • A restart file only when a new engine must start (topology change or branch point).
  • .en files exported for qfep, which reads only .en.

Notes for review

  • qgpu is not added to the QligFEP wheel, to avoid clashing with the qgpu package. Install it from src/core/python (documented in src/core/README.md).
  • Suggestion for Feature/qgpu support python api #139: if RunResult carried every per-state energy term, the GPU engine could write energies.parquet too.
  • Out of scope, described for later: Fortran Q as a library (bind(C) entry points, mpirun python). The main work would be turning the 168 call die sites in md.f90 into error returns.

goodstudyqaq and others added 30 commits November 13, 2026 14:00
Add src/core/README.md covering the layout, the CMake build of qdyn,
running qdyn, and installing and using the qgpu Python package.

qdyn no longer reads CSV input, so drop the --csv-dir option from the
command parser, the --qgpu-input csv mode from test/runTEST.py, and the
--qgpu-input row from test/QGPU_TESTING.md. qdyn now takes only
`qdyn [--gpu] input.inp`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/dev (74f4739) into feature/qgpu_support_python_api (ab84e25).

The only conflicts were 11 modify/delete files that dev removed in its cleanup (benchmark-old, src/bin launchers, src/Qgpu/calc.py and functions.py, and the newbin QligFEP.py and qfep.py). They are resolved in favour of dev's deletion; no live code imports them. The orphaned benchmark-old/FEP_ligand_binding/2.Q7/.gitignore added on the qgpu branch goes with them.
FortranEngine has the Engine interface (update_config, run, get_state, set_state) but starts qdyn or qdynp, optionally under an MPI launcher, once per run(). It writes the .inp file of each run itself and carries the state forward in Q restart files.

Config and Config.from_inp() now work without a native extension: unset settings stay None and Fortran Q applies its own defaults. QGPU_PRECISIONS=none installs the package without CUDA. Behaviour with a native extension installed is unchanged.

Tests compare three stages run through the API with the same stages run from hand-written .inp files, serial and under mpirun: the restart and energy files are byte-identical.
The FEP file overwrites nstates after EQ was sized from the [lambdas] of the input file. With fewer lambdas than states qdyn ran anyway and read the missing lambdas from outside the array.
As the CUDA engine does, so that python workflow.py > log gives one log of all stages on both. Each run also keeps its own log in the work folder.
qdyn writes the name into trajectory headers, so trajectories now match a run from .inp files byte for byte too.
Cluster settings move from the CLUSTER_DICT dictionaries in settings.py to layered YAML: packaged defaults, a per-user file (profiles only) and a project qligfep.yaml (profiles and protocols). The historical clusters ship as packaged profiles and CLUSTER_DICT stays as a read-only view over them, so existing -c names keep working.

Adds the qligfep_config command (init, show, check, build-info) and lets QresFEP accept user-defined Slurm profiles.
qligfep and setupFEP resolve an execution profile and a protocol; -c becomes optional once default_profile is set, and --protocol, --config and --q-bin are new. Command-line flags still win over the protocol.

The run scripts no longer depend on Slurm variables: the same script runs as a Slurm array task or locally, and only FEP_submit.sh differs. This enables local non-equilibrium runs with an MPI profile. Site quirks (partitions, CSB memory directive) come from the profile instead of being hardcoded.

Protocol stages can set their length (time or steps) and any qdyn input keyword. The resolved profile and protocol are written to inputfiles/resolved_config.yaml.
CONFIGURATION.md is a temporary home for the guide until the documentation branch is merged.
build-info records only the modules the user loaded (Lmod stackDepth 0) instead of every dependency, which had produced 18 module load lines for one OpenMPI module. init no longer writes the login shell's modules into a profile when no build manifest exists.

For local execution neq_cores: auto now stays at the profile's ntasks instead of growing to the switch count, and the non-equilibrium core message is logged once per setup.
qligfep_config init no longer writes the only account sacctmgr reports into the profile. Without -A Slurm charges the user's default account; the available accounts are listed in a comment and a note instead.
The packaged profiles no longer carry the project accounts used during development. qligfep and setupFEP accept --account and --partition, which override the profile's slurm settings; without an account jobs are submitted without -A.
Without -A the scripts submit to the user's default Slurm account.
Nothing in the test suite reads it, and the workflow its README describes no longer exists in the repository.
A run now leaves only the restart file the next run starts from: its .inp and log are removed, and so is the restart it replaces. The engine is a context manager, and close() removes the last restart and the links. keep_files=True keeps everything, and the files of a failed run are always kept.

On the Tyk2 protein leg the engine folder held 129 MB of duplicates per replicate.
…g.extra

FortranEngine writes them into its .inp; the CUDA engine rejects them by name. Config.from_sections() builds a Config from .inp content given as data, and the pure-Python .inp reader keeps unknown keywords instead of dropping them.
With --run-plan, qligfep and setupFEP write inputfiles/run_plan.yaml instead of one .inp file per stage and lambda window. The run script calls qligfep_run once per replicate; it fills in the temperature and seed and runs every stage through the qgpu engine API with one engine. The plan is built from the rendered .inp text, so both routes describe the same simulation: tests compare the engine settings of every stage and the output bytes of short runs.

Protocols can list their lambda windows with equilibrium.lambdas. The two production writers are merged into one that also handles lists that are not symmetric around 0.5; the .inp files and run script of existing setups are unchanged.
…e plan

The job called the interpreter's installed QligFEP, which on Habrok is another checkout without the runner. The script now puts the checkout that wrote the plan first on PYTHONPATH.
No flag now gives inputfiles/run_plan.yaml; --inp-files selects one .inp file per stage. --run-plan is still accepted and does nothing. Non-equilibrium setups write .inp files without needing a flag, instead of refusing the plan. setupFEP forwards --inp-files, and qligfep_analyze counts lambda windows from the plan when there are no md*.inp files. Tests that read stage .inp files ask for them with --inp-files.
FortranEngine.get_energies() gives the energy records of the latest run() as qgpu.fortran.Energies (NumPy structured arrays named after q_energies and offdiag_save of nrgy.f90), with or without an energy_file. Config accepts energy > 0 without energy_file; the CUDA Engine refuses that until its bindings can return energies. qgpu.fortran reads and writes .en files losslessly.
…plicate

The plan runner writes replicate.log instead of a log per stage, keeps restart states in memory (a temporary restart file only when the engine is replaced), and writes the energies of every window to energies.parquet. With --qfep it runs qfep on .en files exported to a temporary folder, so qfep.inp is no longer copied and the run script trusts the runner's exit status instead of grepping stage logs. On a failure it keeps the log, the state the failed stage started from and the finished windows. --keep-stage-files writes the per-stage .log, .re and .en as before. qligfep_energies lists, exports (byte for byte) and runs qfep on an energy file. pyarrow becomes a dependency.
qdyn/qdynp --serve <folder> reads stage inputs from stdin (run ... @EnD, save, load, quit) and runs each through the same setup path a new process takes, starting from the state the previous stage left in memory instead of a restart file; [files] final becomes optional. Between stages every module variable is reset to its program-start value by <module>_reset_state routines generated by src/q6/tools/reset_state.py, whose --check is tested. The SAVEd first-call state of nbqplist, nbqplis2 and randm moved to module variables so it is reset too. A failing stage writes the state it started from to <folder>/recovery.re. Without --serve qdyn is unchanged.
FortranEngine(persistent='auto') starts qdyn --serve once and sends every run() to it, so the state stays in memory and no restart file or process start comes between runs; persistent=False keeps one process per run and 'auto' falls back to it for a qdyn without --serve. After a failed run the next run() starts a new process from the state the failed run started from. The engine tests run in both modes; the run plan tests check one qdyn start per replicate.
Engine.run() now returns a qgpu.RunResult with the energies of that run:
per-state Q energies (unweighted, as in the .en file) and Upot/Ukin/Utot
for each `energy` frame, plus the same values at the final coordinates.
This is what BAR/MBAR, adaptive sampling and lambda replica exchange need,
without reading .en files back.

A new EnergyRecorder output keeps the frames in memory on the same steps
as the .en file, so it adds no GPU downloads. qdyn ignores it.

Document RunResult in src/core/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to feature/persistent-qdyn

Engine.run() returns the coworker's RunResult; Engine.get_energies() keeps refusing, now pointing to RunResult for the Q energy totals and to energy_file for the full .en records.
The same result type as the CUDA Engine.run(): q_energy exactly from the .en records, upot/ukin/utot and the final_* values from qdyn's printed energy summaries (two decimals; NaN for frames qdyn did not summarise). Both the per-run and the persistent mode return it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants