Repository navigation
Run plan by default, lean replicate outputs and a persistent Fortran qdyn engine - #143
Draft
David-Araripe wants to merge 272 commits into
Draft
David-Araripe wants to merge 272 commits into
David-Araripe wants to merge 272 commits into
Conversation
Bonded force to gpu
Feature/qgpu temperature
Feature/qgpu radix water
move calc pshell force to gpu
move restrseq force to gpu
Add src/core/README.md covering the layout, the CMake build of qdyn, running qdyn, and installing and using the qgpu Python package. qdyn no longer reads CSV input, so drop the --csv-dir option from the command parser, the --qgpu-input csv mode from test/runTEST.py, and the --qgpu-input row from test/QGPU_TESTING.md. qdyn now takes only `qdyn [--gpu] input.inp`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/dev (74f4739) into feature/qgpu_support_python_api (ab84e25). The only conflicts were 11 modify/delete files that dev removed in its cleanup (benchmark-old, src/bin launchers, src/Qgpu/calc.py and functions.py, and the newbin QligFEP.py and qfep.py). They are resolved in favour of dev's deletion; no live code imports them. The orphaned benchmark-old/FEP_ligand_binding/2.Q7/.gitignore added on the qgpu branch goes with them.
FortranEngine has the Engine interface (update_config, run, get_state, set_state) but starts qdyn or qdynp, optionally under an MPI launcher, once per run(). It writes the .inp file of each run itself and carries the state forward in Q restart files. Config and Config.from_inp() now work without a native extension: unset settings stay None and Fortran Q applies its own defaults. QGPU_PRECISIONS=none installs the package without CUDA. Behaviour with a native extension installed is unchanged. Tests compare three stages run through the API with the same stages run from hand-written .inp files, serial and under mpirun: the restart and energy files are byte-identical.
The FEP file overwrites nstates after EQ was sized from the [lambdas] of the input file. With fewer lambdas than states qdyn ran anyway and read the missing lambdas from outside the array.
As the CUDA engine does, so that python workflow.py > log gives one log of all stages on both. Each run also keeps its own log in the work folder.
qdyn writes the name into trajectory headers, so trajectories now match a run from .inp files byte for byte too.
Cluster settings move from the CLUSTER_DICT dictionaries in settings.py to layered YAML: packaged defaults, a per-user file (profiles only) and a project qligfep.yaml (profiles and protocols). The historical clusters ship as packaged profiles and CLUSTER_DICT stays as a read-only view over them, so existing -c names keep working. Adds the qligfep_config command (init, show, check, build-info) and lets QresFEP accept user-defined Slurm profiles.
qligfep and setupFEP resolve an execution profile and a protocol; -c becomes optional once default_profile is set, and --protocol, --config and --q-bin are new. Command-line flags still win over the protocol. The run scripts no longer depend on Slurm variables: the same script runs as a Slurm array task or locally, and only FEP_submit.sh differs. This enables local non-equilibrium runs with an MPI profile. Site quirks (partitions, CSB memory directive) come from the profile instead of being hardcoded. Protocol stages can set their length (time or steps) and any qdyn input keyword. The resolved profile and protocol are written to inputfiles/resolved_config.yaml.
CONFIGURATION.md is a temporary home for the guide until the documentation branch is merged.
build-info records only the modules the user loaded (Lmod stackDepth 0) instead of every dependency, which had produced 18 module load lines for one OpenMPI module. init no longer writes the login shell's modules into a profile when no build manifest exists. For local execution neq_cores: auto now stays at the profile's ntasks instead of growing to the switch count, and the non-equilibrium core message is logged once per setup.
qligfep_config init no longer writes the only account sacctmgr reports into the profile. Without -A Slurm charges the user's default account; the available accounts are listed in a comment and a note instead.
The packaged profiles no longer carry the project accounts used during development. qligfep and setupFEP accept --account and --partition, which override the profile's slurm settings; without an account jobs are submitted without -A.
Without -A the scripts submit to the user's default Slurm account.
Nothing in the test suite reads it, and the workflow its README describes no longer exists in the repository.
A run now leaves only the restart file the next run starts from: its .inp and log are removed, and so is the restart it replaces. The engine is a context manager, and close() removes the last restart and the links. keep_files=True keeps everything, and the files of a failed run are always kept. On the Tyk2 protein leg the engine folder held 129 MB of duplicates per replicate.
…g.extra FortranEngine writes them into its .inp; the CUDA engine rejects them by name. Config.from_sections() builds a Config from .inp content given as data, and the pure-Python .inp reader keeps unknown keywords instead of dropping them.
With --run-plan, qligfep and setupFEP write inputfiles/run_plan.yaml instead of one .inp file per stage and lambda window. The run script calls qligfep_run once per replicate; it fills in the temperature and seed and runs every stage through the qgpu engine API with one engine. The plan is built from the rendered .inp text, so both routes describe the same simulation: tests compare the engine settings of every stage and the output bytes of short runs. Protocols can list their lambda windows with equilibrium.lambdas. The two production writers are merged into one that also handles lists that are not symmetric around 0.5; the .inp files and run script of existing setups are unchanged.
…e plan The job called the interpreter's installed QligFEP, which on Habrok is another checkout without the runner. The script now puts the checkout that wrote the plan first on PYTHONPATH.
No flag now gives inputfiles/run_plan.yaml; --inp-files selects one .inp file per stage. --run-plan is still accepted and does nothing. Non-equilibrium setups write .inp files without needing a flag, instead of refusing the plan. setupFEP forwards --inp-files, and qligfep_analyze counts lambda windows from the plan when there are no md*.inp files. Tests that read stage .inp files ask for them with --inp-files.
FortranEngine.get_energies() gives the energy records of the latest run() as qgpu.fortran.Energies (NumPy structured arrays named after q_energies and offdiag_save of nrgy.f90), with or without an energy_file. Config accepts energy > 0 without energy_file; the CUDA Engine refuses that until its bindings can return energies. qgpu.fortran reads and writes .en files losslessly.
…plicate The plan runner writes replicate.log instead of a log per stage, keeps restart states in memory (a temporary restart file only when the engine is replaced), and writes the energies of every window to energies.parquet. With --qfep it runs qfep on .en files exported to a temporary folder, so qfep.inp is no longer copied and the run script trusts the runner's exit status instead of grepping stage logs. On a failure it keeps the log, the state the failed stage started from and the finished windows. --keep-stage-files writes the per-stage .log, .re and .en as before. qligfep_energies lists, exports (byte for byte) and runs qfep on an energy file. pyarrow becomes a dependency.
qdyn/qdynp --serve <folder> reads stage inputs from stdin (run ... @EnD, save, load, quit) and runs each through the same setup path a new process takes, starting from the state the previous stage left in memory instead of a restart file; [files] final becomes optional. Between stages every module variable is reset to its program-start value by <module>_reset_state routines generated by src/q6/tools/reset_state.py, whose --check is tested. The SAVEd first-call state of nbqplist, nbqplis2 and randm moved to module variables so it is reset too. A failing stage writes the state it started from to <folder>/recovery.re. Without --serve qdyn is unchanged.
FortranEngine(persistent='auto') starts qdyn --serve once and sends every run() to it, so the state stays in memory and no restart file or process start comes between runs; persistent=False keeps one process per run and 'auto' falls back to it for a qdyn without --serve. After a failed run the next run() starts a new process from the state the failed run started from. The engine tests run in both modes; the run plan tests check one qdyn start per replicate.
Engine.run() now returns a qgpu.RunResult with the energies of that run: per-state Q energies (unweighted, as in the .en file) and Upot/Ukin/Utot for each `energy` frame, plus the same values at the final coordinates. This is what BAR/MBAR, adaptive sampling and lambda replica exchange need, without reading .en files back. A new EnergyRecorder output keeps the frames in memory on the same steps as the .en file, so it adds no GPU downloads. qdyn ignores it. Document RunResult in src/core/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to feature/persistent-qdyn Engine.run() returns the coworker's RunResult; Engine.get_energies() keeps refusing, now pointing to RunResult for the Q energy totals and to energy_file for the full .en records.
The same result type as the CUDA Engine.run(): q_energy exactly from the .en records, upot/ukin/utot and the final_* values from qdyn's printed energy summaries (two decimals; NaN for frames qdyn did not summarise). Both the per-run and the persistent mode return it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Windowed FEP replicates run from one run plan, leave a handful of files instead of ~530, and run every stage in one
qdynpprocess. Results stay byte-identical to the conventional route.Configuration: profiles and protocols (
490dda51…963aaa30)inp:overrides) drive the FEP setup and the generated job scripts. Documented inCONFIGURATION.md.Part 0: the run plan is the default (
072ce680,c9f86cf8,0a3a6b2d)qligfepwritesrun_plan.yaml. One Python runner (QligFEP.run_plan) runs all stages of a replicate throughqgpu.FortranEngine.--inp-filesselects the old.inproute.--run-planis accepted as a hidden no-op. NEQ FEP still uses.inpfiles.Configfield travel inConfig.extra(1558fbf8).Parts 1–2: lean outputs and energies in memory (
fcd793a0,7cfbd4ee,2a5dc5d1)FortranEngine.get_energies()returns a run's energies in memory.energies.parquet(all windows, written once at the end and atomically),qfep.out,energies.csv,replicate.log, and trajectories if requested. No.inp,.re,.enor per-stage logs.qligfep_energiesexportsenergies.parquetto.enfiles, byte for byte.--qfep).--productionkeeps 4 files per replicate.Part 3: one persistent
qdynprocess per replicate (ee5de273Fortran,d04dcbb9Python)qdyn[p] --serve <folder>reads stages one after another. Between stages it resets module state and keeps coordinates, velocities and polarisation in memory, then repeats the whole setup. Every stage therefore takes the same code path as a fresh process.src/q6/tools/reset_state.pygenerates the<module>_reset_stateroutines.--check(tested) verifies they are current.dieduring a stage writesrecovery.re, so the engine can restart from the last good state.FortranEngine(persistent="auto"|True|False). One-process-per-stage stays available as the reference and fallback.RunResultfromFortranEngine.run()(fe6a9848)q_energyis exact (from the.enrecords).upot/ukin/utot/final_*are parsed from qdyn's log (2 decimals, NaN for frames without a printed summary).Verification on Habrok
Tyk2 ejm_31→ejm_45, both legs, set up without route flags, compared with the conventional replicates in
/scratch/p289890/python-api-fortran-tyk2/:.enexported fromenergies.parquet.dcd,qfep.out,energies.csv*Slowed down by other jobs sharing the node, not by the engine. To check, one Slurm job ran the same short protein replicate (14 stages) four times in a row on the same 8 cores, alternating two ways of running it:
qdynpfor every stage. It reads the topology and the previous stage's restart file, simulates, writes a restart file and exits. That is 14 program starts.qdynponce and sends it the 14 stages. Coordinates and velocities stay in memory, with no restart files. That is 1 program start.Runs 1 and 2 of each kind are repeats. The order per-stage-1, persistent-1, persistent-2, per-stage-2 means a change in node load during the job affects both kinds about equally. The two per-stage repeats differing by 6 s shows the timing noise.
Columns:
mpirun/qdynp, reading inputs, restart files, and Python.All four runs gave byte-identical energies and trajectories. The MD loop takes the same time in both modes, so the persistent engine does not slow the simulation. Skipping 13 program starts saves ~10 s per replicate (~0.7 s per stage). The node was shared, not exclusive.
Also checked:
inp:override (force_rmsviaConfig.extra) gives byte-identical results on the plan route and the.inproute.qdynp --serverun is identical to two processes.Testing
The engine tests run in both modes (per-stage and persistent) through an autouse fixture.
Not verified
qdyn --serveon Habrok (only locally).--servewith ansrunlauncher (mpi_command). Onlympirunwas tested.RunResult(fe6a9848) and the merge of Feature/qgpu support python api #139 on Habrok (tested locally only).Still written temporarily (in temporary folders, removed on success)
stage.inp, the inputqdyn --servereads, overwritten every stage.run_NNNN.enper window: read into memory and deleted right away.replicate.log.recovery.re, only when a stage fails..enfiles exported for qfep, which reads only.en.Notes for review
src/core/python(documented insrc/core/README.md).RunResultcarried every per-state energy term, the GPU engine could writeenergies.parquettoo.bind(C)entry points,mpirun python). The main work would be turning the 168call diesites inmd.f90into error returns.