MD-tools v0.5.0¶

Released 2026-09-10, tagged openmm-v0.5.0 on dev. This records what changed, what was verified, and — in Known limitations immediately below — what it does not do. No PyPI upload.

Full suite at the tagged commit: 2056 passed, 3 skipped. The three skips are one conditional branch that wrote no state CSV and two opt-in artefacts; none hides a missing dependency.

Known limitations¶

These are not defects to be found later. They are documented properties of this release, and the first one is the one to read before planning a study.

A build is not reproducible from its inputs, on the ligand and peptide-like routes. AM1-BCC partial charges differ between two builds of the same .smi, same configuration, same machine. Measured on a 43-atom cyclo(Gly-Asp-Arg): 41 of 43 atoms differed, one moving from -0.63047675 to -0.71247675 e, with the builds run sequentially. It is molecule-dependent, not universal — a 79-atom c(RGDfV) reproduced exactly, 0 of 79 atoms differing — so the absence of a difference is luck rather than a guarantee for any particular input.

The cause is in the installed source and is understood: build-top seeds its ETKDG embedding and refuses an unseeded one, but the OpenFF toolkit then discards that conformer — AmberToolsToolkitWrapper.assign_partial_charges clears _conformers and regenerates when use_conformers is not passed, and md_tools.openmm.system does not pass it. The charges therefore come from a conformer that is neither the one in the build provenance nor the same one twice.

What this means in practice. Nothing is wrong within a single build: the charges are real AM1-BCC for the conformer used, and every dataset this produces is internally consistent and self-describing. What cannot be done is regenerate a build from its inputs, or difference two builds expecting to see only an intended change. Keep the built system.xml and topology; do not plan to rebuild them.

For method development this is not a workaround but the intended shape: a project comparing sampling or analysis efficiency against reference simulations loads the SAME system.xml and topology those simulations used — kept with the reference data under $MD_DATA/common/ — rather than building its own. Two arms cannot then differ in their charges, because there is only one build. docs/scientific-defaults.md §14 states it where someone planning a comparison will meet it.

Protein routes are unaffected. ff14SB and ff19SB assign library charges, which are deterministic. This is the small-molecule charge route only.

It is not fixed here deliberately: passing use_conformers= would change the charges of every future build, which is a scientific decision rather than a bug fix. Tracked as entry 6 in docs/backlog.md, with the trigger that should reopen it.

The rest. docs/backlog.md carries the remaining open entries, each with its impact and a concrete trigger. None has been shown to corrupt, discard or misreport scientific output, or to prevent recovery of an interrupted run.


starting commit	`64f4ed0e394ddeec47d3970cd0e4b3c941b71e2a`
preservation tag	`pre-v0.5-doc-cleanup` → annotated `9d5f36d` → `64f4ed0`, verified on `origin`
platform	Linux 6.8, Python 3.12.14, OpenMM 8.6, CUDA (1 × A5000, 8 × RTX 3080)

Everything removed below is recoverable, from one of two points. Material that existed before the cleanup is reachable from the preservation tag. The one exception is reports/explicit_solvent/20260815_ff19sb_peptide_route/, which was added after the tag was made and is therefore not in it; it is reachable from commit 1690140fc77eea95137654b1e107cf9cb6100613. Git history is the archive; this note is the summary, not a chronological dump of every edit.

What MD-tools is now¶

One executable, md-openmm, and four public commands:

md-openmm build-top        a structure        -> built.xml + built.pdb + built.log
md-openmm build-md         a protocol config  -> run scripts, .in files and run.sh
md-openmm md-run           an Amber-like .in  -> a stage, a ladder, or AIS switching paths
md-openmm data-register    a finished tree    -> a verified dataset under $MD_DATA

AIS is a build-md protocol and an md-run input, not a command of its own; md-run is a subcommand, not a second executable. There is one dataset contract (v2), one configuration layout, and one small set of documentation.

Configuration is browsable¶

The repository root held a mode-120000 symlink, so GitHub rendered configs as a symlink blob. It is now an ordinary tree, with exactly one copy of each file:

configs/machine/user.config.example
configs/sys/build-top.config
configs/md/{cMD,REST2,rREST2,AIS}.config

They ship as wheel data files under <prefix>/share/md-tools/configs/, and md_tools.configs.example_root() asks the installed distribution's own metadata where they went. No symlink, no package-data mirror, no second hand-edited copy.

The examples are no longer generated from the schemas. The guarantee that replaced byte-equality is stronger: every example resolves, through the real resolver, to the model's own defaults. An example can be reworded freely; it cannot describe a default the code does not apply.

AIS through `build-md`¶

protocol: AIS in the same strict resolver as cMD, REST2 and rREST2.

tau is the only public, persisted coordinate. s and sqrt(s) are derived inside the scaler and are no longer written to the observation table — a second stored coordinate is a second thing that can disagree with the first.
Every length is an exact integer step count. switching_duration_ps became switching_steps and observation_interval_steps; logs derive ps for the reader. Three divisibility rules are enforced, each refused with the arithmetic that would fix it.
Both endpoints are included, and the schedule states that observation 0 precedes all work: it is the source configuration before any parameter change and before any propagation, so its cumulative work is exactly zero rather than nearly zero.
Unchanged: the work convention ΔW_j = U(τ_{j+1}, x_j) − U(τ_j, x_j), fixed volume with a barostat refused, independent per-path seeds, completed paths never appended to, and outputs validated before completion is declared.
AIS is not part of the cMD stage chain. It consumes an ensemble that already exists, so run.sh takes the source as a required third argument and refuses without it.

Contract v2¶

	v1	v2
canonical path	`{namespace}/{yyyy-mm}/{dataset_name}`	`{year}/{project_name}/{data_name}`
shared datasets	reserved `baseline/` namespace	`{year}/common/{project}/{data}`
dated segment	creation month	completion year, checked against the records
extensions	operational record	ported, plus a required checkpoint digest

v2 is a new model, not a widened v1: one model accepting both shapes is the state in which a wrong path goes undetected. A v1 manifest fails on schema_version, naming the difference.

Extensions. Completed and archived datasets are immutable; extending one creates a new dataset with a new id that records the parent and component; new output goes to a component the new dataset owns, never a link into the parent; lengths are mutually consistent; status and timestamps are ordered; a completed extension cannot finish after the dataset containing it was declared complete.

v2 additionally requires source_checkpoint_sha256, and source_checkpoint_copied for an in-place extension — which writes into the very component it restarted from, so the run can overwrite its own restart point. Without the copy the join stops being reproducible the moment the continuation passes it, and nothing in the record would show that it had.

Registration validates the record, cross-checks it against dataset.yaml, and re-hashes the restart checkpoint if it travelled with the data. It never opens the parent.

schemas/dataset-v2.0.schema.json and schemas/extension-v2.0.schema.json are generated from their pydantic models and drift-checked by tests.

Deliberate removals¶

removed	lines	why
`openmm/sysgen.py`	820	the `sys-gen` route
`openmm/mdgen.py`	613	the `md-gen` route
`openmm/md_data_contract.py`	495	contract v1
`openmm/simple.py`, `emit.py`	1 684	the `setup` route and its launchers
`cli/md_template.py`, `install/`	1 371	the environment installer; replaced by `pip install`
`tests/test_stage_layout.py` and 5 others	~2 500	the generated-project layout they tested
`docs/{examples,implementation,journal,journals,legacy}/`	—	superseded documentation
`claudecode-instructions/`	36 files	executed instruction records
`templates/{ais_run,cmd_run,rest2_run,rest2_equilibrate,stage_run,runtime_record,preflight}.py` and `{extend,run_all,stage_run}.sh`	3 740	the scripts the retired route generated
`.github/workflows/release.yml`	118	it invoked four commands that no longer exist

The nine builder functions build-top and the replica runtime actually use were extracted verbatim into openmm/builders.py before sysgen.py was deleted.

No test was removed for failing. Each assertion was classified as migrated, obsolete with a named deleted feature, or still blocking. Force-field provenance moved into the build record when forcefield.json went; the checksum-manifest properties moved to registry.inventory; the AIS switcher, omega and work-telescoping tests were rebuilt on build-top output; the barostat trace now reads in_system/active/frequency_steps from the stage record.

Two migrations found guarantees with no new home and closed them rather than dropping the test:

a stage did not record the -c parent state it consumed. It now records it with its SHA-256, and registration re-hashes recorded inputs, so a parent rewritten after a child consumed it is caught before it becomes false ancestry;
nothing refused a large timestep on unrepartitioned hydrogens. It is now checked against the masses serialised in the System, which at run time is a fact rather than a request.

One release blocker found and fixed¶

.gitignore carried a bare build/ rule for Python artifacts. A bare pattern matches a directory of that name at any depth, so it also matched src/md_tools/build/ — the package implementing build-top and build-md.

Those five modules had never been committed. They were absent from origin/dev and from the preservation tag; a clone of this repository did not contain two of the three commands the project exists to provide. Everything passed locally because the files were on the machine that wrote them.

Fixed by anchoring the rules (/build/, /dist/, /data/, /logs/) and committing the modules. A test in test_suite_integrity.py now walks src/, tests/ and configs/ and fails if git check-ignore matches any source file. Verified by cloning dev and building a working wheel from the clone.

One current MD configuration model¶

build-md resolved protocol: with integer step counts, while openmm/defaults.py, openmm/config.py and openmm/stages.py still defined and validated the retired methods: model in duration_ns and switching_duration_ps. resolve_md_config and stage_plan each existed twice, under the same name, and both looked authoritative — which is the failure mode: a reader picks one, and there is no way to tell from the call site which model they are in.

Reachability was established with an import-aware map over src/ and tests/. A name-based map was tried first and gave the wrong answer, because the whole problem is that the two models share names.

removed	why
`openmm/stages.py`	the second `stage_plan`; no importer in `src/` at all
`openmm/config.resolve_md_config` + `_required`, `_check_ais`, `_check_barostat_frequency`, `_check_tau`, `_check_timestep`	reachable only from the retired resolver
`openmm/config.check_timestep_against_masses`	a configuration-level guess at what `runtime/stage.py` decides from the masses serialised in the built System
`openmm/config.{load_yaml, sha256_of_document}`	no caller
`openmm/defaults.{md_defaults, ais_defaults, default_document, canonical_method, METHODS, AIS_TAU_START, AIS_TAU_END, AIS_OBSERVATIONS, EXPLICIT_PROTEIN_FORCEFIELD}`	the retired MD half of a mixed module
`templates/md_stages.production_stage_document`	the last code reading the retired duration vocabulary; no caller
`reports/explicit_solvent/20260815_ff19sb_peptide_route/`	five raw logs, 20 KB, referenced by no test, document, CI job or packaging rule

split / renamed	to
`openmm/config.py`	`openmm/system_config.py` — the `build-top` system resolver, which sat beside `build/md.py` sharing a function name with it
`openmm/defaults.py`	`openmm/system_defaults.py` — `build-top` defaults only
`write_yaml`	`openmm/yaml_io.py` — the runtime importing a module called `config` to write a reservoir declaration misdescribed what depends on what
`templates/openmm_md.py`	`templates/replica_executor.py` — an internal module may not be named after a retired executable
`tests/test_openmm_md_interface.py`	`tests/test_replica_executor_interface.py`

Retained deliberately. Everything under templates/ the runtime reaches; resolve_sys_config and its three physical-pairing checks; openff_resource; sys_defaults; dataset_defaults; and the scientific constants — including CONSERVATIVE_PADDING_NM and the HMR pair, which no src module imports but which live tests assert the shipped example against. Nothing was removed for living under templates/.

Two duplicates found on the way. ConfigError was defined in both openmm/config.py and build/strict.py; the CLI catches the build.strict one, so a refusal raised by builders.py would have exited 1 with a bare exception instead of 2 with a clean message. Not reachable today — forcefield.water is not user-settable and the empty-SMILES case is caught earlier, both confirmed by running the command — but one reachable raise away, so builders.py now raises what the CLI handles. And barostat_interval_steps defaulted to a literal 25 in build/md.py while DEFAULT_BAROSTAT_FREQUENCY_STEPS declared 25 elsewhere; build/md.py imports it now, which is what its test already claimed.

Tests were classified, not deleted. The barostat, tau, timestep and AIS source-tau assertions moved to the live resolver. source_tau is gone as a user-declared field, and that is an improvement the test now states: a user restating the source's tau could restate it wrongly, and configuration had no way to tell, so runtime/ais.py checks the source's own record instead. The 0.3.x provenance fixture keeps methods:/duration_ns and says in its docstring that this is historical on-disk data a search is meant to find.

Retired command names: where they may still appear¶

Every occurrence of the eight retired names was classified. All that remain are in two categories, which are how the guarantee is kept rather than violations of it:

explicit negative assertions — tests/test_cli.py, test_wheel.py, test_own_replica_exchange.py, test_platform_policy.py, the CI job's must-fail loops, README.md and CLAUDE.md;
labelled history — comments and docstrings that say what a retired route was replaced by, in tests/conftest.py, test_fair_provenance.py, test_md_project_contract_gates.py, test_ais_build_md.py, docs/README.md and docs/replica-exchange.md.

Three occurrences claimed a retired command was current and were rewritten; one module and one test file were renamed. CLAUDE.md's rule was too narrow — "outside a release note … it is stale text" made the negative tests look like violations — and now names both legitimate places explicitly.

CI¶

.github/workflows/release.yml was not evidence for this package: it invoked show-default, sys-config, sys-gen and md-gen, none of which exist. It is replaced by .github/workflows/ci.yml, which runs on pull requests, on pushes to dev and main, and on manual dispatch — so it speaks for the branch under review rather than only for a tag. Tag-driven validation moves to v*; openmm-v* named this package under its former name.

The job builds the wheel, installs it with no source tree importable, and then runs every interface check from a directory outside the checkout — proven, not assumed, by asserting md_tools resolves inside site-packages. It validates the four commands, refuses the retired ones, locates and resolves the shipped examples through distribution metadata, generates every protocol and compiles what is written, compares both exported schemas against their models, and runs the non-GPU lane with a skip treated as a failure.

Two md-run checks were added to it, both of which a runner with no GPU can make: md-run -h must answer and document every documented flag, and -ng outside an MPI launch must be refused. Neither needs a device, and both are exactly the kind of thing that breaks silently.

It runs no dynamics and says so in a job notice. There is no CUDA runner; the GPU lane is local and is the table below.

Its first run failed, and finding that was the point. Run 33523698117 on dev: every packaging and interface step passed on a real runner — the wheel, the commands, the refused subcommands, the shipped examples, all six bundles, both schemas — and the test lane failed on 8 skips. cpptraj and Amber's reference rem.log were being read from absolute paths under one workstation's software tree, so the only tests that check our Amber trajectory and H-REMD log against the real parser had been skipping on every machine but that one, while the suite reported success. AmberTools is installed on the runner; the paths simply did not exist there.

Fixed by finding cpptraj on PATH and vendoring the 3.6 KB Amber reference into tests/data/, and test_no_test_hard_codes_a_path_on_one_machine now fails if a test locates anything through /home, /data*, /scratch or /Users. The guard was verified against the original defect rather than assumed to work.

Run 33551381960 is green: 664 passed, 0 skipped, in 1 m 42 s — the same count this machine gets, which is the point. Those eight tests now run on the runner instead of reporting success without having executed.

Tests¶

These counts were measured at commit 13c0ed8 and are superseded. They are kept because they were true then and the CI run below can still be opened; they are not a claim about the current tree. A test count that is not attached to a commit stays true-looking across every change that invalidates it, which is the failure this note is now guarded against. The current evidence, with its own exact commit, is 20260903-consolidation-evidence.md.

Every lane below was run on this machine, in this order, on commit 13c0ed8. Environment: Python 3.12.14, OpenMM 8.6, CUDA 9 devices (1 × A5000, 8 × RTX 3080); the GPU lane was pinned to devices 1–4.

lane	command	result	wall
fast	`python -m pytest tests -m "not slow and not gpu"`	925 passed, 118 deselected, 0 skipped	91.5 s
GPU + slow	`python -m pytest tests -m "gpu or slow"`	118 passed, 925 deselected, 0 skipped	396.1 s
CI, GitHub runner	run `33644617673` on `13c0ed8`	success	—
wheel	`python -m build --wheel`, install `--no-deps` into a clean venv	built and installed, `md_tools` resolving inside `site-packages`	—

No test was skipped in either lane. A skip is not neutral here: it means a test that should have run did not, and a green summary would then be reporting coverage the run did not have. CI treats a skip in the non-GPU lane as a failure for the same reason.

Run from a directory outside the checkout, against the installed wheel:

check	result
`md-openmm --version`	`md-tools 0.5.0.dev0`
the four public commands	`build-top`, `build-md`, `md-run`, `data-register` all answer `-h`
the Amber flag contract	every documented `md-run` command parses with `-s` a System and `-x` a trajectory (`test_documented_commands_run.py`)
a full run from outside the checkout	`build-top`, `build-md`, then `./run.sh` completing a cMD chain and a 4-rank REST2 ladder, all on CUDA
generated project portability	no `.py`, `.in`, `run.sh` or `resolved.config` in the generated project names the checkout
retired subcommands	`sys-config`, `sys-gen`, `md-gen`, `setup`, `show-default` all refused
retired console scripts	`openmm-md`, `md-template`, `md-data-register`, `md-data-finish` absent from `PATH`
shipped config examples	6/6 located through distribution metadata at `<prefix>/share/md-tools/configs/`, all resolving
bundle generation	cMD split, cMD implicit, cMD all-in-one, REST2, rREST2, AIS — all generate, every emitted script compiles
AIS bundle shape	carries no equilibration chain
runtime packages	`md_tools.{md,rest2,remd,ais}` ship; `openmm/templates` absent and `md_tools.openmm.templates` not importable
public API	every documented name imports from the wheel
generated scripts	one import and one call each; no definition, argparse, OpenMM import or embedded declaration
`dataset-v2.0.schema.json`	matches the model that validates
`extension-v2.0.schema.json`	matches the model that validates

What the CUDA lane covers¶

GPU tests ran on CUDA. No CPU execution was substituted for CUDA evidence anywhere.

required	covered by
explicit and GBn2 `build-top`	`test_scientific_defaults.py` (ff14SB/TIP3P/PME/1 nm, GBn2/mbondi3 without SASA, ff19SB/OPC)
split and all-in-one cMD	`test_cmd_cuda_smoke.py` — new in this release candidate
REST2 state trajectories, exchanges, `rem.log`, checkpoint, continuation	`test_grouped_run_end_to_end.py`, `test_extension_directory.py`
rREST2 reservoir identity and velocity provenance	`test_rrest2_cuda_smoke.py` — new in this release candidate
AIS source selection, tau schedule, work convention, independent paths, restart, completion	`test_ais.py`, `test_ais_build_md.py`
`md-run` under `mpirun`: a two-rank ladder, a hundred AIS paths, device placement, path identity across worker counts, restart	`test_md_run_mpi_gpu.py` — new

The last two rows were gaps: both properties had unit-level coverage against stubs and no end-to-end run. They are closed rather than reported.

A stable import API, and generated files that are entry points¶

The behaviour used to live in the generated files. That made every generated directory a copy: fixing a defect meant regenerating every project ever produced, and two directories generated a month apart ran different code while claiming the same protocol.

#!/usr/bin/env python
from md_tools.md import run_generated_stage
raise SystemExit(run_generated_stage(__file__, "min"))

Twelve lines including the docstring, and the same shape for md.py, REST2.py, rREST2.py and AIS.py. resolved.config beside the script is the single resolved declaration -- located from __file__ so a directory can be moved, re-validated by the strict resolver at execution, and bound into the checkpoint fingerprint by digest.

package	exports
`md_tools.md`	`PositionalRestraint`, `ReportingConfig`, `run_stage`, `run_generated_stage`, `run_generated_workflow`
`md_tools.rest2`	`REST2Scaler`, `ScalingSelection`
`md_tools.remd`	`REMDRunner`, `NeighborExchangeRule`, `run_remd`, `run_generated_remd`
`md_tools.remd.reservoir`	`ReservoirRefreshRule`
`md_tools.ais`	`run_generated_ais`

Where the code went¶

openmm/templates/ held twenty modules of installed runtime code under a name that said it held templates.

from	to
`templates/replica_*.py`, `exchange_rules`, `rrest2_reservoir`, `source_ensemble`, `rem_log`, `amber_trajectory`, `state_trajectories`	`md_tools/remd/`
`templates/rest2_scaling.py`, `templates/hamiltonian_identity.py`	`md_tools/rest2/`
`templates/md_stages.py`, `templates/phase_space.py`	`md_tools/md/`
`runtime/stage.py`	`md_tools/md/stage.py`
`runtime/replica.py`	`md_tools/remd/generated.py`
`runtime/ais.py`, `openmm/ais.py`	`md_tools/ais/`

Nothing was deleted for living there. Fourteen helpers in md_stages.py were removed after confirming they had no caller anywhere in src/ or tests/.

The bare-import mechanism, and what it was hiding¶

Those modules imported each other by bare name, which resolved only because the runtime put their directory on sys.path. While it was there, remd/statistics.py shadowed the standard library's statistics for anything that imported it — pytorch_lightning failed with module 'statistics' has no attribute 'mean' during this migration, which is how it was found.

There is now no sys.path insertion anywhere in src/, and a test asserts it stays that way.

rREST2 composes rather than forks¶

REST2Scaler + REMDRunner + NeighborExchangeRule + ReservoirRefreshRule. The reservoir rule is generic and lives in md_tools.remd.reservoir rather than in a file named after its first user, so a future reservoir REMD method reuses it instead of copying the driver again.

The exchange mathematics that was duplicated between rest2_scaling.py and replica_engine.py is now in one place, after the two were verified bit-identical over 2000 random inputs.

One implementation of each low-level operation¶

The migration claimed this and did not have it. A check afterwards found sha256_file defined in four modules, derive_seed in three, and resolve_platform in two.

sha256_file is now only in build/record.py — all four produced identical digests, which is precisely why nobody noticed there were four. It lives there because hashing a file needs no OpenMM and that module is the lowest-level of the four.

The other two are different operations that shared a name, which is a worse problem than a duplicate, and they are renamed rather than merged:

openmm.seeds.derive_seed → derive_build_seed. A SHA-256 used while a System is built. The run-time md_tools.md.derive_seed is the repository's older multiplicative hash and cannot be changed to match it: AIS selects its starting frames from it, so a different derivation would silently change which configurations every existing AIS project starts from.
remd.engine.resolve_platform → build_platform. It constructs a Platform from a name already decided. md_tools.md.resolve_platform decides the name and refuses a CPU fallback, because a silent fallback still finishes, still writes a trajectory and still says complete.

A difference worth knowing about, since resolved. Those two platform policies were not the same: a ladder given no explicit platform, on a machine with no visible CUDA device, ran on the CPU — where a stage would refuse. It was recorded in the run context, so such a run was never invisible, but it was not refused, and tightening it was a behaviour change that needed its own validation.

That validation was done in the md-run work. There is now ONE resolver, md_tools.openmm.platform_policy.resolve_platform_request, and stages, ladders and AIS all go through it: CUDA unless --cpu, with no fallback anywhere. Listing the CUDA platform is not the same as having a usable device — conda-forge ships the plugin unconditionally — so the resolver opens and discards a one-particle Context to prove it, which puts the failure before a minimisation rather than after one. The record carries requested_policy, explicit-cpu or default-cuda, so a CPU result can never be mistaken for an unnoticed fallback.

Repository metadata¶

The description is set, and now says what the package is:

Standalone, pip-installable OpenMM tools for building systems, generating MD workflows, and registering MD datasets.

It previously described MD-templates, typo included. Two earlier attempts were refused with HTTP 403: Resource not accessible by personal access token -- a fine-grained PAT without repository-administration permission. That was a permission to grant rather than something to work around, and once granted the same gh repo edit succeeded unchanged.

`md-openmm md-run`: the Amber-like execution interface¶

Four subcommands now -- build-top, build-md, md-run, data-register -- and still exactly one executable. pyproject.toml installs one console script and a test asserts it.

md-openmm md-run -i min.in -p built.pdb -s built.xml -r min.xml -log min.log
mpirun -n 8 md-openmm md-run -ng 8 -i REST2.in -p built.pdb -s built.xml -odir REST2/
mpirun -n 8 md-openmm md-run -ng 8 -i AIS.in  -p built.pdb -s built.xml \
          -source-traj hot/cMD.dcd -odir AIS/

md-run is a surface, not a second implementation. Each protocol is handed to the same function a generated script calls -- stage_main, replica_main, ais_main -- so nothing is reachable from one and not the other.

build-md now writes an .in beside each .py, and run.sh drives md-openmm md-run rather than python <stage>.py, because that is the interface a person types. Every generated .in resolves to exactly the resolved.config beside it; that round-trip is asserted for all four protocols and every generated input. resolved.config is authoritative -- md-run writes it into -odir with the .in file's sha256, so the chain from what a person wrote to what ran is a link rather than a claim.

What running it on real hardware found¶

Five defects, none of which any suite could see, all found the first time the four tiers were actually launched on this machine's nine GPUs:

defect	consequence
md-run dispatched on protocol, not stage	`min.in` of a REST2 workflow went to the replica executor, which had no coordinates
four ranks renamed one ladder log	`REST2.log.partial` -> `REST2.log`, `FileNotFoundError`; `solute.yaml`, `_protocol.py` and the group file were written concurrently too
AIS handed OpenMM the box the file stored	"Periodic box vectors must be in reduced form" killed every path at its first frame -- a truncated octahedron is not in reduced form. A latent AIS defect.
a fixed-tau workflow could not be generated	`dynamics.tau > 0` is fixed-volume, but `stage_plan` applied that to production only; the NPT equilibration was generated and then refused at run time, four stages in
`-ng` accepted and ignored for AIS	a flag that silently does nothing, which is worse than one that is refused

And one that the surface exposed rather than caused: ff19SB + GBn2 was not refused anywhere. _check_protein_solvation_pairing was written for exactly that pair, but on the implicit branch _sys_document took the protein force field from sys_defaults and discarded what the configuration asked for -- so by the time the check ran, the document said ff14SB. An ff19SB + GBn2 request was silently built as ff14SB + GBn2. It is refused in _check_pairings now, the last point where the evidence still exists.

The input files, and which one is authoritative¶

build-md now writes an .in beside each .py, and run.sh drives md-openmm md-run rather than python <stage>.py -- that is the interface a person types, and a script using a different one is a second way to run the same thing that can drift from the documented way.

resolved.config is authoritative. md-run writes it into -odir before anything integrates, with the .in file's sha256 in its header, so the chain from what a person wrote to what ran is a link rather than a claim. Editing resolved.config between a run and its continuation changes the fingerprint and the continuation is refused; editing the .in afterwards changes nothing at all and its recorded digest simply stops matching, which is how you find out. A -odir that already holds a different resolved.config is refused rather than overwritten.

Every generated .in resolves back to exactly the resolved.config beside it. That round-trip is asserted for all four protocols and every generated input, and it found two defects that would otherwise have made an input mean something else:

floats written with %g: 300.0 became 300 and read back as an integer;
keys matched by reconstructing their target from their name, which silently dropped every input deliberately spelled differently from the field it sets -- random_seed -> dynamics.seed, reservoir_enabled -> reservoir.enabled.

A third was found only by a CUDA smoke test whose configuration set dynamics.platform: the input language could not express that field at all, so it was dropped between the .in and the resolved.config it came from. The check is now over the MODEL -- every field of every resolved block must be writable in an .in file -- rather than over whichever example a test happened to use.

Evidence¶

Real CUDA, real MPI, on 1x A5000 + 8x RTX 3080. Explicit ALA in TIP3P, 1796 particles, unless stated.

By hand, through the installed command:

cMD: full chain (min, eq_nvt_posres, eq_npt_posres, eq_npt_free, cMD) through md-run, every stage status: completed, acceleration: CUDA (default-cuda).
fixed-tau cMD, tau = 0.5: NVT throughout including equilibration, 20000 steps = 40 ps.
REST2, 4 states: mpirun -n 4 md-openmm md-run -ng 4, one rank per CUDA device (one rank per device (4 ranks, 9 devices)), 4 fixed-state trajectories remd0.nc..remd3.nc, Amber-format rem.log, restart.json, all ranks rank_status: completed.
AIS, 100 paths of 1 ps: mpirun -n 8 md-openmm md-run -ng 8, paths distributed 13/13/13/13/12/12/12/12, 100 trajectories AIS_traj0000.nc..AIS_traj0099.nc each carrying the box, 11 observations per path, and observation 0 exactly zero in all 100.
Platform policy: --cpu records requested_policy: explicit-cpu; CUDA_VISIBLE_DEVICES="" md-run refuses before integrating anything; --cpu --platform CUDA is refused as a contradiction rather than resolved by precedence.
-ng preflight: -ng 6 against a 4-state ladder, mpirun -n 2 against a 4-state ladder, and -ng 4 with no launcher are each refused before a Context opens, naming all three numbers.

From the wheel, in a clean environment, outside the checkout:

python -m build --wheel
python -m venv --system-site-packages /tmp/wheelenv          # conda stack visible, package not
/tmp/wheelenv/bin/pip install --no-deps dist/md_tools-0.5.0.dev0-py3-none-any.whl

one console script, md-openmm; entry_points.txt contains nothing else;
md_tools.__file__ resolves into site-packages, not the checkout;
build-top then build-md then ./run.sh completed a cMD chain, and a second run.sh completed a 4-rank REST2 ladder with platform : CUDA in all four ranks' .out files;
no generated file names the checkout: grep for the repository path across every *.py, *.in, run.sh and resolved.config in the generated project returns nothing.

Automated:

# counts as measured at an earlier commit in this note's history; run them, do not trust them
python -m pytest tests -m "not slow and not gpu"    # 833 passed, 107 deselected
python -m pytest tests -m "gpu or slow"             # 107 passed, on CUDA
python -m build --wheel

Six corrections to the `md-run` surface¶

Review of the md-run work found six contracts that were wrong rather than incomplete. Each is a promise the code made and did not keep, so each is corrected here with a regression test written first.

1. The short flags are Amber's¶

This supersedes the mapping in which -x named built.xml.

md-openmm md-run -i min.in -p built.pdb -s built.xml -c prev.xml \
          -o min.out -x min.dcd -r min.xml -log min.log

flag	meaning	Amber
`-i`	run input	mdin
`-o`	human-readable simulation output	mdout
`-p`	topology / reference PDB	prmtop
`-c`	input restart	inpcrd / restrt
`-r`	output restart	restrt
`-x`	output trajectory	mdcrd
`-s`	serialised OpenMM System	—

-x has meant mdcrd for decades. Using it for the System here trained people to type the one command that would write a trajectory over the System their run needs, so both mistakes are refused by name: a .dcd in -s, an .xml in -x, and the two resolving to one path. -s is required; -x is optional, defaulting to <stage>.dcd, and is refused for AIS, which writes one trajectory per path that no single name could describe. allow_abbrev=False: --traj errors instead of quietly becoming --trajectory.

2. `-o` and `-log` are two files¶

They were aliased, on the argument that this package writes one readable record. That conflated two audiences and made a person open a machine record and scroll past a hundred lines of hashes to see whether a simulation was moving.

.out is a running account — stage, settings, a progress table of step, time, potential, kinetic and total energy, temperature, volume, density and speed, then how it ended. Line-buffered, so tail -f works. .log is unchanged: the provenance record read_record parses. Each names the other, they default to <name>.out and <name>.log, and only a genuine collision is refused.

3. The platform is a property of the machine¶

dynamics.platform is gone from the scientific schema, from every shipped protocol config and from the .in language. It is now:

machine:
  openmm:
    platform: CUDA        # or CPU, for a deliberate machine-wide default
    precision: mixed
    device_policy: local_rank

in the user configuration, found through the discovery order that already existed and defaulting to CUDA / mixed / local_rank when the file or the block is absent. data-register --init writes it out. A configuration written before the block existed is still valid. dynamics.platform is refused with this migration rather than an unknown-key suggestion: the key was not misspelled, it moved, and a protocol config that carried it carried one machine's hardware into every repository it was shared through.

--platform is gone from md-run; a per-run platform flag would be a second authority for a machine property. --cpu is the one override, --device says which GPU and never whether, and the two are refused together. The record now distinguishes the three ways a platform can be chosen, because a CPU result has three possible causes and only one of them is nobody's decision: built-in default, machine.openmm, --cpu (command line). It also carries the CUDA driver version alongside the OpenMM version.

4. Every accepted AIS reporting option now does something¶

reporting.info_printout and reporting.checkpoint_printout were validated and then dropped. The runtime never saw the reporting block at all, so both were accepted, consequential-looking and inert — the one thing a strict input language must never be.

Four independent cadences, each dividing switching_steps on its own:

setting	controls
`ais.observation_interval_steps`	work observations — this one is the method
`reporting.crd_printout_solute`	frames in `AIS_trajNNNN.nc`
`reporting.info_printout`	`path_NNNN/system.csv`: potential, kinetic and total energy, temperature, volume, density
`reporting.checkpoint_printout`	how often the path becomes resumable

The rule forcing crd_printout_solute == observation_interval_steps is gone. It answered "how often is a configuration written" with the answer to "how often is work measured", and work every 10 steps with frames every 50 is an ordinary thing to want.

A density bug the state table exposed immediately: the dalton-per-nm³ to g/mL factor was 1.66053906660 where it should be 1.66053906660e-3, so TIP3P water came out at 947 g/mL. Wrong by 1000, and it looks like a number rather than a mistake — which is why the constant is now written as the ratio it is, with the arithmetic in a comment.

5. Exact mid-path resume¶

An OpenMM binary checkpoint plus a sidecar holding everything outside the Context: accumulated work, work since the last observation, and how many work rows, frames and state rows are on disk. Restoring one without the other resumes a simulation into somebody else's bookkeeping.

On --resume, every fingerprint — System, topology, source ensemble, schedule, path id, source frame, seeds — is checked before anything loads, then the streams are cut back to the counts the sidecar vouches for, so a record interrupted mid-write is dropped rather than appended to.

mdtraj's NetCDF writer offers r and w and no append, and that turned out to be the right constraint: a frame interrupted mid-write reads as present without being complete, so appending onto an unverified tail would be wrong even where the format allowed it. A resume rewrites the vouched-for frames into a fresh file and continues. Frames are staged inside the path directory and published to AIS_trajNNNN.nc in one atomic move after validation — until then nothing at the run root could be mistaken for a finished path. A completed path is skipped, and its checkpoint is deleted so nothing invites a resume of finished work.

6. MPI fails closed¶

barrier() caught ImportError and returned. mpirun -n 8 on a machine without a working mpi4py therefore ran eight processes that never met: eight independent simulations writing over one set of output paths, each believing it was the whole ladder, and the result looked complete.

md_tools.remd.mpi imports mpi4py, requires MPI initialised and not finalised, and compares the launcher's rank and size against the communicator's and both against -ng and the replica count — all before an output directory, resolved.config, log or Context exists. A mismatch names all four numbers, because naming the two that differ leaves the reader to work out which of the others to change. Serial is untouched: world size 1 imports nothing and a no-op barrier is correct when there is genuinely nobody to wait for. mpi4py is an extra (pip install md-tools[mpi]), and its absence is never a silent downgrade.

Genuine formats¶

md_tools.openmm.trajectory decides what a file IS from its leading bytes — CORD at offset 4 for DCD, CDF\x01 or the HDF5 signature for NetCDF — never from its suffix.

ordinary MD writes DCD through OpenMM's native reporter, and refuses a .nc trajectory name: OpenMM has no native NetCDF writer, and renaming a DCD does not convert it;
AIS accepts a genuine DCD (the ordinary fixed-τ cMD source) or a genuine NetCDF, and refuses one whose suffix and contents disagree, naming both;
AIS and REST2 outputs stay genuine NetCDF, and the tests open them with mdtraj rather than trusting the extension.

What running it found¶

Four defects the suites could not see, all from launching the real commands:

defect	consequence
AIS raced on its log under MPI	an explicit `-log` was used verbatim by every rank, so N ranks renamed one temporary and the run died with a `FileNotFoundError` that said nothing about the cause
`--resume` stopped at md-run	it was never forwarded to the AIS runtime, so every interrupted path silently restarted from its source frame while the command reported success
a stage resumed across platforms	OpenMM's "Checkpoint was created with a different Platform" surfaced as an internal-looking error; it is a fact about the two commands, not the simulation
the density factor	see above

Evidence for the six corrections¶

Hardware: 1x NVIDIA RTX A5000 + 8x NVIDIA GeForce RTX 3080, driver 580.173.02, OpenMM 8.6.0.dev-c6173db, Python 3.12. Explicit ALA in TIP3P, 1796 particles, unless stated.

# counts as measured at an earlier commit in this note's history; run them, do not trust them
python -m pytest tests -m "not slow and not gpu"   # 833 passed, 107 deselected, 0 skipped   84.5 s
python -m pytest tests -m "gpu or slow"            # 107 passed, 833 deselected, 0 skipped  308.9 s
python -m build --wheel                            # md_tools-0.5.0.dev0-py3-none-any.whl

By hand, through the installed command:

cMD, corrected flags. md-openmm md-run -i cMD.in -p ../built.pdb -s ../built.xml -o cMD.out -x cMD.dcd -r cMD.xml -log cMD.log -- the whole chain via run.sh, every stage completed, acceleration: CUDA. cMD.dcd verified as a genuine DCD by the byte sniffer and by mdtraj (5 frames, 1796 atoms). -x cMD.nc refused before any output, naming DCD.
.out and .log are two files. cMD.out carries the settings, a progress table of step, time, potential/kinetic/total energy, temperature, volume, density and speed, and status completed; cMD.log carries the machine record. Each names the other.
REST2, 4 ranks, 4 devices. mpirun -n 4 md-openmm md-run -ng 4 -i REST2.in -p ../built.pdb -s ../built.xml -c eq_npt_free.xml -x REST2.nc -r restart.json -o REST2.out -log REST2.log. 23.2 s. platform : CUDA device=0..3 (one rank per device (4 ranks, 9 devices)), precision mixed -- one line per rank, four distinct devices. remd0.nc..remd3.nc each verified as genuine NetCDF with 5 frames and 1796 atoms; rem.log and restart.json present.
AIS, 100 paths, 8 ranks, from a genuine DCD source. mpirun -n 8 md-openmm md-run -ng 8 -i AIS.in -p ../built.pdb -s ../built.xml -source-traj ../hot/cMD.dcd -odir . -o AIS.out -log AIS.log. 3 m 06 s. AIS_traj0000.nc..AIS_traj0099.nc, contiguous and all genuine NetCDF; paths shared 13/13/13/13/12/12/12/12; AIS_work.csv 2600 rows, uniquely keyed by (path, step) and sorted by it; observation 0 exactly zero in all 100 paths.
The four cadences, all different. switching_steps 250, observations every 10, frames every 25, state rows every 50, checkpoints every 125 gave 26 work rows, 11 frames and 6 state rows at steps 0, 50 … 250 -- three cadences, three counts.
Interruption and resume. A real SIGKILL of the real command mid-path, then --resume: path 1: resuming at step 1600 of 4000 (9 work row(s), 5 frame(s) kept), finishing with the same 21 observations, 6 state rows and 11 frames an uninterrupted path produced, no duplicated steps, the cumulative work still the running sum of the increments across the boundary, the completed path 0 byte-identical, and the checkpoint and sidecar removed.
Platform provenance. No user configuration -> platform_origin: built-in default, resolved_platform: CUDA. machine.openmm.platform: CPU -> platform_origin: machine.openmm, resolved_platform: CPU. --cpu against a CUDA machine -> platform_origin: --cpu (command line).
MPI fails closed. MD_TOOLS_FORCE_NO_MPI4PY=1 mpirun -n 2 md-openmm md-run -ng 2 ... refuses, naming mpi4py, and the output directory is not created. A serial run in the same environment completes normally.

From the wheel, in a clean environment, outside the checkout (tests/test_documented_commands_run.py):

one console script, md-openmm, and the four subcommands;
md_tools.__file__ inside site-packages, with PYTHONPATH scrubbed so the checkout cannot be reached;
the canonical stage command run end to end, every flag producing the artefact its documentation names, and built.xml still a serialised System afterwards -- the accident the old -x mapping invited;
--platform and -x built.xml refused by the installed command;
no generated file naming the checkout.

Correction: the previous head was reported as complete and its CI was red¶

1e2e5b9 is described above as passing every acceptance criterion. Its GitHub Actions run, 33634080764, failed, and so did 33640278414 on the commit after it. The local lanes were run and reported honestly; CI was never looked at, and the summary should not have said "complete" without it. The evidence above stands as local evidence and is labelled as such; it was never evidence about the GitHub head.

The failing step was md-run refuses -ng outside an MPI launch. It passed an XML through -x and omitted the mandatory -s, so argparse refused the command before the MPI check ran: the step exited non-zero for the wrong reason, and the refusal it is named for was never exercised. A test that fails for the wrong reason is worse than no test, because it looks like coverage. It now uses the current flag contract, detects an argparse refusal explicitly instead of accepting it as success, and asserts that -odir was not created.

Corrective round: preflight, MPI, configuration and restart¶

Review of 1e2e5b9 found six contracts that were wrong rather than incomplete. The theme is that a safe outer wrapper was hiding an unsafe runtime.

One MPI authority¶

Two permissive implementations survived outside md_tools.remd.mpi -- executor.barrier and driver.Coordinator -- both spelling:

try:
    from mpi4py import MPI
except ImportError:
    return

which is true of a serial run and false of every other kind. Under mpirun -n 8 with a broken mpi4py they produced eight processes each believing it was the only one, all eight driving every state, all eight writing the same files. The failure is invisible precisely because it only occurs when MPI is broken.

Coordinator is now a subclass of the one Coordination and adds nothing; barrier delegates. A test walks the AST of every module in the package and fails if mpi4py is imported anywhere else -- the AST rather than the text, so a docstring that quotes the retired code to explain it is not mistaken for the code.

Coordination.fail makes a rank-local error collective: it prints the reason from the rank that has it and calls Abort, so a multi-rank failure is a stopped job rather than a hung one. A rank that raises alone leaves the others blocked in the next collective until a scheduler kills them.

The guard is in the runtime, not the wrapper¶

replica_main and ais_main validate the LAUNCH themselves, before -odir exists, because the generated REST2.py, rREST2.py and AIS.py call them directly. A guard that lives only in md-openmm md-run is a property of one entry point, and a safe wrapper over an unsafe runtime is worse than no wrapper: it makes the unsafe path look tested. AIS gained -ng, checked against the communicator.

This round covered the MPI launch and not the rest of the preflight, and the claim below was broader than the tests supported. A generated script still created its output directory, its .out, its .log, solute.yaml, _protocol.py and a group file before discovering that the machine configuration was malformed or that CUDA could not open a Context. That is corrected in "Preflight, shared by the dispatcher and the direct runtimes" further down, where the coverage is stated against the tests that demonstrate it.

Preflight before any output¶

md_tools.run.preflight runs every check -- flag roles, resolved-path collisions, input existence, source format, the MPI launch, the machine configuration, --cpu/--device, the device policy, platform availability, and a real CUDA Context -- before -odir, resolved.config, the .out, the .log, a group file, a trajectory or a checkpoint exists. md_run_main previously wrote resolved.config first, so a refused run left a directory that reads as a run which happened.

Collisions compare RESOLVED paths, so -o sub/../run.out and -log ./run.out are recognised as one file; string comparison missed that and one would have truncated the other.

Absent configuration is not invalid configuration¶

machine_openmm_settings caught every RegistrationError and answered with the built-in defaults. Bad YAML, a duplicate key, an unknown field, an invalid platform -- all became CUDA/mixed/local_rank, so the machine ran on settings nobody chose and the file that said otherwise was never mentioned again.

A genuinely absent file still resolves to the defaults. An existing broken one is fatal, read through the strict duplicate-rejecting loader, and every refusal names the file. MD_TOOLS_CONFIG naming a missing file is a broken reference rather than an absence: somebody meant that path, and defaults would hide the typo.

Truthful provenance, and a device policy that does something¶

machine.openmm.platform: CPU was recorded as explicit-cpu, indistinguishable from a per-run --cpu. The record now carries platform_selection (built-in-default | machine-config | cli-override) and cli_cpu_override separately.

device_policy: openmm was accepted by validation and never consulted -- AIS chose a local-rank device before the machine setting was read. The policy is now read first, and both values do what they say: local_rank assigns one rank per visible device, openmm sets no DeviceIndex and lets OpenMM choose. A configuration field with no runtime effect is worse than an absent one, because somebody set it and believes it took hold.

Crash-atomic AIS checkpoints¶

The previous commit overwrote the binary checkpoint and then replaced the sidecar. Atomic replacement of the sidecar does not make the PAIR atomic: a crash in between left a new Context checkpoint beside old accumulated work, and resuming that continues from step 3000 while believing it is at step 2000. It re-emits rows that exist and reports the work of a path nobody ran. Nothing fails.

md_tools.ais.checkpoint is a generation transaction:

path_0000/checkpoints/generation_000013.chk   written to a NEW name, fsynced, digested
path_0000/checkpoints/generation_000013.json  the sidecar, carrying that digest, fsynced
path_0000/current_checkpoint.json             replaced atomically, LAST

A crash before the pointer lands leaves the previous generation committed; a crash after it leaves a pointer whose files were already durable. Resume follows only the pointer and verifies the checkpoint's sha256 before loading -- never "the newest generation", because the newest file is exactly what a crash leaves behind.

tests/test_checkpoint_transaction.py injects a crash at each of the five boundaries, twice: once with a previous generation committed and once without, and asserts what read_committed returns in every case.

Evidence for the corrective round¶

Hardware: 1x NVIDIA RTX A5000 + 8x NVIDIA GeForce RTX 3080, driver 580.173.02, OpenMM 8.6.0.dev-c6173db, Python 3.12.

python -m pytest tests -m "not slow and not gpu"   # 925 passed, 118 deselected, 0 skipped   91.5 s
python -m pytest tests -m "gpu or slow"            # 118 passed, 925 deselected, 0 skipped  396.1 s
python -m build --wheel                            # md_tools-0.5.0.dev0-py3-none-any.whl

CI, on the commit itself. 33644617673 on 13c0ed8: success. The -ng step now refuses for the right reason, and it checks that it did: an argparse refusal is detected explicitly rather than counted as a pass, and -odir is asserted absent afterwards.

The CI command, reproduced against the installed wheel outside the checkout with PYTHONPATH scrubbed:

$ md-openmm md-run -ng 4 -i ladder.in -p missing.pdb -s missing.xml -odir out
md-run: -ng 4 was requested but this process was not started by an MPI launcher: the world size is 1.
  -ng says how many processes coordinate; it does not create them.
  mpirun -n 4 md-openmm md-run -ng 4 ...

argparse did not refuse it; the message names the launcher; out/ was not created. An -ng contradiction is knowable from the command line and the environment alone, so it is now checked before anything touches the filesystem -- somebody debugging a launch script should be told about the launch, not about their paths.

Real multi-rank CUDA, from the installed wheel:

REST2, 4 ranks, 4 devices. platform : CUDA device=0..3 (one rank per device (4 ranks, 9 devices)), precision mixed -- one line per rank, four distinct devices. remd0.nc..remd3.nc, rem.log, restart.json, REST2.out and REST2.log all written.
AIS, 100 paths, 8 ranks, from a genuine DCD source. 3 m 35 s. AIS_traj0000.nc..AIS_traj0099.nc, contiguous and all genuine NetCDF, 11 frames each; paths shared 13/13/13/13/12/12/12/12; AIS_work.csv 2600 rows uniquely keyed by (path, step) and sorted by it; observation 0 exactly zero in all 100; and no current_checkpoint.json left behind -- a completed path keeps no transaction. The record reads platform_selection: built-in-default, cli_cpu_override: false, device_policy: local_rank.

MPI fail-closed, with a real launcher (tests/test_mpi_fail_closed.py, 8 tests): mpirun -n 2 with mpi4py unavailable refuses through md-run AND through the generated REST2.py and AIS.py wrappers, and creates no output directory in any of the four combinations; a launcher/-ng disagreement is refused naming all the numbers; a rank-local failure takes the whole communicator down and names the rank that failed. Every one has a timeout, because the failure being tested for is a hang and a test that waits forever cannot detect one.

Crash-atomic checkpoints (tests/test_checkpoint_transaction.py, 21 tests): a crash injected at each of the five transaction boundaries, twice -- once with a previous generation committed and once without. Before the pointer replacement, the previous generation is what resumes and the half-written one is ignored; after it, the new generation resumes and both its files are already durable. A tampered checkpoint, a missing checkpoint, a missing sidecar and an unreadable pointer are each refused rather than half-loaded.

Machine configuration (tests/test_machine_configuration.py, 24 tests): twelve ways of being wrong, each refused with the file named; four ways of being right, including no file at all and a legacy configuration predating machine.openmm. Three of the invalid cases run through the real command and assert the filesystem is untouched.

Wheel, outside the checkout: md_tools.__file__ resolves inside .../wh/lib/python3.12/site-packages/md_tools/__init__.py with PYTHONPATH scrubbed; one console script; the four subcommands; a full cMD chain and a 4-rank REST2 ladder run from it.

Preflight, shared by the dispatcher and the direct runtimes¶

The previous round put a shared preflight behind md-openmm md-run and described the direct runtimes as covered. Only the MPI launch was. Everything else -- the machine configuration, the platform, path collisions, input existence, topology/System agreement -- was still checked after stage_main, replica_main and ais_main had begun writing. A generated min.py opened its .out and .log at line 212 and read the machine configuration at line 325.

md_tools.run.preflight is now mode-aware and CONSUMED, not merely called:

preflight_stage(...)  -> StagePreflight     # split stages and all-in-one workflows
preflight_ladder(...) -> LadderPreflight    # REST2 and rREST2
preflight_ais(...)    -> AISPreflight       # switching paths

Typed results rather than one dictionary with optional keys: a coordination that is sometimes None-meaning-serial and sometimes None-meaning-unchecked is a value whose meaning depends on who is reading it, and the reader who gets it wrong is the one running eight ranks over one set of files. Each runtime then USES the result -- checked.acceleration, checked.coordination, checked.record() -- rather than resolving the platform, the machine settings or the MPI world a second time. stage_main and ais_main each had their own copy of that resolution; both are gone.

Two duplicate policies were removed with them: stage_main compared -o against -log on its own (missing -x, -r and -chk, and firing before the shared check, so the message depended on which implementation reached the case first), and checked the trajectory suffix separately.

The all-in-one workflow validates the WHOLE chain before its first stage writes anything. Per-stage preflight could not have caught a broken configuration until stage four, with three stages of output already on disk.

stage_parser and the workflow parser gained -odir, -o and --cpu, so a stage run directly takes the same flags as one run through md-openmm md-run.

What the tests demonstrate¶

tests/test_direct_runtime_preflight.py -- 87 tests, all through generated scripts as subprocesses, never through md-run. Five entry points (a split stage, the all-in-one workflow, REST2, rREST2, AIS) against every refusal:

condition	asserted
malformed YAML, duplicate keys, unsupported platform/precision/device policy	refused, naming the setting
blank `user.person_id`, missing `user.name`, non-mapping `user`	refused, naming the field
`MD_TOOLS_CONFIG` pointing at a missing file	refused as a broken reference
missing `-p` or `-s`	refused, naming the file
topology and System with different particle counts	refused, naming both counts
`-o sub/../run.out` with `-log ./run.out`	refused as one file
an output that would overwrite an input	refused
CUDA that cannot open a Context	refused (deterministic seam; no GPU needed)
an AIS source that is missing, or not a trajectory	refused
a plural MPI launch with no usable `mpi4py`	refused

Each snapshots the destination by digest and asserts it is absent, or byte-for-byte unchanged. An existing directory holding a previous run's results is included: a refusal that damages what it was protecting is as bad as one that creates a false record. And each asserts a diagnostic specific to its condition -- a test satisfied by an argparse error would prove nothing, since argparse runs first and refuses everything equally.

Two ordering tests: with a broken configuration AND an unusable CUDA present, the configuration is what is reported, because the platform cannot be resolved until the settings that name it have been read; and --check refuses the same things a real run does, rather than validating less.

Complete user-configuration validation¶

user.person_id and user.name are required and non-empty wherever a configuration exists, through ONE validator (check_user_identity) shared by registration and simulation. Copying the identity rules into a second one is how the two come to disagree about what "required" means.

An absent configuration still resolves to the built-in defaults; that is the case this exists to keep separate from a file that is present and wrong.

`explicit_cpu` means one thing¶

selection	name	explicit_cpu	cli_cpu_override	platform_selection	requested_policy
built-in	CUDA	false	false	built-in-default	default-cuda
machine CUDA	CUDA	false	false	machine-config	default-cuda
machine CPU	CPU	false	false	machine-config	machine-cpu
`--cpu`	CPU	true	true	cli-override	explicit-cpu

Machine CPU previously recorded explicit_cpu: true -- a claim about what somebody typed, made on the basis of a file they wrote months earlier on a different machine. tests/test_platform_provenance.py asserts every field of every row.

AIS recovery, through the real path runner¶

tests/test_checkpoint_transaction.py proves the commit is atomic. That is necessary and not sufficient: recovery has to reconcile four things written by different code at different moments -- trajectory frames, work rows, state rows, and the committed sidecar's counters.

tests/test_ais_recovery_integration.py drives run_one_path itself over a real four-particle OpenMM Simulation on the Reference platform. Not a mock: every object the runner touches is the real type, so saveCheckpoint, loadCheckpoint and XmlSerializer exercise production code. What is small is the system, not the fidelity.

Eleven fault boundaries, each armed to fire after a generation is already committed:

the five commit boundaries (after-checkpoint-write … after-pointer-replace);
six stream boundaries -- before and after a frame, a work row and a state row becomes durable.

After each, the resumed path is asserted to finish with exactly the counts an uninterrupted run produces, on the same step grid with no duplicates, with the cumulative work still the running sum of its increments across the boundary, and the same source frame. The committed counters are the authority: a frame that reached the disk after the last commit is dropped and rewritten, never kept because the file happened to be longer.

Also asserted: an uncommitted newer generation is ignored by the runner; a corrupt committed pair is refused with a message that says which directory to delete and that the other paths survive; a completed path clears its transaction and a later --resume leaves its trajectory byte-identical; and a sidecar whose fingerprint names a different run is refused before anything loads.

Remaining¶

Recorded rather than hidden.

The instruction's own file count. It asks for "all seven" configuration examples while the tree it draws enumerates six. The tree is the specification and six is what ships; the test asserts the exact set rather than any count, so it fails both on a missing file and on an undeclared one.

The ladder's default process model. Run without mpirun, a ladder is a single process holding every state and letting OpenMM choose the device. This is correct and is what the older smoke test exercises. A real multi-process ladder is now in the automated lane too: tests/test_md_run_mpi_gpu.py runs a two-rank ladder under mpirun, asserts one trajectory per fixed state and no third, and asserts that each rank recorded WHICH device it took and by what rule.

AIS results are not bit-identical across worker counts, and cannot be. Path IDENTITY is: paths_for_rank is a pure function, so path n starts from the same source frame with the same seeds and writes AIS_traj000n.nc whatever the world size, and that was checked at 1 and 3 workers over 12 paths. The work VALUES differ by up to 0.53 kJ/mol against a 4.16 kJ/mol spread across paths -- CUDA mixed-precision reduction order, amplified along 500 steps of a chaotic trajectory. That is a property of the hardware, not of the scheduling, and no amount of care in the distribution would remove it.

Work values are not bit-reproducible; path identity is. Restated here because the corrections did not change it. paths_for_rank is a pure function, so path n starts from the same source frame with the same seeds and writes AIS_traj000n.nc whatever the worker count. The VALUES differ by CUDA's reduction order, amplified along a chaotic trajectory.

A stage cannot be resumed across platforms. An OpenMM binary checkpoint belongs to the platform that wrote it. That is OpenMM's constraint, not something this package can work around; what changed is that it now says so in those words instead of surfacing as an internal-looking exception.

ais.tau_start is asserted, never verified. Nothing in a coordinate trajectory records the Hamiltonian it was sampled under. The log says so and records tau_verified_from_file: false. The AIS method page had claimed the opposite -- that the source's own record was checked against ais.tau_start -- and that claim is corrected.

One deliberate deviation from the instruction. Its example AIS input writes source_frame_selection = random. That is refused, with uniform_random named in the message. Accepting both would give one concept two spellings, which is the rule the rest of the parser is built on -- a configuration language with several spellings for one value has several ways to typo it. Every other key in that example parses, and each is asserted to reach the field it sets.

tau_start is asserted, never verified. Nothing in a coordinate trajectory records the Hamiltonian it was sampled under, so an AIS run cannot check that its source really is at tau_start. The log says so in those words, and records the assertion as tau_verified_from_file: false, so a reader checks it against the run that produced the file rather than assuming it was checked here.

An AIS resume is not bitwise reproducible, and is not asserted to be. The resumed path produces the same number of work rows, frames and state rows at the same protocol steps, with the same source frame, seeds and filename, and a cumulative work that is still the running sum of its increments across the boundary. The VALUES differ from an uninterrupted run: the Context restarts from a checkpoint, and CUDA's reduction order is not fixed. Requiring bitwise equality would be requiring something the hardware does not offer.

A rank-local abort is MPI_Abort, which is abrupt by design. Ranks that were mid-write when another rank failed leave whatever the filesystem had. That is the correct trade -- the alternative is a job that hangs in a collective until a scheduler kills it -- and it is why a path publishes its trajectory only after it is complete and validated, so an abrupt end leaves nothing that reads as finished work.

Not done here, by instruction. dev is not merged to main, nothing is published to PyPI, no v0.5.0 tag exists, and the version remains 0.5.0.dev0. This is a release candidate for review, not a release.

The AIS tau-basis decomposition¶

Why¶

A switching path recorded one number per update: the total work. Reweighting that nonequilibrium ensemble onto a different Hamiltonian — a different tau schedule, a different endpoint, or a Hummer–Szabo estimator evaluated at a tau the path never visited — needs the potential as a function of tau. A total at the one tau that was run does not contain that function, and re-running the path at another tau is not reweighting.

The identity¶

Write the coupling amplitude as a = 1 − τ. Every scale factor in this REST2 convention is a power of it:

group	factor	terms
unscaled	`a⁰`	bonds, angles, excluded omega torsions, environment–environment nonbonded
linear	`a¹`	solute–environment nonbonded, the whole generalized-Born energy
quadratic	`a²`	solute–solute nonbonded and 1–4, eligible solute torsions, solute CMAP

so at frozen coordinates the potential is exactly a quadratic polynomial in a:

U(τ, x) = U_unscaled(x) + a·U_linear(x) + a²·U_quadratic(x)

This is an identity, not a model, and it holds for the whole NonbondedForce — including the PME reciprocal sum, the Ewald self-energy and the long-range dispersion correction. Each of those is a quadratic form in the per-particle charge (or in sqrt(ε)), and the scaler multiplies solute charges by a and solute epsilons by a², so a pair term carries a^(number of solute partners). Nothing is decomposed per pair, and the validated force construction is untouched.

How it is measured¶

Three potential-energy evaluations per switching update, at controlled amplitudes a ∈ {0, ½, 1}, at frozen coordinates, then one exact quadratic through them. Three points determine a quadratic, so the fit contributes no model error: the only error is the floating-point error of the three evaluations. The nodes are as far apart as the domain allows, because the fit divides by their differences; a = 0 switches every scaled term off and therefore reads U_unscaled directly rather than inferring it.

τ = 1 is not a legal protocol coordinate and scaling_for_tau still refuses it. The probe goes through scaling_for_amplitude and TauSwitcher.set_amplitude, which is the same restore-and- rescale code a real switch uses — a separate probe implementation would be a second Hamiltonian, agreeing with the first only until one of them was edited.

Superseded below. The section that follows describes the FIRST version of this output, in which the potential columns carried the pre-switch work basis under names that read as potentials at the row's saved frame. That is scientifically wrong for reweighting and is corrected in "The frame-aligned Hummer–Szabo correction" at the end of this document. It is left here because a reader holding a v1 file needs to know what its columns meant.

Columns (schema v1 — superseded)¶

Twelve, appended to every per-path observation row and to AIS_work.csv (never inserted among the existing ones, so a reader keyed on position keeps reading the same numbers):

potential_unscaled_kj_mol              potential_linear_basis_kj_mol
potential_quadratic_basis_kj_mol       potential_linear_contribution_kj_mol
potential_quadratic_contribution_kj_mol potential_total_reconstructed_kj_mol
delta_work_unscaled_kj_mol             delta_work_linear_kj_mol
delta_work_quadratic_kj_mol            total_work_unscaled_kj_mol
total_work_linear_kj_mol               total_work_quadratic_kj_mol

The basis values are tau-independent and are what a reweighting reads; the contributions are those values as they enter U at that row's tau. Both are written, because deriving one from the other needs the convention, and a column that needs a convention to read is a column that gets read wrongly. Component totals and the schema identity also go to AIS_paths.csv, the checkpoint state, the completion manifest and the provenance record.

Which instant a row describes: the work columns are cumulative to that protocol step; the component potential columns are the frozen-coordinate measurement of the last parameter update contributing to the row. potential_total_reconstructed_kj_mol is therefore exactly the U(τ_after, x_frozen) whose difference from U(τ_before, x_frozen) is that update's total work — which is what makes it comparable to a directly measured number rather than to nothing.

What is checked rather than assumed¶

The total work is still measured directly, U(τ_{j+1}, x_j) − U(τ_j, x_j), from two evaluations of the real Hamiltonian. The components are derived independently, from the fit. The run then requires them to agree:

ΔW_total = ΔW_unscaled + ΔW_linear + ΔW_quadratic

and refuses rather than recording a decomposition of a work value it cannot reproduce. Deriving the total from the components would make that identity true by construction and test nothing.

ΔW_unscaled is exactly 0.0 and is written as the literal it is: the unscaled group has no tau dependence and the coordinates do not move during a parameter update. The column is kept so the zero is visible in the file rather than implied by its absence — and a nonzero value there is the signature of a force that acquired a tau dependence the three-group model does not describe.

Numerical tolerance¶

Documented per OpenMM precision, as (relative, absolute floor) in kJ/mol (md_tools.ais.decomposition.RECONSTRUCTION_TOLERANCES):

precision	relative	floor
`double`	1e-9	1e-6
`mixed`	1e-6	1e-3
`single`	5e-5	1e-1

The reconstruction is a fixed linear combination of three energies measured on one Context, so its error is the energies' own, amplified by the fit coefficients (magnitudes 1, 3 and 4 for these nodes). single is deliberately loose: a solvated potential is order 1e5 kJ/mol and a single-precision nonbonded sum carries roughly 1e-6 relative error, so a tighter tolerance would report the platform's arithmetic as a defect in the decomposition.

Cost, measured¶

Three extra potential-energy evaluations per switching update and no extra integration steps. Recorded in provenance rather than estimated:

decomposition:
  potential_energy_evaluations_per_update: 3
  switching_energy_evaluations: <total for this run>
  switching_energy_evaluation_seconds: <measured wall clock>

Measured on CUDA, at parameter_update_interval_steps: 1, on the two system sizes this branch verifies (test_the_decomposition_cost_is_measured_on_a_large_system):

system	particles	probe evaluations	seconds	per evaluation
ALA, implicit GBn2	22	63	0.286	4.55 ms
ALA, explicit TIP3P	1796	63	3.404	54.0 ms

Read the second row before choosing an update interval. Twelve times the cost per evaluation on a system eighty times larger says the probe is not dominated by the energy evaluation but by the parameter push: updateParametersInContext walks every particle and every exception of the NonbondedForce, three times per update, plus a fourth to restore the path's own tau.

So on a solvated system the decomposition is cheap per step and expensive per update. At the default parameter_update_interval_steps: 1 it costs roughly 0.16 s per switching step, which for a 500-step path is around 80 s of probing per path — comparable to the dynamics. A larger update interval reduces it proportionally and changes the physics not at all in the components: the work is still exact, because the identity is exact at whatever amplitudes the schedule visits. What a larger interval changes is the switching protocol itself, which is a decision to make on its own merits.

This is the honest number, not a favourable one. The 22-particle figure alone would have been misleading, which is why the lane measures both.

The identity holds through PME. On the explicit lane the worst |Σ components − measured total| over the whole work table was 4.7e-5 kJ/mol on energies of order 1e4, at mixed precision — which is the reciprocal sum, the Ewald self-energy and the long-range dispersion correction all decomposing exactly as the algebra says they must. No implicit lane can test that, because none of those terms exists in one.

Tested¶

tests/test_ais_decomposition.py builds systems where the answer is known term by term and checks that each lands in the group the convention names — bonds and angles wholly unscaled, GB in the linear basis and not the quadratic one (which is the retired v1 Hamiltonian), 1–4 and CMAP and eligible torsions quadratic, and an excluded omega torsion whose energy moves from the quadratic group into the unscaled one while τ = 0 is unchanged. Also the forward/reverse sign convention, frozen-coordinate telescoping, and that a probe restores its tau even when an evaluation raises.

tests/test_ais_recovery_integration.py covers the accumulators across an interruption at every transaction and stream boundary, an observation cadence wider than the update cadence, and refusal of a checkpoint written under another basis version or under none. The identity is verified on real CUDA in tests/test_cuda_coverage_matrix.py::test_ais_decomposition_lane.

Hummer–Szabo¶

The three basis values are what an estimator needs to evaluate U(τ) at a tau the path never visited, which is what reweighting a nonequilibrium ensemble onto another Hamiltonian requires. This package still computes no estimator. The columns are the input one would consume.

Installed-wheel verification, from outside the checkout¶

The lane §8 exists for, and the only one that found this defect:

python -m build
pip install --no-deps dist/md_tools-0.5.0.dev0-py3-none-any.whl
cd /tmp/wheellane            # NOT the checkout
python -c "import md_tools; print(md_tools.__file__)"
# .../site-packages/md_tools/__init__.py  — the wheel, not the source tree

Executed there, on real CUDA, from the installed package:

what	result
`md-openmm -h`, `md-openmm md-run -h`, `md-openmm --version`	four commands, Amber flags, `md-tools 0.5.0.dev0`
`build-top` implicit ALA + `build-md`	22 particles, 12 generated files
`md-run` cMD, 40 steps	`CUDA`/`mixed`, `platform_selection: machine-config`, completed
`mpirun -n 2 md-run` REST2, 2 states	`remd0.nc`, `remd1.nc`, completed
`mpirun -n 2 md-run` AIS, 4 paths	4 path trajectories, 20 work rows, 22 columns
the work-sum identity on that AIS table	0 rows of 20 violate it
decomposition provenance	3 evaluations/update, 30 total, 0.141 s

A defect only this lane could find. The group-file parser resolves a relative path against the group file — Amber's rule, and what everyone who writes one by hand expects. The writer emitted -p and -s exactly as they arrived on the command line, which are relative to the working directory. The two conventions agree for as long as a ladder is launched from its own output directory, which is what every test in the suite did and what a person running md-openmm md-run -odir ./run from their project root does not. Run from anywhere else, every rank looked for built.pdb beside the group file. The writer now emits paths relative to the group file, and test_the_group_file_is_written_with_paths_relative_to_itself pins both halves of the convention together with the working directory deliberately somewhere else.

The frame-aligned Hummer–Szabo correction (schema v2)¶

What was wrong¶

A switching update moves the parameters at frozen coordinates x_j, then propagates the configuration to x_{j+1}. Schema v1 wrote the trajectory frame for x_{j+1} and, on the same row, the potential basis measured at x_j — one propagation earlier — under column names that read as potentials at that frame.

A Hummer–Szabo reweighting from such a file pairs the work of one configuration with the energy of another. Nothing raises. Every number is plausible, every internal consistency check passes, and the error is invisible in the output.

Two probes, at two coordinates¶

probe	coordinate	what it is for
work basis	frozen pre-switch `x_j`	decomposes the incremental and accumulated work — the right coordinate, because the convention is `U(τ_{j+1}, x_j) − U(τ_j, x_j)`
observation potential	the coordinate the row SAVES, `x_t`, under that row's τ	the energy basis a Hummer–Szabo reweighting consumes

A row naming a coordinate_frame_index carries potentials recomputed at that frame. A row with no saved coordinate — the cadences are independent — leaves those cells empty by schema rather than borrowing a neighbour's.

The three groups, named by their power of λ¶

With a = 1 − τ and λ = a²:

U(τ, x) = U_non_scaled(x)
        + sqrt(λ) · U_sqrt_scaled(x)          sqrt(λ) = a
        + λ       · U_lin_scaled(x)           λ = a²

column	group	terms
`potential_non_scaled_kj_mol`	`λ⁰`	bonds, angles, excluded omega torsions, environment–environment
`potential_sqrt_scaled_kj_mol`	`sqrt(λ)`	solute–environment nonbonded, the whole generalized-Born energy
`potential_lin_scaled_kj_mol`	`λ`	solute–solute nonbonded and 1–4, eligible solute torsions, solute CMAP
`potential_reconstructed_kj_mol`	—	the identity above, evaluated
`potential_direct_kj_mol`	—	an independent `getState(getEnergy=True)` at the same coordinate and τ

"Linear" and "quadratic" were ambiguous until you said linear in what; these names say it.

Both totals are written so the identity is checkable from the file, without recomputing anything and without trusting that somebody checked it once.

Work components keep distinct names, delta_work_* and total_work_*, over the same three groups. delta_work_kj_mol is the sum of every switch since the previous emitted work observation — one switch when the work and switching cadences agree, their ratio otherwise. It is stated in DECOMPOSITION_SCHEMA["delta_work_meaning"] rather than left to be inferred.

`AIS_hs.csv`¶

The frame-aligned table, holding only rows whose coordinate was saved:

path_id, source_frame, observation_index, switch_step, coordinate_frame_index, tau,
total_work_kj_mol, potential_non_scaled_kj_mol, potential_sqrt_scaled_kj_mol,
potential_lin_scaled_kj_mol, potential_reconstructed_kj_mol, potential_direct_kj_mol

Every row's work and energy describe one configuration, so a reweighting reads it directly instead of filtering AIS_work.csv — and the failure mode of forgetting to filter is silent.

Schema v2, and v1 refused¶

Not a rename. v1's numbers describe a different coordinate, so v1 rows cannot be mixed into a v2 table and a v1 checkpoint cannot be resumed into a v2 run. Both are refused by name, with the reason.

Evaluation cost, counted properly¶

switching_energy_evaluations counted basis probes and omitted the two direct evaluations every switch already performed — understating a path's cost by exactly the part that predates the decomposition. Replaced by:

evaluation_counters:
  basis_probe_energy_evaluations:      # getState inside probes, work and observation alike
  direct_work_energy_evaluations:      # U(tau_j, x_j) and U(tau_{j+1}, x_j), two per switch
  observation_energy_evaluations:      # the direct potential at each frame-aligned row
  total_potential_energy_evaluations:  # their sum
  parameter_updates:                   # counted SEPARATELY: four per probe against three
                                       # evaluations, and the expensive half when solvated
  discarded_energy_evaluations:        # carried across a resume — work thrown away was paid for
  probe_seconds:

An "energy evaluation" is one Context.getState(getEnergy=True); a "parameter update" is one updateParametersInContext or setParameter push. Both are recorded because on a solvated system they diverge sharply.

Verified numerically, not asserted¶

tests/test_ais_hs_frame_alignment.py reopens the real AIS_trajNNNN.nc, reads the coordinates of the exact frame a row names, builds a fresh OpenMM Context, and recomputes all three groups and the direct potential. Sharing no state with the run is the point: a row being consistent with the runtime's other numbers proves nothing about which coordinate they belong to, which is how v1 was self-consistent and wrong.

test_the_potentials_are_not_those_of_the_previous_frame asserts the old behaviour now fails — recomputing at frame N−1 must disagree with row N. Without it, every other assertion would pass on a file uniformly shifted by one frame.

Five distinct cadences (switch/2, work/6, frame/10, system/20, checkpoint/15 over 120 steps), so frame-aligned and unaligned rows both occur; interruption and resume at nine boundaries, checking for duplicate, missing and mismatched rows; implicit and explicit PME.

Measured, at CPU single precision:

quantity	maximum error
implicit, recorded vs independently recomputed	1.6e-4 kJ/mol
implicit, Σ components vs measured work	2.4e-4 kJ/mol
explicit PME, recorded vs recomputed (on 2.1e4 kJ/mol)	3.0e-1 kJ/mol
explicit PME, reconstructed vs direct, in the file	9.2e-5 kJ/mol

The frame round-trip tolerance is documented and scaled by the total energy, not by each component: AMBER NetCDF stores coordinates as float32, and a basis value is a combination of three probe energies each of order the total, with fit coefficients 1, 3 and 4. A bound proportional to a −9 kJ/mol component inside a −2.1e4 kJ/mol system would fail for reasons unrelated to alignment.