Reproducible evaluation for biomolecular prediction models.
Run it three times. Report the spread.
Biomolecular prediction models are evaluated almost entirely on aggregate accuracy against public benchmarks. Three things that practice hides:
Run-to-run instability, observed but unexplained. Wan, Zhang, Xue and Coveney (arXiv:2603.05532) evaluated Boltz-2 on 16,780 compounds for 3CLPro and 21,702 for TNKS2. They re-ran a subset — 793 and 799 ligands — and found predicted affinities correlated well between runs (Pearson 0.913 and 0.962) but that "absolute differences in predicted binding affinities between runs can reach ∼1.5 kcal/mol." They do not explain the source, and note they held the pre-trained weights fixed, so training-seed uncertainty is not what they measured.
Set that beside the compression figure below: if the model spans roughly 2 kcal/mol in total and moves up to 1.5 kcal/mol between runs of the same input, run-to-run noise is approaching the entire usable dynamic range.
And nobody has measured pose reproducibility at all. Wan et al. report RMSDs against experimental structures, not between repeated runs. No published work I can find measures how far Boltz-2's poses move when only the seed changes. That is the gap this exists to close.
Regression to the mean. Community benchmarking (Rowan Scientific) repeatedly finds Boltz-2 compressing predictions into a roughly 2 kcal/mol band regardless of the true dynamic range. A model can post an acceptable Pearson correlation while resolving none of the potency differences it is being used to resolve.
Benchmark contamination. Graber et al. (Nature Machine Intelligence 2025) detected "nearly 600 such similarities" between PDBbind training and CASF complexes, "involving 49% of all CASF complexes." Retrained on their leak-free CleanSplit, Pafnucy's RMSE rose from 1.046 to 1.484 — the model was partly reciting rather than generalising.
Note the direction of that evidence carefully: Pafnucy's Pearson correlation did not degrade on the clean split (0.716 → 0.746). Error got worse while correlation held, which is precisely why a single headline metric is a poor way to judge these models.
These findings exist. What does not exist is a way to reproduce them. They are currently spread across one preprint, one company blog, a tracking blog and assorted individual researchers — and none of it is pinned, repeatable, or aggregated.
That is the gap Triplicate is for.
Pins every variable that determines a result — model version, weights hash, container digest, seed, input — into a content-addressed manifest, then runs the same input N times in isolated containers and reports how far the answers spread.
Determinism is the point. Divergence you can measure is a finding. Divergence you cannot reproduce is an anecdote.
Phase 3 of 4. Provenance core, execution substrate and the first experiment are written and tested. Nothing has been run against a real model, and no result here is publishable yet.
One caveat worth stating plainly: the Modal executor's result handling is covered by tests, but the path that actually launches a container requires live Modal credentials and a GPU, and has not been exercised.
from triplicate.manifest import Manifest
from triplicate.executor import run_replicates
from triplicate.modal_executor import ModalExecutor
base = Manifest(
model="boltz",
model_version="2.0.0",
weights_sha256="...",
container_digest="sha256:...", # mutable tags are rejected
seed=0,
input_sha256="...",
)
report = run_replicates(ModalExecutor(predict), base, n=3)
report.complete # False if any replicate failed
report.spread(lambda out: out["rmsd"]).range # how far apart the answers landedFailed replicates are dropped rather than substituted. A timeout is not a value, and imputing one would manufacture agreement that never happened — so complete is reported separately from the statistics.
pip install -e ".[dev]"
pytest| Phase | Scope | State |
|---|---|---|
| 1. Provenance core | Content-addressed manifests, replicate derivation, spread and range-compression statistics | Done |
| 2. Execution substrate | Per-run isolated containers, weight verification, hard timeouts at the container boundary. No shared namespace, no ambient credentials | Done — 42 tests. Live Modal path unexercised |
| 3. First experiment | Boltz-2 workload, pose comparison, 3CLPro replicate run | Built, not run. Parser verified against real modelcif output; boltz predict itself unexercised. Runnable on a single GPU without Docker or Modal |
| 4. The evaluation | Widen to leak-free splits and the affinity claim; publish tool and preprint | Planned |
A broader tool layer was originally planned for phase 3 and has been deferred. Producing one real number matters more than breadth, and the first experiment needs none of it.
examples/kaggle_variance.py runs the same experiment on a single GPU box — Kaggle, Colab, a workstation — using a subprocess-per-run local executor instead of containers.
pip install boltz==2.0.0
python examples/kaggle_variance.py --smiles "CC(C)(C)c1ccc(cc1)N([C@H](c2cccnc2)C(=O)NC3CCCCC3)C(=O)c4c[nH]cn4" --n 3Boltz requires Python >=3.10,<3.13. Triplicate itself runs on 3.11-3.13, so the ceiling comes from Boltz — check python --version before wondering why the install failed. 2.0.0 is pinned for reproducibility, not currency; 2.2.1 is the latest at time of writing, and pinning whichever version you actually intend to characterise is the point.
That SMILES is X77, the non-covalent Mpro inhibitor from PDB 6W63 — 34 heavy atoms, no warhead. Provenance and the reasoning for choosing it over the covalent alternatives is in examples/data/ligands.md. Expect 34 atoms per pose; a different count means the ligand did not parse as intended and the RMSDs are not trustworthy.
It hashes whatever checkpoint Boltz downloaded and pins that, so every replicate is verified against the same file. With no container there is no image digest, so the environment is recorded as a hash of the package versions instead — weaker, since it captures neither the CUDA driver nor the base OS.
Read the result accordingly: a divergence found this way is real, a convergence is not a promise about other hardware.
The local executor is for one user on one machine and does not isolate anything — a subprocess inherits the parent's environment. Do not point it at inputs you did not write. What it does preserve is what affects the measurement: a fresh interpreter per replicate, so no state carries between runs, and a timeout that kills the process rather than asking a thread to stop.
examples/3clpro_variance.py runs SARS-CoV-2 main protease against Boltz-2 N times, varying only the seed, and reports the widest pairwise pose disagreement. 3CLPro is one of the two targets Wan et al. used, so the affinity behaviour there is at least documented.
The script flags anything above 2 Å, the conventional threshold for a correct pose in docking evaluation. Note what that means here: it is normally applied between a prediction and a crystal structure. Applying it between two runs of the same model asks a different and easier question — if replicates cannot agree with each other to within the tolerance used to call a pose right, the disagreement is larger than the accuracy being claimed.
python examples/3clpro_variance.py \
--digest sha256:... --weights-sha256 ... --fasta 3clpro.fasta --n 5Pose comparison uses no superposition, deliberately. Two poses from the same receptor frame are already comparable, and aligning them first would rotate away the disagreement being measured — a ligand in the wrong pocket would superpose onto the right one and score well.
Phase 2 deliberately avoids the common pattern of exec()-ing model-generated code in the host process under a threading timeout. That design is defensible for a single-user tool on a trusted machine and disqualifying for a shared one: the executing code inherits the host's environment and credentials, state leaks between runs through a shared namespace, and a thread blocked in a C extension cannot be interrupted — so the timeout returns while the work carries on consuming the machine.
PoseBusters (Buttenschoen, Morris, Deane, Chemical Science 2024) showed that deep-learning docking methods produced physically impossible poses — steric clashes, broken stereochemistry — while posting favourable RMSD numbers. Once physical validity was required, classical Vina and Gold beat every deep-learning method tested.
PB-valid then became a metric the whole field reports. One careful independent evaluation changed how an entire literature presents its results. That is the intended shape of this project: a tool and a paper, not a platform.
PoseBench (Morehead et al., Nature Machine Intelligence 2026) extended that scrutiny to AlphaFold3, Boltz-1 and Chai-1.
- Immutable digests only. A container tag like
latestsilently changes what ran; the manifest refuses it. - Range over standard deviation for stability checks. One divergent run out of five is a reproducibility failure even when the other four cluster.
- Report validity alongside accuracy. Accuracy without physical validity is the exact failure mode PoseBusters was built to catch.
- Leak-controlled splits by default — PDBbind CleanSplit, Leak Proof PDBBind. Evaluating on raw PDBbind/CASF measures memorization.
AGPL-3.0. The code is free to read, run, modify and self-host. Running a modified version as a network service requires releasing those modifications.
Chosen deliberately over MIT: a reproducibility tool is worth little if someone can fork it, close it, and host the authoritative version. Note the tradeoff — this is a heavier licence than the Apache-2.0 and MIT norms of the surrounding ecosystem, and it may deter some academic contribution.
This repository contains no third-party code. The models it evaluates, and any weights or datasets used with it, carry their own licences — several are non-commercial, and that is the user's obligation to check.
Intellectual debts are cited inline: PoseBusters and PoseBench for the evaluation posture, and Wan et al. and Graber et al. for the findings that motivated this.