A small toolkit for running forward tests honestly: write the test down before you see the data, pin the code that will evaluate it, look exactly once, and use intervals that stay valid when you peek. It works for any forward test: an A/B test, a forecasting model, a trading signal. Standard library only, Python 3.10+.
Three habits quietly wreck forward tests.
- Peeking inflates false positives. A 95% interval is only 95% if you look once, at a sample size fixed in advance. Check it after every observation and stop when it excludes zero, and the chance of a false "significant" result is several times the nominal 5%.
- Analysis code drifts. If the script that scores the test is edited after you have seen partial results, the test is no longer the one you registered.
- Forgotten trials. If you ran twenty variants and report one, the right multiple-testing denominator is twenty, not one.
prereg makes each of these mechanical to avoid.
spec -> freeze -> collect -> open look -> analyse -> seal result
(amend / repin, if needed, before the look opens)
All dates and timestamps are UTC.
- Spec. A JSON file: hypothesis, metric, decision rule, sample bar,
held-out window,
look_not_before, and the scripts that will do the evaluation (paths relative to the spec file). - Freeze.
freezerecords a SHA-256 of every listed script, a hash of the registered fields (spec_sha256), and aregisteredtimestamp. It requires at least one script,heldout_start <= heldout_end, andlook_not_beforeafterheldout_end. A registered spec cannot be frozen again. - Amend. Changes are appended as dated amendments; original fields are
never edited.
--repinre-hashes the scripts (for a bug fix) and records the old and new hashes. Refused once the look is open. - Collect. Wrap data loading in
guard_heldoutso code cannot read held-out dates early. - Open the look.
open-lookrefuses beforelook_not_before, refuses if the spec or a pinned script changed, and refuses a second look. It unlocks the guard. - Seal the result. Run the pinned analysis, then
seal-lookrecords the result file's SHA-256. One seal only;verifyflags a sealed result that later changes. - Log every test in the ledger so the multiple-testing denominator is always known.
Tamper-evident, not tamper-proof. The spec carries hashes of its
registered fields and of its own state, and opening the look also writes a
<spec>.look marker, so deleting look_opened does not allow a second look.
Anyone who edits the file and recomputes the hashes can still rewrite
history. The guarantee is strongest when the spec is committed to git (or
otherwise timestamped) at freeze time.
spec.json, next to evaluate.py:
{
"hypothesis": "variant B converts better than variant A",
"metric": "difference in conversion rate (B - A)",
"decision_rule": "lower confidence bound of the difference is above 0",
"sample_bar": 2000,
"heldout_start": "2026-03-01",
"heldout_end": "2026-03-31",
"look_not_before": "2026-04-01",
"scripts": ["evaluate.py"]
}$ python -m prereg freeze spec.json
frozen at 2026-02-20T14:02:11Z
3f9a1c07be52 evaluate.py
$ python -m prereg verify spec.json
OK: all pinned scripts match
$ python -m prereg amend spec.json --what "sample_bar 2000 -> 2500" --why "traffic estimate was wrong"
amendment recorded on 2026-02-24
$ python -m prereg open-look spec.json # before 2026-04-01
error: too early: look_not_before is 2026-04-01, today is 2026-03-20
$ python -m prereg open-look spec.json # on or after 2026-04-01
look opened on 2026-04-01; held-out data is unlocked
$ python -m prereg seal-look spec.json result.json
result sealed 9c1e5d0a77b3 result.json
$ python -m prereg ledger add --name variant-b --n 2500 --outcome "lower bound > 0" --p-value 0.012
recorded; ledger now has 1 test(s)In Python:
from prereg import guard_heldout, confidence_sequence, benjamini_hochberg
with guard_heldout("spec.json", dates): # raises until the look is open
data = load(dates)
bounds = confidence_sequence(observations, alpha=0.05, a=0.0, b=1.0)
lower, upper = bounds[-1]tests/test_cs.py draws Bernoulli(0.5) data (bounded in [0, 1], true mean
0.5), 300 observations per path, and asks: on how many paths does the
interval ever miss the true mean at any step?
- A fixed-n 95% normal interval, re-checked after every observation, misses on 44.13% of paths (nominal 5%).
- The confidence sequence misses on 2.47% of paths, within the 5% budget. It is not exactly 5% because the mixture bound is a bound, not an equality (Ville's inequality, and only 300 steps are checked).
These are the seeded values the test prints (seed 7, 1500 paths).
The price is width: the sequence is wider than a fixed-n interval at any single sample size. That width is what you pay to be allowed to look at every step. The test asserts the sequence stays within alpha plus Monte Carlo slack and that the naive interval fails far more often.
confidence_sequence(xs, alpha, a, b) returns running (lower, upper)
bounds on the mean of observations in [a, b], valid simultaneously over all
times with probability at least 1 - alpha. It uses the sub-Gaussian
normal-mixture boundary of Howard et al. (2021) with Hoeffding's variance
proxy (b - a)^2 / 4. The exact formula and assumptions (known bounds,
independent or martingale-difference observations) are in the docstring of
prereg/cs.py.
prereg_ledger.jsonl is append-only, one JSON object per test (name, date,
n, outcome, optional p-value). count() is the denominator. bonferroni
and benjamini_hochberg read the ledger directly; tests with no p-value
count as p = 1 rather than disappearing.
pip install -e ".[dev]"
pytest
ruff check .- Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2).
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society: Series B, 57(1).
- Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. PNAS, 115(11).
MIT