Skip to content
ThatOneSharpDudePublic

About

Pre-registration for forward tests: hash-pinned specs, one-look enforcement, held-out guard, anytime-valid confidence sequences, and a trial ledger.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

9 Commits

Folders and files

Repository files navigation

prereg

CI

A small toolkit for running forward tests honestly: write the test down before you see the data, pin the code that will evaluate it, look exactly once, and use intervals that stay valid when you peek. It works for any forward test: an A/B test, a forecasting model, a trading signal. Standard library only, Python 3.10+.

The problem

Three habits quietly wreck forward tests.

  • Peeking inflates false positives. A 95% interval is only 95% if you look once, at a sample size fixed in advance. Check it after every observation and stop when it excludes zero, and the chance of a false "significant" result is several times the nominal 5%.
  • Analysis code drifts. If the script that scores the test is edited after you have seen partial results, the test is no longer the one you registered.
  • Forgotten trials. If you ran twenty variants and report one, the right multiple-testing denominator is twenty, not one.

prereg makes each of these mechanical to avoid.

Workflow

spec  ->  freeze  ->  collect  ->  open look  ->  analyse  ->  seal result
          (amend / repin, if needed, before the look opens)

All dates and timestamps are UTC.

  1. Spec. A JSON file: hypothesis, metric, decision rule, sample bar, held-out window, look_not_before, and the scripts that will do the evaluation (paths relative to the spec file).
  2. Freeze. freeze records a SHA-256 of every listed script, a hash of the registered fields (spec_sha256), and a registered timestamp. It requires at least one script, heldout_start <= heldout_end, and look_not_before after heldout_end. A registered spec cannot be frozen again.
  3. Amend. Changes are appended as dated amendments; original fields are never edited. --repin re-hashes the scripts (for a bug fix) and records the old and new hashes. Refused once the look is open.
  4. Collect. Wrap data loading in guard_heldout so code cannot read held-out dates early.
  5. Open the look. open-look refuses before look_not_before, refuses if the spec or a pinned script changed, and refuses a second look. It unlocks the guard.
  6. Seal the result. Run the pinned analysis, then seal-look records the result file's SHA-256. One seal only; verify flags a sealed result that later changes.
  7. Log every test in the ledger so the multiple-testing denominator is always known.

Tamper-evident, not tamper-proof. The spec carries hashes of its registered fields and of its own state, and opening the look also writes a <spec>.look marker, so deleting look_opened does not allow a second look. Anyone who edits the file and recomputes the hashes can still rewrite history. The guarantee is strongest when the spec is committed to git (or otherwise timestamped) at freeze time.

Worked example

spec.json, next to evaluate.py:

{
  "hypothesis": "variant B converts better than variant A",
  "metric": "difference in conversion rate (B - A)",
  "decision_rule": "lower confidence bound of the difference is above 0",
  "sample_bar": 2000,
  "heldout_start": "2026-03-01",
  "heldout_end": "2026-03-31",
  "look_not_before": "2026-04-01",
  "scripts": ["evaluate.py"]
}
$ python -m prereg freeze spec.json
frozen at 2026-02-20T14:02:11Z
  3f9a1c07be52  evaluate.py

$ python -m prereg verify spec.json
OK: all pinned scripts match

$ python -m prereg amend spec.json --what "sample_bar 2000 -> 2500" --why "traffic estimate was wrong"
amendment recorded on 2026-02-24

$ python -m prereg open-look spec.json             # before 2026-04-01
error: too early: look_not_before is 2026-04-01, today is 2026-03-20

$ python -m prereg open-look spec.json             # on or after 2026-04-01
look opened on 2026-04-01; held-out data is unlocked

$ python -m prereg seal-look spec.json result.json
result sealed 9c1e5d0a77b3  result.json

$ python -m prereg ledger add --name variant-b --n 2500 --outcome "lower bound > 0" --p-value 0.012
recorded; ledger now has 1 test(s)

In Python:

from prereg import guard_heldout, confidence_sequence, benjamini_hochberg

with guard_heldout("spec.json", dates):      # raises until the look is open
    data = load(dates)

bounds = confidence_sequence(observations, alpha=0.05, a=0.0, b=1.0)
lower, upper = bounds[-1]

Peeking, simulated

tests/test_cs.py draws Bernoulli(0.5) data (bounded in [0, 1], true mean 0.5), 300 observations per path, and asks: on how many paths does the interval ever miss the true mean at any step?

  • A fixed-n 95% normal interval, re-checked after every observation, misses on 44.13% of paths (nominal 5%).
  • The confidence sequence misses on 2.47% of paths, within the 5% budget. It is not exactly 5% because the mixture bound is a bound, not an equality (Ville's inequality, and only 300 steps are checked).

These are the seeded values the test prints (seed 7, 1500 paths).

The price is width: the sequence is wider than a fixed-n interval at any single sample size. That width is what you pay to be allowed to look at every step. The test asserts the sequence stays within alpha plus Monte Carlo slack and that the naive interval fails far more often.

Confidence sequence

confidence_sequence(xs, alpha, a, b) returns running (lower, upper) bounds on the mean of observations in [a, b], valid simultaneously over all times with probability at least 1 - alpha. It uses the sub-Gaussian normal-mixture boundary of Howard et al. (2021) with Hoeffding's variance proxy (b - a)^2 / 4. The exact formula and assumptions (known bounds, independent or martingale-difference observations) are in the docstring of prereg/cs.py.

Ledger

prereg_ledger.jsonl is append-only, one JSON object per test (name, date, n, outcome, optional p-value). count() is the denominator. bonferroni and benjamini_hochberg read the ledger directly; tests with no p-value count as p = 1 rather than disappearing.

Development

pip install -e ".[dev]"
pytest
ruff check .

References

  • Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2).
  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate. Journal of the Royal Statistical Society: Series B, 57(1).
  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. PNAS, 115(11).

License

MIT

About

Pre-registration for forward tests: hash-pinned specs, one-look enforcement, held-out guard, anytime-valid confidence sequences, and a trial ledger.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages