The 1000-parse benchmark has a fixed 1.0s wall-clock bound and has blocked two merges on runner variance #475

tests/v2/test_benchmark.py::test_facade_thousand_names_under_a_second asserts elapsed < 1.0 for 1000 facade parses. The file's own docstring says the bound is

Deliberately generous -- guards against order-of-magnitude regressions, does not gate normal variance.

It is now doing the opposite: every failure so far has been within 8% of the bound, and it has held up two merges.

Observed on CI

All on build (3.12); the other four Python versions have never failed it.

#466    1.03s   1.01s     two failures; master's own 3.12 job re-ran green
#472            pass
#474    1.06s   1.08s     two failures; master's own 3.12 job re-ran green

The numbers have drifted upward across the cycle (1.01–1.03 in one session, 1.06–1.08 hours later), which is the signature of the runner rather than of either branch.

Measured locally for #474, three ways

Branch against its own parent commit, same machine, interleaved:

The first is ~1% and inside the parent's own run-to-run spread; the second is indistinguishable; on the third the branch is faster. No local method reproduces a slowdown, yet CI failed that branch twice while master passed.

Why that matters more than the flake itself

A maintainer facing a red 3.12 cannot tell from this signal whether to investigate a regression or press re-run. On #474 the honest answer after three measurements was "I cannot reconcile the local evidence with CI" — which is exactly the state a guard test should not leave someone in. The cost is real: two merges held, and several hours of measurement spent distinguishing noise from signal on a check that cannot make the distinction itself.

Options

• A relative bound. Time a fixed reference workload in the same process and assert a ratio. This is what makes the check portable across runners, and it is what the neighbouring test_parse_cost_grows_no_worse_than_linearly already does in spirit — it bounds GROWTH in input length rather than absolute cost, and it has never flaked.
• A wider absolute bound, with the headroom stated and dated, so a future narrowing is a decision rather than a drift.
• Non-blocking on CI, kept as a local smoke check.

Scope note

This is about the ABSOLUTE-cost tests (test_parse_thousand_names_under_a_second and test_facade_thousand_names_under_a_second). The scaling tests beside them are a different instrument and are not implicated — the file's docstring already explains why neither subsumes the other, and that reasoning stands.