Repository navigation
Description
Symptom
tools/run_all_tests.py dispatches a hardcoded list of 10 suites (:140-142, plus x509 added conditionally at :154 = the 11 CLAUDE.md cites). Four modules that define a run_tests() entry point — i.e. are dispatchable by that runner, in its own convention — are never invoked:
dispatched: 10 (+ x509 conditionally)
define run_tests(): 15
omitted: ecdh_zero_check, finished_verify, hs_sequence, tls_deframer
CLAUDE.md:602 says the runner "dispatches 11 suites and omits test_finished_verify and test_chained_hmac". The count is right; the omission list is not — it names two, and at least four dispatchable suites are absent.
Why this matters more than a stale sentence
Two of the four omitted suites are the tests for the security fixes merged today.
tools/test_ecdh_zero_check.py— the all-zero X25519 shared-secret check (No all-zero check on the X25519 shared secret: a low-order key_share makes the session passively decryptable #153). Its anti-vacuity control asserts the honest case's 32 bytes against an independent Python X25519, so it is one of the strongest tests in the tree.tools/test_hs_sequence.py— the handshake-sequence gate (TLS 1.3 handshake message sequence is unenforced: a MITM completes the handshake with no Certificate at all #152), the fix for a working server-impersonation exploit.
Both were written this week, both were verified red-then-green before merge, and neither is run by the project's aggregate runner. The same is true of test_finished_verify — the suite created because a 2026-08-12 audit found the server-Finished path had no test at all — and test_tls_deframer, which mutation-proved the 512-content-record page-dispatch bug.
So the pattern is not random: the suites covering the highest-risk paths are precisely the ones the runner does not invoke. That was already noted in #161's inventory; this is the runner-side half of it.
Root cause
The suite list is a hardcoded array, so a new suite is only run if someone remembers to add it. Nothing enforces the coupling, and nothing notices when it drifts. pytest.ini has the equivalent problem solved: tools/test_pytest_boundary.py asserts by AST that every pure-logic tools/test_*.py appears in testpaths, and fails until you list it.
Suggested fix
Mirror the boundary guard: assert that every tools/test_*.py defining run_tests() is either in the runner's list or on an explicit, commented exclusion list. That makes adding a suite without wiring it a build/test failure rather than a silent omission, and it is the same shape of fix that already works one directory over.
Note the trap flagged in #161: a manifest is itself a vacuity surface. Without the guard, a runner that confidently reports a TOTAL over a list nobody maintains is worse than no runner, because the number looks like coverage.
Also correct CLAUDE.md:602
Either enumerate the omissions accurately, or — better, per #161's rule — stop restating a list that lives in the code and point at it instead. Duplicating it is what produced this defect.
Flagged during the #168 documentation-drift audit; verified independently before filing.
Activity
JC-000 commentedon Aug 31, 2026
Correction: my "four" understates it. There are two different gaps here.
The originating audit (#168) said eight; I filed four. Both are defensible but they count different things, and by filing only the smaller number I understated a coverage gap — in an issue whose whole subject is an understated coverage gap. Correcting rather than leaving it.
Measured (
grep -n '^def run_tests' tools/test_*.py):Gap A — dispatchable but not dispatched (4). Modules that define a module-level
run_tests(), i.e. already speak the runner's interface. The runner could call these today; nothing but the hardcoded list stops it.test_hs_sequencerun_tests(transport, labels)— #152, the impersonation-exploit fixtest_ecdh_zero_checkrun_tests(transport, labels)— #153test_finished_verifyrun_tests(transport, labels)— the audit-mandated server-Finished suitetest_tls_deframerrun_tests(transport, labels, cert_der, pubkey_xy, wiki_leaf=None)Gap B — not run by the runner at all (6 more). Standalone
main()programs with no module-levelrun_tests(), so the runner cannot dispatch them as written:test_x509_name,test_ecdsa_kat_oracle,test_p384_overlay_hazard,test_viewer,test_transcript_large,test_chained_hmac.Total not covered by the aggregate runner: 10.
Why the distinction changes the work
run_tests()interface, or accept them as standalone and make that explicit. Notetest_tls_deframershows the interface is not uniform even within Gap A: its signature takes cert material the others do not, so a guard must accommodate that rather than assume a fixed shape.That is what #168 meant by "fixing it means deciding what the runner should dispatch", and it is a fair reason to have left it out of a docs PR.
One consequence worth stating
test_chained_hmachas norun_tests()either — so CLAUDE.md:602's claim that the runner "omitstest_finished_verifyandtest_chained_hmac" names one module from Gap A and one from Gap B, without distinguishing them. The sentence is not merely incomplete; it conflates two different reasons for absence, which is probably why the omission was never chased.The recommendation in the issue body stands, and is now sharper: fix Gap A with a guard modelled on
test_pytest_boundary.py's AST check, and file Gap B separately as an interface question. Do not let a single "wire everything up" task hide the fact that six of them cannot be wired up without a decision first.registry suite: check the NAME of our rows, own by leading token; uci…
registry suite: fail closed when a row of ours names no known owner (…
registry suite: "c64-wireguard" owns, every row must parse; comment n…