Skip to content

run_all_tests.py silently omits four dispatchable suites, including both security tests merged this week #169

Closed

Description

@JC-000

Symptom

tools/run_all_tests.py dispatches a hardcoded list of 10 suites (:140-142, plus x509 added conditionally at :154 = the 11 CLAUDE.md cites). Four modules that define a run_tests() entry point — i.e. are dispatchable by that runner, in its own convention — are never invoked:

dispatched:              10 (+ x509 conditionally)
define run_tests():      15
omitted:                 ecdh_zero_check, finished_verify, hs_sequence, tls_deframer

CLAUDE.md:602 says the runner "dispatches 11 suites and omits test_finished_verify and test_chained_hmac". The count is right; the omission list is not — it names two, and at least four dispatchable suites are absent.

Why this matters more than a stale sentence

Two of the four omitted suites are the tests for the security fixes merged today.

Both were written this week, both were verified red-then-green before merge, and neither is run by the project's aggregate runner. The same is true of test_finished_verify — the suite created because a 2026-08-12 audit found the server-Finished path had no test at all — and test_tls_deframer, which mutation-proved the 512-content-record page-dispatch bug.

So the pattern is not random: the suites covering the highest-risk paths are precisely the ones the runner does not invoke. That was already noted in #161's inventory; this is the runner-side half of it.

Root cause

The suite list is a hardcoded array, so a new suite is only run if someone remembers to add it. Nothing enforces the coupling, and nothing notices when it drifts. pytest.ini has the equivalent problem solved: tools/test_pytest_boundary.py asserts by AST that every pure-logic tools/test_*.py appears in testpaths, and fails until you list it.

Suggested fix

Mirror the boundary guard: assert that every tools/test_*.py defining run_tests() is either in the runner's list or on an explicit, commented exclusion list. That makes adding a suite without wiring it a build/test failure rather than a silent omission, and it is the same shape of fix that already works one directory over.

Note the trap flagged in #161: a manifest is itself a vacuity surface. Without the guard, a runner that confidently reports a TOTAL over a list nobody maintains is worse than no runner, because the number looks like coverage.

Also correct CLAUDE.md:602

Either enumerate the omissions accurately, or — better, per #161's rule — stop restating a list that lives in the code and point at it instead. Duplicating it is what produced this defect.

Flagged during the #168 documentation-drift audit; verified independently before filing.

Activity

  1. JC-000 commented on Aug 31, 2026

    @JC-000
    OwnerAuthor

    Correction: my "four" understates it. There are two different gaps here.

    The originating audit (#168) said eight; I filed four. Both are defensible but they count different things, and by filing only the smaller number I understated a coverage gap — in an issue whose whole subject is an understated coverage gap. Correcting rather than leaving it.

    Measured (grep -n '^def run_tests' tools/test_*.py):

    Gap A — dispatchable but not dispatched (4). Modules that define a module-level run_tests(), i.e. already speak the runner's interface. The runner could call these today; nothing but the hardcoded list stops it.

    module signature
    test_hs_sequence run_tests(transport, labels) — #152, the impersonation-exploit fix
    test_ecdh_zero_check run_tests(transport, labels) — #153
    test_finished_verify run_tests(transport, labels) — the audit-mandated server-Finished suite
    test_tls_deframer run_tests(transport, labels, cert_der, pubkey_xy, wiki_leaf=None)

    Gap B — not run by the runner at all (6 more). Standalone main() programs with no module-level run_tests(), so the runner cannot dispatch them as written: test_x509_name, test_ecdsa_kat_oracle, test_p384_overlay_hazard, test_viewer, test_transcript_large, test_chained_hmac.

    Total not covered by the aggregate runner: 10.

    Why the distinction changes the work

    • Gap A is wiring. Four names into a list, plus a guard so it cannot drift again. Low risk, immediate value, and it is where the security tests are.
    • Gap B is a design decision — give each a run_tests() interface, or accept them as standalone and make that explicit. Note test_tls_deframer shows the interface is not uniform even within Gap A: its signature takes cert material the others do not, so a guard must accommodate that rather than assume a fixed shape.

    That is what #168 meant by "fixing it means deciding what the runner should dispatch", and it is a fair reason to have left it out of a docs PR.

    One consequence worth stating

    test_chained_hmac has no run_tests() either — so CLAUDE.md:602's claim that the runner "omits test_finished_verify and test_chained_hmac" names one module from Gap A and one from Gap B, without distinguishing them. The sentence is not merely incomplete; it conflates two different reasons for absence, which is probably why the omission was never chased.

    The recommendation in the issue body stands, and is now sharper: fix Gap A with a guard modelled on test_pytest_boundary.py's AST check, and file Gap B separately as an interface question. Do not let a single "wire everything up" task hide the fact that six of them cannot be wired up without a decision first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions