live-e2e: the preflight, the bench and the report judge DEEPSEEK_API_KEY by three different rules — a short or mock key passes the gate and then kills the run #2214

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: cd55215b64db48e58e3218f4e5d2a819
Branch: dev/3-live-e2e-key-reading
Base: 77a6b4d
At: 2026-11-02T17:53:24Z

Owner: Dev-3
Tier: T1
Kind: bug (one rule, three readings)
Source: FlyBot Dev-3 bench READY (P-428), verified by FlyBot Lead on 77a6b4d4

Dedupe (searched before opening)

type:issue searches for live-e2e + key, preflight + key, sanitize + secret + redact: the open hit is #794 (the nightly run is red every night because the repository secret is missing — the umbrella this belongs to); #811/#813 (closed) are about a mapped secret reaching a step ungated, not about how the key is judged. Nothing covers the three readings disagreeing. No Policy-sign needed: .github/scripts/live-e2e-preflight.mjs is not in policy/gates.json gateFiles.

What happens today

Three places decide whether DEEPSEEK_API_KEY is usable, with three different rules:

• .github/scripts/live-e2e-preflight.mjs:29 — typeof key === 'string' && key.length > 0: any non-empty string is "configured", publishes key=true, and the workflow's six steps all go ahead.
• dev/live-e2e/bench.mjs:32 — !apiKey || apiKey === 'mock-key' || apiKey.length < 16 → die().
• dev/live-e2e/report.mjs:36 (guardRun) — the same expression → one problem, the run is infra_failed.

So a key that is 8–15 characters long, or the literal mock-key, passes the preflight and then kills the run — exactly the "red every night for a reason nobody can act on" shape #794 is about.

Reproduction (measured by Dev-3 at 77a6b4d4)

• 8-character key abcdefgh: preflight key=true, 0-byte warning, 0-byte summary; bench.mjs exits 1 with DEEPSEEK_API_KEY is missing, too short, or the mock's; a real key is required (never printed); guardRun problems ["DEEPSEEK_API_KEY is missing, too short, or the mock key"].
• mock-key: same disagreement.
• 32 characters of a: all three let it through — length can only judge shape, not authenticity; that is expected and must not change.

Expected behaviour

One reading, three callers. A blank / missing key, a mock-key, and a key shorter than 16 characters are all "not usable", decided in one place, and every caller says the same thing about it. The preflight reports key=false for those and explains why in its own warning; the bench and the run guard refuse with the same reason text.

Consequence for the nightly: a short or mock key skips with a warning instead of turning the run red — the actionable sentence lives in the run summary, which is what #794 asks for. A key of the right length still runs (nobody can tell a fake from a real one without the endpoint).

Scope

• dev/live-e2e/key.mjs (new) — keyReading(value) → { usable, reason: 'missing' | 'mock' | 'short' | null } and keyProblem(reason); blank and missing both read as missing.
• .github/scripts/live-e2e-preflight.mjs, dev/live-e2e/bench.mjs, dev/live-e2e/report.mjs — all three read that module; the preflight adds the "set but not usable" warning (distinct from the missing warning), the other two print the shared reason.
• dev/live-e2e/test/key-reading.test.mjs (new) + .github/scripts/live-e2e-preflight.test.mjs — the red-first tests below; one existing fixture is widened (plan('sk-live-e2e') 11 → 28 characters) so it does not judge itself unusable; no assertion is changed.
• dev/live-e2e/README.md + docs/subsystems/live-e2e.md — the invariant ("one reading, three callers"); dev/live-e2e/ is an arch-docs owned path, so the sentence ships in the same PR.

Not in scope: dev/live-e2e/sanitize.mjs:17 (secret.length >= 8 decides whether the key is echoed) — after this change an 8–15 character key never reaches a live run, so that threshold now only affects test fixtures; leave it and note it. Nothing under core/, plugins/, patches/ds-bot/ or core/client.js.

Red first

On a clean 77a6b4d4 tree carrying only the new/edited tests:

• dev/live-e2e/test/*.test.mjs → 26 tests / 24 pass / 2 fail:
  • the preflight and the run guard accept exactly the same keys — the preflight and the run guard disagree about a blank key, true !== false
  • the bench starter reads the key through the same module, with no threshold of its own — the bench keeps its own length threshold
• .github/scripts/live-e2e-preflight.test.mjs → 5 / 4 / 1: a key that is present but unusable skips like a missing one, and says why — the preflight lets " " through

After the change: 26/26 and 5/5, with the observed outputs 8 characters → preflight key=false plus a 459-byte "set but not usable" warning and bench DEEPSEEK_API_KEY is shorter than 16 characters; …; mock-key → preflight false, guard "is the mock's key"; " " → read as missing, original warning; 32 characters → unchanged, still runs.

Acceptance

1. The three callers agree on every input in the table above ('', missing, ' ', mock-key, 8 chars, 15 chars, 16 chars, 32 chars).
2. A short or mock key skips the nightly with a warning that names the reason; it does not fail the run.
3. Red-first counts above reproduce on the base tree; npm run verify green (.github/scripts 599, live-e2e 26, 4516/4516) and arch-docs: OK (8 changed file(s)).