Skip to content

Fix flaky private-identifier assertion in Experiment tests - #2169

Merged
JacobStephens2 merged 2 commits into
mainfrom
issue-2167
Nov 13, 2026
Merged

JacobStephens2 merged 2 commits into
mainfrom
issue-2167

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

Closes #2167.

The request privacy assertion matched 421 inside legitimate Experiment and Trial paths. Check decoded JSON string values with the numeric identifier token rule from Kind::mentions, accepting hashes while still rejecting 421, PR 421, #421, and /pull/421.

- reject any serialized request containing "421"
+ reject identifier tokens in decoded request string values

The change stays in the tests and retains the quoted-title/full-reference-commit absence assertions and all five positive Session evidence leak-hit assertions.

Evidence

Execution evidence

The baseline was 4d24f79d170c8e72be238f405efc4d3215962375. This deterministic path collision reproduced the reported assertion failure:

mkdir -p target/triage-2167/66d4ba83bed53da05d397c944421ecef8054d4580ff918593d7af28cc778500e
TMPDIR="$PWD/target/triage-2167/66d4ba83bed53da05d397c944421ecef8054d4580ff918593d7af28cc778500e" cargo test --locked --test experiments process::process_session_evidence_check_all_private_fix_identifiers_and_abbreviated_commits -- --exact --nocapture
  • Before: exit 101, panic at tests/experiments/process.rs:111:17; test result: FAILED. 0 passed; 1 failed. The printed request contained 421 only in repository and output_dir paths; inputs remained main, issue-7, /issues/7, and /pull/1.
  • After, same command: exit 0; test result: ok. 1 passed; 0 failed.
  • Control before the fix: the same focused test with 421 changed to 42a in TMPDIR passed (exit 0, 1 passed). This distinguishes the path collision from a leaked private input.

The retained regression runs through the same request assertion, with literal examples covering hashes, adjacent Unicode letters/digits/underscores/hyphens, nested objects/arrays, identifier tokens and JSON-escaped digits:

cargo test --locked --test experiments process::request_privacy_check_distinguishes_identifier_tokens_from_hashes -- --exact --nocapture
  • Before: exit 101; the hash-only request produced left: true, right: false; 0 passed, 1 failed.
  • After, same command: exit 0; 1 passed, 0 failed.

The required focused command also passed normally (exit 0, 1 passed):

cargo test --locked --test experiments process::process_session_evidence_check_all_private_fix_identifiers_and_abbreviated_commits -- --exact --nocapture

cargo fmt --all -- --check passed. scripts/affected-tests main selected the full Rust suite (cargo test, because nextest is unavailable) and passed: PASS cargo test (1348.67s), including compilation. The full run also passed both tests above. CI passed for commit 8039826b, including all four Rust partitions and the coverage gate, site tests and Console tests.

Test seams

  • Trial request privacy assertion: decoded request.json string values distinguish private identifier tokens from digits embedded in hashes.
  • Experiment CLI: the existing process Session evidence test retains all five positive leak-hit assertions and request title/commit absence checks.

Review

Independent reviews against refs/remotes/origin/main: Standards 0 findings; Spec 0 findings. Both reviewers read the complete changed file. Unread changed files: none. Generated-output exclusions: none.

Unaddressed findings

  • Standards: none.
  • Spec: none.
  • Security: none.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

The change can be reverted directly.

Blast Radius: tests

Only tests/experiments/process.rs changes.

Built with codex codex-cli 0.167.2 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 960f026 into main Nov 13, 2026
12 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-2167 branch November 13, 2026 23:39
@JacobStephens2 JacobStephens2 mentioned this pull request Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: private-identifier check matches '421' inside random Experiment hashes

1 participant