[JacobStephens2]

Summary

Failing browser-test summaries now include their error messages, dedented and capped at 15 lines per test, with the number of omitted lines when truncated.

 print_failures(log)
   Rust panic → print the existing panic summary
-  TAP error: |- → print only the header
-  spec failure → print only the test name
+  TAP error block → collect indented message lines
+  spec failure → collect error lines until a stack frame
+  dedent message; print at most 15 lines and the omission count

Diagnostic keys and stack frames stay out, including native (<anonymous>) and async (index N) frames. One-line errors and Rust summaries retain their existing behavior. docs/testing.md describes the expanded browser summary. The regression tests cover both reporters, truncation, and message lines beginning with at.

Evidence

Execution evidence

Test seam: the stdout summary and exit status of scripts/affected-tests, exercised through its CLI with real Git and stand-in test runners.

• Reproduced at the supplied merge base 32e7398fdf4700fd42532ad89dfc528c774ca21f, in a temporary detached worktree with only the branch's regression-test file copied in. No baseline production code was changed. Python 3.12.3, Linux.

  Exact command on both checkouts:

  python3 -m unittest discover -s scripts -p test_affected_tests.py -k test_failed_browser_cases_print_multiline_error_messages

  Before: FAILED (failures=2). Both TAP and spec failed with AssertionError: 'Call log:' not found. TAP printed not ok 2 - copies its show command followed by bare error: |-; spec printed the test names without the message.

  After: Ran 1 test in 0.429s / OK on the existing branch fix.

  Confirmed cause: the old regex matched only the TAP error: header and never collected its indented block. Spec error lines matched no output rule.

• The current Spec review found a native-frame omission gap. Its exact real-Node v24.21.0 command was run before deciding the finding: a JSON.parse('{') failure leaked at JSON.parse (<anonymous>) into the summary. The same command passed after the frame matcher was extended. A retained regression exercises the public CLI:

  python3 -m unittest discover -s scripts -p test_affected_tests.py -k test_spec_stack_frames_without_locations_are_omitted

  Before: FAILED (failures=2), with at JSON.parse (<anonymous>) and at async Promise.all (index 0) unexpectedly present.

  After: Ran 1 test in 2.548s / OK.

• Final Python tooling validation:

  python3 -m unittest discover -s scripts -p 'test_*.py'

  Ran 57 tests in 29.272s / OK, including the original reproduction, both cap cases, one-line errors, passing tests, and unchanged Rust-output fixtures.

• Required local affected-check invocation:

  CARGO_BUILD_JOBS=1 CARGO_PROFILE_TEST_DEBUG=0 NEXTEST_TEST_THREADS=3 scripts/affected-tests main

  The selector conservatively selected the full Rust suite. Compilation succeeded in 15m 27s. After 1,112 tests passed, the duplicate local run was intentionally interrupted once CI was confirmed green on the pushed head, following docs/testing.md's instruction to reuse successful CI. The local command reports exit 100 from that interruption (three in-flight tests marked SIGINT); it is not claimed as a local full-suite pass.

  The successful CI test job for f51cc32d ran cargo nextest run and reported:

  Summary [1415.027s] 4112 tests run: 4112 passed (3 slow), 97 skipped
  

  CI also passed all 59 Python tooling tests, formatting, clippy, and the site and Console jobs. No source changes followed that validated head.

Visual evidence

Not applicable: this change affects terminal summaries; no rendered UI changed.

Merge Danger

Door: two-way

The parser change can be reverted independently.

Blast Radius: tooling

A misidentified block boundary could omit or add summary lines. Complete logs remain available for diagnosis.

Review coverage

Reviewed the whole branch against refs/remotes/origin/main (pinned to 4b9e6a4e8415e3d3a58a5f618613a1de2377e0a7), followed by review of the native-frame fix. Standards raised 0 findings. Spec raised 1 finding, reproduced and fixed with its regression test retained.

Both reviewers read all three changed files: docs/testing.md, scripts/affected-tests, and scripts/test_affected_tests.py. No changed files remain unread. No generated outputs were excluded. The reports and finding counts are saved in the Session's .thirdshift-review-5Y7MDT/ artifacts.

Unaddressed findings

The current review has no unaddressed findings. These design suggestions from the previous review remain declined, with the decisions and evidence retained for the whole branch.

Standards

• Prior Spec: thirdshift, turn a GitHub issue into a ready-for-review PR unattended #2, tuple state / possible Primitive Obsession: declined. scripts/affected-tests:48–59 documents and unpacks the three fields locally. Plan decision: keep this function-local parser state as a tuple; no caller must construct or interpret it, so a new domain type would not simplify the interface.
• Prior Walking skeleton: a fresh run on the happy path #3 (part), ERROR_LINES name: declined. scripts/affected-tests:30–31 defines it as the number of browser-message lines printed; the slice at line 40 uses that meaning directly.
• Prior Pre-flight checks #4, Rust and browser states in one loop: declined. The issue's Key interfaces explicitly ask print_failures to follow an error block “the way it already follows a Rust panic message.” Plan decision: extend this parser locally; splitting it into separate parsers is outside this fix.
• Prior The Failed run path #5, repeated strip() calls: declined. scripts/affected-tests:53–83 uses stripping for distinct indentation, frame, and header checks. Plan decision: retain this local style; the review identified no behavioral defect or documented rule requiring cached stripped text.
• Prior Numbered Issue branches after a merged or closed PR #7, a helper for TAP/spec fixtures: declined. scripts/test_affected_tests.py:538–556 keeps each reporter's literal format visible and independent. Plan decision: retain fixtures beside their assertions, consistent with the neighboring reporter tests at line 509.
• Prior Keeping the PR mergeable: merge the Base branch and conflict Repair #8, conditional prefix in an assertion: declined. scripts/test_affected_tests.py:565 encodes the one expected difference: spec prefixes the message with Error: while TAP does not. Both reporter branches are exercised by the passing original-reproduction command above.

Spec

• Prior Pre-flight checks #4, hand-written site descriptions (informational): no edit needed. The run rg -n '1898|affected-tests|failure summary|panic message|browser test' site/index.html site/how-it-works site/architecture returned no matches. Those pages contain no description of this summary to correct; docs/testing.md is updated.

No declined finding needs a Spec or Day shift decision, so no triage issue was required.

Closes #1898

Security review outcome

No unaddressed introduced findings.

Built with codex codex-cli 0.164.0 · gpt-6.1-sol · xhigh