Skip to content

fix(smoke): capture a page before testing it, and explain a failed check (#217) - #223

Merged
plainprogrammer merged 7 commits into
mainfrom
217-backup-smoke-collection-page
Nov 6, 2026
Merged

plainprogrammer merged 7 commits into
mainfrom
217-backup-smoke-collection-page

Conversation

@plainprogrammer

@plainprogrammer plainprogrammer commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Refs #217 (not Closes: the CI trigger isn't reproduced, so #217 closes after green Image (linux/arm64) jobs on main).

What is wrong

bin/backup-smoke's check of the restored collection page was curl -s … /collection | grep -q "Smoke Dragon" || fail … in a script under set -euo pipefail. grep -q exits at the first read that holds a match; if curl still has bytes to write it gets EPIPE (exit 23), and pipefail fails the pipeline although the lot was on the page. The same pipe-into-an-early-reader shape was in the script's port and CSRF-token cuts (| head -n 1) and in bin/image-smoke's port cut.

Evidence (the mechanism is proven; the CI trigger is not)

Probe: a server sends a page shaped like the restored /collection (13,516 bytes, first Smoke Dragon at byte 11,235 with about 2.3 KB after it, lines as the real page has) in two writes, and the script's exact check runs 100 times per case, with GNU grep 3.11 inside the collector image (CI's runners use GNU grep):

split gap curl | grep -q (before) capture, then grep (after)
none 0/100 0/100
6,951 5 ms 0/100 0/100
12,000 0 0/100 0/100
12,000 5 ms 2/100 0/100
13,000 5 ms 100/100 false failures 0/100
  • The real page does reach curl in varying pieces: curl --trace-ascii against one restored instance showed one receive of 13,516 bytes, 2,873+4,096+6,547 and 6,957+6,559.
  • The CI sightings in Flaky: backup smoke on arm64 fails "the lot isn't on the collection page" after a restore #217 (arm64 only, restores b and c, amd64 green on the same commits) are consistent with this; they are not proof of it.
  • macOS's BSD grep buffers before it matches, so the host never shows it; bin/image-smoke:37 already says "no pipe can end early under pipefail" for its own task output.
  • Not reproduced on the real app. 20 clean unmodified local runs, then about 3,800 pipe trials against the real restored instance (BSD grep idle and loaded; GNU grep from a --network host container through docker-proxy, idle and with the colima VM's 4 CPUs saturated): 0 failures. Nothing here failed before the change, so the clean runs below are not a before/after.
  • bin/upgrade-smoke and script/upgrade/ have no shell pipe (Ruby harness).

Change

  • spec/smoke_scripts_spec.rb (written first, red on bin/backup-smoke lines 51, 63, 107 and bin/image-smoke line 26, committed before the fix): sweeps bin/backup-smoke, bin/image-smoke and bin/upgrade-smoke for a pipe into head or a quitting grep, with negative controls for its matcher and a comment listing what it can't see.
  • bin/backup-smoke: the collection page is saved to a file with its status, size and redirect (collection_has_lot) and then searched; the port and CSRF token are cut without head; sign_in prints its status when it fails. When the lot check fails it prints the status, the byte count, the redirect target, whether the cookie jar holds the session_id cookie (the sign-in's; _collector_session is set by the form alone) and the first 300 bytes of the body.
  • bin/image-smoke: the port cut without head.
  • docs/architecture/release-and-deploy.md: the rule, the guard, the diagnostics, the honest limit, and Flaky: backup smoke on arm64 fails "the lot isn't on the collection page" after a restore #217 in Sources. .rubocop.yml: the new spec in the RSpec/DescribeClass excludes.
  • Two lessons and one memory (chore(memory)), approved by the coordinator.
  • No spec (direct fix, confirmed by the coordinator), no Dockerfile or workflow edit.

Proof

  • Guard: red on the four lines before the fix, 3 examples green after.
  • bin/backup-smoke collector:smoke217 on a local arm64 image (docker 29 through colima; podman isn't installed here and the anonymous pull of edge is denied, so the image was built from this checkout): 20 of 20 passes of the script as first fixed, then 20 of 20 of the final script. Each run 16–32 s. Again, not a before/after.
  • The diagnostics were exercised on throwaway copies: a missing lot prints answered: 200 13516 [status, bytes, redirect], the session_id line count and the page start; a wrong password prints sign-in … answered 422.
  • bin/image-smoke collector:smoke217: OK.
  • Full suite at the push: 2,832 examples, 0 failures (bin/rspec --dry-run count; the pre-push bin/ci passed at e1532e6).
  • CI on head e1532e6, run 42284830800: all green, Image (linux/arm64) (2m44s, job 128124758438) and Image (linux/amd64) included. One green arm64 job is weak evidence at the observed rate (2 of the last ~11 main runs), which is why this is Refs: Flaky: backup smoke on arm64 fails "the lot isn't on the collection page" after a restore #217 closes after green arm64 jobs on main.

Implementation review (Opus, read-only)

Verdict first: NEEDS CHANGES, then fixed to SPEC-ALIGNED in 01be01e. One major finding: the cookie diagnostic counted _collector_session instead of the login's session_id; fixed. Minor, applied: curl's status kept when curl fails, 0 printed for a missing jar, the exclude's alphabetical place, the probe's 5 ms gap stated, "about 3,800". Minor, noted: the guard reads line by line (it can't see a trailing |, |&, egrep, sed q; written into the spec), and sweeping the Ruby bin/upgrade-smoke is a guard for a future pipe.

🤖 Generated with Claude Code

plainprogrammer and others added 7 commits November 6, 2026 14:39
… scripts (#217)

The smoke scripts run under pipefail; curl | grep -q fails the pipeline when
curl still has bytes after grep has matched and gone. The guard sweeps
backup-smoke, image-smoke and upgrade-smoke. It is red on backup-smoke lines
51, 63 and 107 and image-smoke line 26 until the next commit.

Refs #217

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ck (#217)

bin/backup-smoke piped curl into grep -q under pipefail: once grep matched and
exited, curl's next write failed (exit 23) and the check reported the lot
missing although it was on the page. A page the size of the restored
/collection, delivered in two writes, failed 100 of 100 times with GNU grep.
The collection page is now saved to a file and searched, the port and CSRF
token are cut without a pipe into head, and image-smoke's port lookup the same.

When the check fails it prints the status, byte count, redirect target, whether
the cookie jar holds a session cookie and the first 300 bytes of the body; a
failed sign-in prints its status.

Refs #217

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds the rule, the guard that keeps it, the failure diagnostics, the honest
limit (the arm64 trigger was not reproduced; the issue closes on green arm64
jobs on main) and #217 to the release and deploy area doc's Sources.

Refs #217

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…iled fetch (#217)

Implementation review findings: the diagnostic counted _collector_session, which
the sign-in form sets alone, instead of session_id, the login's cookie; a curl
failure replaced the status line it had printed; the guard's limits and the
trial's 5 ms gap are stated; the exclude is in order.

Refs #217

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A memory for the macOS dev machine's container engine (docker via colima, no
podman, the mktemp shim, the denied anonymous ghcr pull) and two lessons: a pipe
into grep -q under pipefail, and a probe that finds nothing needing a positive
control.

Refs #217

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@plainprogrammer
plainprogrammer marked this pull request as ready for review November 6, 2026 20:09
@plainprogrammer plainprogrammer self-assigned this Nov 6, 2026
@plainprogrammer
plainprogrammer merged commit 59c1309 into main Nov 6, 2026
9 checks passed
@plainprogrammer
plainprogrammer deleted the 217-backup-smoke-collection-page branch November 6, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant