[plainprogrammer]

Closes #177

What was wrong

The upgrade proof stops each instance it boots and compares databases only after a graceful stop (0 or 143). It sometimes got 137: the container was killed at the end of the 30 s grace period. Sightings: runs 42131733342 (PR #165), 42124366255 (main) and 42658829910 (PR #234, the upgrade instance stopped 2 s after its boot began), all Image (linux/amd64).

Cause (reproduced locally, arm64 Docker, image built from this branch)

A start-up race in Solid Queue's Puma plugin (gem 1.7.0; 1.7.2 doesn't change it). Puma's after_booted forks the Solid Queue supervisor; on a stop, after_stopped sends it INT and then calls Process.wait with no bound.

• Window 1, before the forked supervisor has trapped signals (about 0.1 s after Puma listens; up to ~2 s on a starved CPU): the supervisor inherits Puma's INT handler and swallows the INT, carries on, and Puma waits forever. The engine kills the container at the grace period: exit 137. Every 137 log has Stopping Solid Queue... before Started Supervisor(fork), and no terminated gracefully after.
• Window 2, the supervisor is up but its dispatcher, worker and scheduler are still booting: their TERM is lost, so the supervisor waits Solid Queue's 5 s shutdown_timeout and sends QUIT. A stop then takes 5-6 s (4-12 s at 0.3 CPU, past the 10 s default of docker compose stop).

The harness stopped an instance at its first /up answer (polled every 2 s), which lands in those windows by chance, more often on a slower runner.

Measured before the fix, same image:

Stop fired at	Result
first /up poll, idle / job queued / under load / after a second boot (18 stops)	all exit 143, 0.4-1.0 s
first /up answer, polled every 50 ms (15 stops)	3 took about 5.8 s (window 2)
Puma's "Use Ctrl-C to stop" line, polled every 20 ms (15 stops)	8 exit 137 at 30.5-31.1 s, 1 took 6.2 s
the same trigger with the container on 0.3 CPU (12 stops)	all 3.9-12.1 s

The fix (smoke side; no app change, no spec: maintainer's ruling)

• Upgrade::Harness#stop_and_snapshot first waits (wait_for_queue: up to 15 log reads 2 s apart, 28 s) for the Started line of each of Solid Queue's dispatcher, worker and scheduler, then stops. On timeout it fails (spec 031 FR-5 allows no warning that lets the job pass): the failure names the missing processes and quotes the container's last 20 log lines, and the instance isn't stopped. (The approved brief said proceed on timeout; the implementation review found that breaks FR-5, so this is a deviation, on purpose.)
• A non-graceful stop now fails with how long the stop took and the container's last 20 log lines.
• bin/image-smoke and bin/backup-smoke only rm -f their containers and never stop gracefully, so they need no wait.
• Operators: the same window exists for docker compose stop / Kamal when the stop arrives within about 1-2 s of the app becoming ready. The maintainer ruled it a known limit: no app-side fix, nothing filed upstream. It is stated in docs/architecture/release-and-deploy.md (accepted limits, An upgrade, constants table, Flaky: Image (linux/amd64) smoke fails with exit 137 at container shutdown #177 in Sources) and in README's Self-hosting section: nothing committed is lost (SQLite's write-ahead log recovers), a job running at that moment can be retried at /admin/jobs.

Proof

• Failing specs first (spec/script/upgrade/harness_spec.rb: waits for all three processes before the stop, fails naming what is missing without stopping, prints the stop's duration and last log lines); FakeEngine's default log now carries the three Started lines. 139 examples green in spec/script/upgrade and spec/smoke_scripts_spec.rb; the suite has 2860 examples.
• The harness's stop sequence on a real container (wait for /up, then for the three Started lines in the log, polling every 50 ms, then docker stop -t 30): 30 of 30 graceful, exit 143, 0.4-0.9 s; 15 of 15 on 0.3 CPU, exit 143, 1.9-6.5 s.
• Real bin/upgrade-smoke against the locally built image as previous and current (--previous-ref HEAD): 5 of 5 passed (41-78 s), none stopping anyway or failing.
• Image jobs on this PR are the CI proof; see the checks below.

Implementation review (Opus, read-only; no spec, so recorded here)

Verdict: ISSUES FOUND, no Critical; all fixed in 7c225321. Findings: the note-and-continue path broke spec 031 FR-5 (now a failure); the docs said three sightings by count and tied the operator window to the container's start rather than the app being ready; the constants table's line references were stale; "as after a power loss" undersold a kill and ignored a running job; "30 s" was 28 s of sleeps; JOB_CONCURRENCY > 1 isn't covered (now a comment on QUEUE_PROCESSES); the first example wouldn't catch all? becoming any? (it does now). Coverage confirmed: every stop that compares databases goes through stop_and_snapshot (previous, upgraded, restored, next); every release tag locks the same solid_queue and puma, so the previous release's log has the same lines; the failure's 20 log lines carry no email, card or location (the seeds run through docker exec, and the stopped instances serve only /up, /registration/new and /session/new).

Owed to the maintainer

• The app-side option (async mode, or an owned copy of the plugin with a bounded wait) and an upstream report, taken by the coordinator as a separate decision.
• Commits: 65590894 (the fix), d86e7870 (docs), 7c225321 (review fixes), a6ea6d7c (lessons and memory).

🤖 Generated with Claude Code