[plainprogrammer]

After a refresh's worker crashes, /admin/catalog now reads "Interrupted…" with "Refresh now" enabled and a link to the failed job, instead of "Running." with the button disabled for up to 15 minutes.

Closes #60

Cause

Catalog::Refresh#call closes its run only from inside the job. When the worker dies, Solid Queue fails the job itself (ProcessPrunedError, ProcessExitError, ProcessMissingError) and the run row stays running. Catalog::RefreshOperation#state asked the queue only once a run was 15 minutes without progress, so the page read "Running." and counted the run as in flight. A second defect had the same cause: a refresh started then was skipped as "already running" by Catalog::RefreshRun.start!.

What changed

• Part A (the issue). A running run whose own job is in the queue's Failed list is interrupted: on the page, in the recent-runs list and in catalog:status (which now asks the operation, so the three agree). It is not in flight, so the button is enabled and the page stops polling, and the existing "See its failed job" link shows. A refresh started then closes that run and runs.
• Part B (beyond the issue's text; yours to confirm). The same holds after the failed job is discarded: a running run that recorded a job id whose job is no longer an unfinished job in the queue is interrupted. Without this, Interrupted → Discard brings "Running." back. Runs with no job id keep the 15-minute rule.
• 2-minute grace (Catalog::RefreshRun::JOB_ENDED_GRACE). A run counts as left behind by a failed or gone job only once its last progress is over 2 minutes old. This protects a refresh that is still alive after Solid Queue wrongly pruned its worker (host suspend, container pause). A real pruned crash is already 5 minutes stale, so it costs nothing there; after a crash the queue reports quickly (a killed forked worker, orphaned claims at boot) the page can read "Running." for up to 2 minutes.
• Queued state. A left-behind run (crashed or stalled) reads "Queued." while another refresh job is in flight, instead of "Interrupted" with a disabled button and no reason.
• The queue is read before the runs, so a job finishing normally between the two reads can't show as interrupted.
• Spec 015 (version 1.2.1): glossary "Interrupted", AC-3.2, AC-3.3, a narrower AC-3.4, new AC-3.9, FR-1, Error Scenarios. README and CLAUDE.md match. No view, CSS, copy or migration change.

Decisions for you

1. Keep part B? The reviewer recommends keeping it. Dropping it means reverting 0ec6839, abbb376 and 631d99d with a small manual resolution in start!, and keeping the queue.read calls in RefreshOperation#run and #recent_runs, which part A also needs.
2. Two minutes for the grace.

Verification

• bin/ci on 2d595d3: passed. RSpec 1262 examples, 0 failures, 1 pending (the art agreement spec, unrelated).
• New examples failed before their code (part A 13, part B 12, review fixes 8 + 2 + 5).
• Two independent read-only reviews (Opus): the first checked the state logic against Solid Queue 1.7.0's source for each ordering (claimed without a run, retry_on, admin retry, graceful release, finished jobs not preserved, the concurrency lock) and its findings were fixed; the second found no defects in the fixes.
• Not verified: a live crash or suspend with a real worker in development, and the page in a browser. The evidence is specs and Solid Queue's source.

Known limits and follow-ups

• Right after a suspended host resumes, a live run's last progress is as stale as its worker's heartbeat. Until the worker next records progress (normally 2 s, up to about 70 s in a hung request) the run can read interrupted, and a refresh started in that window runs beside it. Recorded in spec 015's Error Scenarios.
• In that window, the first worker's later finish! would overwrite the closed run's status, because finish! doesn't check the run is still running. This predates the branch (the stall rule has the same exposure). Read in the code, not tested.
• MTG::Art::Operation has the original flaw in its own code: "Building." with its button disabled for up to 10 minutes after a crash. Not changed here.
• MTG::Scryfall::Client sleeps for Scryfall's Retry-After with no upper bound, which can stall a refresh without progress.
• Catalog::RefreshJob.perform_now from a console records a job id with no queue row, so that run reads as interrupted after 2 minutes without progress. No shipped path does this.
• Review suggestions left open: the queue_read_at guard is unreachable now that the grace exists (harmless defence in depth; two of its examples pass without it); the :running factory sits exactly on the grace boundary; facts can show "Retrying at" under a plain "Queued." summary; the glossary doesn't mention the queued exception.
• Commits abbb376 and 631d99d say "the coordinator ruled", which is session wording for the agent that supervised this work.
• This and A retried job shows "Attempts 0" #61's PR both edit spec 015's changelog; whichever merges second needs a small rebase. docs/roadmap.md still lists Catalog page reads "Running." after a crashed refresh #60.

🤖 Generated with Claude Code