Recover Runs after a crash or reboot, show live Runs, and coordinate Runs across machines #695

Problem

thirdshift lacks three things that a durable-execution engine such as Temporal would provide. The research in #694 (docs/research/temporal-for-thirdshift.md, §7) names them:

1. Recovery after a crash or reboot. The README says: "A process killed outright, with kill -9 or by a power cut, releases nothing: the issue stays in-progress for you to relabel." The Claim is never released, the Worktree and branch are left behind, and a Continuation has to be started by hand. The agent session itself is lost, even though most Harnesses can Resume it by session id.
2. A live view of Runs. Today a Run's state lives in the Command log, the Activity log and the Session logs, read after the fact. Nothing shows which Runs are in flight, which stage each is in, or which session is running.
3. Coordination across machines. Each machine runs its own Passes from its own crontab. Nothing stops two machines from working on the same issue, apart from the in-progress label. Nothing spreads work across machines, and nothing finds which machine a Run is on (make a way to easily see in progress thirdshift runs on machine and cut one off #574 hit this).

What done looks like

• A Run interrupted by kill -9, an OOM kill or a reboot is noticed and handled without a human. It either carries on from its last stage, resuming the agent session by id where the Harness allows, or ends cleanly: the Claim released, the issue relabelled, and the work preserved as for any other failed Run.
• One place shows every active Run on every machine: the issue, the machine, the stage, the session, and links to its logs. make a dashboard to view all present thirdshift activity on one server at a glance #368 and make a way to easily see in progress thirdshift runs on machine and cut one off #574 ask for parts of this.
• Runs on different machines cannot collide on one issue. The work can spread across machines within each host's capacity (about 2 building Runs on the current 4-core / 8 GB host).

Open questions for triage

• Adopt Temporal, or build these into thirdshift directly? The research recommends against adopting Temporal now. It would mean a long-lived Worker daemon, which reverses ADR 0009, plus a server or Temporal Cloud, determinism and Workflow-versioning rules, and a rewrite of the Delivery and Repair loop. Its suggested first step is a small prototype: one Rust Workflow, one heartbeated claude -p session Activity with --resume on retry, the Worker killed mid-session, and the dev server's memory measured on this host.
• Should this be split into three issues (crash recovery, live view, multi-machine)? Crash recovery is useful alone. The other two may share a design.

Related

• make a dashboard to view all present thirdshift activity on one server at a glance #368: a dashboard of all thirdshift activity on one server
• make a way to easily see in progress thirdshift runs on machine and cut one off #574: see in-progress Runs on a machine and cut one off
• make sending ctrl c on running implement result in switching in-progress label to ready-for-agent label #613: Ctrl-C on a running implement should hand the issue back to ready-for-agent
• Recover from temporary Harness service failures within a bounded retry budget #584: recover from temporary Harness service failures