pags up should attach newly configured runner agents without restart #229

Problem

When pags up is already running, newly subscribed/configured runtime agents do not get attached to that runner. The user has to restart pags up before the new agent can use the local runner.

This showed up while configuring terminal test agents: the machine was online, but the new instances showed nodeOnline: true and connected: false until the runner was restarted with the new instance ids.

This is codebase-confirmed behavior, not just an observed UI glitch: the API route explicitly documents the state where a machine is online but a newly created agent is not attached because pags up started before that agent existed.

Current behavior from code

pags up snapshots runtime-capable instances once at startup:

• packages/cli/src/commands/up.ts:64-83 fetches /v1/instances/my/instances and filters active runtime agents.
• packages/cli/src/commands/up.ts:99-109 updates the TUI state from that fixed list.
• packages/cli/src/commands/up.ts:117-119 spawns runner connect <fixed instance ids...>.
• packages/cli/src/commands/up.ts:232-251 handles r, but if the child is still alive it only redraws status; it does not refetch instances or attach new ones.

runner connect also assumes a fixed instance set:

• packages/cli/src/commands/runner/command.ts:55-66 declares connect <instanceIds...> and receives the list from argv.
• packages/cli/src/commands/runner/command.ts:96-100 waits for the local runner, calls connectViaRelay(instanceIds, ...), then waits for runner exit.
• packages/cli/src/commands/runner/relay.ts:14-19 accepts instanceIds: string[] as a captured array.
• packages/cli/src/commands/runner/relay.ts:25-43 registers runtime metadata once for each id in that array.
• packages/cli/src/commands/runner/relay.ts:45-51 opens one relay socket per id in that array.
• packages/cli/src/commands/runner/relay.ts:64-73 heartbeats only that same fixed array.
• packages/cli/src/commands/runner/relay.ts:76-170 manages reconnect for one specific instance socket; there is no account/node-level socket or attach/detach manager.

Server-side evidence:

• workers/api/src/routes/instances.ts:414-418 explains the exact failure mode: machine online, but this agent is not attached because pags up was started before the agent existed and never opened that instance's socket.
• workers/api/src/relay-do.ts:69-90 rejects a second live runner for the same instance with 4409 unless --force is used, so auto-discovery must avoid noisy multi-machine takeover loops.

Desired behavior

A running unscoped pags up should attach newly configured runtime agents in near-realtime, without requiring a manual restart.

Expected flow:

1. User runs pags up once.
2. User subscribes to/configures a new runtime agent in the UI.
3. The active runner detects the new eligible instance.
4. It registers runtime metadata for that instance, opens its relay socket, and starts heartbeating it.
5. UI changes from nodeOnline: true, connected: false to connected: true automatically after the discovery interval.

Use the term near-realtime for the first implementation. True realtime would require a new account/node-level control channel or server push path; the current relay architecture only has per-instance sockets, so a brand-new instance has no socket to push over.

Proposed implementation

Start with CLI polling. Do not add server push for this ticket; issue #83 already tracks real-time status push instead of polling.

Runner membership manager

Refactor connectViaRelay(instanceIds, ...) so the runner owns a mutable membership set instead of a captured immutable array.

The manager should track attached instances, for example:

Map<instanceId, {
  name?: string;
  socket/control handle;
  reconnect/backoff state;
  conflict/backoff state;
}>

Add an attachInstance(instance) path that:

• POSTs /v1/instances/:id/runtime with the same local URL, token, capabilities, runner version, runner node, and force flag used today.
• Mints a relay token.
• Opens the relay socket.
• Adds the instance to the heartbeat set.
• Emits a useful CLI/TUI log line.

Add a detachInstance(instanceId) path that:

• Closes the local relay socket/control handle.
• Removes the instance from heartbeat.
• Emits a useful CLI/TUI log line.
• Avoids aggressive DB deletion unless the server contract is clarified; routing already checks RelayDO socket liveness, not only DB status.

Discovery loop

Add a watch/discovery mode to runner connect, for example --watch-instances, and have normal unscoped pags up use it.

When watch mode is enabled, poll /v1/instances/my/instances every 15-30 seconds and compute eligible instances:

• status === "active"
• capabilities.runtime != null

Attach newly eligible ids and detach ids that disappear, become inactive, or no longer need a local runtime.

pags up --instance X must remain fixed-scope and must not auto-attach other agents. This is already called out by the existing scoped restart comment in packages/cli/src/commands/up.ts:236-238 and should remain protected by tests.

Multi-machine and pinning constraints

This is the main correctness risk.

Auto-discovery must not blindly attach every eligible runtime agent from every online machine:

• If an instance is pinned to another runner node, do not attach it from this node unless --force/takeover semantics explicitly say to.
• If another live runner already owns the relay and RelayDO returns 4409, stop or heavily back off for that instance instead of retrying forever.
• If unpinned, attaching from the current node is acceptable, but avoid flapping if another node won the race.
• Preserve existing --force behavior for deliberate takeover.

Relevant code:

• workers/api/src/relay-do.ts:69-90 emits 4409 for another live runner.
• workers/api/src/lib/runner-client.ts checks relay liveness and honors runner-node pinning during routing.
• workers/api/src/lib/runtime-nodes.ts includes runner node in the relay object name.

Manual refresh improvement

As part of this ticket, make the r key useful while the child is alive:

• Either trigger an immediate discovery pass in watch mode, or clearly show that discovery is already active and when the last scan ran.
• Do not require the child to be dead before a newly configured agent can be picked up.

TUI updates

Update visible state when membership changes:

• Agent count should reflect the current attached/discovered set.
• Logs should include events such as Attached new agent: <name> (<id prefix>...) and Detached agent: <name> (<id prefix>...).
• Errors such as 4409 should surface as actionable conflict/takeover messages rather than endless generic reconnect noise.

Acceptance criteria

• Starting pags up, then subscribing to/configuring a new coding/browser/terminal runtime agent, attaches that agent without restarting the CLI.
• The Settings runner-node panel for the new instance changes to connected automatically after the poll interval.
• Existing connected instances remain connected while the membership set changes.
• pags up --instance X remains scoped to X and does not auto-attach other agents.
• Pressing r while the runner is alive triggers or reflects a discovery refresh instead of only redrawing stale state.
• Heartbeats include newly attached instances and stop for detached instances.
• A runner pinned to another machine is not auto-attached by this machine unless forced.
• A 4409 conflict does not create an infinite noisy reconnect loop for auto-discovered instances.
• Tests cover add/remove membership, scoped mode behavior, manual refresh, pinned-other-node behavior, and relay conflict/backoff.

Related issues

• CODER-007 — (Stretch) Real-time status push instead of polling (Durable Object / relay) #83 covers a later server-push/realtime status architecture. Do not block this polling MVP on that.
• [bug] Task cancel/delete/read routes still dispatch through stale default runtime rows #218 covers stale default runtime rows and task routing. This ticket should avoid making that worse, but does not need to solve it.
• [docs] Update Coder docs for runner-node picker, engines, and package paths #211 covers docs updates for runner-node behavior.