A raw-model start with an unknown model launches anyway and fails with 'prompt could not be delivered' #384

Problem

A raw-model start with a model no provider offers launches anyway and fails after launch. The person sees only "the session started but the prompt could not be delivered; it was ended". Seen in real use on 2026-10-11: asked to start a codex session on a project, the Chief of Staff sent model: "codex" and got that message twice. It then guessed at the cause.

Root cause

A start with a raw model string is not checked before launch. "codex" matched no provider prefix, so relay started a plain chat session with that model. The model endpoint answered HTTP 404: model not found, the first prompt was not delivered, and the session was ended. The 404 never reaches the caller.

Fix

Before launch, check a raw model against the models the resolved provider offers. Refuse an unknown model with an error that names it and lists the valid choices. Starts by agent-list entry (relay#371) already check up front; this covers projects with no agent list and raw starts on every door.

Follow-on from the relay#371 contract, open question 8 (approved by the owner on 2026-10-12).

ship-blocker: no (a start with an unknown model fails late with a vague message; nothing lost, no security effect)

Contract

Planner, 2026-10-18. Bug, standalone (not an epic). Branch fix/384-raw-model-offer off main; one PR to main with Closes #384. Evidence: the reproduction on main bf176b8 in the comments. Threat model: no promise moves; this is an early refusal, not access control. The model endpoint's grant stays the enforcement. ship-blocker: no (unchanged).

Smallest fix: the check that agent-entry starts already run (checkEntryModel) runs for raw model starts too, in the one core every door shares (AuthorizeLaunch). No new abstraction.

Acceptance criteria

1. Relay refuses a fresh start whose raw model its kind does not offer, before any launch.
2. A refused start leaves no session and starts no agent process.
3. The refusal is 400 model_not_offered.
4. The message names the requested model and the kind relay derived from it.
5. The message lists the offered models that the project allows, at most 50.
6. Each listed model has the spelling that relay session start --model accepts.
7. When the model is codex, pi or claude, the message also says how to spell a model of that kind.
8. When relay cannot read the kind's model list, it refuses the start with 503 models_unavailable.
9. A start whose model the kind offers starts as before.
10. A Claude alias or a full Claude tier id starts as before, with no model list check.
11. The check applies to POST /api/sessions, relay session start (flags and --file) and the bridge session.start op.
12. The check applies to POST /api/chief-of-staff/sessions, for a model start and for a chat-template start.
13. A Chief of Staff refusal carries the code in error and the listed models in choices.
14. A refused start ends the session.launch event denied, with the refusal code as reason and the requested model in model.
15. A refused Chief of Staff start ends the chief_of_staff.start event the same way.
16. Resume, agent-entry starts, terminal starts and drop-in keep their current checks.
17. A start that an earlier check refuses keeps that check's code, for example model_not_allowed, model_system_only or read_only_needs_claude.

Paths

• Simple path: a person picks a model in eve's New Session picker, which lists only offered models, so this refusal does not show there. A person who asks the Chief of Staff in words ("start a codex session") gets the refusal with choices, and the Chief of Staff can retry with one of them.
• Power path: relay session start --model M (or --file), with relay model list for the valid ids. No new flag and no config: the offer comes from the catalogue relay already reads.

Interfaces

cmd/relay/agent_models.go:

// newModelOffer: KindChat joins KindPi's case. Both offer the model
// endpoint's non-system row ids; an empty list or a read error answers
// errModelsUnavailable.
case KindPi, KindChat:

// rawModelOffered reports whether a raw session model names a model in its
// kind's offer: codex drops "codex/"; pi keeps <id> of "pi/<provider>/<id>"
// (a malformed pi model is never offered); chat matches the id, or <x> of
// "llama/<x>" or "mlx/<x>", the spellings docs/model-endpoint.md accepts.
func rawModelOffered(kind, model string, offered []string) bool

cmd/relay/session_launch.go:

// checkRawModel refuses a fresh chat, codex or pi start with no entry whose
// model the kind does not offer. A list that cannot be read refuses the start.
func checkRawModel(req LaunchRequest, proj *config.Project, fields sessionLaunchAuditFields) *LaunchRefusal

• AuthorizeLaunch calls it when req.Entry == "", !req.Resume, req.Model != "" and req.Kind is chat, codex or pi.
• The call sits after the persist_session_invalid and read-only checks and before buildSessionRequest, so every existing refusal keeps its code (criterion 17).
• model_system_only in launch stays before it, unchanged.
• Refusals use deniedRefusal, so the denied session_launch audit row is written as today. The audit row shape does not change.

Refusals (texts are exact; <model> passes through capAuditText(…, maxRefusalNameRunes)):

• Choices: the kind's offer, each spelled project.SessionModel(config.AgentEntry{Kind: kind, Model: o}) (codex/<slug>, pi/relay-router/<id>, a chat id unchanged). They are kept when projectAllowsModel passes and capped at maxRefusalChoices. They go in LaunchRefusal.Choices.
• Kind-name hint: when <model> is exactly codex, pi or claude, the message ends with ; for a codex session use codex/<model>, ; for a pi session use pi/relay-router/<model> or ; for a claude session use haiku, sonnet or opus.

Wire (no shape changes):

• POST /api/sessions: {"error": "<message>"} with the status above.
• relay session start and bridge session.start: exit 1, and stderr carries the message.
• POST /api/chief-of-staff/sessions: {"error": "<code>", "message": "<message>", "choices": [...]}. choices is omitted when empty.
• Events: session.launch (in launchWithEvent) and chief_of_staff.start end denied with reason set to the code. Both also set model to the capped requested model, only on model_not_offered and models_unavailable. kind and project_id are set as today.

File ownership

• No file is shared. T2 reuses the features package's helpers (agStart, agProject, agSimple, agStartArgvs, cosDo, cosRefusal, g1sListSessions, requireEvent) and does not edit them or e2e/harness/*. A helper it lacks goes in its own file.
• Docs: routes.md gets a raw-model step beside "Start by agent entry" and lists model_not_offered and models_unavailable for raw starts. cli.md session start gets one sentence on the refusal. events.md adds model to the session.launch and chief_of_staff.start field cells.
• FEATURES.md: row G1.01 adds the refusal, the proof items event:session.launch=denied/model_not_offered and event:session.launch=denied/models_unavailable, and the T2 tests. Row G1.26 adds the refusal, event:chief_of_staff.start=denied/model_not_offered and e2e:TestBug384CosRawModelNotOffered. The names below are fixed so the rows can cite them.

Acceptance mapping and test surface

All criteria are e2e feature tests (integration) in T2's file. Nothing is Playwright: eve's screens do not change. The test-machine pass is T3's real-app check.

• Criteria 16 and 17 are held by the existing suite: TestAgentStartModelNotOffered, TestCosStartEntryRefusals, TestReadOnlyProjectsSession, TestCodexModelOutsideAllowedRefused, TestModelOutsideAllowedRefused, TestSystemModelChatRefused and the resume tests.
• Red on main: on main each refusal test gets 201 or exit 0 where it asserts the refusal. The door is the CLI and HTTP route that exist on main.
• Break-and-restore: make checkRawModel return nil; TestBug384RawModelNotOfferedRefused goes red. T2 records that break in breaks.json.

Existing tests and journeys checked. I expect none to break, so no fixture changes:

• e2e raw non-Claude starts: TestChatSessionAnswers, TestSystemModelChatRefused, TestModelKeyRequired and TestCosStartsChatTemplate use a fake host that offers fake-echo. TestFakePiAnswers picks its id from the list. TestFakeCodexAnswers and the codex starts in g1_terminals_test.go use codex/fake-echo, which the fake codex offers.
• TestReadOnlyProjectsSession starts pi/fake/fake-echo with no model host. It stays green only because the check runs after the read-only check; T1 must keep that order.
• TestModelOutsideAllowedRefused and TestCodexModelOutsideAllowedRefused are refused by model_not_allowed first.
• cmd/devboxverify api-phase journeys start Claude models (haiku) or a blank model.
• The screen-phase chat and codex journeys (nightly only) pick their model from GET /api/models, and the codex journey already FAILs when codex/gpt-6-luna is not listed. eve's journeys also pick from the offered list.

Waits

• T1: none; the build only.
• T2:
  • Relay boot: harness.Start's ready signal.
  • Session host: the service.manifest.register event (WaitSessionHost).
  • Model host: the model.host.register event (WaitModelHost).
  • The refusal is decided before any launch, so the HTTP response or the CLI process exit is the completion handle. "No session" and "no agent start" assertions read after it.
  • The event line: read by trace after the response (requireEvent).
  • Positive starts: the 201 response; no turn.
  • The audit row is written by an async queue with no flush signal, so T2 does not assert it. Row G1.18 already covers the denied row for every launch refusal.
• T3:
  • stack-up prints STACK up r384 <stack.json>.
  • Each harness and suite run ends with its process exit.
  • The real-app CLI ends with its process exit.
  • After ./build.sh, wait for service list to show relaysessions running.

Verification task (T3, last task, on the PR head)

• repo-suite: (cd e2e && go test -race ./features/) on the head.
• devboxverify:
  1. Unlock the devid keychain, then run npm ci in an eve main checkout.
  2. ./stack-up.sh r384 --relay <head checkout> --eve <eve main checkout>.
  3. From the head checkout: go run ./cmd/devboxverify --stack <stack.json> --phase api.
  4. ./stack-down.sh r384.
• real-app:
  1. Hold the devlock (RESTART/SCREEN/WORLD) and install the head with ./build.sh.
  2. On a local project with no agent list, run relay session start --project <id> --model codex. It must exit 1, name codex and list the offered models. Before: exit 0 started session … (repro on bf176b8).
  3. Same with --model nosuchmodel-xyz.
  4. --model haiku still starts; stop that session.
  5. Reinstall from main under the same lock.
• Post Verified: <head sha> · repo-suite: <result> · devboxverify: <result> · real-app: <result> · log: <path>.

The stack run takes no devlock. Only the real-app install does.

Budget

• T1: 4 code files and 4 docs. About +80/-5 lines of Go and +25 lines of docs.
• T2: 1 new file of about 250 lines, plus a few lines in breaks.json.
• Going past either is a question for the coordinator.

Open questions (recommendation first; the coordinator takes it)

1. INSIDE. Claude raw models: skip the offer check. The Claude offer lists only aliases, and a full tier id is a Claude start by design (relay#494), so a check would refuse valid starts.
2. INSIDE. List unavailable: fail closed with 503 models_unavailable, as entry starts do. This keeps the repo's "every default fails closed" rule. Today such a start launches and then fails on its first message, so no journey loses a working path.
   • User-noticeable: with the model broker down, a chat start now answers 503 at once.
   • Log it in the run ledger.
3. INSIDE. Host projects: a raw codex start is checked against this Mac's codex list, as relay#371 question 5 decided. eve's picker already shows that list for a host project.
4. INSIDE. Hint text for codex, pi and claude (criterion 7): add it. It speaks to the incident, where the caller sent a kind name as a model and then guessed.
5. INSIDE. Choices use the --model spelling (codex/…, pi/relay-router/…), not the bare ids that entry refusals list, so a caller can send one back unchanged.
6. BEYOND. fakerelay still accepts any raw model. Recommend no change here. If an eve test needs this refusal, file a follow-on with needs-review and from-dev.