Repository navigation
Open
Labels
deferredApproved but not scheduled: never startApproved but not scheduled: never startenhancementNew feature or requestNew feature or request
Description
Story
As the owner, I want verify's model-backed journeys to run without a paid provider, so that a run costs nothing and a provider rate limit never blocks it.
Acceptance criteria
- A verify run completes the chat journeys with no call to a paid provider.
- The chat journeys pass as often on the local model as on Haiku across 10 consecutive runs.
- A run with the provider unreachable reports PASS for the chat journeys, not BLOCKED.
- The Claude Code journeys either run through the model map to the local model or stay on Haiku by setting.
- Each run's output names the model it used.
- Switching back to Haiku needs one setting and no code change.
What I'd notice
Verify stops reporting BLOCKED for provider rate limits. Nothing else changes.
Tasks
- Spike: point only the chat journeys at a local model with
RELAY_VERIFY_MODELand run 10 times. - Compare the pass rate with Haiku and record it on this issue.
- If stable, route the Claude Code journeys through a model map entry to the local model.
- Name the model in the run output.
- Document the setting in the devboxverify README.
Context
Deferred: the owner has more confidence in Haiku today and wants verify stable first. Swap in the local model once it is.
What exists already, read from the docs and not yet tried:
- The chat journeys take their model from
RELAY_VERIFY_MODEL(defaultChat) out ofGET /api/models. - Relay serves local models (
mlx/<alias>,llama/<alias>) and OpenAI-compatible endpoints. - Claude Code sessions reach relay's model endpoint through
ANTHROPIC_BASE_URLand can be mapped to another model. - relayLLM is bound on the VM bridge, so the verify VM can reach a model served on the host.
Risks to measure in the spike:
- A small model may call tools less reliably, which makes the journeys flakier.
- Some assertions depend on answer wording (research-citations).
- MLX needs the host's GPU, so verify then depends on the host being up with memory free.
- Fix the seed and temperature to keep runs repeatable.
Metadata
Metadata
Assignees
Labels
deferredApproved but not scheduled: never startApproved but not scheduled: never startenhancementNew feature or requestNew feature or request
Activity