Economy Mode: Honest E2E Validation, Reliable Activation, and Cost Measurement #585

Economy Mode: Honest Assessment, Reliable Activation, and E2E Validation

Problem Statement

Economy mode claims to save money by substituting cheaper models. We need proof. Early testing in agent compute showed economy mode using MORE tokens overall, potentially negating per-token savings. We cannot ship a feature that claims to save money if it doesn't.

Current State (as of v0.9.1)

What economy mode does today:

• Model substitution ONLY via ECONOMY_MODEL_MAP in packages/squad-sdk/src/config/models.ts:523-533
• Premium → Standard: opus → sonnet-4.5 (5x cheaper per token ✅)
• Standard → Fast: sonnet → gpt-4.1 (15-19x cheaper per token ✅)
• Fast → Cheapest: haiku → gpt-4.1 (4-5x cheaper per token ✅)
• Only affects Layers 3-4 (auto-selection). Layers 0-2 (explicit user overrides) are immune.

What economy mode does NOT do:

• No prompt shortening
• No agent count reduction
• No context window optimization
• No token budget enforcement
• No quality-aware gating (cheaper model may generate more tokens)

Activation paths today:

1. squad economy on — persistent config write to .squad/config.json
2. --economy CLI flag — session-scoped env var SQUAD_ECONOMY_MODE=1
3. SQUAD_ECONOMY_MODE=1 env var — direct
4. Coordinator skill (.squad/skills/economy-mode/SKILL.md) — user says "use economy mode" but this depends on LLM interpreting the phrase. NOT reliable.

The Core Concern

Per-token pricing is definitively cheaper for every swap. But total cost depends on total tokens consumed:

• If gpt-4.1 produces 15x more tokens than sonnet on the same task, savings are zeroed out
• If cheaper models fail more and require retries/fallbacks, savings are negative
• We have zero data on total-token-consumption per task in economy vs normal mode

Where We Want To Be

1. Reliable activation: A user should be able to type ONE command or set ONE flag and be in economy mode. No concepts to learn. No coordinator skill dependency. Binary: you're in economy mode or you're not.
2. Honest measurement: E2E tests that run identical conversations in normal and economy mode, measuring total tokens, total cost, task success rate, and output quality.
3. Reproducible experiments: Tests that can be re-run on every release to validate economy mode claims.
4. Clear documentation: If economy mode saves 40% on a typical session, say so with data. If it saves 5%, say that. If it costs MORE in some scenarios, document those too.

Measurement Plan

Metrics to capture per conversation:

• Total input tokens (all agents combined)
• Total output tokens (all agents combined)
• Total estimated cost (using MODEL_CATALOG pricing)
• Number of agent spawns
• Number of fallbacks/retries
• Task completion success (did the work actually get done?)
• Model used per spawn (verify economy map was applied)

Test scenarios (E2E, realistic conversations):

1. Code generation: "Write a REST API endpoint with tests" — normal vs economy
2. Bug fix: "Fix this failing test" (provide a broken test) — normal vs economy
3. Multi-agent fan-out: "Team, add error handling to module X" — normal vs economy
4. Documentation: "Write API docs for this module" — normal vs economy
5. Triage: "Label and prioritize these 5 issues" — normal vs economy
6. Mixed session: Multiple commands in one session — normal vs economy
7. Retry scenario: Give a task that's likely to need revision — measure retry cost
8. Edge case: Economy mode with Layer 0/1/2 overrides — verify they're respected

Validation approach:

• Each test runs twice: once with squad economy off, once with squad economy on
• CostTracker captures all usage events
• Compare: total cost, total tokens, success rate
• Report: "Economy mode saved X% on cost, used Y% more/fewer tokens, Z% success rate"

Reliable Activation Assessment

Current activation is TOO complex. A user shouldn't need to understand "layers" or "coordinator skills." Assess:

• Is squad economy on/off truly reliable? Does it persist across sessions?
• Does --economy flag propagate to ALL spawned agents in a session?
• Can a user activate economy mode mid-session reliably (not LLM-dependent)?
• Is there a visual indicator showing economy mode is active?
• What happens if config.json and --economy flag conflict?

Acceptance Criteria

• E2E test suite that runs real multi-turn conversations in both modes
• Cost comparison report generated automatically
• Tests prove (or disprove) that economy mode saves money
• Reliable, one-command activation verified
• If economy mode is broken, fix the model map or document limitations honestly
• No marketing claims that aren't backed by reproducible test data

Files to investigate

• packages/squad-sdk/src/config/models.ts (ECONOMY_MODEL_MAP, resolveModel, estimateCost, MODEL_CATALOG)
• packages/squad-sdk/src/agents/model-selector.ts (SDK resolveModel, selectModelForTask)
• packages/squad-cli/src/cli/commands/economy.ts (CLI command handler)
• packages/squad-cli/src/cli-entry.ts (--economy flag, env var propagation)
• packages/squad-sdk/src/runtime/cost-tracker.ts (CostTracker for measurement)
• .squad/skills/economy-mode/SKILL.md (coordinator skill — assess reliability)
• test/economy-mode.test.ts (existing 34 unit tests — need E2E additions)