Repository navigation
feat(agent): describe images for text-only models - #622
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughThe agent now locks a session’s model after messages begin, sends resolved vision capability, and configures image tooling for direct vision or bounded contextual descriptions. Frontend model selection, backend requests, usage tracking, and shell classifier parameters were updated with tests. ChangesVision-aware agent flow
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant AgentMode
participant agentRuntimeService
participant agent_send_message
participant MapleDeveloperClient
participant ImageDescriptionProvider
AgentMode->>agentRuntimeService: send visionCapable and selected model
agentRuntimeService->>agent_send_message: submit agent request
agent_send_message->>MapleDeveloperClient: configure vision-aware developer tools
MapleDeveloperClient->>ImageDescriptionProvider: describe image when primary model lacks vision
ImageDescriptionProvider-->>MapleDeveloperClient: return contextual text
Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Deploying maple with
|
| Latest commit: |
e1c9012
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://468bbfd7.maple-ca8.pages.dev |
| Branch Preview URL: | https://codex-maple-contextual-image.maple-ca8.pages.dev |
* agent: let a started task switch models A task was locked to its model after the first message (#622), because a local agent could not follow a change of context window or vision. The Pi runtime follows a switch itself now: `set_model` records it, clamps the thinking level and tells extensions, `read_image` follows the model's vision, and the check before each prompt compacts against the new window. - Runtime: a run on another model switches the task's session, loaded or not; a cold task opens on the model it last used and then switches, so every switch is recorded the same way. The task's stored model follows from the start of the run. A switch takes the account's thinking default as it is now, as Pi's switch takes its settings' default. - `set_task_model` takes a selection for any task's next run; the store no longer refuses a model patch after the first message. - Thinking: a started task previews and keeps a pick for the model it will switch to, without switching before its next run. - ACP: the model selector keeps every model after the first prompt, a started task accepts another model, and a task whose model is gone loads on the default instead of being refused. - Desktop: the model menu lists every model in a task, with no lock icon or note. A pick applies from the next message; the selector follows a switch the task makes here or on another surface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * agent: note a model switch in the transcript A run that switches a started task to another model records a Maple notice entry, "Switched to Kimi K3", and shows it live as the run starts. It renders through the existing notice row, as "Stopped by user" does, so reopening the task shows it from the stored entries. ACP keeps notices out of its stream and shows the model in its selector. Pi's own `model_change` entries are not projected: telling a switch from the first selection or a new window for the same model needs the previous model, which the incremental and chunked timeline rebuilds do not carry, and the store has no catalog names. The catalog's label precedence moves from the composer to the runtime, so the notice and the model menu name models the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * agent: summarize on the previous model after a switch to a smaller window Pi compacts with the session's model, in one request. Before the first prompt on a model with a smaller window, its threshold check compacts a history the new model may not be able to read: a 500k-token DeepSeek task moved to a 262k model sends the summary request to the 262k model. A Maple extension on Pi's `before_compact` takes that case. When the branch's last complete reply comes from another model with a larger window, that model writes the summary with Pi's own `compact`. The model is read from the stored reply, so this holds across restarts. When the catalog no longer offers it, or its summary fails, the new model summarizes the history trimmed to its window, oldest messages first. The extension returns no summary otherwise: on the same model, after a switch to a window at least as large, when the new model takes the whole history, and when both attempts fail. Pi then compacts as before. The pi crates are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(agent): model switching and the upstream fidelity rule - UPSTREAM.md: the pi crates mirror upstream Pi, bugs included; only blockers are fixed locally, review findings that match upstream are recorded and left, and Maple behavior goes in `maple-agent`, usually as an extension. - README and the scenario checklist describe switching models mid-task and the compaction after a switch to a smaller window. - The catalog's reason for leaving aliases out no longer rests on the model lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * agent: show a switch notice once when a run reloads its history The switch notice was published live without the identity of its stored row. A compaction during the same run reloads the task's history and merges the run's live rows by that identity, so the desktop showed the notice twice until the task was opened again. The run now links the live row to the notice's entry, as it links message rows, and the row carries the stored identity once it commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
read_imagebehavior for vision-capable Agent modelsValidation
cargo fmt --check, and ClippySummary by CodeRabbit
New Features
Bug Fixes
Tests