Skip to content

feat(agent): describe images for text-only models - #622

Merged
AnthonyRonning merged 1 commit into
masterfrom
codex-maple-contextual-image-helper
Jul 13, 2026
Merged

AnthonyRonning merged 1 commit into
masterfrom
codex-maple-contextual-image-helper

Conversation

@AnthonyRonning

@AnthonyRonning AnthonyRonning commented Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • keep Goose native read_image behavior for vision-capable Agent models
  • use a hidden, thinking-disabled Gemma 4 completion to turn images into text-only tool results for nonvision models, guided by explicit task context
  • derive vision support from the advanced model catalog and lock the Agent model after the first message
  • bound and isolate the helper request, strip raw images on every nonvision outcome, and account for helper usage without corrupting primary context-window metrics

Validation

  • pre-commit suite: production frontend build, 77 frontend tests, and 103 Rust tests
  • TypeScript, ESLint, Prettier, cargo fmt --check, and Clippy
  • provider-boundary test confirms Gemma model selection, image serialization, no tools, and thinking disabled
  • rebuilt macOS app E2E: GLM receives a useful text description for a remote image; Gemma keeps the native image path without a context parameter
  • independent final Rust and frontend review passes

Open in Devin Review

Summary by CodeRabbit

  • New Features

    • Added support for detecting model vision capabilities, including aliases.
    • Improved image-reading tools for models with and without vision support.
    • Locked the selected agent model after a session begins.
    • Added vision capability information to agent message requests.
  • Bug Fixes

    • Prevented sessions from switching models after messages are sent.
    • Disabled unsupported image and reasoning features when unavailable.
  • Tests

    • Added coverage for model locking, vision capability resolution, image handling, and session state detection.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8183f655-90f2-4b16-9caf-cf6262c10d8b

📥 Commits

Reviewing files that changed from the base of the PR and between 6df18c8 and e1c9012.

📒 Files selected for processing (9)
  • frontend/src-tauri/src/agent.rs
  • frontend/src-tauri/src/agent/developer_tools.rs
  • frontend/src-tauri/src/agent/shell_permission.rs
  • frontend/src/components/AgentMode.tsx
  • frontend/src/services/agentModels.test.ts
  • frontend/src/services/agentModels.ts
  • frontend/src/services/agentRuntimeService.ts
  • frontend/src/services/agentTimeline.test.ts
  • frontend/src/services/agentTimeline.ts

📝 Walkthrough

Walkthrough

The agent now locks a session’s model after messages begin, sends resolved vision capability, and configures image tooling for direct vision or bounded contextual descriptions. Frontend model selection, backend requests, usage tracking, and shell classifier parameters were updated with tests.

Changes

Vision-aware agent flow

Layer / File(s) Summary
Model capability and selector state
frontend/src/services/agentModels.ts, frontend/src/services/agentTimeline.ts, frontend/src/components/AgentMode.tsx, frontend/src/services/agentRuntimeService.ts, frontend/src/services/*test.ts
Model and alias vision metadata are resolved, started sessions lock model selection, and outgoing requests include visionCapable.
Session model locking and vision wiring
frontend/src-tauri/src/agent.rs
Session setup validates model changes, accepts vision capability, and passes it into developer-client configuration.
Contextual image tool fallback
frontend/src-tauri/src/agent/developer_tools.rs
read_image supports direct image inspection for vision models and bounded text descriptions, including metadata preservation, usage recording, and coverage tests.
Thinking-disabled classifier parameters
frontend/src-tauri/src/agent/shell_permission.rs
Shell permission classification uses the shared thinking-disabled request-parameter helper and updated tests.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Sequence Diagram(s)

sequenceDiagram
  participant AgentMode
  participant agentRuntimeService
  participant agent_send_message
  participant MapleDeveloperClient
  participant ImageDescriptionProvider
  AgentMode->>agentRuntimeService: send visionCapable and selected model
  agentRuntimeService->>agent_send_message: submit agent request
  agent_send_message->>MapleDeveloperClient: configure vision-aware developer tools
  MapleDeveloperClient->>ImageDescriptionProvider: describe image when primary model lacks vision
  ImageDescriptionProvider-->>MapleDeveloperClient: return contextual text
Loading

Poem

A rabbit hops through model gates,
And locks the choice when chatting starts.
Images bloom or turn to words,
With bounded prompts and careful hearts.
Vision flags dance down the stream—
Carrots salute the clever scheme! 🐇

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding image description support for text-only agent models.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex-maple-contextual-image-helper

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying maple with  Cloudflare Pages  Cloudflare Pages

Latest commit: e1c9012
Status: ✅  Deploy successful!
Preview URL: https://468bbfd7.maple-ca8.pages.dev
Branch Preview URL: https://codex-maple-contextual-image.maple-ca8.pages.dev

View logs

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

@AnthonyRonning
AnthonyRonning merged commit c3f1f9c into master Jul 13, 2026
18 checks passed
@AnthonyRonning
AnthonyRonning deleted the codex-maple-contextual-image-helper branch July 13, 2026 19:35
AnthonyRonning added a commit that referenced this pull request Nov 1, 2026
* agent: let a started task switch models

A task was locked to its model after the first message (#622), because a
local agent could not follow a change of context window or vision. The Pi
runtime follows a switch itself now: `set_model` records it, clamps the
thinking level and tells extensions, `read_image` follows the model's
vision, and the check before each prompt compacts against the new window.

- Runtime: a run on another model switches the task's session, loaded or
  not; a cold task opens on the model it last used and then switches, so
  every switch is recorded the same way. The task's stored model follows
  from the start of the run. A switch takes the account's thinking
  default as it is now, as Pi's switch takes its settings' default.
- `set_task_model` takes a selection for any task's next run; the store no
  longer refuses a model patch after the first message.
- Thinking: a started task previews and keeps a pick for the model it
  will switch to, without switching before its next run.
- ACP: the model selector keeps every model after the first prompt, a
  started task accepts another model, and a task whose model is gone
  loads on the default instead of being refused.
- Desktop: the model menu lists every model in a task, with no lock icon
  or note. A pick applies from the next message; the selector follows a
  switch the task makes here or on another surface.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* agent: note a model switch in the transcript

A run that switches a started task to another model records a Maple
notice entry, "Switched to Kimi K3", and shows it live as the run starts.
It renders through the existing notice row, as "Stopped by user" does, so
reopening the task shows it from the stored entries. ACP keeps notices out
of its stream and shows the model in its selector.

Pi's own `model_change` entries are not projected: telling a switch from
the first selection or a new window for the same model needs the
previous model, which the incremental and chunked timeline rebuilds do
not carry, and the store has no catalog names.

The catalog's label precedence moves from the composer to the runtime, so
the notice and the model menu name models the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* agent: summarize on the previous model after a switch to a smaller window

Pi compacts with the session's model, in one request. Before the first
prompt on a model with a smaller window, its threshold check compacts a
history the new model may not be able to read: a 500k-token DeepSeek
task moved to a 262k model sends the summary request to the 262k model.

A Maple extension on Pi's `before_compact` takes that case. When the
branch's last complete reply comes from another model with a larger
window, that model writes the summary with Pi's own `compact`. The model
is read from the stored reply, so this holds across restarts. When the
catalog no longer offers it, or its summary fails, the new model
summarizes the history trimmed to its window, oldest messages first.

The extension returns no summary otherwise: on the same model, after a
switch to a window at least as large, when the new model takes the whole
history, and when both attempts fail. Pi then compacts as before. The pi
crates are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(agent): model switching and the upstream fidelity rule

- UPSTREAM.md: the pi crates mirror upstream Pi, bugs included; only
  blockers are fixed locally, review findings that match upstream are
  recorded and left, and Maple behavior goes in `maple-agent`, usually as
  an extension.
- README and the scenario checklist describe switching models mid-task
  and the compaction after a switch to a smaller window.
- The catalog's reason for leaving aliases out no longer rests on the
  model lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* agent: show a switch notice once when a run reloads its history

The switch notice was published live without the identity of its stored
row. A compaction during the same run reloads the task's history and
merges the run's live rows by that identity, so the desktop showed the
notice twice until the task was opened again.

The run now links the live row to the notice's entry, as it links message
rows, and the row carries the stored identity once it commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant