Skip to content

Dato Agent: avoid Anthropic output-limit failures on large writes - #160

Merged
roger-datocms merged 7 commits into
datocms:masterfrom
rodrigobarona:fix/dato-agent-output-limit
Nov 4, 2026
Merged

roger-datocms merged 7 commits into
datocms:masterfrom
rodrigobarona:fix/dato-agent-output-limit

Conversation

@rodrigobarona

Copy link
Copy Markdown
Contributor

Fixes #158.

What

  • Output budget: resolveAnthropicMaxOutputTokens now uses the 64k budget at every reasoning effort, still capped by the model's advertised maximum (and 16k when the maximum is unknown). Adaptive thinking shares max_tokens with the reply and the tool input, so the previous 16k cap at low/medium/high left too little room for multi-record write scripts. Anthropic bills only generated tokens.
  • One-shot recovery: when a response stops with max_tokens, the runtime discards it instead of ending the turn. The truncated assistant message is not kept, and none of its tool calls were dispatched. It then retries once, with a user-turn instruction to keep reasoning brief and split large writes into smaller scripts. Any partial visible text stays in the transcript, followed by a paragraph break. A second truncation in the same turn still ends with the existing retryable incomplete error. model_context_window_exceeded is unchanged.

Tests

  • Effort/budget table updated.
  • New: truncated response containing a tool_use is discarded, MCP is never called, and no approval is raised. The retry request has no assistant turn and carries the recovery instruction.
  • New: a second max_tokens stops with the retryable incomplete error.
  • npm run typecheck, npm run lint, npm test (952 passed).

- Use the 64k output budget (bounded by the model maximum) at every effort;
  adaptive thinking shares max_tokens with the reply and tool input.
- When a response stops at max_tokens, discard it (none of its tool calls ran)
  and retry once with an instruction to split the work into smaller scripts.

Co-authored-by: Cursor <cursoragent@cursor.com>
@roger-datocms

Copy link
Copy Markdown
Member

(Claude here [Opus 5.5])

Triage

Status: still relevant. Master (6f57d960) still caps low/medium/high at 16k (dato-agent/src/lib/agentRuntime.ts:5065). It also ends the turn on any max_tokens stop (agentRuntime.ts:6122). The PR's base is current master.

Mergeability: MERGEABLE, merge state CLEAN. The repo runs no CI checks on this PR. I ran them locally on 4534285e:

  • npm run typecheck: pass
  • npm run lint: pass
  • npm test: 952/952 pass

Review

The diagnosis matches the code. Adaptive thinking shares max_tokens with the reply and the tool_use input. 16k leaves too little room for a multi-locale write script.

Correct:

  • The runtime discards the truncated assistant message before any tool dispatch. streamSingleResponse emits no tool activity, so the UI shows no orphan tool call or approval.
  • Recovery runs once per loop state. cloneAnthropicLoopState spreads the flag, so approval resumes keep it.
  • Mid-loop, the instruction lands as a text block after the tool_result blocks. The Messages API accepts that order.
  • The cap still respects the model's advertised maximum. It falls back to 16k when the maximum is unknown.
  • model_context_window_exceeded behaviour stays unchanged.

Minor points (non-blocking):

  • The discarded response's visible text stays in the transcript, e.g. "Creating all sections now." The user can read it as a claim about work that never ran.
  • systemPrompt.ts:209 still asks for "one focused script … so a correct request normally needs one approval". That pushes the first attempt toward the oversized script. A hint to split writes beyond a few records would avoid the wasted first attempt.
  • A runaway low-effort turn can now generate up to 64k tokens before it stops. That means 4x the worst-case latency and spend. This is acceptable, but it belongs in the release notes.

Recommendation

Merge after a maintainer review. Address the systemPrompt.ts:209 wording here or in a follow-up. Fixes #158.

roger-datocms and others added 6 commits November 4, 2026 13:05
Every step of a tool loop resends the tools, the system prompt and the
whole history. Without cache_control, each step paid full input price
for all of it, so splitting a large write into smaller scripts multiplied
input cost and latency.

Mark the system prompt (which also caches the tools before it) and set
top-level automatic caching, so each step reads the previous step's
prefix. A live smoke turn with five steps dropped from $0.0104 to $0.0041.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live tests run the real runtime (createAgentRuntime) against a real
provider and the real DatoCMS Remote MCP, in a forked environment of a
seed project, and assert on the project's end state. Evals compare
request variants and write tables of rates and costs. Neither runs in
`npm test` or gates anything: `npm run test:live` and `npm run eval`.

- e2e/setup: seed the project schema; mint a project-scoped MCP token
- e2e/lib: env loading, fetch tap for usage and request rewrites,
  environment forks, the approve-everything turn driver, spend ledger
  with a per-key cap (default $10)
- smoke and output-limit live tests: one write; a 30-record large write;
  the max_tokens discard-and-retry path under a forced low ceiling
- output-limit eval: limit x script guidance x task budget x caching

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max_tokens is a hard cutoff the model cannot see, so capping it below
what the model supports only truncates large requests; script size is
the system prompt's job. Use the maximum discovered from the Models API
(128k on current models) and fall back to 64k, the smallest maximum of
any active Claude model. Anthropic bills only generated tokens, and
max_tokens does not count toward output rate limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a reply hits the output limit, the runtime now shows a status line
saying nothing in it ran and that it is redoing the work in smaller
steps, so the extra approvals that follow make sense. When the retry
also fails, or the chat outgrows the context window, the error names a
remedy the editor can act on: ask for fewer records at a time, or start
a new chat. OpenAI's incomplete reasons (max_output_tokens,
content_filter) get the same treatment instead of a raw reason code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The config screen is developer-facing, and few people know a model's
limits offhand. Anthropic's model list reports both, so label each
option, e.g. "Claude Haiku 5.5 · 1M context · 128K output". OpenAI's
list reports neither, so its options keep their plain labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roger-datocms

Copy link
Copy Markdown
Member

I made a few additional changes:

  • Raised max output length to 128k
  • Added prompt caching to the Anthropic models
  • Surfaced max-output errors to the editor
  • Added some new tests
1-model-selector 2-output-limit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dato Agent: Anthropic turns fail with "reached its output limit" on multi-record writes

2 participants