Skip to content

fix: report responses cut off at max_tokens - #599

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fermibubble:report-truncated-responses
Nov 18, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fermibubble:report-truncated-responses

Conversation

@fermibubble

@fermibubble fermibubble commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

When a reply hits max_tokens, the API returns finish_reason="length", but nothing checks it. The cut-off reply is parsed as if it were complete, and the iteration fails with "No valid diffs found in response", or "Unmatched SEARCH/REPLACE delimiter" when the cut lands inside a block. Both point at the model's output format (see #241), so it's easy to look in the wrong place.

Thinking models count reasoning tokens toward max_tokens, so this comes up a lot. In a 40-iteration run with gemini-3.5-flash and max_tokens: 8192, 20 iterations failed with "No valid diffs found", and every one of them used exactly 8188 tokens on top of the prompt. The 20 that worked used 2,540 to 7,290. Sending one of the run's prompts to the API directly, 2 of 4 replies came back with finish_reason="length": one stopped mid-sentence before the diff, the other partway through its block. With the default max_tokens: 4096, all 4 were cut off. The log gives no hint, because Gemini's completion_tokens leaves out thinking tokens (completion: 326 out of a total of 13676). The same run with max_tokens: 32768 had no failed iterations out of 40.

Changes:

  • OpenAILLM._call_api logs a warning with the model and the limit when finish_reason == "length", and adds finish_reason to last_usage.
  • The iteration error says when the reply was cut off. From a real run:
    Iteration 1 error [diff]: No valid diffs found in response (response was cut off at llm.max_tokens) | tokens: 11547 (prompt: 3359, completion: 325)

Cut-off replies whose complete blocks still parse are applied as before. I left that alone, but can reject them instead if you'd rather.

Tests: new tests/test_openai_finish_reason.py and a TestWorkerTruncatedResponse class in tests/test_diff_format_validation_worker.py. 4 of the 5 new tests fail on main. Full suite (python -m unittest discover -s tests -p "test_*.py"): 1081 OK, 7 skipped.

@CLAassistant

CLAassistant commented Nov 12, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

A reply that hits max_tokens comes back with finish_reason="length", but nothing checked it. The cut-off reply was parsed as if it were complete, and the iteration failed with "No valid diffs found in response" (or an unmatched delimiter error), which points at the model's output format instead of the limit. Thinking models count reasoning tokens toward max_tokens, so this happens a lot with them. _call_api now logs a warning with the model and the limit and records finish_reason in last_usage, and the iteration error says when the reply was cut off.
@fermibubble
fermibubble force-pushed the report-truncated-responses branch from 0638478 to d6b082e Compare November 18, 2026 02:26
@codelion
codelion merged commit 4c0fa22 into algorithmicsuperintelligence:main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants