Repository navigation
fix(llm): accept list-shaped message content from chat completions - #1663
Open
localhostdotexe wants to merge 4 commits into
Open
localhostdotexe wants to merge 4 commits into
localhostdotexe wants to merge 4 commits into
Conversation
…sestrix#1357) Mistral and other OpenAI-compatible endpoints can send message content as a list of typed blocks (thinking + text) instead of a string. The SDK expects a string and fails validation on the first call. Add a chat-completions model that flattens that content: text blocks become the message text, thinking blocks become reasoning_content, unknown blocks are skipped.
StrixProvider.get_model now swaps the stock OpenAIChatCompletionsModel for BlockTolerantChatCompletionsModel. Custom subclasses and the LiteLLM route are unchanged.
Local server that answers like Mistral's reasoning models. Shows the stock SDK model failing on both paths and the tolerant model returning the text with the thinking kept as a reasoning item.
Contributor
[Medium impact] The follow-up changes appear safe to merge. SummaryThis PR lets chat-completions models accept replies made of content blocks while keeping text and reasoning separate.
Reviews (2) · Last reviewed commit: "test: don't leak globals from the provid..." · Reviewed by Greptile |
configure_sdk_model_defaults sets litellm.api_key/api_base/headers, the SDK default route and OPENAI_API_KEY process-wide. Register them with monkeypatch so later tests don't inherit the fake endpoint.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1357
Mistral's reasoning models (and some other OpenAI-compatible servers) send
message.contentas a list of blocks instead of a string, both in full responses and in every stream delta:[{"type": "thinking", "thinking": [{"type": "text", "text": "..."}]}, {"type": "text", "text": "OK"}]The SDK's chat-completions converter assumes a string, so the first call dies with a
ResponseOutputText(orResponseTextDeltaEventwhen streaming) validation error and Strix reports LLM CONNECTION FAILED.This flattens that content before the SDK reads it.
StrixProvider.get_modelnow returnsBlockTolerantChatCompletionsModel(newstrix/llm/content_blocks.py) in place of the stockOpenAIChatCompletionsModel. It overrides_fetch_response, so streaming and non-streaming are handled in one spot. Text blocks become the message text, thinking blocks go toreasoning_contentso the SDK keeps them as a reasoning item, and unknown block types are skipped. String content is untouched, and the LiteLLM route and custom subclasses are left alone.I went with a subclass rather than patching
chatcmpl_converterso nothing changes globally, and it's easy to drop if the SDK starts handling this itself.tests/test_content_blocks.pyruns against a local server that replies the way Mistral does. It checks that the stock model still fails (so the test really exercises the bug), that both paths return the text plus a reasoning item, that a thinking-only turn keeps its tool call, that string content is unchanged, and that the provider wiring is in place. ruff and mypy are clean, and the rest of the suite has the same failures here as on main.