Skip to content

set_model_response error responses are treated as final output, bypassing ReflectAndRetryToolPlugin retry #4525

Closed

Description

@chaerrypick01

Description

When SetModelResponseTool raises a ValidationError (e.g., LLM passes null for required string fields), the ReflectAndRetryToolPlugin correctly catches the error via on_tool_error_callback and returns a ToolFailureResponse. However, get_structured_model_response() in _output_schema_processor.py only checks the tool name (set_model_response) without inspecting whether the response is an error or a valid result. This causes the error response to be treated as the final model output, completely bypassing the plugin's retry mechanism.

Steps to Reproduce

  1. Create an LlmAgent with output_schema (e.g., required str fields) and at least one tool
  2. Use a non-Vertex AI model (e.g., LiteLLM) so can_use_output_schema_with_tools() returns False and SetModelResponseTool is injected
  3. Enable ReflectAndRetryToolPlugin with max_retries=3
  4. Trigger the agent where the LLM calls set_model_response with null for a required string field

Expected Behavior

ReflectAndRetryToolPlugin should retry the set_model_response call up to max_retries times, giving the LLM a chance to correct its output.

Actual Behavior

The ToolFailureResponse (with response_type: "ERROR_HANDLED_BY_REFLECT_AND_RETRY_PLUGIN") is immediately treated as the final model response. No retry occurs.

Root Cause

In base_llm_flow.py (lines ~649-659 and ~695-705):

yield function_response_event

if json_response := _output_schema_processor.get_structured_model_response(
    function_response_event
):
    final_event = _output_schema_processor.create_final_model_response_event(
        invocation_context, json_response
    )
    yield final_event  # Error response becomes final output

And in _output_schema_processor.py (lines 111-114):

for func_response in function_response_event.get_function_responses():
    if func_response.name == 'set_model_response':
        return json.dumps(func_response.response, ensure_ascii=False)  # No validation

get_structured_model_response() matches only on func_response.name == 'set_model_response' and returns the response as-is, without checking if it's a valid schema result or an error/reflection response from a plugin.

Suggested Fix

get_structured_model_response() should validate the response against the agent's output_schema before treating it as the final output. If validation fails (e.g., the response is a ToolFailureResponse from a plugin, not actual schema data), return None so the LLM loop continues and the model can retry.

def get_structured_model_response(
    function_response_event: Event,
    output_schema: type[BaseModel] | None = None,
) -> str | None:
    if (
        not function_response_event
        or not function_response_event.get_function_responses()
    ):
        return None

    for func_response in function_response_event.get_function_responses():
        if func_response.name == 'set_model_response':
            response = func_response.response
            # Validate against output_schema before accepting as final response
            if output_schema and isinstance(response, dict):
                try:
                    output_schema.model_validate(response)
                except ValidationError:
                    return None  # Not valid schema data; let the LLM loop continue
            return json.dumps(response, ensure_ascii=False)

    return None

This approach is plugin-agnostic — it doesn't need to know about ReflectAndRetryToolPlugin or any specific error response format. Any response that doesn't match the output_schema (error responses, reflection guidance, etc.) is naturally skipped.

Note on throw_exception_if_retry_exceeded=False

When ReflectAndRetryToolPlugin is configured with throw_exception_if_retry_exceeded=False and max retries are exceeded, the plugin returns a "do not use this tool again" guidance message instead of raising an exception. With this fix, that message would also fail schema validation and return None, causing the LLM loop to continue. However, since output_schema requires the LLM to eventually call set_model_response, this creates a contradictory situation where the plugin tells the LLM not to use the tool it must use. This is a pre-existing design tension between set_model_response and throw_exception_if_retry_exceeded=False, not introduced by this fix. The default setting (throw_exception_if_retry_exceeded=True) handles this correctly by raising an exception to stop the loop.

Environment

  • google-adk: 1.23.0+
  • Model: Gemini 2.0 Flash via LiteLLM (non-Vertex AI)
  • Python: 3.11

Activity

  1. added
    core[Component] This issue is related to the core interface and implementation
    on Feb 18, 2026
  2. added theissue type on Feb 18, 2026
  3. surajksharma07 commented on Feb 18, 2026

    @surajksharma07
    Collaborator

    @chaerrypick01 confirmed this in get_structured_model_response() (_output_schema_processor.py:112), which matches only on func_response.name == 'set_model_response' and returns the response as-is, so a ToolFailureResponse from the plugin gets blindly promoted to final model output before any retry can happen.

    The fix is to add a guard before the return json.dumps(...) line: check if isinstance(response, dict) and response.get("response_type") == "ERROR_HANDLED_BY_REFLECT_AND_RETRY_PLUGIN": return None — this causes the flow to skip creating the final event and lets the LLM loop continue for retry.

    Your schema-validation approach (output_schema.model_validate(response)) also works and is more plugin-agnostic,
    but it requires updating both call sites in base_llm_flow.py (~lines 1023 and 1069) to pass invocation_context.agent.output_schema as well.

    Either way, not fixed in v1.25.0 — raising this for a patch.

  4. surajksharma07 commented on Feb 18, 2026

    @surajksharma07
    Collaborator

    @seanzhou1023 Please have a look into it.

  5. self-assigned this
    on Aug 6, 2026
  6. GWeale commented on Aug 6, 2026

    @GWeale
    Collaborator

    the fix #6368 is on main but not in any release yet, so upgrading today will still reproduce it. it ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementationneeds review[Status] The PR/issue is awaiting review from the maintainer

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions