Skip to content

chat wrapper (run_llm_agent_as_node) extracts task-delegation FCs from partial=True progressive-SSE chunks and breaks — the FC event is never persisted, orphaning the synthesized task FR and permanently poisoning the session #6583

Closed

Description

@kenwilly

Environment

  • Python 3.13
  • google-adk==2.0.0
  • Default ADK feature: PROGRESSIVE_SSE_STREAMING (on by default)
  • Triggered under RunConfig(streaming_mode=StreamingMode.SSE)

Bug description

Under RunConfig(streaming_mode=SSE), ADK's StreamingResponseAggregator runs in
PROGRESSIVE_SSE_STREAMING mode (default-on) and marks every intermediate chunk
partial=True — including the chunk that carries a task-delegation FunctionCall. The
non-partial aggregate that re-carries the FC is only produced by aggregator.close()
after the model stream fully drains.

ADK's chat-mode wrapper, google.adk.workflow._llm_agent_wrapper.run_llm_agent_as_node,
extracts task-delegation FCs from every yielded event with no partial check,
dispatches the specialist, and breaks out of the agent's event generator. Two defects
follow:

  1. The FC event is never persisted. The Runner only appends events with
    partial is not True to the session; the break closes the generator before the
    non-partial aggregate is ever yielded. The synthesized task FunctionResponse is
    persisted — so the very next contents build sees an orphaned FR (a function response
    with no matching function call) and raises
    ValueError: No function call event found for function responses ids: {...}. The
    session is permanently poisoned: every later turn replays the same history and
    re-raises.
  2. Dispatch can fire with truncated args. Progressive SSE streams FC arguments
    across chunks; extracting from the first partial chunk that names the FC can dispatch
    the specialist with incomplete (even empty) input.

This defect is the dominant producer of the orphaned-function-response poisoned
state reported in the companion draft
docs/upstream-reports/02-orphaned-function-response-poisoning.md: the wrapper
synthesizes and persists a task-delegation function response while never persisting
the matching function call.

Reproduction

A self-contained Python repro drives a real Runner + coordinator(mode='chat') +
task specialist whose stub coordinator LLM streams the delegation FC in the exact
progressive-SSE shape (a partial=True chunk first, then the partial=False aggregate
re-carrying the same FC with the same id). No downstream workaround is installed — this
is the raw ADK 2.0.0 behaviour.

"""
Minimal repro: ADK 2.0.0 progressive-SSE task-dispatch defect.

Run:  cd <repo with google-adk==2.0.0> && python repro_partial_sse.py
Expected: the coordinator's re-entry contents build raises the
orphaned-response ValueError; the persisted session shows the task FR with no
matching task FC.
"""

from __future__ import annotations

import asyncio
import traceback
from collections.abc import AsyncIterator
from typing import Any

from google.adk.agents import LlmAgent
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_response import LlmResponse
from google.genai.types import Content, FunctionCall, Part

_COORDINATOR_CALLS: list[int] = []
_SPECIALIST_CALLS: list[int] = []


class _StreamingDispatchCoordinatorLlm(BaseLlm):
    """Coordinator stub emitting the progressive-SSE dispatch shape.

    Call 1 yields the task FC twice, exactly as ADK's StreamingResponseAggregator
    does under PROGRESSIVE_SSE_STREAMING: a partial=True chunk first, then the
    partial=False aggregate re-carrying the same FC (same id).
    """

    model: str = "streaming_dispatch_stub"

    @classmethod
    def supported_models(cls) -> list[str]:
        return ["streaming_dispatch_stub"]

    async def generate_content_async(  # type: ignore[override]
        self, llm_request: Any, stream: bool = False
    ) -> AsyncIterator[LlmResponse]:
        _COORDINATOR_CALLS.append(1)
        if len(_COORDINATOR_CALLS) == 1:
            fc = FunctionCall(
                name="ga_specialist",
                args={"request": "Top landing pages for June 2026"},
                id="fc-dispatch-1",
            )
            # 1. partial chunk carrying the task FC — every intermediate
            #    progressive-SSE chunk is marked partial=True.
            yield LlmResponse(
                content=Content(role="model", parts=[Part(function_call=fc)]),
                partial=True,
            )
            # 2. non-partial aggregate re-carrying the same FC (same id) —
            #    only this is persisted by the Runner.
            yield LlmResponse(
                content=Content(role="model", parts=[Part(function_call=fc)]),
                partial=False,
                turn_complete=True,
            )
        else:
            yield LlmResponse(
                content=Content(
                    role="model",
                    parts=[Part.from_text(text="Here are your top landing pages.")],
                ),
                partial=False,
                turn_complete=True,
            )


class _FinishingSpecialistLlm(BaseLlm):
    """Task specialist stub: immediately calls finish_task with its result."""

    model: str = "finishing_specialist_stub"

    @classmethod
    def supported_models(cls) -> list[str]:
        return ["finishing_specialist_stub"]

    async def generate_content_async(  # type: ignore[override]
        self, llm_request: Any, stream: bool = False
    ) -> AsyncIterator[LlmResponse]:
        _SPECIALIST_CALLS.append(1)
        fc = FunctionCall(
            name="finish_task",
            args={"result": "1. /pricing 2. /blog 3. /home"},
            id=f"fin-{len(_SPECIALIST_CALLS)}",
        )
        yield LlmResponse(
            content=Content(role="model", parts=[Part(function_call=fc)]),
            turn_complete=True,
        )


async def _run_repro() -> None:
    from google.adk.runners import Runner
    from google.adk.sessions import InMemorySessionService
    from google.genai import types as genai_types

    specialist = LlmAgent(
        name="ga_specialist",
        model=_FinishingSpecialistLlm(),
        mode="task",
        instruction="GA specialist",
        disallow_transfer_to_parent=True,
    )
    coordinator = LlmAgent(
        name="coordinator",
        model=_StreamingDispatchCoordinatorLlm(),
        mode="chat",
        instruction="Coordinator",
        tools=[],
        sub_agents=[specialist],
    )

    session_service = InMemorySessionService()
    session = await session_service.create_session(
        app_name="partial_dispatch_repro", user_id="test_user"
    )
    runner = Runner(
        agent=coordinator,
        app_name="partial_dispatch_repro",
        session_service=session_service,
    )

    print("Running a progressive-SSE-shaped dispatch turn (raw ADK, no guard)...\n")
    try:
        async for event in runner.run_async(
            user_id=session.user_id,
            session_id=session.id,
            new_message=genai_types.Content(
                role="user",
                parts=[genai_types.Part.from_text(text="Top landing pages June 2026?")],
            ),
        ):
            pass
        print("NO RAISE — the defect may be fixed upstream; re-check on this ADK version.")
    except ValueError as exc:
        print("RAISED ValueError (orphaned task FR — the progressive-SSE poison):\n")
        print(f"  type: {type(exc).__module__}.{type(exc).__name__}")
        print(f"  message: {exc}\n")
        print("Traceback (key frames):")
        print("----")
        print(traceback.format_exc())
        print("----")

    persisted_session = await session_service.get_session(
        app_name="partial_dispatch_repro",
        user_id=session.user_id,
        session_id=session.id,
    )
    persisted = list(persisted_session.events) if persisted_session else []

    def _fc_ids(name: str) -> list[str]:
        out = []
        for e in persisted:
            for fc in e.get_function_calls():
                if fc.name == name:
                    out.append(fc.id)
        return out

    def _fr_ids(name: str) -> list[str]:
        out = []
        for e in persisted:
            for fr in e.get_function_responses():
                if fr.name == name:
                    out.append(fr.id)
        return out

    print("\n── Persisted session events (post-turn) ──")
    for i, e in enumerate(persisted):
        role = getattr(e.content, "role", "?") if e.content else "?"
        parts = []
        for p in (e.content.parts if e.content else []):
            if p.function_call:
                parts.append(f"FC:{p.function_call.name}(id={p.function_call.id})")
            elif p.function_response:
                parts.append(f"FR:{p.function_response.name}(id={p.function_response.id})")
            elif p.text:
                parts.append(f"text:{p.text[:40]!r}")
        print(f"  [{i}] role={role} partial={e.partial}  parts={parts}")

    fc_ids = _fc_ids("ga_specialist")
    fr_ids = _fr_ids("ga_specialist")
    print()
    print(f"Persisted coordinator task FC ids: {fc_ids}")
    print(f"Persisted synthesized task FR ids: {fr_ids}")
    print(f"Specialist dispatched {len(_SPECIALIST_CALLS)}x for one task FC")
    print()
    if not fc_ids and fr_ids:
        print(
            "BUG CONFIRMED: the coordinator's task-delegation FunctionCall event\n"
            "was NOT persisted (the wrapper broke out on the partial=True chunk,\n"
            "before the non-partial aggregate was ever yielded), while the\n"
            "synthesized task FunctionResponse WAS persisted — an orphaned FR.\n"
            "Every later contents build raises the orphaned-response ValueError.\n"
            "Defect 3 is the dominant producer of defect 2's poisoned state."
        )
    elif fc_ids and fr_ids:
        print(
            "No orphan — the FC was persisted. If this is ADK 2.0.0 with the\n"
            "guard removed, the defect may be fixed upstream; re-check."
        )


if __name__ == "__main__":
    asyncio.run(_run_repro())

Observed output (run against google-adk==2.0.0, Python 3.13):

Running a progressive-SSE-shaped dispatch turn (raw ADK, no guard)...

RAISED ValueError (orphaned task FR — the progressive-SSE poison):

  type: builtins.ValueError
  message: No function call event found for function responses ids: {'fc-dispatch-1'}

Traceback (key frames):
----
  File ".../google/adk/workflow/_llm_agent_wrapper.py", line 380, in run_llm_agent_as_node
    async for event in run_iter:
      ...
        break
  File ".../google/adk/flows/llm_flows/base_llm_flow.py", line 935, in _preprocess_async
    async for event in agen:
      yield event
  File ".../google/adk/flows/llm_flows/contents.py", line 74, in run_async
    llm_request.contents = _get_contents(
  File ".../google/adk/flows/llm_flows/contents.py", line 637, in _get_contents
    result_events = _rearrange_events_for_latest_function_response(
        filtered_events
    )
  File ".../google/adk/flows/llm_flows/contents.py", line 224, in _rearrange_events_for_latest_function_response
    raise ValueError(
        ...
    )
ValueError: No function call event found for function responses ids: {'fc-dispatch-1'}

----

── Persisted session events (post-turn) ──
  [0] role=user partial=None  parts=["text:'Top landing pages June 2026?'"]
  [1] role=model partial=None  parts=['FC:finish_task(id=fin-1)']
  [2] role=user partial=None  parts=['FR:finish_task(id=fin-1)']
  [3] role=user partial=None  parts=['FR:ga_specialist(id=fc-dispatch-1)']
  [4] role=? partial=None  parts=[]

Persisted coordinator task FC ids: []
Persisted synthesized task FR ids: ['fc-dispatch-1']
Specialist dispatched 1x for one task FC

BUG CONFIRMED: the coordinator's task-delegation FunctionCall event
was NOT persisted (the wrapper broke out on the partial=True chunk,
before the non-partial aggregate was ever yielded), while the
synthesized task FunctionResponse WAS persisted — an orphaned FR.
Every later contents build raises the orphaned-response ValueError.
Defect 3 is the dominant producer of defect 2's poisoned state.

The persisted-event dump is the smoking gun: event [3] is FR:ga_specialist(id=fc-dispatch-1)
with no matching FC:ga_specialist anywhere in the session (Persisted coordinator task FC ids: []). The specialist dispatched exactly once, so the orphan is not a double
dispatch — it is the wrapper breaking on the partial=True chunk before the
partial=False aggregate (the only event the Runner persists) was ever yielded.

Root cause

google/adk/workflow/_llm_agent_wrapper.py, function _extract_task_delegation_fcs
(line 56) has no partial check:

def _extract_task_delegation_fcs(
    event: Event, tools_dict: dict
) -> list[types.FunctionCall]:
  """Return task-delegation FCs from this event."""
  from ..tools.agent_tool import _TaskAgentTool

  return [
      fc
      for fc in event.get_function_calls()        # ← no `event.partial` gate
      if fc.id
      and fc.name in tools_dict
      and isinstance(tools_dict[fc.name], _TaskAgentTool)
  ]

It is called from run_llm_agent_as_node (line 382) inside the loop that breaks on any
extracted task FC (line 388):

  async with aclosing(run_method) as run_iter:
    async for event in run_iter:
      yield event
      task_fcs = _extract_task_delegation_fcs(event, tools_dict)   # line 382
      for fc in task_fcs:
        output = await _dispatch_task_fc(agent, fc, ctx)
        yield _synthesize_task_fr_event(fc, output)
      if task_fcs:
        had_task_fc = True
        break   # line 388 — closes run_iter before the non-partial aggregate

Under progressive SSE, the first chunk carrying the task FC is partial=True. The wrapper
extracts from it, dispatches, synthesizes the task FR, and breaks. The Runner only
persists events with partial is not True, so:

  • the synthesized task FunctionResponse (yielded by the wrapper itself, not by the
    inner generator's partial=True chunk) is persisted, but
  • the coordinator's FunctionCall event is only ever carried by the partial=False
    aggregate, which the break never reached → it is never persisted.

The result is an orphaned function response, which the next _preprocess_async contents
build raises on (contents.py:224 via _get_contents at contents.py:637).

Expected behaviour

Task-delegation FC extraction should be gated on non-partial events, exactly as ADK's
own process_llm_agent_output does for the non-streamed path — dispatch from the final
aggregate, which the Runner persists and which carries the complete FC args. Concretely,
_extract_task_delegation_fcs (or its caller) should return no FCs for events whose
partial is True, so the break only fires on the non-partial aggregate.

Observed behaviour

Without the gate, the wrapper dispatches from the partial=True chunk, breaks before
the non-partial aggregate is yielded, and the session is left with a synthesized task FR
and no matching task FC — permanently poisoned by the orphaned-response raise documented
in the companion report.

Related

  • Consumption defect (companion report): docs/upstream-reports/02-orphaned-function-response-poisoning.md
    — the orphaned-FR ValueError this defect produces is the same raise that turns any
    orphaned FR into a dead session. This report is the producer; that report is the
    consumer. Fixing the wrapper's partial gate removes the dominant producer; fixing the
    contents processor's fatal raise (companion) removes the fatal consumption.
  • Mixed-turn FC drop (companion report): docs/upstream-reports/01-mixed-turn-fc-drop.md
    — the same run_llm_agent_as_node break (line 388) is also the root of the mixed-turn
    regular-FC drop. Both defects share the wrapper's break-on-task-FC design; a single
    "drain remaining events before break / gate on non-partial" fix sketch addresses
    both.

Activity

  1. added theissue type on Aug 4, 2026
  2. added
    agent engine[Component] This issue is related to Vertex AI Agent Engine
    on Aug 4, 2026
  3. bferanmi806-sketch commented on Aug 4, 2026

    @bferanmi806-sketch

    I’d like to work on this. I plan to add a narrow guard so task-delegation function calls are dispatched only from non-partial events, then add an offline progressive-SSE regression test verifying complete arguments, single dispatch, balanced persisted FC/FR history, and a usable subsequent turn. Please let me know if this direction aligns with the intended behavior.

  4. a2105z commented on Aug 4, 2026

    @a2105z
    Contributor

    @bferanmi806-sketch @kenwilly - also interested, let me know if I can help - maybe we can work together.

  5. self-assigned this
    on Aug 5, 2026
  6. bferanmi806-sketch commented on Aug 5, 2026

    @bferanmi806-sketch

    Thanks I see @llalitkumarrr is assigned now. I’d already investigated the progressive-SSE failure path and outlined a regression-test plan above. I’ll avoid duplicating the implementation, but I’m happy to share context, review the patch, or help test complete arguments, single dispatch, balanced persisted FC/FR history, and a usable subsequent turn.

  7. llalitkumarrr commented on Aug 6, 2026

    @llalitkumarrr
    Collaborator

    Hello @bferanmi806-sketch and @a2105z,

    Could you please share the details of your analysis with us? This information would be very helpful.

    Additionally, @kenwilly mentioned version adk=2.0.0. Do we get the same results with version 2.6.0?

  8. added
    core[Component] This issue is related to the core interface and implementation
    request clarification[Status] The maintainer need clarification or more information from the author
    and removed
    agent engine[Component] This issue is related to Vertex AI Agent Engine
    on Aug 6, 2026
  9. bferanmi806-sketch commented on Aug 6, 2026

    @bferanmi806-sketch

    Thanks for asking. Here is the failure path I traced:

    1. Under progressive SSE, StreamingResponseAggregator.process_response() marks every intermediate model chunk partial=True, including chunks carrying function-call names or partial arguments.

    2. The completed function call, with fully accumulated arguments, is emitted only by StreamingResponseAggregator.close() as a partial=False aggregate.

    3. In run_llm_agent_as_node, the chat-mode loop yields each event and then immediately passes it to _extract_task_delegation_fcs(). That helper currently has no event.partial guard.

    4. When a task-delegation FC is present on a partial chunk, the wrapper dispatches the child task, synthesizes its FR, and then breaks out of the inner generator.

    5. This closes the stream before the non-partial aggregate is reached. The partial FC event is not persisted, while the synthesized task FR is persisted.

    6. On coordinator re-entry, contents reconstruction sees the task FR without its matching FC and raises:

    ValueError: No function call event found for function responses ids: {...}

    The same early extraction can also dispatch with incomplete streamed arguments.

    For the version question: I checked the v2.6.0 source tag. The missing partial-event guard and the early break are still present, and the progressive-SSE aggregator still emits complete FC data only in its non-partial close result.

    The same code path is also present in v2.6.2 and current main, so the issue appears to remain applicable at source level.

    I have not yet run the complete reproduction script against an installed 2.6.0 environment, so I would distinguish this from a confirmed runtime rerun.

    The narrow fix I had in mind is to ignore task-delegation FCs on partial=True events and dispatch only from the final non-partial aggregate.

    The regression test should verify:

    • the specialist receives the complete arguments;
    • one task FC causes exactly one dispatch;
    • the persisted task FC and synthesized task FR have matching IDs;
    • the next coordinator turn remains usable.

    I’m also happy to review or run the reproduction against the proposed patch.

  10. a2105z commented on Aug 6, 2026

    @a2105z
    Contributor

    Hello @llalitkumarrr — sharing the analysis you asked for (and answering the 2.6.0 question). Aligning with @bferanmi806-sketch’s earlier write-up; happy to coordinate so we don’t duplicate the patch.

    Version check (2.0.0 / 2.6.0 / current main)

    I re-ran kenwilly’s progressive-SSE-shaped repro against current main / installed google-adk==2.6.2 (editable checkout).

    Producer defect still reproduces on current main. After one turn:

    • persisted coordinator task FC ids: []
    • persisted synthesized task FR ids: ['fc-dispatch-1']
    • specialist dispatched exactly once
    • log line: Dropping function responses with no matching function call: ['fc-dispatch-1']

    So the wrapper still breaks on the partial=True chunk before the Runner-persisted aggregate is yielded.

    Consumption symptom changed after 2.0.0. On 2.0.0 the next contents build raised:

    ValueError: No function call event found for function responses ids: {...}

    On current main, contents.py can prune orphaned FRs (Dropping function responses…, from 73e86254) instead of always hard-failing on every path. That softens the “permanently poisoned session” raise, but it does not fix the producer: history is still missing the task FC, dispatch can still fire from a truncated partial chunk, and the prune path is a warning / history rewrite rather than correct FC/FR pairing.

    Source-level, the missing event.partial gate and the early break are still present on the v2.6.0 tag, v2.6.2, and current main.

    Failure path (pinpoint)

    1. Progressive SSE emission (StreamingResponseAggregator under PROGRESSIVE_SSE_STREAMING, default-on):

      • Intermediate model chunks are partial=True, including chunks that already name a task-delegation FunctionCall (args may still be streaming / incomplete).
      • The completed FC (full args, same id) is emitted as the non-partial aggregate (aggregator close() / final yield).
    2. Persistence rule (runners.py):

      • Runner only appends events when event.partial is not True.
      • So the only FC event that can land in session history is the non-partial aggregate.
    3. Chat wrapper extraction has no partial gate — src/google/adk/workflow/_llm_agent_wrapper.py:

    def _extract_task_delegation_fcs(event, tools_dict):
      # … no `if event.partial: return []`
      return [
          fc
          for fc in event.get_function_calls()  # ← fires on partial chunks too
          if fc.id and fc.name is not None
          and isinstance(tools_dict.get(fc.name), _TaskAgentTool)
      ]
    1. Call site breaks the generator early — same file, run_llm_agent_as_node (~L470–487 on current main):
    async for event in run_iter:
      yield event
      task_fcs = _extract_task_delegation_fcs(event, tools_dict)
      if task_fcs:
        # … optional mixed-turn FR drain …
        for fc in task_fcs:
          output = await _dispatch_task_fc(agent, fc, ctx)
          yield _synthesize_task_fr_event(fc, output)  # persisted (not partial)
        had_task_fc = True
        break  # closes run_iter before non-partial aggregate is yielded
    1. Resulting session shape (smoking gun):

      • Task FR synthesized by the wrapper is persisted.
      • Matching task FC never reaches the non-partial yield → never persisted.
      • Next contents assembly sees FR without FC (raise on 2.0.0; prune/warn on current main).
    2. Second defect on the same path: if the first partial chunk has incomplete streamed args, the specialist can be dispatched with truncated / empty input even when a later aggregate would have carried the full args.

    Contrast with existing ADK pattern

    process_llm_agent_output in the same module already skips partial events:

    if (
        event.get_function_calls()
        or event.partial   # ← already gated
        or not event.content
        or event.content.role != 'model'
    ):
      return

    Task-delegation extraction should follow that same rule: never act on partial=True model chunks.

    What needs to be fixed

    Narrow, surgical fix (producer):

    In _extract_task_delegation_fcs (or immediately at the call site before dispatch), return no FCs when event.partial is true:

    if event.partial:
      return []

    That forces dispatch + break to wait for the non-partial aggregate that the Runner actually persists, with complete args.

    Out of scope for this bug (related but separate):

    Regression checks the fix should prove

    1. Specialist receives complete FC args (not the empty/truncated partial chunk).
    2. One task FC → exactly one dispatch.
    3. Persisted session has matching task FC id and synthesized task FR id.
    4. Coordinator re-entry / next turn stays usable (no orphan FR; no prune of the task FR).

    Status / coordination

    I’ve confirmed the above on current main, implemented the partial gate + unit/e2e coverage on branch fix/progressive-sse-task-fc-partial-gate, and can open a PR whenever that’s useful.

    Since you’re assigned (@llalitkumarrr): happy to open the PR, hand the patch over, or just review/test yours — whatever avoids double work. @bferanmi806-sketch’s direction matches this; I just wanted the version re-run + exact call-site pin for the clarification label.

  11. removed
    request clarification[Status] The maintainer need clarification or more information from the author
    on Aug 7, 2026
  12. llalitkumarrr commented on Aug 7, 2026

    @llalitkumarrr
    Collaborator

    Hello @a2105z and @bferanmi806-sketch,

    Thank you for sharing your input. We looked into this further based on your suggestions. Implementing a partial gate in _extract_task_delegation_fcs looks like a clean way to resolve this issue.

    @a2105z, your proposed implementation seems to address this and we verified with the reproduction code provided above that it should fix the issue. Please feel free to open a pull request. We will review it and it can be merged once approved.

  13. a2105z commented on Aug 7, 2026

    @a2105z
    Contributor

    Ok thank you, I will immediately get started and open a PR by EOD.

  14. added a commit that references this issue on Aug 7, 2026
    44c5913
  15. a2105z commented on Aug 7, 2026

    @a2105z
    Contributor

    Hello @llalitkumarrr — thanks for the green light and for verifying the partial-gate approach against the repro.

    PR is up here: #6638

    It implements the event.partial gate in _extract_task_delegation_fcs, plus unit + progressive-SSE e2e coverage (complete args, single dispatch, matched FC/FR ids, usable next turn).

    cc @bferanmi806-sketch @kenwilly — happy for eyes/review if useful; tried to keep this aligned with the analysis already shared on this thread.

  16. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Aug 10, 2026
  17. added a commit that references this issue on Aug 24, 2026
    816ada5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementationneeds review[Status] The PR/issue is awaiting review from the maintainer

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions