Skip to content

Resumable flow: a parallel tool call that never executed is never replayed when a sibling call was answered #7108

Closed

Description

@vasilisnasopoulos

🔴 Required Information

Describe the Bug:

In resumable mode, when one event carries parallel function calls and only some of them produced a response before the interruption, decide_step_resume() returns CONTINUE. The calls that never executed are never replayed: they do not run, no error is raised, and the flow proceeds to the next LLM call as if it had answers it never received.

The two extremes are handled correctly — nothing answered replays, everything answered continues. Only the partial case, which is the case resumption exists for, is lost.

The decision asks whether any answer came back rather than whether every call was answered:

  • _resume_utils.py:192 — any(fr.name not in call_names for fr in answers) matches on names, so an answer to one call reads as an answer to the event.
  • _resume_utils.py:251 — not call_ids & answered_ids is an intersection, so one answered id clears the whole set.

call_ids and answered_ids are both already in scope at that point.

Steps to Reproduce:

  1. pip install google-adk==2.9.0
  2. Save the script under Minimal Reproduction Code as repro.py
  3. python repro.py

Expected Behavior:

A call with no response is replayed, whether or not a sibling call in the same event was answered — cases A and B below should be replay_calls.

Observed Behavior:

A. parallel calls, different names, only c1 executed
    executed=['c1']  never executed=['c2']
    decide_step_resume -> continue     expected replay_calls LOST

B. parallel calls, same name twice, only c1 executed
    executed=['c1']  never executed=['c2']
    decide_step_resume -> continue     expected replay_calls LOST

C. NEGATIVE CONTROL — all executed (mirrors your own test)
    executed=['c1', 'c2']  never executed=none
    decide_step_resume -> continue     expected continue     ok

D. NEGATIVE CONTROL — none executed
    executed=none  never executed=['c1', 'c2']
    decide_step_resume -> replay_calls expected replay_calls ok

E. NEGATIVE CONTROL — single call, not executed
    executed=none  never executed=['c1']
    decide_step_resume -> replay_calls expected replay_calls ok

Environment Details:

  • ADK Library Version: google-adk 2.9.0 (the same lines are present on main)
  • Desktop OS: macOS 26.6.2 (arm64)
  • Python Version: 3.12.13

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A — the repro calls decide_step_resume() directly and needs no model

🟡 Optional Information

Additional Context:

The correct idiom is already in this file, 130 lines above, in _pause_left_calls_unanswered:

# `issubset`, not `&`: this asks whether *any* awaited id is still open, so a
# partially answered pause keeps waiting. `decide_resume` asks the opposite
# question of its own ids -- whether *none* are answered -- and drops
# `issubset` for that reason. The two are not interchangeable.
return bool(awaited) and not awaited.issubset(answered)

That comment states the distinction exactly; the replay decision is the second place that needs it.

test_parallel_calls_all_answered_continue covers the fully answered case, and its comment says it exists so a fully answered event is not replayed and the tools do not run twice. That guard is right — the partial case appears to be the gap it left, and it has no test.

I have deliberately not proposed a patch, because the fix is a design choice I am not in a position to make: replaying the event runs all of its calls, so avoiding duplicate execution of the calls that already succeeded means either replaying per call or filtering the event down to the unanswered ids. Both change behaviour beyond this function.

Scope — what I did and did not verify:

  • Verified: decide_step_resume() in isolation, with the event shapes used by tests/unittests/flows/llm_flows/test_resume_utils.py.
  • Not verified: an end-to-end run against a live model, or how often a real interruption lands between sibling responses. Whether this is reachable in practice depends on when responses are persisted relative to the crash, which I have not measured.
  • The Ctx class in the repro is mine, standing in for InvocationContext; it supplies only the three members decide_step_resume reads.

Minimal Reproduction Code:

"""ADK resumable flow, real entry point decide_step_resume().
A parallel tool call that never executed is never replayed: the flow continues
to the LLM as if it had an answer it never got."""
from google.genai import types
from google.adk.events.event import Event
from google.adk.flows.llm_flows._resume_utils import decide_step_resume, ResumeAction

class Ctx:
    """Only what decide_step_resume touches."""
    def __init__(self, events): self._events, self.is_resumable = events, True
    def _get_events(self, current_invocation=True, current_branch=True): return self._events
    def should_pause_invocation(self, ev): return False

def call_event(pairs):
    return Event(author='agent', invocation_id='inv-1',
        content=types.Content(role='model', parts=[
            types.Part(function_call=types.FunctionCall(id=i, name=n, args={}))
            for i, n in pairs]))

def response_event(name, cid):
    return Event(author='user', invocation_id='inv-1',
        content=types.Content(role='user', parts=[
            types.Part(function_response=types.FunctionResponse(
                id=cid, name=name, response={'r': 'ok'}))]))

def run(label, calls, ran, tools, expect):
    events = [call_event(calls)] + [response_event(n, i) for i, n in calls if i in ran]
    d = decide_step_resume(Ctx(events), {t: object() for t in tools})
    missing = [i for i, _ in calls if i not in ran]
    bad = missing and d.action is ResumeAction.CONTINUE
    print(f'{label}\n    executed={sorted(ran) or "none"}  never executed={missing or "none"}')
    print(f'    decide_step_resume -> {d.action.value:12s} expected {expect:12s}'
          f' {"LOST" if bad else "ok"}\n')

run('A. parallel calls, different names, only c1 executed',
    [('c1','ask'), ('c2','fetch')], {'c1'}, ['ask','fetch'], 'replay_calls')
run('B. parallel calls, same name twice, only c1 executed',
    [('c1','ask'), ('c2','ask')], {'c1'}, ['ask'], 'replay_calls')
run('C. NEGATIVE CONTROL — all executed (mirrors their own test)',
    [('c1','ask'), ('c2','fetch')], {'c1','c2'}, ['ask','fetch'], 'continue')
run('D. NEGATIVE CONTROL — none executed',
    [('c1','ask'), ('c2','fetch')], set(), ['ask','fetch'], 'replay_calls')
run('E. NEGATIVE CONTROL — single call, not executed',
    [('c1','ask')], set(), ['ask'], 'replay_calls')

How often has this issue occurred?:

  • Always (100%) — deterministic for the inputs above.

Related: #7076 is a different defect in the same subsystem (who may author a dispatched call); this one is about which calls are replayed.

Activity

  1. Zsanz3 commented on Sep 14, 2026

    @Zsanz3

    I'd like to take a small fix for this.

    Approach: In decide_step_resume, replace the any-answered short-circuit with a full-coverage check (aligned with _pause_left_calls_unanswered's issubset logic), so a parallel tool call that never executed is still replayed when only a sibling was answered. Add a focused unit test for the partial-parallel case in test_resume_utils.py.

    Happy to adjust if maintainers prefer a different shape.

  2. surajksharma07 commented on Sep 14, 2026

    @surajksharma07
    Collaborator

    @vasilisnasopoulos decide_resume genuinely treats one answered sibling as coverage for the whole parallel call, exactly the any-vs-issubset gap you pointed at. #7115 already applies that same issubset fix to both names and ids (currently stuck on CLA). Could you pull that branch and confirm it flips your cases A/B to replay_calls on your setup? One thing to flag: it only fixes the decision, replay still re-runs the whole event so an already-answered sibling like c1 gets executed again, that part's still open. Worth testing that side too before anyone pushes a separate PR for it.

  3. vasilisnasopoulos commented on Sep 14, 2026

    @vasilisnasopoulos
    Author

    Ran #7115 against the cases from the report. Both flip, and it fixes two more I had not filed.

    decide_step_resume, same event shapes as tests/unittests/flows/llm_flows/test_resume_utils.py:

    case main #7115
    A. parallel, different names, only c1 answered continue (call lost) replay_calls
    B. parallel, same name twice, only c1 answered continue (call lost) replay_calls
    C. control — both answered continue continue
    D. control — neither answered replay_calls replay_calls
    E. control — single call, unanswered replay_calls replay_calls

    The lro_ids.issubset(answered_ids) half covers two cases I had missed, where main also continues past a call that never produced a response:

    case main #7115
    F. two long-running calls, only c1 answered continue pause
    G. long-running c1 answered, plain c2 never ran continue replay_calls
    H. control — plain c1 answered, long-running c2 outstanding pause pause
    I. control — two long-running, both answered continue continue

    tests/unittests/flows/llm_flows/test_resume_utils.py: 31 passed on main, 34 on the branch. Whole tests/unittests/flows: 720 passed on main, 723 on the branch, same single pre-existing failure on both (test_tool_call_rearranger.py::test_drop_orphaned_calls_prevents_unclosed_tool_use_in_anthropic_conversion). No regressions.

    On the replay side you flagged: it is real, and the repo already names both the harm and the mechanism.

    _replay_function_calls (base_llm_flow.py:418) hands the whole call_event to _postprocess_handle_function_calls_async, so every call on it runs again, including the answered sibling. The comment in your own test at test_resume_utils.py:262 calls that out as the thing to avoid — "a fully answered event is replayed and both tools run a second time".

    handle_function_calls_async already takes filters: Optional[set[str]] (functions.py:289), applied at _batch_tool_executor.py:248 as fc.id in filters. So the replay path can pass the uncovered ids that _needs_call_replay has already computed, and re-run only those. Nothing new is needed.

    Two things a narrower replay would want covering, since neither is exercised today: an uncovered call whose id is None (no id to filter on, so it has to stay unfiltered), and a same-name pair where only the twin is uncovered, which is case B — filtering by name there would drop the call that never ran.

  4. vasilisnasopoulos commented on Sep 14, 2026

    @vasilisnasopoulos
    Author

    Measured the replay side, since you flagged it as still open. It is real, and
    it costs a real execution: with #7115 applied, the partial-parallel case now
    replays correctly but runs the already-answered tool a second time.

    Counted through handle_function_calls_async, the call the replay path makes,
    with two counting tools and a real InvocationContext:

    branch decision tools that actually ran
    main continue none — c2 is lost, the original report
    #7115 replay_calls ask, fetch — c1 runs twice
    #7115 + the patch below replay_calls fetch only

    The mechanism is already in the repo, so the patch is small. filters on
    handle_function_calls_async (functions.py:289) is applied by call id at
    _batch_tool_executor.py:248, and _needs_call_replay has already computed
    which ids are uncovered. This carries that set on the decision and passes it
    down.

    Not a pull request: I am not signing a CLA for this, so take it as a patch or
    as a description, whichever is easier. Verified on top of #7115:

    • tests/unittests/flows/llm_flows/test_resume_utils.py: 34 passed, unchanged
    • tests/unittests/flows: 723 passed, 1 failed — the same pre-existing
      test_tool_call_rearranger failure main has

    Edge cases I ran, all through real tool execution:

    case replay ids tools that ran
    J. one call carries no id, the other never ran None both — nothing to filter on, so replay everything
    K. same name twice, only c1 answered {c2} ask once
    L. control — nothing answered at all None none; still pause, per test_unanswered_plain_call_pauses
    M. three calls, only the middle one missing {c2} fetch only

    J is the one worth keeping: a call with no id cannot be named in a filter, so
    filtering there would drop the very call the replay exists for. Returning
    None falls back to today's behaviour instead.

    diff --git a/src/google/adk/flows/llm_flows/_resume_utils.py b/src/google/adk/flows/llm_flows/_resume_utils.py
    index d60d09f..cdb99c6 100644
    --- a/src/google/adk/flows/llm_flows/_resume_utils.py
    +++ b/src/google/adk/flows/llm_flows/_resume_utils.py
    @@ -57,6 +57,15 @@ class ResumeDecision:
     
       action: ResumeAction
       event: Event | None = None
    +  replay_ids: frozenset[str] | None = None
    +  """Which of the event's calls to re-run, by id. `None` means all of them.
    +
    +  Replaying the whole event re-runs calls that already answered, which the
    +  parallel-call tests name as the harm to avoid. This carries the uncovered
    +  ids so the replay can run only those. It is `None` whenever any call on the
    +  event has no id, because there is then nothing to filter that call on and
    +  dropping it would lose the call the replay exists for.
    +  """
     
       def replay_event(self) -> Event:
         """The event whose calls to run. Only a REPLAY_CALLS decision carries one.
    @@ -201,6 +210,20 @@ def _needs_call_replay(
       return names_uncovered or ids_uncovered or from_sub_branch
     
     
    +def _uncovered_ids(
    +    calls: list[types.FunctionCall],
    +    answered_ids: set[str],
    +) -> frozenset[str] | None:
    +  """The ids on `calls` with no answer, or `None` if they cannot be filtered.
    +
    +  Every call needs an id for this to be usable: a call without one cannot be
    +  named in a filter, so the only safe answer there is to replay everything.
    +  """
    +  if any(fc.id is None for fc in calls):
    +    return None
    +  return frozenset(fc.id for fc in calls if fc.id not in answered_ids)
    +
    +
     def decide_resume(
         invocation_context: InvocationContext,
         events: list[Event],
    @@ -275,7 +298,11 @@ def decide_resume(
             call_ids=concrete_call_ids,
             answered_ids=answered_ids,
         ):
    -      return ResumeDecision(ResumeAction.REPLAY_CALLS, call_event)
    +      return ResumeDecision(
    +          ResumeAction.REPLAY_CALLS,
    +          call_event,
    +          replay_ids=_uncovered_ids(calls, answered_ids),
    +      )
     
       return ResumeDecision(ResumeAction.PAUSE if pause else ResumeAction.CONTINUE)
     
    diff --git a/src/google/adk/flows/llm_flows/base_llm_flow.py b/src/google/adk/flows/llm_flows/base_llm_flow.py
    index 80632a7..30d6762 100644
    --- a/src/google/adk/flows/llm_flows/base_llm_flow.py
    +++ b/src/google/adk/flows/llm_flows/base_llm_flow.py
    @@ -420,9 +420,13 @@ class BaseLlmFlow(ABC):
           invocation_context: InvocationContext,
           model_response_event: Event,
           llm_request: LlmRequest,
    +      filters: set[str] | None = None,
       ) -> AsyncGenerator[Event, None]:
         """Runs `model_response_event`'s function calls, re-issuing event ids.
     
    +    `filters` names the call ids to run, leaving the calls that already
    +    answered alone. `None` runs every call on the event.
    +
         A node that interrupts mid-call raises `NodeInterruptedError`, which is a
         `BaseException` specifically so intermediate handlers do not swallow it.
         It is left to propagate: `NodeRunner` catches it and reads the interrupt
    @@ -430,7 +434,7 @@ class BaseLlmFlow(ABC):
         """
         async with Aclosing(
             self._postprocess_handle_function_calls_async(
    -            invocation_context, model_response_event, llm_request
    +            invocation_context, model_response_event, llm_request, filters
             )
         ) as agen:
           async for event in agen:
    @@ -466,7 +470,10 @@ class BaseLlmFlow(ABC):
         if resume_decision.action is ResumeAction.REPLAY_CALLS:
           async with Aclosing(
               self._replay_function_calls(
    -              invocation_context, resume_decision.replay_event(), llm_request
    +              invocation_context,
    +              resume_decision.replay_event(),
    +              llm_request,
    +              resume_decision.replay_ids,
               )
           ) as agen:
             async for event in agen:
    @@ -685,9 +692,10 @@ class BaseLlmFlow(ABC):
           invocation_context: InvocationContext,
           function_call_event: Event,
           llm_request: LlmRequest,
    +      filters: set[str] | None = None,
       ) -> AsyncGenerator[Event, None]:
         if function_response_event := await functions.handle_function_calls_async(
    -        invocation_context, function_call_event, llm_request.tools_dict
    +        invocation_context, function_call_event, llm_request.tools_dict, filters
         ):
           auth_event = functions.generate_auth_event(
               invocation_context, function_response_event
  5. surajksharma07 commented on Sep 14, 2026

    @surajksharma07
    Collaborator

    @vasilisnasopoulos That's a clean before/after/after. Main loses c2, #7115 alone runs both ask and fetch (c1 duplicated), your patch runs fetch only. Checked the diff against current main and it applies exactly as written: _replay_function_calls, the REPLAY_CALLS call site and _postprocess_handle_function_calls_async in base_llm_flow.py all match your hunk contexts line for line (none of that file is touched by #7115) and handle_function_calls_async's filters: Optional[set[str]] = None plus the fc.id in filters gate in _batch_tool_executor.py were already exactly what you said, nothing invented just wired through. The J case (any call missing an id → None, replay everything) is the right conservative fallback given a filter can't name what it can't identify.

    This is functionally the maintainer-level fix for the duplicate-execution half we flagged as still open, done properly. Event-count/tracing invariants elsewhere in base_llm_flow.py aren't touched and filtering happens at the existing filters seam rather than slicing the event itself.

    Since you're not signing a CLA for this one: @Zsanz3 given your branch is what this patch sits on top of and you're already going through CLA for #7115 would you fold these two commits in as a third on your branch rather than opening a second PR? It's a direct tested extension of your own fix (34/34 on test_resume_utils.py, 723 passed on the full flows suite with the same single pre-existing failure main has) so one PR covering both halves keeps this to one review instead of two blocked ones.

  6. added
    core[Component] This issue is related to the core interface and implementation
    on Sep 15, 2026
  7. added 5 commits that reference this issue on Sep 15, 2026
    3a5b37d
    daa5df5
    a49640d
    34c5427
    6f1b0ba
  8. vasilisnasopoulos commented on Sep 29, 2026

    @vasilisnasopoulos
    Author

    Thanks for the fix, and for covering both halves: replaying only the calls that never ran is exactly what was missing.

    One edge I checked against _unexecuted_calls_event on its own (not end to end): when a response carries no id, it answers its call by name, and answered_names is a set. So with two parallel calls to the same tool and one id-less response, both count as answered:

    calls response replayed
    ask c1, fetch c2 ask, no id [c2] ✅
    ask c1, ask c2 ask, id c1 [c2] ✅
    ask c1, ask c2 ask, no id nothing: c2 never ran

    Counting id-less responses per name instead of a set would keep it to one call per response, e.g. consume them in call order. Not sure how often id-less responses meet same-name parallel calls in practice, so flagging it rather than asserting it.

  9. added a commit that references this issue on Sep 29, 2026
    7298e09
  10. added 2 commits that reference this issue on Sep 30, 2026
    2c16d9d
    88c7cab
  11. surajksharma07 commented on Oct 6, 2026

    @surajksharma07
    Collaborator

    Good catch on the id-less case @vasilisnasopoulos. @SoroushRF picked it up as #7387/#7388 and that landed on main yesterday as 531f410. It does what you suggested: id-less responses are counted per name with a Counter and consumed in call order and explicit ids are matched first without using one up.

    Ran your three rows against _unexecuted_calls_event. On 2.11.0 the first two replay [c2] and the third still replays nothing. On current main all three replay [c2]. test_resume.py plus test_base_llm_flow.py pass 161 there.

    So the original fix (7298e09) is out in v2.11.0 and the id-less part will ship in the next release. Until then keeping call ids on the responses you restore avoids it. Nothing left open here from my side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions