Repository navigation
Description
🔴 Required Information
Describe the Bug:
In resumable mode, when one event carries parallel function calls and only some of them produced a response before the interruption, decide_step_resume() returns CONTINUE. The calls that never executed are never replayed: they do not run, no error is raised, and the flow proceeds to the next LLM call as if it had answers it never received.
The two extremes are handled correctly — nothing answered replays, everything answered continues. Only the partial case, which is the case resumption exists for, is lost.
The decision asks whether any answer came back rather than whether every call was answered:
_resume_utils.py:192—any(fr.name not in call_names for fr in answers)matches on names, so an answer to one call reads as an answer to the event._resume_utils.py:251—not call_ids & answered_idsis an intersection, so one answered id clears the whole set.
call_ids and answered_ids are both already in scope at that point.
Steps to Reproduce:
pip install google-adk==2.9.0- Save the script under Minimal Reproduction Code as
repro.py python repro.py
Expected Behavior:
A call with no response is replayed, whether or not a sibling call in the same event was answered — cases A and B below should be replay_calls.
Observed Behavior:
A. parallel calls, different names, only c1 executed
executed=['c1'] never executed=['c2']
decide_step_resume -> continue expected replay_calls LOST
B. parallel calls, same name twice, only c1 executed
executed=['c1'] never executed=['c2']
decide_step_resume -> continue expected replay_calls LOST
C. NEGATIVE CONTROL — all executed (mirrors your own test)
executed=['c1', 'c2'] never executed=none
decide_step_resume -> continue expected continue ok
D. NEGATIVE CONTROL — none executed
executed=none never executed=['c1', 'c2']
decide_step_resume -> replay_calls expected replay_calls ok
E. NEGATIVE CONTROL — single call, not executed
executed=none never executed=['c1']
decide_step_resume -> replay_calls expected replay_calls ok
Environment Details:
- ADK Library Version:
google-adk 2.9.0(the same lines are present onmain) - Desktop OS: macOS 26.6.2 (arm64)
- Python Version: 3.12.13
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A — the repro calls
decide_step_resume()directly and needs no model
🟡 Optional Information
Additional Context:
The correct idiom is already in this file, 130 lines above, in _pause_left_calls_unanswered:
# `issubset`, not `&`: this asks whether *any* awaited id is still open, so a
# partially answered pause keeps waiting. `decide_resume` asks the opposite
# question of its own ids -- whether *none* are answered -- and drops
# `issubset` for that reason. The two are not interchangeable.
return bool(awaited) and not awaited.issubset(answered)That comment states the distinction exactly; the replay decision is the second place that needs it.
test_parallel_calls_all_answered_continue covers the fully answered case, and its comment says it exists so a fully answered event is not replayed and the tools do not run twice. That guard is right — the partial case appears to be the gap it left, and it has no test.
I have deliberately not proposed a patch, because the fix is a design choice I am not in a position to make: replaying the event runs all of its calls, so avoiding duplicate execution of the calls that already succeeded means either replaying per call or filtering the event down to the unanswered ids. Both change behaviour beyond this function.
Scope — what I did and did not verify:
- Verified:
decide_step_resume()in isolation, with the event shapes used bytests/unittests/flows/llm_flows/test_resume_utils.py. - Not verified: an end-to-end run against a live model, or how often a real interruption lands between sibling responses. Whether this is reachable in practice depends on when responses are persisted relative to the crash, which I have not measured.
- The
Ctxclass in the repro is mine, standing in forInvocationContext; it supplies only the three membersdecide_step_resumereads.
Minimal Reproduction Code:
"""ADK resumable flow, real entry point decide_step_resume().
A parallel tool call that never executed is never replayed: the flow continues
to the LLM as if it had an answer it never got."""
from google.genai import types
from google.adk.events.event import Event
from google.adk.flows.llm_flows._resume_utils import decide_step_resume, ResumeAction
class Ctx:
"""Only what decide_step_resume touches."""
def __init__(self, events): self._events, self.is_resumable = events, True
def _get_events(self, current_invocation=True, current_branch=True): return self._events
def should_pause_invocation(self, ev): return False
def call_event(pairs):
return Event(author='agent', invocation_id='inv-1',
content=types.Content(role='model', parts=[
types.Part(function_call=types.FunctionCall(id=i, name=n, args={}))
for i, n in pairs]))
def response_event(name, cid):
return Event(author='user', invocation_id='inv-1',
content=types.Content(role='user', parts=[
types.Part(function_response=types.FunctionResponse(
id=cid, name=name, response={'r': 'ok'}))]))
def run(label, calls, ran, tools, expect):
events = [call_event(calls)] + [response_event(n, i) for i, n in calls if i in ran]
d = decide_step_resume(Ctx(events), {t: object() for t in tools})
missing = [i for i, _ in calls if i not in ran]
bad = missing and d.action is ResumeAction.CONTINUE
print(f'{label}\n executed={sorted(ran) or "none"} never executed={missing or "none"}')
print(f' decide_step_resume -> {d.action.value:12s} expected {expect:12s}'
f' {"LOST" if bad else "ok"}\n')
run('A. parallel calls, different names, only c1 executed',
[('c1','ask'), ('c2','fetch')], {'c1'}, ['ask','fetch'], 'replay_calls')
run('B. parallel calls, same name twice, only c1 executed',
[('c1','ask'), ('c2','ask')], {'c1'}, ['ask'], 'replay_calls')
run('C. NEGATIVE CONTROL — all executed (mirrors their own test)',
[('c1','ask'), ('c2','fetch')], {'c1','c2'}, ['ask','fetch'], 'continue')
run('D. NEGATIVE CONTROL — none executed',
[('c1','ask'), ('c2','fetch')], set(), ['ask','fetch'], 'replay_calls')
run('E. NEGATIVE CONTROL — single call, not executed',
[('c1','ask')], set(), ['ask'], 'replay_calls')How often has this issue occurred?:
- Always (100%) — deterministic for the inputs above.
Related: #7076 is a different defect in the same subsystem (who may author a dispatched call); this one is about which calls are replayed.
Activity
Zsanz3 commentedon Sep 14, 2026
I'd like to take a small fix for this.
Approach: In
decide_step_resume, replace the any-answered short-circuit with a full-coverage check (aligned with_pause_left_calls_unanswered'sissubsetlogic), so a parallel tool call that never executed is still replayed when only a sibling was answered. Add a focused unit test for the partial-parallel case intest_resume_utils.py.Happy to adjust if maintainers prefer a different shape.
surajksharma07 commentedon Sep 14, 2026
@vasilisnasopoulos decide_resume genuinely treats one answered sibling as coverage for the whole parallel call, exactly the any-vs-issubset gap you pointed at. #7115 already applies that same issubset fix to both names and ids (currently stuck on CLA). Could you pull that branch and confirm it flips your cases A/B to replay_calls on your setup? One thing to flag: it only fixes the decision, replay still re-runs the whole event so an already-answered sibling like c1 gets executed again, that part's still open. Worth testing that side too before anyone pushes a separate PR for it.
vasilisnasopoulos commentedon Sep 14, 2026
Ran #7115 against the cases from the report. Both flip, and it fixes two more I had not filed.
decide_step_resume, same event shapes astests/unittests/flows/llm_flows/test_resume_utils.py:c1answeredcontinue(call lost)replay_callsc1answeredcontinue(call lost)replay_callscontinuecontinuereplay_callsreplay_callsreplay_callsreplay_callsThe
lro_ids.issubset(answered_ids)half covers two cases I had missed, wheremainalso continues past a call that never produced a response:c1answeredcontinuepausec1answered, plainc2never rancontinuereplay_callsc1answered, long-runningc2outstandingpausepausecontinuecontinuetests/unittests/flows/llm_flows/test_resume_utils.py: 31 passed on main, 34 on the branch. Wholetests/unittests/flows: 720 passed on main, 723 on the branch, same single pre-existing failure on both (test_tool_call_rearranger.py::test_drop_orphaned_calls_prevents_unclosed_tool_use_in_anthropic_conversion). No regressions.On the replay side you flagged: it is real, and the repo already names both the harm and the mechanism.
_replay_function_calls(base_llm_flow.py:418) hands the wholecall_eventto_postprocess_handle_function_calls_async, so every call on it runs again, including the answered sibling. The comment in your own test attest_resume_utils.py:262calls that out as the thing to avoid — "a fully answered event is replayed and both tools run a second time".handle_function_calls_asyncalready takesfilters: Optional[set[str]](functions.py:289), applied at_batch_tool_executor.py:248asfc.id in filters. So the replay path can pass the uncovered ids that_needs_call_replayhas already computed, and re-run only those. Nothing new is needed.Two things a narrower replay would want covering, since neither is exercised today: an uncovered call whose
idisNone(no id to filter on, so it has to stay unfiltered), and a same-name pair where only the twin is uncovered, which is case B — filtering by name there would drop the call that never ran.vasilisnasopoulos commentedon Sep 14, 2026
Measured the replay side, since you flagged it as still open. It is real, and
it costs a real execution: with #7115 applied, the partial-parallel case now
replays correctly but runs the already-answered tool a second time.
Counted through
handle_function_calls_async, the call the replay path makes,with two counting tools and a real
InvocationContext:maincontinuec2is lost, the original report#7115replay_callsask,fetch—c1runs twice#7115+ the patch belowreplay_callsfetchonlyThe mechanism is already in the repo, so the patch is small.
filtersonhandle_function_calls_async(functions.py:289) is applied by call id at_batch_tool_executor.py:248, and_needs_call_replayhas already computedwhich ids are uncovered. This carries that set on the decision and passes it
down.
Not a pull request: I am not signing a CLA for this, so take it as a patch or
as a description, whichever is easier. Verified on top of
#7115:tests/unittests/flows/llm_flows/test_resume_utils.py: 34 passed, unchangedtests/unittests/flows: 723 passed, 1 failed — the same pre-existingtest_tool_call_rearrangerfailuremainhasEdge cases I ran, all through real tool execution:
id, the other never ranNonec1answered{c2}askonceNonepause, pertest_unanswered_plain_call_pauses{c2}fetchonlyJ is the one worth keeping: a call with no id cannot be named in a filter, so
filtering there would drop the very call the replay exists for. Returning
Nonefalls back to today's behaviour instead.surajksharma07 commentedon Sep 14, 2026
@vasilisnasopoulos That's a clean before/after/after. Main loses c2, #7115 alone runs both ask and fetch (c1 duplicated), your patch runs fetch only. Checked the diff against current main and it applies exactly as written: _replay_function_calls, the REPLAY_CALLS call site and _postprocess_handle_function_calls_async in base_llm_flow.py all match your hunk contexts line for line (none of that file is touched by #7115) and handle_function_calls_async's filters: Optional[set[str]] = None plus the fc.id in filters gate in _batch_tool_executor.py were already exactly what you said, nothing invented just wired through. The J case (any call missing an id → None, replay everything) is the right conservative fallback given a filter can't name what it can't identify.
This is functionally the maintainer-level fix for the duplicate-execution half we flagged as still open, done properly. Event-count/tracing invariants elsewhere in base_llm_flow.py aren't touched and filtering happens at the existing filters seam rather than slicing the event itself.
Since you're not signing a CLA for this one: @Zsanz3 given your branch is what this patch sits on top of and you're already going through CLA for #7115 would you fold these two commits in as a third on your branch rather than opening a second PR? It's a direct tested extension of your own fix (34/34 on test_resume_utils.py, 723 passed on the full flows suite with the same single pre-existing failure main has) so one PR covering both halves keeps this to one review instead of two blocked ones.
fix(flows): replay unanswered sibling on parallel tool resume
fix(flows): replay unanswered sibling on parallel tool resume
fix(flows): replay unanswered sibling on parallel tool resume
fix(flows): replay unanswered sibling on parallel tool resume
fix(flows): replay unanswered sibling on parallel tool resume
vasilisnasopoulos commentedon Sep 29, 2026
Thanks for the fix, and for covering both halves: replaying only the calls that never ran is exactly what was missing.
One edge I checked against
_unexecuted_calls_eventon its own (not end to end): when a response carries no id, it answers its call by name, andanswered_namesis a set. So with two parallel calls to the same tool and one id-less response, both count as answered:askc1,fetchc2ask, no id[c2]✅askc1,askc2ask, id c1[c2]✅askc1,askc2ask, no idCounting id-less responses per name instead of a set would keep it to one call per response, e.g. consume them in call order. Not sure how often id-less responses meet same-name parallel calls in practice, so flagging it rather than asserting it.
fix: replay a parallel tool call that never ran when a sibling answered
fix(plugins): record error-bearing tool results as TOOL_ERROR in BigQ…
fix(plugins): record content_formatter failure class in BigQuery anal…
surajksharma07 commentedon Oct 6, 2026
Good catch on the id-less case @vasilisnasopoulos. @SoroushRF picked it up as #7387/#7388 and that landed on main yesterday as 531f410. It does what you suggested: id-less responses are counted per name with a Counter and consumed in call order and explicit ids are matched first without using one up.
Ran your three rows against _unexecuted_calls_event. On 2.11.0 the first two replay [c2] and the third still replays nothing. On current main all three replay [c2]. test_resume.py plus test_base_llm_flow.py pass 161 there.
So the original fix (7298e09) is out in v2.11.0 and the id-less part will ship in the next release. Until then keeping call ids on the responses you restore avoids it. Nothing left open here from my side.