Repository navigation
Description
** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.
Describe the bug
When running an agent via async for event in runner.run_async(...),
pydantic.BaseModel.model_dump_json() is invoked an unexpectedly large number of times within a single user request.
This results in a noticeable performance impact (CPU time and latency), especially when large objects such as LlmRequest, GenerateContentConfig, and Schema are repeatedly serialized even though they appear to be immutable within the same run.
I am not certain whether this behavior is expected by design or if it is an unintended side effect of the event-driven architecture, so I am reporting this as a potential performance issue / design concern rather than a definitive bug.
To Reproduce
Below is a minimal reproduction that triggers the behavior.
from google.adk.runners import Runner
from src.agents.agent import sample_agent
runner = Runner(agent=sample_agent)
async def main():
async for event in runner.run_async(
input="hello",
):
pass # simply consuming events
# run with asyncio.run(main())
To measure the issue, I monkey-patched BaseModel.model_dump_json to record:
- call count
- latency
- serialized size
Observed behavior
For a single user request, model_dump_json() is called thousands of times.
Example aggregated metrics (per single request):
google.adk.events.event.Event n= 177
google.adk.models.llm_request.LlmRequest n= 1886
google.adk.models.llm_response.LlmResponse n= 3772
google.adk.sessions.session.Session n= 69
google.genai.types.Content n= 61
google.genai.types.FunctionDeclaration n= 7048
google.genai.types.Schema n= 49380
google.genai.types.GenerateContentConfig n= 1886
Large immutable objects (e.g. schema and config) appear to be repeatedly serialized during event emission.
Expected behavior
One of the following (or clarification if current behavior is intentional):
- Fewer repeated
model_dump_json()calls for immutable objects within a single run - Caching or memoization of serialized representations
- Documentation clarifying that this level of serialization is expected when using
run_asyncwith event streaming
Screenshots
N/A
Desktop (please complete the following information):
- OS: Linux
- Python version(python -V): 3.13
- ADK version(pip show google-adk): 1.21.0
Model Information:
- Are you using LiteLLM: Yes
- Which model is being used(e.g. gemini-2.5-pro): gpt-4.1-mini
Additional context
I suspect this may be related to the event-driven architecture where the same state objects are serialized for each emitted event.
I would appreciate guidance on:
- whether this behavior is expected
- whether there are recommended patterns to avoid excessive serialization
- or whether this could be optimized within ADK itself
If this is primarily caused by incorrect usage on my side, I would be happy to adjust my implementation accordingly.
Activity
challenger71498 commentedon Jan 24, 2026
Hi @ohs30359-nobuhara,
I tried to reproduce your issue by:
But in my case the invocation count of
BaseModel.model_dump_json()only called a few.I will share my code and the execution result, please let me know if something is missing or incorrect.
Profiler python code
An agent execution example python code
Execution result
Please let me know if there is a correction needed.
Your opinions are greatly appreciated :)
Thank you.
ohs30359-nobuhara commentedon Jan 26, 2026
@challenger71498 Thank you very much for taking the time to reproduce this and for sharing the code and results.
With the setup you provided, I agree that the behavior I observed is likely not reproducible, and I apologize for not clearly communicating the assumptions on my side earlier.
In my actual implementation, the agent configuration is significantly more complex:
Given this structure, my current hypothesis is that the observed behavior is correlated with the number of agents actually involved in a single run, rather than being triggered by a simple single-agent setup.
In particular, repeated event emission across many agents (and tools) may cause the same immutable Pydantic objects (e.g. schemas and configuration objects) to be serialized multiple times within a single user request.
Unfortunately, I’m unable to share the exact production code due to internal constraints. However, I will try to prepare a reduced but structurally similar example (with nested sub-agents and tools) to better approximate the conditions under which the issue appears.
Thank you again for investigating this, and I appreciate any further guidance on whether this behavior is expected in more complex agent graphs, or if there are recommended patterns to mitigate excessive serialization in such scenarios.
challenger71498 commentedon Jan 26, 2026
Hi @ohs30359-nobuhara, thank you for your detailed response! Note that I am not a maintainer of ADK, I'm just an another user.
With your description, I agree that to reproduce the issue the structure of the agent should be complicated. It would be great if you provide an example as you mentioned. :)
This sounds intriguing. As you mentioned this behaviour would be triggered by same Pydantic objects are serialized multiple times, without caching it. In a normal circumstances this would not cause problems, but in a heavy I/O load situation this may be a problem.
Just in case, grouping the count of the invocation of
model_dump_json()by serialized string may help to determine whether ADK actually serializes the same object.An example of the output would be like:
The project I am working on also utilizes ADK heavily, a multi-agent system which uses about ~20 sub-agents. Maybe profiling it would help analyzing the behaviour of ADK. I will try profiling it if I can.
llalitkumarrr commentedon Feb 17, 2026
Hello @ohs30359-nobuhara ,
We recommend upgrading to the latest
google-adkversion and testing whether this resolves the issue. If you continue to observe the issue after the upgrade, please share your detailed observations with us to assist in our analysis.ohs30359-nobuhara commentedon Feb 20, 2026
Thank you for your support.
After investigating this issue for some time, I found that
model_dump_jsonis called under the following conditions:1. google.genai.types.Schema
When Optional or Union is specified as the argument type for Tools registered to an Agent,
model_dump_jsonis invoked every time an event is triggered in the Runner.2. The following three classes
For the classes below, model_dump_json appears to be called every time a Runner event is triggered when OpenTelemetry-
■ related libraries are enabled:
I confirmed that enabling the following modules causes the dump to be executed:
3. google.adk.models.llm_response.LlmResponse
For this class,
model_dump_jsonis invoked when:Example:
Summary
Overall, many
model_dump_jsoncalls are executed for telemetry collection and integration purposes,and this appears to impact performance.
While the behavior related to google.genai.types.Schema can be avoided,
in a production service environment it is difficult to choose not to collect telemetry at all.
Therefore, it would be helpful if mitigation options like the following were available:
Are there any mitigation strategies currently being considered for this issue?
llalitkumarrr commentedon Mar 16, 2026
Hello @wuliang229 ,
Kindly take a look at this.
dexhunter commentedon Jul 11, 2026
I isolated one narrow path related to the telemetry serialization described here:
trace_call_llmcurrently prepares telemetry configuration and serializes request/response content even when the resolved span is not recording. A focused early return preserves recording-span behavior and reduced the non-recording helper benchmark from 139.820 us to 0.123 us.I prepared #6371 with recording/non-recording regression coverage, but I opened it before receiving the explicit scope confirmation requested by the contribution guide. I have moved it back to draft. Would maintainers like this narrow fix to proceed, or should I close it and wait for a broader direction on this issue?
perf(telemetry): skip non-recording LLM trace work
dexhunter commentedon Jul 13, 2026
I traced one concrete part of this report to
trace_call_llm: it serializes the request and response before setting span attributes even when the resolved span is not recording. Those values cannot be retained in that case.I have a small draft fix in #6371 that resolves the span first and returns only when
span.is_recording()is false. Recording spans keep the existing attribute and serialization path. The regression test covers explicit and current spans and verifies that request/response serialization is skipped only for non-recording spans.In a helper-level benchmark with content capture enabled and a large deterministic response, the non-recording path changed from a
153.571 us/callmedian to0.122233 us/call. This is not an end-to-end latency claim and does not attempt to solve every serialization path reported here.The direction came from a 12-step autoresearch run and was then rebuilt and validated independently: public trajectory.
Would this narrow non-recording-span guard be acceptable as a contribution for this issue? I have kept #6371 in draft until the scope is confirmed.
perf(telemetry): skip non-recording LLM trace work
perf(telemetry): skip non-recording LLM trace work
perf: skip non-recording LLM trace work
saitejavn commentedon Jul 30, 2026
Hey @ohs30359-nobuhara, Thank you for the investigation. I'm running into this exact issue. Were you able to resolve it or is it still open?
ohs30359-nobuhara commentedon Jul 30, 2026
We haven't found a fundamental solution to this issue yet.
I'm not sure if this will be helpful, but I'd like to share the mitigation measures we're currently using:
Because of how Telemetry is implemented, model_dump_json is invoked for every chunk.
This means that the smaller and more frequent the SSE chunks are, the higher the overhead becomes.
As a result, buffering SSE chunks at the proxy layer has been especially effective.
perf: skip non-recording LLM trace work