Skip to content

Event streaming in run_async triggers repeated Pydantic serialization #4233

Closed

Description

@ohs30359-nobuhara

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Describe the bug
When running an agent via async for event in runner.run_async(...),
pydantic.BaseModel.model_dump_json() is invoked an unexpectedly large number of times within a single user request.

This results in a noticeable performance impact (CPU time and latency), especially when large objects such as LlmRequest, GenerateContentConfig, and Schema are repeatedly serialized even though they appear to be immutable within the same run.

I am not certain whether this behavior is expected by design or if it is an unintended side effect of the event-driven architecture, so I am reporting this as a potential performance issue / design concern rather than a definitive bug.

To Reproduce
Below is a minimal reproduction that triggers the behavior.

from google.adk.runners import Runner
from src.agents.agent import sample_agent

runner = Runner(agent=sample_agent)

async def main():
    async for event in runner.run_async(
        input="hello",
    ):
        pass  # simply consuming events

# run with asyncio.run(main())

To measure the issue, I monkey-patched BaseModel.model_dump_json to record:

  • call count
  • latency
  • serialized size

Observed behavior
For a single user request, model_dump_json() is called thousands of times.
Example aggregated metrics (per single request):

google.adk.events.event.Event                        n=    177
google.adk.models.llm_request.LlmRequest             n=   1886
google.adk.models.llm_response.LlmResponse           n=   3772
google.adk.sessions.session.Session                  n=     69
google.genai.types.Content                           n=     61
google.genai.types.FunctionDeclaration               n=   7048
google.genai.types.Schema                            n=  49380
google.genai.types.GenerateContentConfig             n=   1886

Large immutable objects (e.g. schema and config) appear to be repeatedly serialized during event emission.

Expected behavior
One of the following (or clarification if current behavior is intentional):

  • Fewer repeated model_dump_json() calls for immutable objects within a single run
  • Caching or memoization of serialized representations
  • Documentation clarifying that this level of serialization is expected when using run_async with event streaming

Screenshots
N/A

Desktop (please complete the following information):

  • OS: Linux
  • Python version(python -V): 3.13
  • ADK version(pip show google-adk): 1.21.0

Model Information:

  • Are you using LiteLLM: Yes
  • Which model is being used(e.g. gemini-2.5-pro): gpt-4.1-mini

Additional context
I suspect this may be related to the event-driven architecture where the same state objects are serialized for each emitted event.

I would appreciate guidance on:

  • whether this behavior is expected
  • whether there are recommended patterns to avoid excessive serialization
  • or whether this could be optimized within ADK itself

If this is primarily caused by incorrect usage on my side, I would be happy to adjust my implementation accordingly.

Activity

  1. added
    live[Component] This issue is related to live, voice and video chat
    on Jan 22, 2026
  2. added theissue type on Jan 22, 2026
  3. challenger71498 commented on Jan 24, 2026

    @challenger71498

    Hi @ohs30359-nobuhara,

    I tried to reproduce your issue by:

    • implement pydantic profiler for model_dump_json by monkeypatching
    • run a simple agent with LiteLLM
    • tested at
      • the latest commit of the main branch (324796b)
      • v1.21.0
    • tested 3 cases
      • using Gemini 2.5 Flash directly, no LiteLLM
      • LiteLLM with Gemini 2.5 Flash
      • LiteLLM with GPT 4.1 mini

    But in my case the invocation count of BaseModel.model_dump_json() only called a few.

    I will share my code and the execution result, please let me know if something is missing or incorrect.

    Profiler python code
    """Monkey-patch for profiling Pydantic BaseModel.model_dump_json calls."""
    
    from __future__ import annotations
    
    import time
    from collections import defaultdict
    from typing import Any
    
    from pydantic import BaseModel
    
    
    class SerializationStats:
        """Statistics for model_dump_json calls."""
    
        def __init__(self):
            self.call_count = 0
            self.total_latency_ms = 0.0
            self.total_size_bytes = 0
    
        def record(self, latency_ms: float, size_bytes: int):
            """Record a serialization call."""
            self.call_count += 1
            self.total_latency_ms += latency_ms
            self.total_size_bytes += size_bytes
    
        @property
        def avg_latency_ms(self) -> float:
            """Average latency per call in milliseconds."""
            return self.total_latency_ms / self.call_count if self.call_count > 0 else 0
    
        @property
        def avg_size_bytes(self) -> float:
            """Average size per call in bytes."""
            return self.total_size_bytes / self.call_count if self.call_count > 0 else 0
    
    
    # Global storage for stats
    _serialization_stats: dict[str, SerializationStats] = defaultdict(SerializationStats)
    
    # Store original method
    _original_model_dump_json = BaseModel.model_dump_json
    
    
    def _patched_model_dump_json(self, **kwargs) -> str:
        """Patched version of model_dump_json that records metrics."""
        # Get fully qualified class name
        class_name = f"{self.__class__.__module__}.{self.__class__.__name__}"
    
        # Time the serialization
        start_time = time.perf_counter()
        result = _original_model_dump_json(self, **kwargs)
        end_time = time.perf_counter()
    
        # Record metrics
        latency_ms = (end_time - start_time) * 1000
        size_bytes = len(result.encode("utf-8"))
        _serialization_stats[class_name].record(latency_ms, size_bytes)
    
        return result
    
    
    def install_profiler():
        """Install the monkey-patch for profiling model_dump_json."""
        BaseModel.model_dump_json = _patched_model_dump_json
    
    
    def uninstall_profiler():
        """Remove the monkey-patch and restore original behavior."""
        BaseModel.model_dump_json = _original_model_dump_json
    
    
    def reset_stats():
        """Clear all collected statistics."""
        _serialization_stats.clear()
    
    
    def get_stats() -> dict[str, SerializationStats]:
        """Get all collected statistics."""
        return dict(_serialization_stats)
    
    
    def print_stats(
        sort_by: str = "count",
        min_calls: int = 1,
        top_n: int | None = None,
    ):
        """Print formatted statistics.
    
        Args:
          sort_by: Sort key - 'count', 'latency', 'size', or 'name'
          min_calls: Minimum number of calls to include in output
          top_n: Only show top N entries (None for all)
        """
        stats = get_stats()
    
        # Filter by minimum calls
        filtered_stats = {
            name: stat for name, stat in stats.items() if stat.call_count >= min_calls
        }
    
        if not filtered_stats:
            print("No statistics collected yet.")
            return
    
        # Sort based on criteria
        if sort_by == "count":
            sorted_items = sorted(
                filtered_stats.items(), key=lambda x: x[1].call_count, reverse=True
            )
        elif sort_by == "latency":
            sorted_items = sorted(
                filtered_stats.items(),
                key=lambda x: x[1].total_latency_ms,
                reverse=True,
            )
        elif sort_by == "size":
            sorted_items = sorted(
                filtered_stats.items(), key=lambda x: x[1].total_size_bytes, reverse=True
            )
        elif sort_by == "name":
            sorted_items = sorted(filtered_stats.items(), key=lambda x: x[0])
        else:
            raise ValueError(
                f"Invalid sort_by value: {sort_by}. "
                "Must be 'count', 'latency', 'size', or 'name'"
            )
    
        # Limit to top N if specified
        if top_n is not None:
            sorted_items = sorted_items[:top_n]
    
        # Print header
        print("\nPydantic model_dump_json() Statistics")
        print("=" * 100)
        print(
            f"{'Class Name':<60} {'Calls':>8} {'Total ms':>12} {'Avg ms':>10}"
            f" {'Total KB':>10} {'Avg B':>8}"
        )
        print("-" * 100)
    
        # Print stats
        for class_name, stat in sorted_items:
            print(
                f"{class_name:<60} {stat.call_count:>8} "
                f"{stat.total_latency_ms:>12.2f} {stat.avg_latency_ms:>10.4f} "
                f"{stat.total_size_bytes / 1024:>10.2f} {stat.avg_size_bytes:>8.0f}"
            )
    
        print("=" * 100)
        print(f"Total unique classes: {len(sorted_items)}")
    
    
    def print_stats_simple(min_calls: int = 1):
        """Print simple statistics matching the example format.
    
        Args:
          min_calls: Minimum number of calls to include in output
        """
        stats = get_stats()
    
        # Filter and sort by call count
        filtered_stats = {
            name: stat for name, stat in stats.items() if stat.call_count >= min_calls
        }
    
        if not filtered_stats:
            print("No statistics collected yet.")
            return
    
        sorted_items = sorted(
            filtered_stats.items(), key=lambda x: x[1].call_count, reverse=True
        )
    
        # Find max class name length for padding
        max_name_len = max(len(name) for name, _ in sorted_items)
    
        # Print stats
        for class_name, stat in sorted_items:
            print(f"{class_name:<{max_name_len}} n={stat.call_count:>7}")
    An agent execution example python code
    """Example usage of the Pydantic profiler."""
    
    from __future__ import annotations
    
    from google.adk.agents import Agent
    from google.adk.models.lite_llm import LiteLlm
    from google.adk.runners import Runner
    from google.adk.sessions.in_memory_session_service import InMemorySessionService
    from google.genai.types import Content, Part
    
    import pydantic_profiler
    
    
    async def main():
        # Install the profiler BEFORE creating any agents or running them
        pydantic_profiler.install_profiler()
    
        model = LiteLlm(
            # model="gemini/gemini-2.5-flash",
            model="openai/gpt-4.1-mini",
        )
    
        # Create and run your agent
        agent = Agent(
            name="test_agent",
            model=model,
            instruction="You are a helpful assistant.",
        )
    
        session_service = InMemorySessionService()
        session = await session_service.create_session(
            app_name="profiler_example",
            user_id="test_user",
        )
    
        runner = Runner(
            app_name="profiler_example",
            agent=agent,
            session_service=session_service,
        )
    
        # Run some interactions
        print("Running agent interactions...")
        async for event in runner.run_async(
            user_id=session.user_id,
            session_id=session.id,
            new_message=Content(
                parts=[Part(text="Hello, how are you?")],
                role="user",
            ),
        ):
            print(f"Agent response: {event.content.model_dump_json()[:50]}...")
    
        # Print statistics in simple format (like the example)
        print("\n" + "=" * 80)
        print("SIMPLE FORMAT:")
        print("=" * 80)
        pydantic_profiler.print_stats_simple(min_calls=1)
    
        # Print detailed statistics
        print("\n" + "=" * 80)
        print("DETAILED FORMAT:")
        print("=" * 80)
        pydantic_profiler.print_stats(sort_by="count", min_calls=10)
    
        # You can also get raw stats for custom analysis
        stats = pydantic_profiler.get_stats()
        print(f"\nTotal unique Pydantic classes serialized: {len(stats)}\n")
    
        # Clean up
        pydantic_profiler.uninstall_profiler()
    
    
    if __name__ == "__main__":
        import asyncio
    
        asyncio.run(main())
    Execution result
    Running agent interactions...
    Agent response: {"parts":[{"media_resolution":null,"code_execution...
    
    ================================================================================
    SIMPLE FORMAT:
    ================================================================================
    google.genai.types.Content                 n=      2
    google.adk.models.llm_response.LlmResponse n=      1
    
    ================================================================================
    DETAILED FORMAT:
    ================================================================================
    No statistics collected yet.
    
    Total unique Pydantic classes serialized: 2
    

    Please let me know if there is a correction needed.
    Your opinions are greatly appreciated :)
    Thank you.

  4. ohs30359-nobuhara commented on Jan 26, 2026

    @ohs30359-nobuhara
    Author

    @challenger71498 Thank you very much for taking the time to reproduce this and for sharing the code and results.

    With the setup you provided, I agree that the behavior I observed is likely not reproducible, and I apologize for not clearly communicating the assumptions on my side earlier.

    In my actual implementation, the agent configuration is significantly more complex:

    • The top-level agent has approximately 15 sub-agents registered.
    • Among those, 5–6 sub-agents themselves have additional sub-agents (typically 2–3 each).
    • All sub-agents have around 3 tools registered.
    • As a result, a single runner.run_async(...) execution involves a fairly large agent tree and tool graph.

    Given this structure, my current hypothesis is that the observed behavior is correlated with the number of agents actually involved in a single run, rather than being triggered by a simple single-agent setup.
    In particular, repeated event emission across many agents (and tools) may cause the same immutable Pydantic objects (e.g. schemas and configuration objects) to be serialized multiple times within a single user request.

    Unfortunately, I’m unable to share the exact production code due to internal constraints. However, I will try to prepare a reduced but structurally similar example (with nested sub-agents and tools) to better approximate the conditions under which the issue appears.

    Thank you again for investigating this, and I appreciate any further guidance on whether this behavior is expected in more complex agent graphs, or if there are recommended patterns to mitigate excessive serialization in such scenarios.

  5. challenger71498 commented on Jan 26, 2026

    @challenger71498

    Hi @ohs30359-nobuhara, thank you for your detailed response! Note that I am not a maintainer of ADK, I'm just an another user.

    Given this structure, my current hypothesis is that the observed behavior is correlated with the number of agents actually involved in a single run, rather than being triggered by a simple single-agent setup.

    With your description, I agree that to reproduce the issue the structure of the agent should be complicated. It would be great if you provide an example as you mentioned. :)

    In particular, repeated event emission across many agents (and tools) may cause the same immutable Pydantic objects (e.g. schemas and configuration objects) to be serialized multiple times within a single user request.

    This sounds intriguing. As you mentioned this behaviour would be triggered by same Pydantic objects are serialized multiple times, without caching it. In a normal circumstances this would not cause problems, but in a heavy I/O load situation this may be a problem.

    Just in case, grouping the count of the invocation of model_dump_json() by serialized string may help to determine whether ADK actually serializes the same object.

    An example of the output would be like:

    google.adk.events.event.Event                        n=    20    value="{foo: bar, baz: 123}"
    google.adk.events.event.Event                        n=    17    value="{foo: bar, baz: 1234}" # Same type but different value
    google.adk.models.llm_request.LlmRequest             n=    12    value="{abc: 16374, def: 0.123}"

    The project I am working on also utilizes ADK heavily, a multi-agent system which uses about ~20 sub-agents. Maybe profiling it would help analyzing the behaviour of ADK. I will try profiling it if I can.

  6. llalitkumarrr commented on Feb 17, 2026

    @llalitkumarrr
    Collaborator

    Hello @ohs30359-nobuhara ,

    We recommend upgrading to the latest google-adk version and testing whether this resolves the issue. If you continue to observe the issue after the upgrade, please share your detailed observations with us to assist in our analysis.

  7. ohs30359-nobuhara commented on Feb 20, 2026

    @ohs30359-nobuhara
    Author

    Thank you for your support.

    After investigating this issue for some time, I found that model_dump_json is called under the following conditions:

    1. google.genai.types.Schema

    When Optional or Union is specified as the argument type for Tools registered to an Agent,
    model_dump_json is invoked every time an event is triggered in the Runner.

    2. The following three classes
    For the classes below, model_dump_json appears to be called every time a Runner event is triggered when OpenTelemetry-

    ■ related libraries are enabled:

    • google.adk.models.llm_request.LlmRequest
    • google.genai.types.FunctionDeclaration
    • google.genai.types.GenerateContentConfig

    I confirmed that enabling the following modules causes the dump to be executed:

    from openinference.instrumentation import TracerProvider
    from openinference.instrumentation.google_adk import GoogleADKInstrumentor
    from opentelemetry.sdk.resources import Resource
    
    tp = TracerProvider(resource=Resource.create({"service.name": "test"}))
    GoogleADKInstrumentor().instrument(tracer_provider=tp)
    

    3. google.adk.models.llm_response.LlmResponse
    For this class, model_dump_json is invoked when:

    run_config=RunConfig(support_cfc=True)
    

    Example:

    async for event in runner.run_async(
        user_id=session.user_id,
        session_id=session.id,
        new_message=Content(parts=[Part(text=prompt)], role="user"),
        run_config=RunConfig(support_cfc=True),  # <-- here
    ):
    

    Summary
    Overall, many model_dump_json calls are executed for telemetry collection and integration purposes,
    and this appears to impact performance.

    While the behavior related to google.genai.types.Schema can be avoided,
    in a production service environment it is difficult to choose not to collect telemetry at all.

    Therefore, it would be helpful if mitigation options like the following were available:

    • Introducing a sampling rate for telemetry
    • Replacing model_dump_json with a lighter-weight serialization method (e.g., toJson or similar)

    Are there any mitigation strategies currently being considered for this issue?

  8. llalitkumarrr commented on Mar 16, 2026

    @llalitkumarrr
    Collaborator

    Hello @wuliang229 ,

    Kindly take a look at this.

  9. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Mar 16, 2026
  10. removed their assignment
    on Mar 26, 2026
  11. added
    core[Component] This issue is related to the core interface and implementation
    and removed
    live[Component] This issue is related to live, voice and video chat
    on Jun 22, 2026
  12. dexhunter commented on Jul 11, 2026

    @dexhunter
    Contributor

    I isolated one narrow path related to the telemetry serialization described here: trace_call_llm currently prepares telemetry configuration and serializes request/response content even when the resolved span is not recording. A focused early return preserves recording-span behavior and reduced the non-recording helper benchmark from 139.820 us to 0.123 us.

    I prepared #6371 with recording/non-recording regression coverage, but I opened it before receiving the explicit scope confirmation requested by the contribution guide. I have moved it back to draft. Would maintainers like this narrow fix to proceed, or should I close it and wait for a broader direction on this issue?

  13. dexhunter commented on Jul 13, 2026

    @dexhunter
    Contributor

    I traced one concrete part of this report to trace_call_llm: it serializes the request and response before setting span attributes even when the resolved span is not recording. Those values cannot be retained in that case.

    I have a small draft fix in #6371 that resolves the span first and returns only when span.is_recording() is false. Recording spans keep the existing attribute and serialization path. The regression test covers explicit and current spans and verifies that request/response serialization is skipped only for non-recording spans.

    In a helper-level benchmark with content capture enabled and a large deterministic response, the non-recording path changed from a 153.571 us/call median to 0.122233 us/call. This is not an end-to-end latency claim and does not attempt to solve every serialization path reported here.

    The direction came from a 12-step autoresearch run and was then rebuilt and validated independently: public trajectory.

    Would this narrow non-recording-span guard be acceptable as a contribution for this issue? I have kept #6371 in draft until the scope is confirmed.

  14. added a commit that references this issue on Jul 13, 2026
    0c63a89
  15. saitejavn commented on Jul 30, 2026

    @saitejavn

    Thank you for your support.

    After investigating this issue for some time, I found that model_dump_json is called under the following conditions:

    1. google.genai.types.Schema

    When Optional or Union is specified as the argument type for Tools registered to an Agent, model_dump_json is invoked every time an event is triggered in the Runner.

    2. The following three classes For the classes below, model_dump_json appears to be called every time a Runner event is triggered when OpenTelemetry-

    ■ related libraries are enabled:

    • google.adk.models.llm_request.LlmRequest
    • google.genai.types.FunctionDeclaration
    • google.genai.types.GenerateContentConfig

    I confirmed that enabling the following modules causes the dump to be executed:

    from openinference.instrumentation import TracerProvider
    from openinference.instrumentation.google_adk import GoogleADKInstrumentor
    from opentelemetry.sdk.resources import Resource
    
    tp = TracerProvider(resource=Resource.create({"service.name": "test"}))
    GoogleADKInstrumentor().instrument(tracer_provider=tp)
    

    3. google.adk.models.llm_response.LlmResponse For this class, model_dump_json is invoked when:

    run_config=RunConfig(support_cfc=True)
    

    Example:

    async for event in runner.run_async(
        user_id=session.user_id,
        session_id=session.id,
        new_message=Content(parts=[Part(text=prompt)], role="user"),
        run_config=RunConfig(support_cfc=True),  # <-- here
    ):
    

    Summary Overall, many model_dump_json calls are executed for telemetry collection and integration purposes, and this appears to impact performance.

    While the behavior related to google.genai.types.Schema can be avoided, in a production service environment it is difficult to choose not to collect telemetry at all.

    Therefore, it would be helpful if mitigation options like the following were available:

    • Introducing a sampling rate for telemetry
    • Replacing model_dump_json with a lighter-weight serialization method (e.g., toJson or similar)

    Are there any mitigation strategies currently being considered for this issue?

    Hey @ohs30359-nobuhara, Thank you for the investigation. I'm running into this exact issue. Were you able to resolve it or is it still open?

  16. ohs30359-nobuhara commented on Jul 30, 2026

    @ohs30359-nobuhara
    Author

    We haven't found a fundamental solution to this issue yet.

    I'm not sure if this will be helpful, but I'd like to share the mitigation measures we're currently using:

    • Configure a custom telemetry sampling rate.
    • Place a proxy between the application and the LLM to buffer SSE chunks.

    Because of how Telemetry is implemented, model_dump_json is invoked for every chunk.
    This means that the smaller and more frequent the SSE chunks are, the higher the overhead becomes.
    As a result, buffering SSE chunks at the proxy layer has been especially effective.

  17. added a commit that references this issue on Aug 11, 2026
    7fd438e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

core[Component] This issue is related to the core interface and implementationneeds review[Status] The PR/issue is awaiting review from the maintainer

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions