Skip to content

opentelemetry ValueError: "Token was created in a different Context" with ParallelAgent and LlmAgent #860

Closed as duplicate

Description

@divanshu-goyal-rc

When using google.adk.agents.ParallelAgent with multiple google.adk.agents.LlmAgent instances as sub-agents, ValueError exceptions from the opentelemetry.context module are consistently logged. These errors occur after each sub-agent completes its processing. While the core functionality of the agents (e.g., LLM calls and output generation) appears to be unaffected, these errors create noise in the logs and suggest a potential issue with context management within the ADK framework or its interaction with opentelemetry.

Steps to Reproduce:

  1. Define multiple LlmAgent instances, each with its own model, instructions, and output schema.
  2. Define an AsyncExitStack and initialize tools (e.g., MCPToolset).
  3. Instantiate a ParallelAgent, providing the LlmAgent instances to its sub_agents parameter.
  4. The root agent setup returns the ParallelAgent instance and the AsyncExitStack.
  5. Run the agent (e.g., by making a request to the FastAPI endpoint if using google.adk.cli.fast_api).

Expected Behavior:
The ParallelAgent and its sub-agents should execute without opentelemetry context errors being logged.

Actual Behavior:
After each sub-agent finishes its execution and its output is generated, the following type of error is logged multiple times:

ERROR - __init__.py:157 - Failed to detach context
Traceback (most recent call last):
  File "/path/to/your/env/lib/pythonX.Y/site-packages/opentelemetry/context/__init__.py", line 155, in detach
    _RUNTIME_CONTEXT.detach(token)
  File "/path/to/your/env/lib/pythonX.Y/site-packages/opentelemetry/context/contextvars_context.py", line 53, in detach
    self._current_context.reset(token)
ValueError: <Token var=<ContextVar name='current_context' default={} at 0xXXXXXXXXX> at 0xYYYYYYYYY> was created in a different Context

(Note: Replace /path/to/your/env/lib/pythonX.Y/ and memory addresses with actuals from your logs if desired)

Environment:

  • Operating System: macOS (darwin 23.5.0)
  • Python Version: 3.12
  • Google ADK Version: 0.4.0
  • Minimal Reproducible Code (agent.py):
    import asyncio
    from google.adk.agents import LlmAgent, BaseAgent, ParallelAgent
    from pydantic import BaseModel, Field
    from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset, SseServerParams
    from contextlib import AsyncExitStack
    import enum
    
    # ... (Out_of_office_category, Out_of_office_output_schema definitions) ...
    # ... (New_lead_category, New_lead_output_schema definitions) ...
    # ... (Book_meeting_category, Book_meeting_output_schema definitions) ...
    # ... (General_type_reply_category, General_type_reply_output_schema definitions) ...
    
    # [PASTE YOUR Enum and BaseModel Schema definitions here for completeness]
    
    class Out_of_office_category(enum.Enum):
        resignation = "resignation"
        holiday = "holiday"
        other = "other"
        not_out_of_office = "not_out_of_office"
    
    class Out_of_office_output_schema(BaseModel):
        out_of_office: bool = Field(description="Whether the email is out of office or not")
        category: Out_of_office_category = Field(description="The category of the email")
        reason: str = Field(description="The reason for the email being out of office")
    
    class New_lead_category(enum.Enum):
        not_right_person_add_new_lead_new_email = "not_right_person_add_new_lead_new_email"
        right_person_add_new_lead_new_email = "right_person_add_new_lead_new_email"
        right_person_add_new_lead_existing_email = "right_person_add_new_lead_existing_email"
        not_add_new_lead = "not_add_new_lead"
    
    class New_lead_output_schema(BaseModel):
        new_lead: bool = Field(description="Whether the email contains a new lead or not")
        new_lead_category: New_lead_category = Field(description="The category of the email")
        reason: str = Field(description="The reason for the email being a new lead or not")
    
    class Book_meeting_category(enum.Enum):
        book_meeting = "book_meeting"
        book_meeting_details = "book_meeting_details"
        not_book_meeting = "not_book_meeting"
    
    class Book_meeting_output_schema(BaseModel):
        book_meeting: bool = Field(description="Whether the email contains a request to book a meeting or not")
        book_meeting_category: Book_meeting_category = Field(description="The category of the email")
        reason: str = Field(description="The reason for the email being a book meeting or not")
    
    class General_type_reply_category(enum.Enum):
        positive = "positive"
        negative = "negative"
    
    class General_type_reply_output_schema(BaseModel):
        general_type_reply_category: General_type_reply_category = Field(description="The category of the email")
        reason: str = Field(description="The reason for the email being a positive or negative reply")
    
    
    async def reply_agent():
        common_exit_stack = AsyncExitStack()
        tools, _ = await MCPToolset.from_server(
            connection_params=SseServerParams(
              url="http://44.214.120.160:8002/sse" # Example URL
          ),
          async_exit_stack=common_exit_stack
        )
    
        out_of_office_agent = LlmAgent(
                model="gemini-2.0-flash",
                name="out_of_office_agent",
                description="check whether the email is out of office or not...",
                instruction=\"\"\"...\"\"\", # Shorten for brevity or refer to full agent code
                output_schema=Out_of_office_output_schema,
                output_key="out_of_office_agent"
            )
        
        new_lead_agent = LlmAgent(
            model="gemini-2.0-flash",
            name="new_lead_agent",
            description="Check whether the email contains a new lead...",
            instruction=\"\"\"...\"\"\",
            output_schema=New_lead_output_schema,
            output_key="new_lead_agent"
        )
    
        book_meeting_agent = LlmAgent(
            model="gemini-2.0-flash",
            name="book_meeting_agent",
            description="Check whether the email contains a request to book a meeting...",
            instruction=\"\"\"...\"\"\",
            output_schema=Book_meeting_output_schema,
            output_key="book_meeting_agent"
        )
        
        general_type_reply_agent = LlmAgent(
            model="gemini-2.0-flash",
            name="general_type_reply_agent",
            description="Classify the email into 'positive', 'negative'...",
            instruction=\"\"\"...\"\"\",
            output_schema=General_type_reply_output_schema,
            output_key="general_type_reply_agent"
        )
    
        return out_of_office_agent, new_lead_agent, book_meeting_agent, general_type_reply_agent, common_exit_stack
    
    
    async def main():
        out_of_office_agent, new_lead_agent, book_meeting_agent, general_type_reply_agent, common_exit_stack = await reply_agent()
        reply_intent_agent = ParallelAgent(
                name="replyagent",
                description="Analyze the email and determine the intent of the email.",
                sub_agents=[out_of_office_agent, new_lead_agent, book_meeting_agent, general_type_reply_agent]
            )
        return reply_intent_agent, common_exit_stack
    
    root_agent = main()

Additional Context:

  • The sub-agents themselves successfully process inputs and produce the expected outputs via LLM calls. The opentelemetry errors seem to occur around the completion/logging phase of each sub-agent's lifecycle within the ParallelAgent.

Activity

  1. added
    core[Component] This issue is related to the core interface and implementation
    on May 22, 2025
  2. added theissue type on May 22, 2025
  3. calvingiles commented on May 23, 2025

    @calvingiles
    Contributor

    Disclaimer: not googler.

    We have seen this, and it seems to be caused by another error, and the trace error is masking the underlying error.

  4. Colstuwjx commented on May 26, 2025

    @Colstuwjx
    Contributor

    similar issue reproduced on my local, it would be great if we could fix this issue.

  5. Jacksunwei commented on May 28, 2025

    @Jacksunwei
    Collaborator

    Could you retry with 1.0.0. I think we've fixed this already at head.

  6. Colstuwjx commented on May 28, 2025

    @Colstuwjx
    Contributor

    Hi @Jacksunwei , unfortuantely, I had just reproduced this issue by using 1.0.0 . See the error logs below:

    Error in event_generator: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Permission denied on resource project XXX.', 'status': 'PERMISSION_DENIED', 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'CONSUMER_INVALID', 'domain': 'googleapis.com', 'metadata': {'containerInfo': 'XXX', 'consumer': 'projects/XXX', 'service': 'aiplatform.googleapis.com'}}, {'@type': 'type.googleapis.com/google.rpc.LocalizedMessage', 'locale': 'en-US', 'message': 'Permission denied on resource project XXX.'}, {'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Google developers console', 'url': 'https://console.developers.google.com'}]}]}}
    Traceback (most recent call last):
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/cli/fast_api.py", line 697, in event_generator
        async for event in runner.run_async(
        ...<8 lines>...
          yield f"data: {sse_event}\n\n"
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/runners.py", line 197, in run_async
        async for event in invocation_context.agent.run_async(invocation_context):
        ...<2 lines>...
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/agents/base_agent.py", line 147, in run_async
        async for event in self._run_async_impl(ctx):
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/agents/llm_agent.py", line 278, in _run_async_impl
        async for event in self._llm_flow.run_async(ctx):
          self.__maybe_save_output_to_state(event)
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 279, in run_async
        async for event in self._run_one_step_async(invocation_context):
          last_event = event
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 309, in _run_one_step_async
        async for event in self._postprocess_async(
        ...<4 lines>...
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 381, in _postprocess_async
        async for event in self._postprocess_handle_function_calls_async(
        ...<2 lines>...
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 471, in _postprocess_handle_function_calls_async
        async for event in agent_to_run.run_async(invocation_context):
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/agents/base_agent.py", line 147, in run_async
        async for event in self._run_async_impl(ctx):
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/agents/llm_agent.py", line 278, in _run_async_impl
        async for event in self._llm_flow.run_async(ctx):
          self.__maybe_save_output_to_state(event)
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 279, in run_async
        async for event in self._run_one_step_async(invocation_context):
          last_event = event
          yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 305, in _run_one_step_async
        async for llm_response in self._call_llm_async(
        ...<8 lines>...
            yield event
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 520, in _call_llm_async
        async for llm_response in llm.generate_content_async(
        ...<16 lines>...
          yield llm_response
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/models/google_llm.py", line 146, in generate_content_async
        response = await self.api_client.aio.models.generate_content(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ...<3 lines>...
        )
        ^
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/genai/models.py", line 7124, in generate_content
        response = await self._generate_content(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            model=model, contents=contents, config=parsed_config
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/genai/models.py", line 6118, in _generate_content
        response_dict = await self._api_client.async_request(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            'post', path, request_dict, http_options
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/genai/_api_client.py", line 789, in async_request
        result = await self._async_request(http_request=http_request, stream=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/genai/_api_client.py", line 733, in _async_request
        await errors.APIError.raise_for_async_response(response)
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/genai/errors.py", line 129, in raise_for_async_response
        raise ClientError(status_code, response_json, response)
    google.genai.errors.ClientError: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Permission denied on resource project XXX.', 'status': 'PERMISSION_DENIED', 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'CONSUMER_INVALID', 'domain': 'googleapis.com', 'metadata': {'containerInfo': 'XXX', 'consumer': 'projects/XXX', 'service': 'aiplatform.googleapis.com'}}, {'@type': 'type.googleapis.com/google.rpc.LocalizedMessage', 'locale': 'en-US', 'message': 'Permission denied on resource project XXX.'}, {'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Google developers console', 'url': 'https://console.developers.google.com'}]}]}}
    Failed to detach context
    Traceback (most recent call last):
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/opentelemetry/trace/__init__.py", line 587, in use_span
        yield span
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
        yield span
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/opentelemetry/trace/__init__.py", line 452, in start_as_current_span
        yield span
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 537, in _call_llm_async
        yield llm_response
    GeneratorExit
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/opentelemetry/context/__init__.py", line 155, in detach
        _RUNTIME_CONTEXT.detach(token)
        ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
      File "/opt/homebrew/anaconda3/envs/adk-local/lib/python3.13/site-packages/opentelemetry/context/contextvars_context.py", line 53, in detach
        self._current_context.reset(token)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
    ValueError: <Token var=<ContextVar name='current_context' default={} at 0x102417970> at 0x11f7caec0> was created in a different Context
    

    As we could see, while the llm invocation failed, the tracing job has also thrown panic.

  7. removed their assignment
    on May 29, 2025
  8. stefanadelbert commented on May 30, 2025

    @stefanadelbert

    I'm getting the same on 1.1.1.

  9. antonpp commented on Jun 2, 2025

    @antonpp

    Have same issue without parallel agent just some tools and an AgentTool.

  10. petitoff commented on Jun 11, 2025

    @petitoff

    This is problem with google opentelemetry-exporter-gcp-trace that works with OpenTelemetry 1.34.x. The latest available version is still 1.9.0, which is incompatible with the newer OpenTelemetry API. I tested the temporary patch that works. You can disable that telemetry with code below in file where you have root agent.

    # Disable OpenTelemetry to avoid context management issues with incompatible GCP exporter
    os.environ["OTEL_SDK_DISABLED"] = "true"
    
    # Suppress OpenTelemetry warnings
    logging.getLogger("opentelemetry").setLevel(logging.ERROR)
    

    This is only temp fix.

  11. rcleveng commented on Jun 25, 2025

    @rcleveng

    I'm also seeing it randomly with 1.42 and had not before.

  12. rcleveng commented on Jun 26, 2025

    @rcleveng

    This is problem with google opentelemetry-exporter-gcp-trace that works with OpenTelemetry 1.34.x. The latest available version is still 1.9.0, which is incompatible with the newer OpenTelemetry API. I tested the temporary patch that works. You can disable that telemetry with code below in file where you have root agent.

    What is the issue number to track the opentelemetry-exporter-gcp-trace issue?

  13. pyralix commented on Jun 29, 2025

    @pyralix

    I'm seeing this issue in 1.5 when using ParallelAgents

  14. 6 remaining items

  15. tuanchris commented on Jul 20, 2025

    @tuanchris

    +1

  16. mykytaterentiev commented on Jul 22, 2025

    @mykytaterentiev

    +1

  17. idreamsahan commented on Jul 22, 2025

    @idreamsahan

    +1 Have the same issue will try the temp fix. thanks

  18. pallaoro commented on Jul 23, 2025

    @pallaoro

    +1

  19. aabmass commented on Jul 24, 2025

    @aabmass
    Member

    I don't think this bug is related to opentelemetry-exporter-gcp-trace. For some context on what is happening see the following issues

    I would recommend this fix python/cpython#118944 (comment) as a quick and correct workaround if the issue is an async iterator. Changing from start_as_current_span() -> start_span() is not a great fix because it breaks span parent propagation

  20. jjbi123 commented on Jul 24, 2025

    @jjbi123

    +1

  21. svelezdevilla commented on Jul 25, 2025

    @svelezdevilla

    +1

  22. rcleveng commented on Jul 25, 2025

    @rcleveng

    So the plan will be to update everywhere with:

    with contextlib.aclosing(stracer.start_as_current_span(...)):
      # existing code under trace span
    

    is that right? That sounds good, I hadn't seen aclose in contextlib before

  23. KiranRaghavendra1248 commented on Jul 28, 2025

    @KiranRaghavendra1248

    +1 - Having same issue

  24. me-her commented on Jul 29, 2025

    @me-her
    Contributor

    +1

  25. added
    tracing[Component] This issue is related to OpenTelemetry tracing
    and removed
    core[Component] This issue is related to the core interface and implementation
    on Aug 12, 2025
  26. Jacksunwei commented on Aug 12, 2025

    @Jacksunwei
    Collaborator

    We're using #1670 to track the effort to fix this. This is a non-trival one, I have to say..

  27. added
    mcp[Component] This issues is related to MCP support
    on Aug 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

mcp[Component] This issues is related to MCP supporttracing[Component] This issue is related to OpenTelemetry tracing

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions