Skip to content

ERROR:opentelemetry.context:Failed to detach context #1670

Closed

Description

@pawan-dream11

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Describe the bug
We have deployed our agent. and getting this issue -

Traceback (most recent call last):
File "/home/myuser/.local/lib/python3.11/site-packages/opentelemetry/context/init.py", line 155, in detach
_RUNTIME_CONTEXT.detach(token)
File "/home/myuser/.local/lib/python3.11/site-packages/opentelemetry/context/contextvars_context.py", line 53, in detach
self._current_context.reset(token)
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x3e20a81529d0> at 0x3e209c1a1340> was created in a different Context

dependencies list in requirements.txt is as follow:-

python-dotenv>=1.0.0
asyncio
google-adk==1.4.2
absl-py==2.2.1
cloudpickle==3.1.1
flake8-pyproject==1.2.3
aiohttp==3.10.11
uv==0.7.14
requests>=2.32.3

To Reproduce
Steps to reproduce the behavior:

  1. Install above dependencies
  2. Run adk web
  3. Open '....'
  4. See error

Desktop (please complete the following information):

  • Python version(python -V): 13.3
  • ADK version(pip show google-adk): 1.4.2

Model Information:
For example, which model is being used. gemini-2.5-pro

Activity

  1. added
    tracing[Component] This issue is related to OpenTelemetry tracing
    on Jun 26, 2025
  2. Apollovishwas commented on Jun 27, 2025

    @Apollovishwas

    Same Issue, Weirdly, it happens once in a while.

  3. sumitg commented on Jul 2, 2025

    @sumitg

    Same! Want to upvote this!

  4. maxbaluev commented on Jul 9, 2025

    @maxbaluev

    Same! It happened with me when i am dynamically create agents and pass them to parallel or sequential agents, because opentelemetry have different contextvar(

  5. paward35 commented on Jul 16, 2025

    @paward35

    +1

  6. svelezdevilla commented on Jul 17, 2025

    @svelezdevilla

    Same here...
    Happening every time we try using a ParallelAgent with sub_agents created at runtime.

  7. vtwoptwo commented on Jul 17, 2025

    @vtwoptwo

    what i did was just create a patch in the library that removes the error logging in the exception - i still see the logs of my traces and the code of the agents works fine

  8. PrakhyathBits commented on Jul 17, 2025

    @PrakhyathBits
  9. heinolaubscher commented on Jul 18, 2025

    @heinolaubscher

    +1

  10. self-assigned this
    on Jul 18, 2025
  11. Jacksunwei commented on Jul 21, 2025

    @Jacksunwei
    Collaborator

    I think this is likely due to with statement over python generator.

    My theory so far, is: after with statement (for otel span tracing) is out of scope and tries to restore the otel token back to original one, the current token is no long the previous one, since python generator is not strictly FILO stack.

    I'm looking into how to solve this. I'd appreciate if anyone can help with a minimal reproducible agent setup for either Sequential or Parallel Agent.

    Thanks for your all's patience!


    @jawoszek could you look into this further?

  12. 39 remaining items

  13. alexanderad commented on Sep 27, 2025

    @alexanderad

    So I guess the correct way would be something like this (same thing, just no need for try/finally, and notably we can keep break) as per https://docs.python.org/3/library/contextlib.html#contextlib.aclosing

    Significantly, aclosing() supports deterministic cleanup of async generators when they happen to exit early by break or an exception

    with contextlib.aclosing (for Python >= 3.10+):

    import contextlib
    
    async with contextlib.aclosing(runner.run_async(...)) as agen:
        async for event in agen:
            # do your thing
            # notably, you can use "break" if required, e.g.:
            if event.is_final_response():
                # ...
                break

    with a "polyfill" from google.adk (to support Python < 3.10):

    from google.adk.utils.context_utils import Aclosing
    
    async with Aclosing(runner.run_async(...)) as agen:
        async for event in agen:
            # ...
  14. weisiang9678 commented on Sep 30, 2025

    @weisiang9678

    So I guess the correct way would be something like this (same thing, just no need for try/finally, and notably we can keep break) as per https://docs.python.org/3/library/contextlib.html#contextlib.aclosing

    Significantly, aclosing() supports deterministic cleanup of async generators when they happen to exit early by break or an exception

    with contextlib.aclosing (for Python >= 3.10+):

    import contextlib

    async with contextlib.aclosing(runner.run_async(...)) as agen:
    async for event in agen:
    # do your thing
    # notably, you can use "break" if required, e.g.:
    if event.is_final_response():
    # ...
    break
    with a "polyfill" from google.adk (to support Python < 3.10):

    from google.adk.utils.context_utils import Aclosing

    async with Aclosing(runner.run_async(...)) as agen:
    async for event in agen:
    # ...

    thanks for the idea... this solved the issue

  15. jawoszek commented on Oct 6, 2025

    @jawoszek
    Collaborator

    Right, all three options should be equivalent apart from requiring different Python/ADK versions:

    # requires Python >= 3.10
    import contextlib
    
    async with contextlib.aclosing(runner.run_async(...)) as agen:
        async for event in agen:
            # do your thing
            break
    # requires ADK >= 1.11.0
    from google.adk.utils.context_utils import Aclosing
    
    async with Aclosing(runner.run_async(...)) as agen:
        async for event in agen:
            # do your thing
            break
    # works in all Python versions, but is the most verbose
    agen = runner.run_async(...)
    try: 
        async for event in agen:
            # do your thing
            break
    finally: 
        await agen.aclose() 
  16. schlaepf commented on Oct 9, 2025

    @schlaepf
    Contributor

    I also ran into this issue and tried your proposed fixes @jawoszek (thanks btw!). But I'm running ADK behind a FastAPI endpoint which returns a StreamingResponse (i.e. events emitted by ADK are streamed to the caller of the endpoint). If I implement one of @jawoszek's approaches nothing gets returned. I suspect it's b.c. the stream gets closed too early. Does anyone else have this issue?

  17. jawoszek commented on Oct 9, 2025

    @jawoszek
    Collaborator

    @schlaepf could you share a snippet of the endpoint code where the break is?

  18. schlaepf commented on Oct 10, 2025

    @schlaepf
    Contributor

    I don't have a break statement anywhere in my code, but this snippet should illustrate the problem.

        from fastapi.responses import StreamingResponse
        try:
            event_stream = runner.run_async(
                ...
                run_config=RunConfig(streaming_mode=StreamingMode.SSE),
            )
        finally:
            await event_stream.aclose()
    
        return StreamingResponse(event_stream, media_type="text/event-stream")

    The error happens when I remove the finally block. The error also shows up when I remove the run_config=RunConfig(streaming_mode=StreamingMode.SSE) line.
    On the other hand, if I keep the finally block in the code the endpoint doesn't return any events (as described previously).

  19. jawoszek commented on Oct 18, 2025

    @jawoszek
    Collaborator

    @schlaepf apologies for the delay.

    Without explicit break I'm not able to easily reproduce. Below works as expected, no detach errors:

    Working code without break
    import os
    from typing import AsyncGenerator
    
    import agent
    import fastapi
    from fastapi.responses import StreamingResponse
    from google.adk.agents.run_config import RunConfig
    from google.adk.agents.run_config import StreamingMode
    from google.adk.runners import InMemoryRunner
    from google.adk.sessions import Session
    from google.genai import types
    from opentelemetry import trace
    from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
    from opentelemetry.sdk.trace import export
    from opentelemetry.sdk.trace import TracerProvider
    
    app = fastapi.FastAPI()
    
    app_name = 'my_app'
    user_id_1 = 'user1'
    
    runner = InMemoryRunner(
        agent=agent.root_agent,
        app_name=app_name,
    )
    
    
    async def response_generator(
        content: types.Content, session: Session
    ) -> AsyncGenerator[str, None]:
      event_stream = runner.run_async(
          user_id=user_id_1,
          session_id=session.id,
          new_message=content,
          run_config=RunConfig(streaming_mode=StreamingMode.SSE),
      )
      async for event in event_stream:
        yield str(event)
    
    
    @app.post('/')
    async def request_handler(query: str) -> StreamingResponse:
    
      content = types.Content(role='user', parts=[types.Part.from_text(text=query)])
      session = await runner.session_service.create_session(
          app_name=app_name, user_id=user_id_1
      )
    
      return StreamingResponse(
          response_generator(content, session), media_type='text/event-stream'
      )
    
    
    if __name__ == '__main__':
      import uvicorn
    
      provider = TracerProvider()
      project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')
      if not project_id:
        raise ValueError('GOOGLE_CLOUD_PROJECT environment variable is not set.')
      print('Tracing to project', project_id)
      processor = export.BatchSpanProcessor(
          CloudTraceSpanExporter(project_id=project_id)
      )
      provider.add_span_processor(processor)
      trace.set_tracer_provider(provider)
    
      uvicorn.run(app, host='0.0.0.0', port=8000, log_level='debug')

    I was able to reproduce the detach error if I add a break that triggers the detach error:

    Code with break that triggers detach error
    import os
    from typing import AsyncGenerator
    
    import agent
    import fastapi
    from fastapi.responses import StreamingResponse
    from google.adk.agents.run_config import RunConfig
    from google.adk.agents.run_config import StreamingMode
    from google.adk.runners import InMemoryRunner
    from google.adk.sessions import Session
    from google.genai import types
    from opentelemetry import trace
    from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
    from opentelemetry.sdk.trace import export
    from opentelemetry.sdk.trace import TracerProvider
    
    app = fastapi.FastAPI()
    
    app_name = 'my_app'
    user_id_1 = 'user1'
    
    runner = InMemoryRunner(
        agent=agent.root_agent,
        app_name=app_name,
    )
    
    
    async def response_generator(
        content: types.Content, session: Session
    ) -> AsyncGenerator[str, None]:
      event_stream = runner.run_async(
          user_id=user_id_1,
          session_id=session.id,
          new_message=content,
          run_config=RunConfig(streaming_mode=StreamingMode.SSE),
      )
      async for event in event_stream:
        yield str(event)
        break # triggers detach error
    
    
    @app.post('/')
    async def request_handler(query: str) -> StreamingResponse:
    
      content = types.Content(role='user', parts=[types.Part.from_text(text=query)])
      session = await runner.session_service.create_session(
          app_name=app_name, user_id=user_id_1
      )
    
      return StreamingResponse(
          response_generator(content, session), media_type='text/event-stream'
      )
    
    
    if __name__ == '__main__':
      import uvicorn
    
      provider = TracerProvider()
      project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')
      if not project_id:
        raise ValueError('GOOGLE_CLOUD_PROJECT environment variable is not set.')
      print('Tracing to project', project_id)
      processor = export.BatchSpanProcessor(
          CloudTraceSpanExporter(project_id=project_id)
      )
      provider.add_span_processor(processor)
      trace.set_tracer_provider(provider)
    
      uvicorn.run(app, host='0.0.0.0', port=8000, log_level='debug')

    Adding try-finally-aclose or aclosing in the generator allows to fix the detach errors:

    Code with break and aclose - there are no detach errors
    import contextlib
    import os
    
    import agent
    import fastapi
    from fastapi.responses import StreamingResponse
    from google.adk.agents.run_config import RunConfig
    from google.adk.agents.run_config import StreamingMode
    from google.adk.runners import InMemoryRunner
    from google.genai import types
    from opentelemetry import trace
    from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
    from opentelemetry.sdk.trace import export
    from opentelemetry.sdk.trace import TracerProvider
    
    app = fastapi.FastAPI()
    
    app_name = 'my_app'
    user_id_1 = 'user1'
    
    runner = InMemoryRunner(
        agent=agent.root_agent,
        app_name=app_name,
    )
    
    
    async def response_generator(content, session):
      event_stream = runner.run_async(
          user_id=user_id_1,
          session_id=session.id,
          new_message=content,
          run_config=RunConfig(streaming_mode=StreamingMode.SSE),
      )
      try:
        async for event in event_stream:
          yield str(event)
          break # triggers detach error
      finally:
        await event_stream.aclose()
    
    
    @app.post('/')
    async def request_handler(query: str):
    
      content = types.Content(role='user', parts=[types.Part.from_text(text=query)])
      session = await runner.session_service.create_session(
          app_name=app_name, user_id=user_id_1
      )
    
      return StreamingResponse(
          response_generator(content, session), media_type='text/event-stream'
      )
    
    if __name__ == '__main__':
      import uvicorn
    
      provider = TracerProvider()
      project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')
      if not project_id:
        raise ValueError('GOOGLE_CLOUD_PROJECT environment variable is not set.')
      print('Tracing to project', project_id)
      processor = export.BatchSpanProcessor(
          CloudTraceSpanExporter(project_id=project_id)
      )
      provider.add_span_processor(processor)
      trace.set_tracer_provider(provider)
    
      uvicorn.run(app, host='0.0.0.0', port=8000, log_level='debug')
    Code with break and aclosing - there are no detach errors
    import contextlib
    import os
    
    import agent
    import fastapi
    from fastapi.responses import StreamingResponse
    from google.adk.agents.run_config import RunConfig
    from google.adk.agents.run_config import StreamingMode
    from google.adk.runners import InMemoryRunner
    from google.genai import types
    from opentelemetry import trace
    from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
    from opentelemetry.sdk.trace import export
    from opentelemetry.sdk.trace import TracerProvider
    
    app = fastapi.FastAPI()
    
    app_name = 'my_app'
    user_id_1 = 'user1'
    
    runner = InMemoryRunner(
        agent=agent.root_agent,
        app_name=app_name,
    )
    
    
    async def response_generator(content, session):
      async with contextlib.aclosing(
          runner.run_async(
              user_id=user_id_1,
              session_id=session.id,
              new_message=content,
              run_config=RunConfig(streaming_mode=StreamingMode.SSE),
          )
      ) as agen:
        async for event in agen:
          yield str(event)
          break # triggers detach error
    
    
    @app.post('/')
    async def request_handler(query: str):
    
      content = types.Content(role='user', parts=[types.Part.from_text(text=query)])
      session = await runner.session_service.create_session(
          app_name=app_name, user_id=user_id_1
      )
    
      return StreamingResponse(
          response_generator(content, session), media_type='text/event-stream'
      )
    
    
    if __name__ == '__main__':
      import uvicorn
    
      provider = TracerProvider()
      project_id = os.environ.get('GOOGLE_CLOUD_PROJECT')
      if not project_id:
        raise ValueError('GOOGLE_CLOUD_PROJECT environment variable is not set.')
      print('Tracing to project', project_id)
      processor = export.BatchSpanProcessor(
          CloudTraceSpanExporter(project_id=project_id)
      )
      provider.add_span_processor(processor)
      trace.set_tracer_provider(provider)
    
      uvicorn.run(app, host='0.0.0.0', port=8000, log_level='debug')

    As you've suspected - in your example the event_stream is closed too early (finally is triggered before returning StreamingResponse). The solution would be to have aclosing/aclose in a separate generator, as in my examples above.

    Note - I'm thinking that in theory if we pass output of runner.run_async directly to StreamingResponse we shouldn't need any additional closing, as we don't introduce any additional async generators. But in my testing - raw output of runner.run_async doesn't work with StreamingResponse so we need to transform it with additional generator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

tracing[Component] This issue is related to OpenTelemetry tracing

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions