Skip to content

during run evaluation of a test case inferences could be NoneType which has no len attribute and error is not caught #2729

Closed

Description

@ShaharKatz

Describe the bug

When running eval via the adk-web UI i recieve the following error within the eval module of adk-python.
While debugging, it's clearly shown that the error catching code assumes that inferences is not None Type and a seperate error should be raised in that case.

if len(inference_result.inferences) != len(eval_case.conversation):
  raise ValueError(
      'Inferences should match conversations in eval case. Found'
      f'{len(inference_result.inferences)} inferences '
      f'{len(eval_case.conversation)} conversations in eval cases.'
  )

17/run_eval HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/adk_web_server.py", line 742, in run_eval
eval_case_results = await _collect_eval_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/cli_eval.py", line 188, in _collect_eval_results
async for eval_result in agen:
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 162, in evaluate
inference_result, eval_case_result = await evaluation_task
^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
return f.result() # May raise f.exception().
^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 151, in run_evaluation
return await self._evaluate_single_inference_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 208, in _evaluate_single_inference_result
if len(inference_result.inferences) != len(eval_case.conversation):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()

To Reproduce
Steps to reproduce the behavior:

  1. Install '...'
  2. Run '....'
  3. Open '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
After defining the test case and pressing the Run Evaluation button the error happens.

Image

Desktop (please complete the following information):

  • OS: macOS Sequia 15.3.1
  • Python version(python -V): Python 3.12.9
  • ADK version(pip show google-adk): 1.12

Model Information:
model agnostic issue.

Additional context
Add any other context about the problem here.

Activity

  1. added theissue type on Aug 26, 2025
  2. ShaharKatz commented on Aug 26, 2025

    @ShaharKatz
    ContributorAuthor

    The underlying error that caused the bug is: User-provided Session id is not supported for VertexAISessionService.
    This is caught in the inference_result in the _evaluate_single_inference_result function as seen by the error_message attribute that it has:

    inference_result.error_message
    'User-provided Session id is not supported for VertexAISessionService.'
    
    Image
  3. ShaharKatz commented on Aug 26, 2025

    @ShaharKatz
    ContributorAuthor

    I've opened an issue for the underlying issue here #2731.
    It's still valuable for this issue to be addressed separately.

  4. dorz-saymine commented on Nov 19, 2025

    @dorz-saymine

    What is the status of this bug?
    I'm using adk 1.17.0 and getting this error without using VertexAISessionService just the default MemorySessionService

  5. ShaharKatz commented on Nov 19, 2025

    @ShaharKatz
    ContributorAuthor

    @dorz-saymine Nearly complete, couple more days.

  6. surajksharma07 commented on Dec 3, 2025

    @surajksharma07
    Collaborator

    @ShaharKatz This issue is still present in ADK 1.17.0.

    The root cause is at local_eval_service.py:229 where the code calls len(inference_result.inferences) without checking if inferences is None.

    This happens when inference fails (e.g., session service errors, network failures) - the InferenceResult has status=FAILURE and error_message populated, but inferences remains None.

    Fix: Add None check before len() call and return NOT_EVALUATED status for failed inferences.

    Defense-in-depth approach:
    (1) filter failed inferences in evaluate() method,
    (2) add safety check in _evaluate_single_inference_result().

    Thanks!

  7. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Dec 3, 2025
  8. surajksharma07 commented on Dec 3, 2025

    @surajksharma07
    Collaborator

    @ankursharmas Please have a look into it.

  9. ShaharKatz commented on Dec 3, 2025

    @ShaharKatz
    ContributorAuthor

    I see @surajksharma07, thanks!

    I've created a PR that should fix this issue specifically.

    I would appreciate any guidance that you and @ankursharmas can give.

  10. 6 remaining items

  11. removed
    need attention[Status] This issue is important but the maintainers haven't started working on it
    on Dec 5, 2025
  12. dorz-saymine commented on Dec 7, 2025

    @dorz-saymine

    @ShaharKatz thanks for creating a pr to fix the issue.
    @surajksharma07 any idea when we can merge the pr?

  13. dorz-saymine commented on Dec 16, 2025

    @dorz-saymine

    any update? still waiting for the fix

  14. surajksharma07 commented on Dec 16, 2025

    @surajksharma07
    Collaborator

    @dorz-saymine Thanks for checking on this.

    PR #3805 by @ShaharKatz implementing the two-layer fix (filter failed inferences in evaluate() + safety check in _evaluate_single_inference_result()) is currently under review and has not yet been merged into any release.

    The PR has been thoroughly reviewed and @ShaharKatz has implemented all suggested changes (None check placement, proper logging, EvalStatus.NOT_EVALUATED return).
    Team is reviewing it and working to get it merged.
    It's not in v1.19.0, v1.20.0, or v1.21.0 yet.

    As a temporary workaround, you can wrap your eval runs in try-except to catch the TypeError and log the failed inference case, though this won't solve the underlying issue.
    The proper fix in PR #3805 will handle this gracefully by returning NOT_EVALUATED status for failed inferences.

  15. dorz-saymine commented on Dec 17, 2025

    @dorz-saymine

    @dorz-saymine Thanks for checking on this.

    PR #3805 by @ShaharKatz implementing the two-layer fix (filter failed inferences in evaluate() + safety check in _evaluate_single_inference_result()) is currently under review and has not yet been merged into any release.

    The PR has been thoroughly reviewed and @ShaharKatz has implemented all suggested changes (None check placement, proper logging, EvalStatus.NOT_EVALUATED return). Team is reviewing it and working to get it merged. It's not in v1.19.0, v1.20.0, or v1.21.0 yet.

    As a temporary workaround, you can wrap your eval runs in try-except to catch the TypeError and log the failed inference case, though this won't solve the underlying issue. The proper fix in PR #3805 will handle this gracefully by returning NOT_EVALUATED status for failed inferences.

    Ok I will try the do the try/catch and log.
    This raises another question for me, why are the evaluation failing at all.
    I usually run my evaluations in parallel, more then half are successful, and I restart the failed once, and it takes a few attempts for all of them to complete

  16. surajksharma07 commented on Dec 17, 2025

    @surajksharma07
    Collaborator

    @dorz-saymine The TypeError crash from PR #3805 is still present in v1.19.0/v1.20.0/v1.21.0 (not merged yet), which masks the real failure causes.

    When inferences fail (before the crash), check inference_result.error_message in your logs - parallel runs often hit API rate limits, network timeouts, or session service errors under concurrent load.

    The try/catch workaround will let you see these underlying errors.
    Common parallel failure patterns: rate limiting from too many concurrent LLM calls, session service timeouts, or resource exhaustion.

    Consider adding delays between parallel eval batches or reducing concurrency to identify if it's a rate limit issue.

  17. removed
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Dec 19, 2025
  18. dorz-saymine commented on Dec 21, 2025

    @dorz-saymine

    @surajksharma07 thanks for your reply.
    I have upgrade just now from v1.17.0 to v.1.21.0 and I had no inference errors at all.

  19. surajksharma07 commented on Dec 22, 2025

    @surajksharma07
    Collaborator

    @dorz-saymine The upgrade to v1.21.0 fixed the underlying causes of your inference failures (session service stability, OAuth handling, content validation), which is why you're not encountering errors.

    However, the NoneType bug in local_eval_service.py:240 (line 240: for idx, actual in enumerate(inference_result.inferences)) still exists and PR #3805 is not yet merged.

    Since your inferences are succeeding, the bug doesn't trigger.

    If you do experience inference failures in the future (network issues, API errors), the crash could reoccur.

    PR #3805 will add defensive None checks to handle this gracefully.

    For now, v1.21.0 should work well for your use case.

    Thanks!

  20. ShaharKatz commented on Jan 10, 2026

    @ShaharKatz
    ContributorAuthor

    Hey @surajksharma07, I saw v1.22.0 was released 2 days ago, are we planning to merge #3805 anytime soon?

  21. surajksharma07 commented on Jan 10, 2026

    @surajksharma07
    Collaborator

    @ShaharKatz PR #3805 is still under review and has not been merged into v1.22.0.
    The NoneType bug at local_eval_service.py:272 remains unfixed in the latest release.

    Current status: @ankursharmas is reviewing the PR with requested changes.
    I see there's a "request clarification" label - please check if there are outstanding questions from the maintainers that need your response.
    Once you address the feedback, the PR can move forward for merge consideration.

    Thank you for your thorough work on this fix and your patience.
    We'll update this thread once PR #3805 is merged and included in a release.

  22. ShaharKatz commented on Jan 13, 2026

    @ShaharKatz
    ContributorAuthor

    Thanks @surajksharma07 for your swift reply.
    Regarding the "request clarification" label, see comment in PR #3805, I would be happy to provide any clarification needed.
    Thank you so much and looking forward to any updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions