Repository navigation
Description
Describe the bug
When running eval via the adk-web UI i recieve the following error within the eval module of adk-python.
While debugging, it's clearly shown that the error catching code assumes that inferences is not None Type and a seperate error should be raised in that case.
if len(inference_result.inferences) != len(eval_case.conversation):
raise ValueError(
'Inferences should match conversations in eval case. Found'
f'{len(inference_result.inferences)} inferences '
f'{len(eval_case.conversation)} conversations in eval cases.'
)17/run_eval HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/adk_web_server.py", line 742, in run_eval
eval_case_results = await _collect_eval_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/cli_eval.py", line 188, in _collect_eval_results
async for eval_result in agen:
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 162, in evaluate
inference_result, eval_case_result = await evaluation_task
^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
return f.result() # May raise f.exception().
^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 151, in run_evaluation
return await self._evaluate_single_inference_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 208, in _evaluate_single_inference_result
if len(inference_result.inferences) != len(eval_case.conversation):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
To Reproduce
Steps to reproduce the behavior:
- Install '...'
- Run '....'
- Open '....'
- See error
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
After defining the test case and pressing the Run Evaluation button the error happens.
Desktop (please complete the following information):
- OS: macOS Sequia 15.3.1
- Python version(python -V): Python 3.12.9
- ADK version(pip show google-adk): 1.12
Model Information:
model agnostic issue.
Additional context
Add any other context about the problem here.
Activity
ShaharKatz commentedon Aug 26, 2025
The underlying error that caused the bug is:
User-provided Session id is not supported for VertexAISessionService.This is caught in the
inference_resultin the_evaluate_single_inference_resultfunction as seen by theerror_messageattribute that it has:ShaharKatz commentedon Aug 26, 2025
I've opened an issue for the underlying issue here #2731.
It's still valuable for this issue to be addressed separately.
dorz-saymine commentedon Nov 19, 2025
What is the status of this bug?
I'm using adk 1.17.0 and getting this error without using VertexAISessionService just the default MemorySessionService
ShaharKatz commentedon Nov 19, 2025
@dorz-saymine Nearly complete, couple more days.
surajksharma07 commentedon Dec 3, 2025
@ShaharKatz This issue is still present in ADK 1.17.0.
The root cause is at
local_eval_service.py:229where the code callslen(inference_result.inferences)without checking ifinferencesis None.This happens when inference fails (e.g., session service errors, network failures) - the InferenceResult has
status=FAILUREanderror_messagepopulated, butinferencesremains None.Fix: Add None check before len() call and return NOT_EVALUATED status for failed inferences.
Defense-in-depth approach:
(1) filter failed inferences in evaluate() method,
(2) add safety check in _evaluate_single_inference_result().
Thanks!
surajksharma07 commentedon Dec 3, 2025
@ankursharmas Please have a look into it.
Handle None inferences in eval results for issue google#2729
ShaharKatz commentedon Dec 3, 2025
I see @surajksharma07, thanks!
I've created a PR that should fix this issue specifically.
I would appreciate any guidance that you and @ankursharmas can give.
6 remaining items
dorz-saymine commentedon Dec 7, 2025
@ShaharKatz thanks for creating a pr to fix the issue.
@surajksharma07 any idea when we can merge the pr?
dorz-saymine commentedon Dec 16, 2025
any update? still waiting for the fix
surajksharma07 commentedon Dec 16, 2025
@dorz-saymine Thanks for checking on this.
PR #3805 by @ShaharKatz implementing the two-layer fix (filter failed inferences in evaluate() + safety check in _evaluate_single_inference_result()) is currently under review and has not yet been merged into any release.
The PR has been thoroughly reviewed and @ShaharKatz has implemented all suggested changes (None check placement, proper logging, EvalStatus.NOT_EVALUATED return).
Team is reviewing it and working to get it merged.
It's not in v1.19.0, v1.20.0, or v1.21.0 yet.
As a temporary workaround, you can wrap your eval runs in try-except to catch the TypeError and log the failed inference case, though this won't solve the underlying issue.
The proper fix in PR #3805 will handle this gracefully by returning NOT_EVALUATED status for failed inferences.
dorz-saymine commentedon Dec 17, 2025
Ok I will try the do the try/catch and log.
This raises another question for me, why are the evaluation failing at all.
I usually run my evaluations in parallel, more then half are successful, and I restart the failed once, and it takes a few attempts for all of them to complete
surajksharma07 commentedon Dec 17, 2025
@dorz-saymine The TypeError crash from PR #3805 is still present in v1.19.0/v1.20.0/v1.21.0 (not merged yet), which masks the real failure causes.
When inferences fail (before the crash), check inference_result.error_message in your logs - parallel runs often hit API rate limits, network timeouts, or session service errors under concurrent load.
The try/catch workaround will let you see these underlying errors.
Common parallel failure patterns: rate limiting from too many concurrent LLM calls, session service timeouts, or resource exhaustion.
Consider adding delays between parallel eval batches or reducing concurrency to identify if it's a rate limit issue.
dorz-saymine commentedon Dec 21, 2025
@surajksharma07 thanks for your reply.
I have upgrade just now from v1.17.0 to v.1.21.0 and I had no inference errors at all.
surajksharma07 commentedon Dec 22, 2025
@dorz-saymine The upgrade to v1.21.0 fixed the underlying causes of your inference failures (session service stability, OAuth handling, content validation), which is why you're not encountering errors.
However, the NoneType bug in
local_eval_service.py:240(line 240:for idx, actual in enumerate(inference_result.inferences)) still exists and PR #3805 is not yet merged.Since your inferences are succeeding, the bug doesn't trigger.
If you do experience inference failures in the future (network issues, API errors), the crash could reoccur.
PR #3805 will add defensive None checks to handle this gracefully.
For now, v1.21.0 should work well for your use case.
Thanks!
ShaharKatz commentedon Jan 10, 2026
Hey @surajksharma07, I saw v1.22.0 was released 2 days ago, are we planning to merge #3805 anytime soon?
surajksharma07 commentedon Jan 10, 2026
@ShaharKatz PR #3805 is still under review and has not been merged into v1.22.0.
The NoneType bug at
local_eval_service.py:272remains unfixed in the latest release.Current status: @ankursharmas is reviewing the PR with requested changes.
I see there's a "request clarification" label - please check if there are outstanding questions from the maintainers that need your response.
Once you address the feedback, the PR can move forward for merge consideration.
Thank you for your thorough work on this fix and your patience.
We'll update this thread once PR #3805 is merged and included in a release.
ShaharKatz commentedon Jan 13, 2026
Thanks @surajksharma07 for your swift reply.
Regarding the "request clarification" label, see comment in PR #3805, I would be happy to provide any clarification needed.
Thank you so much and looking forward to any updates.
Handle None inferences in eval results for issue #2729 (#3805)
Handle None inferences in eval results for issue google#2729 (google#…