Skip to content

TypeError in LocalEvalSampler when metric evaluation fails #5403

Closed

Description

@msteiner-google

When running adk optimize, if a metric evaluation fails (e.g., due to a transient API error, rate limiting, or a JSONDecodeError from the LLM judge), the LocalEvalSampler crashes with a TypeError. This happens because the valuation logic gracefully catches the exception and returns a result with a None score, but the sampler subsequently tries to round this None value.

  Error Logs

    TypeError: type NoneType doesn't define __round__ method
   
    Traceback (most recent call last):
      ...
      File ".../google/adk/optimization/local_eval_sampler.py", line 362, in sample_and_score
        self._extract_eval_data(eval_set_id, eval_results)
      File ".../google/adk/optimization/local_eval_sampler.py", line 292, in _extract_eval_data
        "score": round(eval_metric_result.score, 2),  # accurate enough
    TypeError: type NoneType doesn't define __round__ method

Root Cause

In google/adk/evaluation/local_eval_service.py, the _evaluate_metric_for_eval_case method catches all exceptions during evaluation:

except Exception as e:
  logger.error(...)
  # We use an empty result.
  evaluation_result = EvaluationResult(
     overall_eval_status=EvalStatus.NOT_EVALUATED
  )

The EvaluationResult (and its nested PerInvocationResult) defaults its score field to None.

In google/adk/optimization/local_eval_sampler.py, the _extract_eval_data method iterates through these results and attempts to round the score
without checking if it is None:

for eval_metric_result in per_invocation_result.eval_metric_results:
  eval_metric_results.append({
      "metric_name": eval_metric_result.metric_name,
      "score": round(eval_metric_result.score, 2),  # <--- CRASH HERE
      "eval_status": eval_metric_result.eval_status.name,
})

Reproduction Steps

  1. Configure an agent for optimization using adk optimize.
  2. Include a metric that relies on an LLM judge (e.g., rubric_based_tool_use_quality_v1).
  3. Trigger a scenario where the judge evaluation fails (e.g., simulate a network error or a malformed judge response).
  4. The process will crash during data extraction instead of reporting a 0.0 score or skipping the failed case.

Proposed Fix

The sampler should handle None scores gracefully, either by defaulting them to 0.0 or skipping the rounding step for un-evaluated metrics.

# google/adk/optimization/local_eval_sampler.py
   
"score": round(eval_metric_result.score, 2) if eval_metric_result.score is not None else 0.0,

Activity

  1. added
    eval[Component] This issue is related to evaluation
    on Apr 20, 2026
  2. added theissue type on Apr 20, 2026
  3. sanketpatil06 commented on Apr 22, 2026

    @sanketpatil06

    Hi @msteiner-google , Thanks for the details. To help us get a full understanding of the issue, would you mind sharing the complete code? This will allow us to reproduce exactly as you see it. Thanks!

  4. msteiner-google commented on Apr 22, 2026

    @msteiner-google
    Author

    Hey @surajksharma07 sure thing: https://github.com/msteiner-google/adk-samples . More specifically here's my monkeypatch to make sure the adk optimize run's to completion. Here instead you find the agent. You can clone the repo, and run make optimize. If you comment out the apply_patch and the optimization loop encounters some stutter it will fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluationrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions