Repository navigation
Conversation
… a response A callback that returns a replacement LlmResponse rarely sets usage_metadata. finalize_model_response_event copies only non-None fields into the Event, so the yielded final event and the persisted session event carry no usage for that model call: the BigQuery analytics plugin's event path, the A2A converters and per-call cost attribution from session history all see nothing. usage_metadata measures the model call (tokens billed by the provider), not the content of the response, so replacing the content does not change it. Inherit it from the replaced response when the replacement leaves it unset, alongside the partial/turn_complete inheritance from 39e4538. An explicitly set value on the replacement is still respected, and the callback-owned object is not mutated. Renames the helper to _inherit_unset_model_fields to match its widened scope. Follow-up to google#7035.
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue:
usage_metadata, erasing the model call from token accounting #7451partialduring SSE streaming - every delta is persisted to the session as a final event #7035, fix(flows): inherit unset streaming fields on after_model_callback replacement #7036Problem:
Follow-up to #7035 / #7036. A callback-built replacement now inherits
partialandturn_complete, butusage_metadatawas left out of thatfix's scope and is still dropped.
finalize_model_response_eventcopiesonly non-None fields into the
Event, so when a rebuildingafter_model_callbackleavesusage_metadataunset, the yielded final eventand the persisted session event carry no usage for that model call. The
BigQuery analytics plugin's event path, the A2A converters,
agent_test_runner, and any per-call cost attribution from session historyall see nothing. Happens in streaming and non-streaming mode alike (repro in
the linked issue).
Solution:
Extend the existing inheritance in
flows/llm_flows/core/_finalizer.pytousage_metadata, with the same semantics as 39e4538: inherit only when thereplacement leaves the field unset, copy-on-inherit so the callback-owned
object is not mutated, and an explicitly set value always wins (a callback
that made an additional model call can report its own combined usage). The
helper is renamed
_inherit_unset_model_fieldsto match its widened scope;it is private and had no other references.
Scope is deliberately one field.
grounding_metadataalready has its ownhandling in the same function, and
finish_reason/error_codeare leftto the callback because a guardrail replacement may legitimately change
finish semantics. Rationale for treating
usage_metadataas model-owned:it measures the call as billed by the provider, not the content, so there is
no reading under which
Noneis the more accurate value after a contentrewrite.
Testing Plan
Unit Tests:
Added to
tests/unittests/flows/llm_flows/test_base_llm_flow_partial_handling.py,mirroring the tests from #7036:
test_after_model_callback_replacement_inherits_usage_metadata— arebuilt replacement carries the original's usage; the callback's object is
not mutated.
test_after_model_callback_replacement_explicit_usage_metadata_respected— a replacement that sets its own usage is not overridden.
test_run_async_sse_rebuilt_final_keeps_usage_metadata— flow-level SSErun with a fake model that reports usage on the final response only:
events are
partial == [True, True, None]andusage_metadata == [None, None, <usage>].Manual End-to-End (E2E) Tests:
Ran the repro script from the linked issue (fake model, no API key) against
this branch. Before:
REBUILD non-streamandREBUILD SSEboth showfinal_event.usage=None persisted.usage=None. After: all four runs showusage=128, and thepartialflags from #7036 are unchanged.Checklist
Additional context
Formatted with
isort+pyinkperpyproject.toml(no changes needed).