gemini_context_cache_manager.create gate uses request total tokens, not cacheable prefix tokens

Summary

GeminiContextCacheManager._create_new_cache_with_contents decides whether to call caches.create() using a token count that represents the full request (system_instruction + tools + all chat history + user turn), but the actual caches.create() call only sends the cacheable prefix (system_instruction + tools + contents[:cache_contents_count]).

This mismatch causes caches.create() to fail with 400 INVALID_ARGUMENT ("The cached content is of tokens. The minimum token count to start caching is 4096.") on every cold-prefix request whose chat history happens to push the full token count over 4096 while the prefix itself stays below.

Worse, on failure the manager returns CacheMetadata(fingerprint=..., contents_count=..., cache_name=None) (no cache_name), so the next request with the same fingerprint hits the same failure path and tries to create again. Infinite create-fail loop, zero cache hits, flood of Failed to create cache: 400 INVALID_ARGUMENT warnings.

Affected version

google-adk == 1.34.1

Environment

Model: gemini-3.5-flash on Vertex AI
ContextCacheConfig(cache_intervals=10, ttl_seconds=1800, min_tokens=4096)
App-level config (App.context_cache_config)
~24h prod observation: hundreds of failures/min, cache_hit_pct ~= 0% across all routed models

Reproduction (code path walkthrough)

google/adk/flows/llm_flows/context_cache_processor.py:81
```
llm_request.cacheable_contents_token_count = previous_token_count
```
previous_token_count is the prior response's usage_metadata.prompt_token_count — i.e. the full prompt the model just saw (incl. chat history + tools + system_instruction + the latest user turn).

google/adk/models/gemini_context_cache_manager.py:312-324 (gate inside _create_new_cache_with_contents):

if cacheable_contents_token_count < ctx_cache_config.min_tokens:
    logger.info("Previous request too small to cache: ...")
    return None
if cacheable_contents_token_count < _GEMINI_MIN_CACHE_TOKENS:
    logger.info("Token count below Gemini minimum: ...")
    return None

This uses the full prompt count from step 1 as the gate.

google/adk/models/gemini_context_cache_manager.py:372-441 (_create_gemini_cache):

cache_contents = llm_request.contents[:cache_contents_count]   # line 388
...
cache_config = CreateCachedContentConfig(
    system_instruction=llm_request.config.system_instruction,
    tools=llm_request.config.tools,
    contents=cache_contents,
    ...
)
await self.genai_client.aio.caches.create(model=..., config=cache_config)   # line 423

The actual caches.create() call only sends the prefix slice. The server measures that prefix, finds it < 4096, and rejects with 400.

On failure (gemini_context_cache_manager.py:~133), the manager returns:
```
CacheMetadata(fingerprint=..., contents_count=cache_contents_count)
```
with no cache_name. The next request matches the same fingerprint, sees no cache_name, and goes back through _create_new_cache_with_contents — same gate, same failure.

Observed log spam (production)

WARNING Failed to create cache: 400 INVALID_ARGUMENT.
{'error': {'code': 400, 'message': 'The cached content is of 1820 tokens. The minimum token count to start caching is 4096.', 'status': 'INVALID_ARGUMENT'}}

Token counts in real failures: 1820, 1857, 1991, 1829 — all well below 4096 because the prefix (system_instruction + tools + few-content sub-agents) is small even when the full chat history pushes the gate's measurement above 4096.

Suggested fix

The gate must be measured on the same payload caches.create() will receive. Two options:

Estimate prefix tokens at gate time. Compute system_instruction + tools + contents[:cache_contents_count] token count locally and compare against min_tokens / _GEMINI_MIN_CACHE_TOKENS. This avoids the extra count_tokens API round trip if you can use the on-device tokenizer; otherwise one client.models.count_tokens call is cheap relative to letting the request fail at the server.
Cache the negative result by fingerprint. Even with the gate fix, transient API rejections still happen; refusing to retry the same fingerprint for ttl_seconds would prevent the infinite-loop log flood.

Either fix on its own breaks the loop; both together would be ideal.

Workaround for users

Set App.context_cache_config = None to disable caching entirely until a fix lands. There is no per-agent disable in 1.34.1 — App.context_cache_config is global (llm_agent.py:266), and the only safe workaround is full disable.

For anyone bisecting: this is not a 1.34.1 regression. ADK 1.34.0 has the identical bug.

google/adk/models/gemini_context_cache_manager.py and google/adk/flows/llm_flows/context_cache_processor.py are byte-identical between 1.34.0 and 1.34.1:

$ diff 1.34.0/google/adk/models/gemini_context_cache_manager.py \
       1.34.1/google/adk/models/gemini_context_cache_manager.py
(empty)

$ diff 1.34.0/google/adk/flows/llm_flows/context_cache_processor.py \
       1.34.1/google/adk/flows/llm_flows/context_cache_processor.py
(empty)

Both versions ship the _GEMINI_MIN_CACHE_TOKENS = 4096 constant, the cacheable_contents_token_count-based gate, and the contents[:cache_contents_count] prefix slice at caches.create(). The mismatch existed at least since 1.34.0.

Skip the bisect — it's load-bearing in both, fix needed on main.

@rannn505

@rannn505 the gate in _create_new_cache_with_contents uses cacheable_contents_token_count (full prompt tokens) but caches.create() only receives the prefix slice, so the two counts never match on long conversations.

One approach worth trying: subclass GeminiContextCacheManager and override _create_new_cache_with_contents to estimate tokens on contents[:cache_contents_count] + system_instruction + tools (mirroring the existing _estimate_request_tokens pattern at line 339) and gate on that prefix count instead.

Could you give this a try and confirm whether it stops the 400s on your end?

In the meantime, setting App.context_cache_config = None disables caching globally as a quick interim workaround while you test!

@surajksharma07

@surajksharma07 thanks for the suggested patch. We tried it in production and have feedback.

What we did

Implemented a FixedGeminiContextCacheManager subclass that overrides _create_new_cache_with_contents to estimate tokens on the prefix-only request (mirroring _estimate_request_tokens at line 339). Conservative gate: only attempt cache when estimated prefix tokens >= max(_GEMINI_MIN_CACHE_TOKENS * 1.5, cache_config.min_tokens) to give headroom against the char/4 heuristic vs Vertex's real tokenization.

Result on the 400 errors

Patch successfully eliminated the 400 INVALID_ARGUMENT spam. The pre-check correctly skips requests that wouldn't make valid caches. ✅

Result on production latency

Had to revert within hours due to latency regression:

Request avg latency: ~2x slowdown
p95 hit our 600s task-queue timeout
Queue stalled (30 processing slots full, growing backlog of DEAD tasks)

Suspected root causes (latency, not the original 400 bug)

Every cacheable request makes a caches.create() API roundtrip when the prefix doesn't match an existing cache — adds ~150ms+ per request even when the gate decides "attempt"
Cache cleanup race: many 404 NOT_FOUND warnings when ADK tries to delete caches that already expired. Suggests cleanup runs synchronously in the request path
OTel side effect: Calling end() on an ended span warnings spiked while caching was active, dropped to 0 after revert. Likely double-span-end from sync cleanup hitting cancelled async context (related: OpenTelemetry contextvars leak across asynchronous generator boundaries in BaseAgent and Runner #5722)

Suggested fix directions

The fundamental issue isn't the gate — it's that cache create + cleanup happen synchronously in the request path. Three directions:

Memoize cache creation per prefix fingerprint. Reuse cached prefixes within TTL instead of calling caches.create() again for near-identical prefixes seconds apart
Make cache creation non-blocking. First request returns without waiting for caches.create(); subsequent requests benefit once the cache is ready
Add a config flag ContextCacheConfig.create_cache: bool = True. Lets operators disable cache creation while keeping the rest of the caching infrastructure (e.g., to use pre-created caches without auto-create overhead)

Happy to test any candidate patch — we have full observability (Cloud Trace + OTel) ready for fast feedback.

— @rannn505 (production deployment of ADK 1.34.1 across multiple internal services)

Also filed a related feature request for the trace pollution side: #5889

@rannn505

@rannn505 The gate fix confirming the 400s are gone validates the root cause, and the latency regression is a separate, deeper problem worth addressing properly.

The latency is coming from two places that both sit on the synchronous request path in handle_context_caching:

Cleanup round-trip (line 103 in gemini_context_cache_manager.py):

await self.cleanup_cache(old_cache_metadata.cache_name)

Every time a cache is found to be invalid (expired), this fires await self.genai_client.aio.caches.delete(...) synchronously before anything else proceeds.
When Vertex has already expired the cache server-side (TTL elapsed), the delete comes back 404 — but you've still paid the full API round-trip. Changing this line to asyncio.create_task(self.cleanup_cache(...)) fires the delete in the background and removes it from the hot path entirely.
The cleanup_cache method at line 455 already swallows all exceptions, so this is safe to background.

caches.create() latency (~150ms, line 423 in _create_gemini_cache):

cached_content = await self.genai_client.aio.caches.create(model=..., config=cache_config)

This is awaited inside _create_gemini_cache, which is itself awaited from handle_context_caching at line 116.
Every cache-creation attempt blocks the full request path for the duration of the Vertex API call.

For Direction 2 (non-blocking create): maintain a class-level _pending_creates: dict[str, asyncio.Task[Optional[CacheMetadata]]] keyed by fingerprint.
When a cold fingerprint is seen, fire asyncio.create_task(_create_gemini_cache(...)), store the task, and return a fingerprint-only CacheMetadata for this request (cache miss, no blocking).
On the next request with the same fingerprint, check task.done() — if done, retrieve the result; if still in flight, proceed as a cache miss again.
This removes caches.create() from the critical path completely.
The asyncio.create_task() also gives the task a clean copy of the current context, which reduces the surface for the OTel double-end() you're seeing from the #5722 contextvars leak.

For Direction 1 (dedup concurrent creates): if multiple requests hit the same cold fingerprint simultaneously (likely under your 30-slot load), each currently fires its own caches.create() and creates redundant caches.
An asyncio.Event per fingerprint would let the first caller create while others await the event result — preventing the N-parallel-create storm.
This complements Direction 2 rather than replacing it.

For Direction 3 (create_cache: bool = True flag): simplest to land. A one-field addition to ContextCacheConfig and a two-line guard at the top of _create_new_cache_with_contents (line 292):

if not llm_request.cache_config.create_cache:
    return None

This lets you validate the cache-lookup path with pre-created caches before re-enabling auto-create, and gives operators a safe escape hatch independent of the architectural fixes above.

Suggested path: Direction 3 + the create_task cleanup change are the safest, smallest PRs and would give you immediate relief without touching the create flow.
Direction 2 is the right fix for the 150ms issue — worth a PR once the fingerprint lifecycle and task cleanup (expired task entries, TTL-aligned eviction) is worked out.

Happy to look at a draft for any of these; your Cloud Trace setup should make the before/after comparison straightforward!

fix(models): gate Gemini cache creation on cacheable prefix tokens

Navigation Menu

Repository navigation

gemini_context_cache_manager.create gate uses request total tokens, not cacheable prefix tokens #5847

Summary

Affected version

Environment

Reproduction (code path walkthrough)

Observed log spam (production)

Suggested fix

Workaround for users

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

gemini_context_cache_manager.create gate uses request total tokens, not cacheable prefix tokens #5847

Description

Summary

Affected version

Environment

Reproduction (code path walkthrough)

Observed log spam (production)

Suggested fix

Workaround for users

Activity

rannn505 commented on May 25, 2026

surajksharma07 commented on May 26, 2026

rannn505 commented on May 29, 2026

rannn505 commented on May 29, 2026

surajksharma07 commented on Jun 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions