Skip to content

feat(server): continue long idle threads in a fresh provider session - #64

Draft
ScriptedAlchemy wants to merge 8 commits into
mainfrom
feat/cache-aware-context-snip
Draft

ScriptedAlchemy wants to merge 8 commits into
mainfrom
feat/cache-aware-context-snip

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Why

Cache reads are 70% of list-price spend across 30 days of real sessions, because every call re-sends the whole thread. Once a thread's prompt cache has expired, the next request is uncached anyway, so a fresh session with less history costs little extra and makes every later call cheaper.

What changed

  • A new freshSessionOnColdReturn server setting is off by default. When it is on, a person's message to a thread that is over 100k tokens and has been idle for an hour skips resume. The thread continues in a new native session through the existing context handoff.
  • Web and mobile settings get a toggle, and the portable handoffs guide gets a section.
  • The replay that priced this change and its alternatives moved to a private repo, next to the private chats it reads. This branch no longer carries it.

Scope

  • The change covers every provider, because it reuses the existing resume-fallback path.
  • Commands, agent messages, system messages, and threads with background work never trigger a reset.
  • Before an elective reset, the reconstructed handoff must fit the target model and the resumed session must still have no background work. Otherwise T3 keeps the usable existing session.
  • Idle time includes accepted root-turn activity in the current native conversation, so a long turn's old token report cannot make a recently completed turn appear idle.
  • The change does not use a decision model to choose what survives a reset. Real models are still being evaluated on that question in the same private repo, and none has earned a place yet.

Tradeoffs

  • Resetting during active work saves more on Codex (10.5% at over 150k tokens) but interrupts the work, so this change starts with the safest moment.
  • Claude's server-side clear_tool_uses lost money or roughly broke even at every setting replayed.
  • No provider has a supported way to edit one old tool result.

Blast radius

The setting is off by default, so nothing changes until someone turns it on. When it is on, the agent loses older context in that session. The agent can read omitted history back through the thread-reading tool, but nobody has measured the effect on answer quality.

Verification

  • All 80 ProviderTurnStartService.test.ts tests pass, including three added regression tests. They cover a recent completion after old telemetry, an oversized replacement handoff, and background work that appears during handoff preparation. All three failed before the fixes.
  • vp run check:changed passes on the latest commit: formatting, lint, both Knip passes, GraphQL, typecheck across 36 packages, seven test files, and React compiler checks.
  • A 30-day replay at API rates, which charges for measured re-reading, shows Claude saving 18.7% ($56 of $302) and Codex saving 5.0% ($149 of $2,969). The replay with no resets matches T3's priced cost within 0.02%, and moving it out of this repo reproduced its output byte for byte. Claude's 95% confidence interval on 52 sessions reaches just below zero.
  • This change has had no live run. Nobody has used the toggles in a real client, so there are no screenshots yet.

GPT-6 Astra Pro in ChatGPT did the follow-up review and fixes.

Claude Opus 5.5 in Claude Code, through T3 Code.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2aa714b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy force-pushed the feat/cache-aware-context-snip branch from aac7be6 to c506c63 Compare November 5, 2026 05:01
ScriptedAlchemy and others added 3 commits November 4, 2026 21:41
Cache reads dominate list-price spend, but any cut has to repay the uncached
rebuild it causes. These scripts replay local transcripts against T3's own
pricing code to price cuts, compaction, and seed selection, with no model calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A long thread re-sends its whole history on every call. Once its prompt cache
has expired, the next request is uncached anyway, so that is the cheapest moment
to drop old history.

With the new setting on, a person's message to a thread over 100k tokens, after
an hour idle, skips resume and continues through the existing context handoff.
Commands, agent messages, and threads with background work are left alone. The
setting is off by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume the existing native thread before deciding whether to start fresh, preserving external-writer locks and background work discovered during resume. Route the environment-wide setting through the correct settings search scope.
@ScriptedAlchemy
ScriptedAlchemy force-pushed the feat/cache-aware-context-snip branch from 26e1b68 to 5a95650 Compare November 5, 2026 05:41
ScriptedAlchemy and others added 2 commits November 4, 2026 22:29
…eeded

Builds a labelled set from local transcripts, split by session, then scores and
fine-tunes decision models on it and compares them with plain recency. The set
holds chat text, so it stays local. The fit wrapper works around a one-row batch
crash in mlx-decision 1.1.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy marked this pull request as draft November 5, 2026 07:15
The replay and the relevance scoring read private chats and now run next to
that data in a private repo. The cold-return setting does not depend on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ScriptedAlchemy and others added 2 commits November 5, 2026 16:49
Bring the idle-return branch up to date with the merged worktree cleanup fixes and pstack model expectations. Preserve T3_MOBILE_TEST_PLATFORM in the shared provider test environment so the Android preview regression tests run with the intended platform.

The existing idle-return changes remain intact. Validation passed 82 ProviderTurnStartService tests, 176 selected check:changed tests, formatting, lint, both knip scans, GraphQL, and React compiler checks. The aggregate typecheck launcher is blocked by the environment's seccomp supervisor.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant