Repository navigation
feat(usage): see token burn and cost by T3 build and Labs setting - #69
Closed
ScriptedAlchemy wants to merge 8 commits into
Closed
ScriptedAlchemy wants to merge 8 commits into
ScriptedAlchemy wants to merge 8 commits into
Conversation
Pure join of priced usage records to run attempts by session, and a thread-resampled bootstrap that only returns a ratio when the data supports one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ettings One row per attempt, written before the running commit so a retry cannot lose it. Failure to write leaves the attempt unstamped and never stops the turn. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A server.getUsageCohortReport read joins priced transcript records to run attempts and compares experiment arms with a thread-resampled bootstrap. Behind diagnostics:read and a usageCohortReports capability. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An on-demand section for one environment, hidden when the server lacks the capability, plus a Labs row that opens Usage and a short user guide section. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A nominal 99% bootstrap flagged 2.75% of random splits of real threads, because spend is heavy-tailed. A report holds up to 14 verdicts, so about a third of reports would have shown a false Changed. A 99.8% interval over 4,000 resamples flags 0.57% and still catches 97% of a 30% prompt size cut. Also keep a stamping defect from failing a turn, count a run on two models once in arm totals, and share the report's 92 day limit with clients. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build selector no longer shows a raw value, the self-selection note appears once instead of on every card, a wait longer than the report's window says so, and the Labs row and guide say where the section lives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A web client rejected the whole report when a newer server retuned its interval, because the confidence was a strict literal on the wire. Unknown comparisons and rows now drop on their own instead of failing the report, and the failure state gets a Retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A comparison counted every thread toward its 30 thread minimum, but only threads on a provider, model, and speed both sides ran shape the estimate. Thirty threads per side with one in common returned "Changed, 2x, [2, 2]". A stratum now counts for as many threads as its smaller side, for both the minimum and the weight, so one thread cannot speak for a model the other side ran heavily. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
There was no way to tell whether a merged PR or a Labs setting changed token burn or cost. Usage has no split by build or setting, and runs record neither.
What changed
diagnostics:read, and its response tolerates client and server version differences.How far to trust a verdict
Measured on about 375 real threads per side over 14 days, split at random, with a known cut applied to one side.
Real comparisons are not random, and the report says so above the cards. When arm assignment followed each thread's main model, as a default model switch around a build would, false "Changed" verdicts on prompt tokens reached 4.7% (14 of 300). Dollars stayed at 0 of 300 at that setting. Removing secondary model use took it to 0 of 300, so these are real differences between threads that dip into a model and threads that live on it, not a counting error.
Merge order
Land this before turning on #64, so a baseline exists. Stamps start when a build with this code runs, and nothing is backfilled. A setting's on and off comparison starts once a build that knows that setting runs, so #64 needs its own row in
usageExperiments.ts.The on and off card will not confirm #64's savings. A reset applies to about 75 person-sent returns across 26 threads in 14 days of the data measured here. That is too few and too noisy to read on one machine, even with a holdback. #64 should record eligibility and whether the reset applied.
Left out
Verification
vp run check:changedpasses, with 36 typechecks and 327 tests in 18 files.Screenshots
None attached. This repo is public and the only data I have shows real spend, so the maintainer decides whether to post a capture. Before this change the section does not exist.
Claude Opus 5.5 in Claude Code, through T3 Code. GPT-6 Astra and Claude Fable 5.1 reviewed it independently, and GPT-6 Astra found the sparse overlap defect fixed in the last commit.
🤖 Generated with Claude Code