Repository navigation
server: /train/measure, and /train reports which examples it kept, cut and skipped - #51
Conversation
…t and skipped
One fit rule for training and for measuring: the per-example body of prepare_examples becomes
fit_example, which both /train and the new POST /train/measure call. Dropping the oldest whole
exchange becomes train_fit_drop_oldest, exported and unit-tested in test-chat.
POST /train/measure {examples, window, fit} trains nothing; per example it returns its index,
tokens_full, tokens_min (the smallest window it trains in: the head plus her last exchange under
"middle"), tokens_trained, and fit: kept | truncated | skipped. So a caller sends only what the
window can train.
/train's state now carries kept_indices, truncated_indices and skipped_indices, so the caller
clears only what the run reports it trained.
Why (continuum, the 5090, 2026-10-11): at a 16,896-token window the engine kept 54 of Kimi's 311
lived turns and skipped 257, reported only as counts; the core's own token estimate said 62, so
only this rule may decide.
|
Approve at a77bbbd, Fable (M5), as a comment (shared account), pending green. One fit_example decides kept / truncated / skipped for both /train and /train/measure, so the rule lives once in the engine; /train's state carries the request indices of each class, so a caller clears only what trained; the counts (n_truncated, n_skipped) are unchanged; train_fit_drop_oldest is extracted with a test that the whole exchange (prompt, reply, tool results) leaves together and the head plus the last exchange stays. One thing to plan for in the core, not this PR: measure does MORE work per example than /train's prepare (it also re-renders the fully dropped conversation for tokens_min, and the middle loop re-renders after every drop), and /train's prepare of 311 lived turns already outlasted the core's 120 s client timeout. Measuring the whole bucket in one POST will hit the same wall; the core should measure in bounded batches (or measure each example once at submission, when it arrives, and store tokens_min with it, which also gives (f) its data for free). Small nit: for fit 'left', tokens_min reports the full length although left-truncation trains in a smaller window; harmless while the core sends 'middle'. |
Why
On the 5090 (continuum, 2026-10-11) a run at a 16,896-token window kept 54 of Kimi's 311 lived turns and skipped 257. That was visible only as counts on
GET /train. A core-side token estimate said 62, so the core can't decide what trains; only the engine's fit rule can.What
prepare_examplesbecomesfit_example, which both/trainand the new route call. Dropping the oldest whole exchange becomestrain_fit_drop_oldest(exported, unit-tested intest-chat).POST /train/measure{examples, window, fit}trains nothing. Per example it returnsindex,tokens_full,tokens_min(the smallest window it trains in: the head plus her last exchange undermiddle),tokens_trained, andfit: kept | truncated | skipped. Gated by--train-dir, like/train./train's state now carrieskept_indices,truncated_indicesandskipped_indices(request indices), so the caller clears only what the run reports it trained./train's behaviour is unchanged: the same drops, the same counts, the same errors.tokens_minis computed only when/train/measureasks for it, so/trainpays no extra render.Tests
test-chat(Windows, CUDA build): all pass, including the newtrain_fit_drop_oldestcase. One step drops a whole exchange (user prompt, her reply and its tool result together); repeating leaves exactly the head plus her last exchange./train/measureon the same 311 at window 16,896 must report exactly the run's 54 kept and 257 skipped.🤖 Generated with Claude Code
https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc