Skip to content

server: /train/measure, and /train reports which examples it kept, cut and skipped - #51

Merged
joelteply merged 1 commit into
masterfrom
train/measure-and-kept-indices
Oct 11, 2026
Merged

joelteply merged 1 commit into
masterfrom
train/measure-and-kept-indices

Conversation

@joelteply

Copy link
Copy Markdown

Why

On the 5090 (continuum, 2026-10-11) a run at a 16,896-token window kept 54 of Kimi's 311 lived turns and skipped 257. That was visible only as counts on GET /train. A core-side token estimate said 62, so the core can't decide what trains; only the engine's fit rule can.

What

  • One fit rule. The per-example body of prepare_examples becomes fit_example, which both /train and the new route call. Dropping the oldest whole exchange becomes train_fit_drop_oldest (exported, unit-tested in test-chat).
  • POST /train/measure {examples, window, fit} trains nothing. Per example it returns index, tokens_full, tokens_min (the smallest window it trains in: the head plus her last exchange under middle), tokens_trained, and fit: kept | truncated | skipped. Gated by --train-dir, like /train.
  • /train's state now carries kept_indices, truncated_indices and skipped_indices (request indices), so the caller clears only what the run reports it trained.

/train's behaviour is unchanged: the same drops, the same counts, the same errors. tokens_min is computed only when /train/measure asks for it, so /train pays no extra render.

Tests

  • test-chat (Windows, CUDA build): all pass, including the new train_fit_drop_oldest case. One step drops a whole exchange (user prompt, her reply and its tool result together); repeating leaves exactly the head plus her last exchange.
  • Acceptance on hardware once the core uses it: /train/measure on the same 311 at window 16,896 must report exactly the run's 54 kept and 257 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc

…t and skipped

One fit rule for training and for measuring: the per-example body of prepare_examples becomes
fit_example, which both /train and the new POST /train/measure call. Dropping the oldest whole
exchange becomes train_fit_drop_oldest, exported and unit-tested in test-chat.

POST /train/measure {examples, window, fit} trains nothing; per example it returns its index,
tokens_full, tokens_min (the smallest window it trains in: the head plus her last exchange under
"middle"), tokens_trained, and fit: kept | truncated | skipped. So a caller sends only what the
window can train.

/train's state now carries kept_indices, truncated_indices and skipped_indices, so the caller
clears only what the run reports it trained.

Why (continuum, the 5090, 2026-10-11): at a 16,896-token window the engine kept 54 of Kimi's 311
lived turns and skipped 257, reported only as counts; the core's own token estimate said 62, so
only this rule may decide.
@joelteply

Copy link
Copy Markdown
Author

Approve at a77bbbd, Fable (M5), as a comment (shared account), pending green. One fit_example decides kept / truncated / skipped for both /train and /train/measure, so the rule lives once in the engine; /train's state carries the request indices of each class, so a caller clears only what trained; the counts (n_truncated, n_skipped) are unchanged; train_fit_drop_oldest is extracted with a test that the whole exchange (prompt, reply, tool results) leaves together and the head plus the last exchange stays. One thing to plan for in the core, not this PR: measure does MORE work per example than /train's prepare (it also re-renders the fully dropped conversation for tokens_min, and the middle loop re-renders after every drop), and /train's prepare of 311 lived turns already outlasted the core's 120 s client timeout. Measuring the whole bucket in one POST will hit the same wall; the core should measure in bounded batches (or measure each example once at submission, when it arrives, and store tokens_min with it, which also gives (f) its data for free). Small nit: for fit 'left', tokens_min reports the full length although left-truncation trains in a smaller window; harmless while the core sends 'middle'.

@joelteply
joelteply merged commit af1a3c9 into master Oct 11, 2026
10 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant