[ScriptedAlchemy]

Thread split suggestions almost never fire usefully. On a held-out test of Claude and Codex threads, the prompt and threshold shipped in #55 caught 1 of 33 real new tasks.

This PR switches the shared prompt to the contextual prompt, lowers the threshold from 0.94 to 0.6538, and skips the model request for drafts under five words. The feature stays passive and opt-in, and the eval keeps the old prompt as baseline for comparison.

421 held-out Claude and Codex messages	AUC	Real new tasks caught	False suggestions	Precision among known labels
main: baseline, 0.94	0.749	1/33	3/375	25.0% (1/4)
This PR: contextual, 0.6538, five words	0.886	8/33	6/375	57.1% (8/14)

The table calculates precision from the reported true and false suggestions and excludes uncertain labels, as the evaluator does. The earlier 12.5% and 44.4% figures used a different denominator, and reconciling them needs the private score receipts. These corrections are arithmetic and do not come from a new model evaluation.

Both offline runners now support --min-message-words 5 to reproduce the shipped five-word minimum. The experiment records that choice in its scoring receipts and frozen calibration selection, and it keeps the original baseline's eligibility.

• Recall rose by 8.6 to 36.7 points, on a 95% interval that resamples whole threads.
• False suggestions are at 1.6%, above the 1% calibration target. The increase over main is within noise, at -0.5 to +2.3 points.
• An earlier test on 927 mostly Cursor messages used the same prompt and threshold without the word minimum. It had 20.0% weighted recall and 1.09% false suggestions.

The experiment froze the threshold on calibration data and chose the word minimum before it read the test split.

Limits

• The data is one user's history. The test excludes threads with credential-shaped text, which skews the sample toward shorter threads.
• The prompt misses most new tasks, so it suits a hint and not automatic splitting.
• These numbers come from direct model scoring. Only the updated unit test covers the service's HTTP path, and no live replay covers it.

Verification

• Scoped formatting, lint, both Knip passes, GraphQL validation, the native addon build, and selected tests pass.
• The new short-message test row fails against main's service code.
• python3 -m unittest discover -s evals/thread-split passes (7 tests), including short-draft and receipt-policy regressions.
• The scoped TypeScript tests pass (4 tests), and direct server typechecking passes. The local aggregate typecheck wrapper cannot initialize its seccomp supervisor in this environment.

Still in progress

This PR stays a draft while a local improvement run finishes. It may gain a shorter history or a retrained final layer if either beats this configuration on a fresh sealed test. The run has these results so far:

• Combining small models gave no gain beyond noise.
• Separate thresholds by thread length did slightly worse.
• The last 3 turns instead of the full history gave a small recall gain on Claude and Codex, stayed within noise overall, and ran 3x faster.
• The retrained final layer is still running. mlx-decision 1.1.0 drops a saved pplx yes/no fit on load, so any trained model needs a fix or a workaround for that before it can ship.

GPT-6 Astra Pro in ChatGPT did the follow-up review and fixes.

Claude Opus 5.5 in T3 Code, through Claude Code.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: a15be6d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR