[guilyx]

What and why

This PR fixes the 12 verified findings from the review of jupyddl/learn, the generators and the benchmark harness. Several of them made a metric look better than it was.

Finding	Before	After
Ranking ties	Counted as wins: a constant model scored top-1 = 1.0, and checkpoint selection preferred collapsed epochs	Identical siblings are dropped. A tie with k siblings earns 1/(k+1). A constant model now scores chance (0.379)
"Held out" top-1	Computed over the training groups	split keeps the held-out groups, and selection and reporting use them. The CLI says "held out" only when the groups really were
gripper and workshop generators	Ignored the seed, so 20 seeds gave 1 distinct problem and tuning/validation/held-out sets were duplicates	The seed varies the instance. The same seed still gives the same text
blocksworld and random-strips generators	Could produce instances already solved at the start (75 of 200 blocksworld seeds at size 2)	Re-rolled. Every other seed is byte-identical (checked on 200 seeds per size)
Malformed bundles	Loaded without complaint, and the pure and NumPy paths disagreed	Refused with a clear ValueError (shape, finiteness, keys)
set_flat	Corrupted the model before raising	Length is checked first
Negated facts	All shared one feature slot (not)	Keyed on not-<predicate>
DAgger/bootstrap split	Leaked across train/validation	Grouped on the base instance name
Benchmark configs	wastar weight 1 and weight 5 were merged into one row; the heuristic recorded was the one asked for, not the one that ran	config label includes the planner arguments; the heuristic that actually ran is recorded (api.resolve_heuristic)
summarise_transfer	Averaged over solved runs only, so a heuristic that solved just the easy instance looked 45× better	Unsolved runs are charged; mean_cost is over instances every configuration solved
train()	Silently returned the initial weights when given nothing to rank	Raises
learn_heuristic	Consumed a generator passed as sizes, leaving nothing to iterate	Size arguments are turned into lists on entry

Published research numbers

The published figures are left exactly as measured: they match rl-data.json, the Research view and the video. .docs/learned-heuristics.md and .docs/rl-for-search.md gain a dated Corrections, 2026-11-03 section, written in the style of the existing correction notes:

• Top-1, re-measured (takes seconds): blocksworld 0.923 → 0.929; logistics 0.656 → 0.667; gripper 1.000 → 1.000.
  • The gripper 1.000 was not evidence: a constant model also scored 1.000, and its held-out instances duplicated the training ones.
• Checkpoints moved: blocksworld now keeps epoch 12 instead of 20, and logistics epoch 22 instead of 9. The downstream figures came from different checkpoints and are marked "not re-measured".
• The imitation-only "366 expansions" averaged only the 9 instances it solved. Charging the failure gives 6 329.

Behaviour changes to note

• gripper/workshop: every generated instance changes. demos/workshop no longer matches generate, and the README now says so.
• Learned models: a model trained on a domain with negative preconditions sees different features. The CHANGELOG says to retrain.

How it was checked

• New tests: about 30 tests (54 cases) across test_learn.py, test_generator.py and test_benchmark.py. 44 of the new cases fail on the previous code; the rest are guards that must pass on both. They also pass with NumPy unavailable.
• My own run after rebasing onto main: 569 passed, 1 skipped. flake8 and black are clean, and web/dist is rebuilt.

Checklist

• flake8 jupyddl tests tools and pytest pass locally
• python tools/build_web.py re-run; web/dist committed
• No planner guarantee changes
• CHANGELOG.md updated

The history has a merge -s ours of the already-merged #169 tip so the push fast-forwards. Squash-merge with the PR title.

---

Generated by Claude Code