[derek73]

Each phase 2j batch (#896 through #912) spent about ten minutes on a hand-written measure.sh and rewrote about ten probe scripts: shape grids, corpus incidence counts and a usage lookup. #913 lists about 22 more words, so this PR commits the tooling. It changes no parser behavior. There are three commits.

1. tools/vocab/word_report.py WORD... [--base <checkout>] [--tree <checkout>] [--rank]

• grid: the word in 25 shapes, each in three cases and under all three name orders. The shapes:

  • a head, alone, with a period, before a suffix or a credential;
  • behind a title, a joined title pair, a style or a modifier;
  • its own X of Y phrase;
  • the surname slot;
  • trailing with a period (H5);
  • after a family comma, before a trailing salutation (H8), ending the given part;
  • an empty head (C15), a nickname-only head;
  • after a full name and a comma, past a second comma (C2);
  • the article epithet.

  A shape's three case spellings fold into one block where they read alike once lowercased, and a text's orders fold into one all line where they agree. With --base, only what moved is printed, folded the same way; an order-specific move still prints for its order alone. For sheikh against a pre-Read the name Sheikh and its spellings as names outside a title phrase (#752 phase 2j, #853) #912 checkout the report is 120 lines; unfolded, it ran to about 330.

• corpora: in every fetched local corpus, the names that open with the word, and the names whose part after the first comma opens with it, counted by reading: title, given or other. In psc-individuals, the names opening with the word are also counted by the box the filer typed it into (title or forename). The report names the corpora it could not read, and a damaged corpus ends the run. Counts only; a test checks that no corpus name reaches the output.

• usage: name_usage.py --no-wikidata --json's rows, with each local corpus's credit trimmed.

• ranking, alone with --rank, the triage Should Baron, Bishop, Judge and the other title words borne as names read as names outside a title phrase? (Baron Davis → title 'Baron', no given name) #913 asks for: the title readings at both positions, beside the PSC forename-box and title-box filings and the forename box over the two (28/34). Words are ordered by the forename box's count, then by its share, then by titles; a count can't be rebuilt from a share. It needs no change to the tree.

Each tree is read in a child process of its own. The two children run at once, and each refuses a nameparser imported from outside its tree (snapshot.check_source). snapshot.read's spawn pool is now snapshot.parallel, which this child and commit 2 share, worker check included.

#913's 22 words, ranked on 2026-10-27, with dean, king, prince and sheikh (already moved) as controls:

word	opens	title	comma	title	titles	PSC forename box	PSC title box	forename/both
dean	1,337	0	1,707	1	1	164	1	164/165
sheikh	473	0	15	0	0	32	13	32/45
prince	410	13	24	0	13	28	6	28/34
king	2,097	0	53	0	0	13	0	13/13
sir	481	479	16	16	495	8	277	8/285
lady	392	386	15	15	401	5	160	5/165
queen	105	18	14	14	32	3	0	3/3
marquise	10	9	2	2	11	1	0	1/1
empress	3	3	0	0	3	1	0	1/1
baron	288	83	30	30	113	0	16	0/16
general	79	78	11	11	89	0	4	0/4
bishop	653	23	11	11	34	0	17	0/17
canon	45	19	1	1	20	0	11	0/11
colonel	9	8	11	11	19	0	5	0/5
deacon	46	6	8	8	14	0	1	0/1
lawyer	24	1	1	1	2	0	1	0/1
saint	56	47	3	3	50	0	0	-
marquis	114	12	26	26	38	0	0	-
burgess	385	10	9	9	19	0	0	-
judge	101	1	12	12	13	0	0	-
chancellor	40	5	7	7	12	0	0	-
ranger	22	9	3	3	12	0	0	-
author	1	1	3	3	4	0	0	-
majesty	0	0	1	1	1	0	0	-
sailor	12	1	0	0	1	0	0	-
pharaoh	1	0	0	0	0	0	0	-

titles counts every name that reads the word as a title, real titles included. The PSC boxes separate the two:

• Forename box: a given name the title reading takes, which a move to the title phrase gives back.
• Title box: a real title, which the move would cost.

The forename count orders the table, with the title box beside it. So sir (8/285) and lady (5/165) rank above queen (3/3) on count, while their title boxes, 277 and 160, show the cost of moving them. sheikh's 13 title-box filings are the 13 PSC agreement records #912 moved away from agreeing, so the split predicts a batch's cost before it is cut. A - means no filer typed the word in either box, and the titles column is all there is.

2. --jobs for the agreement score and the spelling views, and an alike view with a kept base

• local_agreement.py scores through snapshot.parallel (--jobs, default every CPU), and its --base child does too. Under jobs, score() refuses a non-default Parser, since a worker cannot be sent one.
• local_corpora.py's pairs --agreement and groups parse each distinct name once, in workers.
• New local_corpora.py alike SOURCE [--base <checkout>] view. It reads jrc-names persons, nepal-candidates romanized/Devanagari pairs and tse-donors filed/registry pairs. With --base it prints the "now alike / now apart" counts each 2j PR quoted, replacing the scratch pairs.py, which parsed the base again for every commit.
  • The base's flags are kept under alike/, keyed by the fetch, the base package's digest, the interpreter and alike_fingerprint.
  • The fingerprint's hand-kept list has the same guard _FOLD_CODE has.
  • Flags are read into a scratch folder of the run's own and renamed into place only after the digest, taken again, is unchanged.
  • Keys are entity ids or record indexes, never names.
• plant_corpus now carries the real source's corpus_keys, so a nepal corpus planted for a --base child passes the child's key check.

Measured 2026-10-27 on 12 CPUs, load average 3 to 15 from other sessions. Each output was byte-identical before and after (the report and the record's entries):

run	before	after
local_agreement.py psc-individuals --record	88.9s	15.4s
groups jrc-names	55.6s	11.8s
pairs tse-donors --agreement (--jobs 1 vs default)	52.1s	15.8s

3. tools/differential/measure.py --base <checkout>

The steps, run one at a time, each but cldr_agreement.py parallel inside itself:

1. the committed snapshot (snapshot.py, the check test_snapshot.py makes);
2. cldr_agreement.py;
3. snapshot.py --local --base for every fetched corpus;
4. local_agreement.py psc-individuals --base;
5. alike --base for each source that pairs spellings.

• Only steps 3–5 read --base. Steps 1–2 compare against the records committed on the branch. The summary labels each step with what it compared against. It also names the local corpora that are not fetched, and a run where no step read --base exits 1.
• Output: each step's whole output goes to a file under --out, by default under the tools' cache, since local moves are listed by name.
• summary.txt: keeps each step's status, its time and its unindented lines: the first few, the last ones (where the verdicts are) and any "not measured" line. It never keeps the indented lines, where the tools list names.
• Crashes: a crash's traceback is indented below a line naming only its type, since its message may hold a name.
• Exit codes: a failed step fails the run; a step that finds moves does not.

On 2026-10-27 it ran 14 steps in 368s against a base copy of master, recording every base for the first time. The base differed only in nameparser/vocab/README.md, which the package digest hashes.

Review fixes (folded into the commits)

Two review rounds, the second with four reviewers (code, tests, comments, silent failures). Each fix has a test, and the first round's were mutation-checked against the old code.

• measure.py read a crashed step as "moves". Every tool refuses with SystemExit("message"), which exits 1 like a move. Each step now runs through measure's --step wrapper, which turns a message exit or an uncaught exception into exit 2.
• Names in the summary.
  • A name holding \r, U+2028 or U+0085 could have reached summary.txt. Output is now read as bytes and cut at \n only.
  • snapshot's grouped moves escape a literal line break inside a name.
  • A crash's exception message is kept off the summary.
  • No fetched corpus holds such a name today.
• Unread corpora passing for zeros.
  • --rank with corpora unfetched or damaged printed zeros. The report now says which corpora it read, and a damaged one ends the run.
  • measure.py left unfetched corpora out of the plan and still said "none failed". It now names them, and fails a run where nothing read --base.
• The summary's line cap kept the first lines and dropped the verdicts, including CLDR's "JA not measured". It now keeps the head, the tail and any "not measured" line.
• A failed name_usage.py crashed word_report with a TypeError, because its note was a list. It now prints the reason.
• word_report's base and head children ran one after the other. They are now fed from threads.
• Doc fixes: steps 1–2 are not compared against --base, alike counts pairs as well as persons, the parallel floor applies to every caller, and the --rank and --base wording.

Docs

tools/differential/README.md now covers --jobs, the alike view, the dated timings and a new "Measuring a change" section. nameparser/vocab/README.md has a word_report.py paragraph and points at measure.py in step 6. AGENTS.md's "Adding a word to the vocabulary" points at both.

Notes

• Editing snapshot.py and the scorer changes the kept-baseline and scorer keys. Every kept local baseline and kept agreement record is recorded once more on its next --base run.
• word_report.py's child reads through snapshot.parallel only above 5,000 texts. That path is tested in process, with the floor lowered, since a child's floor cannot be patched.
• alike counts flips between alike and apart only. A person whose spellings all change their reading but still agree is no move there; the local snapshot shows it.
• Not covered: the family-first half of Should Baron, Bishop, Judge and the other title words borne as names read as names outside a title phrase? (Baron Davis → title 'Baron', no given name) #913 (a title borne as a surname, Graf Steffi). --rank reads the default order only; the grid shows the family-first orders per word.

Tests: uv run pytest 31,979 passed; serial 43 passed; the touched modules pass on 3.15; ruff and mypy are clean.

🤖 Generated with Claude Code