Add jrc-names, a local corpus of JRC-Names person spellings, with a groups view

AGENTS.md

@@ -278,7 +278,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_
 
 **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`.
 
-**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either.
+**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either.
 
 ## Gotchas
 

docs/design/mechanisms.md

@@ -414,7 +414,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')"
 
 **How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone, or a local corpus whose records carry the fields their filers typed, with its record kept in the cache beside it (`local_agreement.py`). The local score tells a change's author what moved among real filed names, by field, and gives agreement by group and how often each disagreement shape occurs. It cannot guard anything: no CI run and no reviewer sees it, and no test fails when it moves (the tool itself exits 1). Its truth is the filer's, so a box filled wrongly (a job typed as a title) reads as a miss unless the scorer sets it aside, and the scorer has to say what it sets aside. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`).
 
-A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775).
+A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775).
 
 **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one.
 

tests/v2/test_local_corpora.py

@@ -41,9 +41,9 @@
 # the fetches read, so a patch reaches the code only when it is set on
 # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD).
 # A second load by path (load_tool) would be a copy nothing reads.
-core, ctgov, fec, tse, tw, ukec, psc = (
+core, ctgov, fec, tse, tw, ukec, psc, jrc = (
     importlib.import_module(f"local_{key}")
-    for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc"))
+    for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc"))
 
 _SOURCE = "ctgov-officials"
 _DATE = "2026-10-21T09:00:05"
@@ -2587,20 +2587,19 @@ def test_no_module_imports_a_sibling_constant() -> None:
     # tse.TSE_MAX_BAD), and every other reader reads it through that
     # module. A `from local_x import CONSTANT` would bind a second name a
     # patch on local_x cannot reach, and a test could pass with its guard
-    # off. The modules are SOURCES' own, so a new source is covered. PSC is
-    # the source's name, a key of SOURCES that local_agreement reads, and
-    # no fetch's setting.
+    # off. The modules are SOURCES' own, so a new source is covered. PSC and
+    # JRC are sources' names, keys of SOURCES, and no fetch's setting.
     import ast
     sources = {source.fetch.__module__ for source in local.SOURCES.values()}
-    assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc)}
+    assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc)}
     siblings = {"local_corpora", "local_core", *sources}
     imported = {(path.stem, alias.name)
                 for path in (Path(str(local.__file__)).parent / f"{s}.py"
                              for s in sorted(siblings))
                 for node in ast.walk(ast.parse(path.read_text(encoding="utf-8")))
                 if isinstance(node, ast.ImportFrom) and node.module in siblings
                 for alias in node.names if alias.name.lstrip("_").isupper()}
-    assert imported == {("local_corpora", "PSC")}
+    assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC")}
 
 
 def test_the_fold_lists_everything_its_functions_call_and_read() -> None: