From 2bcc6041fd59c0d52eb1aed83e291b61c401aa4e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 24 Oct 2026 12:57:27 -0700 Subject: [PATCH 1/2] Add jrc-names, a local corpus of JRC-Names person spellings, and a groups view JRC-Names (EU Joint Research Centre) lists each person entity with the spellings of its name harvested from news text in many scripts: the one local corpus holding several spellings of one person across scripts and transliterations, and 17,074 Arabic-script spellings. local_jrc.py downloads entities.gzip into the cache, pinned by sha256 (the file served on 2026-10-24 was last modified 2024-03-15), keeps the unmarked person rows as {entity, name, script} records and drops the language-marked rows, which are the JRC's own morphological variants (inflected forms, Arabic with a glued conjunction or article): 20,819 rows, every one beside an unmarked spelling of the same person. A download that is not the pinned file replaces nothing. Bad lines, too few persons and too many marked rows are gated by stated shares; a cut gzip or a file ending inside a line is refused; a corpus fetched under another pin or read version is refused on load. local_corpora.py groups compares the parse structure of each person's spellings (title, suffix, given, family, family length and position), counting persons, by pair of scripts, and lists the titles and suffixes read in some spellings only as vocabulary leads. The JRC-Names EULA bars redistribution and derivative works, so nothing read from it is committed; tests use invented names only. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/mechanisms.md | 2 +- tests/v2/test_local_corpora.py | 13 +- tests/v2/test_local_jrc.py | 385 ++++++++++++++++++++++++++++ tools/differential/README.md | 13 +- tools/differential/local_corpora.py | 198 +++++++++++++- tools/differential/local_jrc.py | 277 ++++++++++++++++++++ 7 files changed, 877 insertions(+), 13 deletions(-) create mode 100644 tests/v2/test_local_jrc.py create mode 100644 tools/differential/local_jrc.py diff --git a/AGENTS.md b/AGENTS.md index d9bf88f3..297cee32 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -278,7 +278,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. ## Gotchas diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index a1b9b93a..c26dab6c 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -414,7 +414,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" **How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone, or a local corpus whose records carry the fields their filers typed, with its record kept in the cache beside it (`local_agreement.py`). The local score tells a change's author what moved among real filed names, by field, and gives agreement by group and how often each disagreement shape occurs. It cannot guard anything: no CI run and no reviewer sees it, and no test fails when it moves (the tool itself exits 1). Its truth is the filer's, so a box filled wrongly (a job typed as a title) reads as a miss unless the scorer sets it aside, and the scorer has to say what it sets aside. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`). -A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). +A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one. diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index e9c4eb1f..1837000d 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -41,9 +41,9 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc = ( importlib.import_module(f"local_{key}") - for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc")) + for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -2587,12 +2587,11 @@ def test_no_module_imports_a_sibling_constant() -> None: # tse.TSE_MAX_BAD), and every other reader reads it through that # module. A `from local_x import CONSTANT` would bind a second name a # patch on local_x cannot reach, and a test could pass with its guard - # off. The modules are SOURCES' own, so a new source is covered. PSC is - # the source's name, a key of SOURCES that local_agreement reads, and - # no fetch's setting. + # off. The modules are SOURCES' own, so a new source is covered. PSC and + # JRC are sources' names, keys of SOURCES, and no fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} - assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc)} + assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc)} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" @@ -2600,7 +2599,7 @@ def test_no_module_imports_a_sibling_constant() -> None: for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))) if isinstance(node, ast.ImportFrom) and node.module in siblings for alias in node.names if alias.name.lstrip("_").isupper()} - assert imported == {("local_corpora", "PSC")} + assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC")} def test_the_fold_lists_everything_its_functions_call_and_read() -> None: diff --git a/tests/v2/test_local_jrc.py b/tests/v2/test_local_jrc.py new file mode 100644 index 00000000..ae1d5818 --- /dev/null +++ b/tests/v2/test_local_jrc.py @@ -0,0 +1,385 @@ +"""Offline checks on tools/differential/local_jrc.py, the jrc-names +local corpus, and on local_corpora.py's groups view, which reads it. + +Every name here is invented: the JRC-Names licence bars redistributing +the resource or anything derived from it, so not one of its names, nor +a reading of one, may reach the repository. What these pin is the +fetch's refusals (a file other than the pinned one, a gzip cut short, a +file ending inside a line, a layout or a marking that has moved), what +it keeps and counts, and the view's counting, every count a person's. +""" +import gzip +import hashlib +import importlib +import urllib.request +from pathlib import Path + +import pytest + +from ._differential_fixtures import load_tool, own_cache, plant_corpus + +local = load_tool("local_corpora") +# The module the fetch reads its constants from, by name: a patch set on +# a second load by path would reach nothing (test_local_corpora.py). +jrc = importlib.import_module("local_jrc") + +_JRC = "jrc-names" + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> Path: + return own_cache(monkeypatch, tmp_path_factory) + + +#: The header line, one person spelled three ways (one of them marked +#: Russian, one repeated), one person whose words run "++", an +#: organisation, and one person spelled twice, once marked Arabic with a +#: glued "and": 9 lines. +_LINES = ["#", "1\tP\tu\tAnn+Lee", "1\tP\tu\tАнна+Ли", "1\tP\tru\tАнны+Ли", + "1\tP\tu\tAnn+Lee", "2\tP\tu\tBo++Wu", "3\tO\tu\tAcme+Corp", + "4\tP\tar\tوأحمد+علي", "4\tP\tu\tأحمد+علي"] + + +def _body(lines: list[str], end: bytes = b"\r\n") -> bytes: + return b"".join(line.encode() + end for line in lines) + + +def _cache(monkeypatch: pytest.MonkeyPatch, raw: bytes, *, pin: bool = True) -> Path: + """`raw` as the cached download, pinned unless `pin` is False; its path.""" + path = local.directory(_JRC) / jrc.JRC_FILE + path.write_bytes(raw) + if pin: + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(raw).hexdigest()) + return path + + +def _cached(monkeypatch: pytest.MonkeyPatch, lines: list[str], **kw: object) -> Path: + return _cache(monkeypatch, gzip.compress(_body(lines, **kw))) # type: ignore[arg-type] + + +def _lenient(monkeypatch: pytest.MonkeyPatch) -> None: + """Shares a fixture of 9 lines can pass: its header is 1 in 9, and + its marked rows 2 in 7.""" + monkeypatch.setattr(jrc, "JRC_MAX_BAD", 1) + monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 1) + + +def test_a_jrc_fetch_keeps_each_plain_spelling_once_and_counts_the_rest( + monkeypatch: pytest.MonkeyPatch) -> None: + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + corpus = local.fetch(_JRC) + assert corpus.records == [ + {"entity": "1", "name": "Ann Lee", "script": "Latin"}, + {"entity": "1", "name": "Анна Ли", "script": "Cyrillic"}, + {"entity": "2", "name": "Bo Wu", "script": "Latin"}, + {"entity": "4", "name": "أحمد علي", "script": "Arabic"}] + assert corpus.meta["data_date"] == jrc.JRC_DATE + assert corpus.meta["detail"] == { + "sha256": jrc.JRC_SHA256, "read_version": jrc.JRC_READ_VERSION, + "lines": 9, "lines_read_past": 1, + "organisation_rows": 1, "person_rows": 7, "marked_variants_dropped": 2, + "marked_variants_by_language": "ar:1 ru:1", "repeated_spellings": 1, + "entities": 3, "entities_with_one_spelling": 2, + "spellings_holding_a_replacement_character": 0} + assert corpus.header().endswith("4 spellings (local corpus jrc-names)") + + +def test_a_jrc_file_that_is_not_the_pinned_one_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + _cache(monkeypatch, gzip.compress(_body(_LINES)), pin=False) + with pytest.raises(ValueError, match="not the pinned JRC_SHA256.*fetch jrc-names --refresh"): + local.fetch(_JRC) + assert local.load(_JRC) is None + + +@pytest.mark.parametrize("damage", ["cut short", "not a gzip", "corrupt"]) +def test_a_jrc_gzip_that_is_not_whole_names_the_way_out( + monkeypatch: pytest.MonkeyPatch, damage: str) -> None: + # Pinned to the damaged bytes: the pin is what stops this file, and + # a pin moved to a cut download must still not read as a corpus. + whole = gzip.compress(_body(_LINES)) + raw = {"cut short": whole[:-20], "not a gzip": _body(_LINES), + "corrupt": whole[:20] + bytes(b ^ 0xFF for b in whole[20:40]) + whole[40:]}[damage] + _cache(monkeypatch, raw) + _lenient(monkeypatch) + with pytest.raises(ValueError, match="not a whole gzip.*fetch jrc-names --refresh"): + local.fetch(_JRC) + + +def test_a_jrc_file_ending_inside_a_line_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + _cache(monkeypatch, gzip.compress(_body(_LINES)[:-6])) + _lenient(monkeypatch) + with pytest.raises(ValueError, match="ends inside a line"): + local.fetch(_JRC) + + +@pytest.mark.parametrize("bad", [ + "5\tP\tu", "5\tP\tu\tAnn+Lee\tX", "x5\tP\tu\tAnn+Lee", "5\tQ\tu\tAnn+Lee", + "5\tP\tRU\tAnn+Lee", "5\tP\tu\t+", b"5\tP\tu\tPe\xf1a".decode("latin-1"), + "\u00b2\tP\tu\tAnn+Lee"], + ids=["three columns", "five columns", "an id not digits", "another type", + "a language not a code", "an empty name", "not UTF-8", + "an id of a digit not ASCII"]) +def test_a_jrc_line_read_past_is_counted_up_to_its_share( + monkeypatch: pytest.MonkeyPatch, bad: str) -> None: + # 2 lines of 10 read past: refused at a share under 2/10, read at it. + lines = [*_LINES, bad] + raw = (_body(lines[:-1]) + bad.encode("latin-1" if "\xf1" in bad else "utf-8") + + b"\r\n") + _cache(monkeypatch, gzip.compress(raw)) + monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 1) + monkeypatch.setattr(jrc, "JRC_MAX_BAD", 2 / 10 - 1e-9) + with pytest.raises(ValueError, match="2 of 10 lines .* are not 4 well-formed columns"): + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_MAX_BAD", 2 / 10) + assert local.fetch(_JRC).meta["detail"]["lines_read_past"] == 2 + + +def test_a_jrc_file_with_too_few_persons_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + # A person type renamed reads every person row as read past, or, as + # here, a file of organisations: 1 person row of 3. + _cached(monkeypatch, ["1\tP\tu\tAnn+Lee", "2\tO\tu\tAcme", "3\tO\tu\tAcme+Two"]) + _lenient(monkeypatch) + with pytest.raises(ValueError, match="1 of 3 lines .* name a person"): + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_MIN_PERSONS", 1 / 3) + assert len(local.fetch(_JRC).records) == 1 + + +def test_a_jrc_file_marking_too_many_spellings_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + # 2 of the fixture's 7 person rows carry a language. + _cached(monkeypatch, _LINES) + monkeypatch.setattr(jrc, "JRC_MAX_BAD", 1) + monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 2 / 7 - 1e-9) + with pytest.raises(ValueError, match="2 of 7 person rows .* carry a language"): + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 2 / 7) + assert len(local.fetch(_JRC).records) == 4 + + +class _Download: + """A streamed download stating its whole length.""" + + def __init__(self, body: bytes) -> None: + self.body = body + self.headers = {"Content-Length": str(len(body))} + + def __enter__(self) -> "_Download": + return self + + def __exit__(self, *exc: object) -> None: + return None + + def read(self, n: int) -> bytes: + chunk, self.body = self.body[:n], self.body[n:] + return chunk + + +def _serve(monkeypatch: pytest.MonkeyPatch, *bodies: bytes) -> list[str]: + queue, asked = list(bodies), [] + + def urlopen(request: urllib.request.Request, timeout: int) -> _Download: + asked.append(request.full_url) + if not queue: + pytest.fail(f"an unexpected download of {request.full_url}") + return _Download(queue.pop(0)) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + return asked + + +def test_a_jrc_fetch_downloads_the_file_when_it_is_missing_and_once( + monkeypatch: pytest.MonkeyPatch) -> None: + raw = gzip.compress(_body(_LINES)) + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(raw).hexdigest()) + _lenient(monkeypatch) + asked = _serve(monkeypatch, raw) + assert local.main(["fetch", _JRC]) == 0 + assert local.main(["fetch", _JRC]) == 0 + assert asked == [jrc.JRC_URL] + assert (local.directory(_JRC) / jrc.JRC_FILE).read_bytes() == raw + + +def test_a_jrc_refresh_downloads_the_file_again(monkeypatch: pytest.MonkeyPatch) -> None: + raw = gzip.compress(_body(_LINES)) + _cache(monkeypatch, b"an older download") + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(raw).hexdigest()) + _lenient(monkeypatch) + asked = _serve(monkeypatch, raw) + assert local.main(["fetch", _JRC, "--refresh"]) == 0 + assert asked == [jrc.JRC_URL] + assert len(local.require(_JRC).records) == 4 + + +@pytest.mark.parametrize("cached", [True, False], ids=["a refresh", "a first fetch"]) +def test_a_jrc_download_that_is_not_the_pinned_file_replaces_nothing( + monkeypatch: pytest.MonkeyPatch, cached: bool) -> None: + # The JRC serves only its current file: a refresh writing a newer one + # over the pinned one would leave nothing to rebuild the corpus from. + pinned = gzip.compress(_body(_LINES)) + path = local.directory(_JRC) / jrc.JRC_FILE + if cached: + path.write_bytes(pinned) + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(pinned).hexdigest()) + _lenient(monkeypatch) + newer = gzip.compress(_body(_LINES[:3])) + _serve(monkeypatch, newer) + with pytest.raises(ValueError, match="not the pinned JRC_SHA256: it is kept at"): + local.main(["fetch", _JRC, "--refresh"] if cached else ["fetch", _JRC]) + assert path.with_name(jrc.JRC_FILE + ".new").read_bytes() == newer + assert path.read_bytes() == pinned if cached else not path.exists() + + +def test_a_jrc_download_kept_by_a_mismatch_is_adopted_once_pinned( + monkeypatch: pytest.MonkeyPatch) -> None: + raw = gzip.compress(_body(_LINES)) + local.directory(_JRC).joinpath(jrc.JRC_FILE + ".new").write_bytes(raw) + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(raw).hexdigest()) + _lenient(monkeypatch) + assert _serve(monkeypatch) == [] and len(local.fetch(_JRC).records) == 4 + assert (local.directory(_JRC) / jrc.JRC_FILE).read_bytes() == raw + + +def test_a_jrc_corpus_read_another_way_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch) -> None: + # A change to script() would otherwise leave the cached records' + # labels standing under code that no longer gives them. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_READ_VERSION", jrc.JRC_READ_VERSION + 1) + with pytest.raises(ValueError, match=f"read by read version 1.*fetch {_JRC} --refresh"): + local.load(_JRC) + + +def test_a_jrc_corpus_read_from_another_file_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch) -> None: + # Moving the pin would otherwise leave the old file's corpus standing + # under the new pin's data date. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_SHA256", "0" * 64) + with pytest.raises(ValueError, match=f"read from the file with sha256 .*reads 0{{64}}" + f".*fetch {_JRC} --refresh"): + local.load(_JRC) + + +@pytest.mark.parametrize(("name", "script"), [ + ("Ann Lee", "Latin"), ("José Peña", "Latin"), ("Анна Ли", "Cyrillic"), + ("أحمد علي", "Arabic"), ("王小明", "Han"), ("アン・リー", "Katakana"), + ("ルーシー", "Katakana"), ("ｱﾝ", "Katakana"), ("Ｊｏｈｎ", "Latin"), + ("ʿAli", "Latin"), ("Maªa", "Latin"), ("佐々木", "Han"), ("のゞ々", "Hiragana"), + ("Мар\u02bcяна", "Cyrillic"), ("ʿ", "none"), ("Ann Ли", "mixed"), + ("- ? -", "none")]) +def test_a_spelling_s_script_is_the_script_of_its_letters(name: str, script: str) -> None: + # ー, the prolonged sound mark, is a letter Unicode files under + # KATAKANA-HIRAGANA; a halfwidth or fullwidth letter names its form + # before its script; ʿ, ʼ, ª and 々 are written in several scripts + # and name none: none is a second script. + assert jrc.script(name) == script + + +# -- the groups view ------------------------------------------------------------- + +def _spelled(*spellings: tuple[str, str]) -> object: + """A jrc-names corpus holding each (entity, name) of `spellings`.""" + return local.Local(_JRC, {"data_date": jrc.JRC_DATE}, + [{"entity": entity, "name": name, "script": jrc.script(name)} + for entity, name in spellings]) + + +def test_a_reading_is_the_structure_of_the_parse_never_its_words() -> None: + assert local.reading("Ann Lee") == (False, False, True, True, 1, "last") + assert local.reading("Dr. Ann de la Lee Jr.") == (True, True, True, True, 3, "last") + assert local.reading("Lee, Ann") == (False, False, True, True, 1, "first") + assert local.reading("Ann") == (False, False, True, False, 0, "none") + assert local.reading("Анна Ли") == local.reading("Ann Lee") + + +def test_the_groups_view_counts_persons_by_the_scripts_of_their_spellings() -> None: + found = local.group_counts(_spelled( + ("1", "Ann Lee"), ("1", "Анна Ли"), + ("2", "Bo Wu"), ("2", "Dr. Bo Wu"), ("2", "Бо Ву"), + ("3", "Cy Ng"))) # one spelling: no group + assert (found.persons, found.spellings) == (2, 5) + assert found.groups == {"any": 2, "Cyrillic / Latin": 2, "Latin / Latin": 1} + # Person 2's Latin spellings differ between themselves, and its + # Cyrillic one reads as one of them does: the scripts agree. + assert found.agree == {"any": 1, "Cyrillic / Latin": 2} + assert found.differ == {("any", "title"): 1, ("Latin / Latin", "title"): 1} + assert found.titles == {"dr": 1} and found.suffixes == {} + + +def test_the_groups_view_finds_two_scripts_that_read_a_field_apart() -> None: + # Every Latin spelling reads a title and no Cyrillic one does. + found = local.group_counts(_spelled(("1", "Dr. Ann Lee"), ("1", "Dr Ann Lee"), + ("1", "Анна Ли"))) + assert found.differ == {("any", "title"): 1, ("Cyrillic / Latin", "title"): 1} + assert found.agree == {"Latin / Latin": 1} + + +def test_two_scripts_read_alike_only_where_one_spelling_matches_one_whole() -> None: + # The Cyrillic spelling reads a title and a suffix, and each Latin one + # reads one of the two: no field is read apart, yet no spelling in one + # script reads as any in the other does. + found = local.group_counts(_spelled(("1", "Dr. Ann Lee"), ("1", "Ann Lee Jr."), + ("1", "Др. Анна Ли мл."))) + assert local.reading("Др. Анна Ли мл.")[:2] == (True, True) + assert ("Cyrillic / Latin", "title") not in found.differ + assert ("Cyrillic / Latin", "suffix") not in found.differ + assert "Cyrillic / Latin" not in found.agree + + +def test_the_groups_view_counts_a_person_once_however_many_spellings_differ() -> None: + found = local.group_counts(_spelled(*[ + ("1", name) for name in + ("Ann Lee", "Dr. Ann Lee", "Ann Lee Jr.", "Ann de Lee", "Lee")])) + assert found.groups == {"any": 1, "Latin / Latin": 1} + # Every spelling reads a given name; "Lee" alone reads no family. + assert found.differ == {(pair, field): 1 for pair in ("any", "Latin / Latin") + for field in local.GROUP_FIELDS if field != "given"} + assert found.titles == {"dr": 1} and found.suffixes == {"jr": 1} + + +def test_the_groups_view_lists_only_titles_many_persons_share( + capsys: pytest.CaptureFixture[str]) -> None: + rows = [(str(i), name) for i in range(10) for name in ("Ann Lee", "Dr. Ann Lee")] + rows += [("10", "Bo Wu"), ("10", "Prof. Bo Wu")] + print(local.format_groups(local.group_counts(_spelled(*rows)), 5)) + out = capsys.readouterr().out + assert out.startswith("11 persons with two or more spellings (22 spellings)") + assert "titles read in some spellings only: dr 10" in out + assert "prof" not in out + + +@pytest.mark.parametrize(("records", "match"), [ + ([], "holds no records"), + ([{"name": "ANA", "registry": "ANA"}], "carry no entity and script"), +], ids=["no records", "records with no entity"]) +def test_the_groups_view_refuses_records_it_cannot_group( + records: list[dict[str, str]], match: str) -> None: + with pytest.raises(ValueError, match=match): + local.group_counts(local.Local("tse-donors", {"data_date": jrc.JRC_DATE}, records)) + + +def test_the_groups_command_reports_a_fetched_corpus( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + capsys.readouterr() + assert local.main(["groups", _JRC]) == 0 + out = capsys.readouterr().out + assert out.startswith("JRC-Names (European Commission Joint Research Centre)") + assert "1 persons with two or more spellings (2 spellings)" in out + assert "100.0% alike Cyrillic / Latin" in out + + +def test_the_groups_command_refuses_a_source_with_no_entities( + monkeypatch: pytest.MonkeyPatch) -> None: + plant_corpus(monkeypatch, local, "ukec-individuals", [{"name": "Ann Lee"}], + jrc.JRC_DATE, "names") + with pytest.raises(SystemExit, match="carry no entity and script"): + local.main(["groups", "ukec-individuals"]) diff --git a/tools/differential/README.md b/tools/differential/README.md index 9eeaf5a1..4e9ee477 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -235,7 +235,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`; `PSC_EXTRACT_VERSION` does it for extracts): the sha256 checks prove the files are the ones fetched, not that today's code would write them, and the review that added it found a changed label read under the old records with no warning. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -263,6 +263,7 @@ Any reading of 5,000 entries or more, committed or local, parses in a worker pro | `tw-donors` | the distinct names of every individual donor in every campaign finance report Taiwan's Control Yuan publishes on its political donation platform (ardata.cy.gov.tw, whose zips data.gov.tw also lists): 454 reports of 64 elections on 2026-10-22, presidential, legislative and local: every report from 2018's elections on, and the residual reports of 2014's and 2016's. Each report's zip is cached under the name the platform lists it by, which carries the date it was built, so a refresh downloads only the reports published or rebuilt since. Names as a candidate's staff typed them into a filing, published in full where the ID, address and phone beside them are masked. Of the 308,667 names dated 2026-10-05, 296,820 were three Han characters, 5,449 four and 5,260 two; the population for unspaced Han names written family first, and for the married double surname (冠夫姓) a four-character name often is. Under the Open Government Data License, version 1.0, with the platform's own bar on using the data for profit, credit investigation or another improper purpose | names in spoken order; Latin script (80 of the names dated 2026-10-05 held a Latin letter); the comma (3 of them held a fullwidth one, none an ASCII one); case; names outside Taiwan; donors whose names the report does not publish, such as anonymous gifts; Japanese kanji names, which the Han script cannot tell from Chinese | | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | | `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. The one local corpus whose names carry their fields, scored by `local_agreement.py` | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | +| `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256 and to the code that cuts and folds a word, so a refetch or a change to the fold rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. @@ -278,6 +279,16 @@ uv run python tools/differential/local_corpora.py pairs tse-donors --agreement On the fetch of 2026-10-22, 205,925 of `tse-donors`' 854,531 pairs differ. The commonest shapes: words left out (22.0%: `DE` left out of 13,412 pairs, `DA` 5,906, `DOS` 3,033, then second surnames, `SILVA` 2,869), accents only (21.1%), words shortened (15.8%), words misspelled (14.9%) and words left out and shortened (11.8%). Read the agreement per shape, since a shape can change the surname legitimately: a pair that leaves out the last surname ends the family in another word, which is likely most of why only 44% of words left out agree (not measured apart from the other causes). Accents only cannot change the surname, so its 1,304 of 43,552 pairs that disagree are the parser reading one name two ways: by their last words, `JÚNIOR` against `JUNIOR` in 866 (#843) and `SÁ` against `SA` in 255, where `MARIA DE SA` reads `SA` as a credential and `MARIA DE SÁ` as the surname. The run took 39s. +`jrc-names` reads a gzip downloaded once into the cache (`fetch --refresh` downloads it again), and refuses one whose sha256 is not `JRC_SHA256`: the JRC says the resource is updated daily, but the file served on 2026-10-24 was last modified on 2024-03-15, so a new file is measured and the pin moved by hand rather than read under constants measured on another. A download that is not the pinned file replaces nothing: it is kept beside the cached one as `entities.gzip.new`, to be measured, and adopted without another download once the pin moves to it, since the JRC serves only its current file and the pinned one could not be fetched again. `meta.json` records the pin and `JRC_READ_VERSION`, and a corpus fetched under another of either is refused on load, so a change to how a spelling is read or its script labelled is not counted under the old records. It keeps the person rows (`P`, 1,338,463 of 1,445,013 lines) and reads past the organisations (`O`). A row the JRC marks with a language (`ru`, `ar`, `sl`, `et`, `pl` and others, 20,819 of the person rows) is a morphological variant of a spelling it leaves unmarked (`u`): an inflected form, or an Arabic spelling with و, ل or ال glued on. Every marked row's entity held an unmarked spelling, and 18,594 of the 20,819 had as many words as one and differed from it only in endings or in glued fronts of three letters or fewer (the rule is stated exactly at `JRC_UNMARKED` in `local_jrc.py`); the fetch drops them, counted by language in `meta.json`'s detail. The unmarked spellings still hold inflected forms the JRC did not mark (by the same rule, 22,732 of the 66,673 distinct unmarked Cyrillic spellings are another Cyrillic spelling of the same person), which are kept. Words joined by `+` are read as spaces. A line that is not four well-formed columns (the file's first line, `#`, is one) is counted, and past `JRC_MAX_BAD` refuses the file, as do a gzip cut short, a file ending inside a line, fewer person rows than `JRC_MIN_PERSONS` (a type renamed) and more marked rows than `JRC_MAX_MARKED` (a file that marks every spelling). The licence bars redistributing the resource or anything derived from it, so its snapshot, its views' output and every file derived from it stay in the cache, and what is cited from it is counts and shapes written with made-up names. A snapshot of it (`snapshot.py --local jrc-names`) recorded 1,315,051 names in 30s on 12 CPUs (2026-10-24, 128 MB). + +`groups` reads a source whose records carry an entity and a script (`jrc-names`) as several spellings of one person. It parses every spelling of each person with two or more and compares the readings' structure: whether a title, a suffix, a given name and a family name are read, how many words the family is, and where it stands among the name's words (first, last, all of them, inside, or nowhere). Words are not compared, so spellings in two scripts can be; but where the family stands moves with a script's order, so a script read family first (Hangul, spaced Han) differs from one read given first in that field by convention, not by error. It counts PERSONS, not pairs, so a person spelled 600 ways weighs what one spelled twice does. A pair of scripts is one row: within one script (`Latin / Latin`) a field differs where the person's spellings in it read it more than one way, and they read alike where all read one way; across two (`Cyrillic / Latin`) a field differs where no spelling in one reads it as any in the other does, so two Latin spellings that differ between themselves are not counted against the Cyrillic one, and they read alike where some spelling in one reads, in every field, as some spelling in the other. It then lists the titles and suffixes of the persons some of whose spellings read one and others none (two spellings with two different titles and no untitled one add nothing), by the persons sharing one, ten at least: the lead is a word the vocabulary reads as a title or suffix that is a name in some spelling, or a title spelled in a way it does not know. + +``` +uv run python tools/differential/local_corpora.py groups jrc-names +``` + +On the fetch of 2026-10-24 (data dated 2024-03-15), 93.7% of the 218,822 persons with two or more spellings read alike in every spelling, and the run took 50s. Within a script the disagreement is mostly the writing, not the parse. Katakana / Katakana reads alike for 0.2% of its 4,289 persons because the JRC writes a transliterated name three ways: unspaced and undivided, which the parser cannot divide and reads as a given name alone (4,347 of the 13,039 Katakana spellings), divided by `・` with no space (4,337), and spaced (4,355), the last two read given then family. Han / Han, 2.2% of 2,420, holds unspaced Han, read as one family word, beside spaced Han, read family first, and Han divided by `·`, read in its written order. Arabic / Arabic reads alike for 67.8% of 3,329 persons, and the Arabic comma is most of why: each of the 736 persons with a spelling holding one differs, of the 1,072 that differ, since the comma moves the family to the front. Across scripts read in the same order most persons read alike (Cyrillic / Latin 99.3% of 20,837, Arabic / Latin 98.2% of 5,347, Greek / Latin 98.6% of 915); Hangul against a script read given first reads alike for 3% or fewer in every such row `--top 40` lists (Hangul / Latin 2.0% of 1,247), its family standing first, while Han / Hangul, both read family first, is 87.1% of 1,002. The titles read in some spellings only start `sheikh` 50, `bard` 33, `shaikh` 27, `executive` 22 and include `mahdi` 17, a common given name; the suffixes `jr` 62, `sa` 30, `ii` 27. + ## Corpus provenance `corpus.jsonl` is checked in as a test fixture. It was built by diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index a9c5fe64..d6372cf2 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -13,6 +13,7 @@ uv run python tools/differential/local_corpora.py fetch ctgov-officials uv run python tools/differential/local_corpora.py tally ctgov-officials uv run python tools/differential/local_corpora.py pairs tse-donors + uv run python tools/differential/local_corpora.py groups jrc-names uv run python tools/differential/snapshot.py --local ctgov-officials `fetch` writes three files into local// under @@ -24,7 +25,8 @@ the study and role; one distinct FEC, Taiwan or UK donor name, alone; one distinct pair of a TSE donor's filed name and the tax registry's name for the same donor; one distinct PSC filed name, with its four - boxes and nationality. The counts are counted from here. + boxes and nationality; one distinct spelling of a JRC-Names person, + with its entity and script. The counts are counted from here. - corpus.jsonl -- the distinct names, in the corpus line format snapshot.py and compare.py --corpus read. - meta.json -- the source's own date for the data, what was fetched, and @@ -51,8 +53,8 @@ the listing gives). What each source is, what it is the population for and what its terms allow is its module's docstring: local_ctgov.py (ctgov-officials), local_fec.py (fec-individuals), local_tse.py -(tse-donors), local_tw.py (tw-donors), local_ukec.py (ukec-individuals) -and local_psc.py (psc-individuals). +(tse-donors), local_tw.py (tw-donors), local_ukec.py (ukec-individuals), +local_psc.py (psc-individuals) and local_jrc.py (jrc-names). """ from __future__ import annotations @@ -89,6 +91,7 @@ import local_fec # noqa: E402 import local_tse # noqa: E402 from local_fec import check_fec, fetch_fec, renew_fec # noqa: E402 +from local_jrc import JRC, check_jrc, fetch_jrc, renew_jrc # noqa: E402 from local_psc import PSC, check_psc, fetch_psc, renew_psc # noqa: E402 from local_tse import check_tse, fetch_tse # noqa: E402 from local_tw import fetch_tw # noqa: E402 @@ -182,6 +185,11 @@ def index(self) -> WordIndex: "(a public register: Companies House sets no terms on its use, " "and data protection law applies to the names)", unit="filed names", fetch=fetch_psc, renew=renew_psc, check=check_psc), + JRC: Source( + credit="JRC-Names (European Commission Joint Research Centre), person " + "entities (licensed under the JRC-Names EULA: personal use, no " + "redistribution or derivative works)", + unit="spellings", fetch=fetch_jrc, renew=renew_jrc, check=check_jrc), } @@ -518,6 +526,176 @@ def pair_counts(local: Local, *, agreement: bool = False) -> PairCounts: if agreement else None, accent_splits) +#: What the groups view compares between two spellings of one person, +#: in the order it reports them: whether parse() reads a title, a suffix, +#: a given name and a family name, how many words the family is, and +#: where it stands among the name's words (reading). +GROUP_FIELDS = ("title", "suffix", "given", "family", "family words", "family at") +#: The name-bearing roles, among which reading places the family. +_NAME_ROLES = frozenset({"given", "middle", "family"}) + + +Reading = tuple[bool, bool, bool, bool, int, str] + + +def _read(name: str) -> tuple[Reading, str, str]: + """reading(name), and the title and the suffix parse() reads.""" + from nameparser import parse + + parsed = parse(name) + roles = [str(t.role) for t in parsed.tokens if str(t.role) in _NAME_ROLES] + family = [i for i, role in enumerate(roles) if role == "family"] + if not family: + at = "none" + elif len(family) == len(roles): + at = "whole" + elif family == list(range(len(family))): + at = "first" + elif family == list(range(len(roles) - len(family), len(roles))): + at = "last" + else: + at = "inner" + return ((bool(parsed.title), bool(parsed.suffix), bool(parsed.given), + bool(parsed.family), len(parsed.family.split()), at), + parsed.title, parsed.suffix) + + +def reading(name: str) -> Reading: + """The structure parse() gives `name`, one value per GROUP_FIELDS: + title read, suffix read, given found, family found, the family's + words, and where the family stands among the given, middle and + family words: "first", "last", "whole" (it is all of them), "inner" + or "none". Counts and positions, never the words themselves, so two + spellings in different scripts compare.""" + return _read(name)[0] + + +def _differing(one: set[Reading], other: set[Reading] | None = None) -> set[str]: + """The GROUP_FIELDS in which the readings `one` disagree among + themselves (more than one value), or, given `other`, in which no + reading of `one` agrees with any of `other` (no value in common): two + Latin spellings that differ between themselves do not make the + Latin and the Cyrillic spellings differ where a Cyrillic one reads + as either Latin one does.""" + if other is None: + return {f for i, f in enumerate(GROUP_FIELDS) if len({r[i] for r in one}) > 1} + return {f for i, f in enumerate(GROUP_FIELDS) + if {r[i] for r in one}.isdisjoint({r[i] for r in other})} + + +@dataclass(frozen=True) +class GroupCounts: + """What group_counts finds, every count a count of PERSONS, not of + spellings or of pairs, so that a person spelled 600 ways weighs what + one spelled twice does. Keyed by a pair of scripts ("Cyrillic / + Latin", "Latin / Latin"), and by "any" for all of a person's + spellings taken together.""" + #: The persons with two or more spellings, and those spellings. + persons: int + spellings: int + #: pair -> the persons with two spellings in those scripts. + groups: Counter[str] + #: pair -> those whose spellings read alike: within one script, all + #: of them read one way; across two, some spelling in one reads, in + #: every field, as some spelling in the other does. + agree: Counter[str] + #: (pair, a GROUP_FIELDS member) -> those whose spellings disagree in + #: the field (_differing): within one script's for "Latin / Latin", + #: between the two scripts' for "Cyrillic / Latin". + differ: Counter[tuple[str, str]] + #: The titles, and the suffixes, of a person some of whose spellings + #: read one and others none, folded by word_key, counted once a + #: person: two spellings reading two titles (Sheikh, Shaikh) with no + #: untitled spelling beside them add nothing. + titles: Counter[str] + suffixes: Counter[str] + + +def group_counts(local: Local) -> GroupCounts: + """Each person with two or more spellings, from a source whose + records carry an "entity" and a "script": every spelling's reading + compared by the pair of scripts the spellings are written in, as + _differing says. A distinct name is parsed once. A source without + entities raises ValueError.""" + if not local.records: + raise ValueError(f"{local.source} holds no records") + if any("entity" not in r or "script" not in r for r in local.records): + raise ValueError(f"{local.source}'s records carry no entity and script " + f"to group spellings by") + by_entity: dict[str, list[tuple[str, str]]] = {} + for record in local.records: + by_entity.setdefault(record["entity"], []).append((record["script"], record["name"])) + read: dict[str, tuple[Reading, str, str]] = {} + groups: Counter[str] = Counter() + agree: Counter[str] = Counter() + differ: Counter[tuple[str, str]] = Counter() + titles: Counter[str] = Counter() + suffixes: Counter[str] = Counter() + persons = spellings = 0 + for variants in by_entity.values(): + if len(variants) < 2: + continue + persons += 1 + spellings += len(variants) + by_script: dict[str, set[Reading]] = {} + written: Counter[str] = Counter() + for script, name in variants: + if name not in read: + read[name] = _read(name) + by_script.setdefault(script, set()).add(read[name][0]) + written[script] += 1 + everything = set().union(*by_script.values()) + # pair -> (the fields read apart, whether the spellings read alike) + compared = {"any": (_differing(everything), len(everything) == 1)} + scripts = sorted(by_script) + for i, first in enumerate(scripts): + for second in scripts[i:]: + one = by_script[first] + if first != second: + compared[f"{first} / {second}"] = ( + _differing(one, by_script[second]), + not one.isdisjoint(by_script[second])) + elif written[first] > 1: + compared[f"{first} / {first}"] = (_differing(one), len(one) == 1) + for pair, (fields, alike) in compared.items(): + groups[pair] += 1 + if alike: + agree[pair] += 1 + differ.update((pair, field) for field in fields) + for counter, at in ((titles, 0), (suffixes, 1)): + if len({r[at] for r in everything}) > 1: + counter.update({word_key(read[name][1 + at]) for _, name in variants + if read[name][0][at]} - {""}) + return GroupCounts(persons, spellings, groups, agree, differ, titles, suffixes) + + +def format_groups(found: GroupCounts, top: int) -> str: + """The report `groups` prints: per pair of scripts, the persons with + two spellings in them, the share whose spellings read alike + (GroupCounts.agree), and the share whose spellings read each field + apart (_differing); then the titles and suffixes read in some of a + person's spellings and in none of the others, listed only where + PAIR_WORD_FLOOR persons share one. Counts and words only.""" + lines = [f"{found.persons:,} persons with two or more spellings " + f"({found.spellings:,} spellings); by the scripts of two of them:"] + pairs = ["any", *sorted((p for p in found.groups if p != "any"), + key=lambda p: (-found.groups[p], p))[:top]] + for pair in pairs: + n = found.groups[pair] + if not n: + continue + shares = ", ".join(f"{field} {found.differ[(pair, field)] / n:.1%}" + for field in GROUP_FIELDS if found.differ[(pair, field)]) + lines.append(f" {n:>9,} {found.agree[pair] / n:6.1%} alike {pair}" + + (f"; differ in {shares}" if shares else "")) + for words, what in ((found.titles, "titles read in some spellings only"), + (found.suffixes, "suffixes read in some spellings only")): + listed = [(w, n) for w, n in words.most_common(top) if n >= PAIR_WORD_FLOOR] + if listed: + lines.append(f"{what}: " + ", ".join(f"{w} {n:,}" for w, n in listed)) + return "\n".join(lines) + + #: Where a name is cut into words: spaces, commas, semicolons, slashes. #: A hyphen stays inside a word (FNP-C), and a period inside a token #: (M.D.) is folded away by word_key. @@ -773,6 +951,12 @@ def main(argv: list[str] | None = None) -> int: pair.add_argument("--agreement", action="store_true", help="also parse each pair that differs and count the " "pairs whose two spellings share a surname") + group = sub.add_parser("groups", help="compare the readings of each person's " + "spellings, by the scripts they are in") + group.add_argument("source", choices=sorted(SOURCES)) + group.add_argument("--top", type=int, default=20, metavar="N", + help="pairs of scripts, and titles and suffixes, to list " + "(default 20)") args = ap.parse_args(argv) if args.command == "fetch": local = None @@ -805,6 +989,14 @@ def main(argv: list[str] | None = None) -> int: print(local.header()) print(format_pairs(found, args.top)) return 0 + if args.command == "groups": + try: + grouped = group_counts(local) + except ValueError as exc: # CorpusDamaged included: the records, read here first + raise SystemExit(str(exc)) from None + print(local.header()) + print(format_groups(grouped, args.top)) + return 0 try: parts, with_comma = tally(local) except CorpusDamaged as exc: # the records, read here first diff --git a/tools/differential/local_jrc.py b/tools/differential/local_jrc.py new file mode 100644 index 00000000..b0d619fe --- /dev/null +++ b/tools/differential/local_jrc.py @@ -0,0 +1,277 @@ +"""The jrc-names local corpus: the person names of the JRC-Names resource. + +jrc-names -- the person entities of the EU Joint Research Centre's +JRC-Names (wt-public.emm4u.eu), each with the spellings of its name +harvested from news text in many languages and scripts: a record is one +spelling, with the entity it spells and the script it is written in. +The one local corpus that holds several spellings of one person across +scripts (tse-donors pairs two spellings, both Latin), so it is the +population for spelling and transliteration variants, read by +local_corpora.py's `groups` view (25,415 persons are spelled in two +scripts or more, a "mixed" or "none" script() not counted), and for +Arabic-script names: 17,074 spellings of +7,833 persons, 5,347 of them spelled in Latin too, in the file dated +2024-03-15. Latin is most of it (1,203,634 of 1,317,586 spellings, then +Cyrillic 66,673, Katakana 13,039 and Han 8,016); the Indic scripts are +few (Devanagari 827 spellings, Bengali 419, Tamil 328, Malayalam 183, +the rest under 80 each). 1,510 of the Arabic spellings hold an Arabic +comma (U+060C), which the parser reads as a comma. + +Licensed under the JRC-Names EULA (Resources/LICENCE-EULA_JRC-Names_2011.pdf +on the same site): a personal, non-exclusive, non-transferable licence +that bars redistribution, sublicensing and derivative works. So nothing +read from it is committed, not a name, not a per-name reading: the +download, the corpus, its snapshot and every file derived from them stay +in the cache, and what a contributor cites is counts and shapes written +with made-up names, as for every local corpus. +""" +from __future__ import annotations + +import gzip +import hashlib +import os +import re +import unicodedata +import zlib +from collections import Counter +from pathlib import Path +from typing import Any + +from local_core import Fetched, _download, _pinned, _refuse_past, directory, fetch_command + + +JRC = "jrc-names" +JRC_URL = "https://wt-public.emm4u.eu/data/entities.gzip" +#: The download, cached beside the corpus. +JRC_FILE = "entities.gzip" +#: The file this tool was written against, pinned: the JRC says the +#: resource is updated daily, but the file served on 2026-10-24 was last +#: modified on the date below. A download with another sha256 is refused +#: until someone has measured it and moved the pin, since a layout the +#: constants below were not measured on would otherwise read as a +#: smaller corpus. +JRC_SHA256 = "43144cee881ba8a81573ae28c089dffce0ca634d342791414628a5d54997a7d0" +#: The pinned file's Last-Modified, its data date: the file states none. +JRC_DATE = "2024-03-15T14:08:28Z" +#: How the file is read into records, stored in meta.json's detail: a +#: change to what _jrc_read keeps or to script() bumps it, so that a +#: corpus read the old way is refused on load (check_jrc) rather than +#: counted under labels the code no longer gives. +JRC_READ_VERSION = 1 +#: A line: entity id, type, language and name, tab-separated, the name's +#: words joined by "+". Lines end in CRLF. +JRC_COLUMNS = 4 +JRC_PERSON, JRC_ORGANISATION = "P", "O" +#: The language of a spelling the JRC did not mark: 1,317,644 of the +#: 1,338,463 person rows. A row with any other language is a +#: morphological variant of one that has it: an inflected form (a +#: Russian name in another case, a Slovene or Polish one with its case +#: ending) or an Arabic one with a word glued to it (و "and", ل "for", +#: ال "the"). Of the 20,819 marked rows in the pinned file, every one's +#: entity also held an unmarked spelling, and 18,594 had the same number +#: of words as one and, word by word in order, either every word the +#: same or sharing at least its first letter with the other and no more +#: than three letters past their common start (an ending), or every word +#: the other with no more than three letters glued to its front (a +#: prefix); so the fetch drops them, counted by language, +#: rather than read an ending or a conjunction as part of a name. The +#: unmarked spellings still hold some inflected forms the JRC did not +#: mark: 22,732 of the 66,673 distinct unmarked spellings whose script() +#: is Cyrillic are, by the ending rule above, another Cyrillic spelling +#: of the same entity (some may be two transliterations rather than an +#: inflection; the two were not measured apart), and they are kept. +JRC_UNMARKED = "u" +_LANGUAGE = re.compile(r"[a-z]{2,3}") +#: The share of lines read past (not JRC_COLUMNS columns, not UTF-8, a +#: type other than JRC_PERSON or JRC_ORGANISATION, an id that is not +#: digits, a language that is not a code, a name that is empty) before +#: the file is refused: 1 of the pinned file's 1,445,013 lines was, its +#: first, "#". +JRC_MAX_BAD = 1 / 10_000 +#: The least share of well-formed lines that are people before the file +#: is refused: 1,338,463 of 1,445,012 (93%) in the pinned file. A type +#: renamed would otherwise read as a corpus of none. +JRC_MIN_PERSONS = 1 / 2 +#: The most person rows that may carry a language before the file is +#: refused: 20,819 of 1,338,463 (1.6%) in the pinned file. A file whose +#: every row came marked would otherwise be dropped whole. +JRC_MAX_MARKED = 1 / 10 + +#: A script's name as the first word of its letters' Unicode names +#: gives it, where that word is not the script's own name. +_SCRIPT_NAMES = {"CJK": "Han", "KATAKANA-HIRAGANA": "Katakana"} +#: First words that name no script: letters written in several (ʿ and ʼ, +#: MODIFIER LETTER, in a Latin and in a Ukrainian Cyrillic spelling; the +#: ordinals ª and º; the iteration mark 々, in kanji and kana names +#: alike), which script() reads past rather than call a second script. +_NO_SCRIPT = frozenset({"MODIFIER", "FEMININE", "MASCULINE", "IDEOGRAPHIC"}) +#: A first word that names a form, not a script: the script is the next +#: word (HALFWIDTH KATAKANA LETTER A, FULLWIDTH LATIN CAPITAL LETTER A). +_FORMS = frozenset({"HALFWIDTH", "FULLWIDTH"}) +_ID = re.compile(r"[0-9]+") + + +def script(name: str) -> str: + """The script `name`'s letters are written in, from the first word + of each letter's Unicode name ("Latin", "Cyrillic", "Arabic", "Han" + for CJK, "Katakana" counting the prolonged sound mark), "mixed" where + its letters are in two or more, and "none" where it has no letter. + Coarser than the parser's own script test, which names only the + scripts it treats apart, and enough to key a pair of spellings by.""" + found = set() + for char in name: + if char.isalpha(): + words = unicodedata.name(char, "UNKNOWN").split() + word = words[1] if words[0] in _FORMS and len(words) > 1 else words[0] + if word not in _NO_SCRIPT: + found.add(_SCRIPT_NAMES.get(word, word.title())) + if not found: + return "none" + return found.pop() if len(found) == 1 else "mixed" + + +def _jrc_download(path: Path) -> None: + """Download JRC_URL to `path`, through a file beside it that replaces + `path` only once its sha256 is JRC_SHA256's: the JRC serves only its + current file, so a refresh that overwrote the pinned one with a newer + one would leave the corpus nothing to be rebuilt from. A download + that is not the pinned file is left beside it, to be measured.""" + new = path.with_name(path.name + ".new") + # A download kept by an earlier mismatch, once the pin has moved to it. + if not (new.exists() and _sha256(new) == JRC_SHA256): + _download(JRC_URL, new) + held = _sha256(new) + if held != JRC_SHA256: + old = (f"the cached file at {path} is untouched" if path.exists() + else "no file is cached") + raise ValueError(f"{JRC_URL} served a file with sha256 {held}, not the " + f"pinned JRC_SHA256: it is kept at {new}, and {old}. " + f"Measure the new file and move JRC_SHA256 and JRC_DATE " + f"to read it") + os.replace(new, path) + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as file: + while block := file.read(1 << 20): + digest.update(block) + return digest.hexdigest() + + +def _jrc_read(path: Path) -> Fetched: + """The person spellings of the file at `path`, read as fetch_jrc + says.""" + counts: Counter[str] = Counter() + marked: Counter[str] = Counter() + kept: set[tuple[int, str]] = set() + last = b"\n" + with gzip.open(path, "rb") as lines: + for raw in lines: + counts["lines"] += 1 + last = raw + try: + text = raw.decode("utf-8") + except UnicodeDecodeError: + counts["bad"] += 1 + continue + cols = text.rstrip("\r\n").split("\t") + if len(cols) != JRC_COLUMNS: + counts["bad"] += 1 + continue + entity, kind, language, written = cols + name = " ".join(word for word in written.split("+") if word).strip() + if (not _ID.fullmatch(entity) or kind not in (JRC_PERSON, JRC_ORGANISATION) + or not (language == JRC_UNMARKED or _LANGUAGE.fullmatch(language)) + or not name): + counts["bad"] += 1 + continue + if kind == JRC_ORGANISATION: + counts["organisations"] += 1 + continue + counts["persons"] += 1 + if language != JRC_UNMARKED: + marked[language] += 1 + continue + if (int(entity), name) in kept: + counts["repeated"] += 1 + continue + kept.add((int(entity), name)) + if not last.endswith(b"\n"): + raise ValueError(f"{path} ends inside a line: the file is cut short; " + f"download it again: {fetch_command(JRC, refresh=True)}") + well_formed = counts["lines"] - counts["bad"] + _refuse_past([(counts["bad"], counts["lines"], JRC_MAX_BAD, + f"lines of {path} are not {JRC_COLUMNS} well-formed columns")], + "the file's layout has changed") + if counts["persons"] < well_formed * JRC_MIN_PERSONS or not counts["persons"]: + raise ValueError(f"{counts['persons']:,} of {well_formed:,} lines of {path} " + f"name a person ({JRC_PERSON}): the file's types have changed") + _refuse_past([(sum(marked.values()), counts["persons"], JRC_MAX_MARKED, + f"person rows of {path} carry a language")], + f"the file no longer marks a plain spelling {JRC_UNMARKED!r}") + entities = Counter(entity for entity, _ in kept) + records = [{"entity": str(entity), "name": name, "script": script(name)} + for entity, name in sorted(kept)] + detail: dict[str, int | str] = { + "sha256": JRC_SHA256, "read_version": JRC_READ_VERSION, "lines": counts["lines"], + "lines_read_past": counts["bad"], "organisation_rows": counts["organisations"], + "person_rows": counts["persons"], + "marked_variants_dropped": sum(marked.values()), + "marked_variants_by_language": " ".join( + f"{lang}:{n}" for lang, n in sorted(marked.items(), key=lambda kv: (-kv[1], kv[0]))), + "repeated_spellings": counts["repeated"], "entities": len(entities), + "entities_with_one_spelling": sum(n == 1 for n in entities.values()), + "spellings_holding_a_replacement_character": sum( + "�" in r["name"] for r in records)} + return Fetched(records, JRC_DATE, detail) + + +def fetch_jrc() -> Fetched: + """One record per distinct spelling of a person entity: its entity + id, the name with its "+" read as spaces (runs of them as one) and + its script(). Read from the gzip cached beside the corpus, + downloaded when missing and again by `fetch --refresh` (a download + that is not the pinned file replaces nothing, _jrc_download), and + refused unless its sha256 is JRC_SHA256's. + + Organisation rows are counted and read past, as are person rows + carrying a language (JRC_UNMARKED says why), by language, and a + spelling an entity repeats. A line that is not JRC_COLUMNS + well-formed columns or not UTF-8 is counted, and past JRC_MAX_BAD + refuses the file; so do a gzip cut short or a file ending inside a + line, fewer person rows than JRC_MIN_PERSONS, and more marked ones + than JRC_MAX_MARKED.""" + path = directory(JRC) / JRC_FILE + if not path.exists(): + _jrc_download(path) + held = _sha256(path) + if held != JRC_SHA256: + raise ValueError(f"{path} has sha256 {held}, not the pinned JRC_SHA256: the " + f"cached file is damaged, or was read under another pin. " + f"Download it again: {fetch_command(JRC, refresh=True)}") + try: + return _jrc_read(path) + except (EOFError, gzip.BadGzipFile, zlib.error) as exc: + raise ValueError(f"{path} is not a whole gzip ({type(exc).__name__}: {exc}); " + f"download it again: {fetch_command(JRC, refresh=True)}") from None + + +def renew_jrc() -> None: + """Download the current file over the cached one, where it is the + pinned file (_jrc_download).""" + _jrc_download(directory(JRC) / JRC_FILE) + + +_check_file = _pinned("sha256", lambda: JRC_SHA256, + "was read from the file with sha256 {held}, and this tool " + "reads {wanted}") +_check_read = _pinned("read_version", lambda: JRC_READ_VERSION, + "was read by read version {held}, and this tool reads " + "version {wanted}") + + +def check_jrc(meta: dict[str, Any]) -> str: + """Why a corpus fetched under another pin, or read another way, is + refused: its file's sha256, then its read version.""" + return _check_file(meta) or _check_read(meta) From 41baecaeabac5f23c6d0650d93d2c5d5d4ad802b Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 24 Oct 2026 13:26:09 -0700 Subject: [PATCH 2/2] Address the PR review of jrc-names and the groups view From the five-part review (code, tests, comments, silent failures, types) of #865: - Reading is a NamedTuple and GROUP_FIELDS is derived from it, so a reordered field cannot mislabel counts; titles and suffixes are read by name, not by offset. - groups counts the persons across two scripts that read alike in no pair of spellings and apart in no one field ("no one field"), lists title and suffix WORDS (a run "Dr. Prof." gives two), says its floor, takes --top of 1 or more, and catches only the corpus's own refusals (Ungroupable, CorpusDamaged), not a ValueError from the parser. - The fetch refuses a file that would shrink in silence: persons left with only marked spellings (JRC_MAX_LOST) and repeated spellings (JRC_MAX_REPEATED). Ids are ASCII digits kept as written, a name's whitespace runs read as one space, and meta.json records the Unicode version script() read, checked on load. - A refresh of the intact pinned file downloads nothing, and a plain fetch adopts a kept download the pin has moved to. - Docs say what is measured: all 20,819 marked rows are dropped on the JRC's marking, 18,594 of them measured as variants; script()'s "none"; the README's worker-process sentence narrowed; figures re-run after a re-fetch under read version 2. Each new guard and rule has a negative control. Co-Authored-By: Claude Opus 5.5 --- tests/v2/test_local_jrc.py | 201 +++++++++++++++++++++++++--- tools/differential/README.md | 10 +- tools/differential/local_corpora.py | 140 +++++++++++++------ tools/differential/local_jrc.py | 162 ++++++++++++++-------- 4 files changed, 392 insertions(+), 121 deletions(-) diff --git a/tests/v2/test_local_jrc.py b/tests/v2/test_local_jrc.py index ae1d5818..df5fd92f 100644 --- a/tests/v2/test_local_jrc.py +++ b/tests/v2/test_local_jrc.py @@ -11,6 +11,7 @@ import gzip import hashlib import importlib +import unicodedata import urllib.request from pathlib import Path @@ -57,11 +58,12 @@ def _cached(monkeypatch: pytest.MonkeyPatch, lines: list[str], **kw: object) -> return _cache(monkeypatch, gzip.compress(_body(lines, **kw))) # type: ignore[arg-type] -def _lenient(monkeypatch: pytest.MonkeyPatch) -> None: - """Shares a fixture of 9 lines can pass: its header is 1 in 9, and - its marked rows 2 in 7.""" - monkeypatch.setattr(jrc, "JRC_MAX_BAD", 1) - monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 1) +def _lenient(monkeypatch: pytest.MonkeyPatch, *, but: str = "") -> None: + """Shares a fixture of 9 lines can pass: its header is 1 in 9, its + marked rows 2 in 7 and its repeat 1 in 7. Every share but `but`.""" + for share in ("JRC_MAX_BAD", "JRC_MAX_MARKED", "JRC_MAX_REPEATED", "JRC_MAX_LOST"): + if share != but: + monkeypatch.setattr(jrc, share, 1) def test_a_jrc_fetch_keeps_each_plain_spelling_once_and_counts_the_rest( @@ -77,10 +79,11 @@ def test_a_jrc_fetch_keeps_each_plain_spelling_once_and_counts_the_rest( assert corpus.meta["data_date"] == jrc.JRC_DATE assert corpus.meta["detail"] == { "sha256": jrc.JRC_SHA256, "read_version": jrc.JRC_READ_VERSION, - "lines": 9, "lines_read_past": 1, + "unicode": unicodedata.unidata_version, "lines": 9, "lines_read_past": 1, "organisation_rows": 1, "person_rows": 7, "marked_variants_dropped": 2, "marked_variants_by_language": "ar:1 ru:1", "repeated_spellings": 1, - "entities": 3, "entities_with_one_spelling": 2, + "entities": 3, "entities_with_marked_spellings_only": 0, + "entities_with_one_spelling": 2, "spellings_holding_a_replacement_character": 0} assert corpus.header().endswith("4 spellings (local corpus jrc-names)") @@ -107,6 +110,45 @@ def test_a_jrc_gzip_that_is_not_whole_names_the_way_out( local.fetch(_JRC) +@pytest.mark.parametrize(("lines", "but", "match"), [ + ([*_LINES, "5\tP\tru\tАнны+Ли"], "JRC_MAX_LOST", "1 of 4 persons .* marked spellings only"), + (_LINES, "JRC_MAX_REPEATED", "1 of 7 person rows .* repeat a spelling"), +], ids=["a person spelled only in marked rows", "a repeated spelling"]) +def test_a_jrc_file_that_would_shrink_in_silence_is_refused( + monkeypatch: pytest.MonkeyPatch, lines: list[str], but: str, match: str) -> None: + _cached(monkeypatch, lines) + _lenient(monkeypatch, but=but) + monkeypatch.setattr(jrc, but, 0) + with pytest.raises(ValueError, match=match): + local.fetch(_JRC) + monkeypatch.setattr(jrc, but, 1 / 4) + detail = local.fetch(_JRC).meta["detail"] + assert (detail["entities_with_marked_spellings_only"], detail["repeated_spellings"]) == ( + (1, 1) if but == "JRC_MAX_LOST" else (0, 1)) + + +def test_a_jrc_file_naming_no_person_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + _cached(monkeypatch, ["#"]) + _lenient(monkeypatch) + with pytest.raises(ValueError, match="0 of 0 lines .* name a person"): + local.fetch(_JRC) + + +def test_a_jrc_spelling_is_read_as_written_but_for_its_spaces( + monkeypatch: pytest.MonkeyPatch) -> None: + # Ids are kept as written ("01" and "1" two entities), runs of "+" + # and spaces are one space, and a replacement character is kept and + # counted. + _cached(monkeypatch, ["1\tP\tu\tAnn+ +Lee", "01\tP\tu\tAnn+Lee", + "2\tP\tu\tBo\ufffd+Wu"]) + _lenient(monkeypatch) + corpus = local.fetch(_JRC) + assert [(r["entity"], r["name"]) for r in corpus.records] == [ + ("01", "Ann Lee"), ("1", "Ann Lee"), ("2", "Bo\ufffd Wu")] + assert corpus.meta["detail"]["spellings_holding_a_replacement_character"] == 1 + assert corpus.meta["detail"]["entities"] == 3 + + def test_a_jrc_file_ending_inside_a_line_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: _cache(monkeypatch, gzip.compress(_body(_LINES)[:-6])) _lenient(monkeypatch) @@ -117,18 +159,20 @@ def test_a_jrc_file_ending_inside_a_line_is_refused(monkeypatch: pytest.MonkeyPa @pytest.mark.parametrize("bad", [ "5\tP\tu", "5\tP\tu\tAnn+Lee\tX", "x5\tP\tu\tAnn+Lee", "5\tQ\tu\tAnn+Lee", "5\tP\tRU\tAnn+Lee", "5\tP\tu\t+", b"5\tP\tu\tPe\xf1a".decode("latin-1"), - "\u00b2\tP\tu\tAnn+Lee"], + "\u0665\tP\tu\tAnn+Lee", "\uff15\tP\tu\tAnn+Lee"], ids=["three columns", "five columns", "an id not digits", "another type", "a language not a code", "an empty name", "not UTF-8", - "an id of a digit not ASCII"]) + "an Arabic-Indic digit id", "a fullwidth digit id"]) def test_a_jrc_line_read_past_is_counted_up_to_its_share( monkeypatch: pytest.MonkeyPatch, bad: str) -> None: # 2 lines of 10 read past: refused at a share under 2/10, read at it. + # The two digit ids are digits to int() and to \\d, and would merge a + # spelling into entity 5's group. lines = [*_LINES, bad] raw = (_body(lines[:-1]) + bad.encode("latin-1" if "\xf1" in bad else "utf-8") + b"\r\n") _cache(monkeypatch, gzip.compress(raw)) - monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 1) + _lenient(monkeypatch, but="JRC_MAX_BAD") monkeypatch.setattr(jrc, "JRC_MAX_BAD", 2 / 10 - 1e-9) with pytest.raises(ValueError, match="2 of 10 lines .* are not 4 well-formed columns"): local.fetch(_JRC) @@ -151,7 +195,7 @@ def test_a_jrc_file_marking_too_many_spellings_is_refused( monkeypatch: pytest.MonkeyPatch) -> None: # 2 of the fixture's 7 person rows carry a language. _cached(monkeypatch, _LINES) - monkeypatch.setattr(jrc, "JRC_MAX_BAD", 1) + _lenient(monkeypatch, but="JRC_MAX_MARKED") monkeypatch.setattr(jrc, "JRC_MAX_MARKED", 2 / 7 - 1e-9) with pytest.raises(ValueError, match="2 of 7 person rows .* carry a language"): local.fetch(_JRC) @@ -215,12 +259,15 @@ def test_a_jrc_refresh_downloads_the_file_again(monkeypatch: pytest.MonkeyPatch) @pytest.mark.parametrize("cached", [True, False], ids=["a refresh", "a first fetch"]) def test_a_jrc_download_that_is_not_the_pinned_file_replaces_nothing( monkeypatch: pytest.MonkeyPatch, cached: bool) -> None: - # The JRC serves only its current file: a refresh writing a newer one - # over the pinned one would leave nothing to rebuild the corpus from. + # The JRC serves only its current file: a download writing a newer + # one over the cached one would leave nothing to rebuild from. (A + # refresh of the pinned file downloads nothing at all; this cached + # file is a damaged one, which the refresh is for.) pinned = gzip.compress(_body(_LINES)) + damaged = pinned[:-20] path = local.directory(_JRC) / jrc.JRC_FILE if cached: - path.write_bytes(pinned) + path.write_bytes(damaged) monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(pinned).hexdigest()) _lenient(monkeypatch) newer = gzip.compress(_body(_LINES[:3])) @@ -228,7 +275,7 @@ def test_a_jrc_download_that_is_not_the_pinned_file_replaces_nothing( with pytest.raises(ValueError, match="not the pinned JRC_SHA256: it is kept at"): local.main(["fetch", _JRC, "--refresh"] if cached else ["fetch", _JRC]) assert path.with_name(jrc.JRC_FILE + ".new").read_bytes() == newer - assert path.read_bytes() == pinned if cached else not path.exists() + assert path.read_bytes() == damaged if cached else not path.exists() def test_a_jrc_download_kept_by_a_mismatch_is_adopted_once_pinned( @@ -241,6 +288,44 @@ def test_a_jrc_download_kept_by_a_mismatch_is_adopted_once_pinned( assert (local.directory(_JRC) / jrc.JRC_FILE).read_bytes() == raw +def test_a_jrc_fetch_adopts_a_kept_download_over_the_old_pinned_file( + monkeypatch: pytest.MonkeyPatch) -> None: + # The pin moved to the download a refresh kept: a plain fetch reads + # it, without calling the old file damaged or downloading again. + _cache(monkeypatch, gzip.compress(_body(_LINES[:3])), pin=False) + raw = gzip.compress(_body(_LINES)) + local.directory(_JRC).joinpath(jrc.JRC_FILE + ".new").write_bytes(raw) + monkeypatch.setattr(jrc, "JRC_SHA256", hashlib.sha256(raw).hexdigest()) + _lenient(monkeypatch) + assert _serve(monkeypatch) == [] and len(local.fetch(_JRC).records) == 4 + assert (local.directory(_JRC) / jrc.JRC_FILE).read_bytes() == raw + + +def test_a_jrc_refresh_of_the_pinned_file_downloads_nothing( + monkeypatch: pytest.MonkeyPatch) -> None: + # The pinned file cannot change, and once the JRC serves another a + # download would fail: a corpus read another way is read again from + # the cached file. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + monkeypatch.setattr(jrc, "JRC_READ_VERSION", jrc.JRC_READ_VERSION + 1) + assert _serve(monkeypatch) == [] + assert local.main(["fetch", _JRC, "--refresh"]) == 0 + assert local.require(_JRC).meta["detail"]["read_version"] == jrc.JRC_READ_VERSION + + +def test_a_jrc_corpus_labelled_under_another_unicode_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch) -> None: + # script() reads the Unicode table this Python carries. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + monkeypatch.setattr(jrc.unicodedata, "unidata_version", "0.0.0") + with pytest.raises(ValueError, match="labelled under Unicode .* reads Unicode 0.0.0"): + local.load(_JRC) + + def test_a_jrc_corpus_read_another_way_is_refused_on_load( monkeypatch: pytest.MonkeyPatch) -> None: # A change to script() would otherwise leave the cached records' @@ -248,8 +333,10 @@ def test_a_jrc_corpus_read_another_way_is_refused_on_load( _cached(monkeypatch, _LINES) _lenient(monkeypatch) local.fetch(_JRC) - monkeypatch.setattr(jrc, "JRC_READ_VERSION", jrc.JRC_READ_VERSION + 1) - with pytest.raises(ValueError, match=f"read by read version 1.*fetch {_JRC} --refresh"): + held = jrc.JRC_READ_VERSION + monkeypatch.setattr(jrc, "JRC_READ_VERSION", held + 1) + with pytest.raises(ValueError, match=f"read by read version {held}, and this tool " + f"reads version {held + 1}.*fetch {_JRC} --refresh"): local.load(_JRC) @@ -290,11 +377,18 @@ def _spelled(*spellings: tuple[str, str]) -> object: for entity, name in spellings]) +def test_a_reading_names_its_fields_as_the_report_does() -> None: + assert local.reading("Ann Lee").family_at == "last" + assert local.GROUP_FIELDS == ("title", "suffix", "given", "family", + "family words", "family at") + + def test_a_reading_is_the_structure_of_the_parse_never_its_words() -> None: assert local.reading("Ann Lee") == (False, False, True, True, 1, "last") assert local.reading("Dr. Ann de la Lee Jr.") == (True, True, True, True, 3, "last") assert local.reading("Lee, Ann") == (False, False, True, True, 1, "first") assert local.reading("Ann") == (False, False, True, False, 0, "none") + assert local.reading("de Mesnil") == (False, False, False, True, 2, "whole") assert local.reading("Анна Ли") == local.reading("Ann Lee") @@ -313,11 +407,38 @@ def test_the_groups_view_counts_persons_by_the_scripts_of_their_spellings() -> N def test_the_groups_view_finds_two_scripts_that_read_a_field_apart() -> None: - # Every Latin spelling reads a title and no Cyrillic one does. + # Every Latin spelling reads a title and no Cyrillic one does; the two + # titled spellings count the person once. found = local.group_counts(_spelled(("1", "Dr. Ann Lee"), ("1", "Dr Ann Lee"), ("1", "Анна Ли"))) assert found.differ == {("any", "title"): 1, ("Cyrillic / Latin", "title"): 1} assert found.agree == {"Latin / Latin": 1} + assert found.titles == {"dr": 1} + + +def test_the_groups_view_lists_no_title_every_spelling_reads() -> None: + # Two titles, two suffixes, and no spelling without: no lead, since + # the vocabulary already reads each spelling's word. + found = local.group_counts(_spelled(("1", "Dr. Ann Lee"), ("1", "Prof. Ann Lee"), + ("2", "Bo Wu Jr."), ("2", "Bo Wu Sr."))) + assert found.titles == {} and found.suffixes == {} and found.differ == {} + + +def test_the_groups_view_lists_a_title_run_word_by_word() -> None: + found = local.group_counts(_spelled(("1", "Dr. Prof. Ann Lee"), ("1", "Ann Lee"))) + assert found.titles == {"dr": 1, "prof": 1} + + +def test_the_groups_view_pairs_every_two_of_a_person_s_scripts() -> None: + # The Arabic spelling reads a title the others do not: only its two + # pairs differ. A spelling of two scripts keys a pair as "mixed". + found = local.group_counts(_spelled( + ("1", "Ann Lee"), ("1", "Анна Ли"), ("1", "الدكتور أحمد علي"), ("1", "Ann Ли"))) + assert local.reading("الدكتور أحمد علي").title + assert found.groups == {"any": 1, "Arabic / Cyrillic": 1, "Arabic / Latin": 1, + "Arabic / mixed": 1, "Cyrillic / Latin": 1, + "Cyrillic / mixed": 1, "Latin / mixed": 1} + assert found.agree == {"Cyrillic / Latin": 1, "Cyrillic / mixed": 1, "Latin / mixed": 1} def test_two_scripts_read_alike_only_where_one_spelling_matches_one_whole() -> None: @@ -330,6 +451,9 @@ def test_two_scripts_read_alike_only_where_one_spelling_matches_one_whole() -> N assert ("Cyrillic / Latin", "title") not in found.differ assert ("Cyrillic / Latin", "suffix") not in found.differ assert "Cyrillic / Latin" not in found.agree + # Counted apart, so that the pair's persons are each alike, apart in + # a field, or here. + assert found.differ[("Cyrillic / Latin", local.NO_ONE_FIELD)] == 1 def test_the_groups_view_counts_a_person_once_however_many_spellings_differ() -> None: @@ -343,6 +467,17 @@ def test_the_groups_view_counts_a_person_once_however_many_spellings_differ() -> assert found.titles == {"dr": 1} and found.suffixes == {"jr": 1} +def test_the_groups_report_gives_each_share_of_a_pair_s_persons( + capsys: pytest.CaptureFixture[str]) -> None: + rows = [("1", "Ann Lee"), ("1", "Dr. Ann Lee"), ("2", "Bo Wu"), ("2", "Bo Wu"), + ("2", "Bo Wu"), ("3", "Cy Ng"), ("3", "Сай Нг")] + print(local.format_groups(local.group_counts(_spelled(*rows)), 1)) + out = capsys.readouterr().out + assert " 2 50.0% alike Latin / Latin; differ in title 50.0%" in out + # top=1: "any" and the largest pair only. + assert "Cyrillic / Latin" not in out + + def test_the_groups_view_lists_only_titles_many_persons_share( capsys: pytest.CaptureFixture[str]) -> None: rows = [(str(i), name) for i in range(10) for name in ("Ann Lee", "Dr. Ann Lee")] @@ -350,7 +485,7 @@ def test_the_groups_view_lists_only_titles_many_persons_share( print(local.format_groups(local.group_counts(_spelled(*rows)), 5)) out = capsys.readouterr().out assert out.startswith("11 persons with two or more spellings (22 spellings)") - assert "titles read in some spellings only: dr 10" in out + assert "title words read in some spellings only (10 persons or more): dr 10" in out assert "prof" not in out @@ -377,6 +512,34 @@ def test_the_groups_command_reports_a_fetched_corpus( assert "100.0% alike Cyrillic / Latin" in out +def test_the_groups_command_does_not_hide_a_parser_error_as_damage( + monkeypatch: pytest.MonkeyPatch) -> None: + # Only the corpus's own refusals become a one-line exit: a ValueError + # from parsing a name is a bug, and keeps its traceback. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + + def broken(name: str) -> None: + raise ValueError("a parser bug") + monkeypatch.setattr(local, "_read", broken) + with pytest.raises(ValueError, match="a parser bug"): + local.main(["groups", _JRC]) + + +@pytest.mark.parametrize("top", ["0", "-1"]) +def test_the_groups_command_refuses_a_top_under_one( + monkeypatch: pytest.MonkeyPatch, top: str) -> None: + # A fetched corpus, so that the refusal is the argument's (argparse + # exits 2), not the missing corpus's. + _cached(monkeypatch, _LINES) + _lenient(monkeypatch) + local.fetch(_JRC) + with pytest.raises(SystemExit) as refused: + local.main(["groups", _JRC, "--top", top]) + assert refused.value.code == 2 + + def test_the_groups_command_refuses_a_source_with_no_entities( monkeypatch: pytest.MonkeyPatch) -> None: plant_corpus(monkeypatch, local, "ukec-individuals", [{"name": "Ann Lee"}], diff --git a/tools/differential/README.md b/tools/differential/README.md index 4e9ee477..305558c5 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -235,7 +235,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`; `PSC_EXTRACT_VERSION` does it for extracts): the sha256 checks prove the files are the ones fetched, not that today's code would write them, and the review that added it found a changed label read under the old records with no warning. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -251,7 +251,7 @@ uv run python tools/differential/snapshot.py --local ctgov-officials --record -- uv run python tools/differential/snapshot.py --local ctgov-officials ``` -Any reading of 5,000 entries or more, committed or local, parses in a worker process per CPU (`--jobs N` to change that), and a local reading does not read the facade, whose v1 compatibility is the committed corpora's question. Measured 2026-10-23 on a 12-CPU machine: a 20,000-name sample cost about 120 µs a name for the 2.0 parse and its renders, and the facade about as much again; a check of `fec-individuals` (1,893,964 names) took 49s, and of `ctgov-officials` (352,104) 9.6s, where one process reading the facade too would take about 7.5 minutes and 85s, estimated from that sample's per-name cost. A local recording is kept gzipped in `baselines/` beside the snapshot (41 MB for `fec-individuals` against 249 MB plain, 2026-10-23), under a key of the fetch, a hash of every file in the read tree's `nameparser/` as it is on disk (`__pycache__` aside), the interpreter's minor version and the code that writes a row (`snapshot.py`, `corpus_build.py`, `shapes.py`); recording that base again copies it rather than parsing. The hash is taken again after the read, and a package that changed meanwhile keeps no baseline; a kept baseline that cannot be read, or holds another number of rows than the corpus has entries, is deleted and the names are read. The four most recently recorded or reused baselines are kept per source, whole and sampled alike, so a few `--sample` recordings can push out a whole one. A snapshot recorded before 2026-10-23 may carry facade rows; a check against one sets them aside and says so. `--sample N` reads a fixed 1-in-N sample, chosen by a hash of the name, into `snapshot.sample-N.jsonl`, for iterating on a change; a sample is never compared with a snapshot of the whole corpus or of another sample, and a count quoted from one says it is a sample. +Any snapshot reading of 5,000 entries or more, committed or local, parses in a worker process per CPU (the `pairs` and `groups` views parse in one) (`--jobs N` to change that), and a local reading does not read the facade, whose v1 compatibility is the committed corpora's question. Measured 2026-10-23 on a 12-CPU machine: a 20,000-name sample cost about 120 µs a name for the 2.0 parse and its renders, and the facade about as much again; a check of `fec-individuals` (1,893,964 names) took 49s, and of `ctgov-officials` (352,104) 9.6s, where one process reading the facade too would take about 7.5 minutes and 85s, estimated from that sample's per-name cost. A local recording is kept gzipped in `baselines/` beside the snapshot (41 MB for `fec-individuals` against 249 MB plain, 2026-10-23), under a key of the fetch, a hash of every file in the read tree's `nameparser/` as it is on disk (`__pycache__` aside), the interpreter's minor version and the code that writes a row (`snapshot.py`, `corpus_build.py`, `shapes.py`); recording that base again copies it rather than parsing. The hash is taken again after the read, and a package that changed meanwhile keeps no baseline; a kept baseline that cannot be read, or holds another number of rows than the corpus has entries, is deleted and the names are read. The four most recently recorded or reused baselines are kept per source, whole and sampled alike, so a few `--sample` recordings can push out a whole one. A snapshot recorded before 2026-10-23 may carry facade rows; a check against one sets them aside and says so. `--sample N` reads a fixed 1-in-N sample, chosen by a hash of the name, into `snapshot.sample-N.jsonl`, for iterating on a change; a sample is never compared with a snapshot of the whole corpus or of another sample, and a count quoted from one says it is a sample. `compare.py --corpus` takes the cached `corpus.jsonl` too, against a release. @@ -279,15 +279,15 @@ uv run python tools/differential/local_corpora.py pairs tse-donors --agreement On the fetch of 2026-10-22, 205,925 of `tse-donors`' 854,531 pairs differ. The commonest shapes: words left out (22.0%: `DE` left out of 13,412 pairs, `DA` 5,906, `DOS` 3,033, then second surnames, `SILVA` 2,869), accents only (21.1%), words shortened (15.8%), words misspelled (14.9%) and words left out and shortened (11.8%). Read the agreement per shape, since a shape can change the surname legitimately: a pair that leaves out the last surname ends the family in another word, which is likely most of why only 44% of words left out agree (not measured apart from the other causes). Accents only cannot change the surname, so its 1,304 of 43,552 pairs that disagree are the parser reading one name two ways: by their last words, `JÚNIOR` against `JUNIOR` in 866 (#843) and `SÁ` against `SA` in 255, where `MARIA DE SA` reads `SA` as a credential and `MARIA DE SÁ` as the surname. The run took 39s. -`jrc-names` reads a gzip downloaded once into the cache (`fetch --refresh` downloads it again), and refuses one whose sha256 is not `JRC_SHA256`: the JRC says the resource is updated daily, but the file served on 2026-10-24 was last modified on 2024-03-15, so a new file is measured and the pin moved by hand rather than read under constants measured on another. A download that is not the pinned file replaces nothing: it is kept beside the cached one as `entities.gzip.new`, to be measured, and adopted without another download once the pin moves to it, since the JRC serves only its current file and the pinned one could not be fetched again. `meta.json` records the pin and `JRC_READ_VERSION`, and a corpus fetched under another of either is refused on load, so a change to how a spelling is read or its script labelled is not counted under the old records. It keeps the person rows (`P`, 1,338,463 of 1,445,013 lines) and reads past the organisations (`O`). A row the JRC marks with a language (`ru`, `ar`, `sl`, `et`, `pl` and others, 20,819 of the person rows) is a morphological variant of a spelling it leaves unmarked (`u`): an inflected form, or an Arabic spelling with و, ل or ال glued on. Every marked row's entity held an unmarked spelling, and 18,594 of the 20,819 had as many words as one and differed from it only in endings or in glued fronts of three letters or fewer (the rule is stated exactly at `JRC_UNMARKED` in `local_jrc.py`); the fetch drops them, counted by language in `meta.json`'s detail. The unmarked spellings still hold inflected forms the JRC did not mark (by the same rule, 22,732 of the 66,673 distinct unmarked Cyrillic spellings are another Cyrillic spelling of the same person), which are kept. Words joined by `+` are read as spaces. A line that is not four well-formed columns (the file's first line, `#`, is one) is counted, and past `JRC_MAX_BAD` refuses the file, as do a gzip cut short, a file ending inside a line, fewer person rows than `JRC_MIN_PERSONS` (a type renamed) and more marked rows than `JRC_MAX_MARKED` (a file that marks every spelling). The licence bars redistributing the resource or anything derived from it, so its snapshot, its views' output and every file derived from it stay in the cache, and what is cited from it is counts and shapes written with made-up names. A snapshot of it (`snapshot.py --local jrc-names`) recorded 1,315,051 names in 30s on 12 CPUs (2026-10-24, 128 MB). +`jrc-names` reads a gzip downloaded once into the cache, and refuses one whose sha256 is not `JRC_SHA256`: the JRC says the resource is updated daily, but the file served on 2026-10-24 was last modified on 2024-03-15, so a new file is measured and the pin moved by hand rather than read under constants measured on another. A download that is not the pinned file replaces nothing: it is kept beside the cached one as `entities.gzip.new`, to be measured, and adopted without another download once the pin moves to it, since the JRC serves only its current file and the pinned one could not be fetched again. For the same reason `fetch --refresh` downloads only where the cached file is not the pinned one; otherwise it reads the cached file again. `meta.json` records the pin, `JRC_READ_VERSION` and the Unicode version `script()` read, and a corpus fetched under another of any is refused on load, so a change to how a spelling is read or its script labelled is not counted under the old records. It keeps the person rows (`P`, 1,338,463 of 1,445,013 lines) and reads past the organisations (`O`). It drops every row the JRC marks with a language (`ru`, `ar`, `sl`, `et`, `pl` and others, 20,819 of the person rows), counted by language in `meta.json`'s detail, on the JRC's marking alone. Where measured, a marked row is a morphological variant of a spelling the JRC leaves unmarked (`u`): an inflected form, or an Arabic spelling with و, ل or ال glued on. Every marked row's entity held an unmarked spelling, and 18,594 of the 20,819 had as many words as one and differed from it only in endings or in glued fronts of three letters or fewer (the rule is stated exactly at `JRC_UNMARKED` in `local_jrc.py`); the other 2,225, 1,680 of them Arabic, fit neither as stated and were not measured further. The unmarked spellings still hold inflected forms the JRC did not mark (by the same rule, 22,732 of the 66,673 distinct unmarked Cyrillic spellings are another Cyrillic spelling of the same person), which are kept. A name's `+` and whitespace are read as single spaces, and an id is kept as written. A line that is not four well-formed columns (the file's first line, `#`, is one) is counted, and past `JRC_MAX_BAD` refuses the file, as do a gzip cut short, a file ending inside a line, person rows under `JRC_MIN_PERSONS` of the lines (a type renamed), more marked rows than `JRC_MAX_MARKED` (a file that marks every spelling), more persons left with only marked spellings than `JRC_MAX_LOST` (none were), and more repeated spellings than `JRC_MAX_REPEATED` (58 were). 99 spellings hold a replacement character (U+FFFD), damage in the source; they are kept and counted. The licence bars redistributing the resource or anything derived from it, so its snapshot, its views' output and every file derived from it stay in the cache, and what is cited from it is counts and shapes written with made-up names. A snapshot of it (`snapshot.py --local jrc-names`) recorded 1,315,051 names in 30s on 12 CPUs (2026-10-24, 128 MB). -`groups` reads a source whose records carry an entity and a script (`jrc-names`) as several spellings of one person. It parses every spelling of each person with two or more and compares the readings' structure: whether a title, a suffix, a given name and a family name are read, how many words the family is, and where it stands among the name's words (first, last, all of them, inside, or nowhere). Words are not compared, so spellings in two scripts can be; but where the family stands moves with a script's order, so a script read family first (Hangul, spaced Han) differs from one read given first in that field by convention, not by error. It counts PERSONS, not pairs, so a person spelled 600 ways weighs what one spelled twice does. A pair of scripts is one row: within one script (`Latin / Latin`) a field differs where the person's spellings in it read it more than one way, and they read alike where all read one way; across two (`Cyrillic / Latin`) a field differs where no spelling in one reads it as any in the other does, so two Latin spellings that differ between themselves are not counted against the Cyrillic one, and they read alike where some spelling in one reads, in every field, as some spelling in the other. It then lists the titles and suffixes of the persons some of whose spellings read one and others none (two spellings with two different titles and no untitled one add nothing), by the persons sharing one, ten at least: the lead is a word the vocabulary reads as a title or suffix that is a name in some spelling, or a title spelled in a way it does not know. +`groups` reads a source whose records carry an entity and a script (`jrc-names`) as several spellings of one person. It parses every spelling of each person with two or more and compares the readings' structure: whether a title, a suffix, a given name and a family name are read, how many words the family is, and where it stands among the given, middle and family words (one run at the front or the end, all of them, none, or otherwise, which 1 of the 1,315,051 distinct spellings is). Words are not compared, so spellings in two scripts can be; but where the family stands moves with a script's order, so a script read family first (Hangul, spaced Han) differs from one read given first in that field by convention, not by error. It counts PERSONS, not pairs, so a person spelled 600 ways weighs what one spelled twice does. A pair of scripts is one row, and a spelling whose letters are in two scripts keys one as `mixed`: within one script (`Latin / Latin`) a field differs where the person's spellings in it read it more than one way, and they read alike where all read one way; across two (`Cyrillic / Latin`) a field differs where no spelling in one reads it as any in the other does, so two Latin spellings that differ between themselves are not counted against the Cyrillic one, and they read alike where some spelling in one reads, in every field, as some spelling in the other. A person across two scripts can be neither: every field has a value in common, yet no one spelling matches; the report counts those as `no one field` (0.4% of Han / Latin, less elsewhere), so each pair's persons are alike, apart in a field, or there. It then lists the title and suffix words of the persons some of whose spellings read one and others none (two spellings with two different titles and no untitled one add nothing), by the persons sharing a word, ten at least: the lead is a word the vocabulary reads as a title or suffix that is a name in some spelling, or a title spelled in a way it does not know. ``` uv run python tools/differential/local_corpora.py groups jrc-names ``` -On the fetch of 2026-10-24 (data dated 2024-03-15), 93.7% of the 218,822 persons with two or more spellings read alike in every spelling, and the run took 50s. Within a script the disagreement is mostly the writing, not the parse. Katakana / Katakana reads alike for 0.2% of its 4,289 persons because the JRC writes a transliterated name three ways: unspaced and undivided, which the parser cannot divide and reads as a given name alone (4,347 of the 13,039 Katakana spellings), divided by `・` with no space (4,337), and spaced (4,355), the last two read given then family. Han / Han, 2.2% of 2,420, holds unspaced Han, read as one family word, beside spaced Han, read family first, and Han divided by `·`, read in its written order. Arabic / Arabic reads alike for 67.8% of 3,329 persons, and the Arabic comma is most of why: each of the 736 persons with a spelling holding one differs, of the 1,072 that differ, since the comma moves the family to the front. Across scripts read in the same order most persons read alike (Cyrillic / Latin 99.3% of 20,837, Arabic / Latin 98.2% of 5,347, Greek / Latin 98.6% of 915); Hangul against a script read given first reads alike for 3% or fewer in every such row `--top 40` lists (Hangul / Latin 2.0% of 1,247), its family standing first, while Han / Hangul, both read family first, is 87.1% of 1,002. The titles read in some spellings only start `sheikh` 50, `bard` 33, `shaikh` 27, `executive` 22 and include `mahdi` 17, a common given name; the suffixes `jr` 62, `sa` 30, `ii` 27. +On the fetch of 2026-10-24 (data dated 2024-03-15), 93.7% of the 218,822 persons with two or more spellings read alike in every spelling, and the run took 50s. Within a script the disagreement is mostly the writing, not the parse. Katakana / Katakana reads alike for 0.2% of its 4,289 persons because the JRC writes a transliterated name three ways: unspaced and undivided, which the parser cannot divide and reads as a given name alone (4,347 of the 13,039 Katakana spellings), divided by `・` with no space (4,337), and spaced (4,355), the last two read given then family. Han / Han, 2.2% of 2,420, holds unspaced Han, read as one family word, beside spaced Han, read family first, and Han divided by `·`, read in its written order. Arabic / Arabic reads alike for 67.8% of 3,329 persons, and the Arabic comma is most of why: each of the 736 persons with a spelling holding one differs, of the 1,072 that differ, since the comma moves the family to the front. Across scripts read in the same order most persons read alike (Cyrillic / Latin 99.3% of 20,837, Arabic / Latin 98.2% of 5,347, Greek / Latin 98.6% of 915); Hangul against a script read given first reads alike for 3% or fewer in every such row `--top 40` lists (Hangul / Latin 2.0% of 1,247), its family standing first, while Han / Hangul, both read family first, is 87.1% of 1,002. The title words read in some spellings only start `sheikh` 50, `bard` 33, `shaikh` 27, `sir` 23, `executive` 22 and include `mahdi` 17, a common given name; the suffix words `jr` 62, `sa` 38, `iii` 27, `ii` 27. ## Corpus provenance diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index d6372cf2..8d61fe83 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -74,7 +74,7 @@ from collections.abc import Callable, Iterable from dataclasses import dataclass, field from pathlib import Path -from typing import Any +from typing import Any, Literal, NamedTuple # The shared fetch and cache helpers (corpus_build.py, local_core.py) and # the source modules (local_.py), found by putting this directory @@ -526,25 +526,48 @@ def pair_counts(local: Local, *, agreement: bool = False) -> PairCounts: if agreement else None, accent_splits) +#: Where reading places the family, as Reading.family_at. +FamilyAt = Literal["first", "last", "whole", "inner", "none"] + + +class Reading(NamedTuple): + """The structure parse() gives one spelling (reading): counts and + positions, never the words, so spellings in two scripts compare.""" + title: bool + suffix: bool + given: bool + family: bool + family_words: int + #: Where the family stands among the given, middle and family words. + family_at: FamilyAt + + #: What the groups view compares between two spellings of one person, -#: in the order it reports them: whether parse() reads a title, a suffix, -#: a given name and a family name, how many words the family is, and -#: where it stands among the name's words (reading). -GROUP_FIELDS = ("title", "suffix", "given", "family", "family words", "family at") +#: in the order it reports them: Reading's fields, as the report names +#: them. +GROUP_FIELDS = tuple(name.replace("_", " ") for name in Reading._fields) #: The name-bearing roles, among which reading places the family. _NAME_ROLES = frozenset({"given", "middle", "family"}) +#: What group_counts counts, beside GROUP_FIELDS, for two scripts whose +#: spellings read alike in no one pair, though every field has a value +#: in common across them: a combination of fields reads apart. +NO_ONE_FIELD = "no one field" -Reading = tuple[bool, bool, bool, bool, int, str] +class _Read(NamedTuple): + """reading(name), and the title and the suffix parse() reads.""" + reading: Reading + title: str + suffix: str -def _read(name: str) -> tuple[Reading, str, str]: - """reading(name), and the title and the suffix parse() reads.""" +def _read(name: str) -> _Read: from nameparser import parse parsed = parse(name) roles = [str(t.role) for t in parsed.tokens if str(t.role) in _NAME_ROLES] family = [i for i, role in enumerate(roles) if role == "family"] + at: FamilyAt if not family: at = "none" elif len(family) == len(roles): @@ -555,19 +578,22 @@ def _read(name: str) -> tuple[Reading, str, str]: at = "last" else: at = "inner" - return ((bool(parsed.title), bool(parsed.suffix), bool(parsed.given), - bool(parsed.family), len(parsed.family.split()), at), - parsed.title, parsed.suffix) + return _Read(Reading(bool(parsed.title), bool(parsed.suffix), bool(parsed.given), + bool(parsed.family), len(parsed.family.split()), at), + parsed.title, parsed.suffix) def reading(name: str) -> Reading: """The structure parse() gives `name`, one value per GROUP_FIELDS: title read, suffix read, given found, family found, the family's words, and where the family stands among the given, middle and - family words: "first", "last", "whole" (it is all of them), "inner" - or "none". Counts and positions, never the words themselves, so two - spellings in different scripts compare.""" - return _read(name)[0] + family words (titles, suffixes, nicknames and maiden names left + out): "first" or "last" where it is one run at that edge, "whole" + where it is all of them, "none" where there is none, and "inner" + otherwise, a run inside or a family split in two: rare (1 of the + 1,315,051 distinct jrc-names spellings on 2026-10-24), and kept so + that such a family is not labelled an edge.""" + return _read(name).reading def _differing(one: set[Reading], other: set[Reading] | None = None) -> set[str]: @@ -583,17 +609,25 @@ def _differing(one: set[Reading], other: set[Reading] | None = None) -> set[str] if {r[i] for r in one}.isdisjoint({r[i] for r in other})} +class Ungroupable(ValueError): + """A corpus group_counts cannot read: no records, or records with no + entity and script. A class of its own, as CorpusDamaged is, so that + the command catches these and not a ValueError from the parser.""" + + @dataclass(frozen=True) class GroupCounts: """What group_counts finds, every count a count of PERSONS, not of spellings or of pairs, so that a person spelled 600 ways weighs what one spelled twice does. Keyed by a pair of scripts ("Cyrillic / Latin", "Latin / Latin"), and by "any" for all of a person's - spellings taken together.""" + spellings taken together; a script() label that is no script + ("mixed", "none") keys a pair like one.""" #: The persons with two or more spellings, and those spellings. persons: int spellings: int - #: pair -> the persons with two spellings in those scripts. + #: pair -> the persons with two spellings in those scripts ("any": + #: every person, so groups["any"] == persons). groups: Counter[str] #: pair -> those whose spellings read alike: within one script, all #: of them read one way; across two, some spelling in one reads, in @@ -601,12 +635,15 @@ class GroupCounts: agree: Counter[str] #: (pair, a GROUP_FIELDS member) -> those whose spellings disagree in #: the field (_differing): within one script's for "Latin / Latin", - #: between the two scripts' for "Cyrillic / Latin". + #: between the two scripts' for "Cyrillic / Latin". (pair, + #: NO_ONE_FIELD) -> those across two scripts that read alike in no + #: pair of spellings yet in no one field apart, so that a pair's + #: persons are each alike, apart in a field, or here. differ: Counter[tuple[str, str]] - #: The titles, and the suffixes, of a person some of whose spellings - #: read one and others none, folded by word_key, counted once a - #: person: two spellings reading two titles (Sheikh, Shaikh) with no - #: untitled spelling beside them add nothing. + #: The words of the titles, and of the suffixes, of a person some of + #: whose spellings read one and others none, folded by word_key, + #: counted once a person: two spellings reading two titles (Sheikh, + #: Shaikh) with no untitled spelling beside them add nothing. titles: Counter[str] suffixes: Counter[str] @@ -615,17 +652,18 @@ def group_counts(local: Local) -> GroupCounts: """Each person with two or more spellings, from a source whose records carry an "entity" and a "script": every spelling's reading compared by the pair of scripts the spellings are written in, as - _differing says. A distinct name is parsed once. A source without - entities raises ValueError.""" + _differing says. A distinct name is parsed once, in this process (a + minute for jrc-names). A source without entities raises + Ungroupable.""" if not local.records: - raise ValueError(f"{local.source} holds no records") + raise Ungroupable(f"{local.source} holds no records") if any("entity" not in r or "script" not in r for r in local.records): - raise ValueError(f"{local.source}'s records carry no entity and script " - f"to group spellings by") + raise Ungroupable(f"{local.source}'s records carry no entity and script " + f"to group spellings by") by_entity: dict[str, list[tuple[str, str]]] = {} for record in local.records: by_entity.setdefault(record["entity"], []).append((record["script"], record["name"])) - read: dict[str, tuple[Reading, str, str]] = {} + read: dict[str, _Read] = {} groups: Counter[str] = Counter() agree: Counter[str] = Counter() differ: Counter[tuple[str, str]] = Counter() @@ -642,7 +680,7 @@ def group_counts(local: Local) -> GroupCounts: for script, name in variants: if name not in read: read[name] = _read(name) - by_script.setdefault(script, set()).add(read[name][0]) + by_script.setdefault(script, set()).add(read[name].reading) written[script] += 1 everything = set().union(*by_script.values()) # pair -> (the fields read apart, whether the spellings read alike) @@ -661,11 +699,13 @@ def group_counts(local: Local) -> GroupCounts: groups[pair] += 1 if alike: agree[pair] += 1 - differ.update((pair, field) for field in fields) - for counter, at in ((titles, 0), (suffixes, 1)): - if len({r[at] for r in everything}) > 1: - counter.update({word_key(read[name][1 + at]) for _, name in variants - if read[name][0][at]} - {""}) + elif not fields: + differ[(pair, NO_ONE_FIELD)] += 1 + differ.update((pair, f) for f in fields) + for counter, mark in ((titles, "title"), (suffixes, "suffix")): + if len({getattr(r, mark) for r in everything}) > 1: + counter.update({key for _, name in variants if getattr(read[name].reading, mark) + for key in words(getattr(read[name], mark))}) return GroupCounts(persons, spellings, groups, agree, differ, titles, suffixes) @@ -673,9 +713,11 @@ def format_groups(found: GroupCounts, top: int) -> str: """The report `groups` prints: per pair of scripts, the persons with two spellings in them, the share whose spellings read alike (GroupCounts.agree), and the share whose spellings read each field - apart (_differing); then the titles and suffixes read in some of a - person's spellings and in none of the others, listed only where - PAIR_WORD_FLOOR persons share one. Counts and words only.""" + apart (_differing), then NO_ONE_FIELD's; then the title and suffix + words read in some of a person's spellings and in none of the + others, listed only where PAIR_WORD_FLOOR persons share one. The + `top` largest pairs are listed, and the `top` commonest words. + Counts and words only.""" lines = [f"{found.persons:,} persons with two or more spellings " f"({found.spellings:,} spellings); by the scripts of two of them:"] pairs = ["any", *sorted((p for p in found.groups if p != "any"), @@ -685,14 +727,16 @@ def format_groups(found: GroupCounts, top: int) -> str: if not n: continue shares = ", ".join(f"{field} {found.differ[(pair, field)] / n:.1%}" - for field in GROUP_FIELDS if found.differ[(pair, field)]) + for field in (*GROUP_FIELDS, NO_ONE_FIELD) + if found.differ[(pair, field)]) lines.append(f" {n:>9,} {found.agree[pair] / n:6.1%} alike {pair}" + (f"; differ in {shares}" if shares else "")) - for words, what in ((found.titles, "titles read in some spellings only"), - (found.suffixes, "suffixes read in some spellings only")): - listed = [(w, n) for w, n in words.most_common(top) if n >= PAIR_WORD_FLOOR] + for counted, what in ((found.titles, "title words read in some spellings only"), + (found.suffixes, "suffix words read in some spellings only")): + listed = [(w, n) for w, n in counted.most_common(top) if n >= PAIR_WORD_FLOOR] if listed: - lines.append(f"{what}: " + ", ".join(f"{w} {n:,}" for w, n in listed)) + lines.append(f"{what} ({PAIR_WORD_FLOOR} persons or more): " + + ", ".join(f"{w} {n:,}" for w, n in listed)) return "\n".join(lines) @@ -933,6 +977,14 @@ def lookup(local: Local, word: str) -> Lookup | None: # -- the command line ---------------------------------------------------------- +def _positive(text: str) -> int: + """An argparse type: a whole number of 1 or more.""" + value = int(text) + if value < 1: + raise argparse.ArgumentTypeError(f"{text} is not 1 or more") + return value + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) sub = ap.add_subparsers(dest="command", required=True) @@ -954,7 +1006,7 @@ def main(argv: list[str] | None = None) -> int: group = sub.add_parser("groups", help="compare the readings of each person's " "spellings, by the scripts they are in") group.add_argument("source", choices=sorted(SOURCES)) - group.add_argument("--top", type=int, default=20, metavar="N", + group.add_argument("--top", type=_positive, default=20, metavar="N", help="pairs of scripts, and titles and suffixes, to list " "(default 20)") args = ap.parse_args(argv) @@ -992,7 +1044,7 @@ def main(argv: list[str] | None = None) -> int: if args.command == "groups": try: grouped = group_counts(local) - except ValueError as exc: # CorpusDamaged included: the records, read here first + except (CorpusDamaged, Ungroupable) as exc: # the records, read here first raise SystemExit(str(exc)) from None print(local.header()) print(format_groups(grouped, args.top)) diff --git a/tools/differential/local_jrc.py b/tools/differential/local_jrc.py index b0d619fe..119b832c 100644 --- a/tools/differential/local_jrc.py +++ b/tools/differential/local_jrc.py @@ -12,7 +12,7 @@ Arabic-script names: 17,074 spellings of 7,833 persons, 5,347 of them spelled in Latin too, in the file dated 2024-03-15. Latin is most of it (1,203,634 of 1,317,586 spellings, then -Cyrillic 66,673, Katakana 13,039 and Han 8,016); the Indic scripts are +Cyrillic 66,673, Arabic 17,074, Katakana 13,039 and Han 8,016); the Indic scripts are few (Devanagari 827 spellings, Bengali 419, Tamil 328, Malayalam 183, the rest under 80 each). 1,510 of the Arabic spellings hold an Arabic comma (U+060C), which the parser reads as a comma. @@ -53,28 +53,33 @@ JRC_SHA256 = "43144cee881ba8a81573ae28c089dffce0ca634d342791414628a5d54997a7d0" #: The pinned file's Last-Modified, its data date: the file states none. JRC_DATE = "2024-03-15T14:08:28Z" -#: How the file is read into records, stored in meta.json's detail: a -#: change to what _jrc_read keeps or to script() bumps it, so that a -#: corpus read the old way is refused on load (check_jrc) rather than -#: counted under labels the code no longer gives. -JRC_READ_VERSION = 1 +#: How the file is read into records, stored in meta.json's detail with +#: the Unicode version script() read: a change to what _jrc_read keeps, +#: to how it builds a name or to script() must bump it by hand, and a +#: corpus read the old way is then refused on load (check_jrc) rather +#: than counted under labels the code no longer gives. +JRC_READ_VERSION = 2 #: A line: entity id, type, language and name, tab-separated, the name's #: words joined by "+". Lines end in CRLF. JRC_COLUMNS = 4 JRC_PERSON, JRC_ORGANISATION = "P", "O" #: The language of a spelling the JRC did not mark: 1,317,644 of the -#: 1,338,463 person rows. A row with any other language is a -#: morphological variant of one that has it: an inflected form (a -#: Russian name in another case, a Slovene or Polish one with its case -#: ending) or an Arabic one with a word glued to it (و "and", ل "for", -#: ال "the"). Of the 20,819 marked rows in the pinned file, every one's -#: entity also held an unmarked spelling, and 18,594 had the same number -#: of words as one and, word by word in order, either every word the -#: same or sharing at least its first letter with the other and no more -#: than three letters past their common start (an ending), or every word -#: the other with no more than three letters glued to its front (a -#: prefix); so the fetch drops them, counted by language, -#: rather than read an ending or a conjunction as part of a name. The +#: 1,338,463 person rows. A row the JRC marks with a language is, where +#: measured, a morphological variant of an unmarked spelling of the same +#: entity: an inflected form (a Russian name in another case, a Slovene +#: or Polish one with its case ending) or an Arabic one with a word glued +#: to it (و "and", ل "for", ال "the"). Of the 20,819 marked rows in the +#: pinned file, every one's entity also held an unmarked spelling, and +#: 18,594 had the same number of words as one and, word by word in +#: order, either every word the same or sharing at least its first +#: letter with the other and no more than three letters past their +#: common start (an ending), or every word the other with no more than +#: three letters glued to its front (a prefix). The other 2,225 (1,680 of +#: them Arabic) fit neither as stated and were not measured further. The +#: fetch drops every marked row, all 20,819, on the JRC's marking alone, +#: counted by language, rather than read an ending or a conjunction as +#: part of a name, and counts the entities left with no spelling +#: (JRC_MAX_LOST). The #: unmarked spellings still hold some inflected forms the JRC did not #: mark: 22,732 of the 66,673 distinct unmarked spellings whose script() #: is Cyrillic are, by the ending rule above, another Cyrillic spelling @@ -96,6 +101,14 @@ #: refused: 20,819 of 1,338,463 (1.6%) in the pinned file. A file whose #: every row came marked would otherwise be dropped whole. JRC_MAX_MARKED = 1 / 10 +#: The most entities that may be left with no spelling, every one of +#: theirs marked, before the file is refused: none of the pinned file's +#: 819,209 were. +JRC_MAX_LOST = 1 / 1000 +#: The most person rows that may repeat a spelling their entity already +#: has before the file is refused: 58 of 1,338,463 in the pinned file. A +#: layout that collapsed rows would otherwise read as a smaller corpus. +JRC_MAX_REPEATED = 1 / 1000 #: A script's name as the first word of its letters' Unicode names #: gives it, where that word is not the script's own name. @@ -115,9 +128,10 @@ def script(name: str) -> str: """The script `name`'s letters are written in, from the first word of each letter's Unicode name ("Latin", "Cyrillic", "Arabic", "Han" for CJK, "Katakana" counting the prolonged sound mark), "mixed" where - its letters are in two or more, and "none" where it has no letter. - Coarser than the parser's own script test, which names only the - scripts it treats apart, and enough to key a pair of spellings by.""" + its letters are in two or more, and "none" where no letter names a + script. A heuristic over the names, not the Unicode Script property + the parser's own script test reads; enough to key a pair of scripts + by in the groups view.""" found = set() for char in name: if char.isalpha(): @@ -130,19 +144,37 @@ def script(name: str) -> str: return found.pop() if len(found) == 1 else "mixed" +def _kept(path: Path) -> Path: + """Where a download that is not the pinned file is kept beside `path`.""" + return path.with_name(path.name + ".new") + + +def _settled(path: Path) -> bool: + """Whether `path` is the pinned file, once a download kept beside it + (_kept) has replaced it where the pin has moved to that download: no + download is needed then.""" + if path.exists() and _sha256(path) == JRC_SHA256: + return True + kept = _kept(path) + if kept.exists() and _sha256(kept) == JRC_SHA256: + os.replace(kept, path) + return True + return False + + def _jrc_download(path: Path) -> None: - """Download JRC_URL to `path`, through a file beside it that replaces - `path` only once its sha256 is JRC_SHA256's: the JRC serves only its - current file, so a refresh that overwrote the pinned one with a newer - one would leave the corpus nothing to be rebuilt from. A download - that is not the pinned file is left beside it, to be measured.""" - new = path.with_name(path.name + ".new") - # A download kept by an earlier mismatch, once the pin has moved to it. - if not (new.exists() and _sha256(new) == JRC_SHA256): - _download(JRC_URL, new) + """Download JRC_URL to `path`, through a file beside it (_kept) that + replaces `path` only once its sha256 is JRC_SHA256's: the JRC serves + only its current file, so a download that overwrote the pinned one + with a newer one would leave the corpus nothing to be rebuilt from, + and the pinned file could not be downloaded again. A download that is + not the pinned file is left beside it, to be measured.""" + new = _kept(path) + _download(JRC_URL, new) held = _sha256(new) if held != JRC_SHA256: - old = (f"the cached file at {path} is untouched" if path.exists() + old = (f"the cached file at {path} is untouched, and cannot be " + f"downloaded again if it is lost" if path.exists() else "no file is cached") raise ValueError(f"{JRC_URL} served a file with sha256 {held}, not the " f"pinned JRC_SHA256: it is kept at {new}, and {old}. " @@ -164,7 +196,8 @@ def _jrc_read(path: Path) -> Fetched: says.""" counts: Counter[str] = Counter() marked: Counter[str] = Counter() - kept: set[tuple[int, str]] = set() + kept: set[tuple[str, str]] = set() + marked_entities: set[str] = set() last = b"\n" with gzip.open(path, "rb") as lines: for raw in lines: @@ -180,7 +213,7 @@ def _jrc_read(path: Path) -> Fetched: counts["bad"] += 1 continue entity, kind, language, written = cols - name = " ".join(word for word in written.split("+") if word).strip() + name = " ".join(written.replace("+", " ").split()) if (not _ID.fullmatch(entity) or kind not in (JRC_PERSON, JRC_ORGANISATION) or not (language == JRC_UNMARKED or _LANGUAGE.fullmatch(language)) or not name): @@ -192,11 +225,12 @@ def _jrc_read(path: Path) -> Fetched: counts["persons"] += 1 if language != JRC_UNMARKED: marked[language] += 1 + marked_entities.add(entity) continue - if (int(entity), name) in kept: + if (entity, name) in kept: counts["repeated"] += 1 continue - kept.add((int(entity), name)) + kept.add((entity, name)) if not last.endswith(b"\n"): raise ValueError(f"{path} ends inside a line: the file is cut short; " f"download it again: {fetch_command(JRC, refresh=True)}") @@ -211,39 +245,52 @@ def _jrc_read(path: Path) -> Fetched: f"person rows of {path} carry a language")], f"the file no longer marks a plain spelling {JRC_UNMARKED!r}") entities = Counter(entity for entity, _ in kept) - records = [{"entity": str(entity), "name": name, "script": script(name)} - for entity, name in sorted(kept)] + lost = len(marked_entities - entities.keys()) + _refuse_past([(lost, len(entities) + lost, JRC_MAX_LOST, + f"persons of {path} have marked spellings only")], + "dropping the marked rows would drop the persons") + _refuse_past([(counts["repeated"], counts["persons"], JRC_MAX_REPEATED, + f"person rows of {path} repeat a spelling")], + "the file's rows may have collapsed") + # Sorted by the number an id is, then as written: "01" and "1" stay + # two entities, as the file writes them. + records = [{"entity": entity, "name": name, "script": script(name)} + for entity, name in sorted(kept, key=lambda k: (int(k[0]), k))] detail: dict[str, int | str] = { - "sha256": JRC_SHA256, "read_version": JRC_READ_VERSION, "lines": counts["lines"], + "sha256": JRC_SHA256, "read_version": JRC_READ_VERSION, + "unicode": unicodedata.unidata_version, "lines": counts["lines"], "lines_read_past": counts["bad"], "organisation_rows": counts["organisations"], "person_rows": counts["persons"], "marked_variants_dropped": sum(marked.values()), "marked_variants_by_language": " ".join( f"{lang}:{n}" for lang, n in sorted(marked.items(), key=lambda kv: (-kv[1], kv[0]))), "repeated_spellings": counts["repeated"], "entities": len(entities), + "entities_with_marked_spellings_only": lost, "entities_with_one_spelling": sum(n == 1 for n in entities.values()), "spellings_holding_a_replacement_character": sum( - "�" in r["name"] for r in records)} + "\ufffd" in r["name"] for r in records)} return Fetched(records, JRC_DATE, detail) def fetch_jrc() -> Fetched: """One record per distinct spelling of a person entity: its entity - id, the name with its "+" read as spaces (runs of them as one) and - its script(). Read from the gzip cached beside the corpus, - downloaded when missing and again by `fetch --refresh` (a download - that is not the pinned file replaces nothing, _jrc_download), and - refused unless its sha256 is JRC_SHA256's. + id, the name with its "+" and its whitespace read as single spaces, + and its script(). Read from the gzip cached beside the corpus, + downloaded when no file is cached (a kept download the pin has moved + to is adopted instead, _settled) and refused unless its sha256 is + JRC_SHA256's. Organisation rows are counted and read past, as are person rows carrying a language (JRC_UNMARKED says why), by language, and a spelling an entity repeats. A line that is not JRC_COLUMNS well-formed columns or not UTF-8 is counted, and past JRC_MAX_BAD refuses the file; so do a gzip cut short or a file ending inside a - line, fewer person rows than JRC_MIN_PERSONS, and more marked ones - than JRC_MAX_MARKED.""" + line, person rows under JRC_MIN_PERSONS of the well-formed lines (or + none), marked rows past JRC_MAX_MARKED, entities left with no + spelling past JRC_MAX_LOST and repeated spellings past + JRC_MAX_REPEATED.""" path = directory(JRC) / JRC_FILE - if not path.exists(): + if not _settled(path) and not path.exists(): _jrc_download(path) held = _sha256(path) if held != JRC_SHA256: @@ -258,9 +305,14 @@ def fetch_jrc() -> Fetched: def renew_jrc() -> None: - """Download the current file over the cached one, where it is the - pinned file (_jrc_download).""" - _jrc_download(directory(JRC) / JRC_FILE) + """What `fetch --refresh` runs first: a download, unless the cached + file is the pinned one already (_settled). The pinned file cannot + change, so downloading it again gains nothing, and once the JRC + serves a newer file a refresh would fail where a corpus read the old + way only needed reading again from the cached file.""" + path = directory(JRC) / JRC_FILE + if not _settled(path): + _jrc_download(path) _check_file = _pinned("sha256", lambda: JRC_SHA256, @@ -269,9 +321,13 @@ def renew_jrc() -> None: _check_read = _pinned("read_version", lambda: JRC_READ_VERSION, "was read by read version {held}, and this tool reads " "version {wanted}") +_check_unicode = _pinned("unicode", lambda: unicodedata.unidata_version, + "was labelled under Unicode {held}, and this Python " + "reads Unicode {wanted}") def check_jrc(meta: dict[str, Any]) -> str: - """Why a corpus fetched under another pin, or read another way, is - refused: its file's sha256, then its read version.""" - return _check_file(meta) or _check_read(meta) + """Why a corpus fetched under another pin, read another way, or + labelled by script() under another Unicode version is refused: its + file's sha256, its read version, then its Unicode version.""" + return _check_file(meta) or _check_read(meta) or _check_unicode(meta)