diff --git a/AGENTS.md b/AGENTS.md index b17435c6..4e049002 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -275,7 +275,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree`; tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree`; tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. ## Gotchas diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 3cc47971..7ff44d67 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -410,11 +410,11 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" **Problem shape.** A new source of names turns up (a registry, a dataset, a dump) and the question is what to build for it: a committed corpus, a column in a tool, a script that writes vocabulary. -**Contract statement.** A name source does one or more of five jobs — WATCH, SCORE, COUNT, DERIVE and SURVEY — and the job decides where its data lives. Only WATCH in CI and SCORE need the names in the repository, so a source whose names cannot be committed still does COUNT and SURVEY, WATCH outside CI, and DERIVE where its terms let a derived list ship. +**Contract statement.** A name source does one or more of five jobs — WATCH, SCORE, COUNT, DERIVE and SURVEY — and the job decides where its data lives. Only WATCH and SCORE in CI need the names in the repository, so a source whose names cannot be committed still does COUNT and SURVEY, WATCH and SCORE outside CI, and DERIVE where its terms let a derived list ship. -**How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`). +**How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone, or a local corpus whose records carry the fields their filers typed, with its record kept in the cache beside it (`local_agreement.py`). The local score tells a change's author what moved among real filed names, by field, and gives agreement by group and how often each disagreement shape occurs. It cannot guard anything: no CI run and no reviewer sees it, and no test fails when it moves (the tool itself exits 1). Its truth is the filer's, so a box filled wrongly (a job typed as a title) reads as a miss unless the scorer sets it aside, and the scorer has to say what it sets aside. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`). -A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). +A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775). **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one. diff --git a/tests/v2/test_local_agreement.py b/tests/v2/test_local_agreement.py new file mode 100644 index 00000000..60205c4d --- /dev/null +++ b/tests/v2/test_local_agreement.py @@ -0,0 +1,280 @@ +"""Offline checks on tools/differential/local_agreement.py. + +The score compares a parse of each filed display name with the boxes the +filer typed it into, so what these pin is the comparison's three +decisions (the fold, the suffix counted in the surname box, the title +box set aside as filer error), each beside the reading with the decision +off, and the record's refusals: no record, a record that is not one, a record of +another fetch or another scorer, and a corpus damaged under the stream. The names are invented. +""" +import json +import sys +from pathlib import Path + +import pytest + +from nameparser import Parser + +from ._differential_fixtures import load_tool + +agreement = load_tool("local_agreement") +corpora = agreement.local_corpora +_PSC = corpora.PSC +_PSC_FIELDS = agreement.FIELDED[_PSC] +_LISTED = agreement._title_words(Parser().lexicon) + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> Path: + """Every test's cache is its own, as in test_local_corpora.py.""" + path = tmp_path_factory.mktemp("cache") + monkeypatch.setenv("NAMEPARSER_TOOLS_CACHE", str(path)) + return path + + +def _filed(title: str = "", forename: str = "", middle_name: str = "", surname: str = "", + nationality: str = "British") -> dict[str, str]: + boxes = {"title": title, "forename": forename, "middle_name": middle_name, + "surname": surname} + return {"name": " ".join(v for v in boxes.values() if v), "nationality": nationality, + **{k: v for k, v in boxes.items() if v}} + + +_AGREES = _filed("Mrs", "Jo", surname="Ng") +_DOUBLE_SURNAME = _filed("", "Ana", surname="García López", nationality="Spanish") +_SUFFIX = _filed("Mr", "Bob", surname="Roe Jr") +_JOB_TITLE = _filed("Managing Partner", "Cy", surname="Doe") +_SHAPE_TITLE = _filed("Insp.", "Di", surname="Fox") + + +def _read(record: dict[str, str]) -> tuple[dict[str, str], bool, bool]: + """The record's reading: the fields that disagree, whether it was + set aside, whether the family agreed by the suffix.""" + reading = agreement.read(record, _PSC_FIELDS, Parser(), _LISTED) + return reading.got, reading.set_aside, reading.by_suffix + + +def test_a_box_agrees_whatever_its_case_periods_and_spacing( + monkeypatch: pytest.MonkeyPatch) -> None: + assert agreement.fold(" Mr. ANN ") == "mr ann" + upper = _filed("MR", "ANN", surname="LEE") + upper["forename"] = "Ann." + assert _read(upper)[0] == {} + # The control: compared as written, the box disagrees. + monkeypatch.setattr(agreement, "fold", lambda text: text) + assert _read(upper)[0] == {"given": "ANN"} + + +def test_a_two_word_surname_read_with_a_middle_name_is_two_misses() -> None: + got = _read(_DOUBLE_SURNAME)[0] + assert got == {"middle": "García", "family": "López"} + assert agreement.shape(_DOUBLE_SURNAME, _PSC_FIELDS, got) == ( + "middle 0->1, family 2->1") + + +def test_a_peeled_suffix_is_compared_inside_the_surname_box() -> None: + assert _read(_SUFFIX) == ({}, False, True) + assert _read(_AGREES) == ({}, False, False) + # The control: the parser's family alone leaves the box short. + parsed = Parser().parse(_SUFFIX["name"]) + assert (parsed.family, parsed.suffix) == ("Roe", "Jr") + + +def test_a_title_box_holding_an_unlisted_word_the_parse_reads_otherwise_is_set_aside( + monkeypatch: pytest.MonkeyPatch) -> None: + got, set_aside, _ = _read(_JOB_TITLE) + assert set_aside and "title" in got + # An unlisted title the parse reads as the title, by its period, is + # scored, and agrees. + assert agreement.unlisted("Insp.", _LISTED) + assert not agreement.unlisted("Mr. and MRS", _LISTED) # a connective is listed + assert _read(_SHAPE_TITLE) == ({}, False, False) + # The control: with every word listed, the job is three misses. + monkeypatch.setattr(agreement, "unlisted", lambda title, listed: False) + assert _read(_JOB_TITLE) == ( + {"title": "", "given": "Managing", "middle": "Partner Cy"}, False, False) + + +def _corpus(monkeypatch: pytest.MonkeyPatch, *records: dict[str, str]) -> None: + """A PSC corpus of `records` fetched into the cache, no network behind it.""" + monkeypatch.setitem(corpora.SOURCES, _PSC, corpora.Source( + corpora.SOURCES[_PSC].credit, "filed names", + lambda: corpora.Fetched(list(records), "2026-10-22", {}))) + corpora.fetch(_PSC) + + +_ALL = (_AGREES, _DOUBLE_SURNAME, _SUFFIX, _JOB_TITLE, _SHAPE_TITLE) + + +def test_the_report_counts_by_group_and_field_and_prints_no_name( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(agreement, "GROUP_FLOOR", 1) + monkeypatch.setattr(agreement, "TITLE_FLOOR", 1) + result = agreement.score(list(_ALL), _PSC_FIELDS, Parser()) + assert result.scored == {"British": 3, "Spanish": 1} + assert result.agreed == {"British": 3} + assert result.misses == {"Spanish": {"middle": 1, "family": 1}} + assert result.set_aside == {"Managing Partner": 1} + assert result.by_suffix == 1 + assert set(result.entries) == {agreement.key(_DOUBLE_SURNAME), agreement.key(_JOB_TITLE)} + lines = agreement.report(result, _PSC_FIELDS, 5) + assert "every field agrees: 3 of 4 (75.0%); 1 of them with the parser's suffix" \ + in "\n".join(lines) + assert " British 3 100.0% 0 0 0 0" \ + in lines + names = {w for r in _ALL for k, v in r.items() if k in ("forename", "surname") + for w in v.split()} + printed = " ".join(lines).replace("Managing Partner", "") + assert not {w for w in names if w in printed.split()} + + +def test_a_stream_of_damaged_records_is_refused_once_read( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + _corpus(monkeypatch, *_ALL) + loaded = corpora.load(_PSC) + assert len(list(agreement.records(loaded))) == len(_ALL) # the control + raw = _own_cache / "local" / _PSC / "raw.jsonl" + raw.write_text(raw.read_text().replace("Jo", "Al")) + with pytest.raises(corpora.CorpusDamaged, match="does not match the sha256"): + list(agreement.records(loaded)) + + +def test_a_rerun_reports_what_moved_since_the_record( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: + _corpus(monkeypatch, *_ALL) + assert agreement.main([_PSC, "--record"]) == 0 + assert agreement.main([_PSC]) == 0 # the control: nothing moved + assert "matches the record" in capsys.readouterr().out + # A parser that reads every name with its words reversed moves them all. + real = Parser.parse + monkeypatch.setattr(Parser, "parse", + lambda self, text: real(self, " ".join(reversed(text.split())))) + assert agreement.main([_PSC]) == 1 + out = capsys.readouterr().out + assert "regression: 3 records (" in out and "progress" not in out + assert "changed: 2 records" in out + + +def test_a_record_of_another_fetch_is_refused( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + _corpus(monkeypatch, *_ALL) + assert agreement.main([_PSC, "--record"]) == 0 + _corpus(monkeypatch, *_ALL[:2]) + with pytest.raises(SystemExit, match="recorded from another fetch"): + agreement.main([_PSC]) + path = _own_cache / "local" / _PSC / agreement.RECORD + header = json.loads(path.read_text().splitlines()[0]) + assert set(header) == {"raw_sha256", "scorer", "data_date", "tree"} + + +def test_a_run_with_no_record_names_the_command_that_writes_one( + monkeypatch: pytest.MonkeyPatch) -> None: + _corpus(monkeypatch, *_ALL) + with pytest.raises(SystemExit, match="--record --tree"): + agreement.main([_PSC]) + + +def test_a_set_aside_title_shared_by_too_few_records_is_not_printed( + monkeypatch: pytest.MonkeyPatch) -> None: + typed = _filed("Jo Bloggs", "Cy", surname="Doe") # a name in the title box + result = agreement.score([typed] * 3 + [_JOB_TITLE], _PSC_FIELDS, Parser()) + assert result.set_aside == {"Jo Bloggs": 3, "Managing Partner": 1} + assert "Bloggs" not in "\n".join(agreement.report(result, _PSC_FIELDS, 5)) + # The control: under a floor of 3 the box shared by 3 records is + # printed, and the one typed once is not. + monkeypatch.setattr(agreement, "TITLE_FLOOR", 3) + printed = "\n".join(agreement.report(result, _PSC_FIELDS, 5)) + assert "Jo Bloggs" in printed and "Managing" not in printed + + +def test_a_group_below_the_floor_has_no_row_and_rows_run_by_size( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(agreement, "GROUP_FLOOR", 2) + rows = [_AGREES, _filed("Mr", "Bo", surname="Wu", nationality=" british "), + _filed("Ms", "Di", surname="Fox", nationality="Irish"), + _filed("Ms", "Al", surname="Ng", nationality="Irish"), + _filed("Ms", "Ed", surname="Lin", nationality="Irish"), _DOUBLE_SURNAME, + _filed("Mr", "Cy", surname="Roe", nationality="")] + result = agreement.score(rows, _PSC_FIELDS, Parser()) + assert result.scored == {"Irish": 3, "British": 2, "Spanish": 1, "(none)": 1} + groups = [line.split()[0] for line in agreement.report(result, _PSC_FIELDS, 5) + if line.startswith(" ") and line.split()[0] in result.scored] + assert groups == ["Irish", "British"] + + +def test_a_line_that_is_not_a_record_is_refused_as_damage( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + _corpus(monkeypatch, *_ALL) + loaded = corpora.load(_PSC) + raw = _own_cache / "local" / _PSC / "raw.jsonl" + for damage in ('{"name": ', '["a list"]', '{"name": "Al Ng", "surname": 1}'): + lines = raw.read_text().splitlines(keepends=True) + raw.write_text("".join(lines[:1] + [damage + "\n"] + lines[1:])) + # Through score(), which reads each record before the stream ends. + with pytest.raises(corpora.CorpusDamaged, match="fetch it again"): + agreement.score(agreement.records(loaded), _PSC_FIELDS, Parser()) + raw.write_text("".join(lines)) + assert len(list(agreement.records(loaded))) == len(_ALL) # the control + + +@pytest.mark.parametrize("body", ["", "{not json\n", '["a list"]\n', + '{"raw_sha256": "x"}\n{"got": {}}\n'], + ids=["empty", "header not JSON", "header not an object", + "a row without its key"]) +def test_a_record_that_is_not_one_is_refused_naming_the_record_command( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path, body: str) -> None: + _corpus(monkeypatch, *_ALL) + assert agreement.main([_PSC, "--record"]) == 0 + (_own_cache / "local" / _PSC / agreement.RECORD).write_text(body) + with pytest.raises(SystemExit, match="is not a record this tool wrote.*--record"): + agreement.main([_PSC]) + + +def test_a_record_written_by_another_scorer_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + _corpus(monkeypatch, *_ALL) + assert agreement.main([_PSC, "--record"]) == 0 + before = agreement.scorer_fingerprint() + # Each part of the code that decides a record's entry moves the + # fingerprint: how a field is compared, which records get an entry, + # and how an entry is keyed. + for name, changed in (("fold", lambda text: text.casefold()), + ("score", lambda rows, fielded, parser: None), + ("key", lambda record: str(record))): + with monkeypatch.context() as patch: + patch.setattr(agreement, name, changed) + assert agreement.scorer_fingerprint() != before, name + with monkeypatch.context() as patch: + patch.setattr(agreement.corpus_build, "jsonl_line", + lambda obj, sort_keys=False: str(obj)) + assert agreement.scorer_fingerprint() != before, "jsonl_line" + assert agreement.scorer_fingerprint() == before + monkeypatch.setattr(agreement, "fold", lambda text: text.casefold()) + with pytest.raises(SystemExit, match="another version of this scorer"): + agreement.main([_PSC]) + + +def test_progress_is_reported_where_a_disagreement_now_agrees( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: + _corpus(monkeypatch, *_ALL) + real = Parser.parse + monkeypatch.setattr(Parser, "parse", lambda self, text: real( + self, "Al Ng" if text == _AGREES["name"] else text)) + assert agreement.main([_PSC, "--record"]) == 0 + monkeypatch.setattr(Parser, "parse", real) + assert agreement.main([_PSC, "--list", "1"]) == 1 + out = capsys.readouterr().out + assert "progress: 1 records (" in out and "regression" not in out + assert f" progress: {agreement.key(_AGREES)}" in out + + +def test_a_tree_that_is_not_the_imported_package_is_refused( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path, tmp_path: Path) -> None: + _corpus(monkeypatch, *_ALL) + monkeypatch.setattr(sys, "path", list(sys.path)) # main puts the tree on it + with pytest.raises(SystemExit, match="not from the source package"): + agreement.main([_PSC, "--record", "--tree", str(tmp_path)]) + assert not (_own_cache / "local" / _PSC / agreement.RECORD).exists() + # The control: this checkout itself is the tree it imported. + root = agreement.HERE.parents[1] + assert agreement.main([_PSC, "--record", "--tree", str(root)]) == 0 diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 4cfee875..a9651d3c 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -10,11 +10,14 @@ """ import csv import email.message +import gzip import hashlib import http.client import io import json +import os import re +import shutil import time import types import unicodedata @@ -1879,6 +1882,406 @@ def urlopen(request: urllib.request.Request, timeout: int) -> _Response: local.fetch(_UKEC) +# -- Companies House people with significant control ------------------------------ + +_PSC = "psc-individuals" +_PSC_DAY = "2026-10-22" +_PSC_PERSON = "individual-person-with-significant-control" + + +def _psc_person(name: str, nationality: str | None = "British", + kind: str = _PSC_PERSON, **elements: str) -> dict[str, Any]: + """A PSC record as the snapshot writes it, its boxes from `elements` + (or split from `name`: forename, then surname), with the fields the + corpus must not keep.""" + if not elements: + forename, _, surname = name.partition(" ") + elements = {"title": "Mr", "forename": forename, "surname": surname} + name = f"Mr {name}" + data: dict[str, Any] = { + "kind": kind, "name": name, "name_elements": elements, + "address": {"premises": "1 Invented Row", "postal_code": "ZZ1 1ZZ"}, + "date_of_birth": {"month": 4, "year": 1970}, "country_of_residence": "England", + "natures_of_control": ["ownership-of-shares-75-to-100-percent"]} + if nationality is not None: + data["nationality"] = nationality + return {"company_number": "00000001", "data": data} + + +def _psc_lines(*records: dict[str, Any] | str) -> bytes: + return b"".join((r if isinstance(r, str) else json.dumps(r)).encode() + b"\n" + for r in records) + + +#: Part 1: a person, a person filed twice (two companies), a company, a +#: super-secure PSC with no name, a person with no nationality, and a +#: person whose name and boxes are all empty. +#: Part 2: the first person again, filed by another company, and an +#: overseas entity's beneficial owner. +_PSC_PARTS = { + 1: _psc_lines(_psc_person("Ann Lee"), _psc_person("Bo Wu"), _psc_person("Bo Wu"), + {"data": {"kind": "corporate-entity-person-with-significant-control", + "name": "Invented Holdings Ltd"}}, + {"data": {"kind": "super-secure-person-with-significant-control"}}, + _psc_person("Cy Doe", nationality=None), + _psc_person("", title="", forename="", surname="")), + 2: _psc_lines(_psc_person("Ann Lee"), + _psc_person("Dr Di Fox Roe", "Irish", "individual-beneficial-owner", + title="Dr", forename="Di", middle_name="Fox", + surname="Roe")), +} + + +def _psc_part(n: int, body: bytes, of: int = 2, day: str = _PSC_DAY, + member: str | None = None) -> bytes: + """Part `n` of `of` as the download serves it: a zip of one JSON-lines file.""" + out = io.BytesIO() + with zipfile.ZipFile(out, "w") as archive: + archive.writestr(member or f"psc-snapshot-{day}_{n}of{of}.txt", body) + return out.getvalue() + + +def _psc_page(*files: str) -> bytes: + return "".join(f'{f} (74Mb)\n' for f in files).encode() + + +def _psc_files(of: int = 2, day: str = _PSC_DAY) -> list[str]: + return [f"psc-snapshot-{day}_{n}of{of}.zip" for n in range(1, of + 1)] + + +def _serve_psc(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, + parts: dict[str, Answer], page: bytes | None = None) -> list[str]: + """Answer the download page with `page` (every file in `parts` + linked) and each part file with its answer: bytes to serve, or an + exception to raise. Returns the file names downloaded.""" + monkeypatch.setenv("NAMEPARSER_TOOLS_CACHE", str(tmp_path)) + monkeypatch.setattr(local, "FEC_CHUNK", 64) # several reads, not one + monkeypatch.setattr(local, "PSC_SAMPLE", 1) # every record, unless a test samples + page = _psc_page(*parts) if page is None else page + asked: list[str] = [] + + def urlopen(request: urllib.request.Request, timeout: int) -> _Response | _Download: + url = request.full_url + if url == local.PSC_PAGE: + return _Response(page) + file = url.rsplit("/", 1)[1] + asked.append(file) + answer = parts.get(file) + if isinstance(answer, Exception): + raise answer + if answer is None: + pytest.fail(f"an unexpected download of {url}") + return _Download(answer, len(answer)) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + return asked + + +def _psc_snapshot(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, + bodies: dict[int, bytes] = _PSC_PARTS, day: str = _PSC_DAY) -> list[str]: + of = len(bodies) + return _serve_psc(monkeypatch, tmp_path, { + file: _psc_part(n, bodies[n], of, day) + for n, file in enumerate(_psc_files(of, day), 1)}) + + +def _gone(file: str) -> urllib.error.HTTPError: + return urllib.error.HTTPError(f"{local.PSC_PAGE}/{file}", 404, "Not Found", + email.message.Message(), None) + + +def test_a_psc_fetch_keeps_each_filed_name_once_and_counts_the_rest( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + monkeypatch.setattr(local, "PSC_MAX_BAD", 1) # a fixture of 9 lines + asked = _psc_snapshot(monkeypatch, tmp_path) + corpus = local.fetch(_PSC) + assert asked == _psc_files() + # In the order of their lines, whose keys are sorted: by forename. + assert corpus.records == [ + {"name": "Mr Ann Lee", "title": "Mr", "forename": "Ann", "surname": "Lee", + "nationality": "British"}, + {"name": "Mr Bo Wu", "title": "Mr", "forename": "Bo", "surname": "Wu", + "nationality": "British"}, + {"name": "Mr Cy Doe", "title": "Mr", "forename": "Cy", "surname": "Doe"}, + {"name": "Dr Di Fox Roe", "title": "Dr", "forename": "Di", "middle_name": "Fox", + "surname": "Roe", "nationality": "Irish"}, + ] + assert corpus.meta["data_date"] == _PSC_DAY + assert corpus.meta["detail"] == { + "parts": 2, "sample": "1 in 1", "lines": 9, local.PSC_KINDS[0]: 6, local.PSC_KINDS[1]: 1, + "records_of_other_kinds": 2, "individuals_without_a_nationality": 1, + "sampled_names_that_are_empty": 1} + assert corpus.header().endswith("4 filed names (local corpus psc-individuals)") + + +def test_a_psc_fetch_writes_no_field_but_the_name_its_boxes_and_the_nationality( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _psc_snapshot(monkeypatch, tmp_path) + local.fetch(_PSC) + folder = tmp_path / "local" / _PSC + kept = set(local.PSC_KEPT) + for path in [folder / "raw.jsonl", *(folder / "parts").iterdir()]: + text = (gzip.decompress(path.read_bytes()) if path.suffix == ".gz" + else path.read_bytes()).decode() + for leaked in ("Invented Row", "ZZ1", "1970", "England", "00000001", + "ownership", "Invented Holdings"): + assert leaked not in text, (path.name, leaked) + for line in text.splitlines(): + row = json.loads(line) + assert "part" in row or set(row) <= kept, (path.name, row) + + +def test_a_psc_sample_keeps_the_same_records_whatever_part_files_them( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + people = [_psc_person(f"Ann Lee{n}") for n in range(40)] + _psc_snapshot(monkeypatch, tmp_path, {1: _psc_lines(*people[:25]), + 2: _psc_lines(*people[25:])}) + monkeypatch.setattr(local, "PSC_SAMPLE", 4) + corpus = local.fetch(_PSC) + kept = corpus.records + assert 0 < len(kept) < 20 and corpus.meta["detail"]["sample"] == "1 in 4" + assert all(local.psc_sampled(_extract_line(r)) for r in kept) + # Filed by other parts, in another order, the same records are kept. + _psc_snapshot(monkeypatch, tmp_path, {1: _psc_lines(*people[30:]), + 2: _psc_lines(*reversed(people[:30]))}) + monkeypatch.setattr(local, "PSC_SAMPLE", 4) + shutil.rmtree(tmp_path / "local" / _PSC / "parts") + assert local.fetch(_PSC).records == kept + # The control: with every record kept, all 40 are there. + monkeypatch.setattr(local, "PSC_SAMPLE", 1) + assert len(local.fetch(_PSC).records) == 40 + + +def _extract_line(record: dict[str, str]) -> str: + """A record as its extract line, the text psc_sampled hashes.""" + return local.corpus_build.jsonl_line(record, sort_keys=True) + + +@pytest.mark.parametrize(("bad", "match"), [ + ({"data": {"kind": _PSC_PERSON, "name": "Mr Ann Lee"}}, + "1 of 2 individuals have no name_elements"), + (_psc_person("Mr Ann Lee", title="Mr", forename="Ann", surname="Lee"), + "1 of 2 individuals' display names are not their four boxes joined"), + ("{not json", "1 of 2 lines are not JSON"), +], ids=["no name_elements", "a name not its boxes joined", "a line not JSON"]) +def test_a_psc_snapshot_whose_layout_has_changed_is_refused( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, + bad: dict[str, Any] | str, match: str) -> None: + good = _psc_person("Bo Wu") + _psc_snapshot(monkeypatch, tmp_path, {1: _psc_lines(bad, good)}) + with pytest.raises(ValueError, match=match): + local.fetch(_PSC) + assert local.load(_PSC) is None + # The control: past the threshold the same snapshot is read, the bad + # record counted and dropped, so the threshold is what refused it. + monkeypatch.setattr(local, "PSC_MAX_BAD", 1) + assert [r["name"] for r in local.fetch(_PSC).records] == ["Mr Bo Wu"] + + +@pytest.mark.parametrize(("page", "match"), [ + (b"

The snapshot has moved.

", "links no snapshot parts"), + (_psc_page(*_psc_files(3)[:2]), r"parts \[1, 2\] of \[3\]"), + (_psc_page(_psc_files()[0], _psc_files(day="2026-10-23")[1]), + r"dated \['2026-10-22', '2026-10-23'\]"), +], ids=["no parts", "a part missing", "parts of two days"]) +def test_a_psc_page_that_lists_no_whole_snapshot_is_refused( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, page: bytes, match: str) -> None: + asked = _serve_psc(monkeypatch, tmp_path, {}, page) + with pytest.raises(ValueError, match=match): + local.fetch(_PSC) + assert asked == [] + + +def test_a_psc_part_gone_mid_fetch_is_refused_and_the_next_run_resumes( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + files = _psc_files() + parts: dict[str, Answer] = {files[0]: _psc_part(1, _PSC_PARTS[1]), + files[1]: _gone(files[1])} + asked = _serve_psc(monkeypatch, tmp_path, parts) + with pytest.raises(ValueError, match="replaced during the fetch; fetch again"): + local.fetch(_PSC) + folder = tmp_path / "local" / _PSC + assert not (folder / "parts").exists() + assert sorted(p.name for p in (folder / "parts.new").iterdir()) == [ + local._extract_name(files[0])] + parts[files[1]] = _psc_part(2, _PSC_PARTS[2]) + assert len(local.fetch(_PSC).records) == 4 + assert asked == [*files, files[1]] # part 1 was not downloaded again + assert not (folder / "parts.new").exists() + + +def test_a_resumed_psc_fetch_drops_what_another_day_left( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + asked = _psc_snapshot(monkeypatch, tmp_path) + staging = tmp_path / "local" / _PSC / "parts.new" + staging.mkdir(parents=True) + stale = staging / "psc-snapshot-2026-10-21_1of2.jsonl.gz" + stale.write_bytes(gzip.compress(b'{"part": "yesterday", "counts": {}}\n')) + (staging / "psc-snapshot-2026-10-22_2of2.zip.part").write_bytes(b"cut") + assert len(local.fetch(_PSC).records) == 4 + assert asked == _psc_files() + assert sorted(p.name for p in (tmp_path / "local" / _PSC / "parts").iterdir()) == [ + local._extract_name(f) for f in _psc_files()] + + +def test_a_failed_psc_refresh_keeps_the_parts_and_the_corpus_it_had( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _psc_snapshot(monkeypatch, tmp_path) + before = local.fetch(_PSC) + newer = _psc_files(day="2026-10-23") + cut = _psc_part(2, _PSC_PARTS[2], day="2026-10-23") + _serve_psc(monkeypatch, tmp_path, {newer[0]: _psc_part(1, _PSC_PARTS[1], day="2026-10-23"), + newer[1]: cut[:-10]}) + with pytest.raises(ValueError, match="fetch again"): + local.main(["fetch", _PSC, "--refresh"]) + assert local.load(_PSC) == before + assert sorted(p.name for p in (tmp_path / "local" / _PSC / "parts").iterdir()) == [ + local._extract_name(f) for f in _psc_files()] + # The control: the same refresh, whole, replaces both. + _serve_psc(monkeypatch, tmp_path, {newer[0]: b"", newer[1]: cut}) + assert local.main(["fetch", _PSC, "--refresh"]) == 0 + after = local.load(_PSC) + assert after is not None and after.meta["data_date"] == "2026-10-23" + + +def test_a_psc_cache_missing_a_part_is_refused_with_the_way_out( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _psc_snapshot(monkeypatch, tmp_path) + local.fetch(_PSC) + (tmp_path / "local" / _PSC / "parts" / local._extract_name(_psc_files()[1]) + ).unlink() + with pytest.raises(ValueError, match=r"holds parts \[1\] of \[2\].*--refresh"): + local.fetch(_PSC) + + +@pytest.mark.parametrize(("body", "match"), [ + (b"not a zip at all", "is not a readable zip"), + (_psc_part(1, _PSC_PARTS[1], of=1, member="renamed.txt"), + "holds no psc-snapshot-2026-10-22_1of1.txt"), +], ids=["not a zip", "no member of its own name"]) +def test_a_psc_part_that_cannot_be_read_is_refused( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, body: bytes, match: str) -> None: + _serve_psc(monkeypatch, tmp_path, {_psc_files(1)[0]: body}) + with pytest.raises(ValueError, match=match): + local.fetch(_PSC) + assert not list((tmp_path / "local" / _PSC / "parts.new").glob("*.zip*")) + + +def _plant(tmp_path: Path, day: str, bodies: dict[int, bytes] = _PSC_PARTS, + folder: str = "parts") -> Path: + """The extracts of a snapshot of `bodies` dated `day`, written into + the cache's `folder` as renew_psc would write them; the folder.""" + where = tmp_path / "local" / _PSC / folder + where.mkdir(parents=True, exist_ok=True) + of = len(bodies) + for n, file in enumerate(_psc_files(of, day), 1): + zipped = tmp_path / file + zipped.write_bytes(_psc_part(n, bodies[n], of, day)) + (where / local._extract_name(file)).write_bytes( + local._psc_extract(zipped, file.replace(".zip", ".txt"))) + return where + + +def test_a_psc_cache_holding_parts_of_two_snapshots_is_refused_before_any_is_read( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _serve_psc(monkeypatch, tmp_path, {}) # no download is expected + monkeypatch.setattr(local, "PSC_MAX_BAD", 1) + parts = _plant(tmp_path, _PSC_DAY) + assert len(local.fetch(_PSC).records) == 4 # the control: one snapshot + _plant(tmp_path, "2026-10-21") + with pytest.raises(ValueError, match=r"holds part 1 twice.*--refresh"): + local.fetch(_PSC) + for path in parts.glob("*2026-10-21*"): + path.unlink() + (parts / local._extract_name(_psc_files(day="2026-10-21")[1])).write_bytes( + (parts / local._extract_name(_psc_files()[1])).read_bytes()) + (parts / local._extract_name(_psc_files()[1])).unlink() + with pytest.raises(ValueError, match=r"dated \['2026-10-21', '2026-10-22'\]"): + local.fetch(_PSC) + + +def test_a_psc_snapshot_whose_person_kind_was_renamed_is_refused( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + people = [_psc_person(f"Ann Lee{n}", kind="individual-person-renamed") for n in range(9)] + owner = _psc_person("Bo Wu", kind=local.PSC_KINDS[1]) + _psc_snapshot(monkeypatch, tmp_path, {1: _psc_lines(*people, owner)}) + with pytest.raises(ValueError, match="1 of 10 lines are .* records: a kind"): + local.fetch(_PSC) + # The control: under the kind's own name, the same snapshot is read. + monkeypatch.setattr(local, "PSC_KINDS", ("individual-person-renamed", local.PSC_KINDS[1])) + shutil.rmtree(tmp_path / "local" / _PSC / "parts") + assert len(local.fetch(_PSC).records) == 10 + + +@pytest.mark.parametrize(("damage", "match"), [ + (lambda body: body[:-10], "is not an extract this tool wrote"), + (lambda body: b"not gzip", "is not an extract this tool wrote"), + (lambda body: gzip.compress(b"{not json\n"), "is not an extract this tool wrote"), + (lambda body: gzip.compress( + gzip.decompress(body).replace(b"_1of2.txt", b"_2of2.txt", 1)), + "its header does not describe it"), + (lambda body: gzip.compress( + gzip.decompress(body).replace(b'"lines": 7', b'"lines": "7"', 1)), + "its header does not describe it"), +], ids=["cut short", "not gzip", "header not JSON", "another part's header", + "a count not a number"]) +def test_a_damaged_psc_extract_is_refused_naming_the_refresh( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, + damage: Callable[[bytes], bytes], match: str) -> None: + _serve_psc(monkeypatch, tmp_path, {}) + monkeypatch.setattr(local, "PSC_MAX_BAD", 1) + parts = _plant(tmp_path, _PSC_DAY) + first = parts / local._extract_name(_psc_files()[0]) + assert b'"lines": 7' in gzip.decompress(first.read_bytes()) + first.write_bytes(damage(first.read_bytes())) + with pytest.raises(ValueError, match=f"{match}.*fetch {_PSC} --refresh"): + local.fetch(_PSC) + + +def test_a_psc_extract_of_another_version_is_neither_read_nor_resumed( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + asked = _psc_snapshot(monkeypatch, tmp_path) + monkeypatch.setattr(local, "PSC_EXTRACT_VERSION", 0) + parts = _plant(tmp_path, _PSC_DAY) + monkeypatch.setattr(local, "PSC_EXTRACT_VERSION", 1) + with pytest.raises(ValueError, match="is not an extract of version 1"): + local.fetch(_PSC) + # Staged for a resume, the old version's extracts are downloaded anew. + os.replace(parts, parts.with_name("parts.new")) + assert len(local.fetch(_PSC).records) == 4 + assert asked == _psc_files() + + +def test_a_psc_corpus_of_another_sample_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _psc_snapshot(monkeypatch, tmp_path) + local.fetch(_PSC) + assert local.load(_PSC) is not None # the control + monkeypatch.setattr(local, "PSC_SAMPLE", 8) + with pytest.raises(ValueError, match=r"keeps 1 in 1 filed names.*keeps 1 in 8" + f".*fetch {_PSC} --refresh"): + local.load(_PSC) + + +def test_a_psc_page_linking_each_part_twice_is_one_snapshot( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + files = _psc_files() + _serve_psc(monkeypatch, tmp_path, { + f: _psc_part(n, _PSC_PARTS[n]) for n, f in enumerate(files, 1)}, + page=_psc_page(*files, *files)) + assert len(local.fetch(_PSC).records) == 4 + + +def test_a_psc_fetch_cut_off_between_its_two_renames_resumes_without_downloading( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + asked = _serve_psc(monkeypatch, tmp_path, {}, _psc_page(*_psc_files())) # any download fails + _plant(tmp_path, "2026-10-21", folder="parts.old") + _plant(tmp_path, _PSC_DAY, folder="parts.new") + assert len(local.fetch(_PSC).records) == 4 + assert asked == [] + folder = tmp_path / "local" / _PSC + assert sorted(p.name for p in folder.iterdir() if p.is_dir()) == ["parts"] + + # -- load ---------------------------------------------------------------------- def _fetched(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, diff --git a/tools/differential/README.md b/tools/differential/README.md index b983caad..4fbfac3f 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -233,12 +233,13 @@ uv run python tools/differential/build_shapes_corpus.py --coverage ## Local corpora -A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. +A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials uv run python tools/differential/local_corpora.py tally ctgov-officials --top 50 uv run python tools/name_usage.py --no-wikidata MA FNP-C +uv run python tools/differential/local_agreement.py psc-individuals ``` To read what a branch moves, record the local snapshot with the base checkout, then check with the branch: @@ -257,9 +258,14 @@ uv run python tools/differential/snapshot.py --local ctgov-officials | `tse-donors` | every individual donor to a candidate in Brazil's 2022 general and 2024 municipal elections (the TSE's candidates' accounts, rows from `Recursos de pessoas físicas`), a record per distinct pair of the name as the campaign filed it (`name`) and the federal tax registry's name for the same donor (`registry`): a typed form with its reference form beside it. Of 854,531 records fetched 2026-10-22 (848,351 distinct filed names), 205,925 differ from their registry name -- an agnomen dropped or abbreviated (`JR` filed for the registry's `JUNIOR` in 1,615, `JÚNIOR` for `JUNIOR` in 975), a word left out. Nearly all in capitals (845,498 filed names), in spoken order with no comma (184 records hold one), and full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO` ends 10,955 distinct filed names, `JUNIOR` 13,784, `NETO` 8,410). Published by the TSE under CC BY | case, mostly; the family-comma shape; credentials and titles, which a donation form has no place for; donors to parties rather than candidates; names outside Brazil; names in any script but Latin | | `tw-donors` | the distinct names of every individual donor in every campaign finance report Taiwan's Control Yuan publishes on its political donation platform (ardata.cy.gov.tw, whose zips data.gov.tw also lists): 454 reports of 64 elections on 2026-10-22, presidential, legislative and local: every report from 2018's elections on, and the residual reports of 2014's and 2016's. Each report's zip is cached under the name the platform lists it by, which carries the date it was built, so a refresh downloads only the reports published or rebuilt since. Names as a candidate's staff typed them into a filing, published in full where the ID, address and phone beside them are masked. Of the 308,667 names dated 2026-10-05, 296,820 were three Han characters, 5,449 four and 5,260 two; the population for unspaced Han names written family first, and for the married double surname (冠夫姓) a four-character name often is. Under the Open Government Data License, version 1.0, with the platform's own bar on using the data for profit, credit investigation or another improper purpose | names in spoken order; Latin script (80 of the names dated 2026-10-05 held a Latin letter); the comma (3 of them held a fullwidth one, none an ASCII one); case; names outside Taiwan; donors whose names the report does not publish, such as anonymous gifts; Japanese kanji names, which the Han script cannot tell from Chinese | | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | +| `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. The one local corpus whose names carry their fields, scored by `local_agreement.py` | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256 and to the code that cuts and folds a word, so a refetch or a change to the fold rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` (1 in 10,000) of either of the first two, or `UKEC_MAX_NAMELESS` (1 in 1,000; 4 of 42,355 rows were nameless on 2026-10-22) of the third, it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report more than half of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than 4 digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. +`psc-individuals` caches an extract of each of the snapshot's 33 parts rather than the zips: 168 MB against 2.44 GB on 2026-10-22, each extract holding only the records and fields the corpus keeps. A download that fails costs one part, and running the fetch again resumes, keeping the parts already read for the same day; the set replaces the cached one only once every part is read, and a part that disappears mid-fetch (the files are overwritten each morning) is refused naming the refresh. The fetch refuses a snapshot where more than one individual in 10,000 has no `name_elements`, or a display name that is not its four boxes joined by spaces, or where fewer than half of the lines are individuals (87% were on 2026-10-22; a kind Companies House renamed would read as another kind), and counts what it skips into `meta.json`'s detail. It reads the cached extracts only as one whole snapshot of one date, and an extract written by another version of the extraction (its file name carries the version) is neither resumed from nor read. Companies House's data-products guidance says it imposes no rules on how the register is used and leaves data protection to the user; the dataset's data.gov.uk entry gives its licence as "TBD", and gov.uk's footer licenses its own page text under the Open Government Licence. The credit line says the names are personal data. + +`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares title, given, middle and family with the filer's boxes, folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. A title box the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by nationality, misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, record it from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. + ## Corpus provenance `corpus.jsonl` is checked in as a test fixture. It was built by diff --git a/tools/differential/local_agreement.py b/tools/differential/local_agreement.py new file mode 100644 index 00000000..3ca5e461 --- /dev/null +++ b/tools/differential/local_agreement.py @@ -0,0 +1,396 @@ +"""Where the parser's reading of a local corpus disagrees with the fields its filers typed. + +The SCORE job (mechanisms.md#NAME-DATA-JOBS ("the job decides where its data lives")) +for a local corpus whose records carry the name's fields beside it: +each record's display name is parsed with the tree, +and each field is compared with the box the filer typed it into. What +cldr_agreement.py does for the committed fields files, over names that +are never committed -- so no CI run and no reviewer sees this score, +and its record lives in the cache beside the corpus, where a re-run +reports what moved since it was recorded: + + uv run python tools/differential/local_agreement.py psc-individuals --record --tree ../master-checkout + uv run python tools/differential/local_agreement.py psc-individuals + +The report prints counts and shapes, never a record: agreement overall +and by the group a source states (a nationality), the misses per field, +the commonest disagreement shapes, a shape being each disagreeing +field's word count as filed and as parsed (`middle 0->1, family 2->1` is +a two-word surname whose first word was read as a middle name), and the +set-aside title boxes at least TITLE_FLOOR records share (a word that +many filers typed is no one person's name, though it may be a name). +`--list N` prints up to N moved records of each kind, names and all, for +reading at one's own terminal. + +What counts: + +- Each field is compared case-folded, with periods dropped and runs of + spaces read as one. The display name is the boxes joined, so the + parser sees the filer's own spelling, and the fold only keeps a box's + case, periods or spacing from reading as a miss. +- There is no suffix box. A suffix the parser peels (`Jr`, `OBE`) stood + last in the display name, so it was typed into the last filled box, + normally the surname: the parser's family is compared with its suffix appended, + and the report counts the agreements that needed it. +- A filed title the parser does not read as the title, holding a word + the vocabulary lists in neither its titles nor its connectives, is + set aside rather than scored: a job (`Managing Partner`) or a name + typed into the title box is the filer's error, and it can move the + parse's other fields too. The set-aside titles are printed with their + counts where at least TITLE_FLOOR records share one (the --top + commonest), as leads for the vocabulary (a title nobody listed lands + there as well). + +The record is keyed to the corpus's records and to the scorer's own +code (scorer_fingerprint), so a refetch, or a change to how a field is +compared, is refused rather than reported as the parser's moves. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +import types +from collections import Counter +from collections.abc import Iterator +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any + +HERE = Path(__file__).resolve().parent +# The shared helpers and the local corpora, found as the corpus builders +# find corpus_build (see its docstring for why the directory is appended). +if str(HERE) not in sys.path: + sys.path.append(str(HERE)) +import corpus_build # noqa: E402 +import local_corpora # noqa: E402 + +if TYPE_CHECKING: + # Imported at run time only once --tree has put the tree on sys.path. + from nameparser import Lexicon, Parser + +RECORD = "agreement.jsonl" + + +@dataclass(frozen=True) +class Fielded: + """How a local corpus states its fields: record key -> the parser's + role it is compared with, in the order the display name joins them, + and the record key the report groups by.""" + fields: dict[str, str] + group: str + + +FIELDED = {local_corpora.PSC: Fielded( + {"title": "title", "forename": "given", "middle_name": "middle", + "surname": "family"}, group="nationality")} +#: The fewest records sharing a set-aside title for the report to print +#: it: a word typed into the title box by this many filers is no one +#: person's name. +TITLE_FLOOR = 100 +#: The fewest records of a group for the report to give it a row. +GROUP_FLOOR = 1_000 + + +def fold(text: str) -> str: + return " ".join(text.replace(".", "").casefold().split()) + + +def _title_words(lexicon: Lexicon) -> frozenset[str]: + return frozenset(lexicon.titles) | frozenset(lexicon.conjunctions) + + +def unlisted(title: str, listed: frozenset[str]) -> bool: + """Whether a filed title holds a word the vocabulary lists in + neither its titles nor its connectives (`listed`).""" + return any(word.strip(".").casefold() not in listed for word in title.split()) + + +@dataclass(frozen=True) +class Reading: + """One record's reading: each field as parsed where it disagrees + with the box, whether the record was set aside, and whether the + family agreed only with the parser's suffix appended.""" + got: dict[str, str] + set_aside: bool = False + by_suffix: bool = False + + def entry(self) -> dict[str, Any]: + return {"got": self.got, **({"set_aside": True} if self.set_aside else {})} + + +def read(record: dict[str, str], fielded: Fielded, parser: Parser, + listed: frozenset[str]) -> Reading: + """`record` parsed and compared box by box (see the module docstring).""" + name = parser.parse(record["name"]) + got = {role: getattr(name, role) for role in fielded.fields.values()} + by_suffix = False + if name.suffix: + with_suffix = f"{got['family']} {name.suffix}".strip() + by_suffix = fold(with_suffix) == fold(record.get(_box(fielded, "family"), "")) + got["family"] = with_suffix + wrong = {role: got[role] for box, role in fielded.fields.items() + if fold(record.get(box, "")) != fold(got[role])} + title = record.get(_box(fielded, "title"), "") + if "title" in wrong and title and unlisted(title, listed): + return Reading(wrong, set_aside=True) + return Reading(wrong, by_suffix=by_suffix and not wrong) + + +def _box(fielded: Fielded, role: str) -> str: + return next(box for box, r in fielded.fields.items() if r == role) + + +def shape(record: dict[str, str], fielded: Fielded, got: dict[str, str]) -> str: + """Each disagreeing field's word count as filed and as parsed.""" + return ", ".join(f"{role} {len(record.get(box, '').split())}->{len(got[role].split())}" + for box, role in fielded.fields.items() if role in got) + + +def key(record: dict[str, str]) -> str: + return corpus_build.jsonl_line(record, sort_keys=True) + + +def group_of(record: dict[str, str], fielded: Fielded) -> str: + return " ".join(record.get(fielded.group, "").split()).title() or "(none)" + + +def records(local: local_corpora.Local) -> Iterator[dict[str, str]]: + """raw.jsonl's records, streamed rather than held, checked against + its sha256 once the last is read, and each holding a name and + nothing but strings as it is read: CorpusDamaged either way, before any caller reports or + records what it read.""" + path = local_corpora.directory(local.source, create=False) / local_corpora.RAW + damaged = local_corpora.CorpusDamaged( + f"{path} does not match the sha256 in {local_corpora.META}; fetch it " + f"again: {local_corpora.fetch_command(local.source, refresh=True)}") + digest = hashlib.sha256() + with path.open("rb") as raw: + for line in raw: + digest.update(line) + try: + record = json.loads(line) + except ValueError: + raise damaged from None + if (not isinstance(record, dict) or not isinstance(record.get("name"), str) + or not all(isinstance(v, str) for v in record.values())): + raise damaged + yield record + if digest.hexdigest() != local.meta["sha256"]: + raise damaged + + +@dataclass +class Score: + """What one run read: per group, the records scored and agreeing and + the misses per field; the set-aside titles; the shapes; and every + record that did not wholly agree, by key.""" + scored: Counter[str] + agreed: Counter[str] + misses: dict[str, Counter[str]] + set_aside: Counter[str] + shapes: Counter[str] + by_suffix: int + entries: dict[str, dict[str, Any]] + + +def score(rows: Iterator[dict[str, str]] | list[dict[str, str]], fielded: Fielded, + parser: Parser) -> Score: + listed = _title_words(parser.lexicon) + result = Score(Counter(), Counter(), {}, Counter(), Counter(), 0, {}) + for record in rows: + reading = read(record, fielded, parser, listed) + if reading.got: + result.entries[key(record)] = reading.entry() + if reading.set_aside: + result.set_aside[record.get(_box(fielded, "title"), "")] += 1 + continue + group = group_of(record, fielded) + result.scored[group] += 1 + if not reading.got: + result.agreed[group] += 1 + result.by_suffix += reading.by_suffix + continue + result.misses.setdefault(group, Counter()).update(reading.got.keys()) + result.shapes[shape(record, fielded, reading.got)] += 1 + return result + + +def _pct(n: int, of: int) -> str: + return f"{100 * n / of:5.1f}%" if of else " n/a" + + +def report(result: Score, fielded: Fielded, top: int) -> list[str]: + roles = list(fielded.fields.values()) + total, agreed = sum(result.scored.values()), sum(result.agreed.values()) + misses: Counter[str] = sum(result.misses.values(), Counter()) + out = [f"scored {total:,}; set aside {sum(result.set_aside.values()):,} " + f"(a title box the parser read otherwise, holding an unlisted word)", + f"every field agrees: {agreed:,} of {total:,} ({_pct(agreed, total).strip()}); " + f"{result.by_suffix:,} of them with the parser's suffix in the surname box", + "misses per field: " + ", ".join(f"{r} {misses[r]:,}" for r in roles), "", + f"by {fielded.group} (at least {GROUP_FLOOR:,} records), misses per field:", + f" {fielded.group:<24} {'records':>10} {'agree':>7} " + + " ".join(f"{r:>8}" for r in roles)] + for group, n in result.scored.most_common(): + if n < GROUP_FLOOR: + break + row = result.misses.get(group, Counter()) + out.append(f" {group[:24]:<24} {n:>10,} {_pct(result.agreed[group], n)} " + + " ".join(f"{row[r]:>8,}" for r in roles)) + out += ["", f"the {top} commonest disagreement shapes (field: words filed->parsed):"] + out += [f" {n:>9,} {s}" for s, n in result.shapes.most_common(top)] + out += ["", f"set-aside title boxes written by at least {TITLE_FLOOR} records:"] + out += [f" {n:>9,} {t}" for t, n in result.set_aside.most_common() + if n >= TITLE_FLOOR][:top] + return out + + +def record_path(source: str) -> Path: + return local_corpora.directory(source, create=False) / RECORD + + +def write_record(path: Path, local: local_corpora.Local, entries: dict[str, dict[str, Any]], + tree: str) -> None: + header = {"raw_sha256": local.meta["sha256"], "scorer": scorer_fingerprint(), + "data_date": local.meta["data_date"], "tree": tree} + lines = [corpus_build.jsonl_line(header)] + lines += [corpus_build.jsonl_line({"key": k, **v}) for k, v in sorted(entries.items())] + corpus_build.write_atomic(path, ("\n".join(lines) + "\n").encode()) + + +def read_record(path: Path, local: local_corpora.Local) -> dict[str, dict[str, Any]]: + """The recorded entries, or SystemExit naming the record command + where there is no record, one that is not a record this tool wrote, + one recorded from another fetch of the corpus, which would report + the records the source gained and lost as moves, or one recorded by + another version of the scorer, which would report its own change as + the parser's.""" + again = f"Record it again from the base checkout: {_record_command(local.source)}" + if not path.exists(): + raise SystemExit(f"no record at {path}: record one first, from the base " + f"checkout: {_record_command(local.source)}") + try: + header, *rows = corpus_build.jsonl_lines(path.read_text(encoding="utf-8")) + recorded = json.loads(header) + entries = {row["key"]: {k: v for k, v in row.items() if k != "key"} + for row in map(json.loads, rows)} + if not isinstance(recorded, dict): + raise TypeError("its header is not an object") + except (OSError, ValueError, KeyError, TypeError, AttributeError) as exc: + raise SystemExit(f"{path} is not a record this tool wrote ({exc}). {again}" + ) from None + if recorded.get("raw_sha256") != local.meta["sha256"]: + raise SystemExit( + f"{path} was recorded from another fetch of {local.source} (data as " + f"of {recorded.get('data_date', 'unknown')}; the corpus now is " + f"{local.meta['data_date']}). {again}") + if recorded.get("scorer") != scorer_fingerprint(): + raise SystemExit(f"{path} was recorded by another version of this scorer, " + f"so its moves would be the scorer's. {again}") + return entries + + +#: What decides a record's reading, besides the parser: scorer_fingerprint +#: hashes them, as local_corpora.fold_fingerprint hashes the word fold. +_SCORER_CODE = ("fold", "_title_words", "unlisted", "read", "_box", "Reading", + "key", "score") + + +def scorer_fingerprint() -> str: + """A hash of the code that turns a parse into a record's entry and + decides which records get one (_SCORER_CODE), of + corpus_build.jsonl_line, which writes an entry's key, and of FIELDED, + read from the running code.""" + module = globals() + parts = [repr(sorted((k, repr(v)) for k, v in FIELDED.items()))] + for name in _SCORER_CODE: + target = module[name] + functions = ([v for v in vars(target).values() if isinstance(v, types.FunctionType)] + if isinstance(target, type) else [target]) + for fn in functions: + parts += local_corpora._code_digest(fn.__code__) + parts += local_corpora._code_digest(corpus_build.jsonl_line.__code__) + return hashlib.sha256("\0".join(parts).encode()).hexdigest() + + +def _record_command(source: str) -> str: + return (f"local_agreement.py {source} --record --tree ") + + +def moves(old: dict[str, dict[str, Any]], new: dict[str, dict[str, Any]], + ) -> dict[str, list[str]]: + """Each record whose reading moved, by kind: `regression` (it agreed + and now does not), `progress` (it disagreed and now agrees), + `changed` (it disagrees, or is set aside, differently). Both sides + read the same fetch, so no record is gained or lost.""" + found: dict[str, list[str]] = {"regression": [], "progress": [], "changed": []} + for k in sorted(set(old) | set(new)): + if k not in old: + found["regression"].append(k) + elif k not in new: + found["progress"].append(k) + elif old[k] != new[k]: + found["changed"].append(k) + return found + + +def summarize(found: dict[str, list[str]], old: dict[str, dict[str, Any]], + new: dict[str, dict[str, Any]]) -> list[str]: + """One line per kind of move, with the fields it moved: a field + counts where the record disagrees on it on either side.""" + out = [] + for kind, keys in found.items(): + if not keys: + continue + fields: Counter[str] = Counter() + for k in keys: + fields.update(set(old.get(k, {}).get("got", {})) | set(new.get(k, {}).get("got", {}))) + out.append(f"{kind}: {len(keys):,} records (" + + ", ".join(f"{f} {n:,}" for f, n in fields.most_common()) + ")") + return out + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("source", choices=sorted(FIELDED)) + ap.add_argument("--record", action="store_true", + help="rewrite the record kept beside the corpus") + ap.add_argument("--tree", metavar="PATH", help="read with another checkout") + ap.add_argument("--top", type=int, default=15, metavar="N") + ap.add_argument("--list", type=int, default=0, metavar="N", + help="print up to N moved records of each kind, names and all") + args = ap.parse_args(argv) + local = local_corpora.require(args.source) + if not args.record: + old = read_record(record_path(args.source), local) + root = Path(args.tree).resolve() if args.tree else HERE.parents[1] + if args.tree: + sys.path.insert(0, str(root)) + import nameparser + from snapshot import check_source + at = check_source(nameparser.__file__, root) + fielded = FIELDED[args.source] + print(local.header()) + try: + result = score(records(local), fielded, nameparser.Parser()) + except local_corpora.CorpusDamaged as exc: + raise SystemExit(str(exc)) from None + print("\n".join(report(result, fielded, args.top))) + if args.record: + write_record(record_path(args.source), local, result.entries, str(at.parent)) + print(f"\nrecorded {len(result.entries):,} records that did not wholly " + f"agree to {record_path(args.source)}") + return 0 + found = moves(old, result.entries) + print() + print("\n".join(summarize(found, old, result.entries)) or "matches the record") + for kind, keys in found.items(): + for k in keys[:args.list]: + print(f" {kind}: {k} recorded {old.get(k)} now {result.entries.get(k)}") + return 1 if any(found.values()) else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 92365c75..bc5bee86 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -5,10 +5,10 @@ real names as people type them, read with the snapshot's machinery, kept out of the repository because the names are real people's and nobody vetted them for publication. It also does WATCH outside CI (snapshot.py --local), -but neither gates CI nor feeds SCORE, whose scorer reads only committed -fields files: no reviewer and no CI run can see it. What reaches the -repository is what a contributor cites: counts, and shapes written with -made-up names. +and, for a source whose records carry the fields their filers typed, +SCORE outside CI (local_agreement.py): no reviewer and no CI run can see +either. What reaches the repository is what a contributor cites: counts, +and shapes written with made-up names. uv run python tools/differential/local_corpora.py fetch ctgov-officials uv run python tools/differential/local_corpora.py tally ctgov-officials @@ -124,6 +124,19 @@ publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says; only the DonorName column is kept. + +psc-individuals -- the people with significant control of UK companies, +and the beneficial owners of overseas entities, from Companies House's +daily snapshot: each person's display name, the four boxes the filer +typed it into (title, forename, middle name, surname) and the stated +nationality, and nothing else. A local corpus whose names carry their +fields (local_agreement.FIELDED), so local_agreement.py scores the +parser against them. A record is a distinct filed name, and the corpus +keeps one in PSC_SAMPLE of them (1,040,708 of 8,322,939 on 2026-10-22). +The snapshot comes in parts (33 on 2026-10-22), whose extracts are +cached beside the corpus (renew_psc says why), and `fetch --refresh` +downloads the day's snapshot again. Companies House sets no terms on the +register's use; the names are personal data, which the credit line says. """ from __future__ import annotations @@ -131,6 +144,7 @@ import csv import functools import dataclasses +import gzip import hashlib import http.client import inspect @@ -138,6 +152,7 @@ import json import os import re +import shutil import sys import time import types @@ -399,16 +414,15 @@ def fetch_ctgov() -> Fetched: FEC_MIN_COMMA = 0.9 -def _fec_download(path: Path) -> None: - """Download FEC_URL to `path` through a .part file, refusing a body +def _download(url: str, path: Path) -> None: + """Download `url` to `path` through a .part file, refusing a body whose length differs from the stated Content-Length, or with no Content-Length at all: a cut zip would otherwise fail later as a - BadZipFile that says nothing about the download. The cached zip is - replaced only once the new one is whole, a failed download leaves no - .part behind, and a zip of another cycle is removed once this one - has arrived.""" - print(f"downloading {FEC_URL} ...", file=sys.stderr) - request = urllib.request.Request(FEC_URL, headers={"User-Agent": USER_AGENT}) + BadZipFile that says nothing about the download. `path` is replaced + only once the new file is whole, and a failed download leaves no + .part behind.""" + print(f"downloading {url} ...", file=sys.stderr) + request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) partial = path.with_name(path.name + ".part") try: got = 0 @@ -419,12 +433,18 @@ def _fec_download(path: Path) -> None: out.write(chunk) got += len(chunk) if not stated.isdigit() or int(stated) != got: - raise ValueError(f"{FEC_URL} sent {got:,} bytes where it stated " + raise ValueError(f"{url} sent {got:,} bytes where it stated " f"{stated or 'no length'}; fetch again") except BaseException: partial.unlink(missing_ok=True) raise os.replace(partial, path) + + +def _fec_download(path: Path) -> None: + """Download FEC_URL over the cached zip (_download), and remove a zip + of another cycle once this one has arrived.""" + _download(FEC_URL, path) for stale in path.parent.glob("indiv*.zip"): if stale != path: stale.unlink() @@ -1393,6 +1413,269 @@ def fetch_ukec() -> Fetched: return dataclasses.replace(_ukec_read(body, before), data_date=when) +# -- Companies House people with significant control ------------------------- + +PSC = "psc-individuals" +PSC_PAGE = "https://download.companieshouse.gov.uk/en_pscdata.html" +#: A part of the day's snapshot as the page links it: its file, date, +#: number, and how many parts there are. +PSC_PART = re.compile(r'href="(psc-snapshot-(\d{4}-\d{2}-\d{2})_(\d+)of(\d+)\.zip)"') +#: The two kinds of record that name a person: a company's PSC, and an +#: overseas entity's registrable beneficial owner, filed in the same boxes +#: (70 of part 1's 461,778 individuals on 2026-10-22). The corporate, +#: legal-person and super-secure kinds name no person or none at all. +PSC_KINDS = ("individual-person-with-significant-control", + "individual-beneficial-owner") +#: The four name boxes, in the order the display name joins them. +PSC_ELEMENTS = ("title", "forename", "middle_name", "surname") +#: What a record keeps: the display name, the boxes and the nationality. +PSC_KEPT = ("name", *PSC_ELEMENTS, "nationality") +#: The share of individuals (of lines, for a line not JSON) a fetch skips +#: before it refuses the snapshot. Of part 1's 461,778 individuals on +#: 2026-10-22 none had no name_elements and every display name was its +#: boxes joined; 2 of the whole snapshot's 14,025,738 were not. +PSC_MAX_BAD = 1 / 10_000 +#: The least share of lines that are PSC_KINDS records before a fetch +#: refuses the snapshot: 14,025,738 of 16,071,681 (87%) on 2026-10-22. +#: A kind renamed by Companies House is counted as another kind, and +#: without this floor the corpus would shrink to whatever kind is left. +PSC_MIN_INDIVIDUALS = 1 / 2 +#: The cached extracts, one per part, beside the corpus. +PSC_PARTS = "parts" +#: The extract format, in each extract's file name: a change to what +#: _psc_extract keeps or refuses bumps it, so an extract written by +#: another version is neither resumed from nor read. +PSC_EXTRACT_VERSION = 1 +_PSC_EXTRACT = re.compile(r"psc-snapshot-(\d{4}-\d{2}-\d{2})_(\d+)of(\d+)\.v(\d+)\.jsonl\.gz") +#: The corpus keeps one filed name in PSC_SAMPLE, chosen by a hash of the +#: record (psc_sampled). The whole snapshot held 8,322,939 distinct filed +#: names on 2026-10-22, 1.09 GB as JSON lines, and a local corpus is held +#: in memory whole (Local.records), and snapshot.py --local reads every +#: name with both APIs. A record's hash does not move with the snapshot, +#: so a refetch keeps or drops the same record as before, and no +#: nationality or shape is favoured. +PSC_SAMPLE = 8 + + +def psc_sampled(line: str) -> bool: + """Whether the corpus keeps the record written as this line.""" + return int(hashlib.sha256(line.encode()).hexdigest()[:8], 16) % PSC_SAMPLE == 0 + + +def _extract_name(file: str) -> str: + """The extract cached for a part's zip.""" + return file.replace(".zip", f".v{PSC_EXTRACT_VERSION}.jsonl.gz") + + +def _psc_extract(path: Path, member: str) -> bytes: + """One downloaded part, read down to what the corpus keeps: gzipped + JSON lines, a header naming the part and holding its counts, and then + its distinct records, each its PSC_KEPT values with the empty ones + left out. A record that is not one of PSC_KINDS is counted + (records_of_other_kinds) and dropped, as is an individual whose + name_elements are missing or not strings, or whose display name is + not its boxes joined by spaces: the score compares the boxes with a + parse of that name. Everything else in a record (address, date of + birth, company number, natures of control) is read past.""" + counts: Counter[str] = Counter() + lines: set[str] = set() + with zipfile.ZipFile(path) as archive: + if member not in archive.namelist(): + raise ValueError(f"{path} holds no {member}") + with archive.open(member) as source: + for line in source: + counts["lines"] += 1 + try: + record = json.loads(line) + except ValueError: + counts["lines_not_json"] += 1 + continue + data = record.get("data") if isinstance(record, dict) else None + kind = data.get("kind") if isinstance(data, dict) else None + if not isinstance(data, dict) or kind not in PSC_KINDS: + counts["records_of_other_kinds"] += 1 + continue + counts[kind] += 1 + name, elements = data.get("name"), data.get("name_elements") + if (not isinstance(name, str) or not isinstance(elements, dict) + or not all(isinstance(elements.get(f, ""), str) + for f in PSC_ELEMENTS)): + counts["individuals_without_name_elements"] += 1 + continue + if " ".join(v for f in PSC_ELEMENTS if (v := elements.get(f))) != name: + counts["names_not_their_boxes_joined"] += 1 + continue + nationality = data.get("nationality") + if not isinstance(nationality, str) or not nationality: + counts["individuals_without_a_nationality"] += 1 + nationality = "" + kept = {"name": name, "nationality": nationality, + **{f: elements.get(f, "") for f in PSC_ELEMENTS}} + lines.add(corpus_build.jsonl_line( + {k: v for k, v in kept.items() if v}, sort_keys=True)) + header = corpus_build.jsonl_line({"part": member, "counts": dict(counts)}) + return gzip.compress("".join(f"{line}\n" for line in [header, *sorted(lines)]).encode()) + + +def _psc_listing() -> tuple[str, list[tuple[int, str]]]: + """The day's snapshot as the download page lists it: its date and + each part's number and file, or ValueError where the page links no + parts, or parts of more than one snapshot, or not every part. A part + linked twice is one part.""" + page = corpus_build.fetch_bytes(PSC_PAGE, user_agent=USER_AGENT, timeout=60) + links = PSC_PART.findall(page.decode("utf-8", "replace")) + if not links: + raise ValueError(f"{PSC_PAGE} links no snapshot parts: the page has changed") + dates, totals = {d for _, d, _, _ in links}, {int(m) for *_, m in links} + numbers = sorted(int(n) for _, _, n, _ in set(links)) + if len(dates) != 1 or len(totals) != 1 or numbers != list(range(1, max(totals) + 1)): + raise ValueError(f"{PSC_PAGE} links parts {numbers} of {sorted(totals)} " + f"dated {sorted(dates)}, not every part of one snapshot; " + f"fetch again") + return dates.pop(), sorted({(int(n), f) for f, _, n, _ in links}) + + +def renew_psc() -> None: + """Download every part of the day's snapshot and keep its extract + (_psc_extract) in parts.new/, then swap parts.new/ in for parts/. + + Parts, not the single zip, and extracts, not zips: the 33 parts of + 2026-10-22 were 2.4 GB, as the single zip was (2,443,088,441 bytes), + and their extracts 168 MB (part 1's 5.6 MB of its 78.6 MB). A + download that fails costs one part, and a refresh run again resumes: + the extracts of the same date and version already in parts.new/ are + kept, and anything else there is removed. The page's files are + overwritten each morning, so a part that is gone mid-fetch most + likely means the snapshot was replaced; the next run lists the new + one.""" + date, parts = _psc_listing() + root = directory(PSC) + staging, target, old = (root / f"{PSC_PARTS}{end}" for end in (".new", "", ".old")) + staging.mkdir(exist_ok=True) + keep = {_extract_name(file) for _, file in parts} + for stale in staging.iterdir(): + if stale.name not in keep: + stale.unlink() + for _, file in parts: + out = staging / _extract_name(file) + if out.exists(): + continue + path, url = staging / file, urllib.parse.urljoin(PSC_PAGE, file) + try: + _download(url, path) + body = _psc_extract(path, file.replace(".zip", ".txt")) + except urllib.error.HTTPError as exc: + if exc.code != 404: + raise + exc.close() + raise ValueError(f"{url} is gone (404): the {date} snapshot was most " + f"likely replaced during the fetch; fetch again") from None + except (zipfile.BadZipFile, zlib.error) as exc: + raise ValueError(f"{file} is not a readable zip ({exc}); fetch again" + ) from None + finally: + path.unlink(missing_ok=True) + corpus_build.write_atomic(out, body) + shutil.rmtree(old, ignore_errors=True) + if target.exists(): + os.replace(target, old) + os.replace(staging, target) + shutil.rmtree(old, ignore_errors=True) + + +def _psc_part(path: Path) -> tuple[dict[str, int], list[str]]: + """One extract's counts and record lines, or ValueError naming the + refresh where it is not an extract _psc_extract wrote for its name.""" + member = re.sub(r"\.v\d+\.jsonl\.gz$", ".txt", path.name) + try: + header, *records = corpus_build.jsonl_lines( + gzip.decompress(path.read_bytes()).decode()) + head = json.loads(header) + counts = head["counts"] + if (head["part"] != member or not isinstance(counts, dict) + or not all(type(n) is int for n in counts.values())): + raise ValueError("its header does not describe it") + except (OSError, EOFError, ValueError, KeyError, TypeError, zlib.error) as exc: + raise ValueError(f"{path} is not an extract this tool wrote ({exc})" + f"{_again(PSC)}") from None + return counts, records + + +def _psc_read(parts: Path) -> Fetched: + """The corpus from the cached extracts: every part of one snapshot, + checked by name before any is read, or ValueError naming the + refresh. A record is a distinct filed name -- the display name, its + four boxes and the nationality -- however many companies file it, + one in PSC_SAMPLE of them kept; a sampled name with every box empty + is counted and dropped. Past PSC_MAX_BAD of the individuals skipped + for either reason _psc_extract skips one, or of the lines not JSON, + or with fewer than PSC_MIN_INDIVIDUALS of the lines individuals, the + snapshot is refused rather than read as a smaller corpus.""" + found: dict[int, Path] = {} + dates, totals = set(), set() + for path in sorted(parts.glob("*.jsonl.gz")): + named = _PSC_EXTRACT.fullmatch(path.name) + if not named or int(named[4]) != PSC_EXTRACT_VERSION: + raise ValueError(f"{path} is not an extract of version " + f"{PSC_EXTRACT_VERSION}{_again(PSC)}") + number = int(named[2]) + if number in found: + raise ValueError(f"{parts} holds part {number} twice{_again(PSC)}") + found[number] = path + dates.add(named[1]) + totals.add(int(named[3])) + if (len(dates) != 1 or len(totals) != 1 + or sorted(found) != list(range(1, max(totals) + 1))): + raise ValueError(f"{parts} holds parts {sorted(found)} of {sorted(totals)} " + f"dated {sorted(dates)}, not every part of one snapshot" + f"{_again(PSC)}") + counts: Counter[str] = Counter() + lines: set[str] = set() + for number in sorted(found): + part_counts, records = _psc_part(found[number]) + counts.update(part_counts) + lines.update(line for line in records if psc_sampled(line)) + individuals = sum(counts[kind] for kind in PSC_KINDS) + if individuals < counts["lines"] * PSC_MIN_INDIVIDUALS: + raise ValueError(f"{individuals:,} of {counts['lines']:,} lines are " + f"{' or '.join(PSC_KINDS)} records: a kind has been " + f"renamed or moved under PSC_KINDS") + for key, of, what in ( + ("lines_not_json", counts["lines"], "lines are not JSON"), + ("individuals_without_name_elements", individuals, + "individuals have no name_elements"), + ("names_not_their_boxes_joined", individuals, + "individuals' display names are not their four boxes joined")): + if counts[key] > of * PSC_MAX_BAD: + raise ValueError(f"{counts[key]:,} of {of:,} {what}: the snapshot's " + f"layout has changed under PSC_ELEMENTS") + records = [json.loads(line) for line in sorted(lines)] + named_records = [record for record in records if "name" in record] + if len(named_records) < len(records): + counts["sampled_names_that_are_empty"] = len(records) - len(named_records) + return Fetched(named_records, dates.pop(), { + "parts": max(totals), "sample": f"1 in {PSC_SAMPLE}", + **{key: counts[key] for key in sorted(counts)}}) + + +def fetch_psc() -> Fetched: + """One in PSC_SAMPLE of the snapshot's distinct filed names + (_psc_read), read from the extracts cached beside the corpus + (downloaded when missing, and again by `fetch --refresh`). The data + date is the date the part files are named by; the snapshot holds + what was filed up to the end of the day before.""" + parts = directory(PSC) / PSC_PARTS + if not parts.is_dir(): + renew_psc() + return _psc_read(parts) + + +def check_psc(meta: dict[str, Any]) -> str: + sample = meta["detail"].get("sample") + return "" if sample == f"1 in {PSC_SAMPLE}" else ( + f"keeps {sample} filed names, and this tool keeps 1 in {PSC_SAMPLE}") + + SOURCES: dict[str, Source] = { "ctgov-officials": Source( credit="ClinicalTrials.gov (NLM) study officials", unit="officials", @@ -1416,6 +1699,12 @@ def fetch_ukec() -> Fetched: credit="UK Electoral Commission register of donations, individual " "donors (reuse terms for the register not stated)", unit="names", fetch=fetch_ukec), + + PSC: Source( + credit="Companies House people with significant control, individuals " + "(a public register: Companies House sets no terms on its use, " + "and data protection law applies to the names)", + unit="filed names", fetch=fetch_psc, renew=renew_psc, check=check_psc), }