diff --git a/AGENTS.md b/AGENTS.md index 297cee327..f4a02b870 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -278,7 +278,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Shekh` in front, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. ## Gotchas diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 1837000d8..8533a7d31 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -10,6 +10,7 @@ counts read as real zeros. The network is a stub answering by URL. """ import csv +import dataclasses import email.message import gzip import hashlib @@ -41,9 +42,9 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc, jrc = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal = ( importlib.import_module(f"local_{key}") - for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc")) + for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -180,6 +181,19 @@ def test_a_fetch_paging_a_study_twice_is_refused( local.fetch(_SOURCE) +@pytest.mark.parametrize("name", ["", " ", None, 3], + ids=["empty", "blank", "null", "a number"]) +def test_a_fetched_record_whose_name_is_empty_or_not_text_is_refused( + monkeypatch: pytest.MonkeyPatch, name: object) -> None: + # The corpus skips a second key's empty value (Source.corpus_keys); + # an empty "name" is a fetch's own fault, refused rather than skipped. + monkeypatch.setitem(local.SOURCES, _SOURCE, local.Source( + "credit", "names", lambda: local.Fetched([{"name": "Ann Lee"}, {"name": name}], + _DATE, {}))) + with pytest.raises(ValueError, match="a fetched record has no name"): + local.fetch(_SOURCE) + + def test_a_fetch_with_no_names_is_refused( monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: # Every official nameless, as a renamed field would leave them: the @@ -1931,6 +1945,742 @@ def urlopen(request: urllib.request.Request, timeout: int) -> _Response: local.fetch(_UKEC) +# -- Nepal's local election candidates ------------------------------------------- + +_NEPAL = "nepal-candidates" +_NEPAL_TOKEN = "a" * 32 + + +def _nepal_row(romanized: object, devanagari: object = "राम बहादुर", **extra: object + ) -> dict[str, object]: + """A candidate as a local body's file writes one, every name invented.""" + return {"CandidateName": devanagari, "CandidateNameEng": romanized, + "Gender": "पुरुष", "Age": 40, "Ward": 1, "PostId": 6, "CandidateID": 1, + "PartyID": 2, "SymbolID": 3, "TotalVoteReceived": 99, **extra} + + +#: Three bodies: a pair twice in one body (once padded) and again in +#: another, the same Devanagari under a second romanization, a candidate +#: with no Devanagari name, and an empty file. +_NEPAL_FILES: dict[int, object] = { + 101: [_nepal_row("Dil Maya B.K.", "दिल माया वि.क."), + _nepal_row("Sita Kumari K.C.", "सीता कुमारी के.सी."), + _nepal_row(" Dil Maya B.K. ", "दिल माया वि.क. "), + _nepal_row("Hari Thapa", "")], + 102: [_nepal_row("Dil Maya B.K.", "दिल माया वि.क."), + _nepal_row("Dil Maya BK", "दिल माया वि.क.")], + 103: [], +} + + +def _nepal_error(url: str, code: int) -> urllib.error.HTTPError: + return urllib.error.HTTPError(url, code, "x", email.message.Message(), None) + + +class _Root(_Response): + """The site's root page, setting `cookies`.""" + + def __init__(self, *cookies: str) -> None: + super().__init__(b"") + self.headers = email.message.Message() + for cookie in cookies: + self.headers["Set-Cookie"] = cookie + + +class _Site: + """result.election.gov.np, offline. The root sets a token; the + handler answers 403 to a request not echoing the current one, else + the lookup from `files`' ids and each file from `files` (a list + served as JSON, bytes as they are, an exception raised; a missing id + is a 404). `expire` makes a token lapse after that many handler + requests, as a session does.""" + + def __init__(self, monkeypatch: pytest.MonkeyPatch, files: dict[int, object], + *, bom: bytes = b"\xef\xbb\xbf", listed: object = None) -> None: + self.files, self.bom, self.listed = files, bom, listed + self.requests: list[urllib.request.Request] = [] + self.tokens = 0 + self.expire: int | None = None + self.used = 0 + monkeypatch.setattr(time, "sleep", lambda seconds: None) + monkeypatch.setattr(nepal, "NEPAL_BODIES", len(files)) + # Pinned as if an empty file had held one candidate before it + # was blanked: every real file did. + monkeypatch.setattr(nepal, "NEPAL_CANDIDATES", sum( + len(rows) or 1 for rows in files.values() if isinstance(rows, list))) + monkeypatch.setattr(urllib.request, "urlopen", self.urlopen) + + @property + def token(self) -> str: + return f"{_NEPAL_TOKEN[:-1]}{self.tokens}" + + def handler(self, path: str) -> str: + return nepal.NEPAL_HANDLER + path + + def urlopen(self, request: urllib.request.Request, timeout: int) -> _Response: + url = request.full_url + self.requests.append(request) + if url == nepal.NEPAL_SITE: + self.tokens += 1 + self.used = 0 + return _Root("ASP.NET_SessionId=s1; path=/; HttpOnly", + f"CsrfToken={self.token}; path=/") + assert url.startswith(nepal.NEPAL_HANDLER) + self.used += 1 + if (request.get_header("X-csrf-token") != self.token + or f"CsrfToken={self.token}" not in (request.get_header("Cookie") or "") + or self.expire is not None and self.used > self.expire): + raise _nepal_error(url, 403) + path = url[len(nepal.NEPAL_HANDLER):] + if path == nepal.NEPAL_LOOKUP: + listed = self.listed if self.listed is not None else [ + {"id": i, "name": "नगरपालिका", "parentId": 1} for i in self.files] + return _Response(self.bom + json.dumps(listed).encode()) + body = int(path.rsplit("/", 1)[1].removesuffix(".json")) + if body not in self.files: + raise _nepal_error(url, 404) + answer = self.files[body] + if callable(answer): + answer = answer() + if isinstance(answer, Exception): + raise answer + if isinstance(answer, bytes): + return _Response(answer) + return _Response(self.bom + json.dumps(answer, ensure_ascii=False).encode()) + + def asked(self) -> list[str]: + return [r.full_url.removeprefix(nepal.NEPAL_HANDLER) for r in self.requests] + + +@pytest.fixture +def site(monkeypatch: pytest.MonkeyPatch) -> _Site: + return _Site(monkeypatch, dict(_NEPAL_FILES)) + + +def _tolerate_nepal_s_small_fixture(monkeypatch: pytest.MonkeyPatch) -> None: + """Let _NEPAL_FILES, whose 6 candidates include one with no + Devanagari name and whose 3 files one that is empty, pass + NEPAL_MAX_NAMELESS, NEPAL_MAX_MISSING and NEPAL_MAX_SHORT.""" + monkeypatch.setattr(nepal, "NEPAL_MAX_NAMELESS", 1) + monkeypatch.setattr(nepal, "NEPAL_MAX_MISSING", 1) + monkeypatch.setattr(nepal, "NEPAL_MAX_SHORT", 1) + + +def test_a_nepal_fetch_keeps_each_pair_once_and_counts_the_rest( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + _tolerate_nepal_s_small_fixture(monkeypatch) + corpus = local.fetch(_NEPAL) + assert corpus.records == [ + {"name": "Dil Maya B.K.", "devanagari": "दिल माया वि.क."}, + {"name": "Dil Maya BK", "devanagari": "दिल माया वि.क."}, + {"name": "Hari Thapa", "devanagari": ""}, + {"name": "Sita Kumari K.C.", "devanagari": "सीता कुमारी के.सी."}] + assert re.fullmatch(r"\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ", corpus.meta["data_date"]) + assert corpus.meta["detail"] == { + "election": "Election2079", "local_bodies": 3, "local_bodies_missing": "", + "local_bodies_empty": "103", "files": 3, "candidates": 6, + "without_the_name_keys": 0, "without_a_romanized_name": 0, + "without_a_devanagari_name": 1, "devanagari_name_without_devanagari": 0, + "romanized_name_not_latin": 0, "sessions_renewed": 0, "largest_file": 4} + assert corpus.meta["corpus_keys"] == ["name", "devanagari"] + assert corpus.header().endswith("4 name pairs (local corpus nepal-candidates)") + assert site.asked() == [nepal.NEPAL_SITE, nepal.NEPAL_LOOKUP, + *(nepal.NEPAL_BODY.format(id=i) for i in (101, 102, 103))] + + +def test_a_nepal_fetch_writes_no_field_but_the_two_names( + site: _Site, monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + # Gender, age, ward, votes and the ids stay in the site's files. + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + raw = (_own_cache / "local" / _NEPAL / "raw.jsonl").read_text() + assert {key for line in raw.splitlines() for key in json.loads(line)} == { + "name", "devanagari"} + assert "पुरुष" not in raw and "99" not in raw + + +def test_a_nepal_corpus_holds_both_scripts_names_for_the_snapshot( + site: _Site, monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + # The romanized name is the record's "name"; the Devanagari one is + # read too, so a snapshot watches both scripts, an empty one skipped. + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + corpus = _own_cache / "local" / _NEPAL / "corpus.jsonl" + texts = sorted(json.loads(line) for line in corpus.read_text().splitlines()) + assert texts == ["Dil Maya B.K.", "Dil Maya BK", "Hari Thapa", "Sita Kumari K.C.", + "दिल माया वि.क.", "सीता कुमारी के.सी."] + assert local.load(_NEPAL).meta["names"] == 6 + assert snapshot.main(["--local", _NEPAL, "--record"]) == 0 + recorded = _own_cache / "local" / _NEPAL / "snapshot.jsonl" + assert len(recorded.read_text().splitlines()) == 6 + + +def test_a_nepal_name_keeps_its_vowel_signs_virama_and_joiners( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + # What the corpus is for: the combining marks and the ZWJ/ZWNJ a + # Devanagari name is written with reach the corpus as written. + joined = "क्\u200dष्मा प्\u200cरसाद" + _Site(monkeypatch, {7: [_nepal_row("Ksma Prasad", joined)]}) + local.fetch(_NEPAL) + corpus = (_own_cache / "local" / _NEPAL / "corpus.jsonl").read_text() + assert joined in {json.loads(line) for line in corpus.splitlines()} + + +def test_every_nepal_request_carries_the_site_s_own_check( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # The site's double-submit check: its CsrfToken cookie echoed in a + # header, with the Referer its pages send, and a User-Agent of a + # browser's form, the site refusing one holding "python". + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + for request in site.requests: + agent = request.get_header("User-agent") or "" + assert agent.startswith("Mozilla/5.0") and "python" not in agent.lower() + for request in site.requests[1:]: + assert request.get_header("X-csrf-token") == site.token + assert request.get_header("Referer") == nepal.NEPAL_SITE + assert request.get_header("Cookie") == f"ASP.NET_SessionId=s1; CsrfToken={site.token}" + + +def test_a_nepal_token_the_site_no_longer_holds_renews_the_session( + site: _Site, monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str]) -> None: + # A fetch can outlast a session: a 403 renews it, counted, and asks again. + _tolerate_nepal_s_small_fixture(monkeypatch) + site.expire = 2 + corpus = local.fetch(_NEPAL) + assert len(corpus.records) == 4 + assert corpus.meta["detail"]["sessions_renewed"] == 1 + assert site.asked().count(nepal.NEPAL_SITE) == 2 + assert "renewing the session after a 403" in capsys.readouterr().err + + +def test_a_nepal_403_after_a_renewed_session_is_raised( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # A file the site keeps refusing is no missing file: the fetch stops. + site.files[102] = lambda: _nepal_error("u", 403) + with pytest.raises(urllib.error.HTTPError, match="403") as raised: + local.fetch(_NEPAL) + raised.value.close() + assert site.asked().count(nepal.NEPAL_BODY.format(id=102)) == 2 + assert local.load(_NEPAL) is None + + +def test_a_nepal_server_error_is_retried_and_its_last_raised( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + errors = iter([_nepal_error("u", 503)]) + site.files[102] = lambda: next(errors, _NEPAL_FILES[102]) + _tolerate_nepal_s_small_fixture(monkeypatch) + assert len(local.fetch(_NEPAL).records) == 4 + site.files[102] = lambda: _nepal_error("u", 503) + with pytest.raises(urllib.error.HTTPError, match="503") as raised: + local.fetch(_NEPAL) + raised.value.close() + assert site.asked().count(nepal.NEPAL_BODY.format(id=102)) == 2 + core.RETRIES + + +_NEPAL_WHOLE = json.dumps(_NEPAL_FILES[101], ensure_ascii=False).encode() + + +@pytest.mark.parametrize("cut", [ + lambda body: body[:-1], + lambda body: body[:body.rindex(b"}") + 1], + lambda body: body[:body.rindex(b"},") + 2], + lambda body: body[:body.rindex("माया".encode()) + 2], + lambda body: b"", +], ids=["before the closing bracket", "after a whole candidate", + "after a candidate's comma", "inside a letter", "empty"]) +def test_a_nepal_file_cut_short_fails_to_parse_and_is_refused( + site: _Site, cut: Callable[[bytes], bytes]) -> None: + # A JSON array cut anywhere short of its end does not parse, so a cut + # file is retried as one cut in transit and never read as a body + # with fewer candidates. + site.files[101] = cut(_NEPAL_WHOLE) + with pytest.raises(ValueError, match="is not UTF-8 JSON"): + local.fetch(_NEPAL) + assert site.asked().count(nepal.NEPAL_BODY.format(id=101)) == core.RETRIES + assert local.load(_NEPAL) is None + + +def test_a_nepal_file_cut_once_is_read_whole_on_the_retry( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + answers = iter([_NEPAL_WHOLE[:-1]]) + site.files[101] = lambda: next(answers, _NEPAL_WHOLE) + _tolerate_nepal_s_small_fixture(monkeypatch) + assert len(local.fetch(_NEPAL).records) == 4 + + +def test_a_nepal_file_with_or_without_a_bom_reads_the_same( + monkeypatch: pytest.MonkeyPatch) -> None: + _tolerate_nepal_s_small_fixture(monkeypatch) + _Site(monkeypatch, dict(_NEPAL_FILES)) + with_bom = local.fetch(_NEPAL).records + _Site(monkeypatch, dict(_NEPAL_FILES), bom=b"") + assert local.fetch(_NEPAL).records == with_bom + + +@pytest.mark.parametrize("answer", [b"{}", b"[1]", b"null", b'{"rows": []}'], + ids=["an object", "not candidates", "null", "a wrapped list"]) +def test_a_nepal_file_that_is_not_a_list_of_candidates_is_refused( + site: _Site, answer: bytes) -> None: + site.files[101] = answer + with pytest.raises(ValueError, match="is not a list of candidates"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize(("listed", "match"), [ + ([{"id": 101}, {"id": 102}], "lists 2 local bodies .* not the 3"), + ([{"id": 101}, {"id": 102}, {"id": 103}, {"id": 104}], "lists 4 local bodies"), + ([{"id": 101}, {"id": 101}, {"id": 102}], r"\(2 distinct ids\)"), + ([{"id": 101}, {"id": "102"}, {"id": 103}], "not a list of local bodies"), + ([{"id": 101}, {"id": True}, {"id": 103}], "not a list of local bodies"), + ({"bodies": []}, "not a list of local bodies"), +], ids=["one fewer", "one more", "an id twice", "an id as text", "an id a boolean", + "not a list"]) +def test_a_nepal_lookup_that_has_changed_is_refused( + monkeypatch: pytest.MonkeyPatch, listed: object, match: str) -> None: + _Site(monkeypatch, dict(_NEPAL_FILES), listed=listed) + with pytest.raises(ValueError, match=match): + local.fetch(_NEPAL) + + +def _nepal_bodies(n: int, missing: int = 0, empty: int = 0) -> dict[int, object]: + """`n` bodies of one candidate each, the first `missing` of them + answered as missing and the `empty` after those answered empty.""" + return {i: [] if i < missing + empty else + [_nepal_row(f"Ram {_letters(i)}", f"राम {i}")] for i in range(missing, n)} + + +@pytest.mark.parametrize(("missing", "empty"), [(7, 0), (8, 0), (0, 7), (0, 8), (4, 3), (4, 4)]) +def test_a_nepal_fetch_is_refused_past_its_real_share_of_missing_or_empty_files( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], + missing: int, empty: int) -> None: + # NEPAL_MAX_MISSING unpatched, and NEPAL_BODIES set back to its + # shipped 753 (which the next test holds): 7 of 753 files missing or + # empty pass, the ids of both kept, and 8 are refused. An empty file + # counts as a missing one: a site that blanked its files would + # otherwise read as a smaller election. + site = _Site(monkeypatch, _nepal_bodies(753, missing, empty)) + monkeypatch.setattr(nepal, "NEPAL_BODIES", 753) + monkeypatch.setattr(nepal, "NEPAL_CANDIDATES", 753) + site.listed = [{"id": i, "name": "", "parentId": 1} for i in range(753)] + if missing + empty == 7: + detail = local.fetch(_NEPAL).meta["detail"] + assert detail["local_bodies_missing"] == " ".join(map(str, range(missing))) + assert detail["local_bodies_empty"] == " ".join(map(str, range(missing, 7))) + assert detail["files"] == 753 - missing + assert "750 of 753 local bodies" in capsys.readouterr().err + else: + with pytest.raises(ValueError, match="8 of 753 local bodies' files are missing" + r" \(HTTP 404\) or empty"): + local.fetch(_NEPAL) + + +def test_the_shipped_nepal_body_count_is_the_lookup_s() -> None: + # The count the share above is taken at, as the lookup listed it. + assert nepal.NEPAL_BODIES == 753 + + +def test_the_nepal_user_agent_is_one_the_site_answers() -> None: + # The site refuses a User-Agent holding "python" or "curl" with a 403 + # (2026-10-24), which would read as a lapsed token and then stop the fetch. + agent = nepal.NEPAL_USER_AGENT.lower() + assert "python" not in agent and "curl" not in agent + + +def _hundred(*bad: dict[str, object]) -> list[dict[str, object]]: + """100 candidates: `bad`, then good ones.""" + good = [_nepal_row(f"Ram {_letters(i)}", f"राम {i}") for i in range(100 - len(bad))] + return [*bad, *good] + + +@pytest.mark.parametrize(("bad", "key", "match"), [ + ({"CandidateID": 1}, "without_the_name_keys", "have no CandidateName or"), + (_nepal_row(" "), "without_a_romanized_name", "have no romanized name"), + (_nepal_row(None), "without_a_romanized_name", "have no romanized name"), + (_nepal_row("- - -"), "without_a_romanized_name", "have no romanized name"), + (_nepal_row("12 34"), "without_a_romanized_name", "have no romanized name"), + (_nepal_row("Ram Zed", "१२३"), "without_a_devanagari_name", "have no Devanagari name"), + (_nepal_row("Ram Zed", "| |"), "without_a_devanagari_name", "have no Devanagari name"), + (_nepal_row("Ram Zed", ""), "without_a_devanagari_name", "have no Devanagari name"), + (_nepal_row("Ram Zed", "राम ज़ेड", CandidateName=None), "without_a_devanagari_name", + "have no Devanagari name"), +], ids=["no name keys", "a blank romanized name", "a null romanized name", + "a placeholder for the romanized name", "digits for the romanized name", + "digits for the Devanagari name", "a placeholder for the Devanagari name", + "a blank Devanagari name", "a null Devanagari name"]) +@pytest.mark.parametrize("count", [1, 2]) +def test_a_nepal_fetch_is_refused_past_its_real_share_of_nameless_candidates( + monkeypatch: pytest.MonkeyPatch, bad: dict[str, object], key: str, match: str, + count: int) -> None: + # At NEPAL_MAX_NAMELESS, unpatched: 1 of 100 passes and 2 are refused. + _Site(monkeypatch, {1: _hundred(*[bad] * count)}) + if count == 1: + assert local.fetch(_NEPAL).meta["detail"][key] == 1 + else: + with pytest.raises(ValueError, match=f"2 of 100 (candidates |)?{match}"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize(("bad", "key", "match"), [ + (_nepal_row("Ram Zed", "Ram Zed"), "devanagari_name_without_devanagari", + "CandidateName values hold no Devanagari"), + (_nepal_row("राम ज़ेड", "राम ज़ेड"), "romanized_name_not_latin", + "CandidateNameEng values are not Latin"), +], ids=["Latin under Devanagari", "Devanagari under the romanized name"]) +@pytest.mark.parametrize("count", [1, 2]) +def test_a_nepal_fetch_is_refused_past_its_real_share_of_names_in_the_wrong_script( + monkeypatch: pytest.MonkeyPatch, bad: dict[str, object], key: str, match: str, + count: int) -> None: + # Two columns swapped, or a romanization dropped, at + # NEPAL_MAX_WRONG_SCRIPT, unpatched: 1 of 100 passes, 2 are refused. + _Site(monkeypatch, {1: _hundred(*[bad] * count)}) + if count == 1: + assert local.fetch(_NEPAL).meta["detail"][key] == 1 + else: + with pytest.raises(ValueError, match=f"2 of 100 (candidates |)?{match}"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize("word", ["Rām Śarmā", "Kṛṣṇa Ṭhākura", "Saʼadi"], + ids=["Latin Extended", "IAST's dotted letters", "a modifier letter"]) +def test_a_nepal_romanized_name_in_any_latin_is_latin( + monkeypatch: pytest.MonkeyPatch, word: str) -> None: + # A romanization may write IAST's dotted letters (Latin Extended + # Additional) or an apostrophe as a letter: 5 of 100 such names would + # refuse the fetch were they read as another script. + _Site(monkeypatch, {1: _hundred(*[_nepal_row(f"{word} {_letters(i)}") + for i in range(5)])}) + assert local.fetch(_NEPAL).meta["detail"]["romanized_name_not_latin"] == 0 + + +@pytest.mark.parametrize("column", ["CandidateNameEng", "CandidateName"]) +def test_a_nepal_column_masked_with_one_word_is_refused( + monkeypatch: pytest.MonkeyPatch, column: str) -> None: + # Each pair stays distinct by the other column, so the pairs alone + # would pass a masked column. + masked = "Xxx" if column == "CandidateNameEng" else "राम" + _Site(monkeypatch, {1: [{**row, column: masked} for row in _hundred()]}) + with pytest.raises(ValueError, match=f"hold 1 distinct {column}: the column may be masked"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize("column", ["CandidateNameEng", "CandidateName"]) +@pytest.mark.parametrize("distinct", [25, 24]) +def test_a_nepal_column_is_refused_below_its_real_share_of_distinct_names( + monkeypatch: pytest.MonkeyPatch, column: str, distinct: int) -> None: + # At NEPAL_MIN_DISTINCT, unpatched: 25 distinct names in 100 + # candidates pass and 24 are refused, the other column distinct. + def word(i: int) -> str: + return f"Ram {_letters(i % distinct)}" if column == "CandidateNameEng" \ + else f"राम {i % distinct}" + _Site(monkeypatch, {1: [{**row, column: word(i)} for i, row in enumerate(_hundred())]}) + if distinct == 25: + assert local.fetch(_NEPAL).records + else: + with pytest.raises(ValueError, match=f"hold 24 distinct {column}"): + local.fetch(_NEPAL) + + +def test_a_nepal_devanagari_word_is_looked_up_in_the_devanagari_names( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # The word index reads every name the corpus holds, so a Devanagari + # word is counted in the records whose Devanagari name writes it, not + # read as a false zero; the romanized names are read as before. + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + loaded = local.load(_NEPAL) + assert loaded is not None + assert local.lookup(loaded, "माया").anywhere == 2 + assert local.lookup(loaded, "Maya").anywhere == 2 + assert local.lookup(loaded, "B.K.").anywhere == 2 # BK folds alike + # A phrase is read in each name, the Devanagari as the romanized. + assert local.lookup(loaded, "दिल माया").anywhere == 2 + assert local.lookup(loaded, "सीता कुमारी").anywhere == 1 + + +@pytest.mark.parametrize("word", ["Bishnu", "Bishnu Kumar"], ids=["a word", "a phrase"]) +def test_a_word_both_of_a_record_s_names_write_counts_the_record_once( + monkeypatch: pytest.MonkeyPatch, word: str) -> None: + # A Devanagari name left partly in Latin shares words with the + # romanization; the index counts records, not names, and a spelling + # once a record. + _Site(monkeypatch, {1: _hundred(_nepal_row("Bishnu Kumar Zed", "Bishnu Kumar राम"))}) + local.fetch(_NEPAL) + loaded = local.load(_NEPAL) + assert loaded is not None + found = local.lookup(loaded, word) + assert found.anywhere == 1 + if word == "Bishnu": + assert found.spellings == {"Bishnu": 1} + + +def test_a_phrase_is_not_matched_across_a_record_s_two_names( + monkeypatch: pytest.MonkeyPatch) -> None: + # Each name is read alone: the romanization's last word and the + # Devanagari name's first are not a phrase. + _Site(monkeypatch, {1: _hundred(_nepal_row("Opal Zed", "Bishnu राम"))}) + local.fetch(_NEPAL) + loaded = local.load(_NEPAL) + assert loaded is not None + assert local.lookup(loaded, "Zed").anywhere == 1 + assert local.lookup(loaded, "Zed Bishnu").anywhere == 0 + + +def test_the_cached_index_is_keyed_to_the_names_it_reads( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # A source that starts or stops reading a record's second name must + # not answer from an index built over the other set, nor load a + # corpus.jsonl built from it. + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + loaded = local.load(_NEPAL) + assert loaded is not None + assert local.lookup(loaded, "माया").anywhere == 2 + monkeypatch.setitem(local.SOURCES, _NEPAL, dataclasses.replace( + local.SOURCES[_NEPAL], corpus_keys=("name",))) + with pytest.raises(local.CorpusDamaged, match=r"record keys \['name', 'devanagari'\]"): + local.load(_NEPAL) + # The index, read from the cache beside the same records, is rebuilt. + assert local.lookup(local.Local(_NEPAL, loaded.meta), "माया").anywhere == 0 + + +def test_a_corpus_fetched_before_its_keys_were_recorded_reads_as_name_alone( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + # Every cached corpus fetched before corpus_keys were recorded was + # built from "name": it loads for a source reading "name" alone. + folder = _fetched(monkeypatch, tmp_path) + _rewrite_meta(folder, corpus_keys=None) + assert local.load(_SOURCE) is not None + + +def test_a_nepal_election_whose_files_hold_no_names_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + # Past the nameless share's own refusal, which a wholly nameless + # election would meet first, the empty corpus is still refused. + monkeypatch.setattr(nepal, "NEPAL_MAX_NAMELESS", 1) + _Site(monkeypatch, {1: [_nepal_row(""), _nepal_row(" ")]}) + with pytest.raises(ValueError, match="hold no candidate with a name"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize("cookies", [[], ["CsrfToken=; path=/"]], + ids=["no cookie", "an empty token"]) +def test_a_nepal_site_that_sets_no_token_is_refused( + site: _Site, monkeypatch: pytest.MonkeyPatch, cookies: list[str]) -> None: + monkeypatch.setattr(urllib.request, "urlopen", lambda request, timeout: _Root(*cookies)) + with pytest.raises(ValueError, match="set no CsrfToken cookie"): + local.fetch(_NEPAL) + + +def test_a_nepal_root_failing_once_is_retried( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + _tolerate_nepal_s_small_fixture(monkeypatch) + failed = _failing_once(monkeypatch, nepal.NEPAL_SITE) + assert len(local.fetch(_NEPAL).records) == 4 + assert failed == [nepal.NEPAL_SITE] + + +def test_a_nepal_corpus_of_another_election_is_refused_on_load( + site: _Site, monkeypatch: pytest.MonkeyPatch, _own_cache: Path) -> None: + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + monkeypatch.setattr(nepal, "NEPAL_ELECTION", "Election2084") + with pytest.raises(local.CorpusDamaged, match="holds the election Election2079"): + local.load(_NEPAL) + + +def test_the_pairs_view_does_not_pair_two_scripts( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # The pairs view compares the letters of two spellings, which a + # Devanagari name and its romanization share none of; the Devanagari + # is kept under its own key, not as a reference name. + _tolerate_nepal_s_small_fixture(monkeypatch) + local.fetch(_NEPAL) + with pytest.raises(SystemExit, match="carry no reference name"): + local.main(["pairs", _NEPAL]) + + +def test_the_shipped_nepal_candidate_count_is_the_files_whole() -> None: + # The count every whole fetch must read, as the 753 files held it. + assert nepal.NEPAL_CANDIDATES == 152_960 + + +@pytest.mark.parametrize("change", [-1, 1], ids=["one fewer", "one more"]) +def test_a_nepal_election_of_another_size_with_every_file_whole_is_refused( + site: _Site, monkeypatch: pytest.MonkeyPatch, change: int) -> None: + # A site that trimmed its files (to the winners, or to one post), or + # added to them, is refused however well formed what is left: with + # every file read the count must be the pinned one, exactly. + _tolerate_nepal_s_small_fixture(monkeypatch) + del site.files[103] # the empty one: every file whole + monkeypatch.setattr(nepal, "NEPAL_BODIES", 2) + monkeypatch.setattr(nepal, "NEPAL_CANDIDATES", 6 - change) + with pytest.raises(ValueError, match="hold 6 candidates, 0 missing or empty"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize(("pin", "refused"), [(7, False), (6, True), (5, True)], + ids=["one a lost file", "the pin itself", "over the pin"]) +def test_a_nepal_election_with_a_file_lost_reads_at_most_the_pin_less_a_candidate_a_file( + site: _Site, monkeypatch: pytest.MonkeyPatch, pin: int, refused: bool) -> None: + # Every real file held a candidate, so with the empty file 103 the 6 + # read must be at most the pin less one: the count landing on the pin + # with a file lost is the site's files grown by what it lost. + _tolerate_nepal_s_small_fixture(monkeypatch) + monkeypatch.setattr(nepal, "NEPAL_CANDIDATES", pin) + if refused: + with pytest.raises(ValueError, match="hold 6 candidates, 1 missing or empty"): + local.fetch(_NEPAL) + else: + assert local.fetch(_NEPAL).meta["detail"]["local_bodies_empty"] == "103" + + +@pytest.mark.parametrize("held", [980, 979]) +def test_a_nepal_election_with_a_file_missing_is_refused_past_its_real_shortfall( + monkeypatch: pytest.MonkeyPatch, held: int) -> None: + # With a file missing the count cannot be exact: at NEPAL_MAX_SHORT, + # unpatched, 20 of 1,000 short pass and 21 are refused. + _Site(monkeypatch, {2: [_nepal_row(f"Ram {_letters(i)}", f"राम {i}") + for i in range(held)]}, + listed=[{"id": i, "name": "", "parentId": 1} for i in (1, 2)]) + monkeypatch.setattr(nepal, "NEPAL_BODIES", 2) + monkeypatch.setattr(nepal, "NEPAL_CANDIDATES", 1000) + monkeypatch.setattr(nepal, "NEPAL_MAX_MISSING", 1) + if held == 980: + assert local.fetch(_NEPAL).meta["detail"]["local_bodies_missing"] == "1" + else: + with pytest.raises(ValueError, match="21 of 1,000 NEPAL_CANDIDATES are missing"): + local.fetch(_NEPAL) + + +def test_a_nepal_site_answering_every_body_with_one_file_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + # A cache serving one file under every id: every file whole and well + # formed, the count matching, and the names four, read 40 times over. + four = [_nepal_row(f"Ram {_letters(i)}", f"राम {i}") for i in range(4)] + _Site(monkeypatch, {i: list(four) for i in range(40)}) + with pytest.raises(ValueError, match="distinct CandidateNameEng: the column may be masked"): + local.fetch(_NEPAL) + + +@pytest.mark.parametrize("missing", ["CandidateName", "CandidateNameEng"]) +@pytest.mark.parametrize("count", [1, 2]) +def test_a_nepal_candidate_without_one_name_key_is_counted( + monkeypatch: pytest.MonkeyPatch, missing: str, count: int) -> None: + bad = {k: v for k, v in _nepal_row("Ram Zed", "राम ज़ेड").items() if k != missing} + _Site(monkeypatch, {1: _hundred(*[bad] * count)}) + if count == 1: + assert local.fetch(_NEPAL).meta["detail"]["without_the_name_keys"] == 1 + else: + with pytest.raises(ValueError, match="2 of 100 candidates have no CandidateName or"): + local.fetch(_NEPAL) + + +def test_a_nepal_lookup_answered_as_missing_is_raised_not_counted( + site: _Site, monkeypatch: pytest.MonkeyPatch) -> None: + # Only a body's file is counted as missing; without the lookup there + # is no election to read. + served = site.urlopen + + def urlopen(request: urllib.request.Request, timeout: int) -> object: + if request.full_url == nepal.NEPAL_HANDLER + nepal.NEPAL_LOOKUP: + site.requests.append(request) + raise _nepal_error(request.full_url, 404) + return served(request, timeout) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + with pytest.raises(urllib.error.HTTPError, match="404") as raised: + local.fetch(_NEPAL) + raised.value.close() + assert site.asked().count(nepal.NEPAL_LOOKUP) == 1 + assert local.load(_NEPAL) is None + + +@pytest.mark.parametrize(("code", "asked"), [(503, None), (403, 1)], + ids=["a server error every time", "a refusal"]) +def test_a_nepal_root_that_does_not_answer_is_raised( + site: _Site, monkeypatch: pytest.MonkeyPatch, code: int, asked: int | None) -> None: + # A 5xx at the root is retried and its last raised; a 403, what the + # site answers a refused User-Agent, is raised at once, never read as + # a site that sets no token. + served = site.urlopen + + def urlopen(request: urllib.request.Request, timeout: int) -> object: + if request.full_url == nepal.NEPAL_SITE: + site.requests.append(request) + raise _nepal_error(request.full_url, code) + return served(request, timeout) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + with pytest.raises(urllib.error.HTTPError, match=str(code)) as raised: + local.fetch(_NEPAL) + raised.value.close() + assert site.asked() == [nepal.NEPAL_SITE] * (asked or core.RETRIES) + + +@pytest.mark.parametrize(("cookies", "token", "header"), [ + (["CsrfToken=abc; path=/; Priority=High", "ASP.NET_SessionId=s1; HttpOnly"], + "abc", "ASP.NET_SessionId=s1; CsrfToken=abc"), + (['CsrfToken="a=b"; path=/'], '"a=b"', 'CsrfToken="a=b"'), + (["CsrfToken=abc", "lang=ne np; path=/"], "abc", "CsrfToken=abc; lang=ne np"), +], ids=["an attribute the stdlib does not know", "a quoted value", "a value with a space"]) +def test_a_nepal_cookie_is_sent_as_the_site_set_it( + monkeypatch: pytest.MonkeyPatch, cookies: list[str], token: str, header: str) -> None: + # An attribute is no cookie, a value is sent as written, and no + # cookie the site set is left out. + monkeypatch.setattr(urllib.request, "urlopen", lambda request, timeout: _Root(*cookies)) + assert nepal._session() == {"User-Agent": nepal.NEPAL_USER_AGENT, + "Referer": nepal.NEPAL_SITE, "X-CSRF-Token": token, + "Cookie": header} + + +@pytest.mark.parametrize("cookie", ["=abc; path=/", "Secure; path=/"], + ids=["no name", "no equals sign"]) +def test_a_nepal_cookie_this_cannot_read_is_refused( + monkeypatch: pytest.MonkeyPatch, cookie: str) -> None: + # Read past, it would leave a cookie out of every request, and the + # fetch die on a 403 that hides why. + monkeypatch.setattr(urllib.request, "urlopen", + lambda request, timeout: _Root("CsrfToken=abc", cookie)) + with pytest.raises(ValueError, match="set a cookie with no name"): + nepal._session() + + +@pytest.mark.parametrize(("records", "match"), [ + ([{"name": "Ann Lee", "devanagari": 3}], "'devanagari' is not text"), + ([{"name": "Ann Lee", "devanagari": " "}], + "no fetched record holds the corpus key 'devanagari'"), + ([{"name": "Ann Lee", "devanagri": "राम"}], + "no fetched record holds the corpus key 'devanagari'"), +], ids=["a value not text", "a value blank", "a key misspelled"]) +def test_a_second_corpus_key_the_records_do_not_fill_is_refused( + monkeypatch: pytest.MonkeyPatch, records: list[dict[str, object]], match: str) -> None: + # Skipped in silence, its names would never reach the snapshot. + monkeypatch.setitem(local.SOURCES, _NEPAL, dataclasses.replace( + local.SOURCES[_NEPAL], fetch=lambda: local.Fetched(records, _DATE, {}), check=None)) + with pytest.raises(ValueError, match=match): + local.fetch(_NEPAL) + assert local.load(_NEPAL) is None + + +@pytest.mark.parametrize("source", sorted(set(local.SOURCES) - {_NEPAL})) +def test_every_other_source_s_corpus_and_index_read_the_name_alone( + monkeypatch: pytest.MonkeyPatch, source: str) -> None: + # A record's other keys (psc's boxes, tse's registry name) are no + # names of the corpus or the word index. + record = {"name": "Ann Lee", "given": "Bo Wu", "devanagari": "Cy Doe", + "registry": "Dee Fox"} + monkeypatch.setitem(local.SOURCES, source, dataclasses.replace( + local.SOURCES[source], fetch=lambda: local.Fetched([record], _DATE, {}), + check=None, renew=None)) + local.fetch(source) + lines = local.corpus_path(source).read_text().splitlines() + assert [json.loads(line) for line in lines] == ["Ann Lee"] + loaded = local.load(source) + assert loaded is not None + assert local.lookup(loaded, "Lee").anywhere == 1 + assert [local.lookup(loaded, w).anywhere for w in ("Wu", "Cy", "Fox")] == [0, 0, 0] + + # -- Companies House people with significant control ------------------------------ _PSC = "psc-individuals" @@ -2591,7 +3341,7 @@ def test_no_module_imports_a_sibling_constant() -> None: # JRC are sources' names, keys of SOURCES, and no fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} - assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc)} + assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal)} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" diff --git a/tools/differential/README.md b/tools/differential/README.md index 305558c56..77d25c6e7 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -235,7 +235,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -264,11 +264,14 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | | `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. The one local corpus whose names carry their fields, scored by `local_agreement.py` | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | +| `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Shekh`, `Guru`) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | -A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256 and to the code that cuts and folds a word, so a refetch or a change to the fold rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. +A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. `psc-individuals` caches an extract of each of the snapshot's 33 parts rather than the zips: 168 MB against 2.44 GB on 2026-10-22, each extract holding only the records and fields the corpus keeps. A download that fails costs one part, and running the fetch again resumes, keeping the parts already read for the same day; the set replaces the cached one only once every part is read, and a part that disappears mid-fetch (the files are overwritten each morning) is refused naming the refresh. The fetch refuses a snapshot where more than one individual in 10,000 has no `name_elements`, or a display name that is not its four boxes joined by spaces, or where fewer than half of the lines are individuals (87% were on 2026-10-22; a kind Companies House renamed would read as another kind), and counts what it skips into `meta.json`'s detail. It reads the cached extracts only as one whole snapshot of one date, and an extract written by another version of the extraction (its file name carries the version) is neither resumed from nor read. Companies House's data-products guidance says it imposes no rules on how the register is used and leaves data protection to the user; the dataset's data.gov.uk entry gives its licence as "TBD", and gov.uk's footer licenses its own page text under the Open Government Licence. The credit line says the names are personal data. +`nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request. + `local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares title, given, middle and family with the filer's boxes, folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. A title box the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by nationality, misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, record it from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. `pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied. diff --git a/tools/differential/local_core.py b/tools/differential/local_core.py index becf51b5a..08a890c45 100644 --- a/tools/differential/local_core.py +++ b/tools/differential/local_core.py @@ -17,7 +17,7 @@ import urllib.error import urllib.request import zipfile -from collections.abc import Callable, Iterable +from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass from pathlib import Path from typing import Any, TypeVar @@ -53,6 +53,12 @@ class Source: #: Why local_corpora.load() should refuse a meta.json this version of the source #: did not write (another FEC cycle), or "" to accept it. check: Callable[[dict[str, Any]], str] | None = None + #: The record keys whose values the corpus and the word index hold as + #: names, an empty value skipped: "name", and for a source whose + #: records carry the name in a second script, that script's key + #: (nepal-candidates' "devanagari"). meta.json records them, and a + #: corpus written under other keys is refused on load. + corpus_keys: tuple[str, ...] = ("name",) #: Attempts per request, for every source that retries. @@ -134,14 +140,16 @@ def _backed_off(exc: Exception, attempt: int, url: str) -> bool: def _retried(url: str, read: Callable[[bytes], _T], delay: float, *, - timeout: int = 60) -> _T: + timeout: int = 60, headers: Mapping[str, str] | None = None) -> _T: """`url`'s body as `read` reads it, under _backed_off's policy; any ValueError from `read` is retried as a body cut short in transit, and - `delay` seconds of sleep follow each request.""" + `delay` seconds of sleep follow each request. `headers` go with each + request, a User-Agent among them replacing USER_AGENT.""" for attempt in range(RETRIES): try: body = corpus_build.fetch_bytes(url, user_agent=USER_AGENT, - timeout=timeout, delay=delay) + timeout=timeout, delay=delay, + headers=headers) try: return read(body) except ValueError as exc: diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 8d61fe837..8bcc64ed9 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -26,9 +26,12 @@ one distinct pair of a TSE donor's filed name and the tax registry's name for the same donor; one distinct PSC filed name, with its four boxes and nationality; one distinct spelling of a JRC-Names person, - with its entity and script. The counts are counted from here. + with its entity and script; one distinct pair of a Nepali + candidate's romanized name and the Devanagari one. The counts are + counted from here. - corpus.jsonl -- the distinct names, in the corpus line format - snapshot.py and compare.py --corpus read. + snapshot.py and compare.py --corpus read: each record's "name", and + for nepal-candidates its Devanagari name too (Source.corpus_keys). - meta.json -- the source's own date for the data, what was fetched, and the sha256 of both files, checked on every load: a truncated file would otherwise read as a run of false zeros. @@ -37,8 +40,8 @@ fetched after it; snapshot.sample-N.jsonl and its .source.json for a --sample N reading; and baselines/, the recordings snapshot.py reuses. - wordindex.json -- the word index tools/name_usage.py looks words up - in, written on first use and keyed to the records and the fold - (cached_index), so a single-word lookup parses neither raw.jsonl's + in, written on first use and keyed to the records, the fold and the + record keys it reads (cached_index), so a single-word lookup parses neither raw.jsonl's records nor the names (load() still reads both files for their sha256). @@ -54,7 +57,8 @@ and what its terms allow is its module's docstring: local_ctgov.py (ctgov-officials), local_fec.py (fec-individuals), local_tse.py (tse-donors), local_tw.py (tw-donors), local_ukec.py (ukec-individuals), -local_psc.py (psc-individuals) and local_jrc.py (jrc-names). +local_psc.py (psc-individuals), local_jrc.py (jrc-names) and +local_nepal.py (nepal-candidates). """ from __future__ import annotations @@ -92,6 +96,7 @@ import local_tse # noqa: E402 from local_fec import check_fec, fetch_fec, renew_fec # noqa: E402 from local_jrc import JRC, check_jrc, fetch_jrc, renew_jrc # noqa: E402 +from local_nepal import check_nepal, fetch_nepal # noqa: E402 from local_psc import PSC, check_psc, fetch_psc, renew_psc # noqa: E402 from local_tse import check_tse, fetch_tse # noqa: E402 from local_tw import fetch_tw # noqa: E402 @@ -149,6 +154,13 @@ def header(self) -> str: return (f"{self.credit}; {self.size:,} " f"{SOURCES[self.source].unit} (local corpus {self.source})") + @property + def corpus_keys(self) -> tuple[str, ...]: + """The record keys whose names the corpus and the word index read + (Source.corpus_keys), or "name" alone for a source not in SOURCES.""" + source = SOURCES.get(self.source) + return source.corpus_keys if source else ("name",) + @functools.cached_property def index(self) -> WordIndex: """The word index, read from its cache beside the corpus or built @@ -179,6 +191,11 @@ def index(self) -> WordIndex: credit="UK Electoral Commission register of donations, individual " "donors (reuse terms for the register not stated)", unit="names", fetch=fetch_ukec), + "nepal-candidates": Source( + credit="Election Commission, Nepal: candidates of the 2079 BS (2022) " + "local elections (result.election.gov.np; no licence stated)", + unit="name pairs", fetch=fetch_nepal, check=check_nepal, + corpus_keys=("name", "devanagari")), PSC: Source( credit="Companies House people with significant control, individuals " @@ -210,12 +227,21 @@ def fetch(source: str) -> Local: fetched = SOURCES[source].fetch() if not fetched.records: raise ValueError(f"{source}: the fetch found no names") - if any(not isinstance(r.get("name"), str) for r in fetched.records): + if any(not isinstance(r.get("name"), str) or not r["name"].strip() + for r in fetched.records): raise ValueError(f"{source}: a fetched record has no name") + keys = SOURCES[source].corpus_keys + for key in keys[1:]: + if any(not isinstance(r.get(key, ""), str) for r in fetched.records): + raise ValueError(f"{source}: a fetched record's {key!r} is not text") + if not any(r.get(key, "").strip() for r in fetched.records): + raise ValueError(f"{source}: no fetched record holds the corpus key " + f"{key!r}: a key misspelled, or a column emptied") raw = "".join(corpus_build.jsonl_line(r, sort_keys=True) + "\n" for r in fetched.records).encode() lines = corpus_build.corpus_lines( - {"text": r["name"], "shape": None} for r in fetched.records) + {"text": r[key], "shape": None} for r in fetched.records + for key in keys if r.get(key, "").strip()) corpus = ("\n".join(lines) + "\n").encode() corpus_build.write_atomic(path / RAW, raw) corpus_build.write_atomic(path / CORPUS, corpus) @@ -223,6 +249,7 @@ def fetch(source: str) -> Local: "data_date": fetched.data_date, "fetched": time.strftime("%Y-%m-%d"), "records": len(fetched.records), "names": len(lines), + "corpus_keys": list(keys), "sha256": hashlib.sha256(raw).hexdigest(), "corpus_sha256": hashlib.sha256(corpus).hexdigest(), "detail": fetched.detail} @@ -254,6 +281,13 @@ def load(source: str) -> Local | None: raise CorpusDamaged(f"{path / META} has no {kind.__name__} {key!r}{again}") if (check := SOURCES[source].check) and (why := check(meta)): raise CorpusDamaged(f"{path / META} {why}{again}") + # A meta.json written before corpus_keys were recorded holds "name"'s + # corpus, as every source's then was. + keys = list(SOURCES[source].corpus_keys) + if meta.get("corpus_keys", ["name"]) != keys: + raise CorpusDamaged(f"{path / META} holds a corpus of the record keys " + f"{meta.get('corpus_keys', ['name'])}, and this tool " + f"reads {keys}{again}") _verified(source, RAW, meta["sha256"]) _verified(source, CORPUS, meta["corpus_sha256"]) @@ -802,20 +836,30 @@ class WordIndex: spellings: dict[str, dict[str, int]] -def _names(local: Local) -> Counter[str]: - """Each distinct name with the records writing it: a name repeats - across a source's records (536,981 officials, 352,104 names on - 2026-10-21).""" - return Counter(record["name"] for record in local.records) +def _names(local: Local) -> Counter[tuple[str, ...]]: + """Each distinct tuple of a record's names (its distinct non-empty + values under the source's corpus_keys, as the corpus reads them), + with the records writing it: a name repeats across a source's + records (536,981 officials, 352,104 names on 2026-10-21). Reading + every key lets a lookup of a Devanagari word in nepal-candidates find + the records whose Devanagari name holds it; keeping a record's names + together counts a word both of them write once.""" + keys = local.corpus_keys + return Counter(tuple(dict.fromkeys(record[key] for key in keys if record.get(key))) + for record in local.records) def word_index(local: Local) -> WordIndex: anywhere: Counter[str] = Counter() after: Counter[str] = Counter() spellings: dict[str, Counter[str]] = {} - for name, n in _names(local).items(): - head, _, tail = name.partition(",") - before, behind = _keyed(head), _keyed(tail) + for names, n in _names(local).items(): + before: list[tuple[str, str]] = [] + behind: list[tuple[str, str]] = [] + for name in names: + head, _, tail = name.partition(",") + before += _keyed(head) + behind += _keyed(tail) after_keys = {key for key, _ in behind} for key in {key for key, _ in before} | after_keys: anywhere[key] += n @@ -892,8 +936,9 @@ def _is_counts(value: object) -> bool: def cached_index(local: Local) -> WordIndex: """`local`'s word index, read from wordindex.json beside the corpus where its key matches and its counts are counts, else built and - written there. The key is the records' sha256 and fold_fingerprint(), - so a refetch, or a change to how a word is cut or folded, rebuilds it + written there. The key is the records' sha256, fold_fingerprint() + and the corpus_keys read, so a refetch, or a change to how a word is + cut or folded or to which of a record's names are read, rebuilds it rather than answering from a stale index. A Local with no sha256 (a test's) is built and not cached. An index file that is not an index, or not this one, is built over, with a line on stderr: it is derived @@ -902,7 +947,7 @@ def cached_index(local: Local) -> WordIndex: if not isinstance(sha, str): return word_index(local) path = directory(local.source, create=False) / INDEX - key = f"{sha}:{fold_fingerprint()}" + key = f"{sha}:{fold_fingerprint()}:{','.join(local.corpus_keys)}" if path.exists(): try: data = json.loads(path.read_text(encoding="utf-8")) @@ -938,15 +983,17 @@ def phrase_counts(local: Local, keys: list[str]) -> tuple[int, int]: return 0, 0 n = len(keys) anywhere = after = 0 - for name, count in _names(local).items(): - head, _, tail = name.partition(",") - first = words(head) - got = first + words(tail) - starts = [i for i in range(len(got) - n + 1) if got[i:i + n] == keys] - if starts: - anywhere += count - if any(i >= len(first) for i in starts): - after += count + for names, count in _names(local).items(): + found = found_after = False + for name in names: + head, _, tail = name.partition(",") + first = words(head) + got = first + words(tail) + starts = [i for i in range(len(got) - n + 1) if got[i:i + n] == keys] + found = found or bool(starts) + found_after = found_after or any(i >= len(first) for i in starts) + anywhere += count * found + after += count * found_after return anywhere, after diff --git a/tools/differential/local_nepal.py b/tools/differential/local_nepal.py new file mode 100644 index 000000000..1daf45136 --- /dev/null +++ b/tools/differential/local_nepal.py @@ -0,0 +1,355 @@ +"""The nepal-candidates local corpus: Nepal's 2079 BS local election. + +nepal-candidates -- every candidate in Nepal's local elections of 2079 +BS (2022), as the Election Commission publishes them on its results +site (result.election.gov.np): one file per local body (municipality or +rural municipality), each candidate's name written twice, in Devanagari +as filed and in the Commission's own romanization. It is the +repository's first large population of South Asian names: romanized +names with surnames written as initials (B.K., K.C., G.C.), honorific +words that are titles in front and names elsewhere (Shekh, Guru), Md., +Muhammad's abbreviation, which its period makes a title by shape, and +Devanagari written with vowel signs and the virama, combining marks the +letter-shape tests do not count as letters (decisions/nfd-shapes), and +with the joiners (ZWJ, ZWNJ). The site states no licence; only the two +name columns are kept. +""" +from __future__ import annotations + +import json +import sys +import time +import urllib.error +import urllib.request +from collections import Counter +from typing import Any + +import local_core +from local_core import Fetched, _backed_off, _pinned, _refuse_past, _retried + + +NEPAL_SITE = "https://result.election.gov.np/" +NEPAL_HANDLER = NEPAL_SITE + "Handlers/SecureJson.ashx?file=" +NEPAL_ELECTION = "Election2079" +NEPAL_LOOKUP = f"JSONFiles/{NEPAL_ELECTION}/Local/Lookup/localbodies.json" +NEPAL_BODY = f"JSONFiles/{NEPAL_ELECTION}/Local/{{id}}.json" +#: The User-Agent this source sends: a browser's form, naming the tool +#: in it as a crawler does. The site refuses (HTTP 403) a User-Agent +#: holding "python" or "curl" (2026-10-24), which the shared +#: local_core.USER_AGENT does in the repository's URL. +NEPAL_USER_AGENT = "Mozilla/5.0 (compatible; nameparser local_corpora)" +#: The seconds of sleep after each request: the 754 requests of a fetch +#: took 28 minutes at 0.3 s on 2026-10-24. +NEPAL_DELAY = 0.3 +#: The local bodies the lookup listed on 2026-10-24. The election is +#: over and its files final, so a lookup listing another number is a +#: changed site, refused rather than read as a smaller election. +NEPAL_BODIES = 753 +#: The candidates the 753 files held on 2026-10-24, pinned as the +#: bodies are: a fetch reading every file whole must read exactly this +#: many, so a site that trimmed its files (to the winners, or to one +#: post) is refused however well formed what is left. +NEPAL_CANDIDATES = 152_960 +#: The share of the listed bodies whose file the site may answer as +#: missing (HTTP 404) or as empty (a list of no candidates) before the +#: fetch is refused: a site that blanked its files would otherwise read +#: as a smaller election. Every listed file held candidates on +#: 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids +#: 5551 to 5554, answered when asked again, so they were refused by a +#: lapsed token (403), not missing. A hundredth (7 bodies) lets a few +#: lost files cost their names rather than the fetch, and refuses a +#: site that has moved its files. +NEPAL_MAX_MISSING = 1 / 100 +#: Where files are missing or empty, the share of NEPAL_CANDIDATES the +#: fetch may fall short by (3,059 candidates, 15 bodies of the average +#: 203): the pin cannot be exact without those files, so a site that +#: also trimmed a few candidates elsewhere is not seen then. Not sized +#: against the largest bodies, which were not measured: losing a few +#: of them refuses the fetch, and the detail's largest_file says how +#: large the largest file read was. +NEPAL_MAX_SHORT = 1 / 50 +#: The record keys read; every other key is read past. +NEPAL_NAME, NEPAL_ROMANIZED = "CandidateName", "CandidateNameEng" +#: The share of candidates without the two name keys, without a +#: romanized name, or without a Devanagari one, past which the fetch is +#: refused, a value holding no letter counted as none (_named): a column +#: renamed or emptied wholesale would otherwise read as a smaller +#: election. On 2026-10-24, 0, 26 and 0 of 152,960. +NEPAL_MAX_NAMELESS = 1 / 100 +#: The share of the candidates kept (those with a romanized name) past +#: which a column holding the wrong script refuses the fetch: a +#: Devanagari name with no Devanagari letter, or a romanized one with a +#: letter outside Latin, is what two columns swapped, or a romanization +#: dropped, would show. 0 and 3 of the 152,934 kept on 2026-10-24 (a +#: romanization left partly in Devanagari). +NEPAL_MAX_WRONG_SCRIPT = 1 / 100 +#: The distinct romanized names, and the distinct Devanagari ones, must +#: each be at least this share of the candidates kept: a column masked +#: with one word would otherwise pass, its pairs kept distinct by the +#: other column. 118,126 and 119,282 of 152,934 (77% and 78%) on +#: 2026-10-24. +NEPAL_MIN_DISTINCT = 1 / 4 + + +def _cookies(headers: list[str]) -> dict[str, str]: + """The cookies `headers` (Set-Cookie values) set, by name, each value + as sent: a header is cut at its first semicolon, where its + attributes start, and at its first equals sign. One that holds no + name raises, rather than leaving a cookie out of every request.""" + cookies: dict[str, str] = {} + for header in headers: + name, sep, value = header.split(";", 1)[0].partition("=") + if not sep or not name.strip(): + raise ValueError(f"{NEPAL_SITE} set a cookie with no name: the " + f"site's check has changed") + cookies[name.strip()] = value.strip() + return cookies + + +def _session() -> dict[str, str]: + """The headers every handler request carries: the site's own + double-submit check, the CsrfToken cookie a GET of the site sets + echoed in X-CSRF-Token, beside every cookie it sets, with the + Referer its pages send. Retried under local_core's policy.""" + request = urllib.request.Request(NEPAL_SITE, headers={"User-Agent": NEPAL_USER_AGENT}) + for attempt in range(local_core.RETRIES): + try: + with urllib.request.urlopen(request, timeout=60) as response: + response.read() + cookies = _cookies(response.headers.get_all("Set-Cookie") or []) + break + except local_core._TRANSIENT as exc: + if not _backed_off(exc, attempt, NEPAL_SITE): + raise + else: + raise AssertionError("unreachable: the last attempt raises") + if not cookies.get("CsrfToken"): + raise ValueError(f"{NEPAL_SITE} set no CsrfToken cookie: the site's " + f"check has changed") + return {"User-Agent": NEPAL_USER_AGENT, "Referer": NEPAL_SITE, + "X-CSRF-Token": cookies["CsrfToken"], + "Cookie": "; ".join(f"{k}={v}" for k, v in sorted(cookies.items()))} + + +def _json(body: bytes, path: str) -> object: + """`body` parsed as UTF-8, with or without a byte-order mark (a + survey on 2026-10-23 saw one, the fetch on 2026-10-24 none). A body + cut short is no JSON, and raises.""" + try: + return json.loads(body.decode("utf-8-sig")) + except (UnicodeDecodeError, ValueError) as exc: + raise ValueError(f"{path} is not UTF-8 JSON ({exc})") from None + + +def _get(path: str, headers: dict[str, str], counts: Counter[str]) -> object: + """`path`'s JSON, retried under local_core's policy. A 403, what the + site answers a token it no longer holds, renews the session in + `headers`, counted, and asks once more (a fetch can outlast + ASP.NET's default twenty-minute session, which the token seems bound + to); a second 403 is raised.""" + url = NEPAL_HANDLER + path + try: + return _retried(url, lambda body: _json(body, path), NEPAL_DELAY, headers=headers) + except urllib.error.HTTPError as exc: + if exc.code != 403: + raise + exc.close() + print(f" renewing the session after a 403: {url}", file=sys.stderr) + counts["sessions_renewed"] += 1 + headers.update(_session()) + return _retried(url, lambda body: _json(body, path), NEPAL_DELAY, headers=headers) + + +def _bodies(headers: dict[str, str], counts: Counter[str]) -> list[int]: + """The ids of the local bodies the lookup lists; a list that is not + NEPAL_BODIES distinct int ids is refused.""" + listed = _get(NEPAL_LOOKUP, headers, counts) + if not isinstance(listed, list) or any( + not isinstance(b, dict) or type(b.get("id")) is not int for b in listed): + raise ValueError(f"{NEPAL_LOOKUP} is not a list of local bodies with ids") + ids = sorted({b["id"] for b in listed}) + if len(ids) != len(listed) or len(ids) != NEPAL_BODIES: + raise ValueError(f"{NEPAL_LOOKUP} lists {len(listed):,} local bodies " + f"({len(ids):,} distinct ids), not the {NEPAL_BODIES:,} " + f"of NEPAL_BODIES: the site has changed") + return ids + + +def _candidates(body: int, headers: dict[str, str], + counts: Counter[str]) -> list[dict[str, Any]] | None: + """`body`'s candidates, or None where the site answers its file as + missing (HTTP 404).""" + path = NEPAL_BODY.format(id=body) + try: + rows = _get(path, headers, counts) + except urllib.error.HTTPError as exc: + if exc.code != 404: + raise + exc.close() + return None + if not isinstance(rows, list) or any(not isinstance(r, dict) for r in rows): + raise ValueError(f"{path} is not a list of candidates") + return rows + + +def _devanagari(text: str) -> bool: + return any("\u0900" <= char <= "\u097f" and char.isalpha() for char in text) + + +#: The letters a romanization may write beside ASCII's: Latin-1 and +#: Latin Extended-A and B, the modifier letters (an apostrophe written +#: as a letter), and Latin Extended Additional, where IAST's dotted +#: letters stand. +_LATIN = (("\u00c0", "\u024f"), ("\u02b0", "\u02ff"), ("\u1e00", "\u1eff")) + + +def _latin(text: str) -> bool: + """Whether every letter of `text` is ASCII or in one of _LATIN's ranges.""" + return all(char.isascii() or any(low <= char <= high for low, high in _LATIN) + for char in text if char.isalpha()) + + +def _named(value: object) -> str: + """`value` stripped, or "" where it is no text or holds no letter: the + Commission wrote a placeholder of dashes or bars (`- - -`) for the + romanization of 26 of 152,960 candidates fetched 2026-10-24, each + beside a Devanagari name.""" + text = value.strip() if isinstance(value, str) else "" + return text if any(char.isalpha() for char in text) else "" + + +def _read(rows: list[dict[str, Any]], pairs: set[tuple[str, str]], + counts: Counter[str]) -> None: + """Add one body's (romanized, Devanagari) pairs to `pairs`. A + candidate without the name keys or a romanized name is counted and + read past; of those kept, one without a Devanagari name is kept with + an empty one, and one with a name in the wrong script is kept, each + counted.""" + for row in rows: + counts["candidates"] += 1 + if NEPAL_NAME not in row or NEPAL_ROMANIZED not in row: + counts["without_the_name_keys"] += 1 + continue + name, romanized = _named(row[NEPAL_NAME]), _named(row[NEPAL_ROMANIZED]) + if not romanized: + counts["without_a_romanized_name"] += 1 + continue + if not name: + counts["without_a_devanagari_name"] += 1 + elif not _devanagari(name): + counts["devanagari_name_without_devanagari"] += 1 + if not _latin(romanized): + counts["romanized_name_not_latin"] += 1 + counts["with_a_romanized_name"] += 1 + pairs.add((romanized, name)) + + +def _refuse_a_short_election(candidates: int, absent: int) -> None: + """Refuse a fetch reading other than NEPAL_CANDIDATES with every file + read (`absent` 0); with `absent` files missing or empty, one reading + more than NEPAL_CANDIDATES less one a file (every file held a + candidate), or short by more than NEPAL_MAX_SHORT.""" + if candidates > NEPAL_CANDIDATES - absent or not absent and candidates != NEPAL_CANDIDATES: + raise ValueError(f"the files read hold {candidates:,} candidates, " + f"{absent:,} missing or empty, against the " + f"{NEPAL_CANDIDATES:,} of NEPAL_CANDIDATES: the site's " + f"files have changed") + _refuse_past([(NEPAL_CANDIDATES - candidates, NEPAL_CANDIDATES, NEPAL_MAX_SHORT, + "NEPAL_CANDIDATES are missing")], + "more than a few missing or empty files account for, unless " + "they were among the largest") + + +def fetch_nepal() -> Fetched: + """Every candidate of every local body the lookup lists, one record + per distinct pair of the romanized name and the Devanagari one. + + The record's "name", the one parse() reads, is the romanized name + (CandidateNameEng), since romanized South Asian names are where the + parser misreads: surnames written as initials read as credentials + where the name has a word to spare (B.K., K.C.; #864), and a leading + Md. read as a title by its period. + The Devanagari name (CandidateName) is kept beside it as + "devanagari", and the corpus holds both (Source.corpus_keys), so a + snapshot watches the script too. A candidate with no romanized name + is dropped; one with no Devanagari name is kept, its "devanagari" + empty. A record is a distinct pair, as tse-donors' are, not a + candidate row: a body lists a candidate once per post, and a common + name recurs across bodies, so a row count would weight a name by how + often it stood; a person with two romanizations is two records. + Gender, age, ward and post are read past, since no view reads them. + + A file the site answers as missing (HTTP 404) is counted, its id + kept in the detail as an empty file's is, and with the empty files + past NEPAL_MAX_MISSING of the bodies refuses the fetch. A 403 renews + the session once (_get); a 5xx, a 429 or a dropped connection is + retried and then raised, and any other 4xx is raised at once. A file + is JSON, so one cut short does not parse, and is retried and then + refused rather than read as fewer candidates. A lookup not listing + NEPAL_BODIES bodies, a candidate count other than NEPAL_CANDIDATES + (or short of it by more than NEPAL_MAX_SHORT where files are + missing or empty), too many candidates without the name keys, a + romanized name or a Devanagari one (NEPAL_MAX_NAMELESS) or with a + name in the wrong script (NEPAL_MAX_WRONG_SCRIPT), and too few + distinct names in either column (NEPAL_MIN_DISTINCT) refuse it too. + The site states no date for the data, so the data date is the + fetch's own, in UTC.""" + when = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + counts: Counter[str] = Counter() + headers = _session() + ids = _bodies(headers, counts) + pairs: set[tuple[str, str]] = set() + missing: list[int] = [] + empty: list[int] = [] + for number, body in enumerate(ids, 1): + if number % 50 == 0: + print(f" {number:,} of {len(ids):,} local bodies", file=sys.stderr) + rows = _candidates(body, headers, counts) + if rows is None: + missing.append(body) + continue + counts["files"] += 1 + if not rows: + empty.append(body) + counts["largest_file"] = max(counts["largest_file"], len(rows)) + _read(rows, pairs, counts) + candidates = counts["candidates"] + _refuse_past([(len(missing) + len(empty), len(ids), NEPAL_MAX_MISSING, + "local bodies' files are missing (HTTP 404) or empty")], + "the site has moved, dropped or emptied its files") + _refuse_a_short_election(candidates, len(missing) + len(empty)) + _refuse_past([(counts["without_the_name_keys"], candidates, NEPAL_MAX_NAMELESS, + f"candidates have no {NEPAL_NAME} or {NEPAL_ROMANIZED} key"), + (counts["without_a_romanized_name"], candidates, NEPAL_MAX_NAMELESS, + "candidates have no romanized name"), + (counts["without_a_devanagari_name"], candidates, NEPAL_MAX_NAMELESS, + "candidates have no Devanagari name")], + "a name column may be renamed or emptied") + kept = counts["with_a_romanized_name"] + _refuse_past([(counts["devanagari_name_without_devanagari"], kept, + NEPAL_MAX_WRONG_SCRIPT, f"{NEPAL_NAME} values hold no Devanagari"), + (counts["romanized_name_not_latin"], kept, NEPAL_MAX_WRONG_SCRIPT, + f"{NEPAL_ROMANIZED} values are not Latin")], + "the two name columns may be swapped or replaced") + if not pairs: + raise ValueError("the local bodies' files hold no candidate with a name") + for column, index in ((NEPAL_ROMANIZED, 0), (NEPAL_NAME, 1)): + seen = len({pair[index] for pair in pairs}) + if seen < kept * NEPAL_MIN_DISTINCT: + raise ValueError(f"{kept:,} candidates hold {seen:,} distinct " + f"{column}: the column may be masked") + detail: dict[str, Any] = { + "election": NEPAL_ELECTION, "local_bodies": len(ids), + "local_bodies_missing": " ".join(map(str, missing)), + "local_bodies_empty": " ".join(map(str, empty)), + **{key: counts[key] for key in ( + "files", "candidates", "without_the_name_keys", + "without_a_romanized_name", "without_a_devanagari_name", + "devanagari_name_without_devanagari", "romanized_name_not_latin", + "sessions_renewed", "largest_file")}} + return Fetched([{"name": romanized, "devanagari": name} + for romanized, name in sorted(pairs)], when, detail) + + +check_nepal = _pinned("election", lambda: NEPAL_ELECTION, + "holds the election {held}, and this tool reads {wanted}")