[derek73]

Adds a seventh local corpus, nepal-candidates: every candidate in Nepal's 2079 BS (2022) local elections, from the Election Commission's results site. It is the repository's first large population of South Asian names. Each record pairs the Commission's romanization with the name in Devanagari as filed. Like every local corpus it is fetched into ~/.cache/nameparser-tools/local/nepal-candidates/ and never committed.

uv run python tools/differential/local_corpora.py fetch nepal-candidates
uv run python tools/differential/snapshot.py --local nepal-candidates

What it is

• Source: result.election.gov.np, through the site's own JSON handler. The handler wants the site's double-submit check: the CsrfToken cookie a GET of the site sets, echoed in X-CSRF-Token, with the Referer its pages send.
• User-Agent: the site answers 403 to any User-Agent holding python or curl, and the shared USER_AGENT holds the repository's URL. Per Derek, this source sends a browser-form string that still names the tool: Mozilla/5.0 (compatible; nameparser local_corpora) (NEPAL_USER_AGENT).
• Size: 753 local-body files, 152,960 candidates and 126,981 distinct (romanized, Devanagari) pairs, fetched 2026-10-24. A fetch takes 28 minutes at 0.3 s a request. No licence is stated, and the credit line says so.
• What a record is:
  • name, the romanized name parse() reads, since that is where the parser misreads; and devanagari.
  • One record per distinct pair, as tse-donors, not per candidate row: a body lists a candidate once per post, and common names recur across bodies.
  • Gender, age, ward and post are read past, since no view uses them.
  • The parliament and provincial files are left out: they carry Devanagari only (about 7k names).
• Both scripts are read:
  • Source.corpus_keys (new, default ("name",)) puts the Devanagari names into corpus.jsonl too, so snapshot.py --local watches both scripts. The word index and name_usage.py read both, counting a record once when its two names share a word.
  • meta.json records the keys, and a corpus built under other keys is refused on load. A meta written before this PR is read as ["name"], so the six existing caches keep loading.
  • tally reads the romanized name only.
• Not in the pairs view: that view compares the letters of two spellings, and a romanization shares none with its Devanagari. The Devanagari is therefore kept under its own key, not as registry, and pairs refuses the source.

What a fetch refuses

The failure that matters is a smaller corpus passing for the whole one, so:

• Lookup: a lookup listing other than NEPAL_BODIES (753) distinct int ids is refused.
• Candidate count: pinned at NEPAL_CANDIDATES (152,960), since the election is final.
  • With every file read, the count must be exactly that. A site that trimmed its files (to winners, to one post) is refused however well formed the rest is, and so is a one-row loss.
  • With files missing or empty, the count must be below the pin by at least one per absent file, since every real file held candidates. A shortfall of up to NEPAL_MAX_SHORT (1/50) is accepted, which means a few candidates trimmed elsewhere are not seen when files are missing. A JSON file cannot itself be cut and still parse.
  • NEPAL_MAX_SHORT was not sized against the largest bodies, which were not measured. Losing a few large files refuses the fetch, and largest_file in the detail says how large the largest was.
• Missing or empty files: a 404 counts the file as missing and an empty list as empty, both ids kept in meta.json. Together, past NEPAL_MAX_MISSING (1/100 of the bodies) they refuse the fetch.
  • Every file answered on 2026-10-24. The 18 the 2026-10-23 survey saw refused, ids 5551–5554 among them, answered when asked again. They were refused requests (a lapsed token answers 403), not missing files.
• Session renewal: a 403 renews the session once, counted in the detail; a second 403 is raised. 5xx, 429 and dropped connections are retried through _retried and then raised.
• Cut bodies: a body cut short does not parse, so it is retried and then refused, never read as fewer candidates. The test cuts it at five points, including inside a multibyte letter.
• Names: too many candidates past NEPAL_MAX_NAMELESS (1/100) without the name keys, a romanized name or a Devanagari one, or with a name in the wrong script (NEPAL_MAX_WRONG_SCRIPT). A value holding no letter counts as no name: 26 romanizations were placeholders such as - - -.
• Masked columns: too few distinct names in either column (NEPAL_MIN_DISTINCT, 1/4; 77% and 78% on 2026-10-24).
• Cookies: a cookie the hand parser cannot read is refused rather than left out of every request.

Every share is tested at its shipped value. Every guard has a negative control: it was mutated out, its test was seen to fail, and the file was restored with git checkout after a WIP commit.

Shared changes

• local_core._retried takes headers.
• Source.corpus_keys.
• local_corpora.fetch refuses a name that is empty or blank. It also refuses a declared second key that is not text or that no record fills.
• Every source's cached wordindex.json rebuilds once: its key now holds the corpus keys, and _names' code changed. The FEC index costs about 15 s once.
• README "Local corpora" (table row and paragraph), and the AGENTS.md sentence on which change reads which local corpus.
• Rebased onto Add jrc-names, a local corpus of JRC-Names person spellings, with a groups view #865 (jrc-names). Its read-version convention does not apply here: the records hold the source's own values, stripped, not values this code derives (as with tse-donors).

Leads, recorded here and not fixed

Measured on this branch (the parser is master's) over the 126,981 pairs fetched 2026-10-24. Counts only:

• Initial-style surnames:
  • B.K./K.C./G.C./B.C./D.C. are read as a suffix in 1,171 of the 2,830 romanized names that end in one. The survey at 48ed448, before every file answered, gave 1,130 of 2,539.
  • Filed as Should Ram Bahadur B.K. keep B.K. as its family name? #864; this corpus is its measurement.
• Romanized words read as titles:
  • Shekh 278, Md/Md. 220 and Mo. 42 (Should a title word written after another title be the given name (Mr Sultan Khan, Mr Md Karim Uddin)? #853), Buddha 79, Guru 44, Gyani 37, Siddha 12, Ku 11.
  • Md. reads as a title by its period (rules/H2's shape rule), while bare Md reads given since Should a title word written after another title be the given name (Mr Sultan Khan, Mr Md Karim Uddin)? #853.
• Devanagari words read as titles: श्री 215, श्रीमती 92, गुरु 44, सुश्री 17, श्रीमान 8.
• A title glued to a given name is read whole as the title: डा. + given (1 name here), and Latin Dr.John Smith. This is capitalized() lowercases a letter that should stay capital, after a period (J.R.R. → J.r.r.) or after an ayn (ʿAlī → ʿalī) #838's "joined word closed by none" spelling.
• Vowel signs, the virama and ZWJ/ZWNJ split no word:
  • 719 Devanagari names tokenize to a word count other than their spaces give. Every one holds a parenthesis (668), a full stop (64) or a comma (51).
  • No name of letters, marks and spaces alone does.

Checks

• Fetched three times on 2026-10-24, the last with this code: identical counts each time. snapshot.py --local nepal-candidates --record read the 237,408 names in 5.7 s, and the check matches.

• uv run ruff check, uv run mypy, uv run pytest (3.11): clean.

• uv run --python 3.15 pytest -W error::ResourceWarning tests/v2/test_local_corpora.py tests/v2/test_local_agreement.py: clean.

• Six review passes:

  • code, three times
  • comments
  • tests
  • silent failures

• Each review's fixes were reviewed again.

• The fetch's progress output names no candidate, and the tests use invented names only.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (cb1ece6) to head (7dd4445).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #866   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4669     4669           
=======================================
  Hits         4625     4625           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.