[derek73]

What

A new local corpus, jrc-names. It holds the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text, plus a new view that compares those spellings: local_corpora.py groups.

It is the one local corpus holding several spellings of one person across scripts and transliterations. tse-donors pairs two spellings, both Latin. It also holds 17,074 Arabic-script spellings (7,833 persons, 5,347 of them also spelled in Latin). The committed corpora hold 315 distinct names with an Arabic letter, 96 of those CLDR's fictional renderings.

Licence. The JRC-Names EULA is personal, non-exclusive and non-transferable, with no redistribution, sublicensing or derivative works. So nothing read from it is committed: not a name, not a per-name reading. The download, corpus, snapshot and views' output all stay under ~/.cache/nameparser-tools/local/jrc-names/. The tests use invented names only. The source docstring, the README row and the credit line every report opens with all say this.

The source (tools/differential/local_jrc.py)

• Pinned by sha256. The JRC says the resource is updated daily, but the file served on 2026-10-24 was last modified 2024-03-15, its data date. The fetch downloads entities.gzip (14.8 MB) into the cache and refuses any other sha256.
  • A download that does not match replaces nothing. It is kept as entities.gzip.new to be measured, and adopted once the pin moves to it. This matters because the JRC serves only its current file, so the pinned one could not be fetched again.
  • For the same reason, fetch --refresh downloads nothing when the cached file is already the pinned one; it reads the cached file again.
  • meta.json records the pin, a read version and the Unicode version script() read. A corpus fetched under another of any of the three is refused on load.
• What is kept. Person rows (P, 1,338,463 of 1,445,013 lines); organisations are counted and read past. A record is one distinct spelling: {entity, name, script}, with + read as spaces.
• Inflected variants: dropped by the JRC's own marking, measured first. The fetch drops every row carrying a language (ru, ar, sl, et, pl, …): all 20,819, counted by language in the detail.
  • Every marked row's entity holds an unmarked (u) spelling.
  • 18,594 of the 20,819 are measured as morphological variants of one: they differ only in word endings (Russian or Slovene case), or in up to three letters glued to the front (Arabic و, ل, ال). The exact rule is stated at JRC_UNMARKED.
  • The other 2,225 (1,680 of them Arabic) fit neither rule as stated, and are dropped on the JRC's marking alone.
  • Unmarked spellings still hold inflected forms the JRC did not mark: by the same rule, 22,732 of the 66,673 distinct Cyrillic spellings. These are kept, and documented as a limit.
• Guards, each with a stated and tested share:
  • lines that are not 4 well-formed columns: not UTF-8, an id not ASCII digits, an unknown type, a language that is not a code, or an empty name (JRC_MAX_BAD; the file's # header is the 1 such line);
  • too few person rows (JRC_MIN_PERSONS);
  • too many marked rows (JRC_MAX_MARKED);
  • persons left with only marked spellings (JRC_MAX_LOST; none in the pinned file);
  • too many repeated spellings (JRC_MAX_REPEATED; 58 in the pinned file);
  • a gzip that is cut, corrupt or not a gzip;
  • a file ending inside a line.
• Fetch cost: 15s. snapshot.py --local jrc-names needed no change: it recorded 1,315,051 names in 30s on 12 CPUs (128 MB).

The groups view

group_counts parses every spelling of each person with two or more (218,822 persons, 717,199 spellings) and compares the structure of the readings, never the words, so spellings in different scripts can be compared. The fields are: title, suffix, given, family, family word count, and where the family stands (first, last, whole, inner, none).

• It counts persons, not pairs, keyed by pair of scripts.
• Within one script, a field differs where the person's spellings in it read the field more than one way.
• Across two scripts, a field differs where the two scripts' spellings share no value for it. "Alike" means some spelling in one reads, in every field, as some spelling in the other. So two Latin spellings that differ from each other are not counted against the Cyrillic one.
• A person across two scripts can be neither "alike" nor apart in any one field. These are counted on a line of their own, "no one field" (0.4% of Han / Latin at most), so each pair's persons are alike, apart in a field, or there.
• It then lists the title and suffix words read in some of a person's spellings and none of the others, as vocabulary leads (at least 10 persons each).

I put it in a view of its own rather than in pair_shape/pair_counts, because those compare the strings of one filed/reference pair, while this compares N spellings by parse structure.

Results on the pinned file (counts only), about 50s:

• 93.7% of persons read alike in every spelling.

• Across scripts read in the same order, nearly all agree:

  Scripts	Alike	Persons
  Cyrillic / Latin	99.3%	20,837
  Arabic / Latin	98.2%	5,347
  Greek / Latin	98.6%	915

  Hangul against a script read given first is ≤3% alike, because its family stands first: a convention, not an error.

• Within a script, the disagreement is mostly writing conventions:

  • Katakana / Katakana, 0.2% alike. Each transliterated name appears unspaced (read as a given name alone), divided by ・, and spaced.
  • Han / Han, 2.2% alike. Unspaced Han is read as one family word; spaced Han is read family first.
  • Arabic / Arabic, 67.8% alike, mostly because of the Arabic comma. 1,510 Arabic spellings hold U+060C, which the parser reads as a comma, putting the family first. Each of the 736 persons with such a spelling differs, of the 1,072 that differ.

• Leads:

  • title words read in some spellings only: sheikh 50, bard 33, shaikh 27, sir 23, executive 22, and mahdi 17, a common given name;
  • suffix words read in some spellings only: jr 62, sa 38, iii 27, ii 27.

Possible follow-ups, none filed: whether Mahdi/Bard belong in TITLES, and how an Arabic comma inside a harvested name should read.

Docs

• README "Local corpora": a table row and two paragraphs, plus a sentence on the read version for any new source whose records hold a value its own code derives.
• local_corpora.py's docstring.
• mechanisms.md#NAME-DATA-JOBS: one sentence.
• The AGENTS.md "Validating a new parsing rule" sentence: which change reads this corpus.

Verification

• ruff, mypy, and the full suite on 3.11: 23,398 passed, plus the serial pass. The local-corpus modules on 3.15: 413 passed.
• 48 negative controls, one per guard and view rule: each guard removed, its test seen to fail, then restored with git checkout.
• Review rounds:

  1. A code review and a docs-claims review. These found the "alike" definition, the refresh that could overwrite the only pinned copy, the Arabic comma (my first draft said no name held a comma), and several counts over mismatched populations.

  2. A review of the fix commit. This found that the cached records' script labels were stale under the fixed script(), which is what the read version now catches. All figures were re-measured after a re-fetch.

  3. After opening: a five-part review (code, tests, comments, silent failures, types), then a simplification pass. Its main findings:

     • four mutations of the groups view that no test caught;
     • a parser ValueError that would have been reported as corpus damage;
     • two ways the fetch could shrink in silence;
     • a refresh that could fail once the JRC moves past the pin;
     • comments claiming more than was measured.

     Reading became a NamedTuple. Each fix has a control, and every figure was re-run after a re-fetch under read version 2.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (af0d614) to head (41baeca).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #865   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4669     4669           
=======================================
  Hits         4625     4625           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.