Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/design/decisions/H9-title-phrase.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,9 @@
Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base <checkout of this commit's parent>`, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base <the same checkout>` (and `fec-individuals`, `tse-donors`).

- **2026-10-26 #752, the v1 title list — A v1 `Constants().titles` HOLDS EVERY TITLE 2.6.0 LISTED THAT IS STILL A TITLE OF EITHER KIND.** The review of phase 2j's fifth and sixth batches found that a word a batch moved from `titles` to `titles_ambiguous` also left v1's default `titles`, which the shim copied from TITLES: `Constants().titles.remove('king')` raised `KeyError` and `'king' in Constants().titles` was False, where 2.6.0 accepted the removal and answered True. Measured against the 2.6.0 wheel (`PYTHONSAFEPATH=1`, `nameparser.__file__` asserted on both sides), word by word over all nine v1 set attributes and the `capitalization_exceptions` keys: `titles` alone had lost words (`bard`, `buddha`, `gyani`, `king`, `mahdi`, `siddha`, `wing`, and `sultan` and `sultana` once the batch above landed), and `suffix_not_acronyms` had gained the seven agnomens and `fils` of phase 2h, a vocabulary addition its release bullet records. `suffix_acronyms_ambiguous` (derived in phase 2a), `given_name_abbreviations` (2k) and the phase 2h sets have no v1 manager that lost a word. The translation: `nameparser/config/_titles_2_6_0.py` freezes 2.6.0's TITLES, generated from the wheel and never edited, and v1's default `titles` is TITLES with every 2.6.0 title now in `titles_ambiguous` added back (`moved`). `Constants._build_snapshot` passes a moved word to `titles_ambiguous` while the caller keeps it, so a default `Constants` still translates to `Lexicon.default()`, and passes it to neither once removed, which turns its modifier, style and phrase readings off with it, as removing any title does: `c.titles.remove('king')` then reads `Hon. King Smith` as title 'Hon.', given 'King', 2.6.0's reading after the same removal. A word 2.6.0 never listed (`dean`, `justice`, `grace`, `major`, `marshal`) is not in `moved`, so `c.titles.add('dean')` still makes it a title everywhere. The gap, accepted because v1's one bit per word cannot carry three readings: adding a moved word in v1 restores the default, the phrase reading, and not 2.6.0's title everywhere, which a caller gets from the 2.0 API (`Lexicon.default().add(titles={'king'})`). Keyed on the compiled sets rather than a hand list, so a later batch that moves another 2.6.0 title (`sheikh`) is covered with no edit, as `sultan` and `sultana` were when the batch above merged in, and `tests/v2/test_config_shim.py` sweeps every moved word by the same derivation: membership, set operators, `remove`, `discard`, re-adding, a replaced list, pickle, deepcopy and copy, 37 of its 39 rows failing on master `6ba78d0d` (the two that pass are the non-empty guard and `dean`'s recipe), with rows added in review for a spelling Lexicon folds onto a moved word (`king。`, `wing。`: the moved test and the modifier, style and phrase intersections compare folded spellings, #582's rule, the second review finding that folding the first alone lost `Wing Commander`) and for a replaced list's unlisted words. The default parse does not move: the snapshot, the CLDR, GND and native agreement records and the facade case rows pass unchanged.
- **2026-10-26 #752, phase 2j, `shaikh` (Derek) — THE SEVENTH TITLE BATCH: THE FIRST STYLE OF THE BATCHES.** The batch decisions/vocabulary-model's 2026-10-24 #752 (Derek) entry put seventh, `sheikh` left a batch of its own. `shaikh` gains a dated `given` reading and is a title only inside a title phrase; no listed phrase holds it, so nothing is reported. `Shaikh Ahmed Khan`, `Mr Shaikh Ahmed` and `Khan, Shaikh` read given 'Shaikh', where 1.4.0 through 2.6.0 read title 'Shaikh' (and `Mr Shaikh`). A style keeps the title (`His Highness Shaikh Ahmed Khan`, as 1.4.0 read it), as does a modifier (`Grand Shaikh Ahmed`) or `of` (`Shaikh of Bahrain`); the period-marked spelling at the front is a title by H2's shape (`Shaikh. Ahmed`). The other sites read as for `king` (the entry above): before a family comma H6 no longer peels it (`Shaikh Smith, Ahmed` → family 'Shaikh Smith', 1.4.0's reading), ending the name (`Ahmed Khan Shaikh.` → family 'Shaikh.', 1.4.0's), alone after a full name and a comma, past a second comma, and under a family-first order the word before a name is the family.
- A STYLE AS A PARTNER. `shaikh` is a style (the second contested one, after `grace`), so where it is a title it partners a contested word behind it (`His Highness Shaikh Sultan Khan` → title 'His Highness Shaikh Sultan'); where it is read as a name word it partners nothing, being no title there (`Shaikh Sultan Khan` → given 'Shaikh', middle 'Sultan', where master read title 'Shaikh Sultan'). Nor does it partner where only H2's shape makes it a title, the partner test counting a contested word a title only where its own phrase did (`_vocab.title_readings`): `Shaikh. Sultan Khan` → title 'Shaikh.', given 'Sultan', where master and 1.4.0 read title 'Shaikh. Sultan', and `Shaikh. Dean Smith` → given 'Dean', now reported `title-or-name`, as 1.4.0 read it. Master read `Grace. Dean Smith` so already, while a modifier so written partners (`Wing. Dean Smith` → title 'Wing. Dean'); the review of this batch found the shape, which rules/H9's H2 bullet now states. No local corpus writes `Shaikh.` before another word. The 2026-10-26 `sultana` and `sultan` entry's note on `Sheikh Sultan Khan` stands for `sheikh`, still a title.
- THE COUNTS, recorded on the word (`uv run python tools/name_usage.py Shaikh`, 2026-10-26): given US SSA 41 male births, Wikidata none, nam_dict not listed; family Census 6,063 (asian/PI 89.61%), Wikidata 49 (India 7, Pakistan 5). Mostly a South Asian surname, which stands last and was a name word already; first in the name it is a given-name element, as PSC's filers write it (below).
- THE COST, accepted by Derek on 2026-10-27 ("accepted"), shown the share of rulers below beside `king`'s and `sultan`'s and that `His Highness` in front keeps the title: a ruler or a member of a ruling house written with the bare title in front reads it as the given name (`Shaikh Hamad bin Isa Al Khalifa` → given 'Shaikh', middle 'Hamad bin Isa', where 1.4.0 through 2.6.0 read title 'Shaikh', given 'Hamad'), the class accepted for `king` and `sultan`, here larger in news text and smaller in record data. In jrc-names 59 persons have a spelling opening with `Shaikh`: by a reading of each, about 38 are Gulf rulers and members of their houses (`Shaikh` is the Bahraini English press's spelling), about 13 clerics and tribal or political leaders, and about 8 people whose name opens with it. In PSC the forename box holds it first in 38 of the 41 moved records and the title box in 2, beside 3 title-box filings newly set aside (below).
- MEASURED (2026-10-26, against master `fb4cafac`, py3.11, every local check with `--base`). PSC (1,039,171 scored, 3 more title boxes set aside, each `Shaikh` filed as a title) 33 records fixed, 1 broken (`Shaikh` filed in the title box), 7 changed, each still wrong for another reason (among them a two-word surname box and a title-box filing); ctgov-officials 8 moved, 7 fixes (4 `Shaikh A Aa, CRED` read with the initial as the given name, 2 `Aa Shaikh, CRED, CRED` whose credential master read as the given name), 1 still reading its credential as the given name; fec-individuals 2, both fixes after a family comma (`AA, SHAIKH` and `AA, SHAIKH AA`: given 'SHAIKH' where master read the title); tse-donors and nepal-candidates none; spelling pairs (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE) jrc-names 0 persons now alike and 12 now apart, every one a person spelled both `Shaikh` and `Sheikh`, which `sheikh`'s batch is to rejoin. No committed corpus name held `shaikh` before the batch's three rule examples, so they and CLDR and GND agreement do not move, and could not have. A v1 `Constants().titles` keeps `shaikh` with no edit (the v1 title list entry above), its sweep in `tests/v2/test_config_shim.py` gaining six rows.
Recompute: `uv run python tools/differential/local_agreement.py psc-individuals --base <checkout of this commit's parent>`, and `uv run python tools/differential/snapshot.py --local ctgov-officials --base <the same checkout>` (and `fec-individuals`, `tse-donors`, `nepal-candidates`).
1 change: 1 addition & 0 deletions docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR
- `dean`, `justice` and `major` gain an office or military reading beside a dated `given` one; `justice`'s exclusion (#606) is replaced by the reading. The batches of 2j declare the rest.
Adding the four fields changes `Lexicon`'s pickle layout, as phase 2h did.
- **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date.
- **2026-10-26 #752, phase 2j, `shaikh` (Derek) — THE SEVENTH TITLE BATCH: `shaikh` has a dated `given` reading**, so the style is a title only inside a title phrase. `Shaikh Ahmed Khan` and `Mr Shaikh Ahmed` read given 'Shaikh'; a ruler written with the bare title does too (`Shaikh Hamad bin Isa`), the class accepted for `king` and `sultan`, the larger share of rulers Derek accepted on 2026-10-27. `sheikh` keeps its own batch. The measurement is decisions/H9's entry of this date.
- **2026-10-26 #752, phase 2j, `sultana` and `sultan` (Derek) — THE FIFTH AND SIXTH TITLE BATCHES, IN ONE PR: both have dated `given` readings**, `sultan`'s on its tr.toml entry, so each is a title only inside a title phrase. `Sultana Begum` and `Sultan Khan` read the given name; a sultan written with the bare title does too (`Sultan Hassanal Bolkiah`), the cost accepted for `king`. Derek combined the two batches on 2026-10-26. The measurement is decisions/H9's entry of this date.
- **2026-10-26 #752, phase 2j, `king` (Derek) — THE FOURTH TITLE BATCH: `king` has a dated `given` reading**, US SSA 37,890 births and Census rank 34 as a surname, so it is a title only inside a title phrase. `King Curtis` and `Mr King Smith` read given 'King'; a monarch written with the bare title does too (`King Henry`), the cost Derek accepted on 2026-10-26. `pope` replaces `king` as the worked example of a title that is also a surname wherever the rules and tests used it. The measurement is decisions/H9's entry of this date.
- **2026-10-26 #752, phase 2j, `buddha`, `siddha` and `gyani` (#869) — THE THIRD TITLE BATCH: each has a declared `given` reading**, `siddha`'s and `gyani`'s without a count, which their notes explain: the first halves of Nepali compound given names, which no source outside nepal-candidates counts. `Buddha Bahadur Tamang` reads given 'Buddha', middle 'Bahadur'; `guru` stays a title for a `ne` pack (the 2026-10-24 #752 (Derek) entry). The measurement is decisions/H9's entry of this date.
Expand Down
7 changes: 5 additions & 2 deletions docs/design/rules/H9-title-phrase.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
"Buddha Bahadur Tamang" → given="Buddha"
"King Curtis" → given="King"
"Sultana Begum" → given="Sultana"
"Mr Shaikh Ahmed" → given="Shaikh"
```
- **A modifier in the leading run of titles before the first comma** has its phrase in a head title after it in that run, in `of` after it, or in the object of `of` or `the` there. A modifier outside that run, ending the name or after a comma, is read as any title is, save one that may be a given name, which is read everywhere as it is in that run.
```
Expand Down Expand Up @@ -68,11 +69,12 @@
```
"Hon. Wing Lau" → given="Wing"
```
- **A period-marked word in the leading run of titles** (H2): the abbreviation shape makes it a title whatever the vocabulary says, save a given name written short or a word a connective joins (below), so the phrase test never reaches it and nothing is reported, at the front of the name or behind another title.
- **A period-marked word in the leading run of titles** (H2): the abbreviation shape makes it a title whatever the vocabulary says, save a given name written short or a word a connective joins (below), so the phrase test never reaches it and nothing is reported, at the front of the name or behind another title; a style that may be a given name, made a title by the shape alone, is no partner of the word behind it, which is then read and reported as at the front, where a modifier is (`Wing. Dean Smith` → title 'Wing. Dean').
```
"Dean. Smith" → title="Dean."
"Wing. Lau" → title="Wing."
"Mr Buddha. Tamang" → title="Mr Buddha."
"Shaikh. Sultan Khan" → given="Sultan"
```
- **A title run** (H3, H4): a word this rule reads as a name word is no title word to the run, so the run stops before it and does not count it.
```
Expand Down Expand Up @@ -120,10 +122,11 @@
"Gyani Zail Singh" → given="Gyani"
"Lord Buddha Tamang" → given="Buddha"
```
- A monarch written with the bare title in front reads it as the given name, a style before it or `of` after it keeping the title; the record data at hand writes the word there almost only as a given name, and monarchs that way mostly in news text.
- A ruler written with the bare title in front reads it as the given name, a style before it or `of` after it keeping the title; the record data at hand writes the word there almost only as a given name, and rulers that way mostly in news text.
```
"King Henry" → given="King"
"Sultan Hassanal Bolkiah" → given="Sultan"
"Shaikh Hamad bin Isa" → given="Shaikh"
"His Majesty King Charles" → title="His Majesty King"
```
- Once a partner has made the word a title, a coordinating join behind it takes the next word into the title too.
Expand Down
1 change: 1 addition & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Release Log
- **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way. Each is a surname too, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752`
- **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752`
- **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). A v1 ``Constants().titles`` still lists ``King``, ``Mahdi`` and the other titles 2.6.0 listed, and removing one reads it as a name everywhere, but adding one there no longer makes it a title outside a phrase; that takes the ``Lexicon`` spelling. :issue:`752`, :issue:`853`
- **Fix the name Shaikh reading as a title.** ``Shaikh Ahmed Khan`` and ``Mr Shaikh Ahmed`` give first ``Shaikh``, where 1.4.0 through 2.6.0 gave title ``Shaikh`` and ``Mr Shaikh``; ``Khan, Shaikh`` reads the same. A ruler written with the bare title reads it as a first name too (``Shaikh Hamad bin Isa`` gives first ``Shaikh``), where a style keeps it (``His Highness Shaikh Hamad``); the spellings ``Sheikh`` and ``Shekh`` are still titles. :issue:`752`, :issue:`853`
- **Fix the given names Sultan and Sultana reading as titles.** ``Sultan Khan`` and ``Sultana Begum`` give first ``Sultan`` and ``Sultana``, where 1.4.0 through 2.6.0 gave title ``Sultan`` and ``Sultana`` with no first name; ``Mr Sultan Khan`` and ``Khan, Sultan`` read the same. A sultan written with the bare title reads it as a first name too (``Sultan Hassanal Bolkiah`` gives first ``Sultan``), where a style keeps it (``His Majesty Sultan Hassanal Bolkiah``). :issue:`752`, :issue:`853`
- **Fix the given name King reading as a title.** ``King Curtis`` and ``Mr King Smith`` give first ``King``, where 1.4.0 through 2.6.0 gave title ``King`` and ``Mr King``; ``SMITH, KING`` reads the same. A monarch written with the bare title reads it as a first name too (``King Henry`` gives first ``King``, last ``Henry``), where a style keeps it (``His Majesty King Henry``). :issue:`752`, :issue:`348`
- **Fix the Nepali given names Buddha, Siddha and Gyani reading as titles.** ``Buddha Bahadur Tamang`` and ``Gyani Maya Gurung`` give first ``Buddha`` and ``Gyani``, middle ``Bahadur`` and ``Maya``, where 1.4.0 through 2.6.0 gave title ``Buddha`` and ``Gyani``, first ``Bahadur`` and ``Maya``; ``Siddha Raj Joshi`` and ``Tamang, Buddha Bahadur`` read the same. ``Guru`` is still a title. The Sikh title written ``Gyani`` reads as a given name too (``Gyani Zail Singh``); its usual spelling ``Giani`` is still a title. :issue:`869`, :issue:`752`
Expand Down
2 changes: 1 addition & 1 deletion nameparser/config/_compiled.py
Original file line number Diff line number Diff line change
Expand Up @@ -673,7 +673,6 @@
'sgtmaj',
'sgtmajmc',
'shaik',
'shaikh',
'shayk',
'shaykh',
'sheik',
Expand Down Expand Up @@ -968,6 +967,7 @@
'mahdi',
'major',
'marshal',
'shaikh',
'siddha',
'sultan',
'sultana',
Expand Down
4 changes: 3 additions & 1 deletion nameparser/vocab/ar.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ style = { addresses = "given", gender = "m", same_as = "sheikh" }

[[word]]
text = "shaikh"
style = { addresses = "given", gender = "m", same_as = "sheikh" }
style = { addresses = "given", gender = "m", same_as = "sheikh", issue = 752, decided = 2026-10-26, note = "a title only inside a title phrase, its given reading the clash (rules/H9): 'Shaikh Ahmed' and 'Mr Shaikh Ahmed' read the given name, unreported, since no listed phrase holds it" }
given = { us_ssa = 41, as_of = 2026-10-26, note = "US SSA: male births; Wikidata holds no given-name statement for it, a lower bound" }
family = { us_census_2010 = 6063, wikidata = 49, as_of = 2026-10-26, note = "Census: asian/PI 89.61%; Wikidata: India 7, Pakistan 5" }

[[word]]
text = "shayk"
Expand Down
Loading
Loading