diff --git a/AGENTS.md b/AGENTS.md index 1df5dc0f1..d2bf7ae33 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,7 +94,10 @@ uv run sphinx-build -b html docs dist/docs # number works too). The mechanism, boundary cases and measurements live # in the issue and docs/design/decisions/, never in the bullet -- # 2.4.0's entry was cut from 10,765 words to 1,491 on 2026-10-11 -# - Merge bullets a user would see as one change, with several links +# - Merge bullets a user would see as one change, with several links, +# and carry every merged bullet's limits over: 2.7's merge of eight +# phase-2j bullets read 'a bare one reverses after a comma' over Vice +# and Grand, which stay titles there, until review ran them # - Version is decided at release time (patch/minor/major per semver) # - Format matches existing entries — see the 2.4.0 block for a current example @@ -129,6 +132,11 @@ uv run sphinx-build -b html docs dist/docs # but statement and Background prose can still be wrong about behavior # that changed # - AGENTS.md itself, for stale commands, architecture notes, or gotchas +# - the pickle-break bullet: pickle every public type (Lexicon, Policy, +# Parser, ParsedName, Ambiguity, HumanName, a locale pack) under the +# last release's wheel and load each in the tree; don't derive it +# from a dataclass diff. 2.7's bullet named Lexicon and Parser and +# missed the locale packs, which carry a Lexicon # And check for open Dependabot PRs on uv.lock (namedivider-python) and merge them # first — pyproject floats >=0.4 so fresh installs get the newest namedivider, but # CI's ja-extra job installs from uv.lock and only tests what the lock pins @@ -207,7 +215,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2 and re-measured 2026-10-25 with the key below, the earlier `dphil` example having become a shipped mask in #797: `CAPITALIZATION_EXCEPTIONS['dclinpsy'] = 'DClinPsy'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. -- `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute); `TITLES_AMBIGUOUS` (a title reading beside a `given` one, derived, compiled INSTEAD of into `TITLES`: a title only inside a title phrase, rules/H9, #752 phase 2i), `TITLE_MODIFIERS` and `TITLE_STYLES` (by kind) and `TITLE_PHRASES` (the two-word phrases `attaches` makes), none with a v1 attribute +- `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset of the title readings by construction, though not of `TITLES`: a word that may also be a given name, such as `king` or `sheikh`, compiles to `TITLES_AMBIGUOUS` instead) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute); `TITLES_AMBIGUOUS` (a title reading beside a `given` one, derived, compiled INSTEAD of into `TITLES`: a title only inside a title phrase, rules/H9, #752 phase 2i), `TITLE_MODIFIERS` and `TITLE_STYLES` (by kind) and `TITLE_PHRASES` (the two-word phrases `attaches` makes), none with a v1 attribute - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a word reading on a word with a name reading, `fils` and the agnomens, gated by the count, rules/S11, since 2026-10-25) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (ported from v1's `_join_bound_first_name`) @@ -257,7 +265,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. **Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. @@ -275,7 +283,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **`HumanName.C` is a property backed by `_C`, but pickles under the public key `'C'`** — `__init__` and direct assignment both validate, so an invalid value (or `None`, removed in 2.0, #261) raises `TypeError` immediately instead of surfacing later deep in parsing (#239). `__getstate__`/`__setstate__` deliberately translate `self._C` ↔ a `'C'` key in the pickled dict (with the `CONSTANTS`-singleton-becomes-`None` sentinel) rather than pickling `_C` directly, so the on-disk pickle format is v1's — don't "simplify" that translation away or 1.4 pickles and tests that hand-build a state dict with a `'C'` key will break. -**Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it via opt-in `Constants`. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. +**Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it via opt-in `Constants`. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Frau` is a first name in some contexts, and `Prince` and `Sheikh` were titles until 2.7 gave them a `given` reading, which moved them to `TITLES_AMBIGUOUS` (rules/H9). It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`) is still out of `TITLES`; `marshal` reads as a title only in a phrase (`Field Marshal`). `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. **The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. The one vocabulary it yields to, since #789 (phase 2k of #752), is a given name written short (`Lexicon.given_name_abbreviations`, a `given` reading with `same_as` in the TOML): `"Chas. Smith"` → `given='Chas.'`, read off classify's `vocab:given-abbreviation` tag in `is_leading_title` and off the lexicon in `_vocab.is_title_shaped`, the two spellings of one test that `test_is_title_shaped_and_is_leading_title_agree` holds together. The second spelling is what rules/C4's comma count and the title phrase's head run ask, so a change to the shape moves comma names nobody wrote it for: #789's first grid held no comma and missed `Chas. Smith, Ed` (given 'Ed', family 'Chas. Smith' before, suffix 'Ed' after), which only mutating `is_title_shaped`'s half alone showed. Put a word of the ambiguous class after a comma behind the shaped word (`X. Smith, Ed`), and a modifier in front of it (`Vice X. Smith, Ed`), in the grid of any change to the shape, and mutate each spelling separately. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane Pope"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") @@ -285,13 +293,13 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Comparing against v1 needs `PYTHONSAFEPATH=1` AND a directory outside the worktree** — `uv run --isolated --no-project --with 'nameparser==1.4.0'` still puts the checkout's `nameparser/` ahead of the pinned wheel on `sys.path`, so the "v1" side silently imports the branch and every comparison reports parity. Run it as `cd && PYTHONSAFEPATH=1 uv run --isolated --no-project --with 'nameparser==1.4.0' python -c "..."`. This produces false confidence rather than an error, so it invalidates results without ever looking wrong. **A `tests/v2/cases.py` row's `parity` means 1.4.0, and nothing checks it** -- the table runs every row against the tree, so a row labelled `parity` that only matches master passes for good. #614 added eight such rows, compared against master; #620 found them by running the wheel this way and relabelled each with the change its reading came from, found by `git bisect run` in a scratch worktree (`git worktree add --detach`, never this one). Run the wheel before writing `parity`. -**A group-side reading of a family comma's given part that asks `tail_reading` disagrees with the given part's own walk, and the disagreement is accepted.** `tail_reading` reads the part as the comma-less name reads it, plus the bare salutations rules/H8 adds after a family comma, which a predictor of this part asks for with `salutations=True` (#835); assign's walk over the given part after a family comma reads by its own predicates -- a shape-only numeral VI to X stays a name word there (`Smith, John van der Berg VI` reads middle 'van der Berg VI'), V before a third comma part is read strictly (#144), and P6 attaches a period-marked title-particle (`St.`) the comma-less reading takes as a title. So a bound or a decline group computes with `tail_reading` acts where the walk then reads no title: #623's chain bound (`Smith, John Mc Prof. VI`) and #637's join decline (`Smith, abdul Prof. VI` → middle 'Prof. VI', `Smith, abdul St.` → family 'St. Smith'). Both were recorded rather than narrowed (Derek: garbage input; decisions/H5 and #P5, rules/P5's Accepted clause). Before writing another group-side predictor of the given part, sweep those three shapes and decide, rather than rediscovering the gap in review. And a change to the trailing peel ITSELF reaches the given part through those predictors without touching them: #718's words-to-spare count for a one-letter numeral, asked through `trailing_run_start`, made the chain take the V in `Smith, Freiherr von Berg V` (given 'von Berg V', unreported) while assign's walk, which never counts there (S4: "read without it"), read it as the suffix -- found by review on a grid of family-comma and empty-head shapes behind a title-particle; the predictor now passes `count_numeral=False`. When a change moves what the peel takes, put a family comma with a title-particle head in its grid. +**A group-side reading of a family comma's given part that asks `tail_reading` disagrees with the given part's own walk, and the disagreement is accepted.** `tail_reading` reads the part as the comma-less name reads it, plus the bare salutations rules/H8 adds after a family comma, which a predictor of this part asks for with `salutations=True` (#835); assign's walk over the given part after a family comma reads by its own predicates -- V before a third comma part is read strictly (#144), staying a name word (`Smith, John van der Berg V, Jr.` reads middle 'van der Berg V'), and P6 attaches a period-marked title-particle (`St.`) the comma-less reading takes as a title. So a bound or a decline group computes with `tail_reading` acts where the walk then reads no title: #623's chain bound (`Smith, John Mc Prof. V, Jr.` → suffix 'Mc, Jr.', middle 'Prof. V') and #637's join decline (`Smith, abdul Prof. V, Jr.` → middle 'Prof. V', `Smith, abdul St.` → family 'St. Smith'); a numeral VI to X was a third such shape until #784 read it as a suffix on both sides. Both were recorded rather than narrowed (Derek: garbage input; decisions/H5 and #P5, rules/P5's Accepted clause). Before writing another group-side predictor of the given part, sweep those two shapes and decide, rather than rediscovering the gap in review. And a change to the trailing peel ITSELF reaches the given part through those predictors without touching them: #718's words-to-spare count for a one-letter numeral, asked through `trailing_run_start`, made the chain take the V in `Smith, Freiherr von Berg V` (given 'von Berg V', unreported) while assign's walk, which never counts there (S4: "read without it"), read it as the suffix -- found by review on a grid of family-comma and empty-head shapes behind a title-particle; the predictor now passes `count_numeral=False`. When a change moves what the peel takes, put a family comma with a title-particle head in its grid. **A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/compare.py` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. The proof has to come from the input class the null result is about: #683 moved `tools/name_usage.py` to a POST and checked it with `shehu`, which is ASCII and so cannot show that the server decodes a Cyrillic body. A misdecoded query still runs and matches nothing, so the Cyrillic zeros it was meant to vouch for proved nothing until `Иван` (17,119 given) went through the same path. The converse trap is a mutation that moves nothing: it shows the code is unexercised by your grids, not that it is dead. #638 was filed calling group's chain re-ask dead on a grid with no title-particle head before a particle run and a dotted acronym; the next review found 400 moves (`Smith, Freiherr van Berg X.Y.Z.`) and the issue closed the same day. Before calling code dead, look for the input that reaches it. A CACHE of readings is the same trap one step removed: a wrong key copies one tree's readings in as another's, and the check then reports nothing. #859 first keyed `snapshot.py --local`'s kept baselines by `git rev-parse HEAD:nameparser`, which answers for the enclosing checkout, so a gitignored copy inside one (`build/lib`) took the checkout's key; two reviewers found it. Key a cached reading by what the import reads -- the files on disk, hashed again after the read -- never by what version control says should be there. And a switch that changes no file defeats any file key: #752's spike set its options by environment variable, so after the first run every option copied that run's cached local readings until the variables joined the key (2026-10-24). A SHARED file of readings is the same trap with no key at all: a local corpus's `snapshot.jsonl` and `psc-individuals`' `agreement.jsonl` are one file per corpus for every worktree, decided by whoever recorded last, and on 2026-10-25 parallel sessions of #752's phase 2 read a sibling's re-record as their own moves -- the silent case being a sibling's record from a branch holding the same change, which reads as "no moves". Use `--base ` whenever another session may be measuring: it reads the base from a baseline keyed by its package's files and never touches the shared file. A kept reading is published under its key only once it is whole and its tree unchanged: `--base`'s first draft had the agreement child write straight to the kept path, through a temporary name every session shared, so two sessions scoring one base could keep a record cut on a line boundary, which every later session would reuse. Write into a folder of the run's own, read it back against a count, take the digest again, then rename it into place. **`SUFFIX_ACRONYMS` and `SUFFIX_WORDS` are DISJOINT by construction (since 2026-10-18; an import-time assert from 2026-09-08) — a word has one suffix kind, and its reading is an acronym or a word (`acronym`), so the vocabulary build refuses a word classified twice; the fix is to choose, not to relax the check** — the two branches normalize differently (the word test strips only edge periods, the acronym test strips all of them), so a word in both is matched by two rules and which one fired is unreadable from outside. `esq` was the single overlap and left `SUFFIX_ACRONYMS`, which is what made the assert possible; the whole algebra that had made the overlap safe, and the criterion that decided which set a word belongs to, are in the `suffix-acronym-collisions` entry of `docs/design/decisions/` if anyone wants them. -**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). +**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES ∪ TITLES_AMBIGUOUS` puts the entry in `titles` or `titles_ambiguous` too, each per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". diff --git a/CITATION.cff b/CITATION.cff index f9690c938..5052f1dc6 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -9,4 +9,4 @@ authors: repository-code: "https://github.com/derek73/python-nameparser" url: "https://github.com/derek73/python-nameparser" license: LGPL-2.1-or-later -version: "2.6.0" +version: "2.7.0" diff --git a/README.rst b/README.rst index 4ba937903..059f58be2 100644 --- a/README.rst +++ b/README.rst @@ -7,11 +7,14 @@ nameparser parses human names into seven fields — title, given, middle, family, suffix, nickname, maiden. Results are immutable, configuration is composable, and locale packs are opt-in. -📣 **nameparser 2.6 is out.** It keeps surnames that earlier releases -read as credentials (``Luo Si``, ``Maria de Sa``, ``Ram Bahadur B.K.``), -reads ``Wm`` and ``Md`` as the given names they abbreviate, recognizes more credentials after a comma -(``John Smith, MBChB``), and adds an opt-in reading for names with two -family names (``Gabriel García Márquez``). The `release log +📣 **nameparser 2.7 is out.** It is the first release working towards +the new language and vocabulary model described in the `vocabulary +model RFC `__. +It introduces more nuanced handling of titles and particles +(``Chief Justice John Roberts`` reads the title ``Chief Justice``, +while ``Justice Smith`` keeps the given name ``Justice``), adjusts the +default vocabulary to better align with example name corpora, and +surfaces alternative name readings in the published API. The `release log `__ has the details. diff --git a/docs/concepts.rst b/docs/concepts.rst index 534b6507b..7328fed95 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -65,7 +65,9 @@ work. A vocabulary layer runs first and claims words for what they *are*, wherever they sit: titles, particles, conjunctions, recognized suffixes, and anything set off by nickname or maiden delimiters. Titles chain, so ``"Asst. Vice Chancellor"`` is one -title; particles join forward, so ``de la`` attaches to ``Vega``. +title, and a title that is also a given name, such as ``Dean``, is +claimed only inside such a phrase (``"Dean of Students"``); particles +join forward, so ``de la`` attaches to ``Vega``. Whatever the vocabulary layer has not claimed is left to a positional layer, which assigns purely by where a word sits: the first unclaimed diff --git a/docs/customize.rst b/docs/customize.rst index fe5264a4c..be37b561d 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -16,11 +16,17 @@ Adding and removing words .. doctest:: - >>> from nameparser import Lexicon, Parser + >>> from nameparser import Lexicon, Parser, parse + >>> parse("Dean Robert Johns").given + 'Dean' >>> lex = Lexicon.default().add(titles={"dean"}) >>> Parser(lexicon=lex).parse("Dean Robert Johns").title 'Dean' +By default ``Dean`` is a title only inside a title phrase such as +``Dean of Students`` (:ref:`titles-also-given-names`); listing it in +``titles`` makes it a title wherever it stands. + :meth:`~nameparser.Lexicon.add` and :meth:`~nameparser.Lexicon.remove` both return a new :class:`~nameparser.Lexicon` — the one you started from (here, ``Lexicon.default()``) is never mutated. Every field @@ -102,41 +108,55 @@ Removing works the same way, and drops the word from recognition: >>> Parser(lexicon=lean).parse("Professor Robert Johns").title '' -A few fields mark a subset of another — ``given_name_titles``, -``after_given_titles``, ``title_modifiers`` and ``title_styles`` over -``titles``, ``particles_ambiguous`` over ``particles``, -``suffix_acronyms_ambiguous`` over ``suffix_acronyms``, -``conjunctions_ambiguous`` over ``conjunctions``, -``honorific_tails`` and ``suffix_words_ambiguous`` over -``suffix_words``, and ``agnomens`` over ``suffix_words_ambiguous``. -Entries belong in the base field too, so add to both and remove from -the marker first. Four of them enforce that — ``particles_ambiguous``, -``suffix_acronyms_ambiguous``, ``honorific_tails`` and -``suffix_words_ambiguous`` raise ``ValueError`` naming the orphans, -because an orphan in each of those does real harm rather than nothing. -The others are deliberately unchecked because an orphan there is -inert, an ``agnomens`` entry being read only for a word -``suffix_words_ambiguous`` marks, and a ``title_modifiers`` or -``title_styles`` entry only for a title: -``given_name_titles`` matches a title run as one space-joined string, or -by that run's last word, so a legitimate entry like ``"sir and dame"`` -is no single word in ``titles``; and a ``conjunctions_ambiguous`` or -``after_given_titles`` entry is only ever read for a word that is a -conjunction or a title, so ``remove(conjunctions={"e"})`` simply works -and the stale marker entry is never consulted. - -``titles_ambiguous`` is the one title field that is no subset of -``titles``: a word in it is a title only inside a title phrase, and a -word in both is a title wherever it stands, which is what listing it in -``titles`` asks. ``title_phrases`` holds the two-word phrases a -``titles_ambiguous`` word may stand in (``"chief justice"``, -``"major general"``), each read only where both words are titles: +Marking a subset of another field +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. doctest:: +Several fields mark a subset of another, qualifying how the base +field's words are read. Their entries belong in the base field too, so +add a word to both, and remove it from the marker first. An entry +outside its base is an *orphan*: four markers refuse one with a +``ValueError`` naming it, because there it would change the parse; the +rest leave it unchecked, because there it is never read. - >>> chief = Lexicon.default().remove(title_phrases={"chief justice"}) - >>> Parser(lexicon=chief).parse("Chief Justice John Roberts").title - 'Chief' +.. list-table:: + :header-rows: 1 + :widths: 30 34 36 + + * - Marker + - Base + - An orphan + * - ``particles_ambiguous`` + - ``particles`` + - raises ``ValueError`` + * - ``suffix_acronyms_ambiguous`` + - ``suffix_acronyms`` + - raises ``ValueError`` + * - ``honorific_tails`` + - ``suffix_words`` + - raises ``ValueError`` + * - ``suffix_words_ambiguous`` + - ``suffix_words`` + - raises ``ValueError`` + * - ``agnomens`` + - ``suffix_words_ambiguous`` + - is never read + * - ``conjunctions_ambiguous`` + - ``conjunctions`` + - is never read + * - ``after_given_titles`` + - ``titles`` + - is never read + * - ``given_name_titles``, ``title_modifiers``, ``title_styles`` + - ``titles`` or ``titles_ambiguous`` + - is never read + +So ``remove(conjunctions={"e"})`` simply works, and the stale +``conjunctions_ambiguous`` entry is never consulted. +``given_name_titles`` is unchecked for a second reason: it matches a +run of titles as one space-joined string, or by the run's last word, so +a legitimate entry like ``"sir and dame"`` is no single word of +``titles``. ``titles_ambiguous`` is no subset of ``titles`` at all; see +:ref:`titles-also-given-names`. .. _title-detection-off: @@ -332,7 +352,9 @@ them — ``suffix_acronyms_ambiguous``, ``suffix_words_ambiguous``, ``suffix_acronyms``, ``suffix_words``, ``particles`` and ``conjunctions``. They add no vocabulary by themselves; they narrow how an existing entry is read when it appears alone, and the parse reports -the fork as an ambiguity. +the fork as an ambiguity. Titles that are also given names work +differently, with a word list of their own, ``titles_ambiguous`` +(:ref:`titles-also-given-names`). Credentials that are also surnames ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -432,12 +454,67 @@ free: needs one adds it back with ``Lexicon.default().add(suffix_acronyms={"cha"})``. -The same conservatism is why ``dean`` above is a title by default only -inside a title phrase ("Dean of Students", "Associate Dean"): "Dean" is -also a common given name, and a default that swallowed it as a title -would misparse "Dean Martin" for everyone. Listing it in ``titles`` -makes it a title wherever it stands. The phrase-only titles are -``Lexicon.titles_ambiguous``. +.. _titles-also-given-names: + +Titles that are also given names +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Many titles are borne as given names too: ``Dean``, ``King``, +``Prince``, ``Grace``, ``Major``, ``Sheikh``. A title addresses the +name after it, so at the front of a name there is no count of words to +read, and what tells the person ``Dean Smith`` from the office in +``Dean of Students`` is the phrase the word stands in. Since 2.7 these +words are listed in ``titles_ambiguous`` rather than ``titles``, and +each is a title only inside a title phrase: behind a style +(``His Majesty King``) or a modifier (``Associate Dean``), before +``of`` (``Dean of Students``), or in a listed two-word phrase +(``Major General``). Anywhere else it is a name word: + +.. doctest:: + + >>> parse("Dean Smith").given + 'Dean' + >>> parse("Associate Dean Smith").title + 'Associate Dean' + >>> parse("Dean of Students Jane Smith").title + 'Dean of Students' + >>> parse("Major General John Smith").title + 'Major General' + +Where such a word opens the name and a listed phrase could have held +it, the parse reports a ``title-or-name`` ambiguity, its alternative +the title reading (:ref:`reading-not-taken`). A word no listed phrase +holds, such as ``Grace`` (a title in ``His Grace``), reports nothing +standing alone: + +.. doctest:: + + >>> dean_alt = parse("Dean Smith").ambiguities[0] + >>> dean_alt.kind.value, dean_alt.alternative.kind.value + ('title-or-name', 'title') + >>> parse("Grace Smith").ambiguities + () + +Four fields hold this vocabulary: + +- ``titles_ambiguous`` — the titles read only inside a phrase. A + word listed in ``titles`` as well is a title wherever it stands, so + ``add(titles={"dean"})`` (the first example on this page) is how to + read one as a title everywhere. +- ``title_modifiers`` — the words that qualify a head title + (``associate``, ``vice``). +- ``title_styles`` — the forms of address a title may stand behind + (``majesty``, ``highness``, ``excellency``). +- ``title_phrases`` — the two-word phrases, each read only where + both words are titles. Removing one reads its words apart: + +.. doctest:: + + >>> parse("Chief Justice John Roberts").title + 'Chief Justice' + >>> chief = Lexicon.default().remove(title_phrases={"chief justice"}) + >>> Parser(lexicon=chief).parse("Chief Justice John Roberts").title + 'Chief' Particles that are also given names ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -588,13 +665,10 @@ listed below. - Effect * - ``name_order`` - one of the three exported order constants - - Assigns positional (no-comma) input to given/middle/family in - this order. Use the exported ``GIVEN_FIRST`` (default), - ``FAMILY_FIRST``, or ``FAMILY_FIRST_GIVEN_LAST`` constants. - Ignored when a comma separates family from given ("Thomas, - John" puts the family name first); a comma that only sets off - suffixes ("John Smith, Jr.") leaves it governing the name part. - See :ref:`name-order`. + - Assigns positional input to given/middle/family in this order: + ``GIVEN_FIRST`` (default), ``FAMILY_FIRST`` or + ``FAMILY_FIRST_GIVEN_LAST``. A family comma overrides it. See + :ref:`name-order`. * - ``script_orders`` - pairs of ``Script`` and an order - Assigns a name in one of these scripts in the paired order, @@ -713,6 +787,14 @@ family-then-given regardless of the configured order: >>> family_first.parse("Thomas, John").family 'Thomas' +A comma that only sets off suffixes leaves the order governing the +name before it: + +.. doctest:: + + >>> family_first.parse("Smith John, Jr.").given + 'John' + A Vietnamese full name needs a third order. It is written family, then middle, then given — the name a person is actually called by is the *last* word, not the second. Family-first order gets the family name @@ -1175,9 +1257,16 @@ instead of extending them, the same trap as ``capitalization_exceptions``: Suffixes not separated by commas ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A semicolon separates post-nominals as a comma does, with no -configuration. Glued to the word before it, it stays in the suffix as -written: +A comma is the usual separator between post-nominals, but sources write +others. A few are read with no configuration, any other can be +declared, and a credential joined to another, or split by a space, +needs neither. + +Separators read without configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A semicolon separates post-nominals as a comma does. Glued to the word +before it, it stays in the suffix as written: .. doctest:: @@ -1185,8 +1274,9 @@ written: >>> name.given, name.family, name.suffix ('Jane', 'Smith', 'RN; CRNA') -So does a hyphen or a slash standing alone after the comma (since 2.7), -which is dropped as a comma between suffixes would be: +So does a hyphen or a slash standing alone, with a space on each side, +after a whole name and its comma or after a further comma (since 2.7). +It is dropped, as a comma between suffixes would be: .. doctest:: @@ -1197,19 +1287,21 @@ which is dropped as a comma between suffixes would be: 'Jane' Glued to a word, a hyphen or slash is the word's own, so -``Jane Smith, RN- CRNA`` reads as it always has, with ``RN-`` as the given -name and ``Jane Smith`` as the family; declare ``" - "`` to read the word -without its glued hyphen, as the semicolon is read. Behind a single word (``Smith, RN - CRNA``) -nothing is split, and before the comma it is a word too, so it never reaches a -hyphenated surname written with spaces (``Smith - Jones``). +``Jane Smith, RN- CRNA`` reads as it always has, with ``RN-`` as the +given name and ``Jane Smith`` as the family. Behind a single word +(``Smith, RN - CRNA``) nothing is split, and before the comma the +hyphen is a word too, so it never reaches a hyphenated surname written +with spaces (``Smith - Jones``). + +Declaring another separator +^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``extra_suffix_delimiters`` handles sources that separate post-nominals -with something else. Undeclared, the separator is a word. -Since 2.4 such a name still parses when a credential opens the part -after the comma, because that makes the whole part the suffix, but the -separator stays in the suffix as written. Declared, it splits the -suffix into groups the way a comma does, and it counts glued to a word -as the semicolon does: +with something else. Undeclared, the separator is a word. Since 2.4 +such a name still parses when a credential opens the part after the +comma, because that makes the whole part the suffix, but the separator +stays in the suffix as written. Declared, it splits the suffix into +groups the way a comma does: .. doctest:: @@ -1221,11 +1313,18 @@ as the semicolon does: >>> name.given, name.family, name.suffix ('Jane', 'Smith', 'RN, CRNA') +A declared separator also counts glued to a word, as the semicolon +does, and stays in the suffix as written there: declaring ``" - "`` +reads ``Jane Smith, RN- CRNA`` as given ``Jane``, family ``Smith``. + Through 2.3 the undeclared reading of ``Jane Smith, RN - CRNA`` was given ``RN``, family ``Jane Smith`` and suffix ``CRNA``, so the delimiter was the only way to get the name right, and through 2.6 it was the only way to split the suffix. +Credentials written joined or split +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + A hyphen or a slash written between two credentials with no space needs no declaring when every part is a listed credential that is not also a name, because the word then reads as one credential (since 2.7). A part @@ -1242,6 +1341,15 @@ which :ref:`all-caps-credentials` reads on its own terms: >>> joined.given, joined.family ('MA/PhD', 'Jane Smith') +A credential written with a space inside it, such as ``Pharm D``, is +read as one credential too (since 2.7), where the name has words to +spare in front of it: + +.. doctest:: + + >>> parse("John Smith, Pharm D").suffix + 'Pharm D' + .. _unlisted-credentials: Credentials the vocabulary doesn't list diff --git a/docs/design/decisions/H5-trailing-title-run.md b/docs/design/decisions/H5-trailing-title-run.md index aa8243c8b..04402b50a 100644 --- a/docs/design/decisions/H5-trailing-title-run.md +++ b/docs/design/decisions/H5-trailing-title-run.md @@ -10,7 +10,7 @@ - Transparency holds for one name word where a title run stands in front of it (`Sir John Prof.` → given 'John'); with none in front, the trailing title decides that word's field as H1 reads it (`Smith Prof.` → family 'Smith', `Smith Sir.` → given 'Smith') — 2026-09-09 (review round 2). - On a main segment the particle chain stops in front of a trailing title (`John van der Berg Prof.` → title 'Prof.', family 'van der Berg'), and a title a connective joined into a longer unit is no single word (`John Smith Prof. and Dr.` → family 'Prof. and Dr.') — 2026-10-07 #614. - After a family comma the given part's chain stops where `tail_reading`'s run begins, and P6 reads the particle as though the title were absent: `Smith, John van der Berg Prof.` → middle 'van der Berg', `Smith, Jan van Prof.` → family 'van Smith' — 2026-10-10 #623. -- Where that bound and the given part's own walk disagree behind a title, garbage shapes read off it, recorded and not fixed (`Smith, John Mc Prof. VI` → suffix 'Mc', middle 'Prof. VI') — 2026-10-10 #623. +- Where that bound and the given part's own walk disagree behind a title, garbage shapes read off it, recorded and not fixed (`Smith, John Mc Prof. V, Jr.` → suffix 'Mc, Jr.', middle 'Prof. V'; the entry's `Smith, John Mc Prof. VI` has read title 'Prof.', suffix 'VI' since #784) — 2026-10-10 #623. Elsewhere: a maiden clause in front of a trailing title is no join that hides it (`Mary Smith née Jones Prof.` → title 'Prof.'), decisions/M2's 2026-09-26 #535 entry; a bare ambiguous acronym ending the given part after a family comma is the credential, so `Doe, John Prof. MA` reads title 'Prof.', suffix 'MA', decisions/S5's 2026-09-18 #531 entry on the comma report's reach; the bound join after a family comma declines a title word this run takes (`Berg, abdul Prof.` → title 'Prof.'), decisions/P5's 2026-10-11 #637 entry; a part past the second comma makes no particle chain, so `John Smith, Jr., van der Berg Prof.` reads title 'Prof.', decisions/C7's 2026-10-11 #636 entry; a report names the field its word lands in, closing the `Dr. John of Prince` detail mismatch, decisions/A1's 2026-10-08 #626 entry; the cost of a period behind an ordinary surname that is title vocabulary (`Mary Jane King.` → title 'King.', family 'Jane') is the trailing-position bullet of decisions/vocabulary-exclusions. diff --git a/docs/design/decisions/generational-words.md b/docs/design/decisions/generational-words.md index f050de945..f61f561d5 100644 --- a/docs/design/decisions/generational-words.md +++ b/docs/design/decisions/generational-words.md @@ -18,7 +18,7 @@ Elsewhere: the run start that lets any unambiguous suffix word of two or more le - pt `júnior`, the accented spelling of `junior`, ships beside it with `same_as = "jr"` and inherits its vetting and its accepted collisions rather than being re-vetted under the two tests above (Junior is a borne surname, 2,146 US Census bearers, where `senior` was refused for 3,864): `José Silva Júnior` reads family 'Silva', suffix 'Júnior', as `José Silva Junior` does — 2026-10-23 (Derek), #843. - The Brazilian agnomens `Filho`, `Neto`, `Netto`, `Sobrinho` and their feminines are generation readings of the `agnomen` kind, gated by the count alone (rules/S11): `Pedro Alves Filho` reads family 'Alves', suffix 'Filho', and `Ana Neto` keeps family 'Neto', both reported — 2026-10-25 #752. -Open: #684 (locale packs for `Jiří Dienstbier ml.` and `st.`, the generational words that collide outside their language); #843 (the agnomens' abbreviations FH, FO, NT and SB). +Open: #684 (locale packs for `Jiří Dienstbier ml.` and `st.`, the generational words that collide outside their language). The agnomens' abbreviations (#843) are decided in decisions/S11 (2026-10-27). ## Log diff --git a/docs/design/rules/C2-parts-past-the-second-comma.md b/docs/design/rules/C2-parts-past-the-second-comma.md index db71b75dc..3b6090b8c 100644 --- a/docs/design/rules/C2-parts-past-the-second-comma.md +++ b/docs/design/rules/C2-parts-past-the-second-comma.md @@ -2,7 +2,7 @@ **When:** a name holds a part past the second comma, as in "John Smith, MD, Bart". -**Then:** parts beyond the second are consumed as suffixes either way, except that a word of the title vocabulary there that is not also suffix vocabulary is a title. +**Then:** parts beyond the second are consumed as suffixes either way, except that a word of the title vocabulary there that is not also suffix vocabulary is a title — a word that may also be a given name (rules/H9's `titles_ambiguous`) is not title vocabulary here outside a title phrase, so bare it stays a suffix, while a phrase is a title. - **A part past the second comma holds no name word**, so the surnames H5 keeps a bare trailing title word from misreading are not there to protect, and a piece holding a suffix word, a title/suffix dual among them, stays the postnominal the slot makes it (#603). ``` diff --git a/docs/design/rules/P5-bound-given-names.md b/docs/design/rules/P5-bound-given-names.md index ffb2d7309..311df0dbb 100644 --- a/docs/design/rules/P5-bound-given-names.md +++ b/docs/design/rules/P5-bound-given-names.md @@ -97,7 +97,7 @@ "Sir abdul salam" family-first → family="" "Sir abdul salam" family-first-given-last → given="abdul salam" ``` -- The given part after a family comma reads the words behind a title by its own walk, and where that walk reads them otherwise than the comma-less name does, the join is declined and no title is read — a shape-only numeral (VI to X), or V before a third comma part, stays a name word there, and a title word that is also a particle is attached by P6 (`Berg, abdul Prof. VI` reads middle 'Prof. VI'; `Berg, abdul St.` reads family 'St. Berg'), each as the same name with an ordinary given word reads. +- The given part after a family comma reads the words behind a title by its own walk, and where that walk reads them otherwise than the comma-less name does, the join is declined and no title is read — V before a third comma part stays a name word there, and a title word that is also a particle is attached by P6 (`Berg, abdul Prof. V, Jr.` reads middle 'Prof. V'; `Berg, abdul St.` reads family 'St. Berg'), each as the same name with an ordinary given word reads. --- **Signal:** neighbor: a bound given-name word and the name word after it, with a word to spare diff --git a/docs/design/rules/S-suffixes-generational-and-credentials.md b/docs/design/rules/S-suffixes-generational-and-credentials.md index 7a5eb21a1..6560e242e 100644 --- a/docs/design/rules/S-suffixes-generational-and-credentials.md +++ b/docs/design/rules/S-suffixes-generational-and-credentials.md @@ -1,4 +1,4 @@ ## Suffixes: generational & credentials (S) -Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. The suffix sets match one written word at a time: a multi-word entry there can never match anything and is warned about at configuration. Only two sets are exempt from that rule -- given_name_titles and, since #434, maiden_markers -- and neither is a suffix set, so within the suffix vocabulary the one-word rule is absolute. That limit is on STORAGE, not on the shape a name may have: adjacent suffix tokens are reassembled after matching (`_vocab.is_wholly_suffix`), so a multi-word credential is reachable as its component words -- `John Smith, MD PhD` has read suffix `MD PhD` since 1.4.0 -- and a caller reaches an unshipped one by adding the words it is made of rather than the phrase (#433). The eight multi-word entries that shipped dead for years span the suffix sets and the titles alike and are the Excluded story in decisions/vocabulary-exclusions. CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library currently reports both in one `suffix` field, a merge #326 examines. The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. +Background: what follows a name is one of two different things — generational suffixes (Jr., III), which attach to the name itself, and credentials (PhD, MD, MBA), which are earned attachments. The suffix sets match one written word at a time: a multi-word entry there can never match anything and is warned about at configuration. Only three sets are exempt from that rule -- given_name_titles, maiden_markers since #434, and title_phrases since #894 -- and none is a suffix set, so within the suffix vocabulary the one-word rule is absolute. That limit is on STORAGE, not on the shape a name may have: adjacent suffix tokens are reassembled after matching (`_vocab.is_wholly_suffix`), so a multi-word credential is reachable as its component words -- `John Smith, MD PhD` has read suffix `MD PhD` since 1.4.0 -- and a caller reaches an unshipped one by adding the words it is made of rather than the phrase (#433). The eight multi-word entries that shipped dead for years span the suffix sets and the titles alike and are the Excluded story in decisions/vocabulary-exclusions. CLDR personNames keeps them as separate fields (`generation`, `credentials`) and formats them differently; this library currently reports both in one `suffix` field, a merge #326 examines. The vocabulary is largely split already: a generational word list and a credential acronym list, plus a short list of acronyms that are also ordinary names (MA, BA) and so are AMBIGUOUS as bare words. diff --git a/docs/design/rules/S8-credentials-trailing-run.md b/docs/design/rules/S8-credentials-trailing-run.md index a1d0283f0..ee1e779c5 100644 --- a/docs/design/rules/S8-credentials-trailing-run.md +++ b/docs/design/rules/S8-credentials-trailing-run.md @@ -2,7 +2,7 @@ **When:** a credential stands after the name core with more words behind it in its part, as in "John Smith PhD Jones". -**Then:** a credential after the name core starts a run to the end of its part (#602): every later word of that part reads as a suffix, except a title word, which reads as a title. +**Then:** a credential after the name core starts a run to the end of its part (#602): every later word of that part reads as a suffix, except a title word, which reads as a title; a word that may also be a given name (rules/H9) is a title word here only inside a title phrase. - **Where the run starts.** An unambiguous suffix word of two or more letters, in any script, standing after two name words where no comma divides the name or before a suffix comma, or after the given word in the part after a family comma, makes every later word of that part a suffix. So does a roman numeral of two or more letters that reads as the generation (S10), whether the vocabulary lists it or its shape alone reads it, and so does a listed member of the ambiguous class its writing settles as the credential, written in capitals in a name of mixed case (S6) — after the given word in the part after a family comma wherever it stands, and elsewhere not where a name word stands behind it: a particle (a particle run or a joined surname counting by its first word), a word in a script with no case, or a word in Title case or in capitals that is no credential, which makes the capitals as likely the writer's own initials (`Maria Ana MA Silva`), or a family name written in capitals, as a credential. ``` diff --git a/docs/migrate.rst b/docs/migrate.rst index 8470fe22b..3c2097c3c 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -61,7 +61,8 @@ about either: naming it, since 2.4 * a ``Lexicon``, ``Policy`` or ``Parser`` pickled by 2.5 or earlier raises ``ValueError`` on load since 2.6, each having gained a field; - re-pickle after upgrading + re-pickle after upgrading. A ``Lexicon``, ``Parser`` or locale pack + pickled by 2.6 raises the same way since 2.7 A silent change: comparing with ``==`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -669,7 +670,8 @@ grep your fixtures for, with what 1.4 read: by its last one (``Her Majesty Queen``, ``Rev Sir``) and added titles that address that way (``Prince``, ``Princess``, ``Swami``, ``Guru`` and others). Other titles still leave the name the family name - (``"Dr Harry"``). + (``"Dr Harry"``). Since 2.7 a bare ``Prince`` or ``Princess`` is a + given name (below); behind a style, as here, it is still a title. - ``"Dr King Jr"`` (2.3) — 1.4 read title ``Dr King``, last ``Jr``. - ``"John Smith Prof."`` (2.3) — a trailing period-marked title: 1.4 read last ``Prof.``, middle ``Smith``. @@ -738,12 +740,6 @@ grep your fixtures for, with what 1.4 read: ``Júnior``. - ``"John Smith, MD; PhD"`` (2.6) — a semicolon between post-nominals reads as a comma: 1.4 read first ``MD;``, last ``John Smith``. -- ``"Jane Smith, Dr. / MD"`` (2.7) — a hyphen or slash standing alone - between post-nominals reads as a comma: 1.4 read first ``/``, last - ``Jane Smith``. -- ``"Marie Bon Durand"`` and ``"Herren Hans"`` (2.7) — ``Bon``, ``Dí`` - and ``Heer`` are no longer particles, nor ``Herren`` and ``Heren`` - titles: 1.4 read last ``Bon Durand`` and title ``Herren``. - ``"Mr Md Karim Uddin"`` and ``"Wm Smith"`` (2.6) — ``Md`` and ``Wm`` are the given names they abbreviate: 1.4 read title ``Mr Md`` and ``Wm``. @@ -754,6 +750,46 @@ grep your fixtures for, with what 1.4 read: read first ``Dr.``. - ``capitalize()`` on ``"j.r.r. tolkien"`` (2.6) — 1.4 wrote ``J.r.r.``; repair now capitalizes after a full stop or an ayn. +- ``"King Curtis"``, ``"Prince Fielder"``, ``"Sultan Khan"`` and + ``"Sheikh Ahmed Khan"`` (2.7) — a title that is also a given name is a + title only inside a title phrase or behind a style (``"His Majesty + King Henry"``), so a bare one opening the name is the first name: 1.4 + read title ``King``, ``Prince``, ``Sultan`` and ``Sheikh``. The same + holds for ``Princess``, ``Sultana``, ``Shaikh``, ``Sheik``, ``Shekh``, + ``Shaik``, ``Cheikh``, ``Mahdi``, ``Bard``, ``Buddha``, ``Siddha``, + ``Gyani`` and ``Wing``, and for a lone qualifier such as ``Grand``. + ``Constants().titles`` still lists these words, so adding one there + changes nothing; the 2.0 API's ``Lexicon.default().add(titles={"king"})`` + makes one a title everywhere. +- ``"Chief Justice John Roberts"`` and ``"Major General John Smith"`` + (2.7) — a title phrase is one title: 1.4 read title ``Chief``, first + ``Justice``, and first ``Major``, middle ``General John``. +- ``"Chas. Smith"`` and ``"Ma. Cristina Reyes"`` (2.7) — a given name + written short is the first name: 1.4 read title ``Chas.`` and ``Ma.``. +- ``"John Smith VI MD"`` and ``"Smith, John VI"`` (2.7) — the numerals + ``VI`` to ``X`` are suffixes as ``III`` is: 1.4 read last ``VI``, + suffix ``MD``, and middle ``VI``. +- ``"John Smith, MD/PhD"`` and ``"John Smith, Pharm D"`` (2.7) — a + credential joined by a hyphen or slash, or split by a space, after a + comma is a suffix: 1.4 read first ``MD/PhD``, and first ``Pharm``, + middle ``D``, each with last ``John Smith``. +- ``"John Smith Ph. D. Jones"`` (2.7) — 1.4 read last ``Jones``, suffix + ``Ph. D.``. +- ``"Smith, John MA Jones"`` (2.7) — a listed credential in capitals in + a mixed-case name starts the suffix: 1.4 read middle ``Jones``, suffix + ``MA``. +- ``"Pedro Alves Filho"`` and ``"Alexandre Dumas fils"`` (2.7) — the + Brazilian agnomens and ``fils`` are suffixes where the name has a word + to spare: 1.4 read middle ``Alves`` and ``Dumas``, last ``Filho`` and + ``fils``. +- ``"Andrew Perkins (JD)"`` (2.7) — a bracketed credential its writing + settles is a suffix: 1.4 read nickname ``JD``. +- ``"Jane Smith, Dr. / MD"`` (2.7) — a hyphen or slash standing alone + between post-nominals reads as a comma: 1.4 read first ``/``, last + ``Jane Smith``. +- ``"Marie Bon Durand"`` and ``"Herren Hans"`` (2.7) — ``Bon``, ``Dí`` + and ``Heer`` are no longer particles, nor ``Herren`` and ``Heren`` + titles: 1.4 read last ``Bon Durand`` and title ``Herren``. .. doctest:: @@ -842,8 +878,6 @@ grep your fixtures for, with what 1.4 read: 'Júnior' >>> HumanName("John Smith, MD; PhD").suffix 'MD; PhD' - >>> HumanName("Jane Smith, Dr. / MD").first - 'Jane' >>> md_karim = HumanName("Mr Md Karim Uddin") >>> md_karim.title, md_karim.first ('Mr', 'Md') @@ -857,3 +891,43 @@ grep your fixtures for, with what 1.4 read: >>> tolkien.capitalize() >>> str(tolkien) 'J.R.R. Tolkien' + >>> [HumanName(n).first for n in ("King Curtis", "Prince Fielder", + ... "Princess Diana", "Sultan Khan", "Sultana Begum", + ... "Sheikh Ahmed Khan", "Shaikh Ahmed Khan", "Sheik Ahmed", + ... "Shekh Ahmed", "Shaik Ahmed", "Cheikh Anta Diop", + ... "Mahdi Karimi", "Bard Larsen", "Buddha Bahadur Tamang", + ... "Siddha Raj Joshi", "Gyani Maya Gurung", "Wing Lau")] + ... # doctest: +NORMALIZE_WHITESPACE + ['King', 'Prince', 'Princess', 'Sultan', 'Sultana', 'Sheikh', 'Shaikh', + 'Sheik', 'Shekh', 'Shaik', 'Cheikh', 'Mahdi', 'Bard', 'Buddha', + 'Siddha', 'Gyani', 'Wing'] + >>> HumanName("His Majesty King Henry").title + 'His Majesty King' + >>> HumanName("Chief Justice John Roberts").title + 'Chief Justice' + >>> HumanName("Major General John Smith").title + 'Major General' + >>> HumanName("Chas. Smith").first, HumanName("Ma. Cristina Reyes").first + ('Chas.', 'Ma.') + >>> HumanName("John Smith VI MD").suffix, HumanName("Smith, John VI").suffix + ('VI MD', 'VI') + >>> HumanName("John Smith, MD/PhD").suffix + 'MD/PhD' + >>> HumanName("John Smith, Pharm D").suffix + 'Pharm D' + >>> HumanName("John Smith Ph. D. Jones").suffix + 'Ph. D. Jones' + >>> HumanName("Smith, John MA Jones").suffix + 'MA Jones' + >>> HumanName("Pedro Alves Filho").suffix + 'Filho' + >>> HumanName("Alexandre Dumas fils").last + 'Dumas' + >>> HumanName("Andrew Perkins (JD)").suffix + 'JD' + >>> HumanName("Jane Smith, Dr. / MD").first + 'Jane' + >>> HumanName("Marie Bon Durand").last + 'Durand' + >>> HumanName("Herren Hans").first + 'Herren' diff --git a/docs/release_log.rst b/docs/release_log.rst index cbedc3010..0e805bc97 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -1,36 +1,34 @@ Release Log =========== -* 2.7.0 - Unreleased +* 2.7.0 - October 27, 2026 - nameparser 2.7 is under development. + **nameparser 2.7** is the first release working towards the new language and vocabulary model described in the `vocabulary model RFC `__. It introduces more nuanced handling of titles and particles, adjusts the default vocabulary to better align with example name corpora, and surfaces alternative name readings in the published API. Highlights: + + - **Titles that are also given names read as titles only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, given ``John``, where 2.6 gave title ``Chief``, given ``Justice``; outside a phrase the word is a name, so ``Justice Smith`` gives given ``Justice``. + - **Credentials joined by a hyphen or a slash, or split by a space, read as one credential.** ``John Smith, MD/PhD`` gives suffix ``MD/PhD``, where 2.6 gave given ``MD/PhD``, family ``John Smith``. + - **Given names written short read as given names.** ``Chas. Smith`` and ``Ma. Cristina Reyes`` give the given name ``Chas.`` and ``Ma.``, where 2.6 read both as titles. + - **An ambiguity report names the reading the parse did not take.** ``Ambiguity.alternative`` says what the word would have been and why. **Additions** - - **Add Ambiguity.alternative, the reading a report's parse did not take.** ``parse("JOHN SMITH MA").ambiguities[0].alternative`` is ``Reading('name' as 'family', 'declared')``: read as a name, ``MA`` would be the family name, a surname the vocabulary lists; ``John Smith PhD Jones`` gives source ``position`` for ``Jones``, a name only by where it stands. The new ``Reading``, ``ReadingKind`` and ``ReadingSource`` are exported from ``nameparser``. The field is not compared, so names and their reports compare as before, and a ``ParsedName`` pickled by 2.6 or earlier still loads, its reports with no alternative. :issue:`752` + - **Add Ambiguity.alternative, the reading a report's parse did not take.** ``parse("JOHN SMITH MA").ambiguities[0].alternative`` is ``Reading('name' as 'family', 'declared')``: read as a name, ``MA`` would be the family name. ``Reading``, ``ReadingKind`` and ``ReadingSource`` are exported from ``nameparser``; the field is not compared, and a ``ParsedName`` pickled by 2.6 still loads. :issue:`752` **Behavior Changes** - - **Fix roman numerals VI to X reading as names where III reads as a suffix.** ``John Smith VI MD`` gives suffix ``VI MD`` and ``Smith, John VI`` suffix ``VI``, as ``III`` does, where 1.4.0 through 2.6.0 gave last ``VI``, suffix ``MD`` and middle ``VI``. ``John Smith V, M.J.`` keeps suffix ``V`` and ``Georg VI.`` gives suffix ``VI.``, where 1.4.0 through 2.6.0 gave last ``John Smith V`` and ``VI.``. A one-letter numeral before a credential is still a name (``John Smith V PhD`` keeps last ``V``), and a Title-case ``Vi`` in a mixed-case name reads as a name (``Nguyen Van Vi`` gives last ``Van Vi``, where 2.2.0 through 2.6.0 gave suffix ``Vi``). :issue:`784`, :issue:`774` + - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, given ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, given ``Justice``. Outside a phrase the word is a name, and ``Dean Smith`` now reports a ``title-or-name`` ambiguity. ``Lexicon.default().add(titles={"dean"})`` still makes a word a title everywhere; for ``King`` and the other words 2.6.0 listed as titles, a v1 ``Constants`` cannot. :issue:`752`, :issue:`853` + - **Fix given names that are also titles, such as King, Prince and Sheikh, reading as titles.** ``King Curtis`` gives given ``King``, family ``Curtis``, where 1.4.0 through 2.6.0 gave title ``King``; ``Prince``, ``Princess``, ``Sultan``, ``Sultana``, ``Sheikh`` and its spellings, ``Mahdi``, ``Bard``, ``Wing``, ``Buddha``, ``Siddha`` and ``Gyani`` read the same, and so does a lone ``Vice`` or ``Grand`` opening the name. A style or a title phrase keeps the title (``His Majesty King Henry``, ``Prince of Wales``), and after a full name and a comma a bare given name among them reverses the name, as ``Dean`` does (``John Smith, Wing`` gives given ``Wing``). :issue:`752`, :issue:`348`, :issue:`853`, :issue:`867`, :issue:`869` + - **Read Chas., Wm. and the other given names written short as given names rather than titles.** ``Chas. Smith`` and ``Ma. Cristina Reyes`` give the given name ``Chas.`` and ``Ma.``, where 1.4.0 through 2.6.0 gave title ``Chas.`` and ``Ma.``. An unlisted abbreviation in front is still a title (``Insp. Jane Morse``); ``Lexicon.given_name_abbreviations`` lists the words. :issue:`789`, :issue:`752` + - **Fix Bon, Dí and Heer joining the next word into the family name, and Herren and Heren reading as titles.** ``Marie Bon Durand`` gives middle ``Bon``, family ``Durand``, where 1.4.0 through 2.6.0 gave family ``Bon Durand``, and ``Herren Hans`` gives given ``Herren``, where they gave title ``Herren``. :issue:`766` + - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives given ``John``, family ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave given ``MD/PhD``, family ``John Smith``. Each part must be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; an unlisted one in capitals after a comma behind a mixed-case name reads as a credential too (``John Smith, PU-PH``), and so can a hyphenated name written in capitals (``KIM, AE-RA`` gives suffix ``AE-RA``). :issue:`752`, :issue:`797` + - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives given ``John``, family ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave given ``Pharm``, middle ``D``, family ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives given ``Pharm``. :issue:`752` + - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones``, and ``John Smith MA`` no longer reports a ``suffix-or-name`` ambiguity. A name written in one case still reports (``JOHN SMITH MA``). :issue:`785`, :issue:`752` + - **Read a bracketed credential that is also a name as a suffix where its writing settles it.** ``Andrew Perkins (JD)`` gives suffix ``JD``, where 1.4.0 through 2.6.0 gave nickname ``JD``, and ``John Smith (MA)`` gives suffix ``MA`` again, as 1.4.0 did. Only capitals in a mixed-case name or a period inside the word settle it, so ``John Smith (Ed)`` and ``JOHN SMITH (MA)`` keep the nickname and report a ``suffix-or-nickname`` ambiguity. :issue:`810`, :issue:`752` + - **Read a hyphen or a slash standing alone between post-nominals as a comma.** ``Jane Smith, Dr. / MD`` gives given ``Jane``, family ``Smith``, where 1.4.0 through 2.6.0 gave given ``/``, family ``Jane Smith``. It counts only after a full name and its comma, or after a second comma, and only with a space on each side, so ``Smith - Jones, John`` and ``John Smith, MD- PhD`` read as before. :issue:`889`, :issue:`156` + - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` gives family ``Alves``, suffix ``Filho``, where 1.4.0 through 2.6.0 gave middle ``Alves``, family ``Filho``; ``fils``, ``Neto``, ``Sobrinho``, their feminines and the abbreviations Brazilian filings write for them (``FH``, ``NT``) read the same. Each is a surname too, so the parse reports a ``suffix-or-name`` ambiguity, and a name of two words keeps it as the family name (``Ana Neto``). :issue:`843`, :issue:`752` + - **Fix roman numerals VI to X reading as names where III reads as a suffix.** ``John Smith VI MD`` gives suffix ``VI MD`` and ``Smith, John VI`` suffix ``VI``, as ``III`` does, where 1.4.0 through 2.6.0 gave family ``VI`` and middle ``VI``. A one-letter numeral before a credential is still a name (``John Smith V PhD``), and so is a Title-case ``Vi`` in a mixed-case name (``Nguyen Van Vi`` gives family ``Van Vi``, where 2.2.0 through 2.6.0 gave suffix ``Vi``). :issue:`784`, :issue:`774` - **Fix a lone letter before a further comma reading as part of a credential.** ``Smith, John PhD I, Jr.`` gives middle ``I``, suffix ``PhD, Jr.``, as 1.4.0 through 2.3.0 did, where 2.4.0 through 2.6.0 gave suffix ``PhD I, Jr.``. :issue:`819` - - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals reads as a credential too after a comma behind two or more name words written in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; a hyphenated name written in Title case keeps its reading even where both parts are listed credentials (``Kim, Ae-Ra``), though its capitals spelling does not (``KIM, AE-RA`` gives suffix ``AE-RA``). :issue:`752`, :issue:`797` - - **Read a credential written with a space inside it, such as Pharm D, as one credential.** ``John Smith, Pharm D`` gives first ``John``, last ``Smith``, suffix ``Pharm D``, where 1.4.0 through 2.6.0 gave first ``Pharm``, middle ``D``, last ``John Smith``. It counts only where the name has words to spare, so ``Smith, Pharm D`` still gives first ``Pharm``, and one opening with a single letter is still two words (``John Smith, M. D.`` still gives first ``M.``). :issue:`752` - - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave last ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives last ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` - - **Read the Brazilian agnomens and French fils as suffixes where the name has a word to spare.** ``Pedro Alves Filho`` and ``Alexandre Dumas fils`` give last ``Alves`` and ``Dumas``, suffix ``Filho`` and ``fils``, where 1.4.0 through 2.6.0 gave middle ``Alves`` and ``Dumas``, last ``Filho`` and ``fils``; ``Neto``, ``Netto``, ``Sobrinho`` and the feminines ``Filha``, ``Neta``, ``Sobrinha`` read the same way, and so do the abbreviations Brazilian filings write for them (``JOSE DA SILVA FH`` gives last ``DA SILVA``, suffix ``FH``, where 1.4.0 through 2.6.0 gave last ``DA SILVA FH``). Each word is a surname too, and its abbreviations are read the same way, so a name of two words keeps it as the last name (``Ana Neto``, ``PEDRO FILHO``) and ``Fils`` in Title case in a mixed-case name stays the last name (``Marie Fils``), the parse reports the choice as a ``suffix-or-name`` ambiguity. ``Lexicon.suffix_words_ambiguous`` lists the words and ``Lexicon.agnomens`` the ones read by the word count alone. :issue:`843`, :issue:`752` - - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752` - - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). A v1 ``Constants().titles`` still lists ``King``, ``Mahdi`` and the other titles 2.6.0 listed, and removing one reads it as a name everywhere, but adding one there no longer makes it a title outside a phrase; that takes the ``Lexicon`` spelling. :issue:`752`, :issue:`853` - - **Fix the name Sheikh and its spellings reading as titles.** ``Sheikh Ahmed Khan``, ``Mr Sheikh Ahmed`` and ``Cheikh Anta Diop`` give first ``Sheikh`` and ``Cheikh``, where 1.4.0 through 2.6.0 gave title ``Sheikh``, ``Mr Sheikh`` and ``Cheikh``; ``Khan, Sheikh`` and the spellings ``Sheik``, ``Shekh`` and ``Shaik`` read the same. A ruler written with the bare title reads it as a first name too (``Sheikh Mohammed bin Rashid`` gives first ``Sheikh``), where a style keeps it (``His Highness Sheikh Mohammed``); ``Shaykh`` is still a title. :issue:`752`, :issue:`853` - - **Fix the given names Prince and Princess reading as titles.** ``Prince Fielder`` and ``Mr Prince Smith`` give first ``Prince``, where 1.4.0 through 2.6.0 gave title ``Prince`` and ``Mr Prince``; ``Smith, Princess`` reads the same. A royal written with the bare title reads it as a first name too (``Prince Harry`` gives first ``Prince``, last ``Harry``), where a style keeps it (``His Royal Highness Prince Harry``), and so does ``of`` (``Prince of Wales``). :issue:`752`, :issue:`348`, :issue:`853` - - **Fix the name Shaikh reading as a title.** ``Shaikh Ahmed Khan`` and ``Mr Shaikh Ahmed`` give first ``Shaikh``, where 1.4.0 through 2.6.0 gave title ``Shaikh`` and ``Mr Shaikh``; ``Khan, Shaikh`` reads the same. A ruler written with the bare title reads it as a first name too (``Shaikh Hamad bin Isa`` gives first ``Shaikh``), where a style keeps it (``His Highness Shaikh Hamad``); the spellings ``Sheikh`` and ``Shekh`` are still titles. :issue:`752`, :issue:`853` - - **Fix the given names Sultan and Sultana reading as titles.** ``Sultan Khan`` and ``Sultana Begum`` give first ``Sultan`` and ``Sultana``, where 1.4.0 through 2.6.0 gave title ``Sultan`` and ``Sultana`` with no first name; ``Mr Sultan Khan`` and ``Khan, Sultan`` read the same. A sultan written with the bare title reads it as a first name too (``Sultan Hassanal Bolkiah`` gives first ``Sultan``), where a style keeps it (``His Majesty Sultan Hassanal Bolkiah``). :issue:`752`, :issue:`853` - - **Fix the given name King reading as a title.** ``King Curtis`` and ``Mr King Smith`` give first ``King``, where 1.4.0 through 2.6.0 gave title ``King`` and ``Mr King``; ``SMITH, KING`` reads the same. A monarch written with the bare title reads it as a first name too (``King Henry`` gives first ``King``, last ``Henry``), where a style keeps it (``His Majesty King Henry``). :issue:`752`, :issue:`348` - - **Fix the Nepali given names Buddha, Siddha and Gyani reading as titles.** ``Buddha Bahadur Tamang`` and ``Gyani Maya Gurung`` give first ``Buddha`` and ``Gyani``, middle ``Bahadur`` and ``Maya``, where 1.4.0 through 2.6.0 gave title ``Buddha`` and ``Gyani``, first ``Bahadur`` and ``Maya``; ``Siddha Raj Joshi`` and ``Tamang, Buddha Bahadur`` read the same. ``Guru`` is still a title. The Sikh title written ``Gyani`` reads as a given name too (``Gyani Zail Singh``); its usual spelling ``Giani`` is still a title. :issue:`869`, :issue:`752` - - **Fix the given names Mahdi and Bard reading as titles.** ``Mahdi Karimi`` and ``Bard Larsen`` give first ``Mahdi`` and ``Bard``, where 1.4.0 through 2.6.0 gave title ``Mahdi`` and ``Bard`` with no first name; ``Mr Mahdi Karimi`` and ``Karimi, Mahdi`` read the same. Inside a title phrase either is still a title (``Mahdi of Sudan``), but the epithet with its article is not: ``Muhammad Ahmad, the Mahdi`` gives first ``the Mahdi``, last ``Muhammad Ahmad``, where 1.4.0 through 2.6.0 gave title ``the Mahdi``. :issue:`867`, :issue:`752` - - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``), except ``Wing``, which is also a given name: ``Lau, Wing Kai`` gives first ``Wing``, middle ``Kai``, where 1.4.0 through 2.6.0 gave title ``Wing``, first ``Kai``. ``John Smith Wing.`` gives last ``Wing.``, as 1.4.0 through 2.2.0 did, where 2.3.0 through 2.6.0 gave title ``Wing.``. Like ``Dean``, it reverses a full name before a comma (``John Smith, Wing`` gives first ``Wing``, last ``John Smith``). :issue:`752` - - **Read Chas., Wm. and the other given names written short as given names rather than titles.** ``Chas. Smith`` and ``Ma. Cristina Reyes`` give first ``Chas.`` and ``Ma.``, where 1.4.0 through 2.6.0 gave title ``Chas.`` and ``Ma.``; ``Thos.``, ``Jas.``, ``Geo.``, ``Robt.``, ``Benj.``, ``Jno.``, ``Saml.``, ``Edw.``, ``Richd.``, ``Wm.`` and the Filipino ``Jo.`` read the same way, behind a title too (``Mrs Ma. Liza Cruz`` gives title ``Mrs``, first ``Ma.``). An unlisted abbreviation in front is still a title (``Insp. Jane Morse``), and ``Lexicon.given_name_abbreviations`` lists the words. :issue:`789`, :issue:`752` - - **Read a bracketed credential that is also a name as a suffix where its writing settles it.** ``John Smith (MA)`` and ``John Smith (M.A)`` give suffix ``MA`` and ``M.A``, as 1.4.0 did, where 2.0.0 through 2.6.0 gave nickname ``MA`` and ``M.A``; ``Andrew Perkins (JD)`` gives suffix ``JD``, where 1.4.0 through 2.6.0 gave nickname ``JD``. Only capitals in a mixed-case name or a period inside the word settle it, and only after the name, so ``John Smith (Ed)``, ``JOHN SMITH (MA)`` and ``Jeffrey (JD) Bricken`` keep the nickname and report a ``suffix-or-nickname`` ambiguity. :issue:`810`, :issue:`752` - - **Read a hyphen or a slash standing alone between post-nominals as a comma.** ``Jane Smith, Dr. / MD`` gives first ``Jane``, last ``Smith``, where 1.4.0 through 2.6.0 gave first ``/``, last ``Jane Smith``; ``John Smith, MD - PhD`` gives suffix ``MD, PhD``, where 1.4.0 through 2.5.0 gave first ``-``, last ``John Smith`` and 2.6.0 gave suffix ``MD - PhD``. It counts only after a whole name and its comma, or after a second comma, and only with a space on each side, so ``Smith - Jones, John`` and ``John Smith, MD- PhD`` read as before; ``extra_suffix_delimiters`` still declares any other separator. :issue:`889`, :issue:`156` - - **Fix Bon, Dí and Heer joining the next word into the last name, and Herren and Heren reading as titles.** ``Marie Bon Durand`` gives middle ``Bon``, last ``Durand``, where 1.4.0 through 2.6.0 gave last ``Bon Durand``; ``Li Dí Wang`` and ``Jan Heer Smit`` read the same. ``Herren Hans`` gives first ``Herren``, where 1.4.0 through 2.6.0 gave title ``Herren``; ``Heren Jan`` reads the same, and ``Doe, Jan Heren`` gives middle ``Heren`` again, where 2.6.0 gave title ``Heren``. :issue:`766` - - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``, ``given_name_abbreviations``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` + - **Fix a split Ph. D. in the middle of a name stopping short of the words after it.** ``John Smith Ph. D. Jones`` gives suffix ``Ph. D. Jones``, as ``John Smith PhD Jones`` does, where 1.4.0 through 2.6.0 gave family ``Jones``, suffix ``Ph. D.``; ``John Smith V Ph. D.`` gives family ``V``, as ``John Smith V PhD`` does, where 1.4.0 through 2.6.0 read ``V`` as a suffix. :issue:`827` + - **A Lexicon, Parser or locale pack pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``, ``given_name_abbreviations``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` * 2.6.0 - October 25, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 31de55721..c703d990d 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1023,6 +1023,9 @@ so branching on a kind needs no import: >>> [t.text for t in name.ambiguities[0].tokens] ['Van'] +Most names report none. A non-empty ``ambiguities`` is a useful signal +for routing a record to human review instead of trusting it silently. + .. _credentials-also-surnames: Credentials that are also surnames @@ -1081,8 +1084,24 @@ nothing — periods make ``M.A.`` unambiguously a credential: >>> parse("John Smith M.A.").ambiguities () -Most names report none. A non-empty ``ambiguities`` is a useful signal -for routing a record to human review instead of trusting it silently. +Titles that are also given names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A title that is also borne as a given name, such as ``Dean``, ``King`` +or ``Major``, is a title only inside a title phrase (since 2.7), and +otherwise a name word. Where it opens the name and a phrase could have +held it, the choice is recorded as ``title-or-name``: + +.. doctest:: + + >>> parse("Dean of Students Jane Smith").title + 'Dean of Students' + >>> dean_smith = parse("Dean Smith") + >>> dean_smith.given, [a.kind.value for a in dean_smith.ambiguities] + ('Dean', ['title-or-name']) + +:ref:`titles-also-given-names` covers the phrases and how to read such +a word as a title everywhere. .. _reading-not-taken: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index e6e680056..1713b1088 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -580,12 +580,14 @@ class Lexicon: Entries are normalized at construction -- lowercased, NFC-composed, edge full stops stripped -- so matching is case-insensitive. Vocabulary entries are single words -- a multi-word entry warns at - construction and can never match. Two fields are exempt, and they + construction and can never match. Three fields are exempt, and they differ in HOW they match: ``given_name_titles`` is looked up against the run of words the parse has ALREADY read as titles -- - the whole run space-joined, or that run's last word -- while + the whole run space-joined, or that run's last word -- ``maiden_markers`` is matched by lookahead, longest first, over - words that need not be markers on their own (``"z domu"``). + words that need not be markers on their own (``"z domu"``), and + ``title_phrases`` holds two-word phrases, stored space-joined and + matched against two adjacent title words (``"chief justice"``). Field docs below show examples, not full contents; inspect any field's shipped vocabulary directly, e.g. ``Lexicon.default().conjunctions``.""" @@ -768,8 +770,9 @@ class Lexicon: title_modifiers: frozenset[str] = frozenset() #: Honorific styles ("hon", "excellency", "highness"): one standing #: before a word of ``titles_ambiguous`` puts that word in a title - #: phrase, as in "The Hon. Justice Smith" (rules/H9). Entries not - #: also in ``titles`` are never consulted. Full default list: + #: phrase, as in "The Hon. Justice Smith" (rules/H9). Entries in + #: neither ``titles`` nor ``titles_ambiguous`` are never consulted. + #: Full default list: #: :data:`~nameparser.config.titles.TITLE_STYLES`. title_styles: frozenset[str] = frozenset() #: Two-word title phrases, a title and the title it qualifies diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index ea188f9fc..dbcd65356 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -13,7 +13,9 @@ overrides it when every name piece is written wholly in one script, or in the Han/Hiragana/Katakana repertoire the #272 kana license shares across pieces); token/piece tags; Lexicon only through tags already -applied by classify (plus the leading-title period rule). +applied by classify (plus the leading-title period rule), and +Lexicon.given_name_titles, which an ambiguity's alternative asks for +rules/P5's given-name-title licence. Implements rules H2, H4, H5, H6, H7, H8, N3, O4, O5, S10, W4, and P6 after a family comma, of docs/design/rules/, diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 25a931e17..2dfb7c491 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -14,8 +14,8 @@ the one thing here that a bucket alone does not decide. Reads: Policy.nickname_delimiters, Policy.maiden_delimiters, Lexicon.maiden_markers, and -Lexicon.suffix_words / suffix_acronyms / suffix_acronyms_ambiguous -through _suffix_shaped. +Lexicon.suffix_words / suffix_words_ambiguous / suffix_acronyms / +suffix_acronyms_ambiguous through _suffix_shaped. Implements rules N1, N2, S1, M1 and M3 of docs/design/rules/ (the #273 matching mechanism); each is cited at its code below. One scan diff --git a/nameparser/_types.py b/nameparser/_types.py index 49ae4c3c8..24df77e9c 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -613,8 +613,8 @@ class AmbiguityKind(StrEnum): #: One name word that is a JOIN carrying title vocabulary reports #: this kind too ("John of Bishop"): the unit is read as a name, #: and whether the title word inside it is a title is the fork. - #: Not the title-vs-given-name collision on a word like "Baron", - #: which is a question about the VOCABULARY and is not this kind. + #: Not a word listed only as a title ("Baron"), whose collision + #: with a name is a question about the VOCABULARY and not this kind. #: A title that may also be a given name ("Dean", rules/H9) reports #: it too, where it opens the name outside a title phrase and no #: comma has marked the given name: "Dean Smith" gives given "Dean". diff --git a/nameparser/_version.py b/nameparser/_version.py index 331594dc1..7c788e9e2 100644 --- a/nameparser/_version.py +++ b/nameparser/_version.py @@ -15,5 +15,5 @@ #: "dev" through the cycle, cleared at release. It normalizes to #: 2.7.0.dev0, which sorts above 2.6.0 and BELOW 2.7.0, so an install #: from master can never masquerade as the release it precedes. -PRE_RELEASE = "dev" +PRE_RELEASE = "" __version__ = ".".join(map(str, VERSION)) + PRE_RELEASE diff --git a/nameparser/config/titles.py b/nameparser/config/titles.py index 49652a587..c1133ce17 100644 --- a/nameparser/config/titles.py +++ b/nameparser/config/titles.py @@ -3,7 +3,9 @@ # kind. tools/vocab/build.py compiles them into _compiled.py and runs the # checks that were import-time asserts here: a title reading with # addresses = "given" is what puts a word in GIVEN_NAME_TITLES, so the -# subset relation holds by construction (decisions/vocabulary-model), as +# subset relation with the title readings holds by construction -- the +# readings, not the TITLES constant, since a word that may also be a +# given name compiles to TITLES_AMBIGUOUS instead (decisions/vocabulary-model), as # a salutation reading with after_given = true does for AFTER_GIVEN_TITLES. from nameparser.config import _compiled @@ -52,7 +54,8 @@ """ Honorific styles ("hon", "excellency", "highness"): one standing before a title that may be a given name makes it part of a title phrase, as in -"The Hon. Justice Smith". A subset of TITLES. +"The Hon. Justice Smith". Each is in TITLES, or in TITLES_AMBIGUOUS +where it may be a given name too ("grace", "sheikh"). """ TITLE_PHRASES = _compiled.TITLE_PHRASES diff --git a/tests/v2/cases.py b/tests/v2/cases.py index c6d2fb423..3edcc679e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -10635,8 +10635,9 @@ def _check_cjk_shape_purity(self) -> None: # ranks (Lord Peter, Lady Diana) and title or surname for every # peer and every wife (Lord Byron, Lady Thatcher), and the set # has no way to say "sometimes" -- the reason `venerable` stayed - # out. Since #752 phase 2j both are titles only inside a title - # phrase (rules/H9), so the prince rows write the style in front. + # out. Since #752 phase 2j prince and princess are titles only + # inside a title phrase (rules/H9), so these rows write the style + # in front. Case("title_run_princess_addresses_by_given_name", "Her Royal Highness Princess Anne", {"title": "Her Royal Highness Princess", "given": "Anne"}, diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 469adc3e8..e981140e0 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -2,7 +2,7 @@ "A1": {"words": 195, "then_words": 171, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 32}, "A2": {"words": 60, "then_words": 31, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "C1": {"words": 454, "then_words": 202, "because_sentences": 1, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 116}, - "C2": {"words": 492, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 1, "accepted_words": 81}, + "C2": {"words": 525, "then_words": 432, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 1, "accepted_words": 81}, "C3": {"words": 260, "then_words": 228, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 173}, "C4": {"words": 529, "then_words": 393, "because_sentences": 1, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, "C5": {"words": 319, "then_words": 79, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 1, "accepted_words": 54}, @@ -44,7 +44,7 @@ "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, "P3": {"words": 639, "then_words": 598, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "P4": {"words": 190, "then_words": 82, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "P5": {"words": 717, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 3, "accepted_words": 198}, + "P5": {"words": 717, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 3, "accepted_words": 192}, "P6": {"words": 1086, "then_words": 703, "because_sentences": 2, "except": 2, "precedence": 5, "accepted": 6, "accepted_words": 459}, "P7": {"words": 138, "then_words": 64, "because_sentences": 3, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 28}, "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, @@ -61,7 +61,7 @@ "S5": {"words": 731, "then_words": 400, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 585, "then_words": 330, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S7": {"words": 602, "then_words": 388, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 105}, - "S8": {"words": 551, "then_words": 386, "because_sentences": 1, "except": 1, "precedence": 2, "accepted": 1, "accepted_words": 63}, + "S8": {"words": 571, "then_words": 406, "because_sentences": 1, "except": 1, "precedence": 2, "accepted": 1, "accepted_words": 63}, "S9": {"words": 463, "then_words": 262, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S10": {"words": 665, "then_words": 246, "because_sentences": 3, "except": 4, "precedence": 3, "accepted": 3, "accepted_words": 163}, "S11": {"words": 511, "then_words": 298, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 107},