diff --git a/AGENTS.md b/AGENTS.md index 73e4a0bae..f4102a668 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -200,6 +200,7 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a word reading on a word with a name reading, `fils` and the agnomens, gated by the count, rules/S11, since 2026-10-25) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) +- `given_names.py` — `GIVEN_NAME_ABBREVIATIONS` (#789, a `given` reading carrying `same_as`, the full name: "chas", "wm", Filipino "ma"), the given names written short that rules/H2's shape yields to; no v1 `Constants` attribute, and nothing else reads it - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set with no v1 `Constants` attribute of its own, so the only v1 knob that reaches it is deleting the conjunction - `maiden_markers.py` — `MAIDEN_MARKERS` (`birth_marker` readings, e.g. "née", "geb.") routing the following name to `maiden`; with an `addresses = "given"` title, the one kind a phrase may be (`z domu`) - `surnames.py` — `KOREAN_SURNAMES`, the census list the 2.0 API splits unspaced hangul on (#271), still written in Python with an import-time `assert_normalized`: a segmentation list, outside the per-word model, like the locale packs' lists. With `maiden_markers.py` it is one of the two data modules `Constants` has **no** attribute for: both reach the parse only through `Constants._snapshot()` → `Lexicon`, so the v1 surface stays frozen and there is no v1 knob to turn either off (the opt-out is the 2.0 `Policy`) @@ -244,7 +245,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). -- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), `suffix_words_ambiguous = SUFFIX_WORDS_AMBIGUOUS ∩ suffix_words` with `agnomens` inside it (#752, likewise: deleting the suffix word turns S11's gate off, `test_snapshot_removing_a_suffix_word_turns_its_gate_off`), `title_modifiers`, `title_styles` and `title_phrases` intersected with the titles (#752 phase 2i, rules/H9, likewise), `titles_ambiguous` passed whole, its words being in no v1 set, except where the caller emptied `titles`, v1's way of turning title detection off, which drops it too (`test_snapshot_cleared_titles_turn_the_title_phrase_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. +- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), `suffix_words_ambiguous = SUFFIX_WORDS_AMBIGUOUS ∩ suffix_words` with `agnomens` inside it (#752, likewise: deleting the suffix word turns S11's gate off, `test_snapshot_removing_a_suffix_word_turns_its_gate_off`), `title_modifiers`, `title_styles` and `title_phrases` intersected with the titles (#752 phase 2i, rules/H9, likewise), `titles_ambiguous` passed whole, its words being in no v1 set, except where the caller emptied `titles`, v1's way of turning title detection off, which drops it too (`test_snapshot_cleared_titles_turn_the_title_phrase_off`), `given_name_abbreviations` passed whole (#789: a word a v1 caller adds to `titles` is still a title, the leading peel asking the titles before the shape), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. - **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard. @@ -274,7 +275,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. @@ -296,7 +297,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it via opt-in `Constants`. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. -**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") +**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. The one vocabulary it yields to, since #789 (phase 2k of #752), is a given name written short (`Lexicon.given_name_abbreviations`, a `given` reading with `same_as` in the TOML): `"Chas. Smith"` → `given='Chas.'`, read off classify's `vocab:given-abbreviation` tag in `is_leading_title` and off the lexicon in `_vocab.is_title_shaped`, the two spellings of one test that `test_is_title_shaped_and_is_leading_title_agree` holds together. The second spelling is what rules/C4's comma count and the title phrase's head run ask, so a change to the shape moves comma names nobody wrote it for: #789's first grid held no comma and missed `Chas. Smith, Ed` (given 'Ed', family 'Chas. Smith' before, suffix 'Ed' after), which only mutating `is_title_shaped`'s half alone showed. Put a word of the ambiguous class after a comma behind the shaped word (`X. Smith, Ed`), and a modifier in front of it (`Vice X. Smith, Ed`), in the grid of any change to the shape, and mutate each spelling separately. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") **Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) @@ -348,7 +349,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` and `sa` were duals too until #853 took `md`'s title reading out, a leading `Md` being the given name Muhammad, and #857 took `sa` out of the vocabulary) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` and `sa` were duals too until #853 took `md`'s title reading out, a leading `Md` being the given name Muhammad, and #857 took `sa` out of the vocabulary) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), save a listed title and, since #789, a given name written short (`"Chas. Smith"` → `given`), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/design/decisions/A1-ambiguity-report.md b/docs/design/decisions/A1-ambiguity-report.md index 03ccf2bba..098f2bad0 100644 --- a/docs/design/decisions/A1-ambiguity-report.md +++ b/docs/design/decisions/A1-ambiguity-report.md @@ -5,7 +5,7 @@ - A report whose fork the rest of the parse resolved to neither branch is withdrawn in assemble, the one place that sees final roles: a connective-or-initial letter roled suffix or title loses that report and keeps its own, so `JOHN QUINCY SMITH I` reports `suffix-or-name` alone — 2026-09-20 #397. - The decision stays at its emitter and assemble words the field from the word's final role (`PendingAmbiguity.field_tail`), so a report names the field its word lands in whatever rule moved it: `, Ma Dr.` reports 'Ma' "read as a family name", and `Kim Min Do` under FAMILY_FIRST reports 'Do' read as a family name; a placeholder in the detail was declined — 2026-10-08 #626. - A report on a part past the second comma names no field: `John Smith, Jr., Dr. Bart` reports "segment 'Dr. Bart' beyond the recognized comma structures; read best-effort"; naming every field the part's words land in was declined — 2026-10-10 #629. -- The listed-member report after a family comma names the role the comma binds, worded at the emitter: `John, Ma.` reads title 'Ma.' and reports "read as a title" — 2026-10-21 #793. +- The listed-member report after a family comma names the role the comma binds, worded at the emitter: `John, Si.` reads title 'Si.' and reports "read as a title" (its example was `John, Ma.` until decisions/H2 made `Ma.` there a given name written short) — 2026-10-21 #793. Elsewhere: P8's "connects nothing", withdrawn once over final roles in assemble, is decisions/S10's 2026-10-18 #722's /simplify pass entry. diff --git a/docs/design/decisions/H2-leading-abbreviation-title.md b/docs/design/decisions/H2-leading-abbreviation-title.md index 177081dc1..643648e40 100644 --- a/docs/design/decisions/H2-leading-abbreviation-title.md +++ b/docs/design/decisions/H2-leading-abbreviation-title.md @@ -5,7 +5,8 @@ - An abbreviation of two or more letters ending in its one period, opening the part that carries the given name (the whole name, or the part after a family comma), is a title even unlisted and even where it is suffix vocabulary, and a single-letter initial is not: `Insp. Jane Morse` → title 'Insp.', `Morse, Det. Insp. Jane` → title 'Det. Insp.', `Esq. Smith` → title 'Esq.', `J. Smith` → given 'J.' — 2026-06-30 #288. - The shape is an unbroken run of letters, so an unlisted abugida abbreviation stays a name word and only vocabulary recognizes it — 2026-06-30 #288. - The shape declines a period-marked word carrying a script with no initials: `田中. 太郎` → family '田中.', where `Smith. John` keeps title 'Smith.' — 2026-09-10. -- C13 wins for a credential opening the part after a family comma (`Smith, Esq. John` → suffix 'Esq. John'), and the shape still reads a title there for a word that starts no S8 run (`Smith, Insp. John`, `Smith, Ma. John` → title) — 2026-10-20 #761. +- C13 wins for a credential opening the part after a family comma (`Smith, Esq. John` → suffix 'Esq. John'), and the shape still reads a title there for a word that starts no S8 run (`Smith, Insp. John`, `Smith, Do. John` → title) — 2026-10-20 #761; its `Smith, Ma. John` example moved to the given name on 2026-10-26 #789. +- A given name written short outranks the shape: a word the vocabulary lists with a `given` reading whose `same_as` is the full name, period-marked where the shape would fire, is the given name and the name reads as it would with the full name there (`Chas. Smith` → given 'Chas.', `Mrs Ma. Liza Cruz` → title 'Mrs', given 'Ma.'). The shipped set is the traditional English abbreviations and the Filipino `ma` and `jo`; a listed title still wins, and the reading acts nowhere else — 2026-10-26 #789. Elsewhere: #316's question 2, a symmetric leading rule reading `Esq. Smith` as a suffix, is declined and the trailing slot reads vocabulary, decisions/H5's 2026-09-08 #316 entry (the undated `Decided 2026-09-08` paragraph below records the hook it closed here); the shape does not fire before a family comma, decisions/H6's 2026-10-17 #701 and 2026-10-18 #710 entries; a letter typed decomposed is the same letter to the shape, decisions/nfd-shapes' 2026-10-12 #652 entry, and the script veto reads the composed spelling, its 2026-10-13 #665 entry; halfwidth katakana is classified since #594 (decisions/W4). @@ -22,3 +23,8 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse - 2026-09-10 — the shape declines a word carrying a script with no initials (#323); the argument, the halfwidth-kana limit and the measurements are in #cjk-full-stops. That limit was closed on 2026-10-03 by #594, which classifies halfwidth katakana (decisions/W4). - 2026-10-20 (Derek), #761 — C13 WINS OVER H2 FOR A CREDENTIAL OPENING THE PART AFTER A FAMILY COMMA. The H section's skeleton migration (PR #779) found the two rules each claiming `Smith, Esq. John`: rules/H2 made an abbreviation opening the part after a family comma a title "even when the word is suffix vocabulary", with an exception only for a part that is nothing but suffix words (C8), while rules/C13 makes a credential opening that part the postnominal part however it goes on. The parser already read it C13's way, suffix 'Esq. John' with `suffix-or-name` under all three orders, and neither rule said so. Derek decided C13 wins, with no behavior change; rules/H2 gains the Precedence bullet and its Then names the exception. What the win covers was measured over the suffix vocabulary, each alphabetic word of two or more letters written title-case with a period in `Smith, John` (2026-10-20, py3.11, the tree at #779): 666 words read as the postnominal part, and 22 still read as a title by H2's shape — the 12 title/suffix duals (`Cpl.`), 7 ambiguous acronyms (`Ma.`, `Do.`) and `Abd.`, `Mc.` and `Vd.` — none of them a word that starts S8's run. So the bullet is scoped to C13's own words rather than to every listed suffix word, which Derek confirmed once shown the 22 (keeping the parse unchanged), and `Smith, Ma. John` is an example beside `Smith, Esq. John` and the unlisted `Smith, Insp. John`. + +- 2026-10-26 #789 (#752, phase 2k) — A GIVEN NAME WRITTEN SHORT OUTRANKS THE SHAPE. The traditional English abbreviations of given names (`Chas.` Charles, `Thos.` Thomas, `Jas.` James, `Geo.` George, `Robt.` Robert, `Benj.` Benjamin, `Jno.` John, `Saml.` Samuel, `Edw.` Edward, `Richd.` Richard, `Wm.` William) and the Filipino `Ma.` (María) and `Jo.` (José) opened the part carrying the given name as titles by the shape, from 1.4.0 through 2.6.0 (checked against the 1.4.0, 2.0.0, 2.3.0 and 2.6.0 wheels: `Chas. Smith` → title 'Chas.', `Ma. Cristina Reyes` → title 'Ma.', given 'Cristina'), and a reader of either would not hesitate: each is the given name. Agreed by Derek on 2026-10-23 as part of #752's design, decisions/vocabulary-model's entry of that date. Each word takes a `given` reading whose `same_as` is the full name, which compiles it into `GIVEN_NAME_ABBREVIATIONS` (`Lexicon.given_name_abbreviations`), and the shape declines a word in it: a vocabulary reading outranks a reading a shape rule derived, the precedence the vocabulary model allows a language pack and applies here in the default. The precedence is the shape's alone. A listed title is asked before the shape and still wins, so a caller who lists `chas` as a title gets the 1.4 reading back; and nothing else reads the set, so the bare word (`Ma Smith`, `Smith, John Ma`, `John Smith MA`) and the dotted word anywhere but the head of the given-name part (`John Smith Ma.` → family 'Ma.', `John Ma. Smith` → middle 'Ma.') read as they did, `ma` keeping its surname and credential readings. + - **What the name reads as.** Wherever the shape read the abbreviation as a title, the name now reads, field for field, as it does with the plain given name `Jack` in the abbreviation's place, its case matched, and wherever the shape read none, as it did: `test_a_given_name_written_short_reads_as_a_given_name` (tests/v2/test_properties.py) holds that for each shipped word in its three period-marked spellings, in the 20 of its 21 slots where the shape or the comma's count reads one, and holds the three bare spellings to the set-emptied reading in all 21, under each of the three orders. One reading is not the full name's: rules/C6's case contrast skips a period-marked word, so `Chas. SMITH, XYZ` reverses (given 'XYZ', family 'Chas. SMITH') as `J. SMITH, XYZ` does and as it did before, where `Charles SMITH, XYZ` reads suffix 'XYZ'; left as it is, the contrast's question being C6's (found by the second review). So under a family-first order `Chas. Smith` reads family 'Chas.', given 'Smith', as `Jack Smith` does: the declared order places the word, which the shape had kept out of every name field. The comma moves too, since rules/C4 counts the name words before a comma by asking the shape: `Chas. Smith, Ed` reads given 'Chas.', suffix 'Ed', as `John Smith, Ed` does, where `Chas.` had counted as a title and the name reversed (given 'Ed', family 'Chas. Smith'). The bare spellings read as they did in every slot of that test, a capitals one behind a comma included (`John Smith, JAS` keeps suffix 'JAS'): its first version had put the set in the wordlist rosters the capitals shape and the name-contrast test read, which reversed that name and `Jas SMITH, XYZ` until the code review found it. + - **Measured** against master 12384fae (2026-10-26), with `snapshot.py` for the committed snapshot, `cldr_agreement.py` for the agreement views, `snapshot.py --local --base ` for ctgov-officials, fec-individuals and tse-donors, and `local_agreement.py psc-individuals --base `: the committed snapshot moves 2 names, `Ma.` opening the name and rules/H2's own example `Smith, Ma. John`, now given 'Ma.'; the CLDR, GND and Wikidata agreement views do not move, and the conventions view gains its one `Ma.` rendering's title. ctgov-officials moves 15 of 536,981 officials, each `Ma.` opening the name in mixed case; fec-individuals moves 24 of 1,893,964 names, each written in capitals after a family comma (`WM.` 20, `MA.` 2, `GEO.` 1, `EDW.` 1); tse-donors moves 2 of 854,531 pairs, `MA.` opening the name. psc-individuals, scored against the fields its filers typed, has 86 records fixed, 0 broken and 2 changed, all 88 `Ma.` opening the forename box: 81 behind a salutation filed in the title box (#853's shape, `Mrs Ma. Liza …`), 7 with no title. The spike of 2026-10-24 counted the same. + - **Vetting**, recorded on each word in en.toml and fil.toml: `tools/name_usage.py` counts each as a given name in US SSA data (Chas 3,223, Wm 2,736, Geo 1,324, Robt 468, Thos 166, Jas 124, Jno 11, Edw 6 births, summed over the years; `Benj`, `Saml` and `Richd` under the source's floor), and none as a credential in NPPES. `Jas` is also a surname (117 bearers in the 2010 US Census) and `Jo` a given name of its own (180,911 US births) and a Korean surname, and `Ma` a Chinese surname and a listed credential; a period-marked word opening the given part is a given name on each of these readings, and the credential reading had never taken it there, the shape reading a title first. `tools/vocab/query.py --word` found no other reading on any of the thirteen but `ma`'s and `wm`'s refused title. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 37328d55f..f6fc77885 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -33,13 +33,14 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762. - The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749. - The titles take ten kinds beside the generic one, `academic` and `occupation` divided by function rather than subject, and the title keep-outs are exclusions — 2026-10-19 #750, title kinds; 2026-10-19 #750, the rest of the titles. -- `wm` is no title: William's abbreviation reads as the given name it is (`Wm Smith` → given 'Wm'), and its title reading is refused in en.toml. The dotted `Wm.` opening the given part is still rules/H2's title; a given reading that outranks H2 belongs to phase 2 (#752) — 2026-10-22 #789. +- `wm` is no title: William's abbreviation reads as the given name it is (`Wm Smith` → given 'Wm'), and its title reading is refused in en.toml — 2026-10-22 #789. The dotted `Wm.` opening the given part reads given too, by the reading below — 2026-10-26 #789. +- A given name written short is a `given` reading whose `same_as` is the full name, normalized and naming no entry, a `family` reading taking none; the word compiles into `GIVEN_NAME_ABBREVIATIONS` (`Lexicon.given_name_abbreviations`), which outranks rules/H2's shape and nothing else (decisions/H2). The traditional English abbreviations in en.toml (`chas`, `wm`) and the Filipino `ma` and `jo` in fil.toml carry one — 2026-10-23 #752; 2026-10-26 #789, phase 2k. - Abbreviations and full words spelled alike across European languages (`dr`, `prof`, `doctor`, `professor`, `docent`) are in `mul.toml`; a credential goes to a file only on evidence that names in that language carry it, so none is in `mul.toml` — 2026-10-19 #750, the rest of the titles; 2026-10-20 #751, the credentials. - `und.toml` holds only `bon` and `dí`, particles of unrecorded origin; every other word, the credentials, conjunctions, bound given names and birth markers included, is in a language file or in `mul.toml` — 2026-10-19 #749; 2026-10-20 #751, the conjunctions. Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment. -Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR, are the 2026-10-23 #752 entry's and the RFC's), #789 (sub-phase 2k: the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given over rules/H2's shape, decided 2026-10-23 #752), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). +Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR, are the 2026-10-23 #752 entry's and the RFC's), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). ## Log @@ -218,3 +219,8 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **2026-10-26 #752, phase 2i (Derek) — `attaches` MAY NAME A TITLE WORD AS WELL AS A KIND, AND `grace` AND `marshal` JOIN THE CONTESTED TITLES.** Amends the `attaches` bullet of the 2026-10-25 #752, phase 2i entry: a kind reaches every word that later joins it, and once `marshal` became a contested military word the military kind made `Sgt Marshal Smith` a title phrase. A list entry that is no kind must be a word with a title reading (a build check, its mutation in tests/v2/test_vocab.py). `major` attaches to `general` and `sergeant` and `sgt` to `major`. `grace` (style) and `marshal` (military, its #606 exclusion replaced) carry dated `given` readings. The measurement and Derek's answers are decisions/H9's entry of this date. - **2026-10-26 #752, phase 2j, `wing` — THE FIRST TITLE BATCH: `wing` HAS A DATED `given` READING.** The order is the 2026-10-24 #752 (Derek) entry's. Decision 10 had already read `wing` as a name at the head of the name, so the reading reaches the parts decision 10 left as listed, after a comma and ending the name: `Lau, Wing Kai` reads given 'Wing', and `John Smith Wing.` family 'Wing.'. The rule and the measurement are decisions/H9's entry of this date. - **2026-10-26 (Derek), #785 — PHASE 2g NARROWED: a capitals member with a name word behind it, and no family comma, does not start the credential's run**, the name word being evidence of initials; `Maria Ana MA Silva`, `Jean Pierre MA DUPONT` and `Mary Jane MA King` keep the readings 1.4.0 through 2.6.0 gave, and `John Smith MA Jones` keeping family 'Jones' is the cost. The record is decisions/S6's entry of this date. +- **2026-10-26 #789, #752, phase 2k — A GIVEN NAME WRITTEN SHORT IS A `given` READING WITH `same_as`, AND IT OUTRANKS rules/H2's SHAPE.** Carries out item 14 of the 2026-10-23 #752 entry above and the second half of #789; the first half, bare `wm` out of the titles, is the 2026-10-22 #789 entry. SUPERSEDES: in the README's property table, `same_as` as a property of every category but `name`. What changes in the vocabulary: + - A `given` reading may carry `same_as`, the full given name the word abbreviates (`chas` is `charles`). On a name reading it names a name, not an entry, so check 7 asks only that it be normalized and not the word itself, and a `family` reading is refused one. The reading compiles the word into a new set, `GIVEN_NAME_ABBREVIATIONS` (`Lexicon.given_name_abbreviations`, re-exported from `nameparser.config.given_names`), so an entry holding only such a reading is not the inert name reading the build warns of. No `Constants` attribute reaches it, and the shim passes it whole: a v1 caller who lists the word as a title has it read as one, the leading peel asking the title vocabulary before the shape. + - The set's one reader is rules/H2's shape test, which declines a word in it, so the vocabulary reading outranks the shape-derived one, decision 20 applied in the default. It is not a derived marker: no second vocabulary reading is weighed, only the shape's. `ma`, which already had given, family and credential readings in en.toml, takes its abbreviation reading in fil.toml, and the credential's ambiguous marker is unchanged, its name readings already deriving it. `jo` had no reading before, and its `given` reading derives nothing. + - The words, with their counts on each entry: `benj`, `chas`, `edw`, `geo`, `jas`, `jno`, `richd`, `robt`, `saml`, `thos` and `wm` in en.toml (the last keeping its refused title, whose reason now names this reading), `jo` and `ma` in fil.toml. What moved, the vetting and the grid are decisions/H2's entry of this date. Recompute the list: `uv run python -c "from nameparser import Lexicon; print(sorted(Lexicon.default().given_name_abbreviations))"`. + Adding the field changes `Lexicon`'s pickle layout, as phases 2h and 2i did. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index f1a2dd68a..c3986881e 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -453,7 +453,7 @@ A second axis cuts across the jobs: whether the names are real or invented, labe - Release notes describe intent; TAGS are ground truth. Before trusting a changelog cross-reference about what shipped where, run `git diff -- ` — the "1.2.1 pickle shim" claim recurred twice from a changelog line describing a thing that never shipped. - Never pipe a check's output when its exit code matters. Under zsh, `snapshot.py | tail` makes `$?` tail's status. Redirect to a file and read the file. - Mutation-test a new guard before believing it, and mutate the thing the guard watches — a survivor usually means the fixture satisfies the invariant for free. A guard over a HAND COPY of a pipeline set has two such holes, and #744's review found both in one test of tools/vocab/build.py's check 2, which copies the tokenizer's sets because the tool cannot import the package it writes. Pinning a copy to its source sees only the classes someone copied: the first test pinned `DIVIDERS` alone, and emoji, which the tokenizer strips under `Policy.strip_emoji`, had no copy and no check, so `dr★` compiled and the test passed. Its replacement swept every codepoint `_tokenize._ignorable` drops but accepted any refusal short of a TOML error, so the space in `dr x` passed because the phrase-title check refused it, and a class check 2 never learned would pass too wherever a later check happened to refuse its test text. Drive the sweep from the pipeline's own predicate, match the refusal by the checked rule's own messages, and record the control (hiding the emoji check behind its pinned constant fails the sweep: `test_every_character_the_tokenizer_ignores_is_refused` in tests/v2/test_vocab.py, which now leaves the space out, a phrase's word separator). The predicate reaches only what it decides, though: the interpunct between East Asian characters (rules/T3) is divided by context in `_tokenize_region`, never by `_ignorable`, so hiding that check passes the sweep (measured in #746's review, 2026-10-19). What holds it is its `MUTATIONS` row (`text-with-an-east-asian-interpunct`, input `先·生`); the `EAST_ASIAN_RANGES` and `INTERPUNCT` pins hold only the copies, and stay green with the check gone. Name such a class where the guard stands, and the test that holds it, rather than let the sweep imply it. -- A claim about what a vocabulary entry DOES is a claim about the parse WITHOUT it, so take that parse (`Parser(lexicon=Lexicon.default().remove(titles={...}))`) and compare: the parse with the entry present shows only that the output is consistent with it. #773's first note on `wm` said that removing it would move `Wm. Smith`, read off the parse with the word listed; with the word removed, `Wm. Smith` reads exactly as before, rules/H2's leading-abbreviation title claiming `Wm.` either way. The corrected note then said only the bare `Wm Smith` moves, and that was wrong too (#789, 2026-10-24): it took the off-switch parse at the head of the name only, and a trailing `Wm.` (rules/H5) and a `Wm.` before a family comma (rules/H6), which read vocabulary rather than shape, moved as well. A shape rule shadows an entry only in the spelling and the slot it reads, so take the off-switch parse for each spelling, bare and period-marked, in each position a rule reads the word in. +- A claim about what a vocabulary entry DOES is a claim about the parse WITHOUT it, so take that parse (`Parser(lexicon=Lexicon.default().remove(titles={...}))`) and compare: the parse with the entry present shows only that the output is consistent with it. #773's first note on `wm` said that removing it would move `Wm. Smith`, read off the parse with the word listed; with the word removed, `Wm. Smith` read exactly as before, rules/H2's leading-abbreviation title then claiming `Wm.` either way. The corrected note then said only the bare `Wm Smith` moves, and that was wrong too (#789, 2026-10-24): it took the off-switch parse at the head of the name only, and a trailing `Wm.` (rules/H5) and a `Wm.` before a family comma (rules/H6), which read vocabulary rather than shape, moved as well. A shape rule shadows an entry only in the spelling and the slot it reads, so take the off-switch parse for each spelling, bare and period-marked, in each position a rule reads the word in. - Verify the restore, not just the mutation: diff against the pre-mutation copy; do not trust the harness's own restore report. - Run all the gates, not the ones you remember: ruff runs before mypy and pytest in CI, and each has caught what the others passed. - Purge __pycache__ between same-length source mutations; stale bytecode makes a changed file measure as unchanged. diff --git a/docs/design/rules/H-titles-and-honorifics.md b/docs/design/rules/H-titles-and-honorifics.md index 45099f4b0..ca3a923e0 100644 --- a/docs/design/rules/H-titles-and-honorifics.md +++ b/docs/design/rules/H-titles-and-honorifics.md @@ -1,5 +1,5 @@ ## Titles & honorifics (H) -Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08); the one exception is a listed salutation ending the given part after a family comma, the comma having marked the surname's place (H8). Some title words need a partner to be titles at all: a title that is also borne as a given name, and a word that only qualifies a head title, are titles only inside a title phrase (H9), so "Dean Smith" and "Wing Lau" read given names where "Dean of Students" and "Wing Commander" read titles. An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break +Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary save a given name written short; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08); the one exception is a listed salutation ending the given part after a family comma, the comma having marked the surname's place (H8). Some title words need a partner to be titles at all: a title that is also borne as a given name, and a word that only qualifies a head title, are titles only inside a title phrase (H9), so "Dean Smith" and "Wing Lau" read given names where "Dean of Students" and "Wing Commander" read titles. An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break "Rabbi Cohen". diff --git a/docs/design/rules/H2-leading-abbreviation-title.md b/docs/design/rules/H2-leading-abbreviation-title.md index abc0a5473..c1c8aade1 100644 --- a/docs/design/rules/H2-leading-abbreviation-title.md +++ b/docs/design/rules/H2-leading-abbreviation-title.md @@ -2,7 +2,7 @@ **When:** a period-marked word opens the part of the name that carries the given name, as in "Rev. John Smith". -**Then:** an abbreviation opening the part of the name that carries the given name — the whole name, or the part after a family comma — reads as a title even when unlisted, provided it is an unbroken run of two or more letters ending in its one period, the shape outranking vocabulary where it fires, so that a period-marked opening word is a title even when the word is suffix vocabulary, save a credential opening the part after a family comma (C13). +**Then:** an abbreviation opening the part of the name that carries the given name — the whole name, or the part after a family comma — reads as a title even when unlisted, provided it is an unbroken run of two or more letters ending in its one period, the shape outranking vocabulary where it fires, so that a period-marked opening word is a title even when the word is suffix vocabulary, save a credential opening the part after a family comma (C13) and a given name written short. - **A single-letter initial does not** (T5), and neither does anything with interior periods, hyphens or digits. ``` @@ -36,7 +36,18 @@ ``` "Smith, Esq. John" → suffix="Esq. John" "Smith, Insp. John" → title="Insp." - "Smith, Ma. John" → title="Ma." + "Smith, Do. John" → title="Do." + ``` +- **A given name written short** ("Chas. Smith", "Ma. Cristina Reyes"): an abbreviation the vocabulary reads as a given name written short is that given name, outranking the shape — traditional English abbreviations such as Chas. and Wm., and the Filipino Ma. (María) and Jo. (José) — so the name reads as it would with the full given name in that place, behind a title too ("Mrs Ma. Liza Cruz"), and C4's count of the name words before a comma, which asks this shape, counts it as one; its period still keeps it out of the case contrast C6 reads, as an initial's does ("Chas. SMITH, XYZ" reads as "J. SMITH, XYZ"). Only the shape yields: a word listed as a title is still one, and written bare, or anywhere but the head of the part that carries the given name, the word reads as it would unlisted, so "Ma" keeps its surname and credential readings. + ``` + "Chas. Smith" → given="Chas." + "Smith, Chas." → given="Chas." + "Smith, Ma. John" → given="Ma." + "Mrs Ma. Liza Cruz" → given="Ma." + "Wm. H. Smith III" → given="Wm." + "Chas. Smith, Ed" → suffix="Ed" + "John Smith Ma." → family="Ma." + "Smith, John Ma" → middle="Ma" ``` **Accepted:** @@ -56,5 +67,5 @@ ``` --- -**Signal:** shape: an unlisted word of two or more letters ending in its one period, opening the given-name part -**history:** decisions/H2 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py +**Signal:** shape: an unlisted word of two or more letters ending in its one period, opening the given-name part, which a given name written short declines +**history:** decisions/H2 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/design/rules/H4-input-that-is-nothing-but-vocabulary-still-has-to-name-somebody.md b/docs/design/rules/H4-input-that-is-nothing-but-vocabulary-still-has-to-name-somebody.md index b9d819186..c05737905 100644 --- a/docs/design/rules/H4-input-that-is-nothing-but-vocabulary-still-has-to-name-somebody.md +++ b/docs/design/rules/H4-input-that-is-nothing-but-vocabulary-still-has-to-name-somebody.md @@ -35,10 +35,11 @@ **Precedence:** -- **A dotted lone credential** ("Jr."): `Jr.` is post-nominal vocabulary and still reads as a title, H2's opening-abbreviation shape outranking the vocabulary where it fires — so the suffix half never sees a dotted lone credential. +- **A dotted lone credential** ("Jr."): `Jr.` is post-nominal vocabulary and still reads as a title, H2's opening-abbreviation shape outranking the vocabulary where it fires — so the suffix half never sees a dotted lone credential. A credential that is also a given name written short is that given name instead, by H2's own precedence ("Ma."). ``` "Jr." → title="Jr." "Jr." → ambiguities=() + "Ma." → given="Ma." ``` **Accepted:** diff --git a/docs/design/rules/H9-title-phrase.md b/docs/design/rules/H9-title-phrase.md index 7f0874f05..b789989dd 100644 --- a/docs/design/rules/H9-title-phrase.md +++ b/docs/design/rules/H9-title-phrase.md @@ -63,7 +63,7 @@ ``` "Hon. Wing Lau" → given="Wing" ``` -- **A period-marked word at the front of the name** (H2): the abbreviation shape makes it a title whatever the vocabulary says, so the phrase test never reaches it and nothing is reported. +- **A period-marked word at the front of the name** (H2): the abbreviation shape makes it a title whatever the vocabulary says, save a given name written short, so the phrase test never reaches it and nothing is reported. ``` "Dean. Smith" → title="Dean." "Wing. Lau" → title="Wing." diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 6abae2caf..d69a73466 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -124,7 +124,7 @@ The front of a name has no words-to-spare cue: a title addresses the name after Phase 2i carried this out as [rules/H9](rules/H9-title-phrase.md) (the title phrase), with `of` after a modifier as a partner too (`Member of Parliament`) and a modifier in a connective's object (`President of the United States`); a title that may be a given name compiles out of `titles`, so one a caller lists there is a title everywhere. Which title words get a given reading is decided one word at a time (decision 16), in measured batches, and `Dean Smith` keeps reading given `Dean` (decision 13). A title that only ever stands in front stays a title (`lord`, `sir`, `lady`), and a word that opens names in one tradition but is a title class in another (`guru`, with its Devanagari twin) waits for a language pack. Where the rule reads a contested title as a name, the parse reports `title-or-name`. -**Abbreviated given names** use the same pool from the other side (#789): a `given` reading with `same_as` the full name (`chas` → Charles, Filipino `ma` → María) is to outrank the leading-abbreviation title rule's shape, so that `Ma. Cristina Santos Reyes` reads given `Ma.` where today it reads title `Ma.`. +**Abbreviated given names** use the same pool from the other side (#789): a `given` reading with `same_as` the full name (`chas` → Charles, Filipino `ma` → María) is to outrank the leading-abbreviation title rule's shape, so that `Ma. Cristina Santos Reyes` reads given `Ma.` where it read title `Ma.`. Phase 2k (2026-10-26) did this: the reading compiles into `GIVEN_NAME_ABBREVIATIONS`, which [rules/H2](rules/H2-leading-abbreviation-title.md)'s shape declines, and nothing else reads it. ### Brackets @@ -176,7 +176,7 @@ Consequences that follow from the definitions: - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. - **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages (a Portuguese particle, an English credential, a Vietnamese surname), so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work. This reverses the premise of #684, which proposes a French pack to read `fils` as a suffix: under decisions 6 and 15, a French pack keeps both of French's readings, and `Marie Fils` would still need the gate to keep her family name. -**A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). Shape rules (period abbreviation, initials, capitals) emit readings into the same pool as vocabulary from phase 2 on, rather than final answers, so a pack's preference can act on them. Phase 2 is to settle one such clash in the default, since every language loads there: Filipino `Ma.`, which abbreviates María, is to get a `given` reading in `fil.toml` that outranks the leading period-abbreviation rule (`Ma. Cristina Santos Reyes`, title `Ma.` today); a pack's part is to prefer such readings where the default leaves both standing. +**A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). Shape rules (period abbreviation, initials, capitals) emit readings into the same pool as vocabulary from phase 2 on, rather than final answers, so a pack's preference can act on them. Phase 2 is to settle one such clash in the default, since every language loads there: Filipino `Ma.`, which abbreviates María, is to get a `given` reading in `fil.toml` that outranks the leading period-abbreviation rule (`Ma. Cristina Santos Reyes`, title `Ma.` until phase 2k gave it that reading on 2026-10-26); a pack's part is to prefer such readings where the default leaves both standing. **What moves in phase 3:** the locale packs' vocabulary, still in Python today, becomes language files; `default = false`, given its meaning in phase 2, marks the readings a pack turns on, for words too risky even gated; and the reserved names `prefer` and `strict` gain their meaning. How a tag filters (does `pt` match `pt-BR`?) and how a third-party language file joins the pool and the snapshot are open under #753. diff --git a/docs/migrate.rst b/docs/migrate.rst index 97a4174fe..b7deab611 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -267,8 +267,9 @@ fields: Some ``Lexicon`` fields have no ``CONSTANTS`` attribute at all — ``conjunctions_ambiguous``, ``maiden_markers``, ``surnames``, -``honorific_tails``, ``after_given_titles``, ``suffix_words_ambiguous`` -and ``agnomens`` — so they are reached through the 2.0 API only. +``honorific_tails``, ``after_given_titles``, ``suffix_words_ambiguous``, +``agnomens`` and ``given_name_abbreviations`` — so they are reached +through the 2.0 API only. .. warning:: diff --git a/docs/modules.rst b/docs/modules.rst index 13a5c27cb..b864eb353 100644 --- a/docs/modules.rst +++ b/docs/modules.rst @@ -227,6 +227,8 @@ HumanName.config Defaults :members: .. automodule:: nameparser.config.bound_given_names :members: +.. automodule:: nameparser.config.given_names + :members: .. automodule:: nameparser.config.conjunctions :members: .. automodule:: nameparser.config.maiden_markers diff --git a/docs/release_log.rst b/docs/release_log.rst index 379f43227..afe7a8860 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -15,7 +15,8 @@ Release Log - **Read a credential that is also a name, written in capitals in a mixed-case name, as settled.** ``John Smith MA`` reports no ``suffix-or-name`` ambiguity, where 2.0.0 through 2.6.0 reported one, and ``Smith, John MA Jones`` gives suffix ``MA Jones``, where 2.0.0 through 2.6.0 gave middle ``MA Jones`` and 1.4.0 middle ``Jones``, suffix ``MA``. A name written in one case still reports (``JOHN SMITH MA``), and with no comma a name word behind the capitals keeps them a name word, since they may be initials (``Maria Ana MA Silva`` keeps last ``Silva``). :issue:`785`, :issue:`752` - **Read a title that is also a given name as a title only inside a title phrase.** ``Chief Justice John Roberts`` gives title ``Chief Justice``, first ``John``, where 1.4.0 through 2.6.0 gave title ``Chief``, first ``Justice``; ``Field Marshal Bernard Montgomery``, ``Dean of Students Jane Smith`` and ``Major General John Smith`` give the phrase as the title too. Outside a phrase the word is still a name, so ``Dean Smith`` and ``Grace Smith`` give first ``Dean`` and ``Grace``; ``Dean Smith`` now reports a ``title-or-name`` ambiguity unless a comma has marked the given name (``Smith, Dean``), while ``Grace``, a title only in a phrase such as ``His Grace``, reports nothing. Listing the word as a title still makes it one everywhere (``Lexicon.default().add(titles={"dean"})``). :issue:`752`, :issue:`853` - **Fix a lone title qualifier at the head of a name reading as a title.** ``Wing Lau`` and ``Mr Wing Lau`` give first ``Wing``, where 1.4.0 through 2.6.0 gave title ``Wing`` and ``Mr Wing``; a qualifier such as ``Wing``, ``Vice`` or ``Grand`` is a title only with a title after it (``Wing Commander Lau``). After a comma it is still a title (``John Smith, Vice``), except ``Wing``, which is also a given name: ``Lau, Wing Kai`` gives first ``Wing``, middle ``Kai``, where 1.4.0 through 2.6.0 gave title ``Wing``, first ``Kai``. ``John Smith Wing.`` gives last ``Wing.``, as 1.4.0 through 2.2.0 did, where 2.3.0 through 2.6.0 gave title ``Wing.``. Like ``Dean``, it reverses a full name before a comma (``John Smith, Wing`` gives first ``Wing``, last ``John Smith``). :issue:`752` - - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` + - **Read Chas., Wm. and the other given names written short as given names rather than titles.** ``Chas. Smith`` and ``Ma. Cristina Reyes`` give first ``Chas.`` and ``Ma.``, where 1.4.0 through 2.6.0 gave title ``Chas.`` and ``Ma.``; ``Thos.``, ``Jas.``, ``Geo.``, ``Robt.``, ``Benj.``, ``Jno.``, ``Saml.``, ``Edw.``, ``Richd.``, ``Wm.`` and the Filipino ``Jo.`` read the same way, behind a title too (``Mrs Ma. Liza Cruz`` gives title ``Mrs``, first ``Ma.``). An unlisted abbreviation in front is still a title (``Insp. Jane Morse``), and ``Lexicon.given_name_abbreviations`` lists the words. :issue:`789`, :issue:`752` + - **A Lexicon or Parser pickled by an earlier release no longer loads.** Loading one raises ``ValueError`` naming the new fields (``suffix_words_ambiguous``, ``agnomens``, ``titles_ambiguous``, ``title_modifiers``, ``title_styles``, ``title_phrases``, ``given_name_abbreviations``); re-pickle after upgrading. ``ParsedName`` and ``HumanName`` pickles still load. :issue:`752` * 2.6.0 - October 25, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 9f1573da2..46f4cf2b2 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -861,8 +861,8 @@ Titles you didn't configure ----------------------------- A trailing period marks an abbreviation, and the parts of a name that -get abbreviated are the ones standing outside it — titles and -post-nominals. So a word ending in a period, in no vocabulary list at +get abbreviated are mostly the ones standing outside it — titles and +post-nominals; the exception, a given name written short, is below. So a word ending in a period, in no vocabulary list at all, is read as a title when it stands where a title stands: at the front of the part that carries the given name. That is what lets unfamiliar ranks, honorifics and abbreviations work without configuring @@ -890,8 +890,7 @@ rule applies there in exactly the same way: Where the inference stops ~~~~~~~~~~~~~~~~~~~~~~~~~ -The rule is bounded in five ways, so it doesn't swallow ordinary -names: +The rule is bounded so it doesn't swallow ordinary names: - **Single initials** are left alone (``J.``). - **Abbreviations with interior periods** are left alone (``E.T.``). @@ -905,9 +904,18 @@ names: slot holds a surname too: ``Xyz. Smith, John`` keeps family ``Xyz. Smith``. A listed title is still read there, so ``Dr. Smith, John`` gives title ``Dr.``. +- **A given name written short** is that given name (since 2.7): the + traditional English abbreviations such as ``Chas.``, ``Geo.`` and + ``Wm.``, and the Filipino ``Ma.`` (María) and ``Jo.`` (José). + ``Lexicon.given_name_abbreviations`` lists them; written bare, or + anywhere else in the name, each word reads as it would unlisted. .. doctest:: + >>> parse("Chas. Smith").given + 'Chas.' + >>> parse("Mrs Ma. Liza Cruz").given + 'Ma.' >>> parse("J. Smith").given 'J.' >>> parse("E.T. Jones").given diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index 44a638085..925540f05 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1045,6 +1045,7 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: removal path. """ from nameparser.config.conjunctions import CONJUNCTIONS_AMBIGUOUS + from nameparser.config.given_names import GIVEN_NAME_ABBREVIATIONS from nameparser.config.maiden_markers import MAIDEN_MARKERS from nameparser.config.suffixes import ( AGNOMENS, GLUED_HONORIFICS, SUFFIX_WORDS_AMBIGUOUS) @@ -1196,6 +1197,15 @@ def matchable(field: str) -> frozenset[str]: # through this intersection. conjunctions_ambiguous=CONJUNCTIONS_AMBIGUOUS & conjunctions, bound_given_names=bound, + # likewise no v1 manager: an abbreviated given name + # reading over rules/H2's shape is 2.7 behavior (#789), + # so the set rides in the snapshot only, passed whole. + # No v1 knob reaches it, and none needs to: a word a v1 + # caller ADDS to `titles` is a title wherever it stands, + # since the leading peel asks the title vocabulary before + # it weighs the shape, as it did in 1.4 + # (`c.titles.add("chas")`) + given_name_abbreviations=GIVEN_NAME_ABBREVIATIONS, # v1 Constants has no manager for these (#274 is 2.0 # behavior); the data module is the only source maiden_markers=MAIDEN_MARKERS, diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 5aada7e35..2192a7cf2 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -25,6 +25,7 @@ "titles", "given_name_titles", "suffix_acronyms", "suffix_words", "suffix_acronyms_ambiguous", "particles", "particles_ambiguous", "conjunctions", "conjunctions_ambiguous", "bound_given_names", + "given_name_abbreviations", "maiden_markers", "surnames", "honorific_tails", "after_given_titles", "suffix_words_ambiguous", "agnomens", "titles_ambiguous", "title_modifiers", "title_styles", "title_phrases", @@ -668,6 +669,17 @@ class Lexicon: #: Full default list: #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. bound_given_names: frozenset[str] = frozenset() + #: Given names written short ("chas" for Charles, "wm" for William, + #: Filipino "ma" for María). Period-marked and opening the part of + #: the name that carries the given name, an entry reads as that + #: given name, where an unlisted abbreviation there reads as a + #: title by its shape (rules/H2): "Chas. Smith" reads given + #: "Chas.", "Xyz. Smith" title "Xyz.". An entry also in ``titles`` + #: is a title there, which is what listing it as one asks; anywhere + #: else an entry is read as it would be unlisted. Full default + #: list: + #: :data:`~nameparser.config.given_names.GIVEN_NAME_ABBREVIATIONS`. + given_name_abbreviations: frozenset[str] = frozenset() #: Marker words introducing a birth surname, routed to the maiden #: field ("née", "geb.", "rozená", ...). An entry may be a PHRASE #: ("z domu"): entries are matched by lookahead, longest first, so @@ -1030,6 +1042,7 @@ def remove(self, **entries: Iterable[str]) -> Lexicon: def _default_lexicon() -> Lexicon: # v1 data modules are the single source of vocabulary through 2.x. from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES + from nameparser.config.given_names import GIVEN_NAME_ABBREVIATIONS from nameparser.config.capitalization import CAPITALIZATION_EXCEPTIONS from nameparser.config.conjunctions import ( CONJUNCTIONS, CONJUNCTIONS_AMBIGUOUS) @@ -1067,6 +1080,7 @@ def _default_lexicon() -> Lexicon: conjunctions=CONJUNCTIONS, conjunctions_ambiguous=CONJUNCTIONS_AMBIGUOUS, bound_given_names=BOUND_GIVEN_NAMES, + given_name_abbreviations=GIVEN_NAME_ABBREVIATIONS, maiden_markers=MAIDEN_MARKERS, surnames=KOREAN_SURNAMES, honorific_tails=GLUED_HONORIFICS, diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 0b8030042..750fbd0a2 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -33,7 +33,8 @@ "vocab:after-given-title", "vocab:title-ambiguous", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", "vocab:suffix-word-ambiguous", "vocab:agnomen", -"vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", +"vocab:particle-ambiguous", "vocab:bound-given", +"vocab:given-abbreviation", "vocab:maiden-marker", "vocab:maiden-marker-cont"; in a namespace of its own, "shape:acronym"; and in another, "split:credential" and "split:member", written on the first word of a credential split across @@ -67,7 +68,8 @@ from nameparser._lexicon import _normalize, _spells_an_initial from nameparser._policy import CapsSuffixes from nameparser._pipeline._state import ( - AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, SHAPE_ACRONYM_TAG, + AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, + GIVEN_ABBREVIATION_TAG, SHAPE_ACRONYM_TAG, SPLIT_CREDENTIAL_TAG, SPLIT_MEMBER_TAG, TITLE_AMBIGUOUS_TAG, ParseState, PendingAmbiguity, WorkToken, copy_with, @@ -219,6 +221,12 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, tags.add("initial") if n in lex.bound_given_names: tags.add("vocab:bound-given") + # rules/H2: "an abbreviation the vocabulary reads as a given name + # written short is that given name" -- the reading H2's shape test + # weighs (`_pieces.is_leading_title`); a membership, so any spelling + # carries it and only that test acts on it + if n in lex.given_name_abbreviations: + tags.add(GIVEN_ABBREVIATION_TAG) # maiden markers are NOT tagged here: an entry may be a phrase whose # words are not markers on their own, and this function sees one # token with no neighbours. `_vocab.tag_marker_runs` does the whole diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index dd5ac90fd..fd7891330 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -58,8 +58,8 @@ from typing import NamedTuple from nameparser._pipeline._state import ( - AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, NAME_ROLES, - SHAPE_ACRONYM_TAG, SUFFIX_OR_UNREAD, AmbiguityKind, PendingAmbiguity, + AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, + GIVEN_ABBREVIATION_TAG, NAME_ROLES, SHAPE_ACRONYM_TAG, SUFFIX_OR_UNREAD, AmbiguityKind, PendingAmbiguity, WorkToken, ) from nameparser._pipeline._vocab import ( @@ -383,7 +383,10 @@ def is_leading_title(piece: Sequence[int], ptags: Set[str], # own test file) checks it over the union of both predicates' # example tables rather than leaving it to a sentence. return (bool(_PERIOD_ABBREV.match(text)) - and (text.isascii() or not in_initialless_script(text))) + and (text.isascii() or not in_initialless_script(text)) + # rules/H2's precedence: a given name written short outranks + # the shape ('Chas.', #789) -- classify's tag, a C-level test + and GIVEN_ABBREVIATION_TAG not in tokens[piece[0]].tags) def leading_titles(pieces: Sequence[Sequence[int]], diff --git a/nameparser/_pipeline/_state.py b/nameparser/_pipeline/_state.py index 2ad1ca947..3e5ccb969 100644 --- a/nameparser/_pipeline/_state.py +++ b/nameparser/_pipeline/_state.py @@ -103,6 +103,12 @@ class WorkToken: #: the fork to report where the word opens the name. TITLE_AMBIGUOUS_TAG = "vocab:title-ambiguous" +#: On a word of `Lexicon.given_name_abbreviations`, a given name written +#: short ('Chas.', 'Ma.'): rules/H2's shape test declines a word +#: carrying it, so the given reading outranks the shape-derived title +#: (`_pieces.is_leading_title`, `_vocab.is_title_shaped`). +GIVEN_ABBREVIATION_TAG = "vocab:given-abbreviation" + #: EITHER way a token joins the ambiguous credential class -- the #: vocabulary's claim and the writing's. The two emitters that report #: a fork the peel called and declined ask exactly this: `_group`'s diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index f059ac9a9..fd66ffc13 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -202,11 +202,13 @@ def in_initialless_script(text: str) -> bool: # carries the given name — the whole name, or the part after a # family comma — reads as a title even when unlisted" # (history: decisions/H2) -def is_title_shaped(text: str) -> bool: - """Whether TEXT wears H2's shape alone -- vocabulary-free, the - LISTED half being a separate lookup at each caller's own site - (is_title_piece/lexicon.titles). `name_word_count` below is the - only CALLER; `_pieces.is_leading_title` asks the same question +def is_title_shaped(text: str, lexicon: Lexicon) -> bool: + """Whether TEXT reads as a title by H2's shape -- the LISTED half + being a separate lookup at each caller's own site + (is_title_piece/lexicon.titles), and the one vocabulary read here + the given name written short, which outranks the shape (rules/H2, + #789: 'Chas.' is no title). `name_word_count` and `title_readings` + below are its callers; `_pieces.is_leading_title` asks the same question but keeps this body inline, its path being hot (every leading piece of every parse) where this one is cold (comma names only) -- the measured figure lives at that inline copy and is not @@ -230,7 +232,8 @@ def is_title_shaped(text: str) -> bool: parse, this running four times per piece. """ return (bool(_PERIOD_ABBREV.match(text)) - and (text.isascii() or not in_initialless_script(text))) + and (text.isascii() or not in_initialless_script(text)) + and _normalize(text) not in lexicon.given_name_abbreviations) def is_initial_shaped(text: str) -> bool: @@ -1099,11 +1102,14 @@ def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: shape's "unlisted" (`caps_shape_candidate`), where a word a caller listed even as a SURNAME must never become a credential. - The two left out: `title_phrases`, whose entries are two words and - so never one, and `titles_ambiguous`, a title that may be a given + The three left out: `title_phrases`, whose entries are two words and + so never one; `titles_ambiguous`, a title that may be a given name, which outside a title phrase is read as the unlisted word it is there (rules/H9: "and elsewhere a name word") -- a phrase is two - words and a comma part read by this shape is one.""" + words and a comma part read by this shape is one; and + `given_name_abbreviations`, a given name written short (#789), + whose one reader is H2's shape, so anywhere else its word reads as + the unlisted word it is ('John Smith, JAS' keeps suffix 'JAS').""" for field in _LISTED_FIELDS: if n in getattr(lexicon, field): return True @@ -1111,14 +1117,17 @@ def in_any_wordlist(n: str, lexicon: Lexicon) -> bool: _LISTED_FIELDS = tuple(f for f in _VOCAB_FIELDS - if f not in ("title_phrases", "titles_ambiguous")) + if f not in ("title_phrases", "titles_ambiguous", + "given_name_abbreviations")) #: The wordlists that claim a word as something OTHER than name text. #: `surnames` and `bound_given_names` are left out: they say a word IS #: a name word ('Kim', 'Abdul'), the opposite claim; and so is #: `titles_ambiguous`, a title that may be a given name (rules/H9), with -#: `title_phrases`, whose two-word entries never match one word. +#: `title_phrases`, whose two-word entries never match one word; and so +#: is `given_name_abbreviations`, a given name written short (#789), left +#: out of `_LISTED_FIELDS` above. _NON_NAME_FIELDS = tuple(f for f in _LISTED_FIELDS if f not in ("surnames", "bound_given_names")) @@ -1505,7 +1514,7 @@ def title_readings(texts: Sequence[str], folded: Sequence[str], lead = 0 if head: while lead < len(folded) and (reading[lead] or folded[lead] in conjunctions - or is_title_shaped(texts[lead])): + or is_title_shaped(texts[lead], lexicon)): lead += 1 # right to left, so each modifier knows whether a head title stands # behind it before the run is broken; a connective carries the run @@ -1518,7 +1527,7 @@ def title_readings(texts: Sequence[str], folded: Sequence[str], i < last and folded[i + 1] in _PHRASE_LINKS and folded[i + 1] in conjunctions) behind = reading[i] - elif reading[i] or (head and is_title_shaped(texts[i])): + elif reading[i] or (head and is_title_shaped(texts[i], lexicon)): behind = True elif n not in conjunctions: behind = False @@ -1575,7 +1584,7 @@ def name_word_count(texts: Sequence[str], lexicon: Lexicon, has_particle = True suffix = (is_suffix_lenient(text, lexicon) if lenient else is_suffix_strict(text, lexicon, letters_one_case)) - names.append(not (suffix or title or is_title_shaped(text))) + names.append(not (suffix or title or is_title_shaped(text, lexicon))) if not has_particle: return sum(names) count = 0 diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index ed6476284..41ce8cfde 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -1546,6 +1546,22 @@ 'عبد', }) +GIVEN_NAME_ABBREVIATIONS: frozenset[str] = frozenset({ + 'benj', + 'chas', + 'edw', + 'geo', + 'jas', + 'jno', + 'jo', + 'ma', + 'richd', + 'robt', + 'saml', + 'thos', + 'wm', +}) + MAIDEN_MARKERS: frozenset[str] = frozenset({ 'född', 'fødd', diff --git a/nameparser/config/given_names.py b/nameparser/config/given_names.py new file mode 100644 index 000000000..c9919e07e --- /dev/null +++ b/nameparser/config/given_names.py @@ -0,0 +1,18 @@ +# The words, and the vetting notes that stand beside them, live in +# nameparser/vocab/*.toml; tools/vocab/build.py compiles them into +# _compiled.py (decisions/vocabulary-model). +from nameparser.config import _compiled + +#: Given names written short ("chas" for Charles, "wm" for William, +#: Filipino "ma" for María): a word with a ``given`` reading whose +#: ``same_as`` is the full name. Period-marked and opening the part of +#: the name that carries the given name, one reads as that given name, +#: where an unlisted abbreviation there reads as a title by its shape +#: alone ("Chas. Smith" reads given "Chas.", "Xyz. Smith" title +#: "Xyz."). A listed title outranks it, and anywhere else the word is +#: read as it would be unlisted. +GIVEN_NAME_ABBREVIATIONS: frozenset[str] = _compiled.GIVEN_NAME_ABBREVIATIONS + +# Star imports read __all__ and never the module __getattr__ -- see the +# note in prefixes.py. +__all__ = ["GIVEN_NAME_ABBREVIATIONS"] diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 088dc4f86..0266b78a8 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -132,7 +132,7 @@ A reading's key is its **kind**. The table lists the kinds in their canonical or | `given` | `name` | nothing | | `family` | `name` | nothing | -A suffix reading goes to `SUFFIX_ACRONYMS` when its `acronym` property is true and to `SUFFIX_WORDS` otherwise; the column shows where each kind goes with `acronym` unwritten. A word has at most one suffix kind (check 4). +A suffix reading goes to `SUFFIX_ACRONYMS` when its `acronym` property is true and to `SUFFIX_WORDS` otherwise; the column shows where each kind goes with `acronym` unwritten. A word has at most one suffix kind (check 4). A `given` reading carrying `same_as` goes to `GIVEN_NAME_ABBREVIATIONS`, a given name written short (see [Name readings](#name-readings)). `generation` is a word that numbers or ranks its bearer among relatives of the same name, written after the name: junior and senior and their translations, `fils`, the roman numerals the vocabulary lists. `agnomen` is the Brazilian and Portuguese kind of it (`filho`, `neto`, `sobrinho` and their feminines): it is written in the case of the name around it, so its writing settles nothing, and an agnomen reading needs a name reading on its word (check 11), which gates it by the count alone ([rules/S11](../../docs/design/rules/S11-suffix-word-that-is-also-a-name.md), a suffix word that is also a name). The column shows an agnomen with that name reading. @@ -170,7 +170,7 @@ The `name` kinds compile to nothing on their own. They record that a word is bor ## Properties -A property is **behavioral** when it compiles into a set and so changes how names are parsed, **stated** when it compiles to nothing but the build holds it to the word's readings, and **descriptive** when it is checked and recorded but compiled to nothing. Each is allowed only on the categories listed; a `name` reading takes its own keys (see [Name readings](#name-readings)). Two of the sets are not properties at all but [derived markers](#derived-markers), read off a word's name readings. +A property is **behavioral** when it compiles into a set and so changes how names are parsed, **stated** when it compiles to nothing but the build holds it to the word's readings, and **descriptive** when it is checked and recorded but compiled to nothing, save `same_as` on a `given` reading, which compiles the word into `GIVEN_NAME_ABBREVIATIONS` (see [Name readings](#name-readings)). Each is allowed only on the categories listed; a `name` reading takes its own keys (see [Name readings](#name-readings)). Two of the sets are not properties at all but [derived markers](#derived-markers), read off a word's name readings. | Property | Type | Applies to | Unwritten | Effect | |---|---|---|---|---| @@ -187,7 +187,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | `issue` | descriptive | every category but `name` | | an issue number, or a list of them | | `decided` | descriptive | every category but `name` | | a TOML date | | `expands` | descriptive | every category but `name` | | the meaning the entry was added for, not every meaning | -| `same_as` | descriptive | every category but `name` | | another word of the same category that this one is a spelling of (`jnr` is `jr`) | +| `same_as` | descriptive | every category | | another word of the same category that this one is a spelling of (`jnr` is `jr`); on a `given` reading, the full given name the word abbreviates, which compiles the word into `GIVEN_NAME_ABBREVIATIONS` (see [Name readings](#name-readings)) | | `native` | descriptive | every category, a `name` reading included | | a native-script spelling | | `gender` | descriptive | every category, a `name` reading included | | `"f"`, `"m"` or `"n"`, only where the word gives a definitive clue | @@ -222,6 +222,7 @@ A `given` or `family` reading takes its own keys: | `native` | a native-script spelling | | `note` | free text | | `gender` | `"f"`, `"m"` or `"n"` | +| `same_as` | on a `given` reading only: the full given name the word abbreviates, normalized (`chas` is `charles`), which puts the word in `GIVEN_NAME_ABBREVIATIONS`: written with its period at the head of the part of the name that carries the given name, the word is that given name, where an unlisted abbreviation there is a title by its shape ([rules/H2](../../docs/design/rules/H2-leading-abbreviation-title.md), the leading-abbreviation title), so `Chas. Smith` reads given `Chas.`. Anywhere else the reading acts on nothing. A name is no word of the vocabulary, so this `same_as`, unlike a reading's, names no entry, and an entry holding only such a reading is not inert | A name reading may stand without a count, as a particle kept possibly given under C-i's default with no source counting it does, but it then needs a `note` saying why it is declared (check 3). @@ -232,7 +233,7 @@ An exclusion records that a reading was considered and refused, with the reason, - **A refused reading** is a word-level table: `[word.excluded.generation]` with a `reason`, or `excluded = { generation = "the reason" }`. It refuses that kind in every file, and also every other kind that compiles into the same set, so a generic `suffix = {}` cannot put back a refused `generation`. A category name refuses the whole category (`excluded = { suffix = "..." }`). A `name` kind refuses only itself. - **A refused property** sits on a reading the word keeps: `[word.honorific.excluded]` with `attached = "the reason"`. It refuses any value of that property other than the one it takes unwritten, on any reading of the category; a refused `cased`, which belongs to the word, on any reading of the word. -The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `père` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error. +The value is a reason string, or a table with `reason` and optionally `issue`, `decided` and `note`. The exclusion goes in the file of the language whose reading it refuses. An entry may hold exclusions and no vocabulary reading, as `père` does. An entry with no reading and no exclusion is refused as inert, and one holding only a name reading is a warning, which the test suite makes an error, unless that reading is a `given` reading carrying `same_as`, which compiles. To see what has been refused, and why: `uv run python tools/vocab/query.py --excluded generation`. diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index dee11928b..3550b0379 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -2,7 +2,8 @@ schema = 1 [[word]] text = "wm" -excluded = { title = { reason = "William, abbreviated, and no title of any kind: #68 asked that 'Wm. Scott Magargee III' not read given name 'Wm.', a request judged mistaken, since Wm is the given name; a bare 'WM' opened the given part 69 times in the FEC donor list (re-measured 2026-10-24 on 08d130ed), every one read as a title. The dotted 'Wm.' opening the given part is still taken by the leading period-abbreviation shape rule whether listed or not", issue = [68, 789], decided = 2026-10-22 } } +excluded = { title = { reason = "William, abbreviated, and no title of any kind: #68 asked that 'Wm. Scott Magargee III' not read given name 'Wm.', a request judged mistaken, since Wm is the given name; a bare 'WM' opened the given part 69 times in the FEC donor list (re-measured 2026-10-24 on 08d130ed), every one then read as a title. The dotted 'Wm.' opening the given part reads given too, by the given reading below, which outranks the leading period-abbreviation shape rule (#789, phase 2k of #752)", issue = [68, 789], decided = 2026-10-22 } } +given = { us_ssa = 2736, as_of = 2026-10-26, same_as = "william", note = "William written short; US SSA births summed over both sexes (F 86, M 2,650); Wikidata labels no one Wm" } [[word]] text = "goodman" @@ -4979,3 +4980,43 @@ birth_marker = { gender = "f" } [[word]] text = "nee" birth_marker = { issue = 274, same_as = "née", gender = "f", note = "née typed without its accent" } + +[[word]] +text = "benj" +given = { wikidata = 2, as_of = 2026-10-26, same_as = "benjamin", note = "Benjamin written short; no US SSA year reaches 5 births; Wikidata is a lower bound" } + +[[word]] +text = "chas" +given = { us_ssa = 3223, wikidata = 43, as_of = 2026-10-26, same_as = "charles", note = "Charles written short; US SSA births summed over both sexes (F 10, M 3,213); nam_dict band 1 in the U.S.A.; Wikidata is a lower bound" } + +[[word]] +text = "edw" +given = { us_ssa = 6, as_of = 2026-10-26, same_as = "edward", note = "Edward written short; US SSA male births only" } + +[[word]] +text = "geo" +given = { us_ssa = 1324, wikidata = 47, as_of = 2026-10-26, same_as = "george", note = "George written short; US SSA births summed over both sexes (F 16, M 1,308); also a given name of its own (Wikidata: Romania 8, Belgium 6, Italy 5; nam_dict band 1 in Italy, the Netherlands and Switzerland), a given name on either reading; Wikidata is a lower bound" } + +[[word]] +text = "jas" +given = { us_ssa = 124, wikidata = 1, as_of = 2026-10-26, same_as = "james", note = "James written short; US SSA births summed over both sexes (F 13, M 111); also a Punjabi given name (nam_dict band 2 in India/Sri Lanka), and a surname (US Census 2010 117 bearers, Wikidata 25), which a period-marked word opening the given part is not; Wikidata is a lower bound" } + +[[word]] +text = "jno" +given = { us_ssa = 11, as_of = 2026-10-26, same_as = "john", note = "John written short; US SSA male births only" } + +[[word]] +text = "richd" +given = { same_as = "richard", note = "Richard written short; no US SSA year reaches 5 births and Wikidata labels no one Richd: declared as the traditional English abbreviation, beside its set (#789)" } + +[[word]] +text = "robt" +given = { us_ssa = 468, as_of = 2026-10-26, same_as = "robert", note = "Robert written short; US SSA male births only" } + +[[word]] +text = "saml" +given = { same_as = "samuel", note = "Samuel written short; no US SSA year reaches 5 births and Wikidata labels no one Saml: declared as the traditional English abbreviation, beside its set (#789)" } + +[[word]] +text = "thos" +given = { us_ssa = 166, as_of = 2026-10-26, same_as = "thomas", note = "Thomas written short; US SSA male births only; Wikidata labels no one Thos as a given name" } diff --git a/nameparser/vocab/fil.toml b/nameparser/vocab/fil.toml index 5bc5eda3d..c80829399 100644 --- a/nameparser/vocab/fil.toml +++ b/nameparser/vocab/fil.toml @@ -18,3 +18,11 @@ given = { us_ssa = 565, as_of = 2026-10-25, note = "US SSA births summed over bo [word.preposition] expands = "of the, de la contracted: \"Dela Cruz\"" note = "Dela is a short form of Adela, and an African name meaning saviour/redeemer -- so the Filipino surname particle is not the only reading" + +[[word]] +text = "jo" +given = { us_ssa = 180911, wikidata = 974, as_of = 2026-10-26, same_as = "josé", note = "José written short, in Philippine usage; the counts are of Jo, a given name of its own (US SSA births summed over both sexes, F 179,057, M 1,854; Wikidata a lower bound) and a Korean surname, so a period-marked Jo. opening the given part is a given name on every reading (#789)" } + +[[word]] +text = "ma" +given = { same_as = "maría", note = "María written short, in Philippine usage; no source counts the abbreviation apart from Ma, whose given, family and credential readings are en.toml's; it acts only where the leading period-abbreviation shape rule would have read a title, a period-marked word opening the given part (#789)" } diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 129ebcf54..dde2fed2e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1050,12 +1050,15 @@ def _check_cjk_shape_purity(self) -> None: "Dr.' is; 1.4.0 read title 'Si.' with first 'John'. The " "report says 'read as a title' (test_cases.py's field " "sweep holds it), where it had said 'read as a credential'"), - Case("comma_period_ma_is_a_title", "John, Ma.", - {"title": "Ma.", "family": "John"}, + Case("comma_period_ma_is_a_given_name", "John, Ma.", + {"given": "Ma.", "family": "John"}, ambiguities=("suffix-or-name",), - classification="fix(comma-family)", - notes="the same shape for the oldest marked member: 1.4.0 read " - "title 'Ma.' with first 'John'"), + classification="fix(#789)", + notes="the same shape for the oldest marked member, which is " + "also Filipino María written short: a given name written " + "short outranks rules/H2's shape (#789), so 'Ma.' is the " + "given name where 2.0.0 through 2.6.0 read it as H2's " + "title, and 1.4.0 read title 'Ma.' with first 'John'"), # -- #793's comment (the #794 review): the #791 members in a suffix # run, behind a title, written all-lower, and where no report # belongs. These pin today's readings; none moved in #793. @@ -11808,11 +11811,55 @@ def _check_cjk_shape_purity(self) -> None: Case("wm_bare_behind_a_title_is_the_given_name", "Dr. Wm Smith", {"title": "Dr.", "given": "Wm", "family": "Smith"}, classification="fix(#789)"), - Case("wm_dotted_opening_the_name_is_h2s_title", "Wm. Smith", - {"title": "Wm.", "family": "Smith"}, - notes="rules/H2's shape reads an unlisted abbreviation here, " - "listed or not; a given reading that outranks it is " - "phase 2's (#752), and this row should move with it"), + Case("wm_dotted_opening_the_name_is_the_given_name", "Wm. Smith", + {"given": "Wm.", "family": "Smith"}, + classification="fix(#789)", + notes="a given name written short outranks rules/H2's shape " + "(#752, phase 2k): 1.4.0 through 2.6.0 read title 'Wm.'"), + # -- #789, #752 phase 2k: a given name written short outranks rules/H2's + # shape. The reading moves only where the shape fired; the bare word + # and the dotted word anywhere else read as they did. + Case("chas_dotted_opening_the_name_is_the_given_name", "Chas. Smith", + {"given": "Chas.", "family": "Smith"}, + classification="fix(#789)", + notes="1.4.0 through 2.6.0 read title 'Chas.'"), + Case("chas_dotted_behind_a_title_is_the_given_name", "Dr. Chas. Smith", + {"title": "Dr.", "given": "Chas.", "family": "Smith"}, + classification="fix(#789)", + notes="the shape chained it into the title run: 1.4.0 through " + "2.6.0 read title 'Dr. Chas.' with no given name"), + Case("ma_dotted_behind_a_salutation_is_the_given_name", + "Mrs Ma. Liza Cruz", + {"title": "Mrs", "given": "Ma.", "middle": "Liza", + "family": "Cruz"}, + classification="fix(#789)", + notes="#853's PSC shape, Filipino María written short behind a " + "salutation: 1.4.0 through 2.6.0 read title 'Mrs Ma.', " + "given 'Liza'"), + Case("jo_dotted_opening_the_name_is_the_given_name", "Jo. Rizal Cruz", + {"given": "Jo.", "middle": "Rizal", "family": "Cruz"}, + classification="fix(#789)", + notes="Filipino José written short: 1.4.0 through 2.6.0 read " + "title 'Jo.', given 'Rizal'"), + Case("wm_dotted_after_a_family_comma_in_capitals", "SMITH, WM. DAVID", + {"given": "WM.", "middle": "DAVID", "family": "SMITH"}, + classification="fix(#789)", + notes="the FEC donor list's shape (20 names): 1.4.0 through " + "2.6.0 read title 'WM.', given 'DAVID'"), + Case("chas_dotted_under_family_first_is_placed_by_the_order", + "Chas. Smith", + {"given": "Smith", "family": "Chas."}, + policy=Policy(name_order=FAMILY_FIRST), + classification="fix(#789)", + notes="out of the title, the word is a name word the declared " + "order places, as 'Jack Smith' reads under it"), + Case("ma_bare_opening_the_name_is_unchanged", "Ma Smith", + {"given": "Ma", "family": "Smith"}, + notes="the shape never read a bare word, so the reading has " + "nothing to outrank"), + Case("ma_dotted_inside_the_name_is_unchanged", "John Ma. Smith", + {"given": "John", "middle": "Ma.", "family": "Smith"}, + notes="the shape reads only the head of the given part"), Case("wm_dotted_trailing_is_a_name_word", "John Smith Wm.", {"given": "John", "middle": "Smith", "family": "Wm."}, notes="H5 read a listed `wm` as a title here from 2.3.0 " diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index e2cad9b78..d08be9068 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -661,6 +661,7 @@ def test_leading_title_shape_refuses_an_initialless_script( @pytest.mark.parametrize("text", [ "Xyz.", "Dr.", "Xyz", "X.", "田中.", "김민준.", "たなか.", "Kim김.", "J.", + "Chas.", "MA.", ]) def test_is_title_shaped_and_is_leading_title_agree(text: str) -> None: # The two spellings of H2's shape test -- `is_title_shaped`'s own @@ -676,8 +677,11 @@ def test_is_title_shaped_and_is_leading_title_agree(text: str) -> None: # answers True the same way, for the same reason: listed or not # is a question neither predicate asks), not because it exercises # the shape branch on either side. + # 'Chas.' and 'MA.' are given names written short, which outrank the + # shape (#789): the inline copy reads classify's tag, this one the + # lexicon, and the two must agree. state = _through_group(text + " Smith") - assert is_title_shaped(text) == is_leading_title( + assert is_title_shaped(text, state.lexicon) == is_leading_title( state.pieces[0][0], state.piece_tags[0][0], state.tokens) diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 2ce0a9f3c..948910235 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -41,6 +41,9 @@ # no run either. suffix_acronyms=frozenset({"md", "vd", "ma"}), suffix_acronyms_ambiguous=frozenset({"ma"}), + # and `ma` is a given name written short too (Filipino María, #789), + # read so only period-marked at the head of the given part + given_name_abbreviations=frozenset({"ma"}), conjunctions=frozenset({"y"}), bound_given_names=frozenset({"abdul"}), ) diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 73209063f..6c195eeee 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -544,12 +544,15 @@ def test_the_caps_exclusion_covers_every_vocabulary_field() -> None: assert caps_shape_candidate("ZZQX", base, on, False) # H9: two fields are left out on purpose -- a title that may # be a given name reads, outside a phrase, as the unlisted word it - # is there, and a phrase is two words - for field in ("titles_ambiguous", "title_phrases"): + # is there, and a phrase is two words -- and #789's third: a given + # name written short is read only by H2's shape + for field in ("titles_ambiguous", "title_phrases", + "given_name_abbreviations"): assert caps_shape_candidate( "ZZQX", base.add(**{field: {"zzqx"}}), on, False), field for field in _VOCAB_FIELDS: - if field in ("titles_ambiguous", "title_phrases"): + if field in ("titles_ambiguous", "title_phrases", + "given_name_abbreviations"): continue extra: dict[str, set[str]] = {field: {"zzqx"}} if field in ("honorific_tails", "suffix_words_ambiguous"): @@ -574,16 +577,24 @@ def test_is_title_shaped_is_h2_s_shape_alone() -> None: # (#289/#516, quality-review finding): both must answer alike for # name_word_count's comma-form count not to disagree with the # leading peel about what a title is. - assert is_title_shaped("Xyz.") # unlisted, H2-shaped + lex = Lexicon.default() + assert is_title_shaped("Xyz.", lex) # unlisted, H2-shaped # LISTED or not is a vocabulary question this predicate never # asks -- 'Dr.' wears the same shape 'Xyz.' does, and answers the # same way; the caller's own listed lookup is what tells them # apart (is_title_piece/lexicon.titles, at each call site) - assert is_title_shaped("Dr.") - assert not is_title_shaped("Xyz") # no trailing period - assert not is_title_shaped("X.") # one letter: an initial, - # not an abbreviation - assert not is_title_shaped("田中.") # initialless script (#323) + assert is_title_shaped("Dr.", lex) + assert not is_title_shaped("Xyz", lex) # no trailing period + assert not is_title_shaped("X.", lex) # one letter: an initial, + # not an abbreviation + assert not is_title_shaped("田中.", lex) # initialless script (#323) + # rules/H2's precedence: a given name written short outranks the + # shape (#789) -- the one vocabulary the predicate reads, and only + # that: the same word unlisted wears the shape again + assert not is_title_shaped("Chas.", lex) + assert not is_title_shaped("MA.", lex) + assert is_title_shaped("Chas.", lex.remove( + given_name_abbreviations={"chas"})) def test_name_word_count_counts_names_not_tokens() -> None: diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 209705e26..da6ba6605 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -18,14 +18,14 @@ "D1": {"words": 73, "then_words": 26, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "D2": {"words": 65, "then_words": 37, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "H1": {"words": 427, "then_words": 270, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 4, "accepted_words": 370}, - "H2": {"words": 316, "then_words": 166, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 3, "accepted_words": 203}, + "H2": {"words": 477, "then_words": 172, "because_sentences": 1, "except": 1, "precedence": 2, "accepted": 3, "accepted_words": 203}, "H3": {"words": 142, "then_words": 72, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 113}, - "H4": {"words": 390, "then_words": 183, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 1, "accepted_words": 127}, + "H4": {"words": 410, "then_words": 183, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 1, "accepted_words": 127}, "H5": {"words": 757, "then_words": 289, "because_sentences": 2, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 117}, "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, - "H9": {"words": 616, "then_words": 263, "because_sentences": 3, "except": 0, "precedence": 5, "accepted": 6, "accepted_words": 211}, + "H9": {"words": 622, "then_words": 263, "because_sentences": 3, "except": 0, "precedence": 5, "accepted": 6, "accepted_words": 211}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 714, "then_words": 435, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py index 09bd9344e..bf3fd0571 100644 --- a/tests/v2/test_config_shim.py +++ b/tests/v2/test_config_shim.py @@ -1325,3 +1325,15 @@ def test_snapshot_cleared_titles_turn_the_title_phrase_off() -> None: lexicon, _, _ = c._snapshot() assert lexicon.titles_ambiguous == frozenset() assert HumanName("Dean of Students Jane Smith", constants=c).title == "" + + +def test_snapshot_a_given_name_written_short_listed_as_a_title_is_a_title() -> None: + # #789: given_name_abbreviations rides in the snapshot whole, no v1 + # manager reaching it; the v1 recipe that lists the word as a title + # (`c.titles.add("chas")`) gets 1.4's reading back, the leading peel + # asking the titles before the shape the set outranks + c = Constants() + assert HumanName("Chas. Smith", constants=c).first == "Chas." # baseline + c.titles.add("chas") + assert "chas" in c._snapshot()[0].given_name_abbreviations + assert HumanName("Chas. Smith", constants=c).title == "Chas." diff --git a/tests/v2/test_facade_cases.py b/tests/v2/test_facade_cases.py index 49c7df4a9..7e00799d4 100644 --- a/tests/v2/test_facade_cases.py +++ b/tests/v2/test_facade_cases.py @@ -110,6 +110,9 @@ # #864: the dotted switch turned off, which the v1 shim does not # translate (decisions/S9) "paired_initials_under_the_switch_off_read_the_same", + # #789: a given name written short under a declared family-first + # order, which v1 cannot express + "chas_dotted_under_family_first_is_placed_by_the_order", # #718: S10's count under a declared family-first order, which v1 # cannot express "a_lone_numeral_counts_in_units_under_family_first", diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index fa9332f03..46a950936 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3605,3 +3605,94 @@ def fields(name: ParsedName) -> tuple[object, ...]: failures.append(f"{order!r}: {text!r}") assert not failures, (f"{len(failures)} bare salutation(s) read unlike " f"the period-marked twin:\n" + "\n".join(failures[:10])) + + +#: rules/H2's precedence grid (#789): each slot an abbreviation's shape +#: reads (the head of the given part, behind a title, after a family +#: comma with and without a word before it, before a nickname, a +#: particle, a title-particle, a connective, a maiden marker or a +#: trailing credential), and before a comma ahead of a word of the +#: ambiguous credential class, where the comma's own count asks the +#: shape, a modifier's title phrase among it (`Smith, Ed`), each paired +#: with the same text holding a plain given name in the abbreviation's +#: place; and the bare spellings, which must read as with the set +#: emptied, in two slots of their own (`{} SMITH, XYZ`, `John Smith, {}`, +#: where a wordlist roster once made a bare one a name, #789's review). +#: The 13 shipped abbreviations in six spellings, 21 slots, three +#: orders: 4,914 parses with their control, about 1.5s on py3.11 +#: (measured 2026-10-26). Mutation-checked the same day, on the 19 +#: slots before the review added two: with the peel's half of the +#: precedence off (`_pieces.is_leading_title`) 2,103 of the dotted pairs +#: fail, with the comma count's (`_vocab.is_title_shaped`) 234, and with +#: `_vocab.title_readings`' two calls alone 117; with the field back in +#: `_vocab._LISTED_FIELDS`, 66 of the 21-slot grid. +_ABBREVIATION_SLOTS = ( + "{} Smith", "{} H. Smith", "Smith, {}", "Smith, {} John", ", {} Smith", + "Mr. {} Smith", "Mrs {} Liza Cruz", "Dr. {} Smith, PhD", + "(Bob) {} Smith", "{} Smith Jr.", "{} de la Vega", + "{} Freiherr von Berg", "Rev. Sr. {} Smith", "{} y Gasset", + "{} nee Smith", "Smith Jr., {} John", "{} Smith, Ed", "{} Vice Smith, Ed", "Vice {} Smith, Ed", + "{} SMITH, XYZ", "John Smith, {}") + + +def test_a_given_name_written_short_reads_as_a_given_name() -> None: + """rules/H2's precedence bullet (#789): "an abbreviation the + vocabulary reads as a given name written short is that given name". + Two-sided, against the same lexicon with the set emptied: where the + shape reads the abbreviation as a title there, or the comma's count + reads it as one, it reads, field for + field, as the plain given name `Jack` reads in its place; and where + the shape reads no title (`Smith, MA.`, the credential its capitals + settle), the reading is the one the emptied set gives. Ambiguity + kinds are not compared: `ma` keeps its credential reading, and the + report that reading draws after a comma. The bare spellings, which + the shape never reads, must read exactly as with the set emptied in + every slot, the capitals behind a comma included ('John Smith, JAS' + keeps suffix 'JAS', #789's review). Every slot must see the + shape fire somewhere, so a pass is not a grid the shape never + reaches.""" + lexicon = Lexicon.default() + off = lexicon.remove( + given_name_abbreviations=set(lexicon.given_name_abbreviations)) + + def fields(name: ParsedName, plain: str = "", head: str = "") -> dict[str, str]: + return {k: v.replace(plain, head) if plain else v + for k, v in name.as_dict().items() if v} + + failures, fired = [], set() + for order in (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST): + policy = Policy(name_order=order) + parser, control = Parser(policy=policy), Parser(lexicon=off, policy=policy) + # the stand-in in capitals behind a mixed-case name would be an + # unlisted capitals credential itself ('John Smith, JACK'), which + # no given name written short is: read it with that shape off. + # OFF reaches every word of the text, so a slot compared through + # the stand-in must hold no other capitals word ('{} SMITH, XYZ' + # is compared only with the set emptied) + stand_in = Parser(policy=dataclasses.replace( + policy, unlisted_caps_suffixes=CapsSuffixes.OFF)) + for word in sorted(lexicon.given_name_abbreviations): + # the bare spellings too, which the set must never move + for head, plain in ((word.title() + ".", "Jack"), + (word.upper() + ".", "JACK"), + (word + ".", "jack"), + (word.title(), ""), (word.upper(), ""), + (word, "")): + for slot in _ABBREVIATION_SLOTS: + text = slot.format(head) + got = fields(parser.parse(text)) + unlisted = control.parse(text) + if plain and (head in unlisted.title.split() + or slot.endswith(", Ed")): + fired.add(slot) + want = fields(stand_in.parse(slot.format(plain)), plain, head) + else: + want = fields(unlisted) + if got != want: + failures.append(f"{order!r}: {text!r} {got} != {want}") + assert not failures, (f"{len(failures)} abbreviation(s) misread:\n" + + "\n".join(failures[:10])) + # '{} SMITH, XYZ' is there for the bare spellings: before a family + # comma the shape never fires (rules/H6) + assert fired == set(_ABBREVIATION_SLOTS) - {"{} SMITH, XYZ"}, ( + set(_ABBREVIATION_SLOTS) - fired) diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 93ecf7808..6daece877 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -85,6 +85,7 @@ def test_the_compiled_sets_are_the_default_lexicon(shipped: dict[str, str]) -> N conjunctions=c["CONJUNCTIONS"], conjunctions_ambiguous=c["CONJUNCTIONS_AMBIGUOUS"], bound_given_names=c["BOUND_GIVEN_NAMES"], + given_name_abbreviations=c["GIVEN_NAME_ABBREVIATIONS"], maiden_markers=c["MAIDEN_MARKERS"], surnames=lex.surnames, honorific_tails=c["GLUED_HONORIFICS"], after_given_titles=c["AFTER_GIVEN_TITLES"], @@ -417,6 +418,12 @@ def _w(body: str) -> str: ("same-as-cycle-through-a-second-kind", {"nl.toml": _w('[[word]]\ntext = "c"\npreposition = { may_be_given = false }\n\n[[word]]\ntext = "a"\npreposition = { may_be_given = false, same_as = "c" }\narticle = { may_be_given = false, same_as = "b" }\n\n[[word]]\ntext = "b"\narticle = { may_be_given = false, same_as = "a" }')}, "same_as loops, a -> b -> a;"), # a chain running INTO a loop: the loop is named, not the chain ("same-as-chain-into-a-loop", {"en.toml": _w('[[word]]\ntext = "xa"\ngeneration = { same_as = "xb" }\n\n[[word]]\ntext = "xb"\ngeneration = { same_as = "xc" }\n\n[[word]]\ntext = "xc"\ngeneration = { same_as = "xb" }')}, "same_as loops, xb -> xc -> xb;"), + # 7, a name reading's half (#789): a given reading's same_as is the + # full name it abbreviates, normalized and never the word itself, and + # a family reading takes none + ("same-as-on-a-family-reading", {"en.toml": _w('[[word]]\ntext = "smyth"\nfamily = { note = "x", same_as = "smith" }')}, "only a `given` reading takes one"), + ("same-as-on-a-given-reading-unnormalized", {"en.toml": _w('[[word]]\ntext = "chas"\ngiven = { note = "x", same_as = "Charles" }')}, "not in normalized form; write 'charles'"), + ("same-as-on-a-given-reading-names-itself", {"en.toml": _w('[[word]]\ntext = "chas"\ngiven = { note = "x", same_as = "chas" }')}, "names the word itself"), ] @@ -648,6 +655,17 @@ def test_a_name_reading_on_nothing_warns() -> None: build.compile_files(files) +def test_a_given_name_written_short_compiles_and_is_not_inert() -> None: + # #789: the same_as of a given reading names a name, which no entry + # holds, and puts the word in GIVEN_NAME_ABBREVIATIONS -- so the entry, + # holding only that reading, is no inert name reading (`_compile` + # turns the warning into an error), and a plain given reading beside + # it on another word still marks nothing + c = _compile({"en.toml": _w('[[word]]\ntext = "chas"\ngiven = { note = "x", same_as = "charles" }')}) + assert c["GIVEN_NAME_ABBREVIATIONS"] == frozenset({"chas"}) + assert _sets_fed("given") == set() + + def test_a_check_failure_names_the_refresh_command_in_main( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: out = tmp_path / "_compiled.py" diff --git a/tools/differential/conventions_agreement.json b/tools/differential/conventions_agreement.json index 1cec27cd2..6014b0fb5 100644 --- a/tools/differential/conventions_agreement.json +++ b/tools/differential/conventions_agreement.json @@ -128,13 +128,9 @@ }, "given_names": { "expected": "Ma. Teresa", - "got": "Teresa Cruz" + "got": "Ma. Teresa Cruz" }, - "text": "Ma. Teresa Cruz Bautista", - "title": { - "expected": "", - "got": "Ma." - } + "text": "Ma. Teresa Cruz Bautista" }, "conventions/Michael Wong Kar Ming/default": { "family": { diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index af8570b92..251bba110 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -80,6 +80,8 @@ "Carod i" "Carod i Rovira, Josep" "Carod y de Rovira i" +"Chas. Smith" +"Chas. Smith, Ed" "Chief Dean and Mary Smith" "Chief Justice John Roberts" "DOE, JANE MA VD PHD" @@ -396,6 +398,7 @@ "John Smith MD-PhD" "John Smith Ma" "John Smith Ma Jones" +"John Smith Ma." "John Smith Mc V" "John Smith Msc.Ed." "John Smith Ph. D. Jones" @@ -576,6 +579,7 @@ "Lord Chancellor" "López, Ana García" "MSc Dr. med. univ." +"Ma." "Mai Le née Nguyen" "Major General John Smith" "Major Smith" @@ -615,6 +619,7 @@ "Mr. Jack and Jill" "Mr. Johnson" "Mr. and Mrs. Wing Lau" +"Mrs Ma. Liza Cruz" "Mrs. Garcia" "MÜLLER WEIß, HANS" "Ménil Christophe de" @@ -734,6 +739,8 @@ "Smith, A.B." "Smith, Abd" "Smith, Chancellor of the Exchequer MP" +"Smith, Chas." +"Smith, Do. John" "Smith, Dr." "Smith, Dr. Jr." "Smith, Esq." @@ -745,6 +752,7 @@ "Smith, John" "Smith, John MA Jones" "Smith, John MD/PhD" +"Smith, John Ma" "Smith, John PSM I, Jr." "Smith, John PhD I, Jr." "Smith, John PhD I." @@ -827,6 +835,7 @@ "Wing Lau" "Wing Smith, John" "Wing. Lau" +"Wm. H. Smith III" "Xyz. John Smith" "Xyz. Smith, John" "Xyz. van Johnson" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 794fa4b57..f9ae19571 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -504,6 +504,8 @@ {"name": "Charles Claude de La Billarderie d'Angiviller", "given": "Charles", "middle": "Claude", "family": "de La Billarderie d'Angiviller", "initials": "C. C. B. d.", "capitalized_force": "Charles Claude de la Billarderie D'Angiviller"} {"name": "Charles Wilkins", "given": "Charles", "family": "Wilkins", "initials": "C. W."} {"name": "Charles van der van der Berg", "given": "Charles", "family": "van der van der Berg", "initials": "C. B."} +{"name": "Chas. Smith", "given": "Chas.", "family": "Smith", "initials": "C. S."} +{"name": "Chas. Smith, Ed", "given": "Chas.", "family": "Smith", "suffix": "Ed", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "C. S.", "capitalized_force": "Chas. Smith ED"} {"name": "Chattel, Frederik Jacobus van Rossum", "given": "Frederik", "middle": "Jacobus van Rossum", "family": "Chattel", "initials": "F. J. R. C."} {"name": "Chemistry Jane Smith", "given": "Chemistry", "middle": "Jane", "family": "Smith", "initials": "C. J. S."} {"name": "Cherice J. (Johnson) Williams", "given": "Cherice", "middle": "J.", "family": "Williams", "nickname": "Johnson", "initials": "C. J. W."} @@ -1701,6 +1703,7 @@ {"name": "John Smith MEng PhD", "given": "John", "middle": "Smith", "family": "MEng", "suffix": "PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. M.", "capitalized_force": "John Smith Meng PhD"} {"name": "John Smith Ma", "given": "John", "middle": "Smith", "family": "Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. M."} {"name": "John Smith Ma Jones", "given": "John", "middle": "Smith Ma", "family": "Jones", "initials": "J. S. M. J."} +{"name": "John Smith Ma.", "given": "John", "middle": "Smith", "family": "Ma.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. M."} {"name": "John Smith Mc V", "given": "John", "family": "Smith", "suffix": "Mc V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MC V"} {"name": "John Smith Mr.", "title": "Mr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith Msc.Ed.", "given": "John", "family": "Smith", "suffix": "Msc.Ed.", "initials": "J. S."} @@ -2297,7 +2300,8 @@ {"name": "Ma III, Jack Jr", "given": "Jack", "family": "Ma", "suffix": "III, Jr", "initials": "J. M."} {"name": "Ma, Andrew", "given": "Andrew", "family": "Ma", "initials": "A. M."} {"name": "Ma, Jack", "given": "Jack", "family": "Ma", "initials": "J. M."} -{"name": "Ma. Teresa Cruz Bautista", "title": "Ma.", "given": "Teresa", "middle": "Cruz", "family": "Bautista", "initials": "T. C. B."} +{"name": "Ma.", "given": "Ma.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "M."} +{"name": "Ma. Teresa Cruz Bautista", "given": "Ma.", "middle": "Teresa Cruz", "family": "Bautista", "initials": "M. T. C. B."} {"name": "Maazị Christopher Chukwudike Amadi MP", "given": "Maazị", "middle": "Christopher Chukwudike", "family": "Amadi", "suffix": "MP", "initials": "M. C. C. A."} {"name": "Mac Miller", "given": "Mac", "family": "Miller", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "M. M."} {"name": "Mac a’ Ghobhainn Chorùna, Mgr. Dòmhnall Iain Bàn am post", "order": "FAMILY_FIRST", "title": "Mgr.", "given": "Dòmhnall", "middle": "Iain Bàn", "family": "Mac a’ Ghobhainn Chorùna", "suffix": "am post", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "D. I. B. a. G. C.", "capitalized_force": "Mgr. Dòmhnall Iain Bàn mac A’ Ghobhainn Chorùna AM Post"} @@ -2557,6 +2561,7 @@ {"name": "Mr. and Mrs. John Smith", "title": "Mr. and Mrs.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Mr. and Mrs. John and Jane Smith", "title": "Mr. and Mrs.", "given": "John and Jane", "family": "Smith", "initials": "J. J. S."} {"name": "Mr. and Mrs. Wing Lau", "title": "Mr. and Mrs.", "given": "Wing", "family": "Lau", "initials": "W. L."} +{"name": "Mrs Ma. Liza Cruz", "title": "Mrs", "given": "Ma.", "middle": "Liza", "family": "Cruz", "initials": "M. L. C."} {"name": "Mrs. Garcia", "title": "Mrs.", "family": "Garcia", "initials": "G."} {"name": "Ms A.J. Smith", "title": "Ms", "given": "A.J.", "family": "Smith", "initials": "A. S."} {"name": "Ms Hon Solo", "title": "Ms Hon", "family": "Solo", "initials": "S."} @@ -3140,7 +3145,9 @@ {"name": "Smith, Abd", "family": "Smith", "suffix": "Abd", "initials": "S.", "capitalized_force": "Smith ABD"} {"name": "Smith, Cecilia Angela", "given": "Cecilia", "middle": "Angela", "family": "Smith", "initials": "C. A. S."} {"name": "Smith, Chancellor of the Exchequer MP", "title": "Chancellor of the Exchequer", "family": "Smith", "suffix": "MP", "initials": "S."} +{"name": "Smith, Chas.", "given": "Chas.", "family": "Smith", "initials": "C. S."} {"name": "Smith, Chi", "given": "Chi", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "C. S."} +{"name": "Smith, Do. John", "title": "Do.", "given": "John", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "Smith, Dr.", "title": "Dr.", "family": "Smith", "initials": "S."} {"name": "Smith, Dr. John", "title": "Dr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, Dr. Jr.", "title": "Dr.", "family": "Smith", "suffix": "Jr.", "initials": "S."} @@ -3163,6 +3170,7 @@ {"name": "Smith, John MA Jones", "given": "John", "family": "Smith", "suffix": "MA Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "Smith, John MD/PhD", "given": "John", "family": "Smith", "suffix": "MD/PhD", "initials": "J. S."} {"name": "Smith, John MEng", "given": "John", "middle": "MEng", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M. S.", "capitalized_force": "John Meng Smith"} +{"name": "Smith, John Ma", "given": "John", "middle": "Ma", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M. S."} {"name": "Smith, John PSM I, Jr.", "given": "John", "middle": "I", "family": "Smith", "suffix": "PSM, Jr.", "initials": "J. I. S."} {"name": "Smith, John PhD I, Jr.", "given": "John", "middle": "I", "family": "Smith", "suffix": "PhD, Jr.", "initials": "J. I. S."} {"name": "Smith, John PhD I.", "given": "John", "family": "Smith", "suffix": "PhD I.", "initials": "J. S."} @@ -3205,7 +3213,7 @@ {"name": "Smith, MD; PhD", "family": "Smith", "suffix": "MD; PhD", "initials": "S."} {"name": "Smith, MEng", "given": "MEng", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. S.", "capitalized_force": "Meng Smith"} {"name": "Smith, Ma", "given": "Ma", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. S."} -{"name": "Smith, Ma. John", "title": "Ma.", "given": "John", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "Smith, Ma. John", "given": "Ma.", "middle": "John", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. J. S."} {"name": "Smith, Major. John", "title": "Major.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, Mary-Jane", "given": "Mary-Jane", "family": "Smith", "initials": "M. S."} {"name": "Smith, Ms Jane", "title": "Ms", "given": "Jane", "family": "Smith", "initials": "J. S."} @@ -3537,6 +3545,7 @@ {"name": "Wićazowa, Jěwa-Marja Hańža", "given": "Jěwa-Marja", "middle": "Hańža", "family": "Wićazowa", "initials": "J. H. W."} {"name": "Wićazowa, dr. Jěwa-Marja Hańža jr. phd", "order": "FAMILY_FIRST", "title": "dr.", "given": "Jěwa-Marja", "middle": "Hańža", "family": "Wićazowa", "suffix": "jr. phd", "initials": "J. H. W.", "capitalized_force": "Dr. Jěwa-Marja Hańža Wićazowa Jr. PhD"} {"name": "Wj. Monir Mómen Monir Mómen Balóch", "title": "Wj.", "given": "Monir", "middle": "Mómen Monir Mómen", "family": "Balóch", "initials": "M. M. M. M. B."} +{"name": "Wm. H. Smith III", "given": "Wm.", "middle": "H.", "family": "Smith", "suffix": "III", "initials": "W. H. S."} {"name": "Wolf von Engelhardt", "given": "Wolf", "family": "von Engelhardt", "initials": "W. E."} {"name": "Wolf, Ada Cornelia Eva Sophia van den", "given": "Ada", "middle": "Cornelia Eva Sophia", "family": "van den Wolf", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "A. C. E. S. W."} {"name": "Wolf, Ada Cornelia Eva Sophia von", "given": "Ada", "middle": "Cornelia Eva Sophia", "family": "von Wolf", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "A. C. E. S. W."} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 10504b5f9..bc9d58206 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -50,7 +50,10 @@ its category (`cased`: of the word); run before 4 and 5, so the recorded reason is what a contributor sees; 7. `same_as` names another word of the same category, `cased` spells - its word's letters (#459), at most one `cased` per word; + its word's letters (#459), at most one `cased` per word; on a + `given` reading `same_as` is the full given name the word + abbreviates, a name rather than a word of the vocabulary, written + normalized, and a `family` reading takes none; 8. a phrase only as a birth_marker or an addresses = "given" title, and never with a `cased`, which is looked up per word; 9. a bound given name that is also a particle has a given reading; @@ -82,7 +85,8 @@ together, each marker set in its base) need no check: a flag cannot exist without its reading, and a derived marker is read off the reading it marks. The one soft finding, a -name reading on a word with no vocabulary reading and no exclusion, is +name reading on a word with no vocabulary reading and no exclusion +(save an abbreviated given name, which compiles), is a VocabWarning, which the shipped-data test turns into an error. """ from __future__ import annotations @@ -249,12 +253,18 @@ def _alnum(text: str) -> str: #: derived marker reads these, and a name reading beside another word #: kind marks nothing, no gate reading that clash yet. GATED_WORDS = frozenset({"generation", "agnomen"}) -#: Descriptive properties: recorded, validated, compiled to nothing. +#: Descriptive properties: recorded, validated, compiled to nothing -- +#: save `same_as` on a `given` reading, which is a NAME_KEYS entry, not +#: this set's, and compiles the word into GIVEN_NAME_ABBREVIATIONS. DESCRIPTIVE = frozenset({"note", "issue", "decided", "expands", "same_as", "native", "gender"}) #: What a name reading (`given`, `family`) may carry. COUNT_SOURCES = frozenset({"us_census_2010", "us_ssa", "wikidata"}) -NAME_KEYS = COUNT_SOURCES | {"as_of", "native", "note", "gender"} +#: `same_as` on a `given` reading is the full name the word abbreviates +#: (`chas` -> charles), which compiles it into GIVEN_NAME_ABBREVIATIONS: +#: a period-marked abbreviation opening the given part is that given +#: name, not rules/H2's title (#789) +NAME_KEYS = COUNT_SOURCES | {"as_of", "native", "note", "gender", "same_as"} #: The keys of an `excluded` value's table, and of an `attached` table. EXCLUSION_KEYS = frozenset({"reason", "issue", "decided", "note"}) ATTACHED_KEYS = frozenset({"side", "issue", "decided", "note"}) @@ -767,6 +777,15 @@ def _check_reading(r: Reading, pool: Pool) -> None: raise VocabError(f"{where}: a name reading with no count needs a " f"`note` saying why it is declared") _check_descriptive(where, p) + if "same_as" in p: + if r.kind != "given": + raise VocabError(f"{where}: same_as on a name reading names the " + f"given name a word abbreviates, so only a " + f"`given` reading takes one") + if _normalize(p["same_as"]) != p["same_as"]: + raise VocabError(f"{where}: same_as {p['same_as']!r} is not in " + f"normalized form; write " + f"{_normalize(p['same_as'])!r}") pool.readings.append(r) return stated = STATED.get(r.category, frozenset()) @@ -1012,7 +1031,9 @@ def check_pool(pool: Pool) -> None: if target == r.text: raise VocabError(f"{r.where()}: same_as names the word itself; it " f"names the word this one is a spelling of") - if target is not None and not any( + # a given reading's same_as is the full name it abbreviates, and + # a name is no word of the vocabulary + if target is not None and r.category != "name" and not any( t.category == r.category for t in by_text.get(target, ())): raise VocabError(f"{r.where()}: same_as {target!r} has no " f"{r.category} reading") @@ -1107,7 +1128,9 @@ def check_pool(pool: Pool) -> None: # evidence is what a name reading usually is -- informs nothing excluded_texts = {x.text for x in pool.exclusions if x.prop is None} for text, rs in by_text.items(): - if all(r.category == "name" for r in rs) and text not in excluded_texts: + # an abbreviated given name compiles (GIVEN_NAME_ABBREVIATIONS) + if (all(r.category == "name" and "same_as" not in r.props for r in rs) + and text not in excluded_texts): warnings.warn(f"{rs[0].where()}: a name reading on a word with no " f"vocabulary reading is inert", VocabWarning, stacklevel=2) @@ -1145,6 +1168,7 @@ def _behavior(r: Reading, prop: str) -> object: ("CONJUNCTIONS", "conjunctions"), ("CONJUNCTIONS_AMBIGUOUS", "conjunctions"), ("BOUND_GIVEN_NAMES", "bound_given_names"), + ("GIVEN_NAME_ABBREVIATIONS", "given_names"), ("MAIDEN_MARKERS", "maiden_markers"), ("SUFFIX_WORDS", "suffixes"), ("GLUED_HONORIFICS", "suffixes"), @@ -1171,6 +1195,7 @@ class Compiled(TypedDict): CONJUNCTIONS: frozenset[str] CONJUNCTIONS_AMBIGUOUS: frozenset[str] BOUND_GIVEN_NAMES: frozenset[str] + GIVEN_NAME_ABBREVIATIONS: frozenset[str] MAIDEN_MARKERS: frozenset[str] SUFFIX_WORDS: frozenset[str] GLUED_HONORIFICS: frozenset[str] @@ -1204,6 +1229,7 @@ def compile_pool(pool: Pool) -> Compiled: conjunctions: set[str] = set() initial_conjunctions: set[str] = set() bound: set[str] = set() + abbreviations: set[str] = set() markers: set[str] = set() words: set[str] = set() glued: set[str] = set() @@ -1251,6 +1277,8 @@ def compile_pool(pool: Pool) -> Compiled: bound.add(t) elif r.category == "relation": markers.add(t) + elif r.kind == "given" and "same_as" in p: + abbreviations.add(t) contested = titles & given - particles out = Compiled( GIVEN_NAME_TITLES=frozenset(given_titles), @@ -1283,6 +1311,10 @@ def compile_pool(pool: Pool) -> Compiled: CONJUNCTIONS=frozenset(conjunctions), CONJUNCTIONS_AMBIGUOUS=frozenset(initial_conjunctions), BOUND_GIVEN_NAMES=frozenset(bound), + # a given name written short (`chas`, Filipino `ma`): opening the + # given part period-marked, it is the given name, which outranks + # rules/H2's shape (#789) + GIVEN_NAME_ABBREVIATIONS=frozenset(abbreviations), MAIDEN_MARKERS=frozenset(markers), SUFFIX_WORDS=frozenset(words), GLUED_HONORIFICS=frozenset(glued),