Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/decisions/salutation-titles.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# salutation-titles — salutations, offices and ranks for the languages TITLES had none for (2026-10-04, #606)

Closes #606 (#662 later added owura and bapak, last bullet). No parser code moves: 41 words join TITLES and one, knt (Knight, beside kt), joins SUFFIX_ACRONYMS with a `Knt` case mask. The class is decided once — a word that addresses or ranks a person and is written in FRONT of the name — and each word is then argued on #vocabulary-collisions C-i, the position test, because TITLES has no ambiguous subset and the only two answers available are ship and do not ship. The per-word lists, grouped by language, are the titles carrying `issue = 606` (`uv run python tools/vocab/query.py --issue 606 --kind title`; one line per file a word is in; the #606 block of nameparser/config/titles.py until 2026-10-18, then TOML comments in nameparser/vocab/und.toml, and since 2026-10-19 (#750) the salutations' entries in their language files, the offices and counts still in und.toml).
Closes #606 (#662 later added owura and bapak, last bullet). No parser code moves: 41 words join TITLES and one, knt (Knight, beside kt), joins SUFFIX_ACRONYMS with a `Knt` case mask. The class is decided once — a word that addresses or ranks a person and is written in FRONT of the name — and each word is then argued on #vocabulary-collisions C-i, the position test, because TITLES has no ambiguous subset and the only two answers available are ship and do not ship. The per-word lists, grouped by language, are the titles carrying `issue = 606` (`uv run python tools/vocab/query.py --issue 606 --kind title`; one line per file a word is in; the #606 block of nameparser/config/titles.py until 2026-10-18, then TOML comments in nameparser/vocab/und.toml, and since 2026-10-19 (#750) the salutations' entries in their language files, and the offices', the noble Counts' and knight's too since the second part of #750 the same day).

- **The graf class, decided (Derek, 2026-10-04).** A word borne as a name, but RARELY as the first word of one, ships anyway, on the precedent of graf, which shipped though Steffi Graf exists. This is a judgment on each word's incidence in the leading position — the position C-i asks about — and not a rule, and it is an EXCEPTION to C-i rather than an application of it: C-i is an iff with no threshold, and herra, batoni, hrabia and kreivi were measured as borne in that position, so C-i says do not ship them (TITLES having no ambiguous subset to mark them in). What the exception adds is a weighing C-i does not make, a small measured incidence against the benefit of the title reading, and it is scoped to TITLES and to the words this entry names. It licenses nothing in the particle or suffix sets, and a later TITLES word that wants it is argued on its own counts. The title claim acts only in front, so most bearers are untouched in the forms they write: `Gladys Knight` and `Knight, Gladys` keep family Knight. The cost falls on a name that BEGINS with the word, the cost `Graf Steffi` already paid: `Greve Anna` reads title Greve, family Anna, and a given name is lost the same way, `Herra Wijaya` reading title Herra, family Wijaya. Declaring `FAMILY_FIRST` does not help, since the title claim is read before the order (`Hrabia Anna` reads title Hrabia under it); the family comma does. Two further limits, both pre-existing: a trailing period makes the word the title by rules/H5 (`Gladys Knight.` reads title `Knight.`, as `Mary Jane King.` does), and every word this entry adds widens that rule's reach, H5 claiming any listed title behind a period. Accepted rather than prevented, and stated at rules/H1's Accepted clause. The issue proposed the class for three words — greve (Scandinavian Count), graaf (Dutch Count), knight. The spot-check the issue asked for (forebears.io, 2026-10-04, worldwide counts rounded) found ten more of its "no known collision" words borne, and Derek extended the decision to all of them rather than excluding them: herra, batoni, ponas, counsel, ponia, the Filipino ginoo, ginang and binibini, and two whose registers often write the surname first, hrabia and kreivi. Each word's counts are on the `note` of its title reading in one of its files (herra's in fi.toml), a note beginning "borne" (`uv run python tools/vocab/query.py --issue 606 --kind title`; this bullet held them until 2026-10-19, #750). Those last two, and the given-name uses of herra and batoni, are the leading-position bearers this class accepts, and were decided on with their counts in hand. The line is NOT reproducible from this record, and that is said rather than hidden: the accepted side was counted, while the excluded side — thiru, puan, pan, rodina, ông/bà — was judged from general knowledge of each word, as #606 drafted it, with no count taken (the count source refused further lookups the same day). Each was judged a COMMON leading-position name, against counts in the hundreds on the accepted side; gróf was excluded for want of any evidence at all. Each refusal is an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`). Whoever decides κόμης or the next salutation should count the excluded words first, and is free to find the line in a different place. Those notes are where a sweep meets the decision as it meets the word (the `# borne` comment in titles.py until 2026-10-18).
- **Accents are not folded, and two words depend on it.** `_lexicon._normalize` lowercases and composes NFC and does nothing else to letters, so paní and härra cannot match Pani or Harra (the counts are on their notes). Accent-folding the lexicon would turn both into collisions; whoever proposes it re-argues this entry. frú depends on nothing: the unaccented fru (Swedish/Danish Mrs) already ships, so the Cameroonian surname Fru (its count on fru's note in sv.toml) already reads as a title in `Fru Anna` and did before this entry; that pre-existing collision is recorded here rather than argued, being no part of #606.
Expand Down
18 changes: 14 additions & 4 deletions docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,18 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
- **`und.toml` is exempt.** Its entries keep the Python lists' order, which its vetting comments still point along (a neighbour "above" or "below"); each category's classification moves its words out, into files that are checked.

- **2026-10-19 #750 — title kinds, and the salutations and given-addressing titles classified (part 1 of #750; kinds approved by Derek in chat, 2026-10-19).** No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. Ten title kinds join the generic one, defined in nameparser/vocab/README.md: `salutation`, `style`, `academic`, `religious`, `nobility`, `military`, `office`, `occupation`, `kinship` and `modifier`. Two lines are drawn on purpose. `academic` and `occupation` divide by FUNCTION, not subject: a form of address (`Dr`, the Egyptian `مهندس`) against a job described before a name (`chemist`, `novelist`), so the descriptive words, the collision-prone half of the old bulk import, sit in one kind any later gate can name. `modifier` is decision 10's parked class (`vice`, `grand`, `his`, `1st`), recorded as data now and behaving as every title does until that rule is designed. A household kind was proposed and folded into `occupation` (it would have held about five words).
- **The hard cases, decided with the kinds.** `prince`, `princess`, `sir` and `dame` are `nobility`, the last two as chivalric ranks, and `lord`, `lady` and `knight` will be when their kinds are assigned; `master` is a `salutation`, the style for a boy that is why it addresses by the given name; `father`, `mother`, `brother` and `sister` are `religious`, the use a title makes of them, while `aunt` and `uncle` are `kinship`; the sheikh cluster is a `style`, used for rulers, elders and scholars alike; `chief` will be `nobility` and `deputy` an `office`, not modifiers, since each stands alone as a title ("Chief Obafemi Awolowo"); the Bengali মোঃ and মোসাঃ are salutations, keeping the note that মোঃ abbreviates a name.
- **The hard cases, decided with the kinds.** `prince`, `princess`, `sir` and `dame` are `nobility`, the last two as chivalric ranks, and `lord`, `lady` and `knight` will be when their kinds are assigned (done in part 2, the same day); `master` is a `salutation`, the style for a boy that is why it addresses by the given name; `father`, `mother`, `brother` and `sister` are `religious`, the use a title makes of them, while `aunt` and `uncle` are `kinship`; the sheikh cluster is a `style`, used for rulers, elders and scholars alike; `chief` will be `nobility` and `deputy` an `office`, not modifiers, since each stands alone as a title ("Chief Obafemi Awolowo"); the Bengali মোঃ and মোসাঃ are salutations, keeping the note that মোঃ abbreviates a name.
- **Gender only on a marked word** (decision 9): a feminine form or one half of a fixed pair (`mr` and `mrs`, `king` and `queen`), and `mx` as `"n"`; an unmarked noun (`actor`, `pope`, `doctor`) records none.
- **Files.** Each reading went to the file of every language whose names it occurs in, a Latin transliteration to its source language's file as #749 put the Latin `bin` in ar.toml (`swami` in hi.toml and bn.toml, the sheikh cluster in ar.toml, `cheikh` in fr.toml too). Six files are new: ak, et, id, ro, sw and vi. The few abbreviations shared across European languages (`dr`, `prof`) will go to `mul.toml` with the rest of their kind; no salutation qualified, each language writing its own.
- **The title keep-outs became exclusions**, in the file of the language whose reading they refuse, with their counts as name readings where counts were taken (malam, bay, shehu): `uv run python tools/vocab/query.py --excluded title`. The per-word counts in decisions/salutation-titles moved onto the `note` of each word's title reading, and the prose points at the query. The reasons the given-addressing comments gave for leaving `lord`, `lady`, `venerable`, `rabbi` and `imam` out of GIVEN_NAME_TITLES are notes on those words in und.toml until their kinds are assigned.
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation --count`, `--kind title --file und --count` (the titles still unclassified), `--excluded title`.
- **Files.** Each reading went to the file of every language whose names it occurs in, a Latin transliteration to its source language's file as #749 put the Latin `bin` in ar.toml (`swami` in hi.toml and bn.toml, the sheikh cluster in ar.toml, `cheikh` in fr.toml too). Six files are new: ak, et, id, ro, sw and vi. The few abbreviations shared across European languages (`dr`, `prof`) will go to `mul.toml` (done in part 2, the same day) with the rest of their kind; no salutation qualified, each language writing its own.
- **The title keep-outs became exclusions**, in the file of the language whose reading they refuse, with their counts as name readings where counts were taken (malam, bay, shehu): `uv run python tools/vocab/query.py --excluded title`. The per-word counts in decisions/salutation-titles moved onto the `note` of each word's title reading, and the prose points at the query. The reasons the given-addressing comments gave for leaving `lord`, `lady`, `venerable`, `rabbi` and `imam` out of GIVEN_NAME_TITLES are notes on those words in und.toml until their kinds are assigned (they moved with them in part 2, the same day).
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation --count`, `--kind title --file und --count` (the titles still in `und.toml`, all of them generic until part 2 emptied it the same day, leaving one generic title, `wm`, in en.toml; `--kind title` names the whole category, so no query counts the generic kind alone), `--excluded title`.

- **2026-10-19 #750 — the rest of the titles classified (part 2 of #750).** No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The 644 titles still generic in `und.toml` on e943665e took their kinds, all but `wm` (below), and left for the files of the languages whose names they occur in, so `und.toml` holds no title. Part 1's hard cases are applied as decided there: `lord`, `lady`, `knight` and `chief` are `nobility`, `deputy` an `office`, and `dr` and `prof` are in mul.toml.
- **English holds most of them**: the bulk import's occupations, the US military ranks and their abbreviations, the offices, and the modifiers. `modifier` takes what is a title only inside a phrase: the qualifiers of ranks and offices (`vice`, `rear`, `petty`, `field`), the domain words of the descriptive titles (`baseball`, `software`, `tax`), the ordinals, and the court words of #9's list of judges' titles (`designated`, `docket`, `pro`, `edmi`).
- **Address, not subject, decides `academic`.** Besides the doctors and professors of every script, it takes the English words used to address a professional (`advocate`, `attorney`, `engineer`, `nurse`, and `md` as the medical degree written before a name), the German abbreviations written before a name (`ra`, Rechtsanwalt; `wp`, Wirtschaftsprüfer; `pd` and `priv.-doz`; `dipl.-ing`), and the Devanagari `पंडित` and `पं`, a scholar's title, as part 1's refusal of the Latin `pandit` already keyed it.
- **`wm` keeps the generic kind**, in en.toml: it is William's abbreviation, made a title for #68 so that `Wm. Scott Magargee III` would not read given `Wm.`, and no title kind describes it. Removing it moves a parse, though only of the bare spelling: `Wm Smith` reads title `Wm` with the word listed and given `Wm` without it, while the period-abbreviation shape rule takes `Wm.` either way (measured 2026-10-19 with `Lexicon.default().remove(titles={'wm'})`). So it is left to its own change.
- **mul.toml** takes `docent`, `doctor` and `professor` beside `dr` and `prof`: full words spelled alike in several European languages, listed once rather than in a file per language.
- **A transliteration goes to its source language**, as in part 1: the Islamic titles to ar.toml (`imam`, `mufti`, `mahdi`, `murshid`, `hajji`, `vizier`), or to fa.toml where the Latin spelling is the Persian form (`ayatollah`, `akhoond`, `mullah`, `pir`); `rabbi` to he.toml and `rebbe` to yi.toml; `maharajah`, `maharani` and `subedar` to hi.toml; `giani` and `gyani` to pa.toml; `tsar` and `tsarina` to ru.toml; `effendi` to tr.toml; `vardapet` to hy.toml. The Buddhist, Jain and Zoroastrian titles (`arhat`, `bodhisattva`, `buddha`, `siddha`, `tirthankar`, `saoshyant`) are in en.toml: no source-language name is written with these Latin spellings, which are English loanwords. Four files are new: bin (Edo, `edohen` and `ekegbian`), ff (Fula, `lamido`), mi (Māori, `rangatira`) and yi.
- **Gender** follows part 1's rule. The man's title of a noble pair is marked (`count`, `baron`, `graf`) as `king` is, and so is a `-man` word whose `-woman` pair the list holds (`businessman`); a descriptive noun beside its `-ess` form marks only the feminine (`actress`, `heiress`). `sultan` records none, Ottoman usage having given it to imperial women too (its note in tr.toml).
- **Comments became fields.** The #269, #343 and #344 block comments over the native-script titles are notes on one entry of each group (the Hebrew quote-boundary probe on `ד"ר`, the Bengali doctor split on `ড`); `duke`'s one-line keep-out is an exclusion in en.toml with its counts; the notes part 1 left on `lord`, `lady`, `venerable`, `rabbi` and `imam` moved with them. The readings `und.toml` shares with a moved title, all credentials (`cpl`, `cpo`, `cpt`, `csm`, `lt`, `md`, `ra`, `sa`, `sgm`, `vc`, and `ms` from part 1), sit in a section of their own, as #749's did.
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind title --file und --count`, `--file en --kind modifier --count`, `--file mul`.
1 change: 1 addition & 0 deletions docs/design/mechanisms.md
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')"
- Release notes describe intent; TAGS are ground truth. Before trusting a changelog cross-reference about what shipped where, run `git diff <tagA> <tagB> -- <file>` — the "1.2.1 pickle shim" claim recurred twice from a changelog line describing a thing that never shipped.
- Never pipe a check's output when its exit code matters. Under zsh, `snapshot.py | tail` makes `$?` tail's status. Redirect to a file and read the file.
- Mutation-test a new guard before believing it, and mutate the thing the guard watches — a survivor usually means the fixture satisfies the invariant for free. A guard over a HAND COPY of a pipeline set has two such holes, and #744's review found both in one test of tools/vocab/build.py's check 2, which copies the tokenizer's sets because the tool cannot import the package it writes. Pinning a copy to its source sees only the classes someone copied: the first test pinned `DIVIDERS` alone, and emoji, which the tokenizer strips under `Policy.strip_emoji`, had no copy and no check, so `dr★` compiled and the test passed. Its replacement swept every codepoint `_tokenize._ignorable` drops but accepted any refusal short of a TOML error, so the space in `dr x` passed because the phrase-title check refused it, and a class check 2 never learned would pass too wherever a later check happened to refuse its test text. Drive the sweep from the pipeline's own predicate, match the refusal by the checked rule's own messages, and record the control (hiding the emoji check behind its pinned constant fails the sweep: `test_every_character_the_tokenizer_ignores_is_refused` in tests/v2/test_vocab.py, which now leaves the space out, a phrase's word separator). The predicate reaches only what it decides, though: the interpunct between East Asian characters (rules/T3) is divided by context in `_tokenize_region`, never by `_ignorable`, so hiding that check passes the sweep (measured in #746's review, 2026-10-19). What holds it is its `MUTATIONS` row (`text-with-an-east-asian-interpunct`, input `先·生`); the `EAST_ASIAN_RANGES` and `INTERPUNCT` pins hold only the copies, and stay green with the check gone. Name such a class where the guard stands, and the test that holds it, rather than let the sweep imply it.
- A claim about what a vocabulary entry DOES is a claim about the parse WITHOUT it, so take that parse (`Parser(lexicon=Lexicon.default().remove(titles={...}))`) and compare: the parse with the entry present shows only that the output is consistent with it. #773's first note on `wm` said that removing it would move `Wm. Smith`, read off the parse with the word listed; with the word removed, `Wm. Smith` reads exactly as before, rules/H2's leading-abbreviation title claiming `Wm.` either way, and only the bare `Wm Smith` moves (title `Wm` listed, given `Wm` removed). A shape rule can shadow an entry in one spelling and leave it load-bearing in another, so take the off-switch parse for each spelling, bare and period-marked.
- Verify the restore, not just the mutation: diff against the pre-mutation copy; do not trust the harness's own restore report.
- Run all the gates, not the ones you remember: ruff runs before mypy and pytest in CI, and each has caught what the others passed.
- Purge __pycache__ between same-length source mutations; stale bytecode makes a changed file measure as unchanged.
Expand Down
Loading
Loading