Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/decisions/vocabulary-exclusions.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Excluded (Lexicon.honorific_tails — a glued tail peels only if it could never

Excluded (TITLES):

- The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, and #660's shehu — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error.
- The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, #660's shehu, and #789's wm (2026-10-22) — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error.
- shehu's counts do not settle a graf-class weighing, and must not be read as if they did: its one leading-position figure, the Wikidata given-name count on its entry, is a lower bound (Shagari's own entry has no given-name statement), and its other counts are family-name bearers, a position C-i does not ask about. The evidence that Shehu leads a name is not a count. It is Shagari himself, and CLDR's Hausa sample data, whose six renderings carrying the word all file it as the given name (`grep '"given": "Shehu"' tools/differential/cldr_fields.jsonl`). It arrived with an early bulk import (1a51555f) and was never put to C-i before #660. Whoever re-argues it counts Hausa given-name use first; the accepted cost is the honorific use, `Shehu Abubakar` reading given Shehu as any untitled name does, and the reading belongs in an opt-in pack if anyone asks for it. malam is the opposite case: CLDR's Hausa data files Malam as the title (`grep '"title": "Malam"' tools/differential/cldr_fields.jsonl`), so its refusal is a decline on the counts, not a shehu-style removal.
- The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`).
- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5.
Expand Down
13 changes: 9 additions & 4 deletions docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
- The names later phases will use (`default`, `prefer`, `strict`, the position and attachment keys) are build errors naming their phase, not inert data; locale-pack vocabulary and the Korean surname list stay in Python — 2026-10-18 #737.
- Every language file is in build-checked order (kind, then the `same_as` path from its root head, then the codepoints of the text), the check naming the entry to move, running last, and refusing a `same_as` loop of any length; `und.toml` is exempt — 2026-10-19 #747; 2026-10-21 #762.
- The particles take eight kinds beside the generic one, `expands` carrying a particle's gloss and `note` its vetting, and the particle keep-outs are exclusions — 2026-10-19 #749.
- The titles take ten kinds beside the generic one, `academic` and `occupation` divided by function rather than subject, and the title keep-outs are exclusions; `wm` alone keeps the generic kind — 2026-10-19 #750, title kinds; 2026-10-19 #750, the rest of the titles.
- The titles take ten kinds beside the generic one, `academic` and `occupation` divided by function rather than subject, and the title keep-outs are exclusions — 2026-10-19 #750, title kinds; 2026-10-19 #750, the rest of the titles.
- `wm` is no title: William's abbreviation reads as the given name it is (`Wm Smith` → given 'Wm'), and its title reading is refused in en.toml. The dotted `Wm.` opening the given part is still rules/H2's title; a given reading that outranks H2 belongs to phase 2 (#752) — 2026-10-22 #789.
- Abbreviations and full words spelled alike across European languages (`dr`, `prof`, `doctor`, `professor`, `docent`) are in `mul.toml`; a credential goes to a file only on evidence that names in that language carry it, so none is in `mul.toml` — 2026-10-19 #750, the rest of the titles; 2026-10-20 #751, the credentials.
- `und.toml` holds only `bon` and `dí`, particles of unrecorded origin; every other word, the credentials, conjunctions, bound given names and birth markers included, is in a language file or in `mul.toml` — 2026-10-19 #749; 2026-10-20 #751, the conjunctions.

Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment.

Open: #752 (phase 2, derived ambiguity), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary).
Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given, outranking rules/H2's shape; phase 2's), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary).

## Log

Expand Down Expand Up @@ -100,10 +101,10 @@ Open: #752 (phase 2, derived ambiguity), #753 (phase 3, language packs), #754 (p
- **The title keep-outs became exclusions**, in the file of the language whose reading they refuse, with their counts as name readings where counts were taken (malam, bay, shehu): `uv run python tools/vocab/query.py --excluded title`. The per-word counts in decisions/salutation-titles moved onto the `note` of each word's title reading, and the prose points at the query. The reasons the given-addressing comments gave for leaving `lord`, `lady`, `venerable`, `rabbi` and `imam` out of GIVEN_NAME_TITLES are notes on those words in und.toml until their kinds are assigned (they moved with them in part 2, the same day).
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation --count`, `--kind title --file und --count` (the titles still in `und.toml`, all of them generic until part 2 emptied it the same day, leaving one generic title, `wm`, in en.toml; `--kind title` names the whole category, so no query counts the generic kind alone), `--excluded title`.

- **2026-10-19 #750 — the rest of the titles classified (part 2 of #750).** No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The 644 titles still generic in `und.toml` on e943665e took their kinds, all but `wm` (below), and left for the files of the languages whose names they occur in, so `und.toml` holds no title. Part 1's hard cases are applied as decided there: `lord`, `lady`, `knight` and `chief` are `nobility`, `deputy` an `office`, and `dr` and `prof` are in mul.toml.
- **2026-10-19 #750 — the rest of the titles classified (part 2 of #750)** (its `wm` decision superseded by 2026-10-22 #789, and its measurement corrected in place). No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The 644 titles still generic in `und.toml` on e943665e took their kinds, all but `wm` (below), and left for the files of the languages whose names they occur in, so `und.toml` holds no title. Part 1's hard cases are applied as decided there: `lord`, `lady`, `knight` and `chief` are `nobility`, `deputy` an `office`, and `dr` and `prof` are in mul.toml.
- **English holds most of them**: the bulk import's occupations, the US military ranks and their abbreviations, the offices, and the modifiers. `modifier` takes what is a title only inside a phrase: the qualifiers of ranks and offices (`vice`, `rear`, `petty`, `field`), the domain words of the descriptive titles (`baseball`, `software`, `tax`), the ordinals, and the court words of #9's list of judges' titles (`designated`, `docket`, `pro`, `edmi`).
- **Address, not subject, decides `academic`.** Besides the doctors and professors of every script, it takes the English words used to address a professional (`advocate`, `attorney`, `engineer`, `nurse`, and `md` as the medical degree written before a name), the German abbreviations written before a name (`ra`, Rechtsanwalt; `wp`, Wirtschaftsprüfer; `pd` and `priv.-doz`; `dipl.-ing`), and the Devanagari `पंडित` and `पं`, a scholar's title, as part 1's refusal of the Latin `pandit` already keyed it.
- **`wm` keeps the generic kind**, in en.toml: it is William's abbreviation, made a title for #68 so that `Wm. Scott Magargee III` would not read given `Wm.`, and no title kind describes it. Removing it moves a parse, though only of the bare spelling: `Wm Smith` reads title `Wm` with the word listed and given `Wm` without it, while the period-abbreviation shape rule takes `Wm.` either way (measured 2026-10-19 with `Lexicon.default().remove(titles={'wm'})`). So it is left to its own change.
- **`wm` keeps the generic kind**, in en.toml: it is William's abbreviation, made a title for #68 so that `Wm. Scott Magargee III` would not read given `Wm.`, and no title kind describes it. Removing it moves a parse, though only of the bare spelling: `Wm Smith` reads title `Wm` with the word listed and given `Wm` without it, while the period-abbreviation shape rule takes `Wm.` either way (measured 2026-10-19 with `Lexicon.default().remove(titles={'wm'})`). So it is left to its own change. (Corrected 2026-10-24, #789: the measurement was wrong on its date. It took the off-switch parse at the head of the name only; a trailing `Wm.`, rules/H5's since 2.3.0, and a `Wm.` before a family comma, rules/H6's since 2.5.0, read vocabulary and move with the bare spelling.)
- **mul.toml** takes `docent`, `doctor` and `professor` beside `dr` and `prof`: full words spelled alike in several European languages, listed once rather than in a file per language.
- **A transliteration goes to its source language**, as in part 1: the Islamic titles to ar.toml (`imam`, `mufti`, `mahdi`, `murshid`, `hajji`, `vizier`), or to fa.toml where the Latin spelling is the Persian form (`ayatollah`, `akhoond`, `mullah`, `pir`); `rabbi` to he.toml and `rebbe` to yi.toml; `maharajah`, `maharani` and `subedar` to hi.toml; `giani` and `gyani` to pa.toml; `tsar` and `tsarina` to ru.toml; `effendi` to tr.toml; `vardapet` to hy.toml. The Buddhist, Jain and Zoroastrian titles (`arhat`, `bodhisattva`, `buddha`, `siddha`, `tirthankar`, `saoshyant`) are in en.toml: no source-language name is written with these Latin spellings, which are English loanwords. Four files are new: bin (Edo, `edohen` and `ekegbian`), ff (Fula, `lamido`), mi (Māori, `rangatira`) and yi.
- **Gender** follows part 1's rule. The man's title of a noble pair is marked (`count`, `baron`, `graf`) as `king` is, and so is a `-man` word whose `-woman` pair the list holds (`businessman`); a descriptive noun beside its `-ess` form marks only the feminine (`actress`, `heiress`). `sultan` records none, Ottoman usage having given it to imperial women too (its note in tr.toml).
Expand All @@ -122,3 +123,7 @@ Open: #752 (phase 2, derived ambiguity), #753 (phase 3, language packs), #754 (p
- **Comments became fields.** The block over the single-letter connectives (#383, #479, #397) was class-level, held by decisions/P8 and rules/P3, and is gone; its per-word facts are notes on `e` and `i`, and its keep-outs, `y` and the Cyrillic `и`, `і` and `й`, are `excluded = { may_be_initial = … }` (decisions/vocabulary-exclusions' conjunctions_ambiguous block keeps the class-level part). The #269 blocks over the Cyrillic, Greek and Arabic connectives and the Arabic-script bound given names became an `issue` on each and notes on `і`, `و` and `عبد`; `abd`'s comment became its note in ar.toml, beside the credential note part 1 left in en.toml. Spellings carry `same_as` where one word has several (`abdel` and `abdal` of `abdul`, `abou` of `abu`, `й` of `і`, `nee` of `née`, the е spellings of the ё ones), and gender is recorded where the form marks it (`née` and `né`, `geborene`, `rozená`, the Russian participles).
- **Two keep-outs became data that had been prose.** The Scandinavian abbreviation `f.`, which collides with the initial F., is an exclusion in da.toml. `roz`, removed from the birth markers on 2026-08-26, is an exclusion in cs.toml with its reason, measured again: with it listed, `Rosalind Roz Smith` reads maiden 'Smith' and takes Rosalind for the family, where the 2026-08-26 record says it left no family name at all.
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --file und`, `--kind conjunction`, `--kind bound_given --file ar --count`, `--kind birth_marker`, `--excluded may_be_initial`, `--excluded birth_marker`.

- **2026-10-22 #789 (Derek) — `wm` LEAVES THE TITLES.** SUPERSEDES: the `wm` bullet of 2026-10-19 #750, the rest of the titles. #68 asked that `Wm. Scott Magargee III` not read given name `Wm.`, and `wm` was listed as a title for it in 0.5.5. The request was mistaken: `Wm` is William written short, and a name it opens is a man named William, which a reader of the name would not hesitate over. The case came back from the FEC donor list (`tools/differential/local_corpora.py`, fec-individuals), where bare `WM` opened the given part of 69 names (re-measured 2026-10-24 on 08d130ed), every one read as a title with the given name taken from the next word or left empty. The title reading is refused in en.toml with that reason, so a contributor who adds it back meets it as a build error.
- **The dotted spelling is split by position, not by spelling.** rules/H2's shape reads an unlisted period-marked abbreviation opening the given part as a title, so `Wm. Smith` keeps title 'Wm.' and `Smith, Wm.` too; making `Wm.`, `Chas.`, `Geo.`, `Ma.` or `Jo.` a given name there is a reading that outranks a shape rule, phase 2's question (#752, D14(b) of its spec), and tests/v2/cases.py pins today's `Wm. Smith` so that change shows. Where VOCABULARY reads the word, the dotted spelling moves with the bare one: a trailing `Wm.` was rules/H5's title (`John Smith Wm.` → family 'Wm.', `Smith, John Wm.` → middle 'Wm.', the readings of 1.4.0 through 2.2.0), and a `Wm.` opening the part before a family comma was rules/H6's (`Wm. Smith, John` → family 'Wm. Smith', the reading of 1.4.0 through 2.4.0). The 2026-10-19 bullet said removal moves only the bare spelling, and is corrected in place.
- **Measured** against 08d130ed (`snapshot.py --local` with the base recorded by `--tree`): no committed corpus holds the token `wm`, so the snapshot and the CLDR, GND and Wikidata agreement views could not move and did not; psc-individuals' field agreements do not move either. fec-individuals moves 77 of 1,893,964 names: 69 with bare `WM` opening the given part, 3 with bare `WM` behind another title in it, 3 with a trailing `WM.` and 2 with `WM.` before the family comma. ctgov-officials moves 1 of 352,104, `Wm` behind a title before a family comma.
Loading
Loading