diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 4e1c5645..dc889e96 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -48,3 +48,12 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - **The keep-outs became exclusions**, in the file of the language whose reading they refuse: `ben` and `ال` in ar.toml, `בר` in he.toml, `nic` and unaccented `o` in ga.toml, `bt` and `bte` in ms.toml (`uv run python tools/vocab/query.py --excluded particle`). decisions/vocabulary-exclusions' particle block keeps its class-level bullets and points at the query for the rest. - **Two words keep the generic kind in `und.toml`**: `bon` and `dí`, in the prefix list since the earliest history (2011 and 2012) with no record of which names they come from, and with no particle use found. Both are borne as names (the counts are on their entries); removing them moves a compiled set, so it is left to its own change. - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind particle --count` (readings, one per file a word is in), `--kind clan`, `--file nl --kind particle`. + +- **2026-10-19 #747 — a build-checked word order (check 10).** No compiled set moves: the 41 language files that were out of order on cc030b9c were sorted (an entry moving with any comment above it), `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The order, stated in nameparser/vocab/README.md's Word order section, is kind in `CATEGORY`'s order, then the word's `same_as` path from its root head, then text, so the spellings of one word sit together under it before #750 moves the titles out of `und.toml`. + - **Checked, not rewritten.** `tomllib` only reads, and a comment above an entry belongs to it, which no TOML reader can see; a rewriter would have to round-trip the files' comments. No checked file holds one on 2026-10-19 (every per-entry comment is in `und.toml`), but a word leaving `und.toml` before its category is classified carries its vetting comment with it. So the build refuses a file out of order, naming the entry to move (the first outside a longest run already in order, so a word appended at the end is named rather than its neighbours) and its place against that run, so following the messages takes the fewest moves (#767's second review found the place first given beside a neighbour itself out of place, which took up to 22 moves where 10 would do). + - **Run last.** Check 10 runs after every other check, so a contributor adding a refused word to a language file meets the exclusion's reason (check 6) rather than an order error, the precedence check 6 is placed first for; the first version ran it at the end of loading, ahead of check 6, and the review of #767 found an excluded word appended to en.toml reported as out of order. + - **The same_as path, not the direct target, and the file's own path.** Keyed by the direct target, a spelling of a spelling sorts apart from the word it spells: ar.toml's `ed-` (same_as `el-`, itself same_as `al-`) would follow `ul-`. The path keeps each spelling under its own head inside the root's group, and it follows only the file's own entries, so a word that is a head in one file and a spelling in another does not make one file's order depend on the other (#767's review found the first version, which followed a map taking each word's first file, accepting `jnr` before `jr` in en.toml once de.toml made `jr` a spelling). + - **No same_as loop, over every same_as written.** Check 7 refuses only a same_as naming its own word. Check 10 refuses any longer loop, in every file, `und.toml` included, over every same_as of every reading: no check holds a word to one head across files or kinds, and the first version, which walked one head per word, passed a loop running through a second file's or a second kind's same_as. + - **Codepoint order of the normalized text.** Defined for every script and every machine, and no one language's collation; the cost is that within a script a precomposed accented letter follows `z` (`aḍ-` after `az-`), and that a traditional and a simplified spelling with no `same_as` between them sort apart (`課長`, `课长`). + - **A refusal sorts under the kind it refuses.** Forty-six language-file entries on 2026-10-19 (`senior`, `fils`, the Slavic `ml` and `st`) hold no vocabulary reading, only an exclusion and sometimes the name reading that is its evidence; they sort beside the words they were refused as. + - **`und.toml` is exempt.** Its entries keep the Python lists' order, which its vetting comments still point along ("the bare-κ trap above"); each category's classification moves its words out, into files that are checked. diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 6e70be8d..66fa8b4b 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -79,9 +79,23 @@ Two rules about the `text`: - It is written already normalized: lowercase, NFC, no edge periods, single-spaced. The build refuses any other spelling and prints the one to write. - It is one word, except for a `birth_marker` or a title with `addresses = "given"`, the two readings matched as a whole run (`z domu`). +## Word order + +Every file but `und.toml` lists its words in one order, and the build refuses a file that does not (check 10), naming the entry to move and where it belongs: + +1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`fils`, `senior`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else. +2. **Then by the word it is a spelling of**: a reading's `same_as`, or the word's own text when it has none. A head comes first and its spellings follow it, so `jr` is followed by `jnr` and `junior`, and `sr` by `snr`. A spelling that has spellings of its own heads a group inside its head's: in `ar.toml`, `el-` follows the `a…-` spellings of `al-` with its own assimilated forms after it, and `ul-` comes last. Only the file's own entries are followed, so a word that is a head in one file and a spelling in another is placed in each file by that file alone. +3. **Then by the word's text.** + +Text is compared by codepoint. The texts are already normalized, so this is the order of the normalized text, whatever the script: Latin before Cyrillic before Arabic, and within a script an accented letter after `z` (`aḍ-` follows `az-` and `aš-`). It is defined for every script and the same on every machine, and it is no one language's collation. A traditional and a simplified spelling with no `same_as` between them sort apart (`課長` and `课长` in `zh.toml`). + +The build checks the order and never rewrites a file, because a comment above an entry belongs to it and must move with it, which a TOML reader cannot see. When the build says an entry belongs somewhere else, move it with its comment. The order is checked last, so any other refusal (an exclusion's reason, a `same_as` naming no word) is reported first. The same check refuses a `same_as` chain that loops, in every file. + +`und.toml` is exempt. Its entries keep the order of the Python lists they were converted from, and its vetting comments still point at their neighbours ("the TITLES block below"); a word leaving it for a language file takes its place in that file's order. + ## Kinds -A reading's key is its **kind**. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix and particle categories are classified (two particles of unrecorded origin, `bon` and `dí`, keep the generic kind in `und.toml`); the others carry their generic kind. +A reading's key is its **kind**. The table lists the kinds in their canonical order, which is the [word order](#word-order)'s first key and `CATEGORY`'s order in `tools/vocab/build.py`. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix and particle categories are classified (two particles of unrecorded origin, `bon` and `dí`, keep the generic kind in `und.toml`); the others carry their generic kind. | Kind | Category | Compiled into | |---|---|---| @@ -188,7 +202,7 @@ Record a count on the word as a `given` or `family` reading, with the date you t 1. **Look up the word's other readings.** `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file. An exclusion there is a decision already made; a reading in another category is a clash you are about to create, which is fine when position settles it (`do`) and needs an argument when it does not. 2. **Check it against real names, not memory.** `uv run python tools/name_usage.py ` gives the counts, and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs the matching flag (`may_be_given`, `may_be_name`) or should stay out. The root `AGENTS.md`'s "Adding a word to the vocabulary" lists the traps that have bitten before. -3. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), with what the check found on it: `note`, `issue`, a dated name reading, or an `excluded` entry if you decided against it. +3. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), in its place in the [word order](#word-order), with what the check found on it: `note`, `issue`, a dated name reading, or an `excluded` entry if you decided against it. 4. **Rebuild**: `uv run python tools/vocab/build.py`. It refuses what it can detect, naming the word and, where one file is at fault, the file, and rewrites `nameparser/config/_compiled.py`. Commit both. 5. **Run the tests**: `uv run pytest tests/v2/test_vocab.py` for the build, then the full suite, and read what `tests/v2/test_snapshot.py` reports as moved (the root `AGENTS.md`, "Validating a new parsing rule"). diff --git a/nameparser/vocab/am.toml b/nameparser/vocab/am.toml index 3590abb8..22c20e70 100644 --- a/nameparser/vocab/am.toml +++ b/nameparser/vocab/am.toml @@ -1,9 +1,9 @@ schema = 1 [[word]] -text = "ጁንየር" +text = "ጁኒየር" generation = { issue = 679 } [[word]] -text = "ጁኒየር" +text = "ጁንየር" generation = { issue = 679 } diff --git a/nameparser/vocab/ar.toml b/nameparser/vocab/ar.toml index 8f950f5c..7955b9ad 100644 --- a/nameparser/vocab/ar.toml +++ b/nameparser/vocab/ar.toml @@ -1,16 +1,16 @@ schema = 1 [[word]] -text = "الابن" -generation = { issue = 679, note = "the son" } +text = "الأب" +generation = { issue = 679, note = "the father" } [[word]] text = "الإبن" generation = { issue = 679, note = "the son" } [[word]] -text = "الأب" -generation = { issue = 679, note = "the father" } +text = "الابن" +generation = { issue = 679, note = "the son" } [[word]] text = "جونيور" @@ -21,8 +21,11 @@ text = "سينيور" generation = { issue = 679 } [[word]] -text = "bint" -patronymic = { may_be_given = false, expands = "daughter of; native-script بنت", gender = "f" } +text = "al" +[word.article] +may_be_given = true +expands = "the" +note = "Al Gore: 'Al' is an ordinary English given name. Also the Arabic definite article, which is the particle reading" [[word]] text = "al-" @@ -33,67 +36,67 @@ expands = "the" note = "the Arabic article as an index form writes it, hyphen attached and set behind the given name (\"Rashid, Harun al-\", \"Zaʿīm, Ḥusnī az-\"). Written apart with its hyphen the word is the article and nothing else, so it and its assimilated spellings are never-given where the bare 'al' (Al Gore) may be given. A closed list rather than a shape rule, by decision on #700; ASCII hyphen-minus only" [[word]] -text = "ul-" +text = "ad-" [word.article] may_be_given = false issue = 700 -expands = "the, after a case ending: \"Zia ul-Haq\"" +expands = "the, assimilated before د" same_as = "al-" [[word]] -text = "at-" +text = "adh-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ت" +expands = "the, assimilated before ذ" same_as = "al-" [[word]] -text = "aṯ-" +text = "an-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ث, DMG" +expands = "the, assimilated before ن" same_as = "al-" [[word]] -text = "ath-" +text = "ar-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ث" +expands = "the, assimilated before ر" same_as = "al-" [[word]] -text = "ad-" +text = "as-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before د" +expands = "the, assimilated before س" same_as = "al-" [[word]] -text = "aḏ-" +text = "ash-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ذ, DMG" +expands = "the, assimilated before ش" same_as = "al-" [[word]] -text = "adh-" +text = "at-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ذ" +expands = "the, assimilated before ت" same_as = "al-" [[word]] -text = "ar-" +text = "ath-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ر" +expands = "the, assimilated before ث" same_as = "al-" [[word]] @@ -104,14 +107,6 @@ issue = 700 expands = "the, assimilated before ز" same_as = "al-" -[[word]] -text = "as-" -[word.article] -may_be_given = false -issue = 700 -expands = "the, assimilated before س" -same_as = "al-" - [[word]] text = "aš-" [word.article] @@ -121,27 +116,27 @@ expands = "the, assimilated before ش, DMG" same_as = "al-" [[word]] -text = "ash-" +text = "aḍ-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ش" +expands = "the, assimilated before ض, DMG" same_as = "al-" [[word]] -text = "aṣ-" +text = "aḏ-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ص, DMG" +expands = "the, assimilated before ذ, DMG" same_as = "al-" [[word]] -text = "aḍ-" +text = "aṣ-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ض, DMG" +expands = "the, assimilated before ص, DMG" same_as = "al-" [[word]] @@ -153,19 +148,19 @@ expands = "the, assimilated before ط, DMG" same_as = "al-" [[word]] -text = "aẓ-" +text = "aṯ-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ظ, DMG" +expands = "the, assimilated before ث, DMG" same_as = "al-" [[word]] -text = "an-" +text = "aẓ-" [word.article] may_be_given = false issue = 700 -expands = "the, assimilated before ن" +expands = "the, assimilated before ظ, DMG" same_as = "al-" [[word]] @@ -173,81 +168,121 @@ text = "el-" article = { may_be_given = false, issue = 700, expands = "the, Egyptian spelling", same_as = "al-" } [[word]] -text = "et-" +text = "ed-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ت" +expands = "the, Egyptian spelling assimilated before د" same_as = "el-" [[word]] -text = "eth-" +text = "edh-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ث" +expands = "the, Egyptian spelling assimilated before ذ" same_as = "el-" [[word]] -text = "ed-" +text = "en-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before د" +expands = "the, Egyptian spelling assimilated before ن" same_as = "el-" [[word]] -text = "edh-" +text = "er-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ذ" +expands = "the, Egyptian spelling assimilated before ر" same_as = "el-" [[word]] -text = "er-" +text = "es-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ر" +expands = "the, Egyptian spelling assimilated before س" same_as = "el-" [[word]] -text = "ez-" +text = "esh-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ز" +expands = "the, Egyptian spelling assimilated before ش" same_as = "el-" [[word]] -text = "es-" +text = "et-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before س" +expands = "the, Egyptian spelling assimilated before ت" same_as = "el-" [[word]] -text = "esh-" +text = "eth-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ش" +expands = "the, Egyptian spelling assimilated before ث" same_as = "el-" [[word]] -text = "en-" +text = "ez-" [word.article] may_be_given = false issue = 700 -expands = "the, Egyptian spelling assimilated before ن" +expands = "the, Egyptian spelling assimilated before ز" same_as = "el-" +[[word]] +text = "ul-" +[word.article] +may_be_given = false +issue = 700 +expands = "the, after a case ending: \"Zia ul-Haq\"" +same_as = "al-" + +[[word]] +text = "ال" +[word.excluded.article] +reason = "normally attached to the word it determines, and too ambiguous standing alone (#269)" +issue = 269 + +[[word]] +text = "ben" +[word.excluded.patronymic] +reason = "Ben is a common given name standing exactly where a forward-joining particle claim acts: \"Ahmad Ben Husain\" reads middle 'Ben', and admitting it would take that token into the family. Removed in v0.2.5 and re-affirmed 2026-09-01 (decisions/v1-xfail-triage); a caller who needs the patronymic reading adds it to their own Lexicon. The pressure to re-add is real and recurring: 'Ahmad ben Husain' wants family 'ben Husain', the v1 suite carried that as an xfail for years, and a sweep of Arabic and Hebrew patronymic particles is exactly how it comes back" + +[[word]] +text = "bin" +[word.patronymic] +may_be_given = true +expands = "son of" +gender = "m" +note = "Kept ambiguous deliberately: #269 judged the Latin transliteration separately from the native-script بن, which IS never-given" + +[[word]] +text = "bint" +patronymic = { may_be_given = false, expands = "daughter of; native-script بنت", gender = "f" } + [[word]] text = "ibn" patronymic = { may_be_given = false, expands = "son of; native-script ابن", gender = "m" } +[[word]] +text = "ابن" +[word.patronymic] +may_be_given = false +issue = 269 +expands = "son of: ibn, alternate spelling" +gender = "m" +note = "never a bare given name, like بن; mirrors Latin 'ibn'" + [[word]] text = "بن" [word.patronymic] @@ -266,23 +301,6 @@ expands = "daughter of: bint" gender = "f" note = "never a bare given name, like بن; mirrors Latin 'bint'" -[[word]] -text = "ابن" -[word.patronymic] -may_be_given = false -issue = 269 -expands = "son of: ibn, alternate spelling" -gender = "m" -note = "never a bare given name, like بن; mirrors Latin 'ibn'" - -[[word]] -text = "آل" -[word.clan] -may_be_given = false -issue = 269 -expands = "family of: \"Al Saud\"" -note = "Distinct from the article ال, excluded in this file; a clan prefix, never a bare given name." - [[word]] text = "abu" [word.teknonymic] @@ -292,21 +310,6 @@ expands = "father of" gender = "m" note = "\"Abu Bakr\" -- Abu reads as the given name; also a bound given-name word, so it joins forward (P5)" -[[word]] -text = "al" -[word.article] -may_be_given = true -expands = "the" -note = "Al Gore: 'Al' is an ordinary English given name. Also the Arabic definite article, which is the particle reading" - -[[word]] -text = "bin" -[word.patronymic] -may_be_given = true -expands = "son of" -gender = "m" -note = "Kept ambiguous deliberately: #269 judged the Latin transliteration separately from the native-script بن, which IS never-given" - [[word]] text = "أبو" [word.teknonymic] @@ -327,12 +330,9 @@ gender = "m" note = "left ambiguous as أبو is, both spellings vetted together (#269)" [[word]] -text = "ben" -[word.excluded.patronymic] -reason = "Ben is a common given name standing exactly where a forward-joining particle claim acts: \"Ahmad Ben Husain\" reads middle 'Ben', and admitting it would take that token into the family. Removed in v0.2.5 and re-affirmed 2026-09-01 (decisions/v1-xfail-triage); a caller who needs the patronymic reading adds it to their own Lexicon. The pressure to re-add is real and recurring: 'Ahmad ben Husain' wants family 'ben Husain', the v1 suite carried that as an xfail for years, and a sweep of Arabic and Hebrew patronymic particles is exactly how it comes back" - -[[word]] -text = "ال" -[word.excluded.article] -reason = "normally attached to the word it determines, and too ambiguous standing alone (#269)" +text = "آل" +[word.clan] +may_be_given = false issue = 269 +expands = "family of: \"Al Saud\"" +note = "Distinct from the article ال, excluded in this file; a clan prefix, never a bare given name." diff --git a/nameparser/vocab/as.toml b/nameparser/vocab/as.toml index 8e659966..5396b4af 100644 --- a/nameparser/vocab/as.toml +++ b/nameparser/vocab/as.toml @@ -1,11 +1,13 @@ schema = 1 [[word]] -text = "কনিষ্ঠ" -generation = { issue = 679 } +text = "ক" +[word.excluded.generation] +reason = "the abbreviation ক. is the single-letter initial shape" +issue = 679 [[word]] -text = "জ্যেষ্ঠ" +text = "কনিষ্ঠ" generation = { issue = 679 } [[word]] @@ -13,7 +15,5 @@ text = "জুনিয়ৰ" generation = { issue = 679 } [[word]] -text = "ক" -[word.excluded.generation] -reason = "the abbreviation ক. is the single-letter initial shape" -issue = 679 +text = "জ্যেষ্ঠ" +generation = { issue = 679 } diff --git a/nameparser/vocab/be.toml b/nameparser/vocab/be.toml index 09a252b1..c9fd5af6 100644 --- a/nameparser/vocab/be.toml +++ b/nameparser/vocab/be.toml @@ -1,15 +1,17 @@ schema = 1 [[word]] -text = "малодшы" -generation = { issue = 679 } +text = "мал" +[word.excluded.generation] +reason = "мал. is also \"figure\"" +issue = 679 [[word]] text = "малодшая" generation = { issue = 679 } [[word]] -text = "старэйшы" +text = "малодшы" generation = { issue = 679 } [[word]] @@ -17,7 +19,5 @@ text = "старэйшая" generation = { issue = 679 } [[word]] -text = "мал" -[word.excluded.generation] -reason = "мал. is also \"figure\"" -issue = 679 +text = "старэйшы" +generation = { issue = 679 } diff --git a/nameparser/vocab/bg.toml b/nameparser/vocab/bg.toml index 27dfa7cb..e7794c91 100644 --- a/nameparser/vocab/bg.toml +++ b/nameparser/vocab/bg.toml @@ -1,37 +1,37 @@ schema = 1 [[word]] -text = "младши" -generation = { issue = 661 } +text = "мл" +[word.generation] +issue = 679 +note = "мл.; #679 found nothing it collides with in Cyrillic" [[word]] -text = "младша" -generation = { issue = 661 } +text = "млада" +[word.excluded.generation] +reason = "Mlada is a borne given name, and under a family-first order the given name is the trailing word" +issue = 661 [[word]] -text = "старши" -generation = { issue = 661 } +text = "млади" +generation = { issue = 661, note = "the Younger" } [[word]] -text = "старша" +text = "младша" generation = { issue = 661 } [[word]] -text = "млади" -generation = { issue = 661, note = "the Younger" } +text = "младши" +generation = { issue = 661 } [[word]] text = "стари" generation = { issue = 661, note = "the Elder" } [[word]] -text = "мл" -[word.generation] -issue = 679 -note = "мл.; #679 found nothing it collides with in Cyrillic" +text = "старша" +generation = { issue = 661 } [[word]] -text = "млада" -[word.excluded.generation] -reason = "Mlada is a borne given name, and under a family-first order the given name is the trailing word" -issue = 661 +text = "старши" +generation = { issue = 661 } diff --git a/nameparser/vocab/bn.toml b/nameparser/vocab/bn.toml index 3e04ce4b..70d95286 100644 --- a/nameparser/vocab/bn.toml +++ b/nameparser/vocab/bn.toml @@ -9,18 +9,18 @@ text = "সিনিয়র" generation = { issue = 679 } [[word]] -text = "সাহেব" +text = "বাবু" [word.honorific] issue = [346, 344, 343] -note = "Saheb (রহমান সাহেব)" +note = "Babu, TRAILING (অমল বাবু), while Devanagari बाबू is a LEADING title (#344); the split is real and must not be harmonized" [word.honorific.excluded] attached = "जी ends बनर्जी, मुखर्जी, चटर्जी, so a glued peel would strand a bare virama (बनर् + जी); the 殿 criterion in a non-CJK script (decisions/indic-honorifics)" [[word]] -text = "বাবু" +text = "সাহেব" [word.honorific] issue = [346, 344, 343] -note = "Babu, TRAILING (অমল বাবু), while Devanagari बाबू is a LEADING title (#344); the split is real and must not be harmonized" +note = "Saheb (রহমান সাহেব)" [word.honorific.excluded] attached = "जी ends बनर्जी, मुखर्जी, चटर्जी, so a glued peel would strand a bare virama (बनर् + जी); the 殿 criterion in a non-CJK script (decisions/indic-honorifics)" diff --git a/nameparser/vocab/bs.toml b/nameparser/vocab/bs.toml index 67b475e9..7d28aaa2 100644 --- a/nameparser/vocab/bs.toml +++ b/nameparser/vocab/bs.toml @@ -1,17 +1,17 @@ schema = 1 [[word]] -text = "mlađi" +text = "mlađa" generation = { issue = 661 } [[word]] -text = "mlađa" +text = "mlađi" generation = { issue = 661 } [[word]] -text = "stariji" +text = "starija" generation = { issue = 661 } [[word]] -text = "starija" +text = "stariji" generation = { issue = 661 } diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index d7b00d66..7dd5430d 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -1,21 +1,21 @@ schema = 1 -[[word]] -text = "mladší" -generation = { issue = 661 } - -[[word]] -text = "starší" -generation = { issue = 661 } - [[word]] text = "ml" [word.excluded.generation] reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" issue = [661, 679, 684] +[[word]] +text = "mladší" +generation = { issue = 661 } + [[word]] text = "st" [word.excluded.generation] reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" issue = [661, 679, 684] + +[[word]] +text = "starší" +generation = { issue = 661 } diff --git a/nameparser/vocab/de.toml b/nameparser/vocab/de.toml index 208caf44..6f271e0f 100644 --- a/nameparser/vocab/de.toml +++ b/nameparser/vocab/de.toml @@ -5,25 +5,36 @@ text = "auf" preposition = { may_be_given = false, issue = 18, expands = "upon: \"auf der Heide\"" } [[word]] -text = "dem" -article = { may_be_given = false, issue = 18, expands = "the, dative" } - -[[word]] -text = "der" -article = { may_be_given = false, expands = "the: \"von der Leyen\"" } +text = "tho" +preposition = { may_be_given = true, issue = 18, expands = "to, Low German zu" } [[word]] text = "vom" preposition = { may_be_given = false, expands = "from the, von dem contracted" } +[[word]] +text = "von" +[word.preposition] +may_be_given = true +expands = "of, from" +note = "Von Miller: a real given name, so this stays ambiguous even though \"von Braun\" is misparsed today. The case that shows a word can look like a pure particle, BE misparsed as one, and still have a bearer" + [[word]] text = "zu" preposition = { may_be_given = false, expands = "at, to; nobiliary: \"zu Guttenberg\"" } +[[word]] +text = "dem" +article = { may_be_given = false, issue = 18, expands = "the, dative" } + [[word]] text = "den" article = { may_be_given = true, expands = "the: \"von den Steinen\"" } +[[word]] +text = "der" +article = { may_be_given = false, expands = "the: \"von der Leyen\"" } + [[word]] text = "freiherr" [word.rank] @@ -41,14 +52,3 @@ issue = 18 expands = "baroness" gender = "f" note = "the feminine of freiherr, written and chained the same way (see its note); unlike freiherr it is not in TITLES" - -[[word]] -text = "tho" -preposition = { may_be_given = true, issue = 18, expands = "to, Low German zu" } - -[[word]] -text = "von" -[word.preposition] -may_be_given = true -expands = "of, from" -note = "Von Miller: a real given name, so this stays ambiguous even though \"von Braun\" is misparsed today. The case that shows a word can look like a pure particle, BE misparsed as one, and still have a bearer" diff --git a/nameparser/vocab/el.toml b/nameparser/vocab/el.toml index e98449c4..f1651479 100644 --- a/nameparser/vocab/el.toml +++ b/nameparser/vocab/el.toml @@ -1,9 +1,9 @@ schema = 1 [[word]] -text = "τζούνιορ" +text = "σίνιορ" generation = { issue = 679 } [[word]] -text = "σίνιορ" +text = "τζούνιορ" generation = { issue = 679 } diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index 500e16ac..1522af97 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -1,13 +1,5 @@ schema = 1 -[[word]] -text = "esq" -courtesy = {} - -[[word]] -text = "esquire" -courtesy = { same_as = "esq" } - [[word]] text = "jr" generation = {} @@ -20,6 +12,11 @@ generation = { same_as = "jr" } text = "junior" generation = { same_as = "jr" } +[[word]] +text = "senior" +[word.excluded] +generation = { reason = "Senior is a surname (3,864 US Census bearers)", issue = 679 } + [[word]] text = "sr" generation = {} @@ -39,18 +36,12 @@ text = "vet" status = { issue = 111, note = "bare, not '(vet)', for the reason recorded on ret" } [[word]] -text = "senior" -[word.excluded] -generation = { reason = "Senior is a surname (3,864 US Census bearers)", issue = 679 } +text = "esq" +courtesy = {} [[word]] -text = "mc" -[word.clan] -may_be_given = false -issue = 360 -expands = "son of, Mac contracted" -same_as = "mac" -note = "Unlike 'mac' (Mac Miller) it is not a standalone given name. Measured misparse: \"Mc Donald\" gave given='Mc'. Also in SUFFIX_ACRONYMS as Master of Ceremonies, but that is trailing position and unaffected -- \"John Smith MC\" still reads suffix (#360)." +text = "esquire" +courtesy = { same_as = "esq" } [[word]] text = "mac" @@ -59,6 +50,15 @@ may_be_given = true expands = "son of" note = "Mac Miller: a real given name. This is the case that shows #360's vowel heuristic was not the criterion -- 'mac' and 'mc' differ by attestation, not by shape" +[[word]] +text = "mc" +[word.clan] +may_be_given = false +issue = 360 +expands = "son of, Mac contracted" +same_as = "mac" +note = "Unlike 'mac' (Mac Miller) it is not a standalone given name. Measured misparse: \"Mc Donald\" gave given='Mc'. Also in SUFFIX_ACRONYMS as Master of Ceremonies, but that is trailing position and unaffected -- \"John Smith MC\" still reads suffix (#360)." + [[word]] text = "st" [word.saint] diff --git a/nameparser/vocab/es.toml b/nameparser/vocab/es.toml index 1cfae122..9c96a016 100644 --- a/nameparser/vocab/es.toml +++ b/nameparser/vocab/es.toml @@ -4,6 +4,20 @@ schema = 1 text = "de" preposition = { may_be_given = false, expands = "of" } +[[word]] +text = "del" +[word.preposition] +may_be_given = true +expands = "of the" +note = "BOTH readings attested: Del Shannon (given name) and del Toro (particle). The ambiguity flag is the answer" + +[[word]] +text = "la" +[word.article] +may_be_given = true +expands = "the, fem. sg." +note = "La Shawn, La Toya: a given-name element. The Romance article is the particle reading; its PLURALS ('las', 'los') are never-given, being no one's name" + [[word]] text = "las" [word.article] @@ -20,20 +34,6 @@ issue = [360, 390] expands = "the, masc. pl." note = "Same as 'las' -- and the pair is why the #390 fold narrowing regressed \"de los Santos\" to given='Santos', family='de los'." -[[word]] -text = "del" -[word.preposition] -may_be_given = true -expands = "of the" -note = "BOTH readings attested: Del Shannon (given name) and del Toro (particle). The ambiguity flag is the answer" - -[[word]] -text = "la" -[word.article] -may_be_given = true -expands = "the, fem. sg." -note = "La Shawn, La Toya: a given-name element. The Romance article is the particle reading; its PLURALS ('las', 'los') are never-given, being no one's name" - [[word]] text = "san" [word.saint] diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index de327300..4a4c3230 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -12,13 +12,6 @@ text = "père" [word.excluded] generation = { reason = "pack-shaped: collides outside French", issue = [679, 684] } -[[word]] -text = "de" -[word.preposition] -may_be_given = false -expands = "of" -note = "\"De\" IS a borne Bengali and Odia surname -- trailing, so it never reaches this rule. The case that forced C-i's positional qualifier." - [[word]] text = "d'" [word.preposition] @@ -35,6 +28,20 @@ issue = 700 expands = "the same, typographic apostrophe" same_as = "d'" +[[word]] +text = "de" +[word.preposition] +may_be_given = false +expands = "of" +note = "\"De\" IS a borne Bengali and Odia surname -- trailing, so it never reaches this rule. The case that forced C-i's positional qualifier." + +[[word]] +text = "du" +[word.preposition] +may_be_given = true +expands = "of the, de le contracted" +note = "Du is a Chinese surname (Du Fu), leading under a family-first reading" + [[word]] text = "l'" article = { may_be_given = false, expands = "the, elided: \"L'Enfant\"" } @@ -47,21 +54,6 @@ issue = 700 expands = "the same, typographic apostrophe" same_as = "l'" -[[word]] -text = "ste" -[word.saint] -may_be_given = false -issue = 360 -expands = "Sainte, contracted" -note = "Has a vowel, which is why the criterion is \"abbreviation of a word that is never itself a name\" rather than the vowel shape #360 first proposed. Measured misparse: \"Ste Marie\" gave given='Ste'." - -[[word]] -text = "du" -[word.preposition] -may_be_given = true -expands = "of the, de le contracted" -note = "Du is a Chinese surname (Du Fu), leading under a family-first reading" - [[word]] text = "la" article = { may_be_given = true, expands = "the, fem. sg." } @@ -76,3 +68,11 @@ note = "Lê is among the most common Vietnamese surnames and leads under a famil [[word]] text = "st" saint = { may_be_given = true, expands = "Saint, abbreviated" } + +[[word]] +text = "ste" +[word.saint] +may_be_given = false +issue = 360 +expands = "Sainte, contracted" +note = "Has a vowel, which is why the criterion is \"abbreviation of a word that is never itself a name\" rather than the vowel shape #360 first proposed. Measured misparse: \"Ste Marie\" gave given='Ste'." diff --git a/nameparser/vocab/ga.toml b/nameparser/vocab/ga.toml index d4b9978f..4508375e 100644 --- a/nameparser/vocab/ga.toml +++ b/nameparser/vocab/ga.toml @@ -1,21 +1,31 @@ schema = 1 [[word]] -text = "sóisearach" +text = "sinsearach" [word.excluded.generation] reason = "a rank word, not a generation; Irish writes Óg and Mór" issue = 661 [[word]] -text = "sinsearach" +text = "sóisearach" [word.excluded.generation] reason = "a rank word, not a generation; Irish writes Óg and Mór" issue = 661 +[[word]] +text = "mac" +clan = { may_be_given = true, expands = "son of" } + [[word]] text = "mc" clan = { may_be_given = false, issue = 360, expands = "son of, Mac contracted", same_as = "mac" } +[[word]] +text = "nic" +[word.excluded.clan] +reason = "the Irish \"daughter of\" beside 'ní', but also an abbreviation of Nicolas and Nicole and a given name of its own (Nic Cage): a leading Nic would be marked ambiguous at best, and its home is an opt-in Irish pack" +issue = 604 + [[word]] text = "ní" [word.clan] @@ -26,6 +36,12 @@ gender = "f" cased = "Ní" note = "The female form beside 'ó'; never a given name (#604). Its sibling 'nic' is excluded, in this file. The mask is the word's own title case: Irish writes these capitalized, and a mask outranks the particle lowercase (rules/R4), so the dotted 'n.í.' repairs to 'N.Í.'." +[[word]] +text = "o" +[word.excluded.clan] +reason = "bare O is a common periodless initial, so a particle reading would turn the middle initial of 'Liam O Murchú', and of every 'John O Smith', into a surname; the anglicized form is glued (O'Neil) and needs no particle" +issue = 604 + [[word]] text = "ó" [word.clan] @@ -36,10 +52,6 @@ gender = "m" cased = "Ó" note = "Never a given name (#604). The one ONE-LETTER particle, so the letter written with its period is still the initial it spells -- 'Juan Ó. Pérez' is Óscar, not a surname (rules/P7). Unaccented 'o' is excluded, in this file. The mask is the word's own title case, as on 'ní'." -[[word]] -text = "mac" -clan = { may_be_given = true, expands = "son of" } - [[word]] text = "ua" [word.clan] @@ -49,15 +61,3 @@ expands = "descendant of, the older spelling: \"Seán Ua Buachalla\"" same_as = "ó" cased = "Ua" note = "No given-name use found, but searched less thoroughly than 'ó' and 'ní', and C-i defaults to AMBIGUOUS under uncertainty -- so the leading \"Ua Buachalla\" keeps the given reading and reports it, while mid-name the chain is the same either way (#604). The mask is the word's own title case, as on 'ní': the dotted 'u.a.' repairs to 'U.A.'" - -[[word]] -text = "nic" -[word.excluded.clan] -reason = "the Irish \"daughter of\" beside 'ní', but also an abbreviation of Nicolas and Nicole and a given name of its own (Nic Cage): a leading Nic would be marked ambiguous at best, and its home is an opt-in Irish pack" -issue = 604 - -[[word]] -text = "o" -[word.excluded.clan] -reason = "bare O is a common periodless initial, so a particle reading would turn the middle initial of 'Liam O Murchú', and of every 'John O Smith', into a surname; the anglicized form is glued (O'Neil) and needs no particle" -issue = 604 diff --git a/nameparser/vocab/gd.toml b/nameparser/vocab/gd.toml index 0d84c54f..543af32a 100644 --- a/nameparser/vocab/gd.toml +++ b/nameparser/vocab/gd.toml @@ -1,9 +1,9 @@ schema = 1 -[[word]] -text = "mc" -clan = { may_be_given = false, issue = 360, expands = "son of, Mac contracted", same_as = "mac" } - [[word]] text = "mac" clan = { may_be_given = true, expands = "son of" } + +[[word]] +text = "mc" +clan = { may_be_given = false, issue = 360, expands = "son of, Mac contracted", same_as = "mac" } diff --git a/nameparser/vocab/he.toml b/nameparser/vocab/he.toml index f4c5f628..5dedf059 100644 --- a/nameparser/vocab/he.toml +++ b/nameparser/vocab/he.toml @@ -12,13 +12,19 @@ generation = { issue = 679, note = "U+05F3 geresh" } text = "ג’וניור" generation = { issue = 679, note = "U+2019 apostrophe" } +[[word]] +text = "האב" +generation = { issue = 679, note = "the father" } + [[word]] text = "הבן" generation = { issue = 679, note = "the son" } [[word]] -text = "האב" -generation = { issue = 679, note = "the father" } +text = "סניור" +[word.excluded.generation] +reason = "borne as a surname in Israel (אלי סניור, בוריס סניור)" +issue = 679 [[word]] text = "ז\"ל" @@ -39,10 +45,16 @@ text = "שליט״א" courtesy = { issue = 269, note = "the same, U+05F4 gershayim", same_as = "שליט\"א" } [[word]] -text = "סניור" -[word.excluded.generation] -reason = "borne as a surname in Israel (אלי סניור, בוריס סניור)" -issue = 679 +text = "bar" +[word.patronymic] +may_be_given = true +expands = "son of, Aramaic: \"Bar-Lev\"" +gender = "m" +note = "Bar Refaeli; a common modern Israeli given name. The native-script בר is excluded, in this file" + +[[word]] +text = "bat" +patronymic = { may_be_given = true, expands = "daughter of; native-script בת", gender = "f" } [[word]] text = "בן" @@ -53,6 +65,12 @@ expands = "son of: ben" gender = "m" note = "No Latin-script collision, and not a standalone given name in Hebrew usage." +[[word]] +text = "בר" +[word.excluded.patronymic] +reason = "Bar is a common modern Israeli given name, and the surname spelling is hyphenated anyway (Bar-Lev)" +issue = 269 + [[word]] text = "בת" [word.patronymic] @@ -61,21 +79,3 @@ issue = 269 expands = "daughter of: bat" gender = "f" note = "no Latin-script collision, and not a standalone given name in Hebrew usage, as בן" - -[[word]] -text = "bar" -[word.patronymic] -may_be_given = true -expands = "son of, Aramaic: \"Bar-Lev\"" -gender = "m" -note = "Bar Refaeli; a common modern Israeli given name. The native-script בר is excluded, in this file" - -[[word]] -text = "bat" -patronymic = { may_be_given = true, expands = "daughter of; native-script בת", gender = "f" } - -[[word]] -text = "בר" -[word.excluded.patronymic] -reason = "Bar is a common modern Israeli given name, and the surname spelling is hyphenated anyway (Bar-Lev)" -issue = 269 diff --git a/nameparser/vocab/hi.toml b/nameparser/vocab/hi.toml index f21ed3da..328acce1 100644 --- a/nameparser/vocab/hi.toml +++ b/nameparser/vocab/hi.toml @@ -1,11 +1,11 @@ schema = 1 [[word]] -text = "जूनियर" +text = "जुनियर" generation = { issue = 661, note = "junior, one of its two u spellings" } [[word]] -text = "जुनियर" +text = "जूनियर" generation = { issue = 661, note = "junior, one of its two u spellings" } [[word]] diff --git a/nameparser/vocab/hr.toml b/nameparser/vocab/hr.toml index 357591a1..d37a641c 100644 --- a/nameparser/vocab/hr.toml +++ b/nameparser/vocab/hr.toml @@ -1,29 +1,29 @@ schema = 1 [[word]] -text = "mlađi" -generation = { issue = 661 } +text = "ml" +[word.excluded.generation] +reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" +issue = [661, 679, 684] [[word]] text = "mlađa" generation = { issue = 661 } [[word]] -text = "stariji" -generation = { issue = 661 } - -[[word]] -text = "starija" +text = "mlađi" generation = { issue = 661 } -[[word]] -text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] - [[word]] text = "st" [word.excluded.generation] reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" issue = [661, 679, 684] + +[[word]] +text = "starija" +generation = { issue = 661 } + +[[word]] +text = "stariji" +generation = { issue = 661 } diff --git a/nameparser/vocab/hu.toml b/nameparser/vocab/hu.toml index c53c2e4a..214cd17e 100644 --- a/nameparser/vocab/hu.toml +++ b/nameparser/vocab/hu.toml @@ -1,13 +1,13 @@ schema = 1 [[word]] -text = "ifj" +text = "id" [word.excluded.generation] -reason = "written BEFORE the name (ifj. Kovács János), which H2's shape rule already reads as a title" +reason = "written before the name, and a bare id would make any trailing ID a suffix" issue = 661 [[word]] -text = "id" +text = "ifj" [word.excluded.generation] -reason = "written before the name, and a bare id would make any trailing ID a suffix" +reason = "written BEFORE the name (ifj. Kovács János), which H2's shape rule already reads as a title" issue = 661 diff --git a/nameparser/vocab/hy.toml b/nameparser/vocab/hy.toml index 234d136d..f8db8151 100644 --- a/nameparser/vocab/hy.toml +++ b/nameparser/vocab/hy.toml @@ -1,11 +1,11 @@ schema = 1 -[[word]] -text = "կրտսեր" -generation = { issue = 679 } - [[word]] text = "ավագ" [word.excluded.generation] reason = "an Armenian given name, in lists that write the surname first" issue = 679 + +[[word]] +text = "կրտսեր" +generation = { issue = 679 } diff --git a/nameparser/vocab/is.toml b/nameparser/vocab/is.toml index cdea4953..5df0594c 100644 --- a/nameparser/vocab/is.toml +++ b/nameparser/vocab/is.toml @@ -1,9 +1,9 @@ schema = 1 [[word]] -text = "yngri" +text = "eldri" generation = { issue = 679 } [[word]] -text = "eldri" +text = "yngri" generation = { issue = 679 } diff --git a/nameparser/vocab/it.toml b/nameparser/vocab/it.toml index be00c4c3..59293769 100644 --- a/nameparser/vocab/it.toml +++ b/nameparser/vocab/it.toml @@ -1,5 +1,25 @@ schema = 1 +[[word]] +text = "d'" +preposition = { may_be_given = false, issue = 700, expands = "of, elided di" } + +[[word]] +text = "d’" +[word.preposition] +may_be_given = false +issue = 700 +expands = "the same, typographic apostrophe" +same_as = "d'" + +[[word]] +text = "da" +preposition = { may_be_given = true, expands = "from, of" } + +[[word]] +text = "dal" +preposition = { may_be_given = true, expands = "from the" } + [[word]] text = "de" preposition = { may_be_given = false, expands = "of" } @@ -17,16 +37,16 @@ expands = "the same, typographic apostrophe" same_as = "de'" [[word]] -text = "d'" -preposition = { may_be_given = false, issue = 700, expands = "of, elided di" } +text = "degli" +preposition = { may_be_given = false, expands = "of the, masc. pl." } [[word]] -text = "d’" -[word.preposition] -may_be_given = false -issue = 700 -expands = "the same, typographic apostrophe" -same_as = "d'" +text = "dei" +preposition = { may_be_given = false, expands = "of the, masc. pl." } + +[[word]] +text = "del" +preposition = { may_be_given = true, expands = "of the" } [[word]] text = "dell'" @@ -41,24 +61,11 @@ expands = "the same, typographic apostrophe" same_as = "dell'" [[word]] -text = "l'" -article = { may_be_given = false, expands = "the, elided" } - -[[word]] -text = "l’" -[word.article] -may_be_given = false -issue = 700 -expands = "the same, typographic apostrophe" -same_as = "l'" - -[[word]] -text = "degli" -preposition = { may_be_given = false, expands = "of the, masc. pl." } - -[[word]] -text = "dei" -preposition = { may_be_given = false, expands = "of the, masc. pl." } +text = "della" +[word.preposition] +may_be_given = true +expands = "of the, fem. sg." +note = "Della Reese: an ordinary given name" [[word]] text = "delle" @@ -72,25 +79,6 @@ preposition = { may_be_given = false, expands = "of the, regional variant of dei text = "dello" preposition = { may_be_given = false, expands = "of the, masc. sg." } -[[word]] -text = "da" -preposition = { may_be_given = true, expands = "from, of" } - -[[word]] -text = "dal" -preposition = { may_be_given = true, expands = "from the" } - -[[word]] -text = "del" -preposition = { may_be_given = true, expands = "of the" } - -[[word]] -text = "della" -[word.preposition] -may_be_given = true -expands = "of the, fem. sg." -note = "Della Reese: an ordinary given name" - [[word]] text = "di" [word.preposition] @@ -98,6 +86,18 @@ may_be_given = true expands = "of" note = "BOTH: Di as a short form of Diana, and the Italian particle (DiCaprio)" +[[word]] +text = "l'" +article = { may_be_given = false, expands = "the, elided" } + +[[word]] +text = "l’" +[word.article] +may_be_given = false +issue = 700 +expands = "the same, typographic apostrophe" +same_as = "l'" + [[word]] text = "la" article = { may_be_given = true, expands = "the, fem. sg." } diff --git a/nameparser/vocab/ja.toml b/nameparser/vocab/ja.toml index 9fc44505..6dec2744 100644 --- a/nameparser/vocab/ja.toml +++ b/nameparser/vocab/ja.toml @@ -1,15 +1,32 @@ schema = 1 +[[word]] +text = "シニア" +[word.excluded.generation] +reason = "left out (Derek, 2026-10-14) beside its Hebrew twin; its only count is Wikidata's Senior surname label, 96 bearers, which measures the English surname" +issue = 679 + [[word]] text = "ジュニア" generation = { issue = 679 } [[word]] -text = "先生" -[word.professional_honorific] -issue = [307, 308] -note = "zh Mr. / ja teacher-master, an honorific in both" -attached = { side = "after", issue = 308 } +text = "二世" +[word.generation] +issue = 671 +note = "the generational numeral \"II\"" +[word.generation.attached] +side = "after" +issue = 707 +note = "#707's one ends-with hit, 厄德二世, is the given name Odo with a regnal numeral in its label" + +[[word]] +text = "医学博士" +[word.credential] +acronym = false +issue = 671 +note = "Doctor of Medicine" +attached = { side = "after", issue = 707 } [[word]] text = "博士" @@ -20,39 +37,31 @@ note = "doctorate holder, shared Han; an attested Japanese given name (ひろし excluded = { attached = "glued 田中博士 IS Tanaka Hiroshi, an attested given name" } [[word]] -text = "教授" -[word.professional_honorific] -issue = [307, 308] -note = "professor, shared Han" -attached = { side = "after", issue = 308 } - -[[word]] -text = "様" +text = "くん" [word.honorific] issue = [307, 308] -note = "formal Mr./Ms. (the mail-addressing honorific)" +note = "the kana spelling of 君, which ships where 君 cannot: 王君 is a complete Chinese name, while hiragana is unavailable to Chinese" attached = { side = "after", issue = 308 } [[word]] -text = "氏" +text = "さん" [word.honorific] issue = [307, 308] -note = "news-style Mr. (田中氏); a rare Japanese surname reading exists, but a 氏-surnamed person writes it FIRST" -excluded = { attached = "王氏 is a historical name form (\"the Wang woman\")" } +note = "the everyday honorific, kana" +attached = { side = "after", issue = 308 } [[word]] -text = "殿" +text = "ちゃん" [word.honorific] issue = [307, 308] -note = "formal, official/rank-flavored; spaced is safe because a 殿-surnamed person's name LEADS and the suffix gate is trailing-only" -[word.honorific.excluded] -attached = "some ninety Japanese surnames end in it, 鵜殿 (Udono) and 真殿 (Madono) with four-figure populations, so a peel would cut a family name in two" +note = "familiar/diminutive" +attached = { side = "after", issue = 308 } [[word]] -text = "さん" +text = "様" [word.honorific] issue = [307, 308] -note = "the everyday honorific, kana" +note = "formal Mr./Ms. (the mail-addressing honorific)" attached = { side = "after", issue = 308 } [[word]] @@ -64,35 +73,26 @@ same_as = "様" attached = { side = "after", issue = 308 } [[word]] -text = "くん" +text = "殿" [word.honorific] issue = [307, 308] -note = "the kana spelling of 君, which ships where 君 cannot: 王君 is a complete Chinese name, while hiragana is unavailable to Chinese" -attached = { side = "after", issue = 308 } +note = "formal, official/rank-flavored; spaced is safe because a 殿-surnamed person's name LEADS and the suffix gate is trailing-only" +[word.honorific.excluded] +attached = "some ninety Japanese surnames end in it, 鵜殿 (Udono) and 真殿 (Madono) with four-figure populations, so a peel would cut a family name in two" [[word]] -text = "ちゃん" +text = "氏" [word.honorific] issue = [307, 308] -note = "familiar/diminutive" -attached = { side = "after", issue = 308 } - -[[word]] -text = "二世" -[word.generation] -issue = 671 -note = "the generational numeral \"II\"" -[word.generation.attached] -side = "after" -issue = 707 -note = "#707's one ends-with hit, 厄德二世, is the given name Odo with a regnal numeral in its label" +note = "news-style Mr. (田中氏); a rare Japanese surname reading exists, but a 氏-surnamed person writes it FIRST" +excluded = { attached = "王氏 is a historical name form (\"the Wang woman\")" } [[word]] -text = "議員" +text = "先生" [word.professional_honorific] -issue = 671 -note = "legislator" -attached = { side = "after", issue = 707 } +issue = [307, 308] +note = "zh Mr. / ja teacher-master, an honorific in both" +attached = { side = "after", issue = 308 } [[word]] text = "国会議員" @@ -102,12 +102,11 @@ note = "Diet member" attached = { side = "after", issue = 707 } [[word]] -text = "医学博士" -[word.credential] -acronym = false -issue = 671 -note = "Doctor of Medicine" -attached = { side = "after", issue = 707 } +text = "教授" +[word.professional_honorific] +issue = [307, 308] +note = "professor, shared Han" +attached = { side = "after", issue = 308 } [[word]] text = "社長" @@ -117,22 +116,23 @@ note = "president (ja), society head (zh); a job title used as a form of address attached = { side = "after", issue = 707 } [[word]] -text = "部長" +text = "課長" [word.professional_honorific] issue = 671 -note = "department head (ja), minister (zh); a form of address after a name" -[word.professional_honorific.excluded] -attached = "長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). The measurement found no such bearer, as it found none for the words that shipped; the refusal rests on the argument, which a lower-bound zero does not refute, so a re-measure that again comes back zero is no reason to lift it" +note = "section chief; a form of address after a name" +attached = { side = "after", issue = 707 } [[word]] -text = "課長" +text = "議員" [word.professional_honorific] issue = 671 -note = "section chief; a form of address after a name" +note = "legislator" attached = { side = "after", issue = 707 } [[word]] -text = "シニア" -[word.excluded.generation] -reason = "left out (Derek, 2026-10-14) beside its Hebrew twin; its only count is Wikidata's Senior surname label, 96 bearers, which measures the English surname" -issue = 679 +text = "部長" +[word.professional_honorific] +issue = 671 +note = "department head (ja), minister (zh); a form of address after a name" +[word.professional_honorific.excluded] +attached = "長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). The measurement found no such bearer, as it found none for the words that shipped; the refusal rests on the argument, which a lower-bound zero does not refute, so a re-measure that again comes back zero is no reason to lift it" diff --git a/nameparser/vocab/km.toml b/nameparser/vocab/km.toml index a7a072a3..ad7e2258 100644 --- a/nameparser/vocab/km.toml +++ b/nameparser/vocab/km.toml @@ -1,11 +1,11 @@ schema = 1 -[[word]] -text = "ជូនៀរ" -generation = { issue = 679 } - [[word]] text = "ជូនៀ" [word.excluded.generation] reason = "a given name; ជូនៀរ ships" issue = 679 + +[[word]] +text = "ជូនៀរ" +generation = { issue = 679 } diff --git a/nameparser/vocab/ko.toml b/nameparser/vocab/ko.toml index 0464360f..d6b0fdfd 100644 --- a/nameparser/vocab/ko.toml +++ b/nameparser/vocab/ko.toml @@ -1,19 +1,12 @@ schema = 1 -[[word]] -text = "주니어" -generation = { issue = 679 } - [[word]] text = "시니어" generation = { issue = 679 } [[word]] -text = "씨" -[word.honorific] -issue = [307, 308] -note = "Mr./Ms.; standardly spaced in Korean orthography" -attached = { side = "after", issue = 308 } +text = "주니어" +generation = { issue = 679 } [[word]] text = "박사" @@ -27,38 +20,37 @@ issue = 308 note = "safe glued where its Han twin 博士 is not: that collision is Japanese (博士 = ひろし)" [[word]] -text = "박사님" -[word.professional_honorific] -issue = [307, 308] -note = "the same, honorific form; 선생님, 교수님 and 박사님 are the standard professional honorifics and ship together" -[word.professional_honorific.attached] -side = "after" +text = "교수" +[word.excluded.honorific] +reason = "reads as a common noun as readily as an address term; 교수님 ships" issue = 308 -note = "peels off 김민준박사님 whole rather than stranding 박사 in the given name" [[word]] -text = "선생님" -[word.professional_honorific] +text = "군" +[word.honorific] issue = [307, 308] -note = "teacher/respected elder" -[word.professional_honorific.attached] -side = "after" -issue = 308 -note = "longest-first peels 선생님 off 김선생님 whole, rather than leaving 김선생 to segment into family 김 and given 선생" +note = "young man (\"Master\"); as a single-syllable given name in final position vanishingly rare" +excluded = { attached = "김지군 is a given name ending in it" } [[word]] -text = "교수님" -[word.professional_honorific] +text = "님" +[word.honorific] issue = [307, 308] -note = "professor (honorific form)" +note = "the bare honorific of online/formal address; a single hangul syllable, the 양/군 risk class, but with no hanja reading it cannot sit inside a Sino-Korean given name" attached = { side = "after", issue = 308 } [[word]] -text = "군" +text = "선생" +[word.excluded.honorific] +reason = "reads as a common noun as readily as an address term; 선생님 ships" +issue = 308 + +[[word]] +text = "씨" [word.honorific] issue = [307, 308] -note = "young man (\"Master\"); as a single-syllable given name in final position vanishingly rare" -excluded = { attached = "김지군 is a given name ending in it" } +note = "Mr./Ms.; standardly spaced in Korean orthography" +attached = { side = "after", issue = 308 } [[word]] text = "양" @@ -69,20 +61,28 @@ note = "young woman (\"Miss\"); also a top-tier surname (Yang), but a surname LE attached = "김지양 is a given name ending in it, and 양 is a top-tier surname besides" [[word]] -text = "님" -[word.honorific] +text = "교수님" +[word.professional_honorific] issue = [307, 308] -note = "the bare honorific of online/formal address; a single hangul syllable, the 양/군 risk class, but with no hanja reading it cannot sit inside a Sino-Korean given name" +note = "professor (honorific form)" attached = { side = "after", issue = 308 } [[word]] -text = "선생" -[word.excluded.honorific] -reason = "reads as a common noun as readily as an address term; 선생님 ships" +text = "박사님" +[word.professional_honorific] +issue = [307, 308] +note = "the same, honorific form; 선생님, 교수님 and 박사님 are the standard professional honorifics and ship together" +[word.professional_honorific.attached] +side = "after" issue = 308 +note = "peels off 김민준박사님 whole rather than stranding 박사 in the given name" [[word]] -text = "교수" -[word.excluded.honorific] -reason = "reads as a common noun as readily as an address term; 교수님 ships" +text = "선생님" +[word.professional_honorific] +issue = [307, 308] +note = "teacher/respected elder" +[word.professional_honorific.attached] +side = "after" issue = 308 +note = "longest-first peels 선생님 off 김선생님 whole, rather than leaving 김선생 to segment into family 김 and given 선생" diff --git a/nameparser/vocab/lt.toml b/nameparser/vocab/lt.toml index c15e50b8..5c8a23d7 100644 --- a/nameparser/vocab/lt.toml +++ b/nameparser/vocab/lt.toml @@ -1,15 +1,17 @@ schema = 1 [[word]] -text = "jaunesnysis" -generation = { issue = 661 } +text = "jaun" +[word.excluded.generation] +reason = "the abbreviation jaun., refused with the full words shipping" +issue = 679 [[word]] text = "jaunesnioji" generation = { issue = 661 } [[word]] -text = "vyresnysis" +text = "jaunesnysis" generation = { issue = 661 } [[word]] @@ -17,7 +19,5 @@ text = "vyresnioji" generation = { issue = 661 } [[word]] -text = "jaun" -[word.excluded.generation] -reason = "the abbreviation jaun., refused with the full words shipping" -issue = 679 +text = "vyresnysis" +generation = { issue = 661 } diff --git a/nameparser/vocab/mr.toml b/nameparser/vocab/mr.toml index d829bdf3..4f55fa96 100644 --- a/nameparser/vocab/mr.toml +++ b/nameparser/vocab/mr.toml @@ -1,5 +1,11 @@ schema = 1 +[[word]] +text = "क" +[word.excluded.generation] +reason = "the abbreviation क. is the single-letter initial shape" +issue = 679 + [[word]] text = "ज्युनियर" generation = { issue = 679 } @@ -23,9 +29,3 @@ issue = [346, 344, 343] note = "the Marathi Saheb spelling" [word.honorific.excluded] attached = "जी ends बनर्जी, मुखर्जी, चटर्जी, so a glued peel would strand a bare virama (बनर् + जी); the 殿 criterion in a non-CJK script (decisions/indic-honorifics)" - -[[word]] -text = "क" -[word.excluded.generation] -reason = "the abbreviation क. is the single-letter initial shape" -issue = 679 diff --git a/nameparser/vocab/ms.toml b/nameparser/vocab/ms.toml index c485e822..4c24479b 100644 --- a/nameparser/vocab/ms.toml +++ b/nameparser/vocab/ms.toml @@ -5,23 +5,23 @@ text = "bin" patronymic = { may_be_given = true, expands = "son of: \"Ahmad bin Ismail\"", gender = "m" } [[word]] -text = "binte" +text = "binti" [word.patronymic] may_be_given = true issue = 604 -expands = "daughter of, Singapore spelling" -same_as = "binti" +expands = "daughter of: \"Ina binti Navalamar\"" gender = "f" -note = "kept beside 'binti' (#604)" +note = "The female partner of 'bin'. Binti is also a given name (Swahili, \"daughter\"), so ambiguous like 'bin' (#604). The abbreviations 'bt' and 'bte' are excluded, in this file" [[word]] -text = "binti" +text = "binte" [word.patronymic] may_be_given = true issue = 604 -expands = "daughter of: \"Ina binti Navalamar\"" +expands = "daughter of, Singapore spelling" +same_as = "binti" gender = "f" -note = "The female partner of 'bin'. Binti is also a given name (Swahili, \"daughter\"), so ambiguous like 'bin' (#604). The abbreviations 'bt' and 'bte' are excluded, in this file" +note = "kept beside 'binti' (#604)" [[word]] text = "bt" diff --git a/nameparser/vocab/nl.toml b/nameparser/vocab/nl.toml index 7dd12f23..be5103bf 100644 --- a/nameparser/vocab/nl.toml +++ b/nameparser/vocab/nl.toml @@ -1,45 +1,5 @@ schema = 1 -[[word]] -text = "'t" -article = { may_be_given = false, expands = "het, the: \"'t Hooft\"" } - -[[word]] -text = "’t" -[word.article] -may_be_given = false -issue = 700 -expands = "the same, typographic apostrophe" -same_as = "'t" - -[[word]] -text = "de" -article = { may_be_given = false, expands = "the: \"de Vries\"" } - -[[word]] -text = "der" -article = { may_be_given = false, expands = "the: \"van der Berg\"" } - -[[word]] -text = "het" -article = { may_be_given = false, issue = 18, expands = "the" } - -[[word]] -text = "op" -preposition = { may_be_given = false, issue = 18, expands = "at, on: \"op den Berg\"" } - -[[word]] -text = "ter" -preposition = { may_be_given = false, expands = "at the, te der contracted: \"ter Horst\"" } - -[[word]] -text = "vd" -[word.preposition] -may_be_given = false -issue = 18 -expands = "van der, abbreviated" -note = "Also the British Volunteer Decoration, a suffix acronym: two non-name readings, so C-ii decides it on frequency and the Dutch one wins (decisions/vocabulary-collisions)." - [[word]] text = "aan" preposition = { may_be_given = true, issue = 18, expands = "on, at: \"aan den Toorn\"" } @@ -52,16 +12,8 @@ issue = 18 expands = "aan, archaic spelling: \"aen de Landerwer\"" [[word]] -text = "den" -article = { may_be_given = true, expands = "the: \"van den Berg\"" } - -[[word]] -text = "heer" -[word.rank] -may_be_given = true -issue = 18 -expands = "lord, the nobiliary 'heer van'" -note = "added from #18's test \"Johan de heer Van Kampen\" (family 'de heer Van Kampen'), which appears to be two real names run together (Johannes de Heer, a Dutch Protestant figure, and the surname Van Kampen): the string occurs elsewhere only in another name parser's test data (hex.pm full_name_splitter 1.0.3). No real name supports the rank reading, and its may_be_given = true is the unexamined default: nobody has checked Heer as a given name. Removing it moves PARTICLES, so it is left to #766" +text = "op" +preposition = { may_be_given = false, issue = 18, expands = "at, on: \"op den Berg\"" } [[word]] text = "te" @@ -71,6 +23,10 @@ issue = 18 expands = "at: \"te Riele\"" note = "Te Awanui-a-Rangi: a Māori given-name element, and in the corpus. Looks like the Dutch preposition 'te', which is why attestation rather than etymology decides" +[[word]] +text = "ter" +preposition = { may_be_given = false, expands = "at the, te der contracted: \"ter Horst\"" } + [[word]] text = "tho" [word.preposition] @@ -96,3 +52,47 @@ preposition = { may_be_given = true, issue = 18, expands = "van de contracted: \ [[word]] text = "vander" preposition = { may_be_given = true, expands = "van der contracted" } + +[[word]] +text = "vd" +[word.preposition] +may_be_given = false +issue = 18 +expands = "van der, abbreviated" +note = "Also the British Volunteer Decoration, a suffix acronym: two non-name readings, so C-ii decides it on frequency and the Dutch one wins (decisions/vocabulary-collisions)." + +[[word]] +text = "'t" +article = { may_be_given = false, expands = "het, the: \"'t Hooft\"" } + +[[word]] +text = "’t" +[word.article] +may_be_given = false +issue = 700 +expands = "the same, typographic apostrophe" +same_as = "'t" + +[[word]] +text = "de" +article = { may_be_given = false, expands = "the: \"de Vries\"" } + +[[word]] +text = "den" +article = { may_be_given = true, expands = "the: \"van den Berg\"" } + +[[word]] +text = "der" +article = { may_be_given = false, expands = "the: \"van der Berg\"" } + +[[word]] +text = "het" +article = { may_be_given = false, issue = 18, expands = "the" } + +[[word]] +text = "heer" +[word.rank] +may_be_given = true +issue = 18 +expands = "lord, the nobiliary 'heer van'" +note = "added from #18's test \"Johan de heer Van Kampen\" (family 'de heer Van Kampen'), which appears to be two real names run together (Johannes de Heer, a Dutch Protestant figure, and the surname Van Kampen): the string occurs elsewhere only in another name parser's test data (hex.pm full_name_splitter 1.0.3). No real name supports the rank reading, and its may_be_given = true is the unexamined default: nobody has checked Heer as a given name. Removing it moves PARTICLES, so it is left to #766" diff --git a/nameparser/vocab/pt.toml b/nameparser/vocab/pt.toml index 066e6c83..17001aae 100644 --- a/nameparser/vocab/pt.toml +++ b/nameparser/vocab/pt.toml @@ -1,5 +1,9 @@ schema = 1 +[[word]] +text = "da" +preposition = { may_be_given = true, expands = "of the, fem. sg." } + [[word]] text = "das" [word.preposition] @@ -12,14 +16,6 @@ note = "Added #360: its own plural partner 'dos' was already here. \"Das\" IS a text = "de" preposition = { may_be_given = false, expands = "of" } -[[word]] -text = "dos" -preposition = { may_be_given = false, expands = "of the, masc. pl." } - -[[word]] -text = "da" -preposition = { may_be_given = true, expands = "of the, fem. sg." } - [[word]] text = "do" [word.preposition] @@ -27,6 +23,10 @@ may_be_given = true expands = "of the, masc. sg." note = "Vietnamese Đỗ leads a great many surnames" +[[word]] +text = "dos" +preposition = { may_be_given = false, expands = "of the, masc. pl." } + [[word]] text = "santa" saint = { may_be_given = true, expands = "Saint, fem." } diff --git a/nameparser/vocab/ru.toml b/nameparser/vocab/ru.toml index 02a4ee65..f8a933a0 100644 --- a/nameparser/vocab/ru.toml +++ b/nameparser/vocab/ru.toml @@ -1,29 +1,29 @@ schema = 1 [[word]] -text = "младший" -generation = { issue = 661 } +text = "мл" +[word.generation] +issue = 679 +note = "мл.; #679 found nothing it collides with in Cyrillic" [[word]] text = "младшая" generation = { issue = 661 } [[word]] -text = "старший" -generation = { issue = 661 } - -[[word]] -text = "старшая" +text = "младший" generation = { issue = 661 } -[[word]] -text = "мл" -[word.generation] -issue = 679 -note = "мл.; #679 found nothing it collides with in Cyrillic" - [[word]] text = "ст" [word.excluded.generation] reason = "barely attested after a name, and a plausible false positive among two-letter Cyrillic abbreviations; deferred since #269" issue = [269, 661, 679] + +[[word]] +text = "старшая" +generation = { issue = 661 } + +[[word]] +text = "старший" +generation = { issue = 661 } diff --git a/nameparser/vocab/sd.toml b/nameparser/vocab/sd.toml index e5b781b1..c903db62 100644 --- a/nameparser/vocab/sd.toml +++ b/nameparser/vocab/sd.toml @@ -1,7 +1,7 @@ schema = 1 [[word]] -text = "جونیئر" +text = "جونئیر" generation = { issue = 679 } [[word]] @@ -9,7 +9,7 @@ text = "جونيئر" generation = { issue = 679 } [[word]] -text = "جونئیر" +text = "جونیئر" generation = { issue = 679 } [[word]] diff --git a/nameparser/vocab/sk.toml b/nameparser/vocab/sk.toml index 4420b1cb..33b27a14 100644 --- a/nameparser/vocab/sk.toml +++ b/nameparser/vocab/sk.toml @@ -1,29 +1,29 @@ schema = 1 [[word]] -text = "mladší" -generation = { issue = 661 } - -[[word]] -text = "starší" -generation = { issue = 661 } +text = "ml" +[word.excluded.generation] +reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" +issue = [661, 679, 684] [[word]] text = "mladšia" generation = { issue = 679 } [[word]] -text = "staršia" -generation = { issue = 679 } - -[[word]] -text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] +text = "mladší" +generation = { issue = 661 } [[word]] text = "st" [word.excluded.generation] reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" issue = [661, 679, 684] + +[[word]] +text = "staršia" +generation = { issue = 679 } + +[[word]] +text = "starší" +generation = { issue = 661 } diff --git a/nameparser/vocab/sl.toml b/nameparser/vocab/sl.toml index 85dcec33..43716d56 100644 --- a/nameparser/vocab/sl.toml +++ b/nameparser/vocab/sl.toml @@ -1,29 +1,29 @@ schema = 1 [[word]] -text = "mlajši" -generation = { issue = 679 } +text = "ml" +[word.excluded.generation] +reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" +issue = [661, 679, 684] [[word]] text = "mlajša" generation = { issue = 679 } [[word]] -text = "starejši" -generation = { issue = 679 } - -[[word]] -text = "starejša" +text = "mlajši" generation = { issue = 679 } -[[word]] -text = "ml" -[word.excluded.generation] -reason = "pack-shaped: real and standard, but collides with ML initials in other languages written in Latin script" -issue = [661, 679, 684] - [[word]] text = "st" [word.excluded.generation] reason = "pack-shaped: real and standard, but St. is Saint in other languages written in Latin script" issue = [661, 679, 684] + +[[word]] +text = "starejša" +generation = { issue = 679 } + +[[word]] +text = "starejši" +generation = { issue = 679 } diff --git a/nameparser/vocab/sr.toml b/nameparser/vocab/sr.toml index 960eb94c..5508a701 100644 --- a/nameparser/vocab/sr.toml +++ b/nameparser/vocab/sr.toml @@ -1,15 +1,11 @@ schema = 1 -[[word]] -text = "mlađi" -generation = { issue = 661 } - [[word]] text = "mlađa" generation = { issue = 661 } [[word]] -text = "stariji" +text = "mlađi" generation = { issue = 661 } [[word]] @@ -17,15 +13,21 @@ text = "starija" generation = { issue = 661 } [[word]] -text = "млађи" +text = "stariji" generation = { issue = 661 } +[[word]] +text = "мл" +[word.generation] +issue = 679 +note = "мл.; #679 found nothing it collides with in Cyrillic" + [[word]] text = "млађа" generation = { issue = 661 } [[word]] -text = "старији" +text = "млађи" generation = { issue = 661 } [[word]] @@ -33,7 +35,5 @@ text = "старија" generation = { issue = 661 } [[word]] -text = "мл" -[word.generation] -issue = 679 -note = "мл.; #679 found nothing it collides with in Cyrillic" +text = "старији" +generation = { issue = 661 } diff --git a/nameparser/vocab/ta.toml b/nameparser/vocab/ta.toml index aafad877..cd1f69fd 100644 --- a/nameparser/vocab/ta.toml +++ b/nameparser/vocab/ta.toml @@ -5,9 +5,9 @@ text = "இளையவர்" generation = { issue = 679 } [[word]] -text = "மூத்தவர்" +text = "ஜூனியர்" generation = { issue = 679 } [[word]] -text = "ஜூனியர்" +text = "மூத்தவர்" generation = { issue = 679 } diff --git a/nameparser/vocab/ti.toml b/nameparser/vocab/ti.toml index 3590abb8..22c20e70 100644 --- a/nameparser/vocab/ti.toml +++ b/nameparser/vocab/ti.toml @@ -1,9 +1,9 @@ schema = 1 [[word]] -text = "ጁንየር" +text = "ጁኒየር" generation = { issue = 679 } [[word]] -text = "ጁኒየር" +text = "ጁንየር" generation = { issue = 679 } diff --git a/nameparser/vocab/tr.toml b/nameparser/vocab/tr.toml index 56980563..eaf18603 100644 --- a/nameparser/vocab/tr.toml +++ b/nameparser/vocab/tr.toml @@ -1,13 +1,13 @@ schema = 1 -[[word]] -text = "kıdemli" -[word.excluded.generation] -reason = "a rank word (Kıdemli onbaşı); Turkish writes Jr. or a parenthesis, (oğul)" -issue = 661 - [[word]] text = "baba" [word.excluded.generation] reason = "collides inside Turkish, and Turkish puts the junior word before the name" issue = 679 + +[[word]] +text = "kıdemli" +[word.excluded.generation] +reason = "a rank word (Kıdemli onbaşı); Turkish writes Jr. or a parenthesis, (oğul)" +issue = 661 diff --git a/nameparser/vocab/uk.toml b/nameparser/vocab/uk.toml index 679d1e07..6049c9d9 100644 --- a/nameparser/vocab/uk.toml +++ b/nameparser/vocab/uk.toml @@ -1,7 +1,7 @@ schema = 1 [[word]] -text = "старша" +text = "молодша" generation = { issue = 661 } [[word]] @@ -9,5 +9,5 @@ text = "молодший" generation = { issue = 661 } [[word]] -text = "молодша" +text = "старша" generation = { issue = 661 } diff --git a/nameparser/vocab/und.toml b/nameparser/vocab/und.toml index 1528fc36..2bdd89f3 100644 --- a/nameparser/vocab/und.toml +++ b/nameparser/vocab/und.toml @@ -165,7 +165,7 @@ title = { addresses = "given" } # name), bare شيخ (Shaikha is a common female given name), أمير # and سلطان (Amir/Sultan, common given names), and the 'د.' # abbreviation -- edge-period normalization would leave bare 'د', -# which swallows single-letter initials (the bare-κ trap above). +# which swallows single-letter initials (the bare-κ trap below). [[word]] text = "الدكتور" title = { addresses = "given", note = "\"the doctor\" (m)" } diff --git a/nameparser/vocab/ur.toml b/nameparser/vocab/ur.toml index f87aef7c..0007c828 100644 --- a/nameparser/vocab/ur.toml +++ b/nameparser/vocab/ur.toml @@ -1,7 +1,7 @@ schema = 1 [[word]] -text = "جونیئر" +text = "جونئیر" generation = { issue = 679 } [[word]] @@ -9,7 +9,7 @@ text = "جونيئر" generation = { issue = 679 } [[word]] -text = "جونئیر" +text = "جونیئر" generation = { issue = 679 } [[word]] diff --git a/nameparser/vocab/zh.toml b/nameparser/vocab/zh.toml index 79c52ad2..fe47fb2f 100644 --- a/nameparser/vocab/zh.toml +++ b/nameparser/vocab/zh.toml @@ -1,11 +1,44 @@ schema = 1 [[word]] -text = "先生" -[word.professional_honorific] +text = "二世" +[word.generation] +issue = 671 +note = "the generational numeral \"II\"" +[word.generation.attached] +side = "after" +issue = 707 +note = "#707's one ends-with hit, 厄德二世, is the given name Odo with a regnal numeral in its label" + +[[word]] +text = "医学博士" +[word.credential] +acronym = false +issue = 671 +note = "Doctor of Medicine" +attached = { side = "after", issue = 707 } + +[[word]] +text = "博士" +[word.credential] +acronym = false issue = [307, 308] -note = "zh Mr. / ja teacher-master, an honorific in both" -attached = { side = "after", issue = 308 } +note = "doctorate holder, shared Han; an attested Japanese given name (ひろし, Hiroshi), but the doctorate reading vastly dominates the spaced trailing position" +excluded = { attached = "glued 田中博士 IS Tanaka Hiroshi, an attested given name" } + +[[word]] +text = "醫學博士" +[word.credential] +acronym = false +issue = 671 +note = "Doctor of Medicine" +attached = { side = "after", issue = 707 } + +[[word]] +text = "君" +[word.excluded.honorific] +reason = "王君 is a complete Chinese name, 君 being a common given-name final; its kana spelling くん ships" +issue = 308 [[word]] text = "女士" @@ -22,44 +55,12 @@ note = "Miss" attached = { side = "after", issue = 308 } [[word]] -text = "博士" -[word.credential] -acronym = false -issue = [307, 308] -note = "doctorate holder, shared Han; an attested Japanese given name (ひろし, Hiroshi), but the doctorate reading vastly dominates the spaced trailing position" -excluded = { attached = "glued 田中博士 IS Tanaka Hiroshi, an attested given name" } - -[[word]] -text = "教授" +text = "先生" [word.professional_honorific] issue = [307, 308] -note = "professor, shared Han" +note = "zh Mr. / ja teacher-master, an honorific in both" attached = { side = "after", issue = 308 } -[[word]] -text = "二世" -[word.generation] -issue = 671 -note = "the generational numeral \"II\"" -[word.generation.attached] -side = "after" -issue = 707 -note = "#707's one ends-with hit, 厄德二世, is the given name Odo with a regnal numeral in its label" - -[[word]] -text = "議員" -[word.professional_honorific] -issue = 671 -note = "legislator" -attached = { side = "after", issue = 707 } - -[[word]] -text = "议员" -[word.professional_honorific] -issue = 671 -note = "legislator" -attached = { side = "after", issue = 707 } - [[word]] text = "國會議員" [word.professional_honorific] @@ -82,20 +83,11 @@ note = "Justice of the Peace (Hong Kong)" attached = { side = "after", issue = 707 } [[word]] -text = "医学博士" -[word.credential] -acronym = false -issue = 671 -note = "Doctor of Medicine" -attached = { side = "after", issue = 707 } - -[[word]] -text = "醫學博士" -[word.credential] -acronym = false -issue = 671 -note = "Doctor of Medicine" -attached = { side = "after", issue = 707 } +text = "教授" +[word.professional_honorific] +issue = [307, 308] +note = "professor, shared Han" +attached = { side = "after", issue = 308 } [[word]] text = "社長" @@ -105,12 +97,11 @@ note = "president (ja), society head (zh); a job title used as a form of address attached = { side = "after", issue = 707 } [[word]] -text = "部長" +text = "社长" [word.professional_honorific] issue = 671 -note = "department head (ja), minister (zh); a form of address after a name" -[word.professional_honorific.excluded] -attached = "長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). The measurement found no such bearer, as it found none for the words that shipped; the refusal rests on the argument, which a lower-bound zero does not refute, so a re-measure that again comes back zero is no reason to lift it" +note = "simplified; 长 never occurs in Japanese text, so the entry touches no Japanese name" +attached = { side = "after", issue = 707 } [[word]] text = "課長" @@ -120,18 +111,18 @@ note = "section chief; a form of address after a name" attached = { side = "after", issue = 707 } [[word]] -text = "社长" +text = "議員" [word.professional_honorific] issue = 671 -note = "simplified; 长 never occurs in Japanese text, so the entry touches no Japanese name" +note = "legislator" attached = { side = "after", issue = 707 } [[word]] -text = "部长" +text = "议员" [word.professional_honorific] issue = 671 -note = "simplified twin of 部長" -excluded = { attached = "the simplified twin of 部長, declined with it (#707)" } +note = "legislator" +attached = { side = "after", issue = 707 } [[word]] text = "课长" @@ -141,7 +132,16 @@ note = "simplified twin of 課長" attached = { side = "after", issue = 707 } [[word]] -text = "君" -[word.excluded.honorific] -reason = "王君 is a complete Chinese name, 君 being a common given-name final; its kana spelling くん ships" -issue = 308 +text = "部長" +[word.professional_honorific] +issue = 671 +note = "department head (ja), minister (zh); a form of address after a name" +[word.professional_honorific.excluded] +attached = "長 alone is a borne given name, so 阿部長 can be 阿部 + 長 (#707). The measurement found no such bearer, as it found none for the words that shipped; the refusal rests on the argument, which a lower-bound zero does not refute, so a re-measure that again comes back zero is no reason to lift it" + +[[word]] +text = "部长" +[word.professional_honorific] +issue = 671 +note = "simplified twin of 部長" +excluded = { attached = "the simplified twin of 部長, declined with it (#707)" } diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index b331b6b1..f023a550 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -130,6 +130,8 @@ def test_moving_a_word_between_files_changes_nothing(shipped: dict[str, str]) -> def test_an_exclusion_in_one_file_refuses_the_reading_in_another( shipped: dict[str, str]) -> None: files = dict(shipped) + # appended to an ordered file, out of place: the reason still comes + # first, check 10 running last files["en.toml"] += '\n[[word]]\ntext = "fils"\ngeneration = {}\n' with pytest.raises(build.VocabError, match="767 US Census"): _compile(files) @@ -267,6 +269,45 @@ def _w(body: str) -> str: ("elided-particle-without-its-twin", {"und.toml": _w('[[word]]\ntext = "d\'"\nparticle = { may_be_given = false }')}, "U\\+2019 spelling"), ("elided-particle-kind-without-its-twin", {"fr.toml": _w('[[word]]\ntext = "l\'"\narticle = { may_be_given = false }')}, "U\\+2019 spelling"), ("one-kind-acronym-and-word", {"en.toml": _w('[[word]]\ntext = "abc"\ncourtesy = {}'), "und.toml": _w('[[word]]\ntext = "abc"\ncourtesy = { acronym = true }')}, "acronym is"), + # 10: canonical word order. A swapped pair, within a kind and across + # kinds and categories + ("words-out-of-order", {"en.toml": _w('[[word]]\ntext = "sr"\ngeneration = {}\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "'sr' \\(generation\\) belongs after 'jr'"), + ("kinds-out-of-order", {"en.toml": _w('[[word]]\ntext = "esq"\ncourtesy = {}\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "'esq' \\(courtesy\\) belongs after 'jr'"), + ("categories-out-of-order", {"de.toml": _w('[[word]]\ntext = "von"\npreposition = { may_be_given = false }\n\n[[word]]\ntext = "dr"\ntitle = {}')}, "'von' \\(preposition\\) belongs after 'dr'"), + # a same_as variant sorts after its head, not by its own text: in + # plain alphabetical order jnr would come first + ("same-as-variant-before-its-head", {"en.toml": _w('[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "'jnr' \\(generation\\) belongs after 'jr'"), + # and beside it: alphabetically 'younger' would follow 'sr' + ("same-as-variant-away-from-its-head", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "sr"\ngeneration = {}\n\n[[word]]\ntext = "younger"\ngeneration = { same_as = "jr" }')}, "'sr' \\(generation\\) belongs after 'younger'"), + # an entry recording only a refusal sorts under the kind it refuses + ("refusal-out-of-its-kind", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "esq"\ncourtesy = {}\n\n[[word]]\ntext = "senior"\nexcluded = { generation = "x" }')}, "'esq' \\(courtesy\\) belongs after 'senior'"), + ("refusal-first-in-the-file", {"en.toml": _w('[[word]]\ntext = "senior"\nexcluded = { generation = "x" }\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "'senior' \\(sorted as generation, the kind it refuses\\) belongs after 'jr'"), + # the entry named is the one to move: a word appended at the end, + # not the neighbours it displaced + ("appended-word-out-of-place", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "sr"\ngeneration = {}\n\n[[word]]\ntext = "aa"\ngeneration = {}')}, "'aa' \\(generation\\) belongs before 'jr'"), + # placed against the run kept in order, not beside a neighbour that + # is out of place itself: of c d b a, b goes before c, not after a + ("place-named-against-the-kept-run", {"en.toml": _w('[[word]]\ntext = "c"\ngeneration = {}\n\n[[word]]\ntext = "d"\ngeneration = {}\n\n[[word]]\ntext = "b"\ngeneration = {}\n\n[[word]]\ntext = "a"\ngeneration = {}')}, "'b' \\(generation\\) belongs before 'c'"), + ("appended-word-mid-file", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "sr"\ngeneration = {}\n\n[[word]]\ntext = "ta"\ngeneration = {}\n\n[[word]]\ntext = "tb"\ngeneration = {}\n\n[[word]]\ntext = "ka"\ngeneration = {}')}, "'ka' \\(generation\\) belongs after 'jr'"), + # a refused category sorts as its first kind + ("refused-category-out-of-place", {"en.toml": _w('[[word]]\ntext = "zzz"\nexcluded = { relation = "x" }\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "'zzz' \\(sorted as birth_marker, the kind it refuses\\) belongs after 'jr'"), + # the file's own chains decide its order, not another file's: de + # makes jr a spelling of aa, and en must still put its jr first + ("other-files-chain-does-not-place-a-head", {"de.toml": _w('[[word]]\ntext = "aa"\ngeneration = {}\n\n[[word]]\ntext = "jr"\ngeneration = { same_as = "aa" }'), "en.toml": _w('[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }\n\n[[word]]\ntext = "jr"\ngeneration = {}')}, "en.toml: 'jnr' \\(generation\\) belongs after 'jr'"), + # every other refusal is seen before the order's: a same_as typo + ("same-as-typo-before-the-order", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "sr"\ngeneration = {}\n\n[[word]]\ntext = "younger"\ngeneration = { same_as = "jrx" }')}, "same_as 'jrx' has no suffix reading"), + # a chain must end for the key to exist; und.toml's too, though + # its order is not checked + ("same-as-cycle", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }\n\n[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }')}, "same_as loops, jnr -> jr -> jnr;"), + ("same-as-cycle-in-und", {"und.toml": _w('[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }\n\n[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }')}, "same_as loops, jnr -> jr -> jnr;"), + # over every same_as written: a loop across files, one another file's + # head for the same word would hide, and one through an entry's + # second kind of a category + ("same-as-cycle-across-files", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }'), "fr.toml": _w('[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }')}, "same_as loops"), + ("same-as-cycle-behind-another-files-head", {"de.toml": _w('[[word]]\ntext = "x"\ngeneration = {}\n\n[[word]]\ntext = "jr"\ngeneration = { same_as = "x" }'), "en.toml": _w('[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }\n\n[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }')}, "same_as loops, jnr -> jr -> jnr;"), + ("same-as-cycle-through-a-second-kind", {"nl.toml": _w('[[word]]\ntext = "c"\npreposition = { may_be_given = true }\n\n[[word]]\ntext = "a"\npreposition = { may_be_given = true, same_as = "c" }\narticle = { may_be_given = true, same_as = "b" }\n\n[[word]]\ntext = "b"\narticle = { may_be_given = true, same_as = "a" }')}, "same_as loops, a -> b -> a;"), + # a chain running INTO a loop: the loop is named, not the chain + ("same-as-chain-into-a-loop", {"en.toml": _w('[[word]]\ntext = "xa"\ngeneration = { same_as = "xb" }\n\n[[word]]\ntext = "xb"\ngeneration = { same_as = "xc" }\n\n[[word]]\ntext = "xc"\ngeneration = { same_as = "xb" }')}, "same_as loops, xb -> xc -> xb;"), ] @@ -319,10 +360,92 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: "und.toml": _w('[[word]]\ntext = "fils"\ncredential = {}')}, {"pt-BR.toml": OK, "zh-Hant.toml": _w('[[word]]\ntext = "社長"\nprofessional_honorific = {}'), "es-419.toml": _w('[[word]]\ntext = "sr"\ngeneration = {}')}, + # check 10's order, met: kinds in CATEGORY order, a head before + # its spellings, a refusal under the kind it refuses, a name + # reading placing nothing beside a refusal, a comment moving with + # its entry; und.toml is exempt, so its reverse order passes + {"en.toml": _w('[[word]]\ntext = "dr"\ntitle = {}\n\n' + '[[word]]\ntext = "jr"\ngeneration = {}\n\n' + '[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }\n\n' + '[[word]]\ntext = "junior"\ngeneration = { same_as = "jr" }\n\n' + '[[word]]\ntext = "senior"\nfamily = { us_census_2010 = 3864, as_of = 2026-10-16 }\n' + 'excluded = { generation = "x" }\n\n' + '# a comment above an entry belongs to it\n' + '[[word]]\ntext = "sr"\ngeneration = {}\n\n' + '[[word]]\ntext = "esq"\ncourtesy = {}'), + "und.toml": _w('[[word]]\ntext = "von"\nparticle = { may_be_given = false }\n\n' + '[[word]]\ntext = "sir"\ntitle = {}')}, + # a spelling's own spellings follow it, under the root head: by + # the direct same_as alone, ed- would sort after ul-, under el- + {"ar.toml": _w('[[word]]\ntext = "al-"\narticle = { may_be_given = false }\n\n' + '[[word]]\ntext = "ad-"\narticle = { may_be_given = false, same_as = "al-" }\n\n' + '[[word]]\ntext = "el-"\narticle = { may_be_given = false, same_as = "al-" }\n\n' + '[[word]]\ntext = "ed-"\narticle = { may_be_given = false, same_as = "el-" }\n\n' + '[[word]]\ntext = "ul-"\narticle = { may_be_given = false, same_as = "al-" }')}, + # a word sorts under its EARLIEST kind, not its first written; + # a vocabulary reading outranks a refusal beside it; a refused + # category sorts as its first kind + {"en.toml": _w('[[word]]\ntext = "dr"\ntitle = {}\n\n' + '[[word]]\ntext = "md"\ncredential = {}\ntitle = {}\n\n' + '[[word]]\ntext = "jr"\ngeneration = {}\n\n' + '[[word]]\ntext = "aaa"\ncredential = {}\nexcluded = { generation = "x" }\n\n' + '[[word]]\ntext = "zzz"\nexcluded = { relation = "x" }')}, + # another file's chain does not place this file's head + {"de.toml": _w('[[word]]\ntext = "aa"\ngeneration = {}\n\n' + '[[word]]\ntext = "jr"\ngeneration = { same_as = "aa" }'), + "en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n' + '[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }')}, + # texts compare by codepoint, script by script + {"sr.toml": _w('[[word]]\ntext = "mlađi"\ngeneration = {}\n\n' + '[[word]]\ntext = "млађи"\ngeneration = {}')}, ): _compile(files) +def _apply_order_messages(texts: list[str]) -> int: + """Move entries as check 10's messages say until it passes; the moves.""" + moves = 0 + while True: + words: list[dict[str, Any]] = [{"text": t, "generation": {}} for t in texts] + try: + build.check_order({"en.toml": words}) + return moves + except build.VocabError as e: + m = re.search(r"'(\w+)' \(generation\) belongs (after|before) '(\w+)'", str(e)) + assert m, str(e) + word, side, other = m.groups() + texts.remove(word) + texts.insert(texts.index(other) + (side == "after"), word) + moves += 1 + + +def test_following_the_order_messages_takes_the_fewest_moves() -> None: + # every order of six words: the fewest moves that sort a list is its + # length less its longest run already in order, and following the + # messages takes exactly that many (#767's second review found the + # first version taking up to 22 moves where 10 would do) + from itertools import permutations + for order in permutations("abcdef"): + ranks = [ord(c) - ord("a") for c in order] + longest = [1] * 6 + for i in range(6): + for j in range(i): + if ranks[j] < ranks[i]: + longest[i] = max(longest[i], longest[j] + 1) + assert _apply_order_messages(list(order)) == 6 - max(longest), order + + +def test_the_order_check_ends_on_its_own() -> None: + # check_order called alone, without check 7 having refused a + # self-reference first, and on a chain longer than Python's + # recursion limit: it returns, it does not hang or recurse out + build.check_order({"en.toml": [{"text": "a", "preposition": {"same_as": "a"}}, + {"text": "b", "preposition": {}}]}) + chain = [{"text": f"w{i:04d}", "generation": {"same_as": f"w{i + 1:04d}"}} + for i in range(sys.getrecursionlimit() + 100)] + build.check_order({"und.toml": chain}) + + def test_every_character_the_tokenizer_ignores_is_refused() -> None: # check 2's hand copies are held to the pipeline, and the sweep is # driven by the tokenizer's own predicate, so a separator or strip @@ -630,8 +753,8 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: def test_the_readme_lists_every_kind_with_its_category_and_set() -> None: rows = _readme_table("## Kinds") - assert {_code(r[0])[0]: _code(r[1])[0] for r in rows} == build.CATEGORY - assert len(rows) == len(build.CATEGORY) # a kind listed twice + # in CATEGORY's order, which is check 10's canonical order of the kinds + assert [(_code(r[0])[0], _code(r[1])[0]) for r in rows] == list(build.CATEGORY.items()) for kind, _, fed in rows: assert set(_code(fed)) == _sets_fed(_code(kind)[0]), kind diff --git a/tools/vocab/build.py b/tools/vocab/build.py index bd6d2602..e1cc2cdc 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -53,10 +53,13 @@ 8. a phrase only as a birth_marker or an addresses = "given" title, and never with a `cased`, which is looked up per word; 9. a bound given name that is also a particle has may_be_given; +10. no same_as chain loops, in any file, and the words of every file + but und.toml in canonical order (_order_key): by kind, then by the + word a spelling is the same_as of, then by text; numbered by subject, not by when they run: 1-3, 8 and 7's case mask -run per entry as each file loads, and 4-7 and 9 over the whole pool in -check_pool, 6 first; +run per entry as each file loads, and 4-7, 9 and 10 over the whole pool +in check_pool, 6 first and 10 last; plus the data invariants that were asserts: may_be_initial only on a cased single letter, and an elided particle in both apostrophes (#700). @@ -177,6 +180,18 @@ def _alnum(text: str) -> str: } #: The categories a word-level `excluded` may name besides the kinds. CATEGORIES = frozenset(CATEGORY.values()) +#: CATEGORY's order is the canonical order of the kinds, and of the +#: categories by their first kind: check 10 sorts a file's words by it, +#: and the README's kind table lists the kinds in it. A category's kinds +#: are contiguous, so ordering by kind orders by category first. +KIND_ORDER = {kind: i for i, kind in enumerate(CATEGORY)} +_RUNS = [c for i, c in enumerate(CATEGORY.values()) + if i == 0 or c != list(CATEGORY.values())[i - 1]] +assert len(_RUNS) == len(set(_RUNS)), "a category's kinds are listed apart in CATEGORY" +#: The file exempt from check 10: its entries keep the order of the +#: Python lists they were converted from, which the vetting comments +#: above them still refer to ("the TITLES block below"). +UNORDERED = "und.toml" #: Behavioral properties per category: each compiles into a field, and #: its allowed values are exactly what today's pipeline does. @@ -288,6 +303,8 @@ class Pool: """Every reading and exclusion in the files, in file order.""" readings: list[Reading] = field(default_factory=list) exclusions: list[Exclusion] = field(default_factory=list) + #: each file's entries as written, for check 10's word order + entries: dict[str, list[dict[str, Any]]] = field(default_factory=dict) # ---------------------------------------------------------------- loading @@ -390,9 +407,166 @@ def load(files: Mapping[str, str]) -> Pool: if not isinstance(entry, dict): raise VocabError(f"{name}: `word` must be an array of tables ([[word]])") _load_entry(name, entry, seen, pool) + pool.entries[name] = words return pool +# ------------------------------------------------------------ word order +def _sort_kind(e: Mapping[str, Any]) -> str: + """The kind an entry sorts under: the earliest of its vocabulary + readings in canonical order; else, for an entry that records only a + refusal (`fils`, `senior`), the earliest kind it excludes, a + category counting as its first kind; else its earliest name reading. + A name reading is evidence for something else, so it places the + entry only alone.""" + kinds = [k for k in e if k in CATEGORY] + vocabulary = [k for k in kinds if CATEGORY[k] != "name"] + if vocabulary: + return min(vocabulary, key=KIND_ORDER.__getitem__) + refused = [k if k in CATEGORY else next(c for c in CATEGORY if CATEGORY[c] == k) + for k in e.get("excluded", ())] + return min(refused or kinds, key=KIND_ORDER.__getitem__) + + +def _heads(words: Iterable[Mapping[str, Any]]) -> dict[tuple[str, str], str]: + """Each word's same_as in each category, from one file's entries: + the same_as of the word's earliest reading of that category that + carries one. A same_as naming its own word is left out, for check 7 + to refuse, so following these always ends.""" + heads: dict[tuple[str, str], str] = {} + for e in words: + for k in sorted((k for k in e if k in CATEGORY), key=KIND_ORDER.__getitem__): + if e[k].get("same_as") not in (None, e["text"]): + heads.setdefault((CATEGORY[k], e["text"]), e[k]["same_as"]) + return heads + + +def _order_key(e: Mapping[str, Any], + heads: Mapping[tuple[str, str], str]) -> tuple[int, tuple[str, ...]]: + """Check 10's sort key: the entry's kind (_sort_kind), then its + same_as PATH -- the word it is a spelling of, that word's own head, + and so on as far as the file's own entries go, written root first + and ending in the entry's text. So a head sorts before its spellings + (`jr`, then `jnr` and `junior`), a spelling's own spellings follow + it (`el-` under `al-`, then el-'s assimilated forms), and an entry + with no same_as is a path of one, sorted by its text. Only the + file's own chains are followed (`heads` is _heads of the file), so + no other file decides its order. Texts compare by codepoint: they + are already normalized (check 2), so this is the codepoint order of + the normalized text, whatever the script -- defined everywhere, + stable across Python versions and locales, and no one language's + collation.""" + kind = _sort_kind(e) + category = CATEGORY[kind] + path = [e["text"]] + head = e[kind].get("same_as") if kind in e else None + if head == path[0]: + head = None # check 7's to refuse + # _heads holds no self-reference and _check_same_as_cycles has + # refused every longer loop, so this ends + while head is not None: + path.append(head) + head = heads.get((category, head)) + return KIND_ORDER[kind], tuple(reversed(path)) + + +def _check_same_as_cycles(entries: Mapping[str, list[dict[str, Any]]]) -> None: + """Check 10's first half: no same_as chain loops, in any file, + und.toml's included, over EVERY same_as written -- a word may name a + different head in each file or each kind, and a loop through any of + them is refused, not only through the one a file's order follows.""" + edges: dict[tuple[str, str], set[str]] = {} + for words in entries.values(): + for e in words: + for k, v in e.items(): + if k in CATEGORY and "same_as" in v and v["same_as"] != e["text"]: + edges.setdefault((CATEGORY[k], e["text"]), set()).add(v["same_as"]) + done: set[tuple[str, str]] = set() + for start in sorted(edges): + # depth first, iteratively (a chain can be longer than Python's + # recursion limit): meeting a word already on the open path is a + # loop, and the loop is the path from that word on, since a + # chain may run INTO a loop its first word is not part of + path = [start] + todo = [iter(sorted(edges[start]))] + while path: + head = next(todo[-1], None) + if head is None: + done.add(path.pop()) + todo.pop() + continue + node = (start[0], head) + if node in path: + loop = [t for _, t in path[path.index(node):]] + [head] + raise VocabError(f"{head!r} {start[0]}: same_as loops, " + f"{' -> '.join(loop)}; a spelling names the " + f"word it is a spelling of, which does not " + f"name it back") + if node not in done: + path.append(node) + todo.append(iter(sorted(edges.get(node, ())))) + + +def _misplaced(ranks: list[int]) -> tuple[int, set[int]]: + """The entry to move and the entries to keep: the first entry + outside a longest run of entries already in their relative order, + and that run. Placed against the run, each move the check names + lengthens it by one, so following the messages takes the fewest + moves, and a word appended at the end of a file is the one named, + not the neighbours it displaced. Ties keep the later entries, so of + two swapped words the first is named.""" + best = [1] * len(ranks) + prev = [-1] * len(ranks) + for i in range(len(ranks)): + for j in range(i): + if ranks[j] < ranks[i] and best[j] + 1 >= best[i]: + best[i], prev[i] = best[j] + 1, j + i = max(range(len(ranks)), key=lambda k: (best[k], k)) + kept = set() + while i != -1: + kept.add(i) + i = prev[i] + return min(set(range(len(ranks))) - kept), kept + + +def check_order(entries: Mapping[str, list[dict[str, Any]]]) -> None: + """Build check 10: no same_as chain loops, and every file but + und.toml lists its words in _order_key's order. Checked, never + rewritten: a comment above an entry belongs to it and must move + with it, which a TOML reader cannot see. Run last, so a file out of + order is reported only once every other check has passed: a + contributor sees an exclusion's reason (6) or a same_as typo (7) + before being told where the entry goes.""" + _check_same_as_cycles(entries) + for name, words in sorted(entries.items()): + if name == UNORDERED: + continue + heads = _heads(words) + keys = [_order_key(e, heads) for e in words] + ordered = sorted(range(len(words)), key=keys.__getitem__) + if ordered == list(range(len(words))): + continue + ranks = [0] * len(words) + for at, i in enumerate(ordered): + ranks[i] = at + m, kept = _misplaced(ranks) + # its place among the kept entries, which are in order already + below = [i for i in kept if ranks[i] < ranks[m]] + kind = _sort_kind(words[m]) + what = kind if kind in words[m] else f"sorted as {kind}, the kind it refuses" + if below: + place = f"after {words[max(below, key=ranks.__getitem__)]['text']!r}" + else: + above = min(kept, key=ranks.__getitem__) + place = f"before {words[above]['text']!r}" + raise VocabError( + f"{name}: {words[m]['text']!r} ({what}) belongs {place}. A file " + f"lists its words by kind in the README's table order, then by " + f"the word a spelling is the same_as of, head first, then by " + f"text; move the entry with any comment above it ({UNORDERED} " + f"is exempt)") + + #: Hand copies of what the tokenizer divides or strips at under the #: default Policy (this tool cannot import the pipeline without the #: package), each held to its source by tests/v2/test_vocab.py, which @@ -669,7 +843,7 @@ def _overrides(r: Reading, x: Exclusion) -> bool: # ------------------------------------------------------------- compiling def check_pool(pool: Pool) -> None: - """The cross-file build checks (4-7, 9) and the data invariants.""" + """The cross-file build checks (4-7, 9, 10 last) and the data invariants.""" by_text: dict[str, list[Reading]] = {} for r in pool.readings: by_text.setdefault(r.text, []).append(r) @@ -760,6 +934,10 @@ def check_pool(pool: Pool) -> None: f"and its U+2019 spelling; add " f"{text.replace(a, b)!r}") + # 10: no same_as loop, and each file but und.toml in word order -- + # last, so every other refusal is seen first + check_order(pool.entries) + # soft: a name reading compiles to nothing, so one on a word with no # vocabulary reading and no exclusion anywhere -- an exclusion's # evidence is what a name reading usually is -- informs nothing