Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,12 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
- **The keep-outs became exclusions**, in the file of the language whose reading they refuse: `ben` and `ال` in ar.toml, `בר` in he.toml, `nic` and unaccented `o` in ga.toml, `bt` and `bte` in ms.toml (`uv run python tools/vocab/query.py --excluded particle`). decisions/vocabulary-exclusions' particle block keeps its class-level bullets and points at the query for the rest.
- **Two words keep the generic kind in `und.toml`**: `bon` and `dí`, in the prefix list since the earliest history (2011 and 2012) with no record of which names they come from, and with no particle use found. Both are borne as names (the counts are on their entries); removing them moves a compiled set, so it is left to its own change.
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind particle --count` (readings, one per file a word is in), `--kind clan`, `--file nl --kind particle`.

- **2026-10-19 #747 — a build-checked word order (check 10).** No compiled set moves: the 41 language files that were out of order on cc030b9c were sorted (an entry moving with any comment above it), `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The order, stated in nameparser/vocab/README.md's Word order section, is kind in `CATEGORY`'s order, then the word's `same_as` path from its root head, then text, so the spellings of one word sit together under it before #750 moves the titles out of `und.toml`.
- **Checked, not rewritten.** `tomllib` only reads, and a comment above an entry belongs to it, which no TOML reader can see; a rewriter would have to round-trip the files' comments. No checked file holds one on 2026-10-19 (every per-entry comment is in `und.toml`), but a word leaving `und.toml` before its category is classified carries its vetting comment with it. So the build refuses a file out of order, naming the entry to move (the first outside a longest run already in order, so a word appended at the end is named rather than its neighbours) and its place against that run, so following the messages takes the fewest moves (#767's second review found the place first given beside a neighbour itself out of place, which took up to 22 moves where 10 would do).
- **Run last.** Check 10 runs after every other check, so a contributor adding a refused word to a language file meets the exclusion's reason (check 6) rather than an order error, the precedence check 6 is placed first for; the first version ran it at the end of loading, ahead of check 6, and the review of #767 found an excluded word appended to en.toml reported as out of order.
- **The same_as path, not the direct target, and the file's own path.** Keyed by the direct target, a spelling of a spelling sorts apart from the word it spells: ar.toml's `ed-` (same_as `el-`, itself same_as `al-`) would follow `ul-`. The path keeps each spelling under its own head inside the root's group, and it follows only the file's own entries, so a word that is a head in one file and a spelling in another does not make one file's order depend on the other (#767's review found the first version, which followed a map taking each word's first file, accepting `jnr` before `jr` in en.toml once de.toml made `jr` a spelling).
- **No same_as loop, over every same_as written.** Check 7 refuses only a same_as naming its own word. Check 10 refuses any longer loop, in every file, `und.toml` included, over every same_as of every reading: no check holds a word to one head across files or kinds, and the first version, which walked one head per word, passed a loop running through a second file's or a second kind's same_as.
- **Codepoint order of the normalized text.** Defined for every script and every machine, and no one language's collation; the cost is that within a script a precomposed accented letter follows `z` (`aḍ-` after `az-`), and that a traditional and a simplified spelling with no `same_as` between them sort apart (`課長`, `课长`).
- **A refusal sorts under the kind it refuses.** Forty-six language-file entries on 2026-10-19 (`senior`, `fils`, the Slavic `ml` and `st`) hold no vocabulary reading, only an exclusion and sometimes the name reading that is its evidence; they sort beside the words they were refused as.
- **`und.toml` is exempt.** Its entries keep the Python lists' order, which its vetting comments still point along ("the bare-κ trap above"); each category's classification moves its words out, into files that are checked.
18 changes: 16 additions & 2 deletions nameparser/vocab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,23 @@ Two rules about the `text`:
- It is written already normalized: lowercase, NFC, no edge periods, single-spaced. The build refuses any other spelling and prints the one to write.
- It is one word, except for a `birth_marker` or a title with `addresses = "given"`, the two readings matched as a whole run (`z domu`).

## Word order

Every file but `und.toml` lists its words in one order, and the build refuses a file that does not (check 10), naming the entry to move and where it belongs:

1. **By kind**, in the order of the table under [Kinds](#kinds), which lists the categories in turn and each category's kinds together. A word with several readings sorts under its first kind in that order. A word whose only vocabulary reading is a refusal (`fils`, `senior`) sorts under the kind it refuses, a refused category counting as its first kind, and a name reading places a word only when it has nothing else.
2. **Then by the word it is a spelling of**: a reading's `same_as`, or the word's own text when it has none. A head comes first and its spellings follow it, so `jr` is followed by `jnr` and `junior`, and `sr` by `snr`. A spelling that has spellings of its own heads a group inside its head's: in `ar.toml`, `el-` follows the `a…-` spellings of `al-` with its own assimilated forms after it, and `ul-` comes last. Only the file's own entries are followed, so a word that is a head in one file and a spelling in another is placed in each file by that file alone.
3. **Then by the word's text.**

Text is compared by codepoint. The texts are already normalized, so this is the order of the normalized text, whatever the script: Latin before Cyrillic before Arabic, and within a script an accented letter after `z` (`aḍ-` follows `az-` and `aš-`). It is defined for every script and the same on every machine, and it is no one language's collation. A traditional and a simplified spelling with no `same_as` between them sort apart (`課長` and `课长` in `zh.toml`).

The build checks the order and never rewrites a file, because a comment above an entry belongs to it and must move with it, which a TOML reader cannot see. When the build says an entry belongs somewhere else, move it with its comment. The order is checked last, so any other refusal (an exclusion's reason, a `same_as` naming no word) is reported first. The same check refuses a `same_as` chain that loops, in every file.

`und.toml` is exempt. Its entries keep the order of the Python lists they were converted from, and its vetting comments still point at their neighbours ("the TITLES block below"); a word leaving it for a language file takes its place in that file's order.

## Kinds

A reading's key is its **kind**. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix and particle categories are classified (two particles of unrecorded origin, `bon` and `dí`, keep the generic kind in `und.toml`); the others carry their generic kind.
A reading's key is its **kind**. The table lists the kinds in their canonical order, which is the [word order](#word-order)'s first key and `CATEGORY`'s order in `tools/vocab/build.py`. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix and particle categories are classified (two particles of unrecorded origin, `bon` and `dí`, keep the generic kind in `und.toml`); the others carry their generic kind.

| Kind | Category | Compiled into |
|---|---|---|
Expand Down Expand Up @@ -188,7 +202,7 @@ Record a count on the word as a `given` or `family` reading, with the date you t

1. **Look up the word's other readings.** `uv run python tools/vocab/query.py --word <word>` lists every reading and exclusion of it in every file. An exclusion there is a decision already made; a reading in another category is a clash you are about to create, which is fine when position settles it (`do`) and needs an argument when it does not.
2. **Check it against real names, not memory.** `uv run python tools/name_usage.py <word>` gives the counts, and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs the matching flag (`may_be_given`, `may_be_name`) or should stay out. The root `AGENTS.md`'s "Adding a word to the vocabulary" lists the traps that have bitten before.
3. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), with what the check found on it: `note`, `issue`, a dated name reading, or an `excluded` entry if you decided against it.
3. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), in its place in the [word order](#word-order), with what the check found on it: `note`, `issue`, a dated name reading, or an `excluded` entry if you decided against it.
4. **Rebuild**: `uv run python tools/vocab/build.py`. It refuses what it can detect, naming the word and, where one file is at fault, the file, and rewrites `nameparser/config/_compiled.py`. Commit both.
5. **Run the tests**: `uv run pytest tests/v2/test_vocab.py` for the build, then the full suite, and read what `tests/v2/test_snapshot.py` reports as moved (the root `AGENTS.md`, "Validating a new parsing rule").

Expand Down
4 changes: 2 additions & 2 deletions nameparser/vocab/am.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
schema = 1

[[word]]
text = "ጁንየር"
text = "ጁኒየር"
generation = { issue = 679 }

[[word]]
text = "ጁኒየር"
text = "ጁንየር"
generation = { issue = 679 }
Loading
Loading