Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/design/decisions/A1-ambiguity-report.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,22 @@
# A1 — the ambiguity report

## Standing

- A report whose fork the rest of the parse resolved to neither branch is withdrawn in assemble, the one place that sees final roles: a connective-or-initial letter roled suffix or title loses that report and keeps its own, so `JOHN QUINCY SMITH I` reports `suffix-or-name` alone — 2026-09-20 #397.
- The decision stays at its emitter and assemble words the field from the word's final role (`PendingAmbiguity.field_tail`), so a report names the field its word lands in whatever rule moved it: `, Ma Dr.` reports 'Ma' "read as a family name", and `Kim Min Do` under FAMILY_FIRST reports 'Do' read as a family name; a placeholder in the detail was declined — 2026-10-08 #626.
- A report on a part past the second comma names no field: `John Smith, Jr., Dr. Bart` reports "segment 'Dr. Bart' beyond the recognized comma structures; read best-effort"; naming every field the part's words land in was declined — 2026-10-10 #629.
- The listed-member report after a family comma names the role the comma binds, worded at the emitter: `John, Ma.` reads title 'Ma.' and reports "read as a title" — 2026-10-21 #793.

Elsewhere: P8's "connects nothing", withdrawn once over final roles in assemble, is decisions/S10's 2026-10-18 #722's /simplify pass entry.

Open: none in this file.

## Log

- 2026-09-20 #397 (second review) — A REPORT MUST NOT CONTRADICT THE READING BESIDE IT, and the shape could not arise before this cycle. `classify`'s `conjunction-or-initial` fork offers exactly two readings and its detail says which it took ("it is read as an initial"). Where the rest of the parse then roles that letter SUFFIX, the fork resolved to NEITHER branch: `JOHN QUINCY SMITH I` carried the connective-or-initial beside a `suffix-or-name` saying the same token reads as a generational suffix, and `john smith i`, `HENRY I`, `SMITH, JOHN I`, `JOSEP CAROD I ROVIRA III` and `abdul e i` were the same. `i` is the first word that is both a MARKED connective and suffix vocabulary, so no parse before #397 could reach it — which is why the rule is written down now rather than having been.
DECIDED: the report is WITHDRAWN where the role is SUFFIX or TITLE, in `_pipeline/_assemble.py`, the one place that sees final roles and the ambiguity list together — and beside a withdrawal that was already there, for a report whose referents did not survive assembly. That is the narrow reading of mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE rather than an exception to it: classify still owns the fork and still takes it where it is taken; what goes is a report whose subject the parse went on to read as something else. Emitting later was the alternative and was rejected as more invasive for the same answer — the fork's inputs (the case class, the own-words span) live in classify and moving the emitter moves them.
MEASURED: over the review's 43,925-name grid under eight configurations, 9,806 rows on 1,637 distinct names lose exactly one `conjunction-or-initial` and nothing else — no report of any other kind moves and none is gained. On the deduped corpus-union-cases population the only name affected is `JOHN QUINCY SMITH I`, a row this round added, so the ledger's three-name `fix(#397) the Catalan link reports a connective-or-initial in a one-case name` rule is untouched: all three of its names role the letter MIDDLE and all three still report. The invariant is `tests/v2/test_properties.py::test_no_report_contradicts_the_reading_beside_it`, which states it twice — no connective-or-initial on a token roled SUFFIX or TITLE, and no token named by two reports whose details assert different readings — and fails on 8,204 rows at `dc3bdf9c`, 0 here.
- 2026-10-08 #626 — A REPORT NAMES THE FIELD ITS WORD LANDS IN. Six of assign's reports worded the field from the role the word held at assign (`read as a given name`), and rules in post_rules still move a word after that: H1 moves the name behind a title to the family, P1's family-first fold takes the particle run and one unit and lays the rest out by the order, P6's attachment takes a trailing name word into the family at the end of a name read family-first, and under opt-in policies the patronymic rotations (O1, O2) and `middle_as_family`'s fold (O3) move one too. So `Attorney General of Minnesota` reported 'General of Minnesota' "read as a given name by convention" with the unit in the family (since 2.3.0, #491's join clause); `Kim Min Do` under FAMILY_FIRST told the caller 'Do' was "read as a middle name" with 'Do' in the family, and `de Kim Ma` under FAMILY_FIRST said 'Ma' was a middle name with 'Ma' the given name (both this cycle, since #289 made the Title-case word a name word); `, Ma Dr.` said 'Ma' after the comma was "read as the given name" with 'Ma' in the family (this cycle, since #289 added that comma report; 2.3.0 reads the same fields and reports nothing); and `Van Ivan Petrovich` under the East Slavic patronymic rule said the leading 'Van' was "read as a given name" after O1 rotated it into the family (at 2.3.0 too). #614's second review had fixed one instance of the class (`Freiherr von Berg Ma`) by moving its report to the stage that knows the outcome.
- 2026-10-08 #626 (its literal wording for the comma's credential reading superseded in part by #793 below) — A REPORT NAMES THE FIELD ITS WORD LANDS IN. Six of assign's reports worded the field from the role the word held at assign (`read as a given name`), and rules in post_rules still move a word after that: H1 moves the name behind a title to the family, P1's family-first fold takes the particle run and one unit and lays the rest out by the order, P6's attachment takes a trailing name word into the family at the end of a name read family-first, and under opt-in policies the patronymic rotations (O1, O2) and `middle_as_family`'s fold (O3) move one too. So `Attorney General of Minnesota` reported 'General of Minnesota' "read as a given name by convention" with the unit in the family (since 2.3.0, #491's join clause); `Kim Min Do` under FAMILY_FIRST told the caller 'Do' was "read as a middle name" with 'Do' in the family, and `de Kim Ma` under FAMILY_FIRST said 'Ma' was a middle name with 'Ma' the given name (both this cycle, since #289 made the Title-case word a name word); `, Ma Dr.` said 'Ma' after the comma was "read as the given name" with 'Ma' in the family (this cycle, since #289 added that comma report; 2.3.0 reads the same fields and reports nothing); and `Van Ivan Petrovich` under the East Slavic patronymic rule said the leading 'Van' was "read as a given name" after O1 rotated it into the family (at 2.3.0 too). #614's second review had fixed one instance of the class (`Freiherr von Berg Ma`) by moving its report to the stage that knows the outcome.
DECIDED: the DECISION stays at its site and the WORDING of the field moves to assemble, the one place that knows final roles (the same reasoning as the 2026-09-20 withdrawal above). An emitter whose detail names a field a later rule could change writes the sentence up to the field and hands the rest as `PendingAmbiguity.field_tail` (`""` where the field ends the sentence); assemble joins them around `FIELD_READING` of the first referent's final role. All six such emitters in assign use it -- H4's all-suffix name, title-or-name's join clause, O5's lone word, the trailing peel's picks, the leading ambiguous particle and the comma report on the word after a family comma -- including those right on every measured input, so no later rule can make them wrong. A report in a name emptied for want of content is withdrawn rather than shipped half-worded (no emitter reaches that today). A placeholder in the detail was declined: token text can carry any characters, and a name containing the placeholder would corrupt its own report. Reports worded where the field is settled keep their literal wording: a comma's credential reading, the suffix run's and the numeral fork's, H4's "the name" and P6's own "joins the family". The comma report's given-side wording moves from "the given name" to "a given name" with the rest.
MISSED, then caught in review: the first draft converted five emitters and its grid had no empty-head comma with a trailing title, the shape AGENTS.md's comma sweep already asks for; four of the review's five readers found the sixth emitter on `, Ma Dr.`. The draft also named only H1 and P6 as movers; P1's fold was found by stepping the stages, and the review of the fix found the opt-in movers O1 to O3, which no grid over the three orders reaches -- reverting the leading-particle emitter passed the whole suite until rows under those policies pinned it.
MEASURED 2026-10-08, each report's field phrase compared against its tokens' final roles. RECIPE: texts are the 1,536 names of `tools/differential/corpus*.jsonl` and 347 `tests/v2/cases.py` texts (the table at 0c54662b, the parent of #614's first commit, less the corpus names; today's table adds rows this entry's review wrote for the shapes the grid missed, so it reads more wrong claims on master, 6 more as measured on the review), plus every head in {'John', 'John Smith', 'John Q. Smith', 'Mary Ann Smith', 'John van Smith', 'Juan de la Vega', 'John van', 'anh van', 'Jan Freiherr von Berg', 'Freiherr von Berg', 'Freiherr St', 'St', 'Dr. St', 'Freiherr', 'Dr.', 'Mr. and Mrs.', 'Attorney General of', 'abdul salam', 'Juan y Garcia', 'Jane Doe nee van der Berg', 'Dr. John Smith', 'JOHN SMITH', 'john smith', 'Smith', 'de Mesnil', 'Dr. de Mesnil', 'Kim Min', 'Kim Min Jun', 'Ortega y Gasset', 'van', 'Do', ''} followed by one or two words (repetition allowed) from the list [PhD, MA, Ma, DO, Do, do, vd, mc, Jr, III, V, V., VI, X.Y.Z., Ph. D., Esq., Prof., Sir, Ed, van, and, Jones, ba, John, Smith, Minnesota, de, St, Freiherr], space-joined and stripped; 20,000 draws after Python's `random.seed(626)`, each `random.choice(heads)`, then `random.randint(3, 4)` words each `random.choice(words)`, space-joined and stripped, both lists in the order written here; then, unstripped, `{head}, {word}`, `Smith, {head} {word}` with any double space replaced by one, and `{word}, {head}` for every head and word, and `, {w1} {w2}` and `(Bob), {w1} {w2}` for every ordered pair of words; the empty text dropped and duplicates removed -- 53,517 texts; each parsed under the three orders, so the opt-in movers are outside these figures; a claim is a match of `(reads? as|joins) (<phrase>)` for the phrases `_FIELD_CLAIMS` in tests/v2/test_cases.py lists plus "the given name", and it is wrong when its tokens' roles are not among the fields the phrase names. On master (26cdb891) 2,725 of 89,092 claims were wrong, in three emitters: title-or-name's join clause (1,575), the trailing peel's picks (1,114) and the comma report (36); on the tree, 0. The comparator is master, which words the field at assign; the tree words it at assemble. Fingerprint against master over the differential corpora, master's case-table names and a seeded comma grid (13,090 names) under six policies (the three orders, `lenient_comma_suffixes=False`, a `/` suffix delimiter and `CapsSuffixes.OFF`): no role, kind, initials or rendering moves; the details that change are 6,395 comma reports on 1,086 names reworded from "the given name" to "a given name", 85 on 15 names now naming the family they land in, and 40 title-or-name join reports on 10 names now naming the family. Pinned by `test_a_report_names_the_field_its_word_lands_in` in tests/v2/test_cases.py, over every case row under all three orders except the locale rows (a row with its own policy only as declared), whose recorded control is 14 failures on master, and by `test_the_field_sweep_sees_the_claims_it_checks`, which records how many claims that sweep checks; the opt-in movers by the rows `Van Ivan Petrovich` (East Slavic), `Van Ali Veli oglu` (Turkic) and `Do Bishop Do` (FAMILY_FIRST with `middle_as_family`), which the sweep reads as declared.
Expand Down
11 changes: 11 additions & 0 deletions docs/design/decisions/A2-empty-name.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
# A2 — the empty name

## Standing

- An input with no letter or digit parses to the empty name, where v1 kept `parse(".")` as given '.': `.` and `.,` read every field empty — 2026-07 PR #288.
- A report born from the punctuation survives the emptying, so malformed input stays distinguishable from blank input: `(` reports `unbalanced-delimiter` — 2026-07 PR #288.

Elsewhere: #629's declined field-naming report would have withdrawn a report from a name emptied of content (`", , -"`), against this rule; decisions/A1's 2026-10-10 #629 entry.

Open: none in this file.

## Log

- 2026-07 (v2 core, PR #288) — v1 kept parse(".") as first="."; 2.0 empties content-free input instead. The born-empty-ambiguity survival was a review fix in the rc1 arc: an unbalanced-delimiter report must outlive the emptying, or malformed input becomes indistinguishable from blank input.

10 changes: 10 additions & 0 deletions docs/design/decisions/D1-segmenterless-activation-warning.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# D1 — the segmenterless-activation warning

## Standing

- `parser_for(locales.JA)` without a segmenter warns and does not raise: the inert registration is itself a pinned property, and a warning is filterable by the caller who wants it; the Japanese install hint appears only when a Japanese script is among the dead ones, a hangul-only gap getting the generic remedies — 2026-08 #337, WARN rather than raise
- The warning re-emits from `parser_for`'s frame, so the location reported is the caller's own call — 2026-08 #337, the warning exists because
- The message offers code that type-checks, `Policy(segment_scripts=frozenset())`, pinned by a test, with the bare tuple literal in the denylist test — 2026-08-07 #339.

Open: none in this file.

## Log

- 2026-08 #337 — WARN rather than raise, deliberately: registering the JA pack without the extra must stay safe (the inert registration is itself a pinned property), and a warning is filterable by the caller who wants exactly that inertness. The Japanese install hint is conditional — it appears only when a Japanese script is among the dead scripts; a hangul-only gap gets the generic remedies.
- 2026-08 #337 — the warning exists because parser_for(locales.JA) without segmenter= used to build a parser that silently behaved like a working one minus the feature; it re-emits from the parser_for frame so the reported location is the caller's own call.
- 2026-08-07 #339 — diagnostics that hand the reader code must hand code that type-checks: the message's offered deactivation used a bare tuple literal for segment_scripts — an arg-type error under mypy in a py.typed package — and became frozenset(), pinned by a test asserting the offered spelling. The known-bad spelling is in the denylist test. The structural fact behind the recurrence: autodoc renders docstrings into the API reference, so docs/*.rst is NOT the boundary of "the docs" — a guide and the reference can teach different spellings on the same rendered page, invisibly to any .rst-only sweep.
Expand Down
10 changes: 10 additions & 0 deletions docs/design/decisions/D2-construction-raises-parse-never-does.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# D2 — construction raises, parse never does

## Standing

- Every raise in the locale-apply path is a plain `TypeError` or `ValueError`, so the rewrap with the locale's code cannot break on an exotic exception signature — 2026-07 PR #288.

The two Declined lines below (2026-07-27) stand as written: no zh-vs-ja kind, and `order` stays reserved and unemitted for script-resolved names, while `segmentation` fires on a genuine multi-split fork (`夏侯惇` under `locales.ZH` reports it, and `남궁민수` by default).

Open: none in this file.

## Log

- 2026-07 (v2 core, PR #288) — every raise in the locale-apply path is a plain TypeError/ValueError so the wrap-with-locale-code rewrap cannot break on exotic exception signatures.

Declined (ambiguity kinds for script-resolved names, 2026-07-27):
Expand Down
Loading
Loading