From 94d4bc31feabcbd98a1e47c2c886316b660c9ad2 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 02:57:26 -0700 Subject: [PATCH] Read a comma with nothing before it as a family comma naming no surname (rules/C15, #892) A name opening with a comma had no rule: clauses in H8, O6 and M2 and review findings in five decisions files each decided it locally, and every comma change had to sweep the shape. rules/C15 states it once: the part after the comma reads as the given part after a family comma reads, with no surname supplied. H1 is the one exception -- a title and one name word read that word as the family (', Dr. John'), the comma naming no surname to decide the field by. No reading moves. test_an_empty_head_reads_as_its_family_comma_twin holds ', X' to 'Smith, X' less the surname over 3,519 given parts per order, with H1's 1,075 rows per order counted and a recorded negative control (#613's early return fails 1,098 rows). The twin's surname is written in the given part's case: a mixed-case 'Smith' brings a case contrast (S6) the empty head lacks, which is why ', MA' reads given 'MA' as 'SMITH, MA' does. nameparser/AGENTS.md's comma sweep points at the invariant for fields and kinds, keeping the empty head in report-detail grids. Co-Authored-By: Claude Opus 5.5 --- .../C15-comma-with-nothing-before-it.md | 19 +++ docs/design/decisions/README.md | 1 + .../rules/C15-comma-with-nothing-before-it.md | 54 +++++++ .../rules/H1-title-and-one-name-word.md | 5 + .../H8-salutation-ending-the-given-part.md | 2 +- docs/design/rules/O6-two-family-names.md | 2 +- docs/design/rules/README.md | 1 + nameparser/AGENTS.md | 4 +- nameparser/_pipeline/_comma.py | 9 +- tests/v2/rules_budget.json | 7 +- tests/v2/test_properties.py | 135 ++++++++++++++++++ tools/differential/corpus_rules.jsonl | 13 ++ tools/differential/snapshot.jsonl | 13 ++ 13 files changed, 253 insertions(+), 12 deletions(-) create mode 100644 docs/design/decisions/C15-comma-with-nothing-before-it.md create mode 100644 docs/design/rules/C15-comma-with-nothing-before-it.md diff --git a/docs/design/decisions/C15-comma-with-nothing-before-it.md b/docs/design/decisions/C15-comma-with-nothing-before-it.md new file mode 100644 index 000000000..2f48840d8 --- /dev/null +++ b/docs/design/decisions/C15-comma-with-nothing-before-it.md @@ -0,0 +1,19 @@ +# C15 — a comma with nothing before it + +## Standing + +- A comma with nothing before it is a family comma naming no surname, the part after it read as the given part after a family comma reads, held by an invariant over two inputs rather than by a sweep shape: `, X` reads as `Smith, X` reads, less the surname, its fields and ambiguity kinds alike; a particle P6 attaches is then the family alone (`, Jane van` → family 'van') — 2026-10-27 #892. +- H1 is the one exception: a title and one name word, or a bound pair standing for it, read as the family (`, Dr. John` → family 'John', `, Dr. abdul Rahman` → family 'abdul Rahman'), the comma naming no surname to decide the field by, as a comma with nothing but titles after it names no given part — 2026-10-27 (Derek). + +Open: whether the code written for the empty head alone (#890's floor for `, Dr. fils`, the P6 attachment site in post_rules) can go, the question #892's fourth point asks; the invariant answers it by mutation, one site at a time. + +## Log + +- 2026-10-27 #892 — A COMMA WITH NOTHING BEFORE IT IS A FAMILY COMMA NAMING NO SURNAME. The shape had no rule: clauses in rules/H8, rules/O6 and rules/M2 and review findings in decisions/C1, S10, A1 and H5 each decided it locally, and nameparser/AGENTS.md made every comma change sweep it. Real names almost never take it: 10 of about 6 million local-corpus names (FEC 4, tse 3, PSC 2, tw-donors 1, the last opening with a fullwidth comma, `，邱詩涵`, which reads as `, 邱詩涵` does), the issue's count of 2026-10-25 missing the fullwidth one; recounted 2026-10-27 by scanning each local corpus's names for one opening, after spaces, with `,` or `，`. The alternative, dropping the comma so `, John Smith` reads as `John Smith`, was declined: the shape's one source is an empty surname field in listing data (#613's review, `, PhD` and `, Jr.`), and reading it as a whole name contradicts that source. + - Measured on master (8a65ee18) before any change, over the grid `tests/v2/test_properties.py::test_an_empty_head_reads_as_its_family_comma_twin` builds (10 title runs × 32 bodies × 11 trailing words, 3,519 given parts under each of three orders): every given part reads as its twin but 1,075 per order, all of them H1's, so the rule landed with no code change. A first comparison's eight differing given parts holding `MA`, among them `, MA`, `, (Bob) MA`, `, J. MA` and `, JOHN SMITH MA`, were the twin's doing, not the empty head's: a surname written `Smith` brings a case contrast the given part lacks, so `Smith, MA` reads suffix 'MA' by its capitals (rules/S6) while `, MA` and `SMITH, MA` both read given 'MA'. The test writes the twin's surname in the given part's case for that reason, and rules/C15's Accepted bullet records the reading. + - The particle-and-suffix words `vd` and `mc` ending the name behind H1's family reading (`, Dr. Jones vd`) are the exception's knock-on, not a second one: the twin attaches the word to its surname (rules/P6), the empty head has no family to attach it to once H1 has made the lone word the family, and it reads as the suffix. Behind a given-name title H1 reads no family and P6 attaches as in the twin (`, Sir Jones vd` → family 'vd'). + - The exception is stated by its result, not by the comma-less name: `, Jane Mrs` reads family 'Jane' where `Jane Mrs` reads family 'Mrs' (a bare salutation is a title only after a family comma, rules/H8), and `, Dr. abdul Rahman` reads family 'abdul Rahman' where `Dr. abdul Rahman` reads given 'abdul', family 'Rahman'. A first draft compared against the comma-less name and the design-docs review found it false on 493 of the 1,075 H1 rows per order. + - The invariant compares fields and ambiguity KINDS, never a report's detail: `, Sir Jones vd` reports that 'vd' "joins the family the comma named", a comma that names none, and passes. Report-detail grids keep the empty head (nameparser/AGENTS.md). + - Recompute: `uv run pytest -n0 "tests/v2/test_properties.py::test_an_empty_head_reads_as_its_family_comma_twin"`; its docstring records the negative control. + +- 2026-10-27 (Derek) — H1 STAYS THE ONE EXCEPTION, stated as the comma deciding nothing. "Keep H1 as the one exception, stated as a comma with nothing before it not deciding the field." `, Dr. John` keeps family 'John', where its twin `Smith, Dr. John` reads given 'John': rules/H1 already says a comma with nothing but titles after it names no given part and so does not decide the field, and a comma with nothing before it names no family, the mirror case. Following the invariant instead would have read given 'John' for every such row, 1,075 per order on the grid; rules/H8's Accepted bullet records the trailing spellings' family reading (`, Jane Mrs`, and the period-marked one since H5, #316). diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 04bb680b5..9769dec53 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -88,6 +88,7 @@ Entry conventions: - [C12 — a suffix word ending the name before the comma](C12-suffix-word-ending-the-name-before-the-comma.md) - [C13 — a credential opening the part after the comma](C13-credential-opening-the-part-after-the-comma.md) - [C14 — strict mode: a suffix word written as an initial](C14-strict-mode-a-suffix-word-written-as-an-initial.md) +- [C15 — a comma with nothing before it](C15-comma-with-nothing-before-it.md) ## Name order (O) diff --git a/docs/design/rules/C15-comma-with-nothing-before-it.md b/docs/design/rules/C15-comma-with-nothing-before-it.md new file mode 100644 index 000000000..4c47e6a67 --- /dev/null +++ b/docs/design/rules/C15-comma-with-nothing-before-it.md @@ -0,0 +1,54 @@ +# C15 — a comma with nothing before it + +**When:** a name opens with a comma that has nothing written before it, as in ", John Smith". + +**Then:** the comma is a family comma (C1) naming no surname, and the part after it reads as the given part after a family comma reads, with no surname supplied for it. + +**Because:** a comma with nothing before it is how a listing writes a record whose surname field is empty, as CSV data writes one. The writer marked where the surname goes and left it empty, so the parser supplies none, and a floor that keeps a family word does not apply, the input naming no family to lose. Read this way the shape has no reading of its own: the part reads as it reads behind any surname. + +``` +", John Smith" → given="John" +", John Smith" → middle="Smith" +", John Smith" → family="" +", John Smith Jr." → suffix="Jr." +", John (Bob) Smith" → nickname="Bob" +", PhD" → suffix="PhD" +", Jr." → suffix="Jr." +", Sir John" → given="John" +``` + +**Except:** + +- **A title and one name word** (H1): the comma names no surname, so it does not decide the field of that word, or of a bound pair standing for it, and the title's lone name word reads as the family. A particle that is also a suffix word, ending the name behind a title that addresses by surname, then has no family to attach to (P6) and reads as the suffix. + ``` + ", Dr. John" → family="John" · boundary + ", Jane Mrs" → family="Jane" + ", Dr. abdul Rahman" → family="abdul Rahman" + ", Dr. Jones vd" → suffix="vd" + ``` + +**Precedence:** + +- **A particle ending the name** (P6) attaches to the family after this comma as after any family comma, and with no surname written the family is that particle alone: `, Jane van` reads family `van`. P6 wins because the particle is in the part after the comma, which this rule reads as any given part is read. + ``` + ", Jane van" → family="van" + ", Sir Jones vd" → family="vd" + ``` + +**Accepted:** + +- A writer with a stray comma in front of a whole name gets the listing reading rather than the name's: `, John Smith` reads given `John`, middle `Smith`, where `John Smith` reads family `Smith`. The shape's one source is an empty surname field, and dropping the comma would contradict it. + ``` + ", John Smith" → family="" + "John Smith" → family="Smith" + ``` +- The surname's case is evidence the empty head does not have. A word of the ambiguous credential class written alone in capitals reads as a name, the name being written wholly in one case (S6), where a surname written in mixed case before the comma makes the same word the credential. + ``` + ", MA" → given="MA" + "SMITH, MA" → given="MA" + "Smith, MA" → suffix="MA" + ``` + +--- +**Signal:** comma: nothing written before the first comma +**history:** decisions/C15 · **implemented:** nameparser/_pipeline/_comma.py diff --git a/docs/design/rules/H1-title-and-one-name-word.md b/docs/design/rules/H1-title-and-one-name-word.md index e14cab6b7..ce5edcf0e 100644 --- a/docs/design/rules/H1-title-and-one-name-word.md +++ b/docs/design/rules/H1-title-and-one-name-word.md @@ -20,6 +20,11 @@ "Sir John" family-first → given="John" "John III, Dr." → family="John" · boundary ``` +- **Nor does a comma with nothing before it**, which names no surname (C15): the title run decides the field of the one name word after the comma, the family unless the run addresses by given name. + ``` + ", Dr. John" → family="John" + ", Sir John" → given="John" + ``` **Because:** a title normally addresses by surname, so a title followed by a single name word usually names the family; but a given-name title addresses by given name. Several titles written together are one form of address, and the one that does the addressing is the last — the words in front of it rank the person rather than name them. What stands beside the name word — a suffix, a nickname, a maiden name — does not make the name any longer, so it does not decide this reading. diff --git a/docs/design/rules/H8-salutation-ending-the-given-part.md b/docs/design/rules/H8-salutation-ending-the-given-part.md index a9ea44f1f..c1c1d7712 100644 --- a/docs/design/rules/H8-salutation-ending-the-given-part.md +++ b/docs/design/rules/H8-salutation-ending-the-given-part.md @@ -65,7 +65,7 @@ ``` "Doe, Jane Miss" → title="Miss" ``` -- With nothing before the comma, the one given word the salutation leaves reads as a title's lone name word reads (H1), the family, as the period-marked spelling has read it since H5 (2.3.0, #316); written alone, that word is the given name. +- With nothing before the comma, the one given word the salutation leaves reads as a title's lone name word reads (H1, C15's one exception), the family, as the period-marked spelling has read it since H5 (2.3.0, #316); written alone, that word is the given name. ``` ", Jane Mrs" → family="Jane" ", Jane" → given="Jane" diff --git a/docs/design/rules/O6-two-family-names.md b/docs/design/rules/O6-two-family-names.md index a2196905c..7faa700d9 100644 --- a/docs/design/rules/O6-two-family-names.md +++ b/docs/design/rules/O6-two-family-names.md @@ -48,7 +48,7 @@ "López, Ana García" two_family_names → family="López" "García López, Ana" two_family_names → family="García López" ``` -- **A family comma with nothing before it still names no family**, and the name after it reads as the default reads it. +- **A family comma with nothing before it still names no family** (C15), and the name after it reads as the default reads it. ``` ", Ana García López" two_family_names → family="" ``` diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 128341fc3..b8528ffa5 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -182,6 +182,7 @@ Terms the rules use without defining them each time, each with the rule that def - [C12 — a suffix word ending the name before the comma](C12-suffix-word-ending-the-name-before-the-comma.md) - [C13 — a credential opening the part after the comma](C13-credential-opening-the-part-after-the-comma.md) - [C14 — strict mode: a suffix word written as an initial](C14-strict-mode-a-suffix-word-written-as-an-initial.md) +- [C15 — a comma with nothing before it](C15-comma-with-nothing-before-it.md) ## [Name order (O)](O-name-order.md) diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md index c61a039c6..4eb944bdb 100644 --- a/nameparser/AGENTS.md +++ b/nameparser/AGENTS.md @@ -6,7 +6,7 @@ Loaded when a session reads or edits anything under `nameparser/`: the pipeline, ### Four comma shapes -No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. +No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put a nickname-only head (`(Bob), PhD`) in every comma grid. The empty head no longer needs a row of its own in a grid that compares fields: rules/C15 reads it as a family comma naming an empty family, and `test_an_empty_head_reads_as_its_family_comma_twin` (tests/v2/test_properties.py) holds `, X` to `Smith, X` less the surname, fields and ambiguity kinds, over a grid of given parts, H1's title-and-one-name-word reading the one exception (#892). A change that breaks it gets a generic fix or an entry in decisions/C15, never code for the empty head alone; widen the test's grid with the change's own shapes instead. ### Counting the one-word path @@ -26,7 +26,7 @@ Read a generated grid's moves by SHAPE, not by corpus membership: #614's first d ### A report's field -The second was in what the grids compared: they compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin (#614). So compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. +The second was in what the grids compared: they compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin (#614). So compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. C15's invariant holds that shape's fields and kinds but never a report's detail, so a grid checking details keeps the empty head (`, Sir Jones vd` reports 'vd' joining "the family the comma named", a comma C15 says names none). ### Exclusion lists and shared predicates diff --git a/nameparser/_pipeline/_comma.py b/nameparser/_pipeline/_comma.py index 409fb5c5b..621085d9a 100644 --- a/nameparser/_pipeline/_comma.py +++ b/nameparser/_pipeline/_comma.py @@ -314,11 +314,10 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: return decide(state, False) seg0, seg1 = state.segments[0], state.segments[1] # Nothing after the comma: nothing to read. An EMPTY part BEFORE it - # is read on, being no whole name (`_whole_name` answers no), so - # ', PhD' binds its credential under the family comma as 2.3.0 read - # it; returning here on an empty head left the part to the listing - # walk, which made 'PhD' the given name and ', Jr.' a title (#613's - # PR review; empty surname fields in CSV data write exactly this) + # is read on, being no whole name (`_whole_name` answers no): + # rules/C15: "the part after it reads as the given part after a + # family comma reads" -- so ', PhD' binds its credential as + # 'Smith, PhD' does; decisions/C15 (#892), decisions/C1 (#613) if not seg1: return state tokens = list(state.tokens) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 3ef5ee419..34c247c6d 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -15,16 +15,17 @@ "C12": {"words": 316, "then_words": 107, "because_sentences": 2, "except": 2, "precedence": 1, "accepted": 2, "accepted_words": 59}, "C13": {"words": 259, "then_words": 74, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 0, "accepted_words": 0}, "C14": {"words": 231, "then_words": 111, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, + "C15": {"words": 253, "then_words": 31, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 2, "accepted_words": 101}, "D1": {"words": 73, "then_words": 26, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "D2": {"words": 65, "then_words": 37, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "H1": {"words": 427, "then_words": 270, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 4, "accepted_words": 370}, + "H1": {"words": 463, "then_words": 306, "because_sentences": 3, "except": 1, "precedence": 0, "accepted": 4, "accepted_words": 370}, "H2": {"words": 508, "then_words": 172, "because_sentences": 1, "except": 1, "precedence": 3, "accepted": 3, "accepted_words": 203}, "H3": {"words": 142, "then_words": 72, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 113}, "H4": {"words": 410, "then_words": 183, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 1, "accepted_words": 127}, "H5": {"words": 757, "then_words": 289, "because_sentences": 2, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 117}, "H6": {"words": 316, "then_words": 79, "because_sentences": 3, "except": 4, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 339, "then_words": 270, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, - "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, + "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 131}, "H9": {"words": 704, "then_words": 263, "because_sentences": 3, "except": 0, "precedence": 5, "accepted": 9, "accepted_words": 354}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 714, "then_words": 435, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, @@ -38,7 +39,7 @@ "O3": {"words": 47, "then_words": 18, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O4": {"words": 91, "then_words": 57, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "O5": {"words": 478, "then_words": 160, "because_sentences": 2, "except": 6, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "O6": {"words": 572, "then_words": 138, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 7, "accepted_words": 244}, + "O6": {"words": 573, "then_words": 138, "because_sentences": 2, "except": 5, "precedence": 4, "accepted": 7, "accepted_words": 244}, "P1": {"words": 415, "then_words": 287, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 4, "accepted_words": 319}, "P2": {"words": 315, "then_words": 279, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, "P3": {"words": 639, "then_words": 598, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 366c269f7..165dd20bb 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -266,6 +266,141 @@ def test_the_comma_agreement_exceptions_are_all_still_exceptions( assert not parser.parse(comma_form).suffix, comma_form +# rules/C15's grid: the given parts behind the empty head, as title +# runs before, bodies, and trailing words, crossed. Each list holds a +# shape a comma rule has met: a title-particle and a dual second in a +# run (nameparser/AGENTS.md's head lists), a given-name title, one +# only inside a phrase, a particle run and a particle tail, a bound +# given name, a nickname alone and inside, a maiden clause, a word of +# the ambiguous credential class in each case, a generation word and +# an agnomen that are names too, a given name written short, and the +# trailing title, salutation and numerals the trailing peel reads. +_EMPTY_HEAD_TITLES = ("", "Dr.", "Mr", "Sir", "Freiherr", "St.", + "Rev. Sr.", "Dean", "Queen", "dr.") +_EMPTY_HEAD_BODIES = ( + "", "John", "John Smith", "Ma", "MA", "abdul Rahman", "de Mesnil", + "Jane van", "Jones vd", "Jones mc", "John (Bob) Smith", "(Bob)", + "Jane née Jones", "John Q.", "J.", "JOHN SMITH", "john smith", + "JOHN", "Jane Mrs", "John Smith Jr.", "John V", "Jane fils", "fils", + "Neto", "PhD", "Jr.", "Jane PhD", "Ana García López", "Ed", + "von Berg", "Chas.", "Van") +_EMPTY_HEAD_TAILS = ("", "Jr.", "PhD", "Dr.", "Mrs", "III", "V", "MA", + "Ma", "fils", "X.Y.Z.") +#: Rows rules/C15's one exception takes, H1 reading the title's lone +#: name word as the family (', Dr. John'): 1,075 under each of the +#: three orders. A count, so the exception cannot grow unseen +#: (measured 2026-10-27). +_EMPTY_HEAD_H1_ROWS = 3225 + + +def _empty_head_view(name: ParsedName, drop_first: bool = False + ) -> tuple[dict[str, str], list[str]]: + words: dict[str, list[str]] = {role.value: [] for role in Role} + for tok in name.tokens: + words[tok.role.value].append(tok.text) + if drop_first: + # the twin's surname, from whichever field it reached: a + # given-name title addresses it as the given name ('Smith, Sir + # PhD', rules/H1) + first = name.tokens[0] + words[first.role.value].remove(first.text) + return ({role: " ".join(w) for role, w in words.items()}, + sorted(a.kind.name for a in name.ambiguities)) + + +def _surname_in_the_case_of(given_part: str) -> str: + """The twin's surname, written in the case the given part's own + words are written in: a surname in another case would bring the + case contrast the empty head lacks ('Smith, MA' reads suffix 'MA' + by its capitals, rules/S6, where ', MA' and 'SMITH, MA' read given + 'MA'). Delimited words are not the name's own (rules/S6).""" + letters = [c for c in re.sub(r"\(.*?\)", "", given_part) + if c.isalpha()] + if letters and all(c.isupper() for c in letters): + return "SMITH" + if letters and all(c.islower() for c in letters): + return "smith" + return "Smith" + + +def test_an_empty_head_reads_as_its_family_comma_twin() -> None: + """rules/C15's invariant: ', X' reads as 'Smith, X' reads, less the + surname, its fields and ambiguity kinds alike, under every order. + Less the surname only: a particle the twin attaches to it + (rules/P6) is the family alone here (', Jane van' -> family 'van'). + Kinds, not details: a report's wording is not compared. + + The one exception is H1's, asserted rather than skipped: where the + twin reads a title and name words in the given field alone, the + empty head reads them as the family (', Dr. John' -> family + 'John'), the comma naming no surname to decide by. A particle that + is also a suffix word, which the twin attached to its surname + (rules/P6), then has no family to attach to and is the suffix + (', Dr. Jones vd': family 'Jones', suffix 'vd'). + + This replaces the empty head as a shape every comma sweep must hold + (nameparser/AGENTS.md): a change that breaks it gets a generic fix + or an entry in decisions/C15, never code of its own. 10,557 rows + (3,519 per order), about 2 s serially (measured 2026-10-27). + + RECORDED NEGATIVE CONTROL, measured 2026-10-27: restoring the + defect #613's review found, the comma decision returning early on + an empty head (`if not seg1 or not seg0` in `_comma.decide`), fails + 1,098 rows, ', PhD' reading given 'PhD' among them. + """ + failures: list[str] = [] + h1_rows = 0 + rows = 0 + for label, order in (("given-first", GIVEN_FIRST), + ("family-first", FAMILY_FIRST), + ("family-first-given-last", + FAMILY_FIRST_GIVEN_LAST)): + parser = Parser(policy=Policy(name_order=order)) + for parts in itertools.product(_EMPTY_HEAD_TITLES, + _EMPTY_HEAD_BODIES, + _EMPTY_HEAD_TAILS): + given_part = " ".join(p for p in parts if p) + if not given_part: + continue + rows += 1 + empty = _empty_head_view(parser.parse(", " + given_part)) + twin_text = _surname_in_the_case_of(given_part) + ", " \ + + given_part + twin = _empty_head_view(parser.parse(twin_text), + drop_first=True) + if empty == twin: + continue + fields, kinds = twin + if (fields["title"] and fields["given"] + and not fields["middle"]): + moved = dict(fields, given="", family=fields["given"]) + if fields["family"]: + # the tail P6 attached to the twin's surname, a + # particle that is also a suffix word: with no + # family to attach to it is the suffix, and the + # twin's report on the attachment goes with it + moved["suffix"] = " ".join( + w for w in (fields["family"], fields["suffix"]) + if w) + kinds = list(kinds) + if "SUFFIX_OR_NAME" in kinds: + kinds.remove("SUFFIX_OR_NAME") + if empty == (moved, kinds): + h1_rows += 1 + continue + failures.append( + f"{label}: {', ' + given_part!r} reads {empty} but " + f"{twin_text!r} less its surname reads {twin}") + assert rows == 3 * 3519, rows + assert not failures, ( + f"{len(failures)} given part(s) disagree:\n" + + "\n".join(failures[:15])) + assert h1_rows == _EMPTY_HEAD_H1_ROWS, ( + f"H1's exception took {h1_rows} rows, recorded " + f"{_EMPTY_HEAD_H1_ROWS}: read the rows it gained or lost " + f"before re-recording") + + #: The one-case-head exception class, and the recorded size of it. #: Structural rather than a name list: a name whose OWN words are #: written in one case, where the member's own writing is the only diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 08f77b51c..3ba2c0b63 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -8,11 +8,23 @@ "(MA) John Smith" ", Ana García López" ", Dean" +", Dr. John" +", Dr. Jones vd" +", Dr. abdul Rahman" ", Dr. fils" ", Jane" ", Jane Mrs" +", Jane van" +", John (Bob) Smith" +", John Smith" +", John Smith Jr." +", Jr." +", MA" ", MA Smith" ", MD/PhD" +", PhD" +", Sir John" +", Sir Jones vd" ".," "1. née Smith" "A.B. Vajpayee" @@ -727,6 +739,7 @@ "SMITH VD, JOHN" "SMITH, DEAN MR." "SMITH, JOHN DO" +"SMITH, MA" "SMITH, SAM E." "SMITH, VD DE LA MA" "SMITH, VD MA" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 460b5e447..5f8dc6b52 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -14,12 +14,24 @@ {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} {"name": ", Dean", "given": "Dean", "initials": "D."} +{"name": ", Dr. John", "title": "Dr.", "family": "John", "initials": "J."} +{"name": ", Dr. Jones vd", "title": "Dr.", "family": "Jones", "suffix": "vd", "initials": "J.", "capitalized_force": "Dr. Jones VD"} +{"name": ", Dr. abdul Rahman", "title": "Dr.", "family": "abdul Rahman", "initials": "a. R.", "capitalized_force": "Dr. Abdul Rahman"} {"name": ", Dr. fils", "title": "Dr.", "family": "fils", "initials": "f.", "capitalized_force": "Dr. Fils"} {"name": ", Jane", "given": "Jane", "initials": "J."} {"name": ", Jane Mrs", "title": "Mrs", "family": "Jane", "initials": "J."} +{"name": ", Jane van", "given": "Jane", "family": "van", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "J. v.", "capitalized_force": "Jane Van"} {"name": ", John", "given": "John", "initials": "J."} +{"name": ", John (Bob) Smith", "given": "John", "middle": "Smith", "nickname": "Bob", "initials": "J. S."} +{"name": ", John Smith", "given": "John", "middle": "Smith", "initials": "J. S."} +{"name": ", John Smith Jr.", "given": "John", "middle": "Smith", "suffix": "Jr.", "initials": "J. S."} +{"name": ", Jr.", "suffix": "Jr."} +{"name": ", MA", "given": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "M.", "capitalized": "Ma"} {"name": ", MA Smith", "given": "MA", "middle": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "M. S.", "capitalized_force": "Ma Smith"} {"name": ", MD/PhD", "suffix": "MD/PhD"} +{"name": ", PhD", "suffix": "PhD"} +{"name": ", Sir John", "title": "Sir", "given": "John", "initials": "J."} +{"name": ", Sir Jones vd", "title": "Sir", "given": "Jones", "family": "vd", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "J. v.", "capitalized_force": "Sir Jones Vd"} {"name": ".,"} {"name": "1 & 2, 3 4 5, Mr.", "title": "Mr.", "given": "3", "middle": "4 5", "family": "1 & 2", "initials": "3. 4. 5. 1. 2."} {"name": "1. née Smith", "given": "1.", "maiden": "Smith", "initials": "1."} @@ -2994,6 +3006,7 @@ {"name": "SMITH VD, JOHN", "given": "JOHN", "family": "SMITH", "suffix": "VD", "initials": "J. S.", "capitalized": "John Smith VD"} {"name": "SMITH, DEAN MR.", "title": "MR.", "given": "DEAN", "family": "SMITH", "initials": "D. S.", "capitalized": "Mr. Dean Smith"} {"name": "SMITH, JOHN DO", "given": "JOHN", "family": "DO SMITH", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "J. S.", "capitalized": "John do Smith"} +{"name": "SMITH, MA", "given": "MA", "family": "SMITH", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "M. S.", "capitalized": "Ma Smith"} {"name": "SMITH, SAM E.", "given": "SAM", "middle": "E.", "family": "SMITH", "initials": "S. E. S.", "capitalized": "Sam E. Smith"} {"name": "SMITH, VD DE LA MA", "family": "SMITH VD DE LA MA", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "S. M.", "capitalized": "Smith vd de la Ma"} {"name": "SMITH, VD MA", "family": "SMITH VD MA", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: credential as suffix (vocabulary)"], "initials": "S. M.", "capitalized": "Smith vd Ma"}