Repository navigation
Should a comma turn off the glued-honorific peel? ("田中さん, 太郎" keeps さん in the family name) #312
Description
Activity
Decision: option 2, in a shape the original three options did not name. Recording the reasoning while it is fresh.
The choice does not decide the hard part
Both option 2 and option 3 still have to answer the same question: what the peel's site is when the name spans segments. Under
FAMILY_COMMA,김, 민준씨issegments == ((0,), (1,))with the honorific attached to the given name, across the comma. A separate stage does not dodge that — it still has to pick a rule. So the choice is about structure, not about the answer.What option 3 costs, beyond the mechanics listed above
Two of the "costs" in the issue body are really one invariant and one contract:
- It makes the token-count exemption plural.
tests/v2/pipeline/test_state.pypins a per-stage ownership map, andscript_segmentis deliberately absent from the token-count assert — "the one contract this stage is exempt from." That single exemption is what keeps the anti-Strange parsing of name w lastname prefix and title before and after #100 span discipline enforceable: exactly one stage may change the token count, so exactly one place needs index-remap scrutiny. Option 3 makes it two. _PEELED_TAGbecomes inter-stage protocol. Today the peel emits it and the segmenter's neighbour test reads it, same module — which is why its comment says it needs no home in_types, unlikeFOLDED_TAG. Split them and it inheritsFOLDED_TAG's full obligation: a home in_typesand a strip on the way out (_parser.pystripsFOLDED_TAGfrom public tokens). Option 3 would remove a same-stage coupling by adding a cross-stage one.
The shape chosen
Keep one stage, restructure it into symmetric siblings:
_peel_honorific_tail(state)and_split_surname_site(state), each owning its own gates, withscript_segmentreduced to universal preconditions plus two calls:if original.isascii(): return # universal preconditions if not segments: return state = _peel_honorific_tail(state) # vocabulary-licensed if structure is FAMILY_COMMA: return # segmentation-only gates if interpunct_offsets: return ... surname / segmenter halfThis gets most of option 3's legibility with none of the invariant or tag cost, and it makes the distinction the issue is about visible in the code rather than only in prose. It also blunts option 2's stated weakness: with the peel above the gate block, a new gate naturally lands in the block below it, so re-capturing the peel takes an insertion against the visual grain rather than an ordinary edit.
What would have changed the answer, and the evidence that arrived
A stage earns itself when it has more than one inhabitant. The question was whether more vocabulary-conditional token surgery is coming.
Speculating on the candidates, they are mostly leading peels, and a leading peel is harder to vet than a trailing one — the leading position is where surnames live, and the argument that clears 양 in #308 is exactly that a surname LEADS so the trailing-only gate never sees it. The Chinese familiar prefixes 老/小 (老王, 小王 — among the commonest informal address forms in Mandarin) fail that test outright: 小 is a common given-name character, and peeling it would cut 小明 — this repo's own worked example — in half.
But #317 (Thai) is a real counterexample: นาย/นาง/นางสาว are closed-class address terms glued to the given name, plausibly clean to vet, and they need a head peel routed to
title. So a second inhabitant is more likely than the 老/小 analysis alone suggested — and the sibling shape accommodates a_peel_honorific_headas a third sibling without prejudging the stage question.If a head peel ever ships, that is the moment the two-inhabitant argument for option 3 becomes real and informed, and the refactor is no harder then than now.
Still to decide when this is implemented
The multi-segment site question, which neither option answers: under
FAMILY_COMMAthe name spanssegments[0]andsegments[1], and김, 민준씨puts the honorific on the given-name side. Moving the peel above the gates without answering this fixes the pre-comma case and leaves the post-comma one glued — trading today's clean rule ("a family comma turns the peel off entirely") for a new asymmetry inside the comma case.- It makes the token-count exemption plural.
Scope corrected, and widened
Re-measuring before implementing found that this issue's headline example was wrong, and that the fix it proposed does not work. Both are corrected in the body; the detail is here.
The example was backwards
田中さん 太郎and田中さん, 太郎agree — neither peels. The peel site is the last NON-POST-NOMINAL token, which is 太郎 in the spaced form, so nothing is there to peel. That example was written before #311's scan-back landed and never re-measured.Measured, the real disagreements:
AGREE 田中さん 太郎 family 田中さん, given 太郎 田中さん, 太郎 family 田中さん, given 太郎 DIFFER 김 민준씨 family 김, given 민준, suffix 씨 김, 민준씨 given '민준씨', family 김 DIFFER 田中 太郎さん family 田中, given 太郎, suffix さん 田中, 太郎さん given '太郎さん', family 田中 DIFFER 田中さん PhD family 田中, suffix 'さん, PhD' 田中さん, PhD title PhD, family 田中さん DIFFER 威廉·莎士比亚 さん family 莎士比亚, suffix さん 威廉·莎士比亚さん family 莎士比亚さんTwo of the four are the honorific glued to the given name, which under a family comma lives in
segments[1]— where the peel never looks. One is the 间隔号, which has the identical problem for the identical reason.Option 2 as filed does not work
Moving the peel above the gates and changing nothing else:
- fixes
田中さん, PhD✓ - does nothing for
김, 민준씨or田中, 太郎さん—segments[0]is just 김/田中 ✗ - makes
田中さん, 太郎peel, introducing a disagreement where one did not exist ✗
One of four fixed, two missed, one broken.
What the issue now covers
The site, not only the gates. Two changes that only work together:
- Site: the last non-post-nominal token across ALL segments, not just
segments[0]. Flattening segments rather than the raw token stream preserves the extracted-nickname exclusion thatko_honorific_glued_given_nicknamepins. - Placement: after the universal preconditions (ASCII-only original, empty segments), before the FAMILY_COMMA and 间隔号 gates.
Prototyped and measured: every glued/spaced pair above then agrees on the honorific,
田中さん, 太郎stays unchanged, and three tests fail, all by design —test_interpunct_divided_name_never_peels,ja_honorific_glued_family_comma, and its facade twin.test_family_comma_skips_the_peelkeeps passing but becomes misnamed: the peel is no longer skipped because of the comma, it simply finds no site.Structure stays as decided above — siblings under one stage, not a new stage.
Net: after this the peel consults no comma structure and no dot, only its own vocabulary and the two universal preconditions. That is the claim #308 made for it and did not deliver.
The ASCII bail is deliberately still in front of the peel; its known wart (a caller-configured Latin tail fires only when some other token in the name is non-ASCII) stays documented and out of scope here.
- fixes
- added 7 commits that reference this issue
on Aug 2, 2026 - added a commit that references this issue
on Aug 2, 2026 - added 2 commits that reference this issue
on Aug 13, 2026 - added 12 commits that reference this issue
on Aug 22, 2026 - added 2 commits that reference this issue
on Oct 8, 2026
#308 added a peel: a listed CJK honorific glued to the end of a name token is split off and routed to
suffix. It runs insidescript_segment, and therefore inherits that stage's two structural opt-outs — the family comma and the 间隔号. Neither gate was argued for the peel specifically; both came with the placement.That produces spelling disagreements of exactly the kind #308 set out to remove — though not the ones this issue originally named (corrected 2026-08-01):
The honorific is glued to the GIVEN name in both, which under a family comma lives in
segments[1]— where the peel never looks.#308 treated this shape as a defect worth fixing elsewhere —
ko_honorific_glued_given_trailing_suffixexists precisely so김민준씨 Jr.andDr 김민준씨, Jr.agree — and then left the identical disagreement standing under a family comma (ja_honorific_glued_family_comma). Two rows in the same table are pinned to opposite conclusions about the same phenomenon.Why the gates may not be the peel's to inherit
Both gates answer where does a name divide into surname + given. AGENTS.md states the comma doctrine as "script-conditional behavior is ignored where a comma already decides the family", and the interpunct gate as "a divided name is a transcription — its pieces are syllable groups".
The peel does not ask that question. It asks a position-independent one: does this token end in a word that can never end a name? A comma elsewhere in the string does not change the answer, and the vetting behind
GLUED_HONORIFICSis likewise position-independent. The peel is also explicitly not gated onsegment_scriptsfor exactly this reason — the vocabulary carries its own license rather than borrowing the script's.Measured, if the peel simply moves above both gates
Four tests fail: the two stage tests pinning the gates,
ja_honorific_glued_family_comma, and its facade twin.The part that makes this design work, not a two-line move
Look at the last row. Under
FAMILY_COMMAthe name spans two segments:The peel scans
segments[0]only. So moving the call fixes the pre-comma side and leaves the post-comma side glued — trading today's clean rule ("a family comma turns the peel off entirely") for a new asymmetry inside the comma case. Doing it properly means deciding what the peel's site is when the name spans segments, and for김, 민준씨the honorific is attached to the given name — a different question from the one the peel answers today.Options
segmentandscript_segment. Makes the ordering structural rather than a comment and ends gate inheritance by construction. Costs:_PEELED_TAGbecomes cross-module,_split/_longest_entryneed a shared home, and "the one stage licensed to change the token count" becomes two.Timing
2.1.0 is unreleased. If this lands before the release, no user ever sees the inconsistency; if it doesn't, option 1's documentation is needed so the boundary is at least stated rather than accidental.
Found by an altitude review on #311.