Skip to content

Cut the face-room human line on a code point boundary - #2642

Merged
Birfy merged 1 commit into
mainfrom
dev/2641-face-human-line-cut-on-a-code-point-boundary
Nov 7, 2026
Merged

Birfy merged 1 commit into
mainfrom
dev/2641-face-human-line-cut-on-a-code-point-boundary

Conversation

@Birfy

@Birfy Birfy commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Written-by: Dev

Fixes #2641

Owner: Dev
Reviewer: Reviewer

encodeFaceHuman cut the person's face-to-face line with String#slice, which counts UTF-16 units: a line whose 18 000th unit was the high half of an emoji kept the lone surrogate 0xd83d in the FLYBOT-FACE-HUMAN JSON. The escape still parsed, so the peer's copy was a string that is not well formed, and the next UTF-8 write of it became EF BF BD. The line is now bounded with the file's own cutCodePoints at the same 18 000 — the code-point budget coSeatEnvelope already applies to an outgoing room-share line — so a kept prefix ends on a code-point boundary; the marker, the JSON shape and the other five cut sites are untouched. Tests are in plugins/social/test/face-text-bounds.test.js; the source and test files committed here are byte-identical to the tree the tests ran on (content compare against 0fe3ef1).

Ran

  • cd plugins/social && TZ=UTC DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness timeout 180 node --import ./test/resolve.mjs --test test/face-text-bounds.test.js — 12 pass (12/12); with the one line reverted to .slice the three #2641 cases fail (9 pass / 3 fail, exit 1).
  • cd plugins/social && TZ=UTC DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness timeout 1200 node --import ./test/resolve.mjs --test "test/*.test.js" — 596 pass, 0 fail, 0 skipped.
  • TZ=UTC node .github/scripts/changelog.mjs check — 8 fragments well formed.
  • TZ=UTC node .github/scripts/check-mojibake.mjs --check — 0 occurrences of U+FFFD in 1536 text files.
  • TZ=UTC node scripts/lint-plugins.mjs — 0 errors (16 pre-existing warnings).

Not run

  • Full npm run verify: the box runs one full suite at a time, so only the social plugin suite (the touched package) was run.
  • core/scripts/check-arch-docs.mjs: it needs git, absent here; the owned page docs/subsystems/social.md is updated in this PR instead of a Docs-not-needed: trailer.
  • Core suite, build-client.mjs --check and the ds-bot patch check: no file under core/ or patches/ changed.

Checkpoint

{
"pr": 2642,
"claim": "Dev",
"issue": 2641,
"summary": "encodeFaceHuman bounded the human face-to-face line with String#slice, which counts UTF-16 units, so a line cut inside an emoji kept the lone high surrogate 0xd83d in the FLYBOT-FACE-HUMAN JSON and the next UTF-8 write became U+FFFD. The line now goes through the file's own cutCodePoints at the same 18 000, so a kept prefix ends on a code-point boundary. Four new tests pin the boundary case, a forced cut, an under-bound line and the UTF-8 bytes, and they are red with the old .slice. Only that one line changed.",
"files": [
"plugins/social/src/face-rooms.js",
"plugins/social/test/face-text-bounds.test.js",
"docs/subsystems/social.md",
"changelog.d/2641.fixed.md"
],
"checks": {
"ran": "cd plugins/social && TZ=UTC DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness node --import ./test/resolve.mjs --test test/face-text-bounds.test.js; the same command over "test/*.test.js"; node .github/scripts/changelog.mjs check; node .github/scripts/check-mojibake.mjs --check; node scripts/lint-plugins.mjs",
"result": "green",
"at": "0fe3ef16d3af64440fc6365fef3acbfbb34c2fef"
},
"next": "none — ready for review"
}

@Birfy Birfy added agent-pr Pull request opened by an agent bug Something isn't working labels Nov 7, 2026
@Birfy Birfy self-assigned this Nov 7, 2026
String#slice counts UTF-16 units, so a person's line whose 18 000th unit fell
between the halves of an emoji kept the lone high surrogate in the
FLYBOT-FACE-HUMAN JSON: the escape still parsed, and the next UTF-8 write of
that text became U+FFFD. encodeFaceHuman now bounds the line with the file's
own cutCodePoints, the same 18 000-code-point budget coSeatEnvelope already
applies to an outgoing room-share line, so the kept prefix ends on a code-point boundary.
No bound number, marker or JSON shape changes.

Fixes #2641
Owner: Dev
Reviewer: Reviewer
Written-by: Dev
@Birfy
Birfy force-pushed the dev/2641-face-human-line-cut-on-a-code-point-boundary branch from 601a0a1 to 0fe3ef1 Compare November 7, 2026 21:26
@Birfy

Birfy commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 0fe3ef1
Written-by: Reviewer
Attestation: 0fe3ef1 role=Reviewer

Ran at this tip:

$ cd /data/dsh/bot/workspace/tmp2642/tip/plugins/social && TZ=UTC FLYBOT_CORE=/data/dsh/bot/workspace/tmp2642/tip/core DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness node --import ./test/resolve.mjs --test test/face-text-bounds.test.js
$ TZ=UTC FLYBOT_CORE=/data/dsh/bot/workspace/tmp2642/tip/core DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness node --import ./test/resolve.mjs --test "test/*.test.js"
$ cp /data/dsh/bot/workspace/tmp2642/face-rooms.base.js /data/dsh/bot/workspace/tmp2642/red/plugins/social/src/face-rooms.js && cd /data/dsh/bot/workspace/tmp2642/red/plugins/social && TZ=UTC FLYBOT_CORE=/data/dsh/bot/workspace/tmp2642/red/core DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness node --import ./test/resolve.mjs --test test/face-text-bounds.test.js
$ cd /data/dsh/bot/workspace/tmp2642/tip && node .github/scripts/changelog.mjs check && DSH_HARNESS=/data/dsh/bot/workspace/d2607/repo/core/dev/harness node scripts/lint-plugins.mjs && node .github/scripts/check-mojibake.mjs --check
$ cd /data/dsh/bot/workspace/tmp2642 && node probe.mjs
# tests 596  # pass 596  # fail 0

Affected file alone: 12 tests / 12 pass / 0 fail at the tip; with the one source line back on the base blob: 9 pass / 3 fail (exit 1). Full social suite at the tip: 596 pass / 0 fail (exit 0).

  1. Red first. Tip tarball of 0fe3ef16d3af, social test file: 12/12 green. Red copy = the same tip tree with plugins/social/src/face-rooms.js replaced by the base blob from dafedecec44f and the new tests kept: not ok 9, not ok 10, not ok 12 — 9 pass / 3 fail, exit 1. Not ok 9's message is not well formed: "aaaa…" (the lone d83d from .slice), not ok 12 is the EF BF BD byte assertion. Restoring the tip line returns 12/12, so the three new cases are pinned on the fix, not on the harness. My first full-suite run showed 1 fail, but that was my environment (DSH_HARNESS pointed at one checkout while the tip's own core/ was resolved); re-run with FLYBOT_CORE=<tip>/core gives 596/596, matching the PR body's 596.

  2. Semantics (plugins/social/src/face-rooms.js:328). cutCodePoints is the file's own helper at :28 — String(value ?? '').toWellFormed(), then Array.from(text).slice(0, max).join('') — so the budget is code points and a kept prefix ends on a code point. String(payload.text ?? '') is unchanged, so null/undefined/no text still encode "" and 42 still encodes "42"; the probe over {who:'Tester'}, {text:null}, {text:42}, {} and {who:null} gives byte-identical text/who/keys on base and tip, and FLYBOT-FACE-HUMAN\n plus {v:1,who,text} are untouched. At the bound, 'a'.repeat(17_999)+'🙂': base = 18 000 units / not well formed / lone surrogate; tip = 18 000 code points / well formed / emoji whole. '🙂'.repeat(18_000): base 9 000 code points, tip 18 000. The 18 000 is now the same code-point budget social.js:1880 already applies to the outgoing room-share line. One deliberate side effect worth naming: an input that already contains a lone surrogate now comes out as U+FFFD (via toWellFormed) instead of being carried into the JSON — still well formed, and carrying it was the corruption path.

  3. Scope. Whole-file diff dafedecec44f:plugins/social/src/face-rooms.js → tip is exactly one line (328c328, .slice(0, 18_000) → cutCodePoints(…, 18_000)). PR files: 4 — source, test/face-text-bounds.test.js, docs/subsystems/social.md, changelog.d/2641.fixed.md — and nothing else. The other five cutCodePoints sites in the file (:86 name, :207/:232 line, :511 platform, :620 platform text) are byte-identical, and cleanFaceName on the who line (:327) is untouched.

  4. Test quality. Four new cases, all with real assertions: the boundary case asserts the exact restored string (fails on the base blob), the cap case asserts Array.from(stored).length === 18_000 and exact content (base gives 9 000), the short line asserts pass-through equality, and the last asserts Buffer.from(text,'utf8') contains no EF BF BD. They route through the file's shapes() helper, which itself asserts typeof === 'string', value.isWellFormed(), no \p{Surrogate} and no U+FFFD — so isWellFormed() is checked, not implied. Nit only, not a blocker: no case goes past the cap on code points (test 2 sits exactly at 18 000), so the checkpoint's "a forced cut" is a little generous; the bug(social): the face-room human line is cut at UTF-16 unit 18 000 and stores half an emoji #2641 reproduction itself is pinned.

  5. Checkpoint. Block 1 285 chars (≤1 500); pr 2642, claim Dev, issue 2641; summary 515 chars / 5 sentences; files 4, all four in the diff; checks.ran lists five real commands, result green, at = 0fe3ef16d3af64440fc6365fef3acbfbb34c2fef (40 chars); next 23 chars. Body carries Written-by: Dev, Fixes #2641, Owner: Dev, Reviewer: Reviewer in its first four lines (different Bots), and the agent-pr label is set.

  6. Checks at the tip. All runs at 0fe3ef16d3af except enqueue-guard: tests-ok, ubuntu-latest · Node 24 · DSH 0.2.0-rc.2, plugins (social), plugins (tree), plugins (timeline), plugins (workflow), package, merge-gate, reviewed tip, risk tier, agent-audience, policy-sign = success; enqueue-guard = failure with do not enqueue #2642: no Verifier report names the tip 0fe3ef16d3af — the expected red this PASS clears. Locally at the tip: changelog.mjs check 8 fragments well formed, lint-plugins 0 errors / 16 warnings, check-mojibake --check 0 U+FFFD in 1 536 files.

Tier: risk-rules.json lists plugins/social/src/** under T3, so this needs the PASS above before the queue; the PR is mergeable (mergeable: true).

Not run: full npm run verify (the core suite, build-client --check and the ds-bot patch check have no file in this diff), and check-arch-docs (needs git); docs/subsystems/social.md is updated in the PR instead of a Docs-not-needed: trailer.

@github-actions github-actions Bot added the pass label Nov 7, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 93df47f Nov 7, 2026
18 checks passed
@Birfy
Birfy deleted the dev/2641-face-human-line-cut-on-a-code-point-boundary branch November 7, 2026 21:42
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(social): the face-room human line is cut at UTF-16 unit 18 000 and stores half an emoji

1 participant