[miguel-heygen]

What

A project page saved as UTF-16 now previews, renders and lints as its text. Before, its preview was black and the render had nothing to draw. Every read of a project page now honours the byte order mark at the start of the file, as a browser does: a UTF-16LE or UTF-16BE mark decodes the file as UTF-16, and a UTF-8 mark is dropped.

Why

Windows PowerShell 5.1 writes files from >, Out-File and Set-Content as UTF-16LE with a byte order mark, so an agent working in a Windows shell can save a page that way. The preview bundler, the render compiler, the asset reader and lint each read project files as UTF-8. A UTF-16 page then came out as <\0!\0d\0o\0c\0t\0y\0p\0e… right after <body>, the root composition was missing, and the preview stayed black at 00:00. The same file opened directly in Chrome looks fine, because Chrome honours the mark, so the page itself seemed correct.

Related work

The CLI already decodes PowerShell's UTF-16 for JSON flags (packages/cli/src/utils/jsonFlag.ts). This does the same for the files a bundle reads.

How

• parsers/src/assetResolution.ts: decodeTextHonoringBom is the one decoder. readProjectFile, which reads sub-compositions and assets, uses it, and readTextFile(path) exports it for whole-file reads.
• core htmlBundler.ts (preview: the index, sub-compositions, stylesheets, scripts), producer htmlCompiler.ts compileForRender and renderOrchestrator.ts (the render entry and the index shell for a template entry), and lint project.ts (the root and composition pages) read through readTextFile.
• A file with no mark reads as UTF-8, as before. A UTF-16BE file with an odd byte count drops the dangling byte instead of throwing.

Test plan

fails without the fix: packages/core/src/compiler/htmlBundler.test.ts › bundleToSingleHtml › reads a page saved as UTF-16LE by its byte order mark, as a browser does

• Unit tests added/updated
  • parsers assetResolution.test.ts: UTF-16LE, UTF-16BE, UTF-8 with a mark and plain UTF-8 read to the same text through readTextFile and readProjectFile.
  • core htmlBundler.test.ts: a root page and a sub-composition in each encoding bundle with no NUL or U+FEFF, root and sub-composition text present.
  • producer htmlCompiler.encoding.test.ts: compileForRender of a UTF-16LE root page and sub-composition has no NUL, keeps the root and the scene's text.
  • lint project.test.ts: a UTF-16LE page gets the same findings as the same page in UTF-8.
  • Against main every new test fails. On 0.8.206 a UTF-16LE page bundled with 193 NULs after <body>.
  • All four files pass 3 runs in a row (parsers 39, bundler 211, producer 1, lint 76). Typecheck for parsers, core, producer and lint, oxlint, oxfmt, the comment citation and comment-share checks and the fallow audit are clean.
• Manual testing performed (not on Windows)
• Documentation updated (if applicable)
• Comments follow CONTRIBUTING.md "Comments"

[heygengenesis]

Verdict: REQUEST CHANGES · 2 blockers · 0 important · 0 nits · head ac1246ad

1. Blocker: export is still broken for a UTF-16 page. hyperframes render never goes through htmlBundler. compileStage → compileForRender reads the entry with readFileSync(htmlPath, "utf-8") (producer/src/services/htmlCompiler.ts:2040). Sub-compositions and assets go through readProjectFile → readFileSync(fd, "utf-8") (parsers/src/assetResolution.ts:240). renderOrchestrator.ts:3140/3150 reads the template-entry and index shell the same way. A PowerShell-saved index.html therefore still compiles to NUL-laden text and renders nothing. That is the "no export" half of the title, and the body says the render is fixed. Please move readText somewhere both sides can import (for example parsers, next to readProjectFile) and use it at those reads, and add a compileForRender UTF-16LE test. lint/src/project.ts:217/251 has the same issue, so lint will report the wrong root.
2. Blocker: Comments fails (run 40552161421). RAISED comment share: packages/core/src/compiler/htmlBundler.ts is 10.5% (181/1729), at the base 10.4% (179/1717). The 2-line JSDoc on readText is what pushes it up. A name such as readTextHonoringBom can carry that meaning instead.

Scope

The decode logic itself is correct. LE strips the BOM. BE copies before swap16, so the read buffer isn't mutated, and it trims an odd trailing byte. UTF-8 strips U+FEFF. Studio preview (adapter.bundle, and the cli studioServer → bundleToSingleHtml) is fixed by this PR. CI was still running when I posted. No prior reviews.

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 135 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[miguel-heygen]

Both fixed in 7d9bbce.

1. Export and lint: the decoder now lives in parsers (decodeTextHonoringBom in assetResolution.ts). readProjectFile uses it, and readTextFile exports it. htmlBundler, compileForRender, both renderOrchestrator reads and lint's project.ts page reads go through it. New tests: htmlCompiler.encoding.test.ts (UTF-16LE root and sub-composition through compileForRender), a parsers test for all four encodings, and a lint test that a UTF-16LE page lints like the UTF-8 one. Each fails on main.
2. Comment share: the doc comment moved out of htmlBundler.ts with the decoder. comment-ratchet reports all 5 changed source files at or under their share at the merge-base.

[heygengenesis]

Verdict: APPROVE · 0 blocker · 0 important · 0 nits · ac1246ad → 7d9bbce3

✅ Both blockers are resolved. Every reader now decodes through decodeTextHonoringBom in parsers: readProjectFile, compileForRender, the entry and index-shell reads in renderOrchestrator, htmlBundler, and lint's page reads. A compileForRender UTF-16LE test covers the render path, and Comments is green.

CI

In CI run 40559434763, the Studio edit accuracy gate failed one case (seqplainkeys-none-pct-r0-nested-z100, undo 2 of 3). Every other CI job in that run passed. Run 40559499429 on the same head passed everything, the gate included, so I'm treating that one failure as a flake. Regression and Windows render are green.

[heygengenesis]

Request changes: the body has no fails without the fix: line.

Fix PRs in this lane have to name one test that fails without the fix, on a line that starts fails without the fix:. The Test plan does say "Against main every new test fails", but it doesn't name a test on that line. Something like this would do:

fails without the fix: packages/parsers/src/assetResolution.test.ts › reads UTF-16LE … (use the real test name, or the htmlBundler.test.ts UTF-16 case)

Once the body has that line I'll approve the same head. A body-only edit needs no new commit.

Code at a0d4d3d9, no blockers found:

• decodeTextHonoringBom handles the FF FE and FE FF marks, strips a UTF-8 mark, and leaves no-mark files as UTF-8, same as before. On an odd-length UTF-16BE file it drops the dangling byte before swap16, so that doesn't throw.
• Every caller of safeReadFile in htmlBundler.ts reads text (HTML, CSS, JS), so honoring the mark there can't corrupt binary reads.
• readProjectFile still reads through one fd, and the type check and read still see the same file.
• CI: the regression shard-10 failure is a Docker cache fetch error (could not fetch content descriptor … not found) that happened before any test ran. The change didn't cause it, but it needs a re-run. CI and Windows render verification were still running when I checked.

[miguel-heygen]

Added the line to the body. I ran the new htmlBundler.test.ts against main's code and it fails there: bundleToSingleHtml > reads a page saved as UTF-16LE by its byte order mark, as a browser does. The branch is also rebased on current main (head a0d4d3d9).

[heygengenesis]

Approve. The blocker from my last review is fixed, and I found no other blocker at a0d4d3d9.

• The new line: fails without the fix: packages/core/src/compiler/htmlBundler.test.ts › bundleToSingleHtml › reads a page saved as UTF-16LE by its byte order mark, as a browser does. That test is the UTF-16LE case of the it.each at line 219.
• Checked locally:
  • At this head the test passes: 1 passed, 213 skipped.
  • I put htmlBundler.ts back to the merge-base version and kept the parsers change. The test then fails with AssertionError: expected '<!DOCTYPE html>…' not to contain '\u0000'.
• Code: my earlier notes still hold. decodeTextHonoringBom is correct for all three marks and for files without one. Every caller of safeReadFile reads text.
• CI on this head: CI, Windows render verification, preview-regression, CodeQL, Player perf and PR captures pass. The first regression run failed on the Docker cache fetch error, before any test ran. A second regression run was still going when I checked.