[pyramation]

Summary

pgpm import failed on two kinds of input:

1. Non-ASCII text before a statement. Statement spans from the parser are UTF-8 byte offsets, but pgpm/import sliced the JS string with them (UTF-16 offsets). After the first multibyte character (for example an em dash in a comment), every later statement was mis-cut and failed to reparse:

   -- owner — the db
   CREATE SCHEMA app;
   CREATE TABLE app.users (id int PRIMARY KEY);

   → SqlError: syntax error at or near "EATE".
   statementText / reorderRevertStatements now slice the bytes with the sliceStatementBytes / sqlSourceBytes helpers pgpm/transform already uses. They are now exported from @pgpmjs/transform. The source is encoded once per script, not once per statement, so large dumps stay linear.
2. Scripts where every statement lands in misc/ (for example only REVOKE / ALTER DEFAULT PRIVILEGES / DO). restructureChanges passed an empty string to the parser, which threw. It now returns { changes: [], warnings: [] }.

Tests: pgpm/import/__tests__/import-rows.test.ts (UTF-8 slicing, misc-only script) and pgpm/transform/__tests__/granularity-driver.test.ts (empty input). On main these fail (2 failed in import-rows, 1 in granularity-driver). With this change they pass: import 25/25, transform 125/125.

Link to Devin session: https://app.devin.ai/sessions/e14e75c37ff2404493d13f540df3d54f
Open in Devin Desktop: https://app.devin.ai/desktop/session/e14e75c37ff2404493d13f540df3d54f?variant=devin
Requested by: @pyramation

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[tenki-reviewer]

Review complete. 🟡 1 medium

💬 Inline comments (1)

• 🟡 Add 4-byte and combining-mark cases to byte-slice tests — import-rows.test.ts:164

---

This PR switches statement extraction from character-based slicing to UTF-8 byte offsets so libpg-query span data stays correct for dumps containing non-ASCII text, and hardens the transform granularity driver's empty-input path. Coverage: byte-safe slicing in the import pipeline and empty-input behavior in the granularity driver, with tests exercising accented text and empty restructure inputs.

Files	Change
pgpm/import/src/import.ts	Slices statements using UTF-8 byte offsets instead of character indices, with a byte-slice helper handling the final statement's undefined length.
pgpm/transform/src/granularity-driver.ts	Early-returns on empty input in the restructure path without emitting flatten diagnostics.
pgpm/transform/src/index.ts	Re-exports the updated transform entry point.
pgpm/import/__tests__/import-rows.test.ts	Adds row tests covering non-ASCII statement text through the import pipeline.
pgpm/transform/__tests__/granularity-driver.test.ts	Adds tests for empty-input restructure behavior.

_{Reviewed commit: 890c82e}

[tenki-reviewer]

Fixes UTF-8 byte-offset statement slicing in the pgpm import pipeline and empty-input handling in the transform granularity driver, with updated tests.

Key findings

• 🟡 Add 4-byte and combining-mark cases to byte-slice tests — import-rows.test.ts:164

[tenki-reviewer]

🟡 suggestion · medium

Add 4-byte and combining-mark cases to byte-slice tests

The new byte-offset test (pgpm/import/tests/import-rows.test.ts:164) covers only an em dash (3 bytes) and é (2 bytes), never a 4-byte code point such as an emoji, whose surrogate-pair UTF-16 length (2 chars vs 4 bytes) is exactly where byte vs JS-index slicing diverges most. Combining marks (e.g. e\u0301) and the documented len === undefined end-of-source branch in sliceStatementBytes (pgpm/transform/src/byte-slice.ts:24) are likewise untested, though the module doc cites emoji as a motivating case. A regression in surrogate or final-statement handling would pass the current suite.

📋 Prompt for AI Agents

In pgpm/import/tests/import-rows.test.ts around lines 164-179, extend the UTF-8 byte-offset test (or add sibling tests): one whose source includes an emoji such as 'users 🎉' inside a COMMENT string and a combining-mark sequence such as 'cafe\u0301', asserting the row content contains the uncorrupted literal; plus a unit test calling sliceStatementBytes(Buffer.from('prefix — SELECT 1;', 'utf8'), byteOffsetOf('SELECT'), undefined) asserting the result equals 'SELECT 1;' to pin the len-undefined end-of-source branch.