Skip to content

feat(encoder)!: walk the buffer with a cursor in JS codecs - #26

Draft
Tom32i wants to merge 2 commits into
mainfrom
feat/js-codec-cursor
Draft

Tom32i wants to merge 2 commits into
mainfrom
feat/js-codec-cursor

Conversation

@Tom32i

@Tom32i Tom32i commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #25

JS codecs no longer keep the offset by hand. They receive a Cursor that moves forward on every read and write, the same way Go codecs use *bytes.Buffer. A composite codec is now just its fields in order:

encode(cursor, data) {
    this.idCodec.encode(cursor, data.playerId);
    this.idCodec.encode(cursor, data.avatarId);
}

Changes

  • New Cursor (src/encoder/Cursor.js): a DataView plus a position, with write*/read* methods for Uint8, Int8, Uint16, Int16, Uint32, Int32, Float32 and Float64, plus writeBytes/readBytes, and reserve(byteLength) to skip bytes and fill them in later (it returns their offset). It's exported from netcode, netcode/client, netcode/server and netcode/encoder.
  • BinaryEncoder creates one cursor per encode/decode call. That's one DataView per message instead of one per field. A shared, reused cursor was tried and benchmarked: it was no faster (each new buffer needs a new DataView anyway), and nested calls on the same encoder corrupted the outer message.
  • All packaged codecs now use encode(cursor, data) and decode(cursor). getByteLength(data) is unchanged.
  • String codecs: each string is encoded once, straight into the buffer (TextEncoder.encodeInto), and the length prefix is filled in from the bytes written. getByteLength counts UTF-8 bytes without encoding (utf8ByteLength, with a native ASCII check for longer strings). Cursor gains writeUtf8/readUtf8.
  • String length limits (security fix, JS and Go): strings longer than their length prefix allows (255 bytes for StringCodec, 65,535 for StringLongCodec) are now refused instead of wrapping the prefix. Before, a wrapped prefix let a crafted string forge the fields that follow it in a composite codec. For example, a player's 259-byte name relayed by the server decoded as "Bob" plus a forged role and x. JS throws RangeError (in getByteLength, so before allocating), and Go's BinaryEncoder.Encode now recovers codec panics and returns a new ErrEncodeFailed error. No measurable performance cost.
  • UIntLongCodec uses arithmetic instead of binary strings. Values too large for the byte length still throw RangeError.
  • Client catches decode errors like the server does. Errors thrown by event handlers still propagate.
  • doc/codecs.md: rewrote the custom codec example, added a reference for the Cursor methods, and added a v3 → v4 migration guide.
  • Tests: every codec test checks that the cursor moves by exactly getByteLength(value). Added a composite-codec test and a BinaryEncoder round-trip test.
  • Demo: added a position event { id, x, y } using a custom composite codec: demo/PositionCodec.js (cursor API, shared by the JS client and server), plus the matching Go PositionCodec in demo/server.go.
  • Fixes along the way:
    • test/encoder.test.js now imports from src instead of 'netcode', which resolved to an unbuilt dist/ and failed.
    • The FloatPrecisionCodec example in doc/codecs.md passed an instance; the constructor expects the class.

Performance

How many times faster this PR is than v3 (main), in messages per second through BinaryEncoder (2× = twice as many messages per second). Node 22, Apple Silicon, median of 3 runs, after warmup:

Message Encode (× faster) Decode (× faster)
UInt8 1.7× 1.8×
Composite { id, x, y } 1.6× 3.0×
Short string (12 chars) 2.2× 2.8×
String with emoji 2.4× 2.8×
Long string (4.6 KB, StringLongCodec) 3.7× 1.9×
UIntLongCodec(6) timestamp 4.5× 9.7×

Breaking change

The JS codec signatures change from (buffer, offset) to (cursor), so this needs a major version (v4.0.0). Strings over a codec's limit now throw (JS) or return ErrEncodeFailed (Go) instead of being sent corrupted. The version bump is left for the release commit. Custom codecs (e.g. the 41 client codecs in curvytron2) need migrating; see "Upgrading from v3" in doc/codecs.md.

The wire format is unchanged. The same 7 messages (uint8, 6-byte long int, bool, short and long strings, int16, float64) encode to identical bytes in JS and Go, so v3 and v4 clients can talk to each other and to the Go server. The Go side is not touched.

Test plan

  • npm test: 85 passed (includes UTF-8 length vs TextEncoder, nested encoder calls, UIntLongCodec byte order and range)
  • npm run lint, npm run build
  • go test ./go/... (includes string limits: 255/65,535 bytes accepted, one more refused, multi-byte counted in bytes)
  • Compared JS and Go encoded bytes for the same messages: identical
  • Demo run: a headless client with the demo's events against demo/server.js and demo/server.go. All events, including position, go through in both directions.

@Tom32i
Tom32i force-pushed the feat/js-codec-cursor branch 3 times, most recently from 70acb2e to b794f30 Compare October 14, 2026 15:01
Codecs now receive a Cursor that advances on every read/write:
encode(cursor, data) / decode(cursor). Composite codecs just call their
sub-codecs in order, like the Go side with *bytes.Buffer.

- Cursor: typed write*/read* methods, writeBytes/readBytes,
  writeUtf8/readUtf8, and reserve() to fill bytes in later.
- BinaryEncoder creates a cursor per call (nested calls stay safe).
- String codecs encode once, straight into the buffer, with a UTF-8
  byte count that does not encode (utf8ByteLength).
- String codecs (JS and Go) refuse strings longer than their length
  prefix allows, instead of wrapping it: a wrapped prefix let a crafted
  string forge the fields that follow it. JS throws RangeError, Go's
  BinaryEncoder.Encode returns ErrEncodeFailed.
- UIntLongCodec uses arithmetic instead of binary strings.
- Client catches decode errors like the server does.
- Demo: composite PositionCodec event, in JS and Go.
- Docs: cursor API, v3 -> v4 migration, FloatPrecisionCodec example,
  string limits, Go encode/decode errors.
- Tests import from src and cover the cursor, UTF-8 length, nested
  calls and string limits.

BREAKING CHANGE: JS codec signatures change from (buffer, offset) to
(cursor). Strings over the codec limit now throw (JS) or return
ErrEncodeFailed (Go) instead of being sent corrupted. The wire format is
unchanged.

Closes #25
data * factor is not exact in floating point: 0.29 * 100 is
28.999999999999996, which the int codec truncated to 28, so 0.29 decoded
as 0.28. Both JS and Go now round to the nearest integer, halves away
from zero, so a JS and a Go sender produce the same bytes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JS codecs: walk the buffer with a cursor, as the Go side does

1 participant