Skip to content

perf(vm): honest max_registers — drop register slack on both VM engines - #327

Merged
davydog187 merged 4 commits into
mainfrom
perf/drop-register-buffer
Jun 14, 2026
Merged

davydog187 merged 4 commits into
mainfrom
perf/drop-register-buffer

Conversation

@davydog187

@davydog187 davydog187 commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Makes codegen's max_registers an honest register peak and sizes both VM engines' register tuples to it with no slack buffer. This closes #312 (drop the interpreter's +16 buffer) and unifies that with the dispatcher-side fix merged in #359 — so there is now one register-sizing source of truth across the interpreter and the dispatcher.

Rebased onto current main (post-#359) and reconciled. See "Relationship to #359" below.

Background

Both engines over-allocated every call frame's register tuple to mask the same latent bug:

The buffers existed because peak_reg/next_reg tracking in codegen could undercount max_registers for a fixed destination register in deeply nested constructs (e.g. a concat chain inside a table-constructor list field nested in several for bodies emitted {:concatenate, 53, _, _} while reporting max_registers = 53). These shapes contain multi-return calls inside set_list, which the bytecode encoder rejects, so they only ever execute on the interpreter — which is why the dispatcher could already run buffer-free and the interpreter could not.

The fix

Make max_registers honest at the root, then drop both buffers:

  • instruction_peak/1 (new, in codegen.ex) walks the emitted instruction stream — recursing into numeric_for / generic_for / while / repeat / test / short-circuit bodies — and returns the true high-water register index + 1. It is folded into max_registers at both prototype-finalization sites.
  • Both engines now size to max(max_registers, param_count) with no buffer. Dynamic-range writers (:call results, unbounded :vararg, multi-return distribution) still grow the tuple lazily — ensure_regs_capacity/2 in the interpreter, grow_regs/2 in the dispatcher. The interpreter's unbounded :vararg (count == 0) site now grows before writing, matching the dispatcher.

Relationship to #359 (merged)

#359 recovered the #324 dispatcher regression by having the dispatcher compute its own exact reg_file_size by walking the emitted bytecode — a workaround for the codegen undercount. Now that instruction_peak/1 makes max_registers honest, that separate walker is redundant. This PR removes it (drops the reg_file_size field and the max_register_used/register_positions walker from Lua.Compiler.Bytecode) and points the dispatcher back at max(max_registers, param_count). The max_registers invariant test keeps its independent bytecode-level walker as a cross-check against instruction_peak/1 and now asserts against max_registers.

Net: the dispatcher's runtime path is byte-identical to #359 (same tuple sizes), and the interpreter joins it.

Correctness

  • mix test (default + --include slow --include differential --include lua53, the official Lua 5.3 suite): 2374 passed, 19 skipped.
  • The generative short-circuit harness that overflowed when the buffer was first removed now passes (its register peak is covered by instruction_peak/1).
  • mix format --check-formatted clean; compiles with --warnings-as-errors.

Benchmarks (Apple M-series, drift-controlled by Luerl)

  • dispatcher_vs_interpreter fib(25): interpreter ~95 ms → ~72 ms (~26% faster); dispatcher–interpreter gap closes to ~1.02× (not inverted).
  • fibonacci fib(30), dispatcher path: ~1.125× slower than Luerl — unchanged from perf(vm): size dispatcher register files from emitted bytecode #359 (the dispatcher path is byte-identical), confirming no regression from the reconciliation.
  • No other workload slower.

Closes #312.

@davydog187
davydog187 marked this pull request as ready for review June 2, 2026 23:10
@davydog187

Copy link
Copy Markdown
Contributor Author

Automated review (workflow ship-perf-issues) — 2 rounds, now stable

This PR was opened and reviewed by an AI workflow. Posting the reviewers' own findings for the human record.

Round 1 — HIGH (fixed in ef60cea)

instruction_size backstop omitted :greater_than / :greater_equal (lib/lua/compiler/codegen.ex)

The backstop that replaces the dropped +16 buffer must count every statically-fixed destination register. Codegen emits {:greater_than, dest, ..} / {:greater_equal, dest, ..} (siblings of :lt/:le, which are in the allowlist), but both fell through to the do: 0 catch-all. A >/>= whose destination is the function's high-water register — inside the same deeply-nested shape that motivated this PR — would under-count max_registers by one, with no buffer left to absorb it: a runtime ArgumentError on put_elem, not a compile failure. Exactly the #312 failure class.
→ Addressed by the follow-up commit perf(vm): count > and >= destinations in max_registers backstop.

Round 1 also flagged (low): the instruction_size(_instr), do: 0 catch-all silently hides future register-writing opcodes — the same fragility that produced this bug.

Round 2 — stable, quality-only findings remain

  • low — lib/lua/vm/vm.ex:22: the top-level chunk still allocates max_registers + 16 with a comment that now contradicts the PR's thesis. Not a regression (extra slack is safe), but it means the new instruction_peak backstop is not exercised on the main-chunk path — every adversarial probe had to be wrapped in local function f() … end to hit the buffer-free path. Either drop the +16 here too, or rewrite the comment to say it's intentional defense-in-depth.
  • low: two stale dispatcher comments + dead :scope handling (cleanup).

Verification reported by the reviewers

Full suite 2153 passed / 19 skipped; the :slow Lua 5.3 official suite passed; a 400–600 case fuzz of nested concat/arith in table constructors inside nested for loops on the buffer-free :call path produced zero register-overflow crashes. Benchmark: interpreter fib(25) ~21% faster; dispatcher/interpreter gap closed from 1.21× → 1.03× (not inverted).

Comment thread lib/lua/compiler/codegen.ex Resolved
The interpreter sized every callee register tuple at
`max(max_registers, param_count) + 16`, adding 16 slots of pure
allocation churn per frame. For deep recursion (fib(28) ~ 1.4M frames)
this dominated the dispatcher-vs-interpreter gap: the dispatcher's
`init_regs/2` has always run buffer-free.

Removing the buffer surfaced why it existed: the interpreter-only
instruction-list codegen could undercount `max_registers` for a fixed
destination register in deeply nested constructs (a concat chain inside
a table-constructor list field nested in several `for` bodies emitted
`{:concatenate, 53, _, _}` while reporting `max_registers = 53`). The
+16 buffer was masking that off-by-one; the bytecode encoder rejects
these multi-return shapes, so they only ever reach the interpreter and
never the dispatcher.

Rather than keep a fudge buffer, make `max_registers` honest:
`instruction_peak/1` walks the emitted stream (recursing into loop and
short-circuit bodies) and folds the true high-water register index into
`max_registers` at both prototype-finalization sites. Dynamic-range
writers (`:call`, unbounded `:vararg`) still grow the tuple lazily via
`ensure_regs_capacity/2`; the interpreter's unbounded `:vararg` site now
grows before writing, matching the dispatcher's `write_varargs/4`.

Correctness preserved: full `mix test` (2153) plus the `:slow` level-4
short-circuit harness (the deepest register-pressure case, previously
the overflow source) pass.

Benchmarks (Apple M4):
  dispatcher_vs_interpreter fib(25): interpreter 95.00ms -> 74.83ms;
    dispatcher/interpreter gap 1.21x -> 1.03x (closed, not inverted).
  fibonacci fib(30) lua (chunk): 846.9ms -> 808.8ms.

Closes #312
instruction_size/1 is the sound high-water backstop for max_registers
now that the +16 register buffer is gone on the interpreter :call path.
It already routed :less_than/:less_equal/:equal/:not_equal through the
allowlist, but omitted their mirrors :greater_than/:greater_equal, which
the executor writes (put_elem at the comparison dest). The omission fell
through to the catch-all clause and reported 0 slots, so a > or >=
landing on a function's high-water register could under-count
max_registers by one and crash with :badarg on the buffer-free path.

Add :greater_than/:greater_equal to the allowlist, and expose
instruction_peak/1 (@doc false) so the backstop's completeness is pinned
directly — peak_reg incidentally covers these dests in organic programs,
so the corpus walker alone cannot catch the gap.

Closes #312
Folds the dispatcher-side register sizing (#359) into the interpreter-side
honest-max_registers fix (#312). With codegen's instruction_peak/1 backstop
making max_registers a true bound on every statically-fixed destination,
the separate per-prototype reg_file_size walker #359 added to work around
the codegen undercount is redundant.

Remove it: drop the reg_file_size field and the max_register_used/
register_positions walker from Lua.Compiler.Bytecode, and size both engines
to max(max_registers, param_count) with no slack. Runtime-dynamic writes
(vararg spread, multi-return) still grow on demand (grow_regs/2,
ensure_regs_capacity/2). The max_registers invariant test keeps its
independent bytecode-level walker as a cross-check against instruction_peak/1
(which walks the pre-encoding instruction stream) and now asserts against
max_registers.

One register-sizing source of truth across both engines.
@davydog187
davydog187 force-pushed the perf/drop-register-buffer branch from ef60cea to 62f8ab3 Compare June 14, 2026 12:27
@davydog187 davydog187 changed the title perf(vm): drop +16 register buffer on interpreter :call path perf(vm): honest max_registers — drop register slack on both VM engines Jun 14, 2026
…k buffer

Addresses review feedback on the register-sizing path.

instruction_size/1 enumerated ~34 register-writing opcodes in an allowlist
with a silent `do: 0` catch-all. The list duplicated opcode knowledge that
already lives in the Instruction constructors, the dispatcher, the bytecode
encoder, and the invariant test, with nothing keeping them in sync — a new
register-writing opcode left off the list fell through to zero, undercounted
max_registers with no slack buffer to absorb it, and crashed at runtime (the
#312 failure class; how greater_than/greater_equal slipped through earlier).

Replace the allowlist with the rule it was spelling out: the destination of
every ordinary value-producing opcode is operand 1. The walker now
enumerates only the genuine exceptions — opcodes that write a range or a
fixed offset off a base, recurse into nested bodies, or write no register
(stores, returns, breaks, gotos, labels, source lines) — and defaults
everything else to operand 1. A new load/arithmetic/comparison opcode is
sized correctly with no change to the walker, and the default can only ever
over-count by a slot (benign slack), never undercount into the #312 crash.
A new guard test drives a representative instruction for every emitted
opcode through the walker and pins its slot count.

Lua.VM.execute sized the main chunk's register file to max_registers + 16
with a comment that contradicted the no-slack-buffer thesis. Drop it: size
to max(max_registers, param_count) like every other call frame, so the
backstop is exercised on the main-chunk path too. Runtime-dynamic writes
still grow the tuple lazily.

Also removes the dead Instruction.scope/2 constructor and its
instruction_size clause (emitted nowhere, no engine handler).
@davydog187
davydog187 force-pushed the perf/drop-register-buffer branch from 5036f34 to 2d1eb3f Compare June 14, 2026 13:53
@davydog187
davydog187 merged commit e3a6d0c into main Jun 14, 2026
5 checks passed
@davydog187
davydog187 deleted the perf/drop-register-buffer branch June 14, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(vm): drop +16 register buffer on interpreter :call path

1 participant