Skip to content

perf(vm): size dispatcher register files from emitted bytecode - #359

Merged
davydog187 merged 1 commit into
mainfrom
perf/dispatcher-exact-register-sizing
Jun 14, 2026
Merged

davydog187 merged 1 commit into
mainfrom
perf/dispatcher-exact-register-sizing

Conversation

@davydog187

Copy link
Copy Markdown
Contributor

Summary

Recovers the deep-recursion performance regression tracked in #324. fib(30) goes from ~1.32× slower than Luerl to ~1.11× (Apple M-series, drift-controlled by running Luerl identically before/after), with no other workload slower.

What the investigation found

The issue (and the rc.1/rc.2 CHANGELOG "Known issues") attributed the regression to the per-call depth bookkeeping added in #283. Bisecting fib(30) on this machine told a different story:

Ref fib(30) Note
v1.0.0-rc.0 ~590 ms baseline
#283 (3a4a0c8) ~812 ms depth bookkeeping (+15% at the time)
v1.0.0-rc.2 ~800 ms
#347 (df79f8f) ~996 ms +26% — register slack
origin/main ~943 ms

Two findings:

  1. feat(vm): configurable max call depth #283's depth bookkeeping is no longer the cause. Removing it entirely from current code recovers ~0% — later commits absorbed its cost. Inlining/cheapening the check (the issue's suggested fix) measures flat.
  2. The dominant regressor is perf(vm): compile bitwise ops and set_list multi-return tails #347 — a "perf" commit that added a blanket +16 register-slack buffer to init_callee_regs/4, which runs on every compiled-closure call. fib(30) is ~2.7M calls, each over-allocating a register tuple (e.g. 10 → 26 slots). That's the +26%.

The fix

The +16 slack existed to cover prototypes whose register file can transiently exceed max_registers (multi-return constructor tails like {x, f()}). But every runtime-dynamic write in the dispatcher already grows the tuple on demand via grow_regs/2 (vararg spread, multi-return result distribution, {:multi, …} expansion, pad_nils). The slack's only real job was masking a latent static undercount in codegen's max_registers.

So:

  • Compute an exact per-prototype reg_file_size in Lua.Compiler.Bytecode by walking the emitted bytecode for the highest register index any op references — authoritative even where codegen undercounts.
  • Size the dispatcher's register tuples to exactly that, no slack.
  • grow_regs/2 continues to cover runtime-dynamic growth.

This recovers the perf for all compiled prototypes (not just fib — register tuples shrink across the board) and fixes the latent undercount the slack had been hiding (see the short-circuit/concat regression case added to the invariant test).

Review-stabilization cycle

An adversarial review caught a critical latent bug in the first version of this change (verified with a reproduction): the :self opcode (obj:m()) writes register base+1 — not one of its syntactic operands — so the walker undercounted and a zero-arg method call crashed with :badarg. Fixed by handling :self structurally in both the production walker and the invariant test's walker, plus:

  • an execution-level regression test (obj:m() through the dispatcher), and
  • a :self corpus entry in max_registers_invariant_test.exs.

A follow-up focused audit (independent peak re-derivation across 88 programs + 15 end-to-end executions) confirmed no other opcode has the same bug class.

Validation

  • mix test (default + --include slow --include differential --include lua53, i.e. the official Lua 5.3 suite): all green (2368 passed with the heavy suites).
  • Both new regression tests fail without the :self fix and pass with it.
  • mix format --check-formatted: clean.
  • Full benchmark suite: every workload equal or faster vs origin/main (lua/luerl ratio).

Follow-ups (out of scope)

  • Codegen's max_registers still undercounts for some expression shapes (now harmless on the dispatcher; the interpreter retains its own +16 buffer). Worth fixing at the root in codegen's peak tracking.

Closes #324.

The dispatcher reserved a blanket +16 register-slack buffer on every
prototype's register tuple (added in #347 to cover multi-return
constructor tails). `init_callee_regs/4` runs on every compiled-closure
call, so this over-allocated a fresh tuple on every frame and regressed
call-dense, work-light workloads ~26% — fib(30) most visibly (issue #324).

Replace the slack with an exact per-prototype `reg_file_size`, derived in
`Lua.Compiler.Bytecode` by walking the emitted bytecode for the highest
register index any op references. This is authoritative even where
codegen's `max_registers` undercounts the transient peak (a real
undercount the slack had been masking — see the short-circuit/concat
regression case added to the invariant test). Runtime-dynamic writes
(vararg spread, multi-return distribution) still grow the tuple on demand
via `grow_regs/2`, so the static size only covers the syntactic peak.

fib(30): ~1.32x slower than Luerl -> ~1.11x (drift-controlled), no slower
on any other workload.

Closes #324.
@davydog187
davydog187 merged commit 3c97cce into main Jun 14, 2026
5 checks passed
@davydog187
davydog187 deleted the perf/dispatcher-exact-register-sizing branch June 14, 2026 11:50
davydog187 added a commit that referenced this pull request Jun 14, 2026
Folds the dispatcher-side register sizing (#359) into the interpreter-side
honest-max_registers fix (#312). With codegen's instruction_peak/1 backstop
making max_registers a true bound on every statically-fixed destination,
the separate per-prototype reg_file_size walker #359 added to work around
the codegen undercount is redundant.

Remove it: drop the reg_file_size field and the max_register_used/
register_positions walker from Lua.Compiler.Bytecode, and size both engines
to max(max_registers, param_count) with no slack. Runtime-dynamic writes
(vararg spread, multi-return) still grow on demand (grow_regs/2,
ensure_regs_capacity/2). The max_registers invariant test keeps its
independent bytecode-level walker as a cross-check against instruction_peak/1
(which walks the pre-encoding instruction stream) and now asserts against
max_registers.

One register-sizing source of truth across both engines.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deep recursion ~25% slower since rc.0 — cheapen per-call depth-limit bookkeeping

1 participant