Skip to content

compiler: key scope resolution by node id and dedupe upvalue descriptors - #400

Merged
davydog187 merged 6 commits into
mainfrom
perf/compiler-front
Jul 27, 2026
Merged

davydog187 merged 6 commits into
mainfrom
perf/compiler-front

Conversation

@davydog187

@davydog187 davydog187 commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Two measurement-backed fixes to the compiler front end, plus the harness fix CI needed to go green. The compiler changes are behaviour-preserving except where they move us toward PUC-Lua, and one of them fixes a latent miscompile found along the way.

1. Key scope resolution by integer node ids

Scope.State.var_map keyed its per-node entries by the AST node itself, so every write and every codegen read hashed the node's whole subtree. The worst keys were the most useful ones: on the playground metatables snippet an %Expr.Function{} key took ~2us to :erlang.phash2 and ~0.7us per Map.get, and the 34 keys of that 370-byte chunk held 2,912 words of AST between them.

Every node now carries a chunk-unique meta.id, stamped by one pass over the parsed chunk (Lua.AST.Ids, called from Lua.Parser.parse_chunk/1), and var_map keys on that integer. Scope and codegen both go through Scope.node_key/1, which falls back to the node term when a node has no id — hand-built ASTs (tests, Lua.AST.Builder) keep working unchanged.

It also fixes a latent miscompile

Every empty block parses to the same term — %Lua.AST.Block{stmts: [], meta: nil} — so all the empty blocks in a chunk shared a single {:block_close_threshold, block} entry and the last one resolved decided the close-upvalue watermark for all of them. db.lua alone has six. A block that borrowed a lower watermark closed cells belonging to live enclosing locals:

local t = {}
do
  local a, b, c, d = 1, 2, 3, 4
  local n = 0
  local get = function() return n end
  local set = function(v) n = v end
  for i = 1, 1 do end
  set(42)
  t[1] = get()
  t[2] = n
end
for i = 1, 1 do end
return t[1], t[2]

PUC-Lua prints 42 42; main returns 42 0 (both empty loop bodies compiled to close_upvalues 5, so the first one detached n's cell). Pinned in test/lua/vm/upvalue_test.exs, both new tests fail on main.

2. Dedupe upvalue descriptors per function

ensure_upvalue/4 checked whether the parent already held an upvalue for a name, but never whether the function being given one did. Every reference to a free variable appended a fresh descriptor: fib carried [{:parent_local, 0, "_ENV"}, {:parent_local, 1, "fib"}, {:parent_local, 1, "fib"}], and luassert's util.lua has a closure with eight identical {:parent_local, 0, "_ENV"} slots.

The duplicates were pure waste. Closure creation resolves {:parent_local, reg, _} through the single open-upvalue cell for reg and {:parent_upvalue, i, _} through the enclosing closure's slot i, so identical descriptors always denoted the same cell. The extra slots only cost a build_upvalues/5 iteration and a Map.get(open_upvalues, reg) each at closure creation, widened every closure term, and forced redundant get_upvalue instructions.

A function now reuses the slot when it already carries an identical descriptor. The dedupe is on the whole {kind, index, name} tuple, not the name, and is per-function: shadowed captures that share a name stay apart. The insert also finds the index and appends in one traversal instead of length/1 plus ++ [descriptor].

Behaviour change: debug.getupvalue/setupvalue numbering

local a, b = 1, 2
local function f() return a + b + a end
-- PUC-Lua 5.x: a, b          (verified against the reference interpreter)
-- before:      _ENV, a, b, a
-- after:       _ENV, a, b

One slot per upvalue rather than one per reference — this moves us toward PUC. (The leading _ENV on every nested function is pre-existing and unchanged.) No test pinned the duplicated numbering; the existing debug tests all use functions that reference each free variable once.

The set_upvalue i, r + get_upvalue d, i -> move d, r peephole is deliberately not here — it belongs to the codegen-peephole PR rather than to scope.ex.

3. Website sandbox: cut the worker link before untrapping exits

CI's website job caught a knife-edge race this PR tips over, in Website.LuaSandbox.run/1. It spawn_links the worker and traps exits only for its own duration. On the wall-clock timeout it sent Process.exit(worker, :kill) and returned immediately, leaving the worker's exit signal in flight — measured at 10–20us round trip, while the after clause restores trap_exit in a fraction of that. Whether the caller saw a harmless {:EXIT, _, :killed} message or an untrapped :killed signal that took it down came down to a handful of microseconds.

Measured on this machine, worker-death latency after Process.exit/2: 10–12us on main, 16–22us on this branch — enough to flip playground/timeout (limit) from pass to a deterministic ** (EXIT from #PID<...>) killed. Nothing about the snippet's compiled output changed: the emitted instruction stream and the worker's heap profile are byte-for-byte identical on both builds.

The result branch already handled this — unlink, then flush a signal that may already be queued. That is now unlink_worker/1, used on the timeout path too. Process.unlink/1 guarantees no exit signal from that link is delivered after it returns, so the caller is safe to stop trapping regardless of timing. website/test/website/lua_sandbox_test.exs pins it; the new timeout test fails without the fix.

The website suite now passes 5/5 consecutive runs (54 tests); before the fix it failed 5/5.

Measurements

MIX_ENV=prod, medians of 500-iteration batches over 41 reps (20/21 for the larger files). Two paired runs against main; ranges show the spread, the machine was not quiesced.

Small snippets, us:

snippet Scope.resolve Compiler.compile parse_chunk front end total
metatables (370 B) 18.5–19.0 -> 6.7–7.5 39.2–45.0 -> 21.3–27.2 1.9–2.0 -> 4.4–4.7 41.2–47.0 -> 25.8–31.9
tables 8.7–10.1 -> 4.1–4.4 23.7–26.7 -> 16.2–17.4 1.4–1.5 -> 2.9–3.1 25.1–28.2 -> 19.1–20.5
fib 4.1–4.5 -> 2.5–2.7 11.8–13.3 -> 9.2–10.2 1.0 -> 1.8 12.8–14.3 -> 11.0–12.0

Larger sources, us:

file Scope.resolve Compiler.compile parse_chunk
luassert util.lua (11.4 KB) 322–363 -> 91–99 688–955 -> 287–328 54–60 -> 87–143
luassert spy.lua (7.1 KB) 192–217 -> 73–76 482–610 -> 219–249 32–36 -> 53–57
strings.lua (13.3 KB) 244–262 -> 100–104 1411–1591 -> 1026–1128 89–95 -> 116–122
closure.lua (5.5 KB) 260–263 -> 114–121 720–724 -> 413–476 56–59 -> 83–88
constructs.lua (7.5 KB) 238–240 -> 83–89 754–902 -> 483–544 63–67 -> 146–156

The numbering pass is the whole parse_chunk regression; measured on its own it is 1.7us for the metatables chunk. It buys back 5–10x that in Scope.resolve and codegen.

Runtime (Lua.eval!) is unchanged within noise, as expected — this is all compile-time work.

Validation

  • mix format --check-formatted, mix compile --force --warnings-as-errors, mix docs --warnings-as-errors, mix dialyzer — all clean.
  • mix test --include lua53 --include slow --include differential: 2654 passed, 0 failures, 16 skipped. closure.lua and nextvar.lua from the official suite exercise upvalues heavily.
  • Differential dump of every prototype (instructions, upvalue_descriptors, upvalue_names) for all 29 Lua 5.3 suite files, the luassert/say integration sources and the snippet battery, plus evaluated results for 14 closure-heavy programs, diffed main vs this branch. Every difference is a get_upvalue/set_upvalue index renumbering, an upvalue_descriptors/upvalue_names list losing duplicates, or one of the three close_upvalues thresholds in db.lua that the empty-block fix corrects. No other opcode moved, and the only evaluated result that changed is the debug.getupvalue listing above.
  • No prototype in that surface carries a duplicate descriptor any more (486 did before). Asserted in test/lua/compiler/upvalue_descriptor_test.exs.
  • Website suite: 54 passed, 5/5 consecutive runs.

`Scope.State.var_map` keyed its per-node entries by the AST node itself,
so every write and every codegen read hashed the node's whole subtree.
The worst keys were the most useful ones: an `%Expr.Function{}` key on
the playground metatables snippet took ~2us to `:erlang.phash2` and
~0.7us per `Map.get`, and the 34 keys of that 370-byte chunk held 2,912
words of AST between them.

Give every node a chunk-unique `meta.id`, assigned by a single pass over
the parsed chunk (`Lua.AST.Ids`, called from `Lua.Parser.parse_chunk/1`),
and key `var_map` by that integer. Scope and codegen both go through
`Scope.node_key/1`, which falls back to the node term when a node has no
id, so hand-built ASTs keep working unchanged.

This also fixes a latent miscompile. Every empty block parses to the
same term (`%Lua.AST.Block{stmts: [], meta: nil}`), so all the empty
blocks in a chunk shared one `{:block_close_threshold, block}` entry and
the last one resolved decided the close-upvalue watermark for all of
them. A block that borrowed a lower watermark closed cells belonging to
live enclosing locals:

    local t = {}
    do
      local a, b, c, d = 1, 2, 3, 4
      local n = 0
      local get = function() return n end
      local set = function(v) n = v end
      for i = 1, 1 do end
      set(42)
      t[1] = get()
      t[2] = n
    end
    for i = 1, 1 do end
    return t[1], t[2]   -- PUC-Lua: 42 42;  before: 42 0

Medians over 500 iterations, MIX_ENV=prod, metatables snippet:

    Scope.resolve    18.0us -> 7.4us
    Compiler.compile 39.2us -> 23.4us
    parse_chunk       1.9us -> 3.6us (the numbering pass, 1.7us measured
                                      on its own)

Net ~12us off compiling that chunk; ~5us off the `tables` snippet. On
larger sources the gap widens: luassert's `util.lua` (11KB) goes from
322us to 91us in `Scope.resolve`.
`ensure_upvalue/4` checked whether the *parent* already held an upvalue
for a name, but never whether the function being given one did. Every
reference to a free variable therefore appended a fresh descriptor:
`fib` carried `[{:parent_local, 0, "_ENV"}, {:parent_local, 1, "fib"},
{:parent_local, 1, "fib"}]`, and luassert's `util.lua` has a closure
with eight identical `{:parent_local, 0, "_ENV"}` slots.

The duplicates were pure waste. Closure creation resolves
`{:parent_local, reg, _}` through the single open-upvalue cell for
`reg` and `{:parent_upvalue, i, _}` through the enclosing closure's
slot `i`, so identical descriptors always denoted the same cell — the
extra slots only cost a `build_upvalues/5` iteration and a
`Map.get(open_upvalues, reg)` each at closure creation, widened every
closure term, and forced redundant `get_upvalue` instructions.

Reuse the slot when the function already carries an identical
descriptor. Dedupe on the whole `{kind, index, name}` tuple, not the
name: equal tuples provably denote the same cell, and the search is
per-function so shadowed captures that share a name stay apart. The
insert now also finds the index and appends in one traversal instead of
`length/1` plus `++ [descriptor]`.

This moves `debug.getupvalue`/`setupvalue` numbering toward PUC-Lua,
which allocates one slot per upvalue rather than per reference:

    local a, b = 1, 2
    local function f() return a + b + a end
    -- PUC-Lua 5.x: a, b       (verified against the reference interpreter)
    -- before:      _ENV, a, b, a
    -- after:       _ENV, a, b

Medians over 500 iterations, MIX_ENV=prod, metatables snippet:
`Scope.resolve` 7.4us -> 6.7us, `Compiler.compile` 23.4us -> 21.4us.

A differential run over the Lua 5.3 suite files, the luassert
integration sources and a battery of closure-heavy programs shows no
prototype left with a duplicate descriptor and identical results
everywhere.
`LuaSandbox.run/1` spawn_links its worker and traps exits only for its own
duration. On the wall-clock timeout it sent `Process.exit(worker, :kill)`
and returned straight away, leaving the worker's exit signal in flight —
the signal comes back 10-20us later, and the `after` clause restores
`trap_exit` in a fraction of that. Whether the caller saw a harmless
`{:EXIT, _, :killed}` *message* or an untrapped `:killed` *signal* that
took it down came down to a few microseconds either side.

That is what the `playground/timeout` example test was riding: any change
to how long the worker takes to die flips it, and it fails deterministically
once it lands on the wrong side.

The result branch already handled this — unlink, then flush a signal that
may already be queued. Pull that into `unlink_worker/1` and use it on the
timeout path too. `Process.unlink/1` guarantees no exit signal from that
link is delivered after it returns, so once it has run the caller is safe
to stop trapping regardless of timing.

`test/website/lua_sandbox_test.exs` pins it: the new timeout test fails
without this change and passes with it.
@davydog187

Copy link
Copy Markdown
Contributor Author

🤖 Automated review — Opus + Sonnet consolidated findings

Two independent AI reviewers (Opus, Sonnet) reviewed this PR. Both verified the core refactor end-to-end: every var_map write (24 sites) and read (17 sites) routes through node_key/1, id threading has no duplicate/skip, the upvalue-descriptor dedup is provably sound (frozen parent snapshot → deterministic tuples), and repeat-until scope semantics are unchanged. The fixed miscompile is real and correctly pinned. Two MEDIUMs on completeness:

🟠 MEDIUM — the fixed collision is still live on the public hand-built-AST path

lib/lua/compiler/scope.ex:74-79 + lib/lua/parser.ex:116

Ids are stamped in Parser.parse_chunk/1, not in the compiler — but Lua.AST.Builder is public and Lua.Compiler.compile/2 accepts any Chunk.t(). Nothing on that path calls Ids.assign/1, so node_key/1 falls back to the raw node term. Verified: a Builder chunk with two empty for_num bodies emits close_upvalues: 4 for both (the inner needs 8) — the exact detach-a-live-cell miscompile this PR fixes — and with same-named loop vars it also collapses {:for_num_var_reg, _} (the issue #146 class). The node_key/1 docstring's "falls back to the node term itself, which is … just as unique" is factually wrong — the node term is precisely what wasn't unique.

Fix direction: call Ids.assign/1 from Compiler.compile/2 (or Scope.resolve/2) — the PR's own idempotence test shows re-running is safe — and correct the docstring.

🟠 MEDIUM (latent) — silent catch-all in Ids.number/2 (found by both reviewers)

lib/lua/ast/ids.ex:218 — the catch-all defp number(node, next), do: {node, next} leaves an unrecognized node and its entire subtree unnumbered. Dead code today (all 15 Expr + 15 Statement types + Chunk/Block are covered), but any future node type silently reintroduces exactly the miscompile class this PR fixes, with no test to catch it. Suggest deleting the catch-all (fail loudly) and/or a test asserting no meta.id is nil across a realistic corpus.

🟡 LOW

  • lib/lua/ast/meta.ex — Meta.merge/2 rebuilds through new/2 and silently drops id (and metadata). No lib callers today, but any post-parse caller demotes nodes back into the collision class.
  • test/lua/vm/upvalue_test.exs:1128-1132 — the new sibling-thresholds test does Keyword.fetch! on the loop body instruction list; it passes only because an empty body is coincidentally [{:close_upvalues, n}], and _target/_base are named backwards. Match the body list explicitly.
  • test/lua/compiler/upvalue_descriptor_test.exs — the debug.getupvalue test pins "_ENV,a,b" where PUC-Lua gives "a,b" (pre-existing divergence, flagged in the PR body) with no in-test note — the eventual fix will read as a regression. Add a comment.
  • lib/lua/compiler/scope.ex:616-619 — upvalue_index/3 re-scans the parent's descriptor list by name to recover an index ensure_upvalue/4 just returned. Correct today; avoidable O(n) on the path being optimized.
  • website/lib/website/lua_sandbox.ex — the {:EXIT, _other, reason} cancellation branch still kills the worker without unlink_worker/1, the same race this PR fixes on the timeout path (narrow: caller must cancel and catch the exit).

Test-coverage notes

No test for: the Builder-path compile (MEDIUM #1), all-ids-non-nil over a realistic chunk (MEDIUM #2), or two non-empty structurally identical sibling blocks (empty blocks are just the easiest way to make two blocks equal, not the only way).

Fixes for the above will be pushed to this branch shortly.

…odes

Ids were stamped only in `Lua.Parser.parse_chunk/1`, but `Lua.AST.Builder`
is public and `Lua.Compiler.compile/2` accepts any `Chunk.t()`. On that
path `Scope.node_key/1` fell back to the raw node term, so two
structurally identical empty blocks shared one `{:block_close_threshold,
block}` entry and the second block's watermark decided both — the same
miscompile class this branch fixes for parsed chunks.

Move the stamp into `Lua.Compiler.compile/2`. Assignment is
deterministic, so a parsed chunk re-stamps to the same ids and the
parsed path is unchanged.

`Lua.AST.Ids.number/2` also had a silent catch-all that left an
unrecognized node and its whole subtree unnumbered. Since the raw-term
fallback collides rather than merely being slow, a missing clause is now
a raise, covered by a walk over every parsed suite and integration
fixture asserting all ids are present and unique.

Also drop the by-name re-scan in `ensure_upvalue/4`: the recursive call
already returns the index it just created.

Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
`merge/2` rebuilt the result through `new/2`, which resets `id` and
`metadata`. Merging a wider span into a node therefore dropped its
node id and any attached comments. Keep both from the left operand,
which is the node being widened.

Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
The `{:EXIT, _other, reason}` branch killed the worker and exited
without the `unlink_worker/1` dance the timeout path uses, leaving the
same trap-exit race: `Process.exit/2` only sends the kill, so a caught
`exit/1` upstream could see the worker's `:killed` after `trap_exit`
was restored.

Also note in the upvalue descriptor test that the `debug.getupvalue`
assertion pins this VM's `_ENV`-for-every-closure divergence from
PUC-Lua on purpose.

Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
@davydog187

Copy link
Copy Markdown
Contributor Author

Pushed fixes for the review findings (8bce94c, f8005ed, a3cf7d9). Id stamping moved from Lua.Parser.parse_chunk/1 into Lua.Compiler.compile/2, so the collision is closed on the public hand-built-AST path too — verified by a new regression test that compiles a Lua.AST.Builder chunk with two structurally identical empty for bodies at different watermarks and fails without the stamp. Lua.AST.Ids no longer has a silent catch-all (a missing clause raises), backed by a walk over every parsed suite/integration fixture asserting all node ids are present and unique; Scope.node_key/1's docstring no longer claims the raw-term fallback is "just as unique". Also: Meta.merge/2 now carries id and metadata from the left operand, ensure_upvalue/4 uses the index it was handed instead of re-scanning by name, the :numeric_for test assertion now matches the body instruction list explicitly with correctly named bindings, the debug.getupvalue test notes it pins our _ENV divergence from PUC-Lua on purpose, and the website sandbox's cancellation branch mirrors the timeout path's unlink_worker/1. Full suite green (2636 passed), website 54 passed, --warnings-as-errors and --check-formatted clean in both projects.

Comment thread lib/lua/ast/ids.ex
Comment on lines +2 to +15
@moduledoc """
Stamps AST nodes with chunk-unique identifiers.

Compiler passes keep per-node tables — which registers a `local` claimed,
which upvalue a `Var` resolved to, the close-upvalue watermark of a block.
Keying those tables by the node term makes every read and write hash the
node's whole subtree, so the deepest nodes (function bodies, blocks) — the
ones that make the most useful keys — are also the most expensive ones.

`assign/1` walks a parsed chunk once and writes a distinct integer into
each node's `meta.id`, letting those tables key on a single word instead.
Ids are unique across the whole chunk, including the bodies of nested
functions.
"""

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe we want this as a public-facing module. We should follow the pattern for other internal modules

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this might be handled by the config in mix.exs

@davydog187
davydog187 merged commit 6444055 into main Jul 27, 2026
5 checks passed
@davydog187
davydog187 deleted the perf/compiler-front branch July 27, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant