Repository navigation
compiler: key scope resolution by node id and dedupe upvalue descriptors - #400
Conversation
`Scope.State.var_map` keyed its per-node entries by the AST node itself,
so every write and every codegen read hashed the node's whole subtree.
The worst keys were the most useful ones: an `%Expr.Function{}` key on
the playground metatables snippet took ~2us to `:erlang.phash2` and
~0.7us per `Map.get`, and the 34 keys of that 370-byte chunk held 2,912
words of AST between them.
Give every node a chunk-unique `meta.id`, assigned by a single pass over
the parsed chunk (`Lua.AST.Ids`, called from `Lua.Parser.parse_chunk/1`),
and key `var_map` by that integer. Scope and codegen both go through
`Scope.node_key/1`, which falls back to the node term when a node has no
id, so hand-built ASTs keep working unchanged.
This also fixes a latent miscompile. Every empty block parses to the
same term (`%Lua.AST.Block{stmts: [], meta: nil}`), so all the empty
blocks in a chunk shared one `{:block_close_threshold, block}` entry and
the last one resolved decided the close-upvalue watermark for all of
them. A block that borrowed a lower watermark closed cells belonging to
live enclosing locals:
local t = {}
do
local a, b, c, d = 1, 2, 3, 4
local n = 0
local get = function() return n end
local set = function(v) n = v end
for i = 1, 1 do end
set(42)
t[1] = get()
t[2] = n
end
for i = 1, 1 do end
return t[1], t[2] -- PUC-Lua: 42 42; before: 42 0
Medians over 500 iterations, MIX_ENV=prod, metatables snippet:
Scope.resolve 18.0us -> 7.4us
Compiler.compile 39.2us -> 23.4us
parse_chunk 1.9us -> 3.6us (the numbering pass, 1.7us measured
on its own)
Net ~12us off compiling that chunk; ~5us off the `tables` snippet. On
larger sources the gap widens: luassert's `util.lua` (11KB) goes from
322us to 91us in `Scope.resolve`.
`ensure_upvalue/4` checked whether the *parent* already held an upvalue
for a name, but never whether the function being given one did. Every
reference to a free variable therefore appended a fresh descriptor:
`fib` carried `[{:parent_local, 0, "_ENV"}, {:parent_local, 1, "fib"},
{:parent_local, 1, "fib"}]`, and luassert's `util.lua` has a closure
with eight identical `{:parent_local, 0, "_ENV"}` slots.
The duplicates were pure waste. Closure creation resolves
`{:parent_local, reg, _}` through the single open-upvalue cell for
`reg` and `{:parent_upvalue, i, _}` through the enclosing closure's
slot `i`, so identical descriptors always denoted the same cell — the
extra slots only cost a `build_upvalues/5` iteration and a
`Map.get(open_upvalues, reg)` each at closure creation, widened every
closure term, and forced redundant `get_upvalue` instructions.
Reuse the slot when the function already carries an identical
descriptor. Dedupe on the whole `{kind, index, name}` tuple, not the
name: equal tuples provably denote the same cell, and the search is
per-function so shadowed captures that share a name stay apart. The
insert now also finds the index and appends in one traversal instead of
`length/1` plus `++ [descriptor]`.
This moves `debug.getupvalue`/`setupvalue` numbering toward PUC-Lua,
which allocates one slot per upvalue rather than per reference:
local a, b = 1, 2
local function f() return a + b + a end
-- PUC-Lua 5.x: a, b (verified against the reference interpreter)
-- before: _ENV, a, b, a
-- after: _ENV, a, b
Medians over 500 iterations, MIX_ENV=prod, metatables snippet:
`Scope.resolve` 7.4us -> 6.7us, `Compiler.compile` 23.4us -> 21.4us.
A differential run over the Lua 5.3 suite files, the luassert
integration sources and a battery of closure-heavy programs shows no
prototype left with a duplicate descriptor and identical results
everywhere.
`LuaSandbox.run/1` spawn_links its worker and traps exits only for its own
duration. On the wall-clock timeout it sent `Process.exit(worker, :kill)`
and returned straight away, leaving the worker's exit signal in flight —
the signal comes back 10-20us later, and the `after` clause restores
`trap_exit` in a fraction of that. Whether the caller saw a harmless
`{:EXIT, _, :killed}` *message* or an untrapped `:killed` *signal* that
took it down came down to a few microseconds either side.
That is what the `playground/timeout` example test was riding: any change
to how long the worker takes to die flips it, and it fails deterministically
once it lands on the wrong side.
The result branch already handled this — unlink, then flush a signal that
may already be queued. Pull that into `unlink_worker/1` and use it on the
timeout path too. `Process.unlink/1` guarantees no exit signal from that
link is delivered after it returns, so once it has run the caller is safe
to stop trapping regardless of timing.
`test/website/lua_sandbox_test.exs` pins it: the new timeout test fails
without this change and passes with it.
🤖 Automated review — Opus + Sonnet consolidated findingsTwo independent AI reviewers (Opus, Sonnet) reviewed this PR. Both verified the core refactor end-to-end: every 🟠 MEDIUM — the fixed collision is still live on the public hand-built-AST path
Ids are stamped in Fix direction: call 🟠 MEDIUM (latent) — silent catch-all in
|
…odes
Ids were stamped only in `Lua.Parser.parse_chunk/1`, but `Lua.AST.Builder`
is public and `Lua.Compiler.compile/2` accepts any `Chunk.t()`. On that
path `Scope.node_key/1` fell back to the raw node term, so two
structurally identical empty blocks shared one `{:block_close_threshold,
block}` entry and the second block's watermark decided both — the same
miscompile class this branch fixes for parsed chunks.
Move the stamp into `Lua.Compiler.compile/2`. Assignment is
deterministic, so a parsed chunk re-stamps to the same ids and the
parsed path is unchanged.
`Lua.AST.Ids.number/2` also had a silent catch-all that left an
unrecognized node and its whole subtree unnumbered. Since the raw-term
fallback collides rather than merely being slow, a missing clause is now
a raise, covered by a walk over every parsed suite and integration
fixture asserting all ids are present and unique.
Also drop the by-name re-scan in `ensure_upvalue/4`: the recursive call
already returns the index it just created.
Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
`merge/2` rebuilt the result through `new/2`, which resets `id` and `metadata`. Merging a wider span into a node therefore dropped its node id and any attached comments. Keep both from the left operand, which is the node being widened. Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
The `{:EXIT, _other, reason}` branch killed the worker and exited
without the `unlink_worker/1` dance the timeout path uses, leaving the
same trap-exit race: `Process.exit/2` only sends the kill, so a caught
`exit/1` upstream could see the worker's `:killed` after `trap_exit`
was restored.
Also note in the upvalue descriptor test that the `debug.getupvalue`
assertion pins this VM's `_ENV`-for-every-closure divergence from
PUC-Lua on purpose.
Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
|
Pushed fixes for the review findings (8bce94c, f8005ed, a3cf7d9). Id stamping moved from |
| @moduledoc """ | ||
| Stamps AST nodes with chunk-unique identifiers. | ||
|
|
||
| Compiler passes keep per-node tables — which registers a `local` claimed, | ||
| which upvalue a `Var` resolved to, the close-upvalue watermark of a block. | ||
| Keying those tables by the node term makes every read and write hash the | ||
| node's whole subtree, so the deepest nodes (function bodies, blocks) — the | ||
| ones that make the most useful keys — are also the most expensive ones. | ||
|
|
||
| `assign/1` walks a parsed chunk once and writes a distinct integer into | ||
| each node's `meta.id`, letting those tables key on a single word instead. | ||
| Ids are unique across the whole chunk, including the bodies of nested | ||
| functions. | ||
| """ |
There was a problem hiding this comment.
I don't believe we want this as a public-facing module. We should follow the pattern for other internal modules
There was a problem hiding this comment.
Actually this might be handled by the config in mix.exs
Two measurement-backed fixes to the compiler front end, plus the harness fix CI needed to go green. The compiler changes are behaviour-preserving except where they move us toward PUC-Lua, and one of them fixes a latent miscompile found along the way.
1. Key scope resolution by integer node ids
Scope.State.var_mapkeyed its per-node entries by the AST node itself, so every write and every codegen read hashed the node's whole subtree. The worst keys were the most useful ones: on the playground metatables snippet an%Expr.Function{}key took ~2us to:erlang.phash2and ~0.7us perMap.get, and the 34 keys of that 370-byte chunk held 2,912 words of AST between them.Every node now carries a chunk-unique
meta.id, stamped by one pass over the parsed chunk (Lua.AST.Ids, called fromLua.Parser.parse_chunk/1), andvar_mapkeys on that integer. Scope and codegen both go throughScope.node_key/1, which falls back to the node term when a node has no id — hand-built ASTs (tests,Lua.AST.Builder) keep working unchanged.It also fixes a latent miscompile
Every empty block parses to the same term —
%Lua.AST.Block{stmts: [], meta: nil}— so all the empty blocks in a chunk shared a single{:block_close_threshold, block}entry and the last one resolved decided the close-upvalue watermark for all of them.db.luaalone has six. A block that borrowed a lower watermark closed cells belonging to live enclosing locals:PUC-Lua prints
42 42;mainreturns42 0(both empty loop bodies compiled toclose_upvalues 5, so the first one detachedn's cell). Pinned intest/lua/vm/upvalue_test.exs, both new tests fail onmain.2. Dedupe upvalue descriptors per function
ensure_upvalue/4checked whether the parent already held an upvalue for a name, but never whether the function being given one did. Every reference to a free variable appended a fresh descriptor:fibcarried[{:parent_local, 0, "_ENV"}, {:parent_local, 1, "fib"}, {:parent_local, 1, "fib"}], and luassert'sutil.luahas a closure with eight identical{:parent_local, 0, "_ENV"}slots.The duplicates were pure waste. Closure creation resolves
{:parent_local, reg, _}through the single open-upvalue cell forregand{:parent_upvalue, i, _}through the enclosing closure's sloti, so identical descriptors always denoted the same cell. The extra slots only cost abuild_upvalues/5iteration and aMap.get(open_upvalues, reg)each at closure creation, widened every closure term, and forced redundantget_upvalueinstructions.A function now reuses the slot when it already carries an identical descriptor. The dedupe is on the whole
{kind, index, name}tuple, not the name, and is per-function: shadowed captures that share a name stay apart. The insert also finds the index and appends in one traversal instead oflength/1plus++ [descriptor].Behaviour change:
debug.getupvalue/setupvaluenumberingOne slot per upvalue rather than one per reference — this moves us toward PUC. (The leading
_ENVon every nested function is pre-existing and unchanged.) No test pinned the duplicated numbering; the existingdebugtests all use functions that reference each free variable once.The
set_upvalue i, r+get_upvalue d, i->move d, rpeephole is deliberately not here — it belongs to the codegen-peephole PR rather than toscope.ex.3. Website sandbox: cut the worker link before untrapping exits
CI's website job caught a knife-edge race this PR tips over, in
Website.LuaSandbox.run/1. Itspawn_links the worker and traps exits only for its own duration. On the wall-clock timeout it sentProcess.exit(worker, :kill)and returned immediately, leaving the worker's exit signal in flight — measured at 10–20us round trip, while theafterclause restorestrap_exitin a fraction of that. Whether the caller saw a harmless{:EXIT, _, :killed}message or an untrapped:killedsignal that took it down came down to a handful of microseconds.Measured on this machine, worker-death latency after
Process.exit/2: 10–12us onmain, 16–22us on this branch — enough to flipplayground/timeout (limit)from pass to a deterministic** (EXIT from #PID<...>) killed. Nothing about the snippet's compiled output changed: the emitted instruction stream and the worker's heap profile are byte-for-byte identical on both builds.The result branch already handled this — unlink, then flush a signal that may already be queued. That is now
unlink_worker/1, used on the timeout path too.Process.unlink/1guarantees no exit signal from that link is delivered after it returns, so the caller is safe to stop trapping regardless of timing.website/test/website/lua_sandbox_test.exspins it; the new timeout test fails without the fix.The website suite now passes 5/5 consecutive runs (54 tests); before the fix it failed 5/5.
Measurements
MIX_ENV=prod, medians of 500-iteration batches over 41 reps (20/21 for the larger files). Two paired runs againstmain; ranges show the spread, the machine was not quiesced.Small snippets, us:
Scope.resolveCompiler.compileparse_chunkLarger sources, us:
Scope.resolveCompiler.compileparse_chunkutil.lua(11.4 KB)spy.lua(7.1 KB)strings.lua(13.3 KB)closure.lua(5.5 KB)constructs.lua(7.5 KB)The numbering pass is the whole
parse_chunkregression; measured on its own it is 1.7us for the metatables chunk. It buys back 5–10x that inScope.resolveand codegen.Runtime (
Lua.eval!) is unchanged within noise, as expected — this is all compile-time work.Validation
mix format --check-formatted,mix compile --force --warnings-as-errors,mix docs --warnings-as-errors,mix dialyzer— all clean.mix test --include lua53 --include slow --include differential: 2654 passed, 0 failures, 16 skipped.closure.luaandnextvar.luafrom the official suite exercise upvalues heavily.upvalue_descriptors,upvalue_names) for all 29 Lua 5.3 suite files, the luassert/say integration sources and the snippet battery, plus evaluated results for 14 closure-heavy programs, diffedmainvs this branch. Every difference is aget_upvalue/set_upvalueindex renumbering, anupvalue_descriptors/upvalue_nameslist losing duplicates, or one of the threeclose_upvaluesthresholds indb.luathat the empty-block fix corrects. No other opcode moved, and the only evaluated result that changed is thedebug.getupvaluelisting above.test/lua/compiler/upvalue_descriptor_test.exs.