Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 40 additions & 3 deletions .agents/plans/A1-table-nil-on-missing-key.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
id: A1
title: Empty/missing-key table reads return nil instead of crashing
issue: 162
pr: null
pr: 179
branch: fix/table-nil-on-missing-key
base: main
status: ready
status: review
direction: A
unlocks:
- errors.lua
Expand Down Expand Up @@ -96,4 +96,41 @@ Compare suite count before/after; capture in PR body.

## Discoveries

(populated during implementation)
The four explicit success-criteria cases (`local t = {}; return t[5]`,
out-of-bounds array read, missing string field, `t[5] == nil`) **already
returned `nil` correctly** when this plan was picked up. Same for
metatable `__index` resolution (function and table forms), `rawget`,
`pairs`/`ipairs` over empty tables, and `next({})`. The underlying fix
appears to have landed earlier — most likely as part of the CPS executor
refactor in PR #156.

The error message the plan describes (`Lua runtime error: key N not found
in: %{}`) does still appear in some Lua 5.3 suite files (`sort.lua`,
`strings.lua`, `verybig.lua`), but the source is **not** a missing
`table.data` key. It is `Map.fetch!(state.open_upvalues, reg)` in
`lib/lua/vm/executor.ex` at lines 301 and 310 — an open-upvalue tracking
bug that surfaces with the same exception class. Tracked separately in
plan **A15** (`fix/open-upvalue-missing-cell`).

`nextvar.lua` fails with a different shape (`attempt to concatenate a nil
value`); reproducer points at the same area as the for-loop register
regression covered by plan **A14**.

This PR therefore does not change runtime behaviour. It locks in the
existing correct behaviour with a regression test file
(`test/lua/vm/table_index_test.exs`, 10 cases) so the bug cannot
silently come back, and opens A15 to handle the unlocked-files
slice that is in fact a different bug.

## What changed

- New file: `test/lua/vm/table_index_test.exs` — 10 regression tests
covering missing-key reads (empty table, out-of-bounds array, missing
string field, `nil` comparison), metatable `__index` fall-through
(function form, table form, and direct-hit short-circuit), and stdlib
helpers (`rawget`, `next`, `pairs`).
- New plan: `.agents/plans/A15-open-upvalue-missing-cell.md` — covers the
`set_open_upvalue` / `get_open_upvalue` crash that this plan originally
attributed to table-data reads.
- No production code changes. Suite count unchanged. `mix test` 1284 → 1294
(10 new tests, 0 failures).
130 changes: 130 additions & 0 deletions .agents/plans/A15-open-upvalue-missing-cell.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
id: A15
title: set_open_upvalue / get_open_upvalue crash when cell is missing
issue: null
pr: null
branch: fix/open-upvalue-missing-cell
base: main
status: ready
direction: A
unlocks:
- sort.lua
- strings.lua
- verybig.lua
---

## Goal

Fix the `Map.fetch!` crash inside `set_open_upvalue` and `get_open_upvalue`
in `lib/lua/vm/executor.ex` (lines 301 and 310 at the time of writing).
When these handlers run with a register that has no entry in
`state.open_upvalues`, the VM raises:

```
Lua runtime error: key N not found in: %{}
```

instead of behaving correctly. This was discovered while shipping plan A1:
A1's stated table-read bug already passes, but the same error message also
appears here from a completely unrelated path.

## Reproduction

`test/lua53_tests/sort.lua` fails partway through with:

```
testing (parts of) table library
testing unpack
Lua runtime error: key 3 not found in:

%{}
```

Stacktrace ends at `lib/lua/vm/executor.ex:310` inside `set_open_upvalue`.
`strings.lua` and `verybig.lua` produce the same error shape with
different keys.

## Out of scope

- The for-loop register regression in A14 (separate bug, separate symptom).
- Any other suite failure that surfaces *after* this fix lands.
- Refactors of the upvalue / closure pipeline beyond what is needed.

## Success criteria

- [ ] `mix test` passes (no regressions vs. baseline).
- [ ] `mix test test/lua/vm/table_index_test.exs` still green (sanity).
- [ ] New unit test in `test/lua/vm/upvalue_test.exs` (or appropriate
existing file) reproduces the original sort.lua-style failure as a
minimal Lua snippet and asserts it does not crash.
- [ ] At least one of `sort.lua`, `strings.lua`, `verybig.lua` makes more
progress than before (passes outright if no further bugs surface;
otherwise fails later, at a different site, and that site is logged
in `## Discoveries`).

## Implementation notes

The crash is `Map.fetch!(state.open_upvalues, reg)` against an empty (or
not-yet-populated) `open_upvalues`.

Read the lifecycle:

```bash
grep -nE "open_upvalues|open_upvalue|set_open_upvalue|get_open_upvalue" \
lib/lua/vm/executor.ex
```

`open_upvalues` is reset to `%{}` at:

- VM entry (line ~30)
- Tailcall / call boundaries (lines ~59, 551, 1252, 1470, 1515, 1564)
- Loop-end cleanups (`Map.reject(...)` at lines ~197, 230, 416, 453)

Cells are only *created* by the `closure` handler around line 470, which
adds `{reg => cell_ref}` when a nested prototype captures a parent local.

Hypothesis: a code path emits `set_open_upvalue` / `get_open_upvalue` for
a register that was never claimed by a `closure`. Either:

1. The compiler emits these instructions when it shouldn't, or
2. A `Map.reject` cleanup is too eager and removes a cell that is still
live, or
3. `open_upvalues` is reset (`%{}`) without first promoting live cells
into closed-upvalue cells, leaving dangling instructions.

Reduce sort.lua to the smallest snippet that reproduces, then bisect from
there. A starting reduction:

```bash
elixir -e 'try do Lua.eval!(File.read!("test/lua53_tests/sort.lua")) rescue e -> IO.puts(Exception.format_stacktrace(__STACKTRACE__)) end'
```

Tag the snippet, copy into `test/lua/vm/upvalue_test.exs`, and iterate.

Likely fix shape: either make `set_open_upvalue` / `get_open_upvalue` fall
back to creating the cell on demand (mirroring `closure`'s logic at line
470), or fix the offending compiler emission / cleanup site.

## Verification

```bash
mix format
mix compile --warnings-as-errors
mix test
mix test test/lua/vm/upvalue_test.exs
mix test --only lua53
```

Capture suite delta in PR body.

## Risks

- Falling back to "create cell on demand" hides real compiler bugs. Prefer
to find the upstream cause first.
- Eager cleanup via `Map.reject` may be the root cause. Tightening it
could leak upvalue cells; verify with the metatable / closure regression
tests in `test/lua/vm/metatable_test.exs` and any closure tests.

## Discoveries

(populated during implementation)
164 changes: 164 additions & 0 deletions test/lua/vm/table_index_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
defmodule Lua.VM.TableIndexTest do
use ExUnit.Case, async: true

alias Lua.Compiler
alias Lua.Parser
alias Lua.VM
alias Lua.VM.State
alias Lua.VM.Stdlib

# Reading a missing key from a table must return nil, never raise. This
# covers the language-level contract from Lua 5.3 §3.4.7: an indexing
# access `t[k]` for a key that is not present in `t` evaluates to nil
# (after consulting `__index` if a metatable is set).
#
# The cases below pin that contract against silent regression on the
# most common shapes: empty tables, out-of-bounds array reads, missing
# string fields, comparisons with nil, metatable `__index` fall-through
# in both function and table form, the "direct hit must not consult
# __index" short-circuit, and the stdlib helpers (`rawget`, `next`,
# `pairs`) that all share the same lookup path.

describe "missing key reads return nil" do
test "empty table returns nil for any integer key" do
code = """
local t = {}
return t[5]
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [nil], _state} = VM.execute(proto, state)
end

test "array out-of-bounds read returns nil" do
code = """
local t = {1, 2, 3}
return t[10]
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [nil], _state} = VM.execute(proto, state)
end

test "missing string key on a non-empty table returns nil" do
code = """
local t = {a = 1}
return t.b
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [nil], _state} = VM.execute(proto, state)
end

test "missing key compares equal to nil" do
code = """
local t = {}
return t[5] == nil
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [true], _state} = VM.execute(proto, state)
end
end

describe "metatable __index still resolves (regression guard)" do
test "function __index falls through on missing key" do
code = """
local t = setmetatable({}, {__index = function(_, k) return "got_" .. k end})
return t.foo
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, ["got_foo"], _state} = VM.execute(proto, state)
end

test "table __index falls through on missing key" do
code = """
local t = setmetatable({}, {__index = {x = 42}})
return t.x
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [42], _state} = VM.execute(proto, state)
end

test "direct hits do not trigger __index" do
# __index that would explode if invoked. A direct hit on `x` must
# not consult it.
code = """
local mt = {__index = function() error("__index should not be called") end}
local t = setmetatable({x = 1}, mt)
return t.x
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [1], _state} = VM.execute(proto, state)
end
end

describe "stdlib helpers tolerate missing keys" do
test "rawget returns nil for missing key" do
code = """
local t = {}
return rawget(t, "x")
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [nil], _state} = VM.execute(proto, state)
end

test "next returns nil for empty table" do
code = """
local t = {}
return next(t)
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [nil, nil], _state} = VM.execute(proto, state)
end

test "pairs over empty table makes zero iterations" do
code = """
local t = {}
local n = 0
for _, _ in pairs(t) do
n = n + 1
end
return n
"""

assert {:ok, ast} = Parser.parse(code)
assert {:ok, proto} = Compiler.compile(ast, source: "test.lua")
state = Stdlib.install(State.new())

assert {:ok, [0], _state} = VM.execute(proto, state)
end
end
end
Loading