Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 40 additions & 3 deletions .agents/plans/A3-comment-leak.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
id: A3
title: Comment tokens leak past lexer in calls.lua path
issue: 164
pr: null
pr: 182
branch: fix/lexer-comment-leak
base: main
status: ready
status: review
direction: A
unlocks:
- calls.lua
Expand Down Expand Up @@ -77,4 +77,41 @@ mix test --only lua53

## Discoveries

(populated during implementation)
- The lexer is correct: it deliberately emits `{:comment, type, text, pos}`
tokens so AST consumers can attach them via `Lua.Parser.Comments`. The
parser already skips comments at *some* boundaries (statement,
expression prefix, end-of-block), but missed comments at **list/sequence
separator boundaries**: function-argument lists and table constructors.
After parsing one expression and before peeking for the next `,` (or
the closing `)`/`}`), a trailing line-comment plus optional standalone
comments would flow straight into `expect/3` and crash the executor with
`no case clause matching: {:comment, :single, ...}`.
- Fix is local: a `skip_comments/1` helper applied at three sites in
`Lua.Parser` — `parse_expr_list_acc/2` (between expr and comma),
`parse_expr_list_until/2` (between last expr and terminator), and
`parse_table_fields/2` (between field and `,`/`;`/`}`). The helper is
scoped: in `parse_expr_list_acc`, comments are only consumed when the
next significant token is a comma, so trailing line-comments belonging
to a parent statement can still be attached by `parse_stmt`.
- `calls.lua` no longer crashes on the comment leak. It now fails earlier
on line 6 (`local debug = require "debug"`) because `require` is
sandboxed — a separate, pre-existing concern out of scope here.
- `string.pack` is also unimplemented; the original repro pattern from
`calls.lua:374` would hit that next. Both are tracked elsewhere.

## What changed

- `lib/lua/parser.ex`: added a `skip_comments/1` helper near the
`peek`/`consume` token utilities. Applied it at three boundaries:
`parse_expr_list_acc/2` (gated on comma lookahead so trailing
comments still reach `parse_stmt`), `parse_expr_list_until/2`
(before terminator peek), and `parse_table_fields/2` (before and
after each field).
- `test/lua/parser/comment_test.exs`: new `describe` block "comments
inside expression lists (regression for A3)" with 7 regression
tests covering function-arg lists and table constructors, plus a
test that walks the full parsed AST asserting no `{:comment, ...}`
tuple survives into the structure passed to codegen.
- Suite: 1301 → 1309 tests passing, 0 failures, 0 regressions.
- PR: #182. No follow-up issues opened — `require` sandboxing and
`string.pack` were pre-existing.
9 changes: 6 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,12 @@ under the [`0.5.0` milestone](https://github.com/tv-labs/lua/milestone/1).
- ~~**A0**: 64-bit integer overflow wrapping for arithmetic and bitwise ops
(Lua 5.3 §3.4.1; deliberate divergence from Luerl bignum semantics).~~
Shipped in PR #177.
- **A1**: Empty/missing-key table reads return `nil` (unblocks ~6 files).
- **A2**: Long-string `[[ … ]]` lexer handles embedded `]` and level brackets `[==[`.
- **A3**: Comment tokens leak past lexer in `calls.lua`.
- ~~**A1**: Empty/missing-key table reads return `nil` (unblocks ~6 files).~~
In review in PR #179.
- ~~**A2**: Long-string `[[ … ]]` lexer handles embedded `]` and level brackets `[==[`.~~
In review in PR #180.
- ~~**A3**: Comment tokens leak past lexer in `calls.lua`.~~
In review in PR #182.
- **A4**: Pre-load Lua stdlibs into `package.loaded` so `require"io"` resolves.

### Per-file assertions (one PR each, ≤ ½ day)
Expand Down
37 changes: 35 additions & 2 deletions lib/lua/parser.ex
Original file line number Diff line number Diff line change
Expand Up @@ -948,13 +948,22 @@ defmodule Lua.Parser do
end

defp parse_table_fields(tokens, acc) do
# Skip leading comments (and any orphaned trailing comments from a
# previous field) before deciding whether we've hit the terminator
# or another field.
tokens = skip_comments(tokens)

case peek(tokens) do
{:delimiter, :rbrace, _} ->
{:ok, acc, tokens}

_ ->
case parse_table_field(tokens) do
{:ok, field, rest} ->
# Trailing comments can sit between a field and its separator
# (or the closing brace). Skip them before the punctuation peek.
rest = skip_comments(rest)

case peek(rest) do
{:delimiter, :comma, _} ->
{_, rest2} = consume(rest)
Expand Down Expand Up @@ -1101,9 +1110,15 @@ defmodule Lua.Parser do
defp parse_expr_list_acc(tokens, acc) do
case parse_expr(tokens) do
{:ok, expr, rest} ->
case peek(rest) do
# Comments can appear between an argument and the comma (e.g.
# `f(1 -- one\n, 2)` or a trailing line-comment in calls.lua).
# Only swallow them here if doing so exposes a comma — otherwise
# leave the stream untouched so the statement-level comment
# collection in `parse_stmt` can still attach a trailing comment
# to the right node.
case peek(skip_comments(rest)) do
{:delimiter, :comma, _} ->
{_, rest2} = consume(rest)
{_, rest2} = consume(skip_comments(rest))
parse_expr_list_acc(rest2, [expr | acc])

_ ->
Expand All @@ -1120,6 +1135,8 @@ defmodule Lua.Parser do
end

defp parse_expr_list_until(tokens, terminator) do
tokens = skip_comments(tokens)

case peek(tokens) do
{:delimiter, ^terminator, _} ->
{_, rest} = consume(tokens)
Expand All @@ -1128,6 +1145,10 @@ defmodule Lua.Parser do
_ ->
case parse_expr_list(tokens) do
{:ok, exprs, rest} ->
# Trailing comments can sit between the last expression and
# the terminator (e.g. `f(1, 2 -- note\n)`).
rest = skip_comments(rest)

case expect(rest, :delimiter, terminator) do
{:ok, _, rest2} ->
{:ok, exprs, rest2}
Expand All @@ -1150,6 +1171,18 @@ defmodule Lua.Parser do
defp consume([token | rest]), do: {token, rest}
defp consume([]), do: {nil, []}

# Drop leading comment tokens from a token stream.
#
# The lexer emits `{:comment, type, text, pos}` tuples (deliberately, so
# tooling can attach them to the AST via `Lua.Parser.Comments`). They are
# collected at statement and expression-prefix boundaries, but inside list
# constructs (function arguments, table fields, parenthesized lists) a
# trailing or interleaved comment must be skipped before peeking for a
# separator (`,`, `;`) or terminator (`)`, `}`, `]`). Without this skip,
# a comment leaks into `expect/3` and crashes the executor.
defp skip_comments([{:comment, _, _, _} | rest]), do: skip_comments(rest)
defp skip_comments(tokens), do: tokens

# Expect a specific token type
defp expect(tokens, expected_type) do
case peek(tokens) do
Expand Down
164 changes: 164 additions & 0 deletions test/lua/parser/comment_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -378,4 +378,168 @@ defmodule Lua.Parser.CommentTest do
end
end
end

describe "comments inside expression lists (regression for A3)" do
# Shrunken repro from test/lua53_tests/calls.lua:374, which crashed
# the executor with `no case clause matching: {:comment, :single, ...}`.
# The lexer emits comment tokens (deliberately, for AST consumers), but
# the parser was failing to skip them between expressions in a function
# argument list — so a trailing comment before `)` leaked into expect/3.

test "trailing comment after final argument before closing paren" do
code = ~s[local x = math.max(1, 2, 3 -- LUAC_INT\n)\nreturn x]
assert {:ok, _chunk} = Parser.parse(code)
assert {[3], _} = Lua.eval!(code)
end

test "trailing comment after every argument across multiple lines" do
code = """
local function f(a, b, c) return a + b + c end
local x = f(
1, -- first
2, -- second
3 -- third
)
return x
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[6], _} = Lua.eval!(code)
end

test "multiple consecutive comments before closing paren" do
# Mirrors the exact shape from calls.lua: a trailing line comment
# plus an additional standalone comment before the close paren.
code = """
local function f(a) return a end
local x = f(
42 -- LUAC_INT
-- another note (padding...)
)
return x
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[42], _} = Lua.eval!(code)
end

test "comment between argument and comma" do
code = """
local function f(a, b) return a + b end
return f(1 -- one
, 2)
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[3], _} = Lua.eval!(code)
end

test "long comment between arguments" do
code = """
local function f(a, b) return a + b end
return f(1, --[[ block ]] 2)
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[3], _} = Lua.eval!(code)
end

test "comments inside table constructor" do
code = """
local t = {
1, -- one
2, -- two
3 -- three
-- trailing standalone
}
return #t
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[3], _} = Lua.eval!(code)
end

test "no :comment tuples leak into the AST passed to the compiler" do
# The lexer deliberately emits {:comment, ...} tokens for AST
# consumers, but they are stored as Meta annotations — they must
# never appear as raw tokens inside the parsed AST that flows to
# codegen and the executor. Walk every value in the parsed chunk
# and assert no comment tuple is reachable except through Meta's
# comments field.
code = """
local function f(a, b, c) return a + b + c end
local x = f(
1, -- one
2, -- two
3 -- three
-- trailing
)
local t = {
10, -- ten
20, -- twenty
-- standalone
}
return x, #t
"""

assert {:ok, chunk} = Parser.parse(code)
refute contains_comment_tuple?(chunk)
end

test "exact calls.lua:374 shape parses and evaluates" do
# Direct repro of the failing pattern from calls.lua (without
# the actual string.pack invocation, which is unrelated).
code = """
local function pack(...) return select("#", ...) end
local n = pack(
"\\27Lua", -- signature
5*16 + 3, -- version 5.3
0, -- format
"data", -- data
4, -- sizeof(int)
8, -- sizeof(size_t)
4, -- size of instruction
8, -- sizeof(lua integer)
8, -- sizeof(lua number)
0x5678 -- LUAC_INT
-- LUAC_NUM may not have a unique binary representation (padding...)
)
return n
"""

assert {:ok, _chunk} = Parser.parse(code)
assert {[10], _} = Lua.eval!(code)
end
end

# Recursively check whether a parsed AST contains any raw
# `{:comment, ...}` lexer tuple as a sub-term, ignoring the Meta
# `comments` field where they're legitimately stored as maps.
defp contains_comment_tuple?(%Meta{} = _meta) do
# Comments stored on Meta are %{type: ..., text: ..., position: ...} maps,
# not raw tuples, so traversing into Meta is unnecessary.
false
end

defp contains_comment_tuple?({:comment, _, _, _}), do: true

defp contains_comment_tuple?(%_{} = struct) do
struct
|> Map.from_struct()
|> contains_comment_tuple?()
end

defp contains_comment_tuple?(map) when is_map(map) do
Enum.any?(map, fn {_k, v} -> contains_comment_tuple?(v) end)
end

defp contains_comment_tuple?(list) when is_list(list) do
Enum.any?(list, &contains_comment_tuple?/1)
end

defp contains_comment_tuple?(tuple) when is_tuple(tuple) do
tuple |> Tuple.to_list() |> contains_comment_tuple?()
end

defp contains_comment_tuple?(_), do: false
end
Loading