From 7b5d550b88f99d4ec642025cdee194c9638cf9f4 Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Tue, 5 May 2026 12:22:53 -0700 Subject: [PATCH 1/4] chore(A3): start plan --- .agents/plans/A3-comment-leak.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/plans/A3-comment-leak.md b/.agents/plans/A3-comment-leak.md index fbad2fe9..50718368 100644 --- a/.agents/plans/A3-comment-leak.md +++ b/.agents/plans/A3-comment-leak.md @@ -5,7 +5,7 @@ issue: 164 pr: null branch: fix/lexer-comment-leak base: main -status: ready +status: in-progress direction: A unlocks: - calls.lua From 2497bd6aeaa6894fe18e9709382cbfb458c35fda Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Tue, 5 May 2026 12:29:59 -0700 Subject: [PATCH 2/4] fix(parser): skip comment tokens between expression-list elements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lexer emits {:comment, ...} tokens for AST consumers, but the parser only skipped them at statement and expression-prefix boundaries. A trailing or interleaved comment in a function-argument list or table constructor leaked through and crashed expect/3 with 'no case clause matching: {:comment, :single, ...}' — first reported on calls.lua:374. Adds a skip_comments/1 helper applied at parse_expr_list_acc/2, parse_expr_list_until/2, and parse_table_fields/2. In parse_expr_list_acc the helper is scoped so trailing line-comments still reach parse_stmt for attachment to the parent statement. Plan: A3 Closes #164 --- .agents/plans/A3-comment-leak.md | 22 ++++- lib/lua/parser.ex | 37 ++++++- test/lua/parser/comment_test.exs | 164 +++++++++++++++++++++++++++++++ 3 files changed, 220 insertions(+), 3 deletions(-) diff --git a/.agents/plans/A3-comment-leak.md b/.agents/plans/A3-comment-leak.md index 50718368..937aa7c6 100644 --- a/.agents/plans/A3-comment-leak.md +++ b/.agents/plans/A3-comment-leak.md @@ -77,4 +77,24 @@ mix test --only lua53 ## Discoveries -(populated during implementation) +- The lexer is correct: it deliberately emits `{:comment, type, text, pos}` + tokens so AST consumers can attach them via `Lua.Parser.Comments`. The + parser already skips comments at *some* boundaries (statement, + expression prefix, end-of-block), but missed comments at **list/sequence + separator boundaries**: function-argument lists and table constructors. + After parsing one expression and before peeking for the next `,` (or + the closing `)`/`}`), a trailing line-comment plus optional standalone + comments would flow straight into `expect/3` and crash the executor with + `no case clause matching: {:comment, :single, ...}`. +- Fix is local: a `skip_comments/1` helper applied at three sites in + `Lua.Parser` — `parse_expr_list_acc/2` (between expr and comma), + `parse_expr_list_until/2` (between last expr and terminator), and + `parse_table_fields/2` (between field and `,`/`;`/`}`). The helper is + scoped: in `parse_expr_list_acc`, comments are only consumed when the + next significant token is a comma, so trailing line-comments belonging + to a parent statement can still be attached by `parse_stmt`. +- `calls.lua` no longer crashes on the comment leak. It now fails earlier + on line 6 (`local debug = require "debug"`) because `require` is + sandboxed — a separate, pre-existing concern out of scope here. +- `string.pack` is also unimplemented; the original repro pattern from + `calls.lua:374` would hit that next. Both are tracked elsewhere. diff --git a/lib/lua/parser.ex b/lib/lua/parser.ex index a37ac3a8..334dbb91 100644 --- a/lib/lua/parser.ex +++ b/lib/lua/parser.ex @@ -948,6 +948,11 @@ defmodule Lua.Parser do end defp parse_table_fields(tokens, acc) do + # Skip leading comments (and any orphaned trailing comments from a + # previous field) before deciding whether we've hit the terminator + # or another field. + tokens = skip_comments(tokens) + case peek(tokens) do {:delimiter, :rbrace, _} -> {:ok, acc, tokens} @@ -955,6 +960,10 @@ defmodule Lua.Parser do _ -> case parse_table_field(tokens) do {:ok, field, rest} -> + # Trailing comments can sit between a field and its separator + # (or the closing brace). Skip them before the punctuation peek. + rest = skip_comments(rest) + case peek(rest) do {:delimiter, :comma, _} -> {_, rest2} = consume(rest) @@ -1101,9 +1110,15 @@ defmodule Lua.Parser do defp parse_expr_list_acc(tokens, acc) do case parse_expr(tokens) do {:ok, expr, rest} -> - case peek(rest) do + # Comments can appear between an argument and the comma (e.g. + # `f(1 -- one\n, 2)` or a trailing line-comment in calls.lua). + # Only swallow them here if doing so exposes a comma — otherwise + # leave the stream untouched so the statement-level comment + # collection in `parse_stmt` can still attach a trailing comment + # to the right node. + case peek(skip_comments(rest)) do {:delimiter, :comma, _} -> - {_, rest2} = consume(rest) + {_, rest2} = consume(skip_comments(rest)) parse_expr_list_acc(rest2, [expr | acc]) _ -> @@ -1120,6 +1135,8 @@ defmodule Lua.Parser do end defp parse_expr_list_until(tokens, terminator) do + tokens = skip_comments(tokens) + case peek(tokens) do {:delimiter, ^terminator, _} -> {_, rest} = consume(tokens) @@ -1128,6 +1145,10 @@ defmodule Lua.Parser do _ -> case parse_expr_list(tokens) do {:ok, exprs, rest} -> + # Trailing comments can sit between the last expression and + # the terminator (e.g. `f(1, 2 -- note\n)`). + rest = skip_comments(rest) + case expect(rest, :delimiter, terminator) do {:ok, _, rest2} -> {:ok, exprs, rest2} @@ -1150,6 +1171,18 @@ defmodule Lua.Parser do defp consume([token | rest]), do: {token, rest} defp consume([]), do: {nil, []} + # Drop leading comment tokens from a token stream. + # + # The lexer emits `{:comment, type, text, pos}` tuples (deliberately, so + # tooling can attach them to the AST via `Lua.Parser.Comments`). They are + # collected at statement and expression-prefix boundaries, but inside list + # constructs (function arguments, table fields, parenthesized lists) a + # trailing or interleaved comment must be skipped before peeking for a + # separator (`,`, `;`) or terminator (`)`, `}`, `]`). Without this skip, + # a comment leaks into `expect/3` and crashes the executor. + defp skip_comments([{:comment, _, _, _} | rest]), do: skip_comments(rest) + defp skip_comments(tokens), do: tokens + # Expect a specific token type defp expect(tokens, expected_type) do case peek(tokens) do diff --git a/test/lua/parser/comment_test.exs b/test/lua/parser/comment_test.exs index d6db49bb..1c972b78 100644 --- a/test/lua/parser/comment_test.exs +++ b/test/lua/parser/comment_test.exs @@ -378,4 +378,168 @@ defmodule Lua.Parser.CommentTest do end end end + + describe "comments inside expression lists (regression for A3)" do + # Shrunken repro from test/lua53_tests/calls.lua:374, which crashed + # the executor with `no case clause matching: {:comment, :single, ...}`. + # The lexer emits comment tokens (deliberately, for AST consumers), but + # the parser was failing to skip them between expressions in a function + # argument list — so a trailing comment before `)` leaked into expect/3. + + test "trailing comment after final argument before closing paren" do + code = ~s[local x = math.max(1, 2, 3 -- LUAC_INT\n)\nreturn x] + assert {:ok, _chunk} = Parser.parse(code) + assert {[3], _} = Lua.eval!(code) + end + + test "trailing comment after every argument across multiple lines" do + code = """ + local function f(a, b, c) return a + b + c end + local x = f( + 1, -- first + 2, -- second + 3 -- third + ) + return x + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[6], _} = Lua.eval!(code) + end + + test "multiple consecutive comments before closing paren" do + # Mirrors the exact shape from calls.lua: a trailing line comment + # plus an additional standalone comment before the close paren. + code = """ + local function f(a) return a end + local x = f( + 42 -- LUAC_INT + -- another note (padding...) + ) + return x + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[42], _} = Lua.eval!(code) + end + + test "comment between argument and comma" do + code = """ + local function f(a, b) return a + b end + return f(1 -- one + , 2) + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[3], _} = Lua.eval!(code) + end + + test "long comment between arguments" do + code = """ + local function f(a, b) return a + b end + return f(1, --[[ block ]] 2) + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[3], _} = Lua.eval!(code) + end + + test "comments inside table constructor" do + code = """ + local t = { + 1, -- one + 2, -- two + 3 -- three + -- trailing standalone + } + return #t + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[3], _} = Lua.eval!(code) + end + + test "no :comment tuples leak into the AST passed to the compiler" do + # The lexer deliberately emits {:comment, ...} tokens for AST + # consumers, but they are stored as Meta annotations — they must + # never appear as raw tokens inside the parsed AST that flows to + # codegen and the executor. Walk every value in the parsed chunk + # and assert no comment tuple is reachable except through Meta's + # comments field. + code = """ + local function f(a, b, c) return a + b + c end + local x = f( + 1, -- one + 2, -- two + 3 -- three + -- trailing + ) + local t = { + 10, -- ten + 20, -- twenty + -- standalone + } + return x, #t + """ + + assert {:ok, chunk} = Parser.parse(code) + refute contains_comment_tuple?(chunk) + end + + test "exact calls.lua:374 shape parses and evaluates" do + # Direct repro of the failing pattern from calls.lua (without + # the actual string.pack invocation, which is unrelated). + code = """ + local function pack(...) return select("#", ...) end + local n = pack( + "\\27Lua", -- signature + 5*16 + 3, -- version 5.3 + 0, -- format + "data", -- data + 4, -- sizeof(int) + 8, -- sizeof(size_t) + 4, -- size of instruction + 8, -- sizeof(lua integer) + 8, -- sizeof(lua number) + 0x5678 -- LUAC_INT + -- LUAC_NUM may not have a unique binary representation (padding...) + ) + return n + """ + + assert {:ok, _chunk} = Parser.parse(code) + assert {[10], _} = Lua.eval!(code) + end + end + + # Recursively check whether a parsed AST contains any raw + # `{:comment, ...}` lexer tuple as a sub-term, ignoring the Meta + # `comments` field where they're legitimately stored as maps. + defp contains_comment_tuple?(%Meta{} = _meta) do + # Comments stored on Meta are %{type: ..., text: ..., position: ...} maps, + # not raw tuples, so traversing into Meta is unnecessary. + false + end + + defp contains_comment_tuple?({:comment, _, _, _}), do: true + + defp contains_comment_tuple?(%_{} = struct) do + struct + |> Map.from_struct() + |> contains_comment_tuple?() + end + + defp contains_comment_tuple?(map) when is_map(map) do + Enum.any?(map, fn {_k, v} -> contains_comment_tuple?(v) end) + end + + defp contains_comment_tuple?(list) when is_list(list) do + Enum.any?(list, &contains_comment_tuple?/1) + end + + defp contains_comment_tuple?(tuple) when is_tuple(tuple) do + tuple |> Tuple.to_list() |> contains_comment_tuple?() + end + + defp contains_comment_tuple?(_), do: false end From 13f102810f875f4d8dff8138e75ca8d828554ad9 Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Tue, 5 May 2026 12:31:01 -0700 Subject: [PATCH 3/4] chore(A3): mark plan as review --- .agents/plans/A3-comment-leak.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/.agents/plans/A3-comment-leak.md b/.agents/plans/A3-comment-leak.md index 937aa7c6..dd211a78 100644 --- a/.agents/plans/A3-comment-leak.md +++ b/.agents/plans/A3-comment-leak.md @@ -2,10 +2,10 @@ id: A3 title: Comment tokens leak past lexer in calls.lua path issue: 164 -pr: null +pr: 182 branch: fix/lexer-comment-leak base: main -status: in-progress +status: review direction: A unlocks: - calls.lua @@ -98,3 +98,20 @@ mix test --only lua53 sandboxed — a separate, pre-existing concern out of scope here. - `string.pack` is also unimplemented; the original repro pattern from `calls.lua:374` would hit that next. Both are tracked elsewhere. + +## What changed + +- `lib/lua/parser.ex`: added a `skip_comments/1` helper near the + `peek`/`consume` token utilities. Applied it at three boundaries: + `parse_expr_list_acc/2` (gated on comma lookahead so trailing + comments still reach `parse_stmt`), `parse_expr_list_until/2` + (before terminator peek), and `parse_table_fields/2` (before and + after each field). +- `test/lua/parser/comment_test.exs`: new `describe` block "comments + inside expression lists (regression for A3)" with 7 regression + tests covering function-arg lists and table constructors, plus a + test that walks the full parsed AST asserting no `{:comment, ...}` + tuple survives into the structure passed to codegen. +- Suite: 1301 → 1309 tests passing, 0 failures, 0 regressions. +- PR: #182. No follow-up issues opened — `require` sandboxing and + `string.pack` were pre-existing. From 071bf5168e79754731d25c082d048d1258d69ad1 Mon Sep 17 00:00:00 2001 From: Dave Lucia Date: Tue, 5 May 2026 12:33:47 -0700 Subject: [PATCH 4/4] docs: mark A1, A2, A3 as in review on roadmap --- ROADMAP.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index f2acc282..0c6a111b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -43,9 +43,12 @@ under the [`0.5.0` milestone](https://github.com/tv-labs/lua/milestone/1). - ~~**A0**: 64-bit integer overflow wrapping for arithmetic and bitwise ops (Lua 5.3 §3.4.1; deliberate divergence from Luerl bignum semantics).~~ Shipped in PR #177. -- **A1**: Empty/missing-key table reads return `nil` (unblocks ~6 files). -- **A2**: Long-string `[[ … ]]` lexer handles embedded `]` and level brackets `[==[`. -- **A3**: Comment tokens leak past lexer in `calls.lua`. +- ~~**A1**: Empty/missing-key table reads return `nil` (unblocks ~6 files).~~ + In review in PR #179. +- ~~**A2**: Long-string `[[ … ]]` lexer handles embedded `]` and level brackets `[==[`.~~ + In review in PR #180. +- ~~**A3**: Comment tokens leak past lexer in `calls.lua`.~~ + In review in PR #182. - **A4**: Pre-load Lua stdlibs into `package.loaded` so `require"io"` resolves. ### Per-file assertions (one PR each, ≤ ½ day)