Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 79 additions & 7 deletions .agents/plans/A9-pm-suite.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
id: A9
title: Fix pm.lua pattern-matching assertion
issue: 170
pr: null
pr: 188
branch: fix/pm-suite
base: main
status: ready
status: review
direction: A
unlocks:
- pm.lua
Expand All @@ -23,10 +23,16 @@ generic `assertion failed` — needs triage.

## Success criteria

- [ ] `mix test` passes (≥ 1273, no regressions)
- [ ] Each fixed assertion has a unit test in
`test/lua/vm/string_test.exs` or `test/lua/vm/pattern_test.exs`.
- [ ] `pm.lua` passes end-to-end.
- [x] `mix test` passes (≥ 1273, no regressions)
- [x] Each fixed assertion has a unit test in
`test/lua/vm/string_test.exs` or `test/lua/vm/lexer_test.exs`.
- [ ] `pm.lua` passes end-to-end. **Narrowed.** The triage on this plan
uncovered five distinct, layered bugs blocking pm.lua. The first
four are fixed in this PR (see Discoveries) and now pm.lua reaches
line 112. The fifth — an executor crash on `string.char(range(0, 255))`
caused by multi-return tail-call register expansion — is a separate
concern far from the pattern engine and is split into a follow-up
plan A9a (`fix/pm-suite-executor`).

## Implementation notes

Expand Down Expand Up @@ -65,4 +71,70 @@ mix test test/lua/vm/string_test.exs

## Discoveries

(populated during implementation)
Triage walked the file in order, fixing each blocker as it surfaced. Each
fix has a unit test attached; pm.lua acts as the integration regression.

1. **Lexer never decoded numeric escapes.** `"a\0o"` parsed as four bytes
(`a`, `\`, `0`, `o`) instead of three. Affected pm.lua from line 21
onward — almost every assertion uses `\0`, `\200`, `\xff`, etc.
- Added `\ddd`, `\xXX`, `\u{...}` handling to `lib/lua/lexer.ex`,
including UTF-8 encoding for codepoints up to 0x7FFFFFFF.
- Updated the existing "invalid escape" test, which incorrectly
asserted that `\1` and `\x` should pass through verbatim. Per the
Lua 5.3 reference manual §3.1, `\ddd` is a valid decimal escape
and a bare `\x` is a malformed escape (parse-time error).

2. **Pattern engine missing `%g`, `%G`, `%x`, `%X` character classes.**
pm.lua line 51 (`%g%g%g+`) was the first user; many later lines use
`%x` for hex-digit matching.

3. **`string.gsub` crashed when an end-anchored zero-width pattern
matched at the end of the subject.** The `pos > len` cleanup branch
passed a negative length to `:erlang.binary_part/3`. Fixed by
clamping the remainder length to zero.

4. **`string.sub(s, i, j)` returned the wrong value when `j == 0`.**
The `normalize_index/2` helper collapsed both `0` and `1` to the
same 0-based offset, so the empty `[1,0]` range became a one-char
match. Replaced with PUC-Lua's `posrelat`-then-clamp algorithm.

5. **Position captures `()` were treated as zero-width string captures
instead of numeric byte positions.** The `f1` helper in pm.lua (and
anything using `()` for position tracking) returned the wrong types.
Compiler now emits `:position_capture` for `()` and the matcher
records `pos + 1` as a number.

6. **Out of scope (split to A9a):** pm.lua line 112's
`string.char(range(0, 255))` triggers an executor crash —
`:erlang.setelement/3` is called with an index past the end of the
register tuple when a tail-recursive multi-return funnels 256 values
into a fixed-size register file. This is a pure executor bug
unrelated to patterns; addressing it requires register growth in
`Lua.VM.Executor.do_frame_return/6` and is tracked in
`.agents/plans/A9a-pm-suite-executor.md`.

## What changed

PR: #188

Files touched:

- `lib/lua/lexer.ex` — decimal/hex/unicode numeric escapes, plus
Bitwise import for the UTF-8 encoder.
- `lib/lua/vm/stdlib/pattern.ex` — `%g`/`%G`/`%x`/`%X` classes;
position-capture compile + match; gsub clamp on end-overflow.
- `lib/lua/vm/stdlib/string.ex` — PUC-Lua `posrelat`-then-clamp
semantics for `string.sub`.
- `test/lua/lexer_test.exs` — new tests for `\ddd`, `\xXX`,
`\u{...}`, malformed `\x`, plus relaxed "unknown escape"
passthrough; the old "invalid escape" test (which contradicted the
Lua 5.3 reference) was replaced.
- `test/lua/vm/string_test.exs` — regression tests for `string.sub`
edge cases, `%g`/`%x` classes, position captures, NUL-in-pattern,
and the gsub end-overflow.

Suite delta: 1327 → 1342 unit tests (+15 regressions). lua53 official
suite: 4/24 ready (unchanged). pm.lua remains in `@skipped_tests` —
moving it to `@ready_tests` is gated on A9a.

Follow-up: [`.agents/plans/A9a-pm-suite-executor.md`](A9a-pm-suite-executor.md).
93 changes: 93 additions & 0 deletions .agents/plans/A9a-pm-suite-executor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
id: A9a
title: Fix multi-return register expansion exposed by pm.lua
issue: null
pr: null
branch: fix/pm-suite-executor
base: main
status: ready
direction: A
unlocks:
- pm.lua (continuation of A9)
---

## Goal

Make `pm.lua` runnable past line 112 by fixing the executor's multi-return
handling. Once unblocked, finish triaging any remaining pm.lua failures
and land the suite test under `@ready_tests`.

## Out of scope

- Pattern-engine work (covered by A9 / future A9b if needed).
- General executor refactors. Make the smallest change that grows the
register tuple safely on multi-return.

## Success criteria

- [ ] `string.char(<256 returns from a recursive multi-return function>)`
no longer crashes.
- [ ] `pm.lua` passes end-to-end (or the next failure surfaces and is
either fixed in scope or split into A9b).
- [ ] `pm.lua` is moved from `@skipped_tests` to `@ready_tests` in
`test/lua53_suite_test.exs`.
- [ ] Unit test added in `test/lua/vm/call_stack_test.exs` (or new file)
that exercises a recursive multi-return ≥ 100 values feeding a
variadic native call.
- [ ] `mix test` passes (≥ current count, no regressions).

## Implementation notes

Failure (reproducible from `pm.lua` line 112):

```
:erlang.setelement(26, {tuple of 25 elements}, 255)
lib/lua/vm/executor.ex:1143 do_frame_return/6
```

Triggered by:

```lua
local function range(i, j)
if i <= j then return i, range(i+1, j) end
end
local abc = string.char(range(0, 255))
```

`range(0, 255)` returns 256 values via tail-position multi-return. The
`-2` (multi-return expansion) branch in `do_frame_return/6` writes them
into the caller's register tuple starting at `base`, but the tuple was
allocated by the compiler for the syntactic call site and isn't sized
for 256 expanded slots.

Likely fix: in the `-2` arm of `do_frame_return/6`, ensure
`caller_regs` is grown to at least `base + length(results_list)` slots
before the `Enum.reduce/3` writes happen. The same precaution probably
belongs in the fixed-N arm (`n > 0`) and in `continue_after_call/12`.

Use `:erlang.tuple_size/1` to check capacity, and rebuild the tuple
with appended `nil`s if needed. Keep the change to a single helper
function so the three call sites stay aligned.

## Verification

```bash
mix format
mix compile --warnings-as-errors
mix test
mix test --only lua53
mix test test/lua/vm/call_stack_test.exs
mix test test/lua/vm/string_test.exs
```

## Risks

- Growing the register tuple changes a hot-path invariant. Make sure
the new path is taken only when actually needed (`needed_size >
tuple_size(regs)`), so the common case stays a single `put_elem`.
- Coroutines and pcall use the same return path; verify their tests
still pass before opening the PR.

## Discoveries

(populated during implementation)
105 changes: 105 additions & 0 deletions lib/lua/lexer.ex
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ defmodule Lua.Lexer do
Tokenizes Lua source code into a list of tokens with position tracking.
"""

import Bitwise

@type position :: %{line: pos_integer(), column: pos_integer(), byte_offset: non_neg_integer()}
@type token ::
{:keyword, atom(), position()}
Expand Down Expand Up @@ -353,6 +355,45 @@ defmodule Lua.Lexer do
scan_string(remaining, str_acc, acc, new_pos, start_pos, quote)
end

# \xXX hex escape: exactly two hex digits
defp scan_string(<<?\\, ?x, h1, h2, rest::binary>>, str_acc, acc, pos, start_pos, quote)
when h1 in ?0..?9 or h1 in ?a..?f or h1 in ?A..?F do
if hex?(h2) do
byte = hex_value(h1) * 16 + hex_value(h2)
scan_string(rest, str_acc <> <<byte>>, acc, advance_column(pos, 4), start_pos, quote)
else
{:error, {:invalid_escape, pos}}
end
end

defp scan_string(<<?\\, ?x, _rest::binary>>, _str_acc, _acc, pos, _start_pos, _quote) do
{:error, {:invalid_escape, pos}}
end

# \u{XXX} unicode escape (UTF-8 encoded codepoint)
defp scan_string(<<?\\, ?u, ?{, rest::binary>>, str_acc, acc, pos, start_pos, quote) do
case scan_unicode_escape(rest, 0, 0) do
{:ok, codepoint, digits, after_brace} when codepoint <= 0x7FFFFFFF ->
utf8 = encode_lua_utf8(codepoint)
# consumed: \u{ + digits + }
scan_string(after_brace, str_acc <> utf8, acc, advance_column(pos, 4 + digits), start_pos, quote)

_ ->
{:error, {:invalid_escape, pos}}
end
end

# \ddd decimal escape: 1-3 decimal digits, value must fit in a byte
defp scan_string(<<?\\, d1, rest::binary>>, str_acc, acc, pos, start_pos, quote) when d1 in ?0..?9 do
{value, digits, remaining} = read_decimal_escape(d1 - ?0, 1, rest)

if value > 255 do
{:error, {:invalid_escape, pos}}
else
scan_string(remaining, str_acc <> <<value>>, acc, advance_column(pos, 1 + digits), start_pos, quote)
end
end

defp scan_string(<<?\\, esc, rest::binary>>, str_acc, acc, pos, start_pos, quote) do
# Escape sequence
case escape_char(esc) do
Expand All @@ -377,6 +418,70 @@ defmodule Lua.Lexer do
scan_string(rest, str_acc <> <<c>>, acc, advance_column(pos, 1), start_pos, quote)
end

# Read up to two more decimal digits for a \ddd escape
defp read_decimal_escape(value, digits, <<d, rest::binary>>) when d in ?0..?9 and digits < 3 do
next = value * 10 + (d - ?0)

if next > 255 do
{value, digits, <<d, rest::binary>>}
else
read_decimal_escape(next, digits + 1, rest)
end
end

defp read_decimal_escape(value, digits, rest), do: {value, digits, rest}

# Read hex digits inside \u{...}
defp scan_unicode_escape(<<?}, rest::binary>>, value, digits) when digits > 0 do
{:ok, value, digits + 1, rest}
end

defp scan_unicode_escape(<<c, rest::binary>>, value, digits) when digits < 8 do
if hex?(c) do
scan_unicode_escape(rest, value * 16 + hex_value(c), digits + 1)
else
:error
end
end

defp scan_unicode_escape(_rest, _value, _digits), do: :error

defp hex?(c), do: c in ?0..?9 or c in ?a..?f or c in ?A..?F

defp hex_value(c) when c in ?0..?9, do: c - ?0
defp hex_value(c) when c in ?a..?f, do: c - ?a + 10
defp hex_value(c) when c in ?A..?F, do: c - ?A + 10

# Encode a codepoint as UTF-8. Lua 5.3 accepts codepoints up to 0x7FFFFFFF
# (6-byte UTF-8), beyond what Erlang's :unicode module emits, so handle the
# full range manually.
defp encode_lua_utf8(c) when c < 0x80 do
<<c>>
end

defp encode_lua_utf8(c) when c < 0x800 do
<<0b110_00000 ||| c >>> 6, 0b10_000000 ||| (c &&& 0b111111)>>
end

defp encode_lua_utf8(c) when c < 0x10000 do
<<0b1110_0000 ||| c >>> 12, 0b10_000000 ||| (c >>> 6 &&& 0b111111), 0b10_000000 ||| (c &&& 0b111111)>>
end

defp encode_lua_utf8(c) when c < 0x200000 do
<<0b11110_000 ||| c >>> 18, 0b10_000000 ||| (c >>> 12 &&& 0b111111), 0b10_000000 ||| (c >>> 6 &&& 0b111111),
0b10_000000 ||| (c &&& 0b111111)>>
end

defp encode_lua_utf8(c) when c < 0x4000000 do
<<0b111110_00 ||| c >>> 24, 0b10_000000 ||| (c >>> 18 &&& 0b111111), 0b10_000000 ||| (c >>> 12 &&& 0b111111),
0b10_000000 ||| (c >>> 6 &&& 0b111111), 0b10_000000 ||| (c &&& 0b111111)>>
end

defp encode_lua_utf8(c) do
<<0b1111110_0 ||| c >>> 30, 0b10_000000 ||| (c >>> 24 &&& 0b111111), 0b10_000000 ||| (c >>> 18 &&& 0b111111),
0b10_000000 ||| (c >>> 12 &&& 0b111111), 0b10_000000 ||| (c >>> 6 &&& 0b111111), 0b10_000000 ||| (c &&& 0b111111)>>
end

# Escape character mapping
defp escape_char(?a), do: {:ok, ?\a}
defp escape_char(?b), do: {:ok, ?\b}
Expand Down
24 changes: 22 additions & 2 deletions lib/lua/vm/stdlib/pattern.ex
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,10 @@ defmodule Lua.VM.Stdlib.Pattern do

defp gsub_from(subject, pos, len, _pattern, _repl, max_n, count, acc)
when pos > len or (max_n != nil and count >= max_n) do
# Append remaining subject
remaining = binary_part(subject, pos, len - pos)
# Append remaining subject (clamp pos so we never read past end_of_string)
remaining =
if pos < len, do: binary_part(subject, pos, len - pos), else: ""

{IO.iodata_to_binary([Enum.reverse(acc), remaining]), count}
end

Expand Down Expand Up @@ -187,6 +189,11 @@ defmodule Lua.VM.Stdlib.Pattern do
Enum.reverse([:anchor_end | acc])
end

defp compile_elements("()" <> rest, acc) do
# Position capture: records the current byte position (1-based) at this point.
compile_elements(rest, [:position_capture | acc])
end

defp compile_elements("(" <> rest, acc) do
compile_elements(rest, [:capture_start | acc])
end
Expand Down Expand Up @@ -285,6 +292,11 @@ defmodule Lua.VM.Stdlib.Pattern do
match_elements(subject, pos, rest, full, [{pos, nil} | cstack], captures)
end

# Position capture — record the current 1-based byte position as a number
defp match_elements(subject, pos, [:position_capture | rest], full, cstack, captures) do
match_elements(subject, pos, rest, full, cstack, captures ++ [pos + 1])
end

# Capture end
defp match_elements(subject, pos, [:capture_end | rest], full, [{start, nil} | cstack], captures) do
captured = binary_part(subject, start, pos - start)
Expand Down Expand Up @@ -477,6 +489,14 @@ defmodule Lua.VM.Stdlib.Pattern do
defp match_char_class(ch, ?c), do: ch < 32 or ch == 127
defp match_char_class(ch, ?C), do: not (ch < 32 or ch == 127)

# %g — printable characters except space (ASCII 0x21..0x7E)
defp match_char_class(ch, ?g), do: ch in 0x21..0x7E
defp match_char_class(ch, ?G), do: ch not in 0x21..0x7E

# %x — hexadecimal digit
defp match_char_class(ch, ?x), do: ch in ?0..?9 or ch in ?a..?f or ch in ?A..?F
defp match_char_class(ch, ?X), do: not (ch in ?0..?9 or ch in ?a..?f or ch in ?A..?F)

# Escaped literal (non-alphanumeric after %)
defp match_char_class(ch, literal), do: ch == literal

Expand Down
Loading
Loading