Skip to content

fix(pattern): honour ^ anchor in gsub and literal caret in gmatch - #406

Merged
davydog187 merged 2 commits into
deviceio:mainfrom
smn:fix-gsub-anchor
Jul 28, 2026
Merged

davydog187 merged 2 commits into
deviceio:mainfrom
smn:fix-gsub-anchor

Conversation

@smn

@smn smn commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What

Fix two related string pattern bugs where a leading ^ was mishandled:

  • string.gsub ignored the ^ anchor. An anchored pattern only
    matches at the start of the subject (Lua 5.3 §6.4.1), so gsub must
    attempt at most one replacement at position 0 and leave the remainder
    untouched. We were discarding the anchored flag from compile/1 and
    scanning every position, so ("hello"):gsub("^h", "H")-style calls
    replaced interior matches too.

  • string.gmatch treated a leading ^ as an anchor. In gmatch a
    leading ^ has no special meaning (Lua 5.3 §6.4.3) — PUC-Lua's
    gmatch_aux never strips it, so match sees it as an ordinary
    character. We were honouring it as an anchor, which prevented iteration.

How

  • gsub/gsub_stateful now route through a new do_gsub/6 that
    branches on the anchored flag:
    • anchored → match once at position 0, splice the replacement onto the
      untouched remainder, return a count of 0 or 1. max_n <= 0 short-
      circuits to zero replacements.
    • unanchored → existing gsub_from/10 full scan.
  • gmatch recompiles the pattern with the caret kept literal when
    compile/1 reports it as anchored, mirroring PUC-Lua's behaviour.

Tests

New test/lua/vm/stdlib/pattern_anchor_test.exs covering:

  • Anchored string.gsub — leading-only replacement, no match when
    the subject doesn't start with the pattern, multi-char runs replaced at
    most once, empty anchored match, leading-whitespace trim preserving
    interior/trailing whitespace, n = 0 suppression, captures reaching a
    function replacement exactly once, and whole-subject matches.
  • Leading caret in string.gmatch — ^ does not anchor and matches
    only as a literal character.
  • Anchored Pattern.gsub/4 — at-most-once replacement and max_n = 0.

Pattern.compile/1 returns an {anchored, elements} tuple, but only find/3
used the flag - gsub/4, gsub_stateful/5, and gmatch/2 discarded it.
Because compile strips the leading ^, those entry points matched the
remaining pattern at every position, so string.gsub("xax", "^x", "Y")
returned "YaY", 2 instead of "Yax", 1, silently corrupting the common
leading-trim idiom s:gsub("^%s+", "").

Route gsub and gsub_stateful through an anchored branch that tries the
match only at position 0 and replaces at most once, mirroring PUC-Lua
str_gsub's anchor flag. An n of 0 still suppresses the replacement, and
an empty anchored match (e.g. ^a* on "bbb") replaces once at the start.

gmatch needed the opposite treatment: per Lua 5.3 (and PUC-Lua's
gmatch_aux, which never strips the caret), a leading ^ in gmatch is not
an anchor but an ordinary literal character - so recompile the pattern
keeping the caret literal instead of dropping it.

Expected values in the regression tests are verified against PUC-Lua
5.3.6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3LBtzmXgAMHA8QiGPa5Ek
@davydog187

Copy link
Copy Markdown
Contributor

🤖 Automated review — Opus + Sonnet consolidated findings

Two independent AI reviewers (Opus, Sonnet) reviewed this PR, both by differential testing against a real Lua interpreter. The fix is correct: a 10,200-case differential fuzz (20 subjects × 34 ^-leading patterns × string/function/table replacements × n ∈ {nil, 0, 1, 2, −1} × gsub/gmatch/find/match) produced 0 divergences from PUC-Lua on this branch, vs 3,224 divergences on pre-fix main — so both bugs are real and the fix is complete over that space. Error/state-threading probes (callback errors, side-effect preservation, nested gsub in an anchored callback, %2 invalid index) all match PUC too. Semantics were cross-checked against lstrlib.c (str_gsub's if (anchor) break;, gmatch_aux never stripping ^). Thanks @smn — nice fix.

Everything below is take-it-or-leave-it polish, no blockers:

🟡 LOW

  • lib/lua/vm/stdlib/pattern.ex:77 (and pattern_anchor_test.exs:8) — the comment cites "Lua 5.3 §6.4.3", which doesn't exist (5.3 has §6.4.1 Patterns and §6.4.2 Pack format); the rule lives in the string.gmatch entry of §6.4. Also, the "prevented iteration" framing is backwards: pre-fix gmatch("aaa","^a") silently dropped the anchor and over-matched (3 unanchored matches), rather than preventing iteration.
  • lib/lua/vm/stdlib/pattern.ex:81-85 — gmatch compiles a ^-leading pattern twice (compile/1's result is discarded, then compile_elements(pattern, []) re-parses). A String.starts_with?(pattern, "^") check would express the intent without the wasted parse. (Both reviewers flagged this independently.)
  • lib/lua/vm/stdlib/pattern.ex:169-180 — the anchored do_gsub clause re-implements the match → apply_replacement → splice sequence gsub_from/10 already owns; verified equivalent today, but a future gsub_from fix (e.g. the %f frontier work) would silently miss the anchored path.
  • lib/lua/vm/stdlib/pattern.ex:165 — the max_n != nil and max_n <= 0 guard passes non-numeric max_n via term ordering (reachable only through the Elixir-level Pattern.gsub/4; string.gsub coerces first). is_integer(max_n) and max_n <= 0, applied once before the anchored/unanchored split, would be tighter.
  • Style: the positional-boolean dispatch do_gsub(subject, true/false, …) would read better as gsub_anchored/gsub_unanchored (or :anchored/:unanchored atoms).

Suggested test additions (all verified to behave correctly already — just unpinned)

  • Anchored gsub with n > 1: ("aaa"):gsub("^a", "X", 3) == "Xaa", 1 — this is the test that pins "a positive limit still allows at most one replacement", the core claim of the fix.
  • Anchored gsub with a table replacement (that branch never sees the anchored path in any test) and with a failing $ (("abc\n"):gsub("^%a*$", "X") → unchanged, 0).
  • gmatch literal-caret beyond the plain case: captures (("^a^b"):gmatch("^(%a)") → a, b), a quantifier bound to the caret, caret-plus-$.
  • Worth noting: the vendored suite's anchored cases (pm.lua:95/:148/:161) passed before this fix by accident (unanchored scan happens to match at position 0 for the trim idiom), so this PR's unit file is the real guard.

Out of scope (observed while testing; will be filed as issues, not this PR's problem)

  • print() of a non-UTF-8 binary raises (:io.put_chars argument error) where PUC writes raw bytes — e.g. print(("日本"):gsub("^.", "X")) crashes even though the gsub result is byte-identical to PUC.
  • "invalid capture index %2 in replacement string" is Lua 5.4 wording; PUC 5.3 says "invalid capture index %2".

The anchored gsub path was only pinned at n = 0, and the literal-caret
gmatch path had no coverage for captures, quantifiers, or a trailing end
anchor. Add the missing cases:

- anchored gsub with n > 1 (2, 3, 5) still replaces at most once and
  reports a count of 1 — the core claim of the anchor fix
- anchored gsub with a negative n leaves the subject untouched, count 0
- anchored gsub with a table replacement, which previously never reached
  the anchored branch in any test
- an anchored pattern whose trailing $ fails to match, count 0
- gmatch literal caret with a capture, yielding each captured letter
- gmatch literal caret with a quantifier bound to it
- gmatch literal caret followed by an end anchor
- Pattern.gsub/4 directly, for max_n above 1 and for a negative max_n

Also correct a spec citation: the gmatch caret rule lives in the
string.gmatch entry of Lua 5.3 §6.4, not in a nonexistent §6.4.3.

All expected values were verified against the reference interpreter
(PUC-Lua 5.4.8 and 5.5.0, which agree on every case here) before being
written into the assertions. No lib behaviour changes.

Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
@davydog187

Copy link
Copy Markdown
Contributor

Thanks for this — nice catch, and the diagnosis is exactly right. I ran a differential fuzz of the patched gsub/gmatch paths against PUC-Lua (10,200 cases) and got 0 divergences, so the fix is good as-is.

I pushed one test-only commit (e2ddae1) to the branch, filling in a few anchor cases that weren't pinned yet:

  • anchored gsub with n > 1 — the most important one, since it's the core claim of the fix and the file only covered n = 0. ("aaa"):gsub("^a", "X", 3) -> "Xaa", 1.
  • anchored gsub with a negative n, and with a table replacement (the table branch never reached the anchored path in any test)
  • an anchored pattern whose trailing $ fails
  • gmatch literal caret with a capture, with a quantifier bound to the caret, and followed by an end anchor

All expected values were taken from the reference interpreter rather than written by hand. I also corrected the §6.4.3 citation in the comments — 5.3 has no such section; the gmatch caret rule lives in the string.gmatch entry of §6.4.

No lib/ behaviour changed. Feel free to squash or reword that commit however you like.

@davydog187
davydog187 merged commit bca8b14 into deviceio:main Jul 28, 2026
5 checks passed
@smn

smn commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

awesome, thank you!

@davydog187 davydog187 mentioned this pull request Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants