Skip to content

fix(parser): report syntax errors at the real offending token - #357

Merged
davydog187 merged 2 commits into
mainfrom
fix/parser-error-position-commit-on-progress
Jun 14, 2026
Merged

davydog187 merged 2 commits into
mainfrom
fix/parser-error-position-commit-on-progress

Conversation

@davydog187

Copy link
Copy Markdown
Contributor

Problem

A user pasted a workflow program with a genuine syntax error — an unclosed call whose ) is never supplied before the enclosing end:

w:step("w.h42zel", "code", "Code", { continueOnError = false }, function(output)
    local usercode = function()
      device.doesntExist(
    end
    return usercode()
end)

The parser reported:

Parse Error at line 23, column 62: Expected :delimiter::rparen, got :keyword::function

…with the caret on the function keyword on the outer w:step(...) line — ~3 lines above the real mistake. Reference Lua (luac -p) points at the end (unexpected symbol near 'end').

Root cause

The error-swallowing recovery branch in parse_expr_list_acc/2:

{:error, reason} ->
  if acc == [] do
    {:error, reason}
  else
    {:ok, Enum.reverse(acc), tokens}   # discards the real deep error
  end

Parsing the w:step(...) argument list, 4 arguments parse, then the 5th (the function literal) fails deep inside (the unclosed device.doesntExist(). Because acc is non-empty, the branch throws away that real error and returns the partial list with the stream still pointed at function; the caller's expect(rparen) then blames function.

This was the only error-swallowing recovery site in the parser (audited).

Fix

Adopt the recursive-descent commit-on-progress rule: recover (treat the list as ended) only when the sub-parse made no progress — the next token simply can't begin an expression. A committed failure (input consumed, or EOF) propagates as the real error. Progress is detected via the monotonic byte_offset already present on every token.

Also surface genuinely-unclosed delimiters precisely: parse_call_args, parse_index, parse_table, and parenthesised expressions now capture their opening position and, when the closer is missing at EOF, emit the existing Error.unclosed_delimiter/2 ("Unclosed ( … add a closing ) to match the opening at line N") pointing at the opener instead of a bare got :eof.

After the fix, the reported program points at the end, matching luac.

Tests

New test/lua/parser/error_position_test.exs:

  • Golden corpus (CI-deterministic): pins the reported line for deep errors in call args / table fields / return lists, over-propagation guards (a stray token after a complete list still reports at the boundary), and unclosed constructs.
  • Differential (@tag :differential, skipped without luac): cross-checks our reported line against luac -p for every concrete-offending-token case — all agree.

Verification

  • mix test — 2353 passed, 0 failures (19 pre-existing skips)
  • mix test test/lua/parser/error_position_test.exs --include differential — 13 passed
  • mix format --check-formatted clean

🤖 Generated with Claude Code

A deeply-nested sub-parse failure was swallowed by the partial-list
recovery branch in `parse_expr_list_acc/2`: once one or more arguments
had parsed, any later failure returned the partial list with the stream
left pointing at the start of the failed expression, so the caller's
`expect(rparen)` blamed that token at the list boundary instead of the
actual mistake.

For example:

    w:step("a", "b", "c", {x = 1}, function()
      bar(
    end)

reported "Expected :delimiter::rparen, got :keyword::function" at the
`function` keyword, three lines above the unclosed `bar(` / unexpected
`end`. Reference Lua (`luac -p`) points at the `end`; we now agree.

Adopt the recursive-descent "commit on progress" rule: only recover
(treat the list as ended) when the sub-parse made no progress — i.e. the
next token simply cannot begin an expression. A committed failure (input
consumed, or EOF) propagates as the real error. Progress is detected via
the monotonic `byte_offset` already on every token.

Also surface genuinely-unclosed delimiters precisely: `parse_call_args`,
`parse_index`, `parse_table`, and parenthesised expressions now capture
their opening position and, when the closer is missing at EOF, emit the
existing `Error.unclosed_delimiter/2` ("Unclosed (… add a closing ) to
match the opening at line N") pointing at the opener rather than a bare
"got :eof" at end-of-file.

Add `test/lua/parser/error_position_test.exs`: a CI-deterministic golden
corpus pinning the reported line for deep errors, over-propagation
guards, and unclosed constructs, plus a `:differential` test that
cross-checks the reported line against `luac -p` when available.
The new luac-differential error-position tests shell out to a reference
`luac` and assert exact line-number agreement. That is environment- and
version-dependent: CI has no compatible `luac`, so the suite (which only
excluded `:slow`) ran them and they failed. They were always intended as
an opt-in local check alongside the CI-deterministic golden corpus.

Add `:differential` to the default exclude list; run them explicitly with
`mix test --include differential`.
@davydog187
davydog187 merged commit a1d6dc9 into main Jun 14, 2026
5 checks passed
@davydog187
davydog187 deleted the fix/parser-error-position-commit-on-progress branch June 14, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant