You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parser emits invalid UTF-8 for a non-ASCII "unexpected character" #394
The lexer is byte-oriented, so an unexpected non-ASCII character is reported by its
lone UTF-8 lead byte, and the parser splices that raw byte into the error message. The
resulting Lua.Parser.Error / Lua.CompilerException message — and Lua.Parser.Error.to_map/2's message, despite its "wire-safe" contract — is not
valid UTF-8, so Jason.encode! raises invalid byte 0xE2 on it.
Found via a Sauron LiveView that JSON-encodes codegen parse errors: a pasted
box-drawing character (│, from curl copied out of a bordered terminal) bricked the
editor because the unencodable payload raised inside LiveView's diff.
Repro
{:error,[err|_]}=Lua.Parser.parse_structured("local x = │")m=Lua.Parser.Error.to_map(err,"local x = │")String.valid?(m.message)#=> false (ends in a lone 0xE2)Jason.encode!(m)#=> ** (Jason.EncodeError) invalid byte 0xE2
Root cause (3 linked defects)
lib/lua/lexer.ex:272 — the unexpected-character clause matches a single byte <<c, _::binary>> and emits {:unexpected_character, c, pos}, capturing only the
UTF-8 lead byte of a multibyte character. Match <<cp::utf8, _::binary>> and carry
the full codepoint, with a separate fallback clause for genuinely-invalid UTF-8
bytes (report invalid byte 0xNN).
lib/lua/parser.ex:1505 — "Unexpected character: #{<<char>>}" splices the raw
byte. Render the codepoint safely (<<cp::utf8>>) and name it U+XXXX.
Column / byte-offset counting — string/comment/identifier scanners advance advance_column(pos, 1) per byte (e.g. lexer.ex:306, lexer.ex:343), so position.column is miscounted past any multibyte character (in a string, comment,
or after ..). Advance column per codepoint so positions are UTF-8-correct.
Also
to_map/2's docstring promises "No ANSI escapes appear in any string field" but makes
no UTF-8-validity guarantee; defect #2 means the "wire-safe" map isn't actually
wire-safe. Worth making that guarantee real, with a regression test.
Activity