Skip to content

Parser emits invalid UTF-8 for a non-ASCII "unexpected character" #394

Closed

Description

@davydog187

Summary

The lexer is byte-oriented, so an unexpected non-ASCII character is reported by its
lone UTF-8 lead byte, and the parser splices that raw byte into the error message. The
resulting Lua.Parser.Error / Lua.CompilerException message — and
Lua.Parser.Error.to_map/2's message, despite its "wire-safe" contract — is not
valid UTF-8, so Jason.encode! raises invalid byte 0xE2 on it.

Found via a Sauron LiveView that JSON-encodes codegen parse errors: a pasted
box-drawing character (│, from curl copied out of a bordered terminal) bricked the
editor because the unencodable payload raised inside LiveView's diff.

Repro

{:error, [err | _]} = Lua.Parser.parse_structured("local x = │")
m = Lua.Parser.Error.to_map(err, "local x = │")
String.valid?(m.message)   #=> false  (ends in a lone 0xE2)
Jason.encode!(m)           #=> ** (Jason.EncodeError) invalid byte 0xE2

Root cause (3 linked defects)

  1. lib/lua/lexer.ex:272 — the unexpected-character clause matches a single byte
    <<c, _::binary>> and emits {:unexpected_character, c, pos}, capturing only the
    UTF-8 lead byte of a multibyte character. Match <<cp::utf8, _::binary>> and carry
    the full codepoint, with a separate fallback clause for genuinely-invalid UTF-8
    bytes (report invalid byte 0xNN).
  2. lib/lua/parser.ex:1505 — "Unexpected character: #{<<char>>}" splices the raw
    byte. Render the codepoint safely (<<cp::utf8>>) and name it U+XXXX.
  3. Column / byte-offset counting — string/comment/identifier scanners advance
    advance_column(pos, 1) per byte (e.g. lexer.ex:306, lexer.ex:343), so
    position.column is miscounted past any multibyte character (in a string, comment,
    or after ..). Advance column per codepoint so positions are UTF-8-correct.

Also

to_map/2's docstring promises "No ANSI escapes appear in any string field" but makes
no UTF-8-validity guarantee; defect #2 means the "wire-safe" map isn't actually
wire-safe. Worth making that guarantee real, with a regression test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions