Skip to content

Fix unbounded recursion on cyclic tables at the eval boundary - #407

Merged
davydog187 merged 3 commits into
deviceio:mainfrom
fedme:cyclic-table-fix
Jul 27, 2026
Merged

davydog187 merged 3 commits into
deviceio:mainfrom
fedme:cyclic-table-fix

Conversation

@fedme

@fedme fedme commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

The problem

The most common Lua OOP idiom creates a table that contains itself:

local Client = {}
Client.__index = Client   -- Client's own "__index" entry points back at Client

function Client.new(opts)
  return setmetatable({}, Client)
end

return Client

Client.__index = Client is how instances delegate method lookup to the module table, so virtually every hand-rolled Lua "class" has this shape. Returning such a table from Lua.eval!/3 never terminates on 1.0.1. Both eval-boundary walks recurse into the cycle and allocate a fresh snapshot of the whole table at every level:

  • decode: true (the default): Lua.VM.Value.decode/2 recursing through the {:tref, id}
  • decode: false: Lua.VM.Display.peek_table building the peek for Display.Table

Memory grows until something dies. We hit this in production hosting sandboxed Lua apps: worker processes run with max_heap_size set, so the BEAM killed them mid-walk with an untrappable :killed while loading any module using this idiom (observed climbing past 2.9GB uncapped). The minimal reproduction is three lines:

local T = {}
T.__index = T
return T

The fix

Thread a set of ancestor table ids through both walks and stop at the point of recurrence:

  • decode: true: the walk terminates by leaving the table's {:tref, id} reference at the recurrence. This mirrors the existing contract, where functions already pass through decoded tables as opaque references; a cyclic table is just another value with no acyclic plain-data representation. No new option and no behavior change for acyclic data.
  • decode: false: the recurring occurrence carries peek: :circular and renders as #Lua.Table<id: N, circular>.

The ancestor set is path-based, so shared references that do not form a cycle (the same table under sibling keys) still decode and peek in full. Tests cover self-cycles, mutual cycles, shared non-cyclic references, and the sibling-key false-positive case.

One design note worth flagging: with reference substitution, someone serializing decoded data discovers a cycle at encode time (cannot encode {:tref, 12}) rather than at the eval boundary. The alternative is raising a clear error when decode meets a cycle. We went with substitution for consistency with function passthrough, but happy to switch if you prefer the strict version.

The standard Lua OOP idiom (T.__index = T) creates tables that contain
themselves. Both eval-boundary walks recursed into them forever,
growing memory without bound: Value.decode in decode: true mode and
Display.peek_table in decode: false mode.

The walk now terminates at the point of recurrence: decode leaves the
table's {:tref, id} reference there, mirroring how functions already
pass through as opaque references, and Display renders a :circular
peek. Shared non-cyclic references decode in full.
@fedme

fedme commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

@smn this is the issue I mentioned

@davydog187

Copy link
Copy Markdown
Contributor

Thanks!

🍍 🍍 🍍 🍍 🍍

@fedme
fedme force-pushed the cyclic-table-fix branch from a14f554 to b93f805 Compare July 27, 2026 12:22
OTP 28's opacity checker false-positives on the MapSet the walkers
capture in their entry closures (call_without_opaque on every MapSet
call), failing CI. Specs on the private clauses don't appease it. A
plain map with the ids as keys is the same structure MapSet wraps, so
behavior is identical and there is no opaque type left to police.
@fedme
fedme force-pushed the cyclic-table-fix branch from b93f805 to 8c6f432 Compare July 27, 2026 12:34
@fedme

fedme commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks!

🍍 🍍 🍍 🍍 🍍

Thank you for reviewing it! I've switched MapSet to a plain Map because Dialyzer was complaining about it.

Decoding now leaves a bare `{:tref, id}` wherever a table contains
itself, so that marker can travel back into `Value.encode/3` when a
caller re-encodes a decoded value. There was no clause for it, so the
call died with a bare FunctionClauseError.

Refuse it explicitly with a `Lua.RuntimeException` that names the
cause and points at `decode: false`, following the struct-refusal
clause above it. Passing the tref through live is not an option: the
id is only meaningful against the state it was decoded from, so
accepting it would let callers forge references into VM state.

Also strengthen the sibling-keys decode test to assert the full
decoded structure instead of `is_list/1`, and cover a cycle reached
through the sequence branch of the display peek (`t[1] = t`), which
had no test. Reformat two files the fix commit left unformatted.
@davydog187
davydog187 merged commit d178ef3 into deviceio:main Jul 27, 2026
5 checks passed
@davydog187 davydog187 mentioned this pull request Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants