Repository navigation
Fix unbounded recursion on cyclic tables at the eval boundary - #407
Merged
Merged
Conversation
The standard Lua OOP idiom (T.__index = T) creates tables that contain
themselves. Both eval-boundary walks recursed into them forever,
growing memory without bound: Value.decode in decode: true mode and
Display.peek_table in decode: false mode.
The walk now terminates at the point of recurrence: decode leaves the
table's {:tref, id} reference there, mirroring how functions already
pass through as opaque references, and Display renders a :circular
peek. Shared non-cyclic references decode in full.
Contributor
Author
|
@smn this is the issue I mentioned |
Contributor
|
Thanks! 🍍 🍍 🍍 🍍 🍍 |
OTP 28's opacity checker false-positives on the MapSet the walkers capture in their entry closures (call_without_opaque on every MapSet call), failing CI. Specs on the private clauses don't appease it. A plain map with the ids as keys is the same structure MapSet wraps, so behavior is identical and there is no opaque type left to police.
Contributor
Author
Thank you for reviewing it! I've switched |
davydog187
approved these changes
Jul 27, 2026
Decoding now leaves a bare `{:tref, id}` wherever a table contains
itself, so that marker can travel back into `Value.encode/3` when a
caller re-encodes a decoded value. There was no clause for it, so the
call died with a bare FunctionClauseError.
Refuse it explicitly with a `Lua.RuntimeException` that names the
cause and points at `decode: false`, following the struct-refusal
clause above it. Passing the tref through live is not an option: the
id is only meaningful against the state it was decoded from, so
accepting it would let callers forge references into VM state.
Also strengthen the sibling-keys decode test to assert the full
decoded structure instead of `is_list/1`, and cover a cycle reached
through the sequence branch of the display peek (`t[1] = t`), which
had no test. Reformat two files the fix commit left unformatted.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The most common Lua OOP idiom creates a table that contains itself:
Client.__index = Clientis how instances delegate method lookup to the module table, so virtually every hand-rolled Lua "class" has this shape. Returning such a table fromLua.eval!/3never terminates on 1.0.1. Both eval-boundary walks recurse into the cycle and allocate a fresh snapshot of the whole table at every level:decode: true(the default):Lua.VM.Value.decode/2recursing through the{:tref, id}decode: false:Lua.VM.Display.peek_tablebuilding the peek forDisplay.TableMemory grows until something dies. We hit this in production hosting sandboxed Lua apps: worker processes run with
max_heap_sizeset, so the BEAM killed them mid-walk with an untrappable:killedwhile loading any module using this idiom (observed climbing past 2.9GB uncapped). The minimal reproduction is three lines:The fix
Thread a set of ancestor table ids through both walks and stop at the point of recurrence:
decode: true: the walk terminates by leaving the table's{:tref, id}reference at the recurrence. This mirrors the existing contract, where functions already pass through decoded tables as opaque references; a cyclic table is just another value with no acyclic plain-data representation. No new option and no behavior change for acyclic data.decode: false: the recurring occurrence carriespeek: :circularand renders as#Lua.Table<id: N, circular>.The ancestor set is path-based, so shared references that do not form a cycle (the same table under sibling keys) still decode and peek in full. Tests cover self-cycles, mutual cycles, shared non-cyclic references, and the sibling-key false-positive case.
One design note worth flagging: with reference substitution, someone serializing decoded data discovers a cycle at encode time (
cannot encode {:tref, 12}) rather than at the eval boundary. The alternative is raising a clear error when decode meets a cycle. We went with substitution for consistency with function passthrough, but happy to switch if you prefer the strict version.