Repository navigation
Description
Ergonomic encode/decode for deflua (an Lua.Encoder protocol + auto-marshalling)
Under the current VM, a deflua receives table and userdata arguments as opaque refs ({:tref, _} / {:udref, _}) that must be passed to Lua.decode!/2 before use, and to return a table or userdata it must call Lua.encode!/2, which allocates in the VM heap and therefore returns a new state that has to be threaded back out as {value, lua}.
In practice this pushes three repeating shapes into every non-trivial embedding:
-
Per-argument decode helpers. Most functions that take a table/options arg grow a private helper to turn the ref into an Elixir map/list (with passthrough for already-decoded values). These are near-identical across modules and drift in their edge cases (nil vs
[]vs binary vs integer). -
Manual
encode!+ state threading on every return:deflua compare(a_ref, b_ref), lua do a = decode_userdata!(lua, a_ref) b = decode_userdata!(lua, b_ref) {:ok, score, result_image} = Image.compare(a, b) {ref, lua} = Lua.encode!(lua, {:userdata, result_image}) {[score, ref], lua} end
The
{ref, lua} = Lua.encode!(...)/{[...], lua}dance is pure noise that obscures intent and is easy to get wrong (forget to threadlua, return the wrong tuple shape, etc.). -
Hand-written struct → table converters. Because structs can't be auto-encoded, each one that crosses into Lua needs a bespoke flattener, and that shape lives far from the struct definition.
Proposal
Three additions, each independently useful, opt-in, and backwards compatible.
A. An Lua.Encoder protocol
Let Elixir terms describe how they become Lua values, so structs marshal themselves:
defprotocol Lua.Encoder do
@doc "Return the Lua-materializable representation of `term`."
def to_lua(term)
end
defimpl Lua.Encoder, for: Point do
def to_lua(p), do: %{x: p.x, y: p.y}
end
defimpl Lua.Encoder, for: Texture do
def to_lua(texture), do: {:userdata, texture} # opaque handle
endKey design point — keep the protocol pure. Protocol-based encoders like Jason.Encoder work because encoding has no shared state (an impl emits iodata). Lua encoding is not like that: materializing a table mutates the VM heap and yields a new state, so a to_lua(term, lua) :: {ref, lua} protocol would force every impl to thread state and would not compose for nested values.
The fix is to separate description from materialization:
Lua.Encoder.to_lua/1is pure — it returns a plain Elixir descriptor (numbers, strings, lists, maps,{:userdata, term}, and nested structs that also implement the protocol). No state, no refs, no threading.- The library walks that descriptor once and materializes it into the heap, threading state internally. Callers never see
Lua.encode!or a ref.
This keeps the familiar "describe, don't serialize" shape while adapting it to a stateful target, and it makes nested encoding compose for free.
B. deflua return auto-encoding
If deflua ran its return value through the materializer above, the explicit {ref, lua} = Lua.encode!(...) pattern disappears:
# today
deflua compare(a, b), lua do
{:ok, score, img} = Image.compare(decode(a), decode(b))
{ref, lua} = Lua.encode!(lua, {:userdata, img})
{[score, ref], lua}
end
# proposed — return plain Elixir; the library materializes + threads state
deflua compare(a, b) do
{:ok, score, img} = Image.compare(a, b)
[score, img] # `img` is a %Texture{} -> Lua.Encoder selects userdata
enddeflua already auto-encodes scalars and lists; this extends the same treatment to userdata and structs via the protocol, so the explicit-state form (, lua do) is only needed when a function genuinely needs the VM (e.g. calling Lua.call_function/3), not merely to return a value.
C. deflua argument auto-decoding (opt-in)
Eliminate the per-argument decode helpers by letting deflua decode table/userdata args before the body runs:
# whole-function
deflua configure(name, opts), decode: true do
# `opts` is already an Elixir map/list, not a {:tref, _}
end
# or per-argument, when only some args should be decoded
deflua draw(image :: userdata, regions :: table), lua do
...
enddecode: true would map to a canonical decode: refs are decoded; already-decoded or empty values pass through unchanged, so direct Elixir callers (and tests) keep working without a %Lua{}.
Before / after
# today
deflua find(image_ref, query), lua do
image = decode_userdata!(lua, image_ref)
results = for r <- Engine.find(image, query), do: Region.cast(r)
{ref, lua} = Lua.encode!(lua, Enum.map(results, ®ion_to_map/1))
{[results != [], ref], lua}
end
# proposed
deflua find(image :: userdata, query) do
results = for r <- Engine.find(image, query), do: Region.cast(r)
[results != [], results] # %Region{}s encode via Lua.Encoder; state threaded by lib
endNo decode helper, no region_to_map/1, no Lua.encode!, no {value, lua}.
Design considerations / open questions
- State threading is the crux. The pure-descriptor split (A) is what makes auto-encode (B) viable without leaking
%Lua{}into user code. Worth prototyping the materializer first to confirm nested structs and{:userdata, _}compose cleanly. - Backwards compatibility. All three are additive and opt-in. Existing
{value, lua}returns and manualLua.encode!/decode!keep working; auto-encode only changes behavior for return values that previously would have errored (structs/userdata). - Built-ins +
@derive.Lua.Encoderimpls forMap/List/structs plus@derive {Lua.Encoder, only: [...]}for field selection would cover the common struct-flattening cases out of the box. - Consistent decode errors. A shared "decode this userdata or raise" helper would give callers uniform, well-scoped error messages instead of each re-implementing the pattern match.
- Decode symmetry. A full
Lua.Decoderprotocol is probably unnecessary; arg auto-decode (C) covers the common need.
Why
Embedding the VM in a large API surface — structs crossing the boundary in both directions, functions called both through the VM and directly from Elixir — currently forces a lot of mechanical marshalling boilerplate at every call site. Shipping A–C would let defluas read like ordinary Elixir functions that take and return ordinary Elixir values.
Activity
docs(release): refresh roadmap to rc.3, record perf baseline, scope t…