Skip to content

Tables are never freed: a long-lived state grows ~1.6 MB per 1,000-row render #429

Open

Description

@shinyobjectz

ROADMAP lists GC and weak tables as a deliberate non-goal for 1.0. This issue is about what that costs a host that keeps one state alive and does real work in it, plus a narrow proposal that doesn't need the full gc.lua semantics.

Numbers

A render that builds 1,000 small row tables, joins them into a string and returns the string. No table survives the call:

local function render(n)
  local rows = {}
  for i = 1, n do rows[i] = { tag = "li", attrs = { class = "row" }, children = { "item " .. i } } end
  local out = {}
  for i, r in ipairs(rows) do out[i] = "<" .. r.tag .. ">" .. r.children[1] .. "</" .. r.tag .. ">" end
  return table.concat(out)
end
for i = 1, ITER do render(1000) end
renders map_size(state.tables) state size (:erts_debug.flat_size) time
1 3,014 1.6 MB 20 ms
10 30,032 16 MB 101 ms
50 150,112 82 MB 1,125 ms

(main @ 3f2aead, Apple M4, OTP 29. collectgarbage("count") returns 0.0.)

Memory grows linearly with work done, not with live data. Time grows faster than linearly, presumably because the BEAM copies an ever-larger state. In our host (an agent's computer that runs Lua under a 64 MB process heap), about 20 server-rendered HTML pages in one run hit the cap. Each page renders in about 17 ms on its own.

Proposal: collect at safe points

The hard part of collecting during a run is that live registers are Erlang locals in the dispatcher. There is one point where no Lua frame is live: between host calls, after Lua.eval! or Lua.call_function has returned.

  • Lua.gc(lua) (or an opt-in :gc option applied on return) marks from the roots: _G, package.loaded, the metatables for each type, the upvalue cells of reachable closures, and the tables reachable from them.
  • Values the host still holds ({:tref, id} handed out by Lua.get!, decoded returns, and so on) would need pinning, or would be documented as invalid after gc.
  • Then sweep tables and upvalue_cells.

This frees a long-lived state's garbage without touching the hot path. A later step could run the same mark during a run at collectgarbage() calls, with roots taken from the frames.

We'd be glad to send a PR for the between-calls version if the shape is acceptable. Related: #425.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions