Skip to content

feat(lua,rust): a LuaJIT FFI call reaches the Rust function and the operation handler it names - #3302

Closed
Ahaeflig wants to merge 1 commit into
colbymchenry:mainfrom
Ahaeflig:feat/lua-rust-bridge
Closed

Ahaeflig wants to merge 1 commit into
colbymchenry:mainfrom
Ahaeflig:feat/lua-rust-bridge

Conversation

@Ahaeflig

Copy link
Copy Markdown

A fresh take on #2331, which you closed with three asks. Each is done:

  • The daemon socket change is split out. It's now its own PR, fix(mcp): two CODEGRAPH_DIR indexes of one project each get their own daemon #3301.
  • The extraction version is 29, the next after main's 28.
  • It's validated on the native kernel. Lua and Rust go to the kernel by default, so that's the path users run. The whole suite ran with the kernel staged and CODEGRAPH_KERNEL_EXPECT=1. A new kernel test indexes a Lua + Rust project on both paths and requires the same graph. The real repositories below were indexed on both paths and compared row by row.

Beyond those asks, three things changed since #2331:

  • Ordinary Lua resolution is left alone. Connect LuaJIT FFI calls to Rust operation handlers #2331 stopped a Lua call through a parameter from resolving to a module of the same name, but only in projects that also held Rust, so the same Lua code resolved two ways depending on what else was in the repository.
  • Calls inside closures are credited. A call inside a closure now counts for the function around it.
  • Only some Lua files are analyzed. The pass now skips Lua files that can't add an edge.

The problem

A LuaJIT program that loads a Rust library calls it through the FFI by symbol. Often it goes through one shared export that runs a different handler for each operation name, like core.call("task.run", …) → #[no_mangle] extern "C" fn wire_send(op, …) → match op { "task.run" => run(), … }. Static extraction stops at both steps. The Lua call names a C symbol that no Lua file defines, so callers, impact and explore flows end on the Lua side. And reading through the export's match would make every Lua caller reach every handler.

What it does

A synthesis pass, run only in projects that hold both Lua and Rust, works from the source alone:

  • FFI calls. A call through a proven FFI receiver (require("ffi"), ffi.C, ffi.load(...) and their aliases) links by exact symbol to the one Rust C-ABI export with that symbol. If two exports share it, neither is linked.
  • Operations. A literal operation is followed through Lua and Rust wrappers that forward it, and linked to the one handler its match arm selects.
  • The shared export is a boundary. The call into it is kept as a transport reference that every walk treats as a stop: its callers and impact list the Lua call sites, while code behind it doesn't fan back out to every Lua caller.
  • Closures. A call inside a closure counts for the function around it, as the extractor's own calls do. It forwards only that function's own parameters, never the closure's.

It refuses rather than guesses on shadowed or mutated receivers, ambiguous exports, guarded or #[cfg] arms, and unsupported transforms. There are no built-in names. docs/design/lua-rust-bridge.md has the rules, the limits and the validation.

On cost:

  • Lua files. The pass analyzes a Lua file only if it can matter: the files holding the literal "ffi", then, round by round, the files that name a module where a carrier was found. A carrier is a function that forwards an operation into a dispatching export.
  • Stored analyses. Each file's analysis is stored in the index under a fingerprint of the analyzers and grammars, so a sync re-analyzes only the files that changed.
  • Other projects. Projects without both languages are unaffected.

The hop wording lives in graph/synthesized-hop.ts, and src/index.ts reaches the new sync gate through engine().

What I ran

Linux x64 (WSL2), Node 24.19.0, main at f72c9d2 against this branch.

  • Full suite, native kernel staged (npm run build:kernel, CODEGRAPH_KERNEL_EXPECT=1): 11,283 passed, 2 expected failures, 58 skipped (platform-gated), none failed.
  • The bridge's tests on the wasm path (CODEGRAPH_KERNEL=0), together with every Lua and Rust resolution test: 53 files, 461 passed.
  • __tests__/kernel-lua-rust-bridge.test.ts (new). It indexes one Lua + Rust project on each path. On the kernel arm it requires both languages to route to the kernel with no file deferred. It requires the same nodes, edges, bridge edges, callers, callees and impact from both. The release gate's kernel-*.test.ts glob picks it up.

Real repositories, each indexed fresh with main and with this branch, on both extraction paths. On every corpus the two paths gave byte-identical graphs, as a SHA-256 over every node and edge row (timestamps and the edge rowid aside), for both builds:

Corpus Files Nodes Edges, main → this Bridge edges
RunRustFromLua 2 34 52 → 60 8 FFI
Kong/atc-router 31 584 1,715 → 1,715 none (see Not done)
A private game monorepo (1,158 Lua, 181 Rust files) 2,535 89,864 276,724 → 276,873 149: 125 operation, 12 FFI, 2 transport, 10 getter member calls
Ruff (Rust control) 5,082 83,558 323,022 → 323,022 none; every row identical
  • The monorepo, row by row. Node rows match main's exactly, and an edge-by-edge diff finds only the bridge's additions.
  • Separate operations stay apart. Three Lua entry points each reach exactly the handlers their literal operations select, with no call path to the others. For example, a live-battle loopback's send reaches command, concede and cancel, never derive, history or create.
  • Explore and callers. One codegraph_explore query returns each flow, and the native export's callers list its Lua call site as transport.
  • Re-index and sync stability. Re-indexing any corpus in place keeps its counts, and the monorepo's edges match a fresh index's row for row. Edit-and-restore syncs leave every count as it was.

Cost, timed on the monorepo with the native kernel on an otherwise idle machine, alternating the builds. Each figure is the median of six fresh indexes or three sync rounds:

main this
Fresh index 14.5 s 21.8 s
CLI sync in a new process: a Lua edit / its restore 2.3 / 2.0 s 3.4 / 3.1 s
Same, the native library's Rust crate root 1.8 / 1.6 s 3.0 / 2.8 s
Same, a Python tool 2.3 / 2.0 s 3.4 / 3.1 s

The bridge pass is 7.8 s of that fresh index and the longest synthesis pass. It analyzes all 181 Rust files but only 280 of the 1,158 Lua files. Analyzing every Lua file gives the identical graph at 25.8 s. If the remaining cost matters, the next steps would be following the Rust side from the exports in the same way, or splitting the analyses across the resolver workers. Ruff, with no Lua, runs no bridge pass.

Not done

  • Agent A/B. The coverage playbook asks for one, and it hasn't been run. I'd rather say so than claim a result.
  • A large positive corpus. I haven't found one; Ruff is only a control.
  • The OpenResty binding layout isn't followed yet. In it, a loader returns ffi.load(path) on one branch, and the handle is exported to other modules as cdefs.clib, as in Kong's atc-router above. That needs two proofs the analysis doesn't make yet, so those calls stay unlinked rather than guessed.
  • Other platforms. macOS and Windows weren't run. The change touches no paths, sockets, watchers or processes.

Existing indexes need a re-index after upgrading, and so does a project after a Cargo manifest edit. No version, dependency or kernel code changed.

🤖 Generated with Claude Code

…peration handler it names

A LuaJIT program calls its Rust library through the FFI by symbol, often
through one shared export that runs a different handler for each operation
name. Static extraction stopped at both: the Lua call named a C symbol no Lua
file defines, and the export's match reached every handler.

A synthesis pass, gated on projects holding both languages, links a call
through a proven FFI receiver to the one Rust C-ABI export with its symbol,
and a literal operation forwarded through Lua and Rust wrappers to the handler
its match arm selects. The call into a dispatching export stays as a transport
reference that every walk treats as a boundary: the export's callers and
impact list its Lua call sites, and code behind it doesn't fan back out to
every Lua caller. A call inside a closure counts for the function around it,
as the extractor's own calls do.

The pass analyzes only the Lua files that can matter: those holding the
literal "ffi", then each round those naming a module a carrier was found in.
Each file's analysis is stored in the index under a fingerprint of the
analyzers and grammars, so a sync re-analyzes only what changed, and a sync
that brings a project its first Lua and first Rust files refreshes the bridge.

Lua and Rust go to the native kernel by default. The bridge parses with the
wasm grammars and builds the same graph from either extractor, which
kernel-lua-rust-bridge.test.ts checks. EXTRACTION_VERSION is 29.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Ahaeflig

Copy link
Copy Markdown
Author

Closing this in favour of three smaller PRs that can be reviewed one at a time:

Before splitting it, I reviewed this PR adversarially and fixed what turned up. Each problem has a test that fails without its fix. The larger ones:

  • Proofs that didn't hold. A colon call surrendered its own module, a nil callee in an early round withdrew a library, and a package.path write invalidated every module.
  • Operations in the wrong arm. An Ok(op) binding was kept wrapped, a multi-statement arm wasn't seen as dispatch, and unwrap_or and its kin lost the operation.
  • Cost. A sync re-analyzed every Lua and Rust file. Each file's analysis is now stored in the index, so only an edited file is analyzed again. Files are picked by the require literals that can name a module, not by substring.

Each new PR's description and docs/design/lua-rust-bridge.md list what was run.

🤖 Generated with Claude Code

@Ahaeflig Ahaeflig closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant