Skip to content

Finish multicore adaptive JIT: real worker execution, deadlock fix, correct JIT ABI, safe recompile - #11

Merged
MonkeyIsNull merged 1 commit into
mainfrom
vfm-multicore-adaptive-jit
Oct 8, 2026
Merged

MonkeyIsNull merged 1 commit into
mainfrom
vfm-multicore-adaptive-jit

Conversation

@MonkeyIsNull

Copy link
Copy Markdown
Owner

Finishes the previously-experimental multicore adaptive JIT and, in doing so, fixes several real correctness/safety bugs it depended on. Produced via a plan -> 5-lens committee (concurrency, codegen/ABI, memory, API/design, tests) -> synthesize -> implement -> review -> fix cycle.

What was broken

  • Worker never executed JIT — an int result = 1; placeholder (vfm.c). Compiled code was never run.
  • vfm_multicore_execute_batch deadlocked — per-batch create-then-pthread_join on workers that only exit on destroy.
  • Adaptive emitters used a divergent/wrong ABI and emitted silent nops for unimplemented opcodes.
  • Use-after-free in adaptive_jit_recompile — freed the live page before recompiling.
  • The single-core ARM64 JIT had never actually been executed by any test (jit_test skipped execution; test_vfm only checked interpreter fallback), and its execution was in fact broken: X0 treated as a VM-state struct while callers pass X0=packet; PUSH immediate truncated to 16 bits; fixed-offset stack addressing that only worked for one PUSH.

What this does

  • Canonical JIT ABI on both arches: uint64_t fn(const uint8_t *packet, uint16_t len, uint64_t *stack64, vfm_u128_t *stack128) — no dependence on struct layout, so one read-execute page is valid across threads. ARM64 emitters rewritten (index-based stack, full 64-bit immediates); each emitter oracle-gated against the interpreter, demoting unproven ones (trusted set: PUSH/ADD/RET). vfm_execute JIT/interpreter return senses unified.
  • Worker executes for real: each core gets its own interpreter-only VM that borrows the shared compiled page (single owner, freed once); worker runs vfm_execute per packet (JIT when present, per-core interpreter fallback otherwise).
  • Deadlock fixed: persistent worker pool + generation/completed_count barrier under a pool mutex (replaces per-batch create/join).
  • Safe recompile: compile-into-temp + swap-on-success; never frees the live page on failure; cores stay valid. Multicore JITs only the oracle-validated {PUSH, ADD, RET} set on both arches, declining the rest to the interpreter.

Blocking bug fixed during review

A use-after-free in the single-core JIT cache teardown (vfm_destroy + vfm_load_program reload): it nulled jit_cache_entry before testing it, so vfm_jit_free munmap'd a cache-owned page, leaving a dangling cache pointer a later same-hash lookup would execute. The new ABI made single-core JIT execution reachable for the first time, exposing it. Now captures cache ownership before releasing and frees only non-cache-owned pages; also munmaps a compiled page when vfm_jit_cache_store fails. Permanent ASan regression added.

Tests

  • jit_test now executes the single-core JIT and asserts JIT == interpreter oracle (PUSH/ADD/RET, >0xFFFF immediates, LD16 decline, cache-teardown UAF).
  • New multicore suite: 126 correctness cases (6 programs x {1,2,8} cores x batch sizes incl. remainders/empty) vs the oracle, recompile swap, declined-recompile UAF regression, per-instance counters, 500-batch no-deadlock (timeout-guarded), program lifetime. ASan + TSan targets.

Known follow-up (out of scope)

The single-core x86-64 LD8 emitter is itself wrong (MOV r8 + a 64-bit self-move leaves high bits garbage; needs MOVZX) — the x86 analogue of the ARM64 bug fixed in #10. The multicore path already declines it to the interpreter; single-core x86 JIT consumers remain affected. Tracked separately.

Verification

  • macOS/arm64: 16/16 + JIT-vs-oracle execution + 126 multicore cases + recompile + UAF regression + 500-batch no-deadlock + ASan clean + TSan clean.
  • Docker linux/amd64: 16/16 + multicore matches the oracle (incl. proto_tcp) + recompile + no-deadlock + ASan clean. TSan is not runnable under QEMU, so it was run natively on macOS.

…orrect JIT ABI, safe recompile

The multicore adaptive JIT was unfinished and unsafe: the worker never executed JIT (an int result=1 placeholder), vfm_multicore_execute_batch deadlocked (per-batch create-then-join on workers that never terminate), the adaptive emitters used a divergent/wrong ABI and emitted silent nops for unimplemented opcodes, and adaptive_jit_recompile freed the live JIT page before recompiling (use-after-free if a recompile declined). Investigation also showed the single-core ARM64 JIT had never actually been executed by any test (jit_test skipped execution and only checked the compile pointer; test_vfm only asserted interpreter fallback), and its execution was in fact broken: the compiled code treated X0 as a VM-state struct while callers pass X0=packet, PUSH truncated immediates to 16 bits, and stack addressing used fixed offsets that only worked for a single PUSH.

Single-core first, oracle-validated: define one canonical JIT ABI on both arches -- uint64_t fn(const uint8_t *packet, uint16_t len, uint64_t *stack64, vfm_u128_t *stack128) -- removing all dependence on struct layout so a single read-execute page is valid across threads. Rewrite the ARM64 emitters to index-based stack addressing and full 64-bit immediates (movz/movk), retarget the packet-reading emitters to X0, and gate each emitter against the interpreter oracle, demoting any that cannot be proven (trusted ARM64 set: PUSH/ADD/RET). Unify vfm_execute so the JIT and interpreter return senses agree.

Multicore: back each core with its own interpreter-only vfm_state_t that borrows the shared compiled page (single owner, freed exactly once); the worker runs vfm_execute per packet (JIT when the page is present, per-core interpreter fallback otherwise). Replace the deadlocking per-batch create/join with a persistent worker pool plus a generation/completed_count barrier under a pool mutex. Recompile uses compile-into-temp + swap-on-success, never freeing the live page on failure so cores stay valid. The multicore path JITs only the oracle-validated {PUSH, ADD, RET} set on both arches, declining everything else to the interpreter.

Blocking use-after-free fixed: the single-core JIT cache teardown (vfm_destroy and the vfm_load_program reload path) nulled jit_cache_entry before testing it, so the free guard always held and vfm_jit_free munmap'd a cache-owned page, leaving the cache bucket with a dangling pointer that a later same-hash lookup would execute. The new ABI made single-core JIT execution reachable for the first time, exposing this. Capture cache ownership before releasing and free only non-cache-owned pages; also munmap a freshly compiled page when vfm_jit_cache_store fails (leak). Added a permanent ASan regression (jit_test cache-teardown case + a test-jit-asan target).

Tests: jit_test now EXECUTES the single-core JIT and asserts JIT == interpreter oracle (PUSH/ADD/RET, >0xFFFF immediates, LD16 decline, cache-teardown UAF). New multicore suite: 126 correctness cases (6 programs x {1,2,8} cores x batch sizes incl. remainders/empty) vs the oracle, recompile swap, declined-recompile UAF regression, per-instance counters, a 500-batch no-deadlock test (timeout-guarded so a hang fails), and program lifetime. ASan and TSan build targets added.

Known follow-up, NOT in scope here: the single-core x86-64 LD8 emitter is itself wrong (MOV r8 + a 64-bit self-move leaves the high bits garbage; it needs MOVZX), the x86 analogue of the ARM64 bug fixed in PR #10. The multicore path already declines it to the interpreter, but single-core x86 JIT consumers remain affected. Tracked separately.

Verified on macOS/arm64 (16/16 + JIT-vs-oracle execution + 126 multicore cases + recompile + UAF regression + 500-batch no-deadlock + ASan clean + TSan clean) and Docker linux/amd64 (16/16 + multicore matches the oracle incl. proto_tcp + recompile + no-deadlock + ASan clean; TSan is not runnable under QEMU so it was run natively on macOS).
@MonkeyIsNull
MonkeyIsNull merged commit 34ec4c7 into main Oct 8, 2026
2 checks passed
@MonkeyIsNull
MonkeyIsNull deleted the vfm-multicore-adaptive-jit branch October 8, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant