Skip to content

Harden and oracle-validate the single-core x86-64 JIT - #12

Merged
MonkeyIsNull merged 1 commit into
mainfrom
vfm-x86-jit-oracle-validate
Oct 8, 2026
Merged

MonkeyIsNull merged 1 commit into
mainfrom
vfm-x86-jit-oracle-validate

Conversation

@MonkeyIsNull

Copy link
Copy Markdown
Owner

Follow-up to #11. The x86 jit_test execute-vs-oracle section was skipped, so the single-core x86 emitters had never been run against the interpreter — the exact blind spot that hid the ARM64 bugs. This enables that harness (x86/QEMU; cleanly skipped on arm64) and audits every single-core x86 emitter.

Fixed

  • LD8 — was MOV r8,[rdi+off] (writes only bits 0–7) + a 64-bit self-move, leaving bits 8–63 as register-allocator garbage so compares after LD8 diverged. Now MOVZX r32, byte (full zero-extend). This is the x86 twin of the ARM64 bug Fix JIT silently dropping unsupported opcodes (port/proto filters wrong on Apple Silicon) #10 fixed.
  • LD16 — was a 16-bit MOV + BSWAP r16 (architecturally undefined for a 16-bit operand). Now MOVZX r32, word + ROR r16,8 for ntohs.
  • LD32 — verified (32-bit MOV auto-zero-extends; BSWAP r32 = ntohl). All three loads now bounds-check and return VFM_ERROR_BOUNDS on OOB offsets, matching the interpreter instead of reading adjacent memory.
  • Register allocator — aliased every allocation past the 8th onto R8 (silent wrong code for >8 cumulative pushes); now liveness-based, declines on exhaustion.
  • Prologue/epilogue — now preserve callee-saved R12–R15 (was an ABI violation at stack depth ≥ 5).

Demoted to interpreter fallback (decline → NULL) instead of emitting wrong code

  • JEQ/JNE/JGT/JLT/JMP — target was computed as offset * 16 (an arbitrary address), with signed jumps for the interpreter's unsigned compares. Correct branching needs a two-pass pc→offset fixup not implemented here.
  • DIV — DIV r64 raises #DE/SIGFPE on a packet-driven zero divisor where the interpreter returns DIVISION_BY_ZERO.
  • LD128/EQ128 — stack-slot order + hard-coded displacements diverged.
  • Validated correct: PUSH/ADD/SUB/MUL/AND/OR/XOR/SHL/SHR/NOT/NEG/DUP/SWAP/POP/RET.

Net: the x86 JIT now runs its proven-safe subset (loads + arithmetic + stack ops) and falls back to the interpreter for the rest — same posture as ARM64. The single-core ARM64 compiler, the multicore path, and the interpreter are untouched.

Tests

test_jit_declines_unsupported_opcode's x86 "supported" sample changed to branch-free {LD16,PUSH,ADD,RET} (the old sample only compiled non-NULL because of the broken branch emitter). New x86 execute-vs-oracle suite in jit_test.

Verification

  • Docker linux/amd64 (gcc, QEMU executes x86): 16/16; x86 JIT == interpreter oracle across loads/arith/compare incl. proto==6 LD8+XOR and an OOB load; ASan clean; LD8 before (diverges) / after (matches 6).
  • macOS/arm64: 16/16; x86 harness cleanly skipped; x86 emitters still compile; no regression to ARM64 execution / multicore / cache-UAF tests.

jit_test's x86 execute-vs-oracle section was skipped, so the single-core x86 emitters had never been run against the interpreter -- the same blind spot that hid the ARM64 bugs until PR #11. Enabled an x86 execute-against-oracle harness (runs under x86/QEMU, cleanly skipped on arm64 hosts) and audited every single-core x86 emitter.

Fixed: LD8 emitted MOV r8,[rdi+off] (writes only bits 0-7) plus a 64-bit self-move, leaving bits 8-63 as register-allocator garbage so compares after LD8 diverged -- now MOVZX r32,byte (zero-extends the full 64-bit reg). LD16 used a 16-bit MOV plus BSWAP r16 (architecturally undefined for a 16-bit operand) -- now MOVZX r32,word plus ROR r16,8 for ntohs. LD32 verified (32-bit MOV auto-zero-extends; BSWAP r32 = ntohl). All three loads now emit a packet-bounds check returning VFM_ERROR_BOUNDS to match the interpreter on out-of-bounds offsets instead of reading adjacent memory.

Also fixed two latent silent-wrong-code bugs found in the audit: the register allocator (a monotonic counter) aliased every allocation past the 8th onto R8 for programs with >8 cumulative pushes -- now liveness-based and declines on exhaustion; and the prologue/epilogue did not preserve callee-saved R12-R15 (an ABI violation at stack depth >= 5) -- now preserved.

Demoted to interpreter fallback (return NULL via a shared decline path: free + munmap) rather than emit wrong code: DIV (DIV r64 raises #DE/SIGFPE on a packet-driven zero divisor where the interpreter returns DIVISION_BY_ZERO); JEQ/JNE/JGT/JLT/JMP (the target was computed as offset*16 -- an arbitrary address -- with signed jumps for the interpreter's unsigned compares; correct branching needs a two-pass pc->offset fixup not implemented here); LD128/EQ128 (stack-slot order and hard-coded relative displacements diverged). PUSH/ADD/SUB/MUL/AND/OR/XOR/SHL/SHR/NOT/NEG/DUP/SWAP/POP/RET validated correct against the oracle.

Updated test_jit_declines_unsupported_opcode's x86 'supported' sample to branch-free opcodes {LD16,PUSH,ADD,RET}: the old sample only compiled non-NULL because the broken branch emitter produced a non-NULL page of wrong code (the blind spot itself). The JGE unsupported->NULL assertion is unchanged. Single-core ARM64 compiler, the multicore path, and the interpreter are untouched.

Verified on Docker linux/amd64 (gcc, QEMU executes x86): 16/16, the x86 JIT==interpreter oracle across loads/arith/compare incl. proto==6 LD8+XOR and an OOB load, ASan clean, and an LD8 before(diverges: returns a dirtied high-bits value)/after(matches: 6) check. macOS/arm64: 16/16, the x86 harness cleanly skipped, the x86 emitters still compile, no regression to ARM64 execution / multicore / cache-UAF tests.
@MonkeyIsNull
MonkeyIsNull merged commit 51713f7 into main Oct 8, 2026
2 checks passed
@MonkeyIsNull
MonkeyIsNull deleted the vfm-x86-jit-oracle-validate branch October 8, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant