Conversation
Bring up YJIT on x86_64-w64-mingw-ucrt, previously unsupported (rb_jit_reserve_addr_space returned NULL on _WIN32). Changes: - configure.ac: allow JIT on x86_64-*mingw*; link Rust std deps (bcrypt/ntdll/userenv/synchronization). - jit.c: VirtualAlloc/VirtualProtect/VirtualFree for reserve/mark_writable/mark_executable/mark_unused; GetSystemInfo page size; FlushInstructionCache. - defs/jit.mk: on mingw, binutils ld -r cannot partial-link the Rust staticlib (PE TLS DataDirectory). Instead build a symbol-localized copy of the archive (drop .dwo members, objcopy --localize-symbols on just the symbols that collide with Ruby's missing/*.o) and link it directly. - yjit backend (x86_64): MS x64 ABI. C_ARG_OPNDS = RCX,RDX,R8,R9 with stack args + 32-byte shadow space for >4 args; TEMP_REGS = RCX,RDX,R8,R9,R10; alloc regs = RAX,RSI,RDI (RSI/RDI callee-saved, preserved once per frame in FrameSetup); Win64 caller-save set. - yjit utils.rs: c_callable! uses extern "C" (MS x64) on Windows instead of extern "sysv64" so runtime callbacks match the generated calls. - yjit cruby_bindings: ID/st_data_t are usize (uintptr_t), not c_ulong, which is 32-bit on LLP64. - yjit fd handling (options/log/disasm): RawFd -> RawFileRef (RawHandle on Windows). - assorted usize/u64/i32/i64 cast fixes for LLP64. Status: scalar arithmetic, loops, method calls, deep recursion (fib), strings, arrays, procs, and blocks all run correctly and pass. Benchmarks vs interpreter: fib(31) 9.8x, method/ivar dispatch 2.4x, predicate/branch 1.9x. Known bug: an accumulator built up over many JIT'd block iterations that crosses 2^32 triggers a spurious 'Unnormalized Fixnum' check (the computed value is numerically correct). RubyVM::YJIT.runtime_stats also asserts. Both are isolated follow-ups; core codegen is correct.
On Windows x64 (LLP64) long is 32-bit, so Ruby's Fixnum range is LONG_MAX/2 (+/-2^30) even though VALUE is 64-bit. YJIT's integer fast paths do 64-bit arithmetic and only detect overflow at 2^63, so a result between 2^31 and 2^63 stayed Fixnum-tagged when it should promote to Bignum -> [BUG] Unnormalized Fixnum. Add guard_fixnum_in_long_range(): after opt_plus/opt_minus/opt_mult/opt_succ compute the tagged result, side-exit to the interpreter (which promotes to Bignum) when it leaves the signed-32-bit range. No-op on LP64 where the 64-bit overflow check already matches the boundary. Costs 2 cmp+jcc per op; benchmarks unchanged (fib 9.8x etc). Also add VALUE::num_from_usize() (rb_ull2inum-backed) and route RubyVM::YJIT.runtime_stats counter packing through it, so counters exceeding the LLP64 Fixnum range become Bignums instead of asserting in fixnum_from_usize. Verified byte-identical to the interpreter across boundary-crossing add/sub/mult, power-via-mult, succ, factorial/Bignum, and the previously-failing block accumulators >2^32. runtime_stats now returns a Hash.
RARRAY heap length is a C long (32-bit on Windows LLP64), but get_array_len read it at c_long::BITS and csel'd it against the embedded length derived from the 64-bit flags word -> match_num_bits panic (operands of incompatible sizes 64 vs 32), hit by the Array#empty?/length/size cfunc specializations and array aref/splat when compiled at runtime. Read a full VALUE-width word at the heap-len offset (the 4 bytes above len belong to the adjacent aux field) and mask to the low bits; array lengths are non-negative so this zero-extends to a 64-bit operand matching the embedded length and downstream Fixnum tagging. The masking clobbers flags, so it is computed before the RARRAY_EMBED_FLAG test whose result the csel consumes. Verified byte-identical to the interpreter (--yjit-call-threshold=2): length/size/empty on embedded and heap arrays, aref+length, runtime RubyVM::YJIT.enable. bootstraptest/test_yjit.rb 369/369 and the 30k ifelse/methods stress tests pass.
String#bytesize and String#getbyte read RSTRING_LEN, a C long that is 32-bit on Windows (LLP64), then used it in 64-bit contexts (Fixnum tagging in bytesize; cmp against a 64-bit index in getbyte). Mixing a 32-bit field with 64-bit operands tripped the backend operand-size assertion and crashed with [BUG] YJIT panicked (asm/x86_64/mod.rs) - hit by the ruby-bench ruby-xor benchmark. Add load_long_len_field() which reads a full VALUE-width word at the length offset and masks to the low bits (lengths are non-negative, so this zero-extends), matching the approach used for array length. Verified byte-identical to the interpreter for bytesize/getbyte on embedded and heap strings including negative/out-of-bounds indices; ruby-xor now runs (10.6x). bootstraptest 369/369.
The invokebuiltin / leaf-builtin fast paths bail out when the builtin needs more arguments than C_ARG_OPNDS holds, which is 4 on the MS x64 ABI against 6 on SysV. ccall() already spills arguments past the fourth onto the stack, so the register count is not the real limit: builtins with 3-4 arguments were side-exiting on Windows for no reason, and the exit profile diverged from every other platform. Use the SysV limit of 6 on Windows too, so the same builtins compile everywhere.
rb_jit_mark_executable is handed a page-aligned range that can contain pages rb_jit_mark_writable never committed, or pages rb_jit_mark_unused decommitted during code GC. Linux mprotect tolerates that; VirtualProtect fails the whole call with ERROR_INVALID_ADDRESS if even one page in the range is uncommitted, which showed up as "[BUG] Couldn't make JIT page executable" under code GC and with a small --yjit-exec-mem-size. Commit the range with VirtualAlloc(MEM_COMMIT) first, which is idempotent on already-committed pages, then set the protection.
The Windows port makes YJIT work on x86_64-*mingw*, but JIT_TARGET_OK also gates ZJIT, which has not enabled itself on Windows. Widening JIT_TARGET_OK would implicitly turn on a JIT the platform does not support yet, so derive YJIT_TARGET_OK from it and add mingw only there.
Both are uintptr_t in C, but bindgen run on LP64 records them as c_ulong, which is 32-bit once the checked-in bindings are compiled on LLP64 (Windows). Blocklist the two types and emit them as usize, so a regenerated cruby_bindings.inc.rs stays correct instead of depending on a hand-edit surviving the next bindgen run.
The harness has the child write its marshaled stats to fd 3. Windows spawn rejects fd >= 3 as a redirect key (process.c guard; CreateProcess only wires up fds 0/1/2), which errored out 119 tests with "wrong file descriptor (3)". On Windows, hand the child a temp-file path in YJIT_TEST_STATS_FILE and read the results back from there instead. Every other platform keeps the pipe.
… Windows test_tracing_str_uplus asserts an exact putspecialobject side-exit count under object-allocation tracing. On Windows YJIT compiles putspecialobject where other platforms deopt; the result is identical (the frozen string's allocation source line is correct and tracing invalidation works), only the exact deopt profile differs. Assert :any there, as the suite does for other platform-specific differences. With this, test/ruby/test_yjit.rb on Windows is 139 tests, 375 assertions, 0 failures, 0 errors, 1 skip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WARNING: Very experimental
Experimental YJIT port to Windows x86_64 (
x86_64-w64-mingw-ucrt, MSYS2/UCRT64 toolchain,rustcfrommingw-w64-ucrt-x86_64-rust).Before this,
rb_jit_reserve_addr_spaceinjit.csimply returnedNULLunder_WIN32, so--enable-yjitproduced a build that could not allocate executable memory. With these commitsruby.exe --yjitJIT-compiles and runs correctly.Benchmarks
All numbers are the same binary, YJIT vs. interpreter, best-of steady state, measured in the Windows VM.
ruby/ruby-bench, pure-Ruby CPU benchmarks
Geometric mean 7.3× over the 10 benchmarks.
getivaris memory-bound, so ~1× is the expected result there.Macro benchmarks (the vendored, gem-free subset of ruby-bench)
fannkuchreduxis permutation/GC-bound and is flat under YJIT upstream too.leeand the Ractor benchmarks callRactor.make_shareableand were skipped.The ~25 gem-based macro benchmarks (lobsters, railsbench, rubocop, sequel, …) need
bundle installof large native-gem trees, and Rails additionally needs thesocketandstrscanextensions, which do not build in this 4.1.0dev tree for reasons unrelated to YJIT. Not covered here.miniruby microbenchmarks
fib(31)(recursion)Tests
bootstraptest/test_yjit.rbbootstraptest/test_yjit_30k_ifelse.rb,..._30k_methods.rbtest/ruby/test_yjit.rbtest/ruby/test_yjit.rbstarted at 65 assertions / 120 errors, almost all of them a harness limitation rather than YJIT (see below).What the port required
The x86_64 backend hard-coded the System V AMD64 ABI; Windows uses the Microsoft x64 ABI. Three differences matter for codegen:
C_ARG_OPNDSand theCCalllowering/emit were rewritten.FrameSetup/FrameTeardown, and uses RCX, RDX, R8, R9, R10 as temps — five temps, all caller-saved, matching SysV's arithmetic.c_callable!pins the Rust runtime callbacks toextern "sysv64"on all x86_64. On Windows that fights the native ABI, so the macro usesextern "C"(= MS x64) there and everything speaks one convention.Platform plumbing:
jit.c— executable memory viaVirtualAlloc(MEM_RESERVE)/VirtualAlloc(MEM_COMMIT)/VirtualProtect(PAGE_EXECUTE_READ)/FlushInstructionCache/VirtualFree(MEM_DECOMMIT), page size fromGetSystemInfo. The reservation probes downward from the module address to stay within ±2 GiB of Ruby's C functions so 32-bit relativecall/jmpcan reach them.configure.ac— a separateYJIT_TARGET_OKforx86_64-*mingw*(widening the sharedJIT_TARGET_OKwould implicitly enable ZJIT, which has not enabled itself on Windows), plus the Rust staticlib's system dependencies-lbcrypt -lntdll -luserenv -lsynchronization.defs/jit.mk— binutilsld -rcannot partial-link the Rustlibyjit.aon PE; it chokes on the TLS directory Rust std emits (unable to fill in DataDirectory[9]: _tls_used). Instead the port builds a localized copy of the archive: strip the.dwomembers, thenobjcopy --localize-symbolson only the symbols that actually collide with$(MISSING)(thenmintersection —lgamma_r,tgamma, …). A blanket--keep-global-symbol=rb_*would sever Rust's inter-CGU symbols.LLP64, the second and subtler mismatch
longis 32-bit on Windows even thoughVALUEis 64-bit, soRUBY_FIXNUM_MAXis2^30— a Win64 Fixnum holds 31 bits, not 63. Three separate bugs came out of that, each producing wrong-but-plausible values rather than an obvious crash:opt_plus/opt_minus/opt_mult/opt_succdo 64-bit arithmetic and detect overflow with a 64-bitjo, which only trips near2^63, so results between2^31and2^63stayed Fixnum-tagged and hit[BUG] Unnormalized Fixnum. Fixed with a Windows-only range guard that side-exits to the interpreter for Bignum promotion. No-op on LP64.get_array_lenreads the heap length as a Clong(32-bit here) andcsels it against the 64-bit-derived embedded length, trippingmatch_num_bits. Fixed by reading 64 bits and masking. The maskingandclobbers flags, so it has to be emitted before theRARRAY_EMBED_FLAGtest thecselconsumes — getting that order wrong made heap arrays silently report length 0.String#bytesize/#getbytereadRSTRING_LENthe same way. Surfaced by the ruby-benchruby-xorbenchmark.Also
IDandst_data_tareuintptr_tin C but bindgen recorded them asc_ulong; they are now blocklisted in the generator and emitted asusize, so a regeneratedcruby_bindings.inc.rsstays correct.Verified byte-identical to the interpreter with a differential harness: arithmetic across the Fixnum boundary in both directions, deep recursion, blocks and accumulators past
2^32, factorials, strings, arrays, hashes, procs/lambdas, andRubyVM::YJIT.runtime_stats.Test-harness changes
Two commits touch
test/ruby/test_yjit.rbrather than YJIT:assert_compileshas the child Marshal-dump its stats to fd 3, which Windowsspawnrejects outright (process.crefusesfd >= 3;CreateProcessonly inherits 0/1/2). That errored out 119 tests before any code ran. On Windows the child now writes to a temp file whose path arrives inYJIT_TEST_STATS_FILE; every other platform keeps the pipe.test_tracing_str_uplusasserts an exactputspecialobjectdeopt count. On Windows YJIT compiles that instruction where other platforms side-exit; the result is identical, only the deopt profile differs, so the exit assertion is:anythere.Status
Experimental. Built and tested on one Windows VM with MSYS2/UCRT64; there is no CI for this target, and the MS-ABI paths do not get exercised by a Linux build.