Repository navigation
Native and JVM builds: switch off the JEP 498 sun.misc.Unsafe warning at its source (Windows data corruption) - #94
Merged
Merged
Conversation
… at its source The native image prints jvector's sun.misc.Unsafe warning (the first reopen of a vector index) to the stderr handle it cached at start-up; on Windows a database file opened after the host's dup2 onto fd 2 can own that handle value and got the four lines (release run 40291551544, a .lsmvecidx file). The image keeps sun.misc.Unsafe's option as a build-time constant, so JavaArgs in the glue jar's native-image.properties sets -Dsun.misc.unsafe.memory.access=allow for the builder. The JVM build gets --sun-misc-unsafe-memory-access=allow as a default (a caller's value wins), next to the System.initialErr redirect of #88. test_log_redirect.py now checks the warning is on no stream and in no file on both builds, and keeps the initialErr redirect covered with the option set back to warn. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The native
arcadedbwheel (GraalVM CE native image) can write the JDK's JEP 498 warning into a database file on Windows. The release run 40291551544 (tag 26.11.1.dev1) failed the parity job "windows-x64, Python 3.11" withtests/test_log_redirect.py::test_redirect_with_the_console_off_sends_nothing_to_the_redirect_target:db0\Doc_0_485397029700.4.262144.v0.lsmvecidxstarted withA terminally deprecated method in sun.misc.Unsafe has been called. The Python 3.12 and 3.13 Windows legs passed, because it is the handle-reuse race of #78 and #88. This PR switches the warning off at its source in both packages, so no stream has it to write.Where the warning comes from in the native image
jvector's
SimpleMappedReader$Suppliercallssun.misc.Unsafe.invokeCleanerwhen it unmaps a graph file, on the first reopen of a vector index. In JDK 25,sun.misc.Unsafe.beforeMemoryAccess()then prints the four-line warning withVM.initialErr(), which isSystem.initialErr: in the image, the stream over the stderr handle that the runtime cached at start-up. #78 moved the log andSystem.out/System.erraway from that handle, and #88 did the same forSystem.initialErrin the JVM build, but the native image has noSystem.setErrhook for it, and nothing in the glue code redirects it. After the host'sdup2onto fd 2 closes the cached handle, a database file opened later can get the same handle value, and the warning lands in it. On Linux the same text goes to the host's redirect target (checked on the dev0 native wheel:ARCADEDB_LOG_CONSOLE=0, fd 2 redirected, four lines in the target).The option is a build-time constant in the image:
sun.misc.Unsafe.MEMORY_ACCESS_OPTIONis astatic finalthat is read fromVM.getSavedProperty("sun.misc.unsafe.memory.access")when the class is initialised, and the image builder folds it into the image.ARCADEDB_NATIVE_ISOLATE_ARGS=-Dsun.misc.unsafe.memory.access=allow(and=deny) at run time did nothing on the existing wheel.The fix
JavaArgs = -Dsun.misc.unsafe.memory.access=allowin the glue jar'snative-image.properties.JavaArgsgoes to the builder's JVM, so every way of building the glue jar gets it. I tested four ways on a minimal image that callsUnsafe.invokeCleaner(GraalVM CE 25.4.4.1.1,P1=1 NEED_GB=13 JOB_MEM_GB=14 quiet_build.sh): without the option the program prints the four lines;-J-Dsun.misc.unsafe.memory.access=allow,-J--sun-misc-unsafe-memory-access=allowandJavaArgsin a jar'snative-image.propertiesprint nothing; a-Dsun.misc.unsafe.memory.access=allowat run time on the first image prints the four lines.--sun-misc-unsafe-memory-access=allowis a default of_build_jvm_args, next to--enable-native-access; a value of the caller's own injvm_argsorARCADEDB_JVM_ARGSwins. This is belt and braces next to JVM build: Java System.out/System.err go to a file the library owns when the console is off (Windows data corruption) #88'sSystem.initialErrredirect, which stays for every other text of that kind. The option reaches HotSpot through JPype (checked: with it the four lines are gone fromarcadedb-stdio.log).docs/modules/ROOT/pages/how-to/logs.adocsays that neither package lets the warning appear, and that the native value is fixed in the image.Tests
tests/test_log_redirect.py: on both builds the warning must be in no database file, not in the redirect target, not on the console and not inarcadedb-stdio.log(the native build was exempt from the warning assertions before). The JVM build also runs the console-off redirect once more withARCADEDB_JVM_ARGS=--sun-misc-unsafe-memory-access=warn, so JVM build: Java System.out/System.err go to a file the library owns when the console is off (Windows data corruption) #88's redirect ofSystem.initialErrstays tested: the warning must then be inarcadedb-stdio.logand nowhere else.tests/test_jvm_args.py: the default, and that a caller's value wins.ARCADEDB_REDIRECT_REPEATS=2) fails on the console-off redirect, console-on and console-off tests, with the four lines in the redirect target and on the console. On the JVM build (Linux, 35 tests intest_log_redirect.pyandtest_jvm_args.py) all pass.Windows proof
Temporary workflow on the branch
tmp-native-win-loop(no PR, deleted after the run), run 40301549781: it builds the Windows native image and wheel from this branch (GraalVM CE 25.4.4.1.1, MSVC, engine 26.10.2), installs the wheel withuv run --withand loops. Each iteration starts a new pytest process (default fd capture) fortest_redirecting_fd_1_and_2_leaves_the_database_files_clean,test_redirect_with_the_console_off_sends_nothing_to_the_redirect_targetandtest_console_log_can_be_turned_offon Windows, withARCADEDB_REDIRECT_REPEATSrepeats of the create, build, close, reopen and query cycle per child.With the fix: 280 iterations (840 test runs), 0 failures.
Control, the Windows wheel of release run 40291551544 (tag 26.11.1.dev1, no fix) under the same tests: 40 iterations at 10 repeats per child (120 test runs), 40 failures.
CI of this PR: Test native run 40301477031: success; Test JVM run 40301476995: success.
🤖 Generated with Claude Code