Repository navigation
fix(jni): attach native worker threads to the JVM as daemons - #11085
Merged
robert3005 merged 2 commits intoNov 2, 2026
Merged
Conversation
Spark writes through a Java stream, so native workers run Java upcalls and attach themselves permanently. They never exit, and as non-daemon threads they kept the JVM alive after main returned. Attach them as daemons first, and detach them when the thread exits, since the jni crate only detaches attachments it made itself. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
An upcall from another thread-local destructor can run after the daemon attachment slot was destroyed. Storing the guard then panics inside a destructor and aborts the process, so leave that attachment to the jni crate. Close the writers in NativeWorkerAttachmentTest before asserting that they all started, and give VortexJvmExitTest's child the parent JVM's --add-opens flags so it can't drift from the Gradle configuration. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
jni-daemon-worker-threads
branch
from
November 2, 2026 13:50
34890d0 to
51c9893
Compare
robert3005
enabled auto-merge (squash)
November 2, 2026 13:54
robert3005
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #11081, which this PR's test needs. Review that one first.
The Spark connector writes through a Java stream (
HadoopWritable), so native worker threads call back into Java and attach themselves to the JVM as non-daemon threads. The workers never exit, so after a Spark write the JVM stays alive whenmainreturns.Changes
with_jvmattaches a thread that isn't attached yet withAttachCurrentThreadAsDaemon, and the jni crate then reuses that attachment. Threads that Java or the crate attached are left alone.Tests
VortexJvmExitTeststarts a child JVM that writes and reads through Spark and returns frommain. The child must exit.NativeWorkerAttachmentTestblocks four writers' first upcalls on four workers. The workers must be daemons, and they must be gone after the pool shrinks to zero.Both fail without the change. The child JVM stays alive, and the worker is attached as a non-daemon thread.
Checked locally:
fmt --checkandclippy --all-targets -D warningsfor vortex-jni,:vortex-jni:test, the five Spark suites andspotlessCheckpass.I used AI tools to help write and review this change.