Skip to content

fix(jni): attach native worker threads to the JVM as daemons - #11085

Merged
robert3005 merged 2 commits into
vortex-data:developfrom
gitRasheed:jni-daemon-worker-threads
Nov 2, 2026
Merged

robert3005 merged 2 commits into
vortex-data:developfrom
gitRasheed:jni-daemon-worker-threads

Conversation

@gitRasheed

Copy link
Copy Markdown
Contributor

Summary

Stacked on #11081, which this PR's test needs. Review that one first.

The Spark connector writes through a Java stream (HadoopWritable), so native worker threads call back into Java and attach themselves to the JVM as non-daemon threads. The workers never exit, so after a Spark write the JVM stays alive when main returns.

Changes

  • with_jvm attaches a thread that isn't attached yet with AttachCurrentThreadAsDaemon, and the jni crate then reuses that attachment. Threads that Java or the crate attached are left alone.
  • A thread-local guard detaches these threads when they exit, because the crate only detaches attachments it made. Without it, every retired worker stays registered with the JVM.
  • Windows keeps the old behaviour, because detaching from a thread-local destructor can deadlock on the loader lock there (jni-rs issue 701).

Tests

  • VortexJvmExitTest starts a child JVM that writes and reads through Spark and returns from main. The child must exit.
  • NativeWorkerAttachmentTest blocks four writers' first upcalls on four workers. The workers must be daemons, and they must be gone after the pool shrinks to zero.

Both fail without the change. The child JVM stays alive, and the worker is attached as a non-daemon thread.

Checked locally: fmt --check and clippy --all-targets -D warnings for vortex-jni, :vortex-jni:test, the five Spark suites and spotlessCheck pass.

I used AI tools to help write and review this change.

gitRasheed and others added 2 commits November 2, 2026 13:49
Spark writes through a Java stream, so native workers run Java upcalls
and attach themselves permanently. They never exit, and as non-daemon
threads they kept the JVM alive after main returned. Attach them as
daemons first, and detach them when the thread exits, since the jni
crate only detaches attachments it made itself.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
An upcall from another thread-local destructor can run after the daemon
attachment slot was destroyed. Storing the guard then panics inside a
destructor and aborts the process, so leave that attachment to the jni
crate.

Close the writers in NativeWorkerAttachmentTest before asserting that
they all started, and give VortexJvmExitTest's child the parent JVM's
--add-opens flags so it can't drift from the Gradle configuration.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the jni-daemon-worker-threads branch from 34890d0 to 51c9893 Compare November 2, 2026 13:50
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 2, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 2, 2026 13:54
@robert3005
robert3005 merged commit 172ea59 into vortex-data:develop Nov 2, 2026
101 of 102 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants