Repository navigation
GH-601: [Gandiva] Synchronize some methods on the Projector - #602
Merged
Merged
Conversation
kou
changed the title
Member
|
@lriggs since you've been working on Gandiva lately, would you be able to help take a look at the changes here? |
kou
changed the title
kou
changed the title
Member
|
We should rebase on main after we merge GH-595. |
Member
Ok I'm clearly not paying attention...you are lriggs 😬 As Kou states, let's rebase this after the other PR. |
Member
|
Could you rebase on main? |
lriggs
force-pushed
the
projectorSynch
branch
from
February 20, 2025 18:06
28cee4e to
9937099
Compare
This comment has been minimized.
This comment has been minimized.
lriggs
force-pushed
the
projectorSynch
branch
from
February 20, 2025 18:13
9937099 to
f852ce6
Compare
Contributor
Author
|
I rebased but don't seem to have permissions to add the bug-fix label. |
lidavidm
approved these changes
Feb 20, 2025
lriggs
added a commit
to dremio/arrow-java
that referenced
this pull request
Oct 8, 2026
…le-large-node parallelism (#33) * apacheGH-601: Stripe Projector/Filter make() locking by cache key instead of a blanket process-wide lock Projector.make() and Filter.make() are declared `static synchronized`, so every LLVM module build in a process serializes behind one class-level monitor -- regardless of whether the concurrent calls target the same expression or completely unrelated ones. On a single JVM packed with many threads (e.g. a large single-node executor), this collapses what should be independent compilations onto one lock, which JFR shows as the dominant source of monitor contention under concurrent query load. The blanket lock was added in apacheGH-601/apache#602 to guard against a genuine TOCTOU race in Gandiva's native object cache: two threads compiling the identical (schema, expressions, selection vector mode, configuration) tuple concurrently can both decide the entry is missing and independently define the same generated LLVM symbol, causing LLVM ORC's duplicate-symbol detection to fire. The real fix for that race belongs in the native Gandiva cache-read path (a separate, non-arrow-java change); until it lands, some form of serialization is still required for genuinely identical concurrent builds. Replace the class-level lock with a small striped lock keyed by a hash of the same tuple that identifies a native cache entry (schema bytes, expression/condition bytes, selection vector mode, configuration id). Concurrent make() calls that hash to the same stripe still serialize -- preserving the invariant that avoids the duplicate-symbol crash -- while unrelated concurrent compilations, which make up the large majority of concurrent traffic, proceed in parallel instead of queuing behind one process-wide lock. Verified against ProjectorTest#testMakeProjectorParallel (the existing apacheGH-601 concurrency repro) and the full Projector/Filter test suite with the real Gandiva JNI library, no regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove lock stripe * remove * spotless * add benchmark for synchronization changes * fix error * fix error --------- Co-authored-by: Chris Pride <cpride@cpride.net> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Multiple threads can attempt to create the same llvm expression in Gandiva. This isn't allowed with the new JIT compiler, so synchronizing will prevent this scenario.
What changes are included in this PR?
Synchronize some methods to avoid adding duplicate llvm expressions.
Are these changes tested?
Yes, through unit tests in Gandiva.
Are there any user-facing changes?
No.
Closes GH-601