[Java][Gandiva] Synchronize some methods on Projector.

Describe the enhancement requested

Our product, which uses Gandiva, was encountering errors from LLVM during intense query processing stress tests. I traced the issue back to the JIT upgrade (apache/arrow#39098), but it seems it only started occurring then because the new JIT engine requires unique symbol names unlike the old engine.

The error from llvm is: Duplicate definition of symbol 'expr_0_0'.

After updating the llvm version to 18 (apache/arrow#45114) you can see this error occasionally when running the Gandiva ProjectorTest.

Failed to make LLVM module due to Failed to add IR module to LLJIT: Duplicate definition of symbol 'expr_0_0'

	at org.apache.arrow.gandiva.evaluator.JniWrapper.buildProjector(Native Method)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:206)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:70)
	at org.apache.arrow.gandiva.evaluator.ProjectorTest.lambda$testMakeProjectorParallel$1(ProjectorTest.java:138)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
[WARNING] Tests run: 44, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.656 s -- in org.apache.arrow.gandiva.evaluator.ProjectorTest

Adding synchronization the the make and evaluate methods in gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java will fix the issue.

diff --git a/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java b/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
index 951cebc88e..a67d20db6b 100644
--- a/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
+++ b/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
@@ -187,7 +187,7 @@ public class Projector {
    * @param configurationId Custom configuration created through config builder.
    * @return A native evaluator object that can be used to invoke these projections on a RecordBatch
    */
-  public static Projector make(
+  public static synchronized Projector make(
       Schema schema,
       List<ExpressionTree> exprs,
       SelectionVectorType selectionVectorType,
@@ -313,7 +313,7 @@ public class Projector {
         outColumns);
   }
 
-  private void evaluate(
+  private synchronized void evaluate(
       int numRows,
       List<ArrowBuf> buffers,
       List<ArrowBuffer> buffersLayout,

apacheGH-601 Synchronie some methods on the Projector.

I believe there's efforts to revert back to the old JIT due to various problems with the new one

Ah I see it looks like we upgraded LLVM instead apache/arrow#45114

Upgrading llvm fixed the symbol link errors, but after running the disabled tests I found this new issue.

apacheGH-601 Synchronie some methods on the Projector.

apacheGH-601: [Gandiva] Synchronize some methods on the Projector (ap…

DX-125697: Gandiva expression-compilation lock contention limits sing…

Navigation Menu

Repository navigation

[Java][Gandiva] Synchronize some methods on Projector. #601

Describe the enhancement requested

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Java][Gandiva] Synchronize some methods on Projector. #601

Description

Describe the enhancement requested

Activity

lriggs commented on Feb 12, 2025

lidavidm commented on Feb 12, 2025

lidavidm commented on Feb 12, 2025

lriggs commented on Feb 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions