Repository navigation
[Java] Gandiva Tests are failing due to linking issues #63
Description
Describe the bug, including details regarding any error messages, version, and platform.
At the moment, java-jars CI in Arrow Java is failing due to a Gandiva issue. This CI in green is much needed to determine the accuracy of certain PRs to Java. In order to keep it green, as discussed we are disabling the failing Gandiva tests temporarily. But needs to be re-enabled once the LLVM related issue is resolved.
Component(s)
Java
Activity
jinchengchenghh commentedon Aug 16, 2024
apache/arrow#43639 (comment) This test also needs to be disabled.
lidavidm commentedon Aug 18, 2024
@praveenbingo @pravindra you seem to be the last committers to have worked on the Java part of Gandiva; do you have any idea about this issue?
[-][Java] Gandiva Tests are failure related to Gandiva [/-][+][Java] Gandiva Tests are failing due to linking issues[/+]vibhatha commentedon Sep 6, 2024
Most of the failing tests are due to the
Could not create LLJIT instanceissue [1]. Also, the ProjectTest is mainly failing once. So it would be better to disable the overall test suite and re-enable it once this has been properly fixed.[1] https://github.com/apache/arrow/actions/runs/10731577229/job/29761887531?pr=43978#step:7:20331
GH-43576: [Java] Gandiva Tests are failing due to linking issues (#43978
lidavidm commentedon Sep 10, 2024
@laurentgo do you have any colleagues who might know what's up with Gandiva?
GH-43576: [Java] Gandiva Tests are failing due to linking issues (apa…
raulcd commentedon Oct 9, 2024
I've moved this to 19.0.0, let me know if this should be a blocker for the release.
lidavidm commentedon Oct 9, 2024
It's possible we're going to ship Gandiva in a known-broken state, so we should just highlight it in the release notes
vibhatha commentedon Oct 10, 2024
+1 for this. I think that makes sense and a sub optimal solution for the problem we have.
lriggs commentedon Nov 2, 2024
Hi, I work with @laurentgo at Dremio and I've started looking into this issue. Let me know if there is a better format for this discussion (email vs GH issue, etc). I wanted to summarize some of the different threads and then talk about how I've worked around the issue here at Dremio.
The test errors seem to all be "Symbols not found: [ llvm_orc_registerEHFrameSectionWrapper ]" which is also tracked here, and a few causes and ideas for fixes are listed here and here for Python.
I also found this issue regarding the symbols in the llvm project.
The poster there believes that requiring the symbols is a bad design. I tend to agree but I'm not sure why it was done in the first place.
The ORC symbol issue seems to stem from apache/arrow#37848. We haven't had this problem at Dremio because we actually reverted that change in our local repo after I found a few llvm expression related test failures. I asked about that here https://lists.apache.org/thread/xzrwx1t37dc88vo5yo5337l16ypqwcrd. I spent considerable time looking into this issue but was never able to track it down.
I'm not sure what to do at this point. One idea is reverting the JIT upgrade while also following up with the llvm community on the symbol issue. @niyue Mentioned following up with them here, not sure if he found anything: apache/arrow#39695 (comment)
vibhatha commentedon Nov 3, 2024
Right. I think we can discuss the technical aspect of the issue here.
Also I started a ML thread to get community support to resolve this issue as this is a bit out of scope for me.
FWIW, I would say it would be fine to get this to a release viable point by using an older version of an upgrade we have done so far. But I solely wouldn't be a judge for that.
Really appreciate your support to look into this matter.
niyue commentedon Nov 5, 2024
@lriggs Regarding the LLVM 17 symbol issue, I recall investigating it previously. It stems from a design change introduced in LLVM 17, and unfortunately, there doesn't appear to be an easy fix at that time. A related issue has been logged here: LLVM Project Issue #74671.
GH-45132: [C++][Gandiva] Update LLVM to 18.1 (#45114)
lriggs commentedon Feb 28, 2025
This has been fixed in #55.
GH-45132: [C++][Gandiva] Update LLVM to 18.1 (#45114)