Repository navigation
ClassNotFoundException MemoryAddress with jdk-19 #212
Description
I'm trying out libpython-clj 2.018 on JDK 19, and I'm getting the following error, both the first time I try to initialize, and when attempting to require libpython-clj2.require:
java.lang.ClassNotFoundException: jdk.incubator.foreign.MemoryAddress
I saw issue #180, which suggested adding :jvm-opts ["--add-modules" "jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"] to my deps.edn for JVM versions 17 and up. I may tinker around with the version a bit, but it will require rebuilding docker images, so it would be helpful to know if anyone has any experience using libpython-clj 2 with JDK 19.
Thanks
Activity
jjtolton commentedon Sep 1, 2022
Pinging @cnuernber on this one.
cnuernber commentedon Sep 1, 2022
So JDK-18, 19 aren't at this time supported as they both have incompatible changes to the pathway from an integer to a direct buffer.
cnuernber commentedon Sep 1, 2022
Specifically 16, 17, 18, 19 all contain incompatible changes there so I have been avoiding upgrading dtype-next to those versions. At this time only JDK-17 is supported beyond JDK-11.
metasoarous commentedon Sep 2, 2022
Got it 👍 Thanks @cnuernber!
I don't have a strong reason for using any particular JDK version, so happy to stick with JDK 17 for now. It's too bad that we're having to fuss over platform "changes" though. In theory, if someone were motivated, could newer versions be supported?
Thanks!
cnuernber commentedon Sep 2, 2022
Of course. Its just work - scanning the api changes and fixing dtype-next.
metasoarous commentedon Sep 2, 2022
I switched to JDK 17, and am seeing the same behavior as before.
For context, I'm pretty sure libpythonclj is finding the correct python installation. And functions like
py/->pythonworks. But others do not (such aspy/as-pythonandpy/->jvm).I have a pretty minimal reproduction of this in a Docker container at https://github.com/metasoarous/libpython-require-error. Please let me know if you see anything obviously wrong there, or have hints about what I should be looking for.
Thanks again
jjtolton commentedon Sep 2, 2022
@metasoarous I see your issue -- and raise you an issue! metasoarous/libpython-require-error#1
metasoarous commentedon Sep 2, 2022
Thanks @jjtolton; That should be resolved now. Please let me know if it gets you to up the issue stack :-)
jjtolton commentedon Sep 2, 2022
Much appreciated. I'll dive in shortly!
cnuernber commentedon Sep 3, 2022
One thing I want to note is that to my knowledge
:jvm-optsonly works in an alias. A top-level:jvm-optsis silently ignored. So I think the example deps.edn needs an alias with it in it and the dockerfile's cmd needs to be updated to include that alias.As noted in the other issue if you don't see exactly this warning:
WARNING: Using incubator modules: jdk.incubator.foreignThen you have not loaded the module.
jjtolton commentedon Sep 3, 2022
@metasoarous , I reconfigured your
deps.ednper @cnuernber 's instructions.Current file:
{:paths ["src"] :deps {org.clojure/clojure {:mvn/version "1.11.1"} techascent/tech.ml.dataset {:mvn/version "6.093"} ;techascent/tech.ml.dataset {:mvn/version "6.086" ;:exclusions [org.slf4j/slf4j-api]} org.bytedeco/mkl-platform-redist {:mvn/version "2020.1-1.5.3"} clj-python/libpython-clj {:mvn/version "2.018"} cider/cider-nrepl {:mvn/version "0.28.5"} com.taoensso/timbre {:mvn/version "5.2.1"} ch.qos.logback/logback-classic {:mvn/version "1.1.3"}} :aliases {:opts {:jvm-opts ["--add-modules" "jdk.incubator.foreign" "--enable-native-access=ALL-UNNAMED"]}}}I now get a different error, but still not working. Note:
root@802025a3b951:/app# java -version openjdk version "17.0.2" 2022-01-18 OpenJDK Runtime Environment (build 17.0.2+8-86) OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)New error:
jjtolton commentedon Sep 3, 2022
Hopefully this is a more tractable error?
jjtolton commentedon Sep 3, 2022
I think to rule out installation issues, I will try to merge the example Dockerfile with the libpython-clj example Dockerfile to try to rule out installation/version issues.
jjtolton commentedon Sep 3, 2022
@metasoarous not sure if this solves the initial problem or not, but I put in a PR to your repo so that at least you have a minimal working setup: metasoarous/libpython-require-error#2
metasoarous commentedon Sep 3, 2022
@cnuernber D'oh! I did not in fact realize that this is a constraint with
:jvm-opts. Not clear to me why that's not possible but 🤷 Thanks for pointing out.@jjtolton Thank you for digging in! As mentioned in #213, I was able to get baseline functionality working by starting
FROM clojure:openjdk-8-tools-deps, which I think is the same version used in the libpython-clj example Dockerfile. I may be fine continuing to use that, which would at least remove any immediate pressure for me, but it would still be nice to sort out connecting to jdk-17 if possible. (PS I had jdk-11 working at one point, but it stopped building some of the python pieces, presumably because the dockerfiles on dockerhub were updated with some weird environmental artifact 🤷)Thanks again!
jjtolton commentedon Sep 3, 2022
No problem, @metasoarous. Will tinker around later and see if I can a Dockerfile for JDK-17 going. Hopefully this gives you enough for now.
metasoarous commentedon Sep 4, 2022
First off, just wanted to confirm @cnuernber's suspicion that I wasn't using
:jdk-optsproperly. Now that I've done that, at least jdk-17 works. I'm happy to stick with that for a good while.It's up to you whether you want to leave this issue open to track jdk-19 support, in case someone is inspired to work on it. I think it would be great for beginner friendliness to the ecosystem if folks aren't having to worry about JDK version so much. Trying to at least support the LTS versions is probably a pretty tenable path forward if you want to keep things simple.
Regardless, I think the README could be made a bit clearer regarding which versions are supported. I'm happy to submit an issue &/or PR for that if you agree.
Thanks again
jjtolton commentedon Sep 4, 2022
jjtolton commentedon Sep 5, 2022
@metasoarous set a PR for a working reference copy with JDK-17
cnuernber commentedon Jan 6, 2023
2.023 and dtype-next version 10.000-beta-18 FIX THIS ISSUE :-)
metasoarous commentedon Jan 6, 2023
@cnuernber AMAZING! Thank you kindly for all your work!
build: Remove :jdk-17 alias
build: Remove :jdk-17 alias
AlexAti commentedon Apr 10, 2023
Hello, just a note for people reaching for a problem similar to mine and end up here, in case it saves someone's time: Java 20 no longer has idk.incubator.foreign as a module, so your programs might have broken if you autoupdated your openjdk version (brew did this to me automatically luckily there's still openjdk@17).