You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am still wondering if this is a bug in libpython-clj or if certain python libraries are incompatible with lib-pythonclj and it's using embedded python
Taking a shot in the dark, I turned of automatic gil management in libpython-clj and ran your example w/o using require-python. This is also running with the julia environment which loads the signal handling jvm stub allowing the jvm process to forward signals to python when python has the appropriate signal handler installed.
It worked - there is a lot to dig through here. Note that disabling the gil management means you have to call exactly the java method take-gil once. When doing the java-api I found out that we use check-gil literally many many times per function call. I think there is a race condition in libpython there. With manual gil management we don't ever check the GIL.
Looks like good progress.
I do not understand enough on python and the GIL, in order to use this correctly, I fear.
I do support your hypothesis of race condition somewhere, as the problem disappears the moment I run the clj file via the clj CLI and not via Cider nrepl.
Just for information.
Playing around with the lockGIL I get soetims an python exception:
Traceback (most recent call last):
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
thread._reset_internal_locks(True)
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7faf5aaf0790>
Traceback (most recent call last):
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
thread._reset_internal_locks(True)
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7faf5aaf0790>
Traceback (most recent call last):
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
thread._reset_internal_locks(True)
File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?
Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?
So there is another variable - object-reference-logging in the ffi namespace. Setting it to true should allow you to see the last object freed before the crash.
The require-python statement really loads a ton of objects and all of these then get released. My guess is if you did (py/import-module "simpletransformers.classification") that would not crash.
Changing to import-module `(def classification (py/import-module "simpletransformers.classification"))
gives an exception:
21:24:06.111 [main] ERROR libpython-clj2.python.ffi - Exception while releasing object
java.lang.IllegalStateException: Recursive update
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1739)
at libpython_clj2.python.ffi$pyobject_type_kwd.invokeStatic(ffi.clj:810)
at libpython_clj2.python.ffi$pyobject_type_kwd.invoke(ffi.clj:807)
at libpython_clj2.python.ffi$wrap_obj_ptr$fn__8977.invoke(ffi.clj:845)
at libpython_clj2.python.gc$track$fn__8386.invoke(gc.clj:37)
at tech.resource.GCReference.run(GCReference.java:25)
at libpython_clj2.python.gc$clear_reference_queue.invokeStatic(gc.clj:52)
at libpython_clj2.python.gc$clear_reference_queue.invoke(gc.clj:49)
at train$eval14435.invokeStatic(train.clj:11)
at train$eval14435.invoke(train.clj:11)
at clojure.lang.Compiler.eval(Compiler.java:7181)
at clojure.lang.Compiler.load(Compiler.java:7640)
at clojure.lang.Compiler.loadFile(Compiler.java:7578)
at clojure.main$load_script.invokeStatic(main.clj:475)
at clojure.main$script_opt.invokeStatic(main.clj:535)
at clojure.main$script_opt.invoke(main.clj:530)
at clojure.main$main.invokeStatic(main.clj:664)
at clojure.main$main.doInvoke(main.clj:616)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:40)
huh...what was released? IF the classification variable is bound then that cannot be what is released...I don't think this is the root of the issue but it is certainly a serious issue.
1:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object - 0x7f36140ccdc0:3573:dict
21:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object - 0x7f34fb76bd90: 6:type
21:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object - 0x7f34e31bd3b0: 4:module
21:30:16.324 [main] INFO libpython-clj2.python.ffi - tracking object - 0x7f34da3c2c70: 4:module
21:30:16.324 [main] INFO libpython-clj2.python.ffi - tracking object - 0x7f34f57d9090: 4:module
21:30:16.379 [main] INFO libpython-clj2.python.ffi - releasing object - 0x7f34da3e8c10: 1::method-wrapper
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f35ae533f00, pid=158138, tid=158148
#
# JRE version: OpenJDK Runtime Environment (11.0.16.1+1) (build 11.0.16.1+1)
# Java VM: OpenJDK 64-Bit Server VM (11.0.16.1+1, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libpython3.9.so+0x133f00] _PyTrash_begin+0x0
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/carsten/Dropbox/sources/predict_disaster_tweet/core.158138)
#
# An error report file with more information is saved as:
# /home/carsten/Dropbox/sources/predict_disaster_tweet/hs_err_pid158138.log
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
I propose to close this issue.
The "crashing" is solved by using libpython-clj
with:
embedded mode
manual GIL management
configure python library to not use multiprocessing extensively
(multiprocessing makes apparently as well problems in certain cpython runtimes such as Colab))
I keep here a discussion #219 about a very occasionally blocking and refer to some example code which is 99% working, as reference.
Activity
behrica commentedon Jan 14, 2022
Repo with reproducible code will follow soon.
behrica commentedon Jan 14, 2022
A repo with code and instructions on how to reproduce the issue is here:
https://github.com/behrica/libpython-clj--194
behrica commentedon Mar 20, 2022
I am still wondering if this is a bug in
libpython-cljor if certain python libraries are incompatible with lib-pythonclj and it's using embedded pythoncnuernber commentedon Mar 21, 2022
Taking a shot in the dark, I turned of automatic gil management in libpython-clj and ran your example w/o using require-python. This is also running with the julia environment which loads the signal handling jvm stub allowing the jvm process to forward signals to python when python has the appropriate signal handler installed.
It worked - there is a lot to dig through here. Note that disabling the gil management means you have to call exactly the java method take-gil once. When doing the java-api I found out that we use check-gil literally many many times per function call. I think there is a race condition in libpython there. With manual gil management we don't ever check the GIL.
cnuernber commentedon Mar 21, 2022
Here is the deps.edn file with the jvm opts I used:
{:paths ["src" "resources"] :deps {org.clojure/clojure {:mvn/version "1.10.3"} clj-python/libpython-clj {:mvn/version "2.018"}} :aliases {:manual-gil {:jvm-opts ["-Dlibpython_clj.manual_gil=true"]}}}behrica commentedon Mar 23, 2022
Looks like good progress.
I do not understand enough on python and the GIL, in order to use this correctly, I fear.
I do support your hypothesis of race condition somewhere, as the problem disappears the moment I run the clj file via the
cljCLI and not via Cider nrepl.behrica commentedon Mar 23, 2022
I can confirm that the above code fixes the issue, but I suppose that it is a workaround, is it ?
behrica commentedon Mar 23, 2022
Just for information.
Playing around with the
lockGILI get soetims an python exception:but code keeps working, so training finshes.
cnuernber commentedon Mar 23, 2022
That is fascinating. I will follow up on this soon.
behrica commentedon Sep 4, 2022
Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?
behrica commentedon Sep 4, 2022
Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?
behrica commentedon Sep 4, 2022
I can still not use
simpletransformerlibrary with libpython-clj.jjtolton commentedon Sep 4, 2022
I'll try to take a look Monday, out of town today. I've used
simpletransformersa few times so I'll see if I can get a working setup.behrica commentedon Sep 5, 2022
Just add, it hsppens in REPL only. Working in CLJ file
jjtolton commentedon Sep 6, 2022
I wasn't able to make much progress with this, unfortunately. Very strange. Glad it is working in clj file.
54 remaining items
behrica commentedon Sep 24, 2022
I have now a "Deadlock" in the "train call", with larger data.
Standard Clojure libpython-clj
Reproducible with my data, every time running as
clj src/train.cljcnuernber commentedon Sep 24, 2022
With manual gil?
Most likely they are kicking off the multithreading module which is forking the process which is failing.
behrica commentedon Sep 24, 2022
Maybe we are getting somewhere...
This crashes reliably, without manual GIL.
behrica commentedon Sep 24, 2022
Depending if called in repl or command line, it sometimes hangs or crashes, but never runs through.
cnuernber commentedon Sep 24, 2022
So there is another variable - object-reference-logging in the ffi namespace. Setting it to true should allow you to see the last object freed before the crash.
The require-python statement really loads a ton of objects and all of these then get released. My guess is if you did (py/import-module "simpletransformers.classification") that would not crash.
behrica commentedon Sep 24, 2022
Changing to
import-module`(def classification (py/import-module "simpletransformers.classification"))gives an exception:
but, it continues....
`
cnuernber commentedon Sep 24, 2022
huh...what was released? IF the classification variable is bound then that cannot be what is released...I don't think this is the root of the issue but it is certainly a serious issue.
behrica commentedon Sep 24, 2022
cnuernber commentedon Sep 24, 2022
I think the gil has to captured for clear-reference-queue. I don't think that is a method that captures the gil automatically.
behrica commentedon Sep 24, 2022
yes, I did that.
Then the above code (without training), runs.
behrica commentedon Sep 24, 2022
Once I add the training to id, I get again all kind of issues, mainly hanging this time, not crashing.
behrica commentedon Sep 24, 2022
threaddump-1664051627417.tdump.gz
behrica commentedon Sep 24, 2022
I attached a thread dump while it is hanging.
behrica commentedon Sep 26, 2022
I propose to close this issue.
The "crashing" is solved by using
libpython-cljwith:
(multiprocessing makes apparently as well problems in certain cpython runtimes such as Colab))
I keep here a discussion #219 about a very occasionally blocking and refer to some example code which is 99% working, as reference.