Repository navigation
segfault when releasing a thread #100
Description
I see apparently at random, a segmentation violation.
Bcktrace is
(gdb) bt
#0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 <signal handler called>
#2 __new_sem_post (sem=0x21) at sem_post.c:36
#3 0x00007f38785080f9 in PyThread_release_lock ()
from /lib/x86_64-linux-gnu/libpython3.9.so.1.0
#4 0x00007f3878570be9 in ?? () from /lib/x86_64-linux-gnu/libpython3.9.so.1.0
#5 0x00007f38788d4a2a in release_interpreter (idata=0x55da959531c0)
at mod_python.c:306
#6 python_handler (req=0x7f387450d0a0, phase=<optimized out>)
at mod_python.c:1573
#7 0x000055da949c4bb0 in ap_run_handler ()
#8 0x000055da949c51a6 in ap_invoke_handler ()
#9 0x000055da949ddabb in ap_process_async_request ()
#10 0x000055da949ddcfe in ap_process_request ()
#11 0x000055da949d9bad in ?? ()
#12 0x000055da949ce860 in ap_run_process_connection ()
#13 0x00007f3878960133 in ?? () from /usr/lib/apache2/modules/mod_mpm_worker.so
#14 0x00007f3878c5eea7 in start_thread (arg=<optimized out>)
at pthread_create.c:477
#15 0x00007f3878b8edef in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Looks like we're trying to release a semaphore we don't hold.
Python 3.9.1 (Debian build), Apache2.4.46-4 (Debian); mod_python from 88e4444 with the libpython fix.
Using mpm_worker if that makes a difference.
Activity
wom-bat commentedon Feb 5, 2021
More debugging --- loaded
libpython-3.9d.sowith debugging syms; the bug is because we're trying to release the curent thread.grisha commentedon Feb 5, 2021
I'm not set up to test/verify any of this so if you get to the bottom of it it'd be fantastic. It's (at first glance) code that stood the test of time, so I wonder if there is something about Python 3.9.1 - perhaps it's doing something differently?
cryptofunallday commentedon Feb 23, 2021
Also been getting this for several months, basically since python went to 3.9 it has been happening. Using version 3.8 seems to make it go away, not really best practices to do that though.
wom-bat commentedon May 19, 2021
Commenting out the call to
PyThreadState_Clear(tstate);near line 306 ofmod_python.cmakes the issue go away.There is only one documented change for this function: from Python 3.9 on it calls the
PyThreadState.on_deletecallback.domruf commentedon Nov 16, 2021
We use httpd docker images which recently switched from debian buster to bullseye. I now also see the Segmentation fault errors with my PythonAuthenHandler.
It looks like the errors happen mainly right after a restart of httpd.
Is there anything I can do to prevent these errors?
i300220 commentedon Jun 16, 2023
This opens a python file I've been running since python 3.9 at least 2 years ago. The file is called several times per second 24/7. I'm now running python 3.11 and is segfaults horribly. Commenting out PyThreadState_Clear(tstate); fixed it.
Just a confirmation that wom-bat patch works.
grisha commentedon Jun 16, 2023
So should that line be wrapped in something like so
i300220 commentedon Jun 16, 2023
Totally makes sense. Thanks!
grisha commentedon Jun 16, 2023
Could you confirm that this works for you? I'm currently not set up to test it. If it works, then a pull-request would be much appreciated, if not let me know and I'll add it in myself. Thank you!
i300220 commentedon Jun 16, 2023
It does. Just recompiled the source from debian 12 with your suggestion. Restarted apache and no segfault.
Can't make a pull request though, permission denied:
$ LANG=C git push origin mb
remote: Permission to grisha/mod_python.git denied to i300220.
fatal: unable to access 'https://github.com/grisha/mod_python.git/': The requested URL returned error: 403
grisha commentedon Jun 16, 2023
That's okay, I took care of it, thanks for your help!
stefanor commentedon Oct 26, 2023
This was fixed as >= 3.10, but reported against 3.9 too. Is that related, or is #122 a different bug
grisha commentedon Oct 26, 2023
@stefanor I don't have the set up to test it - if you can confirm that the fix should include 3.9 too that'd be much appreciated.
i300220 commentedon Oct 26, 2023
I'm afraid it's present in 3.9 too. I didn't realised it immediately. Only when I installed debian 12 and after many useless edits in my python files because I had a lot of trouble and noticed this fix applied to 3.11 has nothing to do with my python files being bad because the bug was suddenly gone. It was related to this bug. Sorry I forgot to report it. I remember clearly I had to switch to old apache module whenever I wanted to run some python 3 code.
In other words, it was fine on debian 10, broken in debian 11 (with 3.9) and fine again in debian 12, thanks to this fix.
grisha commentedon Oct 26, 2023
@i300220 so the fix should really be
does this look reasonable?
i300220 commentedon Oct 26, 2023
@grisha that's right. thanks!
Do not clear thread state for version above and including 3.9