Repository navigation
Py_NewInterpreterFromConfig end with hard crash #123488
Description
Crash report
What happened?
I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my library.
The library is thread safe and supports many languages
- C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php
including other languages with thread support
- C C++ JAVA C# TCL GO
are working fine. Even python works fine without thread support.
Task: now I started to use thread support in python.
I had to change my code to to Multi-phase initialization, isolate the python data on an per-thread-level and finally get an easy thread example working -> ok
I test a client server application. the server create a thread in my library and this thread is initialized with
if (create == MQ_FACTORY_NEW_THREAD) {
//mk_debug_color(MK_YELLOW,"%s","MQ_FACTORY_NEW_THREAD");
static PyInterpreterConfig config = {
.use_main_obmalloc = 0,
.allow_fork = 0,
.allow_exec = 0,
.allow_threads = 1,
.allow_daemon_threads = 0,
.check_multi_interp_extensions = 1,
.gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);
MK_RT_REF.threadData = tstate;
//printV("#2 PyThreadState_Get<%p>", PyThreadState_Get())
if (PyStatus_Exception(status)) {
return MkErrorSetC_3_NULL(status.err_msg, status.func, 300+status.exitcode);
}
// mark new interpreter as "MQ_STARTUP_IS_THREAD"
PyObject *mainM = PyImport_AddModule("__main__");
PyObject *strO = PyUnicode_FromString("MQ_STARTUP_IS_THREAD");
PyObject_SetAttrString(mainM,"__name__", strO);
Py_CLEAR(strO);
// source initial script
FILE *FH = fopen(MqInitGetArg0()->data[1]->storage.first.C, "r");
check_LNG(PyRun_SimpleFileEx(FH,MqInitGetArg0()->data[1]->storage.first.C,true)) {
return MkErrorSetV_1E("script '%s' failed",MqInitGetArg0()->data[1]->storage.first.C);
}
}
I figure out that always after 4 successful simple round trip clinet → server → client the server crash with the message below
- The stack-trace comes from my
MqDisasterSignalhandler.
exec[#6] -> 'NHI1_EXT/debug/bin/python3' 'NHI1_HOME/example/py/MyServer.py' '--thread' '--uds' '--file' '/tmp/test.uds'
Debug memory block at address p=0x7f0012d1a730: API '^@'
18302063728033398269 bytes originally requested
The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
at p-7: 0x00 *** OUCH
at p-6: 0x00 *** OUCH
at p-5: 0x00 *** OUCH
at p-4: 0x00 *** OUCH
at p-3: 0x00 *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 8 pad bytes at tail=0xfdfe7cfe10cfa52d are X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: BackTrace {
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ library : filename : lineno ] function
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ ------- : -------- : ------ ] --------
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/sys_mq.c : 705 ] MqDisasterSignal
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ unknown : unknown : 0 ] unknown
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2408 ] _PyObject_DebugDumpAddress
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2326 ] _PyMem_DebugCheckAddress
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2159 ] _PyMem_DebugRawFree
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 685 ] PyMem_RawFree
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 1853 ] _PyObject_Free
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2163 ] _PyMem_DebugRawFree
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2296 ] _PyMem_DebugFree
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 830 ] PyObject_Free
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1569 ] dictresize
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1194 ] insertion_resize
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1261 ] insertdict
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1865 ] _PyDict_SetItem_Take2
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1883 ] PyDict_SetItem
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7618 ] add_subclass
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7352 ] type_ready_add_subclasses
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7515 ] type_ready
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7553 ] PyType_Ready
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3795 ] type_new_impl
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3929 ] type_new
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 1664 ] type_call
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 240 ] _PyObject_MakeTpCall
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3949 ] type_vectorcall
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 133 ] _PyObject_FastCallDictTstate
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 157 ] PyObject_VectorcallDict
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/bltinmodule.c : 208 ] builtin___build_class__
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/methodobject.c : 438 ] cfunction_vectorcall_FASTCALL_KEYWORDS
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Include/internal/pycore_call.h : 92 ] _PyObject_VectorcallTstate
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 325 ] PyObject_Vectorcall
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/bytecodes.c : 2714 ] _PyEval_EvalFrameDefault
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : nclude/internal/pycore_ceval.h : 89 ] _PyEval_EvalFrame
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/ceval.c : 1683 ] _PyEval_Vector
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/ceval.c : 578 ] PyEval_EvalCode
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1722 ] run_eval_code_obj
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1743 ] run_mod
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1643 ] pyrun_file
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 433 ] _PyRun_SimpleFileObject
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 466 ] PyRun_SimpleFileExFlags
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : py/MqFactoryC_py.c : 195 ] py_mqmsgque_sFactoryCTor
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/MqFactoryS_mq.c : 289 ] MqFactoryInvoke_RT
X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/sys_mq.c : 269 ] MqSysServerThreadMain
The server code (file sourced into the new interpreter) is quite simple.
# Add a code block here, if required
import sys
from pymqmsgque import *
# package-item
class MyServer(MqContextC):
# factory startup
def __init__(self, tmpl=None):
self.ConfigSetServerSetup(self.ServerSetup)
super().__init__(tmpl)
# service to serve all incoming requests for token "HLWO"
def MyFirstService(self):
self.SendSTART()
self.SendSTR(self.ReadSTR() + " World")
self.SendRETURN()
# define a service as link between the token "HLWO" and the callback "MyFirstService"
def ServerSetup(self):
self.ServiceCreate("HLWO",self.MyFirstService)
# package-main
if __name__ == "__main__":
# create the "MyServer" factory… and the object
srv = MqFactoryC.Add(MyServer).New()
try:
srv.LinkCreate(sys.argv)
srv.ProcessEvent(MqWaitOnEventE.FOREVER)
except Exception as ex:
srv.ErrorCatch(ex)
finally:
srv.Exit()the problem is that running this example with a server with valgrind nothing happen (no crash)
the same example using fork or using spawn to startup the server instances works fine.
the problem is only with thread support.
additional Information: because my library only support one server-instance per thread, I choose PyInterpreterConfig_OWN_GIL to be clear that I don't want to have interaction between different python interpreters.
all the communication (between processes and threads) is done by my library. my library uses thread-local-storage to isolate the data per-thread.
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.12.4 (main, Aug 29 2024, 18:00:55) [GCC 13.3.0]
Activity
vstinner commentedon Aug 30, 2024
cc @ericsnowcurrently
ZeroIntensity commentedon Aug 30, 2024
I wonder if this is related to #123134. However, have you called
PyGILState_Ensurebefore callingPy_NewInterpreterFromConfig? According to the docs, you still need the GIL to create a subinterpreter.aotto1968 commentedon Aug 30, 2024
the docs say also…
Note that the PyGILState_* functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_* API is unsupported.
ZeroIntensity commentedon Aug 30, 2024
Right, mixing them is unsupported. You still need to have Python initialized, and then you can create subinterpreters with the main interpreter's GIL -- once the sub interpreter is initialized, it will have its own GIL.
aotto1968 commentedon Aug 30, 2024
First success … If I use a "release" build, (non debug python) than
valgrindreport an erroraotto1968 commentedon Aug 30, 2024
I have an initialized main python interpreter which loads my library and in the library a thread is created etc.. this is the reason why it is workc but only for 4 instances :-(
Task → my current research is to make the type itself thread-local I mean the new type I define in my extension but now the thread startup does not understand the type template (because the original type is from the main-thread and this type has no meaning in the new thread. :-(
The "MyServer" is the initial server from the main thread
ZeroIntensity commentedon Aug 30, 2024
Yeah, this looks like it's probably a duplicate of #123134. I haven't reproduced this on Linux yet, but on Windows, there seems to be a thread safety issue inside the allocator when using a per-interpreter GIL. I'll investigate this further later today. In the meantime, try adding some synchronization between your threads to prevent them from calling
Py_NewInterpreterFromConfigconcurrently.aotto1968 commentedon Aug 30, 2024
even if I do a one thread per step my python is failing after the fourth thread was created (and closed) → there is no "concurrency" right now
ZeroIntensity commentedon Aug 30, 2024
You'll have to share more of the C code, but that's probably just a side-effect of a prior memory error.
aotto1968 commentedon Aug 30, 2024
question : do you use a TYPE per THREAD or a TYPE per process , I mean the C-Python-Type definition
I add the macro
OT_CLASS_TYPE_REFthis is__thread PyTypeObjector justPyTypeObject→ I test this setup.
aotto1968 commentedon Aug 31, 2024
new test case with parallel access . looks like the "race" thing
aotto1968 commentedon Aug 31, 2024
OK - now a success news (with smell)
In the docu about the
PyInterpreterConfig_OWN_GIL(A Per-Interpreter GIL) there was a paragraph about isolation ...and my first question was , does the type object have to be isolated as well?
I started a research about thread local type object
switching to thread-local-types the interpreter does not crash anymore BUT ...
analysis
2.1. tests done with my performance test tool → parallel creating of a per-thread-session
2.2. test up to 3 workers → everything is OK
2.3. test 4 or more workers → a race crash happen…
3.1. there is no (performance)-reason to use a thread at all in python.
3.2. the startup-speed for spawn depends on the speed of the filesystem, a 10 times difference between a mounted file system and a local filesystem is possible. the info from
3.1.was related to an local filesystem.race crash
ZeroIntensity commentedon Aug 31, 2024
After playing around with this, I was unable to reproduce the data race. Though, I did encounter a deadlock when trying to call
Py_NewInterpreterFromConfigwithout the GIL and the properPy_BEGIN_ALLOW_THREADSsection -- are those present in your code? If not, then this is a user error!aotto1968 commentedon Sep 1, 2024
Just to be clear what the flow is:
server.pyserver.pyload an external libraryPy_NewInterpreterFromConfigwithPyInterpreterConfig_OWN_GILIf I do…
than I get :
I think
Py_NewInterpreterFromConfig→PyInterpreterConfig_OWN_GILcreate and set threadstate and gil.If I skip
PyEval_SaveThreadandPyGILState_Ensureeverything works for easy tests BUT In a load test I get after a while13 remaining items
aotto1968 commentedon Sep 1, 2024
I call the
Py_BEGIN_ALLOW_THREADSon the main thread because there are no other threads before thePyServerThread→ just look to thetidcolumn and thepy_mqmsgque_PyServerThreadthe
py_mqmsgque_sFactoryCToris still blocking…ZeroIntensity commentedon Sep 1, 2024
It's difficult for me to follow the code due to the use of macros, but if it's truly the main thread, then I think the problem is the
PyGILState_*calls -- they aren't necessary to callPy_BEGIN_ALLOW_THREADS. Releasing the GIL state in this case might prevent any future API calls (such asPy_Finalize). Though, I could be wrong on that, the GIL state APIs might have a mechanism to deal with that.Regardless, I don't think this is a CPython issue :(
aotto1968 commentedon Sep 1, 2024
Found something special under "load" !! → Q: how to check if "ts" is still valid?
The main problem is that every thread is a server that can terminate itself. So if a server terminates between
PyThreadState_Swap#1andPyThreadState_Swap#2and that server "happens" to be returned astsby the firstPyThreadState_Swap#1, then it just CRASHes.PyThreadStatedisappears.tscan not follow an external thread delete.If I switch to
Py_BEGIN_ALLOW_THREADSinstead ofPyThreadState_SwapthePy_EndInterpreter(interpreter_ts)fails with
Fatal Python error: Py_EndInterpreter: thread is not currentupdate
even if I disable the
Py_EndInterpreterand thepthread_exitthe python crash withDebug memory block at address ... FORBIDDENBYTEZeroIntensity commentedon Sep 1, 2024
AFAIK,
Py_EndInterpreterwill deallocate the thread state, so trying to usetspast that point is invalid. Why are you callingPyThreadState_Swapinstead of just using the actual thread state for the interpreter?aotto1968 commentedon Sep 1, 2024
the problem is that this is just an example → this lost
tscould happen everywhere in the code because of thePyThreadState_Swapapi design.ZeroIntensity commentedon Sep 1, 2024
Ok, but it's not obvious what the clear "fix" would be here.
PyThreadState_Swapinvalidates the old thread state (which is why your example fails), we can't change that.aotto1968 commentedon Sep 1, 2024
I just testing around the
tsis the thread-state before the thread-exit is called →it is NOt the thread stata I want delete bot could be.ZeroIntensity commentedon Sep 1, 2024
In the above example, you're using
PyThreadState_Swapincorrectly.Again, I don't think there's anything here that's a CPython issue :(
I'll submit a PR for some extra documentation on subinterpreters, but I suggest opening a help thread and closing this issue.
aotto1968 commentedon Sep 1, 2024
I already delete this test code because it was just a test → the original problem of crash somewhere is still an issue.
ZeroIntensity commentedon Sep 1, 2024
I can't help with that unless I see the code, but let's move this to discourse. If we find that this is indeed a CPython issue, we can reopen (or better yet, make a new one with a more solid reproducer).
aotto1968 commentedon Sep 1, 2024
Again, I don't think there's anything here that's a CPython issue :(
well I already use my library as an extension for many languages and obvious able I'm not smart enough to use the python thread api. → I'll think this is an python problem :-)
After trying for the second time in my limited lifetime to do something with thread on python and being miserably wiser twice, I must now admit that python is apparently incapable of providing any usable thread-api.
→ the smartest thing to do with thread on python… activate the ignoreThread switch
ZeroIntensity commentedon Sep 1, 2024
Not disagreeing that multithreading from the C API is difficult! We need to document it better. Though, if you want to get this working, I can help you on discourse, instead of polluting the issues page with relentless back-and-forth debugging.
ZeroIntensity commentedon Sep 4, 2024
@encukou, this should be closed as not planned. I'll create a new issue sometime tomorrow to add some better documentation on using PEP 684.
ZeroIntensity commentedon Sep 4, 2024
I've created #123672 to address the documentation problem.