Repository navigation
Multiple PyFrameObjects can be created for a single _PyInterpreterFrame #97002
Description
Activity
- added a commit that references this issue
on Sep 21, 2022 @pablogsal, this looks a lot like #95818. Possibly related to #96975?
I'll dig into it further.
@davfsa, does this crash pretty consistently? I'm unable to reproduce on
main,3.11, orv3.11.0rc2... Pytest just runs the test suite and passes.Do you have concrete (ideally, minimal) steps that I can follow to reproduce it? Or maybe a core dump?
@davfsa, does this crash pretty consistently? I'm unable to reproduce on
main,3.11, orv3.11.0rc2... Pytest just runs the test suite and passes.Do you have concrete (ideally, minimal) steps that I can follow to reproduce it? Or maybe a core dump?
I'll check when I get home, but before opening this issue I ran our test suite several times and was quite consistent. The segfault happens only on our CI, so might be a bit hard to get a core dump from there. I'll try to get one locally.
So far the underlying issue I found happens when aiohttp leaves an open websocket, leading to pytest to wrap it and raise a warning, but we have it setup to error on any warnings.
I'll try go get a minimally repo as soon as I can, but the issue can be found when running our test suite (https://github.com/hikari-py/hikari) using
nox -s pytest.I am able to reproduce this intermittently now, partially minimized. What I'm doing is running:
$ pytest -v tests/hikari/impl/test_rest_bot.py --cov
Sometimes we get those attribute errors in the tracebacks where frames/code/etc. are being replaced with seemingly random objects, and sometimes it just segfaults. So this does look like a use-after-free somewhere, similar to #95818.
Worth noting is that this only seems to happen when
--covis passed. So C tracing seems like a possible culprit.With a debug build, I'm (sometimes) able to observe the following assertion failure during
tests/hikari/impl/test_rest_bot.py::TestRESTBot::test_get_listener:Objects/frameobject.c:1304: PyFrame_GetCode: Assertion `code != NULL' failed. Fatal Python error: Aborted Current thread 0x00007f86102d0280 (most recent call first): File "/home/brandtbucher/cpython/Lib/traceback.py", line 353 in _walk_tb_with_full_positions File "/home/brandtbucher/cpython/Lib/traceback.py", line 416 in _extract_from_extended_frame_gen File "/home/brandtbucher/cpython/Lib/traceback.py", line 688 in __init__ File "/home/brandtbucher/cpython/Lib/traceback.py", line 139 in format_exception File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 72 in unraisable_exception_runtest_hook File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/unraisableexception.py", line 83 in pytest_runtest_setup File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 55 in _multicall File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 259 in <lambda> File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 338 in from_call File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 258 in call_runtest_hook File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 219 in call_and_report File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 124 in runtestprotocol File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 347 in pytest_runtestloop File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 322 in _main File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 268 in wrap_session File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/brandtbucher/env311/lib/python3.11/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 164 in main File "/home/brandtbucher/env311/lib/python3.11/site-packages/_pytest/config/__init__.py", line 187 in console_main File "/home/brandtbucher/env311/bin/pytest", line 8 in <module>
Full dump:
Details
#0 raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x000055878e6ede65 in faulthandler_fatal_error (signum=6) at ./Modules/faulthandler.c:373 #2 <signal handler called> #3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #4 0x00007fdbd477a859 in __GI_abort () at abort.c:79 #5 0x00007fdbd477a729 in __assert_fail_base (fmt=0x7fdbd4910588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55878e7bb321 "code != NULL", file=0x55878e7a2af4 "Objects/frameobject.c", line=1304, function=<optimized out>) at assert.c:92 #6 0x00007fdbd478bf36 in __GI___assert_fail (assertion=assertion@entry=0x55878e7bb321 "code != NULL", file=file@entry=0x55878e7a2af4 "Objects/frameobject.c", line=line@entry=1304, function=function@entry=0x55878e7a3640 <__PRETTY_FUNCTION__.18739> "PyFrame_GetCode") at assert.c:101 #7 0x000055878e55e344 in PyFrame_GetCode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6df0>: frame=frame@entry=) at Objects/frameobject.c:1306 #8 0x000055878e55e37a in frame_getcode (Python Exception <class '__main__.NullPyObjectPtr'> <__main__.PyCodeObjectPtr object at 0x7f09f19d6670>: f=, closure=<optimized out>) at Objects/frameobject.c:97 #9 0x000055878e54223b in getset_get (descr=descr@entry=0x7fdbd4385850, Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=obj@entry=, type=<optimized out>) at Objects/descrobject.c:202 #10 0x000055878e599bf9 in _PyObject_GenericGetAttrWithDict (Python Exception <class 'gdb.error'> There is no member named f_frame.: obj=, name='f_code', dict=dict@entry=0x0, suppress=suppress@entry=0) at ./Include/object.h:133 #11 0x000055878e59a520 in PyObject_GenericGetAttr (obj=<optimized out>, name=<optimized out>) at Objects/object.c:1367 #12 0x000055878e5998b2 in PyObject_GetAttr (Python Exception <class 'gdb.error'> There is no member named f_frame.: v=v@entry=, name='f_code') at Objects/object.c:916 #13 0x000055878e66121b in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0a3d2a0, throwflag=throwflag@entry=0) at Python/ceval.c:3466 #14 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0a3d2a0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #15 gen_send_ex2 (gen=0x7fdbd0a3d250, arg=arg@entry=0x0, presult=presult@entry=0x7ffdc69a9b10, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219 #16 0x000055878e55491f in gen_iternext (gen=<optimized out>) at Objects/genobject.c:594 #17 0x000055878e665e67 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6310, frame@entry=0x7fdbd42d61b0, throwflag=throwflag@entry=0) at ./Include/object.h:133 #18 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d61b0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #19 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #20 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #21 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, args=args@entry=0x7ffdc69a9d30, nargsf=nargsf@entry=4, kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:152 #22 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd406f490>, obj=obj@entry=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=args@entry=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwargs=kwargs@entry={'limit': None, 'compact': True}) at Objects/call.c:482 #23 0x000055878e5bcda1 in slot_tp_init (self=<TracebackException(max_group_width=15, max_group_depth=10) at remote 0x7fdbd0aaca60>, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:7861 #24 0x000055878e5b83e1 in type_call (type=0x55878f938370, args=(<type at remote 0x55878e8e23e0>, ResourceWarning('unclosed <socket.socket fd=13, family=1, type=1, proto=0>',), <traceback at remote 0x7fdbd071fb60>), kwds={'limit': None, 'compact': True}) at Objects/typeobject.c:1112 #25 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, keywords=keywords@entry=('limit', 'compact')) at Objects/call.c:214 #26 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('limit', 'compact'), nargsf=9223372036854775811, args=0x7fdbd42d6188, callable=<type at remote 0x55878f938370>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90 #27 PyObject_Vectorcall (callable=callable@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargsf=<optimized out>, nargsf@entry=9223372036854775811, kwnames=kwnames@entry=('limit', 'compact')) at Objects/call.c:299 #28 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<type at remote 0x55878f938370>, args=args@entry=0x7fdbd42d6188, nargs=<optimized out>, kwnames=('limit', 'compact')) at Python/ceval.c:7305 #29 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6100, frame@entry=0x7fdbd0967c40, throwflag=throwflag@entry=0) at Python/ceval.c:4767 #30 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0967c40, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #31 gen_send_ex2 (gen=gen@entry=0x7fdbd0967bf0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa090, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219 #32 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0967bf0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287 #33 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310 #34 0x000055878e545401 in method_vectorcall_O (func=<method_descriptor at remote 0x7fdbd4387110>, args=0x7ffdc69aa230, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:481 #35 0x000055878e536905 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7ffdc69aa230, callable=<method_descriptor at remote 0x7fdbd4387110>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92 #36 PyObject_VectorcallMethod (name=<optimized out>, args=args@entry=0x7ffdc69aa230, nargsf=2, nargsf@entry=9223372036854775810, kwnames=kwnames@entry=0x0) at Objects/call.c:855 #37 0x000055878e65ca0a in PyObject_CallMethodOneArg (arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, name=<optimized out>, self=<generator at remote 0x7fdbd0967bf0>) at ./Include/cpython/abstract.h:116 #38 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd0b4f710, throwflag=throwflag@entry=0) at Python/ceval.c:2598 #39 0x000055878e553e55 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd0b4f710, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #40 gen_send_ex2 (gen=gen@entry=0x7fdbd0b4f6c0, arg=<_Result(_result=[], _excinfo=None) at remote 0x7fdbd0762450>, presult=presult@entry=0x7ffdc69aa2f0, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:219 #41 0x000055878e554236 in gen_send_ex (gen=0x7fdbd0b4f6c0, arg=<optimized out>, exc=exc@entry=0, closing=closing@entry=0) at Objects/genobject.c:287 #42 0x000055878e5548b8 in gen_send (gen=<optimized out>, arg=<optimized out>) at Objects/genobject.c:310 #43 0x000055878e593230 in cfunction_vectorcall_O (func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=0x7fdbd42d60e0, nargsf=<optimized out>, kwnames=<optimized out>) at ./Include/cpython/methodobject.h:52 #44 0x000055878e53610a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fdbd42d60e0, callable=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:92 #45 PyObject_Vectorcall (callable=callable@entry=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, args=args@entry=0x7fdbd42d60e0, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:299 #46 0x000055878e657908 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=<built-in method send of generator object at remote 0x7fdbd0b4f6c0>, func@entry=<method_descriptor at remote 0x7fdbd4387110>, args=args@entry=0x7fdbd42d60d8, nargs=2, kwnames=0x0) at Python/ceval.c:7299 #47 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d6010, frame@entry=0x7fdbd42d5ec8, throwflag=throwflag@entry=0) at Python/ceval.c:4767 #48 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ec8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #49 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #50 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #51 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aa5d0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:152 #52 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:482 #53 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/typeobject.c:7630 #54 0x000055878e5352c9 in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:343 #55 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>}) at Objects/call.c:355 #56 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_setup', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd3486fc0>, argnames=(), kwargnames=(...), plugin=<module at remote 0x7fdbd32505f0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='threadexception', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd348fb70>, <HookImpl(function=<method at remote 0x7fdbd2769370>, argnames=('item',), kwargnames=(...), plugin=<CaptureManager(_method='fd', _global_capturing=<MultiCapture(in_=<FDCapture(targetfd=0, targetfd_invalid=None, targetfd_save=3, tmpfile=<_io.TextIOWrapper at remote 0x7fdbd2657110>, syscapture=<SysCapture(_old=<_io.TextIOWrapper at remote 0x7fdbd41de120>, name='stdin', tmpfile=<DontReadFromInput() at remote 0x7fdbd26d4830>, _state='started') at remote 0x7fdbd26d4240>, _state='started') at remote 0x7fdbd3d1aef0>, out=<FDCapture(targetfd=1, targetfd_invalid=None, targetfd_save=5, tmpfile=<Enco...(truncated), callargs=callargs@entry=(), kwdict=kwdict@entry={'item': <Function at remote 0x7fdbd0a83f60>}, use_tracing=255) at Python/ceval.c:7347 #57 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5e40, frame@entry=0x7fdbd42d5cc0, throwflag=throwflag@entry=0) at Python/ceval.c:5377 #58 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5cc0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #59 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #60 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #61 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at ./Include/object.h:772 #62 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:328 #63 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635ff0>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwargs=kwargs@entry={}) at Objects/call.c:355 #64 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635ff0>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, 'setup'), kwdict=kwdict@entry={}, use_tracing=255) at Python/ceval.c:7347 #65 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5c18, frame@entry=0x7fdbd42d5ae0, throwflag=throwflag@entry=0) at Python/ceval.c:5377 #66 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5ae0, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #67 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #68 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #69 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3635910>, tuple=tuple@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772 #70 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328 #71 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3635910>, args=args@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355 #72 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3635910>, callargs=callargs@entry=(<Function at remote 0x7fdbd0a83f60>, <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from lower to higher, that i...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347 #73 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d59f0, frame@entry=0x7fdbd42d58a8, throwflag=throwflag@entry=0) at Python/ceval.c:5377 #74 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d58a8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #75 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #76 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #77 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69aadd0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/call.c:152 #78 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=(), kwargs=kwargs@entry={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/call.c:482 #79 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=(), kwds={'item': <Function at remote 0x7fdbd0a83f60>, 'nextitem': <Function(keywords=<NodeKeywords at remote 0x7fdbd0a87ac0>, own_markers=[], extra_keyword_matches=set(), stash=<Stash at remote 0x7fdbd0cc0190>, _report_sections=[], user_properties=[], originalname='test_set_listener', _obj=<method at remote 0x7fdbd0a9b770>, _fixtureinfo=<FuncFixtureInfo at remote 0x7fdbd0a9b6b0>, fixturenames=['mock_rest_bot', 'mock_interaction_server', 'mock_rest_client', 'mock_entity_factory', 'mock_http_settings', 'mock_proxy_settings', 'mock_executor'], funcargs={}, _request=<FixtureRequest(_pyfuncitem=<...>, fixturename=None, _scope=<Scope(_value_='function', _name_='Function', __objclass__=<EnumType(_generate_next_value_=<function at remote 0x7fdbd4109230>, __module__='_pytest.scope', __annotations__={'Function': '_ScopeName', 'Class': '_ScopeName', 'Module': '_ScopeName', 'Package': '_ScopeName', 'Session': '_ScopeName'}, __doc__='\n Represents one of the possible fixture scopes in pytest.\n\n Scopes are ordered from low...(truncated)) at Objects/typeobject.c:7630 #80 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, keywords=keywords@entry=('item', 'nextitem')) at Objects/call.c:214 #81 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('item', 'nextitem'), nargsf=9223372036854775808, args=0x7fdbd42d5890, callable=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90 #82 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('item', 'nextitem')) at Objects/call.c:299 #83 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtest_protocol', _wrappers=[<HookImpl(function=<function at remote 0x7fdbd34875f0>, argnames=('item',), kwargnames=(), plugin=<module at remote 0x7fdbd3250ef0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': True, 'specname': None}, plugin_name='faulthandler', tryfirst=False, trylast=True, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd3254d80>, <HookImpl(function=<function at remote 0x7fdbd33b7ac0>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd33ee4b0>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': False, 'trylast': False, 'specname': None}, plugin_name='unittest', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=True, specname=None) at remote 0x7fdbd33dfa80>, <HookImpl(function=<function at remote 0x7fdbd3643330>, argnames=('item',), kwargnames=(...), plugin=<module at remote 0x7fdbd4139370>, opts={'hookwrapper': True, 'optionalhook': False, 'tryfirst': True, 'trylast': ...(truncated), args=args@entry=0x7fdbd42d5890, nargs=<optimized out>, kwnames=('item', 'nextitem')) at Python/ceval.c:7305 #84 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=frame@entry=0x7fdbd42d5810, throwflag=throwflag@entry=0) at Python/ceval.c:4767 #85 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5810, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #86 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #87 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #88 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3637120>, tuple=tuple@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at ./Include/object.h:772 #89 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:328 #90 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3637120>, args=args@entry=(<Session at remote 0x7fdbd25c6610>,), kwargs=kwargs@entry=0x0) at Objects/call.c:355 #91 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3637120>, callargs=callargs@entry=(<Session at remote 0x7fdbd25c6610>,), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347 #92 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5720, frame@entry=0x7fdbd42d55d8, throwflag=throwflag@entry=0) at Python/ceval.c:5377 #93 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d55d8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #94 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #95 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #96 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab3a0, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:152 #97 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=(), kwargs=kwargs@entry={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/call.c:482 #98 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=(), kwds={'session': <Session at remote 0x7fdbd25c6610>}) at Objects/typeobject.c:7630 #99 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, keywords=keywords@entry=('session',)) at Objects/call.c:214 #100 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('session',), nargsf=9223372036854775808, args=0x7fdbd42d55d0, callable=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90 #101 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('session',)) at Objects/call.c:299 #102 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_runtestloop', _wrappers=[<HookImpl(function=<method at remote 0x7fdbd2733bf0>, argnames=('session',), kwargnames=(), plugin=<CovPlugin(pid=32382, cov_controller=<Central(cov_source=None, cov_report={}, cov_config='.coveragerc', cov_append=False, cov_branch=None, config=<Config() at remote 0x7fdbd26d46a0>, nodeid=None, cov=<Coverage(_no_disk=False, config=<CoverageConfig(attempted_config_files=['.coveragerc', 'setup.cfg', 'tox.ini', 'pyproject.toml'], config_files_read=['/home/brandtbucher/hikari/pyproject.toml'], config_file='/home/brandtbucher/hikari/pyproject.toml', _config_contents=b'# Copyright (c) 2020 Nekokatt\n# Copyright (c) 2021-present davfsa\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the "Software"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the...(truncated), args=args@entry=0x7fdbd42d55d0, nargs=<optimized out>, kwnames=('session',)) at Python/ceval.c:7305 #103 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5568, frame@entry=0x7fdbd42d5430, throwflag=throwflag@entry=0) at Python/ceval.c:4767 #104 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5430, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #105 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #106 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #107 0x000055878e534f24 in _PyVectorcall_Call (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=0x55878e53541c <_PyFunction_Vectorcall>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, tuple=tuple@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at ./Include/object.h:772 #108 0x000055878e53538a in _PyObject_Call (tstate=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:328 #109 0x000055878e5353f2 in PyObject_Call (callable=callable@entry=<function at remote 0x7fdbd3636f10>, args=args@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwargs=kwargs@entry=0x0) at Objects/call.c:355 #110 0x000055878e65797c in do_call_core (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<function at remote 0x7fdbd3636f10>, callargs=callargs@entry=(<Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asyncio_mode=...(truncated), kwdict=kwdict@entry=0x0, use_tracing=255) at Python/ceval.c:7347 #111 0x000055878e66b6f1 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5340, frame@entry=0x7fdbd42d51f8, throwflag=throwflag@entry=0) at Python/ceval.c:5377 #112 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d51f8, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #113 _PyEval_Vector (tstate=0x55878ea0cf78 <_PyRuntime+166328>, func=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6428 #114 0x000055878e535470 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:393 #115 0x000055878e535a72 in _PyObject_FastCallDictTstate (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, args=args@entry=0x7ffdc69ab970, nargsf=nargsf@entry=1, kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:152 #116 0x000055878e535c31 in _PyObject_Call_Prepend (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<function at remote 0x7fdbd3e18100>, obj=obj@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=(), kwargs=kwargs@entry={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/call.c:482 #117 0x000055878e5bcf65 in slot_tp_call (self=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=(), kwds={'config': <Config(option=<Namespace(keyword='', markexpr='', maxfail=0, continue_on_collection_errors=False, confcutdir=None, noconftest=False, keepduplicates=False, collect_in_virtualenv=False, importmode='prepend', basetemp=None, durations=None, durations_min=<float at remote 0x7fdbd43ce4a0>, version=0, plugins=[], traceconfig=False, showfixtures=False, show_fixtures_per_test=False, verbose=1, no_header=False, no_summary=False, reportchars='fE', disable_warnings=False, showlocals=False, tbstyle='auto', showcapture='all', fulltrace=False, color='auto', code_highlight='yes', capture='fd', runxfail=False, pastebin=None, assertmode='rewrite', xmlpath=None, junitprefix=None, doctestmodules=False, doctestreport='udiff', doctestglob=[], doctest_ignore_import_errors=False, doctest_continue_on_failure=False, last_failed_no_failures='all', stepwise=False, stepwise_skip=False, cov_source=[True], cov_report={'': None}, cov_config='.coveragerc', no_cov_on_fail=False, no_cov=False, cov_append=False, cov_branch=None, asy...(truncated)) at Objects/typeobject.c:7630 #118 0x000055878e535764 in _PyObject_MakeTpCall (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, keywords=keywords@entry=('config',)) at Objects/call.c:214 #119 0x000055878e5361fb in _PyObject_VectorcallTstate (kwnames=('config',), nargsf=9223372036854775808, args=0x7fdbd42d51d0, callable=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_call.h:90 #120 PyObject_Vectorcall (callable=callable@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargsf=<optimized out>, nargsf@entry=9223372036854775808, kwnames=kwnames@entry=('config',)) at Objects/call.c:299 #121 0x000055878e6576a2 in trace_call_function (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=<_HookCaller(name='pytest_cmdline_main', _wrappers=[], _nonwrappers=[<HookImpl(function=<function at remote 0x7fdbd3636f10>, argnames=('config',), kwargnames=(), plugin=<module at remote 0x7fdbd3898530>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='main', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33ce950>, <HookImpl(function=<function at remote 0x7fdbd33b6780>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd33da930>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': False, 'hookwrapper': False, 'specname': None}, plugin_name='helpconfig', tryfirst=False, trylast=False, optionalhook=False, hookwrapper=False, specname=None) at remote 0x7fdbd33dcc90>, <HookImpl(function=<function at remote 0x7fdbd3655390>, argnames=('config',), kwargnames=(...), plugin=<module at remote 0x7fdbd363ff50>, opts={'tryfirst': False, 'trylast': False, 'optionalhook': F...(truncated), args=args@entry=0x7fdbd42d51d0, nargs=<optimized out>, kwnames=('config',)) at Python/ceval.c:7305 #122 0x000055878e667ca9 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, frame=0x7fdbd42d5120, frame@entry=0x7fdbd42d5020, throwflag=throwflag@entry=0) at Python/ceval.c:4767 #123 0x000055878e66d738 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdbd42d5020, tstate=0x55878ea0cf78 <_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73 #124 _PyEval_Vector (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, func=func@entry=0x7fdbd41fa620, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:6428 #125 0x000055878e66d81b in PyEval_EvalCode (co=co@entry=<code at remote 0x7fdbd437cf40>, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/ceval.c:1154 #126 0x000055878e6c088b in run_eval_code_obj (tstate=tstate@entry=0x55878ea0cf78 <_PyRuntime+166328>, co=co@entry=0x7fdbd437cf40, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}) at Python/pythonrun.c:1714 #127 0x000055878e6c1028 in run_mod (mod=mod@entry=0x55878f87d5b0, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, flags=flags@entry=0x7ffdc69abe28, arena=arena@entry=0x7fdbd42731c0) at Python/pythonrun.c:1735 #128 0x000055878e6c1134 in pyrun_file (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/brandtbucher/env311/bin/pytest') at remote 0x7fdbd420c380>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fdbd43db770>, '__file__': '/home/brandtbucher/env311/bin/pytest', '__cached__': None, 're': <module at remote 0x7fdbd4272810>, 'sys': <module at remote 0x7fdbd43cbb30>, 'console_main': <function at remote 0x7fdbd373bee0>}, closeit=closeit@entry=1, flags=0x7ffdc69abe28) at Python/pythonrun.c:1630 #129 0x000055878e6c4642 in _PyRun_SimpleFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:440 #130 0x000055878e6c488e in _PyRun_AnyFileObject (fp=fp@entry=0x55878f870090, filename=filename@entry='/home/brandtbucher/env311/bin/pytest', closeit=closeit@entry=1, flags=flags@entry=0x7ffdc69abe28) at Python/pythonrun.c:79 #131 0x000055878e6e808b in pymain_run_file_obj (program_name=program_name@entry='/home/brandtbucher/env311/bin/python', filename=filename@entry='/home/brandtbucher/env311/bin/pytest', skip_source_first_line=0) at Modules/main.c:360 #132 0x000055878e6e83e3 in pymain_run_file (config=config@entry=0x55878e9f2fc0 <_PyRuntime+59904>) at Modules/main.c:379 #133 0x000055878e6e94b8 in pymain_run_python (exitcode=exitcode@entry=0x7ffdc69abf84) at Modules/main.c:601 #134 0x000055878e6e9522 in Py_RunMain () at Modules/main.c:680 #135 0x000055878e6e959c in pymain_main (args=args@entry=0x7ffdc69abfe0) at Modules/main.c:710 #136 0x000055878e6e9668 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:734 #137 0x000055878e496d06 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
Confirmed that it's not
coverage. Setting a simple trace function at the top of the file is enough to make things crash:def trace(frame, event, arg): return trace sys.settrace(trace)
Not sure if it's a symptom or the bug, but
__lltrace__tells me that theRETURN_VALUE(83) opcode in my trace function is being incremented at some point (just before the crash) and executed asIMPORT_STAR(84). The VM then runs off the end of the bytecode, prints that it's reached an unknown opcode, and then I get a segfault during a garbage collection sometime later.I'm fairly confident that the specialization machinery isn't responsible for the changing opcode (I disabled specialization of
LOAD_GLOBAL, and even tried another version of thetracefunction that usedLOAD_FAST... same behavior). Pretty bizarre, so far, but I feel like I'm at least making some progress.Another puzzle piece: the segfault only happens if the file is loaded from a cached PYC. Forcing recompilation with
PYTHONDONTWRITEBYTECODE=1seems to get rid of the crash, and if you modify the file manually, the first run (not cached) will succeed, but the rest (cached) will crash.Again, these could just be action at a distance from the VM getting into some bad state earlier in the program, or changing when and how GC/tracing events happen. But they're interesting, nonetheless.
28 remaining items
Update: We've been working on this at the CPython core dev sprint at Google this week. I finally found the root issue, which is that a poorly-timed GC collection just before tracing the resumption of a generator can cause two duplicate frame objects to be created, which corrupts the VM's internal state in all sorts of fun ways. Thanks for your help reporting and debugging this!
Reacted by Alex Waygood, davfsa and wookie184- added a commit that references this issue
on Oct 6, 2022 Confirmed that this fixes the segfaults in the Hikari test suite. This fix will ship in 3.11.0!
- added a commit that references this issue
on Oct 7, 2022 Confirmed that this fixes the segfaults in the Hikari test suite. This fix will ship in 3.11.0!
Thank you for finding the root cause and fixing it! 🥳
@brandtbucher Since gh-98002 is merged, can this issue be closed or have its
release-blockerlabel removed?I'd prefer to leave it open, and let @pablogsal close it once the cherry-pick to 3.11.0 is finished.
- added a commit that references this issue
on Oct 8, 2022 - added a commit that references this issue
on Oct 11, 2022 - added a commit that references this issue
on Oct 22, 2022
Metadata
Metadata
Assignees
Labels
Projects
- Status
Crash report
I am not really sure under what circumstances this can occur, as I have not been able to isolate the specific piece of code that triggers this. Thankfully, when trying to re-create the segfault to capture a core dump locally, I have found that I instead got a full traceback, which you can find bellow
Error messages
Segfault
Traceback
Your environment