Skip to content

s390x RHEL7 got crash on the refleak test #102351

Closed
Closed
s390x RHEL7 got crash on the refleak test#102351
Assignees
carljm
Labels
type-crashA hard crash of the interpreter, possibly with a core dump

Description

@corona10
corona10
opened on Mar 1, 2023

link: https://buildbot.python.org/all/#/builders/217/builds/1082/steps/5/logs/stdio

test_multiple_watchers (test.test_capi.test_watchers.TestFuncWatchers.test_multiple_watchers) ... ok
./Include/object.h:654: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x3ff7e7ffd80 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Current thread 0x000003ff86cf7700 (most recent call first):
  Garbage-collecting
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/support/__init__.py", line 2085 in _hook
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/setup.py", line 84 in _test_audit_hook
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/test_capi/test_watchers.py", line 479 in test_watcher_raises_error
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/case.py", line 623 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/case.py", line 678 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/unittest/runner.py", line 208 in run
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/support/__init__.py", line 1104 in _run_suite
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/support/__init__.py", line 1230 in run_unittest
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/refleak.py", line 89 in dash_R
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/runtest.py", line 315 in _runtest_inner2
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/main.py", line 353 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/main.py", line 756 in _main
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/main.py", line 711 in main
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/libregrtest/main.py", line 775 in main
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/runpy.py", line 88 in _run_code
  File "/home/dje/cpython-buildarea/pull_request.edelsohn-rhel-z.refleak/build/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testcapi, _testmultiphase, _testsinglephase, _testinternalcapi (total: 4)
make: *** [buildbottest] Aborted
program finished with exit code 2

Linked PRs

Activity

  1. corona10
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Mar 1, 2023
  2. corona10
    changed the title [-]s390x RHEL7 got crash on the refleak test.[/-] [+]s390x RHEL7 got crash on the refleak test[/+] on Mar 1, 2023

  3. corona10 commented on Mar 1, 2023

    @corona10
    corona10
    on Mar 1, 2023
    MemberAuthor

    I will try to debug issue with QEMU..

  4. corona10 commented on Mar 1, 2023

    @corona10
    corona10
    on Mar 1, 2023
    MemberAuthor

    Okay no way to setup through QEMU since I can not get the image of s390x RHEL image.
    So I will try to contract with the machine manager.

  5. corona10 commented on Mar 1, 2023

    @corona10
    corona10
    on Mar 1, 2023
    MemberAuthor

    And also @vstinner share useful approach for this issue: https://pythondev.readthedocs.io/debug_tools.html#debug-crash-in-garbage-collection-visit-decref

  6. corona10
    assigned
    corona10
    and unassigned
    corona10
    on Mar 1, 2023

  7. iritkatriel commented on Mar 1, 2023

    @iritkatriel
    iritkatriel
    on Mar 1, 2023
    Member

    @erlend-aasland

    I've reproduced it on a Mac and bisected to:

    efc985a714b6f43c43ae629183f95618054422ae is the first bad commit
commit efc985a714b6f43c43ae629183f95618054422ae
Author: Erlend E. Aasland <erlend.aasland@protonmail.com>
Date:   Thu Feb 23 16:03:13 2023 +0100

    gh-93649: Split exception tests from _testcapimodule.c (GH-102173)
    
    
    
    Automerge-Triggered-By: GH:erlend-aasland

 Lib/test/test_capi/test_exceptions.py | 145 +++++++++++++++++
 Lib/test/test_capi/test_misc.py       | 131 +--------------
 Modules/Setup.stdlib.in               |   2 +-
 Modules/_testcapi/exceptions.c        | 277 ++++++++++++++++++++++++++++++++
 Modules/_testcapi/parts.h             |   1 +
 Modules/_testcapimodule.c             | 290 +---------------------------------
 PCbuild/_testcapi.vcxproj             |   1 +
 PCbuild/_testcapi.vcxproj.filters     |   3 +
 8 files changed, 434 insertions(+), 416 deletions(-)
 create mode 100644 Lib/test/test_capi/test_exceptions.py
 create mode 100644 Modules/_testcapi/exceptions.c

  8. iritkatriel commented on Mar 1, 2023

    @iritkatriel
    iritkatriel
    on Mar 1, 2023
    Member

    But I think it may be a red herring (when I touch stuff the crash goes away). Maybe we just need to bump the magic number.

  9. iritkatriel
    added a commit that references this issue on Mar 1, 2023

    pythongh-102351: bump MAGIC_NUMBER

    862619a

  10. iritkatriel commented on Mar 1, 2023

    @iritkatriel
    iritkatriel
    on Mar 1, 2023
    Member

    I suspect this PR needed the magic number bump: #101933

  11. iritkatriel commented on Mar 1, 2023

    @iritkatriel
    iritkatriel
    on Mar 1, 2023
    Member

    But I think it may be a red herring (when I touch stuff the crash goes away). Maybe we just need to bump the magic number.

    Bumping magic number didn't solve this: #102359.

  12. erlend-aasland commented on Mar 2, 2023

    @erlend-aasland
    erlend-aasland
    on Mar 2, 2023
    Contributor

    @iritkatriel, thanks for the heads-up, I'll have a look!

  13. erlend-aasland commented on Mar 2, 2023

    @erlend-aasland
    erlend-aasland
    on Mar 2, 2023
    Contributor

    @iritkatriel: what's your repro on macOS? I can't reproduce it (yet) on my M1 running Ventura 13.1.

  14. iritkatriel commented on Mar 2, 2023

    @iritkatriel
    iritkatriel
    on Mar 2, 2023
    Member

    I was running python -m test -j2 -R3:3 test_capi

    running just a submodule of test_capi didn’t fail. I think something to do with module import/setup?

  15. corona10 commented on Mar 2, 2023

    @corona10
    corona10
    on Mar 2, 2023
    MemberAuthor

    When I debug the recount: -2459565876494606883
    It could be overflowed issue or an uninitialized issue.

  16. corona10 commented on Mar 2, 2023

    @corona10
    corona10
    on Mar 2, 2023
    MemberAuthor

    When I debug the recount: -2459565876494606883

    Ah it's because already freed.

  17. iritkatriel commented on Mar 2, 2023

    @iritkatriel
    iritkatriel
    on Mar 2, 2023
    Member

    I think it might be the same issue as #102350.
    I narrowed it down to that test, and if I update my branch with that last commit the crash is gone.

  18. corona10 commented on Mar 2, 2023

    @corona10
    corona10
    on Mar 2, 2023
    MemberAuthor

    I narrowed it down to that test, and if I update my branch with that last commit the crash is gone.

    Ah for my case, it's still happening with the latest commit.

  19. iritkatriel commented on Mar 2, 2023

    @iritkatriel
    iritkatriel
    on Mar 2, 2023
    Member

    Clean build?

  20. corona10 commented on Mar 2, 2023

    @corona10
    corona10
    on Mar 2, 2023
    MemberAuthor

    Clean build?

    Yes with the clean-build.
    Please read my comment when I triggered the refleak test with the PR.
    #102350 (comment)
    This issue was created from the comment.

  21. iritkatriel commented on Mar 2, 2023

    @iritkatriel
    iritkatriel
    on Mar 2, 2023
    Member

    Yes, I just got the crash again with main too. Sorry about the noise.

  22. iritkatriel commented on Mar 2, 2023

    @iritkatriel
    iritkatriel
    on Mar 2, 2023
    Member

    I think the problem comes from test_watcher_raises_error in Lib/test/test_capi/test_watchers.py.

    This test registers a dictionary watcher which raises an exception. That takes us to _PyErr_WriteUnraisableMsg, where Py_DECREF(hook_args); fails because the last item in hook_args (the obj, i.e., the dictionary) has refcnt 0.

    This only happens when I run it with ./python.exe -m test -v -j2 -R3:3 test_capi. If I remove the -j2 it doesn't fail.

    I added assert(Py_REFCNT((PyObject*)mp) > 0); at the beginning of _PyDict_NotifyEvent, and it failed (during make) which seem wrong (apparently this is fine, for PyDict_EVENT_DEALLOCATED events).

  23. carljm
    self-assigned this
    on Mar 3, 2023

  24. carljm commented on Mar 3, 2023

    @carljm
    carljm
    on Mar 3, 2023
    Member

    Further debugging confirmed that the root cause of this issue is #102381. Fixing all code/func/dict dealloc callbacks to temporarily resurrect the object before calling the callback, and to not save references to the object in unraisablehook, makes this issue go away.

  25. carljm commented on Mar 3, 2023

    @carljm
    carljm
    on Mar 3, 2023
    Member

    For posterity: the specific repro here requiring -j2 -R seems to be an artifact of those options causing a GC run to tend to occur while an error-raising func watcher is active, and that GC run deallocates a cycle of unrelated objects from the datetime.py pure-Python fallbacks, including some functions.

  26. carljm commented on Mar 8, 2023

    @carljm
    carljm
    on Mar 8, 2023
    Member

    Fixed in #102382

  27. carljm
    closed this as completedon Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

type-crashA hard crash of the interpreter, possibly with a core dump

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions