Skip to content

PPC64LE RHEL8 3.x buildbot is timing out on test_gdb #104735

Description

@JelleZijlstra

Buildbot https://buildbot.python.org/all/#/builders/559 has been failing since May 12 with a failure like this:

test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction)
Timeout (0:15:00)!
Thread 0x00007fff97a24d90 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/selectors.py", line 415 in select
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 2075 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/subprocess.py", line 1207 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 112 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 229 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-ppc64le/build/Lib/test/test_gdb.py", line 936 in test_pycfunction

It last passed on commit 718b132 and succeeded on a052be4, suggesting that the culprit is one of these commits:

Activity

  1. encukou commented on Jan 18, 2024

    @encukou
    Member

    @vstinner is the PEP-11 powerpc64le maintainer.
    Victor, do you need any help with this issue? I'm afraid I can't help too much, though -- I don't have a PPC64 box available.

  2. encukou commented on Jan 18, 2024

    @encukou
    Member

    Oh, sorry! It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer.
    Unfortunately I will need to step down from this role as I no longer have easy access to the hardware.

  3. vstinner commented on Jan 18, 2024

    @vstinner
    Member

    It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer.

    Yeah, ppc64le with GCC is supported at Tier-2 level: it's the config of RHEL operating system. ppc64le with clang is only supported at Tier-3 level: best effort.

    Buildbot https://buildbot.python.org/all/#/builders/559 has been failing since May 12 with a failure like this (...)

    I made multiple test_gdb changes last year. Is the ppc64le buildbot still failing with timeout time to time? If yes, does someone have links to such recent failures?

  4. vstinner commented on Jan 18, 2024

    @vstinner
    Member

    I made multiple test_gdb changes last year.

    test_gdb.test_pycfunction() is running tons of tests in a single method. I splitted test_gdb.py into a test_gdb test package (multiple .py files): commit 8f324b7 of issue gh-109972.

    If it still fails time to time, maybe we should simply increase the timeout. I noticed that some PPC64LE buildbot workers are way slower than other buildbot workers. I don't recall which ones.

  5. vstinner commented on Jan 18, 2024

    @vstinner
    Member

    Oh, sorry! It turns out that powerpc64le is listed twice in PEP-11, and in the other entry, I am listed as the maintainer. Unfortunately I will need to step down from this role as I no longer have easy access to the hardware.

    If you still care, you can get access to ppc64le machines from Fedora: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers I can give you a SSH access the ppc64 machines maintained by Red Hat, you should be able to easily reproduce the issue in a few minutes (need to clone the repo, build Python with similar build options than the worker, and run the test). Otherwise, we can just downgrade ppc64le+gcc to Tier-2, that's ok.

  6. encukou commented on Jan 24, 2024

    @encukou
    Member

    Let's do both: downgrade to tier 3, but I'll keep an eye on stable buildbots, and use Fedora infra when I around get to investigating this issue.

  7. vstinner commented on Jan 24, 2024

    @vstinner
    Member

    PPC64LE RHEL8 3.x buildbot is timing out on test_gdb

    I lost track of this issue. Does the issue still occur time to time?

  8. encukou commented on Jan 25, 2024

    @encukou
    Member

    No.
    And the buildbot logs from May are gone, so it wouldn't be easy to investigate this. I don't think it's worth it to keep the issue open.

  9. vstinner commented on Jan 25, 2024

    @vstinner
    Member

    Ok, thanks for keeping track of that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions