Repository navigation
gdb stack trace incomplete when enabling Perf support #130856
Description
Bug report
Bug description:
GDB stack trace is incomplete when enabling Perf support under specific optimization options on x86_64. It seems that through various changes the stack trace was broken and/or fixed. I've bisected the relevant commits.
Perf 6.13.4, GDB 16.2, GCC 14.2.1 on x86_64, Fedora 41.
With frame pointers using -X perf:
./configure --enable-shared --without-static-libpython && CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -<optmization>" make
Run LD_LIBRARY_PATH=$PWD gdb -args ./python -X perf -m test -vv test_sys and within the GDB shell run.
Press ctrl+c during execution and run bt. On specific configurations the stack trace is trimmed, usually right after the unknown frames.
python3.12:
- -O3, -O1, -O0, -Og:
Complete gdb stack trace. - -O2:
Incomplete gdb stack trace (stops at the unknown frames). (Didn't work from the beginning, resolved in 3.13)
python3.13:
-
-O3, -O2, -O0, -Og:
Complete gdb stack trace. -
-O1:
Incomplete gdb stack trace (stops at the unknown frames). (regressed on 3.13 here and fixed again in 3.14 here)
python3.14:
-
-O3, -O2, -Og:
Incomplete gdb stack trace (stops at the unknown frames). (-O3, -O2 regressed here, -Og regressed here) -
-O1, -O0:
Complete gdb stack trace.
Without frame pointers using -X perf_jit
./configure --enable-shared --without-static-libpython && CFLAGS="-<optmization>" make
Run LD_LIBRARY_PATH=$PWD gdb -args ./python -X perf_jit -m test -vv test_sys and within the GDB shell run.
Press ctrl+c during execution and run bt. On specific configurations the stack trace is trimmed, usually right after the unknown frames.
python3.12:
- Not supported
python3.13:
-
-O3:
Incomplete gdb stack trace (stops at the unknown frames). (Worked initially, regressed here, fixed in 3.14) -
-O2, -O1, -Og:
Complete gdb stack trace. -
-O0:
Complete gdb stack trace.
Python functions are not visible in Perf output.
python3.14:
-
-O3, -O2, -O1:
Complete gdb stack trace. -
-Og:
Incomplete gdb stack trace (stops at the unknown frames). (Regressed here, same as with frame pointers) -
-O0:
Complete gdb stack trace.
Python functions are not visible in Perf output.
CPython versions tested on:
CPython main branch, 3.14, 3.13, 3.12
Operating systems tested on:
Linux
Activity
stratakis commentedon Mar 4, 2025
cc @pablogsal
stratakis commentedon Mar 5, 2025
Went ahead and tested aarch64 as well. The machine I tested is a VM running on bare metal (one of the buildbots). Unfortunately it seems that GDB doesn't work there at all.
Perf 6.13.4, GDB 16.2, GCC 14.2.1 on aarch64, Fedora 41.
With frame pointers using -X perf:
Python functions are visible in Perf output with frame pointers.
python3.12:
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
python3.13:
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
python3.14:
Backtrace stopped: not enough registers or memory available to unwind further
Backtrace stopped: Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Without frame pointers using -X perf_jit
Python functions are not visible in Perf output with -O0.
python3.12:
python3.13:
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
python3.14:
Backtrace stopped: not enough registers or memory available to unwind further
Backtrace stopped: Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
pablogsal commentedon Mar 10, 2025
I have been investigating this and it's a bit of a mess. Different versions of gdb can basically leverage different heuristics to recontruct the stack trace and the different changes to ceval after the optimization work make this not play well with the algorithms they use. It's basically an impossible battle. The only way is to add gdb support for jit compilers but that slows down Python considerably as gdb demands entire ELF files in memory to be created wich kind of defies the purpose of having a low performance profiler active :(
So I am afraid we need to close this as won't fix 😢
stratakis commentedon Mar 10, 2025
Makes sense, understood. Maybe in the future some better way can be figured out to resolve this.