You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Build Python with frame pointers (-fno-omit-frame-pointer) #96174
I propose to build CPython with -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer on compilers that support these options, at least on Linux with GCC and clang on platforms that support perf trampoline (#96143).
Pitch
The feature is needed for #96143 "Allow the linux perf profiler to see Python calls". Otherwise perf won't be able to reconstruct the call frames and perf report output isn't helpful.
Previous discussion
The feature change proposal https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer explains the usefulness of frame pointers in great detail. Code without frame pointers can be slightly more efficient because it does not have to safe a register on function call and it can repurpose the register for other uses. The potential performance impact is discussed in the Fedora change proposal, too. tl;dr Fedora testing has not shown any significant impact.
Idea: we could enable it by default and drop it with --enable-optimizations
While I laughed when I read it, I'd actually be fine with that. We tell people to use that for the best speed. Linux distro packaging setups either use that or do an equivalent on their own at which point they'd need to adopt adding an explicit -fomit-frame-pointer to their custom build, but that is entirely up to them to manage.
This is one of those compiler flags that makes code execution easier to passively analyze but always costs a little performance. The more registers the less impact on any given platform. ie: x86_64 should feel it more than ARM.
We still build most everything at work with frame pointers.
Also, the pitch should be about unwinders in general, not only the perf work. Perf is orders of magnitude slower without frame pointers and so is any other unwinder (like libunwind). Having to parse dwarf to unwind is super heavy on I/O and CPU. Everywhere this is discussed the pitch is to sacrifice just a tiny bit of perf for better debugging and profiling experience
There is a small performance penalty associated with the change, but in cases where the impact is high (such as the Python interpreter), we’ll continue to omit frame pointers until this is addressed. Our analysis suggests that the penalty on 64-bit architectures is between 1-2% in most cases. We will not make this change on 32-bit architectures where the penalty is higher.
Another extreme example of overhead was the Python scimark_sparse_mat_mult benchmark, which could reach 10%. Fortunately this was analyzed by Andrii Nakryiko (Meta) who found it was a unusual case of a large function where gcc switched from %rsp offsets to %rbp-relative offsets, which took more bytes to store, causing performance issues. I've heard this has since been fixed so that Python can reenable frame pointers by default.
I assume fedora is in the same situation. @stratakis may know more
In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.
I assume fedora is in the same situation. @stratakis may know more
In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.
And while personally I haven't measured the performance of the Fedora Python builds before and after frame pointers, we do optimize aggressively with --enable-optimizations, --with-lto, -O3 and exploring BOLT at the moment.
I am not sure what the best resolution would be for upstream. I've used Perf with nice results with Frame Pointers due to Pablo's work to make Python functions visible on the output, however if I can measure for example a 10% speed increase without them, I'd consider removing them and rely on the support without frame pointers which sometimes can be brittle ofc depending on kernel/compiler and other factors.
I assume fedora is in the same situation. @stratakis may know more
In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.
Excellent news! I am happy to had the wrong understanding. Frame pointers is the correct choice IMHO and I wish we could enable this by default and make it opt-out
With the tailcalling interpreter, and the JIT taking over a larger fraction of the execution time from the interpreter, the performance impact of frame pointers will change.
I think it will lessen, but that's just a guess, so we'll definitely need new measurements for any PEP.
With the tailcalling interpreter, and the JIT taking over a larger fraction of the execution time from the interpreter, the performance impact of frame pointers will change. I think it will lessen, but that's just a guess, so we'll definitely need new measurements for any PEP.
Absolutely. I also think the tail call interpreter should make things better but the PEP will surely need careful analysis of all the costs and consequences of such a decision.
Activity
tiran commentedon Aug 22, 2022
CC @pablogsal @gpshead @markshannon
mdboom commentedon Aug 22, 2022
It wouldn't hurt to run this change through pyperformance to help confirm the lack of performance impact.
encukou commentedon Aug 22, 2022
Please don't trust the unapproved Fedora change proposal too much. Especially its performance numbers: it seems that the flag was only enabled for a small subset of code when they were made.
pablogsal commentedon Aug 22, 2022
Idea: we could enable it by default and drop it with --enable-optimizations
gpshead commentedon Aug 22, 2022
While I laughed when I read it, I'd actually be fine with that. We tell people to use that for the best speed. Linux distro packaging setups either use that or do an equivalent on their own at which point they'd need to adopt adding an explicit
-fomit-frame-pointerto their custom build, but that is entirely up to them to manage.This is one of those compiler flags that makes code execution easier to passively analyze but always costs a little performance. The more registers the less impact on any given platform. ie: x86_64 should feel it more than ARM.
We still build most everything at work with frame pointers.
pablogsal commentedon Aug 22, 2022
Also, the pitch should be about unwinders in general, not only the perf work. Perf is orders of magnitude slower without frame pointers and so is any other unwinder (like libunwind). Having to parse dwarf to unwind is super heavy on I/O and CPU. Everywhere this is discussed the pitch is to sacrifice just a tiny bit of perf for better debugging and profiling experience
thesamesam commentedon Nov 5, 2022
There's been a lot more discussion on the Fedora thread (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/OOJDAKTJB5WGMOZRXTUX7FTPFBF3H7WE/) as of the last few days - worth reading.
As for Python, if it decides to do it, please restrict it to 64-bit arches (ideally just amd64 for now until others have been analysed).
ruslaniv commentedon Dec 8, 2025
This thread is quite old, was there any resolution on this?
pablogsal commentedon Dec 8, 2025
Nop, we are still blocked by an unnaceptable slowdown to make this the default.
gpshead commentedon Dec 8, 2025
data point: I believe Ubuntu 24.10+ and Fedora 38+ ship their python with frame pointers enabled. (ie: the tide is turning)
pablogsal commentedon Dec 8, 2025
But not for python sadly. From https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default:
and:
https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
pablogsal commentedon Dec 8, 2025
I assume fedora is in the same situation. @stratakis may know more
stratakis commentedon Dec 8, 2025
In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.
stratakis commentedon Dec 8, 2025
And while personally I haven't measured the performance of the Fedora Python builds before and after frame pointers, we do optimize aggressively with --enable-optimizations, --with-lto, -O3 and exploring BOLT at the moment.
I am not sure what the best resolution would be for upstream. I've used Perf with nice results with Frame Pointers due to Pablo's work to make Python functions visible on the output, however if I can measure for example a 10% speed increase without them, I'd consider removing them and rely on the support without frame pointers which sometimes can be brittle ofc depending on kernel/compiler and other factors.
pablogsal commentedon Dec 8, 2025
Excellent news! I am happy to had the wrong understanding. Frame pointers is the correct choice IMHO and I wish we could enable this by default and make it opt-out
pablogsal commentedon Dec 8, 2025
Maybe we should do a PEP @gpshead
markshannon commentedon Dec 9, 2025
With the tailcalling interpreter, and the JIT taking over a larger fraction of the execution time from the interpreter, the performance impact of frame pointers will change.
I think it will lessen, but that's just a guess, so we'll definitely need new measurements for any PEP.
pablogsal commentedon Dec 9, 2025
Absolutely. I also think the tail call interpreter should make things better but the PEP will surely need careful analysis of all the costs and consequences of such a decision.
encukou commentedon Apr 15, 2026
For reference: PEP-831 is out!