Skip to content

Build Python with frame pointers (-fno-omit-frame-pointer) #96174

Closed

Description

@tiran

Feature or enhancement

I propose to build CPython with -fno-omit-frame-pointer and -mno-omit-leaf-frame-pointer on compilers that support these options, at least on Linux with GCC and clang on platforms that support perf trampoline (#96143).

Pitch

The feature is needed for #96143 "Allow the linux perf profiler to see Python calls". Otherwise perf won't be able to reconstruct the call frames and perf report output isn't helpful.

Previous discussion

The feature change proposal https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer explains the usefulness of frame pointers in great detail. Code without frame pointers can be slightly more efficient because it does not have to safe a register on function call and it can repurpose the register for other uses. The potential performance impact is discussed in the Fedora change proposal, too. tl;dr Fedora testing has not shown any significant impact.

Activity

  1. tiran commented on Aug 22, 2022

    @tiran
    MemberAuthor

    CC @pablogsal @gpshead @markshannon

  2. mdboom commented on Aug 22, 2022

    @mdboom
    Contributor

    It wouldn't hurt to run this change through pyperformance to help confirm the lack of performance impact.

  3. encukou commented on Aug 22, 2022

    @encukou
    Member

    Please don't trust the unapproved Fedora change proposal too much. Especially its performance numbers: it seems that the flag was only enabled for a small subset of code when they were made.

  4. pablogsal commented on Aug 22, 2022

    @pablogsal
    Member

    Idea: we could enable it by default and drop it with --enable-optimizations

  5. gpshead commented on Aug 22, 2022

    @gpshead
    Member

    Idea: we could enable it by default and drop it with --enable-optimizations

    While I laughed when I read it, I'd actually be fine with that. We tell people to use that for the best speed. Linux distro packaging setups either use that or do an equivalent on their own at which point they'd need to adopt adding an explicit -fomit-frame-pointer to their custom build, but that is entirely up to them to manage.

    This is one of those compiler flags that makes code execution easier to passively analyze but always costs a little performance. The more registers the less impact on any given platform. ie: x86_64 should feel it more than ARM.

    We still build most everything at work with frame pointers.

  6. pablogsal commented on Aug 22, 2022

    @pablogsal
    Member

    Also, the pitch should be about unwinders in general, not only the perf work. Perf is orders of magnitude slower without frame pointers and so is any other unwinder (like libunwind). Having to parse dwarf to unwind is super heavy on I/O and CPU. Everywhere this is discussed the pitch is to sacrifice just a tiny bit of perf for better debugging and profiling experience

  7. thesamesam commented on Nov 5, 2022

    @thesamesam
    Contributor

    There's been a lot more discussion on the Fedora thread (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/OOJDAKTJB5WGMOZRXTUX7FTPFBF3H7WE/) as of the last few days - worth reading.

    As for Python, if it decides to do it, please restrict it to 64-bit arches (ideally just amd64 for now until others have been analysed).

  8. ruslaniv commented on Dec 8, 2025

    @ruslaniv

    This thread is quite old, was there any resolution on this?

  9. pablogsal commented on Dec 8, 2025

    @pablogsal
    Member

    This thread is quite old, was there any resolution on this?

    Nop, we are still blocked by an unnaceptable slowdown to make this the default.

  10. gpshead commented on Dec 8, 2025

    @gpshead
    Member

    data point: I believe Ubuntu 24.10+ and Fedora 38+ ship their python with frame pointers enabled. (ie: the tide is turning)

  11. pablogsal commented on Dec 8, 2025

    @pablogsal
    Member

    data point: I believe Ubuntu 24.10+ and Fedora 38+ ship their python with frame pointers enabled. (ie: the tide is turning)

    But not for python sadly. From https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default:

    There is a small performance penalty associated with the change, but in cases where the impact is high (such as the Python interpreter), we’ll continue to omit frame pointers until this is addressed. Our analysis suggests that the penalty on 64-bit architectures is between 1-2% in most cases. We will not make this change on 32-bit architectures where the penalty is higher.

    and:

    https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html

    Another extreme example of overhead was the Python scimark_sparse_mat_mult benchmark, which could reach 10%. Fortunately this was analyzed by Andrii Nakryiko (Meta) who found it was a unusual case of a large function where gcc switched from %rsp offsets to %rbp-relative offsets, which took more bytes to store, causing performance issues. I've heard this has since been fixed so that Python can reenable frame pointers by default.

  12. pablogsal commented on Dec 8, 2025

    @pablogsal
    Member

    I assume fedora is in the same situation. @stratakis may know more

  13. stratakis commented on Dec 8, 2025

    @stratakis
    Contributor

    I assume fedora is in the same situation. @stratakis may know more

    In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.

  14. stratakis commented on Dec 8, 2025

    @stratakis
    Contributor

    I assume fedora is in the same situation. @stratakis may know more

    In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.

    And while personally I haven't measured the performance of the Fedora Python builds before and after frame pointers, we do optimize aggressively with --enable-optimizations, --with-lto, -O3 and exploring BOLT at the moment.

    I am not sure what the best resolution would be for upstream. I've used Perf with nice results with Frame Pointers due to Pablo's work to make Python functions visible on the output, however if I can measure for example a 10% speed increase without them, I'd consider removing them and rely on the support without frame pointers which sometimes can be brittle ofc depending on kernel/compiler and other factors.

  15. pablogsal commented on Dec 8, 2025

    @pablogsal
    Member

    I assume fedora is in the same situation. @stratakis may know more

    In Fedora there have been long heated debates regarding frame pointers enablement. In the end they are applied by default including our Python builds. You can see an analysis for the Python interpreter here.

    Excellent news! I am happy to had the wrong understanding. Frame pointers is the correct choice IMHO and I wish we could enable this by default and make it opt-out

  16. pablogsal commented on Dec 8, 2025

    @pablogsal
    Member

    Maybe we should do a PEP @gpshead

  17. markshannon commented on Dec 9, 2025

    @markshannon
    Member

    With the tailcalling interpreter, and the JIT taking over a larger fraction of the execution time from the interpreter, the performance impact of frame pointers will change.
    I think it will lessen, but that's just a guess, so we'll definitely need new measurements for any PEP.

  18. pablogsal commented on Dec 9, 2025

    @pablogsal
    Member

    With the tailcalling interpreter, and the JIT taking over a larger fraction of the execution time from the interpreter, the performance impact of frame pointers will change. I think it will lessen, but that's just a guess, so we'll definitely need new measurements for any PEP.

    Absolutely. I also think the tail call interpreter should make things better but the PEP will surely need careful analysis of all the costs and consequences of such a decision.

  19. encukou commented on Apr 15, 2026

    @encukou
    Member

    For reference: PEP-831 is out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    buildThe build process and cross-buildperformancePerformance or resource usagetype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions