Skip to content

Easier flamegraph / profiling support for datafusion benchmarks #2174

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to make flamegraphs to understand performance of PRs like #2146 from @yjshen and other benchmarks in a cross platform, easy to use way. I don't want to switch between Instruments on Mac to perf on linux

It is not easy to get them (see this mailing list topic, and I hit the same "flamegraph takes forever to make" issue when I tried to run this on my development machine

Describe the solution you'd like
Use code in pprof crate -- @mkmik did this in influxdb_iox have used it to good effect in IOx

So this would look like adding an optional feature pprof to benchmark program that would generate flamegraphs and profile.proto format output. You would run it like this:

# --profile also writes flamegraph.svg and profile.proto files
target/release/tpch benchmark datafusion --profile --iterations 3 --path /tpch-parquet --format parquet --query 1

You could use the profile.proto in the (excellent) golang tooling like:

go tool pprof cpu.prof

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Activity

  1. self-assigned this
    on Apr 6, 2022
  2. tustvold commented on Apr 6, 2022

    @tustvold
    Contributor

    Have you checked out cargo-flamegraph? I think it might fit the bill?

  3. alamb commented on Apr 6, 2022

    @alamb
    ContributorAuthor

    I did try to run cargo flamegraph ... and it was spending an absurd amount of time calling addr2line (as reported in the mailing list topic). Perhaps due to some old version of prof or something on the GCP debian machine I was using

  4. Dandandan commented on Apr 6, 2022

    @Dandandan
    Contributor

    FWIW, I never have experienced the in the mailinglist mentioned slowness of flamegraph (or the Hotspot UI I mentioned there), using a recent Pop!_OS version.

  5. tustvold commented on Apr 6, 2022

    @tustvold
    Contributor

    Perhaps this might help? FWIW I also use the hotspot UI, which also has some nice functionality for digging into the profiles, viewing thread activity, etc... It chugs a bit on larger profiles (multiple GB) but otherwise works well enough for my purposes...

    Edit: turns out cargo-flamegraph is already using inferno. Unfortunately pprof also uses it, so switching to using pprof may not help matters...

  6. bobtins commented on Apr 6, 2022

    @bobtins

    @alamb someone drilled deep into why perf is slow.
    TL;DR--GPLv2-license-compatible perf is really inefficient.
    @Dandandan perf can be compiled with libbfd (fast) or without (slow). I am running Ubuntu but the author above only reports slow results on Debian. Maybe Pop!_OS is cheating and compiling with libbfd?
    Summing up these results:

    • me, Ubuntu 20.04: slow
    • me, recompiled perf: fast
    • @Dandandan , Pop!_OS: fast
    • @alamb , MacOS: slow

    @alamb applying the solution from IOx sounds like a great way to go. I haven't been on this for awhile (busy) but I probably would've taken a more laborious approach, perhaps reinventing a wheel.

  7. tustvold commented on Apr 6, 2022

    @tustvold
    Contributor

    @bobtins I think you maybe meant to link to this https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement ?

    The good news is a mitigation was merged at some point last year, and has apparently been released, so perhaps the GCP machines was running an old Debian version? - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=be8ecc57f180415e8a7c1cc5620c5236be2a7e56

  8. bobtins commented on Apr 6, 2022

    @bobtins

    @tustvold I did try out hotspot; it's really nice! It can record data (which just runs perf) or open perf files that were recorded.
    hotspot_3

  9. bobtins commented on Apr 6, 2022

    @bobtins

    @bobtins I think you maybe meant to link to this https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement ?

    Hm, good news, thanks for the updated info.

  10. realno commented on Apr 6, 2022

    @realno
    Contributor

    This is great, looking forward to seeing some results!

  11. alamb commented on Apr 6, 2022

    @alamb
    ContributorAuthor

    Thanks for the suggestions -- I'l try the various tools on this thread and will report back

  12. mkmik commented on Apr 7, 2022

    @mkmik
    Contributor

    FWIW I found the pprof approach to be very useful for online performance analysis (e.g on demand on production services), and less useful when running ad-hoc performance analysis runs locally (e.g. benchmarks)

  13. houqp commented on Apr 7, 2022

    @houqp
    Member

    @alamb have you tried cargo-flamegraph's --no-inline argument?

  14. mingmwang commented on Apr 8, 2022

    @mingmwang
    Contributor

    I had use pprof-rs to benchmark the DataFusion/Arrow Parquet reader performance. It can generate the flamegraph pictures easily.

  15. alamb commented on Jul 21, 2023

    @alamb
    ContributorAuthor

    I don't think there is any work action for this, standard profiling tools work great. Closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions