You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Easier flamegraph / profiling support for datafusion benchmarks #2174
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to make flamegraphs to understand performance of PRs like #2146 from @yjshen and other benchmarks in a cross platform, easy to use way. I don't want to switch between Instruments on Mac to perf on linux
It is not easy to get them (see this mailing list topic, and I hit the same "flamegraph takes forever to make" issue when I tried to run this on my development machine
Describe the solution you'd like
Use code in pprof crate -- @mkmik did this in influxdb_iox have used it to good effect in IOx
So this would look like adding an optional feature pprof to benchmark program that would generate flamegraphs and profile.proto format output. You would run it like this:
# --profile also writes flamegraph.svg and profile.proto files
target/release/tpch benchmark datafusion --profile --iterations 3 --path /tpch-parquet --format parquet --query 1
You could use the profile.proto in the (excellent) golang tooling like:
go tool pprof cpu.prof
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
I did try to run cargo flamegraph ... and it was spending an absurd amount of time calling addr2line (as reported in the mailing list topic). Perhaps due to some old version of prof or something on the GCP debian machine I was using
FWIW, I never have experienced the in the mailinglist mentioned slowness of flamegraph (or the Hotspot UI I mentioned there), using a recent Pop!_OS version.
Perhaps this might help? FWIW I also use the hotspot UI, which also has some nice functionality for digging into the profiles, viewing thread activity, etc... It chugs a bit on larger profiles (multiple GB) but otherwise works well enough for my purposes...
Edit: turns out cargo-flamegraph is already using inferno. Unfortunately pprof also uses it, so switching to using pprof may not help matters...
@alambsomeone drilled deep into why perf is slow.
TL;DR--GPLv2-license-compatible perf is really inefficient. @Dandandan perf can be compiled with libbfd (fast) or without (slow). I am running Ubuntu but the author above only reports slow results on Debian. Maybe Pop!_OS is cheating and compiling with libbfd?
Summing up these results:
@alamb applying the solution from IOx sounds like a great way to go. I haven't been on this for awhile (busy) but I probably would've taken a more laborious approach, perhaps reinventing a wheel.
FWIW I found the pprof approach to be very useful for online performance analysis (e.g on demand on production services), and less useful when running ad-hoc performance analysis runs locally (e.g. benchmarks)
Activity
tustvold commentedon Apr 6, 2022
Have you checked out cargo-flamegraph? I think it might fit the bill?
alamb commentedon Apr 6, 2022
I did try to run
cargo flamegraph... and it was spending an absurd amount of time callingaddr2line(as reported in the mailing list topic). Perhaps due to some old version of prof or something on the GCP debian machine I was usingDandandan commentedon Apr 6, 2022
FWIW, I never have experienced the in the mailinglist mentioned slowness of flamegraph (or the Hotspot UI I mentioned there), using a recent Pop!_OS version.
tustvold commentedon Apr 6, 2022
Perhaps this might help? FWIW I also use the hotspot UI, which also has some nice functionality for digging into the profiles, viewing thread activity, etc... It chugs a bit on larger profiles (multiple GB) but otherwise works well enough for my purposes...
Edit: turns out cargo-flamegraph is already using inferno. Unfortunately pprof also uses it, so switching to using pprof may not help matters...
bobtins commentedon Apr 6, 2022
@alamb someone drilled deep into why perf is slow.
TL;DR--GPLv2-license-compatible perf is really inefficient.
@Dandandan perf can be compiled with libbfd (fast) or without (slow). I am running Ubuntu but the author above only reports slow results on Debian. Maybe Pop!_OS is cheating and compiling with libbfd?
Summing up these results:
@alamb applying the solution from IOx sounds like a great way to go. I haven't been on this for awhile (busy) but I probably would've taken a more laborious approach, perhaps reinventing a wheel.
tustvold commentedon Apr 6, 2022
@bobtins I think you maybe meant to link to this https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement ?
The good news is a mitigation was merged at some point last year, and has apparently been released, so perhaps the GCP machines was running an old Debian version? - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=be8ecc57f180415e8a7c1cc5620c5236be2a7e56
bobtins commentedon Apr 6, 2022
@tustvold I did try out hotspot; it's really nice! It can record data (which just runs perf) or open perf files that were recorded.

bobtins commentedon Apr 6, 2022
Hm, good news, thanks for the updated info.
realno commentedon Apr 6, 2022
This is great, looking forward to seeing some results!
alamb commentedon Apr 6, 2022
Thanks for the suggestions -- I'l try the various tools on this thread and will report back
mkmik commentedon Apr 7, 2022
FWIW I found the pprof approach to be very useful for online performance analysis (e.g on demand on production services), and less useful when running ad-hoc performance analysis runs locally (e.g. benchmarks)
houqp commentedon Apr 7, 2022
@alamb have you tried cargo-flamegraph's
--no-inlineargument?mingmwang commentedon Apr 8, 2022
I had use pprof-rs to benchmark the DataFusion/Arrow Parquet reader performance. It can generate the flamegraph pictures easily.
alamb commentedon Jul 21, 2023
I don't think there is any work action for this, standard profiling tools work great. Closing