flamegraph

[cargo-]flamegraph

A Rust-powered flamegraph generator with additional support for Cargo projects! It can be used to profile anything, not just Rust projects! No perl or pipes required <3

Built on top of @jonhoo's wonderful Inferno all-rust flamegraph generation library!

[!TIP] You might want to also try samply, which provides a more interactive UI using a seamless integration with Firefox's Profiler web UI. It is also written in Rust and has better macOS support.

Quick Start

Install it, and run

# Rust projects
cargo flamegraph

# Arbitrary binaries
flamegraph -- /path/to/binary

How to use flamegraphs: what's a flamegraph, and how can I use it to guide systems performance work?

Installation

[cargo-]flamegraph supports

• Linux: relies on perf
• MacOS: relies on xctrace
• Windows: native support with the blondie library; also works with dtrace on Windows

cargo install flamegraph will make the flamegraph and cargo-flamegraph binaries available in your cargo binary directory. On most systems this is something like ~/.cargo/bin.

Linux

Note: If you're using lld (which is the default since Rust 1.90.0) or mold on Linux, you must use the --no-rosegment flag. Otherwise perf will not be able to generate accurate stack traces (explanation).

For example, Rust 1.90.0 and later:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-Clink-arg=-Wl,--no-rosegment"]

for explicitly configured lld on older toolchains:

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-Clink-arg=-fuse-ld=lld", "-Clink-arg=-Wl,--no-rosegment"]

and for mold:

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-Clink-arg=-fuse-ld=/usr/local/bin/mold", "-Clink-arg=-Wl,--no-rosegment"]

Debian (x86 and aarch)

Note: Debian bullseye packages an outdated version of Rust which does not meet flamegraph's requirements. You should use rustup to install an up-to-date version of Rust, or upgrade to Debian bookworm or newer.

sudo apt install -y linux-perf

Ubuntu (x86)

sudo apt install linux-tools-common linux-tools-generic linux-tools-`uname -r`

Ubuntu (aarch)

sudo apt install linux-tools-common linux-tools-generic linux-tools-`uname -r`

Note: The perf binary is not packaged for all kernel versions. A check and workaround is below.

# Check if the perf binary is missing:
ls -l /usr/lib/linux-tools/`uname -r`/ | grep perf

# If it is there, you can stop here.
# If is missing, check if you have the tools for another kernel version installed:
ls -l /usr/lib/linux-tools/

# If you do, check it contains perf:
# (replace <FROM_KERNEL>)
ls -l /usr/lib/linux-tools/<FROM_KERNEL>/

# If you do, symlink it. This has been tested with 6.14.0-1014-aws running perf from 6.8.0-85-generic
# (replace <FROM_KERNEL> as before)
sudo ln -s /usr/lib/linux-tools/<FROM_KERNEL>/perf /usr/lib/linux-tools/`uname -r`/perf

Ubuntu/Ubuntu MATE (Raspberry Pi)

sudo apt install linux-tools-raspi

Pop!_OS

sudo apt install linux-tools-common linux-tools-generic

Windows

Blondie Backend

This is enabled by default. Windows is supported out-of-the-box, thanks to Nicolas Abram's excellent blondie library.

DTrace on Windows

Alternatively, one can install DTrace on Windows. If found, flamegraph will always prefer using dtrace over the built-in Windows support.

Shell auto-completion

At the moment, only flamegraph supports auto-completion. Supported shells are bash, fish, zsh, powershell and elvish. cargo-flamegraph does not support auto-completion because it is not as straight-forward to implement for custom cargo subcommands. See #153 for details.

How you enable auto-completion depends on your shell, e.g.

flamegraph --completions bash > $XDG_CONFIG_HOME/bash_completion # or /etc/bash_completion.d/

Examples

…

Usage

flamegraph is quite simple. cargo-flamegraph is more sophisticated:

…

Then open the resulting flamegraph.svg with a browser, because most image viewers do not support interactive svg-files.

Enabling perf for use by unprivileged users

To enable perf without running as root, you may lower the perf_event_paranoid value in proc to an appropriate level for your environment. The most permissive value is -1 but may not be acceptable for your security needs etc...

echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid

Improving output when running with --release

Due to optimizations etc... sometimes the quality of the information presented in the flamegraph will suffer when profiling release builds.

To counter this to some extent, you may either set the following in your Cargo.toml file:

[profile.release]
debug = true

Or set the environment variable CARGO_PROFILE_RELEASE_DEBUG=true.

Please note that tests, unit tests and benchmarks use the bench profile in release mode (see here).

Usage with benchmarks

In order to perf existing benchmarks, you should set up a few configs. Set the following in your Cargo.toml file to run benchmarks:

[profile.bench]
debug = true

Use custom paths for perf and dtrace

If PERF or DTRACE environment variable is set, it'll be used as corresponding tool command. For example, to use perf from ~/bin:

env PERF=~/bin/perf flamegraph /path/to/my/binary

Use custom addr2line binary for perf

It has been reported that addr2line can run very slowly in several issues (#74, #199, #294). One solution is to use gimli-rs/addr2line instead of the system addr2line binary. This is suggested in this comment, and you can follow the steps below to set it up:

cargo install addr2line --features=bin

Systems Performance Work Guided By Flamegraphs

Flamegraphs are used to visualize where time is being spent in your program. Many times per second, the threads in a program are interrupted and the current location in your code (based on the thread's instruction pointer) is recorded, along with the chain of functions that were called to get there. This is called stack sampling. These samples are then processed and stacks that share common functions are added together. Then an SVG is generated showing the call stacks that were measured, widened to the proportion of all stack samples that contained them.

The y-axis shows the stack depth number. When looking at a flamegraph, the main function of your program will be closer to the bottom, and the called functions will be stacked on top, with the functions that they call stacked on top of them, etc...

The x-axis spans all of the samples. It does not show the passing of time from left to right. The left to right ordering has no meaning.

The width of each box shows the total time that that function is on the CPU or is part of the call stack. If a function's box is wider than others, that means that it consumes more CPU per execution than other functions, or that it is called more than other functions.

The color of each box isn't significant, and is chosen at random.

Flamegraphs are good for visualizing where the most expensive parts of your program are at runtime, which is wonderful because...

Humans are terrible at guessing about performance!

Especially people who come to Rust from C and C++ will often over-optimize things in code that LLVM is able to optimize away on its own. It's always better to write Rust in a clear and obvious way, before beginning micro-optimizations, allocation-minimization, etc...

Lots of things that would seem like they would have terrible performance are actually cheap or free in Rust. Closures are fast. Initialization on the stack before moving to a Box is often compiled away. Clones are often compiled away. So, clone() away instead of fighting for too long to get the compiler to be happy about ownership!

Then make a flamegraph to see if any of that was actually expensive.

Flamegraphs Are the Beginning, Not the End

Flamegraphs show you the things that are taking up time, but they are a sampling technique to be used for high-level and initial looks at the system under measurement. They are great for finding the things to look into more closely, and often it will be obvious how to improve something based on its flamegraph, but they are really more for choosing the target to perform optimization on than an optimization measurement tool in itself. They are coarse-grained, and difficult to diff (although this may be supported soon). Also, because flamegraphs are based on the proportion of total time that something takes, if you accidentally make something else really slow, it will show all other things as smaller on the flamegraph, even though the entire program runtime is much slower, the items you were hoping to optimize look smaller.

It is a good idea to use Flamegraphs to figure out what you want to optimize, and then set up a measurement environment that allows you to determine that an improvement has actually happened.

• use flamegraphs to find a set of optimization targets
• create benchmarks for these optimization targets, and if appropriate use something like cachegrind and cg_diff to measure cpu instructions and diff them against the previous version.
• Measuring CPU instructions is often better than measuring the time it takes to run a workload in many cases, because it's possible that a background task on your machine ran and caused something to slow down in terms of physical time, but if you actually made an implementation faster, it is likely to have a stronger correlation with reduced total CPU instructions.
• Time spent on the CPU is not the full picture, as time is spent waiting for IO to complete as well, which does not get accounted with tools like perf that only measure what's consuming time on the CPU. Check out Brendan Gregg's article on Off-Cpu Accounting for more information about this!

Performance Theory 101: Basics of Quantitative Engineering

• Use realistic workloads on realistic hardware, or your data doesn't necessarily correspond very much with what will be happening in production
• All of our guesses are wrong to some extent, so we have to measure the effects of our work. Often the simple code that doesn't seem like it should be fast is actually way faster than code that looks optimized. We need to measure our optimizations to make sure that we didn't make our code both harder to read AND slower.
• Measure before you change anything, and save the results in a safe place! Many profiling tools will overwrite their old output when you run them again, so make sure you take care to save the d