bpftune

bpftune - BPF driven auto-tuning

bpftune aims to provide lightweight, always-on auto-tuning of system behaviour. The key benefit it provides are

• by using BPF observability features, we can continuously monitor and adjust system behaviour
• because we can observe system behaviour at a fine grain (rather than using coarse system-wide stats), we can tune at a finer grain too (individual socket policies, individual device policies etc)

The problem

The Linux kernel contains a large number of tunables; these often take the form of sysctl(8) parameters, and are usually introduced for situations where there is no one "right" answer for a configuration choice. The number of tunables available is quite daunting. On a 6.2 kernel we see

# sysctl --all 2>/dev/null|wc -l
1624

See here for an excellent writeup on network-related tunables..

At the same time, individual systems get a lot less care and adminstrator attention than they used to; phrases like "cattle not pets" exemplify this. Given the modern cloud architectures used for most deployments, most systems never have any human adminstrator interaction after initial provisioning; in fact given the scale requirements, this is often an explicit design goal - "no ssh'ing in!".

These two observations are not unrelated; in an earlier era of fewer, larger systems, tuning by administrators was more feasible.

These trends - system complexity combined with minimal admin interaction suggest a rethink in terms of tunable management.

A lot of lore accumulates around these tunables, and to help clarify why we developed bpftune, we will use a straw-man version of the approach taken with tunables:

"find the set of magic numbers that will work for the system forever"

This is obviously a caricature of how administrators approach the problem, but it does highlight a critical implicit assumption - that systems are static.

And that gets to the "BPF" in bpftune; BPF provides means to carry out low-overhead observability of systems. So not only can we observe the system and tune appropriately, we can also observe the effect of that tuning and re-tune if necessary.

Key design principles

• Minimize overhead. Use observability features sparingly; do not trace very high frequency events.
• Be explicit about policy changes providing both a "what" - what change was made - and a "why" - how does it help? syslog logging makes policy actions explicit with explanations
• Get out of the way of the administrator. We can use BPF observability to see if the admin sets tunable values that we are auto-tuning; if they do, we need to get out of the way and disable auto-tuning of the related feature set.
• Don't replace tunables with more tunables! bpftune is designed to be zero configuration; there are no options, and we try to avoid magic numbers where possible.
• Use push-pull approaches. For example, with tcp buffer sizing, we often want to get out of the way of applications and bump up tcp sndbuf and rcvbuf, but at a certain point we run the risk of exhausting TCP memory. We can however monitor if we are approaching TCP memory pressure and if so we can tune down values that we've tuned up. In this way, we can let the system find a balance between providing resources and exhausting them. In some cases, we won't need to tune up values; they may be fine as they are. But in other cases these limits block optimal performance, and if they are raised safely - with awareness of global memory limits - we can get out the way of improved performance. Another concern is that increasing buffer size leads to latency - to handle that, we correlate buffer size changes and TCP smoothed round-trip time; if the correlation between these exceeds a threshold (0.7) we stop increasing buffer size.

Concepts

The key components are

• tuners: each tuner manages tunables and handles events sent from BPF programs to userspace via the shared ring buffer. Each tuner has an associated set of tunables that it manages.

• optional strategies: a tuner can specify multiple strategies; after running for a while a strategy times out and we assess if a better strategy is available. Each strategy specifies a

  • name
  • description
  • timeout
  • evaluation function
  • set of BPF program names in tuner associated with strategy

  Strategies are optional and should be set in the tuner init() method via bpftune_strategies_add(). See test/strategy for a coded example. When a strategy times out, the various evaluation functions are called and the highest-value evaluation dictates the next stratgey.

  Strategies provide a way of providing multiple schemes for auto-tuning the same set of tunables, where the choice is guided by an evaluation of the effectiveness of the strategies.

• events specify a

  • tuner id: which tuner the event is destined for
  • a scenario: what happened
  • an associated netns (if supported)
  • information about the event (IP address etc)

• the tuner then responds to the event guided by the active strategy; increase or decrease a tunable value, etc. Describing the event in the log is key; this allows an admin to understand what changed and why.

Architecture

• bpftune is a daemon which manages a set of .so plugin tuners; each of these is a shared object that is loaded on start-up.
• tuners can be enabled or disabled; a tuner is automatically disabled if the admin changes associated tunables manually.
• tuners share a global BPF ring buffer which allows posting of events from BPF programs to userspace. For example, if the sysctl tuner sees a systl being set, it posts an event.
• each tuner has an associated id (set when it is loaded), and events posted contain the tuner id.
• each tuner has a BPF component (built using a BPF skeleton) and a userspace component. The latter has init(), fini() and event_handler() entrypoints. When an event is received, the tuner id is used to identify the appropriate event handler and its event_handler() callback function is run.
• init, fini and event_handler functions are loaded from the tuner .so object.
• BPF components should include bpftune.bpf.h; it contains the common map definitions (ringbuf, etc) and shared variables such as learning rate and tuner ids that each tuner needs.

Supported tuners

• TCP connection tuner: auto-tune choice of congestion control algorithm. See bpftune-tcp-conn (8)
• IP fragmentation tuner: auto-tune IP fragmentation memory limits to support fragment reassembly. See bpftune-ip-frag (8)
• neighbour table tuner: auto-tune neighbour table sizes by growing tables when approaching full. See bpftune-neigh (8)
• sysctl tuner: monitor sysctl setting and if it collides with an auto-tuned sysctl value, disable the associated tuner. See bpftune-sysctl (8)
• TCP buffer tuner: auto-tune max and initial buffer sizes. See bpftune-tcp-buffer (8)
• net buffer tuner: auto-tune tunables related to core networking. See bpftune-net-buffer (8)
• netns tuner: notices addition and removal of network namespaces, which helps power namespace awareness for bpftune as a whole. Namespace awareness is important as we want to be able to auto-tune containers also. See bpftune-netns (8)
• UDP buffer tuner: auto-tune buffers relating to UDP. See bpftune-udp-buffer (8)

Code organization

Both core bpftune.c and individual tuners use the libbpftune library. It handles logging, tuner init/fini, and BPF init/fini.

Each tuner shared object defines an init(), fini() and event_handler() function. These respectively set up and clean up BPF and handle events that originate from the BPF code.

Getting Started

If building the repository manually, simply run

$ make ; sudo make install

at the top-level of the repository. bpftune also supports a

$ make pkg

target, which will make a bpftune RPM. See ./buildrpm/bpftune.spec

By default, we disable openrc script installation to avoid unneeded package dependencies; to enable this add

--with openrc

to the rpmbuild command.

We can also build with non-standard libdir for distros which do not use /usr/lib64 like CachyOS; in this case to install to /usr/lib instead

$ make libdir=lib
$ sudo make install libdir=lib

To build the following packages are needed (names may vary by distro);

• libbpf, libbpf-devel >= 0.6
• libcap-devel
• bpftool >= 4.18
• libnl3-devel (on some distros like Debian libnl-route-3-dev is needed)
• clang >= 11 (if not using GCC BPF)
• llvm >= 11
• python3-docutils

The bpf components in bpftune can be built via GCC BPF support. See https://gcc.gnu.org/wiki/BPFBackEnd for details on the BPF backend. To build with gcc bpf, specify

$ GCC_BPF=bpf-unknown-none-gcc make

From the kernel side, the kernel needs to support BPF ring buffer (around the 5.6 kernel, though 5.4 is supported on Oracle Linux as ring buffer support was backported), and kernel BTF is required (CONFIG_DEBUG_INFO_BTF=y). Verify /sys/kernel/btf/vmlinux is present.

To enable bpftune as a service

$ sudo service bpftune start

...and to enable it by default

$ sudo systemctl enable bpftune

bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out.

bpftune can also be run in the foreground as a program; to redirect output to stdout/stderr, run

$ sudo bpftune -s

On exit, bpftune will summarize any tuning done.

Queries of bpftune state can be done via bpftune -q.

Performance Co-Pilot (PCP) Support

Support has been added to export bpftune tunable values to Performance Co-Pilot via a PMDA (Performance Metric Domain Agent). It uses bpftune quereies to populate metrics in PCP via a python-based PMDA. See src/pcp/pmdabpftune.python.

To install the PMDA (ensuring pcp and python3-pcp packages are installed first), simply run

$ sudo make install
$ cd /var/lib/pcp/pmdas/bpftune
$ sudo ./Install

Once the above has been done, PCP metrics will be available for bpftune tunables. To see these:

$ pminfo -f bpftune

bpftune.udp_buffer.net.core.rmem_default
    value 212992

bpftune.udp_buffer.net.core.rmem_max
    value 3099438
...

The PMDA is also packaged in bpftune-pcp-pmda; see buildrpm/bpftune.spec.

To create grafana dashboards using bpftune tunables, see the blog entry here for details on setting up grafana to handle PCP metric visualization.

From there it is necessary to ensure that pmlogger is logging the metrics regularly; add something like the following to

/var/lib/pcp/config/pmlogger/config.default

log advisory on default {
        bpftune.tcp_buffer.net.ipv4.tcp_rmem
        bpftune.tcp_buffer.net.ipv4.tcp_wmem
        bpftune.tcp_buffer.net.ipv4.tcp_mem
}

...and restart pmlogger

$ sudo service pmlogger restart

Once that is done, it should be possible to create dashboards with queries of bpftune metrics. For example here is a simple dashboard created for the tcp_buffer tuner:

Ansible Install Play

Information: If you are using an Fedora Upstream based Distribution you have to enable the correct repository based on the system you are using, because the libbpf-devel package is getting shipped on additional repository, based on the Distribution. You can look it up here: https://pkgs.org/search/?q=libbpf-devel

…

Tests

Tests are supplied for each tuner in the tests/ subdirectory. "make test" runs all the tests. Tests use network namespaces to simulate interactions with remote hosts. See .