Baike.dev
Log in
Back to tool/Back to issues
#24067·netdata

[Bug]: ebpf.plugin overflows per-CPU map buffers when possible CPUs exceed online CPUs

Author: methanoyaCreated Sep 27, 2026Updated Sep 27, 2026
Labelsbugneeds triage

Bug description

Found at the first start of an virtual machine with vmware.

ebpf.plugin aborts about one second after it starts on a host where the number of possible CPUs is larger than the number of online CPUs (VMware guest: 4 online, 128 possible). glibc detects heap corruption and netdata disables the plugin:

ebpf.plugin[527688]: malloc(): invalid size (unsorted)
spawn-plugins[527046]: SPAWN SERVER: child with pid 527688 (request 5) coredump'd due to signal 6: /bin/sh -c "exec /usr/libexec/netdata/plugins.d/ebpf.plugin 1 "
netdata[526982]: PLUGINSD: 'host:<hostname>', '/usr/libexec/netdata/plugins.d/ebpf.plugin' (pid 527688) exited abnormally. Disabling it.

Root cause: buffers for per-CPU map lookups are sized with the online CPU count, but the kernel copies one value per possible CPU.

  • src/collectors/ebpf.plugin/ebpf.c:1018: ebpf_nprocs = (int)sysconf(_SC_NPROCESSORS_ONLN); (4 here).
  • A lookup on a BPF_MAP_TYPE_PERCPU_ARRAY / BPF_MAP_TYPE_PERCPU_HASH map copies round_up(value_size, 8) * num_possible_cpus() bytes into the caller's buffer (128 CPUs here).
  • Example, ebpf_mount.c:270 (ebpf_mount_read_global_table()):
    • mount_values = callocz(ebpf_nprocs + 1, sizeof(netdata_idx_t)) allocates 40 bytes on this host.
    • bpf_map_lookup_elem() on tbl_mount (PERCPU_ARRAY) writes 8 × 128 = 1024 bytes into it.

The core dump matches this exactly. There are 1024 zero bytes starting at the 40-byte mount_values buffer, and they overwrite the header of the next heap chunk. The abort happens in whichever thread calls malloc() next, here the cgroup integration thread. That's why the stack trace doesn't point at the mount module:

#8  _int_malloc (av=<main_arena>, bytes=13) at ./malloc/malloc.c:3938
#10 __GI___libc_malloc (bytes=13)
#11 __GI___strdup (s="/run/netdata")
#13 os_run_dir ()
#14 ebpf_cgroup_integration ()

At that moment the mount thread had just completed its first collection.

Affected code (master @ 4464a9ed7, same in v2.11.1):

  • 15 buffers sized with ebpf_nprocs receive per-CPU lookups, in 10 modules: disk, filesystem, hardirq, mdflush, mount, process, shm, softirq, swap and vfs. v2.11.1 also has dcstat and fd.
  • ebpf_read_global_table_stats() (libbpf_api/ebpf_library.c) reads into the process/shm/swap/vfs buffers.
  • With the stock configuration only mount runs, so it's the one that crashes by default. Enabling any of the other modules hits the same problem.

Expected behavior

ebpf.plugin reads per-CPU maps into buffers large enough for every possible CPU and keeps running. Per-CPU sums cover all possible CPUs; offline CPUs contribute zero.

Steps to reproduce

  1. Use a Linux host whose possible CPU count is higher than its online count.
    • Check with cat /sys/devices/system/cpu/possible /sys/devices/system/cpu/online. Here that gives 0-127 and 0-3, and the kernel logs "CPU topo: Allowing 4 present CPUs plus 124 hotplug CPUs".
    • VMware guests commonly look like this.
  2. Install Netdata v2.11.1 with kickstart.sh (native DEB packages) and keep the stock configuration (mount = yes in ebpf.d.conf).
  3. Start netdata.
  4. About one second after ebpf.plugin starts it logs malloc(): invalid size (unsorted) and aborts with SIGABRT, and netdata disables it. Enabling any other affected module (for example process = yes) triggers the same crash.

Installation method

kickstart.sh

System info

bash
Linux <hostname> 7.0.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Aug  1 04:26:38 UTC 2026 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=26.04
/etc/lsb-release:DISTRIB_CODENAME=resolute
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 26.04.1 LTS"
/etc/os-release:PRETTY_NAME="Ubuntu 26.04.1 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION_ID="26.04"
/etc/os-release:VERSION="26.04.1 LTS (Resolute Raccoon)"
/etc/os-release:VERSION_CODENAME=resolute
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:UBUNTU_CODENAME=resolute
/etc/os-release:LOGO=ubuntu-logo

Netdata build info

bash
Packaging:
    Netdata Version ____________________________________________ : v2.11.1
    Installation Type __________________________________________ : binpkg-deb
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : ubuntu 26.04
    Configure Options __________________________________________ : cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_STANDARD=11 -DCMAKE_CXX_STANDARD=17 -DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS=' -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_CXX_FLAGS='  -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_COMPILE_DEFINITIONS='_GNU_SOURCE' -DCMAKE_EXE_LINKER_FLAGS=' -fexceptions -rdynamic' -DCMAKE_SHARED_LINKER_FLAGS=''
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Stock Data Files ___________________________________________ : /usr/share/netdata
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 7.0.0-31-generic
    Operating System ___________________________________________ : Ubuntu
    Operating System ID ________________________________________ : ubuntu
    Operating System ID Like ___________________________________ : debian
    Operating System Version ___________________________________ : 26.04.1 LTS (Resolute Raccoon)
    Operating System Version ID ________________________________ : 26.04
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 4
    CPU Frequency ______________________________________________ : 2400000000
    RAM Bytes __________________________________________________ : 16178053120
    Disk Capacity ______________________________________________ : 137438953472
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : vmware
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip brotli)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
    Memory Allocator ___________________________________________ : system
Database Engines:
    sqlite _____________________________________________________ : 3.53.4
    dbengine (compression) _____________________________________ : YES (zstd lz4)
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : YES
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
    libmnl (library for working with netfilter) ________________ : YES
    stacktraces (library for getting stack traces) _____________ : libbacktrace (mmap, threads, data)
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    macos-logs (monitor macOS unified logs) ____________________ : NO
    windows (monitor Windows systems) __________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    network-viewer (monitor TCP/UDP IPv4/6 sockets) ____________ : YES
    systemd-journal (monitor journal logs) _____________________ : YES
    windows-events (monitor Windows events) ____________________ : NO
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : YES
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : NO
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO
Runtime Information:
    Profile ____________________________________________________ : standalone
    Stream Parent (accept data from Children) __________________ : NO
    Stream Child (send data to a Parent) _______________________ : NO
    Total System Memory ________________________________________ : 16178053120
    Available System Memory ____________________________________ : 3741184000

Additional info

Regression test: the new --unittest step needs the option-dispatch fix, #24062 / #24063. On the current master, --unittest selects CO-RE and starts collecting, so it never reaches the step.

Source: netdata/netdata

View original on GitHubView discussion on GitHub