#915·coroot

Storage report missing for apps on statically provisioned PersistentVolumes (empty volume label)

Author: gecubeCreated Jun 10, 2026Updated Jun 10, 2026

Summary

The per-application Storage report is silently missing for any Kubernetes app whose PersistentVolume has a custom (statically provisioned) name. The node-agent collects all the disk/fs metrics for these containers just fine — but it only sets the volume label when the PV name matches the pvc-<uuid> shape of dynamically provisioned volumes, and the server then drops every k8s volume with an empty volume label and deletes the whole Storage report.

This was the actual root cause behind my earlier report in #914 (which is mostly unrelated Bottlerocket log noise — see my comment there).

Environment

  • EKS v1.35.4, Bottlerocket AMI (kernel 6.12.83), Karpenter node groups
  • coroot-ce chart 0.3.3, coroot-operator 0.9.7, coroot-node-agent 1.33.3
  • EBS CSI driver, mix of dynamically provisioned PVs and statically created PVs (restored from snapshots during DR/migration)

Root cause

1. node-agent extracts the volume label for container_resources_disk_* metrics by matching the OCI mount source path against this regex:

https://github.com/coroot/coroot-node-agent/blob/main/common/volumes.go

go
k8sVolumeDir = regexp.MustCompile(`.+(pvc-[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}).*`)

The kubelet mount path embeds the PV name: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv-name>/mount. For dynamically provisioned PVs the name is pvc-<uuid> and the regex matches. For statically provisioned PVs (e.g. vls-restore, ledger-migration, grafana-dr-restore in my cluster) the regex fails and the volume label is emitted empty.

2. coroot server then skips volumes with an empty name for k8s apps and removes the report entirely if none survive (auditor/storage.go):

go
if isK8s && v.Name.Value() == "" {
    continue
}
...
if !seenVolumes {
    a.delReport(model.AuditReportStorage)
}

Observed behavior

On the same node, with the same agent:

  • loki-write (dynamic PV pvc-167a6074-…) → series carry volume="pvc-167a6074-…" → Storage tab fully populated.
  • victoria-logs (static PV vls-restore), ledger-service (static PV ledger-migration), influxdb, grafana, vmsingle (all restored/static PVs) → identical container_resources_disk_* series exist in Prometheus but without a volume label → no Storage tab at all in the UI.

So the data is collected and stored; only the PV-name resolution and the server-side filter hide it.

Suggested fix

In ParseKubernetesVolumeSource, instead of requiring the pvc-<uuid> name shape, fall back to extracting the directory name under kubernetes.io~csi/ (and the other kubernetes.io~* volume plugin dirs) — that directory is the PV name regardless of how the PV was provisioned:

/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv-name>/mount  →  <pv-name>

Happy to send a PR to coroot-node-agent if that approach sounds acceptable.