Baike.dev
Log in
Back to tool/Back to issues
#18402·rook

osd and osd-prepare pods stuck calling lvs when other osds are down

Author: sathieuCreated Sep 16, 2026Updated Sep 16, 2026
Labelsbug

Is this a bug report or feature request?

  • Bug Report

Context

We had a severe outage in one cluster where the kube-apiserver virtual IP was not reachable from the kubelets (the root cause of this was a bug in kube-vip https://github.com/kube-vip/kube-vip/issues/1654). The outage was long enough that the nodes were moved to Ready=Unknown status.

Once the initial problem was fixed, some OSD pods were restarted. It's unclear if this is by kubelet (probes) or by rook-ceph-operator. The operator had the following logs:

disruption: osd "rook-ceph-osd-6" is down on node "node-2" but no node drain is detected
disruption: osd "rook-ceph-osd-16" is down on node "node-2" but no node drain is detected
[...]
disruption: OSD(s) [6 16 5 14 9 11 4 3 13 8 10 0 12 7 2 17] are down and PGs are not clean. PGs Status: "cluster is not full[truncated]
disruption: OSD failure Domains : ["node-1" "node-2" "node-3" "node-5"]
disruption: Draining Failure Domain: "node-3"
disruption: Set noout on draining Failure Domain: "true"
[...]
exec: exec timeout waiting for process rbd to return. Sending interrupt signal to the process

Now there is no enough OSDs "up". The restaring OSDs deadlock because of lvs:

bash
$ kubectl exec -ti -n rook-ceph rook-ceph-osd-prepare-node-1-hf4tl

# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         146  0.0  0.0   5048  4176 pts/11   Ss   14:33   0:00 bash
root         172  0.0  0.0   7536  3544 pts/11   R+   14:33   0:00  \_ ps auxf
root           1  0.0  0.0 1303752 71096 ?       Ssl  14:17   0:00 /rook/rook ceph osd provision
root         137  0.0  0.0  36960 28884 ?        S    14:17   0:00 /usr/bin/python3 -s /usr/sbin/ceph-volume inventory --format json /dev/sdc2
root         139  0.0  0.0  23884  9848 ?        D    14:17   0:00  \_ /sbin/lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S lv_path=/dev/sdc2 -o lv_tags,lv_path,lv_name,vg_name,l

# ls -l /proc/139/fd
total 0
lr-x------ 1 root root 64 Sep 11 14:15 0 -> 'pipe:[1232136]'
l-wx------ 1 root root 64 Sep 11 14:15 1 -> 'pipe:[1232137]'
l-wx------ 1 root root 64 Sep 11 14:15 2 -> 'pipe:[1232138]'
lrwx------ 1 root root 64 Sep 11 14:15 3 -> /run/lvm/hints
lr-x------ 1 root root 64 Sep 11 14:15 4 -> /dev/rbd0
lr-x------ 1 root root 64 Sep 11 14:15 5 -> /dev/rbd1
lr-x------ 1 root root 64 Sep 11 14:15 6 -> /dev/rbd2
lr-x------ 1 root root 64 Sep 11 14:15 7 -> /dev/rbd3

The lvs process is blocked reading rdb devices which are not available because of too many falling OSDs.

Deviation from expected behavior:

osd and osd-prepare pods are stuck. Even rebooting the nodes requires a hard reset, because the kernel is blocking.

Expected behavior:

osd and osd-prepare pods should not hung, and cluster should recover automatically.

How to reproduce it (minimal and precise):

  • Create a minimal k8s cluster, using kube-vip to serve the IP, ensure kubelets points to this VIP (/etc/kubernetes/kubelet.conf)
  • Install rook-ceph, ...
  • create cephcluster
  • stop the VIP (by moving out kube-vip static manifests)
  • wait nodes to become Ready=Unknow (KUBECONFIG=/etc/kubernetes/admin.con kubectl --server https://127.0.0.1:6443 --insecure-skip-tls-verify get nodes from a control-plane)
  • bring back the VIP
  • watch the pods restart, and lvs command getting in D state

Proposed fix

There is the filter option to lvm.conf.

Current UpdateLVMConfig already changes filter when on PVC.

Our proposed fix is to change filter to [ "r|/dev/rbd.*|", "a|.*|" ] when not on PVC. We successfully tested this with the help of gatekeeper:

bash
$ kubectl exec -n rook-ceph rook-ceph-osd-4-5cfdc648c9-76x8f  -- cat /etc/lvm/lvm.conf > lvm.conf
$ cp lvm.conf lvm.conf.orig
$ sed -i '0,/filter =/ s@# filter = \[ "a|\.\*|" \]@filter = [ "r|/dev/rbd.*|", "a|.*|" ]@' lvm.conf
$ diff -u lvm.conf.orig lvm.conf
--- lvm.conf.orig       2026-09-11 17:20:21.520714934 +0200
+++ lvm.conf    2026-09-11 17:20:36.932690120 +0200
@@ -212,7 +212,7 @@
        #
        # Example
        # Accept every block device:
-       # filter = [ "a|.*|" ]
+       filter = [ "r|/dev/rbd.*|", "a|.*|" ]
        # Reject the cdrom drive:
        # filter = [ "r|/dev/cdrom|" ]
        # Work with just loopback devices, e.g. for testing:
$ kubectl create cm -n rook-ceph lvm-dot-conf --from-file=lvm.conf=lvm.conf -oyaml --dry-run=client | kubectl apply -f -
configmap/lvm-dot-conf configured

The following YAML is applied to inject lvm.conf in pods: injection.yaml

We'll create a PR changing UpdateLVMConfig function.

Source: rook/rook

View original on GitHubView discussion on GitHub