Ceph mons wedge / never hold quorum on Ubuntu 26.04 HA (kernel 7.0.0-31-generic); identical stack healthy on 22.04

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:

On a fresh HA cluster with nodes running Ubuntu 26.04 (kernel 7.0.0-31-generic), the Ceph MONs cannot hold a stable quorum. The cluster briefly reaches HEALTH_OK (~6 minutes) and then flaps to HEALTH_ERR with failed to get status ... timed out. MONs settle into probing/electing and never re-form quorum. At least one MON process wedges: its local admin socket becomes unresponsive (ceph --admin-daemon /run/ceph/ceph-mon..asok mon_status hangs / times out), while the process sits in State: S on futex_do_wait at ~0% CPU (only a few seconds of CPU over 20+ minutes). MON logs repeatedly show:

cephx server client.admin: handle_request failed to decode CephXAuthenticate: End of buffer [buffer:2]

Because quorum never stabilizes, the operator keeps skipping reconcile since ceph health is HEALTH_ERR, and CephBlockPool/rbdpool never leaves Progressing, so the install blocks waiting for it to become Ready.

Key point: the identical Rook + Ceph + Kubernetes stack is healthy on Ubuntu 22.04 HA, and the same 26.04 image works fine in single-node mode (which does not deploy Ceph). The failure appears only on Ubuntu 26.04 in the HA (multi-MON) configuration.

Expected behavior:

MONs form a stable quorum, the MON admin socket stays responsive, ceph status returns HEALTH_OK with mon: 3 daemons, quorum a,b,c, and CephBlockPool/rbdpool reaches Ready — exactly as it does with the same stack on Ubuntu 22.04.

How to reproduce it (minimal and precise):

1. Deploy Rook v1.17.0 on Kubernetes v1.31, CNI Calico (IPIP, tunl0 MTU 1480).
2. Nodes: VMware VMs (vmxnet3 NIC), Ubuntu 26.04 (kernel 7.0.0-31-generic); 3 control-plane MONs + 4 worker OSDs (raw device per worker).
3. CephBlockPool: replicated, size: 2, failureDomain: zone, deviceClass: ssd.
4. Bring up the CephCluster. OSDs come up and the cluster momentarily reaches HEALTH_OK, then MON quorum collapses and rbdpool stays Progressing.

Reproduced on Ceph v19.2.1, v18.2.4 (Reef), and v19.2.6 (latest Squid) → not Ceph-version-specific.

The following were ruled out by direct measurement (all healthy / no effect):

• Clock skew across MONs < 0.3 s; MON disk fsync ~0.4 ms (rocksdb reports no stalls)
• MON↔MON TCP connectivity, including pod-IP-direct (bypassing Service/kube-proxy): all OK
• kube-proxy in iptables mode
• NIC checksum/segmentation offload (vmxnet3) disabled cluster-wide: no change
• Host kernel modules and sysctls present on all nodes: br_netfilter, overlay, ip_vs, nf_conntrack loaded; net.bridge.bridge-nf-call-iptables=1, net.ipv4.ip_forward=1
• dmesg clean on the MON nodes — no OOM, hung-task, or NIC-driver errors
• Forcing msgr v1 (ms_bind_msgr2=false): no change — the admin-socket hang is transport-independent

The only environmental differences from the working Ubuntu 22.04 nodes are the kernel (7.0.0-31-generic vs 5.15) and the nftables default backend. This looks like a MON-process hang specific to the Ubuntu 26.04 kernel, independent of Ceph version and messenger transport.

File(s) to submit:

• CephCluster / CephBlockPool CRs available on request (omitted here) — spec highlights already listed above.

Logs to submit:

• MON logs (all three), operator log, and MON-process thread kernel-stacks are captured and available on request. Key recurring signature: cephx server ... failed to decode CephXAuthenticate: End of buffer. (Not attached here; can provide if needed.)

Cluster Status to submit:

• ceph status from the toolbox times out (monclient(hunting): authenticate timed out) while quorum is unformed; MON admin-socket mon_status shows state: probing/electing, empty quorum. Full kubectl rook-ceph ceph status / health output available on request.

Thank you for your help.