Baike.dev
Log in
Back to tool/Back to issues
#18412·rook

External cluster CephX rotation: rook-ceph-mon.ceph-secret not updated on re-import; --dry-run crashes with --cephx-key-rotate

Author: xakaitetoiaCreated Sep 16, 2026Updated Sep 16, 2026

Two defects on the documented CephX rotation path for external clusters, both hit while remediating CVE-2025-30156.

Environment: Rook v1.20.7, Ceph 19.2.6 (squid), external cluster, --restricted-auth-permission true, consumer cluster imported ~3 months earlier.

1. rook-ceph-mon.ceph-secret is not updated when re-importing

Repro — external cluster already imported, then rotate and re-import:

bash
# provider
python3 create-external-cluster-resources.py \
  --rbd-data-pool-name replicapool --cephfs-filesystem-name myfs \
  --k8s-cluster-name rookstorage --restricted-auth-permission true \
  --run-as-user client.healthchecker \
  --cephx-key-rotate rotate --cephx-key-type aes256k \
  --format bash --output /root/ext.sh

# consumer (secrets already exist from the original import)
set -a; . /root/ext.sh; set +a
bash import-external-cluster.sh

Actual — ceph-username advances to the new generation, ceph-secret keeps the old one:

bash
$ kubectl -n rook-ceph get secret rook-ceph-mon -o jsonpath='{.data.ceph-username}' | base64 -d
client.healthchecker.2
$ kubectl -n rook-ceph get secret rook-ceph-mon -o jsonpath='{.data.ceph-secret}' | base64 -d | sha256sum
e7c10b20…                          # == client.healthchecker  (generation 0)
$ ceph auth get-key client.healthchecker.2 | sha256sum
ded906ae…                          # the key that should have been written

The operator then cannot authenticate and loops indefinitely:

E | object-controller: failed to reconcile CephObjectStore ... failed to retrieve current ceph "mon" version
I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one
monclient(hunting): handle_auth_bad_method server allowed_methods [2] and I support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)

CephCluster stays Progressing with failed to detect and validate ceph version. Note I support [2,1] rather than [1]: a keyring was loaded and cephx was offered, so this is a wrong-key rejection rather than a missing keyring or a too-old client.

Cause — importSecret()'s update branch patches only the username, while its create branch sets both fields and every CSI function patches both userID and userKey:

bash
# import-external-cluster.sh, importSecret() — existing-secret branch
-p "{\"stringData\":{\"$MON_SECRET_CEPH_USERNAME_KEYNAME\":\"$userID\"}}"

# importCsiRBDNodeSecret() — existing-secret branch, for comparison
-p "{\"stringData\":{\"userID\":\"$userID\",\"userKey\":\"$CSI_RBD_NODE_SECRET\"}}"

Expected — the update branch also patches ceph-secret from $ROOK_EXTERNAL_USER_SECRET.

Workaround

bash
kubectl -n rook-ceph patch secret rook-ceph-mon \
  -p "{\"stringData\":{\"ceph-secret\":\"$ROOK_EXTERNAL_USER_SECRET\"}}"
kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator

2. --dry-run crashes when combined with --cephx-key-rotate rotate

bash
$ python3 create-external-cluster-resources.py \
    --rbd-data-pool-name replicapool --cephfs-filesystem-name myfs \
    --k8s-cluster-name rookstorage --restricted-auth-permission true \
    --run-as-user client.healthchecker \
    --cephx-key-rotate rotate --cephx-key-type aes256k \
    --format bash --dry-run

Execute: 'ceph auth ls'
Traceback (most recent call last):
  File "create-external-cluster-resources.py", line 2244, in <module>
    rjObj.main()
  File "create-external-cluster-resources.py", line 2210, in main
    generation = current_generation + 1
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Cause — dry_run() prints without returning, so get_cephx_latest_key_generation() returns None in dry-run mode:

python
def dry_run(self, msg):                      # line 571
    if self._arg_parser.dry_run:
        print("Execute: " + "'" + msg + "'")     # no return

def get_cephx_latest_key_generation(self):   # line 1623
    ...
    if self._arg_parser.dry_run:
        return self.dry_run("ceph " + cmd_json["prefix"])   # -> None

Without --cephx-key-rotate it does not crash, but generation stays None and every entity is named .None, because the naming helpers test if generation != 0.

Expected — --dry-run reports the generation it would use and the entity names it would create, for both cases.

Source: rook/rook

View original on GitHubView discussion on GitHub