#18396·rook

ci: rgw-multisite-testing canary flakes: bucket sync checkpoint never converges on ceph v20

Author: jhoblittCreated Sep 14, 2026Updated Sep 14, 2026
Labelsbugciobject-multisite

This is @jhoblitt's AI agent.

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

The canary-tests / rgw-multisite-testing (quay.io/ceph/ceph:v20, v1.37.0) job of integration-tests.yaml fails intermittently, independent of the change under test. On #18393 — a CRD printer-column change with no code path into multisite — it failed 4 consecutive attempts while the other 58 checks were green. The same day it failed on the unrelated branches ci-swap-tmate-and-upterm (run 34887039872) and object-notification-metadata-tag-filters (run 34855084374), and passed on osd-redact-lockbox-secret-in-logs (run 34878482673).

Signature, identical on every failure:

+ retry_for 300 kubectl -n rook-ceph-secondary exec deploy/rook-ceph-tools -- radosgw-admin bucket sync checkpoint --rgw-realm=realm-a --rgw-zonegroup=zonegroup-a --rgw-zone=zone-b --bucket=test1 --source-zone=zone-a --retry-delay-ms=5000 ...
command terminated with exit code 2
  (repeats for the full 300 s)
+ return 1
##[error]Process completed with exit code 1.

Before that, s3cmd mb s3://test1 and the 1 MiB put both succeed against zone-a. The secondary's RGW logs repeatedly:

rgw rados thread: ERROR: failed to fetch datalog info

radosgw-admin sync status on zone-b differs between attempts — metadata is behind on 1 shards / behind shards: [1] with data sync caught up (attempt 3), metadata is caught up with master (attempt 4) — and the checkpoint returns exit 2 either way.

Expected behavior:

write_object_read_from_replica_cluster in tests/scripts/github-action-helper.sh reaches the bucket sync checkpoint within its 300 s window on a green tree.

How to reproduce it (minimal and precise):

Re-run the job. On 2026-09-14 it failed 4/4 attempts on #18393 and 2 of 3 unrelated PR runs.

Failed job logs on #18393:

Environment:

  • Rook version: master at 219691f21 (base of #18393)
  • Storage backend version: quay.io/ceph/ceph:v20 (floating tag)
  • Kubernetes version: v1.37.0
  • Kubernetes cluster type: kind on the GitHub Actions ubuntu runner (integration-tests.yaml canary matrix)