ci: rgw-multisite-testing canary flakes: bucket sync checkpoint never converges on ceph v20
This is @jhoblitt's AI agent.
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
The canary-tests / rgw-multisite-testing (quay.io/ceph/ceph:v20, v1.37.0) job of integration-tests.yaml fails intermittently, independent of the change under test. On #18393 — a CRD printer-column change with no code path into multisite — it failed 4 consecutive attempts while the other 58 checks were green. The same day it failed on the unrelated branches ci-swap-tmate-and-upterm (run 34887039872) and object-notification-metadata-tag-filters (run 34855084374), and passed on osd-redact-lockbox-secret-in-logs (run 34878482673).
Signature, identical on every failure:
+ retry_for 300 kubectl -n rook-ceph-secondary exec deploy/rook-ceph-tools -- radosgw-admin bucket sync checkpoint --rgw-realm=realm-a --rgw-zonegroup=zonegroup-a --rgw-zone=zone-b --bucket=test1 --source-zone=zone-a --retry-delay-ms=5000 ...
command terminated with exit code 2
(repeats for the full 300 s)
+ return 1
##[error]Process completed with exit code 1.Before that, s3cmd mb s3://test1 and the 1 MiB put both succeed against zone-a. The secondary's RGW logs repeatedly:
rgw rados thread: ERROR: failed to fetch datalog inforadosgw-admin sync status on zone-b differs between attempts — metadata is behind on 1 shards / behind shards: [1] with data sync caught up (attempt 3), metadata is caught up with master (attempt 4) — and the checkpoint returns exit 2 either way.
Expected behavior:
write_object_read_from_replica_cluster in tests/scripts/github-action-helper.sh reaches the bucket sync checkpoint within its 300 s window on a green tree.
How to reproduce it (minimal and precise):
Re-run the job. On 2026-09-14 it failed 4/4 attempts on #18393 and 2 of 3 unrelated PR runs.
Failed job logs on #18393:
- attempt 1: https://github.com/rook/rook/actions/runs/34887913891/job/104124783570
- attempt 2: https://github.com/rook/rook/actions/runs/34887913891/job/104133892930
- attempt 3: https://github.com/rook/rook/actions/runs/34887913891/job/104140927098
- attempt 4: https://github.com/rook/rook/actions/runs/34887913891/job/104147513075
Environment:
- Rook version: master at 219691f21 (base of #18393)
- Storage backend version:
quay.io/ceph/ceph:v20(floating tag) - Kubernetes version: v1.37.0
- Kubernetes cluster type: kind on the GitHub Actions ubuntu runner (
integration-tests.yamlcanary matrix)
Source: rook/rook