basalt-26.3: roachtest: backup-restore/chaos failed
Author: cockroach-teamcityCreated Sep 16, 2026Updated Sep 17, 2026
LabelsC-test-failureO-robotA-disaster-recoveryO-roachtestT-disaster-recoveryrelease-blockerbranch-basalt-26.3
roachtest.backup-restore/chaos failed with artifacts on basalt-26.3 @ 4e3a6026e33f8cd3e7daf4a59cb0e5ad92174612:
Failed with:
(assertions.go:388).Fail:
Error Trace: pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:474
pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:354
pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:133
pkg/cmd/roachtest/test_runner.go:1618
src/runtime/asm_amd64.s:1771
Error: Received unexpected error:
only 0 consecutive intervals of balanced stores
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas.func1
| pkg/roachprod/node_stability.go:287
| github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
| pkg/util/retry/retry.go:329
| github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas
| pkg/roachprod/node_stability.go:271
| github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForBalancedReplicas
| pkg/roachprod/failureinjection/failures/failure.go:367
| github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForRestartedNodesToStabilize
| pkg/roachprod/failureinjection/failures/failure.go:405
| github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*ProcessKillFailure).WaitForFailureToRecover
| pkg/roachprod/failureinjection/failures/process_kill.go:126
| github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*Failer).WaitForFailureToRecover
| pkg/roachprod/failureinjection/failures/failer.go:195
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.injectAndRecoverFailure.func1
| pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:459
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
| pkg/cmd/roachtest/roachtestutil/task/manager.go:201
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
| pkg/cmd/roachtest/roachtestutil/task/manager.go:211
| golang.org/x/sync/errgroup.(*Group).Go.func1
| external/org_golang_x_sync/errgroup/errgroup.go:93
| runtime.goexit
| src/runtime/asm_amd64.s:1771
Wraps: (2) only 0 consecutive intervals of balanced stores
Error types: (1) *withstack.withStack (2) *errutil.leafError
Test: backup-restore/chaos
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/backup-restore/chaos/run_1Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
|---|---|---|
| teamcity-21518936-1789538086-15-n4cpu4-0001 | 10.91.97.139 | |
| teamcity-21518936-1789538086-15-n4cpu4-0002 | 10.91.97.140 | |
| teamcity-21518936-1789538086-15-n4cpu4-0003 | 10.91.97.138 | |
| teamcity-21518936-1789538086-15-n4cpu4-0004 | 10.91.97.163 |
Parameters:
arch=amd64cloud=gcecoverageBuild=falsecpu=4diskCount=0encrypted=truefs=ext4localSSD=truemetamorphicWriteBuffering=trueonlineRestore=falseruntimeAssertionsBuild=false
See: roachtest README
See: How To Investigate (internal)
See: Grafana
Logs were not uploaded to Datadog
Same failure on other branches- #174726 release-26.3.2-rc: roachtest: backup-restore/chaos failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-26.3.2-rc]
- #173553 release-26.3: roachtest: backup-restore/chaos failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-26.3 target-release-26.3.2]
This test on roachdash | Improve this report!
Jira issue: CRDB-68446
Source: cockroachdb/cockroach