#175539·cockroach

basalt-26.3: roachtest: backup-restore/chaos failed

Author: cockroach-teamcityCreated Sep 16, 2026Updated Sep 17, 2026
LabelsC-test-failureO-robotA-disaster-recoveryO-roachtestT-disaster-recoveryrelease-blockerbranch-basalt-26.3

roachtest.backup-restore/chaos failed with artifacts on basalt-26.3 @ 4e3a6026e33f8cd3e7daf4a59cb0e5ad92174612:

Failed with:

(assertions.go:388).Fail: 
	Error Trace:	pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:474
	            				pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:354
	            				pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:133
	            				pkg/cmd/roachtest/test_runner.go:1618
	            				src/runtime/asm_amd64.s:1771
	Error:      	Received unexpected error:
	            	only 0 consecutive intervals of balanced stores
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas.func1
	            	  | 	pkg/roachprod/node_stability.go:287
	            	  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
	            	  | 	pkg/util/retry/retry.go:329
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod.WaitForBalancedReplicas
	            	  | 	pkg/roachprod/node_stability.go:271
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForBalancedReplicas
	            	  | 	pkg/roachprod/failureinjection/failures/failure.go:367
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*GenericFailure).WaitForRestartedNodesToStabilize
	            	  | 	pkg/roachprod/failureinjection/failures/failure.go:405
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*ProcessKillFailure).WaitForFailureToRecover
	            	  | 	pkg/roachprod/failureinjection/failures/process_kill.go:126
	            	  | github.com/cockroachdb/cockroach/pkg/roachprod/failureinjection/failures.(*Failer).WaitForFailureToRecover
	            	  | 	pkg/roachprod/failureinjection/failures/failer.go:195
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.injectAndRecoverFailure.func1
	            	  | 	pkg/cmd/roachtest/tests/backup_restore_roundtrip.go:459
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func1
	            	  | 	pkg/cmd/roachtest/roachtestutil/task/manager.go:201
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/task.(*group).GoWithCancel.func2
	            	  | 	pkg/cmd/roachtest/roachtestutil/task/manager.go:211
	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1771
	            	Wraps: (2) only 0 consecutive intervals of balanced stores
	            	Error types: (1) *withstack.withStack (2) *errutil.leafError
	Test:       	backup-restore/chaos
(require.go:1376).NoError: FailNow called
test artifacts and logs in: /artifacts/backup-restore/chaos/run_1

Cluster Node to Ip Mapping:

Node Public IP Private IP
teamcity-21518936-1789538086-15-n4cpu4-0001 10.91.97.139
teamcity-21518936-1789538086-15-n4cpu4-0002 10.91.97.140
teamcity-21518936-1789538086-15-n4cpu4-0003 10.91.97.138
teamcity-21518936-1789538086-15-n4cpu4-0004 10.91.97.163

Parameters:

  • arch=amd64
  • cloud=gce
  • coverageBuild=false
  • cpu=4
  • diskCount=0
  • encrypted=true
  • fs=ext4
  • localSSD=true
  • metamorphicWriteBuffering=true
  • onlineRestore=false
  • runtimeAssertionsBuild=false
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Logs were not uploaded to Datadog

Same failure on other branches

  • #174726 release-26.3.2-rc: roachtest: backup-restore/chaos failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-26.3.2-rc]
  • #173553 release-26.3: roachtest: backup-restore/chaos failed [A-disaster-recovery C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-26.3 target-release-26.3.2]

/cc @cockroachlabs/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-68446