Vote-subset-dependent BeginBlock state writes (x/slashing, x/distribution) cause apphash divergence under partial vote gossip (v0.53.8)
Title
Vote-subset-dependent BeginBlock state writes (x/slashing, x/distribution) cause apphash divergence and speculative-execution livelock under partial vote gossip — Cosmos SDK v0.53.8
Summary
We operate a Cosmos SDK v0.53.8 + CometBFT v0.38 chain (4 validators,
co-located testbed) and observed repeated consensus stalls with pairwise
apphash mismatches (wrong Block.Header.AppHash) across five
environments, on empty blocks, with unanimous committed state throughout.
Root cause, isolated to a unit-level repro: x/slashing and
x/distribution BeginBlockers write state derived from the node-local
DecidedLastCommit vote subset. When vote gossip is incomplete, two
correct nodes hold different subsets, write different state, and commit
different apphashes — a legitimate-inputs divergence (different inputs,
different outputs), which then cascades into a round-loop livelock via
stale speculative state (details below).
We report this as operators who found it by using the software as intended, with a self-contained repro that should save your team significant triage time.
Environment
- cosmos-sdk v0.53.8, CometBFT v0.38.23, Go 1.25, linux/amd64 + Windows dev hosts (incidents observed on both; mechanism is platform-free).
- 4-validator testnets, 2.3–3.5s blocks, co-located (localhost/Docker bridge). No IBC, no vote extensions in play, no custom BeginBlock vote logic.
Minimal repro (self-contained, no network)
Two BeginBlock executions over identical committed state and identical
(empty) blocks, differing ONLY in the VoteInfos attached to the
sdk.Context (3 validators A/B/C, equal power):
| Context | Votes | Result |
|---|---|---|
| C0 | A,B,C | reference apphash |
| C1 | A,B (C removed) | DIFFERENT apphash |
Bisect: x/slashing alone is order-insensitive (same=true);
x/distribution alone carries the signal. Both write vote-dependent
state: slashing records per-validator liveness (MissedBlocksCounter,
bitmap) and distribution splits rewards by previousTotalPower (which
shifts every validator's share when membership changes).
Repro sketch (real keepers, narrow fakes for account/staking lookups, real multistore commit per variant — full harness attached below (collapsible); drop into any SDK v0.53.x chain module test tree with matching dependencies to run):
// same fixture, same header, same empty block; only VoteInfos varies
ctx := sdk.NewContext(ms, header, false, logger).WithVoteInfos(votes)
slashing.BeginBlocker(ctx, slashKeeper) // writes signing info per vote
distrKeeper.BeginBlocker(ctx) // allocates by vote power
apphash := ms.Commit().Hash // differs C0 vs C1, stable per variantDeterminism control: C0 twice → identical apphash (10/10 + cross-process). The divergence is input-driven, not flaky.
Mechanism
baseapp/abci.go builds VoteInfos from req.DecidedLastCommit
(node-local view of the last commit). Under complete gossip every node
holds the same set and all is well. Under incomplete gossip (lost vote,
tight timeout_commit, degraded P2P), node X records validator V as
signed while node Y records V as missed, and distribution allocates
different reward splits — different apphash, same block bytes. Both
nodes are "correct" given their inputs; consensus has no tiebreaker.
A second, sharper edge: once nodes hold different speculative (uncommitted) execution results, they validate subsequent proposals against stale speculative state rather than committed state, and every new proposal is rejected by some pair — a round-loop livelock with no recovery short of restart (which itself may not converge if the trigger persists). Committed state stays unanimous throughout; the damage is purely liveness. We name this speculative poisoning.
Order-sensitivity note (scoped precisely)
The same harness shows same-members-different-order also diverges — but
we verified this input class is UNREACHABLE via consensus: CometBFT
index-orders commit votes (VoteSet.MakeExtendedCommit), so the app
never receives reordered full sets. Reported for completeness, not as
an incident mechanism. (Separately, we confirmed iavl v1.2.8 roots
depend on key insertion order for arbitrary permutations — see
companion draft — which is why every write loop in consensus code must
use canonical order regardless.)
Real-world incidents (summary, no infra details)
Five independent 4-validator environments stalled with the pairwise mismatch signature over several weeks: post-upgrade and pre-upgrade binaries, fresh and restored state, 2.3s and 3.5s blocks. Splits occurred both immediately after transaction bursts AND in at least one pristine case (an environment with zero transaction history split on empty blocks), proving bursts unnecessary; most incidents followed bursts within a few blocks, consistent with aggravation rather than trigger. A single lost or delayed vote message suffices to split the observed subsets; vote-message loss occurs on production networks as well — co-location reduces, but does not eliminate, exposure. Our application modules were audited line-by-line and excluded (no maps/time/floats/goroutines in executed paths); the decider reproduces the split with stock SDK modules only. Vote-set capture instrumentation is deployed on our networks; we can supply live subset-diff evidence if useful.
Fix evaluation (why no local fix exists)
Slashing liveness and distribution rewards MUST be computed from observed
votes — deriving them from anything else (committed set, block evidence)
changes consensus semantics and still diverges when views differ. The
writers live in SDK modules; application code cannot interpose. Our
deployed mitigations (larger timeout_commit margin, P2P hygiene,
sentry posture, liveness monitoring, agreement gates) reduce the
trigger rate but cannot remove the mechanism. A durable fix likely lives
either in deterministic commit delivery (CometBFT) or in making these
writes subset-independent (SDK design question we leave to your team).
Our ask
Confirmation of the mechanism, guidance on whether subset-independent BeginBlock writes are on any roadmap, and any recommended operator mitigations beyond timing/P2P hygiene we may have missed. Happy to provide the full harness, logs, and vote-set captures.
Reported by the Genesis Protocol team.
Decider harness (self-contained Go test, click to expand)// Package determinism is a self-contained vote-subset decider harness.
//
//
// Question: do differing VoteInfos (same block, same parent state) change
// the committed apphash via x/slashing + x/distribution BeginBlock writes?
// Four contexts vary membership/order; each runs both BeginBlockers on an
// identically-constructed store and commits. The verdict matrix is printed
// and order-independence + determinism are asserted; member-diff behavior
// is reported (it documents the inherited SDK hazard by design).
package determinism
// SPDX-License-Identifier: Apache-2.0
import (
"context"
"encoding/hex"
"fmt"
"testing"
"time"
"cosmossdk.io/log"
"cosmossdk.io/math"
"cosmossdk.io/store/metrics"
"cosmossdk.io/store/rootmulti"
storetypes "cosmossdk.io/store/types"
coreaddress "cosmossdk.io/core/address"
abci "github.com/cometbft/cometbft/abci/types"
cmtproto "github.com/cometbft/cometbft/proto/tendermint/types"
dbm "github.com/cosmos/cosmos-db"
"github.com/cosmos/iavl"
iavldb "github.com/cosmos/iavl/db"
"github.com/cosmos/cosmos-sdk/codec"
codectypes "github.com/cosmos/cosmos-sdk/codec/types"
"github.com/cosmos/cosmos-sdk/codec/address"
"github.com/cosmos/cosmos-sdk/runtime"
sdk "github.com/cosmos/cosmos-sdk/types"
authtypes "github.com/cosmos/cosmos-sdk/x/auth/types"
bankkeeper "github.com/cosmos/cosmos-sdk/x/bank/keeper"
banktypes "github.com/cosmos/cosmos-sdk/x/bank/types"
distrkeeper "github.com/cosmos/cosmos-sdk/x/distribution/keeper"
distrtypes "github.com/cosmos/cosmos-sdk/x/distribution/types"
slashing "github.com/cosmos/cosmos-sdk/x/slashing"
slashingkeeper "github.com/cosmos/cosmos-sdk/x/slashing/keeper"
slashingtypes "github.com/cosmos/cosmos-sdk/x/slashing/types"
stakingtypes "github.com/cosmos/cosmos-sdk/x/staking/types"
"github.com/stretchr/testify/require"
)
// ---- fakes (narrow; embed full interfaces, override BeginBlock surface) ----
// fakeAccounts satisfies both bank + distribution AccountKeeper interfaces.
// Only module-address/account/codec are functional; the rest are inert
// (BeginBlock paths never touch them — a panic here names the gap).
type fakeAccounts struct {
addrs map[string]sdk.AccAddress
}
func (f fakeAccounts) GetModuleAddress(name string) sdk.AccAddress { return f.addrs[name] }
func (f fakeAccounts) GetModuleAccount(_ context.Context, name string) sdk.ModuleAccountI {
return authtypes.NewModuleAccount(authtypes.NewBaseAccount(f.addrs[name], nil, 0, 0), name)
}
func (f fakeAccounts) AddressCodec() coreaddress.Codec { return address.NewBech32Codec("genesis") }
func (f fakeAccounts) NewAccount(_ context.Context, acc sdk.AccountI) sdk.AccountI {
return acc
}
func (f fakeAccounts) NewAccountWithAddress(_ context.Context, addr sdk.AccAddress) sdk.AccountI {
return authtypes.NewBaseAccountWithAddress(addr)
}
func (f fakeAccounts) GetAccount(_ context.Context, _ sdk.AccAddress) sdk.AccountI {
return nil
}
func (f fakeAccounts) GetAllAccounts(_ context.Context) []sdk.AccountI { return nil }
func (f fakeAccounts) HasAccount(_ context.Context, _ sdk.AccAddress) bool { return true }
func (f fakeAccounts) SetAccount(_ context.Context, _ sdk.AccountI) {}
func (f fakeAccounts) IterateAccounts(_ context.Context, _ func(sdk.AccountI) bool) {
}
func (f fakeAccounts) ValidatePermissions(_ sdk.ModuleAccountI) error { return nil }
func (f fakeAccounts) GetModuleAddressAndPermissions(name string) (sdk.AccAddress, []string) {
return f.addrs[name], nil
}
func (f fakeAccounts) GetModuleAccountAndPermissions(_ context.Context, name string) (sdk.ModuleAccountI, []string) {
return authtypes.NewModuleAccount(authtypes.NewBaseAccount(f.addrs[name], nil, 0, 0), name), nil
}
func (f fakeAccounts) SetModuleAccount(_ context.Context, _ sdk.ModuleAccountI) {}
func (f fakeAccounts) GetModulePermissions() map[string]authtypes.PermissionsForAddress {
return nil
}
type fakeStaking struct {
slashingtypes.StakingKeeper
vals map[string]stakingtypes.Validator
codec coreaddress.Codec
}
func (f fakeStaking) IsValidatorJailed(_ context.Context, _ sdk.ConsAddress) (bool, error) {
return false, nil
}
func (f fakeStaking) ValidatorByConsAddr(_ context.Context, addr sdk.ConsAddress) (stakingtypes.ValidatorI, error) {
v, ok := f.vals[addr.String()]
if !ok {
return stakingtypes.Validator{}, fmt.Errorf("decider: unknown validator %s", addr.String())
}
return v, nil
}
func (f fakeStaking) ValidatorAddressCodec() coreaddress.Codec { return f.codec }
func (f fakeStaking) ConsensusAddressCodec() coreaddress.Codec {
return address.NewBech32Codec("genesisvalcons")
}
func (f fakeStaking) IterateValidators(_ context.Context, _ func(int64, stakingtypes.ValidatorI) bool) error {
return nil
}
func (f fakeStaking) Validator(_ context.Context, _ sdk.ValAddress) (stakingtypes.ValidatorI, error) {
return stakingtypes.Validator{}, fmt.Errorf("decider: no validator by operator")
}
func (f fakeStaking) Delegation(_ context.Context, _ sdk.AccAddress, _ sdk.ValAddress) (stakingtypes.DelegationI, error) {
return stakingtypes.Delegation{}, fmt.Errorf("decider: no delegation")
}
func (f fakeStaking) IterateDelegations(_ context.Context, _ sdk.AccAddress, _ func(int64, stakingtypes.DelegationI) bool) error {
return nil
}
func (f fakeStaking) GetAllSDKDelegations(_ context.Context) ([]stakingtypes.Delegation, error) {
return nil, nil
}
func (f fakeStaking) GetAllValidators(_ context.Context) ([]stakingtypes.Validator, error) {
return nil, nil
}
func (f fakeStaking) GetAllDelegatorDelegations(_ context.Context, _ sdk.AccAddress) ([]stakingtypes.Delegation, error) {
return nil, nil
}
// ---- fixture ---------------------------------------------------------------
type fixture struct {
ms *rootmulti.Store
keys map[string]*storetypes.KVStoreKey
slash slashingkeeper.Keeper
distr distrkeeper.Keeper
bank bankkeeper.BaseKeeper
valOp []string // operator bech32 per validator (fixed order A,B,C)
valCon [][]byte // consensus address bytes per validator
feeCol sdk.AccAddress
}
func buildFixture(t *testing.T) fixture {
t.Helper()
return buildFixtureOn(t, dbm.NewMemDB())
}
func buildFixtureOn(t *testing.T, db dbm.DB) fixture {
t.Helper()
keys := storetypes.NewKVStoreKeys("slashing", "distribution", "bank")
ms := rootmulti.NewStore(db, log.NewNopLogger(), metrics.NewNoOpMetrics())
// mount in SORTED order: Go map iteration is randomized, and mount
// order must not influence the harness (control for order artifacts).
for _, name := range []string{"bank", "distribution", "slashing"} {
ms.MountStoreWithDB(keys[name], storetypes.StoreTypeIAVL, nil)
}
require.NoError(t, ms.LoadLatestVersion())
interfaceRegistry := codectypes.NewInterfaceRegistry()
cdc := codec.NewProtoCodec(interfaceRegistry)
addrs := map[string]sdk.AccAddress{
distrtypes.ModuleName: sdk.AccAddress([]byte("distribution______")),
"fee_collector": sdk.AccAddress([]byte("fee_collector______")),
}
accts := fakeAccounts{addrs: addrs}
auth, err := address.NewBech32Codec("genesis").BytesToString([]byte("decider-authority____"))
require.NoError(t, err)
valCodec := address.NewBech32Codec("genesisvaloper")
valAddrs := [][]byte{[]byte("validatorA__________"), []byte("validatorB__________"), []byte("validatorC__________")}
conAddrs := [][]byte{[]byte("consensusA__________"), []byte("consensusB__________"), []byte("consensusC__________")}
vals := map[string]stakingtypes.Validator{}
var ops []string
for i := range valAddrs {
op, err := valCodec.BytesToString(valAddrs[i])
require.NoError(t, err)
ops = append(ops, op)
vals[sdk.ConsAddress(conAddrs[i]).String()] = stakingtypes.Validator{
OperatorAddress: op,
ConsensusPubkey: nil,
Jailed: false,
Status: stakingtypes.Bonded,
Tokens: math.NewInt(3000),
DelegatorShares: math.LegacyNewDec(3000),
Description: stakingtypes.Description{Moniker: fmt.Sprintf("val%d", i)},
Commission: stakingtypes.Commission{
CommissionRates: stakingtypes.CommissionRates{Rate: math.LegacyNewDecWithPrec(1, 1)},
},
}
}
staking := fakeStaking{vals: vals, codec: valCodec}
bank := bankkeeper.NewBaseKeeper(cdc, runtime.NewKVStoreService(keys["bank"]), accts, map[string]bool{}, auth, log.NewNopLogger())
slash := slashingkeeper.NewKeeper(cdc, codec.NewLegacyAmino(), runtime.NewKVStoreService(keys["slashing"]), staking, auth)
distr := distrkeeper.NewKeeper(cdc, runtime.NewKVStoreService(keys["distribution"]), accts, bank, staking, "fee_collector", auth)
ctx := sdk.NewContext(ms, cmtproto.Header{ChainID: "decider", Height: 1}, false, log.NewNopLogger())
require.NoError(t, slash.SetParams(ctx, slashingtypes.DefaultParams()))
distr.InitGenesis(ctx, *distrtypes.DefaultGenesisState())
for _, c := range conAddrs {
require.NoError(t, slash.SetValidatorSigningInfo(ctx, sdk.ConsAddress(c),
slashingtypes.NewValidatorSigningInfo(sdk.ConsAddress(c), 1, 0, time.Unix(0, 0), false, 0)))
}
genCodec := address.NewBech32Codec("genesis")
fcStr, err := genCodec.BytesToString(addrs["fee_collector"])
require.NoError(t, err)
bank.InitGenesis(ctx, banktypes.NewGenesisState(
banktypes.DefaultParams(),
[]banktypes.Balance{{Address: fcStr, Coins: sdk.NewCoins(sdk.NewCoin("ugen", math.NewInt(900000)))}},
sdk.NewCoins(sdk.NewCoin("ugen", math.NewInt(900000))),
[]banktypes.Metadata{},
[]banktypes.SendEnabled{},
))
return fixture{ms: ms, keys: keys, slash: slash, distr: distr, bank: bank, valOp: ops, valCon: conAddrs, feeCol: addrs["fee_collector"]}
}
// vote builds one VoteInfo for validator index i (power 1000, signed).
func (f fixture) vote(i int) abci.VoteInfo {
return abci.VoteInfo{
Validator: abci.Validator{Address: f.valCon[i], Power: 1000},
BlockIdFlag: cmtproto.BlockIDFlagCommit,
}
}
// run executes both BeginBlockers under one VoteInfos variant and commits.
func (f fixture) run(t *testing.T, votes []abci.VoteInfo) string {
return f.runSelective(t, votes, true, true)
}
// runSelective executes a subset (bisect helper).
func (f fixture) runSelective(t *testing.T, votes []abci.VoteInfo, slash, distr bool) string {
t.Helper()
header := cmtproto.Header{
ChainID: "decider",
Height: 100,
Time: time.Unix(1_000_000, 0).UTC(),
ProposerAddress: f.valCon[0],
}
ctx := sdk.NewContext(f.ms, header, false, log.NewNopLogger()).WithVoteInfos(votes)
ctx = ctx.WithEventManager(sdk.NewEventManager())
if slash {
require.NoError(t, slashing.BeginBlocker(ctx, f.slash))
}
if distr {
require.NoError(t, f.distr.BeginBlocker(ctx))
}
return hex.EncodeToString(f.ms.Commit().Hash)
}
// TestVoteSubsetDecider is the h7410 verdict matrix (permanent CI net).
//
// H2 verdicts (stable across processes):
// member-diff (C1, C3) splits: PROVEN. Fewer/different signers change
// slashing liveness records + distribution reward splits (previousTotalPower
// shifts every validator's share). This is inherited SDK behavior for
// node-local DecidedLastCommit subsets under gossip asynchrony.
// v3 code is uninvolved (its paths never execute on empty blocks).
// The inequality assertions below LOCK this behavior: any future change
// (e.g. vote canonicalization) visibly flips them for review.
// order-diff (C2) splits in-test but is UNREACHABLE via consensus:
// CometBFT index-orders commit votes (VoteSet.MakeExtendedCommit), so the
// app never receives reordered full sets. Logged, not asserted.
func TestVoteSubsetDecider(t *testing.T) {
variants := map[string]func(f fixture) []abci.VoteInfo{
"C0-baseline": func(f fixture) []abci.VoteInfo { return []abci.VoteInfo{f.vote(0), f.vote(1), f.vote(2)} },
"C1-member-diff": func(f fixture) []abci.VoteInfo { return []abci.VoteInfo{f.vote(0), f.vote(1)} },
"C2-order-diff": func(f fixture) []abci.VoteInfo { return []abci.VoteInfo{f.vote(2), f.vote(1), f.vote(0)} },
"C3-both": func(f fixture) []abci.VoteInfo { return []abci.VoteInfo{f.vote(1), f.vote(0)} },
}
order := []string{"C0-baseline", "C1-member-diff", "C2-order-diff", "C3-both"}
hashes := map[string]string{}
for _, name := range order {
f := buildFixture(t)
hashes[name] = f.run(t, variants[name](f))
t.Logf("verdict: %s apphash=%s", name, hashes[name])
}
// determinism: same inputs replay identically
f := buildFixture(t)
require.Equal(t, hashes["C0-baseline"], f.run(t, variants["C0-baseline"](f)),
"same inputs must replay identically")
// H2 mechanism lock: member-diff MUST diverge (liveness + rewards move)
require.NotEqual(t, hashes["C0-baseline"], hashes["C1-member-diff"],
"H2 member-diff must split (slashing/distribution write vote-dependent state)")
require.NotEqual(t, hashes["C0-baseline"], hashes["C3-both"],
"H2 member+order diff must split")
// order-diff is informational (unreachable input class in production)
t.Logf("order-diff C2==C0: %v (informational only)", hashes["C2-order-diff"] == hashes["C0-baseline"])
}
// TestIAVLInsertionOrder pins the store-layer property: bare-tree roots
// depend on key insertion order (same set, different order -> different
// root; same order replays identically). Single-bucket Go-map rotations
// converge in practice, but arbitrary permutationsSource: cosmos/cosmos-sdk