[Bug] Wallet state desynchronization on reorg when Layer-1 rollback bypassed (_add_coin_state ignores created_height=None)
What happened?
Summary
In the Chia light wallet architecture, blockchain reorganizations are primarily handled via block-level rollback (Layer 1: wallet_short_sync_backtrack() calling perform_atomic_rollback() when receiving NewPeakWallet).
As a secondary defense-in-depth layer, incoming CoinStateUpdate messages segregate reorged coin states (CoinState(coin, spent_height=None, created_height=None)) in WalletNode.add_states_from_peer() (chia/wallet/wallet_node.py:1104-1118) and dispatch them directly to WalletStateManager.add_coin_states() upfront.
However, inside WalletStateManager._add_coin_state() (chia/wallet/wallet_state_manager.py:1211-1214), handling of this reorg invalidation signal is completely unimplemented:
if coin_state.created_height is None:
# TODO implements this coin got reorged
# TODO: we need to potentially roll back the pool wallet here
passBecause this branch is a bare pass, the wallet has zero coin-level invalidation redundancy. Whenever Layer 1 rollback fails to execute or is bypassed, the wallet suffers permanent on-disk state desynchronization.
Failure Modes Bypassing Layer 1 Rollback
Layer 1 short-sync rollback does not execute or reach the coin in several realistic conditions:
- Peer Disconnection: The full node sends
CoinStateUpdatefollowed byNewPeakWallet. If the connection drops between these messages, the wallet processes theCoinStateUpdate(hitting thepass) but never receives theNewPeakWalletthat triggers backtrack. - Early Rejection of
new_peak_wallet: The peak handler aborts before backtrack if peer timestamp skew > 600s, weight is non-advancing, or header validation fails. - Secondary Peer Sync Rollback Suppression: In
long_sync_from_untrusted(), secondary peer sync tasks run withrollback=False. Peak updates through this path bypassperform_atomic_rollback()completely. - Offline Reorgs / Reconnection: When reconnecting after an offline reorg where canonical headers converge above the fork point.
Impact & Why Long Resync Cannot Recover
- Phantom Balance: The reorged coin remains in SQLite marked
confirmed=1andspent=0. Production RPC endpoints (/get_wallet_balance,/get_transactions) report the phantom coin as spendable. - Spend Construction Lockout: When spending funds,
wallet.select_coins()selects the phantom coin and broadcasts a spend bundle. Consensus nodes reject it withUNKNOWN_UNSPENT, leaving the transaction trapped intx_storeas pending unconfirmed, blocking further spends. - Canonical Re-inclusion Failure: If the reorged coin is subsequently mined on the canonical chain at a new block height (e.g. height 102 vs 100),
_add_coin_state()evaluateselif coin_state.spent_height is None: if local_record is None:. Becauselocal_recordalready exists in SQLite (at height 100), the check evaluates toFalse. The update is dropped, leaving the confirmation height permanently corrupted. - Persistence Across Restarts: Standard background resync calculates
fork_point_syncing = min(current_height - 16, wp_fork_point). If canonical headers converged past the 16-block window, the rollback never reaches the orphaned coin, and the wallet store has no differential deletion logic against full node responses. The only recovery today is deleting the local SQLite database.
Steps to Reproduce
- Connect a light wallet instance and confirm a coin at height 100.
- Deliver a
CoinStateUpdatecontainingCoinState(coin, spent_height=None, created_height=None). - Disconnect the peer prior to
NewPeakWalletor route through secondary peer sync whererollback=False. - Query
/get_wallet_balanceor check SQLitecoin_record: the phantom coin remains confirmed and spendable. - Push a canonical re-inclusion
CoinState(coin, spent_height=None, created_height=uint32(102)):coin_record.confirmed_block_heightremains stuck at 100. - Attempt to spend the coin: transaction is rejected by consensus with
UNKNOWN_UNSPENTand stalls in the unconfirmed queue.
Expected Behavior
- Processing
CoinState(coin, None, None)in_add_coin_state()should:- Delete the coin record from
coin_store. - Reconcile
tx_store: delete incoming/reward transaction records, reset outgoing transactions where the coin was change usingtx_store.tx_reorged(), and purge invalid unconfirmed spends of the phantom coin. - Dispatch
coin_removedWebSocket events viasync_scope.
- Delete the coin record from
- Processing
CoinState(coin, None, new_height)whenlocal_recordexists should updateconfirmed_block_heightin SQLite and update associated transaction records. - Processing
CoinState(coin, None, created_height)whenlocal_recordwas marked spent should restorespent=Falseandspent_height=0.
Version
2.7.4-rc2 (main branch @ commit 1621edbc5; bug present since commit e84ca53a2)
What platform are you using?
Linux
What ui mode are you using?
CLI
Relevant log output
# Root cause code path (chia/wallet/wallet_state_manager.py:1211-1214):
if coin_state.created_height is None:
# TODO implements this coin got reorged
# TODO: we need to potentially roll back the pool wallet here
pass
# Segregation in WalletNode (chia/wallet/wallet_node.py:1114-1118):
for batch in to_batches(reorged_coin_states, chunk_size):
self.log.info(f"Process reorged states: ({len(batch.entries)} / {len(reorged_coin_states)})")
if not await self.wallet_state_manager.add_coin_states(batch.entries, peer, fork_height):
self.log.debug("Processing reorged states failed")
return False
# Empirical Verification Test Suite Output (Live WalletNode / SQLite / RPC):
===========================================================================
CHIA WALLET REORG INGESTION GAP EMPIRICAL VERIFICATION
===========================================================================
[TEST 1] Peer Trust Branching Analysis
[+] Coin confirmed at height 100 in local SQLite DB
[!] Untrusted mode state_update_received: coin still in DB = True
[+] Trusted mode state_update_received: coin still in DB = False
[TEST 2] Production RPC Handler Visibility (/get_wallet_balance, /get_transactions)
[!] RPC confirmed_wallet_balance: 50000000000 mojo
[!] RPC spendable_balance: 50000000000 mojo
[!] RPC tx status: confirmed=True, confirmed_at_height=100
[TEST 3] Next State Transition (Self-Healing Test)
[+] Simulating canonical chain re-inclusion at height 102...
[!] DB confirmed_block_height: 100
[+] CONFIRMED: Height is STUCK at 100 (self-healing failed)!
[TEST 4] Constructing Signed Transaction from Phantom State
[!] Total transactions in tx_store: 2
[!] Pending unconfirmed transactions: 1
[+] CONFIRMED: Wallet constructed & signed spend bundle for phantom coin!
[!] Consensus rejection on broadcast: Err.UNKNOWN_UNSPENT
===========================================================================
ALL 4 EMPIRICAL AUDIT CHECKS COMPLETED AND VERIFIED!
===========================================================================Source: Chia-Network/chia-blockchain