suix_getBalance intermittently returns a huge u128 value for a small-balance address
Title
suix_getBalance intermittently returns a huge u128 value for a small-balance address
Body
Steps to Reproduce Issue
The issue is intermittent and appears to be node-local. The same suix_getBalance request can return a normal value on one fullnode and a huge value on another fullnode at roughly the same time. Repeating the same request later against the same node may return the normal value again.
- Call
suix_getBalancefor this mainnet address and the SUI coin type:
curl --location 'https://rpc.mainnet.sui.io' \
--header 'content-type: application/json' \
--data '{
"method": "suix_getBalance",
"jsonrpc": "2.0",
"params": [
"0x8f1ab753c0dffbb296541b2ad22ada6437f17ed9e102dece186e4a6575ad3e68",
"0x2::sui::SUI"
],
"id": "1"
}'Repeat the same request against multiple fullnodes at approximately the same time.
If one node returns a huge value, compare it with the sum of balances returned by
suix_getAllCoinsfor the same owner and coin type:
suix_getAllCoins(owner, "0x2::sui::SUI")- Repeat the same
suix_getBalancerequest later against the node that returned the huge value.
Optional diagnostic step:
Start a fullnode with the JSON-RPC index cache disabled and retry the same balance query:
DISABLE_INDEX_CACHE=1 sui-node --config-path /path/to/fullnode.yamlThe current source enables this variable when its parsed value is greater than zero:
read_size_from_env("DISABLE_INDEX_CACHE").unwrap_or(0) > 0Expected Result
suix_getBalance should return the actual SUI balance for the address.
If the node detects an internally inconsistent balance state, it should return a JSON-RPC error instead of returning a valid-looking huge balance.
The API should not return a value close to u128::MAX for an address whose actual balance is small.
Actual Result
Occasionally, suix_getBalance returns a huge value close to u128::MAX.
One observed value was:
340282366920938463463374607356230107723This value looks like a signed-to-unsigned conversion of a negative i128 balance:
2^128 = 340282366920938463463374607431768211456
2^128 - 340282366920938463463374607356230107723
= 75,538,103,733 MIST
= 75.538103733 SUISo the observed JSON-RPC value is equivalent to:
(-75_538_103_733_i128) as u128The behavior observed from RPC responses:
- Direct
curlrequests to a fullnode can return the huge value. - Different fullnodes can return different values for the same request at approximately the same time.
- The same node can return to the normal balance later.
Analysis
The suspicious path is the JSON-RPC balance index/cache path.
TotalBalance is stored internally with signed fields:
struct TotalBalance {
balance: i128,
num_coins: i64,
address_balance: u64,
}The JSON-RPC response path converts the signed balance into the unsigned API response. In the affected path, unchecked casts can expose a negative internal balance as a huge unsigned value:
coin_object_count: balance.num_coins as usize,
total_balance: balance.balance as u128,If balance.balance becomes negative due to a transient index/cache inconsistency, balance.balance as u128 wraps the negative value and returns a number close to u128::MAX.
A likely race is:
- The balance index computes a delta for a transaction.
commit_index_batchwrites the new DB state.- Before the cache delta is merged, an RPC balance request has a cache miss.
- That RPC request reads the already-updated DB state and fills the cache with the new balance.
- The commit path then merges the old delta into this already-new cache value.
- For a spend/delete path, this can double-apply the negative delta and make the cached
TotalBalance.balancenegative. - The RPC layer casts that negative
i128tou128, returning a huge value.
This would explain why the issue is intermittent, node-local, and can disappear after the cache entry is rebuilt or invalidated.
Suggested Fix
Since a PR submission was not available from my side, here are the suggested code changes directly.
1. Prevent RPC from exposing negative internal balances as huge u128
File:
crates/sui-json-rpc/src/coin_api.rsAdd a helper that uses checked conversions:
fn build_balance_response(coin_type: String, balance: TotalBalance) -> RpcInterimResult<Balance> {
let coin_object_count = usize::try_from(balance.num_coins).map_err(|_| {
Error::from(SuiErrorKind::ExecutionError(format!(
"negative coin object count in JSON-RPC balance cache for {coin_type}: {}",
balance.num_coins
)))
})?;
let total_balance = u128::try_from(balance.balance).map_err(|_| {
Error::from(SuiErrorKind::ExecutionError(format!(
"negative total balance in JSON-RPC balance cache for {coin_type}: {}",
balance.balance
)))
})?;
Ok(Balance {
coin_type,
coin_object_count,
total_balance,
locked_balance: Default::default(),
funds_in_address_balance: balance.address_balance as u128,
})
}Then use this helper in get_balance:
let balance = self
.internal
.get_balance(owner, coin_type_tag.clone())
.await
.tap_err(|e| {
debug!(?owner, "Failed to get balance with error: {:?}", e);
})?;
build_balance_response(coin_type_tag.to_string(), balance)And in get_all_balances:
all_balance
.iter()
.map(|(coin_type, balance)| build_balance_response(coin_type.to_string(), *balance))
.collect::<RpcInterimResult<Vec<_>>>()This prevents a negative internal balance from being serialized as a valid huge u128 balance.
2. Invalidate negative balance cache entries and recompute from DB
File:
crates/sui-core/src/jsonrpc_index.rsAdd helpers to detect negative cached balance values:
fn contains_negative_total_balance(balance: &SuiResult<TotalBalance>) -> bool {
matches!(balance, Ok(balance) if balance.balance < 0 || balance.num_coins < 0)
}
fn contains_negative_all_balance(balances: &SuiResult<Arc<AllBalance>>) -> bool {
matches!(balances, Ok(balances) if balances
.values()
.any(|balance| balance.balance < 0 || balance.num_coins < 0))
}In get_coin_object_balance, invalidate a negative per-coin cache entry instead of returning it:
let cache_key = (owner, coin_type.clone());
let balance = self.caches.per_coin_type_balance.get(&cache_key);
if let Some(balance) = balance {
if Self::contains_negative_total_balance(&balance) {
debug!(
?owner,
?coin_type,
"invalidating negative balance cache entry"
);
self.caches.per_coin_type_balance.invalidate(&cache_key);
} else {
return balance;
}
}Also reject a negative balance found through the all-balances cache fallback:
let all_balance = self.caches.all_balances.get(&owner);
if let Some(Ok(all_balance)) = all_balance
&& let Some(balance) = all_balance.get(&coin_type)
{
if balance.balance < 0 || balance.num_coins < 0 {
debug!(
?owner,
?coin_type,
"invalidating all-balance cache entry containing negative balance"
);
self.caches.all_balances.invalidate(&owner);
} else {
return Ok(*balance);
}
}In get_all_coin_object_balances, invalidate an all-balances cache entry if any cached coin balance is negative:
if let Some(all_balance) = self.caches.all_balances.get(&owner) {
if Self::contains_negative_all_balance(&all_balance) {
debug!(
?owner,
"invalidating all-balance cache entry containing negative balance"
);
self.caches.all_balances.invalidate(&owner);
} else {
return all_balance;
}
}The DB recomputation path sums persisted coin object balances, so it should rebuild a non-negative value under normal conditions.
Tests Added
Suggested tests for crates/sui-core/src/jsonrpc_index.rs:
test_negative_per_coin_balance_cache_is_recomputed_from_db
test_negative_all_balance_cache_is_recomputed_from_dbSuggested tests for crates/sui-json-rpc/src/coin_api.rs:
test_get_balance_negative_total_balance_returns_error
test_negative_total_balance_returns_errorSuggested validation commands:
cargo fmt --all -- --check
SUI_SKIP_SIMTESTS=1 cargo nextest run -p sui-core test_negative --no-fail-fast
SUI_SKIP_SIMTESTS=1 cargo nextest run -p sui-json-rpc negative_total_balance --no-fail-fastSource: MystenLabs/sui