--max-database-cache-size is applied per RocksDB instance, not as a process-wide budget

Summary

--max-database-cache-size is documented/used as a single cache-size knob, but CombinedDatabase::open opens multiple independent RocksDB instances and passes the same DatabaseConfig.cache_capacity to each of them. That multiplies intended cache RAM by the number of DBs and can push the process toward OOM, especially when operators size the flag as if it were a global budget. There is already an in-code acknowledgment:

// TODO: Use different cache sizes for different databases

in crates/fuel-core/src/combined_database.rs (CombinedDatabase::open).

RocksDB instances opened

CombinedDatabase::open creates a separate RocksDB for each logical database:

#	Database	Always?
1	on_chain	yes
2	off_chain	yes
3	relayer	yes
4	gas_price	yes
5	compression	yes
6	block_aggregation_storage	only with rpc feature (enabled in production)
So production-shaped builds open 6 RocksDBs; non-rpc rocksdb builds open 5.

How the flag is wired

1. CLI (bin/fuel-core/src/cli/run.rs):
   • --max-database-cache-size → max_database_cache_size: Option<usize>
   • Mapped into CombinedDatabaseConfig.database_config.cache_capacity
2. CombinedDatabase::open clones that config for every DB:
   • max_fds is divided (saturating_div(4)) across DBs
   • cache_capacity is not divided — each DB gets the full value (DatabaseConfig { max_fds, ..database_config })
3. Per-instance allocation (crates/fuel-core/src/state/rocks_db.rs in RocksDb::open_with):
   • block cache = capacity / 3
   • row cache = capacity / 3
   • remaining ~1/3 left for other RocksDB memory use per the local comments

Impact

If an operator sets --max-database-cache-size=N expecting ~N bytes of RocksDB cache for the process:

• Actual configured cache budget is closer to 5×N or 6×N (one full capacity per DB), before other RocksDB / process overhead.
• Smaller / colder DBs (e.g. relayer, gas price, compression) do not need the same cache as on-chain/off-chain, so RAM is wasted on them.
• To avoid OOM, operators must set a lower value than the “real” desired on-chain cache, which under-provisions the hot DBs. This is especially painful in memory-constrained environments (k8s limits, sentry/archive nodes) where OOMKill is a real failure mode.

Contrast with FD handling

File descriptors are already shared across DBs (max_fds / 4), while cache capacity is not. Note the divisor 4 also looks stale relative to the current 5–6 DB count — related cleanup opportunity when fixing cache budgeting.

Expected behavior (proposal)

• Expose per-database cache knobs (or a weighted profile) instead of a single flat value that is for a single DB instance only.

Reproduction / inspection

1. Start fuel-core with RocksDB and --max-database-cache-size=<N>.
2. Observe startup log from CombinedDatabase::from_config (single cache size logged once).
3. Confirm in CombinedDatabase::open that each Database::open_rocksdb(...) receives the same cache_capacity.
4. Compare RSS / cgroup memory vs N — growth tracks roughly instances × N, not N.

Related code

• bin/fuel-core/src/cli/run.rs — --max-database-cache-size
• crates/fuel-core/src/combined_database.rs — CombinedDatabase::open (TODO + shared cache_capacity)
• crates/fuel-core/src/state/rocks_db.rs — block/row cache from cache_capacity