coli serve: RAM auto-detect returns 0.0 GB and silently falls back to a tiny expert cache (no warning)

Summary

coli serve (via openai_server.py) sometimes fails to auto-detect available RAM and silently falls back to a very small expert-cache budget, without warning the user. On a 128 GB Apple M3 Max, starting coli serve --model <olmoe_dir> with no --ram/--cap flags produced:

[cache] 1 slots/layer of 64 experts: 2.0 GB budget (88% of what the OS still offers), 2.0 GB dense resident, ...

and /health reported:

"hwinfo":{"cores":16,"ram_total_gb":0.0,"ram_avail_gb":0.0,"gpus":0,"vram_total_gb":0.0,"cpu":"unknown","gpu":""}

i.e. RAM detection returned 0.0 GB total/available on a machine that actually has 128 GB, and the cache sizing logic used a small fallback (2 GB) instead of anything close to the ~85 GB coli doctor calculates as viable for the same model on the same host.

Impact

For a small model (OLMoE-7B, fully resident at ~7 GB with the right cache size), this silently limited decode throughput: with the auto default, effective decode throughput was well below the residency-based projection; passing explicit --ram 96 --cap 64 (matching coli doctor's own recommended plan) raised measured decode from a degraded/uncharacterized baseline to 25.8 tok/s cold / 24.9 tok/s warm on the same hardware and model. There is no warning printed when the RAM auto-detect falls back to 0/small values, so the degradation is invisible unless the user happens to inspect /health or compare against coli doctor's numbers.

Repro

cd c
make olmoe
python3 tools/convert_olmoe_merged.py --repo allenai/OLMoE-1B-7B-0125-Instruct --out ./olmoe_merged
python3 ./coli doctor --model ./olmoe_merged
#  -> RAM 85.0 GB budget · ... cap 64/layer

python3 ./coli serve --model ./olmoe_merged --port 8123 &
curl -s http://127.0.0.1:8123/health
#  -> hwinfo.ram_total_gb: 0.0, hwinfo.ram_avail_gb: 0.0
#  -> serve log: "[cache] 1 slots/layer of 64 experts: 2.0 GB budget ..."

# workaround: explicit --ram/--cap fixes it
python3 ./coli serve --model ./olmoe_merged --port 8123 --ram 96 --cap 64
#  -> hwinfo populates correctly (ram_total_gb: 137.4, ram_avail_gb: 80.2 when reproduced on a second run)
#  -> cache sizes to the full residency plan, decode throughput matches doctor's projection

Reproduced on macOS 26.6.2, Apple M3 Max, 128 GB unified memory, colibri v1.11.0 built from source. It's possible this is specific to how openai_server.py's RAM probe behaves in this environment (no swap configured at the time, sysctl vm.swapusage reported 0 initially) rather than a generic macOS issue, but there's no diagnostic output distinguishing "detected 0 GB, using safe fallback" from "everything is fine" — worth at least a warning/log line when the detector returns 0 or an implausibly low value so users don't silently get a degraded cache.

Suggested fix

• Surface a warning when RAM auto-detection returns 0 (or below some sane floor) instead of silently sizing the cache off it.
• Consider having coli serve reuse the same RAM-detection path coli doctor already uses successfully, if they currently differ.