perf(storage): optimize NVMe-oF and Ceph TCP socket queue depths and BBR congestion control on k3s worker nodes

Performance Optimization Proposal

During high-concurrency storage replication bursts (e.g. Ceph OSD peering, Longhorn volume rebuilds, and distributed Rook snapshots) across bare-metal worker nodes, default Linux socket memory boundaries (rmem_default, wmem_default) and standard cubic congestion control generate micro-burst queue packet drops.

Proposed Parameter Tuning

1. TCP Socket Memory Windows:
   • net.core.rmem_max = 67108864 (64MB)
   • net.core.wmem_max = 67108864 (64MB)
   • net.ipv4.tcp_rmem = 4096 87380 67108864
   • net.ipv4.tcp_wmem = 4096 65536 67108864
2. Ingress Queue Backlog Depth:
   • net.core.netdev_max_backlog = 300000
   • net.core.somaxconn = 65535
3. TCP BBR & Fair Queueing:
   • net.core.default_qdisc = fq
   • net.ipv4.tcp_congestion_control = bbr

These settings eliminate packet drops during 10GbE multi-gigabit storage synchronization while keeping latency minimal.