[Feature Request] Support for INT4 (q4) Quantization in Rust/Candle Backend

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The Rust implementation

Question

Could the team prioritize adding INT4 (Q4_0/Q4_K) quantization support specifically for the moshi-backend Rust implementation, alongside optimized ARM NEON/SIMD kernels for the Mimi codec and Temporal Transformer layers?

I am developing an offline, full-duplex assistant running on a Raspberry Pi 5 paired with the Hailo-10H NPU (40 TOPS).

To achieve a great user experience, the system must maintain Moshi's native 12.5Hz (80ms per frame) cadence without thermal throttling or audio stutter. While the Hailo-10H features 8GB of dedicated memory, the 34 GB/s LPDDR4 bandwidth creates a hard physical bottleneck for a 7.6B model in INT8 (8GB * 12.5Hz$ far exceeds the bus capacity).

To maintain real-time streaming, we must reduce the weight-fetch payload to a 4-bit profile.

Excited to hear from you.