Can't quantize nemotron 3.5
Author: supersimple33Created Aug 19, 2026Updated Aug 19, 2026
System Info
Hello when trying to quantize nemotron 3.5 lightning the model barely reduces in size and it seems most is still saved in bf16. (version 0.50.1 running on fedora with a100s) [transformer 5.15.0, torch 2.13)
Reproduction
tokenizer = AutoTokenizer.from_pretrained(nemo_dir)
model = AutoModelForCausalLM.from_pretrained(
nemo_dir,
quantization_config=quant_config,
)
model.save_pretrained(
quantized_output_dir,
safe_serialization=True,
) # the output here is only roughly 3GB less Expected behavior
The model should be at least half the size
Source: bitsandbytes-foundation/bitsandbytes