Gradient Clipping Doesn't Work in Finetuning

This one's frustrating because the config files SHOW a max_norm parameter, but it doesn't actually do anything during finetuning.

# litgpt/finetune/lora.py:560
unsupported = [(train, ["max_tokens", "max_norm", "tie_embeddings", "lr_warmup_fraction"])]

Meanwhile in pretraining:

# litgpt/pretrain.py:360
fabric.clip_gradients(model, optimizer, max_norm=train.max_norm)

Why this is a problem:

Gradient clipping is pretty important for stable training, especially with:

• Small datasets (where you're more likely to hit bad batches)
• Long sequences
• QLoRA (where quantization noise can cause gradient spikes)
• Any time you're near the edge of numerical stability

The config files in config_hub/finetune/ all have max_norm: set to empty/null. Users will assume this means "no clipping" when it actually means "this parameter is ignored."

The fix:

Just implement it. The code's already there in pretrain.py - copy it over. Make sure it works with:

• Regular LoRA
• QLoRA
• Full finetuning
• Adapter training

Or if you really don't want to support it, remove it from the configs and document why.