#848·nanochat

SFT inherits pre-batch-scaling learning rates from base checkpoint

Author: QwertyemmaCreated Sep 3, 2026Updated Sep 9, 2026

I noticed a possible issue with how learning rates are passed from base training to SFT.

In base_train.py, the learning rates are scaled using batch_lr_scale before being passed to model.setup_optimizer():

unembedding_lr=args.unembedding_lr * batch_lr_scale, embedding_lr=args.embedding_lr * batch_lr_scale, scalar_lr=args.scalar_lr * batch_lr_scale, matrix_lr=args.matrix_lr * batch_lr_scale,

However, user_config is created earlier from the original args:

user_config = vars(args).copy()

and the checkpoint later saves that user_config:

"user_config": user_config,

So the checkpoint contains the learning rates before batch_lr_scale is applied, while the optimizer during base training uses the scaled values.

In chat_sft.py, the SFT script reads the learning rates from the base checkpoint when they aren't explicitly provided:

pretrain_user_config = meta.get("user_config", {})

and then inherits values such as embedding_lr, unembedding_lr, and matrix_lr from it.

This means that when the base training batch size is different from B_REF, SFT can inherit a learning rate that is different from the effective learning rate used during base training.

I'm not completely sure whether user_config is intended to store the original user-provided values or the effective training configuration, so I wanted to raise this before making a PR.

If the intention is for SFT to inherit the effective base-training learning rates, I think the scaled values should be reflected in the checkpoint metadata, or alternatively the scaling information should be stored so that SFT can reconstruct the effective values.