SFT inherits pre-batch-scaling learning rates from base checkpoint
I noticed a possible issue with how learning rates are passed from base training to SFT.
In base_train.py, the learning rates are scaled using batch_lr_scale before being passed to model.setup_optimizer():
unembedding_lr=args.unembedding_lr * batch_lr_scale, embedding_lr=args.embedding_lr * batch_lr_scale, scalar_lr=args.scalar_lr * batch_lr_scale, matrix_lr=args.matrix_lr * batch_lr_scale,
However, user_config is created earlier from the original args:
user_config = vars(args).copy()
and the checkpoint later saves that user_config:
"user_config": user_config,
So the checkpoint contains the learning rates before batch_lr_scale is applied, while the optimizer during base training uses the scaled values.
In chat_sft.py, the SFT script reads the learning rates from the base checkpoint when they aren't explicitly provided:
pretrain_user_config = meta.get("user_config", {})
and then inherits values such as embedding_lr, unembedding_lr, and matrix_lr from it.
This means that when the base training batch size is different from B_REF, SFT can inherit a learning rate that is different from the effective learning rate used during base training.
I'm not completely sure whether user_config is intended to store the original user-provided values or the effective training configuration, so I wanted to raise this before making a PR.
If the intention is for SFT to inherit the effective base-training learning rates, I think the scaled values should be reflected in the checkpoint metadata, or alternatively the scaling information should be stored so that SFT can reconstruct the effective values.
Source: karpathy/nanochat