Make `save_hyperparameters()` robust against different CLI entry points

Author: awaelchliCreated Mar 13, 2024Updated Aug 30, 2026

Labelsbughelp wanted

If you run with

litgpt finetune ...

and when getting to saving a checkpoint, we hit this line: https://github.com/Lightning-AI/litgpt/blob/f951f9334610da35c7ecaa7e26e7ba3ac2504dab/litgpt/finetune/lora.py#L193

which reruns the CLI and parses the args that were passed. But this no longer works because it's not the same parser.

Saving LoRA weights to 'out/finetune/lora-llama2-7b/step-000200/lit_model.pth.lora'
usage: litgpt [-h] [--config CONFIG] [--print_config[=flags]] [--precision PRECISION] [--quantize QUANTIZE] [--devices DEVICES] [--seed SEED] [--lora_r LORA_R]
              [--lora_alpha LORA_ALPHA] [--lora_dropout LORA_DROPOUT] [--lora_query {true,false}] [--lora_key {true,false}] [--lora_value {true,false}]
              [--lora_projection {true,false}] [--lora_mlp {true,false}] [--lora_head {true,false}] [--data.help CLASS_PATH_OR_NAME] [--data DATA]
              [--checkpoint_dir CHECKPOINT_DIR] [--out_dir OUT_DIR] [--logger_name {wandb,tensorboard,csv}] [--train CONFIG] [--train.save_interval SAVE_INTERVAL]
              [--train.log_interval LOG_INTERVAL] [--train.global_batch_size GLOBAL_BATCH_SIZE] [--train.micro_batch_size MICRO_BATCH_SIZE]
              [--train.lr_warmup_steps LR_WARMUP_STEPS] [--train.epochs EPOCHS] [--train.max_tokens MAX_TOKENS] [--train.max_steps MAX_STEPS]
              [--train.max_seq_length MAX_SEQ_LENGTH] [--train.tie_embeddings {true,false,null}] [--train.learning_rate LEARNING_RATE]
              [--train.weight_decay WEIGHT_DECAY] [--train.beta1 BETA1] [--train.beta2 BETA2] [--train.max_norm MAX_NORM] [--train.min_lr MIN_LR] [--eval CONFIG]
              [--eval.interval INTERVAL] [--eval.max_new_tokens MAX_NEW_TOKENS] [--eval.max_iters MAX_ITERS]
error: Unrecognized arguments: finetune lora

A initial hack to fix this was done in #1103. Comment by @carmocca https://github.com/Lightning-AI/litgpt/pull/1103#discussion_r1523182612

How do you think this could be done? Do we need to choose between jsonargparse.CLI or the CLI in main and then pass the correct one to capture_parser?

We could also simplify this by not having a CLI in the scripts themselves.

We need to make it more robust.

Source: Lightning-AI/litgpt

View original on GitHub View discussion on GitHub