No checkpoints saved during pretraining in Google Colab despite log.log_model and save_interval set.

Bug description

Hello,

I've been troubleshooting this issue in a Google Colab environment and wanted to share my findings, as they strongly align with the observations in this thread.

Troubleshooting Steps & Parameters Used:

• Initial setup with bf16-mixed precision (suitable for T4 GPU).
• Adjusted train.max_steps and train.save_interval (e.g., both set to 5 for a short run).
• Ensured log.log_model was explicitly set to true (--log.log_model true).
• Explicitly set log.checkpoint_name to 'last' (--log.checkpoint_name 'last')
• Corrected CLI command format (!python -m litgpt pretrain ...).
• Removed train.max_tokens to avoid premature termination, ensuring max_steps was respected.
• Confirmed litgpt[all] and other dependencies were installed.

Observed Outcome: Despite all these modifications and successful execution of the pretrain command (for short durations, confirming a save should have occurred), no model checkpoints (.ckpt, .pth) or model_config.yaml files were ever saved in the specified out_dir (e.g., out/custom-model/). Only TensorBoard log files were consistently generated within out/custom-model/logs/tensorboard/.

A find command within the output directory confirmed the complete absence of any checkpoint-related files.

This behavior suggests an underlying issue within the litgpt framework regarding checkpoint saving, at least in the Google Colab environment, and is consistent with what others have reported here. My testing indicates that even with explicit log.log_model and train.save_interval settings, no checkpoints are being produced.

I hope this detailed context helps in diagnosing the root cause. Please let me know if there are any further tests I can perform. Given these findings, I believe this issue remains relevant.

Thank you.

Reproduced in studio

No response

What operating system are you using?

Windows

LitGPT Version

I have attached the Colab notebook I used for these experiments so you can directly reproduce the behavior. If needed, I can also provide a minimal version of the notebook with fewer dependencies and a smaller dataset to make debugging easier. Please let me know if there are any specific logging flags, environment variables, or debug modes you would like me to enable in a follow-up run so I can share more detailed logs or traces.

litegpt_1402 (1).ipynb