Add TinyStories to the pretraining docs
Author: rasbtCreated Mar 11, 2024Updated May 23, 2026
Labelsdocumentation
As far as I know, one can pretrain fine using TinyStories:
litgpt/pretrain.py --data litgpt.data.TinyStories Should we add this to the documentation?
Right now, we only have a pretrain_tinyllama.md doc that pretrains on SlimPajama and StarCoder.
What I propose is
- create a general
pretraining.md - inside this document
- explain the general pretraining script
- add the
pretrain_tinyllama.mdcontents as a section there - add a TinyStories section
What do you think?
Source: Lightning-AI/litgpt