#1082·litgpt

Add TinyStories to the pretraining docs

Author: rasbtCreated Mar 11, 2024Updated May 23, 2026
Labelsdocumentation

As far as I know, one can pretrain fine using TinyStories:

python
litgpt/pretrain.py --data litgpt.data.TinyStories 

Should we add this to the documentation?

Right now, we only have a pretrain_tinyllama.md doc that pretrains on SlimPajama and StarCoder.

What I propose is

  • create a general pretraining.md
  • inside this document
    • explain the general pretraining script
    • add the pretrain_tinyllama.md contents as a section there
    • add a TinyStories section

What do you think?