Feature Request: Add a CPU-Friendly "Train an LLM from Scratch" Learning Path for Students

Problem

This repository is an excellent resource for understanding how a Transformer-based language model is built from scratch. However, the current training workflow assumes access to a GPU and relatively capable hardware.

This can make the project difficult for students who only have:

• A normal laptop or desktop

• 4–8 GB RAM

• No dedicated GPU

• Limited storage

• CPU-only PyTorch

The educational goal should not require expensive hardware. Students should be able to understand the complete LLM pipeline and train a small, working language model from scratch on their own computer.

Proposed Solution

Add a dedicated CPU/Low-Hardware Student Mode to the repository.

The goal would be to provide a small configuration that allows students to:

1. Prepare a small dataset

2. Build a tokenizer

3. Understand token embeddings

4. Implement self-attention

5. Implement multi-head attention

6. Build Transformer blocks

7. Train a small decoder-only Transformer

8. Save checkpoints

9. Resume training

10. Generate text from the trained model

All of this should work without requiring a GPU.

Suggested Student Configuration

Instead of starting with millions or billions of parameters, provide a very small educational model.

Example:

Parameters:       ~1M–10M
Context length:   128–256 tokens
Layers:           2–4
Attention heads:  2–4
Embedding size:   128–256
Batch size:       1–8
Dataset:          1–100 MB
Precision:        float32
Device:           CPU

The exact configuration can be tuned based on benchmarking.

The important point is that a student should be able to run:

python train.py --device cpu --config student

and eventually obtain a working checkpoint.

Small Dataset

The current project uses the Pile, which is far too large for a beginner's first CPU experiment.

A much smaller educational dataset should be provided.

For example:

data/
├── student/
│   ├── train.txt
│   └── val.txt

The repository could provide a small sample dataset containing public-domain or appropriately licensed text.

Students could also replace it with their own dataset.

For example:

python prepare_data.py \
    --input data/student/train.txt \
    --output data/student/train.bin

CPU-Friendly Training

The training code should automatically detect available hardware:

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

But explicitly allowing CPU mode would be useful:

python train.py --device cpu

The README should explain that CPU training is intended for learning and experimentation, not for training large production-scale models.

Progressive Learning Path

I think the biggest improvement would be to provide multiple levels.

Level 1 — Tiny Transformer

Target:

~100K–500K parameters

Hardware:

4 GB RAM
CPU only

Purpose:

Understand how the Transformer works.

Expected result:

Training completes on a normal laptop.
Model generates simple text.

Level 2 — Small Language Model

Target:

~1M–10M parameters

Hardware:

8 GB RAM
CPU recommended
GPU optional

Purpose:

Understand actual language-model training.

Level 3 — GPU Training

Keep the existing larger configuration for users who have access to:

Google Colab
Kaggle
NVIDIA GPU

This creates a natural progression:

Tiny Model
     ↓
Small Model
     ↓
GPU Model
     ↓
Larger Models

Benchmarking

It would also be useful to document approximate training times for different hardware.

For example:

Actual numbers should be measured rather than estimated.

Educational CLI

A beginner-friendly CLI could make the project much easier to use:

python train.py --preset tiny

Available presets:

tiny
student
small
gpu

For example:

python train.py --preset tiny --device cpu

Then:

python generate.py \
    --checkpoint checkpoints/tiny.pt \
    --prompt "Once upon a time"

Documentation

Add a section such as:

Train an LLM on a Laptop — No GPU Required

Explain clearly:

You do not need a powerful GPU to learn how an LLM works. This configuration intentionally uses a very small Transformer so students can train and experiment with the complete pipeline on a CPU.

The documentation should also explain the important distinction:

Training a tiny LLM from scratch
                ≠
Training a ChatGPT-scale model

The objective is education and understanding, not competing with large commercial models.

Why This Would Be Valuable

This would make the repository much more accessible to:

• High-school students

• College students

• Beginners learning AI

• Researchers with limited hardware

• Students using old laptops

• People learning PyTorch

• People without access to cloud GPUs

It would also make the repository suitable for classroom assignments and workshops.

Suggested Acceptance Criteria

• CPU-only training works

• No dedicated GPU required

• Small sample dataset included

• Tiny model configuration included

• Student configuration included

• Training can run with 4–8 GB RAM

• Training can resume from checkpoints

• Text generation works from the resulting checkpoint

• CPU training instructions added to README

• Approximate hardware requirements documented

• GPU configuration remains available

• Educational explanation of each component included

Additional Idea

A particularly useful addition would be a "Build Your Own Tiny LLM" notebook.

The notebook could progressively implement:

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Embeddings
 ↓
Positional Information
 ↓
Self Attention
 ↓
Multi-Head Attention
 ↓
Feed Forward Network
 ↓
Transformer Block
 ↓
Language Model Head
 ↓
Loss
 ↓
Backpropagation
 ↓
Training
 ↓
Text Generation

This would turn the repository into a complete hands-on course where a student can understand every major component instead of treating the model as a black box.

Summary

The current repository already provides a strong foundation for learning how to train an LLM from scratch.

I propose adding a CPU-first student track with tiny models, small datasets, low RAM requirements, beginner-friendly commands, and progressive scaling.

The objective is:

"Anyone with a normal laptop should be able to train a tiny language model from scratch and understand exactly how it works."