no-magic

--- --- # no-magic **Because `model.fit()` isn't an explanation.** --- ## What This Is `no-magic` is a curated collection of single-file, dependency-free Python implementations of the algorithms that power modern AI. Each script is a complete, runnable program that trains a model from scratch and performs inference — no frameworks, no abstractions, no hidden complexity. Every script in this repository is an **executable proof** that these algorithms are simpler than the industry makes them seem. The goal is not to replace PyTorch or TensorFlow — it's to make you dangerous enough to understand what they're doing underneath. ## See It In Action

01 — Foundations (14 scripts)

Autoregressive GPT Token-by-token generation	RNN vs GRU Vanishing gradients and gating	LSTM 4-gate memory highway
BPE Tokenizer Iterative pair merging → vocabulary	Word Embeddings Contrastive learning → semantic clusters	RAG Pipeline Retrieve → augment → generate
BERT Bidirectional attention + [MASK] prediction	Convolutional Net Sliding kernels → feature maps	ResNet F(x) + x = gradient highway
Vision Transformer Image patches as tokens	Diffusion Noise → data via iterative denoising	VAE Encode → sample z → decode
GAN Generator vs discriminator minimax	Optimizers SGD vs Momentum vs Adam convergence

**Comparison scripts:** [attention_vs_none.py](01-foundations/attention_vs_none.py) · [rnn_vs_gru_vs_lstm.py](01-foundations/rnn_vs_gru_vs_lstm.py)

02 — Alignment & Training (10 scripts)

LoRA Fine-tuning Low-rank weight injection	QLoRA 4-bit base + full-precision adapters	DPO Alignment Preferred vs. rejected → policy update
PPO (RLHF) Clipped policy gradient for alignment	GRPO Group-relative rewards, no critic	REINFORCE Log P(a) × reward = gradient
Mixture of Experts Sparse routing to specialist MLPs	Batch Normalization Normalize activations → stable training	Dropout Kill neurons → prevent overfitting

**Comparison scripts:** [adam_vs_sgd.py](02-alignment/adam_vs_sgd.py)

03 — Systems & Inference (17 scripts)

Attention Mechanism Q·KT → softmax → weighted V	Flash Attention Tiled O(N) memory computation	RoPE Position via rotation matrices
KV-Cache Memoize keys/values — stop recomputing	PagedAttention OS-style paged KV-cache memory	Quantization Float32 → Int8 = 4x compression
Beam Search Tree search with top-k pruning	Checkpointing O(n) → O(√n) memory via recompute	Model Parallelism Tensor + pipeline across devices
State Space Models Linear-time selective state transitions	Vector Search Exact vs LSH approximate search	BM25 TF → TF-IDF → BM25 evolution
Speculative Decoding Draft fast, verify once	Complex SSM Complex eigenvalues = real + RoPE	Discretization Euler vs ZOH vs Trapezoidal
Roofline Model SISO → MIMO hardware utilization	TurboQuant Data-oblivious quantization via random rotation

04 — Agents & Planning (5 scripts)

Monte Carlo Tree Search UCB1 tree search + random rollouts	ReAct Agent Thought → Action → Observation	Multi-Armed Bandits ε-greedy vs UCB1 vs Thompson Sampling
Minimax + Alpha-Beta Adversarial search with pruning	Memory-Augmented Network Differentiable read/write heads

> All algorithms have animated visualizations. Full 1080p60 videos in [Releases](https://github.com/no-magic-ai/no-magic/releases). > Visualization source and rendering: [no-magic-viz](https://github.com/no-magic-ai/no-magic-viz) — built with [Manim](https://www.manim.community/). ## Philosophy Modern ML education has a gap. There are thousands of tutorials that teach you to call library functions, and there are academic papers full of notation. What's missing is the middle layer: **the algorithm itself, expressed as readable code**. This project follows a strict set of constraints: - **One file, one algorithm.** Every script is completely self-contained. No imports from local modules, no `utils.py`, no shared libraries. - **Zero external dependencies.** Only Python's standard library. If it needs `pip install`, it doesn't belong here. - **Train and infer.** Every script includes both the learning loop and generation/prediction. You see the full lifecycle. - **Runs in minutes on a CPU.** No GPU required. No cloud credits. Every script completes on a laptop in reasonable time. - **Comments are mandatory, not decorative.** Every script must be readable as a guided walkthrough of the algorithm. We are not optimizing for line count — we are optimizing for understanding. See `CONTRIBUTING.md` for the full commenting standard. ## Who This Is For - **ML engineers** who use frameworks daily but want to understand the internals they rely on. - **Students** transitioning from theory to practice who want to see algorithms as working code, not just equations. - **Career switchers** entering ML who need intuition for what's actually happening when they call high-level APIs. - **Researchers** who want minimal reference implementations to prototype ideas without framework overhead. - **Anyone** who has ever stared at a library call and thought: _"but what is it actually doing?"_ This is not a beginner's introduction to programming. You should be comfortable reading Python and have at least a surface-level familiarity with ML concepts. The scripts will give you the depth. ## What You'll Find Here The repository is organized into four tiers based on conceptual dependency: ### 01 — Foundations (14 scripts) Core algorithms that form the building blocks of modern AI systems. GPT, RNN, LSTM, BERT, CNN, ResNet, ViT, GAN, VAE, diffusion, embeddings, tokenization, RAG, and optimizer comparison. Includes comparison scripts for attention mechanisms and recurrent architectures. See [`01-foundations/README.md`](01-foundations/README.md) for the full algorithm list, timing data, and roadmap. ### 02 — Alignment & Training Techniques (10 scripts) Methods for steering, fine-tuning, and aligning models after pretraining. LoRA, QLoRA, DPO, PPO, GRPO, REINFORCE, MoE, batch normalization, dropout/regularization, and optimizer comparison. See [`02-alignment/README.md`](02-alignment/README.md) for the full algorithm list, timing data, and roadmap. ### 03 — Systems & Inference (17 scripts) The engineering that makes models fast, small, and deployable. Attention variants, Flash Attention, KV-cache, PagedAttention, RoPE, quantization, beam search, checkpointing, parallelism, SSMs, vector search, BM25, speculative decoding, complex SSM equivalence, discretization methods, and roofline analysis. See [`03-systems/README.md`](03-systems/README.md) for the full algorithm list, timing data, and roadmap. ### 04 — Agents & Planning (5 scripts) Autonomous reasoning and decision-making. Monte Carlo Tree Search for strategic planning, ReAct agents for tool-augmented reasoning loops, multi-armed bandits for exploration/exploitation, minimax with alpha-beta pruning for adversarial search, and memory-augmented networks for persistent agent memory. See [`04-agents/README.md`](04-agents/README.md) for the full algorithm list, timing data, and roadmap. ## How to Use This Repo ```bash # Clone the repository git clone https://github.com/no-magic-ai/no-magic.git