Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems.
Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
A modern LLM can contain hundreds of billions of parameters, run on extremely expensive accelerators, and still spend most of its inference time doing something that looks embarrassingly sequential: That is the awkward part of autoregressive generation.
The model may process a whole prompt in parallel during the initial prefill, but once generation starts, the next token depends on the previous token.
So generating 100 tokens looks conceptually like running the model 100 times.
And for many serving workloads, that is exactly where the money goes.
A family of techniques tries to break this bottleneck by asking a deceptively simple question: What if the model could predict several future tokens at once, then verify them in parallel?
That idea leads to speculative decoding, Medusa-style multiple decoding heads, and the broader multi-token prediction approach used during training.
The interesting part is that these are not merely "optimization tricks." They change the computational structure of decoding.
This article develops that idea from first principles and then gets into the engineering details.
1.
The problem: your GPU is doing an expensive sequential loop Consider ordinary autoregressive decoding.
Given a prompt: the model predicts: Then it feeds the new sequence back through the model: and predicts the next token.
Then again: and so on.
Formally, the model factorizes the probability of a sequence as: That conditional dependence is what makes language modeling so useful.
It is also what makes decoding annoying.
Prefill is parallel; decode is sequential This distinction matters enormously in production.
Suppose a prompt has 2,000 tokens and we want 200 generated tokens.
During prefill, the transformer can process many positions concurrently: During decoding: The transformer itself is highly parallelizable.
The dependency graph of generation is not.
This creates an unusual hardware situation.
A giant model can spend much of its decoding time limited not by arithmetic throughput but by repeatedly moving a large set of model weights through memory.
This was one of the motivations behind the speculative decoding work by Yaniv Leviathan, Matan Kalman, and Yossi Matias, and later became a central observation in Medusa.
The core question is therefore not: How do we make one forward pass cheaper?
It is: How do we get more than one accepted token out of each expensive forward pass?
That is a much more interesting question.
2.
The first big idea: speculate, then verify Imagine that instead of asking the large model for one token, we had a tiny model that could cheaply guess several: The big model can then evaluate those candidate tokens in parallel.
Suppose the small model proposes: The large model might agree with all four: Great.
One expensive evaluation effectively produced four tokens.
But perhaps the draft is: and the large model says: Then we keep the accepted prefix and let the large model continue from the rejected position.
This is the basic idea of speculative decoding.
It is beautifully simple: Leviathan et al. showed that this could accelerate generation while preserving the output distribution exactly, rather than merely producing "approximately similar" text.
Their 2023 paper reported roughly 2x-3x acceleration on the models they evaluated.
The crucial insight was that autoregressive generation does not mean the expensive model must discover every token sequentially.
It only means that the final accepted sequence has to respect the autoregressive distribution.
Speculation lets us take advantage of the fact that many consecutive tokens are easy to predict.
For example: contains many stretches where the answer is nearly obvious: or or The large model is still being asked to