Large Language Models have quickly become part of everyday software development.
We ask them to explain code, debug errors, generate tests, write Python scripts, summarize documentation, or help us understand an unfamiliar codebase.
Within seconds, we get a response that can feel surprisingly natural.
But what actually happens during those few seconds?
Suppose you type: What is a build system?
The model doesn't simply search through a database for a stored answer, and it doesn't generate the entire response in one shot.
At the heart of an autoregressive LLM is a deceptively simple task: Given the tokens I've seen so far, what token should come next?
Getting to that prediction, however, involves several layers of computation.
At a high level: Let's follow that journey.
1.
Everything Starts With the Prompt Consider: What is a build system?
Humans immediately recognize the words and their meaning.
A neural network needs numbers.
Before the model can process the question, the text passes through a tokenizer.
2.
Tokenization: Breaking Text Into Pieces A tokenizer divides text into smaller units called tokens.
Conceptually, our prompt might become: This is only an illustration.
Actual tokenization depends on the tokenizer used by the model.
A token isn't necessarily a complete word.
It might represent: a complete word part of a word punctuation whitespace combined with text a number part of an identifier a programming-language symbol Each token is mapped to an integer called a token ID.
Conceptually: The IDs above are illustrative.
The important part is the transformation: Human-readable text has become a sequence of numbers the model can process.
But token IDs themselves don't capture useful semantic relationships.
The number , for example, doesn't inherently explain what build means.
That's where embeddings enter the picture.
3.
Embeddings: Turning Tokens Into Vectors Each token is mapped to an embedding—a high-dimensional vector of numbers.
Conceptually: These useful representations are learned during model training rather than manually assigned by engineers.
Embeddings give the neural network much richer representations to work with.
But language depends heavily on context.
Consider: The developer restarted the build because it failed.
What does it refer to?
A human reader can connect it with the build.
The model needs a mechanism for modeling relationships between tokens as well.
That's where self-attention becomes important.
4.
Self-Attention: Which Other Tokens Matter?
Self-attention is one of the core ideas behind the Transformer architecture.
Instead of treating every token independently, attention allows each token's representation to incorporate information from other relevant tokens in the context.
A useful mental model is that each token is asking: Which other tokens should influence my representation right now?
Under the hood, tokens are projected into three important vectors: Query (Q) Key (K) Value (V) An intuitive way to think about them is: Query: What information am I looking for?
Key: What information might I match with?
Value: What information should I contribute if I'm relevant?
The standard scaled dot-product attention operation is: We'll explore the mathematics more deeply in a later article.
For now, the key idea is: Attention allows the representation of a token to depend on its surrounding context.
Transformers also use multiple attention heads, allowing different kinds of relationships to be modeled in parallel.
5.
Through the Transformer Self-attention is part of the larger Transformer architecture.
Token representations pass through a stack of Transformer layers.
A simplified view: Real Transformer blocks also contain mechanisms such as residual connections and normalization.
As the representations move through these layers, they become increasingly contextualized.
The representation associated with build, for example, is no longer simply a static representation of that token.
It now incorporates information about how build is being used in this particular context.
Eventually, the model reaches the central question: What token should come next?
6.
From Transformer Output to Logits After processing the context, the model produces scores for possible next tokens.
These scores are called logits.
Imagine a tiny example: Possible Token Logit A 7.2 The 5.8 Build 3.4 Software 2.7 Banana -4.1 A real model may have a vocabulary containing tens or hundreds of thousands of tokens.
That means the model produces a large vector of scores at every generation step.
Higher logits generally indicate more plausible next tokens.
These scores can then be converted into a probability distribution using softmax.
Conceptually: Token Probability A 55% The 25% Build 10% Software 7% Everything else 3% Again, these numbers are illustrative.
Now the model has possible continuations and their relative likelihoods.
7.
Choosing the Next Token How the next token is chosen depends on the decoding strategy.
Generation can be influenced by techniques such as: temperature top-k sampling top-p sampling greedy or other decoding strategies Suppose the model selects: A The sequence now becomes: What is a build system?
A The model isn't finished.
It predicts again.
8.
The Answer Grows One Token at a Time The model now has: What is a build system?
A It predicts another token.
Perhaps: build Now: What is a build system?
A build Then: system Then: is Then: a The process continues until a stopping condition is reached.
Eventually, we might get: A build system is software that automates the process of compiling, linking, testing, and packaging software.
This is autoregressive generation.
At each step: This is one of the most important ideas for understanding LLMs: The answer isn't normally generated all at once.
It grows token by token.
9.
What Happens When an LLM Generates Code?
The same basic process applies when an LLM generates software.
Suppose we ask: Write a C++ function that returns the larger of two integers.
The model might produce: From our perspective, it looks as though the model understood the requirement and produced a complete C++ function.
But the fundamental generation mechanism hasn't changed.
The output is still produced as a sequence of tokens.
Conceptually: The actual token boundaries depend on the tokenizer.
What's interesting is how the existing context changes what becomes likely next.
After: tokens associated with a C++ function signature become plausible.
Later, after: the prompt, previously generated code, and patterns learned during training make certain continuations more plausible than unrelated ones.
The model doesn't suddenly switch to a dedicated C++ code-generation engine.
The same fundamental loop continues: The same principle applies to Python, Rust, Java, SQL, shell scripts, and other programming languages.
Of course, generating a small function is much easier than generating or modifying a large software component.
For larger tasks, the model may need to maintain relationships involving: function signatures types variable names APIs control flow previously generated code programming-language syntax user requirements across a much larger context.
That raises an interesting question: If an LLM generates code token by token, how can it produce syntactically and sometimes logically consistent programs?
That's a topic worth exploring separately.
For now, the important takeaway is: Natural language and source code look very different to us, but to an LLM they ultimately become sequences of tokens to be modeled and predicted.
10.
Doesn't Repeating the Transformer Get Expensive?
Yes.
Suppose the model has already processed a long prompt and generated hundreds of tokens.
Recalculating all of the attention information for every previous token from scratch for each new token would involve substantial redundant computation.
One important optimization is the KV cache.
During self-attention, the model computes Key and Value representations for to