Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems.
Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
A language model can write Python, explain quantum mechanics, and imitate Shakespeare.
Show it a screenshot of a production dashboard, however, and suddenly the central question becomes: How does a transformer that was trained on text learn what a pixel means?
The naïve answer is: “Give the image to the LLM.” That description hides almost all of the interesting engineering.
Modern multimodal systems are usually compositions of several models: a vision encoder turns pixels into vectors, a connector translates those vectors into something the language model understands, and the LLM then reasons over the resulting representation alongside ordinary text tokens.
That architectural trick has turned the transformer from a language architecture into something much closer to a general-purpose interface for heterogeneous data.
The evolution is worth understanding because it reveals a useful engineering pattern: you often do not need to retrain a giant model to give it a new sensory modality.
You need a good representation and a sufficiently expressive interface between representations.
1.
The basic mental model: pixels become tokens Start with an ordinary LLM.
Its input looks conceptually like: Everything is eventually represented as vectors.
Multimodal transformers exploit this fact.
An image is first converted into a sequence of vectors: The important conceptual shift is this: The LLM does not have to understand pixels directly.
It only has to understand a representation produced by another model.
This is remarkably similar to a software interface.
The vision encoder is one service.
The LLM is another service.
The connector is the API contract.
That analogy becomes surprisingly literal once you look at actual architectures.
Why not just tokenize pixels?
You could imagine treating every RGB pixel as a token.
A 1024 x 1024 image has: Even before considering RGB channels, that's about one million spatial locations.
Self-attention over one million tokens is absurdly expensive.
The computational cost of full self-attention is approximately: where: If you increase by 100x, attention work increases by roughly: So multimodal systems need compression.
The first major engineering problem is therefore: How do we turn a huge image into a manageable sequence of informative vectors?
2.
The first breakthrough: vision itself becomes transformer-friendly The most important precursor was the realization that transformers did not fundamentally require language.
In 2020, Alexey Dosovitskiy and colleagues at Google Research published the Vision Transformer, or ViT.
The idea was beautifully simple.
Take an image: Divide it into patches, say: You get: or: Each patch becomes a vector.
Now the image has become a sequence: which looks suspiciously like a sequence of words.
The transformer does not care that these vectors originated from little squares of an image rather than pieces of text.
This was a profound simplification.
Instead of inventing a completely different architecture for vision, researchers could increasingly reuse the machinery that had made transformers dominant in NLP.
A useful way to think about ViT Suppose each patch is represented by a vector: A linear projection maps the flattened patch into the model dimension: The transformer then processes: with positional information added so that it knows that one patch came from the top-left and another from the bottom-right.
The architecture is now almost indistinguishable from a language transformer at the sequence-processing level.
That opened the door to something much bigger.
What if we could align these visual vectors with language?
That was the next step.
3.
CLIP: teach images and language to share a coordinate system In 2021, OpenAI researcher