This is Part 10 of my "From One User to One Million" series, where we'll build an understanding of System Design by following a simple application as it grows from a single user to millions.
Instead of memorising technologies, we'll learn why they exist by solving real problems as they appear.
In Part 9, we solved the problem of data that had grown too large for a single database.
We split it across multiple shards, each holding its piece of the whole, so that no single machine ever had to carry everything.
At that point, the architecture could scale in almost every direction we'd tried to push it.
Traffic was distributed across application servers.
Repeated database work was absorbed by the cache.
Read traffic was spread across replicas.
Data itself was partitioned across shards.
And yet.
We ended Part 9 by noticing something that none of those solutions addressed.
Some user requests trigger a lot of downstream work.
Saving an order is one thing.
But saving the order, sending a confirmation email, generating an invoice, updating inventory, firing off a notification, recording an analytics event, triggering the recommendation engine: that's an entirely different conversation.
Right now, all of that happens before the user gets a response.
The question we left with was this: what if they didn't have to wait for all of it? -- Section 1: The User Doesn't Need Everything Right Now Before we look at any solution, it's worth asking a simpler question.
When a user places an order, what do they actually need to know before they can move on?
They need to know the order was received.
They need confirmation that the important thing happened: their money was accepted, their items are reserved, the transaction is real.
That's it.
That's what they're waiting for.
They do not need to wait for the confirmation email to land in their inbox.
They do not need to wait for the invoice to be generated and stored somewhere.
They do not need to wait for the analytics system to record that this purchase happened.
They certainly don't need to wait for the recommendation engine to update its model based on what they just bought.
All of that will happen.
It should happen.
But none of it needs to happen before the user gets their confirmation screen.
This seems obvious when you say it directly.
Of course the user doesn't need to wait for the analytics event.
Of course they don't need to hold their breath while the recommendation system recalculates.
But the way most applications are initially built, that's exactly what happens. -- Section 2: Doing Everything Synchronously Here's what a typical order flow looks like when everything is wired together the simple, obvious way.
The user clicked a button and waited almost 700 milliseconds for a confirmation screen.
More than half a second, for a response to something they did in an instant.
And that's assuming nothing goes wrong.
What if the email service is having a slow moment and takes two seconds instead of 200 milliseconds?
The user waits two seconds.
What if the analytics system is down entirely?
The request fails, and the user gets an error, even though the order itself was saved perfectly.
The application has tied its response time and its reliability to every piece of downstream work it performs.
Every slow step makes the user wait longer.
Every failing step makes the whole request fail.
That's not a hardware problem.
It's not a database problem.
It's an architectural problem.
The application is doing work in a sequence when most of that work doesn't actually depend on the steps before it.
The confirmation email doesn't need the invoice to be generated before it can be sent.
The analytics event doesn't need the email to succeed before it can be recorded.
These tasks are all independent of each other.
They're only sequential because that's how they got wired together.
So the question becomes: what would it look like to stop treating them as sequential? -- Section 3: Put the Work in a Queue The insight i