Imagine this: It's salary day.
It's 2 PM.
Millions of people across India suddenly open their UPI apps and start paying rent, sending money to family, paying credit-card bills, and shopping online.
Now here's the system-design interview question: If millions of people make payments at almost exactly the same time, is every request hitting one central server?
What prevents the entire payment system from freezing?
At first glance, it sounds like a scaling problem.
It isn't just a scaling problem.
It's a combination of: horizontal scaling concurrency distributed systems database consistency retries idempotency backpressure failure isolation downstream bottlenecks And that's what makes payment systems such an interesting system-design problem.
First: Don't Imagine One Giant UPI Server A common mental model looks like this: If that were literally true, we'd have a pretty serious problem.
One machine cannot safely process the country's entire payment traffic.
Instead, think about a distributed system: The exact implementation of a real payment network is much more complicated than this diagram, but this is the right system-design mental model.
The important idea is: The system is distributed across many machines and participating institutions.
Step 1: The First Problem — Traffic Spikes Let's take a concrete example.
You want to pay your landlord: At the same moment, millions of other people are doing something similar.
Suddenly: The first question is: How do we handle the additional traffic?
Naive Solution: One Powerful Server We could buy a massive machine.
This is called vertical scaling.
Make the machine bigger.
But there are limits.
Eventually: And there's an even bigger problem.
If the server dies: For a payment system, that's unacceptable.
Solution: Horizontal Scaling Instead of making one machine enormous, add more machines.
If one server handles 20,000 requests/sec, and we need roughly 1 million: We can scale the application tier horizontally.
Now if one server fails: Traffic can be routed to healthy servers.
This gives us our first reusable system-design pattern: Pattern #1: Horizontal Scaling When request volume exceeds the capacity of one machine, distribute requests across many machines.
But We've Created a New Problem Suppose two of your payments arrive simultaneously.
They might land on different servers: Suppose Harsh has: Both servers read the balance at approximately the same time.
Server 1 sees: Server 2 sees: Then: The system has effectively allowed: to be spent from an account containing: That's a race condition.
And this is where payment-system design gets interesting.
Step 2: We Need Atomic State Changes A payment isn't just: Conceptually, we need something closer to: The critical state transition must happen safely under concurrency.
We need a guarantee that two competing operations cannot both incorrectly modify the same financial state.
Depending on the architecture, this can involve: database transactions locking optimistic concurrency control serialization partition ownership carefully designed state machines The important interview lesson isn't: "Use database locks." It's: Identify the shared mutable state and protect the critical transition.
Step 3: Now the Database Becomes the Bottleneck Let's say we've successfully scaled our application servers.
We now have: We have 500 application servers.
But one database.
Now all those servers are fighting for the same resource.
The application tier scales.
The database doesn't.
This is a classic distributed-systems bottleneck: The fastest part of your system doesn't matter if a slower shared dependency limits the entire system.
Step 4: Partition the Work Instead of forcing everything through one database/resource, we can partition data and workload.
Conceptually: The partitioning strategy could be based on something like: The exact choice depends on the system.
This is generally called: Sharding / Partitioning The reusable pattern is: Pattern #2: Partition the bottleneck When one resource can't handle the workload, divide the workload into independent partitions.
But There's Another Problem Imagine our application servers are perfectly healthy.
Our databases are perfectly healthy.
But a downstream bank suddenly becomes overloaded.
Our system can process millions of incoming requests.
But the downstream dependency might only safely process a smaller amount.
This gives us another fundamental principle: The capacity of a distributed system is constrained by its critical bottlenecks and dependencies.
You can't solve a downstream bottleneck by simply adding more application servers.
Step 5: Backpressure Suppose a component can safely process: but we're receiving: If we blindly forward everything: Instead, for work that is safe to process asynchronously, we can introduce a buffer: The queue absorbs temporary bursts.
This gives us: Pattern #3: Backpressure When producers can generate work faster than consumers can process it, slow down producers or buffer the work.
This is where technologies such as Kafka or other messaging systems become useful.
But notice the reasoning.
We didn't start with: "Let's use Kafka." We started with: "Our producer is faster than our consumer.
We need buffering/backpressure." Then we choose a technology.
Can We Queue the Entire Payment?
Not necessarily.
This is an important interview trap.
You shouldn't say: "We'll just put every payment into Kafka." Financial transactions have correctness and latency requirements.
There is a difference between: and: For example: The financial state transition may require strict correctness.
But sending: to a notification service doesn't necessarily need to block the core transaction.
Those side effects can often be asynchronous.
Step 6: Now Imagine the Network Fails Here's where payment systems become really interesting.
You press: The request reaches the payment system.
The money gets debited.
But before the response reaches your phone: Your app says: You think: "Okay, I'll try again." So you press Pay again.
Now the system receives: If the system treats both as new payments: That's obviously unacceptable.
Step 7: Idempotency We need the system to recognize: "This retry is actually the same payment." So the client/request can carry a unique identifier: First request: Retry: The system returns the previous result rather than charging the user again.
Conceptually: This is idempotency.
And it is one of the most important concepts in distributed systems.
Pattern #4: Idempotency A retry of the same logical operation should not produce an additional side effect.
You will see this pattern everywhere: payments order creation payment webhooks message processing API retries distributed jobs Step 8: Why Retries Are Dangerous Retries sound harmless: But imagine 1 million clients doing this.
A failing system can actually make itself more overloaded through retries.
This is called a retry storm.
So production systems need things like: exponential backoff jitter retry limits timeouts circuit breakers idempotency Again, each mechanism exists because a specific failure mode exists.
The Bigger Picture At this point our system is evolving.
We started with: And progressively discovered problems.
Traffic spike Shared state Database bottleneck Downstream overload Network failure Retry duplication Our architecture is now becoming: And around the entire system: The Most Important Lesson When you're asked: "Design a UPI-scale payment system." Don't start drawing 20 boxes.
Start asking:
1.
What's the workload?
2.
What's the critical state?
3.
Where is concurrency dangerous?
4.
What's the bottleneck?
5.
What happens during failure?
6.
What happens when traffic suddenly increases?
7.
What happens when the client retries?
This is the real system-design thought process.
A Reusable Pattern Library By solving this one problem, we've already learned several patterns.
Problem Pattern One server can't handle traffic Horizontal scaling Shared state modified concurr