A number like 100,000 requests per second is a requirement, and the moment you write it down it turns into a set of constraints you can reason about. This is how I would work through it: the arithmetic first, then the decisions, then the parts that fail long before the machine runs out of CPU.
Do the arithmetic before drawing anything
The first useful thing about a throughput target is that it converts into other numbers you can reason about.
Little's law says the number of requests in flight equals the arrival rate multiplied by the time each one spends in the system. If the average response takes 10 milliseconds, then 100,000 requests per second means roughly 1,000 requests are being handled at any instant.
{
"type": "doughnut",
"data": {
"labels": ["Edge and TLS", "Load balancer", "Application", "Serialization", "Cache lookup", "Database (on a miss)"],
"datasets": [{
"data": [5, 2, 12, 4, 3, 24],
"backgroundColor": ["#cba6f7", "#89b4fa", "#a6e3a1", "#f9e2af", "#94e2d5", "#fab387"],
"borderWidth": 0
}]
},
"options": {
"responsive": true,
"plugins": { "legend": { "position": "bottom" } }
}
}
That chart is the budget I allocate, not a measurement. Every stage gets a slice and the slices have to add up to the p99 target of 50 milliseconds, with the database getting the largest slice because it is the only stage I do not fully control. Deciding the budget first is what turns "make it fast" into "make the application stage finish in 12 milliseconds".
The rest of the arithmetic follows the same pattern.
| Question | Working | Answer |
|---|---|---|
| Requests in flight | 100,000/s × 10 ms | about 1,000 concurrent |
| Egress bandwidth | 100,000/s × 2 KB | about 200 MB/s, or 1.6 Gbps |
| Origin load at a 95% cache hit rate | 100,000 × 0.05 | 5,000 database reads/s |
| Instances at 3,000 rps each | 100,000 ÷ 3,000 | 34, so about 70 for real headroom |
Two things stand out. The bandwidth is large but not exotic, and the cache hit rate is doing more work than any amount of clever code. A 95% hit rate turns a database problem into a rounding error, and a 90% hit rate doubles the load the database has to survive. Hit ratio is an architectural decision you make up front, long before there is anything to watch.
The shape of the system
With the budget fixed, the topology mostly writes itself.
<svg viewBox="0 0 960 470" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="API architecture: clients reach an edge and load balancer, the stateless API tier reads from cache and replicas, and writes flow through a queue to workers and the primary database">
<defs>
<marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="currentColor" opacity="0.7"/>
</marker>
</defs>
<g font-family="ui-monospace, monospace" fill="currentColor">
<text x="10" y="24" font-size="13" font-weight="700" opacity="0.6">REQUEST PATH</text>
<rect x="20" y="46" width="150" height="66" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.45"/>
<text x="95" y="74" font-size="13" text-anchor="middle" font-weight="700">Clients</text>
<text x="95" y="94" font-size="11" text-anchor="middle" opacity="0.6">browser / mobile / svc</text>
<rect x="210" y="46" width="180" height="66" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.7" stroke-width="1.5"/>
<text x="300" y="74" font-size="13" text-anchor="middle" font-weight="700">Edge</text>
<text x="300" y="94" font-size="11" text-anchor="middle" opacity="0.6">TLS / HTTP3 / WAF</text>
<rect x="430" y="46" width="150" height="66" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.45"/>
<text x="505" y="74" font-size="13" text-anchor="middle" font-weight="700">Load balancer</text>
<text x="505" y="94" font-size="11" text-anchor="middle" opacity="0.6">L4 anycast</text>
<rect x="620" y="46" width="300" height="66" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.7" stroke-width="1.5"/>
<text x="770" y="74" font-size="13" text-anchor="middle" font-weight="700">API tier (stateless, N instances)</text>
<text x="770" y="94" font-size="11" text-anchor="middle" opacity="0.6">thread-per-core, in-process cache</text>
<line x1="170" y1="79" x2="208" y2="79" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<line x1="390" y1="79" x2="428" y2="79" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<line x1="580" y1="79" x2="618" y2="79" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<rect x="620" y="176" width="140" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="690" y="200" font-size="12" text-anchor="middle">Shared cache</text>
<text x="690" y="218" font-size="10" text-anchor="middle" opacity="0.6">redis / memcached</text>
<rect x="780" y="176" width="140" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="850" y="200" font-size="12" text-anchor="middle">Read replicas</text>
<text x="850" y="218" font-size="10" text-anchor="middle" opacity="0.6">pooled connections</text>
<line x1="700" y1="112" x2="690" y2="174" stroke="currentColor" stroke-opacity="0.55" marker-end="url(#a)"/>
<line x1="820" y1="112" x2="850" y2="174" stroke="currentColor" stroke-opacity="0.55" marker-end="url(#a)"/>
<text x="600" y="150" font-size="10" text-anchor="middle" opacity="0.55">read</text>
<text x="10" y="286" font-size="13" font-weight="700" opacity="0.6">WRITE PATH</text>
<rect x="20" y="308" width="150" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="95" y="332" font-size="12" text-anchor="middle">Queue</text>
<text x="95" y="350" font-size="10" text-anchor="middle" opacity="0.6">durable log</text>
<rect x="210" y="308" width="160" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="290" y="332" font-size="12" text-anchor="middle">Workers</text>
<text x="290" y="350" font-size="10" text-anchor="middle" opacity="0.6">batched writes</text>
<rect x="410" y="308" width="150" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="485" y="332" font-size="12" text-anchor="middle">Primary</text>
<text x="485" y="350" font-size="10" text-anchor="middle" opacity="0.6">single writer</text>
<rect x="600" y="308" width="150" height="58" rx="8" fill="none" stroke="currentColor" stroke-opacity="0.5"/>
<text x="675" y="332" font-size="12" text-anchor="middle">Replication</text>
<text x="675" y="350" font-size="10" text-anchor="middle" opacity="0.6">to read pool</text>
<line x1="170" y1="337" x2="208" y2="337" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<line x1="370" y1="337" x2="408" y2="337" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<line x1="560" y1="337" x2="598" y2="337" stroke="currentColor" stroke-opacity="0.7" marker-end="url(#a)"/>
<line x1="660" y1="112" x2="95" y2="206" stroke="currentColor" stroke-opacity="0.35" stroke-dasharray="5 5" marker-end="url(#a)"/>
<text x="300" y="252" font-size="10" text-anchor="middle" opacity="0.55">writes enqueue and return early</text>
</g>
</svg>
Every arrow in that diagram is a decision about what has to be synchronous. Reads can be served by a cache or a replica. Writes go to a durable queue and return as soon as the queue has them, which keeps the write path short and lets the workers batch. The moment a write has to touch the primary inside the request, the primary becomes the ceiling for the whole system, and no amount of horizontal scaling in the API tier fixes that.
Decision: what actually runs a request
Runtime choice matters less than people expect for the shape of the system, and more than they expect for tail latency.
An event-loop runtime handles a thousand concurrent requests on one thread, which is why the c10k problem stopped being a problem. The catch is that anything blocking on that loop stops every other request, so a synchronous database driver turns a fast runtime into a slow one. A thread-per-core runtime runs several event loops, one per CPU, and uses SO_REUSEPORT so the kernel spreads connections across them. That gives you parallelism without a shared lock on the hot path.
Garbage collection is the part I would think hardest about. A stop-the-world pause is a tail-latency event, and at 100,000 requests per second a 20 millisecond pause stacks 2,000 requests behind it. A runtime with short, concurrent collection or no collection at all is worth real money here. If the language does pause, allocate carefully: reuse buffers, avoid per-request object churn, and watch the allocation rate rather than the heap size.
Decision: the wire between client and server
HTTP/1.1 with keep-alive is fine and simple, but it gives you one in-flight request per connection unless the client opens several. To carry 1,000 concurrent requests over HTTP/1.1 you may need thousands of sockets, and every socket is kernel memory and a file descriptor. HTTP/2 multiplexes many streams on one connection, which cuts the socket count, at the cost of head-of-line blocking when a packet is lost. HTTP/3 moves that problem onto QUIC, where streams are independent, and it is the reason a public API in front of mobile networks should terminate HTTP/3 at the edge even if everything behind it is HTTP/2 or gRPC.
TLS is real CPU, and it is the first place people get surprised. A full handshake is expensive relative to the request itself, so session resumption has to be the default path. Terminate TLS at the edge, keep the connection alive, and let the clients reuse the session. The internal hop between the load balancer and the API tier can drop TLS entirely if the network is trusted, which removes a whole class of CPU cost from the tier that has to scale.
Decision: how to parse and serialize
Parsing is the silent bottleneck. A JSON body costs allocation and CPU to turn into memory, then the response costs the same in reverse, and at 100,000 requests per second that work is a large share of the 12 millisecond application budget. Two ways to reduce it: serialise less, and serialise faster.
Serialising less means returning only the fields the caller asked for, and pushing pagination and filtering into the data layer so the API tier never materialises rows it will throw away. Serialising faster means choosing a compact format for internal hops, where nobody is debugging it by eye. Protobuf or flatbuffers cut both the CPU and the bytes. Keep JSON at the public edge if your clients expect it, and stop using it between services that you also control.
Decision: make the cache the first line of defence
A cache in front of the database is the highest-value component in the whole design. At 95% hit rate the database sees 5,000 reads per second instead of 100,000, and the difference between those two numbers is the difference between a design that works and one that does not.
The details matter more than the choice of cache. Add a small in-process cache for the hottest keys so the shared cache is not a network hop on every request. Guard the misses with request coalescing, so a thousand concurrent requests for the same expired key turn into one database read and a thousand waiters. Add jitter to expiry times, because a thousand keys that expire on the same second will send a wave to the database that looks like a denial of service. Cache negative results too, so a request for something that does not exist does not become a database query every time.
sequenceDiagram
participant C as Client
participant LB as Load balancer
participant API as API instance
participant L1 as In-process cache
participant L2 as Shared cache
participant DB as Replica
C->>LB: GET /items/42
LB->>API: route to an instance
API->>L1: lookup
alt in-process hit
L1-->>API: value
else miss
API->>L2: lookup
alt shared hit
L2-->>API: value
API->>L1: backfill
else miss
API->>API: coalesce waiters
API->>DB: one query
DB-->>API: row
API->>L2: store with jittered TTL
API->>L1: store
end
end
API-->>C: 200, cache hit
Decision: the data layer has fewer knobs than you think
The API tier scales by adding instances. The database does not, so it gets the careful treatment.
Connections are the first constraint. Each connection costs memory on both sides, and a pool that is too large makes the database slower, not faster, because the database spends its time context switching. Pool at the database side, not per instance, and size the pool to the CPU cores the database actually has.
Reads go to replicas, and replicas introduce a consistency decision. A client that writes and immediately reads may not see its own write, so either route that specific read to the primary or accept the staleness explicitly and document it. Sharding comes later, and only when the primary is genuinely out of vertical room, because a shard key is very hard to change once data is on it. Choose a key with high cardinality and even access, or you will discover a hot shard on your busiest day.
Decision: surviving the load you did not plan for
Everything above is about handling the traffic you expect. Most incidents are about the traffic you did not.
The failure mode of a system near saturation is a steep curve rather than a flat one: latency climbs slowly and then goes vertical, because a queue that is filling serves fewer requests, which adds more arrivals to the queue. The way out is to refuse work on purpose before that point. Admission control that sheds a small percentage of traffic keeps the requests you do accept fast, and a fast 429 is worth more to a caller than a slow timeout.
Timeouts are a correctness requirement. Every outbound call needs a deadline shorter than the caller's, retries need exponential backoff with jitter, and a circuit breaker needs to open when a dependency is failing so the failing dependency is not finished off by your retries. Retry storms are a self-inflicted outage, and the only defence is that jitter plus a budget on how much of your traffic can be retries at all.
What breaks first
If I had to bet on the order of failures, it would look like this.
Tail latency amplification comes first. A request that fans out to ten backends is as slow as the slowest of the ten, so a 1% slow backend becomes a 10% slow endpoint. Reduce fan-out and set per-call deadlines.
Then garbage collection and allocation. Trimming the allocation rate on the hot path buys more headroom than a faster CPU.
Then connection churn. A client that reconnects on every request turns a request problem into a TLS problem, and the edge spends its CPU on handshakes instead of requests.
Then hot shards and hot keys. Uniform keys look fine in staging and concentrate under real traffic.
Then logging and I/O. Writing a log line per request at 100,000 requests per second is a disk problem waiting to happen. Sample the logs, and keep the counter metrics cheap.
How I would prove it
Load tests that never lose a packet, from a single machine on the same network, tell you very little. The useful tests are the ugly ones.
Ramp the load slowly to find the knee of the latency curve, where p99 starts to climb, and treat that point as your real capacity. Run a soak at 70% of that for hours, because memory leaks and connection leaks do not show up in a five-minute test. Then spike, well past the knee, and check that the system sheds load and recovers rather than collapsing. Run the generators from several machines, or the generator becomes the bottleneck and you end up measuring your load tool. And always report percentiles, never the average, because the average is where the incidents hide.
What I would do differently
The honest trade-off is that a 100K rps design is expensive and complicated, and most systems do not need it. If your real requirement is 5,000 requests per second, the stateless tier, one cache, and one database with a pool will get you there with a fraction of the operational surface. Reach for sharding, queues, and multi-region only when a number forces you to.
The other thing I would keep from this exercise is the arithmetic. Writing down concurrency, bandwidth, and instance count before choosing any technology makes the conversation concrete, and it usually reveals that the cache or the serialisation is the real lever. Writing that arithmetic down first is the part that transfers to the next system, whatever the number turns out to be.
Reach me at swadhinbiswas.cse@gmail.com or on GitHub and LinkedIn.
No comments yet.