Scaling directions — deciding how to grow
When the estimated load doesn't fit on one machine, there are two directions. Grow one machine, or add machines.
Adding machines has a condition. If the server holds state, you can't add. Meeting that condition takes up most of the scaling discussion.
What you'll learn in this chapter
- The difference between vertical and horizontal, and each one's limits
- Steps to externalize state so horizontal is possible
- What the load balancer distributes by, and health checks
- Delays when auto-scaling machine count
- Offloading reads to caches, writes to queues
- The order in which bottlenecks move
Builds on load numbers from Capacity estimation. Also uses "who holds the cache" from HTTP caching.
What this chapter does not cover
What happens once you insert a queue is outside this chapter's scope. Message resends and duplicates, order guarantees, dead-letter handling — those are inter-service integration topics. Here queues are treated as "mechanisms for smoothing load".
Client-side retry and timeout design is Retries and idempotency; cache headers are HTTP caching.
Two directions
| Direction | What to do | Fits | Limit |
|---|---|---|---|
| Vertical (scale up) | Grow one machine's CPU, memory, disk | Non-splittable processing; write-concentrated databases | Capped at one machine's limit. Often requires a restart |
| Horizontal (scale out) | Add machines with the same role | Parts where requests can be processed independently | Can't add when state is held |
Vertical's benefit is you don't change the application. Change the setting and restart. So it's a reasonable first move.
Two limits. There's a cap, and however big, it's still one machine. One machine going down stops the service. That's Availability and redundancy.
To go horizontal, externalize state
The condition for adding machines to work correctly is one. Any server should produce the same result for a given request.
What breaks this is state placed inside servers.
Three typical states. Move all three to a shared place outside the server.
| State | If held on server | Where to move |
|---|---|---|
| Session | Login breaks the moment routed to another server | Shared datastore. Or hold signed on the cookie |
| Uploaded files | Readable only on the server that saved | Object storage or shared filesystem |
| In-memory cache | Contents diverge per server | Shared cache server. Or limit to use cases where divergence is OK |
Row 3 has an exception. Data that may diverge can be held in each server's memory. For master data updated rarely, that's faster. The dividing line is "what happens when they diverge".
Not externalizing sessions is possible: route the same user always to the same server. But if that server dies, that user's session dies. It also skews load distribution. If it's done by client-IP hash, the key is really the source IP, so everyone behind a shared connection such as an office LAN lands on one server together.
What the load balancer distributes by
The load balancer decides which server receives an incoming request. There are methods; the default is round-robin (in order).
| Method | Distribution | Fits |
|---|---|---|
| Round-robin | One at a time in order | When per-request times are similar (default) |
| Least connections | Server with fewest in-flight connections | Times vary |
| Client IP hash | Same source IP goes to the same server | Want to keep session on the server |
| Least response time | By average response time + connections | Servers have different performance |
nginx defaults to round-robin, with least_conn and ip_hash as options (nginx — Using nginx as HTTP load balancer). least_time (by response time) was only in the commercial version before 1.31.0 (ngx_http_upstream_module). Check the availability for your version when you see the method name.
Which layer to route at is another choice. Route by TCP port only, or by HTTP content (path, hostname)? What's possible differs. That's the L4 vs. L7 distinction from TCP/IP.
Don't route to a dead one
Adding machines is pointless if you keep routing to a dead server. Load balancers exclude non-responding servers from routing.
nginx's reverse proxy judges by actual request success. A server that fails max_fails times within fail_timeout is excluded for the same length of time (ngx_http_upstream_module). After that, it's checked with a real request, and if it succeeds, returned to rotation.
Besides judging from real request outcomes, as above, a dead server can also be found by polling a dedicated check endpoint.
Two design notes.
- Decide whether the check endpoint examines dependencies. Only checking process liveness returns "healthy" even if the database is down. Examining dependencies makes one bad database exclude every server
- A too-fast exclusion decision reduces machines on brief slowness. Fewer machines increase load on the rest, and more get excluded
The second is the typical performance problem turning into an availability problem. See Availability and redundancy.
Delays in auto-scaling
Every cloud has a mechanism for automatic scaling by load. But it doesn't grow instantly.
- Detect the load rise (tens of seconds to minutes of measurement interval)
- Start a server (tens of seconds to minutes)
- Get the app to a responding state (dependency loading, warm-up)
Adds up to minutes. Doesn't catch load peaks that end in minutes. For known peaks like scheduled events, scale up in advance.
Auto-scaling fits gradual changes by time of day. Nightly rise and morning drop are fine.
Offload reads to caches
If reads are heavy, not remaking the same result each time works best.
Cache placement patterns have names.
| Pattern | Steps | Character |
|---|---|---|
| Cache-aside (lazy loading) | On read, check cache; on miss, fetch from database and write back | Only requested data loads. On miss, 3 round trips. Old values may linger after updates |
| Write-through | Update the cache immediately after updating the database | Cache stays fresh. Also loads unread data, consuming space |
The two aren't exclusive — usually combined (Amazon ElastiCache — Caching strategies / AWS — Caching patterns). Either way, set expiries so stale values don't linger.
Different layer from HTTP caching
Browser and CDN caches from HTTP caching and the cache here live in different layers.
| Layer | Who holds | Reduces |
|---|---|---|
| HTTP cache | Browser, CDN, proxy | Requests reaching the server |
| Application cache | Shared cache the servers read | Queries to the database |
The earlier the layer, the greater the effect. If a CDN can return it, neither the server nor the database moves. Conversely, per-user content can't be cached up front, so it's cached in later layers.
Offload writes to queues
Writes can't be scattered by replication. What can be used instead is shifting in time.
Don't do everything within the request — record receipt only, and push processing to the back.
Two things this achieves.
- Faster response. Return without waiting for heavy processing
- Flatten the peak. 1,000 items in a moment; workers consume at a steady rate
Costs. Results don't reflect immediately. You need to design "acknowledged" messaging to users. Failure notification in mid-processing also needs deciding.
Also consider possible duplicate delivery. Many queues guarantee "at least once" delivery at the cost of allowing duplicates. Make the same processing produce the same result on the second run — that's the idempotency covered by Retries and idempotency.
Bottlenecks move
Speed up one place and the next-slowest appears. Often in this order.
Application servers are easy to add, so they're solved first. Then all queries converge on one database, which becomes the next limit. Split the database and you see transfer volume and physical write speed.
The key is predicting where it clogs next. Before scaling apps to 10, see if the database can take 10× queries.
Problems adding doesn't solve
Some problems don't yield to more machines.
- One slow request — 30 seconds per request stays 30 seconds no matter how many machines. Speed it up or make it async
- Contention on a single global resource — everyone updating the same row queues regardless of machine count (Locks and isolation levels)
- Order-sensitive processing — work whose order breaks under parallelism can't be distributed at all
Distinguish the kind of slowness before "slow so add machines".
Decision procedure
- Measure where it clogs — don't add by guess. First distinguish app vs. database
- Try vertical first — if there's headroom in the cap, scale up one machine. No app changes needed
- If going horizontal, enumerate state — session, files, in-memory cache. Check all three can be externalized
- If reads are heavy, consider caching — from the earliest layer. What CDN can return shouldn't reach the server
- If write peaks are the issue, consider queuing — also decide how to communicate delayed reflection to users
- Predict the next bottleneck — verify what limits after adding, then execute
Common misconceptions
"Horizontal scales infinitely." — Only the stateless parts do. The database, shared cache, and external APIs — the shared parts — become the next limit.
"Vertical is old-fashioned." — For write-concentrated databases, vertical can be the most natural solution. Distributing needs consistency design, so if one machine suffices, that's simpler.
"Add a cache and it gets faster." — Only works when the same result is requested repeatedly. Per-user or per-request-changing data doesn't hit. On a miss, the extra round trip makes it slower.
"With auto-scaling, sudden load is fine." — Startup and warm-up take minutes. Doesn't catch peaks ending in minutes. Scale up in advance for known peaks.
Exercises
Q1. Scaling app servers from 1 to 4 made login break frequently. What do you suspect?
Answer and explanation
Session is being held inside the servers.
At 1 server, the same user's requests always went to the same server. At 4, the second request may go to a different server. That server doesn't know the session and treats them as logged-out.
Two remedies:
- Move sessions to a shared datastore — any server sees the same state
- Route the same user to the same server — small change, but that server dying loses the session. Load also skews
The former is the main path. The latter can serve as a stopgap during migration.
Q2. Should the health-check endpoint verify database connectivity?
Answer and explanation
The answer depends on the use. In practice, prepare both.
| What's checked | Use | Checking dependencies makes it |
|---|---|---|
| Process liveness | Restart decision | Unnecessary restarts on dependency issues |
| Full processing including dependencies | Routing exclusion | All machines excluded on one bad database |
Dangerous is using a dependency-checking judgment for routing selection. A single database issue excludes all servers, taking down the whole service.
If you check dependencies, cap the number of servers that can be excluded, or handle dependency issues differently.
Q3. In a service with 95% reads and 5% writes, database load is high. Consider caching or queuing first?
Answer and explanation
Caching first.
95% of load is reads, so applying tools there is natural. Queues reduce write peaks, not total volume. Smoothing 5% doesn't change the remaining 95%.
That said, before caching, verify "is the same result being requested repeatedly". If every request reads different data, caching doesn't hit. In that case, revisiting indexes (Indexes) or read replicas (Data distribution) become the candidates.
Queues can still be needed for 5% writes — when each item is a heavy process. See processing time, not count.
Summary
- Two directions: vertical and horizontal. Vertical is capped at one machine; horizontal scales in count with conditions. Horizontal also has caps — the shared parts
- Horizontal's condition is "any server produces the same result". Externalize sessions, files, in-memory caches
- Load balancer default is round-robin. Choose least-connections when times vary
- A health check that examines dependencies can cause one dependency issue to exclude all machines
- Auto-scaling has minutes of delay. Scale up in advance for known peaks
- Offload reads to caches, writes to queues. The earlier the layer, the greater the effect
- Bottlenecks move. Before adding, see what limits next
- HTTP caching — Return at the earlier layer and set expiries
- nginx — Using nginx as HTTP load balancer — Introduction to distribution defaults and switching
- nginx — ngx_http_upstream_module — Formal definitions per directive. Version and edition conditions live here only
- Amazon ElastiCache — Caching strategies — Trade-offs of lazy loading and write-through
- Practical AWS Guide — ALB — Cloud load balancer configuration and behavior
What to read next
- Availability and redundancy — Reread added machines as failure preparedness
- HTTP caching — Effective caching in the earlier layer