Availability and redundancy — designing for failure

Servers break. Networks fail. These are design premises, not exceptions.

So the question isn't "how to not break". It's "what happens when it breaks". Some designs answer it and some don't.

What you'll learn in this chapter

• Finding single points of failure (SPOF)
• Multiplying availability — series drags down, parallel pulls up
• Standby patterns (active-standby / active-active)
• Failover time and what happens during it
• Degraded operation — give up a part instead of stopping everything
• Not escalating performance problems into availability problems

Prerequisites

Builds on horizontal distribution from Scaling directions. Also uses the three-way failure classification from Retries and idempotency.

What this chapter does not cover

Mechanisms that isolate cascading failures aren't covered. Circuit breakers (temporarily stop calls to non-responding dependencies) and bulkheads (partition resources per dependency so one clog doesn't spread) belong to inter-service integration guides. This chapter focuses on how to structure redundancy within a single service.

Server-side request rate limiting is covered by Laravel API Development Guide — Middleware and rate limiting. Here we cover the limited side (client) behavior, in Retries and idempotency.

Finding SPOFs

A single point of failure (SPOF) is a spot whose failure stops the whole service. Finding them is simple.

Look for boxes there's only one of, in the architecture diagram.

There are two apps. But the load balancer and database are each one. Whichever breaks, service stops. Scaling apps to 10 doesn't touch those two.

Look at lines too, not just boxes. Two servers on the same power, same network device, same building — on the diagram there are two, but really there's one. Cloud "put them in different availability zones" is about removing that sharing.

Count external services too. One payment API is also a SPOF. If it can't be made redundant, decide in advance what to give up when it goes down (degraded operation, below).

Multiplying availability

Availability is "the fraction of time it's up". As numbers, two counterintuitive properties appear.

Series drags down

Missing even one required thing means it doesn't work — that's "in series". The whole availability is the product of each part's (AWS — Availability with dependencies).

const series = (parts: number[]) => parts.reduce((a, p) => a * p, 1);

// four 99.9% parts, all required
console.log((series([0.999, 0.999, 0.999, 0.999]) * 100).toFixed(3) + "%");

99.601%

Lining up four 99.9% parts gives 99.6% overall. More parts → lower. "Each is three 9s so the whole is three 9s" is false.

This directly guides design decisions. Adding a part itself lowers availability. See whether the value of adding compensates for the drop.

This product assumes parts fail independently. Measurements deviate both ways.

• Higher than product — sharing infrastructure makes failures overlap, so downtime doesn't add
• Also higher than product — parts often run at higher availability than the published number. The AWS whitepaper notes that configurations with 2–3 dependencies at 99.99% can achieve ≥ 99.99% (same reference)
• Lower than product — when there are dependencies you didn't count. If something not in the product goes down, uptime is shorter than the calculation

Numbers are tools for narrowing candidates, not guarantees.

Parallel pulls up

Multiple things playing the same role, and any one working is enough — that's "in parallel". Everything is only down when all are simultaneously broken, so the failure probability multiplies (AWS — Availability with redundancy).

const parallel = (a: number, n: number) => 1 - Math.pow(1 - a, n);

console.log("99% × 2: " + (parallel(0.99, 2) * 100).toFixed(4) + "%");
console.log("99% × 3: " + (parallel(0.99, 3) * 100).toFixed(4) + "%");

99% × 2: 99.9900%
99% × 3: 99.9999%

Two 99% servers give 99.99%. Each added machine shrinks the down probability by the failure factor. At 1% failure, 2 machines gives 0.01%; 3 gives 0.0001%. Higher initial availability, greater per-machine effect.

But this assumes independent failure. Same building, same power, same buggy software → simultaneous failure. Redundancy's effect is set by how independent you made them.

Overlapping failures have opposite meaning for series and parallel. For parallel, redundancy stops working, which is bad. For series, downtime doesn't add — numerically favorable. Both come from the "independence" assumption breaking, not something to induce deliberately.

Number of 9s and annual downtime

Availability numbers land better as annual downtime.

const MIN_PER_YEAR = 365 * 24 * 60;
for (const a of [0.99, 0.999, 0.9999, 0.99999]) {
  const down = MIN_PER_YEAR * (1 - a);
  console.log(`${(a * 100).toFixed(3)}% → ${down.toFixed(0)} min/yr (${(down / 60).toFixed(1)} hours)`);
}

99.000% → 5256 min/yr (87.6 hours)
99.900% → 526 min/yr (8.8 hours)
99.990% → 53 min/yr (0.9 hours)
99.999% → 5 min/yr (0.1 hours)

Adding one 9 divides downtime by 10. And the required cost climbs. Whether the difference between 53 and 5 minutes justifies the investment depends on the service.

Redundancy patterns

Even with two machines, how you use the standby differs.

Pattern	Standby state	Failover time	Steady-state resources
Active-standby	Booted but not processing	Detection + switch	Standby is idle
Active-active	Both processing	Just remove from routing	Both used
Cold standby	Stopped	Longest (starts from boot)	Cheapest

Active-active isn't always better. A setup where both write concurrently requires consistency design (Data distribution). Read-centric app servers fit naturally; databases are a different story.

In active-active, also confirm capacity when one goes down. A two-machine setup splits load; with one, load doubles. If one can't handle it, that's not redundancy.

Failover time

Saying "redundant, so it's fine" often misses what happens to the service during the switch.

Failover isn't instantaneous. Stages:

1. Detect the failure — how many failures / how many seconds to conclude no response
2. Promote the standby — make it accept writes if it's a database
3. Switch endpoints — rewrite name resolution, or re-establish connections
4. Clients grab the new endpoint — cached name resolution needs to expire

Amazon RDS Multi-AZ DB instance deployments perform failover by pointing the database name (CNAME) at the standby, taking usually 1–2 minutes from detection to resumed processing (Amazon RDS Multi-AZ). Multi-AZ DB cluster deployments with readable standby typically take under 35 seconds (AWS CLI — failover-db-cluster). The same "Multi-AZ" name covers deployments with different failover times.

During switching, the app fails to connect. So connection retry belongs on the app side. This corresponds to the "indeterminate" case in Retries and idempotency. Requests fired at the moment of switching don't know if they were processed.

Long-held name resolution keeps grabbing the old endpoint after switching. In failover-oriented setups, keep the hold short (DNS).

Degraded operation

When a non-redundant piece breaks, the choice isn't only "stop everything". You can give up the broken feature and keep the rest running.

Broken	Stop everything	Degrade
Recommendation API	Whole site returns 500	Hide the recommendation slot; show the product page
Search engine	Search errors, unusable	Fall back to a newest-first listing
Payment	Purchase errors	Block purchase; keep browsing and cart working

For degraded operation to work, you need to have split features into essential and non-essential at design time. Calls to non-essential dependencies should not drag the whole thing down on failure.

This classification also affects availability calculation. Non-essential dependencies don't enter the series product. If the recommendation API dies and the product page still shows, that API's availability doesn't lower the whole.

Don't turn performance into availability problems

Something that was just slow can become "stops" depending on the reaction.

Three typical cases.

• Health checks too strict — temporary slowness reduces machines, load on the rest rises, and more get excluded (Scaling directions)
• Timeouts too long — connections waiting for responses pile up, new requests can't be accepted. One slow dependency stops the whole thing
• Immediate retries on failure — piles more load on an overloaded partner. Corresponds to "overload" in Retries and idempotency; need to wait before resending

The commonality is the reaction to "slow" is making it worse. Leave room to accept slowness as slowness — set timeouts, cap wait volume, space out retries.

Decision procedure

1. In the diagram, count boxes and lines there's only one of — count external services and physical sharing too
2. Decide acceptable downtime — minutes per year. Without this, investment can't be judged
3. Compute the series product — line up required dependencies and multiply. If short of target, reduce parts or add redundancy
4. Choose the redundancy pattern — confirm capacity when one goes down
5. Estimate failover time — see what happens to clients during it
6. Design degradation — enumerate non-essential dependencies and make failure not drag the whole thing down
7. Inspect reactions to "slow" — timeouts, health checks, retry intervals

Common misconceptions

"Two machines doubles availability." — It's a fraction, not doubled. 99% becomes 99.99%. In downtime, one-hundredth.

"Redundant so it won't go down." — That's for independent failures. Same power, same misconfiguration, same software bug break both together. Redundancy's effect depends on independence.

"Higher availability is always better." — The cost of adding one 9 exceeds the previous. If 53 minutes of yearly downtime is acceptable, the investment for 5 minutes can go elsewhere. The service's nature decides.

"Failover is automatic so ignore it." — During switching, apps fail to connect. Retries and name-resolution hold-time settings are needed.

Exercises

Q1. Five 99.9% parts, all required. What's the whole availability? If the target is 99.9%, what do you do?

Answer and explanation

About 99.5% (0.999 ^ 5 = 0.99501). In annual downtime, about 44 hours — far exceeding the 8.8-hour target.

Two directions:

• Reduce series parts — turn non-essential dependencies into degradation targets. Move recommendation-like features out of the product
• Add redundancy to parts — making 99.9% into two gives 99.9999%, making its contribution to the product nearly nil. But one part left at 99.9% caps the whole at 99.9%, so this direction alone means adding redundancy to all five

Look at the first one first. Redundancy has cost; removing a dependency from "required" is often a design change alone.

Q2. You made the database redundant with active-standby. Can you now say "SPOFs are gone"?

Answer and explanation

You can't. Points remain to verify.

What to check	Why
Do the two fail independently?	Same building, same power → simultaneous
What happens during failover?	Tens of seconds to minutes of unreachable, depending on setup. Decide how to handle
Does failover actually work?	Untested failovers can fail to fire
What about other boxes?	Fixing the database still leaves the load balancer as a SPOF if it's one

"Fails over if broken" is a hypothesis until you actually break it. You need operations that periodically test switching.

Q3. The external payment API stopped responding, and the whole site returned 500. What do you fix?

Answer and explanation

Prevent the payment API failure from taking down non-payment features.

The cause is usually one of:

• Waiting for a response — no timeout or too long. Waiting connections pile up, no new requests accepted
• Exception propagates to the top — payment failure is treated as page-wide error

Order:

1. Set timeouts — decide how long to wait. Without it, a slow dependency stops the whole thing
2. Decide alternative behavior on failure — disable only the purchase button; keep browsing and cart working
3. Space out retries — if overload was the cause, immediate retry worsens it (Retries and idempotency)

The problem is one dependency's failure becoming a whole-system failure. Even if the payment API itself can't be made redundant, the scope of impact is a design choice.

Summary

• SPOFs are found by looking for boxes and lines with just one in the diagram. Count external services and physical sharing
• Series multiplies down. Four 99.9% parts give 99.6%. Adding parts itself lowers availability
• Parallel multiplies failure probability, up. Two 99%s give 99.99%. Only when independent
• Redundancy patterns split by whether the standby is used. Confirm capacity when one goes down
• Failover isn't instant. Clients fail to connect during it, so retries are needed
• Designing degradation moves non-essential dependencies out of availability calculation
• Misdesigned timeouts, health checks, and retries turn performance into availability problems

Related references

• Retries and idempotency — Why requests in flight during failover are "indeterminate"
• AWS — Availability with dependencies — Theoretical max availability from multiplying required dependencies
• AWS — Availability with redundancy — Availability of a redundant setup as the failure-probability product
• Amazon RDS Multi-AZ — Failover by name change, and its timing
• Practical AWS Guide — RDS — Multi-AZ setup and operations

What to read next

• Data distribution — What happens to consistency across multiple machines
• Retries and idempotency — Splitting failure into three and setting resend conditions