Designing for High Availability & Disaster Recovery

It is 2:14 AM. The primary region's managed database has stopped accepting connections — not a slow query, not a failover, just gone. The dashboard shows green across the board because the health checks are hitting the app tier, and the app tier is up. Checkout is not. Orders are silently failing at the payment step, and nobody notices for eleven minutes because the alert that should have fired was scoped to HTTP 5xx rates, and the app was returning 200s with an error payload the frontend swallowed.

By the time someone pages the on-call engineer, there is one on-call engineer, and she is on a flight. The secondary on-call has read access to the runbook but not the IAM role needed to promote the standby database. The runbook, written eight months earlier, references a load balancer console that was replaced in a migration nobody updated the doc for.

Nothing in this story is exotic. It is the median outage. Every piece of it was "handled" on paper — multi-AZ database, documented on-call rotation, a DR runbook — and none of it worked when it mattered, because none of it had been exercised end to end. This post is about the difference between an architecture that looks resilient and one that actually survives contact with a real failure: the availability math, the redundancy models, the failover mechanics, and the discipline of testing recovery instead of assuming it.

---

What the Nines Actually Cost You

"Three nines" gets thrown around as a target without anyone doing the arithmetic. Let's do it once, properly, so the number means something the next time it's in a slide.

Availability	Downtime / year	Downtime / month	Downtime / week
99%	3.65 days	7.3 hours	1.68 hours
99.9% ("three nines")	8.77 hours	43.8 minutes	10.1 minutes
99.95%	4.38 hours	21.9 minutes	5.04 minutes
99.99% ("four nines")	52.6 minutes	4.38 minutes	1.01 minutes
99.999% ("five nines")	5.26 minutes	26.3 seconds	6.05 seconds

The jump from 99.9% to 99.99% looks like a rounding error on a slide. In practice it's the difference between "we can take a maintenance window without paging anyone" and "a single unplanned restart consumes the entire annual error budget." At 99.99%, your total allowed downtime for the whole year is less than an hour, which means your deploy pipeline, your database failover, your DNS TTL, and your on-call response time all have to work correctly on the first try, every time, because you don't have budget left for a second attempt.

This is why the last nine is disproportionately expensive. Going from 99% to 99.9% is mostly about removing obvious single points of failure — a second app server, a managed database with automatic failover, retries with backoff. Going from 99.9% to 99.99% requires active-active infrastructure, automated failover with sub-minute detection, chaos testing, and usually a second region. Going from 99.99% to 99.999% requires all of that plus eliminating dependencies you didn't think were dependencies: your certificate authority, your DNS provider, your cloud provider's control plane (not just the data plane), and your own deploy tooling. Each additional nine is roughly an order of magnitude more expensive to buy, and most businesses don't need to buy it — the mistake is choosing a number because it sounds good rather than because a customer contract or a genuine revenue-per-minute calculation demands it.

Before designing anything, force the actual question: what does one hour of downtime cost this business, in money, in trust, in regulatory exposure? If the honest answer is "an annoyed Slack message," you don't need five nines. If the answer is "we lose a six-figure SLA credit and three enterprise customers churn," now you have a budget to justify the multi-region investment.

---

Single Points of Failure Hide in Places You Don't Draw

Most architecture diagrams show redundant app servers and call it a day. The SPOFs that actually cause outages are rarely the boxes on the diagram — they're the things everyone assumed were "someone else's problem."

The obvious ones, which most teams do handle:

• A single application server or container instance
• A single database instance with no replica
• A single availability zone

The ones that get missed, which are the ones that actually page you at 2 AM:

• DNS — one registrar, one DNS provider, a TTL of 24 hours that means your beautifully engineered failover can't propagate for a day.
• TLS certificate expiry — an auto-renewal job that silently failed three months ago, and nobody notices until the cert expires and every client starts rejecting the handshake. This is one of the most common "self-inflicted disasters" in production systems, and it has nothing to do with server redundancy.
• A single region's control plane — your compute is multi-AZ, but the API you use to launch new compute, attach volumes, or update DNS records lives in a single region's control plane that goes down independently of the data plane. Multi-AZ doesn't save you if the orchestration layer above it is regional.
• A single CI/CD pipeline or artifact registry — you can't roll forward or roll back if the thing that ships the fix is itself down.
• A single identity provider — if your SSO/IdP has an outage, "just log in and fix it" stops being an option, including for the people who need to run the recovery.
• A single on-call person — the rotation exists on paper, but only one engineer actually has the production access, the institutional knowledge, or the muscle memory to execute the runbook. Bus-factor-one is a SPOF exactly like a database with no replica; it just doesn't show up in an infrastructure diagram.
• A single upstream vendor — payment processor, email provider, feature-flag service — treated as "always up" because it's not your code.

The discipline that finds these is not a diagram review, it's a dependency walk: for every request your system serves, list every system, service, and human it touches before a response goes out, and ask "what happens if this one thing is unavailable right now." Do that walk honestly and the DNS provider, the cert renewal job, and the on-call bus factor stop being footnotes and start being line items in the resilience plan.

---

Active-Active vs. Active-Passive: The Trade-Off Nobody States Honestly

Redundancy comes in two shapes, and the difference between them is not a technical nuance — it's a direct trade of money and complexity against failover speed.

Active-passive: one region (or one instance, at smaller scale) serves all live traffic. A standby is provisioned, kept warm or cold, and only takes traffic when the primary fails. This is cheaper to run and dramatically simpler to reason about — there's one source of truth, one set of writes, no distributed consistency problem. The cost is failover time: promoting a standby, updating routing, and confirming it's healthy takes anywhere from tens of seconds to tens of minutes, and during that window you are down.

Active-active: two or more regions serve live traffic simultaneously, usually behind a global load balancer or geo-DNS. If one region fails, traffic already flowing to the healthy region continues uninterrupted, and traffic is rerouted away from the failed region — no promotion step, no cold start. The cost is that you now have a genuine distributed systems problem: writes happening in two places need to converge, and you must pick a consistency model (synchronous cross-region replication, which adds latency to every write, or asynchronous replication, which reintroduces the RPO gap you were trying to eliminate) and you need every stateful component — not just the database, but caches, session stores, feature flags, rate limiters — to support multi-region operation.

Notice the replication arrow between the two databases: that link is where every hard problem in active-active lives. If it's synchronous, a write isn't acknowledged until both regions confirm it, which means your write latency now includes a cross-region round trip — often 60-150ms depending on the regions — on every single write, all the time, to protect against a failure that might happen once a year. If it's asynchronous, writes commit locally and replicate in the background, which keeps latency low but means a region failure can lose the writes that hadn't replicated yet, and two regions accepting conflicting writes to the same record concurrently need an explicit conflict resolution strategy (last-write-wins, vector clocks, CRDTs, or routing all writes for a given entity to one "home" region even in an otherwise active-active setup).

The honest framing for a design review: active-passive if your RTO tolerance is minutes and you want one thing to reason about; active-active if your RTO tolerance is seconds and you're willing to hire the complexity of distributed writes. Most teams that reach for active-active do it because it sounds more impressive in an architecture review, not because they've done the RTO math that justifies it. Don't take on cross-region write conflict resolution to shave an RTO from four minutes to thirty seconds if nobody has shown that thirty seconds matters.

---

Failover: Automatic Sounds Safer Than It Is

Failover mechanisms split into automatic and manual, and the instinct is always "automatic is strictly better — humans are slow and make mistakes at 3 AM." That instinct is wrong often enough that it deserves pushback.

Manual/graceful failover means a human confirms the primary is actually down (not just slow, not just experiencing a transient network blip) and explicitly triggers the promotion. It's slower — you're bounded by human response time and confirmation — but it's much harder to trigger incorrectly.

Automatic failover removes the human from the loop: a health check fails N times in a row, and the system promotes the standby without asking anyone. This is faster and is the only way to hit sub-minute RTOs. But automatic failover has a failure mode of its own that manual failover doesn't: flapping. If your health check is too sensitive, or the failure is a brief network partition rather than a real outage, you can end up promoting a standby, having the "failed" primary come back seconds later, and now you have two nodes that both believe they're primary — split brain — both accepting writes, both about to conflict. An automatic failover system that isn't designed with hysteresis (sustained failure over a window, not one missed check), fencing (guaranteeing the old primary genuinely cannot accept writes once demoted — STONITH-style, "shoot the other node in the head"), and a clear quorum mechanism will, given enough time, cause an outage that a manual process never would have.

The practical middle ground most mature teams land on: automatic detection and automatic alerting, with automatic promotion gated behind a short, deliberately-imposed confirmation window (say, health check must fail continuously for 45-90 seconds before promotion fires), plus fencing that makes it physically impossible for the old primary to keep taking writes after demotion. Fully automatic, zero-delay failover is appropriate only once you've proven — through actual drills, not intent — that your health checks don't false-positive on transient blips.

---

Data Consistency Is the Bill You Pay for Geo-Redundancy

Multi-region deployment buys you protection against a whole region going dark — a real, not hypothetical, failure mode; cloud providers have had full-region outages. But every unit of geo-redundancy you buy is paid for in one of two currencies: latency (synchronous replication, strong consistency, slower writes everywhere) or staleness (asynchronous replication, eventual consistency, a window where regions disagree and a failover can lose recent writes).

There is no configuration that gives you multi-region, strongly consistent, low-latency writes simultaneously — that's the CAP/PACELC trade-off showing up in a design review instead of a textbook. What you can do is stop treating "consistency" as one setting for the whole system and instead ask, per data type, how much staleness is tolerable:

• User session state — usually fine to be eventually consistent or even region-local; a user re-authenticating after a regional failover is annoying, not catastrophic.
• Product catalog / content — near-perfect candidate for asynchronous, eventually-consistent replication; a few seconds of staleness on a product description is invisible.
• Inventory counts and payment state — the place where staleness turns into real business damage (overselling, double-charging) and where you either need synchronous replication for that specific data, or a design that tolerates conflicts explicitly (e.g., reconciling oversells after the fact rather than trying to prevent them with a global lock).

Segmenting consistency requirements by data type, instead of picking one replication strategy for the entire database, is usually how teams get most of the latency benefit of async replication without taking on unlimited risk on the data that actually needs to be right.

---

RPO and RTO: Define Them From the Business, Not the Diagram

RPO (Recovery Point Objective) answers: how much data can we afford to lose, measured in time? If your last backup or replication checkpoint was 10 minutes before the failure, your RPO is effectively 10 minutes — every write in that window is gone.

RTO (Recovery Time Objective) answers: how long can we be down before the damage becomes unacceptable? This is measured from the moment of failure to the moment full service is restored, not from the moment someone starts working on it.

The mistake nearly everyone makes is setting these numbers based on what the current architecture happens to support, then calling it a target. That's backwards. The number should come from the business impact, and the architecture should be built to hit it — or the business should consciously accept a cheaper architecture and a worse number, with eyes open.

Worked Example: Checkout Service for an E-Commerce Platform

Say we run checkout for a mid-size e-commerce platform: 40,000 orders/day, average order value $85, roughly evenly distributed with a peak multiplier of 4x during a flash sale.

Business-driven RPO. Every lost order-write is a customer who clicked "place order," possibly got charged, and now has no record of it — a chargeback risk and a trust problem, not just a support ticket. At average traffic (40,000 orders / 24h ≈ 27.8 orders/minute), a 5-minute RPO means up to ~139 orders could be lost or duplicated in a failure — roughly $11,800 of directly at-risk order value at $85 AOV, before counting refunds and support cost, which historically run 3-5x the direct figure. Leadership decides that's tolerable at 5 minutes but not at 30 (≈833 orders, ~$70,800 exposure). RPO target: 5 minutes, which forces near-synchronous replication specifically for the order and payment tables — the backup/checkpoint cadence and transaction log shipping interval need to guarantee no more than 5 minutes of unreplicated writes at any point.

Business-driven RTO. Checkout being fully down costs the full run-rate of orders — 27.8/minute × $85 ≈ $2,363/minute, rising to ~$9,450/minute during a 4x flash-sale spike, before counting the cart-abandonment tail after service is restored. Leadership sets a hard ceiling: no more than 15 minutes of downtime during a flash sale (≈$142,000 exposure, judged acceptable against the cost of the redundancy needed to beat it), and a looser 60-minute ceiling on ordinary days. RTO target: 15 minutes at peak, 60 minutes otherwise.

Translating those numbers into architecture: a 15-minute peak RTO rules out a fully manual "page someone and wait" process — human diagnosis alone can eat that budget before anyone touches the standby. It requires automated detection, a promotion path rehearsed enough that a sleep-deprived on-call engineer can execute it in under 5 minutes, and a standby that's already warm. The 5-minute RPO rules out nightly-only backups and requires continuous log shipping scoped to the order/payment write path — while the catalog, recommendations, and marketing content can tolerate a far cheaper, staler replication strategy, because losing 30 minutes of "recently viewed items" costs nothing next to losing 30 minutes of orders.

This is the point of doing the math: it tells you which subsystems deserve the expensive treatment and which don't, instead of applying five-nines engineering uniformly across a system where most of it doesn't need it.

---

Backups You Haven't Restored Are a Hypothesis, Not a Backup

A backup job that completes successfully tells you exactly one thing: bytes were written somewhere. It tells you nothing about whether those bytes can be turned back into a working database, whether the schema migrations applied since the backup are compatible, whether the restore process actually finishes inside your RTO window, or whether anyone on the team knows how to run it under pressure.

The number of organizations that discover their backups are corrupted, incomplete, encrypted with a key nobody kept, or simply too large to restore within any reasonable RTO — only during the actual incident — is not small. Backups are a claim. A restore test is what turns the claim into a fact: on a schedule, run a full restore into an isolated environment, time it, and validate against a checksum or business invariants (order counts reconcile, referential integrity holds, nothing silently truncated). If that restore test isn't automated and scheduled, the backup strategy is aspirational.

---

Drills Are Where DR Plans Go to Fail (Which Is the Point)

A disaster recovery runbook that has never been executed is a work of fiction with good production values. It reads coherently, it has diagrams, it has a table of contents — and the first time someone actually tries to follow it during a real outage, they discover step 4 references a console that was deprecated, step 7 assumes an IAM permission that was revoked in a security audit, and step 11 says "notify the database team" with a Slack channel that was archived.

This is the normal outcome, not a sign of an unusually bad team. Documentation drifts from reality continuously — every infra change, every access policy update, every tool migration invalidates some sentence in the runbook, and nobody re-validates it on that cadence. The only way to keep it honest is to periodically execute it for real: a game day, where the team deliberately fails a real component (kill the primary database, sever a region's network path, expire a certificate in staging) during business hours, with stakeholders aware, and follows the actual documented process to recover — the literal keystrokes, not a tabletop discussion of what they'd do.

Game days reliably surface the same categories of gaps: stale runbooks, missing access grants only the original setup person still has, monitoring that doesn't alert on the failure mode being tested, and a competent team that's never coordinated a live recovery and wastes ten minutes figuring out who's driving. Every one of those is far cheaper to find on a Tuesday afternoon than during a real outage. Treat the first drill as an expected source of findings, not a pass/fail test — a plan that survives its first drill unchanged either got lucky or wasn't tested hard enough.

That timeline is a good outcome — under five minutes from failure to full traffic cutover, with an explicit fencing step so the old primary can't accept a stray write after it's been demoted. The realistic version, the first time a team runs this drill, has extra hours added at nearly every step: the page goes to someone without the right access, the "trigger promotion" step is a wiki page instead of a script, and the DNS change takes 20 minutes to propagate because the TTL was never lowered. The gap between the clean sequence diagram and the messy first drill is exactly the value of running the drill before you need it for real.

---

SLA, SLO, and SLI — Three Different Documents for Three Different Audiences

These three terms get used interchangeably and shouldn't be, because they answer different questions to different audiences:

• SLI (Service Level Indicator) — the actual measurement: request success rate, p99 latency, error rate. This is the number your monitoring system produces.
• SLO (Service Level Objective) — the internal target for that indicator: "99.95% of requests succeed over a rolling 30-day window." This is what your team engineers toward and what triggers internal alerting when you're burning error budget too fast.
• SLA (Service Level Agreement) — the external, usually contractual, promise, typically set looser than the internal SLO on purpose, with financial or contractual consequences for missing it (service credits, termination rights).

The relationship that matters: SLA should always be looser than SLO, which should always be looser than what the architecture can actually deliver on a good day. If SLA and SLO are the same number, you have zero margin — the first bad month puts you in breach of a customer contract, not just an internal ding. Everything above — RPO/RTO, the redundancy model, the failover mechanism — exists to make a specific SLO achievable, and the availability math from the first section is the ceiling on which SLO is even physically possible.

---

Load Balancing Choices That Actually Matter for Failover

Load balancing algorithms get discussed as a performance topic, but the choice matters just as much for failover behavior:

• Round-robin distributes requests evenly regardless of backend health beyond a binary up/down check. Simple, but it will keep sending traffic to a node that's technically "up" but degraded (high latency, partial failures) until a health check explicitly marks it down.
• Least-connections routes to whichever backend has the fewest active connections, which naturally shifts load away from a struggling node faster than round-robin, since a slow node accumulates connections.
• Health-check-based routing is the one that actually enables failover, and its value is entirely in how the health check is defined. A shallow check (does the process respond to a TCP ping) will report healthy long after the node has stopped doing useful work — the process is up, but its database connection pool is exhausted, or its dependency is timing out. A deep health check (can this node complete a representative read and write against its actual dependencies within a latency budget) catches the failures that matter, at the cost of more load and more complexity in the check itself.

For DR purposes, the global traffic manager needs to execute the region-level cutover — its health checks need region-level granularity, not just per-node, and its failover decision needs the same anti-flapping protection discussed earlier, or you get traffic bouncing between two regions during a partial degradation, which is worse than staying on the degraded one.

---

Common Mistakes

Calling multi-AZ "disaster recovery." Multi-AZ protects against a rack or data-center-level failure within one region. It does nothing for a regional control-plane outage, a bad deploy that corrupts data in every AZ simultaneously, or a misconfiguration that replicates instantly to every replica. DR means surviving the loss of an entire region or an entire class of failure, not surviving a single machine dying.

Treating "backup completed" as "recoverable." Covered above, worth repeating as its own line item because it's the single most common false sense of security in the room: a green checkmark on a backup job is not evidence the data can be restored. Only a timed, validated restore test is.

Writing a DR runbook and never executing it. A plan that exists only on paper accumulates drift from reality with every infrastructure change until it's actively misleading. If it hasn't been run in the last two quarters, assume it's wrong.

Setting RTO/RPO to match what the architecture happens to support, instead of what the business needs. This inverts the entire exercise — it turns a business risk conversation into an architecture-justifies-itself conversation, and it means you find out the number was wrong exactly once, during the incident that tests it.

Fully automating failover without fencing or hysteresis. Sub-minute automatic failover sounds like the mature choice, but without guarding against transient blips and guaranteeing the old primary can't keep writing after demotion, automatic failover creates split-brain outages that a slower, human-confirmed process would never have triggered.

Ignoring the single on-call person and the single vendor dependency as SPOFs. Redundant servers with a bus-factor-one team, or a fully redundant multi-region architecture that all routes through one payment processor with no fallback, is not actually redundant — it's redundant on the parts that were easy to diagram and fragile on the parts that were easy to ignore.

---

Final Takeaway

High availability and disaster recovery are not features you buy off a cloud provider's pricing page — they're a set of explicit decisions about how much downtime and data loss the business can survive, backed by architecture and, critically, backed by proof that the architecture works under real failure. The math on the nines tells you what you're actually signing up for. The SPOF walk tells you where the plan is fiction. The RPO/RTO exercise, done from business impact rather than architectural convenience, tells you which parts of the system deserve the expensive treatment and which don't. And the drill — the one that goes badly the first time, surfaces the stale runbook and the missing access grant, and gets fixed before the next quarter's real outage — is the only thing that turns a nicely drawn diagram into a system that survives 2:14 AM.

Next in this system design series: Database Scaling & Sharding Strategies, where we'll go deeper into the data-layer trade-offs this post only touched on when discussing multi-region replication.

---

Subscribe to get system design and backend engineering posts in your inbox every two weeks.