Building Resilient Systems: Circuit Breakers, Timeouts, and Bulkheads

It's 2:14 AM. The pager goes off because checkout latency has gone from 200ms to 11 seconds, and the on-call dashboard shows every service in the request path lighting up red — not because they're broken, but because they're all waiting on one thing: a recommendations service that started responding slowly after a downstream cache node fell over.

Nobody touched the recommendations service. Nobody deployed anything. It just got slow, and slow turned out to be worse than down. A dependency that's fully dead fails fast — you get a connection refused, you handle it, you move on. A dependency that's slow eats a thread, then another, then every thread in the pool, and now your service — which has nothing to do with recommendations — can't serve any request, including the ones that never touch recommendations at all.

That's the failure mode this post is about. Not "how do we prevent failures" — you can't, dependencies will always fail — but "how do we make sure one dependency's bad day doesn't become everyone's bad day." The tools are circuit breakers, timeouts, bulkheads, and retries, and the reason they show up together in every serious systems design conversation is that none of them work well in isolation. A circuit breaker without a sane timeout never trips. A retry without backoff turns one outage into ten. We'll go through the mechanics of each, where they interact, and where we've personally gotten them wrong.

Three ways a dependency takes you down with it

Before the tooling, it's worth being precise about what actually happens when a downstream service misbehaves, because the fix depends entirely on which failure mode you're dealing with.

Timeout / hang. The dependency accepts the connection, accepts the request, and then just... sits there. No response, no error, no TCP reset. This is the worst of the three because your calling thread has no signal to react to — it's blocked until something external (a client-side timeout) forces the issue. Slow database queries, a downstream service stuck in GC pause, a thread pool that's already exhausted on the other end — all of these present as a hang, not a crash.

Crash / hard failure. The dependency is unreachable — connection refused, DNS failure, 5xx immediately. This one is actually easier to handle because the failure is fast and unambiguous. Your code gets an exception in milliseconds and can react.

Cascading failure. This is what turns a single dependency's bad day into an incident review. Service A calls B, B is slow, A's threads pile up waiting on B, A's thread pool exhausts, and now A is slow for everyone calling A — including callers who don't care about B at all. If A is itself a dependency of C, the same thing happens one level up. This is how a cache node falling over in one region takes down checkout, search, and account settings simultaneously — they all shared a thread pool, a connection pool, or an event loop with the thing that got slow.

The core insight: crashes are self-limiting, but hangs are contagious. Everything below exists to convert hangs into fast, contained failures before they propagate.

It's worth naming why this matters more today than it did a decade ago. A monolith calling its own database has one failure domain to reason about. A service mesh with forty microservices, each calling three or four others, has a failure graph, and any node in that graph going slow can affect nodes that never directly call it, through shared infrastructure like a connection pool, a database replica, or a message broker sitting a few hops downstream. The more services you have, the more this stops being an edge case and starts being the normal way outages happen — not "the database went down," but "service K got slow, and forty minutes later, twelve unrelated services were returning errors because they'd all quietly depended on something that depended on K."

Circuit breakers: stop hitting the thing that's already down

A circuit breaker wraps a call to a dependency and tracks its recent success/failure rate. Once the failure rate crosses a threshold, the breaker "opens" and stops making the call entirely — it fails immediately, without even attempting the network request, and (ideally) falls back to something else. The idea is stolen directly from electrical circuit breakers: when downstream current is abnormal, cut the circuit before it burns down the wiring.

There are three states, and the transitions between them are the entire mechanism:

• Closed — normal operation. Requests flow through to the dependency. The breaker is counting failures (usually over a rolling window, e.g. last 100 calls or last 10 seconds) and comparing the failure rate against a threshold.
• Open — the failure threshold was breached. All calls fail immediately without touching the network. This is what protects your threads and the downstream service — you stop sending it traffic while it's underwater, and you stop your own threads from blocking on it.
• Half-open — after a configured wait duration, the breaker allows a small number of trial requests through. If they succeed, it closes again. If they fail, it reopens and waits again.

The subtlety that trips people up: a circuit breaker doesn't make the dependency healthier, and it doesn't make your request succeed. What it buys you is two things. First, it stops your own service from wasting threads and connections on calls that are statistically very likely to fail — those resources go to requests that can succeed instead. Second, it stops piling load onto a struggling dependency, which is often the difference between that dependency recovering in thirty seconds versus never recovering because it's still getting hammered by five upstream services all retrying at once.

The threshold and window size matter more than people expect. Too sensitive (e.g., open after 2 failures in a 10-request window) and you'll trip the breaker on ordinary noise — a transient GC pause on the dependency's side shouldn't take you fully open. Too lax and you'll keep sending traffic into a dependency that's clearly degraded, defeating the purpose. We've settled on failure-rate thresholds around 50% over a sliding window of the last 50-100 calls as a reasonable starting point, then tuned per-dependency based on its actual failure characteristics in production.

Timeouts: one number is never enough

A circuit breaker only works if it has a clean signal of "this call failed." Without an aggressive timeout, a hanging call never fails — it just sits in the "in-flight" bucket forever, and the breaker never gets the data it needs to open. So timeouts come first, conceptually, even though they get less attention.

The mistake we see constantly is a single global timeout — "the HTTP client timeout is 30 seconds" — applied uniformly to every request. That number is almost always wrong in two directions at once. It's too long for a call that should return in 50ms (a cache read, a config lookup), so a hang there costs you 30 seconds of a blocked thread when it should have cost you 200ms. And it's often too short for a genuinely expensive operation — a report generation endpoint, a bulk export — that legitimately needs more time.

The finer-grained model splits timeout into layers:

• Connection timeout — how long to wait to establish the TCP connection / TLS handshake. Should be short (1-3 seconds); if you can't even connect, waiting longer doesn't help.
• Read timeout (a.k.a. socket read timeout) — how long to wait for data once the connection is established. This is usually your dominant tuning knob — it should reflect the actual p99 latency of the specific dependency, not a company-wide default.
• Write timeout — relevant for large payloads or slow uplinks; usually less critical than read timeout for typical JSON APIs but matters for file uploads or streaming.
• Overall request timeout / deadline — a budget for the entire logical operation, including retries. This is the one people forget. If your read timeout is 2 seconds and you retry twice, your worst case is 6+ seconds even though each individual attempt looks "fast" on paper. An overall deadline caps the total regardless of how many attempts happen underneath.

The overall deadline matters even more in service-to-service chains. If service A calls B calls C, and each has its own independent 5-second timeout, a worst case where C is slow can let A wait 15 seconds cumulatively even though no individual timeout looks unreasonable. Propagating a deadline (e.g., via a header like X-Request-Deadline or gRPC's built-in deadline propagation) lets each hop know how much budget is actually left and fail fast once it's exhausted, rather than each layer independently waiting its own full timeout.

Rule of thumb we actually use: timeouts should be set from measured p99 latency for that specific dependency plus a margin, not copy-pasted from another service's config. A payment gateway and an internal Redis cache do not deserve the same timeout, even if it's more work to configure them separately.

Bulkheads: don't let one slow client sink the whole ship

Timeouts and circuit breakers reduce how long a bad dependency can hurt you and how often you keep hitting it, but they don't solve the resource-starvation problem on their own. If your service has one shared thread pool of 200 threads handling calls to five different downstream services, and one of those five gets slow, it can still consume all 200 threads before its circuit breaker even has a chance to trip — because the breaker trips on failures, and a slow dependency inside its timeout window doesn't count as a failure yet.

Bulkheads solve this the way they do on a ship: physical compartments so that if one section floods, the others stay dry. In software, that means giving each downstream dependency (or each class of operation) its own isolated pool of resources — usually a dedicated thread pool or a semaphore-based concurrency limiter, and often a dedicated connection pool as well. If the recommendations-service bulkhead is exhausted because recommendations is slow, requests to the inventory service, which has its own separate pool, are completely unaffected.

This is the single biggest lesson from cascading-failure postmortems: shared resource pools are the mechanism by which an unrelated dependency's problem becomes your problem. The fix is almost always "give this call path its own bounded pool," which sounds obvious in hindsight and is almost never done until after the incident that teaches it.

The trade-off is resource efficiency versus isolation. A shared pool uses capacity more efficiently on average — no thread sits idle just because "its" service isn't currently being called — but that efficiency is exactly what lets one dependency's problem spread. Bulkheads sacrifice some average-case efficiency for a hard ceiling on blast radius. In practice, we size bulkheads per-dependency based on that dependency's actual call volume and criticality, not evenly — the payment gateway bulkhead is bigger than the "fetch related articles" bulkhead, because starving the latter is an acceptable degradation and starving the former is not.

Retries: the thing that turns a blip into an outage

Retrying a failed call seems like the most obviously correct thing to do, and it's also the resilience mechanism most likely to make an outage worse if implemented naively. Here's the failure sequence: a dependency gets slightly overloaded and starts failing 10% of requests. Every caller retries immediately on failure. Those retries add another 10%+ of load on top of a service that was already struggling, which pushes the failure rate higher, which triggers more retries, and the system spirals into what's usually called a retry storm — self-inflicted DDoS, entirely internal.

Two things fix this, and you need both:

Exponential backoff with jitter. Instead of retrying immediately, wait progressively longer between attempts (100ms, 200ms, 400ms, 800ms...), and add randomization ("jitter") so that many clients who failed at the same moment don't all retry at exactly the same moment, which just recreates the synchronized-spike problem at a different point in time. AWS's architecture blog has a well-known writeup on why "full jitter" (randomizing the entire backoff window, not just adding a small jitter to a fixed delay) outperforms naive exponential backoff under load — it's worth internalizing that decorrelated jitter, not just backoff, is what prevents synchronized retry waves.

A hard cap on retry count, combined with respecting the overall request deadline mentioned earlier. Three retries is a common default; more than that rarely helps and mostly just delays the failure while consuming more resources. And retries should generally be skipped entirely once a circuit breaker for that dependency is open — retrying against an open circuit is retrying against something you've already determined is failing, which defeats the purpose of having the breaker at all.

There's also a correctness dimension that's easy to overlook: only retry idempotent operations, or operations you've made idempotent via an idempotency key. Retrying a non-idempotent "charge the customer's card" call because you didn't see the response in time can double-charge someone if the first attempt actually succeeded and only the response was lost. This is a common enough production bug that most payment APIs require an idempotency key on write operations specifically to make retries safe.

Fallbacks: degrade instead of fail

A circuit breaker that's open still needs to return something. The naive answer is an error, and sometimes that's the right answer — you can't fake a payment authorization. But for a large class of read paths, you can serve something meaningfully better than a 500: a cached version of the data, a simplified default, or a "best effort" response computed without the failing dependency.

Examples we've actually shipped: when the personalized-recommendations service is unavailable, fall back to a cached "popular this week" list instead of an empty section. When a third-party address-validation API is down, accept the address as entered and flag it for later validation rather than blocking checkout entirely. When a feature-flag service times out, fall back to the last known-good flag values cached in memory rather than defaulting every flag off (or on) blindly.

The design question for a fallback is always "what's the least-wrong thing we can show," and it forces a genuinely useful conversation with product owners about which parts of the experience are load-bearing and which are enhancements. A checkout page that degrades gracefully by dropping recommendations is fine. A checkout page that degrades by failing to compute tax is not — that dependency doesn't get a soft fallback, it gets a circuit breaker with no fallback and an honest error, because a wrong-but-successful answer is worse than a clear failure.

Health checks and self-healing

Circuit breakers, timeouts, and bulkheads are reactive — they respond to a dependency that's already failing. Health checks are how the platform decides whether to keep sending traffic to a given instance in the first place, and they close the loop on recovery.

Two flavors matter: liveness (is the process still running and not deadlocked — if this fails, the orchestrator restarts the container) and readiness (is this instance currently able to serve traffic — if this fails, the load balancer stops routing to it without killing the process). Conflating the two is a common mistake: a readiness check that's too strict (checking every downstream dependency's health) can cause an instance to be pulled from rotation because of a downstream blip that has nothing to do with the instance's own ability to serve traffic, which just shrinks your healthy fleet exactly when you need capacity most.

There's a related mistake in the other direction: a readiness check that's too shallow — one that only checks "is the HTTP port open" — will happily keep an instance in rotation even after its bulkheads are fully exhausted and every real request is failing. A useful middle ground is to have the readiness check reflect the health of the instance's own critical resources (thread pool saturation, open-circuit count) without transitively checking every downstream dependency's availability, since the circuit breakers and bulkheads are already the mechanism designed to contain those failures locally.

Self-healing, in this context, is mostly the combination of the mechanisms above operating without a human: the half-open state automatically testing recovery, the readiness check automatically re-admitting a recovered instance, the orchestrator automatically restarting a deadlocked process. None of it requires paging anyone at 2 AM if it's wired up correctly — which is the actual goal. The failures you still get paged for should be the genuinely novel ones: a new failure mode nobody's coded a fallback for, or a dependency failing in a way that trips every safeguard at once. Everything routine should resolve itself while you're asleep.

A brief note on watching this happen

Everything above tells you what to do when a dependency fails. It says nothing about how you'd know a circuit breaker just tripped, or that retries have tripled your request volume, or which dependency's p99 latency started drifting an hour before it fell over. That's a distributed tracing and metrics problem, and it deserves its own treatment rather than a paragraph here — for now, the short version is: emit circuit breaker state transitions, retry counts, and bulkhead rejection counts as metrics, not just logs, because logs don't page you and metrics do.

From Hystrix to Resilience4j

Netflix's Hystrix was the library that popularized most of these patterns for the JVM ecosystem — circuit breakers, bulkheads via thread pool isolation, and fallback methods were all first-class concepts in Hystrix, and it shaped how a whole generation of engineers thought about resilience. Netflix put it into maintenance mode in 2018, having moved internally toward adaptive concurrency limits rather than static circuit breaker thresholds.

Resilience4j emerged as the community's answer, built for a different era of the JVM: no dependency on RxJava (Hystrix's reactive core), a lightweight functional-interface-based design, and modules you compose independently rather than one large framework you adopt wholesale. It also integrates cleanly with Micrometer for metrics and Spring Boot via annotation-driven configuration, which is a big part of why it became the default choice for new Spring projects. The move wasn't about Hystrix being "wrong" — it worked, at Netflix's scale, for years — it was about the ecosystem wanting something with a smaller footprint and better composability with the reactive and non-reactive stacks that came after RxJava's dominance faded.

Wiring it up with Resilience4j

Here's a representative example combining a circuit breaker, retry, and bulkhead on a call to an external inventory service, with a fallback method for when the circuit is open.

@Service
public class InventoryClient {

    private final RestTemplate restTemplate;

    public InventoryClient(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackStock")
    @Retry(name = "inventoryService")
    @Bulkhead(name = "inventoryService", type = Bulkhead.Type.THREADPOOL)
    public StockLevel getStockLevel(String sku) {
        return restTemplate.getForObject(
            "/inventory/{sku}/stock", StockLevel.class, sku);
    }

    // Fallback signature must match the original method plus the exception.
    private StockLevel fallbackStock(String sku, Throwable ex) {
        // Serve last-known cached value rather than failing the page.
        return stockCache.getOrDefault(sku, StockLevel.unknown(sku));
    }
}

@Configuration
public class ResilienceConfig {

    @Bean
    public CircuitBreakerConfigCustomizer inventoryCircuitBreakerCustomizer() {
        return CircuitBreakerConfigCustomizer.of("inventoryService", builder ->
            builder
                .failureRateThreshold(50)
                .slidingWindowSize(50)
                .waitDurationInOpenState(Duration.ofSeconds(15))
                .permittedNumberOfCallsInHalfOpenState(5)
                .slowCallDurationThreshold(Duration.ofSeconds(2))
                .slowCallRateThreshold(50)
        );
    }

    @Bean
    public RetryConfigCustomizer inventoryRetryCustomizer() {
        return RetryConfigCustomizer.of("inventoryService", builder ->
            builder
                .maxAttempts(3)
                .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
                    Duration.ofMillis(200), 2.0, 0.5))
                .retryOnException(ex -> ex instanceof TransientInventoryException)
        );
    }

    @Bean
    public ThreadPoolBulkheadConfigCustomizer inventoryBulkheadCustomizer() {
        return ThreadPoolBulkheadConfigCustomizer.of("inventoryService", builder ->
            builder
                .maxThreadPoolSize(20)
                .coreThreadPoolSize(10)
                .queueCapacity(10)
        );
    }
}

A few details worth calling out in this config. The slowCallDurationThreshold treats calls slower than 2 seconds as failures for circuit-breaking purposes even if they eventually return a 200 — this is what lets the breaker react to the "hanging" failure mode, not just outright errors. ofExponentialRandomBackoff bakes in jitter automatically rather than requiring you to hand-roll it. And the bulkhead's queue capacity is intentionally small (10) — a large queue just delays the failure and holds up caller threads longer, which defeats the purpose of having a bounded pool in the first place; better to reject fast once the pool and its queue are full.

Common mistakes we've seen (and made)

Retrying inside a circuit breaker without checking its state. If retry and circuit breaker aren't ordered correctly (circuit breaker wrapping retry, so an open breaker skips retries entirely), you can end up retrying against a dependency the breaker already knows is failing, defeating the entire point of tripping it. Resilience4j's default annotation ordering handles this if you apply the annotations correctly, but it's worth verifying with a test, not assuming.

One timeout value for every dependency. Copy-pasting a 30-second HTTP client timeout across every downstream call means your fast cache reads hang for 30 seconds on a bad day, and your legitimately slow batch operations get killed prematurely. Set timeouts per-dependency from measured latency data.

Shared thread pools across unrelated dependencies. This is the single most common root cause we've seen in cascading-failure postmortems. If dependency X and dependency Y share a pool, X's slowness starves Y's capacity even though Y is completely healthy. Bulkhead every external call that has meaningfully different latency or reliability characteristics.

No jitter on retries. Backoff without jitter still synchronizes retries from many clients that failed around the same time, recreating a load spike at each backoff interval instead of smoothing it out. This is subtle because it looks fine in isolation and only shows up under real concurrent load.

Fallbacks that mask real failures silently. A fallback that always returns a plausible-looking cached value with no metric or log signal means you can be running in degraded mode for days without anyone noticing, because nothing ever technically "errors." Every fallback path should increment a counter so it's visible on a dashboard, even when it's doing exactly what it's supposed to do.

Treating the circuit breaker's open state as the end of the response. Opening the circuit stops the bleeding but still needs to resolve to something returned to the caller — an honest error, a fallback, or a cached value. An open circuit with no fallback registered just changes the exception type your caller sees; it doesn't improve their experience unless something downstream of the breaker actually handles that case.

Final takeaway

None of these patterns are exotic, and none of them require a new framework you haven't heard of — Resilience4j, sensible timeout configuration, and separate thread pools per dependency get you most of the way there. What actually requires discipline is treating every external call as something that will eventually fail in a way you didn't plan for, and deciding in advance what "acceptable degradation" looks like for that specific call path. The system that survives a 2 AM dependency outage isn't the one with the cleverest failure handling — it's the one where someone already asked "what happens if this hangs for 30 seconds" for every single downstream call, before it happened in production instead of after.

Next in this system design series: Distributed Tracing & Observability, where we'll cover how to actually see the failure you just built resilience against, before it becomes an incident.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.