Real-Time Updates at Scale: TCP, Load Balancers, and WebSockets Explained

Watch a rideshare app for thirty seconds and the driver's pin moves smoothly across the map. No refresh button. No spinner. It just updates.

That single detail hides three separate engineering problems, and most system design explanations skip straight to "use WebSockets" without answering any of them:

• How does an update actually travel from the server to a browser that's just sitting there, idle?
• What has to be true about the network connection for that to work at all?
• If you put a load balancer in front of your servers — which every production system has — does it even support this?

This post works through those three questions in order: the network layer underneath every connection, the load balancer decision that trips up most designs, and the WebSocket architecture that ties it together.

---

The Real Question Behind "Real-Time"

Strip away the buzzword and a real-time feature is really two separate questions:

Step 1 is a connectivity problem: the client didn't ask for anything, so how does data reach it without polling every second?

Step 2 is a plumbing problem inside your own backend: something happened — a driver's GPS ping, a new chat message, a price change — and now that event has to find its way to the one server process holding the open connection to the right client.

Most teams solve Step 1 by picking WebSockets and stop there. Step 2 is where real systems break, because "notify the client" quietly turns into "notify the client, but only the one server out of fifty that's holding their connection." We'll come back to that after the fundamentals.

---

What "Connection" Actually Means: TCP and HTTP

Before comparing polling, long polling, and WebSockets, it's worth being precise about what a "connection" is, because the load balancer discussion later only makes sense once this is clear.

Every HTTP request — REST call, page load, or the handshake that upgrades into a WebSocket — sits on top of a TCP connection. TCP has to be established before a single byte of HTTP is exchanged, and it has to be torn down afterward:

Three things matter about this diagram for everything that follows:

1. The three-way handshake (SYN, SYN-ACK, ACK) has a cost. It's a full round trip before any application data moves. On a normal REST API, keep-alive connections amortize this cost across many requests. On a mobile network with 100–200ms of latency, it's still noticeable.
2. Closing a connection is also a negotiation (FIN/ACK on both sides), not an instant drop. Load balancers and app servers both need to track connection state during this teardown, which is why "just kill the process" during a deploy causes dropped requests.
3. A plain HTTP request-response cycle opens and closes this cycle every time, unless something keeps the connection alive on purpose. That "something" is exactly what a WebSocket does — it performs one normal HTTP handshake, then asks the server to upgrade the same TCP connection so it never closes.

That last point is the whole trick behind WebSockets: there's no special "real-time protocol" running underneath. It's the same TCP connection an HTTP request would use, just held open indefinitely instead of closed after one response.

---

Why This Breaks Your Load Balancer

Here's where most system design walkthroughs go quiet, because the answer depends on a distinction that's easy to skip: Layer 4 vs. Layer 7 load balancing.

• Layer 7 (application layer) load balancers — think a typical Nginx or ALB config in HTTP mode — terminate the TCP connection from the client, read the HTTP request, and open a new TCP connection to a backend. They understand paths, headers, and cookies, which is why they're the default choice for REST APIs.
• Layer 4 (transport layer) load balancers don't look inside the request at all. They see TCP (or UDP) packets and forward them based on IP and port, without terminating or re-establishing the connection.

For a normal REST endpoint, L7 is usually the right choice — you get routing rules, retries, and request-level observability. For a long-lived WebSocket connection, L7 becomes a liability:

The problem with terminating a WebSocket connection at an L7 proxy is that the proxy now has to hold two long-lived connections open per client — one to the browser, one to the backend — for as long as the session lasts, and re-encode every frame between them. At a few hundred concurrent users that's invisible. At hundreds of thousands of concurrent WebSocket connections, that per-connection memory and CPU overhead on the proxy layer becomes the bottleneck, not your application servers.

An L4 load balancer sidesteps this: it picks a backend once, at connection time (usually via a hashing rule on source IP or a round-robin at connect time), and then just forwards packets. The TCP connection is between the client and the actual WebSocket server the whole time — the load balancer isn't in the data path at the application level at all. That's why AWS's Network Load Balancer (NLB), not the Application Load Balancer, is the documented choice for WebSocket-heavy workloads: it's built for exactly this — millions of long-lived TCP connections with minimal per-connection overhead.

The trade-off you're accepting: L4 load balancers can't route on path or header, and they can't retry a failed request the way an L7 proxy can, because they don't know what a "request" is. For a WebSocket service, that's the right trade — you don't want the load balancer inspecting frames anyway.

---

Putting It Together: A WebSocket Reference Architecture

Take a location-tracking feature — a rider watching a driver's pin move — and lay out the pieces:

Each piece maps directly back to what we've covered:

• User → L4 Load Balancer: one TCP connection, held open for the duration of the app session, distributed across WebSocket Service instances at connect time.
• L4 Load Balancer → WebSocket Service: the raw connection, unterminated, landing on whichever instance was picked. This instance now owns that rider's connection for as long as it stays open.
• WebSocket Service ↔ Driver Location Service: this is Step 2 from the very first diagram — a completely separate concern from the client connection, and the part that's genuinely hard to get right at scale.

Solving Step 2: Getting the Update to the Right Server

Here's the part that catches teams off guard. Say a driver's GPS ping lands on the Driver Location Service. It knows which rider needs the update, but it has no idea which of your fifty WebSocket Service instances is holding that rider's open connection — that instance was chosen essentially at random by the load balancer minutes ago.

The fix is a publish-subscribe layer sitting between the two services — Redis Pub/Sub, Kafka, or a managed equivalent:

1. Every WebSocket Service instance subscribes to a channel keyed by connection ID or user ID for each client it's currently holding.
2. When the Driver Location Service has an update, it publishes to that channel — it doesn't need to know or care which instance is listening.
3. Whichever WebSocket Service instance owns that connection receives the message and pushes it down the open socket.

This is what makes the WebSocket Service instances stateless from the point of view of routing updates, even though each one is holding stateful, long-lived connections. Skip this layer and you end up with the classic bug: "it works in dev with one server, and silently drops half the updates in production with three."

---

WebSockets vs. the Alternatives

WebSockets solve Step 1 elegantly, but they aren't free — every open connection consumes memory on the server and a slot in the load balancer's connection table, and reconnect logic (network drop, phone sleeps, app backgrounds) is entirely your responsibility to build.

Approach	Latency	Server cost at scale	When it's the right call
Short polling	Seconds (poll interval)	Low per-request, high in aggregate	Infrequent updates, simplicity matters more than latency
Long polling	Near real-time	Moderate — connection held only until data or timeout	Real-time-ish needs without WebSocket infra
Server-Sent Events (SSE)	Real-time, one-way	Similar to WebSockets, simpler protocol	Server-to-client only (feeds, notifications), no need for client-to-server on the same channel
WebSockets	Real-time, bidirectional	Highest — every connection is stateful and long-lived	Bidirectional, high-frequency updates (location, chat, collaborative editing)

If updates only flow one direction — a live score ticker, a notification feed — SSE gets you real-time delivery over a plain HTTP connection (still subject to the same L4/L7 trade-off, just simpler to reason about) without building a bidirectional protocol you don't need. Reach for full WebSockets when the client also needs to send frequent updates back, like a chat app or a collaborative editor.

---

Common Mistakes

Putting a stock L7 load balancer in front of a WebSocket service without checking connection limits. Most L7 proxies have a default max-connections-per-worker setting tuned for short-lived HTTP requests. A few thousand idle WebSocket connections can quietly hit that ceiling and start rejecting new users with no obvious error in your app logs.

Assuming "connected" means "reachable." A phone that's locked or backgrounded may hold a TCP connection open from the OS's perspective while the app can't actually process the message. Real systems need application-level heartbeats (ping/pong frames) on top of TCP's own keep-alive, because TCP alone won't tell you the client app is actually alive.

Forgetting Step 2 entirely. Teams design the client-facing WebSocket layer carefully and then wire the backend event source directly to "the WebSocket server" as if there's only one. There isn't, once you've scaled past a single instance — you need the pub/sub fan-out layer from day one, even if you only run two instances in staging.

Treating load balancer choice as an infrastructure afterthought. Whether you terminate WebSockets at L7 or pass them through at L4 changes your connection-scaling ceiling by an order of magnitude. It's an architecture decision, not a config toggle you set once and forget.

---

Final Takeaway

"Add WebSockets" is the easy part of building a real-time feature. The parts that actually determine whether it holds up in production are underneath it: understanding that a WebSocket is just a TCP connection you've agreed to keep open, choosing a load balancer that's built to hold millions of those connections instead of terminating and re-encoding each one, and building the pub/sub layer that lets any server instance figure out which open connection an event belongs to.

Get those three right, and the driver's pin just moves. Get any one of them wrong, and it works perfectly in every demo and falls over the day real traffic shows up.

Next in this system design series: Message Queues & Async Processing, where we'll go deeper into the Kafka/Redis Pub-Sub layer this post leaned on for Step 2 — consumer groups, ordering guarantees, and what happens when a consumer falls behind.

---

Subscribe to get system design and backend engineering posts in your inbox every two weeks.