Distributed Tracing & Observability: Finding the Slow Request Across Ten Services

It's 2:40 PM. Someone on the growth team pastes a message into the incident channel: "checkout is slow, customers are complaining." That's it. No stack trace, no error code, no service name. Just "slow."

You pull up the dashboard for the checkout-service and its P50 latency looks fine — 180ms. But P99 has crept up to 4.2 seconds over the last hour. Somewhere between the API gateway and the response, one in every hundred requests is taking twenty times longer than it should, and you have no idea which of the eleven services in the call path is responsible.

This is the moment where a pile of dashboards stops being useful and you need something that answers a different question. Not "is the system healthy," but "what happened to this specific request." That's what distributed tracing is for, and it's worth being precise about why the other tools in your observability stack can't answer that question on their own.

Three Pillars, Three Different Jobs

Metrics, logs, and traces get bundled together under "observability" so often that it's easy to think they're interchangeable views of the same data. They aren't. Each one is optimized for a different shape of question, and each one is close to useless for the questions the others are good at.

Metrics are aggregates over time — counters, gauges, histograms. Request rate, error rate, P99 latency, queue depth, CPU utilization. They're cheap to store because you're not keeping individual events, just rolled-up numbers in fixed-size time buckets. Metrics are what tell you something is wrong and roughly when it started. They're also what your alerting is built on, because you can define a threshold and page someone when it's crossed. What metrics cannot tell you is why. A P99 latency graph will show you the spike. It will never show you that the spike was caused by a specific tenant's query hitting an unindexed column on a specific replica.

Logs are structured or unstructured event records — one line (or one JSON blob) per thing that happened. They carry detail metrics don't: stack traces, request payloads, specific error messages, business context. Logs are where you go once you have a hypothesis and want to confirm it. The problem is that logs are scoped to a single process. order-service's logs know nothing about what payment-service was doing at the same moment, and grepping through a dozen log streams trying to line up timestamps by hand is exactly the kind of manual correlation that traces exist to eliminate.

Traces are the connective tissue. A trace represents one logical request as it moves through your system, broken into a tree of timed operations called spans. A trace doesn't replace metrics or logs — it tells you where in the system to go look, and gives you the request ID you need to pull the exact log lines that matter, out of the millions that don't.

Put together: metrics tell you something's wrong, traces tell you where, logs tell you why. Skip any one of the three and you're debugging with a blindfold on for part of the investigation. Most teams over-invest in metrics dashboards and under-invest in tracing, because metrics are easier to bolt on after the fact and traces require actual propagation discipline across every service boundary. That asymmetry is exactly why the "slow checkout" incident above is so common — the metrics told you that it's slow, and then everyone sits in a call guessing.

What a Span Actually Is

A span is a single timed unit of work with a name, a start time, a duration, a set of key-value attributes, and — this is the part that makes tracing "distributed" — a reference to its parent span. String enough spans together by parent/child relationships and you get a trace: a tree (usually closer to a mostly-linear chain with a few branches for fan-out calls) that represents the full lifecycle of one request.

Concretely, when a request hits your API gateway, a root span is created: trace_id=7a3f..., span_id=001, name POST /checkout. That trace ID is the thing that ties everything together — every span created anywhere in the system as part of handling this request will carry that same trace ID. When the gateway calls order-service, it creates a child span — span_id=002, parent_span_id=001 — and hands both the trace ID and its own span ID to order-service over the wire. order-service uses 002 as the parent for the spans it creates, and so on, all the way down through inventory-service, payment-service, and whatever database or cache calls happen along the way.

The mechanism for "handing both IDs over the wire" is trace context propagation, and it's the single most important piece of plumbing in the whole system — if it breaks anywhere, your trace doesn't just get a little sloppier, it silently splits into two disconnected traces and you lose the ability to see the full picture.

The modern standard for this is the W3C Trace Context spec, carried as an HTTP header:

traceparent: 00-7a3f2c1e9b4d4a5c8e6f1a2b3c4d5e6f-a1b2c3d4e5f60708-01

Broken down, that's version-traceId-parentSpanId-flags. Every hop reads this header, extracts the trace ID and parent span ID, creates its own child span with a new span ID, and — critically — re-injects an updated traceparent header into whatever outbound call it makes next (HTTP request, gRPC call, message published to a queue). If a service receives the header but forgets to propagate it on its own outbound calls, everything downstream of that service becomes an orphaned trace with no link back to the original request. This is the number one way tracing quietly stops working in real systems, and it's almost never caught until someone goes looking for a trace during an incident and finds it cuts off three hops in.

Here's what that propagation looks like end to end for a single checkout request:

style Gateway fill:#B45309,color:#fff,stroke:#92400E
style Order fill:#4F46E5,color:#fff,stroke:#3730A3
style Payment fill:#4F46E5,color:#fff,stroke:#3730A3
style Inventory fill:#0F766E,color:#fff,stroke:#115E59

Look at that timeline for a second, because it's the whole point of tracing in one picture. payment-service responded in 180ms — fast, not the problem. inventory-service took 3.4 seconds — that's almost the entire 4.1 second total. Without a trace, all you'd know is "checkout took 4.1 seconds." With the trace, you know within about ten seconds of looking at it that inventory-service's /reserve call is where the time went, and you can go straight to that service's logs, filtered by trace ID, to find out whether it was a lock contention issue, a slow query, or a downstream call of its own that's the real culprit one more level down.

Correlation IDs: The Poor Cousin of Full Tracing

Before OpenTelemetry and W3C trace context were standard, and still in plenty of systems today, teams solve a narrower version of this problem with a correlation ID — a single opaque string (often just a UUID) generated at the edge and passed through every service and logged with every line. It's not a full trace: there's no parent/child span structure, no timing breakdown, no notion of which hop is slow. It's purely a way to grep across a dozen services' log streams and pull every line that belongs to one logical request.

Correlation IDs are worth calling out specifically because they're a legitimate, low-effort stepping stone if you don't yet have real tracing wired up. If every log line already includes correlationId=abc-123, you can at least reconstruct the sequence of events for a request, even without timing detail. In practice, most teams that add a correlation ID eventually realize they want the timing and hierarchy information too, and that's the point where the correlation ID field just becomes the trace ID, and structured spans get added on top. If you're instrumenting a system from scratch, there's little reason to build correlation IDs as a separate concept — just adopt trace context propagation from day one and log the trace ID on every line, which gives you both capabilities from a single mechanism.

Sampling: You Cannot Afford Every Trace

Here's the uncomfortable part. If your system does 50,000 requests per second, and each request produces a trace with an average of 15 spans, and each span is a few hundred bytes once you include attributes — you're generating tens of gigabytes of trace data per hour. Store that at full fidelity, forever, and your tracing backend's storage bill will exceed the cost of the infrastructure it's monitoring. Nobody runs 100% sampling in a production system of any real size, and if you're being pitched a tracing setup that assumes you will, push back on the assumption before you push back on the vendor.

Head-based sampling makes the keep/discard decision at the start of the trace, typically at the root span, before anything is known about how the request will turn out. The simplest form is a flat percentage — sample 1% of all requests, tag the decision in the trace context flags, and every downstream service respects that decision (this is why the traceparent header above ends in 01 — that's the sampled flag). Head-based sampling is cheap and simple: no service needs to buffer anything, decisions propagate for free. The downside is obvious once you say it out loud — you're deciding whether a trace is interesting before you know if it was slow or errored. Statistically, at 1% sampling, you'll catch plenty of normal-case traces and miss the vast majority of the rare, expensive failures that you actually wanted to see.

Tail-based sampling flips this: every span is buffered (usually at a collector layer, not on the application itself) until the full trace completes, and then a decision is made — keep it if it errored, keep it if latency exceeded some threshold, keep it if it hit a specific service you're currently worried about, and otherwise sample the rest at a low background rate. This gets you the traces you actually care about — the slow ones, the failed ones — at the cost of running a buffering/aggregation layer that has to hold spans in memory until a trace is judged complete, which adds infrastructure and latency to the collection pipeline itself.

Most production setups worth their salt end up doing both: a modest head-based rate to keep steady-state cost predictable, layered with tail-based rules that guarantee anything erroring or exceeding a latency threshold gets kept regardless of the head-based coin flip. The decision of where to set that dial is not really an engineering decision in isolation — it's a budget decision. "We want to catch every P99+ outlier" and "we want tracing to cost less than 5% of our infra bill" are both legitimate goals that can conflict, and someone needs to own that trade-off explicitly rather than let it default to whatever the SDK ships with.

Where the Data Goes

Spans don't just evaporate into a dashboard. There's a pipeline: your application emits spans, something collects and possibly samples/batches them, something stores them in a form that's queryable by trace ID and by attribute, and something renders them as the waterfall diagrams you actually look at during an incident.

The collector in the middle is doing more work than it looks like from the diagram. It's a separate process (or sidecar, or DaemonSet in Kubernetes) that receives spans from every instrumented service over OTLP (the OpenTelemetry wire protocol), and can batch them, apply tail-based sampling rules, redact sensitive attributes before they ever leave your infrastructure, and fan the data out to one or more backends. Decoupling collection from storage this way means you can swap your trace backend — move from a self-hosted Jaeger cluster to a managed vendor, or run both in parallel during a migration — without touching a single line of application code, because the applications only ever talk to the collector.

On storage: Jaeger and Zipkin are the two long-standing open-source trace backends, both originally built at companies (Uber and Twitter respectively) solving exactly this problem at scale, and both still perfectly reasonable choices if you want to self-host. Grafana Tempo is a newer entrant designed to lean on cheap object storage rather than a dedicated database, trading some query flexibility for much lower storage cost. On the managed side, Datadog APM, Honeycomb, and similar vendors handle collection, storage, and the query UI as one product, which is usually the right trade for a team that doesn't want to operate trace storage infrastructure themselves. None of these choices matter much until you've committed to instrumenting properly — the backend is replaceable; badly propagated context is not.

Instrumentation: OpenTelemetry as the Default Answer

A few years ago, "how do I add tracing" meant picking a vendor's proprietary SDK and getting locked into their span format and their backend. OpenTelemetry (OTel) changed that by becoming the vendor-neutral standard for emitting traces, metrics, and logs, with an API/SDK split that lets you instrument your code once and point the output at whichever backend you want, via the collector described above. Unless you have a specific reason not to, OpenTelemetry is the default answer today, and most managed tracing vendors have converged on accepting OTLP as an ingestion format rather than requiring their own SDK.

There are two levels of instrumentation, and you want both, in layers.

Automatic instrumentation uses bytecode weaving or framework hooks to create spans for the "obvious" boundaries without you writing any span-creation code — incoming HTTP requests, outgoing HTTP client calls, JDBC queries, Kafka producer/consumer calls. For a Spring Boot service, adding the OpenTelemetry Java agent as a -javaagent flag gets you spans for every controller method, every RestTemplate/WebClient call, and every database query, with trace context propagation across all of it, without touching application code. This is the 80% case and it's genuinely good — it means every service in your system produces a baseline trace the day you turn the agent on, with zero code changes.

Manual spans are for the 20% that automatic instrumentation can't see into: a specific block of business logic you suspect is slow, a call to a third-party SDK that doesn't get auto-instrumented, or a case where you want to attach domain-specific attributes (a tenant ID, an order total, a cache hit/miss flag) to a span so you can filter and group traces by them later. The annotation-based approach keeps this lightweight:

@Service
public class InventoryReservationService {

    private static final Tracer TRACER =
            GlobalOpenTelemetry.getTracer("inventory-service");

    private final WarehouseClient warehouseClient;

    public InventoryReservationService(WarehouseClient warehouseClient) {
        this.warehouseClient = warehouseClient;
    }

    @WithSpan("inventory.reserve")
    public ReservationResult reserve(@SpanAttribute("order.id") String orderId,
                                      @SpanAttribute("sku") String sku,
                                      int quantity) {

        Span current = Span.current();
        current.setAttribute("inventory.quantity_requested", quantity);

        // This call hits a legacy warehouse system over a flaky link.
        // Wrap it in its own span so a slow/failing call is visible
        // independently of the rest of this method.
        Span warehouseSpan = TRACER.spanBuilder("warehouse.legacy_call")
                .setSpanKind(SpanKind.CLIENT)
                .startSpan();

        try (Scope scope = warehouseSpan.makeCurrent()) {
            warehouseSpan.setAttribute("warehouse.sku", sku);
            ReservationResult result = warehouseClient.reserve(sku, quantity);
            warehouseSpan.setAttribute("warehouse.reserved", result.isSuccess());
            return result;
        } catch (WarehouseTimeoutException ex) {
            warehouseSpan.recordException(ex);
            warehouseSpan.setStatus(StatusCode.ERROR, "warehouse call timed out");
            throw ex;
        } finally {
            warehouseSpan.end();
        }
    }
}

Two things worth noticing here. First, @WithSpan gives you a clean, declarative span around the whole method with almost no boilerplate — good for marking a meaningful unit of business logic without hand-managing scopes. Second, the manual TRACER.spanBuilder(...) block around the warehouse call is deliberate and more verbose, because that's the specific dependency we don't trust — wrapping it in its own span, with its own attributes and its own exception recording, means that when this call is slow or fails, it shows up as a distinctly named, separately timed segment in the trace, not blended into the surrounding method. That distinction — instrument what's automatic for free, hand-instrument what you specifically suspect — is the practical rule of thumb for where to spend manual effort.

Reading a Trace to Find the Actual Bottleneck

Once tracing is in place, the workflow during an incident looks less like guessing and more like triage. Start from the metric that told you something was wrong — P99 latency on checkout, say — and pull a sample of traces from exactly that time window, filtered to the slow tail (most backends let you query "traces where duration > 2s" directly). Don't look at one trace and generalize; pull five or ten and look for the pattern. If every slow trace bottlenecks in the same downstream span, you have a specific root cause. If the slow span is different each time, you're looking at contention or resource exhaustion rather than a single bad code path, and the fix is architectural, not a one-line patch.

The critical path of a trace is the sequence of spans that actually determines the total duration — not every span that took a long time, but the chain where reducing that span's duration would reduce the overall request time. This distinction matters because traces often have parallel branches: if order-service calls payment-service and inventory-service concurrently and waits for both, and payment-service takes 200ms while inventory-service takes 3.4 seconds, only the inventory branch is on the critical path — shaving time off payment-service does nothing for the total, because the request was already waiting on inventory regardless. Good trace visualizations (Jaeger's waterfall view, for instance) make this visually obvious: the critical path is whatever spans align with the outer edge of the timeline, and everything that finishes before that edge, however slow it looks in isolation, isn't what you need to fix first.

P95 and P99 latency deserve a specific mention here because they're where tracing earns its keep versus where metrics alone mislead you. Averages hide outliers by definition. A service with a 50ms average and a 4-second P99 has a real problem affecting 1% of traffic — at 50,000 requests per second that's 500 requests per second having a bad time, which is not a rounding error, it's an ongoing incident that your average latency graph is actively hiding from you. Tracing is what turns "P99 is bad" from an alert into an actionable root cause, because you can pull the actual P99 traces and see exactly which span is responsible, rather than staring at an aggregate number wondering where to even start looking.

The other habit worth building is comparing a slow trace against a known-good baseline trace from the same endpoint, rather than reading the slow one in isolation. Pull a representative fast trace for POST /checkout from an hour before the incident started, and lay it next to the slow ones you pulled from the bad window. Almost always, most of the tree is identical — same number of spans, same rough durations for payment-service, same shape for the gateway overhead. The difference is usually confined to one or two spans that either ballooned in duration or, just as often, appeared where they didn't exist before — an extra retry span, an extra cache-miss-then-database-fallback pair, a lock-wait span that only shows up under contention. Finding that delta is faster and less error-prone than trying to eyeball "is 3.4 seconds normal for this span" from memory, especially for spans you don't look at every day. If your trace backend supports saving or bookmarking a few baseline traces per critical endpoint, it's worth doing before you need them, not during the incident when everyone's already asking for an ETA.

Common Mistakes

Breaking context propagation at async boundaries. Trace context flows automatically through synchronous HTTP/gRPC calls when you're using an auto-instrumentation agent, but the moment a request crosses into a thread pool, a CompletableFuture, a Kafka producer, or a scheduled job, the context has to be explicitly captured and re-attached — it doesn't just follow the data by magic. Forget this, and every async hop silently starts a brand-new trace with no parent, which is exactly the kind of gap that shows up only when you go looking for a trace during an incident and it just... stops.

Sampling everything, then being surprised by the bill. Turning on 100% sampling in a staging environment feels harmless and is a completely reasonable way to validate instrumentation. Leaving it at 100% in production because nobody revisited the config after launch is how a tracing backend line item quietly becomes a meaningful fraction of your infra spend. Set the sampling rate deliberately, as a budget decision, not a default you forgot to change.

Treating traces as a replacement for logs instead of a companion to them. A span tells you which operation was slow. It rarely carries enough detail to tell you why — that's what a log line with a stack trace or a specific error message gives you. The move that actually pays off is putting the trace ID on every log line, so that once a trace points you at a suspect span, you can pivot straight to the exact log lines for that request without re-deriving a timestamp window and grepping blind.

Instrumenting the app but not the infrastructure in between. A trace that jumps from your service straight to a database with no visibility into the connection pool, or that shows a gap of unexplained time at a message broker, is a trace with a blind spot exactly where infrastructure-level issues (pool exhaustion, broker backpressure) tend to live. Auto-instrumentation for common clients (JDBC, HTTP clients, Kafka clients) closes most of this gap for free — it's worth confirming it's actually enabled rather than assuming the agent caught everything.

No naming or attribute conventions, so nothing is queryable later. A trace full of spans named process or handle with no consistent attributes is technically instrumented and practically useless, because you can't filter or group by anything meaningful when you're trying to find the pattern across ten slow traces. Agreeing on a small set of standard span names and attributes (operation name, tenant/customer ID, resource identifiers) across teams up front saves the much larger cost of redoing instrumentation once someone actually tries to use it during an incident and finds the data is there but unqueryable.

Final Takeaway

Distributed tracing isn't a nice-to-have dashboard feature — it's the tool that turns "the API feels slow" into "the warehouse.legacy_call span in inventory-service is timing out for 1% of requests, and it's the only thing on the critical path." Everything else in this post — context propagation, sampling strategy, OpenTelemetry, critical path analysis — is in service of getting to that sentence faster, with less guessing and fewer people staring at disconnected dashboards during a live incident. Get the propagation right first, decide on sampling deliberately rather than by default, and instrument the 20% of your code that automatic tooling can't see — the rest is mostly about learning to read the waterfall diagram in front of you.

Next in this system design series: Message Queues & Async Processing, where the request you just traced end-to-end stops being a single request at all, and becomes an asynchronous event with its own delivery guarantees.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.