Production Observability with OpenTelemetry, Prometheus, Grafana, and Logs

Two years ago we had an incident that should have taken ten minutes and took two hours. Checkout latency spiked, the on-call engineer got paged, and they had — on paper — everything: distributed traces, a metrics dashboard, centralized logs. What they didn't have was any way to connect the three. The trace showed a slow span in the payment service. The dashboard showed p99 latency climbing. The logs showed a wall of WARN lines with no request ID, no trace ID, nothing to grep on. Three separate systems, three separate stories, and no way to prove which log lines belonged to which slow request.

We fixed the actual bug in twenty minutes once we found it. We spent the other hour and forty minutes trying to find it, because our "observability stack" was really just three monitoring tools bolted together with no shared vocabulary. This post is about the plumbing that makes that hour and forty minutes disappear — instrumenting a service with OpenTelemetry, exposing metrics Prometheus can actually alert on, building a Grafana dashboard someone can read half-asleep, and writing logs that tie back to the trace and the metric spike that got someone paged in the first place.

We already wrote about distributed tracing theory elsewhere — spans, context propagation, sampling strategy. This one skips the theory and goes straight to the parts that break in production: what auto-instrumentation actually gives you, which metrics deserve an alert, and how to keep the pager from becoming background noise.

The pipeline, end to end

Before touching code, it helps to have the whole path in your head, because every outage post-mortem eventually points back to one hop in this chain being missing, mislabeled, or dropped.

Every box in that diagram is a place where correlation can silently break. If the collector strips the trace ID before it reaches your log backend, you've lost the thread. If Prometheus scrapes on a different label schema than your dashboard expects, panels go blank exactly when you need them. We treat this pipeline as a single system to test, not four independent tools that happen to be adjacent.

Instrumenting the service: auto vs. manual

OpenTelemetry's Java agent gets you further than most teams expect on day one. Attach the javaagent, point OTEL_EXPORTER_OTLP_ENDPOINT at your collector, and you get traces for incoming HTTP requests, outbound HTTP clients, JDBC calls, and messaging consumers, all with context propagation handled for you. For a Spring Boot service, that's most of the trace tree without writing a single line of instrumentation code.

# Passed as JAVA_TOOL_OPTIONS or via the -javaagent flag
otel:
  service:
    name: checkout-service
  exporter:
    otlp:
      endpoint: http://otel-collector:4317
      protocol: grpc
  traces:
    sampler: parentbased_traceidratio
    sampler_arg: "0.1"
  metrics:
    exporter: otlp
  logs:
    exporter: otlp

Auto-instrumentation is where you stop, though, the moment you need a metric that describes your business, not your framework. The agent will tell you HTTP request duration for free. It will not tell you "how long did the call to the fraud-scoring vendor actually take, broken down by whether we hit cache or not." That's a metric you have to write, because only you know it matters.

There's also a cost side to auto-instrumentation that teams discover the hard way, usually during a load test right before launch. The javaagent hooks into class loading and adds overhead to every instrumented call — small per call, but it adds up when you're instrumenting every JDBC statement in a service doing tens of thousands of queries a second. We've had to tune the sampler ratio down and disable instrumentation for a couple of extremely hot, extremely low-value internal calls (health checks, readiness probes) specifically because the trace volume was costing more in collector CPU than it was giving back in debugging value. Instrumentation isn't free, and "trace everything at 100%" is a decision you'll regret in either your APM bill or your collector's memory graph within a month of going to production traffic.

The other thing auto-instrumentation won't do is tell you when a span is technically fast but functionally wrong — a cache lookup that returns in two milliseconds because it always misses, say. That's a case where a manual counter (cache_hit_total vs cache_miss_total, tagged by cache name) tells you something the trace never will, because the trace only ever shows you latency, not correctness of behavior.

This is where Micrometer earns its place alongside OpenTelemetry — it's the metrics facade Spring Boot already speaks, and it exports cleanly to a Prometheus registry. We wrap risky external calls with a Timer explicitly, rather than trusting an aggregate HTTP client metric to surface the one dependency that actually causes outages.

@Component
public class FraudScoringClient {

    private final RestClient restClient;
    private final MeterRegistry meterRegistry;

    public FraudScoringClient(RestClient restClient, MeterRegistry meterRegistry) {
        this.restClient = restClient;
        this.meterRegistry = meterRegistry;
    }

    public FraudScore score(Order order) {
        Timer.Sample sample = Timer.start(meterRegistry);
        String outcome = "success";

        try {
            FraudScore result = restClient.post()
                    .uri("/v1/score")
                    .body(order)
                    .retrieve()
                    .body(FraudScore.class);
            return result;
        } catch (RestClientResponseException ex) {
            outcome = "client_error";
            throw ex;
        } catch (ResourceAccessException ex) {
            outcome = "timeout";
            throw ex;
        } finally {
            sample.stop(Timer.builder("fraud_scoring_call_duration_seconds")
                    .description("Latency of calls to the external fraud scoring vendor")
                    .tag("outcome", outcome)
                    .publishPercentileHistogram()
                    .register(meterRegistry));
        }
    }
}

The tag on outcome is doing real work here. Without it, "fraud scoring call duration" is a single blended number that hides the fact that timeouts (which take the full client-configured wait before failing) are dragging the p99 up while successful calls are fast and boring. Every custom metric we add gets at least one label like this — something that lets us slice the aggregate into "the thing that's actually breaking" versus "the thing that's fine."

The trap on the other side is tagging too much. A label with unbounded cardinality — user ID, order ID, raw URL path with path params unsubstituted — will quietly turn your Prometheus instance into a memory leak. We keep labels to values with cardinality in the dozens, not the thousands: outcome, region, tier, HTTP method. Never anything with a UUID in it.

We also learned to be deliberate about histogram bucket boundaries rather than accepting Micrometer's defaults everywhere. publishPercentileHistogram() gives you client-side percentile calculation across whatever bucket boundaries are configured, and the default boundaries are tuned for a generic web request, not necessarily for a call to an external vendor that's contractually supposed to respond in under 300ms. If your buckets are [1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s] and your actual SLA boundary is 300ms, you get a coarse, misleading picture right at the threshold that matters most. We set explicit SLO-aligned buckets (.serviceLevelObjectives(Duration.ofMillis(300), Duration.ofSeconds(1))) on any timer that backs an alert, so the histogram has real resolution exactly where the alert threshold sits.

RED for the app, USE for the box underneath it

Two mnemonics cover almost every metric worth collecting, and they're not interchangeable — using the wrong one for the layer you're looking at is how you end up with dashboards full of numbers nobody can act on.

RED — Rate, Errors, Duration — is for anything that serves requests: HTTP endpoints, gRPC methods, message consumers, the fraud-scoring call above. For every one of these, we want:

• Rate: requests per second, so we can tell "we're getting throttled" apart from "traffic just dropped because it's 3am"
• Errors: the fraction of requests failing, broken down by error type where possible
• Duration: latency distribution — not the average, the percentiles, because the average hides exactly the slow tail that pages people

USE — Utilization, Saturation, Errors — is for resources: CPU, memory, disk, connection pools, thread pools, queue depth. For the JVM heap, for instance:

• Utilization: how much of the heap is in use right now
• Saturation: how much GC pressure exists — pause frequency and duration, not just occupancy
• Errors: OutOfMemoryError counts, thread pool rejection counts

The reason this split matters in practice: RED metrics tell you the user is having a bad time. USE metrics tell you why. When checkout latency (a RED symptom) climbs, the first thing we look at isn't more application metrics — it's whether the database connection pool (a USE metric) is saturated. We alert on RED metrics almost exclusively and use USE metrics for diagnosis, because paging a human for "CPU is at 85%" without knowing whether that's actually hurting anyone is exactly the kind of alert that trains people to ignore the pager.

There's a nuance worth calling out for anyone mapping this onto a service with background work rather than pure request/response — batch jobs, queue consumers, scheduled reconciliation tasks. RED doesn't map cleanly onto "process everything currently in the queue," because there's no single request to time. For those, we lean on a close cousin: throughput (items processed per second), lag (how far behind the consumer is relative to what's been produced), and error rate on individual item processing. Consumer lag in particular is the queue-based equivalent of latency — it's the metric that tells you the user-facing effect (a delayed notification, a stale read replica, a slow webhook) before anyone notices from the outside. Treat it with the same seriousness as p99 latency on a synchronous endpoint, because from the user's perspective, a message sitting in a queue for ten minutes is functionally identical to a slow HTTP response.

We've also found it worth explicitly separating "infrastructure that's shared across services" — the database, the message broker, the Kubernetes node pool — into its own USE-based dashboard and alert set, owned by whoever runs that shared infrastructure, distinct from the application-level RED dashboard each service team owns. Mixing the two into one undifferentiated wall of metrics is how you end up with an application on-call engineer staring at a disk I/O saturation graph for a database they don't have write access to, wasting the first ten minutes of an incident figuring out who to even escalate to.

Alerting on symptoms, not causes

The single biggest lever for alert quality is this rule: alert on what the user experiences, not on what you think caused it. "Error rate above 5% for five minutes" is a symptom — it's true regardless of whether the cause is a bad deploy, a downstream outage, a full disk, or a network partition, and it demands a human look regardless of cause. "CPU above 80%" is a guess about a cause, and it's frequently wrong — a service can run at 90% CPU for hours doing exactly what it's supposed to do under load, with zero user impact.

groups:
  - name: checkout-service.rules
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{
            application="checkout-service", status=~"5.."
          }[5m]))
          /
          sum(rate(http_server_requests_seconds_count{
            application="checkout-service"
          }[5m]))
          > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Checkout error rate above 5%"
          description: >
            Error rate has been above 5% for 5 minutes on checkout-service.
            Check the fraud-scoring dependency and recent deploys first.
          runbook: "https://runbooks.internal/checkout-error-rate"

      - alert: CheckoutHighLatencyP99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_server_requests_seconds_bucket{
              application="checkout-service", uri="/api/checkout"
            }[5m])) by (le)
          ) > 1.5
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Checkout p99 latency above 1.5s"
          description: >
            p99 latency for /api/checkout has exceeded 1.5s for 10 minutes.
          runbook: "https://runbooks.internal/checkout-latency"

Three things in that file are load-bearing, and they're the parts teams skip when they're in a hurry to "add monitoring."

The for: 10m on the latency alert is not laziness — it's a decision that a two-minute latency blip during a deploy rollout doesn't deserve a page, but ten sustained minutes does. Get this window wrong in either direction and you either miss a real incident or wake someone up for a deploy that was already finishing its rollout. We tune the window against our actual deploy cadence and traffic pattern, not a default copied from a blog post.

The runbook annotation is the difference between an alert and a pop quiz. If the page doesn't come with "here's where to look first," the on-call engineer is starting from zero at 3am, which is exactly how our two-hour incident happened. Every alert we ship gets a runbook link before it goes live, not after the first time someone gets paged confused.

And the severity: page label matters because not everything that's worth watching is worth waking someone up for. We route on that label downstream in Alertmanager: severity: page goes to PagerDuty and wakes a human at 3am, severity: ticket opens a low-priority issue for the next business day, and anything below that is just a Grafana annotation nobody gets notified about at all. Getting a new alert's severity wrong in either direction has a cost — mislabel a real symptom as ticket and it sits unread through an actual outage; mislabel routine noise as page and you've spent one more chip out of the team's tolerance for being woken up. We review severity assignment in the same PR review as the alert rule itself, not as an afterthought.

One pattern that's paid for itself repeatedly: a multi-window burn-rate alert instead of a single flat threshold. A flat "error rate above 5% for 5 minutes" alert has to compromise between catching fast, severe spikes and not flapping on brief blips, and any single window is a compromise in one direction or the other. Running two conditions in parallel — a short window (5 minutes) at a high threshold to catch a severe, fast-moving outage, and a longer window (1 hour) at a lower threshold to catch a slow, sustained degradation that never spikes hard enough to trip the short window — catches both failure shapes without needing either threshold to be a guess that covers everything. It's more rule to write, but it's the difference between finding out about a slow leak from the alert versus finding out about it from a customer email three days later.

This is where error budgets change the alerting conversation entirely.

SLOs, SLIs, and the error budget as a shared contract

An SLI (service level indicator) is a number you can measure — "percentage of checkout requests completing in under 500ms," say. An SLO (service level objective) is the target the team agreed on for that number — "99.9% of checkout requests over a rolling 30 days." The error budget is just the arithmetic on the other side of that target: if the SLO is 99.9%, you're allowed 0.1% of requests to fail or be slow before you've broken the promise. Over 30 days of, say, 10 million requests, that's 10,000 requests you can afford to lose.

The reason this matters more than it sounds like it should: without an error budget, every "can we ship this risky change" conversation is a vibe check. With one, it's a number. If we've burned 90% of this month's error budget on an infrastructure migration, the answer to "can we also ship the untested payment retry logic this week" is no, not because anyone's being cautious, but because the team already agreed on the number that governs the decision. If the budget is barely touched, we have room to take a real risk on purpose.

We track SLO burn rate as its own alert, separate from the symptom alerts above — a fast burn (spending a week's budget in an hour) pages immediately; a slow burn (on track to exhaust the month's budget three days early) can wait for business hours. That distinction alone cuts a meaningful chunk of false urgency out of the pager.

The other place the error budget earns its keep is in the retro after an incident. Without a budget, "was this incident bad enough to warrant a blameless post-mortem and a week of remediation work" is a judgment call that tends to correlate with how loud the loudest stakeholder in the room was. With a budget, it's arithmetic: this incident burned 40% of the quarter's allowance in ninety minutes, which is objectively a big deal regardless of whether the CEO happened to be checking out during the outage. It also protects the team in the other direction — a minor, well-contained blip that burned 0.5% of the budget doesn't need the same ceremony as a major outage, and having the number on hand keeps a team from over-processing small incidents just because someone got paged.

We've found it worth setting the SLO itself somewhere below what the infrastructure can theoretically achieve, deliberately. If your actual measured reliability over the last quarter is 99.97% and you set the SLO at 99.99%, you've built a target the team will chronically appear to be failing, which erodes the entire point of having a shared number everyone trusts. We set SLOs a notch below sustained historical performance and tighten them over time as we earn confidence, not the other way around.

A dashboard for 3am, not a wall of panels

We've inherited dashboards with forty panels, six colors per panel, and no indication of which one to look at first. That's not a dashboard, it's an archive. A dashboard built for an on-call engineer who just got paged and has ninety seconds before their brain fully boots up needs to answer one question first: is this still broken, and how badly?

Our service-health dashboard has, in order, top to bottom:

1. Big number row: current error rate, current p99 latency, current request rate — three numbers, red/yellow/green against SLO thresholds, no interpretation required.
2. Time series for the RED metrics, each with the SLO threshold drawn as a horizontal line, so "we're above the line" is a glance, not a calculation.
3. Dependency health: one row per external call the service makes (database, cache, fraud scoring, payment gateway), same red/yellow/green treatment, because most incidents in a service like this are actually a downstream problem.
4. A link to logs, pre-filtered to the current time window and this service, and a link to the trace explorer, same filter. Not embedded — linked, because trying to cram a full log viewer into a dashboard panel is how you get the forty-panel wall in the first place.

Everything past that — JVM internals, thread pool depth, GC pause histograms — lives on a second, "diagnostics" dashboard that the on-call engineer clicks into only after the first one tells them something's actually wrong. Two dashboards, not one dashboard trying to be both a status page and a debugger.

A few smaller conventions have made a bigger difference than we expected. Every panel on the health dashboard uses the same time range and the same refresh interval — Grafana's per-panel override is convenient to build with and miserable to read from, because at 3am the last thing anyone needs is to realize panel three is showing the last hour while panel five is showing the last day. We also fixed the y-axis scale on the latency panels rather than letting Grafana auto-scale; auto-scaling makes a 200ms blip look exactly as dramatic as a 2-second outage, because both fill the panel edge to edge, and that visual lie costs precious seconds of triage while someone figures out whether the graph they're looking at is actually bad or just zoomed in.

We also annotate deploys directly on the dashboard — a vertical marker line fed from the CI/CD pipeline every time a new version rolls out. The single most common root cause of a production incident is a recent deploy, and burying that fact in a separate deployment tool means the on-call engineer has to remember to go check it. Putting it on the same time axis as the latency spike turns "did we just deploy something" from a question into a glance.

Structured logs and the correlation ID

None of the above matters if, once the dashboard says "latency is up" and the trace says "this span is slow," you can't find the log lines for that specific request. The fix is almost embarrassingly simple and almost universally skipped under deadline pressure: every log line carries the trace ID.

With the OpenTelemetry Java agent active, the trace ID and span ID are already in MDC (MappedDiagnosticContext) for the current thread. The only job left is to make sure your log encoder actually emits them, and that your logs are structured JSON rather than freeform text, so a log backend can index on trace_id as a real field instead of hoping a regex finds it in a message string.

<encoder class="net.logstash.logback.encoder.LogstashEncoder">
  <includeMdcKeyName>trace_id</includeMdcKeyName>
  <includeMdcKeyName>span_id</includeMdcKeyName>
  <customFields>{"service":"checkout-service"}</customFields>
</encoder>

Once that's in place, the workflow that used to take two hours takes two minutes: dashboard shows the spike, click through to the trace explorer filtered to that time window, find the slow trace, copy its trace ID, paste it into the log search, and now you have every log line — across every service the request touched — for that exact request, in order. That's the whole point of the pipeline diagram at the top of this post. Metrics tell you something's wrong. Traces tell you where. Logs tell you why. None of the three work without the trace ID gluing them together.

Structured JSON logging also changes what "searching logs" means in a way that's easy to undersell. Freeform text logs mean every investigation starts with someone guessing a regex and iterating. Structured fields mean the question "show me every 5xx response for this customer's account in the last hour, broken down by endpoint" is a query, not an archaeology project. We standardized on a small, consistent field set across every service — trace_id, span_id, service, level, http.method, http.status_code, and a user_id field that's hashed rather than raw, so we get correlation without turning the log backend into a second copy of PII we now have to protect. The hashing matters more than it sounds like it should — the first version of this scheme logged raw user IDs, and it took a security review flagging it to notice we'd built a second, less-audited datastore of personal data purely as a side effect of making debugging easier.

One more detail that's saved us real time: logging the request body (or a redacted summary of it) at DEBUG level only, gated behind a header we can flip per-request in production without a redeploy. Most of the time that log line is silent and costs nothing. When we're chasing a specific customer's specific failure, we flip the header for their traffic only, get exactly the payload that triggered the bug, and flip it back off. It's a middle ground between "never log request bodies" (safe, but useless for the one weird edge case bug you can't repro) and "always log request bodies" (an instant compliance problem and a logging cost line item nobody signed off on).

Common mistakes

Alerting on a metric with no attached action. If an alert fires and the response is "huh, interesting," it shouldn't be an alert — it should be a dashboard panel. Every alert we keep has to answer "what does the on-call engineer do differently in the next five minutes because this fired," and if nobody can answer that in the design review, we delete the alert rather than let it degrade into noise.

Dashboards nobody opens until an incident. A dashboard that only gets looked at during an outage is a dashboard nobody trusts, because nobody's built the muscle memory of what "normal" looks like on it. We make the service-health dashboard part of the daily deploy checklist — glance at it after every deploy — specifically so that during an incident, the on-call engineer already knows what the graphs look like on a good day.

No shared request ID between logs, metrics, and traces. This was our two-hour incident. Metrics said "slow." Traces said "slow here." Logs said nothing anyone could tie to either, because the log lines had no trace ID. Fixing the encoder configuration above was a half-day of work that would have turned that incident into a fifteen-minute one.

Unbounded label cardinality on custom metrics. Tagging a Prometheus metric with user ID, session ID, or a raw untemplated URL path (/orders/38291 instead of /orders/{id}) creates a new time series per unique value. We've seen this alone take down a Prometheus instance by exhausting memory on time series churn, taking every dashboard and alert with it — the outage caused by the observability tooling itself.

Setting for: durations by copying a template instead of your own traffic pattern. A for: 1m window on a low-traffic overnight service will flap on normal statistical noise. A for: 15m window on a high-traffic checkout path during a real outage means fifteen minutes of user pain before anyone gets paged. The right duration depends on your request volume and your deploy cadence, not on what a blog post (including this one) used as an example.

Final takeaway

The tools here — OpenTelemetry, Prometheus, Grafana, a log backend — are table stakes at this point; most teams have some version of all four already running. What separates a team that resolves incidents in minutes from one that resolves them in hours isn't the tool list, it's whether those four systems agree on a shared vocabulary: the same trace ID threading through logs and traces, alerts that fire on what users actually feel instead of a guess at the cause, a dashboard built for a half-awake human instead of a demo to management, and an error budget that turns "can we take this risk" into a number instead of an argument. Build the plumbing once, and the next 3am page is the twenty-minute fix instead of the two-hour scavenger hunt.

Next in this DevOps series: Incident Response & Disaster Recovery, where the alert this post just designed turns into an actual incident someone has to run.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.