Message Queues & Async Processing: Kafka, RabbitMQ, and Event Streaming

A payment succeeds. The order service needs to update inventory, send a confirmation email, notify the warehouse, and kick off fraud scoring. If all four of those happen inline in the request path and the warehouse system is having a bad day, the customer's checkout hangs for eight seconds and then times out — even though their money was already charged.

We've shipped this exact bug. The fix wasn't a faster warehouse API. It was accepting that "order placed" and "warehouse notified" don't need to happen in the same breath, put a message queue between them, and let each side move at its own pace. That's the entire pitch for async processing in one sentence. Everything else in this post is about the ways that pitch goes wrong in production and what to do about it.

The three delivery guarantees, and where each one lies to you

Every message queue vendor advertises a delivery guarantee. There are three, and only one of them is honest about being incomplete.

At-most-once: the message is sent, and if it's lost in transit or the consumer crashes before processing it, nobody retries. This is what you get with fire-and-forget UDP-style delivery, or a Kafka producer configured with acks=0. Nothing re-sends. It's fast and it's simple and it silently drops data under any kind of failure. Use it for metrics you can afford to lose — a page-view counter, a heartbeat — never for anything with money or state attached.

At-least-once: the sender keeps retrying until it gets an acknowledgment, which means a message can be delivered more than once — the ack itself can get lost even though the message was processed fine, so the sender retries a message the consumer already handled. This is the default posture of Kafka with acks=all and enabled retries, and of RabbitMQ with manual acknowledgments and requeue-on-nack. It's the guarantee almost everyone actually runs in production, and it means duplicates are not an edge case — they are an expected, routine occurrence that your consumer must handle correctly every single time.

Exactly-once: the message is delivered and processed exactly one time, no drops, no duplicates. This is the one people want and the one that's hardest to actually get. Kafka has "exactly-once semantics" (EOS) for Kafka-to-Kafka pipelines using idempotent producers and transactional writes across the log — and it's real, but it's scoped to the Kafka cluster boundary. The moment your consumer does something outside Kafka — write to Postgres, call a payment API, send an email — that transactional guarantee doesn't extend there. The instant a message triggers a side effect in a different system, you are back to at-least-once with an idempotency check that you have to build yourself. Anyone who tells you their queue gives you end-to-end exactly-once across two different systems is either wrong or hasn't been paged yet.

The practical takeaway: design every consumer assuming at-least-once delivery, and get "effectively exactly-once" through idempotency at the consumer, not through the broker's marketing copy.

Pub-sub vs. point-to-point: two different jobs

These get conflated constantly, so it's worth being precise.

Point-to-point (queue) is a work-distribution pattern. One producer puts a task on a queue; exactly one consumer (out of possibly many competing consumers) picks it up and processes it. Think "resize this uploaded image" — you want the work done once, by whichever worker is free. Classic RabbitMQ queue semantics, or an SQS standard queue, model this directly.

Publish-subscribe is a broadcast pattern. One event goes out, and every interested subscriber gets its own copy. "Order placed" isn't a task for one worker — it's a fact that the inventory service, the email service, the analytics pipeline, and the fraud service all independently care about, each doing something different with the same event.

Kafka blurs this distinction in a useful way: a topic is inherently pub-sub (every consumer group gets every message), but within a single consumer group, partitions are distributed across group members so that the group as a whole behaves like a point-to-point work queue. That dual nature is exactly why Kafka replaced a lot of RabbitMQ-plus-a-fanout-exchange setups — you get broadcast-to-many-groups and load-balance-within-a-group from the same primitive.

The design question to ask before picking either: is this event a fact that multiple independent systems need to react to (pub-sub), or is it a unit of work that needs to be done exactly once by exactly one worker (point-to-point)? Get this wrong and you end up either processing an order four times or having no fraud pipeline because it was competing with inventory for the same queue.

Kafka internals, at the depth you actually need

Kafka's mental model is a distributed, append-only log, not a queue. That difference explains almost every behavior that surprises people coming from RabbitMQ or SQS.

A topic is a logical stream of events — "orders", "payment-events". It's split into partitions, and a partition is the actual unit of storage and parallelism: an ordered, immutable sequence of records, each identified by a monotonically increasing offset. Producers write to a partition; consumers read from a partition starting at some offset and simply advance through it. Nothing is deleted on read — Kafka retains messages for a configured retention period (or forever, for compacted topics) regardless of whether anyone has consumed them. That's the single biggest conceptual shift: consumers don't remove messages from the broker, they just track their own position in the log.

Each partition is replicated across multiple brokers — typically a replication factor of 3 — with one broker serving as leader for that partition and the others as in-sync replicas (ISRs) that mirror it. Producers and consumers only talk to the leader; if the leader's broker dies, one of the ISRs is elected the new leader. acks=all means the producer waits for the message to be written to all in-sync replicas, not just the leader, before considering the write successful — that's the setting that actually protects you from data loss on a broker failure, and it's non-negotiable for anything you can't afford to lose.

A consumer group is how Kafka does load-balanced consumption: every consumer in the group is assigned a disjoint subset of the topic's partitions, and Kafka rebalances that assignment when consumers join or leave the group. This is why partition count is your hard ceiling on parallelism — five partitions means at most five consumers in a group can be doing work simultaneously; a sixth just sits idle. It's also why you pick your partition count for headroom up front: you can increase partitions later, but doing so reshuffles key-to-partition mapping and breaks any ordering guarantee you were relying on for existing keys.

Notice consumer 1 owns two partitions and consumers 2 and 3 own one each — that's a normal, healthy assignment when you have more partitions than consumers. Add a fourth consumer to this group and Kafka rebalances so each one owns exactly one partition. Add a fifth, and it sits idle until you either add partitions or accept the lower parallelism.

Offsets are the other piece worth being deliberate about. Kafka commits consumer offsets to an internal __consumer_offsets topic, and the timing of that commit relative to your actual processing is the whole ballgame for delivery semantics. Auto-commit on a timer is convenient and dangerous: if the consumer crashes between committing the offset and finishing the side effect, you lose the message (looks like at-most-once). If it crashes between finishing the side effect and committing the offset, you reprocess it on restart (at-least-once). Manual commit, done after the side effect completes, gives you the at-least-once behavior deliberately instead of by accident — which is the one you want, paired with an idempotent consumer.

RabbitMQ's model: exchanges, bindings, queues

RabbitMQ is an AMQP broker, and its model is closer to what you'd draw on a whiteboard for "message routing": a producer never writes directly to a queue. It publishes to an exchange, which routes the message to zero or more queues based on bindings — rules that say "messages matching this routing key/pattern go to that queue." A direct exchange routes by exact routing-key match; a topic exchange routes by wildcard pattern (order.*.created); a fanout exchange ignores the routing key entirely and broadcasts to every bound queue, which is RabbitMQ's version of pub-sub.

Once a message lands in a queue, RabbitMQ deletes it after a consumer acknowledges successful processing — that's the fundamental contrast with Kafka. RabbitMQ queues are transient work lists; Kafka partitions are a durable, replayable log. If you need "replay the last three days of events for a new consumer that just came online," Kafka gives you that for free because the data is still sitting in the log. RabbitMQ can't — once it's acked and gone, it's gone, unless you've separately archived it.

That makes the choice mostly about access pattern, not raw performance:

• Need multiple independent teams to consume the same event stream at different paces, replay history, or build stream-processing pipelines (joins, aggregations) → Kafka.
• Need flexible routing logic (route by content, priority queues, complex topologies), lower operational overhead for a single team's task queue, or per-message TTL and priority → RabbitMQ.
• Need simple, fully managed, "just a queue," minimal ops → SQS, which behaves like a simpler point-to-point RabbitMQ queue with less configurability but much less to run.

We've used Kafka as the backbone for an order-processing event bus (many consumers, need for replay, need for a durable audit trail) and RabbitMQ for internal task queues within a single service (image processing jobs, report generation) where nobody downstream needed the history. Neither is strictly "better" — they solve different shaped problems.

Idempotent consumers: how you actually get exactly-once-ish behavior

Given that at-least-once is the honest default, the consumer has to be safe to run twice on the same message. There are two common approaches, and the right one depends on whether the operation is naturally idempotent.

Naturally idempotent operations need no extra work: UPDATE orders SET status = 'shipped' WHERE id = ? produces the same end state whether it runs once or five times. Design your event handlers to be "set to this state" rather than "increment by this amount" wherever the domain allows it — it eliminates an entire category of duplicate-processing bugs for free.

Operations with side effects that aren't naturally idempotent — charging a card, sending an email, decrementing inventory — need explicit deduplication. The standard pattern is a dedup table keyed on a message ID (or better, a business-level idempotency key like the order ID plus event type), checked and inserted in the same transaction as the side effect:

@Service
public class OrderEventConsumer {

    private final OrderRepository orderRepository;
    private final ProcessedEventRepository processedEventRepository;
    private final InventoryClient inventoryClient;

    public OrderEventConsumer(OrderRepository orderRepository,
                               ProcessedEventRepository processedEventRepository,
                               InventoryClient inventoryClient) {
        this.orderRepository = orderRepository;
        this.processedEventRepository = processedEventRepository;
        this.inventoryClient = inventoryClient;
    }

    @KafkaListener(topics = "order-events", groupId = "inventory-service")
    @Transactional
    public void onOrderPlaced(ConsumerRecord<String, OrderPlacedEvent> record,
                               Acknowledgment ack) {
        String eventId = record.key(); // e.g. orderId + ":ORDER_PLACED"

        if (processedEventRepository.existsById(eventId)) {
            // Duplicate delivery — already handled, just move the offset forward.
            ack.acknowledge();
            return;
        }

        OrderPlacedEvent event = record.value();

        try {
            inventoryClient.reserveStock(event.orderId(), event.lineItems());
            processedEventRepository.save(new ProcessedEvent(eventId, Instant.now()));
            // Commit happens for both the reservation write and the dedup
            // record together, or neither does.
        } catch (InsufficientStockException ex) {
            orderRepository.markBackordered(event.orderId());
            processedEventRepository.save(new ProcessedEvent(eventId, Instant.now()));
        }

        ack.acknowledge(); // manual commit — only after the work is durable
    }
}

Two details matter more than the boilerplate. First, the dedup check and the side effect need to be atomic with each other — if reserveStock succeeds but the process crashes before the dedup row is saved, redelivery reprocesses it and you've double-reserved stock. Wrapping both in a local DB transaction (assuming reserveStock is itself transactional or idempotent-safe) closes that gap; if reserveStock is a call to an external service that isn't idempotent on its own, you need it to accept an idempotency key too, or you've just moved the duplicate-processing problem one hop downstream. Second, ack.acknowledge() happens last, after everything durable has succeeded — that's what makes a crash mid-handler result in redelivery instead of silent loss.

Dead letter queues and retry with backoff

Some messages will never succeed no matter how many times you retry them — a malformed payload, a foreign key that references a row that got deleted, a bug in a new deploy that throws on a specific edge case. Left unhandled, these "poison messages" block the partition behind them (in Kafka, a consumer stuck retrying offset N never advances to N+1) or get requeued forever in a tight loop (in RabbitMQ, burning CPU and filling logs).

The fix is a bounded retry policy with exponential backoff, followed by a dead letter queue (DLQ) — a separate topic or queue where messages get parked after N failed attempts, for a human or a separate remediation process to look at later, without blocking the main stream.

A few things that make this pattern work well rather than just move the problem: distinguish transient failures (downstream service timed out — retry makes sense) from permanent ones (payload fails schema validation — retrying is pointless, go straight to the DLQ). Cap total retry time to something bounded — three attempts with 1s/5s/30s backoff, not infinite retries — because an unbounded retry loop on a single message is functionally the same outage as no retry at all, just slower to notice. And treat the DLQ as a queue that needs an owner and an alert, not a place messages go to be silently forgotten; a DLQ nobody monitors is just a more elaborate way of dropping messages.

Consumer lag: the metric that tells you everything

If you can only alert on one number for an async system, make it consumer lag — the difference between the latest offset produced to a partition and the offset your consumer group has committed. Lag climbing means your consumers are falling behind production, for one of three reasons: a slow downstream dependency, insufficient consumer parallelism relative to partition count, or a consumer that's stuck or crash-looping on a poison message.

The reason lag matters more than almost any other metric is that it's a leading indicator wrapped as a lagging one — by the time lag is visibly high, you're already minutes or hours from an actual customer-facing problem (stale inventory counts, delayed shipping notifications, a fraud check that runs after the fraudulent order already shipped), but the lag metric itself was climbing well before anyone noticed. We've been paged for "orders aren't showing as shipped" incidents that were, on inspection, a consumer lag graph that had been climbing steadily for four hours with no alert configured on it. Set a threshold-based alert on lag (not just "consumer is down," which misses the slow-and-falling-behind case), and graph it per-partition, not just aggregated across the group — an aggregate can look healthy while one hot partition is badly behind.

Ordering: within a partition, not across them

Kafka guarantees strict ordering within a single partition and makes zero guarantee across partitions. If you key messages by orderId, every event for a given order lands on the same partition and is processed in the order it was produced — good, that's usually the guarantee you actually need, since "order created" arriving after "order shipped" for the same order is the failure mode you're trying to avoid. But if you need global ordering across all orders, Kafka fundamentally can't give you that without collapsing to a single partition, which caps your throughput at whatever one consumer can handle.

The practical implication: choose your partition key to match the entity whose ordering actually matters, and don't assume ordering holds for anything broader than that. "Events for the same customer are ordered relative to each other" is achievable and useful. "All events across all customers are globally ordered" is not something a partitioned log is built to give you, and reaching for it usually means the design should use a single partition (accept the throughput ceiling) or drop the requirement (usually it wasn't real — check whether the business logic actually needs cross-entity ordering or just assumed it had it).

Schema evolution: the bug that ships itself two months later

An event schema is a contract, and unlike an API contract, nobody calls it at request time to notice it broke — a producer can start emitting a slightly different shape today, and the consumer that chokes on it might not run against that data for weeks, if it's a batch job, or might break instantly for every real-time consumer, depending on topology. Either way, the failure surfaces far from the change that caused it, which makes it miserable to debug.

Using a schema registry with Avro or Protobuf (Confluent Schema Registry is the common Kafka pairing) turns this from a runtime surprise into a build-time or deploy-time check: the registry enforces compatibility rules — typically backward compatibility (new schema can read old data) as the default requirement — and rejects a producer's schema registration outright if it would break existing consumers. The rules that keep you safe are simple to state and easy to violate by accident: add fields as optional with defaults, never remove a field a consumer still reads, never repurpose a field's meaning or type, and version explicitly when a change genuinely isn't backward compatible rather than trying to force it through.

Even without a formal registry, the discipline matters: treat every event schema as something with real, if invisible, consumers depending on today's shape, and review schema changes with the same seriousness as a public API change — because that's exactly what it is, just without a compiler to catch you.

When async processing is the wrong answer

Queues are not free — they add latency (nothing is instant once you introduce a broker hop), operational surface area (another system to run, monitor, and page for), and a whole category of bugs (duplicates, ordering, lag) that a synchronous call simply doesn't have. Reach for async because a real property of the problem calls for it, not because event-driven architecture is fashionable.

Skip it when you need strong consistency across the write: if the customer needs to see the updated balance the instant the transfer completes, and there's no acceptable window where the two accounts disagree, that's a transactional, synchronous operation — wrapping it in "publish an event and reconcile eventually" trades a real consistency requirement for architectural fashion. Skip it when the end-to-end latency of the async round-trip is worse than just doing the work inline — if the "async" work is a 20ms cache lookup, adding a broker hop to make it "event-driven" adds latency for no benefit; synchronous is faster and simpler. And skip it when the operation must complete before returning success to the caller — if "reserve the seat" can't be allowed to fail silently after you've already told the user their booking is confirmed, don't put it on a queue where a downstream failure means the confirmation you already sent was a lie.

The order-processing example from the opening is the right shape for async precisely because "order placed" and "warehouse notified" can legitimately disagree for a few seconds without anyone lying to the customer. "Payment charged" and "order total calculated" cannot — get that one wrong and you've shipped a bug where customers get charged the wrong amount because two systems disagreed about a number that should never have been eventually-consistent in the first place.

Common mistakes

Treating "at-least-once" as a footnote instead of a design constraint. Teams read the Kafka docs, see "at-least-once delivery," nod, and then write a consumer that increments a counter or calls a non-idempotent payment API with no dedup logic. It works fine in testing, because duplicates are rare under low load and clean shutdowns. It breaks in production during a rebalance, a slow consumer restart, or a network blip — exactly when you can least afford it.

Ignoring consumer lag until it's an outage. Lag is cheap to monitor and expensive to discover after the fact. If there's one dashboard to build before shipping a new Kafka consumer, it's per-partition lag with an alert threshold, not "is the consumer process running" — a wedged consumer that's technically alive but stuck on one message looks perfectly healthy on a process-uptime check.

Choosing a partition key that creates a hot partition. Keying by tenantId seems reasonable until one enterprise customer generates 40% of your traffic, and their partition — and only their partition — falls behind while the other nine sit idle. Look at the actual distribution of your key's cardinality and skew before committing to it, not just its logical correctness.

Unbounded retries on a poison message. A consumer that retries forever on a message it can never successfully process isn't resilient, it's stuck — and depending on your commit strategy, it can block every message behind it in the same partition indefinitely. Bound your retries, and have a DLQ with an owner on the other end of it.

Assuming schema changes are backward compatible without checking. A field rename or type change that looks harmless to the producer team can silently break a consumer team's deserialization months later, in a service the producer team has never heard of. Schema registry enforcement (or at minimum, a documented compatibility review) is cheap insurance against a bug that otherwise surfaces as "why did the recommendation service stop working three weeks after we shipped an unrelated change."

Final takeaway

Message queues buy you resilience to downstream failures and independent scaling of producers and consumers — at the cost of duplicates you must handle, ordering you must design for deliberately, and a lag metric you must watch like a heart-rate monitor. None of that is a reason to avoid them; it's a reason to be precise about what you're actually asking the queue to guarantee before you build a consumer that assumes it guarantees more. Pick pub-sub or point-to-point based on whether the event is a fact or a task. Pick Kafka or RabbitMQ based on whether you need a replayable log or flexible routing. And build every consumer as if it will see the same message twice — because eventually, it will.

Next in this system design series: Designing for High Availability & Disaster Recovery, where we'll zoom out from a single broker's guarantees to what happens when an entire region goes down.

Subscribe to get system design and backend engineering posts in your inbox every two weeks.