Message Queues and Streaming: Decoupling at Scale

TL;DR: Message systems split into two families. A queue (RabbitMQ, SQS) delivers each message to one consumer and deletes it after acknowledgement. A log (Kafka, Kinesis, Pulsar) is an append-only, ordered, durable record that many consumer groups read independently at their own pace. Jay Kreps’s 2013 essay reframed the log as the source of truth from which every downstream system derives its state¹. The practical delivery guarantee is at-least-once with idempotent consumers; “exactly-once” only holds end-to-end within Kafka’s transactional boundary². At scale, LinkedIn processes over 7 trillion messages per day across 4,000+ brokers³, and Netflix processes hundreds of billions of events daily through its Keystone pipeline. The partition is the unit of parallelism, the cap on consumer scaling, and the boundary of ordering.

Learning Objectives

After this module, you will be able to:

• Distinguish a queue from a log and pick the right one for a workload
• Design for at-least-once delivery with idempotent consumers
• Reason about Kafka partitions, consumer groups, and ordering guarantees
• Handle backpressure, dead-letter queues, and poison messages
• Compare Kafka, RabbitMQ, SQS, Kinesis, and Pulsar on real criteria
• Explain why partition count is a permanent architectural decision

Intuition

Think of two services at the post office.

The first is the package counter. You hand a parcel to a clerk. The clerk gives it to exactly one delivery driver. Once delivered, the receipt is shredded. If you want to send the same parcel to two people, you need two parcels. This is a queue: one message, one consumer, gone after processing.

The second is the newspaper archive. Every edition is printed, numbered, and shelved in order. Any subscriber can walk in, find edition #4,217, and read forward from there. A new subscriber does not need to wait for tomorrow’s paper; they can start from any past edition. Adding a subscriber does not slow down existing readers. This is a log: append-only, ordered, replayable, multi-subscriber.

The queue is simpler. The log is more powerful. Most confusion in system design comes from reaching for a queue when you need a log, or paying the operational cost of a log when a queue would suffice.

The rest of this chapter teaches you to tell the difference and pick correctly.

Theory

Queue vs stream: the fundamental distinction

A queue (SQS, RabbitMQ work queue) tracks per-message state: visible, in-flight, acknowledged. Competing consumers drain work faster than one could alone. Once acknowledged, the message is gone. If you add a new consumer service next month, it starts from “now” with no history⁴.

A log (Kafka, Kinesis, Pulsar) stores durable, offset-addressable segments. Each consumer group tracks its own offset. Adding a new consumer group replays history from offset 0 without disturbing existing consumers. The log is not just a transport mechanism; it is the source of truth that downstream systems (search indexes, caches, warehouses, microservices) derive their state from¹.

graph LR
    Start[New async workload] --> Q1{Need replay<br/>or multi-subscriber?}
    Q1 -->|Yes| Log[Use a log:<br/>Kafka / Kinesis / Pulsar]
    Q1 -->|No| Q2{Rich routing<br/>fanout / topic?}
    Q2 -->|Yes| RMQ[RabbitMQ]
    Q2 -->|No| Q3{In AWS<br/>and low-ops?}
    Q3 -->|Yes| SQS[SQS Standard<br/>or FIFO if ordered]
    Q3 -->|No| Other[Redis Streams<br/>or NATS JetStream]

Use a log when you need replay, multi-subscriber fan-out, or CDC. Use a queue when you need simple work distribution with per-message acks and no history.

Delivery guarantees: at-most, at-least, exactly-once

Three levels exist, and only one is honest at scale:

• At-most-once (Kafka acks=0, fire-and-forget): the producer sends and moves on. Messages can be lost on broker crashes.
• At-least-once (the practical default): the producer retries until it gets an ack. On broker crashes, a message may be written twice. Consumers must handle duplicates. Netflix’s Keystone pipeline uses acks=1 (leader-only acknowledgement), a pragmatic middle ground that accepts a small data loss risk on leader failure in exchange for lower latency and higher availability⁵.
• Exactly-once (Kafka EOS): requires enable.idempotence=true (producer sequence numbers deduped broker-side), a transactional.id (atomic writes across partitions plus consumer offset commit in the same transaction), and consumers using isolation.level=read_committed². Throughput overhead is approximately 3% compared to at-least-once with acks=all².

Kafka architecture: topics, partitions, ISR, KRaft

A Kafka topic is split into N partitions. Each partition is a replicated log with one leader and several followers. The ISR (in-sync replicas) is the subset of replicas that have caught up to the leader.

Key invariants:

• acks=all + min.insync.replicas=2 on replication factor 3 guarantees no data loss under single-broker failure⁶.
• Producer idempotence (default since Kafka 3.0) assigns each producer a PID and each batch a sequence number; brokers dedupe retries with max.in.flight.requests.per.connection <= 5⁶.
• Partition throughput: 10-50 MB/s per partition in production⁷.

flowchart LR
    subgraph Producers
        P1[Order Service]
        P2[Payment Service]
    end
    subgraph Kafka["Kafka Cluster (RF=3)"]
        subgraph Topic["topic: orders"]
            PA["Partition 0<br/>Leader: B1<br/>ISR: B1,B2,B3"]
            PB["Partition 1<br/>Leader: B2<br/>ISR: B2,B3"]
            PC["Partition 2<br/>Leader: B3<br/>ISR: B1,B3"]
        end
        DLQ["orders.DLQ"]
    end
    subgraph CG1["Consumer Group: billing"]
        C1[billing-1]
        C2[billing-2]
    end
    subgraph CG2["Consumer Group: search-index"]
        C3[indexer-1]
    end
    P1 -->|"key=order_id"| Topic
    P2 -->|"key=order_id"| Topic
    PA --> C1
    PB --> C2
    PC --> C2
    PA --> C3
    PB --> C3
    PC --> C3
    C1 -.->|"after N retries"| DLQ

KRaft (KIP-500) replaced ZooKeeper with an internal Raft-based metadata quorum. ZooKeeper was deprecated in Kafka 3.5 and fully removed in Kafka 4.0⁸. This eliminates an entire system from the operational footprint and enables faster controller failover.

Tiered storage (KIP-405, GA in Kafka 3.9) offloads older log segments to object storage (S3, GCS), significantly reducing storage cost for long-retention topics while keeping recent data on local disks for hot reads⁹¹⁰.

RabbitMQ and AMQP: exchanges, flexibility, routing

RabbitMQ implements AMQP 0-9-1. Producers publish to an exchange; bindings attach queues to exchanges with a routing key pattern¹¹:

• Direct exchange: routes to queues whose binding key exactly matches the routing key.
• Fanout exchange: ignores routing keys, delivers to every bound queue.
• Topic exchange: uses dotted routing keys (orders.eu.paid) with * (one word) and # (zero or more words) wildcards.

flowchart TD
    P["Order Service"] -->|"routing_key: orders.eu.paid"| EX{"Topic Exchange<br/>(orders)"}
    EX -->|"binding: orders.eu.*"| Q1["queue: eu-billing"]
    EX -->|"binding: orders.*.paid"| Q2["queue: revenue-analytics"]
    EX -->|"binding: orders.#"| Q3["queue: audit"]
    Q1 --> C1["EU Billing Worker"]
    Q2 --> C2["Analytics Worker"]
    Q3 --> C3["Audit Worker"]

Modern RabbitMQ (3.8+) provides quorum queues built on Raft, replacing the deprecated mirrored-queue feature with clear failure semantics¹². However, RabbitMQ throughput (tens of thousands msgs/sec per queue) is an order of magnitude below Kafka (hundreds of thousands per partition)¹³. It is a broker, not a log: once acknowledged, messages are gone. No replay.

Use RabbitMQ when you need rich routing (topic/fanout/headers), low-latency task distribution, or RPC-style request/reply patterns. Use Kafka when you need a durable, replayable event log.

SQS, SNS, Kinesis, Pulsar, and Redpanda

SQS uses a visibility timeout (default 30s): after ReceiveMessage, the message is invisible to other consumers. If the consumer crashes before DeleteMessage, the message reappears. A redrive policy moves messages to a DLQ after maxReceiveCount failed receives¹⁶.

For fan-out in AWS, combine SNS (pub/sub) with SQS (per-subscriber queue): SNS delivers to N SQS queues, each consumed independently. This gives you log-like multi-subscriber semantics without running Kafka.

Consumer patterns: groups, competing consumers, DLQ, backpressure

Consumer groups distribute partitions across members. You cannot have more active consumers per group than partitions. This is the hard ceiling on parallelism⁷.

Cooperative rebalance (KIP-429, Kafka 2.4+): only partitions that need to move are revoked, instead of the older “stop-the-world” protocol where every consumer surrenders every partition on any membership change¹⁷¹⁸.

Backpressure is managed via bounded in-flight windows:

• Kafka: max.poll.records and fetch.max.bytes
• SQS: 120,000 in-flight messages per queue¹⁹
• RabbitMQ: prefetch (QoS) count

Dead Letter Queue (DLQ): after N retries with exponential backoff, produce the failing record to a <topic>.DLQ, commit the offset, and move on. This is the Dead Letter Channel pattern from Hohpe and Woolf’s Enterprise Integration Patterns²⁰. Alert on DLQ depth. Build a redrive tool to re-inject fixed messages.

Lag monitoring: the difference between log-end offset and committed offset. LinkedIn’s Burrow is the canonical consumer-lag monitor²¹.

sequenceDiagram
    participant App as Producer
    participant B as Kafka Broker
    participant C as Consumer
    participant DB as Idempotency Store
    App->>B: send(order_id=42) seq=7
    B-->>App: timeout (no ack)
    App->>B: retry send(order_id=42) seq=7
    Note over B: Idempotent dedupe by PID+seq
    B-->>App: ack offset=1001
    C->>B: poll()
    B-->>C: offset=1001, order_id=42
    C->>DB: INSERT idempotency_key=42
    alt Key is new
        DB-->>C: OK
        C->>C: Process side effect
        C->>B: commit offset=1002
    else Key exists (duplicate)
        DB-->>C: Duplicate
        C->>B: commit offset=1002 (skip work)
    end

Real-World Example

LinkedIn Kafka: 7 trillion messages per day.

LinkedIn is where Kafka was born, and it remains the largest public deployment. As of 2019, LinkedIn operates over 100 Kafka clusters with 4,000+ brokers, 100,000+ topics, and 7 million partitions, processing more than 7 trillion messages per day³. The largest single cluster runs 140+ brokers with 1 million replicas.

Every LinkedIn service runs a Kafka client (or a REST proxy for non-JVM languages). Events flow to regional Kafka clusters, are mirrored cross-region by Brooklin (which replaced MirrorMaker, mirroring more than 7 trillion messages/day between clusters)²², and consumed by Samza stream processors, Hadoop, Pinot, and hundreds of microservices.

Key engineering decisions that kept this running:

• Maintenance mode brokers: brokers flagged for decommission stop receiving new partition assignments, letting SREs safely drain hardware without triggering rebalance storms³.
• Controller memory optimization: reusing UpdateMetadataRequest objects to prevent cascading controller failures in clusters with millions of replicas³.
• Cruise Control: automated partition rebalancing and self-healing across clusters.

The failures are instructive. LinkedIn’s KIPs document production issues: KIP-291 (separating controller from data-plane connections to avoid head-of-line blocking), KIP-354 (maximum log compaction lag, after a compaction stall filled disks), and KIP-380 (detecting outdated control requests after broker bounces)³.

The lesson: at 7 trillion messages per day, every configuration choice (partition count, acks setting, retention policy, rebalance strategy) is the difference between a healthy pipeline and a 3am page.

Trade-offs

Decision rule: Use Kafka for partitioned ordered streams with replay. Use SQS when you do not need ordering or replay and want zero ops. Use RabbitMQ when you need flexible routing patterns. Use Kinesis or Pulsar when your constraints (AWS-native or multi-tenant) demand them.

Common Pitfalls

Exercise

Design Challenge: You are building the event backbone for a fintech platform. Requirements: 100 million events per day, 3 consumer types (fraud detection with sub-second latency, billing with exactly-once semantics, analytics warehouse with hourly batch loads), and strict per-user ordering of financial events.

Key Takeaways

• A queue delivers each message to one consumer and deletes it. A log is an append-only, replayable record that many consumer groups read independently.
• At-least-once with idempotent consumers is the honest, practical delivery guarantee. “Exactly-once” only holds end-to-end within Kafka’s transactional boundary.
• The partition is the unit of parallelism and the cap on consumer scaling. You cannot have more active consumers per group than partitions.
• Ordering is only meaningful per partition. Design your partition key for the ordering you need (typically entity ID).
• Partition count is nearly permanent. Increasing it later reshuffles hash(key) % N, breaking ordering guarantees. Over-provision at creation.
• DLQs are not optional. Poison messages will happen, and without a DLQ they block the entire partition indefinitely.
• Use Kafka for ordered event streams with replay. Use SQS for zero-ops async jobs. Use RabbitMQ for rich routing and task distribution.

Further Reading

• The Log: What every software engineer should know (Jay Kreps, 2013) - the foundational essay that reframes messaging as derived views over a durable log; read this before anything else in the space.
• How LinkedIn customizes Apache Kafka for 7 trillion messages/day - the canonical large-scale operations story; covers controller tuning, maintenance mode, and Cruise Control.
• Exactly-Once Semantics Are Possible (Confluent, 2017) - authoritative walkthrough of idempotent producer + transactions; explains the 3% overhead and the Kafka-to-Kafka boundary.
• Scaling Slack’s Job Queue - the best “add Kafka incrementally without rewriting everything” story; shows Kafka as a durable buffer in front of Redis workers.
• Kafka Inside Keystone Pipeline (Netflix, 2016) - running Kafka in a cloud that treats brokers as cattle; covers the acks=1 trade-off and cold-standby failover.
• Designing Data-Intensive Applications, Ch. 11 (Kleppmann) - conceptual framing of messaging and stream processing; the textbook treatment of exactly-once semantics.
• Enterprise Integration Patterns (Hohpe and Woolf) - the DLQ, routing, and channel patterns everyone reinvents badly; read the Dead Letter Channel and Competing Consumers patterns.
• KIP-500: Replace ZooKeeper with Self-Managed Metadata Quorum - the design document for KRaft; explains why ZooKeeper was the operational bottleneck.

Flashcards

References