Pub/Sub: Fan-Out and Event-Driven Systems

TL;DR: Publish/subscribe is the 1:N messaging pattern: a publisher sends to a topic, and the broker copies the message to every subscription¹. It is the “broadcast” cousin of the queue (which is 1:1). The critical design axis is durability versus latency versus cost. Redis Pub/Sub delivers in sub-milliseconds but drops messages silently if a subscriber disconnects². Google Cloud Pub/Sub replicates to a quorum of clusters before acking, powering 500M+ messages/sec internally³. Kafka gives you replay and consumer groups but forces partition planning⁴. Use Redis Pub/Sub for best-effort realtime; Kafka for durable fanout with replay; SNS+SQS when AWS-native and you want zero ops.

Learning Objectives

After this module, you will be able to:

• Distinguish pub/sub from queueing and point-to-point messaging
• Pick push vs pull delivery based on consumer behavior and scale
• Choose between in-memory (Redis, NATS) and durable (Kafka, Google Pub/Sub, SNS+SQS) substrates
• Design fan-out strategies that handle the celebrity problem without write amplification explosions
• Implement per-key ordering without requiring global ordering (which does not scale)
• Recognize when Redis Pub/Sub will silently lose messages and what to use instead

Intuition

Think of a radio station. The DJ speaks into a microphone. The signal goes out to every tuned-in receiver simultaneously. The DJ does not know how many listeners exist, does not wait for each one to confirm receipt, and does not re-broadcast for someone who turned on their radio late. If you missed the 8am news, it is gone.

Now think of a newspaper subscription. The paper prints one edition and mails a copy to every subscriber on the list. A new subscriber starts receiving tomorrow’s paper, not yesterday’s. If the mailbox is full, the paper piles up (or gets discarded by the carrier).

Pub/sub is both of these models, depending on the system you pick. Redis Pub/Sub is the radio: fire-and-forget, instant, no memory. Kafka is the newspaper with an archive room: every edition is numbered and shelved, and a new subscriber can start reading from edition #1.

The rest of this chapter teaches you when to pick the radio, when to pick the newspaper, and what breaks when you confuse the two.

Theory

The pub/sub pattern

A pub/sub system has four actors: a publisher that emits messages, a topic (the named channel), a subscription (an interest expression), and a subscriber that receives copies of matching messages. Each subscription gets its own copy of every matching message. This is the fundamental difference from a queue: a queue delivers each message to exactly one competing consumer; pub/sub delivers each message to every subscriber¹.

flowchart LR
    subgraph "Pub/Sub (1:N)"
        P1[Publisher] --> T1[Topic]
        T1 --> S1A[Subscriber A]
        T1 --> S1B[Subscriber B]
        T1 --> S1C[Subscriber C]
    end
    subgraph "Queue (1:1)"
        P2[Publisher] --> Q[Queue]
        Q --> C1[Consumer 1]
        Q -.- C2[Consumer 2 idle]
        Q -.- C3[Consumer 3 idle]
    end

Subscriptions can be durable (survive subscriber disconnect: Google Pub/Sub, Kafka consumer groups, MQTT with CleanStart=0) or ephemeral (disappear on disconnect: Redis Pub/Sub, NATS Core)³⁵. Filtering is usually topic-based (subscribe to orders.us) with optional wildcards (MQTT + for one level, # for multi-level), or content-based using filter predicates on message attributes (SNS filter policies, Google Pub/Sub filter expressions)⁶⁷.

Systems compared

Not all pub/sub systems are interchangeable. They sit on a spectrum from “fast and lossy” to “durable and heavy”:

Use Redis Pub/Sub when losing a message is acceptable (cache invalidation, typing indicators). Use Kafka when you need replay, ordering, and multi-consumer-group independence. Use SNS+SQS when you want durable fan-out on AWS without running infrastructure.

Delivery semantics

Three levels exist, and only one is honest at scale:

• At-most-once (Redis Pub/Sub, NATS Core): the broker writes to the subscriber’s socket and forgets. If the subscriber is disconnected or slow, the message is lost. No retry, no ack, no persistence².
• At-least-once (Google Pub/Sub, SNS+SQS, Kafka default): the broker retries until the subscriber acknowledges. Duplicates are possible on publisher retry, broker crash mid-ack, or consumer crash after side-effect but before commit³⁴.
• Exactly-once (Kafka EOS, Google Pub/Sub pull-only, MQTT QoS 2): only holds within one system boundary. The moment you make an external RPC or write to a database, the chain breaks⁴¹¹. MQTT QoS 2 uses a four-packet handshake (PUBLISH/PUBREC/PUBREL/PUBCOMP) that significantly reduces throughput versus QoS 0 due to requiring 4 network round-trips per message¹⁰.

Fan-out strategies

When a producer has N followers, how do you make sure all N see the event? This is the fan-out problem, and it has three solutions:

Fan-out on write (push): At publish time, insert the message into each follower’s inbox. Twitter’s 2013 design used a Redis cluster with each timeline as an 800-entry linked list; publishing a tweet with 20,000 followers meant 20,000 LPUSH operations. This gave 300K QPS timeline reads¹². The failure mode: Lady Gaga had 31 million followers in 2013, producing up to 5 minutes of end-to-end tail latency for a single tweet¹².

Fan-out on read (pull): Do no work at publish time. At read time, query each followed author’s timeline and merge. LinkedIn’s FollowFeed uses this approach, measuring a 62x reduction in data set size versus pre-materialization¹³.

Hybrid: Push for normal users, pull for celebrities. Twitter explicitly stopped fanning out “high-value users,” merging them in at read time, saving “tens of percent of computational resources”¹².

sequenceDiagram
    participant Author
    participant FanoutSvc as Fanout Service
    participant Inbox as Follower Inboxes
    participant ReadSvc as Timeline Read Service
    participant Reader

    Author->>FanoutSvc: POST /post
    alt followers < 100K (normal)
        FanoutSvc->>Inbox: write to N follower inboxes
    else followers >= 100K (celebrity)
        FanoutSvc-->>FanoutSvc: skip write-side fanout
    end
    Reader->>ReadSvc: GET /timeline
    ReadSvc->>Inbox: read pre-materialized entries
    ReadSvc->>ReadSvc: merge celebrity authors at read time
    ReadSvc-->>Reader: sorted timeline

Push vs pull delivery

Push (broker initiates): lowest delivery latency, no polling overhead. Google Pub/Sub push subscriptions POST each message to an HTTPS endpoint. AWS SNS is push-only to subscribers (HTTP, SQS, Lambda). MQTT is push from broker to subscriber¹⁴⁹¹⁰.

Pull (subscriber polls): subscriber controls rate, large batch sizes give high throughput. Kafka consumer groups are pull-only by design. Google Pub/Sub recommends pull for “GBs per second” workloads¹⁴.

The trade-off: push is simpler for consumers (“my code runs when an event happens”) but requires flow-control protocols to handle slow subscribers. Pull adds client-side complexity (cursor management, ack-deadline extension, rebalance handling) but decouples consumer slowness from broker health.

Use push when latency matters and subscribers are fast. Use pull when throughput matters and subscribers process at varying speeds. Chapter 2.10 Real-Time Communication covers the consumer-side delivery mechanisms (WebSockets, SSE, long-polling).

Durability spectrum

Pub/sub systems trade durability for latency along a clear spectrum:

flowchart LR
    A["Redis Pub/Sub<br/>None, sub-ms"] --> B["NATS Core<br/>None, sub-ms"]
    B --> C["Redis Streams<br/>Retention, ~ms"]
    C --> D["NATS JetStream<br/>Replicated, ~10ms"]
    D --> E["Kafka<br/>Retention, tens of ms"]
    E --> F["Google Pub/Sub / SNS+SQS<br/>7-14 days, tens of ms"]

Redis Pub/Sub has zero durability: “Published messages evaporate, regardless if there was any subscriber”². A subscriber that disconnects loses those messages forever. Redis Streams (XADD/XREADGROUP) added a persistent append-only log with consumer groups¹⁵.

SNS+SQS is the AWS durability pattern: SNS is push with no storage, but the subscribed SQS queues persist messages for up to 14 days. A subscriber outage does not lose data¹⁶.

Google Cloud Pub/Sub replicates each publish to a quorum of clusters before acking. On the data plane, the message is written to N clusters and within each cluster to M disks; the publish is acked once written to at least half the clusters³. Unacked messages are retained for 7 days with seek-based replay.

flowchart TD
    P[Publisher] --> SNS[SNS Topic]
    SNS -->|"filter: type=billing"| Q1[SQS: billing]
    SNS -->|"filter: type=email"| Q2[SQS: email]
    SNS -->|"filter: type=search"| Q3[SQS: search-index]
    Q1 --> C1[Billing Worker]
    Q2 --> C2[Email Worker]
    Q3 --> C3[Indexer]

Real-World Example

Discord: fan-out to 1M+ concurrent users in a single guild.

Discord’s Midjourney server has over 10 million members with 1M+ concurrent online users¹⁷. Every message, presence update, and voice event must fan out to every connected session in the guild. This is quadratic: a 100,000-user guild doing 1 message per user generates 10 billion notifications.

Discord’s architecture uses Elixir (Erlang/BEAM) with one process per guild as the central router. Each connected user has a session process. The guild process holds the session list and fans out every event to every session, which forwards over WebSocket to the client.

Three optimizations make this tractable:

1. Passive sessions. Roughly 90% of user-guild connections in large servers are marked “passive.” These skip per-message permission checks and payload serialization. They receive events only when the user foregrounds the guild. This 90% passive rate gives approximately 3x headroom in maximum community size¹⁷.

2. Relay processes. Instead of the guild process sending directly to all sessions, relay processes sit between guild and sessions. Each relay handles up to 15,000 connected sessions, splitting the fan-out across multiple Erlang processes.

3. Manifold Offload. The network send is moved to a separate sender process, insulating the guild from BEAM backpressure when a relay’s network buffer backs up¹⁷.

flowchart LR
    subgraph Guild["Guild Process (1 per server)"]
        GP[Guild Router]
    end
    GP --> R1[Relay 1<br/>15K sessions]
    GP --> R2[Relay 2<br/>15K sessions]
    GP --> R3[Relay N<br/>15K sessions]
    R1 --> S1[Session processes]
    R2 --> S2[Session processes]
    R3 --> S3[Session processes]
    S1 -->|WebSocket| C1[Clients]
    S2 -->|WebSocket| C2[Clients]
    S3 -->|WebSocket| C3[Clients]

The fan-out is local (in-process via Erlang message passing), not over the network for the per-session step. The cost is scheduling plus permission checks, not TCP round-trips. Where Elixir’s immutable data structures became bottlenecks, Discord replaced them with Rust NIFs: at 250,000 items, raw Elixir ordsets worst-case insert was 27,000 microseconds. Discord’s custom Elixir OrderedSet (a skip-list-like structure) improved that to 4-640 microseconds. The final Rust SortedSet NIF delivered 0.61-3.68 microsecond inserts across set sizes from 5,000 to 1,000,000 items¹⁸.

The lesson: at extreme fan-out, the pub/sub primitive is not a broker. It is in-process message passing with careful tiering to keep the quadratic cost manageable.

Trade-offs

Decision rule: Use Redis Pub/Sub for ephemeral notifications where loss is tolerable. Use Kafka when you need replay, per-key ordering, and multiple independent consumer groups. Use SNS+SQS when you are on AWS and want durable fan-out without running infrastructure. Use Google Pub/Sub on GCP. Use NATS JetStream when you want Kafka semantics without Kafka ops.

Common Pitfalls

Exercise

Design Challenge: You are building the notification fan-out for a social app with 100M users. Each user generates an average of 10 events per day (posts, likes, comments). Each event must reach all followers. The average user has 200 followers; 0.1% of users have 1M+ followers. You must preserve per-user ordering (a user’s events arrive in the order they were created). Design the pub/sub architecture: pick a substrate, estimate throughput, handle the celebrity problem, and explain your ordering strategy.

Key Takeaways

• Pub/sub is fan-out: every subscriber gets every matching message. It is not a load-balancer.
• Redis Pub/Sub is ephemeral and will silently drop messages. Use Streams, Kafka, or SNS+SQS when delivery matters.
• Fan-out on write is fast to read but breaks on celebrities. Hybrid (push for normal, pull for celebrities) is required at social-network scale.
• Global ordering does not scale. Design for per-key ordering (user_id, order_id) and accept that cross-key events are unordered.
• Push is simpler for consumers; pull is simpler for the broker. Choose based on where you want to absorb slow consumers.
• SNS+SQS is the simplest durable fan-out on AWS and solves most problems without running infrastructure.
• At-least-once with idempotent consumers is the honest delivery guarantee. “Exactly-once” only holds within one system boundary.

Further Reading

• Google Cloud Pub/Sub Architectural Overview - the single best primary architecture doc for a cloud pub/sub service; covers data-plane/control-plane split, quorum writes, and consistent hashing.
• Fanout Amazon SNS notifications to Amazon SQS queues - AWS’s canonical fan-out pattern with JSON envelope example; the “right default” for durable fan-out on AWS.
• The Architecture Twitter Uses to Deal with 150M Active Users - Raffi Krikorian’s 2013 talk writeup; the celebrity problem origin story with specific numbers on fan-out latency.
• Discord: Maxjourney - Pushing Discord’s Limits - Elixir guild fanout, passive sessions, relay tier; the clearest explanation of quadratic fan-out cost at 1M+ concurrent users.
• LinkedIn FollowFeed - fan-out-on-read at scale with scatter-gather brokers; demonstrates the 62x data reduction versus pre-materialization.
• Slack Engineering: Real-time Messaging - Channel Server / Gateway Server / consistent hashing; production pub/sub for chat at 16M channels per host.
• Enterprise Integration Patterns: Publish-Subscribe Channel - Hohpe and Woolf’s original 2003 pattern definition; still the best one-page reference for the formal semantics.
• OASIS MQTT 5 Specification - the canonical pub/sub protocol for IoT; read for QoS 0/1/2 semantics and the cost of exactly-once delivery.

Flashcards

References