Design a Video Conferencing System (Zoom / Google Meet)

TL;DR. A 20-person video meeting is not 20 video calls. On an SFU it is 20 publishes and 380 subscribes, all within a 150 ms audio budget while packet loss, NAT firewalls, and asymmetric bandwidth conspire against you. The dominant architecture is SFU + simulcast: each publisher encodes 2-3 resolution layers, the SFU selects per subscriber based on TWCC bandwidth estimates, and the server never decodes media. Zoom reported 300M daily meeting participants in April 2020 (a corrected figure after Zoom retracted an earlier “daily active users” claim; participants counts the same person in multiple meetings per day)¹ on this model. The pivotal trade-off is SFU (cheap CPU, heavy subscriber downlink) versus MCU (expensive CPU, minimal subscriber bandwidth).

Learning Objectives

• Design an SFU-based video conferencing system that handles 10M concurrent participants across 50+ global POPs
• Compare P2P mesh, SFU, and MCU topologies and justify SFU as the modern default for rooms of 5-1,000 participants
• Estimate media bandwidth, TURN relay cost, and recording storage for a Zoom-scale deployment
• Justify simulcast over SVC for broad client compatibility and explain the layer-selection mechanism
• Trade off cloud-trust (server can transcode, record, caption) versus E2EE (server sees only ciphertext)

Intuition

A video call between two people is trivial. Each side sends one stream, receives one stream. A WebSocket server exchanges SDP offers, ICE punches through the NAT, and SRTP flows peer-to-peer. Done.

Now add 18 more people. Each participant must publish their camera to 19 others. In a naive mesh, that is 20 x 19 = 380 directed media streams. A typical home uplink of 5 Mbps cannot sustain even five 720p streams at 1.5 Mbps each². The mesh collapses at participant five.

The insight that unlocks the design: put a forwarding server in the middle. Each participant publishes once. The server copies each stream to every subscriber. This is the Selective Forwarding Unit (SFU). It never decodes the video, so CPU stays low. But now the server must decide which resolution each subscriber gets, because a phone on cellular cannot handle 19 streams at 720p. That decision, made 100 times per second per subscriber using bandwidth estimates from TWCC feedback, is what separates a production SFU from a toy.

The second pressure is latency. Humans perceive audio delay above 150 ms as “awkward pauses.” Video can tolerate up to 500 ms. Every hop, every encode, every jitter buffer eats into that budget. An MCU that decodes and re-encodes adds 50-100 ms of latency that an SFU avoids entirely.

Requirements

Clarifying Questions

• Q: Max participants per meeting? Assume: Standard meetings up to 100; webinars up to 1,000; large events up to 100K (broadcast mode).
• Q: SFU, MCU, or P2P? Assume: SFU as default. MCU for PSTN dial-in and recording composition. P2P for 1:1 only.
• Q: End-to-end encryption required? Assume: Cloud-trust by default (enables recording, captions, AI). E2EE as opt-in for sensitive meetings.
• Q: Recording? Assume: Cloud recording with per-track capture and async composition. Live composition for streaming.
• Q: Global from day one? Assume: Yes. Media POPs in 50+ cities. Participants connect to nearest POP.
• Q: Background effects, captions, transcription? Assume: Client-side background blur (MediaPipe). Server-side ASR for captions and transcription.

Functional Requirements

• Create and join meetings with audio, video, and screen share tracks
• Publish 2-3 simulcast layers per video track; SFU selects per subscriber
• Host controls: mute-all, remove participant, lock meeting, waiting room
• Cloud recording with post-meeting playback
• Live captions via server-side ASR
• Breakout rooms (sub-sessions within a meeting)

Non-Functional Requirements

• Load: 500K simultaneous meetings, 10M concurrent participants (avg 20/meeting)
• Latency: p99 audio end-to-end <150 ms, p99 video <500 ms
• Availability: 99.9% for media path
• Bandwidth: Participant uplink 2-5 Mbps, downlink 5-20 Mbps
• Compliance: EU meetings stay in-region; TURN relay for NAT traversal

Capacity Estimation

Key ratios:

• Read:write ratio: Each publish generates N-1 subscribes. A 20-person meeting has a 19:1 fan-out ratio.
• TURN cost: Often the single largest bandwidth line item despite serving only 5-10% of sessions³.
• Reference scale: Zoom reported ~300M daily meeting participants in April 2020 (corrected from an initial “daily active users” claim; participants double-counts users who attend multiple meetings per day)¹. Google Meet grew 30x in three months to ~100M daily participants⁴.

API and Data Model

API Design

POST /v1/meetings
  Body: { "title": "...", "host_id": "...", "settings": {...} }
  Returns: 201 { "meeting_id": "uuid", "join_url": "...", "host_token": "..." }

POST /v1/meetings/{id}/join
  Body: { "participant_id": "...", "device_info": {...}, "bandwidth_hint": 5000 }
  Returns: 200 { "signaling_ws_url": "wss://...", "turn_credentials": {...} }

POST /v1/meetings/{id}/record
  Returns: 202 { "job_id": "uuid" }

GET /v1/meetings/{id}/stats
  Returns: 200 { "participants": 18, "avg_bitrate_kbps": 1200, "packet_loss": 0.02 }

Signaling (WebSocket):

WS /v1/meetings/{id}/signaling
  -> sdp_offer / sdp_answer
  -> ice_candidate (trickled)
  -> participant_joined / participant_left
  -> layer_change_request { "track_id": "...", "max_height": 360 }
  -> mute_changed / speaker_active

Media plane: SRTP over UDP via ICE/DTLS handshake. RTCP feedback: NACK, PLI, TWCC.

Data Model

-- Meeting metadata (PostgreSQL)
CREATE TABLE meetings (
  meeting_id   UUID PRIMARY KEY,
  host_id      UUID NOT NULL,
  title        TEXT,
  settings     JSONB,  -- waiting_room, e2ee_flag, recording_policy
  created_at   TIMESTAMPTZ,
  ended_at     TIMESTAMPTZ
);

-- Session state (Redis, keyed by meeting_id, volatile)
-- Active participants, published tracks, simulcast layer assignments,
-- per-subscriber BWE targets, speaker history

-- Recording manifest (S3 + metadata DB)
-- Per-track chunk list, timing index, composition recipe, job state

-- TURN credentials (Redis, TTL ~4h)
-- Per-session HMAC-signed username/password

High-Level Architecture

flowchart LR
    subgraph Client
        Cam[Camera + Mic] --> Enc[WebRTC encoder<br/>simulcast 180p/360p/720p]
        Dec[WebRTC decoder] --> Disp[Display + Speaker]
    end
    subgraph EdgePOP[Edge POP]
        Enc -->|SRTP/UDP| SFU[SFU media server]
        SFU --> Dec
        Enc <--> TURN[TURN/STUN]
        Sig[Signaling WS] <--> Ctrl[Meeting controller]
    end
    subgraph Cloud
        SFU <-->|private backbone| SFU2[Peer SFU<br/>cross-region]
        SFU --> Rec[Recording<br/>composer]
        SFU --> ASR[Captions ASR]
        Rec --> Store[(S3)]
        ASR -.partials.-> Sig
        Ctrl --- PG[(PostgreSQL)]
        Ctrl --- RD[(Redis)]
    end

Write path (publish): The client encodes camera at 180p, 360p, and 720p simultaneously and sends all three as separate RTP streams (SSRCs) to the SFU. The SFU receives TWCC feedback from each subscriber and selects the appropriate layer per subscriber.

Read path (subscribe): For each subscribed track, the SFU forwards the selected layer’s packets, rewriting RTP sequence numbers and timestamps so the decoder sees one continuous timeline⁵. Subscribers signal maxHeight per source so the SFU can drop high layers early.

Async path: Recording captures per-track to S3; a batch compositor renders the final video after the meeting ends. ASR receives a mixed audio feed and emits caption partials over the signaling channel.

Join-time budget: A healthy join must fit DNS, TLS, signaling, ICE, DTLS, and first media within ~3 seconds or users assume the app is broken.

gantt
    dateFormat X
    axisFormat %L ms
    title Join-time latency waterfall (budget ~3000 ms)
    section Setup
    DNS + TLS handshake       : 0, 300
    Signaling WS + SDP        : 300, 900
    ICE candidate gather      : 400, 1800
    DTLS handshake            : 1800, 2200
    First SRTP audio packet   : 2200, 2500
    First SRTP video packet   : 2500, 3000

Deep Dives

SFU simulcast and layer selection

The core quality mechanism in a production SFU is simulcast with per-subscriber layer selection. The publisher encodes the same camera at 2-3 resolutions (e.g., 180p at 15 fps, 360p at 30 fps, 720p at 30 fps) and uploads all layers simultaneously, costing 2-3x the uplink of a single stream⁶.

For each subscriber, the SFU picks the layer that fits within the subscriber’s estimated downlink capacity. A subscriber viewing a thumbnail gets 180p. The active speaker view gets 720p. Jitsi’s allocation algorithm prioritizes sources by speech activity (dominant speaker first), applies a LastN cap to drop non-visible sources entirely, then greedily allocates layers within the remaining bandwidth budget⁷.

sequenceDiagram
    participant Pub as Publisher
    participant SFU
    participant SubA as Subscriber A<br/>(good link)
    participant SubB as Subscriber B<br/>(weak link)
    Pub->>SFU: SSRC1 (180p, 15fps)
    Pub->>SFU: SSRC2 (360p, 30fps)
    Pub->>SFU: SSRC3 (720p, 30fps)
    SubA-->>SFU: TWCC feedback: 8 Mbps capacity
    SubB-->>SFU: TWCC feedback: 500 kbps, 3% loss
    SFU->>SubA: Forward SSRC3 (720p), rewrite to output SSRC
    SFU->>SubB: Forward SSRC1 (180p), rewrite to output SSRC
    Note over SFU: Layer switch rewrites RTP seq, ts, VP8 PictureID

When the SFU switches layers (e.g., from 720p to 360p because the subscriber’s link degraded), it rewrites RTP SSRC, sequence number, timestamp, and VP8 PictureID/TL0PICIDX so the decoder sees one continuous stream⁵. This rewriting is compatible with E2EE because these fields are packet metadata outside the encrypted frame payload.

SVC alternative: Scalable Video Coding (VP9-SVC, AV1-SVC) encodes a single hierarchical stream where the SFU simply drops higher layers without rewriting. AV1 offers 30-50% better compression than predecessors like VP9 and H.264⁸ and Google Meet has deployed it to support video at 40 kbps⁹. The trade-off: SVC decoder support remains uneven on older browsers, while simulcast VP8/H.264 works everywhere⁶.

Bandwidth estimation with GCC and TWCC

Without knowing each subscriber’s available bandwidth, the SFU cannot pick the right layer. Google Congestion Control (GCC) solves this by inferring capacity from one-way delay gradients¹⁰.

Transport-Wide Congestion Control (TWCC) is the current WebRTC default. Every outgoing RTP packet carries a transport-wide sequence number. The receiver sends periodic RTCP feedback (“I received packet X at time Z”) roughly every 100 ms, bounded between 50-250 ms intervals¹⁰¹¹. The sender computes per-packet delay gradients and loss rates, feeds them into a trendline filter and overuse detector, and an AIMD controller produces a target send rate.

TWCC adapts within approximately one second of a link change¹⁰. This is the difference between “video froze for 5 seconds” and “video got slightly softer for a moment.” Signal found that only two open-source SFUs had “adequate congestion control” when they evaluated options, which motivated their Rust rewrite⁵.

The SFU uses TWCC-derived targets to drive the allocation loop:

1. Order sources by priority (active speaker first)
2. Apply LastN to drop non-visible sources
3. For each remaining source, select the highest layer that fits within the subscriber’s remaining bandwidth budget⁷

End-to-end encryption with SFU forwarding

E2EE in a video conferencing system means the SFU forwards opaque ciphertext and cannot decode, transcode, record, or caption the media. WebRTC’s Encoded Transforms (previously Insertable Streams) expose encoded frames to JavaScript before packetization, allowing the app to encrypt frame payloads while leaving RTP headers in plaintext for the SFU to forward¹²¹³.

Signal’s implementation generates a per-client frame-encryption key at join time. The key is distributed to all participants via Signal’s existing E2EE messaging channel. On any join or leave, every client generates a fresh key and begins using it 3 seconds later, ensuring forward secrecy (joiners cannot decrypt prior media) and post-compromise security (leavers cannot decrypt subsequent media)⁵.

sequenceDiagram
    participant A as Client A
    participant B as Client B
    participant C as Client C (joining)
    participant SFU
    Note over A,B: Encrypting with K1
    C->>SFU: join request
    SFU-->>A: "C joined"
    SFU-->>B: "C joined"
    A->>B: K2 (via E2EE chat)
    A->>C: K2 (via E2EE chat)
    B->>A: B_new_key
    B->>C: B_new_key
    Note over A,C: Wait 3 seconds for key delivery
    A->>SFU: frames encrypted with K2
    B->>SFU: frames with B_new_key
    C->>SFU: frames with C_key
    Note over SFU: SFU forwards ciphertext only

The critical trade-off: enabling E2EE disables server-side transcription, recording composition, noise suppression, and AI summarization because all require plaintext media. Microsoft Teams documents this explicitly: enabling meeting E2EE disables transcription and recording¹⁴. Zoom rolled out E2EE in October 2020 amid FTC scrutiny for misrepresenting encryption (the FTC settlement was announced in November 2020 and finalized in February 2021)¹⁵; IACR researchers later described impersonation attacks against Zoom’s E2EE design (mostly requiring insider collusion with meeting participants), though the authors themselves noted these were “not an immediate threat” to Zoom’s E2EE¹⁶.

Verdict: Scope E2EE narrowly (1:1 calls, sensitive meetings) and surface the trade-off in product UX. Most users prefer transcription and recording over zero-trust encryption.

Real-World Example

Google Meet: scaling 30x in three months during COVID-19.

Google Meet is built entirely on WebRTC. When COVID-19 hit in early 2020, usage grew 30x in three months to approximately 100M daily meeting participants⁴¹⁷. The SRE team organized an incident response structure with distinct workstreams for capacity, dependencies, bottlenecks, “control knobs” (mitigations), and production changes¹⁷.

The first mitigation was a single switch: force default video from HD to SD globally. This one control knob cut per-session bandwidth roughly in half and bought days of capacity runway while new servers provisioned¹⁷.

The second insight was counter-intuitive: “fatter” tasks (4x CPU and RAM per container) handled approximately 1.8x the requests of four baseline containers. Fixed overhead (connection keepalives, logging, monitoring sockets) amortized better on larger instances¹⁷. This is a general lesson for stateful media servers.

Meet’s investment in AV1 encoding paid off for bandwidth-constrained users. AV1 produces usable video at 40 kbps⁹, reaching users on 2G cellular who previously could not sustain any video call. Meta has similarly deployed AV1 for mobile RTC where hardware decoding is available¹⁸.

The architecture shares codec and transport investments with every other Google WebRTC service. Improvements to GCC/TWCC flow into Meet automatically, and vice versa.

Trade-offs

The single biggest meta-decision: SFU versus MCU. SFU wins for interactive meetings because it avoids the encode/decode round-trip latency and keeps server CPU low. MCU wins only when subscriber bandwidth is the binding constraint (dial-in, 2G, VDI thin clients). Every major platform (Zoom, Meet, Teams, Signal) runs SFU-first and bolts MCU behavior on for edge cases²⁵.

Scaling and Failure Modes

At 10x (100M concurrent): SFU nodes saturate. A single node handles 500-2,000 participants sharded by meeting_id. Scale horizontally; keep a meeting on one node when possible. For large meetings (100+ participants), cascade across multiple SFU nodes with inter-node relay over a private backbone¹⁹.

At 100x (1B concurrent): TURN bandwidth becomes the dominant cost. Deploy TURN globally using turns on TLS/443 to survive aggressive middleboxes³. Regional imbalance forces dynamic meeting migration: if a meeting’s participant distribution shifts (3 US, 17 EU), migrate the meeting’s primary SFU to the majority region.

At 1000x: The architecture shifts to CDN-backed broadcast for large events (LL-HLS/WHEP for 10K+ viewers) while keeping SFU for interactive participants.

Failure: SFU node crash mid-meeting. Participants see a freeze. Clients detect via RTCP timeout, re-signal to a healthy node, and re-ICE within 3-5 seconds. Meeting state in Redis enables stateless failover.

Failure: Asymmetric packet loss on one publisher. Everyone else sees that participant freeze. Mitigation: publisher-side FEC, simulcast layer drop to lowest resolution, and a “your connection is unstable” client banner.

Failure: TURN cluster saturation during corporate firewall rollout. Relay rate spikes from 5% to 30%+. Mitigation: capacity-plan TURN as a first-class line item; auto-scale TURN nodes per POP based on relay session count.

Common Pitfalls

Follow-up Questions

1. How do you add E2EE without losing SFU selective forwarding?

Use Encoded Transforms to encrypt frame payloads while leaving RTP headers (SSRC, seq, timestamp) in plaintext. The SFU rewrites headers for layer switching without touching ciphertext. Rotate keys on every join/leave with a 3-second grace period for delivery⁵.

2. How does a 1,000-person webinar differ from a 20-person meeting?

Split into publishers (panelists, SFU-connected) and viewers (receive-only, ABR unicast or LL-HLS). Viewers get a single composited stream from an MCU-like compositor. Only panelists have bidirectional media. Microsoft Teams uses this exact split for town halls²¹.

3. What is the minimum viable PSTN dial-in?

SIP gateway terminates the phone call, transcodes G.711 to Opus, and joins the SFU as a virtual participant. Audio-only, no video. The MCU mixes the dial-in participant’s audio into the room and sends them a mixed-minus feed (everyone except themselves).

4. How do you handle a participant whose camera is broken (black frames)?

Detect stuck frames via RTP timestamp analysis (no new frames for >2 seconds). Notify the client with a “camera may be off” banner. Suspend the track’s simulcast layers to free bandwidth for other participants.

5. What changes for server-side noise suppression?

The SFU routes audio through an ML pipeline (RNNoise or Krisp) before forwarding. This adds 10-20 ms of latency but dramatically improves audio quality in noisy environments. Incompatible with E2EE because the server needs plaintext audio frames.

6. How do you handle cross-region meetings (Tokyo to Sao Paulo)?

Each participant connects to their nearest SFU POP. The SFUs relay between each other over a private backbone. Worst-case audio path stays under 200 ms. The meeting controller tracks participant distribution and can migrate the “primary” SFU to the majority region¹⁹.

Exercise

Exercise 1: Bandwidth budget for a 50-person all-hands

Your company runs a weekly 50-person all-hands. Each participant publishes 3 simulcast layers (180p at 150 kbps, 360p at 500 kbps, 720p at 1.5 Mbps). The active speaker view shows 1 person at 720p and 5 at 360p; remaining 43 are thumbnails at 180p. Calculate the per-subscriber downlink requirement.

Key Takeaways

• SFU + simulcast is the default. P2P is for 1:1; MCU is for bridging and broadcast; everything else is SFU with per-subscriber layer selection.
• TWCC drives quality. Bandwidth estimation and the allocation loop are what separate a production SFU from a toy. Codec choice is secondary.
• Route on meeting, not user. Media locality beats user locality. A fragmented meeting across regions tanks audio quality.
• Decouple recording and captions. They are critical features but must never stall the SFU media hot path.
• E2EE is a product trade-off, not a free switch. It disables transcription, recording, and AI features. Scope it narrowly.
• Budget TURN explicitly. 5-10% of sessions need relay, but it is often the single largest bandwidth cost line.

Further Reading

• How to build large-scale end-to-end encrypted group video calls (Signal). The single best primary source on combining SFU forwarding, simulcast, TWCC, and SFrame E2EE in production Rust.
• Three months, 30x demand: How we scaled Google Meet (Google Cloud). SRE workstreams, capacity forecasting, “fatter tasks,” and fire-escape control knobs during COVID.
• A technical guide to the Zoom Web SDK (Daily). The clearest explanation of Zoom’s proprietary H.264+SVC stack and how it differs from WebRTC.
• Jitsi Videobridge bandwidth allocation algorithm. The production SFU rate-allocation algorithm: priority ordering, LastN, and greedy layer selection.
• Transport-Wide Congestion Control (IETF Internet-Draft). The canonical description of TWCC’s sender-side bandwidth estimation.
• RFC 6716: Opus audio codec. The mandatory WebRTC audio codec: 6-510 kbps, 5-66.5 ms algorithmic delay.
• Improved video calling with faster AV1 encoding (Chrome). Google Meet’s AV1 deployment enabling video at 40 kbps for bandwidth-constrained users.
• LiveKit SFU internals. Concise P2P vs MCU vs SFU framing plus LiveKit’s horizontal-scaling model.

Flashcards

References