Load Balancers: Spreading Traffic, Absorbing Failure

TL;DR: A load balancer accepts a virtual IP, picks a backend, and quietly removes the ones that stop responding. Layer 4 forwards packets by 5-tuple at line rate: Google Maglev saturates a 10 Gbps link on a single machine¹. Layer 7 terminates TCP, parses HTTP, and routes on headers, at the cost of CPU per request. Most production stacks run both: a stateless L4 tier absorbs raw packet volume, and an L7 tier adds routing, retries, and observability².

Learning Objectives

After this module, you will be able to:

• Choose between L4 and L7 load balancing for a given workload
• Explain the trade-offs of round robin, least-connections, consistent-hash, and power-of-two-choices algorithms
• Design health checks that catch real failures without flapping
• Reason about sticky sessions, connection draining, and graceful deploys
• Describe how anycast and DNS-based GSLB route traffic globally
• Compare HAProxy, NGINX, Envoy, and cloud load balancers on features, throughput, and cost

Intuition

Imagine you arrive at an airport terminal with 20 boarding gates. A single gate agent checks every passenger’s ticket, assigns them a gate, and redirects anyone whose gate is closed. Without her, passengers would crowd gate 1 while gates 2 through 20 sit empty.

She does three jobs. First, she distributes passengers across gates so no single gate is overwhelmed. Second, she monitors gate status: if a gate’s jetbridge breaks, she stops sending people there. Third, she handles the paperwork: scanning boarding passes, verifying IDs, and handing back a simplified stub so the gate agent downstream does less work.

A load balancer does the same three jobs for network traffic: distribution, health tracking, and termination (TLS, protocol parsing). The simplest version just reads the destination on your ticket (the 5-tuple) and points you to a gate. A smarter version reads your frequent-flyer status, your connection time, and your luggage weight before deciding. The simple version is faster. The smart version makes better decisions. Production airports use both: a fast first pass at the terminal entrance, then a smarter second pass at the gate cluster.

That is the L4/L7 split in one analogy. The rest of this chapter makes it concrete.

Theory

Layer 4 vs layer 7

An L4 load balancer operates on TCP/UDP packets. It hashes the 5-tuple (source IP, source port, destination IP, destination port, protocol) into a backend slot and forwards the packet without ever reading the payload. Responses typically bypass the LB entirely via Direct Server Return (DSR), so the LB only touches ingress traffic¹³.

An L7 load balancer terminates the TCP connection, decrypts TLS, parses HTTP (or gRPC, WebSocket, HTTP/2), and opens a new connection to the chosen backend. This lets it route by path, header, cookie, or method, retry failed requests, inject observability headers, and enforce per-tenant rate limits⁴⁵.

The cost is CPU. Every request parsed is a request the LB must buffer, decrypt, and re-serialize. L7 proxies top out an order of magnitude lower on requests per second than L4 forwarders on the same hardware.

flowchart LR
    subgraph L4["Layer 4 (packet forwarding)"]
        C1([Client]) -->|TCP packet| LB4[L4 LB<br/>hash 5-tuple]
        LB4 -->|encapsulated packet| B1[Backend]
        B1 -->|direct reply DSR| C1
    end
    subgraph L7["Layer 7 (proxy)"]
        C2([Client]) -->|TLS handshake| LB7[L7 LB<br/>terminate TLS<br/>parse HTTP]
        LB7 -->|new TCP conn| B2[Backend]
        B2 -->|HTTP response| LB7
        LB7 -->|HTTP response| C2
    end

Use L4 when you need raw throughput, protocol-agnostic forwarding, or a DDoS absorption tier. Use L7 when you need path-based routing, retries, auth, or per-request metrics. Most production stacks use both: Cloudflare runs Unimog (L4, XDP) in front of Pingora (L7)²⁶; AWS customers put NLB in front of ALB or Envoy⁷.

Algorithms

Round robin walks an ordered list. It is dead simple and distributes evenly across homogeneous hosts. It ignores actual load: a stalled host keeps receiving traffic until a health check removes it⁸.

Weighted round robin repeats higher-weight hosts proportionally. Use it when backends have different capacities (mixed-generation hardware).

Least-connections / least-outstanding-requests picks the host with the fewest in-flight requests. The naive version herds: when many LB instances simultaneously pick the same “least loaded” backend, that backend spikes.

Power of two choices (P2C) fixes the herding problem. Pick two backends at random; send the request to whichever has fewer outstanding requests. Mitzenmacher’s 2001 result proves that d=2 choices gives an exponential improvement over d=1 (random), while d=3 gives only a constant factor further improvement⁹. This is why Envoy, NGINX (via its random two directive), Netflix Zuul, and Finagle all offer P2C as a load balancing option⁴⁸.

Consistent hashing routes requests by hashing a stable key (user ID, cache key, 5-tuple) into a server slot. When one of N backends is removed, roughly 1/N of keys move. Google’s Maglev hash fills a 65,537-slot lookup table via a permutation algorithm; packet-path selection is a single modulo and array lookup¹¹⁰.

flowchart TD
    P[Incoming packet<br/>5-tuple hash H] --> M{"H mod 65,537"}
    M -->|slot 0| A[Backend A]
    M -->|slot 1| B[Backend B]
    M -->|slot 2| A
    M -->|slot 3| C[Backend C]
    M -->|slot ...| X["... 65,532 more slots ..."]
    M -->|slot 65536| C

Use round robin for homogeneous stateless services. Use P2C least-request as the default for heterogeneous or latency-sensitive workloads. Use consistent hash only when you need cache affinity or flow pinning, and accept that hot keys can overload one backend¹¹.

Health checks and outlier detection

Active health checks probe backends on a schedule: a TCP connect, an HTTP GET to /healthz, or a gRPC health RPC. They find dead backends fast even when traffic is low¹².

Passive (outlier) health checks observe real request traffic. Envoy’s default outlier detector ejects a host after 5 consecutive 5xx responses for a base ejection time of 30 seconds, with linear backoff on re-ejection¹²¹³. Passive checks catch gray failures (slow responses, partial errors) that active checks miss.

Connection draining stops sending new connections to a backend while existing requests finish. In Kubernetes, the preStop hook sleeps for the LB’s deregistration delay before the container receives SIGTERM¹⁴¹⁵.

Slow start ramps traffic to a newly launched backend over a configurable window. Netflix uses 90 seconds to avoid overloading a cold JVM⁸.

stateDiagram-v2
    [*] --> Probation: registered
    Probation --> Healthy: first successful response
    Probation --> Unhealthy: failed probe
    Healthy --> Unhealthy: N consecutive 5xx
    Unhealthy --> Healthy: M consecutive 2xx
    Healthy --> Draining: pre-stop / deregister
    Draining --> Removed: timeout or zero in-flight
    Unhealthy --> Removed: ejected and TTL expired
    Removed --> [*]

Session persistence

Sticky sessions bind a client to a specific backend via a cookie, header, or source-IP hash. AWS ALB uses an AWSALB cookie¹⁶; NLB pins each TCP connection via a 6-tuple flow hash that adds the TCP sequence number, so separate connections from one client can land on different targets¹⁷.

Global load balancing

Above a single datacenter, traffic is steered globally by two mechanisms:

BGP anycast advertises the same VIP from every POP. Internet routers converge on the nearest location for each client. Cloudflare operates in 330+ cities with 500 Tbps of external capacity¹⁸. When a POP goes dark, it withdraws its BGP route and traffic reconverges in seconds, with no DNS TTL delay².

DNS-based GSLB (Route 53, NS1) returns different A records based on client subnet, RTT, or weighted policy. It is coarser: stale caches at ISPs can pin users to dead regions for minutes. Only new DNS lookups are affected; existing TCP connections stay put¹⁹.

flowchart LR
    U1([User EU]) -->|BGP anycast| P1[POP London<br/>L4 LB]
    U2([User US]) -->|BGP anycast| P2[POP Ashburn<br/>L4 LB]
    U3([User APAC]) -->|BGP anycast| P3[POP Tokyo<br/>L4 LB]
    P1 --> B1[Backend pool]
    P2 --> B2[Backend pool]
    P3 --> B3[Backend pool]

Use anycast when you need sub-second failover and can operate BGP. Use DNS GSLB when you need geographic steering without BGP infrastructure, and accept minutes-long failover on stale caches.

TLS termination

Three patterns dominate:

1. Termination at the LB. The LB decrypts, inspects HTTP, and re-originates to the backend (cleartext or re-encrypted). This is the default for AWS ALB, Cloudflare’s L7 tier, and Envoy ingress. It enables header routing, WAF, and retries, but puts all private keys on the LB²⁰.
2. Passthrough. The LB forwards encrypted bytes untouched (L4 or SNI-based routing). AWS NLB TLS listener and Istio PASSTHROUGH mode fit here. No HTTP routing, but end-to-end encryption is preserved²¹.
3. mTLS in a service mesh. Istio and Linkerd terminate outer TLS at the sidecar and re-originate mTLS to the upstream sidecar. Automatic cert rotation, zero-trust between services²¹.

Use termination when you need L7 features. Use passthrough when compliance requires end-to-end encryption. Use mTLS when you need authenticated service-to-service communication.

Real-World Example

Cloudflare Unimog: every server is a load balancer

Cloudflare’s Unimog is an L4 load balancer deployed across their entire edge network: 330+ cities, 500 Tbps of capacity¹⁸. Its defining design choice is that every server in a datacenter is simultaneously a load balancer and an application server. There is no separate LB tier.

How it works. Routers ECMP packets to any server. An XDP program chain runs per packet: l4drop (DDoS mitigation) first, then Unimog. Unimog hashes the 5-tuple, looks up a forwarding table with tens of thousands of buckets (roughly 100x the server count), and GUE-encapsulates the packet to the target server².

Connection persistence. When the forwarding table changes (server added or removed), Unimog uses “daisy chaining” borrowed from the Beamer paper: each bucket has two slots (current and previous). If the current server has no matching TCP socket, it forwards to the previous server. Less than 1% of packets take this “second hop”².

CPU overhead. Less than 1% of each server’s CPU is spent on load-balancing work. XDP runs before the Linux kernel network stack touches the packet, so the cost is near zero².

The oscillation incident. Early on, Cloudflare hit a feedback-loop problem: when a datacenter became overloaded and the control plane (conductor) diverted new connections to less-loaded servers, those servers then degraded while the original recovered, causing oscillation. The fix: teach the conductor to distinguish individual-server degradation from datacenter-wide saturation².

DDoS absorption. In 2025, Cloudflare mitigated a 31.4 Tbps DDoS attack through the l4drop/Unimog chain with no human intervention¹⁸.

The lesson: a stateless L4 LB that runs on every server eliminates the “LB tier” as a separate capacity-planning problem. The LB scales with the fleet.

Trade-offs

Where does the load balancer sit?

Which algorithm picks the backend?

Common Pitfalls

Exercise

Design the load balancing tier for a public API that serves 200k RPS across 80 stateless pods, needs TLS termination, path-based routing to four services, per-tenant rate limiting, and zero-downtime deploys. Decide between ALB, NLB + Envoy, or NGINX, and justify the choice against cost, feature fit, and operational burden.

Key Takeaways

• L4 is fast and dumb; L7 is smart and expensive. Most production stacks use both: L4 in front for packet volume, L7 behind for routing intelligence.
• P2C least-request is the best default algorithm. It self-balances, resists herding, and handles heterogeneous backends without configuration.
• The load balancing algorithm matters less than good health checks and connection draining. A perfect algorithm with broken health checks still sends traffic to dead servers.
• Sticky sessions are a smell. Externalize state to Redis or a JWT; use consistent hash only for cache affinity where losing affinity degrades latency, not correctness.
• Anycast with stateless L4 gives sub-second global failover. DNS-based GSLB is simpler but fails over in minutes due to TTL caching.
• Every server can be a load balancer. Cloudflare’s Unimog proves that eliminating a separate LB tier removes a capacity-planning problem at less than 1% CPU overhead².
• Enable SO_REUSEPORT on any multi-core proxy. It is one config line that eliminates accept-queue contention and can double throughput²⁴.

Further Reading

• Maglev: A Fast and Reliable Software Network Load Balancer (Eisenbud et al., NSDI 2016) - The foundational L4-at-scale paper; introduces the Maglev hash and explains how Google serves all production traffic without shared LB state.
• Unimog: Cloudflare’s Edge Load Balancer (2020) - The most detailed public write-up of a global L4 LB, including daisy-chaining for connection persistence and the oscillation incident.
• Open-sourcing Katran (Meta Engineering, 2018) - How XDP+eBPF lets an L4 LB run colocated with backend applications at 10M+ pps per core.
• Rethinking Netflix’s Edge Load Balancing (2018) - P2C plus server-reported utilization with real production incident data; the best public case for why round robin fails.
• The Power of Two Random Choices: A Survey of Techniques and Results (Mitzenmacher, 2001) - The theoretical basis cited by every major modern LB; proves d=2 is exponentially better than d=1.
• Envoy Load Balancer Documentation - The reference L7 LB’s menu of algorithms (ring hash, Maglev, P2C, weighted round robin) and why each exists.
• Socket Sharding in NGINX 1.9.1 (F5/NGINX, 2015) - The SO_REUSEPORT benchmark that unlocks multi-core scaling; one of the highest-ROI tuning changes for any proxy.
• HAProxy Forwards Over 2M RPS on a Single Arm Instance (2021) - A public software LB benchmark near the top of what one box can do; useful for capacity planning.
• Consistent Hashing and Random Trees (Karger et al., STOC 1997) - The original consistent hashing paper; prerequisite reading for understanding Maglev, ring hash, and jump hash.
• GLB: GitHub’s Open Source Load Balancer (2018) - Rendezvous hashing and the “second chance” technique for stateless connection persistence; a good complement to Unimog.

Flashcards

References