Load Balancer vs Reverse Proxy vs API Gateway

TL;DR. These three overlap in implementation (NGINX, Envoy, and HAProxy can play any role) but differ in intent. A load balancer distributes connections across identical replicas at sub-millisecond cost¹. A reverse proxy terminates TLS, compresses, and buffers at the HTTP layer. An API gateway adds per-route policy: auth, rate limiting, and request transformation at ~5-9 ms p99 overhead². Default to a cloud L4 LB plus a thin reverse proxy. Add a gateway only when you need per-consumer policy across 10+ services.

Learning Objectives

• Compare L4 load balancers, L7 reverse proxies, and API gateways across latency, visibility, and operational cost.
• Identify the workload characteristics that make each layer necessary or redundant.
• Justify a hybrid layering (L4 + gateway + mesh) for a production edge.
• Evaluate real-world systems (Cloudflare Unimog, Netflix Zuul, HAProxy) and explain their architectural choices.

The Core Trade-off

The fundamental tension is visibility versus speed. An L4 load balancer sees only the 5-tuple (source IP, source port, dest IP, dest port, protocol). It forwards packets in kernel space via XDP/eBPF at less than 1% CPU overhead¹. A reverse proxy terminates TLS, parses HTTP headers, and makes per-request routing decisions, adding ~400 microseconds at saturation³. An API gateway executes a filter chain (auth, rate-limit, transform) on every request, adding ~5-9 ms p99². At the extreme, a service mesh sidecar adds ~2.65 ms P90 total across both sidecars in a single service-to-service call (Istio 1.10)⁴, meaning each hop adds roughly half that⁵.

Each layer up the stack gains expressiveness but pays in latency, CPU, and failure surface. The wrong pick shows up as either reimplementing features the right tool already provides or paying for features you never use.

A second tension is blast radius. An L4 LB failure drops connections but has no application state to corrupt. A misconfigured API gateway is a single point of failure for every route behind it, and a bad regex in a rate-limit plugin can take down your entire public API surface.

flowchart LR
    subgraph L4["L4 Load Balancer"]
        direction TB
        A1[Packet forwarding]
        A2[Health checks]
        A3[Source IP preserved]
    end
    subgraph L7["Reverse Proxy"]
        direction TB
        B1[TLS termination]
        B2[HTTP routing]
        B3[Compression]
    end
    subgraph GW["API Gateway"]
        direction TB
        C1[Auth / JWT]
        C2[Rate limiting]
        C3[Request transform]
    end
    L4 -->|"<1% CPU"| L7
    L7 -->|"+400us"| GW
    GW -->|"+5-9ms p99"| Upstream[Service]

Side-by-Side Comparison

The table misleads on one dimension: throughput ceiling. A gateway’s 100K RPS per node sounds low, but you rarely need gateway-level policy on every request. The L4 LB absorbs DDoS and distributes; only routed traffic hits the gateway. In practice, the gateway is not your throughput bottleneck unless you put it where the L4 LB should be.

The dimension that dominates in interviews: gRPC. An L4 LB cannot balance gRPC streams because HTTP/2 multiplexes many RPCs over one TCP connection. One backend gets all the load³. This single fact forces L7 for any gRPC-heavy architecture.

When to Pick a Load Balancer

Use a pure L4 load balancer when:

• Traffic is TCP/UDP and you need raw distribution. Database connections, Redis clusters, DNS, game servers. AWS NLB handles 3M+ RPS at 30 Gbps with native source-IP preservation⁶.
• DDoS absorption is the primary concern. Cloudflare runs Unimog on every server across its 335+ city network; L4drop in the same XDP chain absorbs volumetric attacks before they reach application code¹⁷.
• You need connections that persist for hours or days. VPN tunnels, WebSocket pass-through, Cloudflare Spectrum. L4 preserves the end-to-end TCP session¹.
• Sub-millisecond overhead is a hard requirement. Latency-sensitive trading systems, real-time bidding, intra-cluster east-west traffic where every microsecond counts. Round Robin at L4 is per new connection, not per request⁸.

When to Pick an API Gateway

Use an API gateway when:

• Many services, one public hostname. You have 10+ microservices behind api.example.com and need per-route auth, rate limiting, and versioning. This is the gateway’s natural habitat⁹.
• Per-consumer policy is a business requirement. API keys, usage plans, per-tenant rate limits, developer portals. Netflix routes all external traffic through Zuul for exactly this reason⁹.
• Request transformation or protocol bridging. REST-to-gRPC translation, GraphQL federation, response aggregation across multiple upstreams.
• You are selling API-as-product. Stripe, Twilio, and every API-first company runs a gateway with usage metering, quota enforcement, and key management.

Do not reach for a gateway when a reverse proxy suffices. Kong with zero plugins still adds ~5 ms p99². If you have one service and need only TLS termination and path rewriting, NGINX or Caddy costs less in latency and operations.

The Hybrid Path

Most production systems layer all three with distinct responsibilities:

1. Cloud L4 LB (NLB, Unimog): DDoS absorption, health-based distribution, source-IP preservation.
2. API Gateway (Kong, Envoy Gateway, Zuul): per-route auth, rate limiting, request transformation.
3. Service mesh sidecar (Envoy via Istio): east-west mTLS, retries, circuit breaking, telemetry.

Each layer owns one concern. The anti-pattern is stacking them without distinct responsibilities: NLB, then ALB, then NGINX, then Envoy sidecar, each adding 1-3 ms. If your distributed trace shows proxy hops consuming 20%+ of end-to-end latency, you have redundant layers.

Istio’s Gateway API can subsume a separate ingress NGINX for mesh-native apps¹⁰. If you already run a mesh, evaluate whether the mesh ingress replaces your standalone gateway before adding another hop.

flowchart LR
    Client -->|HTTPS| L4[L4 LB / NLB]
    L4 --> GW[API Gateway]
    GW --> Sidecar1[Envoy Sidecar]
    Sidecar1 --> SvcA[Service A]
    SvcA --> Sidecar2[Envoy Sidecar]
    Sidecar2 --> SvcB[Service B]

Real-World Examples

Cloudflare Unimog (L4). Every server in Cloudflare’s 335+ city network acts as a load balancer⁷. An XDP/eBPF program hashes the 4-tuple, looks up a forwarding table, and GUE-encapsulates the packet to the chosen backend. CPU overhead: less than 1%¹. The system supports connections persisting for days via a two-slot daisy-chaining technique (current DIP and previous DIP per bucket). A conductor control plane reads Prometheus metrics and adjusts bucket counts so heterogeneous server generations converge to equal utilization¹.

Netflix Zuul 2 (API Gateway). All external API traffic (83M+ members at the time of the 2016 post) passes through Zuul clusters⁹. Zuul 1 used one thread per connection (Servlet model). Zuul 2 rewrote on Netty for persistent device connections, gaining ~25% throughput improvement on logging-heavy clusters. Netflix is explicit: “we did not see a significant efficiency benefit in migrating to async”⁹. The real win was connection scaling, not CPU. Filters are hot-reloadable Groovy scripts, letting Netflix update routing logic in minutes across thousands of hosts⁹.

HAProxy (Reverse Proxy benchmark). On a 64-core AWS Graviton2 instance, HAProxy 2.3 forwards 2.04M RPS in HTTP mode, adding ~400 microseconds average latency. With TLS 1.3 (RSA-2048): 1.99-2.01M RPS at ~413 microseconds³. Envoy’s P2C (Power of Two Choices) algorithm picks two hosts at random and routes to the less loaded one, achieving near-optimal distribution at O(1) cost¹¹. The benchmark proves that a single reverse proxy node handles more traffic than most companies will ever see.

Common Mistakes

Decision Checklist

• Is the primary need “distribute across identical replicas” (L4) or “per-route policy” (gateway)?
• Does the protocol require HTTP-level visibility (gRPC, path routing, header inspection)?
• Do you need per-consumer auth, API keys, or usage plans?
• Is TLS terminated at the edge, end-to-end, or re-encrypted via mesh mTLS?
• How many distinct services sit behind this entry point? (1-3: proxy. 10+: gateway.)
• Does your distributed trace show proxy hops consuming >15% of end-to-end latency?

flowchart TD
    Start[Incoming traffic] --> Q1{Need HTTP path routing or header inspection?}
    Q1 -->|No| Q2{Need TLS termination?}
    Q1 -->|Yes| L7[L7 reverse proxy or gateway]
    Q2 -->|No| L4[L4 load balancer]
    Q2 -->|Yes| L7TLS[L7 with TLS termination]
    L7 --> Q3{Per-route auth, rate-limit, or transform?}
    Q3 -->|Yes| GW[API Gateway]
    Q3 -->|No| RP[Reverse Proxy]
    L7TLS --> Q3

Key Takeaways

• L4 load balancers cost sub-millisecond overhead and handle millions of RPS but are blind to HTTP semantics.
• Reverse proxies add ~400 microseconds for TLS termination, compression, and per-request routing.
• API gateways add ~5-9 ms p99 for auth, rate limiting, and transformation; use them only when per-route policy justifies the cost.
• gRPC and HTTP/2 force L7 balancing because L4 cannot distribute multiplexed streams.
• The production default is L4 for distribution, gateway for policy, mesh for east-west mTLS. Remove any layer that does not own a distinct concern.

Further Reading

• Cloudflare Unimog: the most detailed public write-up of a production L4 LB running on every server via XDP/eBPF.
• Netflix Zuul 2: Journey to Asynchronous, Non-Blocking Systems: honest post-migration writeup; read for the “async did not save us CPU” conclusion.
• HAProxy Forwards Over 2M RPS on Graviton2: benchmark methodology is the best practical reference on how to run a proxy benchmark correctly.
• Envoy Supported Load Balancers: canonical reference for P2C, Ring Hash, Maglev, and weighted variants.
• Kong Gateway Performance Benchmarks: reproducible RPS and p99 numbers per plugin configuration; useful for sizing gateways.
• Istio Performance and Scalability: sidecar CPU, memory, and latency numbers across releases.

Flashcards

References