Service Discovery and Service Mesh: Finding and Talking to Services

TL;DR: Service discovery answers “where is service B right now?” for every inter-service call. Solutions range from DNS with short TTLs, to registries like Consul and Kubernetes Services, to mesh-based sidecars that resolve endpoints locally. A service mesh extends discovery by inserting a proxy (Envoy, Linkerd2-proxy) next to every pod, giving you mTLS, retries, circuit breaking, and distributed tracing without changing application code. The cost is real: Istio sidecars consume ~0.20 vCPU and 60 MB RAM per pod at 1,000 RPS¹. Do not adopt a mesh until you have 20+ services and a clear need for the full bundle; if you only need mTLS, cert-manager plus NetworkPolicies is cheaper.

Learning Objectives

After this module, you will be able to:

• Distinguish client-side, server-side, and mesh-based service discovery
• Explain how Kubernetes Services and EndpointSlices route traffic at scale
• Describe the data plane vs control plane split and the xDS API
• Decide whether a service mesh is worth its operational cost for your estate
• Configure mTLS, retries, timeouts, and circuit breaking at the mesh layer
• Compare Istio, Linkerd, and Cilium on performance, features, and overhead

Intuition

You start a new job at a 500-person company. You need to talk to someone in Finance. In a small startup, you would shout across the room. In a big company, you look them up in the corporate directory (the registry), get their desk number (the endpoint), and walk over. If they moved desks last week, the directory has the new location. If they are on vacation, the directory marks them unavailable so you do not waste a trip.

Now imagine the company also installs a receptionist outside every office door. The receptionist checks your badge (mTLS), logs your visit (tracing), retries if the person is briefly away (retries with budget), and turns you away if the person’s queue is too long (circuit breaking). You never learn the routing rules; you just say “I need Finance” and the receptionist handles it.

The corporate directory is service discovery. The receptionist is the sidecar proxy. The HR department that trains all receptionists and keeps their policies in sync is the control plane. That is the entire service mesh in one analogy.

Theory

Service discovery models

Three models exist for resolving a logical service name to a live instance:

Client-side discovery. The caller queries a registry (Eureka, Consul, ZooKeeper) and picks an instance itself. Netflix built Ribbon for this: the Java client fetched endpoints from Eureka and applied client-side load balancing². The advantage is no extra network hop. The disadvantage is that every language needs a mature client library, and Netflix eventually moved to Envoy precisely because maintaining Ribbon across polyglot services became untenable².

Server-side discovery. The caller hits a stable VIP or load balancer that knows the backends. AWS ALB with target groups and Kubernetes ClusterIP Services are canonical examples. Clients stay dumb. The cost is a proxy hop and a shared failure domain at the LB.

Mesh-based discovery. The caller sends to localhost; a sidecar proxy resolves the target using config streamed from a control plane. This is client-side discovery with a local proxy: the application stays trivial, but you pay the sidecar’s CPU and memory overhead.

flowchart LR
    subgraph CS["Client-side discovery"]
        CA[Service A] -->|"1. query registry"| Registry[Eureka / Consul]
        CA -->|"2. direct call"| CB[Service B]
    end
    subgraph SS["Server-side discovery"]
        SA[Service A] -->|"call VIP"| LB[Load Balancer]
        LB --> SB[Service B]
    end
    subgraph MB["Mesh-based discovery"]
        MA[Service A] -->|"localhost"| Sidecar[Sidecar Proxy]
        Sidecar -.->|"xDS"| CP[Control Plane]
        Sidecar -->|"mTLS"| SidecarB[Sidecar Proxy]
        SidecarB --> MBB[Service B]
    end

Use client-side when you control all callers and want latency-aware selection (gRPC with xDS, Finagle). Use server-side as the default for Kubernetes workloads. Use mesh-based when you need uniform policy across many languages.

Kubernetes Service model

Kubernetes Services are the native discovery mechanism for most cloud workloads. A ClusterIP Service gets a virtual IP; kube-proxy on every node programs iptables (or nftables, the recommended successor since Kubernetes 1.33) rules that DNAT traffic toward one of the backend pods³.

EndpointSlices became the default for kube-proxy in Kubernetes 1.19, superseding the monolithic Endpoints object. Before EndpointSlices, a Service with 5,000 pods on 3,000 nodes meant distributing a 1.5 MB Endpoints object per change, producing “more than 22 TB worth of data transferred” across the cluster during a rolling update⁴. EndpointSlices partition endpoints into shards of up to 100, scaling to 100,000+ network endpoints per Service.

At very large scale, kube-proxy rule programming is a known hotspot. Cilium replaces it entirely with eBPF maps for O(1) lookup per packet⁵.

Service mesh: data plane vs control plane

Matt Klein framed the canonical split in 2017: “the data plane is responsible for conditionally translating, forwarding, and observing every network packet… the control plane takes a set of isolated stateless sidecar proxies and turns them into a distributed system”⁶.

The data plane is the set of proxies on the request path. Envoy (C++, used by Istio and Consul Connect), Linkerd2-proxy (Rust, used by Linkerd), and ztunnel (Rust, used by Istio ambient mode).

The control plane (Istiod, Linkerd control plane, Consul servers, Google Traffic Director) generates configuration and streams it to proxies over the xDS suite of APIs: CDS (clusters), EDS (endpoints), LDS (listeners), RDS (routes)⁶. Each sidecar opens a gRPC stream to the control plane and receives eventually-consistent config updates. When a deployment scales up, the control plane watches Kubernetes and emits EDS updates to every affected proxy.

flowchart TD
    K8s[Kubernetes API Server] -->|"watch pods, services"| Istiod[Istiod / Control Plane]
    Istiod -->|"CDS/EDS/LDS/RDS"| P1[Envoy Sidecar A]
    Istiod -->|"CDS/EDS/LDS/RDS"| P2[Envoy Sidecar B]
    Istiod -->|"CDS/EDS/LDS/RDS"| P3[Envoy Sidecar C]
    subgraph PodA["Pod A"]
        AppA[App] -->|"localhost"| P1
    end
    subgraph PodB["Pod B"]
        AppB[App] -->|"localhost"| P2
    end
    P1 -->|"mTLS"| P2
    P1 -->|"mTLS"| P3

What meshes provide

A sidecar gives every service, in any language, the following without an SDK:

• Mutual TLS (mTLS) with automatic cert rotation (default 24-hour TTL in Istio)
• Retries with budgets to prevent retry storms from cascading
• Circuit breakers and outlier detection to eject unhealthy upstreams
• Distributed tracing headers propagated automatically (B3, W3C Trace Context)
• Traffic splitting for canary, blue-green, and A/B deployments
• Per-service authorization policies based on cryptographic identity (SPIFFE)

sequenceDiagram
    participant App as App A
    participant SA as Sidecar A
    participant SB as Sidecar B
    participant AppB as App B
    App->>SA: HTTP GET /api (localhost)
    SA->>SA: lookup cluster via xDS
    SA->>SA: add trace header, apply retry policy
    SA->>SB: mTLS handshake + request
    SB->>SB: verify SPIFFE ID, authz check
    SB->>AppB: HTTP GET /api (localhost)
    AppB-->>SB: 200 OK
    SB-->>SA: 200 OK (mTLS)
    SA-->>App: 200 OK
    Note over SA,SB: On 5xx, SA retries up to budget

Major meshes compared

Istio is the feature-rich option. CNCF graduated in July 2023⁷. Istio 1.24 load-tested at 1,000 services, 2,000 pods, and 70,000 mesh-wide RPS¹. It uses Envoy as the data plane and Istiod as the consolidated control plane. The downside is weight: at 1,000 RPS per sidecar, expect ~0.20 vCPU and 60 MB RAM¹.

Linkerd is the lightweight alternative. In the 2021 Kinvolk benchmark (Linkerd 2.11.1 vs Istio 1.12.0) at 2,000 RPS, Linkerd added 6 ms median latency versus Istio’s 17 ms, and consumed 26 MB per proxy versus Istio’s 156 MB⁸. Both projects have evolved significantly since; Istio ambient mode changes the comparison. Its data plane, Linkerd2-proxy, is a purpose-built Rust micro-proxy using Tokio and Rustls. The design choice: avoid garbage-collected languages to keep tail latency predictable⁹.

Cilium takes a different approach entirely. It uses eBPF programs attached to kernel socket hooks to enforce L4 policy and encryption (WireGuard or IPsec) without any proxy on the request path⁵. L7 features are weaker and kernel-version gated, but the overhead is near zero for L4-only workloads.

Consul Connect bridges the VM and container worlds. If you have a mixed estate (some Kubernetes, some bare-metal, some VMs), Consul’s registry-then-mesh path is the natural fit.

Overhead and alternatives

According to CNCF Annual Survey data, sidecar-based mesh adoption dropped from 50% to 42% between 2023 and 2024¹⁰. The reason is cost: sidecar overhead is roughly fixed per pod, independent of traffic. A side service doing 10 RPS with a 100 MB JVM heap now costs 160 MB total because Envoy adds 60 MB. Multiplied across hundreds of small services, cluster capacity vanishes.

Istio ambient mode splits the mesh: a ztunnel DaemonSet handles mTLS and L4 policy at ~0.06 vCPU and 12 MB per node, while optional waypoint proxies handle L7 routing only where needed¹¹¹. Measured against sidecar’s 0.20 vCPU and 60 MB per pod in the same Istio 1.24 benchmark, that is roughly 3x CPU and 5x memory for L4-only workloads.

Proxyless gRPC embeds xDS support directly in the gRPC library. Google Traffic Director speaks xDS to gRPC clients, eliminating the sidecar hop entirely for services that already use gRPC¹².

flowchart LR
    subgraph Sidecar["Sidecar mode (per-pod Envoy)"]
        A1[App A] --> EA[Envoy ~60 MB]
        EA -->|"mTLS"| EB[Envoy ~60 MB]
        EB --> B1[App B]
    end
    subgraph Ambient["Ambient mode (per-node ztunnel)"]
        A2[App A] --> ZA[ztunnel ~12 MB]
        ZA -->|"HBONE mTLS"| ZB[ztunnel ~12 MB]
        ZB --> B2[App B]
        ZA -.->|"L7 only"| WP[Waypoint Envoy]
        WP -.-> ZB
    end

mTLS bootstrapping with SPIFFE

Meshes mint a short-lived X.509 certificate per workload and require both sides to present and verify certs on every connection. SPIFFE standardizes the identity document (SVID) and the Workload API the proxy uses to fetch it; SPIRE is the reference implementation¹³.

A typical Kubernetes SPIFFE ID looks like spiffe://cluster.local/ns/default/sa/bookinfo-reviews, encoding the namespace and ServiceAccount. Because the identity is a URI, authorization policies refer to workloads symbolically rather than by IP. In Linkerd, each proxy generates its own key on startup, never writes it to disk, and gets a cert signed by the control plane Identity service⁹.

The key insight: mTLS authenticates who is calling. It does not authorize what they can do. You still need explicit AuthorizationPolicies with a deny-by-default posture.

Real-World Example

Lyft Envoy: the origin of the modern service mesh

In 2016, Lyft’s SOA had a problem every polyglot company hits: each service had a half-baked RPC library with inconsistent retries, circuit breakers, and observability. Java services used one retry strategy; Python services used another. Nobody had consistent distributed tracing. Matt Klein built Envoy to solve this.

Envoy launched publicly on September 14, 2016¹⁴. By April 2017, it had over 40 contributors and was being adopted by Google and IBM¹⁴. The key design decisions:

1. C++ for performance. Tail-latency predictability over JVM-based proxies. No GC pauses.
2. Eventually-consistent discovery. Lyft deliberately designed service discovery in Envoy as “eventually consistent and lossy”, in contrast to the strongly-consistent ZooKeeper approach, and reported “extremely high reliability without the maintenance headache”¹⁴.
3. HTTP/2 transparent upgrade. Critical for gRPC adoption across the fleet.
4. Universal xDS APIs. By publishing CDS/EDS/LDS/RDS as open APIs, Klein decoupled the data plane from any control plane. This is why Istio, Consul Connect, and Google Traffic Director all use Envoy today.

Lyft ran Envoy as both an edge proxy and a per-host sidecar on every service pod, replacing per-language RPC libraries entirely. The architecture is the direct ancestor of every modern service mesh: sidecar proxy, local-first routing, control plane pushing config.

The lesson: if you have 3+ languages and inconsistent resilience logic, a shared proxy is cheaper than maintaining N client libraries. But if you are a single-language shop with a mature RPC framework, you may not need the proxy at all.

Trade-offs

Common Pitfalls

Exercise

You have 80 microservices on Kubernetes, 5 teams, a mandate for mTLS everywhere, and existing retry/timeout logic in an in-house Go library. Compare: (a) keep the library, add cert-manager + NetworkPolicies, (b) adopt Linkerd, (c) adopt Istio. Estimate CPU overhead, migration effort, and which operational burdens move where.

Key Takeaways

• Service discovery is mandatory; a service mesh is a deliberate choice. Many teams buy mesh features they never use.
• If you only need mTLS, cert-manager and NetworkPolicies cost far less than a full mesh.
• Envoy is the de facto data plane; Linkerd is lighter and faster (26 MB, 6 ms median overhead), Istio is feature-rich but heavier (60 MB, 17 ms median overhead)⁸.
• The control plane never sits on the request path. If it goes down, existing proxies keep routing with stale config, but new deployments stop working.
• Sidecars buy language-agnostic policy at the cost of 1-5% CPU and 30-200 MB RAM per pod. Ambient mode and eBPF cut this by roughly 3x CPU and 5x memory for L4-only workloads.
• Do not adopt a mesh until you have 20+ services and clear needs for the full bundle (mTLS + retries + tracing + policy + traffic splitting).
• The direction of travel is sidecar-less: Istio ambient, Cilium eBPF, and proxyless gRPC. Evaluate before committing to per-pod sidecars.

Further Reading

• Matt Klein, “Service mesh data plane vs. control plane” (2017) - The taxonomy article that named the data plane/control plane split; required reading before evaluating any mesh.
• Linkerd blog, “Under the hood of Linkerd2-proxy” (2020) - Best single article on why a purpose-built Rust micro-proxy wins on tail latency versus a general-purpose C++ proxy.
• Netflix Tech Blog, “Zero Configuration Service Mesh with On-Demand Cluster Discovery” (2023) - Real migration story from Eureka + Ribbon to Envoy; shows the pain of maintaining per-language discovery clients.
• Kubernetes Blog, “Scaling Kubernetes Networking with EndpointSlices” (2020) - Why the old Endpoints API broke at scale and how EndpointSlices fix it with 100-endpoint shards.
• Istio blog, “Istio: The Highest-Performance Solution for Network Security” (2025) - Ambient mode benchmark numbers showing 75% throughput improvement over 4 releases; the case for sidecar-less.
• William Morgan, “Benchmarking Linkerd and Istio: 2021 Redux” - The most cited independent mesh benchmark; shows Linkerd at 6x less memory and 3x less latency than Istio.
• SPIFFE project documentation - Workload identity standard used by Istio, Consul, cert-manager, and Dapr; understand this before designing zero-trust policies.
• Envoy Proxy documentation - The canonical data plane reference; start with the architecture overview and xDS protocol description.

Flashcards

References