Design an Observability Platform (Datadog / New Relic / Honeycomb)

TL;DR. An observability platform unifies metrics, logs, and traces behind a single correlation key (trace_id) so an engineer pivots from a spiking error-rate graph to the exact failing span and its log lines in one click. At Datadog scale this means 100+ trillion events per day¹, petabytes of storage on S3, and interactive query latency of ~2 ms p50 per fragment¹. The pivotal trade-off is not volume but cardinality: every unique label combination is a new time series, and one unbounded label can OOM your TSDB overnight². The architecture converges on OTLP as the universal wire protocol³, object storage as the durable layer, shuffle-sharded multi-tenant query isolation¹, and tail-based sampling to preserve 100% of error traces without storing everything⁴.

Learning Objectives

• Design a unified ingestion layer that accepts metrics, logs, and traces over a single protocol (OTLP)
• Explain why label cardinality, not raw volume, is the first constraint that kills a metrics platform
• Implement trace-log-metric correlation using trace_id as the universal join key
• Justify storage choices (Mimir vs Prometheus, Loki vs Elasticsearch, Tempo vs Jaeger) for each signal
• Budget per-tenant query compute so one noisy dashboard cannot starve the fleet
• Compare head-based and tail-based sampling and articulate when each applies

Intuition

A single Prometheus server, a Loki instance, and a Jaeger collector each work fine in isolation. You can monitor 50 services with them. The architecture collapses the moment an on-call engineer sees a latency spike on a dashboard and asks: “which specific request caused this, and what did its logs say?”

That question requires a pivot. The metric tells you something is wrong. The trace tells you which request path is wrong. The log tells you why. Without a shared correlation key, the engineer opens three tabs, manually aligns timestamps, and guesses. At 3 AM during an incident, guessing is not acceptable.

The insight that unlocks the design: trace_id is the join key across all three signals. The SDK stamps every log line with the active span’s trace ID. Metrics carry exemplars that point at sample trace IDs. The query engine joins them on click⁵. This is not three pipelines bolted together; it is one product with three storage backends.

Design a Metrics Pipeline covers the metrics pillar in depth. This chapter zooms out to the unified platform: the OTLP ingestion layer that replaces per-signal agents, the cardinality control that protects the TSDB, the multi-tenant isolation that keeps tens of thousands of customers from interfering with each other, and the query engine that returns interactive results over petabytes.

Requirements

Clarifying Questions

• Q: Unified OTLP ingestion, or separate pipelines per signal? Assume: Unified. One SDK, one collector, one protocol (OTLP gRPC/HTTP)³.

• Q: Multi-tenant from day one? Assume: Yes. 10,000+ customers share infrastructure. Per-tenant quotas on ingest, query, and retention.

• Q: Real-time alerting SLA? Assume: Alert evaluation every 30 seconds. Page delivery within 60 seconds of threshold breach.

• Q: APM (distributed tracing, service maps) first-class? Assume: Yes. Traces are a primary signal, not a bolt-on.

• Q: RUM and synthetic monitoring in scope? Assume: RUM (Core Web Vitals: LCP <= 2.5 s, INP <= 200 ms, CLS <= 0.1 at p75)⁶ in scope. Synthetics as a follow-up.

• Q: What is the query latency target? Assume: p50 < 5 ms per fragment, p99 < 300 ms. Dashboards feel interactive.

Functional Requirements

• Ingest metrics, logs, and traces over OTLP (gRPC port 4317, HTTP port 4318)³
• Query and visualize each signal via PromQL, LogQL, and trace-search APIs
• Evaluate alerting rules on schedule with multi-window multi-burn-rate SLO alerts⁷
• Correlate signals: click a metric point to see the trace, click a span to see its logs
• Auto-discover service topology from trace parent-child relationships (service map)
• Per-tenant dashboards, RBAC, and retention policies

Non-Functional Requirements

• Ingestion: 1B events/sec combined (100M metric samples/sec, 800M log events/sec, 100M spans/sec)
• Latency: p50 < 5 ms per query fragment; dashboard refresh < 2 seconds end-to-end¹
• Availability: 99.99% ingestion path; 99.9% query path
• Tenant isolation: one customer’s 30-day scan cannot starve shared query workers
• Retention: 15 days hot, 90 days warm, 1 year cold (compressed, downsampled for metrics)

Capacity Estimation

Key ratios:

• Read:write: ~5:1 on metrics (dashboard refresh), ~0.01:1 on logs (most logs never read)
• Cache hit rate: ~80% result cache, ~70% blob-range cache on Datadog Husky¹
• Fragments touching blob storage: only 0.4% after all cache layers¹

API and Data Model

API Design

POST /v1/traces    (OTLP protobuf, Snappy-compressed)
POST /v1/metrics   (OTLP protobuf)
POST /v1/logs      (OTLP protobuf)

GET  /api/v1/query?query=rate(http_requests_total[5m])&time=<ts>
GET  /api/v1/query_range?query=...&start=<ts>&end=<ts>&step=15s
GET  /api/v1/logs?query={service="api"} |= "error"&start=<ts>&end=<ts>&limit=100
GET  /api/v1/traces/{trace_id}
GET  /api/v1/traces/search?service=api&minDuration=500ms&limit=20

POST /api/v1/alerts/rules   Body: { expr, for, labels, annotations }
GET  /api/v1/service-map?start=<ts>&end=<ts>

Pagination uses cursor-based tokens. Rate limiting returns 429 with Retry-After. All endpoints require tenant API key in DD-API-KEY or Authorization: Bearer header.

Data Model

-- Metrics (columnar TSDB, time-partitioned)
(tenant_id, metric_name, labels{}, timestamp_ms, value)
Partition key: tenant_id + metric_name + label_hash
Compression: Gorilla XOR + delta-of-delta

-- Logs (label-indexed chunks)
(tenant_id, stream_labels{}, timestamp_ns, log_line, trace_id, span_id)
Index: label tuples only (Loki model) or full-text inverted index (ES model)

-- Traces (span store)
(tenant_id, trace_id, span_id, parent_span_id, service, operation,
 start_time_ns, duration_ns, status, attributes{})
Primary lookup: trace_id -> all spans

erDiagram
    METRIC_SERIES ||--o{ SAMPLE : contains
    METRIC_SERIES {
        uuid tenant_id
        string metric_name
        map labels
        int series_id
    }
    SAMPLE {
        int64 timestamp_ms
        float64 value
    }
    LOG_STREAM ||--o{ LOG_ENTRY : contains
    LOG_STREAM {
        uuid tenant_id
        map stream_labels
    }
    LOG_ENTRY {
        int64 timestamp_ns
        string line
        string trace_id
    }
    TRACE ||--o{ SPAN : contains
    TRACE {
        string trace_id
        uuid tenant_id
    }
    SPAN {
        string span_id
        string parent_span_id
        string service
        int64 duration_ns
        map attributes
    }

High-Level Architecture

flowchart LR
    subgraph Hosts["Instrumented Fleet (10M hosts)"]
        SDK[OTel SDK]
    end
    subgraph Ingress["OTLP Ingress Layer"]
        GW[API Gateway<br/>authN + tenant routing]
        COL[OTel Collector<br/>batch + sample + route]
    end
    subgraph Storage["Signal Stores"]
        MET[(Metrics<br/>Mimir / Husky)]
        LOG[(Logs<br/>Loki / Husky)]
        TRC[(Traces<br/>Tempo / Husky)]
        OBJ[(S3 object store)]
    end
    subgraph Query["Query Layer"]
        QE[Unified Query Engine]
        CACHE[Result Cache]
        DASH[Dashboards]
    end
    subgraph Control["Control Plane"]
        QUOTA[Tenant quotas<br/>billing meters<br/>shard routing]
        ALERT[Alert Evaluator<br/>multi-burn-rate]
        PAGE[PagerDuty / Opsgenie]
    end
    SDK -->|OTLP gRPC 4317| GW
    GW --> COL
    COL -->|metrics| MET
    COL -->|logs| LOG
    COL -->|traces| TRC
    MET --> OBJ
    LOG --> OBJ
    TRC --> OBJ
    QE --> MET
    QE --> LOG
    QE --> TRC
    QE --> CACHE
    DASH --> QE
    ALERT -->|PromQL eval| MET
    ALERT -->|page| PAGE
    QUOTA -.->|governs| GW
    QUOTA -.->|governs| QE

Write path: Hosts emit all signals via the OTel SDK over OTLP. The ingress gateway authenticates the tenant API key, applies rate limits, and forwards to an OTel Collector fleet. Collectors batch, apply tail sampling (traces), enforce cardinality caps (metrics), and route each signal to its store. Stores buffer briefly in memory, then flush immutable files to S3.

Read path: Dashboards hit the unified query engine, which routes PromQL to the metrics store, LogQL to the log store, and trace queries to the trace store. A result cache (keyed on tenant_id, query, time_range) absorbs ~80% of repeated dashboard refreshes¹. The query engine joins results on trace_id for cross-signal pivots.

Alert path: The alert evaluator runs PromQL rules every 30 seconds. Multi-window multi-burn-rate logic (14.4x over 1 h AND 5 m; 6x over 6 h AND 30 m) pages only on sustained budget burn⁷. Notifications route through PagerDuty or Opsgenie.

Deep Dives

Unified OTLP ingestion and the OTel Collector

The OpenTelemetry Collector is the single binary that replaces StatsD, Jaeger agent, Fluentd, and vendor-specific agents. Its architecture is a pipeline of receiver -> processor -> exporter, configurable per signal³.

flowchart LR
    subgraph Collector["OTel Collector"]
        RX[OTLP Receiver<br/>gRPC :4317 / HTTP :4318]
        PROM[Prometheus Scraper<br/>legacy pull]
        ML[memory_limiter]
        BATCH[batch processor]
        TS[tail_sampling<br/>keep errors + slow]
        CARD[cardinality_limiter<br/>per-tenant cap]
        EXP[OTLP Exporter]
    end
    RX -->|traces| TS --> BATCH --> EXP
    RX -->|metrics| CARD --> ML --> BATCH
    RX -->|logs| ML --> BATCH
    PROM -->|metrics| CARD
    EXP --> Backend[Metrics / Logs / Traces stores]

W3C Trace Context propagation ensures every service in a request chain carries the same trace_id. The traceparent header encodes version-trace_id(16 bytes)-parent_id(8 bytes)-flags⁹. Every SDK and collector must honor this for cross-vendor correlation.

The tail-sampling processor buffers complete traces (typically 10-30 seconds, configurable via decision_wait), then applies policies: keep 100% of errors, 100% of traces exceeding 500 ms latency, and 1% probabilistic baseline for everything else¹⁰. This replaces head sampling, which decides keep/drop at the root span before any status is known and therefore discards the exact error traces engineers need⁴.

Honeycomb’s Refinery extends this with dynamic sampling (rare-value traces kept more often) and EMA-based rate adjustment that adapts to traffic volume changes⁴.

Cardinality control: the constraint that kills metrics platforms

Every unique (metric_name, labels) combination is a new time series. Prometheus explicitly warns: labels derived from unbounded sets (user ID, request ID, session ID) create a cardinality bomb that breaks queries, dashboards, and alert evaluation¹¹. Datadog prices custom metrics at $0.10 per 100 ingested metrics beyond the 100-200 per host allotment, making cardinality a direct cost problem¹².

Why it is catastrophic: One engineer adds user_id to http_requests_total. Overnight, a 5,000-series metric becomes 50M series. The TSDB ingester OOMs. Dashboards go dark during the incident you need them for².

Detection at ingest: The cardinality limiter processor tracks per-tenant series count. When a metric exceeds the configured cap, new label combinations are dropped with a warning event. Datadog’s “Metrics Without Limits” decouples ingested from indexed: all tags land in storage, but only an allowlisted subset is queryable, breaking the cost curve without losing raw data¹².

Exemplars solve the correlation problem: Adding trace_id as a metric label is the canonical cardinality bomb. Instead, OpenMetrics exemplars attach a sampled trace_id pointer to a metric sample without making it a dimension²⁵. The UI reads the exemplar to jump from metric to trace.

Multi-tenant query isolation via shuffle sharding

At the scale of tens of thousands of customers sharing one query fleet, one tenant’s ad-hoc 30-day log scan reading 10 TB can starve every other tenant’s dashboard. Consistent hashing alone gives every tenant access to every worker, so a heavy scan saturates the entire pool¹.

Datadog Husky uses shuffle sharding: each tenant is mapped to a bounded subset of reader workers. Tenant A gets 9 workers, Tenant B gets 15, Tenant C gets 6¹. A noisy tenant’s scan saturates only its subset; other tenants’ workers remain unaffected.

flowchart TD
    subgraph Pool["Reader Pool (20 workers)"]
        W1((1)) & W2((2)) & W3((3)) & W4((4)) & W5((5))
        W6((6)) & W7((7)) & W8((8)) & W9((9)) & W10((10))
        W11((11)) & W12((12)) & W13((13)) & W14((14)) & W15((15))
        W16((16)) & W17((17)) & W18((18)) & W19((19)) & W20((20))
    end
    TA["Tenant A<br/>shard: 9 workers"] --> W1 & W4 & W7 & W10 & W13 & W15 & W17 & W19 & W20
    TB["Tenant B<br/>shard: 6 workers"] --> W2 & W5 & W8 & W11 & W14 & W18

Caching stacks further reduce load: the result cache (keyed on query + time range) hits ~80% for dashboard refreshes; the blob-range cache hits ~70% for repeated chunk reads; a predicate cache stores expensive filter bitsets with ~11x average CPU savings per hit (peaks above 20x)¹. After all layers, only 0.4% of fragment queries actually touch blob storage¹.

The query orchestrator also tracks per-reader load and routes to a consistent secondary if the primary is overloaded, preserving cache affinity while avoiding hot spots¹.

SLO alerting: multi-window multi-burn-rate

Naive threshold alerts (“error rate > 0.1% for 10 minutes”) either page on insignificant blips or miss slow-burning budget consumption⁷. Google SRE Workbook Ch. 5 defines the production standard:

• Page: 14.4x burn rate over 1 h AND 14.4x over 5 m (consumes 2% of monthly budget in 1 h)
• Page: 6x burn rate over 6 h AND 6x over 30 m (consumes 5% in 6 h)
• Ticket: 1x burn rate over 3 days (consumes 10% in 3 d)⁷

The long window gives precision (avoids false pages). The short window gives fast reset (alert clears 5 minutes after the incident ends, not 1 hour later). The alert evaluator must compute multiple rolling windows per SLO, amortizing TSDB reads across rules.

Real-World Example

Datadog Husky: third-generation event store serving 100+ trillion events per day.

Datadog’s first two generations used Elasticsearch-style shard-and-replica clustering. At scale, a single noisy tenant could destabilize entire clusters. The third generation, Husky, was built from scratch around three principles: storage on S3, stateless compute, and per-tenant isolation¹³.

Architecture: Writers consume from Kafka, buffer events in memory, upload custom columnar files to S3, then commit file metadata to FoundationDB (chosen for its serializable transactions and rigorous simulation testing)¹³. Compactors merge small files into larger ones as a distributed auto-scaling service. Readers execute queries over individual fragment files and return partial aggregates. All components are stateless and scale independently¹³.

Query performance: Husky’s query engine uses a Volcano-style iterator model with lazy row-group decoding. Out of 1,000 sample fragment queries, 300 are pruned by metadata, 560 by the result cache, 78 by column metadata, and 28 by other caches. Only 3.4% read data; only 0.4% touch blob storage¹. Fragment query latency: p50 ~2 ms, p90 ~6 ms, p99 ~257 ms¹.

Business scale: $2.68 billion revenue in FY2024, 26% YoY growth, 462 customers with $1M+ ARR, about 3,610 customers with $100K+ ARR¹⁴. Pricing: $15/host/month infrastructure (Pro), $31/host/month APM, $0.10/GB log ingestion¹⁵.

The key insight: separating storage from compute from day one eliminated the noisy-neighbor problem that plagued the first two generations. Every tenant gets bounded blast radius through shuffle sharding, and the economics of S3 make petabyte retention viable.

Trade-offs

The biggest meta-decision: three-pillars vs. wide events. The three-pillars camp (Grafana LGTM, traditional Datadog) stores metrics, logs, and traces in separate optimized backends joined by correlation. The wide-events camp (Honeycomb) stores arbitrarily-wide structured events and derives metrics at query time¹⁸¹⁹. Three-pillars wins on cost efficiency (each backend is optimized for its access pattern). Wide events win on exploratory debugging (any field is queryable without pre-aggregation). The industry is converging: Datadog internally uses wide events in Husky while exposing three-pillar APIs externally.

Scaling and Failure Modes

At 10x (10B events/sec):

• Ingestion Kafka partitions saturate. Mitigation: increase partition count, add regional ingress clusters.
• Result cache hit rate drops as query diversity grows. Mitigation: tiered caching with per-tenant LRU eviction.

At 100x (100B events/sec):

• Single-region S3 throughput limits. Mitigation: multi-region cells with per-region ingestion and cross-region query federation.
• FoundationDB metadata layer becomes the bottleneck (5-second transaction limit)¹³. Mitigation: shard metadata by tenant prefix.

At 1000x:

• Architectural rewrite: edge pre-aggregation (aggregate metrics at the collector before shipping), tiered storage with aggressive downsampling, and per-tenant dedicated cells for the largest customers.

Failure modes:

• Cardinality explosion: One tenant’s bad label OOMs the metrics ingester. Detection: per-metric series-creation rate alert. Response: cardinality limiter drops new combinations; existing series tombstoned. Recovery: minutes if automated²¹¹.
• Regional S3 outage: Recent data (last 2 hours) still in ingester memory. Historical queries fail gracefully with partial results. Alerting continues from in-memory data.
• Noisy-tenant query storm: Shuffle sharding contains blast radius to the tenant’s worker subset. Per-tenant concurrent-query cap (e.g., 50 parallel queries) prevents even the subset from being fully consumed¹.

Common Pitfalls

Follow-up Questions

1. How do you integrate LLM observability (token usage, hallucination rate, prompt-response traces)?

LLM calls are spans with attributes (llm.token_count, llm.model, llm.prompt_hash). The OTel SDK’s semantic conventions for GenAI carry these as span attributes. Evaluation metrics (hallucination score, relevance) are logged as span events. LLM Evaluation and Observability covers the evaluation pipeline that feeds into this platform.

2. How do you support high-cardinality exploratory queries without pre-built dashboards?

Honeycomb’s wide-event model stores all attributes on every event. BubbleUp queries group by any field at query time without pre-aggregation²⁰²¹. For a three-pillars architecture, this requires a columnar scan engine (like Husky) that can filter on arbitrary attributes without a pre-built index.

3. How do you handle ephemeral serverless workloads (Lambda, Cloud Run) where the host lives for 200 ms?

Push-based OTLP (not pull/scrape). The function’s OTel SDK batches spans and metrics, then flushes synchronously before the runtime freezes. A lightweight extension or sidecar handles the OTLP export. Trace context propagates through the event trigger (HTTP header, SQS message attribute).

4. What is the pricing model, and how do you align invoices with resource use?

Bill on three axes: ingested volume (GB/month for logs, custom metrics count for metrics¹², spans/month for traces), query compute (CPU-seconds), and retention (GB-months in each tier). Datadog’s model ($15/host infra Pro, $31/host APM, $0.10/GB logs)¹⁵ is the reference. “Metrics Without Limits” decouples ingested from queryable to avoid punishing instrumentation¹².

5. How do you onboard a new customer with zero instrumentation?

Auto-instrumentation agents (Java agent, .NET profiler, eBPF-based Beyla²²) attach to processes without code changes. The agent discovers services, emits traces and metrics, and the platform auto-generates a default dashboard and service map from the first 5 minutes of data.

6. Can this platform observe itself (meta-observability)?

The observability platform runs its own OTel SDK pointed at a dedicated internal tenant. Ingestion lag, query latency, and cache hit rates are first-class metrics. A separate, minimal Prometheus instance provides last-resort alerting if the main platform is down.

Exercise

Exercise 1: Tail-sampling budget

Your platform ingests 100M spans/sec. Storage costs $0.02/GB/month on S3. Each span averages 200 bytes. You want to keep 100% of error spans (~0.1% of traffic), 100% of slow spans (latency > 500 ms, ~1% of traffic), and a probabilistic baseline of the rest. What sampling percentage for the baseline keeps total trace storage under 50 TB/month?

Key Takeaways

• trace_id is the product. Three separate pipelines without a correlation key are three products, not one. Build the join from day one⁵.
• Cardinality, not volume, kills metrics. One unbounded label is worse than 10x more well-labeled data²¹¹.
• Tail sampling preserves what matters. Head sampling discards errors probabilistically; tail sampling keeps 100% of errors and slow traces at the cost of a brief buffer (typically 10-30 s)⁴¹⁰.
• Shuffle sharding is mandatory for multi-tenant query. Without it, one noisy tenant starves the fleet¹.
• Object storage is the universal durable layer. Metrics (Mimir), logs (Loki), and traces (Tempo) all converge on S3; they differ only in the compute layer above it¹⁶¹⁷.
• Multi-burn-rate alerting replaces naive thresholds. The 14.4x/6x/1x three-tier model from Google SRE gives precision and fast reset simultaneously⁷.

Further Reading

• Inside Husky’s query engine: real-time access to 100 trillion events. The best public write-up of a Datadog-scale query engine with real latency percentiles and cache-hit numbers.
• Introducing Husky, Datadog’s Third-Generation Event Store. Motivation and design for why Datadog outgrew Elasticsearch and built their own columnar store on S3.
• Google SRE Workbook, Ch. 5: Alerting on SLOs. Canonical source for multi-window multi-burn-rate alerts; the 14.4x/6x/1x table every on-call engineer should memorize.
• W3C Trace Context 1.0 Recommendation. The actual spec for traceparent and tracestate headers that every SDK must implement.
• OpenTelemetry Collector Architecture. Receivers, processors, exporters explained; the foundation for unified ingestion.
• Honeycomb Refinery documentation. How tail-based sampling works in production with dynamic and EMA modes.
• Grafana Mimir: scaling to 1 billion active series. The reference point for open-source metric-store scale with split-and-merge compaction.
• Observability 2.0 and the Database for It. Good synthesis of the wide-events argument with concrete payload examples.

Flashcards

References