Lambda vs Kappa Architecture

TL;DR. Lambda runs two pipelines (batch for correctness, stream for freshness) and pays with dual codebases that drift out of sync¹. Kappa collapses both into one streaming job and replays the retained log for reprocessing¹. Default to Kappa when your streaming engine can express all required analytics and log retention covers your worst-case replay window. Fall back to Lambda (or a Kappa-ish hybrid) when regulatory batch ground truth is required or reprocessing terabytes through a streaming engine is infeasible². The deciding dimension is whether the batch layer earns its operational cost.

Learning Objectives

• Compare Lambda and Kappa across operational complexity, correctness guarantees, reprocessing cost, and freshness latency.
• Identify workload characteristics (retention window, analytics expressibility, regulatory requirements) that favor one architecture over the other.
• Justify a hybrid “Kappa-ish” approach combining streaming freshness with periodic batch reconciliation.
• Evaluate LinkedIn, Uber, Netflix, and Spotify as production systems that made this choice and explain why.

The Core Trade-off

Lambda was proposed by Nathan Marz around 2011³ and codified in his 2015 Manning book⁴. It runs a batch layer (Spark, MapReduce) that recomputes correct views from the full history, a speed layer (Flink, Storm) that computes approximate low-latency views, and a serving layer that merges both at query time. The batch path is the source of truth; the speed path is a best-effort approximation until the next batch run lands.

Kappa was proposed by Jay Kreps in 2014, at the time leading data infrastructure at LinkedIn and co-creator of Kafka (later co-founder and CEO of Confluent)¹. It collapses both pipelines into one: a single streaming job reads from a retained log, and “batch” reprocessing is performed by replaying the log from offset zero with a second instance of the same job. Once the replay catches up, reads cut over to the new output table.

The fundamental tension: Lambda pays a permanent operational tax (two codebases, two on-calls, two scaling profiles, constant drift-debugging) in exchange for batch-grade correctness. Kappa eliminates that tax but requires that the streaming engine can express every aggregation the batch layer handled, and that log retention covers every replay you will ever need. Kreps summarizes it bluntly: “maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems like it would be.”¹

flowchart LR
    subgraph "Lambda: two paths"
        E1[Events] --> B[Batch layer<br/>Spark / MapReduce]
        E1 --> S[Speed layer<br/>Flink / Storm]
        B --> BV[Batch views]
        S --> SV[Speed views]
        BV --> SL[Serving layer<br/>merge]
        SV --> SL
    end
    subgraph "Kappa: one path"
        E2[Events] --> K[(Kafka log<br/>long retention)]
        K --> J[Streaming job<br/>Flink / Samza]
        J --> T[Output table]
    end

Side-by-Side Comparison

The table understates one dimension: drift. In Lambda, the batch and stream paths use different frameworks with different window semantics, null handling, and type coercions. “Programming in distributed frameworks like Storm and Hadoop is complex. Inevitably, code ends up being specifically engineered toward the framework it runs on.”¹ This drift is not a theoretical risk; it is the primary operational pain of Lambda in practice.

The table also understates Kappa’s retention cost. LinkedIn keeps “more than a petabyte of Kafka storage online” to support long-retention replay¹. That is economical for LinkedIn; it may not be for your team.

When to Pick Lambda

Regulatory batch ground truth is required. Financial risk reconciliation, ad-attribution billing, end-of-month close. The batch path is the auditable source of truth; streaming is a live approximation that regulators do not accept².

Reprocessing terabytes through a streaming engine is infeasible. Uber’s sessionization pipeline found that “backfilling more than a handful of days’ worth of data could easily lead to replaying days’ worth of client logs… overwhelming the system’s infrastructure and causing lags.”² When 10 TB reprocesses in hours on Spark but days on Flink, the batch layer earns its cost.

Streaming engines cannot express the required analytics. Cross-day deduplication, global sorts, session stitching across days, and arbitrary windowed joins that Flink handles poorly or not at all².

Separate teams already own batch and streaming. Lambda is often the existing architecture, not a green-field choice. Migration to Kappa is a multi-quarter project; Lambda is the pragmatic default until you can invest.

When to Pick Kappa

One pipeline’s operational burden is your binding constraint. Two pipelines mean two on-call rotations, two code paths, and constant “why do these numbers disagree?” debugging. Kappa eliminates this entire category of bugs¹.

The streaming engine can express all your analytics. Event-sourcing applications, real-time dashboards, CDC-driven warehousing, feature computation for online ML, and fraud detection all fit Kappa well⁵.

Reprocessing is cheap. Your retention window covers the worst-case replay, and replaying a week finishes in hours. Kafka’s cheap consumers and long retention make this viable: “adding the second reprocessing job is just a matter of firing up a second instance of your code but starting from a different position in the log.”¹

Netflix runs over 15,000 Flink jobs processing more than 60 PB per day on its Keystone platform⁵. At that scale, maintaining a parallel batch pipeline for the same logic would be operationally catastrophic.

The Hybrid Path

In 2026, most large pipelines are neither pure Lambda nor pure Kappa. They are “Kappa-ish”: Kafka is the event log, Flink computes streaming features and live dashboards, and a nightly Spark or dbt job writes reconciled aggregates to an Iceberg or Delta lakehouse⁶⁷. The batch job is not a duplicate of the streaming logic; it is a different query (reconciliation, not real-time aggregation) over the same source of truth.

Apache Beam (based on the 2015 Dataflow Model paper⁸) provides a single SDK that compiles to multiple runners (Dataflow, Flink, Spark). The same pipeline runs on bounded or unbounded data. Spotify reportedly processes over 1.4 trillion events per day across more than 38,000 production pipelines using Beam/Scio⁹¹⁰. Their Wrapped 2020 job joined roughly 1 PB in a single pipeline⁹.

Lakehouse table formats (Iceberg, Delta Lake) add ACID transactions and time travel to cloud object stores, so both Flink and Spark can read and write the same table with snapshot isolation⁷¹¹. This eliminates the Lambda-era split between batch stores and streaming stores.

flowchart TD
    E[Event source] --> K[(Kafka)]
    K --> F[Flink streaming]
    K --> IB[Iceberg mirror]
    F --> I[(Iceberg table<br/>ACID)]
    IB --> I
    I --> SP[Nightly Spark / dbt<br/>reconcile]
    SP --> I
    I --> BI[BI / dashboards]
    F --> RT[Real-time features]

Real-World Examples

LinkedIn (2014-present). The birthplace of Kappa. Over 7 trillion Kafka messages per day across 4,000+ brokers and 100,000 topics¹². Kreps’s essay documented the pain of their prior Lambda approach: “We have built various hybrid-Hadoop architectures… none were very pleasant or productive.”¹ New pipelines from 2014 onward follow Kappa with Kafka as the retained log and Samza for processing.

Uber (2020). Uber’s sessionization pipeline (dynamic pricing, fraud detection) rejected both pure Kappa (log replay would overwhelm Kafka infrastructure) and pure batch (would require excess cluster resources). Their solution: treat a Hive table as a streaming source, relaxing watermarks from 10 seconds to 2 hours in backfill mode². One codebase, two modes. Production job: 75 cores, 1.2 TB memory. Backfill: ~10 TB over 9 days².

Netflix Keystone (2018-present). Over 15,000 Flink jobs processing more than 60 PB of data per day⁵. Each job runs in an isolated cluster on Titus containers to bound blast radius⁵. Netflix pairs Keystone streaming with Iceberg tables for batch-like parts of its warehouse (Netflix originally created Iceberg), making it a Kappa-ish hybrid¹¹.

Common Mistakes

Decision Checklist

• Can your streaming engine express all required analytics (windows, joins, dedup)?
• Does your log retention cover the worst-case replay window?
• Is reprocessing through the streaming engine feasible in acceptable time at your data volume?
• Do you have regulatory or accounting requirements for batch-as-ground-truth?
• Do you have one team or two? Two pipelines need two owners.
• How often does the batch path “correct” the streaming path? If rarely, the batch pipeline is waste.
• Can your output store handle 2x storage during Kappa replay cutover?

graph TD
    Start[New pipeline] --> Q1{Retention covers<br/>worst-case replay?}
    Q1 -- no --> Lambda[Lambda or<br/>Iceberg mirror + replay]
    Q1 -- yes --> Q2{Streaming engine<br/>expresses all analytics?}
    Q2 -- no --> Hybrid[Kappa-ish hybrid:<br/>stream + batch reconcile]
    Q2 -- yes --> Q3{Regulatory batch<br/>ground truth required?}
    Q3 -- yes --> Hybrid
    Q3 -- no --> Kappa[Pure Kappa]

Key Takeaways

• Kappa is the default for new pipelines. Lambda’s dual-codebase cost is justified only when batch semantics are genuinely required or streaming reprocessing is infeasible.
• The operational pain of Lambda is not running two pipelines; it is keeping them in sync. Drift between batch and stream outputs is the primary failure mode.
• Most 2026 production systems are Kappa-ish hybrids: Kafka as the log, Flink for speed, Spark/dbt for reconciliation, Iceberg as the unified table.
• Beam and lakehouse formats (Iceberg, Delta) are the modern answer to “why choose?” One codebase, batch or streaming, same table.
• Know your binding constraint: if it is correctness, keep the batch layer. If it is operational simplicity, eliminate it.

Further Reading

• Questioning the Lambda Architecture: Kreps’s 2014 founding Kappa essay; the single most important primary source for this decision.
• Designing a Production-Ready Kappa Architecture: Uber’s honest case study of what had to be modified to make Kappa work at scale.
• The Dataflow Model (Akidau et al., VLDB 2015): the theoretical basis for Beam’s unified batch/streaming; read before betting on a unified SDK.
• How Spotify Optimized the Largest Dataflow Job Ever: 1 PB join using Beam/Scio; the best public Beam-at-scale write-up.
• Keystone Real-time Stream Processing Platform: Netflix’s architecture at trillions of events per day; Kappa-ish at extreme scale.
• Apache Iceberg documentation: the lakehouse table format that eliminates the Lambda-era batch-store vs streaming-store split.

Flashcards

References