Chaos Engineering: Breaking Things on Purpose

TL;DR: Chaos engineering is not breaking things for fun. It is a scientific method: form a hypothesis about how your system handles failure, inject a controlled fault, measure whether steady state holds, and either build confidence or fix what broke. Netflix has run Chaos Monkey in production continuously since July 2011¹. When a real AWS DynamoDB outage hit US-EAST-1 in September 2015 and cascaded through dependent services including SQS, EC2 Auto Scaling, CloudWatch, and the AWS Console, with Auto Scaling backlogs not clearing until roughly 8.5 hours after the event began², Netflix “sidestepped any significant impact” because Chaos Kong drills had already forced the cross-region failover fixes³. The lesson: you do not have resilience until you have tested it under real failure. Blast-radius control is what separates chaos engineering from sabotage.

Learning Objectives

After this module, you will be able to:

• Explain the Principles of Chaos and the four-step experiment framework
• Define steady-state metrics, hypothesis, and rollback criteria for an experiment
• Run a game day with appropriate blast radius and kill-switch conditions
• Choose between Chaos Monkey, Chaos Mesh, LitmusChaos, Istio fault injection, and managed services (AWS FIS, Azure Chaos Studio)
• Build organizational support for intentional production failure injection

Intuition

You own a house with a backup generator. The power company has been reliable for years, so you have never tested the generator. One winter night, an ice storm knocks out power to the whole neighborhood. You flip the transfer switch. Nothing happens. The fuel line is clogged. The battery is dead. The manual is in a box you cannot find in the dark.

Your neighbor, on the other hand, runs the generator for 30 minutes every month. She checks the fuel, tests the transfer switch, and times how long it takes to restore power. When the ice storm hits, her lights come back in 90 seconds.

The difference is not the generator. Both houses have one. The difference is that your neighbor tested hers under controlled conditions before the real failure arrived. She found the clogged fuel line in daylight, on a Saturday, with a hardware store open. You found it at 2 a.m. in a blizzard.

Chaos engineering is the monthly generator test for distributed systems. Resilience Patterns gave you the circuit breakers, retries, and bulkheads. This chapter teaches you how to verify they actually work, before the ice storm arrives.

Theory

The origin story

In August 2008, a corrupted database halted Netflix’s DVD shipping for three days. Leadership concluded that horizontally scalable, fault-tolerant cloud architecture was the only path forward⁴. The migration from monolithic datacenters to AWS ran from 2008 to 2011. The problem: AWS assumes individual VMs fail constantly. If your application cannot tolerate one instance disappearing, it will eventually suffer when AWS terminates that instance unexpectedly.

The solution, announced on the Netflix Tech Blog on July 19, 2011, was Chaos Monkey: a tool that “randomly disables our production instances to make sure we can survive this common type of failure without any customer impact”¹. The name came from “the idea of unleashing a wild monkey with a weapon in your data center to randomly shoot down instances and chew through cables”¹.

By 2012, Netflix had built the Simian Army around Chaos Monkey:

• Latency Monkey injected artificial delays into RESTful calls
• Conformity Monkey terminated instances violating best practices (e.g., not in an auto-scaling group)
• Chaos Gorilla simulated the outage of an entire availability zone

By 2015, the program expanded further:

• Chaos Kong simulated the loss of an entire AWS region by routing all traffic away

In 2014, Netflix published FIT (Failure Injection Testing), which intercepts requests at the Zuul edge proxy, decorates them with failure context, and propagates that context through injection points inside Hystrix, Ribbon, EVCache, and Astyanax⁵. FIT enabled targeting “a specific test account or a specific device” before scaling to a percentage of production.

In 2015, Netflix engineers including Casey Rosenthal, Ali Basiri, Lorin Hochstein, and Aaron Blohowiak published the Principles of Chaos manifesto at principlesofchaos.org, formalizing what Netflix had learned into a general engineering discipline⁶.

The Principles of Chaos

The manifesto (last updated March 2019) prescribes a four-step experiment loop⁶:

1. Define steady state as some measurable output that indicates normal behavior (throughput, error rates, latency percentiles)
2. Hypothesize that the steady state will hold in both control and experimental groups
3. Introduce variables that reflect real-world events (server crashes, network partitions, disk failures)
4. Try to disprove the hypothesis by looking for a steady-state difference

Five advanced principles describe the ideal practice: build hypotheses around steady-state behavior (output, not internals); vary real-world events (prioritize by impact or frequency); run experiments in production (“systems behave differently depending on environment and traffic patterns”); automate experiments to run continuously; and minimize blast radius⁶.

The key distinction: chaos engineering is not fault injection. Fault injection tests a single condition. Chaos engineering is an empirical method that generates new information about complex systems⁷. The product is not discovered bugs. It is continuously re-validated operational readiness.

flowchart LR
    SS[1. Define steady state<br/>measurable output: SLO metrics]
    H[2. Hypothesize<br/>steady state will hold]
    V[3. Inject fault<br/>real-world variable]
    M[4. Measure<br/>control vs. experimental]
    OK{Steady state<br/>preserved?}
    FIX[Fix the system<br/>document findings]
    CONF[Build confidence<br/>automate experiment]
    SS --> H --> V --> M --> OK
    OK -- Yes --> CONF --> SS
    OK -- No --> FIX --> SS

The experiment framework

A rigorous chaos experiment has six components:

Quantified steady state. Not “the system works” but “the Transaction Create API serves the 99th percentile of requests in under 100 ms.” The AWS Well-Architected Reliability pillar provides a hypothesis template: “If specific fault occurs, the workload name workload will describe mitigating controls to maintain business or technical metric impact”⁸.

Explicit hypothesis. A concrete example from AWS: “If 20% of the nodes in the Amazon EKS node-group are taken down, the Transaction Create API continues to serve the 99th percentile of requests in under 100 ms. The EKS nodes will recover within five minutes, and pods will get scheduled and process traffic within eight minutes”⁸.

Controlled, reversible fault injection. The fault must be something you can stop. AWS FIS supports a post-action rollback configuration that “returns the target to the state that it was in before the action was run”⁸.

Blast-radius control. Start with one pod, one account, one user. Widen only after each phase passes.

Kill switch. AWS FIS supports up to five CloudWatch-alarm-based stop conditions per experiment template. The experiment halts automatically when any alarm breaches⁸. An experiment without a kill switch is not chaos engineering. It is sabotage.

Conclusion document. Did steady state hold? If not, what broke, what is the fix, and who owns it?

Failure modes to inject

The taxonomy of injectable faults spans seven categories:

Blast radius control

The Principles of Chaos states the obligation directly: “it is the responsibility and obligation of the Chaos Engineer to ensure the fallout from experiments are minimized and contained”⁶.

In practice, blast radius widens in phases:

flowchart TD
    T[Tabletop<br/>plan hypothesis and runbook]
    D[Dev environment<br/>single pod, team watching]
    GATE1{Dev pass?}
    P1[Production: test account<br/>1 user or 1 device]
    P2[Production: 1% traffic<br/>canary scope]
    P3[Production: 10-50% traffic<br/>real load]
    P4[Production: continuous<br/>Chaos Monkey always-on]
    T --> D --> GATE1
    GATE1 -- Yes --> P1 --> P2 --> P3 --> P4
    GATE1 -- No --> T

Netflix FIT uses Zuul to isolate most failure tests to “only a specific test account or a specific device” before expanding to “a small percentage of production requests”⁵. LinkedIn’s LinkedOut framework uses the LiX A/B testing platform to narrow failure to “a specific user making a specific request with a specific Rest.li method”⁹. Slack’s Disasterpiece Theater adds a human gate: every exercise begins in dev, and the team makes an explicit go/no-go decision before proceeding to production¹⁰.

When to run. Start during business hours with engineers on standby. Never Friday evening. Never during major launches. Netflix runs Chaos Monkey during business hours specifically so failures are learning opportunities rather than outages¹.

Game days and organizational adoption

A game day is a pre-planned chaos exercise with the team watching. The goal is to stress both the system and the humans: runbooks, alerting, on-call response, and cross-team coordination.

Amazon GameDay originated in the early 2000s under Jesse Robbins, whose title was “Master of Disaster.” The ACM Queue round-table describes it as “a program designed to increase resilience by purposely injecting major failures into critical systems semi-regularly to discover flaws and subtle dependencies”¹¹.

Google DiRT (Disaster Recovery Testing) has run annually company-wide since circa 2006. Kripa Krishnan’s 2012 ACM Queue paper describes it as “an annual, company-wide, multi-day Disaster Recovery Testing event”¹². DiRT scenarios include simulated earthquakes and zombie apocalypses, but the wrapper is for realistic failures: datacenter power loss, inter-region network partition, loss of authentication infrastructure. The dramatic framing forces teams to engage rather than check boxes.

Slack Disasterpiece Theater (since January 2018) follows a structured format: hosts document “precisely how they’re going to incite the failure, right down to the commands they’re going to run,” go on record with their confidence level, run in dev first, then make a go/no-go decision for production¹⁰.

The adoption timeline. Most organizations take 6 to 18 months from their first game day to comfortable production chaos. The path:

1. Tabletop exercises (zero risk, high learning)
2. Monthly game days in staging
3. Targeted production experiments with narrow scope
4. Continuous production chaos (Chaos Monkey always-on)

Cultural prerequisites matter more than tooling. LinkedIn Waterbear’s playbook includes “roadshows to other eng teams and orgs, internal tech talks, resilience-focused sessions in new hire onboarding training, and competitive games to encourage people to discover resilience issues”⁹. Chaos without blameless culture produces resentment, not resilience.

Real-World Example

Netflix: from Chaos Monkey to Chaos Kong

Netflix’s chaos engineering program is the industry reference because it spans the full maturity spectrum: random instance termination (Chaos Monkey, 2011), targeted request-level faults (FIT, 2014), AZ outage simulation (Chaos Gorilla), and region evacuation drills (Chaos Kong, 2015).

Architecture. Chaos Monkey is a Go service integrated with Spinnaker that terminates “virtual machine instances and containers that run inside of your production environment” via API calls to whichever cloud backend Spinnaker manages¹³. It is opt-out, not opt-in: eligibility defaults on, making resilience the default assumption.

FIT provides the fine-grained layer. Zuul decorates incoming requests with failure context at the edge. That context propagates through internal service-to-service calls. Each injection point (Hystrix, Ribbon, EVCache, Astyanax) checks the context and simulates the failure if matched⁵.

sequenceDiagram
    participant Client
    participant Zuul as Zuul Edge
    participant Svc as Service
    participant Cache as EVCache
    participant DB as Astyanax
    Client->>Zuul: request (test account)
    Zuul->>Zuul: attach FIT failure context
    Zuul->>Svc: request + failure context
    Svc->>Cache: GET key (with context)
    alt context matches injection point
        Cache-->>Svc: simulated timeout
        Svc->>Svc: fallback path
    else no match
        Cache-->>Svc: real value
    end
    Svc->>DB: query (with context)
    DB-->>Svc: real value
    Svc-->>Zuul: response
    Zuul-->>Client: response

The Chaos Kong validation. On September 20, 2015, a real AWS DynamoDB service event in US-EAST-1 cascaded into multiple dependent services. According to the AWS post-event summary, the DynamoDB error rate stabilized at roughly 55% starting at 2:37am PDT and the service was not restored until 7:10am PDT (~5 hours), with several dependent services affected: SQS cache refreshes failed from ~5:45am until 7:10am PDT, EC2 Auto Scaling APIs were degraded from 2:15am to 7:10am PDT and the backlog of scaling activities did not clear until 10:52am PDT (~8.5 hours from the start), CloudWatch metrics were delayed or missing from 2:35am until 7:29am PDT, and AWS Console logins were impacted from 5:45am to 7:10am PDT. AWS wrote that “there are several other AWS services that use DynamoDB that experienced problems during the event” beyond the named examples². Netflix “sidestepped any significant impact” because Chaos Kong exercises had already exposed and fixed the systemic weaknesses that would have made the outage catastrophic³. The honest inversion: Chaos Kong had disproved Netflix’s confidence in single-region operation many times before September 2015, forcing the cross-region failover fixes that made the real outage survivable.

The Subscriber experiment. In 2015, Netflix injected 30 ms of latency into 50% of Subscriber-to-cache traffic. Steady state was video plays per second. At 50% traffic, steady state deviated. The root cause was not a code bug but thread-pool configuration in an upstream service³. This is what chaos engineering most often finds: misconfigured timeouts, missing fallbacks, undersized thread pools, and unbounded retries.

Defense in depth

A mature chaos program is not a single practice you pick from a menu. The rows below are layered over 6 to 18 months: teams start in staging, graduate to bounded production experiments, then to continuous production chaos, while running game days in parallel the entire time. The “Add this layer when” column says when each layer earns its place, not which one to choose instead of the others.

Common Pitfalls

Exercise

Design a game day for your team: your Redis primary fails over. Define the steady-state metrics, write the hypothesis, choose the blast radius, plan the kill switch, write the runbook for the experiment, and plan the debrief.

Key Takeaways

• Chaos engineering is an empirical method (hypothesis, inject, measure, conclude), not random destruction.
• You do not have resilience until you have tested it under real failure. Staging-only chaos gives false confidence.
• Blast-radius control is what separates chaos engineering from sabotage: start with one pod, widen only after each phase passes.
• Every experiment needs a kill switch. AWS FIS supports up to five alarm-based stop conditions per template.
• What chaos engineering most often finds is not code bugs but misconfigured timeouts, missing fallbacks, undersized thread pools, and stale runbooks.
• Game days test the humans (runbooks, alerting, escalation), not just the system. Run them quarterly regardless of automation maturity.
• Cultural prerequisites (blameless debriefs, management buy-in, tracked action items) matter more than tooling.

Further Reading

• Principles of Chaos Engineering - The manifesto. Short, authoritative, and the single document every chaos program should align on before choosing tools.
• Chaos Engineering: System Resiliency in Practice by Rosenthal and Jones (O’Reilly, 2020) - The canonical book with case studies from Slack, LinkedIn, Capital One, Microsoft, and Google.
• The Netflix Simian Army - The original 2011 blog post that started the discipline; read for the “opt-out not opt-in” design decision.
• Chaos Engineering Upgraded - The 2015 Netflix post formalizing the experiment framework and introducing Chaos Kong region evacuation.
• Disasterpiece Theater - Richard Crowley’s 2019 writeup of Slack’s game-day process; the best guide for teams starting their first exercises.
• Weathering the Unexpected - Kripa Krishnan’s 2012 ACM Queue paper on Google DiRT; the best published source on company-wide game days.
• Resilience Engineering at LinkedIn with Project Waterbear - LinkedIn’s socio-technical approach covering LinkedOut, FireDrill, and D2Tuner; read for the cultural adoption playbook.
• Chaos Mesh documentation - The CNCF incubating Kubernetes-native chaos platform; start with PodChaos and NetworkChaos CRDs.

Flashcards

References