Tutorial: Learn Chaos Engineering from Scratch (2026)
I remember the first time I deliberately killed a production server — my heart was pounding. But watching the system route around the failure seamlessly was exhilarating. That is the promise of chaos engineering: by intentionally breaking things in a controlled way, you build confidence that your system can withstand real failures.
Chaos engineering is the discipline of experimenting on a distributed system to build confidence in its capability to withstand turbulent conditions. Netflix pioneered it with Chaos Monkey, and tools like Gremlin have made it accessible to every engineering team. This tutorial covers the principles, tools, and practices of chaos engineering.
Principles of Chaos Engineering
Chaos engineering follows a scientific method. Start by defining a steady state — measurable normal behavior like response times under 200ms and error rates below 0.1%. Formulate a hypothesis that the system remains in steady state despite a specific failure. Run the experiment by introducing the failure. Compare the result against the hypothesis.
The five principles are: build a hypothesis around steady state, vary real-world events, run experiments in production (or production-like environments), automate experiments to run continuously, and minimize blast radius. Start with small experiments and expand scope as confidence grows.
// Chaos experiment structure
Hypothesis: System maintains error rate < 0.1%
when one EC2 instance is terminated
Metrics monitored:
- HTTP error rate
- P95 response time
- Active instance count
- Queue depthNetflix Chaos Monkey and Simian Army
Chaos Monkey, part of Netflix's Simian Army, randomly terminates EC2 instances in production. It runs during business hours to ensure engineers are available to respond. The Simian Army includes Chaos Gorilla (simulates an AZ failure), Chaos Kong (simulates a region failure), and Latency Monkey (injects network delays).
Netflix open-sourced Chaos Monkey through the Spinnaker platform. It integrates with AWS, Azure, and Google Cloud. Configuration controls the probability of termination, which instance types to target, and exclusion windows. Start with a small probability in a staging environment before enabling it in production.
# Chaos Monkey configuration (Spinnaker)
chaos.enabled: true
chaos.terminationProbability: 0.1
chaos.maxTerminationsPerDay: 3
chaos.excludedAccounts:
- production-us-east-1
- production-eu-west-1
# Schedule: weekdays 9 AM - 5 PM
chaos.scheduling.enabled: true
chaos.scheduling.days: MON-FRI
chaos.scheduling.startHour: 9
chaos.scheduling.endHour: 17Running Chaos Experiments with Gremlin
Gremlin is a commercial chaos engineering platform that simplifies running experiments. It supports attacks on infrastructure (CPU, memory, disk, network), state (process killing, shutdown), and Kubernetes resources (pod deletion, node drain). Attacks can target specific hosts, containers, or entire deployments.
Create an experiment in the Gremlin web console or via API. Define the attack type, target, duration, and blast radius. Gremlin provides safety features — halt conditions, team permissions, and automatic rollback. I start with a 30-second CPU attack on one instance before progressing to longer, broader experiments.
# Gremlin CLI: CPU attack on one host
gremlin attack cpu --capacity 80 --length 60 --target host-123
# Gremlin API: Blackhole network traffic
gremlin attack network blackhole --length 30 --target app-server
# Kubernetes: Kill pods in deployment
gremlin attack kubernetes pod --deployment api-service --length 120Building a Chaos Experiment Pipeline
Treat chaos experiments like code. Define experiments as configuration files, version them in git, and run them in CI/CD. Start with a staging environment, run experiments after deployment, and validate that the system survives. Gradually promote experiments to production.
I use a pipeline structure: deploy the application, run smoke tests, execute chaos experiments, verify steady state, and generate a resilience report. If an experiment fails, the pipeline fails, and the team investigates before the next deployment. This embeds resilience testing into the delivery process.
# Experiment definition (YAML)
apiVersion: chaos.gremlin.com/v1
kind: Experiment
metadata:
name: kill-api-pod
spec:
target:
type: kubernetes
pods:
- deployment: api-service
attack:
type: process_killer
process: java
duration: 60
blastRadius: 1
haltConditions:
- errorRate > 5%Observability and Steady State Validation
Chaos engineering is useless without observability. Before running experiments, ensure you can measure the steady state — metrics, logs, and traces that define normal behavior. During experiments, monitor the same signals in real time. After experiments, verify that metrics return to baseline.
Critical metrics include request latency (p50, p95, p99), error rate, throughput, CPU/memory utilization, database connection pool usage, and queue depth. Set up alerts that trigger when metrics deviate from steady state. A chaos experiment that causes no degradation is as informative as one that does.
# Observability during chaos experiment
"""
Metrics to watch:
http.requests.latency.p95 < 500ms
http.errors.rate < 0.5%
jvm.heap.usage < 70%
db.pool.active < 80
kafka.consumer.lag < 100
"""
# Automated validation
assert metrics.p95_latency < 500, f"Latency too high: {metrics.p95_latency}"
assert metrics.error_rate < 0.5, f"Error rate too high: {metrics.error_rate}%"Chaos Engineering Best Practices and Adoption
Start small — run game days where the team manually introduces failures and practices incident response. Then automate the most common failure scenarios: instance termination, network latency, database failover, and dependency unavailability. Always run experiments during business hours with engineering on standby.
Blast radius is the most important safety concept. Limit each experiment to a single instance, availability zone, or service. Use feature flags to exclude critical customer-facing features during experiments. Document every experiment, including the hypothesis, results, and remediation actions. Chaos engineering is a journey, not a destination.
# Blast radius configuration
experiment:
maxTargets: 1
maxDuration: 300
allowedHours: "09:00-17:00"
excludedServices:
- payment-service
- auth-service
haltOn:
errorRate: 2
p95Latency: 1000
notification:
slack: "#chaos-engineering"
pagerduty: "chaos-engineering-team"Frequently Asked Questions
Is chaos engineering the same as failure testing?
Failure testing confirms that a system handles a known failure. Chaos engineering is more exploratory — you introduce unknown or rare failure modes to discover weaknesses. It is about building confidence through experimentation rather than verifying known scenarios.
Should I run chaos experiments in production?
Start in staging and test environments. Netflix runs Chaos Monkey in production because they have mature observability and incident response. Most teams should begin in pre-production and only move to production when they have safety mechanisms and team readiness.
What is the blast radius in chaos engineering?
Blast radius is the scope of impact an experiment can have. A small blast radius affects one instance or service. A large blast radius affects multiple services or an entire region. Always minimize blast radius, especially when starting chaos engineering.
Can chaos engineering work for monolithic applications?
Yes, but the value is higher for distributed systems. For monoliths, focus on infrastructure attacks (CPU, memory, disk) and dependency failures (database, cache, external APIs). The principles still apply — define steady state, introduce failure, and measure impact.
Originally published on Ayodhyyya. Last updated June 1, 2026.