Chaos Engineering: Testing System Resilience

Core Principles

1. Define steady state (normal behavior metrics)
2. Hypothesize: "System will maintain steady state when X fails"
3. Run experiment in production (or prod-like staging)
4. Learn and fix weaknesses
5. Automate and run continuously

Start small: Production chaos only after staging validation

Types of Chaos Experiments

Infrastructure:
  - Kill random pods (Chaos Monkey)
  - Network partition between services
  - High CPU/memory on nodes
  - Disk I/O latency/errors

Application:
  - Inject latency in service calls
  - Throw exceptions in dependencies
  - Corrupt payloads
  - Cause dependency timeouts

State:
  - Fill disk space
  - Fill database with garbage
  - Clear caches (Redis/CDN)
  - Revert database connections

Litmus Chaos (Kubernetes)

# chaos-experiment.yaml - Kill random pod
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-kill-experiment
spec:
  appinfo:
    appns: production
    applabel: app=api-server
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
        - name: TOTAL_CHAOS_DURATION
          value: '60'      # 60 seconds
        - name: CHAOS_INTERVAL
          value: '10'      # Kill pod every 10s
        - name: FORCE
          value: 'false'   # Graceful termination

# Network chaos: Add 200ms latency to payment service calls
- name: pod-network-latency
  spec:
    components:
      env:
      - name: NETWORK_LATENCY
        value: '200'
      - name: DESTINATION_IPS
        value: "10.0.0.1"  # Payment service IP
      - name: TOTAL_CHAOS_DURATION
        value: '300'

Application-Level Fault Injection

class FaultInjectionMiddleware {
  constructor(private faultRate: number = 0) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (process.env.NODE_ENV !== 'production' || !this.isEnabled()) {
      return fn();
    }

    if (Math.random() < this.faultRate) {
      const fault = this.pickFault();
      console.log(`[CHAOS] Injecting fault: ${fault.type}`);

      switch (fault.type) {
        case 'latency':
          await new Promise(r => setTimeout(r, fault.delayMs));
          break;
        case 'error':
          throw new Error('Injected fault');
        case 'timeout':
          await new Promise(r => setTimeout(r, 30000)); // Cause timeout
          break;
      }
    }

    return fn();
  }

  private isEnabled(): boolean {
    return process.env.CHAOS_ENABLED === 'true';
  }
}

Game Day Planning

## Game Day Template

**Date**: 2024-03-15
**Hypothesis**: "When the payment service is unavailable, the checkout flow degrades gracefully,
                 showing an error message while cart and browsing remain functional."

**Experiment**:
1. Kill all payment-service pods
2. Verify cart/browsing still works
3. Verify checkout shows error (not 500)
4. Verify alert fires within 5 minutes
5. Verify payment service recovers on restart

**Steady State**:
- Checkout success rate > 99%
- Error rate < 0.1%

**Rollback Plan**:
- Restart payment-service pods immediately if...

**Results**:
- What happened:
- Weaknesses found:
- Fixes required:

Measuring Resilience

// Mean Time to Detect (MTTD)
// Mean Time to Recover (MTTR)
// Error Budget consumption

async function measureResilience(experiment: ChaosExperiment): Promise<ResilienceReport> {
  const startTime = Date.now();
  await experiment.inject();

  const detectionTime = await waitForAlert();
  const recoveryTime = await waitForSteadyState();

  return {
    mttd: detectionTime - startTime,
    mttr: recoveryTime - startTime,
    impactDuration: recoveryTime - detectionTime,
    errorsGenerated: await countErrors(startTime, recoveryTime),
  };
}

Chaos engineering transforms theoretical resilience into proven resilience.