SRE Fundamentals: SLOs and Error Budgets

SLIs, SLOs, and SLAs

SLI (Service Level Indicator): A quantitative measure of service behavior
  Examples: request latency, error rate, availability, throughput

SLO (Service Level Objective): Target value for an SLI
  Example: 99.9% of requests respond in < 500ms over 30 days

SLA (Service Level Agreement): Legal/business contract with consequences
  SLO is internal target; SLA is customer-facing commitment
  SLO should be stricter than SLA (buffer for incidents)

Defining Good SLIs

# Availability SLI
sli:
  name: api_availability
  description: Ratio of successful HTTP responses to total requests
  formula: count(http_requests{status!~"5.."}) / count(http_requests)

# Latency SLI
sli:
  name: api_latency
  description: Fraction of requests faster than threshold
  formula: histogram_quantile(0.99, http_request_duration_seconds)
  threshold: 500ms

# Good SLIs:
# - Directly measurable
# - Proportional to user happiness
# - Not too many (3-5 per service)

Setting SLOs

Common SLO levels:
  99%    = 7.3 hours/month downtime  (3 nines)
  99.9%  = 43.8 minutes/month         (3.5 nines)
  99.95% = 21.9 minutes/month
  99.99% = 4.3 minutes/month          (4 nines)

Start conservative:
  1. Measure current performance
  2. Set SLO at current performance - 5%
  3. Tighten over time as you improve reliability

Example SLOs:
  - 99.9% of checkout requests succeed
  - 99.5% of API responses < 200ms (P99)
  - Error rate < 0.1% averaged over 1 hour

Error Budgets

Error Budget = 1 - SLO

SLO = 99.9% availability
Error Budget = 0.1% = 43.8 minutes/month

If 20 minutes of downtime occurred:
  Consumed: 20/43.8 = 45.6% of budget
  Remaining: 54.4% (23.8 minutes)

Policy based on budget:
  > 50% remaining: Deploy freely, experiment
  < 50% remaining: Slow down releases, focus on reliability
  Budget exhausted: Feature freeze, reliability work only

Prometheus-based SLO Monitoring

# SLO: 99.9% availability, measured over 30 days
# Recording rules for efficient queries
groups:
- name: slo_rules
  interval: 30s
  rules:
  # 1-minute error rate
  - record: slo:api_errors:rate1m
    expr: rate(http_requests_total{status=~"5.."}[1m]) / rate(http_requests_total[1m])

  # 1-hour error rate
  - record: slo:api_errors:rate1h
    expr: rate(http_requests_total{status=~"5.."}[1h]) / rate(http_requests_total[1h])

  # 30-day availability
  - record: slo:api_availability:30d
    expr: 1 - (rate(http_requests_total{status=~"5.."}[30d]) / rate(http_requests_total[30d]))

Burn Rate Alerting

# Multi-window, multi-burn-rate alerting
# Alert when error budget is burning too fast
groups:
- name: slo_alerts
  rules:
  # Fast burn: 2% budget in 1 hour (page immediately)
  - alert: SLOBurnRateFast
    expr: |
      (slo:api_errors:rate1h > 14.4 * 0.001)  # 14.4x burn rate
      and
      (slo:api_errors:rate5m > 14.4 * 0.001)
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "SLO burning at 14x rate - exhausts budget in 1 hour"

  # Slow burn: 5% budget in 6 hours (ticket)
  - alert: SLOBurnRateSlow
    expr: |
      (slo:api_errors:rate6h > 6 * 0.001)
      and
      (slo:api_errors:rate30m > 6 * 0.001)
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "SLO burning at 6x rate - exhausts budget in 5 days"

Error Budget Policy

## Error Budget Policy

**SLO**: 99.9% API availability (43.8 min/month budget)

**When budget is healthy (>50% remaining)**:
- Deploy multiple times per day
- Run experiments and A/B tests
- Accepted reliability risk for new features

**When budget is at risk (10-50% remaining)**:
- Code freeze unless critical fixes
- Double review for infrastructure changes
- Run postmortem on top incidents

**When budget is exhausted (<0% remaining)**:
- Complete feature freeze
- All engineering time on reliability
- Weekly review with engineering leadership

SLOs turn reliability from a feeling into a measurable engineering constraint.