Build a robust SRE incident management practice: design effective on-call rotations, run blameless post-mortems, define meaningful SLOs and SLIs, and use error budgets to balance reliability and feature velocity.

SRE Incident Management: On-Call Rotation, Post-Mortems, SLO/SLI/SLA Definitions, and Error Budget Management

Site Reliability Engineering brings software engineering discipline to operations. At its core, SRE is about defining what reliability means, measuring it objectively, and making data-driven trade-offs between reliability and feature velocity. This guide covers the complete incident management lifecycle from on-call design to error budget policies.

The SRE Reliability Framework

Business Requirements --> SLA (external commitment)
       |
       v
Engineering Targets --> SLO (internal objective, more strict than SLA)
       |
       v
Measurements --> SLI (actual metric you measure)
       |
       v
Error Budget = 100% - SLO target

Defining SLIs (Service Level Indicators)

An SLI is a quantitative measurement of service behavior. Choose SLIs that reflect the user experience:

Request-Based SLIs

# Availability SLI
availability_sli = good_requests / total_requests

# Where "good" means:
# - Status code < 500
# - Response time < 500ms
# - Response body is valid JSON

# Latency SLI
latency_sli = requests_under_threshold / total_requests
# e.g., requests completing in < 200ms

PromQL SLI Calculation

# Availability SLI over 30 days
(
  sum(rate(http_requests_total{status!~"5.."}[30d]))
  /
  sum(rate(http_requests_total[30d]))
)

# Latency SLI: fraction of requests < 300ms
(
  sum(rate(http_request_duration_seconds_bucket{le="0.3"}[30d]))
  /
  sum(rate(http_request_duration_seconds_count[30d]))
)

Appropriate SLIs by Service Type

Service Type	Primary SLI	Secondary SLI
User-facing API	Availability	Latency (P99)
Data pipeline	Freshness	Completeness
Storage	Durability	Throughput
Batch job	Completion rate	Duration

Setting SLOs (Service Level Objectives)

SLOs are the internal targets you hold yourself to. They should be tighter than the SLA to give a buffer.

# slo-config.yaml
slos:
  - name: "API Availability"
    description: "Percentage of successful requests"
    sli:
      metric: availability
      query: |
        sum(rate(http_requests_total{status!~"5.."}[30d]))
        / sum(rate(http_requests_total[30d]))
    target: 0.999  # 99.9% = ~43.8 min downtime/month
    window: 30d
    
  - name: "API Latency P99"
    description: "99th percentile request latency under 500ms"
    sli:
      metric: latency
      query: |
        histogram_quantile(0.99,
          sum by (le) (rate(http_request_duration_seconds_bucket[30d]))
        ) < 0.5
    target: 0.95   # 95% of time P99 is under 500ms
    window: 30d

Error Budgets

The error budget is the allowed unreliability:

Error Budget = 1 - SLO Target
99.9% SLO => 0.1% error budget => 43.8 minutes/month
99.5% SLO => 0.5% error budget => 3.65 hours/month
99.0% SLO => 1.0% error budget => 7.3 hours/month

Error Budget Policy

Error Budget > 50% remaining:
  - Feature development proceeds at normal pace
  - Experiments and risky deployments allowed

Error Budget 10-50% remaining:
  - Increase monitoring and alerting sensitivity
  - Postmortem required for all incidents
  - Defer non-critical risky deployments

Error Budget < 10% remaining:
  - Feature freeze: only reliability work
  - All deployments require approval
  - Incident commander assigned

Error Budget exhausted (0%):
  - Full freeze until new budget period
  - Executive notification required
  - Mandatory reliability sprint

On-Call Rotation Design

Rotation Principles

Primary/Secondary pairing — backup for every shift
Follow-the-sun — route alerts to the awake engineer
Load balancing — distribute incidents fairly
Escalation paths — clear chain of contact

# PagerDuty schedule (via Terraform)
resource "pagerduty_schedule" "primary" {
  name      = "Backend Primary On-Call"
  time_zone = "UTC"

  layer {
    name                         = "Weekly Rotation"
    start                        = "2026-01-01T00:00:00+00:00"
    rotation_virtual_start       = "2026-01-01T00:00:00+00:00"
    rotation_turn_length_seconds = 604800  # 1 week

    users = [
      pagerduty_user.alice.id,
      pagerduty_user.bob.id,
      pagerduty_user.carol.id,
      pagerduty_user.dave.id,
    ]
  }
}

resource "pagerduty_escalation_policy" "backend" {
  name = "Backend Escalation"

  rule {
    escalation_delay_in_minutes = 5
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.primary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 10
    target {
      type = "schedule_reference"
      id   = pagerduty_schedule.secondary.id
    }
  }

  rule {
    escalation_delay_in_minutes = 15
    target {
      type = "user_reference"
      id   = pagerduty_user.engineering_manager.id
    }
  }
}

On-Call Runbook Template

# Service: Payment API

## Quick Reference
- Dashboard: https://grafana.example.com/d/payment-api
- Logs: https://kibana.example.com/app/payment-api
- Runbooks: https://wiki.example.com/runbooks/payment-api

## Alert: HighErrorRate
**Severity:** Warning / Critical (>5%)

**Symptoms:** Users cannot complete checkout

**Investigation Steps:**
1. Check error rate dashboard
2. grep logs for 5xx errors: `kubectl logs -n production -l app=payment-api --since=10m | grep "ERROR"`
3. Check database connection pool: `kubectl exec -n production deploy/payment-api -- /health/db`
4. Check upstream payment provider status: https://status.stripe.com

**Remediation:**
- If DB connections exhausted: `kubectl rollout restart deploy/payment-api -n production`
- If upstream provider outage: switch to fallback provider in config map
- If code bug: roll back: `kubectl rollout undo deploy/payment-api -n production`

**Escalation:** If not resolved in 30min, page payment-team-lead

Incident Response Process

Severity Levels

Level	User Impact	Response Time	Examples
SEV-1	Complete outage	Immediate	Checkout down, 100% error rate
SEV-2	Major degradation	15 min	>50% errors, >10x latency
SEV-3	Minor degradation	1 hour	Elevated errors, single region
SEV-4	Minimal impact	Next business day	Non-critical feature down

Incident Commander Role

Incident Commander responsibilities:
1. Declare incident severity
2. Establish communication channel (#incident-YYYY-MM-DD-description)
3. Assign roles: Technical Lead, Communications Lead, Scribe
4. Provide status updates every 15min (SEV-1) or 30min (SEV-2)
5. Make go/no-go decisions for remediations
6. Declare incident resolved
7. Schedule post-mortem within 48 hours

Blameless Post-Mortems

The goal is learning, not blame. Focus on systems and processes, not people.

Post-Mortem Template

# Post-Mortem: Payment Service Outage - 2026-05-15

**Status:** Complete
**Severity:** SEV-1
**Duration:** 47 minutes (14:23 UTC to 15:10 UTC)
**Impact:** ~12,000 users unable to complete checkout; ~$180k estimated revenue impact

## Timeline

| Time (UTC) | Event |
|---|---|
| 14:20 | Deploy of payment-api v2.4.1 started |
| 14:23 | Alert: ErrorRate > 50% (PagerDuty) |
| 14:25 | On-call engineer acknowledged |
| 14:28 | Incident declared SEV-1 |
| 14:35 | Root cause identified: DB connection string misconfiguration |
| 14:50 | Rollback initiated |
| 15:05 | Error rate back to baseline |
| 15:10 | Incident resolved |

## Root Cause

The v2.4.1 deploy introduced a configuration change that used the staging database 
connection string in production. The misconfiguration was not caught in CI because 
integration tests run against a mock, not a real database.

## Contributing Factors

1. No validation that DB connection strings match expected environment
2. Deploy pipeline did not run smoke tests against production after deploy
3. Configuration management allowed staging values in production config files

## Action Items

| Action | Owner | Due Date | Priority |
|---|---|---|---|
| Add config validation that detects staging values in prod | Platform Team | 2026-05-22 | P1 |
| Add post-deploy smoke tests to deployment pipeline | DevOps Team | 2026-05-29 | P1 |
| Implement config schema validation in CI | Backend Team | 2026-06-05 | P2 |
| Review all service configurations for similar issues | All Teams | 2026-06-12 | P2 |

## What Went Well

- Alert fired within 3 minutes of deploy
- Team assembled quickly via incident channel
- Rollback procedure was well-documented and executed quickly

Error Budget Burn Rate Alerts

Calculate burn rate to detect budget exhaustion before it happens:

# Current burn rate (how fast we are consuming error budget)
# 1.0 = consuming at exactly the SLO rate
# 14.4 = consuming 14.4x faster than budget allows (will exhaust in 2 hours at 99.9% SLO)

(1 - (
  sum(rate(http_requests_total{status!~"5.."}[1h]))
  / sum(rate(http_requests_total[1h]))
)) / (1 - 0.999)

Alert thresholds based on time-to-budget-exhaustion:

Burn rate > 14.4 for 1h and 5m = page (budget exhausted in 2 hours)
Burn rate > 6 for 6h and 30m = ticket (budget exhausted in 5 days)

Conclusion

Effective SRE incident management transforms reactive firefighting into a systematic reliability engineering practice. SLIs provide objective measurements of user experience. SLOs set aspirational targets aligned with business needs. Error budgets create a shared language between engineering and product for risk tolerance. Blameless post-mortems generate organizational learning from failures. Together, these practices create a virtuous cycle of improving reliability over time.

页面加载失败

SRE Incident Management: On-Call Rotation, Post-Mortems, SLO/SLI/SLA Definitions, and Error Budget Management

SRE Incident Management: On-Call Rotation, Post-Mortems, SLO/SLI/SLA Definitions, and Error Budget Management

The SRE Reliability Framework

Defining SLIs (Service Level Indicators)

Request-Based SLIs

PromQL SLI Calculation

Appropriate SLIs by Service Type

Setting SLOs (Service Level Objectives)

Error Budgets

Error Budget Policy

On-Call Rotation Design

Rotation Principles

On-Call Runbook Template

Incident Response Process

Severity Levels

Incident Commander Role

Blameless Post-Mortems

Post-Mortem Template

Error Budget Burn Rate Alerts

Conclusion