Prometheus and Grafana Production Setup: Scrape Config, Alerting Rules, and Dashboard Best Practices

Prometheus and Grafana form the backbone of observability for most Kubernetes-based platforms. Prometheus collects and stores time-series metrics; Grafana visualizes them and integrates with Alertmanager for notifications. This guide covers a complete production setup from first scrape to actionable alerts.

Architecture Overview

Application --> Prometheus --> Alertmanager --> PagerDuty / Slack
                    |
                 Grafana (dashboards)
                    |
                 Thanos (long-term storage)

Key components:

• Prometheus server scrapes targets, evaluates rules, stores TSDB data
• Alertmanager deduplicates, groups, and routes alerts
• Pushgateway accepts metrics from batch jobs
• Exporters expose metrics from third-party systems

Installing with kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values prometheus-values.yaml

Production values.yaml

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 50GB
    resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 4Gi
        cpu: 2
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 100Gi
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

grafana:
  adminPassword: "changeme"
  persistence:
    enabled: true
    size: 10Gi

alertmanager:
  alertmanagerSpec:
    resources:
      requests:
        memory: 128Mi

Scrape Configuration

ServiceMonitor for Kubernetes Services

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-api
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: my-api
  namespaceSelector:
    matchNames:
      - production
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: namespace

Static Scrape Config for External Systems

- job_name: 'blackbox-http'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://api.example.com/health
        - https://app.example.com
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

PromQL Essentials

Error Rate

100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

Latency Percentiles

histogram_quantile(0.99,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Resource Utilization

# CPU usage per pod (%)
100 * sum by (pod, namespace) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
) / sum by (pod, namespace) (
  kube_pod_container_resource_limits{resource="cpu"}
)

# Memory usage vs limits
sum by (pod) (container_memory_working_set_bytes{container!=""})
  / sum by (pod) (kube_pod_container_resource_limits{resource="memory"})

Alerting Rules

PrometheusRule Resource

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: api.rules
      interval: 30s
      rules:
        - alert: HighErrorRate
          expr: |
            (
              sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
              /
              sum(rate(http_requests_total[5m])) by (service)
            ) > 0.01
          for: 5m
          labels:
            severity: warning
            team: backend
          annotations:
            summary: "High error rate on {{ $labels.service }}"
            description: "Error rate is {{ $value | humanizePercentage }}"
            runbook_url: "https://runbooks.example.com/high-error-rate"

        - alert: HighLatency
          expr: |
            histogram_quantile(0.99,
              sum by (le, service) (
                rate(http_request_duration_seconds_bucket[5m])
              )
            ) > 0.5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High P99 latency on {{ $labels.service }}"

        - alert: ServiceDown
          expr: up{job=~"my-api.*"} == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Service {{ $labels.job }} is down"

Multi-Window Burn-Rate SLO Alerts

- alert: ErrorBudgetBurnHigh
  expr: |
    (
      job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
    ) and (
      job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001)
    )
  labels:
    severity: critical
    page: "true"
  annotations:
    summary: "High error budget burn rate"
    description: "Burning 14.4x budget. Investigate immediately."

Alertmanager Configuration

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: default
  routes:
    - matchers:
        - severity = critical
      receiver: pagerduty
    - matchers:
        - team = backend
      receiver: slack-backend

receivers:
  - name: default
    slack_configs:
      - channel: '#alerts'

  - name: pagerduty
    pagerduty_configs:
      - service_key: 'PAGERDUTY_KEY'

  - name: slack-backend
    slack_configs:
      - channel: '#backend-alerts'
        send_resolved: true

inhibit_rules:
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: ['alertname', 'service']

Grafana Dashboard Best Practices

Organize dashboards in a hierarchy:

1. Overview — RED metrics (Rate, Errors, Duration) across all services
2. Service — drill-down for a single service
3. Infrastructure — node/cluster resource utilization
4. Business — user-facing KPIs

Variable Templates for Reusable Dashboards

{
  "templating": {
    "list": [
      {
        "name": "namespace",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(kube_pod_info, namespace)",
        "refresh": 2
      },
      {
        "name": "service",
        "type": "query",
        "query": "label_values(http_requests_total{namespace=\"$namespace\"}, service)",
        "refresh": 2
      }
    ]
  }
}

Long-Term Storage with Thanos

containers:
  - name: thanos-sidecar
    image: quay.io/thanos/thanos:v0.35.0
    args:
      - sidecar
      - --tsdb.path=/prometheus
      - --prometheus.url=http://localhost:9090
      - --objstore.config-file=/etc/thanos/objstore.yaml

# objstore.yaml
type: S3
config:
  bucket: my-prometheus-metrics
  endpoint: s3.amazonaws.com
  region: us-east-1

Conclusion

A production Prometheus/Grafana stack requires careful attention to retention policies, alerting logic, and dashboard design. ServiceMonitors make Kubernetes service discovery declarative. Multi-window burn-rate alerts provide SLO-based reliability signaling. Thanos extends storage beyond local TSDB limits. With these patterns, your monitoring stack becomes a first-class reliability tool rather than an afterthought.