Kubernetes Production Patterns: Health Checks, Resource Limits, RBAC, and Observability

The Gap Between "It Works" and "Production Ready"

Getting an application running on Kubernetes is straightforward. Getting it production-ready — with proper resource limits, security policies, observability, and operational runbooks — takes considerably more work.

This guide covers the patterns and configurations that experienced Kubernetes engineers apply before going to production.

Resource Requests and Limits: Getting Them Right

Incorrect resource requests are one of the most common causes of production issues:

resources:
  requests:
    memory: "256Mi"   # REQUESTED: scheduler uses this for placement
    cpu: "100m"       # Node must have 256Mi + 100m available
  limits:
    memory: "512Mi"   # LIMIT: container killed if it exceeds this (OOMKilled)
    cpu: "1000m"      # LIMIT: CPU throttled (not killed) if it exceeds this

Setting the right values:

# Step 1: Deploy without limits first (dev/staging)
# Step 2: Monitor actual usage
kubectl top pods --containers

# Or use Vertical Pod Autoscaler in recommendation mode:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # "Off" = recommendation only, won't change running pods

Common mistakes:

# ❌ No requests — scheduler doesn't know where to place pods
# ❌ Request = Limit — no room for burst, constant throttling
# ❌ Very high CPU limit — starves other pods on the node
# ❌ Very high memory limit — can cause OOM on the node itself

# ✅ Pattern: requests ≈ average usage, limits = 2-3x requests
resources:
  requests:
    memory: "256Mi"   # ~average memory usage
    cpu: "100m"       # ~average CPU usage
  limits:
    memory: "512Mi"   # 2x for spikes
    cpu: "500m"       # 5x for CPU burst (throttling > killing)

PodDisruptionBudgets

PDBs prevent Kubernetes from draining too many pods simultaneously (during node maintenance, upgrades):

# Ensure at least 2 pods are always available
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2      # OR: maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

# Check PDB status
kubectl get pdb
# NAME        MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
# my-app-pdb  2              N/A              1                    5d

# When you try to delete a pod that would violate PDB:
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
# Error: Cannot evict pod as it would violate the pod's disruption budget.

RBAC: Role-Based Access Control

# ServiceAccount for your application (principle of least privilege)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: production
---
# Role: what permissions are allowed (namespace-scoped)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-app-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["my-app-secrets"]  # Specific secret only
  verbs: ["get"]
---
# RoleBinding: bind role to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: production
roleRef:
  kind: Role
  apiGroupGroup: rbac.authorization.k8s.io
  name: my-app-role

# Developer access — read-only to specific namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-viewer
rules:
- apiGroups: ["", "apps", "autoscaling"]
  resources: ["pods", "deployments", "replicasets", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: []  # No exec access for security
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-team-viewer
  namespace: staging
subjects:
- kind: Group
  name: "dev-team"  # From your OIDC provider
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: namespace-viewer
  apiGroup: rbac.authorization.k8s.io

Network Policies

By default, all pods can communicate with all other pods. Network policies add firewall rules:

# Default deny all ingress in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}  # Applies to all pods
  policyTypes:
  - Ingress
---
# Allow specific traffic to the API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend   # Only from frontend pods
    - namespaceSelector:
        matchLabels:
          name: monitoring  # And from monitoring namespace
    ports:
    - protocol: TCP
      port: 3000
---
# Allow database only from API pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-from-api
spec:
  podSelector:
    matchLabels:
      app: postgres
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api
    ports:
    - protocol: TCP
      port: 5432

Secrets Management with External Secrets Operator

Storing secrets in Kubernetes Secrets isn't ideal (base64 encoded, in etcd). Use External Secrets Operator with AWS Secrets Manager or HashiCorp Vault:

# After installing External Secrets Operator:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: db-credentials   # Creates this K8s Secret
    creationPolicy: Owner
  data:
  - secretKey: url          # K8s Secret key
    remoteRef:
      key: production/db    # AWS Secrets Manager path
      property: url         # JSON property in the secret
  - secretKey: password
    remoteRef:
      key: production/db
      property: password

Observability Stack

# ServiceMonitor (Prometheus Operator) — scrape your app's metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-metrics
  labels:
    release: prometheus   # Must match Prometheus operator's selector
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics         # Port name in Service
    path: /metrics
    interval: 30s
---
# PrometheusRule — define alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
spec:
  groups:
  - name: my-app
    rules:
    - alert: HighErrorRate
      expr: |
        rate(http_requests_total{status=~"5..", app="my-app"}[5m]) /
        rate(http_requests_total{app="my-app"}[5m]) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate on my-app"
        description: "Error rate is {{ $value | humanizePercentage }}"
    
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: critical

Deployment Strategies

# Blue-Green Deployment with Services
---
# Blue deployment (current)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: blue
  template:
    metadata:
      labels:
        app: my-app
        version: blue
    spec:
      containers:
      - name: app
        image: my-app:1.0.0
---
# Green deployment (new version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: green
  template:
    metadata:
      labels:
        app: my-app
        version: green
    spec:
      containers:
      - name: app
        image: my-app:2.0.0
---
# Service — switch by changing selector
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  selector:
    app: my-app
    version: blue  # ← Change to "green" to switch traffic
  ports:
  - port: 80
    targetPort: 3000

# Canary deployment with weighted traffic splitting
# (using Nginx Ingress annotations)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"  # 10% to new version
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-v2   # New version
            port:
              number: 80

Init Containers and Sidecars

spec:
  initContainers:
  # Run before main containers start
  - name: migrate-db
    image: my-app:2.0.0
    command: ["./migrate", "--up"]
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: url
  
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 
      'until nc -z postgres-service 5432; do echo waiting; sleep 2; done']
  
  containers:
  - name: app
    image: my-app:2.0.0
  
  # Sidecar containers
  - name: log-shipper
    image: fluent-bit:latest
    # Ships logs from shared volume to logging system
    volumeMounts:
    - name: log-volume
      mountPath: /var/log/app
  
  - name: cloud-sql-proxy  # Common pattern for GCP Cloud SQL
    image: gcr.io/cloudsql-docker/gce-proxy:latest
    command: ["/cloud_sql_proxy", "-instances=project:region:instance=tcp:5432"]

The biggest productivity gain from learning Kubernetes production patterns is understanding that the declarative model means you describe what you want, and the cluster continuously converges to that state. Debugging is mostly a matter of asking: "What's the desired state? What's the actual state? Why don't they match?"

→ Calculate Linux file permissions with the Chmod Calculator.