Service Mesh with Istio: Traffic Management, mTLS, and Observability at Scale

Service meshes move cross-cutting concerns from application code to the infrastructure layer. Istio is the most production-proven implementation.

Architecture

• Istiod: Manages configuration, certificates, and service discovery
• Envoy sidecars: Injected into every pod; intercept all traffic
• Gateways: Ingress/egress traffic control

kubectl label namespace production istio-injection=enabled

Canary Deployment

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
            subset: v1
          weight: 90
        - destination:
            host: payment-service
            subset: v2
          weight: 10
      timeout: 5s
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: "5xx,reset,connect-failure"
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  subsets:
    - name: v1
      labels: {version: v1}
    - name: v2
      labels: {version: v2}
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s

Mutual TLS

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-authz
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/production/sa/order-service"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/api/v*/payments*"]

Fault Injection for Resilience Testing

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: notification-chaos
spec:
  hosts:
    - notification-service
  http:
    - fault:
        delay:
          percentage: {value: 20.0}
          fixedDelay: 500ms
        abort:
          percentage: {value: 5.0}
          httpStatus: 503
      route:
        - destination:
            host: notification-service

Propagate Trace Headers

TRACE_HEADERS = ['x-request-id', 'x-b3-traceid', 'x-b3-spanid',
                 'x-b3-parentspanid', 'x-b3-sampled']

@app.middleware("http")
async def forward_trace_headers(request: Request, call_next):
    request.state.trace_headers = {
        h: request.headers[h] for h in TRACE_HEADERS if h in request.headers
    }
    return await call_next(request)

Istio pays dividends when you have dozens of services needing consistent security and observability.

→ Decode service mesh JWT tokens with the JWT Parser tool.