OpenTelemetry in Production: Distributed Tracing, Metrics, and Log Correlation

Observability answers questions you didn't anticipate when building the system. OpenTelemetry standardizes instrumentation across languages and vendors.

Three Pillars

• Traces: Request path across services (latency, causality)
• Metrics: Aggregated numbers (P99, error rate, throughput)
• Logs: Discrete events with context

Power comes from correlation: metrics spike → failing trace → log explaining why.

Initialize Tracer (Go)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exporter, _ := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("order-service"),
            semconv.ServiceVersion("1.2.0"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

Instrument Functions

var tracer = otel.Tracer("order-service")

func (s *service) CreateOrder(ctx context.Context, req *pb.Request) (*pb.Order, error) {
    ctx, span := tracer.Start(ctx, "CreateOrder",
        trace.WithAttributes(
            attribute.String("customer.id", req.CustomerId),
            attribute.Int("items.count", len(req.Items)),
        ),
    )
    defer span.End()

    order, err := s.db.CreateOrder(ctx, req)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "database error")
        return nil, err
    }
    span.SetAttributes(attribute.String("order.id", order.ID))
    return orderToProto(order), nil
}

Custom Metrics

func initMetrics() {
    exporter, _ := prometheus.New()
    meter := metric.NewMeterProvider(metric.WithReader(exporter)).Meter("order-service")

    orderCounter, _ = meter.Int64Counter("orders_total",
        metric.WithDescription("Total orders created"))
    orderLatency, _ = meter.Float64Histogram("order_duration_seconds",
        metric.WithExplicitBucketBoundaries(0.001, 0.01, 0.1, 0.5, 1.0, 5.0))
}

Log-Trace Correlation

func logWithTrace(ctx context.Context, logger *zap.Logger, msg string, fields ...zap.Field) {
    span := trace.SpanFromContext(ctx)
    if span.IsRecording() {
        sc := span.SpanContext()
        fields = append(fields,
            zap.String("trace_id", sc.TraceID().String()),
            zap.String("span_id", sc.SpanID().String()),
        )
    }
    logger.Info(msg, fields...)
}

OTel Collector Config

receivers:
  otlp:
    protocols:
      grpc: {endpoint: 0.0.0.0:4317}
      http: {endpoint: 0.0.0.0:4318}

processors:
  batch: {timeout: 1s, send_batch_size: 1024}

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls: {insecure: true}
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Observability is a team discipline. The technology is standardized. The work is instrumenting the right events and building culture around reading them.

→ Decode Base64 trace IDs with the Base64 Converter tool.