OpenTelemetry Distributed Tracing: Auto-Instrumentation, Custom Spans, Sampling Strategies, and Jaeger Integration

Distributed tracing answers the question "why was this request slow?" across service boundaries. OpenTelemetry (OTel) provides a vendor-neutral SDK and specification that works with any backend including Jaeger, Tempo, Zipkin, Honeycomb, or Datadog. This guide covers instrumentation from auto-magic to fine-grained manual spans, with production sampling strategies that control cost without losing critical data.

Core Concepts

• Trace is a collection of spans representing a single request journey
• Span is a named, timed operation; spans form a tree
• Context Propagation passes trace context (TraceID, SpanID) across service boundaries via HTTP headers (W3C TraceContext)
• Exporter sends spans to a backend (OTLP, Jaeger, Zipkin)
• Collector receives, processes, and exports telemetry

Architecture

Service A --[OTLP/gRPC]--> OTel Collector --[OTLP]--> Jaeger
Service B --[OTLP/HTTP]--> OTel Collector --[OTLP]--> Grafana Tempo

The Collector decouples your services from the backend, enabling backend switching without code changes.

Node.js Auto-Instrumentation

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/exporter-metrics-otlp-grpc

// tracing.ts - must be loaded before any other module
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { ParentBasedSampler, TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base';

const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': process.env.SERVICE_NAME || 'my-service',
    'service.version': process.env.SERVICE_VERSION || '1.0.0',
    'deployment.environment': process.env.NODE_ENV || 'development',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (req) =>
          req.url?.includes('/health') || req.url?.includes('/metrics'),
      },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enhancedDatabaseReporting: true },
      '@opentelemetry/instrumentation-redis': { enabled: true },
    }),
  ],
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(0.1),
  }),
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

// server.ts
import './tracing';  // must be first import
import express from 'express';

Custom Spans

Auto-instrumentation covers HTTP, database, and cache calls. Add custom spans for business-level operations:

import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service', '1.0.0');

async function processPayment(orderId: string, amount: number) {
  return tracer.startActiveSpan('payment.process', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'order.id': orderId,
      'payment.amount': amount,
      'payment.currency': 'USD',
    },
  }, async (span) => {
    try {
      const fraudResult = await tracer.startActiveSpan('payment.fraud_check', async (fraudSpan) => {
        try {
          const result = await fraudCheckService.check(orderId, amount);
          fraudSpan.setAttributes({
            'fraud.score': result.score,
            'fraud.decision': result.decision,
          });
          return result;
        } finally {
          fraudSpan.end();
        }
      });

      if (fraudResult.decision === 'block') {
        span.setStatus({ code: SpanStatusCode.ERROR, message: 'Blocked' });
        throw new PaymentBlockedError(fraudResult.reason);
      }

      const result = await chargeCard(orderId, amount);
      span.setAttributes({ 'payment.transaction_id': result.transactionId });
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Adding Events (Logs within Spans)

span.addEvent('cache.miss', {
  'cache.key': cacheKey,
  'cache.size_bytes': keySize,
});

span.addEvent('retry.attempt', {
  'retry.count': attemptNumber,
  'retry.delay_ms': delay,
});

Go Instrumentation

// otel.go
package otel

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func InitTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("my-go-service"),
        semconv.ServiceVersion("1.0.0"),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1),
        )),
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

// Instrument HTTP handlers
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

http.Handle("/api/orders", otelhttp.NewHandler(orderHandler, "orders.list"))

Sampling Strategies

Tail-Based Sampling in OTel Collector

Tail sampling makes decisions after the full trace is received:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      # Always keep error traces
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      # Always keep slow traces (>1s)
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 1000 }

      # Keep 5% of successful fast traces
      - name: probabilistic-ok
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

      # Always keep important users
      - name: important-users
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [enterprise, vip]

OpenTelemetry Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 5s
    limit_mib: 512

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector:4317
    tls:
      insecure: true
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger, otlp/tempo]

Jaeger Deployment

# jaeger-all-in-one for development
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  selector:
    matchLabels:
      app: jaeger
  template:
    spec:
      containers:
        - name: jaeger
          image: jaegertracing/all-in-one:1.57
          ports:
            - containerPort: 16686  # UI
            - containerPort: 4317   # OTLP gRPC
            - containerPort: 4318   # OTLP HTTP
          env:
            - name: COLLECTOR_OTLP_ENABLED
              value: "true"
            - name: SPAN_STORAGE_TYPE
              value: elasticsearch
            - name: ES_SERVER_URLS
              value: http://elasticsearch:9200

Trace Correlation with Logs

Link traces to logs for context correlation:

import { trace, context } from '@opentelemetry/api';
import winston from 'winston';

const logger = winston.createLogger({
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.printf(({ level, message, timestamp, ...meta }) => {
      const span = trace.getActiveSpan();
      const traceId = span?.spanContext().traceId;
      const spanId = span?.spanContext().spanId;
      return JSON.stringify({ timestamp, level, message, traceId, spanId, ...meta });
    })
  ),
  transports: [new winston.transports.Console()],
});

Conclusion

OpenTelemetry provides a vendor-neutral, future-proof observability foundation. Auto-instrumentation captures the majority of spans with minimal code changes. Custom spans expose business-level context that infrastructure metrics cannot provide. Tail-based sampling in the Collector controls cost without sacrificing visibility into errors and slow requests. Jaeger or Grafana Tempo provide the UI to navigate these traces and diagnose production issues quickly.