Message Queues: Kafka vs RabbitMQ

Comparison Overview

Kafka:
  - Log-based, messages retained (days/forever)
  - High throughput (millions/sec)
  - Consumer groups replay from any offset
  - Great for event streaming, audit logs, analytics
  - Complex ops (Zookeeper/KRaft, partitions, topics)

RabbitMQ:
  - Traditional message queue (messages consumed then removed)
  - Complex routing (exchanges, bindings)
  - Built-in dead letter queues
  - Great for task queues, RPC, complex routing
  - Easier to operate

Kafka Patterns

import { Kafka, Partitioners } from 'kafkajs';

const kafka = new Kafka({ clientId: 'app', brokers: ['kafka:9092'] });

// Producer with guaranteed delivery
const producer = kafka.producer({
  createPartitioner: Partitioners.DefaultPartitioner,
});
await producer.connect();

async function publishEvent(topic: string, event: object, key?: string): Promise<void> {
  await producer.send({
    topic,
    messages: [{
      key: key || null,
      value: JSON.stringify(event),
      headers: {
        'event-type': topic,
        'timestamp': Date.now().toString(),
      },
    }],
  });
}

// Consumer group (multiple consumers = parallel processing)
const consumer = kafka.consumer({ groupId: 'order-processor' });
await consumer.connect();
await consumer.subscribe({ topics: ['orders', 'payments'], fromBeginning: false });

await consumer.run({
  eachBatch: async ({ batch, resolveOffset, heartbeat }) => {
    for (const message of batch.messages) {
      await processMessage(JSON.parse(message.value!.toString()));
      resolveOffset(message.offset);
      await heartbeat(); // Prevent rebalance timeout
    }
  },
});

RabbitMQ Patterns

import amqp from 'amqplib';

const connection = await amqp.connect(process.env.RABBITMQ_URL!);
const channel = await connection.createChannel();

// Work Queue (task distribution)
await channel.assertQueue('tasks', {
  durable: true,
  arguments: {
    'x-dead-letter-exchange': 'dlx',  // Failed messages go here
    'x-message-ttl': 3600000,         // 1 hour TTL
  }
});

// Publish task
channel.sendToQueue('tasks', Buffer.from(JSON.stringify({ type: 'email', userId: '123' })), {
  persistent: true,          // Survive broker restart
  messageId: generateId(),   // Idempotency
});

// Consume with ack
channel.prefetch(10); // Process max 10 unacked messages
channel.consume('tasks', async (msg) => {
  if (!msg) return;
  try {
    await processTask(JSON.parse(msg.content.toString()));
    channel.ack(msg);              // Success: remove from queue
  } catch (err) {
    const retries = (msg.properties.headers['x-retries'] || 0) + 1;
    if (retries < 3) {
      channel.nack(msg, false, false); // Dead letter for retry
    } else {
      channel.ack(msg);               // Give up: move to dead letter
    }
  }
});

// Dead Letter Queue: retry or investigate failed messages
await channel.assertExchange('dlx', 'direct');
await channel.assertQueue('dead-letters', { durable: true });
await channel.bindQueue('dead-letters', 'dlx', '');

Pub/Sub Patterns

// Kafka: naturally pub/sub (multiple consumer groups)
// Each consumer group gets ALL messages independently
kafka.consumer({ groupId: 'email-service' }).subscribe({ topic: 'user-events' });
kafka.consumer({ groupId: 'analytics-service' }).subscribe({ topic: 'user-events' });

// RabbitMQ: fanout exchange for pub/sub
await channel.assertExchange('events', 'fanout');
const q1 = await channel.assertQueue('email-notifications', { durable: true });
const q2 = await channel.assertQueue('analytics-tracking', { durable: true });
await channel.bindQueue(q1.queue, 'events', '');
await channel.bindQueue(q2.queue, 'events', '');

// Publish to all subscribers
channel.publish('events', '', Buffer.from(JSON.stringify(event)));

Choosing Between Kafka and RabbitMQ

Use Kafka when:
  - Event streaming (Kafka is the source of truth)
  - High throughput (>10k messages/sec)
  - Multiple consumers need same messages
  - Audit log or event replay needed
  - Building data pipelines

Use RabbitMQ when:
  - Task queues (workers pull jobs)
  - Complex routing (topic/header routing)
  - RPC (request-reply pattern)
  - Lower throughput but complex logic
  - Simpler operational requirements

For most microservices, RabbitMQ is simpler to operate; Kafka shines for data pipelines.