Event-Driven Microservices: Decoupling Services with RabbitMQ

Why Events Instead of Direct Calls

Most backend systems start the same way. Service A needs something from Service B, so A calls B over HTTP and waits for a response. For a lot of things, that's the right call. But this backend is split into a handful of services: an API layer up front, an orders service, a product catalog, a notification service, and a few more behind them. A good chunk of what happens doesn't need an answer right away. A customer places an order. They need a fast, reliable response confirming the order was placed. They do not need to wait for a confirmation email to actually leave the building first.

So from the start, anything that falls into that second category, work that's triggered by something happening but doesn't block the response, goes through RabbitMQ instead of a direct call. The order service publishes an event and moves on. Whoever cares about that event picks it up on their own schedule.

The difference matters more than it looks. In the direct-call version, the order service's response time is the sum of its own work, plus however long the notification service takes, plus however long Postmark takes to accept the email. If Postmark is slow, checkout is slow. If the notification service happens to be mid-deploy, checkout can fail outright. None of that has anything to do with whether the order itself was valid or saved correctly. It's an unrelated system leaking its problems into the one place a customer is actually looking at.

The Shape of It

RabbitMQ sits between publishers and consumers, and it never sees a service name. It sees an exchange, a routing key, and whatever queues happen to be bound to that exchange. Producers publish to exchanges. Consumers read from queues. The exchange is what decides which queues get a copy of a given message, based on those bindings.

This system uses a single topic exchange for domain events, platform.events. A topic exchange matches routing keys against patterns: * stands in for exactly one word, # for any number of words. Every event gets a routing key that describes what happened: order.created, order.payment.confirmed, product.stock.updated. The notification service doesn't care about the exchange or which other consumers might be on it. It declares a queue and binds it to the pattern it wants, order.*, and RabbitMQ takes care of the rest.

This is the part that actually delivers the decoupling. The order service has no idea the notification service exists. It publishes order.created and it's done. If some other consumer needs that same event later, it binds its own queue to the same exchange. Nothing about the order service changes.

Publishing an Event

Publishing to an exchange and forgetting about it is fine, until the broker restarts at the wrong moment and a message never actually gets written anywhere. Two things make publishing safe: durable messages, and publisher confirms.

A durable, persistent message survives a broker restart because RabbitMQ writes it to disk instead of keeping it only in memory. A publisher confirm is RabbitMQ telling the publisher, explicitly, that the message made it that far. Without confirms, publish() succeeding only means the message left the application. It doesn't mean RabbitMQ has it.

// events/publisher.js
const EXCHANGE = 'platform.events';

// channel here is a confirm channel:
// connection.createConfirmChannel(), not the plain createChannel()
async function publishEvent(channel, routingKey, payload) {
  const eventId = crypto.randomUUID();

  return new Promise((resolve, reject) => {
    channel.publish(
      EXCHANGE,
      routingKey,
      Buffer.from(JSON.stringify(payload)),
      {
        persistent: true,          // survive a broker restart
        contentType: 'application/json',
        messageId: eventId,        // doubles as the idempotency key downstream
        timestamp: Date.now(),
      },
      (err) => (err ? reject(err) : resolve(eventId))
      // the callback only fires once the broker confirms the write,
      // not when the socket write happens
    );
  });
}

// usage in the order service, right after the order is saved
await publishEvent(channel, 'order.created', {
  orderId: order._id,
  total: order.total,
});

Consuming an Event

On the consuming side, two settings decide how safe and how fair the whole thing is: acknowledgment mode and prefetch.

Auto-ack tells RabbitMQ to consider a message delivered the moment it hands it to a consumer, before the consumer has done anything with it. If the consumer crashes half a second later, the message is just gone. Every consumer here acks manually, after the work is actually done, not before.

Prefetch caps how many unacknowledged messages a single consumer can hold at once. Without it, one instance of the notification service can end up holding thousands of messages it hasn't processed yet, while other instances sit idle waiting for work. A prefetch of 10 keeps work distributed evenly across however many instances happen to be running, and stops a slow instance from hoarding messages it can't keep up with.

// events/consumer.js
async function startConsumer(channel) {
  channel.prefetch(10);

  channel.consume('notification.orders', async (msg) => {
    if (!msg) return;

    try {
      await handleOrderEvent(JSON.parse(msg.content.toString()));
      channel.ack(msg);
    } catch (err) {
      await handleFailure(channel, msg);
    }
  });
}

When Things Fail

Sending an email fails sometimes. Postmark has a bad minute, a network call times out, whatever. The question is what happens to the event when that happens.

Retrying it immediately in a tight loop is a bad idea. It hammers whatever just failed, right when it's least likely to help. So a failed message gets a short break before it's tried again. This is handled with a second queue that exists purely to hold a message for a fixed amount of time. It has no consumer. Its only job is to sit there until a TTL (time to live) expires, at which point RabbitMQ dead letters it, meaning it automatically republishes the message back to the real exchange, using the same routing key it originally had. From the notification queue's point of view, the message just shows up again a minute later, ready for another attempt.

Retries aren't unlimited. Every time a message fails, a counter in its headers goes up. After five attempts, it stops retrying and goes to a dead letter queue instead, a plain queue nothing reads from automatically. That's a deliberate dead end. A message that has failed five times in a row is telling you something is actually broken, not just slow, and the right move is to alert someone, not keep hammering a broken downstream service forever.

// events/topology.js
const MAIN_QUEUE = 'notification.orders';
const RETRY_QUEUE = 'notification.orders.retry';
const DLQ = 'notification.orders.dlq';
const MAX_RETRIES = 5;

async function setupTopology(channel) {
  await channel.assertExchange('platform.events', 'topic', { durable: true });
  await channel.assertExchange('platform.events.retry', 'topic', { durable: true });

  await channel.assertQueue(MAIN_QUEUE, { durable: true });
  await channel.bindQueue(MAIN_QUEUE, 'platform.events', 'order.*');

  // holding queue: nothing consumes this directly. a message sits here
  // until its TTL runs out, then RabbitMQ dead-letters it back to the
  // real exchange, using the routing key it arrived with
  await channel.assertQueue(RETRY_QUEUE, {
    durable: true,
    arguments: {
      'x-message-ttl': 60000, // 1 minute
      'x-dead-letter-exchange': 'platform.events',
    },
  });
  await channel.bindQueue(RETRY_QUEUE, 'platform.events.retry', '#');

  await channel.assertQueue(DLQ, { durable: true });
}

async function handleFailure(channel, msg) {
  const attempt = (msg.properties.headers['x-retry-count'] || 0) + 1;
  const headers = { ...msg.properties.headers, 'x-retry-count': attempt };

  if (attempt > MAX_RETRIES) {
    channel.sendToQueue(DLQ, msg.content, { persistent: true, headers });
  } else {
    // publish with the message's ORIGINAL routing key, not the queue
    // name, so it re-binds correctly once it dead-letters back
    channel.publish('platform.events.retry', msg.fields.routingKey, msg.content, {
      persistent: true,
      headers,
    });
  }

  channel.ack(msg); // it's been re-homed, safe to drop from the main queue
}

Handling the Same Event Twice

RabbitMQ's delivery guarantee is at least once, not exactly once. That's not a limitation, it's a deliberate trade-off most message brokers make, because exactly-once delivery across a network is a genuinely hard problem. In practice it means a message can be delivered twice: a consumer finishes the work but crashes right before sending the ack, so RabbitMQ, never having heard back, redelivers it.

For an event that sends a confirmation email, an unprotected duplicate delivery means a customer gets the same email twice. So every event carries a unique id, and the notification service checks Redis (already in the stack for caching) before doing anything. If that id has been seen before, it acks the message and skips the work. If not, it does the work and records the id.

// handlers/orderEvents.js
async function handleOrderEvent(event) {
  const dedupeKey = `event:${event.id}`;
  const alreadyHandled = await redis.get(dedupeKey);

  if (alreadyHandled) return;

  await sendOrderConfirmationEmail(event);

  await redis.set(dedupeKey, '1', 'EX', 60 * 60 * 24); // 24h dedupe window
}

A Full Walkthrough: Placing an Order

Here's the whole thing end to end, for a single order.

1. A customer checks out. The request goes through the API layer to the order service.
2. The order service validates the order, writes it to the database, and publishes order.created to platform.events with a fresh event id.
3. The order service responds to the customer immediately. As far as checkout is concerned, the order is placed. Nothing past this point can slow that response down.
4. The notification service, consuming notification.orders in the background, receives the event, checks Redis for that event id, doesn't find it, and sends the confirmation email through Postmark.
5. If the send throws, whether Postmark timed out or something else went wrong, the notification service publishes the event to the retry exchange with the retry count bumped, and acks the original message.
6. A minute later the message reappears on notification.orders and gets tried again. If it keeps failing past five attempts, it lands in the dead letter queue and stops retrying on its own.

At no point does the order service know or care whether any of steps four through six succeeded. That's not a gap in the design. That's the point of building it this way.

Where This Stands Today

RabbitMQ moves more than a million events a day across this system's services. Orders, payment confirmations, stock updates, all of it flows through the same exchange. None of it depends on the notification service being fast, or even being up. If it's mid-deploy, or Postmark is having a rough night, events just wait in the queue. Nothing gets lost, and checkout never feels it.

What This Trades Away

None of this is free, and it's worth being honest about the cost.

Quick Reference

Closing Thoughts

None of this is complicated once it's in place. An exchange, a couple of queues, a TTL, a header for counting attempts. What it buys is bigger than the mechanism: services that don't know about each other, don't wait on each other, and don't take each other down when one of them has a bad day.

The pattern scales the same way regardless of what the event actually is. Order confirmations, stock updates, whatever comes next. Bind a queue to the pattern you care about, and the rest of the system never has to know you're there.

This runs on the same backend as the piece on taming database connections at scale, sharing its lazy per-tenant connection manager without either system getting in the other's way. And the one gap left in the publisher code above, what happens if the process crashes between saving something and publishing its event, gets closed in the follow-up on the transactional outbox pattern.

Fewer things waiting on each other means fewer things that can go wrong at once.