Transactional Outbox Pattern

The Problem

When working with event-based and/or message-oriented middleware systems, we face two problems, that might affect our ability for reliable messaging.

Problem 1 : Sending the message to a broker

The first problem arises when we want to send messages to a message broker. This is the single point of failure when it comes to delivering the message. As soon as the message reached the message broker we are almost guaranteed a delivery of the message broker, even if the message broker or the consumers are temporarily stopped or offline. But if the message broker isn’t available when we want to send the message from a producer, we have a problem.

Problem 2: Ensuring consistency

Consider the following pseudo-code that might run when a user places an order:

public void PlaceOrder(Order o) {
  // Update the services own datastore
  StoreOrderInDatabase(o);

  // Send a message to a broker to inform downstream services
  NotifyOrderPlaced(o);
}

To ensure consistent processing of the order, we need to ask ourselves, what happens when our service goes down at certain points during execution?

• Case 1.1:
  • Service goes down bevor the StoreOrderInDatabase method gets called.
    • No update to the database.
    • No message sent.
    • No problem in terms of consistency, but User might try to order again.
• Case 1.2:
  • Service goes down after StoreOrderInDatabase happened.
    • Database got updated, but no message was sent.
    • Leads to data inconsistency, as processing of the order does not continue correctly.
    • Fix might be difficult, our service thinks the order was already received, but downstream services don’t know about it.

So Case 1.1 is a standard error handling issue, where we need to design our application in a way to allow the user the place the order again when something goes wrong, but Case 1.2 is our main concern in terms of correct order processing.

Could we fix the issue by reordering the actions?

public void PlaceOrder(Order o) {
  // Send a message to a broker to inform downstream services
  NotifyOrderPlaced(o);

  // Update the services own datastore
  StoreOrderInDatabase(o);
}

What can go wrong this time?

• Case 2.1:
  • Service goes down before the NotifyOrderPlaced method gets called.
    • Equivalent to Case 1.1 -> no consistency problem
• Case 2.2:
  • Service goes down after the NotifyOrderPlaced method gets called.
    • Now the downstream services get informed to process the order, but our own service hasn’t updated its state in the database.
    • Problem in terms of data consistency. Who has the correct information?

Again Case 2.2 is our main concern and leads to inconsistencies.

The Solution: Transactional Outbox Pattern

To solve the issue we can do the following. Instead of updating the database AND sending a message, we focus on making one consistent database update using a transaction:

// Place Order pseudo code
public void PlaceOrder(Order o) {
  using (var transaction = new DatabaseTransaction()) {
    // Update the services own datastore
    StoreOrderInDatabase(o);

    // Store a message for downstream services in the database to be sent out later
    StoreOutboxMessageInDatabase(o);

    // Commit transaction
    transaction.Commit();
  }
}

Using a transaction we achieve the following: Either all database operations within the transaction succeed and the database gets updated, or the whole transaction will be rolled back, meaning no changes will be made.

All the messages we need to send to the message broker will be stored in a table in the database first. We need to have an additional, independent piece of code (BackgroundService, background Thread, or similar), that periodically checks this table for unsent messages, and sends them out.

// Outbox processor pseudo code
public async Task ProcessOutbox() {
  while (true) {

    var messages = LoadUnsentMessagesFromDatabase();
    foreach (var message in messages) {
      SendMessageToBroker(message);
      message.IsSent = true;
    }
    SaveChangesToDatabase();

    await Task.Delay(1000);
  }
}

This is a very simplified version of an outbox processor, but using this we ensure at-least-once delivery. We might still face problems when scaling the outbox processor to multiple instances.