Skip to content
Deep Thought

Reliable Messaging

Reliable Messaging

What?

  • When sending messages between services we want to make sure that the message will be processed by the receiving service eventually.
  • We are looking for a mechanism that ensures (as much as possible) that every message will be delivered to it’s recipient.

What can go wrong

  • Putting a message into a queue
  • Message being lost while in the queue
    • e.g. when Service goes down
  • Reading message from the queue
    • Could be read, but not deleted from queue
    • Could be deleted but not being processed correctly by recipient.

Strategies

  • Putting a message into a queue
    • Wait for success response from messaging service.

What if there is a timeout and you are not sure if a message has been successfully sent or not?

—> Try to use idempotent insert

Strategies (2)

  • Message being lost while in the queue
    • Message has to be stored persistently (disk/storage)
    • Message Service outage must not affect messages being stored

Strategies (3)

  • Reading message from the queue
    • Receive
    • Process
    • Acknowledge (delete)

Only when acknowledging the received message after it has been processed we make sure it was processed correctly.

  • RabbitMQ
  • Apache Kafka
  • ActiveMQ
  • Azure ServiceBus

They are also called Message Brokers or message-oriented middleware.

Protocols

  • AMQP
  • MQTT

Are implemented by the message brokers.

Mechanisms

We use publish/subscribe to send messages to and receive messages from:

  • Topics
  • Queues

Topic/Queue

All subscribers to a topic receive the same message when a message is published, but only one subscriber to a queue receives the message when the message is sent.

Dead-Letter Queue

  • Is an additional queue created by the message broker.
  • Messages that cannot be delivered due to errors will end up in the dead-letter queue.
  • Usually the message broker tries to redeliver the same message a few times before moving it to the DLQ.
  • DLQ can be used to check for errors. Should be investigated by the developers regularly/on insert.

Why not just use HTTP-Calls to other services?

  • HTTP-Calls couple to services together
    • Sender sends request
    • Recipient receives request
    • Recipient processes request
    • Recipient sends response
    • Sender receives response

Advantages of message brokers

  • Services are more loosely coupled
  • Sender and receiver don’t have to be available at the same time.
  • Messages in the queue act as buffer and can add up before being processed eventually
  • Can be used to distribute tasks (producer/consumer pattern)

Advanced features

  • Structuring
    • Message brokers offer features such as namespaces to logically organize resources.
  • Message sessions
    • Guarantee delivery by using request-response access
  • Routing
    • Message brokers can dynamically re-distribute or re-route to other queues/topics based on rules.
    • Can be used to control execution during runtime
  • Scheduled delivery / message deferral
  • Filters
    • Subscribers can control which messages they are interested in within a certain queue/topic.
  • Duplicate detection
  • Secure access (roles/permissions)
  • Geo-disaster recovery

Azure service bus tier comparison

AzureServiceBusTiers.png