RabbitMQ: Best Practices

RabbitMQ is an open-source message broker that originally implemented AMQP(Advanced Message Queueing Protocol). Message brokers adhere to protocols for communicating between software systems using well-defined messages.

WHY use RabbitMQ?

Message queueing enables software applications to connect and scale. Applications can connect and communicate with each other. Messaging is asynchronous, decouple applications by separating sending and receiving data. Message queueing is the embodiment of various software practices to publish/subscribe, asynchronous processing or work queues. It also acts as a temporary message storage until received. RabbitMQ is a message-broker and comes with its pros and cons.

• Reliability: RabbitMQ can be configured to trade-off performance for reliability (Persistence, Acknowledgements, Publisher Confirms and HA). This point can be taken in for both pro and con as there is a trade-off between throughput and reliability.
• Flexible Routing: Any message before reaching the queue goes through exchange. Exchanges are responsible for handling the routing of messages and comes in several types (Fanout, Direct, Topic, and Custom Implementation)
• Clustering: RabbitMQ servers on the local network can be clustered together, forming a single logical broker. And many more . . .

RabbiMQ comes with its advantages and disadvantages. RabbitMQ provides multiple fine-tuning options for best results. While working on it recently, I came across a few which I want to share here and are as follows —

Connections

RabbitMQ connections require at least 7 TCP packets for handshake. Channels can be opened and closed more frequently if needed. Best practice is to reuse connections and multiplex a connection between threads with channels.

• AMQP connections: 7 TCP packages
• AMQP channel: 2 TCP packages
• AMQP publish: 1 TCP package (more for larger messages)
• AMQP close channel: 2 TCP packages
• AMQP close connection: 2 TCP packages
• Total 14–19 packages (+ Acks)

Heartbeat Timeout

Heartbeats defend against certain network equipment which may terminate ‘idle’ TCP connection when there is no activity on them for a certain period of time. The heartbeat_timeout value defines after what period of time the peer TCP connection should be considered unreachable(down) by RabbitMQ and client libraries. Default value for heartbeat_timeout is 60 seconds. The value is negotiated between the client and RabbitMQ server at the time of connection. When both values are non-zero, the lower of the requested values is used.
Clients send heartbeat frame every (heartbeat_timeout / 2) seconds. If two consecutive frames are missed, the peer is considered to be unreachable and the TCP connection is closed. Clients will have to re-connect afterward. A good practice is to keep heartbeat_timeout between 5 to 20 seconds. Setting low value can lead to false positives due to transient network congestion, short-lived server flow control, etc.

Blocked Connection Timeout

Blocked connection notifications are sent when the broker is running low on resources(memory or disk). A connection.blocked notification is sent to publishing connections the first time RabbitMQ is low on resources.
blocked_connection_timeout value defines after what period of time the peer TCP connection is interrupted and dropped.
A blocked connection may last for an indefinite period of time, stalling the connection and possibly resulting in hang until the connection is unblocked. blocked_connection_timeout is intended to interrupt a connection that has been blocked longer than the given timeout value. Best practice is to keep the value between 150 to 300 seconds.

Message Persistence

The persistence layer has two components: Queue Index and Message Store. The queue index is responsible for maintaining knowledge about where a given message is in a queue, along with whether it has been delivered and acknowledged. Therefore there is one queue index per queue. The message store is a key-value store for messages, shared among all queues in the server. Messages can either be stored directly in the queue index, or written to the message store.
Under memory pressure, the persistence layer tries to write as much out to disk as possible, and remove as much from memory. Following must remain memory:

• Each queue maintains metadata for each unacknowledged message. The message itself can be removed from memory if its destination is the message store.
• The message store needs an index. The default message store index uses a small amount of memory for every message in the store.

If messages are written to queue indices:

• Messages can be written to disk in one operation rather than two; for small messages, this can be substantial gains. The queue index keeps blocks of a fixed number of records in memory; for not-so-small messages, memory consumption can be substantial. Generally, messages with size ~4096 bytes(inclusive of headers) are optimum to store in queue indices. RabbitMQ configuration variable to define the size of messages to be written in queue indices is queue_index_embed_msgs_below.
• Messages stored in queue indices do not require entry in the message store, thus do not have memory cost on that front. Although, unacknowledged messages with destination queue index, are always kept in memory.
• If a message is routed to multiple queues through an exchange, the message will be written to each queue indices. If such a message is written to the message store, only one copy needs to be written.

If there are any more ways to fine-tune RabbitMQ with better performance and reliability, please reach out. Also please refer to this very helpful blog on RabbitMQ Common Mistakes.

Any suggestions or thoughts, let me know:
Insta + Twitter + LinkedIn + Medium | Shivam Aggarwal