Why do we need message queues?
A message queue is a software system that allows applications to communicate by sending and receiving messages. These messages are stored in a queue, which acts as a safe buffer for data that needs to be processed.
The messages are stored in a first-in, first-out (FIFO) order, meaning the first message added to the queue is the first one processed. This allows for asynchronous communication between applications, where one application can send a message and another can receive it at a later time.
Message queues are foundational for a few critical reasons:
- Decoupling: They separate different components of an application so they don’t have to be directly aware of each other. This makes the system far more flexible and easier to maintain.
- Scalability and Fault Tolerance: Instead of requiring all applications to be online and holding a connection at the same time, messages can be stored in the queue and processed when the receiving system is ready.
The Core Concept: Synchronous vs. Asynchronous
To truly understand the need for message queues, you must understand the difference between synchronous and asynchronous processes.
Synchronous messaging
Synchronous messaging means “at the same time.” We engage with this constantly in our daily lives.
When we ask someone a question face-to-face, we wait for the answer. If the answer takes a while, we just have to stand there and wait. The same takes place within many systems you use daily. When your browser requests a web page, you have to wait until that specific web page is returned before you can interact with it.
Asynchronous messaging
A prime example of asynchronous messaging is email. When sending an email to multiple recipients, you are placing it in their “queue” for them to access and read at their convenience. You don’t stand there waiting for them to open it.
As the sender, you have no way of knowing when all recipients have read the message unless you receive a failure notification (like a bounced email). However, even if one email address is incorrect, you can still safely assume the rest of the recipients received the message.
Asynchronous messaging is also prevalent in instant messaging systems like Slack. You can create a channel to “publish” an “event” (a message), which others can “subscribe” to and receive notifications for. You can even build integrations that allow other systems to “consume” specific messages and trigger further actions.
The critical takeaway: The event is merely published to a queue. The sender is not concerned about what the consumers may do with that message.
Event-driven architecture
Event-driven architecture is a way of designing systems where different components communicate by sending and responding to events. An event is simply a message that describes something that has occurred, such as a button being clicked, a payment clearing, or a file being uploaded.
Because the components respond to these events in a loosely coupled way, they don’t need to know the technical details of how the other components work. In simple terms: Event-Driven Architecture relies on systems sending and responding to events, rather than constantly requesting and waiting for data.
The structure of event-driven message queues
To make this concrete, let’s look at one of the most widely used event-driven tools: Kafka.
Here are the main components of that system:
- Topics: These are the channels that messages are sent to and received from. Think of them as categories. For example, a topic could be
Order_Updatesfor an e-commerce system,Temperature_Readingsfor an IoT application, orSystem_Logs. - Producers: The applications or systems that send messages to topics. In the e-commerce example, the website checkout page would be a producer that sends order details to the
Order_Updatestopic. - Brokers: The servers that handle the storage and routing of messages. When a producer sends a message to a topic, it is safely stored on one or more brokers, making it available to consumers.
- Consumers: The downstream applications or systems that read the messages from the topics (e.g., the warehouse system reading
Order_Updatesto start packing a box).
To sum up…
Message queues and event-driven systems are powerful tools for building distributed systems that need to communicate reliably and efficiently.
Kafka-style systems use a publish-subscribe model, where messages are sent to topics and consumed by subscribers. The broker acts as the central manager, containing the topics, storing the messages, and routing them to the appropriate consumers. By mastering this structure, you move away from brittle, point-to-point integrations and towards highly scalable, fault-tolerant architecture.