link: Cloud Architecture
Message brokers
Diagram
Overview
Message Brokers are middleware systems designed to facilitate the exchange of messages between different applications or services. They decouple the sender and receiver, allowing them to communicate asynchronously, which is crucial for building scalable and resilient systems.
NOTE
This allows applications to share information with one another, even if they’re written in different programming languages!
Key Concepts
Components
Summary
- Producer: The application responsible for sending messages. It’s connected with the message broker. In the publish/subscribe pattern, they are called
publishers.- Consumer: The endpoint that consumes messages waiting in the message broker. In the publish/subscribe pattern, they are called
subscribers.- Queue/Topic: A storage mechanism in the message broker used to store messages.
Messaging Systems
Message Brokers are integral to Message-driven Architecture, where the communication between components is achieved through asynchronous messaging. They support various communication patterns and ensure reliable message delivery.
Important
- Message-Driven Architecture: An architectural style where components communicate by sending and receiving messages via a messaging system, allowing for loose coupling and asynchronous interactions.
Topics and Queues
- Queues: In a queue, each message is processed by a single consumer. This pattern is often used in point-to-point messaging, where a message is directed from one sender to one receiver.
- Topics: In a topic, each message can be processed by multiple consumers. This pattern is typically used in publish/subscribe messaging, where messages are broadcast to multiple subscribers.
Messaging Patterns
-
Point-to-Point Messaging (Message Queues): Messages are sent to a specific queue and consumed by a single receiver. This ensures that each message is processed only once.
-
Publish/Subscribe Messaging (Message Topics): Messages are published to a topic and can be consumed by multiple subscribers, allowing for broad distribution of information.
Types of Message Brokers
Summary
Apache Kafka: An Event Streaming platform (is not a message broker) for high-throughput, low-latency data streaming, supporting both publish-subscribe and point-to-point models.
RabbitMQ: An open-source message broker supporting multiple protocols, including AMQP, with various queue types for different needs.
Azure Service Bus: A managed enterprise message broker with queues and publish-subscribe topics, offering advanced features like sessions and dead-lettering.
Google Cloud Pub/Sub: A real-time messaging service for sending and receiving messages between independent applications, supporting topics and subscriptions.
Database Backed Queue: Uses a relational or NoSQL database as a queue. Suitable for applications that already use a database and need a simple queueing mechanism.
Redis: An in-memory data structure store that can be used as a message broker with support for publish-subscribe and list-based queues, offering high performance and low latency.
Amazon MQ: A managed message broker service for Apache ActiveMQ and RabbitMQ that supports both pub/sub and point-to-point messaging patterns.
AWS SQS, AWS SNS: A fully managed service that decouples and scales microservices, distributed systems, and serverless applications.
Apache ActiveMQ: An open-source message broker that supports pub/sub and point-to-point messaging, along with advanced features for reliable messaging.
Quality of Service
Quality of Service
Diagram
Overview
Delivery semantics define how messages are delivered and processed in a messaging system. They are crucial in ensuring that messages are handled correctly according to the application’s needs. There are three primary delivery semantics:
At Most Once
Info
- Definition: A message is delivered not more than once. Messages may be lost, but they are not redelivered.
- Use Cases: Suitable for scenarios where a small amount of data loss is acceptable, such as monitoring metrics or non-critical log data.
- Example: Sending periodic updates where missing one update is not critical, like temperature readings from a sensor.
At Least Once
Info
- Definition: Each message is delivered at least once, but it may be delivered multiple times. No message should be lost.
- Use Cases: Ideal for use cases where message loss is unacceptable, but duplicates can be tolerated or handled, such as logging systems or metrics collection.
- Example: Sending notifications where ensuring the message is received is crucial, even if it means the recipient might get the same notification more than once.
Exactly Once
Link to originalInfo
- Definition: Each message is delivered exactly once, ensuring no duplicates and no message loss. This is the most stringent delivery semantic.
- Use Cases: Critical for financial transactions, payment processing, trading, or any system where duplication or loss of messages cannot be tolerated.
- Example: Processing orders in an e-commerce system where duplicating an order or losing an order can have significant financial implications.
Message Topic vs Event Streaming
Message Topic vs Event Streaming
Overview
Both message brokers and event streaming platforms facilitate communication between different parts of a system, but they serve distinct purposes and are optimized for different types of messaging patterns.
Key Differences
Aspect Message Brokers Event Streaming Platforms Message Handling Manages individual messages that need reliable delivery Manages continuous streams of data Consumer Model Consumers typically receive each message only once Consumers can read from any point in the stream multiple times Examples RabbitMQ, Azure Service Bus, Amazon MQ Apache Kafka, Apache Pulsar, Amazon Kinesis Processing Model Point-to-point or publish/subscribe Primarily publish/subscribe Scalability Scales by adding more consumers Scales horizontally by adding more partitions Use Cases Task distribution, command handling, short-lived messaging Event sourcing, real-time analytics, data pipeline Overview
- Task Distribution: Ideal for scenarios where tasks need to be distributed among multiple workers.
- Short-lived Messaging: Suitable for messaging patterns where messages are processed quickly and do not need to be retained long-term.
- Reliable Delivery: Ensures each message is delivered reliably to one or more consumers.
Event Streaming Platforms:
Link to original
- Event Sourcing: Captures and stores all changes to the state as a series of events, which can be replayed to reconstruct past states.
- Real-time Analytics: Processes streams of data in real-time for analytics and monitoring.
- Data Pipeline: Facilitates the flow of data across various stages of processing, ensuring that data can be consumed by multiple downstream systems independently.
Summary
Message Brokers are essential components in modern distributed systems, enabling asynchronous communication and enhancing the scalability, flexibility, and reliability of applications. By understanding the key concepts, patterns, and types of message brokers, developers can effectively leverage these tools to build robust and responsive systems.
References
Message broker – complete know-how, use cases and a step-by-step guide | TSH.io