Message Queue¶
Why Message Queue¶
- Asynchronously processing
- Decoupling services
- Controlling traffics (adjusting processing speed)
Messing Patterns¶
Message queue¶
Producer produces messages, one message can be consumed by one consumer.
Publish/Subscribe¶
Publisher published a message, every subscriber receives a copy of this message.
Kafka Message Pattern¶
Kafka vs MQTT vs AWS SQS¶
| Kafka | MQTT | AWS SQS | |
|---|---|---|---|
| Product type | Open Source | Open Protocol | managed service |
| Programming language |
Java, Scala | C, C++, Java, Python, Erlang, etc. | Closed, unknown |
| Messing Patterns | Message Queue Streaming |
Publish-Subscribe | Message queue |
| Protocol | TCP-based protocol | TCP/IP-based protocol | HTTP/HTTPS based API Calls |
| Architecture | Distributed with brokers, topics, partitions | Clients, broker | Centralized, Managed by AWS |
| Data Storage | Persistent storage with configurable retention | No built-in storage; messages are transient | Messages are stored in queues until processed |
| Throughput | High throughput for large data streams | Designed for low-bandwidth scenarios | Moderate throughput, scales with queues |
| Message Delivery | At least once (configurable to exactly once or at most once) | At most once (QoS 0), at least once (QoS 1), exactly once (QoS 2) | At least once |
| Latency | Low latency, suitable for high-throughput scenarios | Low latency, designed for real-time communication | Higher latency compared to Kafka and MQTT |
| Scalability | Horizontally scalable by adding brokers | Scalability depends on broker architecture | Automatically scales with demand |
| Reliability | High reliability with replication and fault tolerance | Depends on broker implementation | Highly reliable with built-in fault tolerance |
| Ordering | Guarantees message ordering per partition | No inherent message ordering | No built-in message ordering |
| Message Size | Handles large messages (up to several MB) | Best for small messages (typically up to a few KB) | Supports messages up to 256 KB in size |
| Security | Authentication, Authorization, Encryption | Basic security (varies by broker) | Integrated with AWS IAM for security |
| Integration | Rich ecosystem with various connectors and integrations | Broad support for various IoT devices and platforms | Deep integration with AWS services and applications |
| Cost | Open-source with operational costs for hardware and maintenance | Varies by broker service or self-hosted costs | Pay-as-you-go pricing model |
| Primary Use Case | Event Streaming and Log Aggregation | IoT, Real-Time Data Communication | Asynchronous Task Processing and Decoupling systems |
Choosing the Right Tool
- For high-throughput data pipelines and real-time analytics: Go with Kafka.
- For lightweight, real-time messaging in IoT or low-bandwidth scenarios: Go with MQTT.
- For managed task queues or decoupling services with minimal maintenance: Go with AWS SQS.