Why "just use Kafka" isn't always right
The most common mistake when picking a message queue is choosing the best-known product instead of the guarantees you need. One million messages a day sounds like a lot, but it averages about 11.6 messages per second (1,000,000 ÷ 86,400). At that volume, Kafka, RabbitMQ, NATS JetStream and Redis Streams can all handle the throughput. What decides it is the delivery model, replay, the unit of ordering, and the operational burden.
Start with the delivery model
| Item | Kafka | RabbitMQ | NATS (Core / JetStream) | Redis Streams |
|---|---|---|---|---|
| Core model | Partitioned append-only log | Exchange → queue routing (stream queues also available) | Core: pub/sub; JetStream: persistent streams | Append-only log as a Redis data type |
| After a message is read | Kept for the retention period (consumers track offsets) | Classic and quorum queues delete on ack; streams keep it | Core: not stored; JetStream: kept per retention policy | Kept until trimmed (XTRIM/MAXLEN) |
| Replay | Reset offsets and reprocess | Streams (3.9+) only, not regular queues | Set a JetStream consumer's start position | XRANGE / read from an ID |
| Ordering unit | Within a partition | Per queue (multiple competing consumers can process out of order; Single Active Consumer helps) | Per stream/subject | Per stream |
| Delivery guarantee | At-least-once by default; exactly-once processing via idempotent producer + transactions | At-least-once with publisher confirms + consumer acks | Core: at-most-once; JetStream: at-least-once, exactly-once via message-ID dedup | At-least-once with consumer groups + XACK |
| Fan-out | Each consumer group reads everything independently | fanout/topic exchanges copy to each queue | Every subscriber receives; JetStream per consumer | Each consumer group reads independently |
| High availability | Replication factor (typically 3), KRaft controllers | Quorum queues (Raft) | JetStream cluster replicas (R1/R3/R5) | Redis replication/cluster (asynchronous) |
Some of this changed in recent versions, so don't trust older articles blindly:
- Kafka 4.0 (March 2025) removed ZooKeeper mode; only KRaft is supported. "Kafka means running a ZooKeeper cluster too" is no longer true.
- RabbitMQ 4.0 removed classic queue mirroring. Use quorum queues or streams for replication.
- Since Kafka 3.0, producers default to
enable.idempotence=trueandacks=all(unless conflicting settings are present). - NATS JetStream deduplication uses the
Nats-Msg-Idheader and the stream's duplicate window (2 minutes by default). - In Redis Streams, unacked messages stay in the PEL (pending entries list). Find a dead consumer's messages with
XPENDINGand take them over withXAUTOCLAIM(6.2+).
A 30-second decision table: 6 questions
- Do you need to re-read past messages? (e.g. a new service consuming the last 7 days from the start)
- If you need ordering, at what level? Global, per key (order ID, etc.), or per queue
- Is it a work queue where messages disappear once consumed, or an event log that several teams read independently?
- Are the routing rules complex? (header/pattern routing, priorities, delayed retries, DLQ)
- Must loss be zero? Then you need broker replication, producer confirmations and consumer acks, all three
- Who will run it? Upgrades, disks, rebalances, monitoring
| Situation | First choice | Why | Watch out for |
|---|---|---|---|
| Many consumers read the same events independently; replay and long retention are key | Kafka | Log + consumer groups + offset reset are built in; ecosystem (Connect, Streams, Schema Registry) | Partition count and key design drive ordering and scale. You can add partitions, but the key→partition mapping changes |
| Work queues with retries, DLQ, priorities, complex routing | RabbitMQ | Exchanges, bindings, DLX and TTL are built in; per-message acks | Don't use it as an unbounded backlog store; replicate with quorum queues |
| Lightweight service-to-service messaging, request-reply, edge/IoT, small ops team | NATS (+JetStream) | Single binary, low ops burden, add JetStream persistence only where needed | Core NATS drops messages when no subscriber is listening; create JetStream streams explicitly for persistence |
| You already run Redis and need small-scale async jobs | Redis Streams | No extra infrastructure; consumer groups | Data lives in memory, so retention costs memory. Asynchronous replication can lose recent writes on failover |
Cost: get these numbers right before the price list
Cloud prices change by region and over time, so this article doesn't quote fixed prices. Instead, normalize your inputs so you can quote any product the same way. Example: 1M messages/day × 2 KB average × 7-day retention.
| Input | Calculation | Result |
|---|---|---|
| Average throughput | 1,000,000 ÷ 86,400 | ~11.6 msg/s (measure peak separately) |
| Daily ingest | 1,000,000 × 2 KB | ~2 GB/day |
| Monthly ingest | 2 GB × 30 | ~60 GB/month (use for transfer costs) |
| Retained data | 2 GB × 7 days | ~14 GB |
| With replication factor 3 | 14 GB × 3 | ~42 GB (+ headroom, index/overhead) |
A common mistake is multiplying monthly ingest (60 GB) by the retention days again. With 7-day retention, what sits on disk is 7 days' worth (~14 GB, ~42 GB replicated). At this scale, the minimum broker footprint (usually 3 nodes) and people's time drive the cost far more than storage.
Easy-to-miss line items:
- Cross-AZ transfer: with RF 3 across AZs, producer → leader, leader → followers and consumer reads can all cross AZs. Check your cloud's rates; in Kafka,
client.racklets consumers read from the nearest replica (follower fetching). - Managed-service billing units: broker-hours, partition-hours, stored GB, processed GB: each service differs. Feed the same workload table into each vendor's official calculator.
- Labor for self-hosting: even when instances are cheapest, write down how many hours a month go to upgrades, disk growth and incidents.
Metrics to watch in the first week
| Product | Must-watch metrics | Warning signs |
|---|---|---|
| Kafka | Consumer lag, under-replicated partitions, disk usage | Lag keeps growing; URP > 0 persists |
| RabbitMQ | Queue depth (ready/unacked), memory and disk alarms, connections/channels | Unacked keeps growing (stuck consumer); alarms block publishers |
| NATS JetStream | Consumer pending / ack pending, redeliveries, stream size | Ack pending grows; max deliver reached |
| Redis Streams | Stream length (XLEN), PEL size (XPENDING), memory | PEL never shrinks (dead consumer); near maxmemory |
FAQ
Q. Can RabbitMQ replay messages? A. RabbitMQ Streams (3.9+) retain messages and let you re-read by offset. Classic and quorum queues delete messages on ack, so they aren't meant for replay.
Q. Does Kafka guarantee global ordering? A. Only within a partition. Send the same key (e.g. order ID) to the same partition to get per-key ordering. Global ordering requires a single partition and gives up parallelism.
Q. Is NATS alone loss-free? A. Core NATS is at-most-once; messages are lost if no subscriber is connected. Use JetStream streams with explicit acks when you need durability.
As of 2026-10-15, based on the official Apache Kafka, RabbitMQ, NATS and Redis docs. Check defaults against the docs for the version you run.
Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us
This article has no recorded human check (date and scope) yet.
Comments
Be the first to comment.