Message Queues & Pub/Sub

Concept#

So far, everything in this domain has assumed a request gets handled synchronously, a client asks, a server answers, right now. But a large class of real work doesn't need to happen "right now" from the requester's point of view: sending a confirmation email, resizing an uploaded image, updating a search index, charging a card in the background after an order is placed. Message queues exist to decouple asking for work to happen from actually doing the work, in time.

Why decoupling in time matters#

Imagine an API endpoint that, on every signup, synchronously sends a welcome email before responding to the client. If the email provider is slow or briefly down, every signup request hangs or fails, even though the actual signup (writing a row to the database) succeeded fine. The email-sending concern has been coupled to the request path for no good reason: the user doesn't need to wait for the email to be sent to get their "signup successful" response.

A message queue fixes this: the signup handler writes a message ("send welcome email to user 123") onto a queue and immediately responds to the client. A separate consumer process reads messages off that queue at its own pace and sends the actual email. If the email provider is slow, messages simply pile up in the queue rather than blocking user-facing requests, the queue acts as a buffer absorbing the mismatch in speed between producer and consumer.

Synchronous (tightly coupled):
  Client → API → [write DB] → [send email] → response
                                   ^
                        slow/down email provider blocks EVERYTHING
 
Asynchronous (queue-decoupled):
  Client → API → [write DB] → [enqueue "send email" message] → response
                                                                    (fast!)
                     Consumer (separate process) ← dequeues → sends email
                     (can be slow, can retry, can be temporarily down, none of that affects the user-facing response)

Point-to-point queues vs. pub/sub fan-out#

There are two related but distinct patterns, and mixing them up is a common source of confusion:

Point-to-point queue (e.g. SQS, RabbitMQ queue):
  One message → consumed by exactly ONE consumer (from a pool).
  Use case: distributing WORK across a pool of workers, you want
  each job (resize this image, process this payment) done ONCE.
 
Publish/Subscribe (e.g. SNS, Kafka topics, Redis Pub/Sub):
  One message → delivered to EVERY subscriber independently.
  Use case: broadcasting an EVENT that multiple, independent parts
  of the system each need to react to their own way, e.g. "order
  placed" might need to trigger inventory update, analytics tracking,
  AND a confirmation email, as three separate subscribers, each
  getting their own copy of the same event.

A common real architecture combines both: an event is published to a pub/sub topic, and each interested service has its OWN point-to-point queue subscribed to that topic, giving you fan-out to multiple services, while each individual service still gets exactly-once-per-message processing within its own worker pool.

Delivery guarantees: what a queue actually promises#

Queues differ meaningfully in what they guarantee about how many times a message gets delivered:

At-most-once:   message delivered 0 or 1 times. If a consumer crashes
                mid-processing, the message is simply LOST, never
                retried. Rare in practice; usually a sign of a
                misconfigured or poorly-chosen queue for the use case.
 
At-least-once:  message delivered 1 OR MORE times. If a consumer
                crashes or fails to acknowledge in time, the message
                is redelivered, but this means a consumer MUST
                tolerate processing the same message more than once.
                This is the most common real-world guarantee (SQS,
                RabbitMQ with manual ack, Kafka's default consumer
                behavior).
 
Exactly-once:   message delivered and processed exactly 1 time, no
                more, no less. Genuinely hard to guarantee end-to-end
                in a distributed system, and often either unavailable,
                expensive, or only true within narrow constraints
                (e.g. Kafka's exactly-once semantics apply within its
                own transactional pipeline, not automatically to
                arbitrary external side effects a consumer performs).

Because at-least-once is the practical default nearly everywhere, the real burden shifts to the consumer: it must be written to handle receiving and processing the same message twice without causing a duplicate real-world effect, this is the idempotency problem, covered below.

Idempotency: making "maybe delivered twice" safe#

A consumer operation is idempotent if running it twice has the same effect as running it once. This is the practical answer to at-least-once delivery: rather than fighting to guarantee exactly-once delivery (hard, often impossible end-to-end), design consumers so duplicate delivery simply doesn't matter.

// ❌ NOT idempotent, running this twice charges the customer twice
async function processPayment(message) {
  await chargeCard(message.customerId, message.amount);
}