Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions content/topics/event-streams-and-logs/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
id: '50202977-c30d-4d2e-a1d5-893102a0dcce'
title: 'Event streams and logs'
status: 'draft'
summary: 'Choose retained event history and understand replay, partitions, and consumer positions.'
iconKey: 'queue'
domains:
- 'system-design'
kind: 'concept'
childTopicIds: []
relatedTopicIds:
- '7b77e594-3472-4324-98b9-6705a606f976'
- 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
---

**Choose a retained log for history, replay, or ordered event processing.** Multiple subscribers are optional; one consumer can still need replay.

## Kafka-style log model

- **Topic:** Named event collection, divided into partitions.
- **Partition:** Ordered append-only sequence; the ordering boundary.
- **Offset:** Position within one partition.
- **Consumer group:** Independent processing view; group members share partitions.
- **Retention:** Policy controlling how long/how much history remains.

Reading or committing an offset **does not delete the event**. Another group can read at a different position; a group can replay history that is still retained.

## Boundaries and costs

- **Ordering is per partition**, not across an entire multi-partition topic. Parallel application processing must preserve any required order too.
- **Retention limits replay.** Cleanup or compaction may remove records; a log is not automatically a permanent archive of every event.
- **Offsets track progress**, not an atomic commit of arbitrary external effects. A crash can repeat processing.
- **Partitioning affects capacity and ordering.** Hot keys can bottleneck one partition; additional workers do not fix that automatically.

Other products use different terminology and consumption models; check their guarantees.

## Sources

- [Kafka introduction](https://kafka.apache.org/intro/)
- [Kafka design](https://kafka.apache.org/design/)
42 changes: 42 additions & 0 deletions content/topics/message-delivery-and-processing/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
id: '7b77e594-3472-4324-98b9-6705a606f976'
title: 'Message delivery and processing'
status: 'draft'
summary: 'Recognize delivery guarantees and make retrying consumers safe.'
iconKey: 'queue'
domains:
- 'system-design'
kind: 'decision-aid'
childTopicIds: []
relatedTopicIds: []
---

## Delivery versus business effects

- **At-most-once:** A delivery can be lost; duplicates are avoided within the stated scope.
- **At-least-once:** Redelivery is possible; make effects duplicate-safe.
- **Exactly-once:** Ask **within which boundary?** Broker deduplication or log transactions do not automatically cover external APIs or databases.

A broker accepting a message does not prove business completion. Retention expiry, repeated failures, or dead-lettering can leave a job unfinished.

## Acknowledge after a durable result

1. Receive the message.
2. Apply a duplicate-safe effect and durably record success.
3. Acknowledge the message.

- **Acknowledge too early:** a later crash can lose the job's effect.
- **Crash after the effect but before acknowledgement:** expect processing again.
- **Visibility timeout / acknowledgement deadline:** temporary delivery ownership; extend it for long work where supported. Expiry can allow redelivery.
- **Transient failure:** retry with backoff and an attempt/time budget.
- **Persistent failure:** isolate in a dead-letter queue where supported; alert, investigate, and deliberately retry or resolve it.

For thumbnails, a deterministic output key based on image version and thumbnail variant can make repeating generation safe. Ensure an older job cannot overwrite a newer image's output.

A database deduplication marker must commit with its local effect; marking “seen” before doing work can lose the work after a crash. External effects need their own idempotency or recovery contract.

## Sources

- [RabbitMQ acknowledgements and confirms](https://www.rabbitmq.com/docs/confirms)
- [SQS visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html)
- [Kafka delivery semantics](https://kafka.apache.org/design/#semantics)
39 changes: 39 additions & 0 deletions content/topics/message-queues/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
id: 'dd0ab4c6-2430-46a1-a6b8-5bb116c3e2e7'
title: 'Message queues'
status: 'draft'
summary: 'Distribute background jobs among workers and recover from interrupted processing.'
iconKey: 'queue'
domains:
- 'system-design'
kind: 'concept'
childTopicIds: []
relatedTopicIds:
- '7b77e594-3472-4324-98b9-6705a606f976'
- 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
---

**Choose a work queue to distribute jobs among workers.** For thumbnails, each job needs one successful result, rather than an independent result from every worker.

## Job lifecycle

1. Receive a job.
2. Generate the thumbnail.
3. Persist the result.
4. Acknowledge/delete the message.

- **Workers compete:** adding workers shares the backlog; it does not broadcast each job to every worker.
- **Worker fails before acknowledgement:** a recoverable delivery can be retried. The result may already exist, so repeats must be safe.
- **Queue accepts a message:** publication succeeded; thumbnail generation has not necessarily finished.
- **Backlog grows:** buffer bursts, limit worker concurrency, and monitor oldest-job age. A queue does not create processing capacity.

## Durability is a configuration choice

Use persistent storage and appropriate replication, publication confirmation, and retention settings. An in-memory queue inside the API process cannot preserve pending work across process loss.

For large images, queue an **image ID or object-storage reference**, not the image bytes. Workers need access to the referenced object.

## Sources

- [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability)
- [SQS Standard queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html)
104 changes: 104 additions & 0 deletions content/topics/messaging-implementations-and-hosting/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
id: 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
title: 'Messaging implementations and hosting'
status: 'draft'
summary: 'Name an engine or cloud service that fits the messaging model and operational constraints.'
iconKey: 'queue'
domains:
- 'system-design'
kind: 'decision-aid'
childTopicIds: []
relatedTopicIds: []
---

## Narrow the candidates

- **Background jobs:** RabbitMQ; AWS SQS; GCP Pub/Sub with workers sharing a subscription
- **Independent event subscribers:** RabbitMQ exchanges + queues; AWS SNS + SQS; GCP Pub/Sub
- **Retained log and replay:** Apache Kafka; AWS MSK; GCP Managed Service for Apache Kafka
- **Existing JMS broker integration:** ActiveMQ Classic; Amazon MQ for ActiveMQ Classic

These are starting points, not exclusive capabilities. Choose the model first, then check ordering, replay, failure handling, and existing platform constraints.

## Engines: self-managed or hosted

### RabbitMQ

- **Fit:** job queues and routed pub/sub through exchanges and queues.
- **Tradeoff:** topology, durability, acknowledgements, and capacity still need design. RabbitMQ also supports streams; it is not limited to queues.
- **Hosting:** run it yourself or use a managed provider, including Amazon MQ.

### ActiveMQ Classic

- **Fit:** existing JMS applications and broker/protocol compatibility.
- **Tradeoff:** select and operate the broker topology and verify client/version compatibility.
- **Hosting:** self-managed or Amazon MQ. Specify **Classic**; do not assume Amazon MQ hosts Artemis.

### Apache Kafka

- **Fit:** retained event history, replay, and the Kafka processing ecosystem.
- **Tradeoff:** partition keys, retention, consumer lag, and ordering remain design responsibilities even with managed hosting.
- **Hosting:** self-managed, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or another provider.

## AWS services

### SQS Standard

Independent jobs; no strict ordering needed.

**Tradeoff:** At-least-once delivery; duplicates and out-of-order delivery are possible.

### SQS FIFO

Ordering within message groups and send deduplication matter.

**Tradeoff:** Group design constrains parallelism; retries can still repeat business effects.

### SNS + SQS

Fan-out with a durable queue for each consumer application.

**Tradeoff:** More queues, subscriptions, and failure policies to manage.

### Amazon MQ

Managed RabbitMQ or ActiveMQ Classic; engine/protocol compatibility matters.

**Tradeoff:** Broker capacity and engine-specific configuration still matter.

### MSK

Managed Kafka and its APIs/ecosystem.

**Tradeoff:** Kafka data-model and consumer responsibilities remain.

**Standard and FIFO are SQS queue types, not pricing tiers.** FIFO deduplication is bounded; a received message that is not deleted can become visible again.

## GCP services

### Pub/Sub

Managed fan-out; shared subscriptions can distribute worker jobs.

**Tradeoff:** Service-specific API and subscription, retention, and ordering policies.

### Managed Service for Apache Kafka

Kafka APIs, tooling, and partitioned-log model are required.

**Tradeoff:** Kafka partitions, retention, and consumer behavior still need design.

## Who operates what?

- **Self-managed engine:** your team owns provisioning, upgrades, storage, replication, monitoring, and recovery.
- **Managed engine:** provider operates the broker infrastructure within the service contract; your team still owns topology, configuration, clients, and workload behavior.
- **Managed service API:** no broker cluster to operate directly; accept service-specific APIs, limits, and guarantees.

## Sources

- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [streams](https://www.rabbitmq.com/docs/streams)
- [ActiveMQ Classic features](https://activemq.apache.org/components/classic/documentation/features)
- [Amazon MQ engines](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html)
- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery logic](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html)
- [Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html)
- [Google Pub/Sub versus managed Kafka](https://docs.cloud.google.com/pubsub/docs/choose-pubsub-kafka)
57 changes: 57 additions & 0 deletions content/topics/messaging/delivery-shapes.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
57 changes: 57 additions & 0 deletions content/topics/messaging/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
id: '4ba18cf5-d17f-421b-b707-7c7ebca46c85'
title: 'Messaging'
status: 'draft'
summary: 'Choose a messaging model from workload, fan-out, replay, and ordering requirements.'
iconKey: 'architecture'
domains:
- 'system-design'
kind: 'area'
childTopicIds:
- 'dd0ab4c6-2430-46a1-a6b8-5bb116c3e2e7'
- 'b4252c38-54df-408b-a140-eafa6a5eb5c1'
- '50202977-c30d-4d2e-a1d5-893102a0dcce'
- '7b77e594-3472-4324-98b9-6705a606f976'
- 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
relatedTopicIds: []
---

## Choose from the requirement

### Work queue

Distribute thumbnail jobs among workers.

**Tradeoff:** Retries can repeat work; effects must tolerate duplicates.

### Publish/subscribe

Let billing, email, and analytics each react to an order event.

**Tradeoff:** Each subscription needs its own backlog and failure policy.

### Event stream/log

Retain events for replay or process an ordered sequence.

**Tradeoff:** Manage retention and consumer positions; ordering has a defined boundary.

## How delivery differs

![A work queue distributes different jobs among workers; pub/sub sends an event to each subscription; retained-log consumer groups read at independent positions.](./delivery-shapes.svg)

## Before choosing

- **Pub/sub can be durable:** offline consumers do not automatically lose messages. Retention and subscription settings matter.
- **Streams do not require multiple consumers:** retention, replay, and ordered event processing are stronger selection reasons. Ordering is commonly per partition, not global.
- **Products overlap:** queues, pub/sub, and logs are models, not mutually exclusive product categories.

A **producer** sends messages; a **consumer** processes them; a **broker** is infrastructure that routes or stores them. A **topic** names a publication destination; its delivery semantics depend on the product.

> **Database jobs can also fit.** A durable job table avoids another service, but workers need safe claiming, retries, and recovery; polling adds database load.

## Sources

- [RabbitMQ consumers](https://www.rabbitmq.com/docs/consumers)
- [Google Pub/Sub subscriptions](https://docs.cloud.google.com/pubsub/docs/subscription-overview)
- [Kafka design](https://kafka.apache.org/design/)
39 changes: 39 additions & 0 deletions content/topics/publish-subscribe/topic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
id: 'b4252c38-54df-408b-a140-eafa6a5eb5c1'
title: 'Publish/subscribe'
status: 'draft'
summary: 'Fan out events to independent subscriptions while workers can share each subscription.'
iconKey: 'queue'
domains:
- 'system-design'
kind: 'concept'
childTopicIds: []
relatedTopicIds:
- '7b77e594-3472-4324-98b9-6705a606f976'
- 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2'
---

**Choose pub/sub when independent consumers each need the same event.** For an order event, billing and email need separate deliveries; they should not compete for one shared job.

## Topic versus subscription

- **Topic:** Producer publishes an event to a named destination.
- **Subscription or subscriber queue:** Holds one consumer application's delivery/backlog state.
- **Workers sharing a subscription:** Share that application's workload, where supported.
- **Independent subscriptions:** Each receives its own eligible events.

## Offline consumers

**Pub/sub does not necessarily lose messages while consumers are offline.** Durable subscriptions or subscriber queues retain eligible messages within their configured retention limits. Ephemeral subscriptions may provide only live delivery.

An SNS topic can fan out the same order event to:

- **Billing SQS queue** → billing workers.
- **Email SQS queue** → email workers.

SNS fans out to the queues; their workers can process later. Create subscriptions before relying on delivery; historical replay is product- and retention-dependent.

## Sources

- [SNS fan-out to SQS](https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html)
- [Google Pub/Sub subscriptions](https://docs.cloud.google.com/pubsub/docs/subscription-overview)
1 change: 1 addition & 0 deletions content/topics/system-design/topic.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ domains:
- 'system-design'
kind: 'area'
childTopicIds:
- '4ba18cf5-d17f-421b-b707-7c7ebca46c85'
- '17e411bb-2c99-49e8-93ec-18b767e4a890'
- 'b19de3ee-dc7d-4d9d-9b82-06d997a825e1'
- '7c7a63f6-dbe5-432d-b535-0be64ecab578'
Expand Down
Loading
Loading