diff --git a/content/topics/event-streams-and-logs/topic.md b/content/topics/event-streams-and-logs/topic.md new file mode 100644 index 0000000..854c5a5 --- /dev/null +++ b/content/topics/event-streams-and-logs/topic.md @@ -0,0 +1,40 @@ +--- +id: '50202977-c30d-4d2e-a1d5-893102a0dcce' +title: 'Event streams and logs' +status: 'draft' +summary: 'Choose retained event history and understand replay, partitions, and consumer positions.' +iconKey: 'queue' +domains: + - 'system-design' +kind: 'concept' +childTopicIds: [] +relatedTopicIds: + - '7b77e594-3472-4324-98b9-6705a606f976' + - 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' +--- + +**Choose a retained log for history, replay, or ordered event processing.** Multiple subscribers are optional; one consumer can still need replay. + +## Kafka-style log model + +- **Topic:** Named event collection, divided into partitions. +- **Partition:** Ordered append-only sequence; the ordering boundary. +- **Offset:** Position within one partition. +- **Consumer group:** Independent processing view; group members share partitions. +- **Retention:** Policy controlling how long/how much history remains. + +Reading or committing an offset **does not delete the event**. Another group can read at a different position; a group can replay history that is still retained. + +## Boundaries and costs + +- **Ordering is per partition**, not across an entire multi-partition topic. Parallel application processing must preserve any required order too. +- **Retention limits replay.** Cleanup or compaction may remove records; a log is not automatically a permanent archive of every event. +- **Offsets track progress**, not an atomic commit of arbitrary external effects. A crash can repeat processing. +- **Partitioning affects capacity and ordering.** Hot keys can bottleneck one partition; additional workers do not fix that automatically. + +Other products use different terminology and consumption models; check their guarantees. + +## Sources + +- [Kafka introduction](https://kafka.apache.org/intro/) +- [Kafka design](https://kafka.apache.org/design/) diff --git a/content/topics/message-delivery-and-processing/topic.md b/content/topics/message-delivery-and-processing/topic.md new file mode 100644 index 0000000..51a2f21 --- /dev/null +++ b/content/topics/message-delivery-and-processing/topic.md @@ -0,0 +1,42 @@ +--- +id: '7b77e594-3472-4324-98b9-6705a606f976' +title: 'Message delivery and processing' +status: 'draft' +summary: 'Recognize delivery guarantees and make retrying consumers safe.' +iconKey: 'queue' +domains: + - 'system-design' +kind: 'decision-aid' +childTopicIds: [] +relatedTopicIds: [] +--- + +## Delivery versus business effects + +- **At-most-once:** A delivery can be lost; duplicates are avoided within the stated scope. +- **At-least-once:** Redelivery is possible; make effects duplicate-safe. +- **Exactly-once:** Ask **within which boundary?** Broker deduplication or log transactions do not automatically cover external APIs or databases. + +A broker accepting a message does not prove business completion. Retention expiry, repeated failures, or dead-lettering can leave a job unfinished. + +## Acknowledge after a durable result + +1. Receive the message. +2. Apply a duplicate-safe effect and durably record success. +3. Acknowledge the message. + +- **Acknowledge too early:** a later crash can lose the job's effect. +- **Crash after the effect but before acknowledgement:** expect processing again. +- **Visibility timeout / acknowledgement deadline:** temporary delivery ownership; extend it for long work where supported. Expiry can allow redelivery. +- **Transient failure:** retry with backoff and an attempt/time budget. +- **Persistent failure:** isolate in a dead-letter queue where supported; alert, investigate, and deliberately retry or resolve it. + +For thumbnails, a deterministic output key based on image version and thumbnail variant can make repeating generation safe. Ensure an older job cannot overwrite a newer image's output. + +A database deduplication marker must commit with its local effect; marking “seen” before doing work can lose the work after a crash. External effects need their own idempotency or recovery contract. + +## Sources + +- [RabbitMQ acknowledgements and confirms](https://www.rabbitmq.com/docs/confirms) +- [SQS visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) +- [Kafka delivery semantics](https://kafka.apache.org/design/#semantics) diff --git a/content/topics/message-queues/topic.md b/content/topics/message-queues/topic.md new file mode 100644 index 0000000..92a52f8 --- /dev/null +++ b/content/topics/message-queues/topic.md @@ -0,0 +1,39 @@ +--- +id: 'dd0ab4c6-2430-46a1-a6b8-5bb116c3e2e7' +title: 'Message queues' +status: 'draft' +summary: 'Distribute background jobs among workers and recover from interrupted processing.' +iconKey: 'queue' +domains: + - 'system-design' +kind: 'concept' +childTopicIds: [] +relatedTopicIds: + - '7b77e594-3472-4324-98b9-6705a606f976' + - 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' +--- + +**Choose a work queue to distribute jobs among workers.** For thumbnails, each job needs one successful result, rather than an independent result from every worker. + +## Job lifecycle + +1. Receive a job. +2. Generate the thumbnail. +3. Persist the result. +4. Acknowledge/delete the message. + +- **Workers compete:** adding workers shares the backlog; it does not broadcast each job to every worker. +- **Worker fails before acknowledgement:** a recoverable delivery can be retried. The result may already exist, so repeats must be safe. +- **Queue accepts a message:** publication succeeded; thumbnail generation has not necessarily finished. +- **Backlog grows:** buffer bursts, limit worker concurrency, and monitor oldest-job age. A queue does not create processing capacity. + +## Durability is a configuration choice + +Use persistent storage and appropriate replication, publication confirmation, and retention settings. An in-memory queue inside the API process cannot preserve pending work across process loss. + +For large images, queue an **image ID or object-storage reference**, not the image bytes. Workers need access to the referenced object. + +## Sources + +- [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability) +- [SQS Standard queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) diff --git a/content/topics/messaging-implementations-and-hosting/topic.md b/content/topics/messaging-implementations-and-hosting/topic.md new file mode 100644 index 0000000..75d8c8f --- /dev/null +++ b/content/topics/messaging-implementations-and-hosting/topic.md @@ -0,0 +1,104 @@ +--- +id: 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' +title: 'Messaging implementations and hosting' +status: 'draft' +summary: 'Name an engine or cloud service that fits the messaging model and operational constraints.' +iconKey: 'queue' +domains: + - 'system-design' +kind: 'decision-aid' +childTopicIds: [] +relatedTopicIds: [] +--- + +## Narrow the candidates + +- **Background jobs:** RabbitMQ; AWS SQS; GCP Pub/Sub with workers sharing a subscription +- **Independent event subscribers:** RabbitMQ exchanges + queues; AWS SNS + SQS; GCP Pub/Sub +- **Retained log and replay:** Apache Kafka; AWS MSK; GCP Managed Service for Apache Kafka +- **Existing JMS broker integration:** ActiveMQ Classic; Amazon MQ for ActiveMQ Classic + +These are starting points, not exclusive capabilities. Choose the model first, then check ordering, replay, failure handling, and existing platform constraints. + +## Engines: self-managed or hosted + +### RabbitMQ + +- **Fit:** job queues and routed pub/sub through exchanges and queues. +- **Tradeoff:** topology, durability, acknowledgements, and capacity still need design. RabbitMQ also supports streams; it is not limited to queues. +- **Hosting:** run it yourself or use a managed provider, including Amazon MQ. + +### ActiveMQ Classic + +- **Fit:** existing JMS applications and broker/protocol compatibility. +- **Tradeoff:** select and operate the broker topology and verify client/version compatibility. +- **Hosting:** self-managed or Amazon MQ. Specify **Classic**; do not assume Amazon MQ hosts Artemis. + +### Apache Kafka + +- **Fit:** retained event history, replay, and the Kafka processing ecosystem. +- **Tradeoff:** partition keys, retention, consumer lag, and ordering remain design responsibilities even with managed hosting. +- **Hosting:** self-managed, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or another provider. + +## AWS services + +### SQS Standard + +Independent jobs; no strict ordering needed. + +**Tradeoff:** At-least-once delivery; duplicates and out-of-order delivery are possible. + +### SQS FIFO + +Ordering within message groups and send deduplication matter. + +**Tradeoff:** Group design constrains parallelism; retries can still repeat business effects. + +### SNS + SQS + +Fan-out with a durable queue for each consumer application. + +**Tradeoff:** More queues, subscriptions, and failure policies to manage. + +### Amazon MQ + +Managed RabbitMQ or ActiveMQ Classic; engine/protocol compatibility matters. + +**Tradeoff:** Broker capacity and engine-specific configuration still matter. + +### MSK + +Managed Kafka and its APIs/ecosystem. + +**Tradeoff:** Kafka data-model and consumer responsibilities remain. + +**Standard and FIFO are SQS queue types, not pricing tiers.** FIFO deduplication is bounded; a received message that is not deleted can become visible again. + +## GCP services + +### Pub/Sub + +Managed fan-out; shared subscriptions can distribute worker jobs. + +**Tradeoff:** Service-specific API and subscription, retention, and ordering policies. + +### Managed Service for Apache Kafka + +Kafka APIs, tooling, and partitioned-log model are required. + +**Tradeoff:** Kafka partitions, retention, and consumer behavior still need design. + +## Who operates what? + +- **Self-managed engine:** your team owns provisioning, upgrades, storage, replication, monitoring, and recovery. +- **Managed engine:** provider operates the broker infrastructure within the service contract; your team still owns topology, configuration, clients, and workload behavior. +- **Managed service API:** no broker cluster to operate directly; accept service-specific APIs, limits, and guarantees. + +## Sources + +- [RabbitMQ queues](https://www.rabbitmq.com/docs/queues) and [streams](https://www.rabbitmq.com/docs/streams) +- [ActiveMQ Classic features](https://activemq.apache.org/components/classic/documentation/features) +- [Amazon MQ engines](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/welcome.html) +- [SQS Standard](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html) and [FIFO delivery logic](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html) +- [Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html) +- [Google Pub/Sub versus managed Kafka](https://docs.cloud.google.com/pubsub/docs/choose-pubsub-kafka) diff --git a/content/topics/messaging/delivery-shapes.svg b/content/topics/messaging/delivery-shapes.svg new file mode 100644 index 0000000..78d84af --- /dev/null +++ b/content/topics/messaging/delivery-shapes.svg @@ -0,0 +1,57 @@ + + + Messaging delivery shapes + A work queue shares jobs between workers. Pubsub fans an event out to subscriptions. Log consumer groups track separate positions in retained history. + + + + + + + + Work queue: share jobs + + Producer + + Queue: A B C + + + Worker 1: A + + Worker 2: B + + + Different workers receive different jobs. + Retries can repeat a job. + + Pub/sub: fan out an event + + Producer + + Topic: E + + + Sub A: event E + + Sub B: event E + + + Each subscription receives event E. + Its workers can share that backlog. + + Retained log: read at own pace + + Producer + + E1 E2 E3 E4 + + + Group A: pos 2 + + Group B: pos 4 + + + One partition; independent positions. + Reading does not delete the events. + + diff --git a/content/topics/messaging/topic.md b/content/topics/messaging/topic.md new file mode 100644 index 0000000..c8aee5f --- /dev/null +++ b/content/topics/messaging/topic.md @@ -0,0 +1,57 @@ +--- +id: '4ba18cf5-d17f-421b-b707-7c7ebca46c85' +title: 'Messaging' +status: 'draft' +summary: 'Choose a messaging model from workload, fan-out, replay, and ordering requirements.' +iconKey: 'architecture' +domains: + - 'system-design' +kind: 'area' +childTopicIds: + - 'dd0ab4c6-2430-46a1-a6b8-5bb116c3e2e7' + - 'b4252c38-54df-408b-a140-eafa6a5eb5c1' + - '50202977-c30d-4d2e-a1d5-893102a0dcce' + - '7b77e594-3472-4324-98b9-6705a606f976' + - 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' +relatedTopicIds: [] +--- + +## Choose from the requirement + +### Work queue + +Distribute thumbnail jobs among workers. + +**Tradeoff:** Retries can repeat work; effects must tolerate duplicates. + +### Publish/subscribe + +Let billing, email, and analytics each react to an order event. + +**Tradeoff:** Each subscription needs its own backlog and failure policy. + +### Event stream/log + +Retain events for replay or process an ordered sequence. + +**Tradeoff:** Manage retention and consumer positions; ordering has a defined boundary. + +## How delivery differs + +![A work queue distributes different jobs among workers; pub/sub sends an event to each subscription; retained-log consumer groups read at independent positions.](./delivery-shapes.svg) + +## Before choosing + +- **Pub/sub can be durable:** offline consumers do not automatically lose messages. Retention and subscription settings matter. +- **Streams do not require multiple consumers:** retention, replay, and ordered event processing are stronger selection reasons. Ordering is commonly per partition, not global. +- **Products overlap:** queues, pub/sub, and logs are models, not mutually exclusive product categories. + +A **producer** sends messages; a **consumer** processes them; a **broker** is infrastructure that routes or stores them. A **topic** names a publication destination; its delivery semantics depend on the product. + +> **Database jobs can also fit.** A durable job table avoids another service, but workers need safe claiming, retries, and recovery; polling adds database load. + +## Sources + +- [RabbitMQ consumers](https://www.rabbitmq.com/docs/consumers) +- [Google Pub/Sub subscriptions](https://docs.cloud.google.com/pubsub/docs/subscription-overview) +- [Kafka design](https://kafka.apache.org/design/) diff --git a/content/topics/publish-subscribe/topic.md b/content/topics/publish-subscribe/topic.md new file mode 100644 index 0000000..bcd5ac7 --- /dev/null +++ b/content/topics/publish-subscribe/topic.md @@ -0,0 +1,39 @@ +--- +id: 'b4252c38-54df-408b-a140-eafa6a5eb5c1' +title: 'Publish/subscribe' +status: 'draft' +summary: 'Fan out events to independent subscriptions while workers can share each subscription.' +iconKey: 'queue' +domains: + - 'system-design' +kind: 'concept' +childTopicIds: [] +relatedTopicIds: + - '7b77e594-3472-4324-98b9-6705a606f976' + - 'b34b3eca-ea1a-4d87-94f5-ee46a11192f2' +--- + +**Choose pub/sub when independent consumers each need the same event.** For an order event, billing and email need separate deliveries; they should not compete for one shared job. + +## Topic versus subscription + +- **Topic:** Producer publishes an event to a named destination. +- **Subscription or subscriber queue:** Holds one consumer application's delivery/backlog state. +- **Workers sharing a subscription:** Share that application's workload, where supported. +- **Independent subscriptions:** Each receives its own eligible events. + +## Offline consumers + +**Pub/sub does not necessarily lose messages while consumers are offline.** Durable subscriptions or subscriber queues retain eligible messages within their configured retention limits. Ephemeral subscriptions may provide only live delivery. + +An SNS topic can fan out the same order event to: + +- **Billing SQS queue** → billing workers. +- **Email SQS queue** → email workers. + +SNS fans out to the queues; their workers can process later. Create subscriptions before relying on delivery; historical replay is product- and retention-dependent. + +## Sources + +- [SNS fan-out to SQS](https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html) +- [Google Pub/Sub subscriptions](https://docs.cloud.google.com/pubsub/docs/subscription-overview) diff --git a/content/topics/system-design/topic.md b/content/topics/system-design/topic.md index bce6be8..f366b78 100644 --- a/content/topics/system-design/topic.md +++ b/content/topics/system-design/topic.md @@ -8,6 +8,7 @@ domains: - 'system-design' kind: 'area' childTopicIds: + - '4ba18cf5-d17f-421b-b707-7c7ebca46c85' - '17e411bb-2c99-49e8-93ec-18b767e4a890' - 'b19de3ee-dc7d-4d9d-9b82-06d997a825e1' - '7c7a63f6-dbe5-432d-b535-0be64ecab578' diff --git a/src/app/browse/topic-index-page.spec.ts b/src/app/browse/topic-index-page.spec.ts index 6f4b66e..6f499fe 100644 --- a/src/app/browse/topic-index-page.spec.ts +++ b/src/app/browse/topic-index-page.spec.ts @@ -39,7 +39,7 @@ describe('TopicIndexPage', () => { host.querySelectorAll('.domain-hierarchy__root > .domain-hierarchy__item app-topic-icon'), ).map((icon) => icon.getAttribute('data-icon-key')), ).toEqual(['java', 'architecture', 'architecture', 'database']); - expect(host.querySelector('[role="status"]')?.textContent).toContain('92 topics'); + expect(host.querySelector('[role="status"]')?.textContent).toContain('98 topics'); }); it('renders the default System Design view as an expandable hierarchy', async () => { @@ -59,6 +59,12 @@ describe('TopicIndexPage', () => { (link) => link.querySelector('.domain-hierarchy__title')?.textContent?.trim(), ), ).toEqual([ + 'Messaging', + 'Message queues', + 'Publish/subscribe', + 'Event streams and logs', + 'Message delivery and processing', + 'Messaging implementations and hosting', 'Scale and estimation', 'URL shortener', 'Short URL identifiers', @@ -84,7 +90,7 @@ describe('TopicIndexPage', () => { 'Managed vs self-managed databases', ]); expect((host.querySelector('#topic-domain') as HTMLSelectElement).value).toBe('system-design'); - expect(host.querySelector('[role="status"]')?.textContent).toContain('24 topics'); + expect(host.querySelector('[role="status"]')?.textContent).toContain('30 topics'); }); it('shows draft topics and clears every constraint', async () => { @@ -93,9 +99,9 @@ describe('TopicIndexPage', () => { await harness.fixture.whenStable(); const host = harness.routeNativeElement as HTMLElement; expect(host.querySelector('app-domain-topic-hierarchy')).toBeNull(); - expect(host.querySelector('[role="status"]')?.textContent).toContain('3 draft topics'); - expect(host.querySelectorAll('app-topic-result-list a')).toHaveLength(3); - expect(host.querySelectorAll('app-topic-result-list .draft-status')).toHaveLength(3); + expect(host.querySelector('[role="status"]')?.textContent).toContain('9 draft topics'); + expect(host.querySelectorAll('app-topic-result-list a')).toHaveLength(9); + expect(host.querySelectorAll('app-topic-result-list .draft-status')).toHaveLength(9); expect((host.querySelector('#topic-status') as HTMLSelectElement).value).toBe('draft'); await harness.navigateByUrl( @@ -141,12 +147,12 @@ describe('TopicIndexPage', () => { const host = harness.routeNativeElement as HTMLElement; expect(TestBed.inject(Router).url).toBe('/topics?q=Queue'); - expect(host.querySelector('[role="status"]')?.textContent).toContain('3 topics'); + expect(host.querySelector('[role="status"]')?.textContent).toContain('4 topics'); expect( Array.from(host.querySelectorAll('app-topic-result-list a .result__title')).map((title) => title.textContent?.trim(), ), - ).toEqual(['Queue', 'Concurrent queues', 'PriorityQueue']); + ).toEqual(['Queue', 'Concurrent queues', 'Message queues', 'PriorityQueue']); }); it('presents friendly Topic-type labels while preserving kind query values', async () => { diff --git a/src/app/catalog/catalog.service.spec.ts b/src/app/catalog/catalog.service.spec.ts index d4b05c6..53cc2e7 100644 --- a/src/app/catalog/catalog.service.spec.ts +++ b/src/app/catalog/catalog.service.spec.ts @@ -104,7 +104,7 @@ describe('CatalogService', () => { 'system-design', 'algorithms-data-structures', ]); - expect(service.getDomainTopicCount('system-design')).toBe(24); + expect(service.getDomainTopicCount('system-design')).toBe(30); expect(service.getDomainTopicCount('http')).toBe(8); expect(service.getKindOptions().map(({ key }) => key)).toEqual([ 'area', @@ -131,10 +131,16 @@ describe('CatalogService', () => { service.normalizeBrowseCriteria({ status: 'draft' }), ); expect(draftView.mode).toBe('results'); - expect(draftView.resultCount).toBe(3); + expect(draftView.resultCount).toBe(9); expect(draftView.resultHeading).toBe('Draft topics'); expect(draftView.sections[0].topics.map(({ title }) => title).sort()).toEqual([ + 'Event streams and logs', 'HTML Downloader', + 'Message delivery and processing', + 'Message queues', + 'Messaging', + 'Messaging implementations and hosting', + 'Publish/subscribe', 'URL Frontier', 'Web crawler', ]); @@ -143,7 +149,13 @@ describe('CatalogService', () => { ({ status }) => status === 'draft', ); expect(occurrences.map(({ title }) => title).sort()).toEqual([ + 'Event streams and logs', 'HTML Downloader', + 'Message delivery and processing', + 'Message queues', + 'Messaging', + 'Messaging implementations and hosting', + 'Publish/subscribe', 'URL Frontier', 'Web crawler', ]); @@ -187,6 +199,7 @@ describe('CatalogService', () => { expect(view.sections[0].topics.map(({ title }) => title)).toEqual([ 'Queue', 'Concurrent queues', + 'Message queues', 'PriorityQueue', ]); }); @@ -195,7 +208,7 @@ describe('CatalogService', () => { const view = service.getTopicBrowseView({ query: '', domain: null, kind: null, status: null }); expect(view.mode).toBe('hierarchy'); - expect(view.resultCount).toBe(92); + expect(view.resultCount).toBe(98); expect(view.sections).toEqual([]); expect(view.hierarchyRoots.map(({ title }) => title)).toEqual([ 'Java', @@ -236,7 +249,7 @@ describe('CatalogService', () => { status: null, }); - expect(view.resultCount).toBe(24); + expect(view.resultCount).toBe(30); expect(view.mode).toBe('hierarchy'); expect(view.sections).toEqual([]); expect(view.hierarchyRoots.map(({ title }) => title)).toEqual(['System Design', 'Databases']); @@ -246,8 +259,9 @@ describe('CatalogService', () => { 'Relational databases (SQL)', 'Managed vs self-managed databases', ]); - expect(view.hierarchyRoots[0].matchingTopicCount).toBe(17); + expect(view.hierarchyRoots[0].matchingTopicCount).toBe(23); expect(view.hierarchyRoots[0].children.map(({ title }) => title)).toEqual([ + 'Messaging', 'Scale and estimation', 'URL shortener', 'Web crawler',