Skip to content

WIP: watcher: add optional SQS subscription for crate index changes - #3349

Draft
syphar wants to merge 14 commits into
rust-lang:mainfrom
syphar:crates-io-sqs
Draft

WIP: watcher: add optional SQS subscription for crate index changes#3349
syphar wants to merge 14 commits into
rust-lang:mainfrom
syphar:crates-io-sqs

Conversation

@syphar

@syphar syphar commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

idea is:

not in here yet, but better for prod:
a graceful shutdown for the watcher / subscriber, so when shutting down the daemon we'll wait until in-flight events are handled.

I'll add this while this PR is in dry-run mode.

@syphar syphar self-assigned this May 22, 2026
@github-actions github-actions Bot added the S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed label May 22, 2026
@syphar syphar changed the title watcher: add sqs sdk, add first draft for event payload structs WIP: watcher: add sqs sdk, add first draft for event payload structs May 22, 2026
@rustbot

This comment has been minimized.

Sign in to view
syphar
syphar commented Jun 3, 2026
View reviewed changes
Comment thread crates/lib/docs_rs_crates_io/src/events.rs
@rustbot

This comment has been minimized.

Sign in to view
@syphar syphar mentioned this pull request Jul 1, 2026
Merged
@syphar syphar changed the title WIP: watcher: add sqs sdk, add first draft for event payload structs WIP: watcher: add optional SQS subscription for crate index changes Jul 1, 2026
syphar
syphar commented Jul 17, 2026
View reviewed changes
Comment thread crates/bin/docs_rs_watcher/src/subscriber.rs Outdated
@rustbot

This comment has been minimized.

Sign in to view
@rustbot

This comment has been minimized.

Sign in to view
@syphar syphar mentioned this pull request Aug 26, 2026
Merged
syphar added 2 commits August 27, 2026 00:17
@syphar
Watcher: Start recording metrics for git index change handling
fa408af
@syphar
Squashed commit of the following:
34f34df 
commit d0a6995
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Aug 27 06:50:04 2026 +0200

    kk

commit 254154c
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Aug 27 06:47:26 2026 +0200

    kk

commit 7d44010
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:44:38 2026 +0200

    kk

commit 8ab10bd
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:36:30 2026 +0200

    kk

commit 6ab809e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:32:38 2026 +0200

    kk

commit 6557417
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:31:51 2026 +0200

    kk

commit 5284aef
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:29:42 2026 +0200

    note

commit 106f930
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:28:36 2026 +0200

    kk

commit 0de0661
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:27:12 2026 +0200

    kk

commit 13ab2e3
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:25:59 2026 +0200

    kk

commit ca54cec
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:21:02 2026 +0200

    metrics

commit d4a259f
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:13:28 2026 +0200

    metri

commit 09d1f68
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:02:33 2026 +0200

    kk

commit 5c13b48
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 22:00:55 2026 +0200

    kk

commit c4e9b38
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:59:30 2026 +0200

    kk

commit 0be13cb
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:57:44 2026 +0200

    kk

commit 08217cd
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:54:42 2026 +0200

    kk

commit 8162966
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:52:24 2026 +0200

    mertrics

commit f3ab38c
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:46:32 2026 +0200

    kk

commit 1870b93
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:41:07 2026 +0200

    kk

commit 866ffa6
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:19:10 2026 +0200

    kk

commit eb61bfe
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:16:07 2026 +0200

    test

commit 3520919
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:11:14 2026 +0200

    kk

commit 76582df
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 21:08:12 2026 +0200

    kk

commit 21bf74e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 20:56:57 2026 +0200

    kk

commit 7ef3f00
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 20:52:33 2026 +0200

    ht

commit 39e1985
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 20:46:26 2026 +0200

    fix

commit 6d32c4b
Merge: b4490bf 79f294f
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 26 20:42:30 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit b4490bf
Merge: 51d00df 2990aab
Author: Denis Cornehl <denis@cornehl.org>
Date:   Mon Aug 17 12:45:15 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit 51d00df
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 12 05:19:16 2026 +0200

    kk

commit d7d1862
Merge: 5613a6d b67fb7d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 12 05:10:10 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit 5613a6d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 09:39:35 2026 +0200

    fixes

commit 5758369
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 09:17:41 2026 +0200

    Revert "WIP"

    This reverts commit 3556734.

commit 221230a
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 09:17:03 2026 +0200

    config

commit 3556734
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 05:08:12 2026 +0200

    WIP

commit 0ce6c2a
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 04:40:13 2026 +0200

    tryt more

commit a4150f0
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 04:18:25 2026 +0200

    kk

commit fc47c25
Merge: c2302c2 c03c6d6
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Aug 5 04:11:17 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit c2302c2
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri Jul 17 18:09:54 2026 +0200

    higher vis timeout

commit dd9c9d5
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:37:40 2026 +0200

    sqlx

commit 0644add
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:33:14 2026 +0200

    anme

commit c6e40c0
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:32:16 2026 +0200

    sql

commit 00f0bca
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:29:51 2026 +0200

    pub

commit 307b678
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:28:14 2026 +0200

    tryform

commit 206d1ea
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:23:14 2026 +0200

    fix env

commit 089ba22
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:22:25 2026 +0200

    fix test

commit 3b5a9b1
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:16:30 2026 +0200

    clean

commit a8f480d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:15:11 2026 +0200

    chore(watcher): refine SQS processing buckets

    Add buckets around the one-minute visibility timeout so slow message handlers are easier to spot in metrics.

commit c0f5b01
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:11:22 2026 +0200

    sort

commit ecd7952
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:10:38 2026 +0200

    names

commit 9fdefe3
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 03:03:09 2026 +0200

    kk

commit 1835ae7
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:59:01 2026 +0200

    small things

commit 84e5739
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:40:29 2026 +0200

    more instrument

commit 1bc440c
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:27:25 2026 +0200

    metrics

commit 79ba59d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:17:17 2026 +0200

    feat(watcher): instrument SQS subscriber metrics

    Add starter counters and histograms for SQS intake, failures, retries, processing time, event lag, and applied change types, and record them from the subscriber flow.

commit ea9a7b3
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:11:00 2026 +0200

    added dummy

commit 405ab70
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 02:02:18 2026 +0200

    comments

commit da03230
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:59:22 2026 +0200

    clean

commit 5b1d257
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:53:41 2026 +0200

    fix(watcher): keep registry watcher alive on SQS errors

    Supervise the SQS subscriber separately in mixed mode so SQS failures are logged and restarted without stopping the legacy registry watcher.

commit 98caa4b
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:50:57 2026 +0200

    refactor(watcher): clarify subscriber naming

    Rename the SQS subscriber entrypoints to better reflect the transport, body-handling, and event-processing layers.

commit 8f3bd6e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:48:00 2026 +0200

    nnma

commit 44c0dd6
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:39:55 2026 +0200

    fix test

commit af73c65
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:35:59 2026 +0200

    config, fixes, comment

commit 6c6d694
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:24:32 2026 +0200

    sqlx

commit b752567
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:21:15 2026 +0200

    config

commit 11c8ecb
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:19:17 2026 +0200

    no locks

commit 28ebe71
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:12:41 2026 +0200

    sort

commit 4050149
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 01:04:27 2026 +0200

    dummy

commit fe8ed9d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:59:01 2026 +0200

    fix wait time

commit 7212133
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:58:17 2026 +0200

    logs

commit 697c831
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:51:10 2026 +0200

    no ui

commit e70bafa
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:46:49 2026 +0200

    feat(watcher): add local ElasticMQ support

    Wire ElasticMQ into docker compose for watcher development and allow the watcher SQS client to target a custom endpoint URL.

commit 28015cf
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:42:41 2026 +0200

    kk

commit b6ef366
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:41:05 2026 +0200

    kk

commit f80a634
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:39:23 2026 +0200

    kk

commit 5a9ef57
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:36:06 2026 +0200

    refactor(watcher): simplify subscriber flow

    Replace the transport trait and helper fan-out with a smaller listen/handle/process structure. Keep tests at the decision boundary with MessageOutcome and remove the recursion-limit workaround.

commit c06a1e5
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:22:45 2026 +0200

    rescursion

commit 876f083
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:15:07 2026 +0200

    refactor

commit 59fc37d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Thu Jul 2 00:05:05 2026 +0200

    refactor

commit 4955cbb
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 23:50:47 2026 +0200

    refactor(watcher): isolate SQS subscriber transport

    Split the subscriber into poll, handle, and decode layers behind a small SQS client trait so transport behavior can be unit-tested without an emulator.

commit 4ee1856
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 23:46:03 2026 +0200

    test(watcher): cover subscriber dispatch

    Add unit tests for process_change and process_message in the SQS subscriber and clean up small watcher warnings found during validation.

commit 6ff0666
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 23:36:08 2026 +0200

    msg

commit c82f60b
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 23:25:56 2026 +0200

    err

commit e5149b1
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 23:08:38 2026 +0200

    simp

commit 5f6dde2
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 22:51:00 2026 +0200

    read

commit 7d6c4c2
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 16:29:00 2026 +0200

    wip

commit 216b68d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 16:08:31 2026 +0200

    WIP

commit b380d7e
Merge: c46d068 f4344f7
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 15:55:29 2026 +0200

    Merge branch 'event-structs' into crates-io-sqs

commit c46d068
Merge: d927bc5 964c90a
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 15:54:23 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit f4344f7
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 14:48:58 2026 +0200

    use String for IndexChangeEvent -> version

commit 6a862ee
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jul 1 14:46:18 2026 +0200

    remove "yanked" from event payload

commit 431662e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jun 3 22:36:11 2026 +0200

    add docs_rs_crates_io subcrate for interaction / shared types

commit d927bc5
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sun Jun 14 14:59:54 2026 +0200

    prio

commit 8f49598
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 05:34:30 2026 +0200

    todo

commit bc3b16e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 05:33:48 2026 +0200

    errs

commit e1f4467
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 05:31:14 2026 +0200

    first version

commit d041799
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 04:12:16 2026 +0200

    make deletes repeatable

commit f056b42
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 04:03:26 2026 +0200

    save

commit fed546f
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 03:55:39 2026 +0200

    sort

commit 3639af8
Merge: 26db891 65af70e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat Jun 13 03:55:19 2026 +0200

    Merge branch 'event-structs' into crates-io-sqs

commit 65af70e
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jun 3 22:36:11 2026 +0200

    add docs_rs_crates_io subcrate for interaction / shared types

commit 26db891
Merge: c0436b1 7a6c393
Author: Denis Cornehl <denis@cornehl.org>
Date:   Wed Jun 3 22:28:35 2026 +0200

    Merge branch 'main' into crates-io-sqs

commit c0436b1
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat May 23 09:22:13 2026 +0200

    no rustls

commit bb4d845
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat May 23 08:22:02 2026 +0200

    some cleanup

commit 1e374cf
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat May 23 08:17:50 2026 +0200

    renames

commit e84c343
Author: Denis Cornehl <denis@cornehl.org>
Date:   Sat May 23 08:15:59 2026 +0200

    refactor(events): drop schema version

    Remove the redundant schema_version field from the crates.io event envelope and keep versioning in the typed payloads.

commit b311a59
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 18:57:05 2026 +0200

    chore(lockfile): record watcher url dep

    Update Cargo.lock after making docs_rs_watcher depend directly on url.

commit de207e5
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 18:56:48 2026 +0200

    refactor(watcher): parse SQS queue URL

    Use url::Url for the watcher SQS queue URL config so invalid values fail during config loading.

commit cb7453f
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 18:56:01 2026 +0200

    feat(watcher): add SQS config

    Add watcher config fields for an SQS queue URL and region to support an event-based path.

commit 277ea22
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 18:33:43 2026 +0200

    fix(watcher): make version delete idempotent

    Treat duplicate version deletion events as a no-op so temporary event-based handling can safely replay them.

commit 0b7dd8d
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 17:03:56 2026 +0200

    wider deps

commit 6b053a3
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 16:35:11 2026 +0200

    refactor(events): use chrono timestamps

    Replace time::OffsetDateTime with chrono::DateTime<Utc> for RFC 3339 event timestamps.

commit e52cae8
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 16:32:57 2026 +0200

    refactor(events): use typed event timestamps

    Remove event source metadata and store occurred_at as an RFC 3339 OffsetDateTime.

commit b7403da
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 16:25:14 2026 +0200

    refactor(events): version event payload types

    Rename the current wire payload to ChangeV1 and make the event envelope generic for future schema versions.

commit 87c4384
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 16:14:48 2026 +0200

    feat(events): add event envelope metadata

    Wrap typed change payloads in a conventional event envelope with id, occurred_at, source, and schema_version.

commit fddeb9a
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 15:40:36 2026 +0200

    add shared subcrate for event types

commit 9d1b50b
Author: Denis Cornehl <denis@cornehl.org>
Date:   Fri May 22 15:16:04 2026 +0200

    add aws-sdk-sqs
@rustbot

rustbot commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

☔ The latest upstream changes (possibly #3477) made this pull request unmergeable. Please resolve the merge conflicts.

@syphar syphar closed this Sep 9, 2026
@syphar syphar reopened this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@syphar syphar

Labels

S-waiting-on-review Status: This pull request has been implemented and needs to be reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@syphar @rustbot