From 5458cd6b401ea41d9cffb34127a420de5f8958df Mon Sep 17 00:00:00 2001 From: Harlan GitHub Agent Date: Thu, 12 Nov 2026 12:59:04 +1100 Subject: [PATCH 1/2] docs(adr): decide Skill ingestion on Cloudflare primitives D1 filled to 10 GB on 2026-11-04 because ingestion used it as a blob store, an append-only log, and a job queue with no retention. ADR-0019 records the measured hotspots and decides where each kind of data lives: D1 for relational state with named retention, R2 for SKILL.md bodies, KV ETags for GitHub reads, a Workflow per Repository sync, and Analytics Engine for observations. It sequences the migration as independently shippable steps, each with a rollback. Co-Authored-By: Claude Opus 5.5 --- ...kill-ingestion-on-cloudflare-primitives.md | 294 ++++++++++++++++++ 1 file changed, 294 insertions(+) create mode 100644 docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md diff --git a/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md b/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md new file mode 100644 index 00000000..13cedbf9 --- /dev/null +++ b/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md @@ -0,0 +1,294 @@ +# ADR-0019: Skill ingestion on Cloudflare primitives + +**Status:** Accepted +**Date:** 2026-11-12 + +Amends [ADR-0005](0005-v3-cli-harness-artifact-delivery.md) and [ADR-0015](0015-follow-moved-repositories.md). + +## Context + +`skilld-db` reached D1's 10 GB limit on 2026-11-04. Writes failed for eight days. +Twenty-two scheduled tasks went stale and ingestion stopped. #688 fixed the +cause: `sweep-skill-runs` wrote a Resolution with about 60 KB of attestations +every 50 minutes per Skill, twice, and never deleted one. + +The same shape repeats across ingestion. D1 is a transactional store. The +pipeline uses it as a blob store, an append-only log, and a job queue. Each use +grows without a retention window. + +### Measured on 2026-11-12 + +Bounded reads of the production database, with the Worker's own code. + +| Store | Rows | Bytes (sampled) | Growth | Retention | +| --- | --- | --- | --- | --- | +| `artifact_resolutions` | 114,585 | 34 KB each: `attestation_json` 23 KB, `attestation_statement_json` 10 KB, `check_results_json` 1 KB | one per run check | 3 days since #688 | +| `artifact_attestations` | 114,096 | 23 KB each, a second copy of the attestation | one per ready Resolution | cascades from #688 | +| `artifact_check_results` | 792,206 | about 7 rows per Resolution | with Resolutions | cascades from #688 | +| `skills` | 25,145 | 22 KB each: `rendered_raw` 8.9 KB, `rendered_html` 11.9 KB, `rendered_frontmatter` 0.5 KB, `assets` 0.7 KB. Largest `rendered_raw` 117 KB | one per Skill, rewritten on every content change | none, soft-retired | +| `jobs` | 146,816 | 230 bytes each of payload and retry text | up to 250 an hour from `sync-github-skills` | none | +| `skill_revisions` | 146,515 | 830 bytes each, the commit message | up to 30 per changed SKILL.md | none | +| `scheduled_runs` | 159,973 | small | 6 writes per task run, about 1,850 runs a day | 45 days | +| `activity` | 241,974 | small | one per publish or update | none, user-facing feed | +| `embedding_attempts` | 25,578 | small | one per embedding | none | +| `discovery_ledger` | 80,080 | `evidence_text` | one per mention | none | +| `ai_ready_pages` | about 11,700 | 0.5 KB markdown | sequence at 23,359,165 | module owned | + +The estimate: `artifact_resolutions`, `artifact_attestations`, and their check +results hold about 7 GB of the 10 GB. `skills` holds about 550 MB, of which +`rendered_html` is 300 MB and derivable from `rendered_raw`. + +Reads: 224,109 D1 queries read 147 million rows in the 24 hours before this +ADR, with writes failing the whole time. That is page traffic and the +five-minute tasks, not ingestion. + +[GitHub](https://github.com): the hourly sync reads 250 Repositories. One [GraphQL](https://graphql.org) batch of 100 +summaries answers most of them as unchanged. A changed Repository under 50 +Skills costs about 4 requests: summary, tree, a blob batch, and a commit +batch. REST reads already send `If-None-Match` from a KV ETag cache +(`gh-etag:`{lang="html"}, 7 days), and a 304 spends no quota. The artifact build path +sends no conditional request: a first check of a Skill costs about 8 REST +requests plus a codeload archive. Since #688 a ready Skill is rechecked weekly, +so the sweep spends about 1,000 requests a day instead of 30,000. + +`ai_ready_pages`: the AUTOINCREMENT sequence reached 23.3 million for 11,700 +rows because `seedRoutes` in nuxt-ai-ready attempts one INSERT per sitemap URL +on every seed, and [SQLite](https://sqlite.org) bumps the sequence on every attempted insert, even +one that conflicts and changes nothing. The `retired` sitemap added 10,462 +URLs that answered 410, so each seed inserted them and the indexer deleted them +again, with an FTS delete and insert each time. #690 removed that sitemap. +The remaining churn is a module defect, fixed in nuxt-ai-ready, not here. + +Invocations: 25 scheduled tasks fire about 1,850 times a day. Five of them run +every five minutes and most of those runs find no work, yet each pays six +`scheduled_runs` and `sync_jobs` writes. Queue consumers take one message per +invocation, so a 250-job sync is 250 invocations plus continuations. + +## Decision + +Each kind of data lives in the [Cloudflare](https://cloudflare.com) primitive built for it. D1 keeps +small relational state, and every table that grows names its retention. + +```mermaid +flowchart LR + subgraph Discovery + X[X and Bluesky mentions] --> L[(D1 discovery_ledger)] + HN[HN and skills.sh] --> L + W[GitHub App push webhook] --> Q + L --> Q + end + subgraph Sync + C[Hourly cron: 250 Repositories] -->|GraphQL summary batch| U{tree changed?} + U -->|no| D1S[(D1 repos)] + U -->|yes| Q[[Queue skilld-repo-sync]] + Q --> WF[Workflow: sync one Repository] + WF -->|ETag via KV| GH[GitHub REST] + WF -->|SKILL.md body, by sha256| R2[(R2 skill sources)] + WF -->|row without body| SK[(D1 skills)] + WF -->|outcome| AE[(Analytics Engine skilld_ingest_v1)] + Q -.->|after tries| DLQ[[DLQ]] + end + subgraph Derived + SK --> DR[drain-skill-dirty] + SK --> EMB[embeddings to Vectorize] + R2 --> EMB + R2 --> PG[Skill page, .md, embeddings] + SK --> SW[sweep-skill-runs] + SW --> RES[(D1 Resolutions, 3 days)] + end + subgraph Retention + PR[prune tasks] --> RES + PR --> J[(D1 jobs, 7 days)] + PR --> EA[(D1 embedding_attempts, 30 days)] + end +``` + +### D1 holds relational state with a named retention + +Every table a task appends to carries a retention constant and a prune task. +A table without one is a defect. The windows: + +| Table | Window | Prune | +| --- | --- | --- | +| `artifact_resolutions` and cascades | 3 days, public, settled, not the newest attestation of its Artifact | `sweep-skill-runs` (#688) | +| `jobs` | 7 days after `completed_at` | `prune-finished-jobs` | +| `failed_jobs` | 30 days | `prune-finished-jobs` | +| `job_batches` | once finished and empty | `prune-finished-jobs` | +| `embedding_attempts` | 30 days | `prune-finished-jobs` | +| `scheduled_runs` | 45 days | every observed run | +| `repo_star_observations` | 90 days | every sync | +| `x_post_metrics` | 30 days | `refresh-x-engagement` | +| `skill_revisions` | newest 30 per Skill | sync write, this ADR | +| `activity` | none: a user-facing feed. Reads page by `created_at` | not pruned | +| `discovery_ledger`, `discovery_candidates` | none: one row per Repository ever seen, under 100 KB each thousand | not pruned | + +A prune deletes in batches of at most 500 rows through an index seek, with a +cap on batches per run. It runs before the task's own start row is written, so +it can free space while the database is full. + +### Blobs live in R2, addressed by content + +The SKILL.md body moves to R2. `skills.rendered_raw_sha256` already names its +bytes. The key is `v1/skill-source/sha256//.md` in the +`PUBLIC_ARTIFACTS` bucket, the same shape as an Artifact. Two Skills with the +same body share one object. A put is idempotent, so a retried sync writes the +same bytes to the same key. + +`rendered_html` is not stored. `parseSkillMd` is a pure function of the body. +The Skill page already re-parses the body on every request (it did before this +ADR, for description repair), so the column was dead weight. The rendered page +stays cached by the edge cache as it is today. + +D1 keeps what queries filter and sort on: identity, description, frontmatter +JSON, `rendered_raw_sha256`, `rendered_raw_bytes`, status, counters, scores. + +One server utility, `readSkillSource`, reads the body: R2 by sha256 first, then +the D1 column while it still holds bytes. Every reader goes through it: +the Skill page, the `.md` route, the `/api/skills` body select, embeddings, +rename detection, validation, and the AI-ready markdown source. + +Attestations stay in D1. With the 3-day window a Resolution row is transient, +and the newest attestation of an Artifact is one row per Artifact. The +duplicate in `artifact_attestations` serves the Artifact routes and build +reuse. Moving it to R2 saves nothing once retention holds. + +### GitHub reads are conditional and immutable reads are cached + +`ghRequest` keeps its KV ETag cache for REST. The artifact build path adopts +it for the two mutable reads it makes, `GET /repos/{o}/{r}` and `git/ref/...`. +A 304 costs no quota and keeps the exact-commit pin that ADR-0005 requires. +Immutable objects (trees, commits, blobs by SHA) stay in the existing +`artifact-github:v1:` KV cache for 30 days. + +The GraphQL summary batch is the one unconditional read. It costs 3 requests +an hour for 250 Repositories and answers most of them as unchanged. It stays. + +### A GitHub App push webhook marks a Repository for sync + +`layers/artifact-delivery/server/api/v1/webhooks/github.post.ts` handles only +`installation` events today. It gains `push`: on a push to the default branch +of an installed Repository, it enqueues a `registry/repo-maintenance` sync job +with `unique: true` on `owner/repo`. The job is the same one the hourly cron +creates, so a webhook never produces a second code path. A Repository without +the App keeps the hourly poll. Delivery IDs stay deduplicated in +`github_app_webhook_deliveries`. + +### One Repository sync is a Workflow + +Today a Repository with more than 250 SKILL.md files chains across queue +deliveries through `repo_sync_progress` and a `release(1)`{lang="ts"}, and each delivery +counts as an attempt. The sync becomes a Workflow, `SKILL_SYNC_WORKFLOW`, with +one instance per `(owner, repo, treeSha)`{lang="ts"}: + +1. `summary`: GraphQL summary, move detection (ADR-0015), unchanged short-cut. +2. `tree`: REST tree with ETag, truncation check. +3. `slice-N`: one step per 50 SKILL.md paths. Blob batch, commit batch, parse, + R2 puts, D1 writes under the 8 MB batch budget. Each step retries alone. +4. `finish`: `repos.last_tree_sha`, missing Skills, star observation. + +The instance ID is the idempotency key. A second start for the same tree is +refused by the platform, which replaces the `repo_sync_progress` claim. A step +that throws retries with the Workflow's own backoff, and a step that reports a +permanent failure (404, 410, truncated tree) ends the instance with that +reason. The queue message only starts the instance, so `max_retries` goes back +to a failure budget. + +### Append-only observations go to Analytics Engine + +A new dataset, `skilld_ingest_v1`, takes one data point per sync outcome +(Repository, result, files, GitHub requests, quota remaining, duration), per +embedding attempt, per sweep check, and per prune. These are the numbers the +daily check-in reads and the cost follow-up needs. D1 keeps only the current +state: `sync_jobs` (one row per task), `repos.last_synced_at`, and the pruned +evidence tables above. + +### Durable Objects are not used + +Per-Repository serialisation is the only reason to want one. The Workflow +instance ID gives it for the sync, and the Resolution CAS on `state_version` +gives it for builds. A DO would add a billing line and a migration tag for no +invariant the platform does not already hold. + +### Correctness invariants + +- A sync is idempotent. The same tree SHA writes the same R2 objects and the + same D1 rows. `INSERT OR IGNORE` for revisions, upsert for Skills, a put to a + content-addressed key. +- Exactly-once effects come from keys, never from delivery counts. The Workflow + instance ID, the job `unique_key`, the webhook delivery ID, and the + Resolution `request_key_hash` each refuse a duplicate. +- A GitHub read is conditional where GitHub allows it, and a 304 serves the + cached body. A build still pins the exact commit (ADR-0005). +- A reader never fails because a blob moved. `readSkillSource` falls back to + D1 until the backfill is complete, then the column is nulled, then dropped. +- A prune never deletes a row something still reads. Each keep rule is a + subquery, as in `resolution-retention.ts`. +- A retention window is longer than the longest reader: 3 days for a 60 s + poll and a 15 min sweep settle, 7 days for a daily check-in, 45 days for a + monthly digest. + +### Cost + +Monthly, at the measured volumes, Cloudflare list prices as of 2026-11. + +| Item | Before | After | +| --- | --- | --- | +| D1 storage | 10 GB, full | about 1 GB: `skills` without bodies 150 MB, Resolutions 3 days about 150 MB, the rest | +| D1 rows written | 0 while full; about 60 K a day before | about 40 K a day: the duplicate HTML write, the chained attempts, and the FTS rewrites on body updates go | +| D1 rows read | 147 M a day, about $4.40 | unchanged by this ADR | +| R2 | two buckets, Artifacts only | plus 25 K objects, 225 MB, under $0.01 storage and $0.11 for the backfill Class A puts; 25 K Class B reads a day about $0.27 | +| KV | ETag cache | plus 2 reads per build, within the free tier | +| Queues | 6 K messages a day, about $0.07 | unchanged | +| Workflows | none | 250 instances a day, 5 to 10 steps each; billed as Worker CPU plus $0.00 for the first 1 M steps | +| Analytics Engine | web and CLI datasets | plus about 10 K points a day, $0.08 | +| GitHub | about 7 K requests a day after #688 | about 4 K: build ETags and fewer continuations | + +The cost of this ADR is dominated by Worker CPU, which does not change. Its +value is that the database stops filling. + +## Consequences + +The migration is a sequence of small pull requests. Each is safe alone and +each names its rollback. + +| Step | Change | Rollback | +| --- | --- | --- | +| 1 | `prune-finished-jobs` task (#691) | Delete the task. Nothing reads what it removes. | +| 2 | `skill_revisions` keeps the newest 30 per Skill, pruned in the sync write | Delete the statement. | +| 3 | Dual write: the sync puts the body to R2 and records `rendered_raw_bytes`; `readSkillSource` reads R2 then D1 | Readers fall back to D1 on their own. Delete the put. | +| 4 | A backfill task puts every stored body to R2 in batches of 100, marking `rendered_source_r2_at` | Stop the task. Objects are inert. | +| 5 | Readers go through `readSkillSource`; `rendered_html` is no longer written | Reinstate the column write. Pages re-parse either way. | +| 6 | Build path sends `If-None-Match` for the repo and ref reads | Delete the cache lookup. | +| 7 | Push webhook enqueues a sync for installed Repositories | Drop the event from the handler. | +| 8 | `skilld_ingest_v1` data points from sync, sweep, embeddings, prunes | Delete the writes. | +| 9 | Workflow sync behind a per-pool flag, reviewed Repositories first | Flip the flag. The queue job path stays until step 11. | +| 10 | Null `rendered_raw` and `rendered_html` where `rendered_source_r2_at` is set, in batches | A destructive migration. Harlan runs it. The R2 objects are the source. | +| 11 | Drop `repo_sync_progress`, the two columns, and the continuation code | After 10 and a full week of 9. | + +Cull path: a step that fails in production is reverted alone. Steps 1 to 9 +add code and objects; none of them deletes user-facing data. + +Steps 10 and 11 change the public read path only through `readSkillSource`, +which already serves from R2 by then. They ship with Harlan's sign-off. + +## Rejected alternatives + +**A bigger database.** D1 has one limit. Sharding by table set would split +`skills` from its counters and break every join the pages make. + +**Attestations in R2 now.** Once the 3-day window holds, Resolutions are +transient and the per-Artifact copy is bounded by the number of Artifacts. +Moving them adds a read to every `skilld run` for no space the retention does +not already free. + +**Durable Objects per Repository.** Covered above: the Workflow instance ID +and the Resolution CAS already serialise the two paths that need it. + +**Render HTML at sync, cache in KV.** KV values are limited to 25 MiB and the +rendered page is already edge cached. A pure render at read keeps one source +of truth. + +**Hidden retention through `INSERT OR REPLACE`.** It keeps row counts flat but +rewrites every byte and every FTS entry on each pass, which is the +`ai_ready_pages` defect. A retention window and an explicit DELETE is cheaper +and visible. From cb834ed623d09e7e8ac5645ce3d26b3528a8a4b4 Mon Sep 17 00:00:00 2001 From: Harlan GitHub Agent Date: Thu, 12 Nov 2026 13:05:28 +1100 Subject: [PATCH 2/2] docs(adr): claim a Repository per Workflow instance, fence by tree A key of (owner, repo, treeSha) only deduplicated one tree; two trees of one Repository could write at once. The instance is now the Repository claim and the tree SHA is its generation fence. Co-Authored-By: Claude Opus 5.5 --- ...kill-ingestion-on-cloudflare-primitives.md | 27 ++++++++++++++----- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md b/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md index 13cedbf9..fc5a6a48 100644 --- a/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md +++ b/docs/adr/0019-skill-ingestion-on-cloudflare-primitives.md @@ -178,7 +178,7 @@ the App keeps the hourly poll. Delivery IDs stay deduplicated in Today a Repository with more than 250 SKILL.md files chains across queue deliveries through `repo_sync_progress` and a `release(1)`{lang="ts"}, and each delivery counts as an attempt. The sync becomes a Workflow, `SKILL_SYNC_WORKFLOW`, with -one instance per `(owner, repo, treeSha)`{lang="ts"}: +one instance per Repository, ID `sync:/`{lang="html"}: 1. `summary`: GraphQL summary, move detection (ADR-0015), unchanged short-cut. 2. `tree`: REST tree with ETag, truncation check. @@ -186,12 +186,23 @@ one instance per `(owner, repo, treeSha)`{lang="ts"}: R2 puts, D1 writes under the 8 MB batch budget. Each step retries alone. 4. `finish`: `repos.last_tree_sha`, missing Skills, star observation. -The instance ID is the idempotency key. A second start for the same tree is -refused by the platform, which replaces the `repo_sync_progress` claim. A step -that throws retries with the Workflow's own backoff, and a step that reports a -permanent failure (404, 410, truncated tree) ends the instance with that -reason. The queue message only starts the instance, so `max_retries` goes back -to a failure budget. +The instance ID is the Repository claim. The platform refuses a second start +while an instance with that ID runs, so two syncs of one Repository never +write at once, whichever trees they saw. That replaces the `repo_sync_progress` +claim. The tree SHA the `summary` step read is the instance's generation: +every slice and the finish step write `WHERE repos.last_tree_sha IS NOT +newer` through the same compare the continuation path uses today, and a +tree that moved under a running instance ends it with `restart-required`, +as `syncRepo` does now. The next poll or push starts a fresh instance. + +A step that throws retries with the Workflow's own backoff, and a step that +reports a permanent failure (404, 410, truncated tree) ends the instance with +that reason. The queue message only starts the instance, so `max_retries` +goes back to a failure budget. + +During the rollout a Repository is on one path only: the per-pool flag +chooses the Workflow or the queue job before anything is enqueued, so the two +never overlap on one Repository. ### Append-only observations go to Analytics Engine @@ -217,6 +228,8 @@ invariant the platform does not already hold. - Exactly-once effects come from keys, never from delivery counts. The Workflow instance ID, the job `unique_key`, the webhook delivery ID, and the Resolution `request_key_hash` each refuse a duplicate. +- One writer per Repository at a time. The Workflow instance ID, or today the + `repo_sync_progress` claim, holds that; the tree SHA fences a stale writer. - A GitHub read is conditional where GitHub allows it, and a 304 serves the cached body. A build still pins the exact commit (ADR-0005). - A reader never fails because a blob moved. `readSkillSource` falls back to