Repository navigation
docs(adr): decide Skill ingestion on Cloudflare primitives - #692
Merged
Merged
Conversation
D1 filled to 10 GB on 2026-11-04 because ingestion used it as a blob store, an append-only log, and a job queue with no retention. ADR-0019 records the measured hotspots and decides where each kind of data lives: D1 for relational state with named retention, R2 for SKILL.md bodies, KV ETags for GitHub reads, a Workflow per Repository sync, and Analytics Engine for observations. It sequences the migration as independently shippable steps, each with a rollback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
harlan-github-agent
Bot
added
harlan-agent-running
Contributor
🤖 MERGED
GitHub merged this pull request.
21da4134-9d37-4a2d-9324-185d50c4b2f1 Selected findings run after merge and open separate pull requests. |
harlan-github-agent
Bot
added
harlan-agent-ready
![harlan-github-agent[bot]](https://avatars.githubusercontent.com/in/4579275?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Harlan Agent Kit posted this automated review. AI open source policy.
Automated Review: READY.
A key of (owner, repo, treeSha) only deduplicated one tree; two trees of one Repository could write at once. The instance is now the Repository claim and the tree SHA is its generation fence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
harlan-github-agent
Bot
added
harlan-agent-running
1 of 6 tasks
harlan-github-agent
Bot
added
harlan-agent-ready
Contributor
There was a problem hiding this comment.
Harlan Agent Kit posted this automated review. AI open source policy.
Automated Review: READY.
1 of 6 tasks
harlan-github-agent
Bot
removed
the
harlan-agent-ready
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Linked issue
Related to #126 (daily check-in: database capacity incident). Follows #688 and #691.
❓ Type of change
📚 Description
ADR-0019 maps Skill ingestion end to end and decides its target shape on Cloudflare primitives.
Measured in production on 2026-11-12, with bounded reads:
artifact_resolutions114,585 rows at about 34 KB each, plus a 23 KB duplicate per row inartifact_attestationsand 792,206 check result rows: about 7 GB of the 10 GB. fix(artifact): stop run checks filling D1 and prune old Resolutions #688 now prunes 500 per sweep.skills25,145 rows at about 22 KB each, 550 MB.rendered_htmlis 300 MB of it and is a pure function ofrendered_raw.jobs146,816,skill_revisions146,515,scheduled_runs159,973,embedding_attempts25,578, none with retention until chore(registry): prune finished durable jobs and embedding attempts #691.ai_ready_pagessequence at 23.3 M for 11.7 K rows: every sitemap seed attempts one INSERT per URL, and theretiredsitemap (chore(seo): remove the experiment E retired sitemap #690) added 10,462 URLs that answered 410.Decisions: every growing D1 table names a retention window and a prune; SKILL.md bodies move to R2 by
rendered_raw_sha256behind onereadSkillSourcereader with D1 fallback;rendered_htmlstops being stored; the build path adopts the ETag cache; the GitHub Apppushwebhook enqueues the same sync job the cron creates; one Repository sync becomes a Workflow keyed by(owner, repo, treeSha); observations go to anskilld_ingest_v1Analytics Engine dataset; no Durable Objects.The Consequences section lists eleven steps, each with its rollback. Steps 10 and 11 are destructive and stay with Harlan.
🤖 Generated with Claude Code