Repository navigation
feat(registry): write SKILL.md bodies to R2 by content hash - #693
Merged
Merged
Conversation
`skills.rendered_raw` holds every body in D1, about 9 KB a row and 117 KB at the largest, and D1 reached its 10 GB limit on 2026-11-04. ADR-0019 moves the body to R2 under its SHA-256, which the row already records in `rendered_raw_sha256`. This is the dual-write step. The sync puts each changed body to R2 before the D1 batch, so a row that names a hash never points at a missing object, and a failed put leaves the row's own copy in place. A five-minute task copies the rows that predate the change, 200 a run, with a cursor in the new `registry_cursors` table. `readSkillSource` serves R2 first and falls back to the column, for the readers a later step moves over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
harlan-github-agent
Bot
added
the
harlan-agent-running
harlan-github-agent
Bot
added
harlan-agent-running
Contributor
🤖 MERGED
GitHub merged this pull request.
edd72e85-d5f4-42bd-86e0-ed6d6514c4a5 Selected findings run after merge and open separate pull requests. |
harlan-github-agent
Bot
added
harlan-agent-ready
![harlan-github-agent[bot]](https://avatars.githubusercontent.com/in/4579275?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Harlan Agent Kit posted this automated review. AI open source policy.
Automated Review: READY.
harlan-github-agent
Bot
removed
the
harlan-agent-ready
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Linked issue
Related to #126 (daily check-in: database capacity incident). Step 3 and 4 of ADR-0019 (#692).
❓ Type of change
📚 Description
skills.rendered_rawholds every SKILL.md body in D1: about 9 KB a row, 117 KB at the largest, 25,145 rows on 2026-11-12. D1 is a 10 GB store and reached that limit on 2026-11-04. ADR-0019 moves the body to R2 under its SHA-256, which the row already records inrendered_raw_sha256.This is the dual-write step. Nothing reads from R2 yet and no column changes.
skill-source-store.ts: keyv1/skill-source/sha256/<2>/<sha>.mdinPUBLIC_ARTIFACTS. A put is conditional on the key not existing, R2 checks the bytes against the hash, and a failure is a value, not a throw.readSkillSourceserves R2 first and falls back to the D1 column; a later step moves each reader onto it.syncRepocollects each changed body and puts it to R2 before the D1 batch, so a row that names a hash never points at a missing object. A failed put leaves the row's own copy and counts instats.sourceStoreFailures. The repo-maintenance job passes the bucket; scripts and older tests that omit it write D1 only.backfill-skill-sources(*/5) copies rows that predate the change, 200 a run by rowid, about ten hours for the current table. The cursor lives in the newregistry_cursorstable (migration 0159) and only advances past rows whose put succeeded.Cost: 25 K Class A puts for the backfill, about $0.11, and 225 MB of R2 storage.
Rollback: delete the task and the
sourceBucketoption. The objects are inert and the readers never left D1.docs/arch/cron.mdis regenerated andpnpm cron:paritypasses.🤖 Generated with Claude Code