✨ feat(git0): add partial downloads and lazy history attachment by vtempest · Pull Request #106 · OpenSourceAGI/dev-tools-starter-agent

vtempest · 2026-09-17T05:14:25Z

Adds support for downloading individual files or folders from a repository, and optionally attaching full Git history after extraction completes. This lets users download just the parts they need while keeping the initial download fast.

Key changes

Partial path downloads: New normalizeSubPath(), subPathFilter(), and hoistSubPath() functions enable filtering the tarball to a single file or folder and moving it to the extraction root, so git0 owner/repo/tree/main/packages/ui downloads only that folder.
Git history attachment: New history.ts module with attachGitHistory() and attachGitHistoryLazily() to fetch .git either immediately or in the background after extraction. The lazy variant lets files be on disk and bun install start while history downloads.
URL parsing: parseTarget() in github-api.ts reads GitHub URLs including /tree/ and /blob/ links, extracting branch and path information so users can paste links directly.
CLI flags: New args.ts parser and --path, --branch, --history, --history-only, and --mirror flags. The --path flag also works as a shorthand (git0 owner/repo/path/to/folder).
Comprehensive tests: New test suites for sub-path normalization, filtering, hoisting, history attachment, argument parsing, and target parsing. Benchmark suite comparing tarball + lazy history vs. git clone strategies.
Documentation: Updated README with usage examples, new features, and benchmark results. Added benchmark/README.md explaining why the tarball approach wins on time-to-files.

Implementation details

Sub-path filtering happens in tar's parser (before unpacking), so filtered entries never reach disk — downloading one folder of a large repo writes only that folder.
History is fetched into a staging directory and renamed into place only on success, so interrupted downloads leave no half-written .git.
The printLogo() function is now idempotent to prevent duplicate branding when multiple entry points print it.
All new functions are thoroughly documented with JSDoc including examples and error cases.

https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

`main()` printed the banner before parsing arguments and then `downloadRepoAndSetup()` printed it again, so the most common invocation of all — `git0 owner/repo` — rendered the logo twice and looked broken. Deleting one of the calls would have fixed the common case and left the paths that reach a download without going through `main()` unbranded, so `printLogo` is now idempotent instead: every call site can still say "show the branding here" and the user sees it exactly once. `resetLogo()` lets the tests drive several flows through one process. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

A link copied out of the GitHub UI already says which folder or file you want — `/tree/main/packages/react-dom`, `/blob/master/.continue/agents/x.yaml` — and git0 threw that away and downloaded the repository around it. It is now read and honoured. Entries outside the path are dropped in tar's parser, ahead of the unpacker's `strip`, so they are skipped in the stream and never written; the requested subtree is then hoisted to the root of the target folder, which reads dir-vs-file off the disk rather than guessing at a strip depth the `owner/repo/some/path` shorthand cannot supply. The folder is named after the path rather than the repository, because downloading `debate-ai.com/.continue/agents` into a folder called `debate-ai.com` claims to hold a site and holds two config files. Also adds `--path`, `--branch`/`--ref`, and flag parsing in its own module so it is testable without executing `cli.ts`'s top-level `main()`. A failed tarball request now reports the repository and branch instead of leaving an empty directory for the next step to trip over. `..` in a sub-path is rejected rather than normalized away: the value becomes a prefix match and then a path join, and the only honest reading of `packages/../../etc` in a "download this folder" argument is that something is wrong with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

`--history` ends in the same state `git clone` would have left you in — a working tree with a full `.git` beside it, `git log` working, `git status` clean — without giving up the speed that is git0's whole point. The ordering is the feature. A clone is serial: the remote counts and packs every object in the history, sends the pack, and only then writes files you can run. A codeload tarball is a pre-made, cached, already-compressed snapshot that extracts while it downloads. And history is the one part of a clone that nothing on the critical path waits for — `bun install` does not need `git log` — so it is started after extraction finishes and awaited at the very end, overlapping the IDE launch and the dependency install rather than blocking them. Mechanically: bare clone into a temp directory, renamed into place only once it succeeds so an interrupted download leaves no half-written `.git`; `core.bare` flipped; origin's fetch refspec restored; `git reset --mixed HEAD` to populate the index from HEAD without touching the extracted files. A failed fetch is reported and swallowed — a missing `git` or a dropped connection should cost the user their history, not the project they are already working in. `--history-only` is the other half: a bare clone into `<repo>.git` with no working files, or `--mirror` for every ref, for a backup or a host-to-host move. `benchmark/` measures the claim rather than asserting it — time to files (when the project can run) separately from time to history, across `git clone`, `git clone --depth=1`, git0 and `git0 --history`. It is a `.bench.ts`, so `bun test` does not collect it and it never runs in CI; its pure reporting helpers are unit-tested. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

git0 has a site, and nothing in the README pointed at it. The badge catalog already carries a `website` badge; what was missing was a `websiteUrl` for package headers. It is read off `homepage` rather than a list kept in the script — npm already asks for that field and renders it, so a package with a site has usually filled it in, and the ones that have not left it at the GitHub tree URL the publish flow writes. A badge back to the page you are already on is noise, so a `github.com` homepage yields no badge and every other package's header is unchanged. git0's own `homepage` now points at the site instead of its directory on GitHub, which is also what npm should have been showing. The README gains the link and documents both new features: partial paths, and `--history` with the reasoning for why fetching history last is faster than cloning it first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

The skill is the documentation agents actually load, so it gains the new invocations, a flag table, the mechanics of hoisting and of the bare-to- working conversion, and the four new failure messages with their causes. The package note records the three things a future change is most likely to undo by accident: `printLogo`'s guard is the fix and not a redundancy, `normalizeSubPath` rejecting `..` is a boundary check and not tidying, and awaiting the history download earlier silently gives back the speed the feature exists to keep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

`bun run readmes:check` was already failing on master for these four — their generated headers had drifted from what `sync-package-readmes.mjs` produces. Unrelated to the git0 work in this branch, but CI fails on the whole repo, so the generator's own output is brought back in line. No hand edits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

vercel · 2026-09-17T05:14:31Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Updated
cccp-deployment	Error	Sep 17, 2026 9:11am UTC
dev-tools-help-docs	Error	Sep 17, 2026 9:11am UTC
test-google-login	Error	Sep 17, 2026 9:11am UTC

master's version-bump commit moved git0 to 0.2.99 while this branch had already moved it to 0.3.0 for the new flags. Resolved to 0.3.0 — it is ahead of master's patch bump, and the new public behaviour (`--path`, `--branch`, `--history`, `--history-only`, `--mirror`) is a minor, not a patch. That version line was the only conflict; everything else merged cleanly. Revalidated after the merge: 134 tests pass, the bundle builds, and `readmes:check` is clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

vtempest · 2026-09-17T08:22:44Z

The three red Vercel checks are not this PR's

Vercel – dev-tools-help-docs, Vercel – cccp-deployment and Vercel – test-google-login are all failing, and all three fail for the same pre-existing reason: bun install at the repo root exits non-zero on master.

error: The entry point "source.config.ts" cannot be marked as external
error: postinstall script from "api2ai" exited with 1

The cause is packages/api2ai-mcp-generator/package.json:

"postinstall": "fumadocs-mdx && drizzle-kit push"

fumadocs-mdx needs a source.config.ts, and api2ai-mcp-generator has none — the five source.config.ts files in this repo all belong to other workspaces. Because it is a root-install postinstall, it takes the whole install down with it, so every Vercel project that builds from this monorepo fails before it reaches its own build step. That is why three unrelated projects went red at once.

Why it isn't this branch:

I reproduced it on a clean checkout before making any change in this branch.
packages/api2ai-mcp-generator is not in this diff (git diff --name-only origin/master...HEAD | grep api2ai → 0).
Two of the three failing projects — apps/Cloud-Computer-Control-Panel and apps/dev-tools-help-docs — are not touched by this diff at all. The third's only change is a regenerated README badge header, which cannot fail a build.

No fix for it exists anywhere yet, so there is nothing to port into this PR, and fixing it here would mean editing a package this change has no business touching (CLAUDE.md: one package per change).

Proposed patch, as a separate change:

--- a/packages/api2ai-mcp-generator/package.json
+++ b/packages/api2ai-mcp-generator/package.json
-    "postinstall": "fumadocs-mdx && drizzle-kit push",

Worth doing on its own merits beyond unblocking CI: api2ai publishes to npm, so this postinstall also runs drizzle-kit push — a command that writes to a database — on every machine that installs the package. If the docs generation is genuinely wanted, it belongs in a docs script that the docs build calls, not in postinstall.

This PR's own state: master has been merged in (its version bump to 0.2.99 conflicted with this branch's 0.3.0, resolved to 0.3.0), and revalidated on the merged tree — 134 tests pass under bun test, the bundle builds, bun run readmes:check is clean.

I have no way to re-run a Vercel deployment, so I am not spending a re-run on this.

Generated by Claude Code

codecov · 2026-09-17T08:25:01Z

Codecov Report

❌ Patch coverage is 69.90291% with 62 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/git0-repo-downloader/src/cli.ts	0.00%	50 Missing ⚠️
packages/git0-repo-downloader/src/github-api.ts	76.00%	6 Missing ⚠️
packages/git0-repo-downloader/src/history.ts	89.09%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

#106 landed as a squash commit, so this branch shares no commits with the result even though master already carries every line of it. Merging master back in realigns the two: the merge-base becomes master's tip, and what is left on this branch is the one commit that has not landed yet — the CI install fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KULWViqKDCNmcShYttoU23

claude added 6 commits September 17, 2026 03:51

vtempest merged commit 12a80c3 into master Sep 17, 2026
27 of 32 checks passed

vtempest deleted the claude/adoring-pasteur-9nsaza branch September 17, 2026 08:25

vercel Bot had a problem deploying to Preview – test-google-login September 17, 2026 08:27 Failure

vtempest mentioned this pull request Sep 17, 2026

🔧 fix(ci): install with --ignore-scripts for the git0 test job #107

Merged

vercel Bot had a problem deploying to Preview – dev-tools-help-docs September 17, 2026 09:04 Failure

vercel Bot had a problem deploying to Preview – cccp-deployment September 17, 2026 09:11 Failure

Navigation Menu

✨ feat(git0): add partial downloads and lazy history attachment - #106

✨ feat(git0): add partial downloads and lazy history attachment#106
vtempest merged 7 commits into
masterfrom
claude/adoring-pasteur-9nsaza

vtempest commented Sep 17, 2026

Uh oh!

vercel Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

vtempest commented Sep 17, 2026

Uh oh!

codecov Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vtempest commented Sep 17, 2026

Key changes

Implementation details

Uh oh!

vercel Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vtempest commented Sep 17, 2026

The three red Vercel checks are not this PR's

Uh oh!

codecov Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Sep 17, 2026 •

edited

Loading

codecov Bot commented Sep 17, 2026 •

edited

Loading