Stream a harvested Overdrive feed to disk instead of holding it in memory - #331
Open
jonathangreen wants to merge 5 commits into
Open
Stream a harvested Overdrive feed to disk instead of holding it in memory#331jonathangreen wants to merge 5 commits into
jonathangreen wants to merge 5 commits into
Conversation
jonathangreen
changed the base branch from
save-partial-overdrive-harvest
to
refresh-overdrive-token
August 31, 2026 12:57
An Overdrive harvest runs for hours, and until now anything that stopped it threw away everything it had downloaded. A token that couldn't be refreshed, a request that never succeeded, or a Ctrl-C three hours in all left no output file at all. Errors deep in a harvest used to call sys.exit(), which can't be recovered from -- and inside a request task it tears the event loop down where it stands. They raise OverdriveError now instead. fetch() turns anything that ends a harvest early, cancellation included, into a HarvestAborted carrying the products downloaded so far, and download-feed writes those out before exiting with the same non-zero status as before. Claude-Session: https://claude.ai/code/session_01JF12Q485dQh1J73NA7CjvV
…mory
A harvest collected every product into one dict and handed the whole thing
back at the end, so peak memory grew with the size of the collection. A
consortium-sized feed with metadata and availability ran to gigabytes.
fetch() is now an async iterator that hands each product over as soon as
the last of its requests lands, and drops it. What it holds is bounded by
the requests in flight -- 2 * connections pages, so 8000 products at the
default settings -- rather than by the size of the collection. The one
thing still kept per product is its id, so that a title listed on two
pages isn't harvested twice, at about 80 bytes each.
download-feed writes products out as they arrive. The file is byte for
byte what json.dumps(products, indent=4) produced before, so nothing
downstream has to change, and the array is closed on the way out so an
aborted harvest still leaves valid JSON behind.
Measured with tracemalloc against a mock feed, metadata and availability
on, 20 connections, one measurement per process:
items retained (old) streamed (new) held
2000 38.3 MB 7.8 MB 2000
8000 103.0 MB 52.2 MB 8000
32000 304.0 MB 59.0 MB 8000
128000 1122.7 MB 73.9 MB 8000
Claude-Session: https://claude.ai/code/session_01JF12Q485dQh1J73NA7CjvV
A product whose only outstanding request 404s under --skip-not-found is never going to be completed, so it has to be let go of at that point. The end-of-harvest drain hands it over either way, so a test that only looks at the harvested feed can't tell the difference -- what changes is how long it is held. Watch the high-water mark instead. Claude-Session: https://claude.ai/code/session_01JF12Q485dQh1J73NA7CjvV
Three places encoded how many requests follow each product: the count PendingProducts waits on, the progress bar's total, and the process_request branch that actually enqueues the URLs. The last is the real rule -- one request for metadata, two for availability -- and the other two restated it. requests_per_product() gives that rule a name, with a docstring saying what a count that disagrees with the URLs costs: too high and a product is stranded until the end-of-harvest drain, which is the memory growth this branch exists to fix; too low and it's released before its last response lands, and attaching that response raises KeyError. The progress total reads better for it too. It is pages + items * per_product now, which is the rule, rather than the 1-and-2 split multiplied back out. Claude-Session: https://claude.ai/code/session_01LPuVWiNPYGUM3vpTazw4Yi
fetch() caught CancelledError and re-raised it as HarvestAborted, so that an interrupted harvest could hand back everything it had downloaded. Streaming the feed to disk took most of the reason for that away: everything finished is already written by the time the Ctrl-C lands, and what converting the cancellation buys is only the products still in flight -- at most 2 * connections pages, 6% of a large harvest. That is not worth what it costs. Swallowing a cancellation leaves the harvest uncancellable, and it robs the caller of the KeyboardInterrupt that asyncio.run raises in its place, so an interrupt was reported as a failure and exited 255. Cancellation now propagates untouched, HarvestAborted is for failures only, and download-feed catches the KeyboardInterrupt to report the interrupt and exit 130. Verified end to end against a mock feed, in a foreground process so that SIGINT is the terminal's default rather than the SIG_IGN a background job inherits: one Ctrl-C 23% into a 6000 item harvest exits 130 immediately with no traceback, and leaves 1344 complete products in a file that parses. Claude-Session: https://claude.ai/code/session_01LPuVWiNPYGUM3vpTazw4Yi
jonathangreen
force-pushed
the
stream-overdrive-harvest-to-disk
branch
from
August 31, 2026 14:01
db93a4f to
fa67b35
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
fetch()is now an async iterator. A product is released as soon as its last request lands, so the harvest holds only what is in flight —2 × connectionspages, 8,000 products at the defaults — not the whole collection.download-feedwrites each one as it arrives.Worth a look:
PendingProductscounts the requests each product waits on and releases it at zero. Two cases, both tested: a skipped 404 has to decrement or the product waits until the end of the run, and a title listed on two pages must not be written twice, which the old id-keyed dict handled implicitly.json.dumps(products, indent=4), asserted in a test. The array is closed on the way out, so a partial file still parses.sys.exit()→OverdriveError, four call sites. The prerequisite, not a drive-by:SystemExitraised in a request task escapesasyncio.runpast an enclosingexcept, so nothing could recover.HarvestAbortedcarries only the in-flight products now; the rest is already on disk.Motivation and Context
A consortium-sized feed with metadata and availability ran to gigabytes, all held until the last request landed.
tracemallocagainst a mock feed, 20 connections:Flat once the in-flight window saturates. The residual slope is the id set at ~80 bytes a product, 10 MB of the last row — the price of not writing duplicates. Say the word if you would rather have constant memory and accept them.
Second benefit: a harvest that fails or is interrupted now leaves everything it finished on disk instead of nothing.
How Has This Been Tested?
32 new tests, 235 total against #329's 203.
TestStreamingcovers the bound and both edge cases,TestPendingProductsandTestProductWriterthe bookkeeping and the file format,TestPartialHarvesteach way a harvest can end.Nine mutations tried, all caught — including the two that matter most, an in-flight count one too high (products stranded, memory grows again) and one too low (released before their last response lands).
Ctrl-C checked end to end against a mock feed, in a foreground process so SIGINT is the terminal's default rather than the
SIG_IGNa background job inherits: one interrupt 23% into a 6,000-item harvest exits 130 with no traceback and leaves 1,344 complete products in a file that parses.mypy --strict,pre-commit, and the full suite are clean. Not tested against the live Overdrive API.Checklist