feat(http): segmented parallel download via HTTP Range (aria2-style) - #10
Open
FarnaHerry wants to merge 4 commits into
Open
feat(http): segmented parallel download via HTTP Range (aria2-style)#10FarnaHerry wants to merge 4 commits into
FarnaHerry wants to merge 4 commits into
Conversation
aria2-style multi-connection downloads. download_to_file_parallel() probes with Range: bytes=0-0, then splits the file into segments fetched concurrently into a pre-allocated file by a worker pool pulling from a shared segment index: - maxConnectionsPerFile caps concurrent workers (-x) - maxSegments sets the split count (-s); 0 ties it to the connection count, preserving the pre-feature behavior - minSegmentBytes sets the smallest split worth making (--min-split-size) - interrupted segments resume mid-range on retry (2 retries each) - falls back to the sequential path when the server ignores Range, the resource is empty (416), or the file is too small to split - 206 responses are validated (Content-Range start/total, no chunked framing) so a misbehaving server cannot corrupt the output - progress callbacks stay monotonic across worker threads All existing API is unchanged; default config (maxConnectionsPerFile=1) behaves exactly like before.
…ngo.org New ParallelDownloadTest suite (7 cases) against httpbingo.org, which honors Range on /range/N and serves deterministic content: - segmented result byte-identical to sequential download - maxSegments decoupled from connection count - monotonic parallel progress - fallback when the server ignores Range (200) - small-file fallback to the sequential path - redirect resolution before segmenting - cancellation aborts all workers Also migrate the existing httpbin.org tests to httpbingo.org (same API, more reliable); httpbin.org frequently serves 503s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds aria2-style segmented parallel downloading to
HttpClient: a newdownload_to_file_parallel()entry point plus three tuning knobs onHttpClientConfig. The change is purely additive — with the default config the behavior is byte-for-byte identical to 0.2.9, and no existing code path is modified (the diff againstmastertouches only new code insrc/http.cppm, plus tests/docs/build files).Version bumped 0.2.9 → 0.3.0.
New API surface
Same progress/cancellation semantics and the same
DownloadToFileResultcontract asdownload_to_file().How it works
GET ... Range: bytes=0-0(withConnection: close). Redirects are resolved at this stage so every segment worker later targets the final URL directly.200— the server ignoredRange. The probe response body is the whole file, so it is streamed straight to disk over the probe connection (no wasted round-trip). Handles chunked / Content-Length / close-delimited bodies.416— empty resource; a zero-byte file is written and the call succeeds.206— total size is taken fromContent-Range(with overflow-checked parsing; unknown/*totals fall back to the sequential path).resize_file()'d to the total size, so workers can write disjoint ranges in place with no write-side synchronization.ceil(total / minSegmentBytes)segments (capped bymaxSegments, hard-capped at 2048).maxConnectionsPerFileworkers pull segment indices from a shared atomic counter — this decouples split count from connection count (-svs-x), so slow segments on one connection don't bound the overall layout.connect_fresh) and never touches the shared pool, which is what makes the whole thing thread-safe without locking the client. A worker requests only the remaining bytes of its segment (Range: bytes=start+written-end), so a dropped connection resumes mid-range on retry (2 retries per segment).Content-Rangestart must equal the requested offset and the total must match the probe; chunked 206 responses are rejected (the raw framing bytes would corrupt the pre-allocated file). A misbehaving server fails the download instead of silently writing garbage.std::terminate.Compatibility
maxConnectionsPerFile = 1→download_to_file_parallel()simply forwards todownload_to_file()).Tests
New
ParallelDownloadTestsuite (7 cases, all passing):maxSegments>maxConnectionsPerFile(decoupled split/connection counts)finalUrlis the target URL)"cancelled"Test endpoint migration: the existing
DownloadToFileTestcases were moved from httpbin.org to httpbingo.org (same API, actively maintained). httpbin.org has been serving frequent 503s, which made CI red for reasons unrelated to the code; httpbingo is also the only one of the two whose/range/Nendpoint honorsRange, which the new suite needs.Other
.xlings.json: it pinned mcpp 0.0.87, which broke out-of-the-box builds for anyone with a different toolchain version..gitignore: ignorecompile_commands.json(mcpp build artifact).Checklist
mcpp buildclean (clang 22, C++23 modules)mcpp test— 16/16 download tests pass (Linux x86_64)