Skip to content

feat(http): segmented parallel download via HTTP Range (aria2-style) - #10

Open
FarnaHerry wants to merge 4 commits into
mcpplibs:masterfrom
FarnaHerry:feat/range-parallel-download
Open

feat(http): segmented parallel download via HTTP Range (aria2-style)#10
FarnaHerry wants to merge 4 commits into
mcpplibs:masterfrom
FarnaHerry:feat/range-parallel-download

Conversation

@FarnaHerry

Copy link
Copy Markdown

Summary

Adds aria2-style segmented parallel downloading to HttpClient: a new download_to_file_parallel() entry point plus three tuning knobs on HttpClientConfig. The change is purely additive — with the default config the behavior is byte-for-byte identical to 0.2.9, and no existing code path is modified (the diff against master touches only new code in src/http.cppm, plus tests/docs/build files).

Version bumped 0.2.9 → 0.3.0.

New API surface

export struct HttpClientConfig {
    // ... existing fields unchanged ...
    int maxConnectionsPerFile { 1 };         // concurrency cap (-x). 1 = sequential (default)
    int maxSegments { 0 };                   // split count (-s). 0 = tie to maxConnectionsPerFile
    std::int64_t minSegmentBytes { 1 << 20 };// minimum bytes per segment (--min-split-size)
};

HttpClient::download_to_file_parallel(url, destFile, onProgress, isCancelled);

Same progress/cancellation semantics and the same DownloadToFileResult contract as download_to_file().

How it works

  1. Probe. A dedicated connection issues GET ... Range: bytes=0-0 (with Connection: close). Redirects are resolved at this stage so every segment worker later targets the final URL directly.
  2. Three-way decision:
    • 200 — the server ignored Range. The probe response body is the whole file, so it is streamed straight to disk over the probe connection (no wasted round-trip). Handles chunked / Content-Length / close-delimited bodies.
    • 416 — empty resource; a zero-byte file is written and the call succeeds.
    • 206 — total size is taken from Content-Range (with overflow-checked parsing; unknown/* totals fall back to the sequential path).
  3. Pre-allocate. The destination is created and resize_file()'d to the total size, so workers can write disjoint ranges in place with no write-side synchronization.
  4. Work queue. The file is split into ceil(total / minSegmentBytes) segments (capped by maxSegments, hard-capped at 2048). maxConnectionsPerFile workers pull segment indices from a shared atomic counter — this decouples split count from connection count (-s vs -x), so slow segments on one connection don't bound the overall layout.
  5. Workers. Each worker uses its own TLS connection per attempt (connect_fresh) and never touches the shared pool, which is what makes the whole thing thread-safe without locking the client. A worker requests only the remaining bytes of its segment (Range: bytes=start+written-end), so a dropped connection resumes mid-range on retry (2 retries per segment).
  6. Validation. Each 206 response is checked before writing: Content-Range start must equal the requested offset and the total must match the probe; chunked 206 responses are rejected (the raw framing bytes would corrupt the pre-allocated file). A misbehaving server fails the download instead of silently writing garbage.
  7. Completion. Any segment failure stops the pool, removes the partial file, and reports the first error. Progress callbacks are serialized and kept monotonically increasing across threads. Worker exceptions are contained and reported as download failures instead of reaching std::terminate.

Compatibility

  • No existing API changed; all new config fields default to legacy behavior (maxConnectionsPerFile = 1download_to_file_parallel() simply forwards to download_to_file()).
  • No new dependencies; still mbedTLS-only, C++23 modules.
  • Works through HTTP CONNECT proxies (segment workers use the same proxy path as the pool).

Tests

New ParallelDownloadTest suite (7 cases, all passing):

  • segmented result is byte-identical to a sequential download of the same resource
  • maxSegments > maxConnectionsPerFile (decoupled split/connection counts)
  • parallel progress is strictly monotonic
  • fallback when the server ignores Range (probe gets 200)
  • small file falls back to the sequential path
  • redirect is resolved before segmenting (finalUrl is the target URL)
  • cancellation aborts all workers and reports "cancelled"

Test endpoint migration: the existing DownloadToFileTest cases were moved from httpbin.org to httpbingo.org (same API, actively maintained). httpbin.org has been serving frequent 503s, which made CI red for reasons unrelated to the code; httpbingo is also the only one of the two whose /range/N endpoint honors Range, which the new suite needs.

Other

  • Removed .xlings.json: it pinned mcpp 0.0.87, which broke out-of-the-box builds for anyone with a different toolchain version.
  • .gitignore: ignore compile_commands.json (mcpp build artifact).

Checklist

  • mcpp build clean (clang 22, C++23 modules)
  • mcpp test — 16/16 download tests pass (Linux x86_64)
  • No changes to existing public API
  • README updated

aria2-style multi-connection downloads. download_to_file_parallel()
probes with Range: bytes=0-0, then splits the file into segments
fetched concurrently into a pre-allocated file by a worker pool
pulling from a shared segment index:

- maxConnectionsPerFile caps concurrent workers (-x)
- maxSegments sets the split count (-s); 0 ties it to the connection
  count, preserving the pre-feature behavior
- minSegmentBytes sets the smallest split worth making (--min-split-size)
- interrupted segments resume mid-range on retry (2 retries each)
- falls back to the sequential path when the server ignores Range,
  the resource is empty (416), or the file is too small to split
- 206 responses are validated (Content-Range start/total, no chunked
  framing) so a misbehaving server cannot corrupt the output
- progress callbacks stay monotonic across worker threads

All existing API is unchanged; default config (maxConnectionsPerFile=1)
behaves exactly like before.
…ngo.org

New ParallelDownloadTest suite (7 cases) against httpbingo.org, which
honors Range on /range/N and serves deterministic content:
- segmented result byte-identical to sequential download
- maxSegments decoupled from connection count
- monotonic parallel progress
- fallback when the server ignores Range (200)
- small-file fallback to the sequential path
- redirect resolution before segmenting
- cancellation aborts all workers

Also migrate the existing httpbin.org tests to httpbingo.org (same
API, more reliable); httpbin.org frequently serves 503s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant