speeding up fetch - #1
Merged
Merged
Conversation
bartman
commented
Sep 4, 2026
bartman
commented
Owner
- Batch and concurrent jday fetches for faster fetch
- Document batched fetch performance and jrange weeks
- Add unit tests for batched concurrent fetch
- Count GraphQL response bytes for transfer stats
- Add fetch --stats and honor --force over cache
- Test fetch --stats formatting and --force cache bypass
- Document fetch --stats output format
Sequential per-date GraphQL requests made `wxrust fetch` take ~19s for 152 workouts. Pack up to 10 jday selections into one aliased query and run 8 requests concurrently (~1.2s for the downloads, ~3.4s end-to-end). Tune the HTTP client (idle pool, TCP_NODELAY, gzip) and skip cached dates by file existence instead of parsing.
Record that jrange range is in weeks (max 32), GraphQL aliases work for bulk jday, and fetch uses 10-wide batches with concurrency 8. Note downloadLogs exists but is unused because JEditorData can diverge from jday cache format.
Cover query builders, cache existence checks, get_jdays_batch success/error/cache paths, concurrent chunk reordering, and fetch_command skip/force/network behavior.
Record request count and response body size on each HTTP GraphQL call so fetch --stats can report MB/s.
Print `X workouts, Y seconds, x.y T/s, x.y MB/s` after the progress line. --force (and --diff) skip cache reads so already-cached dates are downloaded from the server again.
Cover transfer-stats output and assert --force still issues a network jday request when a cache file already exists.
Describe the T/s and MB/s summary line printed after fetch.
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.