Retry transient download failures instead of panicking - #40
Merged
Merged
Conversation
A single stalled connection used to abort an entire mirror run because download() unwrapped its reqwest error at every call site. Retry each download a few times with exponential backoff (--retries, default 3) and return a proper error once the attempts are exhausted, so a full sync no longer has to be wrapped in an external retry loop. Also make the downloader more robust: - Check the HTTP status before writing the body and detect truncated responses, which previously could spin forever on a short read. - Do not retry permanent 4xx responses (except 408/429), which matters for the many optional rustup-init downloads that legitimately 404. - Turn the component checksum assertion into a clean error. - Make main() return Result so required downloads propagate rather than panic. Fixes #39.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the retry part of #39.
What changed
download()now retries transient failures (timeouts, connection resets, truncated bodies, 5xx/429) with exponential backoff, configurable via--retries(default 3). Once the attempts are exhausted it returns a proper error instead of the previous.unwrap()panic at the call sites, so one stalled connection no longer throws away the in-flight progress of the current channel.Related robustness fixes in the same code path:
rustup-initdownloads that legitimately 404.assert_eq!.main()returnsResult, so required downloads propagate errors instead of panicking.--retries 0disables retries.Re:
[[artifacts.*]]Intentional. Those entries (
rust-*.msi,rust-*.pkg,rustc-*-src.tar.*) are not used by rustup itself, so they are neither mirrored nor rewritten. Leaving the official URLs in place is correct: rewriting them to the mirror would advertise files the mirror does not serve.Testing
cargo build,cargo test(new unit test for the backoff schedule),cargo fmt --check.cargo clippyclean apart from the pre-existing warning atsrc/main.rs:378.