fix: report download failure when resume give-up occurs mid-stream#1988
Closed
TheYoctoJester wants to merge 1 commit into
Closed
fix: report download failure when resume give-up occurs mid-stream#1988TheYoctoJester wants to merge 1 commit into
TheYoctoJester wants to merge 1 commit into
Conversation
When the network drops during artifact download and the HTTP download resumer exhausts its retry backoff, DownloadResumerClient::CallUserHandler delivered the resulting DownloadResumerError only to the user body handler. For streaming downloads -- where the update state machine consumes the artifact through a DownloadResumerAsyncReader -- that handler intentionally ignores errors and relies on them surfacing through the reader instead. The reader's pending read was never completed, so the artifact read blocked forever: the deployment was never marked failed, no status was reported, and the daemon never returned to the poll loop until it was restarted. On a terminal error, also complete the body reader's pending read with the error, in addition to invoking the user body handler. The buffered SetBodyWriter path is unaffected: there the pending read belongs to an internal AsyncCopy that only logs the error, and the user body handler still delivers the terminal status. Add a regression test that streams a download through MakeBodyAsyncReader, takes the server down until the backoff is exhausted, and asserts the reader receives the DownloadResumerError instead of hanging. Changelog: Fixed mender-update hanging indefinitely (never reporting a deployment failure and never returning to the poll loop) when a network outage during artifact download lasted long enough for the download resumer to give up on resuming. Signed-off-by: Josef Holzmayr <jester@theyoctojester.info> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vpodzime
self-requested a review
July 14, 2026 15:16
TheYoctoJester
marked this pull request as ready for review
July 15, 2026 07:31
|
Merging these commits will result in the following changelog entries: Changelogsmender (fix/resumer-giveup-hangs-async-reader)New changes in mender since master: Bug Fixes
|
|
@TheYoctoJester, start a full client pipeline with:
my commands and optionsYou can prevent me from automatically starting CI pipelines:
You can trigger a client pipeline on multiple prs with:
You can trigger a client pipeline for a specific Mender Client release with:
You can trigger GitHub->GitLab branch sync with:
You can print PR statistics for a repository with:
You can deploy a review app with:
You can run e2e tests against a deployed review app with:
You can cherry pick to a given branch or branches with:
|
Contributor
|
Slightly reworked in #1992. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes an indefinite hang in
mender-updatewhen a network outage duringartifact download lasts long enough for the HTTP download resumer to give up.
The daemon stays
active (running)but wedged: it never reports the deploymentas
Failureand never returns to the poll loop until it is restarted.Reported on the Mender Hub:
https://hub.mender.io/t/bug-mender-update-hangs-after-download-resume-gives-up-never-reports-failure-never-returns-to-poll-loop-5-0-3/8299
(reproduced on 5.0.2–5.0.5; the affected code is identical on
master).Root cause
The artifact body is consumed as a stream through
DownloadResumerAsyncReader.When the resumer exhausts its retry backoff it logs "Giving up on resuming the
download" and delivers the
DownloadResumerErrorviaDownloadResumerClient::CallUserHandler, which routes it to the user bodyhandler. But the update state machine's body handler deliberately ignores
errors (
src/mender-update/daemon/states.cpp), expecting them to surfacethrough the reader instead. The reader's pending read (
last_read_.handler)is never completed, so the artifact read blocks forever — the state machine
never posts
Failure, and (because the daemon's signal handler keeps theio_contextbusy) the recursive event-loop read never returns.The existing tests only exercised give-up with the buffered
SetBodyWritermodel, where the body handler does receive the error — so the gap in the
streaming (reader) path went uncovered.
Fix
In
CallUserHandler, on any terminal error also complete the body reader'spending read with the error, in addition to invoking the user body handler.
The buffered
SetBodyWriterpath is unaffected: there the pending read belongsto an internal
AsyncCopythat only logs the error, and the user body handlerstill delivers the terminal status.
Testing
DownloadResumerTest.ServerDownWhileReadingBodyReportsErrorstreams a download through
MakeBodyAsyncReader, takes the server down untilthe backoff is exhausted, and asserts the reader receives a
DownloadResumerError. It times out (hangs) without the fix and passes withit; the full
http_resumersuite stays green (37/37).qemux86-64image (mender 5.0.3) against Hosted Mender:cutting the network mid-download now produces
Giving up on resuming ... → DownloadCancel → DownloadError scripts → ArtifactCleanup → Pushing deployment status: failure, and the client returnsto the poll loop instead of wedging.
🤖 Generated with Claude Code