Skip to content

fix: report download failure when resume give-up occurs mid-stream#1988

Closed
TheYoctoJester wants to merge 1 commit into
mendersoftware:masterfrom
TheYoctoJester:fix/resumer-giveup-hangs-async-reader
Closed

fix: report download failure when resume give-up occurs mid-stream#1988
TheYoctoJester wants to merge 1 commit into
mendersoftware:masterfrom
TheYoctoJester:fix/resumer-giveup-hangs-async-reader

Conversation

@TheYoctoJester

Copy link
Copy Markdown
Contributor

Summary

Fixes an indefinite hang in mender-update when a network outage during
artifact download lasts long enough for the HTTP download resumer to give up.
The daemon stays active (running) but wedged: it never reports the deployment
as Failure and never returns to the poll loop until it is restarted.

Reported on the Mender Hub:
https://hub.mender.io/t/bug-mender-update-hangs-after-download-resume-gives-up-never-reports-failure-never-returns-to-poll-loop-5-0-3/8299
(reproduced on 5.0.2–5.0.5; the affected code is identical on master).

Root cause

The artifact body is consumed as a stream through DownloadResumerAsyncReader.
When the resumer exhausts its retry backoff it logs "Giving up on resuming the
download"
and delivers the DownloadResumerError via
DownloadResumerClient::CallUserHandler, which routes it to the user body
handler
. But the update state machine's body handler deliberately ignores
errors (src/mender-update/daemon/states.cpp), expecting them to surface
through the reader instead. The reader's pending read (last_read_.handler)
is never completed, so the artifact read blocks forever — the state machine
never posts Failure, and (because the daemon's signal handler keeps the
io_context busy) the recursive event-loop read never returns.

The existing tests only exercised give-up with the buffered SetBodyWriter
model, where the body handler does receive the error — so the gap in the
streaming (reader) path went uncovered.

Fix

In CallUserHandler, on any terminal error also complete the body reader's
pending read with the error, in addition to invoking the user body handler.
The buffered SetBodyWriter path is unaffected: there the pending read belongs
to an internal AsyncCopy that only logs the error, and the user body handler
still delivers the terminal status.

Testing

  • New unit test DownloadResumerTest.ServerDownWhileReadingBodyReportsError
    streams a download through MakeBodyAsyncReader, takes the server down until
    the backoff is exhausted, and asserts the reader receives a
    DownloadResumerError. It times out (hangs) without the fix and passes with
    it; the full http_resumer suite stays green (37/37).
  • End-to-end on a qemux86-64 image (mender 5.0.3) against Hosted Mender:
    cutting the network mid-download now produces
    Giving up on resuming ... → DownloadCancel → DownloadError scripts → ArtifactCleanup → Pushing deployment status: failure, and the client returns
    to the poll loop instead of wedging.

🤖 Generated with Claude Code

When the network drops during artifact download and the HTTP download
resumer exhausts its retry backoff, DownloadResumerClient::CallUserHandler
delivered the resulting DownloadResumerError only to the user body handler.
For streaming downloads -- where the update state machine consumes the
artifact through a DownloadResumerAsyncReader -- that handler intentionally
ignores errors and relies on them surfacing through the reader instead. The
reader's pending read was never completed, so the artifact read blocked
forever: the deployment was never marked failed, no status was reported,
and the daemon never returned to the poll loop until it was restarted.

On a terminal error, also complete the body reader's pending read with the
error, in addition to invoking the user body handler. The buffered
SetBodyWriter path is unaffected: there the pending read belongs to an
internal AsyncCopy that only logs the error, and the user body handler
still delivers the terminal status.

Add a regression test that streams a download through MakeBodyAsyncReader,
takes the server down until the backoff is exhausted, and asserts the
reader receives the DownloadResumerError instead of hanging.

Changelog: Fixed mender-update hanging indefinitely (never reporting a
deployment failure and never returning to the poll loop) when a network
outage during artifact download lasted long enough for the download
resumer to give up on resuming.

Signed-off-by: Josef Holzmayr <jester@theyoctojester.info>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vpodzime
vpodzime self-requested a review July 14, 2026 15:16
@TheYoctoJester
TheYoctoJester marked this pull request as ready for review July 15, 2026 07:31
@mender-test-bot

Copy link
Copy Markdown

Merging these commits will result in the following changelog entries:

Changelogs

mender (fix/resumer-giveup-hangs-async-reader)

New changes in mender since master:

Bug Fixes
  • Fixed mender-update hanging indefinitely (never reporting a
    deployment failure and never returning to the poll loop) when a network
    outage during artifact download lasted long enough for the download
    resumer to give up on resuming.

@mender-test-bot

Copy link
Copy Markdown

@TheYoctoJester, start a full client pipeline with:

  • mentioning me and start client pipeline

my commands and options

You can prevent me from automatically starting CI pipelines:

  • if your pull request title starts with "[NoCI] ..."

You can trigger a client pipeline on multiple prs with:

  • mentioning me and start client pipeline --pr mender/127 --pr mender-connect/255

You can trigger a client pipeline for a specific Mender Client release with:

  • mentioning me and start client pipeline --release 6.0.x (can be given multiple times)
  • by default, a pipeline is triggered for each supported release the component is a part of

You can trigger GitHub->GitLab branch sync with:

  • mentioning me and sync

You can print PR statistics for a repository with:

  • mentioning me and print fast pr stats (Team stats only)
  • mentioning me and print full pr stats (Detailed report)
  • options: --repo <repo>, --team <name>, --all-repos, --exclude-drafts, --exclude-user <user>
  • mentioning me and print full pr stats --repo mender --all-repos --exclude-drafts

You can deploy a review app with:

  • mentioning me and start review app (OS environment)
  • mentioning me and start review app enterprise (Enterprise environment)

You can run e2e tests against a deployed review app with:

  • mentioning me and start review tests (defaults to os environment)
  • mentioning me and start review tests enterprise (for enterprise environment)

You can cherry pick to a given branch or branches with:

  • mentioning me and:
 cherry-pick to:
 * 1.0.x
 * 2.0.x

@vpodzime

Copy link
Copy Markdown
Contributor

Slightly reworked in #1992.

@vpodzime vpodzime closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants