Skip to content

[Cherry 5.1.x]: Fix mender-update hanging when long-term network issues happen during Artifact download #1996

Merged
vpodzime merged 8 commits into
5.1.xfrom
cherry-5.1.x-master-resumer-giveup-hangs-async-reader
Jul 24, 2026
Merged

[Cherry 5.1.x]: Fix mender-update hanging when long-term network issues happen during Artifact download #1996
vpodzime merged 8 commits into
5.1.xfrom
cherry-5.1.x-master-resumer-giveup-hangs-async-reader

Conversation

@mender-test-bot

Copy link
Copy Markdown

Cherry pick of PR: #1992
For you :)

vpodzime and others added 5 commits July 24, 2026 08:27
…erAsyncReader

The last_read_ struct records information about the last call of
AsyncRead() which is a DownloadResumerAsyncReader's function. It
makes more sense to record this information inside that class
then. The strange `friend` declaration allowing a
DownloadResumerAsyncReader instance contained inside
DownloadResumerClient to access DownloadResumerClient's private
members can now be dropped and the hierarchy makes more sense.

Ticket: MEN-9954
Changelog: None
Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>
(cherry picked from commit f493b88)
…stream

When there are network issues during Artifact download and the
HTTP download resumer exhausts its retry backoff, the only way to
deliver the error to the state machine is through the HTTP **body
reader's** handler because the **body handler** passed to the
DownloadResumerClient is only called when all body data is
fetched which normally happens in a completely different state
than the one that starts the download and thus the
UpdateDownloadState doesn't handle any errors in it.

Add a Fail() function to the body reader and use it to make sure
it calls its handler with the respective error.

Also add a comment about how the DownloadResumerClient is used.

Ticket: MEN-9954
Changelog: Fixed mender-update hanging indefinitely (never reporting a
deployment failure and never returning to the poll loop) when a network
outage during artifact download lasted long enough for the download
resumer to give up on resuming.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>
(cherry picked from commit a9263bd)
Unlike the basic http::Client which constructs the reader and
returns it, leaving the API user code to manage the reader's
lifetime, the DownloadResumerClient needs to keep a reference
(shared pointer) to the reader too and thus manage its
lifetime. The reason for this is that while the http::Client's
reader (and its lifetime) is bound to a socket that gets closed
on an error or completion, only the DownloadResumerClient itself
knows when the body reader (providing data from multiple HTTP
requests/responses) won't produce more data, i.e. when no more
(re)tries will be performed.

Tests that were exercising the reader ownership by the API user
code were adjusted to test cancellation of the reader instead.

Ticket: MEN-9954
Changelog: None
Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>
(cherry picked from commit c722497)
…omAsyncReader::Read()

The handler passed to the asynchronous reader can be called with
a different stack (frame) than that of the
ReaderFromAsyncReader::Read() function and then those references
(pointers) lead to random unrelated places. We need to use shared
pointers to make sure the referenced data is valid independently
on the stack.

Ticket: MEN-9954
Changelog: None
Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>
(cherry picked from commit 877de62)
…:Read()

The handler passed to the asynchronous reader can be called
multiple times after adding a call to `AsyncRead()` to the event
loop, not only once when there is data available, but also if the
asynchronous reader encounters an error. Since the event loop is
only run recursively **once** per ReaderFromAsyncReader::Read()
call, we need to make sure it's also stopped only once. Otherwise
it may stop the event loop for good.

Ticket: MEN-9954
Changelog: None
Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>
(cherry picked from commit 4c2c524)
@mender-test-bot

Copy link
Copy Markdown
Author

@mender-test-bot, start a full client pipeline with:

  • mentioning me and start client pipeline

my commands and options

You can prevent me from automatically starting CI pipelines:

  • if your pull request title starts with "[NoCI] ..."

You can trigger a client pipeline on multiple prs with:

  • mentioning me and start client pipeline --pr mender/127 --pr mender-connect/255

You can trigger a client pipeline for a specific Mender Client release with:

  • mentioning me and start client pipeline --release 6.0.x (can be given multiple times)
  • by default, a pipeline is triggered for each supported release the component is a part of

You can trigger GitHub->GitLab branch sync with:

  • mentioning me and sync

You can print PR statistics for a repository with:

  • mentioning me and print fast pr stats (Team stats only)
  • mentioning me and print full pr stats (Detailed report)
  • options: --repo <repo>, --team <name>, --all-repos, --exclude-drafts, --exclude-user <user>
  • mentioning me and print full pr stats --repo mender --all-repos --exclude-drafts

You can deploy a review app with:

  • mentioning me and start review app (OS environment)
  • mentioning me and start review app enterprise (Enterprise environment)

You can run e2e tests against a deployed review app with:

  • mentioning me and start review tests (defaults to os environment)
  • mentioning me and start review tests enterprise (for enterprise environment)

You can cherry pick to a given branch or branches with:

  • mentioning me and:
 cherry-pick to:
 * 1.0.x
 * 2.0.x

Lluis Campos and others added 3 commits July 24, 2026 10:57
Ticket: QA-1517

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Lluis Campos <lluis.campos@northern.tech>
(cherry picked from commit bc42303)
Ticket: QA-1517

Signed-off-by: Lluis Campos <lluis.campos@northern.tech>
(cherry picked from commit 151d64a)
To remove custom extensions of conventional commits standard.

Ticket: QA-1517

Signed-off-by: Lluis Campos <lluis.campos@northern.tech>
(cherry picked from commit d6987de)
@mender-test-bot

mender-test-bot commented Jul 24, 2026

Copy link
Copy Markdown
Author

Merging these commits will result in the following changelog entries:

Changelogs

mender (cherry-5.1.x-master-resumer-giveup-hangs-async-reader)

New changes in mender since 5.1.x:

Bug Fixes
  • Fixed mender-update hanging indefinitely (never reporting a
    deployment failure and never returning to the poll loop) when a network
    outage during artifact download lasted long enough for the download
    resumer to give up on resuming.
    (MEN-9954)

@vpodzime

Copy link
Copy Markdown
Contributor

ci/mender-qa Expected — Waiting for status to be reported

tested on master and the cherry-pick was clean, not running this through the full pipeline

@vpodzime
vpodzime merged commit debdcc4 into 5.1.x Jul 24, 2026
1 check passed
@vpodzime
vpodzime deleted the cherry-5.1.x-master-resumer-giveup-hangs-async-reader branch July 24, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants