Fix mender-update hanging when long-term network issues happen during Artifact download by vpodzime · Pull Request #1992 · mendersoftware/mender

vpodzime · 2026-07-16T12:37:46Z

No description provided.

mender-test-bot · 2026-07-16T12:38:12Z

@vpodzime, start a full client pipeline with:

mentioning me and start client pipeline

my commands and options

You can prevent me from automatically starting CI pipelines:

if your pull request title starts with "[NoCI] ..."

You can trigger a client pipeline on multiple prs with:

mentioning me and start client pipeline --pr mender/127 --pr mender-connect/255

You can trigger a client pipeline for a specific Mender Client release with:

mentioning me and start client pipeline --release 6.0.x (can be given multiple times)
by default, a pipeline is triggered for each supported release the component is a part of

You can trigger GitHub->GitLab branch sync with:

mentioning me and sync

You can print PR statistics for a repository with:

mentioning me and print fast pr stats (Team stats only)
mentioning me and print full pr stats (Detailed report)
options: --repo <repo>, --team <name>, --all-repos, --exclude-drafts, --exclude-user <user>
mentioning me and print full pr stats --repo mender --all-repos --exclude-drafts

You can deploy a review app with:

mentioning me and start review app (OS environment)
mentioning me and start review app enterprise (Enterprise environment)

You can run e2e tests against a deployed review app with:

mentioning me and start review tests (defaults to os environment)
mentioning me and start review tests enterprise (for enterprise environment)

You can cherry pick to a given branch or branches with:

mentioning me and:

 cherry-pick to:
 * 1.0.x
 * 2.0.x

vpodzime · 2026-07-16T12:38:43Z

@lluiscampos I'd appreciate your 👀 on this as well if you have a minute. 🍻

…erAsyncReader The last_read_ struct records information about the last call of AsyncRead() which is a DownloadResumerAsyncReader's function. It makes more sense to record this information inside that class then. The strange `friend` declaration allowing a DownloadResumerAsyncReader instance contained inside DownloadResumerClient to access DownloadResumerClient's private members can now be dropped and the hierarchy makes more sense. Ticket: MEN-9954 Changelog: None Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>

mender-test-bot · 2026-07-16T12:45:17Z

Hello 😺 I created a pipeline for you here: Pipeline-2681806525

Build Configuration Matrix

Key	Value
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
INTEGRATION_REV	master
MENDER_BINARY_DELTA_REV	master
MENDER_CLIENT_SUBCOMPONENTS_REV	main
MENDER_CONFIGURE_MODULE_REV	master
MENDER_CONNECT_REV	master
MENDER_CONTAINER_MODULES_REV	main
MENDER_FLASH_REV	master
MENDER_REV	pull/1992/head
MONITOR_CLIENT_REV	master
RUN_INTEGRATION_TESTS	true
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true

…stream When there are network issues during Artifact download and the HTTP download resumer exhausts its retry backoff, the only way to deliver the error to the state machine is through the HTTP **body reader's** handler because the **body handler** passed to the DownloadResumerClient is only called when all body data is fetched which normally happens in a completely different state than the one that starts the download and thus the UpdateDownloadState doesn't handle any errors in it. Add a Fail() function to the body reader and use it to make sure it calls its handler with the respective error. Also add a comment about how the DownloadResumerClient is used. Ticket: MEN-9954 Changelog: Fixed mender-update hanging indefinitely (never reporting a deployment failure and never returning to the poll loop) when a network outage during artifact download lasted long enough for the download resumer to give up on resuming. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>

Unlike the basic http::Client which constructs the reader and returns it, leaving the API user code to manage the reader's lifetime, the DownloadResumerClient needs to keep a reference (shared pointer) to the reader too and thus manage its lifetime. The reason for this is that while the http::Client's reader (and its lifetime) is bound to a socket that gets closed on an error or completion, only the DownloadResumerClient itself knows when the body reader (providing data from multiple HTTP requests/responses) won't produce more data, i.e. when no more (re)tries will be performed. Tests that were exercising the reader ownership by the API user code were adjusted to test cancellation of the reader instead. Ticket: MEN-9954 Changelog: None Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>

…omAsyncReader::Read() The handler passed to the asynchronous reader can be called with a different stack (frame) than that of the ReaderFromAsyncReader::Read() function and then those references (pointers) lead to random unrelated places. We need to use shared pointers to make sure the referenced data is valid independently on the stack. Ticket: MEN-9954 Changelog: None Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>

…:Read() The handler passed to the asynchronous reader can be called multiple times after adding a call to `AsyncRead()` to the event loop, not only once when there is data available, but also if the asynchronous reader encounters an error. Since the event loop is only run recursively **once** per ReaderFromAsyncReader::Read() call, we need to make sure it's also stopped only once. Otherwise it may stop the event loop for good. Ticket: MEN-9954 Changelog: None Signed-off-by: Vratislav Podzimek <vratislav.podzimek+auto-signed@northern.tech>

mender-test-bot · 2026-07-21T13:57:58Z

Merging these commits will result in the following changelog entries:

Changelogs

mender (master-resumer-giveup-hangs-async-reader)

New changes in mender since master:

Bug Fixes

Fixed mender-update hanging indefinitely (never reporting a
deployment failure and never returning to the poll loop) when a network
outage during artifact download lasted long enough for the download
resumer to give up on resuming.
(MEN-9954)

codecov · 2026-07-21T14:10:36Z

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/common/events/events_io.cpp	69.23%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

vpodzime · 2026-07-21T14:12:48Z

@mender-test-bot start client pipeline

mender-test-bot · 2026-07-21T14:13:10Z

Hello 😺 I created a pipeline for you here: Pipeline-2694097193

Build Configuration Matrix

Key	Value
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
INTEGRATION_REV	master
MENDER_BINARY_DELTA_REV	master
MENDER_CLIENT_SUBCOMPONENTS_REV	main
MENDER_CONFIGURE_MODULE_REV	master
MENDER_CONNECT_REV	master
MENDER_CONTAINER_MODULES_REV	main
MENDER_FLASH_REV	master
MENDER_REV	pull/1992/head
MONITOR_CLIENT_REV	master
RUN_INTEGRATION_TESTS	true
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true

vpodzime · 2026-07-21T14:30:30Z

4 lines in your changes missing coverage.

Btw, these are 3 lines that used to miss coverage that our now 4 lines missing coverage with 1 extra line and 3 lines with minimal changes.

michalkopczan · 2026-07-22T09:08:18Z

 		logger_.Warning("DownloadResumerClient destroyed while request is still active!");
 	}
 	client_.Cancel();
+	resumer_reader_.reset();


Why do you reset a shared_ptr here?If no one is using it, it will be destroyed here automatically. And if someone else is using it, this reset will do nothing from their perspective, their shared_ptr will still point to resumer_reader_.

michalkopczan · 2026-07-22T09:15:35Z

 		DoCancel();
 		user_body_handler_(exp_resp);
+		// we are done, the body reader won't produce any more data
+		resumer_reader_.reset();


Why do we need all those resumer_reader_.reset() in multiple places? Isn't it called only when we already cancelled our DownloadResumerClient, meaning that it will not use this shared_ptr anymore and will be cleared on DownloadResumerClient destruction?

vpodzime · 2026-07-23T07:30:59Z

@mender-test-bot start client pipeline --pr integration/2951

mender-test-bot · 2026-07-23T07:45:40Z

Hello 😺 I created a pipeline for you here: Pipeline-2699405995

Build Configuration Matrix

Key	Value
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
INTEGRATION_REV	pull/2951/head
MENDER_BINARY_DELTA_REV	master
MENDER_CLIENT_SUBCOMPONENTS_REV	main
MENDER_CONFIGURE_MODULE_REV	master
MENDER_CONNECT_REV	master
MENDER_CONTAINER_MODULES_REV	main
MENDER_FLASH_REV	master
MENDER_REV	pull/1992/head
MONITOR_CLIENT_REV	master
RUN_INTEGRATION_TESTS	true
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true

vpodzime requested review from danielskinstad and michalkopczan July 16, 2026 12:37

vpodzime mentioned this pull request Jul 16, 2026

fix: report download failure when resume give-up occurs mid-stream #1988

Closed

vpodzime force-pushed the master-resumer-giveup-hangs-async-reader branch from 50b488b to d998f81 Compare July 16, 2026 12:44

This comment was marked as outdated.

Sign in to view

vpodzime force-pushed the master-resumer-giveup-hangs-async-reader branch 3 times, most recently from 83ebfd5 to 77fcfae Compare July 21, 2026 12:34

vpodzime and others added 3 commits July 21, 2026 15:41

vpodzime force-pushed the master-resumer-giveup-hangs-async-reader branch 2 times, most recently from f5144b4 to 63faa55 Compare July 21, 2026 13:54

vpodzime force-pushed the master-resumer-giveup-hangs-async-reader branch from 63faa55 to d0a4eb0 Compare July 21, 2026 13:57

vpodzime changed the title ~~Fix mender-update handing when long-term network issues happen during Artifact download~~ Fix mender-update hanging when long-term network issues happen during Artifact download Jul 21, 2026

michalkopczan reviewed Jul 22, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

Uh oh!

Conversation

vpodzime commented Jul 16, 2026

Uh oh!

mender-test-bot commented Jul 16, 2026

Uh oh!

vpodzime commented Jul 16, 2026

Uh oh!

This comment was marked as outdated.

mender-test-bot commented Jul 16, 2026

Uh oh!

mender-test-bot commented Jul 21, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelogs

mender (master-resumer-giveup-hangs-async-reader)

Bug Fixes

Uh oh!

codecov Bot commented Jul 21, 2026

Codecov Report

Uh oh!

vpodzime commented Jul 21, 2026

Uh oh!

mender-test-bot commented Jul 21, 2026

Uh oh!

vpodzime commented Jul 21, 2026

Uh oh!

michalkopczan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

michalkopczan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

vpodzime commented Jul 23, 2026

Uh oh!

mender-test-bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mender-test-bot commented Jul 21, 2026 •

edited by atlassian Bot

Loading