Skip to content

GH-4879: Keep async acks for resubmitted records - #4880

Merged
sobychacko merged 4 commits into
spring-projects:mainfrom
sharanggupta:GH-4879
Nov 12, 2026
Merged

sobychacko merged 4 commits into
spring-projects:mainfrom
sharanggupta:GH-4879

Conversation

@sharanggupta

@sharanggupta sharanggupta commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes GH-4879

When the error handler retains records, doPoll() resubmits them and captureOffsets() replaces the offset lists of the batch, so a pending async ack for another partition of the same poll is dropped as an obsolete batch. Resubmitted records are still tracked from their original poll, so captureOffsets() is skipped for them. The test runs against an embedded broker.

@sobychacko

Copy link
Copy Markdown
Contributor

Thanks @sharanggupta, the fix for record listeners is right.

We found one problem with batch listeners. With seekAfterError(false), FailedBatchProcessor commits the records before the failed index directly through the consumer, without going through the acks. Skipping captureOffsets() on the resubmit left those partitions in offsetsInThisBatch, so once the retried batch was acknowledged the consumer stayed paused for async acks and never resumed.

Since we're close to the release, we pushed a follow-up commit that keeps capturing the offsets when records are resubmitted to a batch listener, with a regression test in AsyncAckResubmittedRecordsTests.

We'll squash the commits when merging.

sharanggupta and others added 4 commits November 12, 2026 13:27
When the error handler retains records (`seekAfterError=false`), `doPoll()`
resubmits them and `captureOffsets()` replaces the offset lists of the batch.
A pending async ack for a record of another partition from the same poll then
fails the identity check and is ignored as an obsolete batch, so its offset
is never committed.

Skip `captureOffsets()` for resubmitted records; their offsets are still
tracked from the original poll.

Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>
Replace the mocked consumer with `@EmbeddedKafka`. The first record
delivered keeps its acknowledgment pending whatever its partition, and the
other partition fails once, so the test does not depend on the order of the
partitions in the poll.

Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>
Capture the offsets again when records are resubmitted to a batch listener.
The batch error handler commits the records before the failed one without
acking them, so keeping the original offsets left those partitions pending
and the consumer stayed paused for async acks.
Add a batch listener test.

Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
@sobychacko
sobychacko merged commit 960d757 into spring-projects:main Nov 12, 2026
3 checks passed
spring-builds pushed a commit that referenced this pull request Nov 12, 2026
Fixes: #4879

When the error handler retains records (`seekAfterError=false`), `doPoll()`
resubmits them and `captureOffsets()` replaces the offset lists of the batch.
A pending async ack for a record of another partition from the same poll then
fails the identity check and is ignored as an obsolete batch, so its offset
is never committed.

Skip `captureOffsets()` for resubmitted records; their offsets are still
tracked from the original poll.

* Test the resubmitted records against an embedded broker

Replace the mocked consumer with `@EmbeddedKafka`. The first record
delivered keeps its acknowledgment pending whatever its partition, and the
other partition fails once, so the test does not depend on the order of the
partitions in the poll.

Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>

* Capture offsets again for resubmitted batches

Capture the offsets again when records are resubmitted to a batch listener.
The batch error handler commits the records before the failed one without
acking them, so keeping the original offsets left those partitions pending
and the consumer stayed paused for async acks.
Add a batch listener test.

Co-authored-by: Soby Chacko <soby.chacko@broadcom.com>
(cherry picked from commit 960d757)
sobychacko added a commit that referenced this pull request Nov 12, 2026
Fixes: #4879

 **Auto-cherry-pick to `4.1.x` & `4.0.x`**

When the error handler retains records (`seekAfterError=false`), `doPoll()`
resubmits them and `captureOffsets()` replaces the offset lists of the batch.
A pending async ack for a record of another partition from the same poll then
fails the identity check and is ignored as an obsolete batch, so its offset
is never committed.

Skip `captureOffsets()` for resubmitted records; their offsets are still
tracked from the original poll.

* Test the resubmitted records against an embedded broker

Replace the mocked consumer with `@EmbeddedKafka`. The first record
delivered keeps its acknowledgment pending whatever its partition, and the
other partition fails once, so the test does not depend on the order of the
partitions in the poll.

Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>

* Capture offsets again for resubmitted batches

Capture the offsets again when records are resubmitted to a batch listener.
The batch error handler commits the records before the failed one without
acking them, so keeping the original offsets left those partitions pending
and the consumer stayed paused for async acks.
Add a batch listener test.

Co-authored-by: Soby Chacko <soby.chacko@broadcom.com>
(cherry picked from commit 960d757)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Async acks: pending ack is dropped when the error handler resubmits retained records

2 participants