Repository navigation
GH-4879: Keep async acks for resubmitted records - #4880
Merged
Merged
Conversation
Contributor
|
Thanks @sharanggupta, the fix for record listeners is right. We found one problem with batch listeners. With Since we're close to the release, we pushed a follow-up commit that keeps capturing the offsets when records are resubmitted to a batch listener, with a regression test in We'll squash the commits when merging. |
When the error handler retains records (`seekAfterError=false`), `doPoll()` resubmits them and `captureOffsets()` replaces the offset lists of the batch. A pending async ack for a record of another partition from the same poll then fails the identity check and is ignored as an obsolete batch, so its offset is never committed. Skip `captureOffsets()` for resubmitted records; their offsets are still tracked from the original poll. Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>
Replace the mocked consumer with `@EmbeddedKafka`. The first record delivered keeps its acknowledgment pending whatever its partition, and the other partition fails once, so the test does not depend on the order of the partitions in the poll. Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev>
Capture the offsets again when records are resubmitted to a batch listener. The batch error handler commits the records before the failed one without acking them, so keeping the original offsets left those partitions pending and the consumer stayed paused for async acks. Add a batch listener test. Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
Signed-off-by: Soby Chacko <soby.chacko@broadcom.com>
spring-builds
pushed a commit
that referenced
this pull request
Nov 12, 2026
Fixes: #4879 When the error handler retains records (`seekAfterError=false`), `doPoll()` resubmits them and `captureOffsets()` replaces the offset lists of the batch. A pending async ack for a record of another partition from the same poll then fails the identity check and is ignored as an obsolete batch, so its offset is never committed. Skip `captureOffsets()` for resubmitted records; their offsets are still tracked from the original poll. * Test the resubmitted records against an embedded broker Replace the mocked consumer with `@EmbeddedKafka`. The first record delivered keeps its acknowledgment pending whatever its partition, and the other partition fails once, so the test does not depend on the order of the partitions in the poll. Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev> * Capture offsets again for resubmitted batches Capture the offsets again when records are resubmitted to a batch listener. The batch error handler commits the records before the failed one without acking them, so keeping the original offsets left those partitions pending and the consumer stayed paused for async acks. Add a batch listener test. Co-authored-by: Soby Chacko <soby.chacko@broadcom.com> (cherry picked from commit 960d757)
sobychacko
added a commit
that referenced
this pull request
Nov 12, 2026
Fixes: #4879 **Auto-cherry-pick to `4.1.x` & `4.0.x`** When the error handler retains records (`seekAfterError=false`), `doPoll()` resubmits them and `captureOffsets()` replaces the offset lists of the batch. A pending async ack for a record of another partition from the same poll then fails the identity check and is ignored as an obsolete batch, so its offset is never committed. Skip `captureOffsets()` for resubmitted records; their offsets are still tracked from the original poll. * Test the resubmitted records against an embedded broker Replace the mocked consumer with `@EmbeddedKafka`. The first record delivered keeps its acknowledgment pending whatever its partition, and the other partition fails once, so the test does not depend on the order of the partitions in the poll. Signed-off-by: Sharang Gupta <sharang@sharanggupta.dev> * Capture offsets again for resubmitted batches Capture the offsets again when records are resubmitted to a batch listener. The batch error handler commits the records before the failed one without acking them, so keeping the original offsets left those partitions pending and the consumer stayed paused for async acks. Add a batch listener test. Co-authored-by: Soby Chacko <soby.chacko@broadcom.com> (cherry picked from commit 960d757)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes GH-4879
When the error handler retains records,
doPoll()resubmits them andcaptureOffsets()replaces the offset lists of the batch, so a pending async ack for another partition of the same poll is dropped as an obsolete batch. Resubmitted records are still tracked from their original poll, socaptureOffsets()is skipped for them. The test runs against an embedded broker.