Fix multipart download response metadata for presigned URL and normal paths by jencymaryjoseph · Pull Request #7077 · aws/aws-sdk-java-v2

jencymaryjoseph · 2026-06-25T17:50:19Z

Motivation and Context

When the S3 multipart async client downloads a large object via a presigned URL (using ranged GETs for each part), the response metadata exposed to the customer reflects only the first part — not the full object. Customers see an incorrect contentLength (part size instead of total), a partial contentRange, and meaningless per-part composite checksum values.
This fix rewrites the first part's response into a full-object response before it reaches the customer, across both presigned download paths (parallel toFile and serial toBytes/custom transformers).

Modifications

The fix has two prongs, because the two download architectures hand off the response in different places:

Parallel path (toFile) — a subscriber manages all parts concurrently and owns the result future. We rewrite the response just before completing the future.
ParallelPresignedUrlMultipartDownloaderSubscriber calls toFullObjectResponse(...) before resultFuture.complete(...).
Serial path (toBytes, custom transformers) — parts flow one at a time through the splitting infrastructure, which calls the customer's onResponse() with the first part's response. We inject a responseMapper so the rewrite happens at the onResponse() delivery point, before the customer sees it.
PresignedUrlDownloadHelper uses splitWithResponseRewrite(...).

Both prongs use the same MultipartDownloadUtils.toFullObjectResponse() to do the rewrite:

contentLength → total object size (parsed from Content-Range)
contentRange → bytes 0-(total-1)/total
all checksum value fields → null when checksumType is COMPOSITE (composite checksums are per-part hashes, not valid for the whole object)
all other fields (etag, versionId, etc.) are preserved via toBuilder()

Response-mapper injection (sdk-core, shared infrastructure). Rather than adding a new split method, responseMapper is carried on the existing public SplittingTransformerConfiguration. The default split(config) reads it and threads it into SplittingTransformer / ByteArraySplittingTransformer, which apply it at their onResponse boundary (defaulting to identity when unset). This keeps the public API surface to a single optional config setter — no new method, no instanceof branching — and every transformer (including FileAsyncResponseTransformer) is handled through its own existing split(config).

Presigned 416 fix. Broadened the empty-object fallback catch to also match a raw S3Exception with status 416. The serial path surfaces the raw exception directly (unwrapped), so the original catch on EmptyObjectRangeNotSatisfiableException alone never matched for custom transformers, skipping the fallback. As a follow-up, EmptyObjectRangeNotSatisfiableException will be removed entirely and both paths unified on isRangeNotSatisfiable().

Transfer Manager progress. With the mapper on the config, splitWithResponseRewrite is a normal transformer.split(config) call, so the TM's progress wrapper's own split() runs and counts bytes correctly. No special routing is needed in GenericS3TransferManager.

Testing

Unit tests

MultipartDownloadUtilsTest — toFullObjectResponse(): content-length/range rewrite, checksum nulling for COMPOSITE, preservation for FULL_OBJECT, no-op when Content-Range is absent.
SplittingTransformerConfigurationTest — config carries the response mapper.

WireMock tests

PresignedUrlMultipartDownloadResponseMetadataWiremockTest — toFile and toBytes see full-object metadata.
PresignedUrlMultipartDownloaderSubscriberWiremockTest — 416 fallback works for custom serial transformers (fails without the fix).
S3TransferManagerPresignedUrlListenerWiremockTest — transferInitiated / bytesTransferred / transferFailed fire correctly for presigned downloads across multipart/non-multipart × toFile/toBytes × ranged/non-ranged.

Integration tests

AsyncPresignedUrlExtensionTestSuite — presigned toBytes/toFile metadata assertions, MPU-with-checksum (COMPOSITE) handling.

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

zoewangg · 2026-06-25T19:33:50Z

+    }
+
+    @Test
+    void multipartDownload_checksumModeEnabled_hasCorrectFullObjectMetadata() throws Exception {


Is this test necessary?

zoewangg · 2026-06-25T19:34:29Z

+    }
+
+    @Test
+    void multipartDownload_toBytes_smallObject_hasCorrectFullObjectMetadata() throws Exception {


Can we consolidate this with multipartDownload_toFile_hasCorrectFullObjectMetadata using parameterized tests?

zoewangg · 2026-06-25T19:35:26Z

    }

+    @Test
+    void getObject_withRangeRequest_preservesPartialMetadata() throws Exception {


Same here, let's try to consolidate tests with parameterized tests

zoewangg · 2026-06-25T19:35:38Z

+    }
+
+    @Test
+    void getObject_mpuObjectWithChecksumMode_hasCorrectMetadata() throws Exception {


Same here. How is checksum mode special?

zoewangg · 2026-06-25T19:36:04Z

+    }
+
    // Helper methods
+    private static void uploadMpuObjectWithChecksum() {


Checksum should be enabled by default, any reason we need to upload it with checksum?

If an object is MPU without checksumMode enabled, S3 doesnt return checksum.
If an object is MPU with checksumMode enabled, S3 doesnt returns FULL_OBJECT checksum.
And if uploaded with checksum enabled and with an explicit checksum algorithm like .checksumAlgorithm(ChecksumAlgorithm.CRC32)) S3 returns COMPOSITE checksum.

zoewangg · 2026-06-25T19:37:28Z

+        if (transformer instanceof ByteArrayAsyncResponseTransformer) {
+            return (SplitResult<GetObjectResponse, T>)
+                ((ByteArrayAsyncResponseTransformer<GetObjectResponse>) transformer).split(splitConfig, mapper);
+        }


Any reason we have special logic for ByteArrayAsyncResponseTransformer? ByteArrayAsyncResponseTransformer is an internal API and not supposed to be used across modules

Oh yeah, removed the instanceof and added split(config, mapper) to the AsyncResponseTransformer interface (with ByteArrayAsyncResponseTransformer overriding it). splitWithResponseRewrite() now just calls transformer.split(splitConfig, mapper)

zoewangg · 2026-06-25T19:39:16Z

-                if (cause instanceof EmptyObjectRangeNotSatisfiableException) {
+                // Parallel path wraps it as EmptyObjectRangeNotSatisfiableException;
+                // serial path (toBytes, custom transformers) surfaces raw S3Exception.
+                if (cause instanceof EmptyObjectRangeNotSatisfiableException


Question: what is EmptyObjectRangeNotSatisfiableException?

EmptyObjectRangeNotSatisfiableException is an internal exception created by the parallel subscriber when it gets a 416 from S3 on a ranged request to an empty object. The serial path doesnt go through the subscriber, so the raw 416 S3Exception arrives without being wrapped. Planning to remove this exception class as a follow up and just use isRangeNotSatisfiable() for all paths.

zoewangg · 2026-06-25T19:41:52Z

+             UnaryOperator.identity());
+    }
+
+    private SplittingTransformer(AsyncResponseTransformer<ResponseT, ResultT> upstreamResponseTransformer,


Can we update this ctor to take a Builder parameter? That way, we don't need to create a new ctor.

…parameterized tests

… wrapper

jencymaryjoseph · 2026-06-26T18:01:13Z

-                              ? progressUpdater.wrapForNonSerialFileDownload(
-                                  responseTransformer, GetObjectRequest.builder().build())
-                              : progressUpdater.wrapResponseTransformer(responseTransformer);
+        if (isS3ClientMultipartEnabled()


Fixes test failure for bytesTransferred not firing for presigned toBytes multipart downloads.
That path was routed to wrapForNonSerialFileDownload, which only counts bytes inside its split() override, but the serial download splits and drives onStream directly, bypassing it. Now routed by parallelSplitSupported() so serial toBytes uses wrapResponseTransformerForMultipartDownload (counts in onStream), mirroring the regular download path

zoewangg · 2026-06-26T17:46:22Z

+     * Creates a {@link SplitResult} with a response mapper applied at the upstream {@code onResponse} delivery point.
+     */
+    @SdkInternalApi
+    default SplitResult<ResponseT, ResultT> split(SplittingTransformerConfiguration splitConfig,


IMO all public methods in a public API class are inherently public APIs, so we can't really add SdkInternalApi. Should we consider folding responseMapper into SplittingTransformerConfiguration. That way, we don't have to introduce another method

moved responseMapper onto SplittingTransformerConfiguration and removed the extra split method

zoewangg · 2026-06-26T17:50:18Z

+        this(upstreamResponseTransformer, resultFuture, UnaryOperator.identity());
+    }
+
+    public ByteArraySplittingTransformer(AsyncResponseTransformer<ResponseT, ResponseBytes<ResponseT>>


Why do we need to new ctor? can we just add a new parameter?

Cleaned up, the mapper now comes from the config rather than a separate split(config, mapper),

zoewangg · 2026-06-26T18:07:02Z

-                              : progressUpdater.wrapResponseTransformer(responseTransformer);
+        if (isS3ClientMultipartEnabled()
+            && presignedDownloadRequest.presignedUrlDownloadRequest().range() == null) {
+            if (responseTransformer.split(b -> b.bufferSizeInBytes(1L)).parallelSplitSupported()) {


I'm a bit concerned that invoking responseTransformer.split may have implications, for example, involving a service call (they are harmless in ou implementations today, but we can't guarantee future implementations or custom implementations).

Is there another way?

good point, removed. With the mapper on the config, the wrapper's own split() handles both the rewrite and byte counting.

zoewangg · 2026-06-26T18:08:26Z


    private final Map<Integer, ByteBuffer> buffers;

+    private final UnaryOperator<ResponseT> responseMapper;


Question: don't we need to update FileAsyncResponseTransfomer as well?

No, file's existing split(config) now reads the mapper from the config like every other transformer, so it's handled automatically

zoewangg · 2026-06-26T18:10:08Z

+     * @return full-object response with total content-length, full content-range,
+     *         and checksum values nulled if checksum type is COMPOSITE
+     */
+    public static GetObjectResponse toFullObjectResponse(GetObjectResponse firstPartResponse) {


Should we include other fields such as etag, version ID etc if they are present?

firstPartResponse.toBuilder() copies all fields. the function only overrides contentLength/contentRange and for COMPOSITE checksums nulls checksums.

… split wiring

jencymaryjoseph added 3 commits June 25, 2026 10:19

Add response rewrite infrastructure for multipart download metadata

c3252d8

Fix presigned URL multipart download response metadata and 416 fallback

2397de3

Fix normal multipart download response metadata

c24d9db

jencymaryjoseph requested a review from a team as a code owner June 25, 2026 17:50

zoewangg reviewed Jun 25, 2026

View reviewed changes

jencymaryjoseph added 3 commits June 26, 2026 09:29

Consolidate normal path integration tests with parameterized tests

587e5c6

Address review: Builder ctor, split(config,mapper) interface method, …

d9bc3d6

…parameterized tests

Fix transfer manager presigned serial wrapper to use correct progress…

9cf092f

… wrapper

jencymaryjoseph requested a review from zoewangg June 26, 2026 17:04

jencymaryjoseph commented Jun 26, 2026

View reviewed changes

zoewangg reviewed Jun 26, 2026

View reviewed changes

Move responseMapper to SplittingTransformerConfiguration and simplify…

e628e58

… split wiring


		private final Map<Integer, ByteBuffer> buffers;

		private final UnaryOperator<ResponseT> responseMapper;

Uh oh!

Conversation

jencymaryjoseph commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jencymaryjoseph commented Jun 25, 2026 •

edited

Loading