Support Git-style searchPaths with wildcards in AWS S3 buckets (#2812) by tomy8964 · Pull Request #2958 · spring-cloud/spring-cloud-config

tomy8964 · 2025-07-06T10:38:31Z

Summary

Add full Git-style searchPaths support (placeholders + wildcards) to the AWS S3 backend so that users can migrate existing Git-based configurations without renaming to application.*.

Implements literal, dot-wildcard (.*), single (?) and double (**) wildcard matching
Retains the lookup order .properties → .json → .yml → .yaml when expanding literals or .* patterns
Scans “directory” patterns for nested files
Deduplicates identical keys across multiple patterns

This issue (#2812)
resolves #2812

Changes

Core Logic
- Extended getS3ConfigFileWithSearchPaths(...) to treat {application} and {profile} placeholders identically to the Git backend
- Added support for:
  - Literal + auto-ext: stops at first existing extension
  - Dot-wildcard (.*): expands against supported extensions in priority order
  - Single-character wildcard (?)
  - Multi-level wildcard (**)
  - Directory scan for literal prefixes ending with /
- Ensured seenKeys set to avoid duplicate property sources
Tests
Added new JUnit 5 tests in AwsS3EnvironmentRepositoryTests to cover all scenarios:
- searchPaths_placeholderOnly_shouldResolveExactFile
- searchPaths_wildcardOnly_shouldResolveAllProperties
- searchPaths_placeholderAndWildcard_shouldResolveMatchingKeys
- searchPaths_orderMatters_forPropertySourceOrder
- searchPath_extensionPreserved (dynamic tests for each extension)
- searchPaths_applicationAsDirectory_shouldStillHonorSearchPaths
- multiDocumentYaml_withSearchPaths_shouldNotSplitDocuments
- getLocations_returnsCorrect
- searchPaths_deduplication_shouldOnlyAddOnce
- searchPaths_singleCharacterWildcard_shouldMatchExactlyOneChar
- searchPaths_withEmptyLabel_shouldUseDefaultLabel
- searchPaths_multipleLabels_shouldApplyForEachLabelInReverseOrder
- searchPaths_literalNotFound_shouldReturnEmpty
…plus the additional edge cases for literal-stop, dot-wildcard priority, and nested directory patterns.
Example Usage

spring:
  cloud:
    config:
      server:
        awss3:
          bucket: my-config-bucket
          search-paths:
            - "{label}/{application}"          # literal + auto-ext
            - "{label}/{application}.*"        # dot-wildcard expansion
            - "{label}/common/*.json"          # JSON only in common/
            - "{label}/{application}.yml"      # explicit YML

Additional Notes

Backward Compatibility

This update does not break any existing AWS S3 config setups. The default lookup behavior is unchanged if no placeholders or wildcards are used in searchPaths.

Migration

Existing users migrating from Git-backed config servers can now use their current searchPaths (including wildcards and placeholders) on S3 with no renaming or convention change required.

Documentation

Documentation and usage examples for the enhanced searchPaths will be updated in the relevant documentation files after the merge.
If there are specific locations that require documentation updates, please let me know—I will be happy to update them.

Performance and Cost

Using wildcards (such as * or **) in searchPaths may result in additional AWS S3 API calls (e.g., ListObjects), especially for large buckets or deeply nested directory patterns.
This could lead to increased latency and higher AWS costs.
Users should consider the structure and size of their buckets when designing searchPaths and monitor AWS S3 usage accordingly.

…g-cloud#2812) Fixes spring-cloud#2812 - ListObjectsV2 + AntPathMatcher based matching for *, **, ?, dot-wildcard - auto-ext lookup order (.properties → .json → .yml/.yaml) - directory scan, deduplication, prefix extraction - only active when searchPaths non-empty Signed-off-by: Geonwook Ham <tomy8964@naver.com> Signed-off-by: ham <tomy8964@naver.com>

Signed-off-by: Geonwook Ham <tomy8964@naver.com>

tomy8964 · 2026-05-30T03:26:07Z

Hi @ryanjbaxter!
I have updated the branch to resolve the merge conflicts with the latest main. Could you please take a look when you have some time? Thank you!

Copilot

Pull request overview

This PR extends the AWS S3 environment repository to support Git-style searchPaths (placeholders + wildcard matching), enabling S3-backed config layouts to match existing Git-backed repository conventions.

Changes:

Add searchPaths configuration to the AWS S3 backend and wire it through the factory.
Implement placeholder expansion and wildcard-based S3 key discovery (including directory scans) with deduplication.
Add an extensive JUnit 5 test suite covering placeholder/wildcard resolution scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
`spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepository.java`	Adds `searchPaths` support, wildcard matching via `AntPathMatcher`, S3 list/head probing, and key→property source wrapping.
`spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryFactory.java`	Passes `searchPaths` from properties into the repository constructor.
`spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentProperties.java`	Introduces `searchPaths` as a bindable configuration property.
`spring-cloud-config-server/src/test/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryTests.java`	Adds coverage for placeholder/wildcard behavior, ordering, and special cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Fixes spring-cloud#2812 Rework AwsS3EnvironmentRepository path resolution and profile handling. Reverse the app list before checking/appending the default application and only append the default when it is missing. Simplify property-source iteration and invoke negated-profile property source logic only when searchPaths is empty to avoid duplicate sources. Normalize search path patterns by resolving null label/profile vars, collapsing duplicate slashes and trimming leading '/', and optimize extension probing so existing extensions aren't re-probed. Signed-off-by: Geonwook Ham <tomy8964@naver.com>

…-s3-searchpaths

Fixes spring-cloud#2812 In AwsS3EnvironmentRepositoryTests, avoid mutating a shared 'server' instance by creating a new ConfigServerProperties (serverWithDefaultLabel) and setting its defaultLabel before constructing AwsS3EnvironmentRepository. This prevents side-effects on shared test state and ensures the repository gets the intended default label for the test. Signed-off-by: Geonwook Ham <tomy8964@naver.com>

tomy8964 · 2026-06-02T12:58:41Z

Hi @ryanjbaxter!

I have successfully addressed all the feedback from the Copilot review:

Fixed property source precedence and default application fallback in findOne().
Restored the native Spring Cloud Config profile activation lifecycle (including negated profiles) by refactoring addPropertySources().
Added path normalization (trimming leading slashes and duplicate //) and optimized literal extension probing to avoid redundant S3 calls.
Isolated the test state in AwsS3EnvironmentRepositoryTests by using a fresh ConfigServerProperties instance.

The checks are passing, and it's ready for your review. Could you please take another look when you have some time? Thank you!

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

The for loop header and the first statement are on the same line (for (...) {String resolvedLabel = ...), which is likely to fail formatting checks (and is hard to read). Split the loop body onto its own lines. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Geonwook Ham <tomy8964@naver.com>

Fixes spring-cloud#2812 Generalize getS3ConfigFileWithSearchPaths by adding a Function keyWrapper to handle different key-wrapping strategies (profile-specific and negated-profile cases). Implement wrapKeyWithNegatedConfigFiles to support YAML documents with negated spring.config.activate.on-profile expressions and wire it into property source discovery. Clean up formatting, streamline S3 client calls, and simplify create/wrapper methods. Also remove explicit setOrder in AwsS3EnvironmentRepositoryFactory (return repository directly) and update tests for minor formatting/usage changes. Signed-off-by: Geonwook Ham <tomy8964@naver.com>

tomy8964 · 2026-06-02T14:42:42Z

Hi @ryanjbaxter!

I have successfully addressed the second round of feedback from the Copilot review:

Generalized S3 Search Logic: Refactored getS3ConfigFileWithSearchPaths by introducing a functional Function (keyWrapper) parameter, eliminating duplicate S3 object crawling and pagination logic between profile-specific and negated-profile cases.
Restored Negated Profiles: Implemented wrapKeyWithNegatedConfigFiles to fully support multi-document YAML files with negated profile expressions under the searchPaths architecture.
Formatting & Factory Cleanup: Streamlined the method chains, wrapped arguments to comply with project formatting conventions, and removed the redundant setOrder call in AwsS3EnvironmentRepositoryFactory.
Test Syntax Fixes: Corrected minor YAML formatting issues (missing spaces after colons) within AwsS3EnvironmentRepositoryTests.

The checks are passing again, and the codebase is now much cleaner and more reusable. Could you please take another look when you have some time? Thank you!

ryanjbaxter · 2026-06-03T19:23:36Z


 	public AwsS3EnvironmentRepository(S3Client s3Client, String bucketName, boolean useApplicationAsDirectory,
 			ConfigServerProperties server) {
+		this(s3Client, bucketName, useApplicationAsDirectory, server, null);


Could we pass an empty list here?

Yes, we updated this constructor to pass Collections.emptyList() instead of null.

ryanjbaxter · 2026-06-03T19:24:15Z


 	private int order = DEFAULT_ORDER;

+	private List<String> searchPaths = new ArrayList<>();


This could be Collections.emptyList()

Yes, we initialized searchPaths using Collections.emptyList() to optimize memory usage and guarantee immutability by default.

ryanjbaxter · 2026-06-03T19:25:44Z

+				boolean fileFound = false;
+				List<String> extensionsToProbe = (pattern.endsWith(".properties") || pattern.endsWith(".json")
+						|| pattern.endsWith(".yml") || pattern.endsWith(".yaml")) ? List.of("")
+								: List.of(".properties", ".json", ".yml", ".yaml");


This could be private static final

Yes, we extracted these extension lists into private static final constants (SUPPORTED_EXTENSIONS and EMPTY_EXTENSION) at the class level.

ryanjbaxter · 2026-06-03T19:26:57Z

+
+			if (pattern.endsWith(".*")) {
+				String base = pattern.substring(0, pattern.length() - 2);
+				for (String ext : List.of(".properties", ".json", ".yml", ".yaml")) {


extentions could be reused here

Yes, we replaced it to reuse the static SUPPORTED_EXTENSIONS constant.

ryanjbaxter · 2026-06-03T19:30:14Z

+						}
+
+						String expression = onProfileValue.toString().trim();
+						if (!expression.contains("!") && !expression.contains("&") && !expression.contains("|")


Couldnt you use isSimpleProfileName?

Yes, we refactored isSimpleProfileName to be a package-private static helper method and reused it here to check for simple profile names.

ryanjbaxter · 2026-06-03T19:34:25Z

+			boolean useApplicationAsDirectory, S3Client s3Client, boolean callReadImmediately) {
 		super(application, profile, label, bucketName, useApplicationAsDirectory, s3Client);
-		this.properties = read();
+		if (callReadImmediately) {


Is this necessary? There are subclasses that now call read immediately, so could this implementation just not call read in the constructor? And if so I would prefer to introduce a new class which has the new functionality (not call read immediately) and then leave these as is.

We refactored this entire part. Instead of adding a callReadImmediately flag to the base classes, we consolidated all three key-based config file classes (PropertyConfigFileFromKey, YamlConfigFileFromKey, JsonConfigFileFromKey) into a single S3ConfigFileFromKey class. The base classes remain unchanged (calling read() in the constructor as they originally did), and S3ConfigFileFromKey handles reading appropriately, allowing us to remove the callReadImmediately flag entirely.

ryanjbaxter

Could you also add documentation?

ryanjbaxter · 2026-06-03T19:38:00Z


-		AwsS3EnvironmentRepository repository = new AwsS3EnvironmentRepository(client,
-				environmentProperties.getBucket(), environmentProperties.isUseDirectoryLayout(), server);
-		repository.setOrder(environmentProperties.getOrder());


Why is setOrder no longer called?

We introduced a new constructor in AwsS3EnvironmentRepository that accepts AwsS3EnvironmentProperties directly. Inside this constructor, the order is mapped and set directly from the properties (this.order = properties.getOrder()), making the repository initialization cleaner in the factory.

ryanjbaxter · 2026-06-03T19:42:52Z

 			String label) {
-		List<S3ConfigFile> s3ConfigFiles = getS3ConfigFile(app, profile, label,
-				this::getNonProfileSpecificPropertiesOrJsonConfigFile, this::getNonProfileSpecificS3ConfigFileYaml);
+		List<S3ConfigFile> s3ConfigFiles = searchPaths.isEmpty() ? getS3ConfigFile(app, profile, label,


When searchPaths is set why are nonProfileSpecificPropertiesOrJsonConfigFile not called? Wouldn't this result in application.* files not being included?

When searchPaths is active, the wildcard directory scanning in addProfileSpecificPropertySource automatically discovers both profile-specific and non-profile-specific files (e.g. application.yml and application-profile.yml) matching the patterns. For each YAML file found, it loads both profile-specific and non-profile-specific documents. Skipping it here prevents duplicate S3 API calls and duplicate property sources in the environment.

ryanjbaxter · 2026-06-03T19:44:19Z

+					try {
+						s3Client.headObject(HeadObjectRequest.builder().bucket(bucketName).key(key).build());
+						result.addAll(keyWrapper.apply(key));
+						break;


Wouldn't this stop the for loop after the first file is found and we wouldn't check the rest of the extensions?

Yes, that was a mistake in the previous version. We removed the break statement so that all matching files with different extensions (e.g., both .properties and .yml at the same path) are scanned and loaded, aligning with Git/Native backend behavior.

ryanjbaxter · 2026-06-03T19:45:56Z

+					catch (S3Exception e) {
+						int status = e.statusCode();
+						if (status != 404 && status != 403) {
+							throw e;


Might be worth logging this as INFO in case there is a legitimate configuration error occurring and not just that the file does not exist

Yes, we added an INFO level log statement (LOG.info("Error checking S3 object key: " + key, e);) before rethrowing the exception to make troubleshooting configuration errors easier.

ryanjbaxter · 2026-06-03T19:49:09Z


 	}

+	private List<S3ConfigFile> getS3ConfigFileWithSearchPaths(String application, String profile, String label,


I would prefer this method be broken up into small more digestible chunks to make it easier to reason about and maintain/test

Yes, we broke down getS3ConfigFileWithSearchPaths into smaller helper methods (probeLiteralPattern, scanDirectoryPattern, probeDotWildcardPattern, scanWildcardPattern) to improve readability and testability.

ryanjbaxter · 2026-06-03T19:50:44Z


 }

+class PropertyConfigFileFromKey extends PropertyS3ConfigFile {


These three classes are nearly identical. That combined with the callReadImmediately comment I had above makes it feel like these classes could be refactored

Yes, we completely refactored this by consolidating PropertyConfigFileFromKey, YamlConfigFileFromKey, and JsonConfigFileFromKey into a single S3ConfigFileFromKey class.

ryanjbaxter · 2026-06-03T19:53:25Z

+		this(s3Client, bucketName, useApplicationAsDirectory, server, null);
+	}
+
+	public AwsS3EnvironmentRepository(S3Client s3Client, String bucketName, boolean useApplicationAsDirectory,


It seems better at this point to just have a constructor that takes in AwsEnvironmentProperties

Yes, we added a new constructor AwsS3EnvironmentRepository(S3Client, AwsS3EnvironmentProperties, ConfigServerProperties) that delegates to the canonical constructor, simplifying the factory builder.

ryanjbaxter · 2026-06-03T19:55:33Z

 				"default", "defaultlabel", null, new String[] { "s3://test/defaultlabel" }));
 	}

+	// 1) Placeholder only


We dont need the number comments on the tests

Yes, we removed the numbered comments from the test cases.

…-s3-searchpaths

…dant callReadImmediately flags Signed-off-by: Geonwook Ham <tomy8964@naver.com>

Signed-off-by: Geonwook Ham <tomy8964@naver.com>

tomy8964 · 2026-06-06T03:24:22Z

Hi @ryanjbaxter!
We have successfully addressed all of your feedback and suggestions. Here is a summary of the changes:

Summary of Changes

searchPaths Optimization & Safety
- Initialized searchPaths using Collections.emptyList() by default in both AwsS3EnvironmentProperties and AwsS3EnvironmentRepository.
- Updated the repository constructor to guard against null and pass Collections.emptyList() instead.
- Cleaned up the factory builder by introducing a constructor that accepts AwsS3EnvironmentProperties directly, removing the redundant setOrder call.
Cleanups & Constant Extraction
- Extracted S3 file extension arrays into class-level private static final constants (SUPPORTED_EXTENSIONS and EMPTY_EXTENSION) and reused them.
- Refactored isSimpleProfileName to be a package-private static helper method and reused it in wrapKeyWithNegatedConfigFiles.
- Removed numbered comments in the test cases (AwsS3EnvironmentRepositoryTests).
Subclass Refactoring
- Consolidated the redundant key-based S3ConfigFile subclasses (PropertyConfigFileFromKey, YamlConfigFileFromKey, and JsonConfigFileFromKey) into a single S3ConfigFileFromKey class, completely removing the callReadImmediately flag.
S3 File Probe & Scanning Improvements
- Corrected a bug by removing the break statement in the literal file probing loop, ensuring all matching files with different extensions (e.g., both .properties and .yml at the same path) are properly loaded.
- Added an INFO level log statement when checking S3 object keys before rethrowing unexpected S3Exceptions to assist troubleshooting.
- Refactored getS3ConfigFileWithSearchPaths into smaller, self-documenting helper methods (probeLiteralPattern, scanDirectoryPattern, probeDotWildcardPattern, scanWildcardPattern) to improve readability and maintainability.
Documentation
- Added comprehensive documentation for the S3 search-paths feature in aws-s3-backend.adoc, explaining literal/directory paths, wildcards, placeholders, and providing a YAML configuration example.
  All tests have been run and verified. Please let us know if there is anything else that needs to be addressed. Thank you! 🙏

spring-cloud-issues added the waiting-for-triage label Jul 6, 2025

tomy8964 mentioned this pull request Jul 7, 2025

AwsS3EnvironmentRepository does not comply with Git #2812

Open

Merge branch 'main' into feature/spring-cloudgh-2812-aws-s3-searchpaths

4a2c386

Signed-off-by: Geonwook Ham <tomy8964@naver.com>

ryanjbaxter requested a review from Copilot June 1, 2026 13:42

Copilot started reviewing on behalf of ryanjbaxter June 1, 2026 13:42 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

tomy8964 force-pushed the feature/gh-2812-aws-s3-searchpaths branch from ad8fc5a to daa088a Compare June 2, 2026 12:36

tomy8964 added 2 commits June 2, 2026 21:41

Merge branch 'spring-cloud:main' into feature/spring-cloudgh-2812-aws…

9546357

…-s3-searchpaths

ryanjbaxter requested a review from Copilot June 2, 2026 13:25

Copilot started reviewing on behalf of ryanjbaxter June 2, 2026 13:25 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

tomy8964 and others added 2 commits June 2, 2026 23:04

ryanjbaxter requested changes Jun 3, 2026

View reviewed changes

tomy8964 added 3 commits June 4, 2026 21:48

Merge branch 'spring-cloud:main' into feature/spring-cloudgh-2812-aws…

17bbc58

…-s3-searchpaths

refactor: consolidate key-based S3ConfigFile classes and remove redun…

f7c5afd

…dant callReadImmediately flags Signed-off-by: Geonwook Ham <tomy8964@naver.com>

docs: add SearchPaths S3 DOCS

0568977

Signed-off-by: Geonwook Ham <tomy8964@naver.com>

tomy8964 requested a review from ryanjbaxter June 6, 2026 03:37


		private int order = DEFAULT_ORDER;

		private List<String> searchPaths = new ArrayList<>();


		}

		private List<S3ConfigFile> getS3ConfigFileWithSearchPaths(String application, String profile, String label,


		}

		class PropertyConfigFileFromKey extends PropertyS3ConfigFile {

Conversation

tomy8964 commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Additional Notes

Backward Compatibility

Migration

Documentation

Performance and Cost

Uh oh!

tomy8964 commented May 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomy8964 commented Jun 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomy8964 commented Jun 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanjbaxter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomy8964 commented Jul 6, 2025 •

edited

Loading