Skip to content

Forward catalog static FileIO properties to server-side FileIO#4696

Open
jbonofre wants to merge 1 commit into
apache:mainfrom
jbonofre:file-io-props
Open

Forward catalog static FileIO properties to server-side FileIO#4696
jbonofre wants to merge 1 commit into
apache:mainfrom
jbonofre:file-io-props

Conversation

@jbonofre

Copy link
Copy Markdown
Member

IcebergCatalog.loadFileIOForTableLike only passed tableDefaultProperties (the table-default.*-prefixed subset of catalog properties) to the FileIOFactory. For S3-compatible storage configured with stsUnavailable: true, catalogs typically carry static credentials (s3.access-key-id / s3.secret-access-key) as catalog properties. These were dropped before reaching S3FileIO, which then fell back to the AWS default credentials provider chain and failed when no chain credential was available.

Merge catalogProperties as the base layer under tableProperties. DefaultFileIOFactory still overlays StorageAccessConfig last, so STS-vended subscoped credentials continue to take precedence over static catalog credentials when STS is available.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

`IcebergCatalog.loadFileIOForTableLike` only passed `tableDefaultProperties`
(the `table-default.*`-prefixed subset of catalog properties) to the
`FileIOFactory`. For S3-compatible storage configured with
`stsUnavailable: true`, catalogs typically carry static credentials
(`s3.access-key-id` / `s3.secret-access-key`) as catalog properties.
These were dropped before reaching `S3FileIO`, which then fell back to
the AWS default credentials provider chain and failed when no chain
credential was available.

Merge `catalogProperties` as the base layer under `tableProperties`.
`DefaultFileIOFactory` still overlays `StorageAccessConfig` last, so
STS-vended subscoped credentials continue to take precedence over
static catalog credentials when STS is available.
@github-project-automation github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jun 11, 2026
@jbonofre jbonofre added this to the 1.6.0 milestone Jun 11, 2026
@adutra

adutra commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

catalogs typically carry static credentials (s3.access-key-id / s3.secret-access-key) as catalog properties.

Isn't this a bad practice? In the REST Federation doc page, we say:

Catalog properties are returned to authenticated catalog clients as part of the Iceberg REST /config response. Only place non-sensitive client configuration in catalog properties. Do not put passwords, tokens, access keys, or other secrets into catalog properties.

@dimas-b

dimas-b commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

I agree with @adutra 's comment above that passwords should not be stored in catalog properties.

I believe general-purpose Catalog properties should probably not be forwarded to clients automatically. That will create scope confusion between the Server and the Client. If a particular property has a reason for being forwarded, it should be ok on a case-by-case basis (as for table-default.*).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants