feat(providers): AWS STS AssumeRole refresh strategy and aws-s3 profile

[russellb]

Summary

Add gateway-owned AWS STS AssumeRole as a credential refresh strategy, ship aws and aws-s3 provider profiles, and extend policy validation to support single-label * wildcards in middle DNS labels.

Combined with the SigV4 proxy re-signing from #1638 (now merged), sandboxes can access S3 and other AWS services without ever seeing real credentials — the gateway mints short-lived STS credentials and the proxy re-signs requests on the fly.

Related Issue

Refs #1576

Changes

STS credential refresh (openshell-server)

• New AwsStsAssumeRole variant in ProviderCredentialRefreshStrategy (proto field AWS_STS_ASSUME_ROLE = 6).
• provider_refresh.rs implements the refresh loop: calls sts:AssumeRole using the gateway's ambient AWS credentials (or explicit long-lived keys from refresh material), stores the resulting AccessKeyId, SecretAccessKey, and SessionToken on the provider, and schedules re-rotation before expiry.
• ConfigureProviderRefresh and RotateProviderCredential RPCs gated behind the providers_v2_enabled setting.
• Validates that configuring STS refresh for one provider won't collide with credential keys already held by another provider attached to the same sandbox.

Provider profiles (openshell-providers)

AWS provider profiles follow the same pattern as the existing aws-bedrock profile: a generic base plus service-specific variants.

• providers/aws.yaml — base AWS profile with STS refresh material (role_arn, session_name, external_id, aws_region, optional long-lived keys) but no endpoints or binaries allowlist. Intended for AWS services that don't yet have a dedicated profile — the user attaches their own policy to supply endpoints. Same role google-cloud plays relative to google-vertex-ai.
• providers/aws-s3.yaml — S3-specific profile that adds pre-configured endpoints covering regional (*.s3.*.amazonaws.com), global (*.s3.amazonaws.com with signing_region: us-east-1), and dualstack variants. All endpoints use credential_signing: sigv4 and signing_service: s3. Includes a binaries allowlist for Python, curl, and the AWS CLI.

Policy: single-label * wildcard in middle DNS labels (openshell-policy)

• validate_host_wildcard now permits * as a complete middle label (e.g. *.s3.*.amazonaws.com) in addition to leading-label and intra-label positions. Matches exactly one DNS label via Rego's glob.match with . separator, consistent with TLS wildcard semantics.
• OPA endpoint matching and L7 config lookup updated: host_matches_wildcard_middle rule and endpoint_config_for_middle_wildcard in Rego.
• Validation rejects ** in any position and partial wildcards in middle labels.

TUI (openshell-tui)

• Added AwsStsAssumeRole match arm to the refresh strategy label in the provider settings view.

Documentation

• docs/sandboxes/manage-providers.mdx: STS refresh setup instructions
• examples/aws-s3-sts.md: end-to-end manual test guide (create IAM role, configure provider, test S3 PUT/GET/LIST from sandbox)

Testing

• cargo test -p openshell-providers — 65 passed (STS serde roundtrip, aws/aws-s3 profile parsing, endpoint validation)
• cargo test -p openshell-policy — 91 passed (wildcard DNS validation, middle-label wildcards, credential_signing validation)
• cargo test -p openshell-server — all passed (STS configure v2 gate, success path, credential key collision)
• cargo test -p openshell-supervisor-network — all passed (wildcard host matching, OPA endpoint config)
• cargo test -p openshell-tui — 24 passed
• End-to-end manual test: Podman sandbox → boto3 → S3 PUT/GET/LIST with STS-minted credentials and proxy-side SigV4 re-signing against real AWS (all passed)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

[russellb]

This is still waiting on #1638 to go in first.

[russellb]

@pimlock willing to take a look at this one since you reviewed #1638 ?

[pimlock]

/ok to test cb10f83

[pimlock]

Hi @russellb, I did a first pass, but I run out of time to get through this fully. I needed to check this out locally and run this to understand how it worked (wasn't as familiar with the cred refresh).

I added a few comments.
The biggest item I had was around how the implicit extra env vars that are managed by the refresh, but not defined as such in the provider profile.
I did investigation around this and options. I would need to spend more time figuring out what makes sense, but I will need to come back to it on Thursday. In the meantime, I can share raw notes from agent, but I haven't checked these very deeply yet: https://gist.github.com/pimlock/6a99b3d9c4e19c6efc3b0d2da59b6b63

[pimlock]

This call will fail without setting a dummy value for --credential AWS_ACCESS_KEY_ID=dummy (interestingly, it doesn't even need to be a declared credential, something like --credential FOO=bar will do).

I think ideally we could use the --runtime-credentials flag, but using that fails, because of the STS refresh not being in here:

• OpenShell/crates/openshell-providers/src/profiles.rs

  Lines 478 to 486 in cb10f83

  	pub fn is_gateway_mintable(&self) -> bool {
  	matches!(
  	self.strategy,
  	ProviderCredentialRefreshStrategy::Oauth2RefreshToken
  	| ProviderCredentialRefreshStrategy::Oauth2ClientCredentials
  	| ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt
  	)
  	}
  	}

However that won't fully work either, since the refresh is only configured on the access_key_id and not on the secret key/session token.

That last thing is one thing that I wonder how we can solve -> the only cred that is marked as being refreshed is the access_key_id, but in fact the secret key and session token are as well.

And right now we are missing a way to represent this.

[russellb]

Both parts addressed in cbe9b43.

Runtime credentials: root cause was is_gateway_mintable in openshell-providers omitting AwsStsAssumeRole — it had diverged from the server-side check. Consolidated into one shared function that includes STS, so the profile's credentials are runtime-resolvable and openshell provider create --type aws-s3 --runtime-credentials works with no placeholder --credential. Docs and examples/aws-s3-sts.md updated.

Representing multi-output refresh: went declarative. A refresh now declares the extra credentials it co-mints via additional_outputs, mapping each strategy-produced output to a sibling credential:

refresh:
  strategy: aws_sts_assume_role
  additional_outputs:
    - output: secret_access_key
      credential: secret_access_key
    - output: session_token
      credential: session_token

The resolved output→env-key mapping is pinned into the refresh state at configure time (so later profile edits can't silently redirect where minted values are written), and minting, collision reservation, and active_provider_environment_keys all read it instead of the previously hardcoded AWS list.

Concretely this closes the gap you described: configure now reserves all three keys up front, so "configure STS, don't rotate, another provider claims AWS_SECRET_ACCESS_KEY, attach both" is rejected at attach time instead of failing at mint. The SigV4 signer stays on the standard AWS names, now enforced by profile validation (outputs must resolve to AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN and the primary to AWS_ACCESS_KEY_ID).

[pimlock]

I just noticed that this check is duplicated in

OpenShell/crates/openshell-providers/src/profiles.rs

Lines 478 to 486 in cb10f83

	pub fn is_gateway_mintable(&self) -> bool {
	matches!(
	self.strategy,
	ProviderCredentialRefreshStrategy::Oauth2RefreshToken
	| ProviderCredentialRefreshStrategy::Oauth2ClientCredentials
	| ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt
	)
	}
	}

.

Can we replace this one with the one in openshell-providers?

[russellb]

Consolidated into a single is_gateway_mintable_strategy in openshell-providers; both server call sites (provider_refresh.rs and grpc/provider.rs) now delegate to it. Good catch that these had diverged — the providers-side copy was missing AwsStsAssumeRole, which is exactly what broke --runtime-credentials (your docs comment). Fixing the duplication fixes both. cbe9b43

[pimlock]

I was wondering if there is a way to reuse credentials and only define endpoints in the service-specific profile.

I think the closest we could do today is something like:

• aws.yaml has credentials (what it already has)
• aws-service.yaml has endpoints
• the user needs to attach both providers to the sandbox for this to work

The benefit here would be they configure the aws profile instance once and can reuse it with multiple services.

The problem is that the relationship is implicit and running a sandbox with aws-service, but not aws will fail in runtime during signing. Perhaps there is a way to make that relationship explicit and be able to validate it.

[pimlock]

cc @johntmyers I wonder if something like this came up before:

• there is one credential type for given suite of services
• endpoints for each service are defined separately

In this example:

• provider_1: I want to bring in AWS creds
• provider_2: I want to access S3
• provider_3: I want to access Lambda

[russellb]

This is a nice direction and I think it's the natural next step. The blocker is exactly what you named — the credentials↔endpoints relationship would be implicit and only fail at signing time. Making it explicit and validated is worth its own design pass (it overlaps the additional_outputs model added in this PR), so rather than expand this PR I'd like to track it as a follow-up. Flagging it here as a follow-up for now — happy to open a tracking issue for you and @johntmyers to weigh in on.

[russellb]

@pimlock thank you very much for the review! I know it can be very time-consuming. I will dig into the comments tomorrow.

[pimlock]

/ok to test cbe9b43