What happened?
A private OCI chart pull can fail with 401 unauthorized when provider-helm relies on the default credential chain instead of chart.pullSecretRef, even though the provider runtime identity is able to resolve registry credentials.
This affects Azure Container Registry workload identity because the ACR credential helper returns Docker identity-token credentials: username <token> plus an ACR refresh token. provider-helm does not preserve that identity token before handing credentials to Helm, so the OCI pull falls back to anonymous/no usable credentials.
Example failure:
failed to install release: failed to pull chart: failed to perform "FetchReference" on source: GET "https://example.azurecr.io/v2/helm/private/my-chart/manifests/1.0.0": GET "https://example.azurecr.io/oauth2/token?scope=repository%3Ahelm%2Fprivate%2Fmy-chart%3Apull&service=example.azurecr.io": response status code 401: unauthorized: authentication required
Expected behavior
When the default keychain returns an identity token, provider-helm should preserve it and pass it to Helm/ORAS as an OCI registry refresh token. Registries that already materialize credentials as username plus password/access token should continue to work through the existing username/password path.
The bug is in the credential data shape passed between the default keychain and Helm:
The result is that AuthConfig.IdentityToken is resolved successfully by the default keychain, then discarded before Helm pulls the chart. This is not limited to ACR as a data-shape issue; any Docker credential helper that returns identity-token credentials can hit the same drop.
How can we reproduce it?
- Run
provider-helm v1.2.0, or current main at 9e411cf68a2d5f39c96ea73f3f07838f97151e47, with Azure Workload Identity environment variables available to the provider runtime.
- Grant that identity pull access to a private Azure Container Registry OCI chart repository.
- Create a
Release that references the private ACR OCI chart without chart.pullSecretRef, for example:
apiVersion: helm.m.crossplane.io/v1beta1
kind: Release
metadata:
name: private-chart
namespace: default
spec:
forProvider:
namespace: default
chart:
repository: oci://example.azurecr.io/helm/private
name: my-chart
version: 1.0.0
providerConfigRef:
name: default
kind: ClusterProviderConfig
- Observe that the chart pull fails with ACR
401 unauthorized.
The default keychain can resolve a token through the ACR helper, but the token is stored in AuthConfig.IdentityToken and discarded before Helm pulls the chart.
What environment did it happen in?
Provider Helm version: v1.2.0
Code path also present on main at 9e411cf68a2d5f39c96ea73f3f07838f97151e47.
Registry: private Azure Container Registry OCI chart repository.
Authentication path: provider runtime Azure Workload Identity through the default credential chain, without chart.pullSecretRef.
What happened?
A private OCI chart pull can fail with
401 unauthorizedwhenprovider-helmrelies on the default credential chain instead ofchart.pullSecretRef, even though the provider runtime identity is able to resolve registry credentials.This affects Azure Container Registry workload identity because the ACR credential helper returns Docker identity-token credentials: username
<token>plus an ACR refresh token.provider-helmdoes not preserve that identity token before handing credentials to Helm, so the OCI pull falls back to anonymous/no usable credentials.Example failure:
failed to install release: failed to pull chart: failed to perform "FetchReference" on source: GET "https://example.azurecr.io/v2/helm/private/my-chart/manifests/1.0.0": GET "https://example.azurecr.io/oauth2/token?scope=repository%3Ahelm%2Fprivate%2Fmy-chart%3Apull&service=example.azurecr.io": response status code 401: unauthorized: authentication requiredExpected behavior
When the default keychain returns an identity token,
provider-helmshould preserve it and pass it to Helm/ORAS as an OCI registry refresh token. Registries that already materialize credentials as username plus password/access token should continue to work through the existing username/password path.The bug is in the credential data shape passed between the default keychain and Helm:
docker-credential-acr-envreturns<token>and the ACR refresh token: https://github.com/chrismellard/docker-credential-acr-env/blob/82a0ddb27589107be5a0c4d2ad0269c6ad1762c1/pkg/credhelper/helper.go#L72-L80go-containerregistrymaps helper username<token>toauthn.AuthConfig.IdentityToken: https://github.com/google/go-containerregistry/blob/v0.20.7/pkg/authn/keychain.go#L212-L216provider-helmcurrently copies onlyUsernameandPasswordwhen converting the keychain result:provider-helm/pkg/clients/registryauth/auth.go
Lines 212 to 215 in 9e411cf
RepoCredsonly carriesUsernameandPassword:provider-helm/pkg/clients/helm/repocreds.go
Lines 3 to 7 in 9e411cf
provider-helm/pkg/clients/helm/client.go
Lines 224 to 234 in 9e411cf
auth.Credential.RefreshToken: https://github.com/oras-project/oras-go/blob/v2.6.0/registry/remote/auth/credential.go#L30-L34The result is that
AuthConfig.IdentityTokenis resolved successfully by the default keychain, then discarded before Helm pulls the chart. This is not limited to ACR as a data-shape issue; any Docker credential helper that returns identity-token credentials can hit the same drop.How can we reproduce it?
provider-helmv1.2.0, or currentmainat9e411cf68a2d5f39c96ea73f3f07838f97151e47, with Azure Workload Identity environment variables available to the provider runtime.Releasethat references the private ACR OCI chart withoutchart.pullSecretRef, for example:401 unauthorized.The default keychain can resolve a token through the ACR helper, but the token is stored in
AuthConfig.IdentityTokenand discarded before Helm pulls the chart.What environment did it happen in?
Provider Helm version: v1.2.0
Code path also present on
mainat9e411cf68a2d5f39c96ea73f3f07838f97151e47.Registry: private Azure Container Registry OCI chart repository.
Authentication path: provider runtime Azure Workload Identity through the default credential chain, without
chart.pullSecretRef.