Feat: SPIFFE authentication for operator client registration (Admin API)

[Alan-Cha]

Summary

Implements #1421 - Eliminate admin credentials from client registration using SPIFFE JWT-SVID authentication with Keycloak Admin API.

This PR provides the operator-side implementation. Platform bootstrap automation is in kagenti/kagenti#2135.

Problem

Currently, the operator uses admin credentials to register OAuth clients via Keycloak Admin API. This has several security issues:

• Admin credentials = full realm permissions (create/delete users, realms, clients, roles)
• Long-lived credentials requiring manual rotation
• Operator compromise = full realm admin access
• Admin credentials stored as secrets in every agent namespace

Solution

Use the operator's SPIFFE JWT-SVID to authenticate with Keycloak Admin API using federated-jwt client authenticator.

Architecture

Operator Pod
├─> spiffe-helper sidecar (standalone container in operator pod)
│   └─> writes JWT-SVID to /opt/jwt_svid.token (auto-rotates)
├─> Manager reads JWT-SVID from file
└─> Authenticates to Keycloak:
    POST /realms/kagenti/protocol/openid-connect/token
    - grant_type: client_credentials
    - client_assertion_type: urn:ietf:params:oauth:client-assertion-type:jwt-spiffe
    - client_assertion: <JWT-SVID>
    → Returns access token with manage-clients role
    → Use same Admin API endpoints as before

Benefits

Current (Admin Creds)	With SPIFFE Auth (This PR)
Admin username/password	JWT-SVID (cryptographic)
Full realm admin permissions	Scoped (manage-clients only)
Long-lived credentials	Short-lived (1 hour), auto-rotates
Manual rotation	Automatic (spiffe-helper)
Secrets in all namespaces	No secrets
Audit trail: "admin" user	Audit trail: operator SPIFFE ID

Implementation

Core Changes

internal/keycloak/admin.go:

• Added JWTSVIDGrantToken() method for JWT-SVID authentication
• Uses client_assertion_type: urn:ietf:params:oauth:client-assertion-type:jwt-spiffe
• Returns access token with operator's assigned roles

internal/controller/clientregistration_controller.go:

• Added SPIFFE auth fields: UseSpiffeAuth, OperatorClientID, JWTSVIDPath, Recorder
• Dual authentication path:
  • UseSpiffeAuth=true: Read JWT-SVID from file → authenticate with JWTSVIDGrantToken()
  • UseSpiffeAuth=false: Use admin credentials (legacy, default)
• Both paths use same Admin API endpoints (no DCR)
• Path traversal protection, token exposure prevention, Kubernetes Event recording

cmd/main.go:

• Registers CLI flags: --use-spiffe-auth, --operator-client-id, --jwt-svid-path
• Configures ClientRegistrationReconciler with SPIFFE settings
• Feature is opt-in via CLI flags (default: disabled)

Helm Configuration

charts/kagenti-operator/values.yaml:

spiffe:
  enabled: false
  operatorAuth:
    enabled: false
    jwtAudience: ""       # must be set explicitly; defaults to keycloak.publicUrl/realms/realm
    jwtSVIDPath: "/opt/jwt_svid.token"

charts/kagenti-operator/templates/manager/manager.yaml:

• Conditionally adds spiffe-helper sidecar container when spiffe.operatorAuth.enabled=true
• Conditionally mounts spiffe-workload-api CSI volume and jwt-svid emptyDir volume

charts/kagenti-operator/templates/manager/configmap-spiffe-helper.yaml (new):

• Configures spiffe-helper to write JWT-SVID to /opt/jwt_svid.token

Usage

Enable SPIFFE Authentication

# Via installation script (recommended)
ENABLE_OPERATOR_SPIFFE_AUTH=true ./.github/scripts/local-setup/kind-full-test.sh --skip-cluster-destroy

# Or manually via Helm
helm upgrade kagenti-operator charts/kagenti-operator/ \
  --set spiffe.enabled=true \
  --set spiffe.operatorAuth.enabled=true \
  --set spiffe.operatorAuth.jwtAudience="http://keycloak.example.com/realms/kagenti" \
  -n kagenti-system

Requirements

✅ SPIRE deployed with SPIFFE OIDC Discovery Provider
✅ SPIFFE IdP configured in Keycloak (automated by kagenti/kagenti#2135)
✅ Operator client pre-created in Keycloak with federated-jwt auth (automated by kagenti/kagenti#2135)
✅ Operator client has manage-clients role (automated by kagenti/kagenti#2135)

All requirements are automated when using ENABLE_OPERATOR_SPIFFE_AUTH=true flag.

Testing Status

✅ Code Implementation: Complete
✅ Keycloak Configuration: Validated (SPIFFE IdP + operator client + manage-clients role)
✅ Backward Compatibility: Verified (default behavior unchanged)
✅ Authentication Method: Validated (JWTSVIDGrantToken returns valid access token)
✅ Admin API Integration: Uses existing endpoints (no changes to registration logic)
✅ Full E2E Test: Complete (multiple fresh-cluster runs, all 9 steps passed)

Rollout Plan

1. Phase 1: ✅ Core implementation (this PR)
2. Phase 2: ✅ Platform automation (feat(platform): Add operator SPIFFE authentication bootstrap kagenti#2135)
3. Phase 3: ✅ E2E testing
4. Phase 4: Staging deployment with SPIFFE auth enabled
5. Phase 5: Production rollout
6. Phase 6: Deprecate admin credential path

Backward Compatibility

• ✅ Default behavior unchanged (UseSpiffeAuth=false uses admin credentials)
• ✅ Opt-in via Helm values
• ✅ No breaking changes to existing deployments
• ✅ Both authentication modes work independently
• ✅ Gradual migration supported

Security Improvements

1. Scoped Permissions: Operator gets manage-clients role only (not full admin)
2. No Credential Storage: No secrets in operator namespace or agent namespaces
3. Short-Lived Tokens: JWT-SVID expires after 1 hour, auto-rotates
4. Cryptographic Auth: JWT-SVID signed by SPIRE (stronger than passwords)
5. Audit Trail: Keycloak logs show operator's SPIFFE ID (not generic "admin")

Related PRs

• feat(platform): Add operator SPIFFE authentication bootstrap kagenti#2135: Platform bootstrap automation (Keycloak setup, Helm hook job, installation scripts)
• Together these PRs enable complete zero-credential operator authentication

Related Issues

• Implements Eliminate admin credentials from operator client registration #410
• Builds on: Operator should read keycloak-admin-secret from kagenti-system, not agent namespaces #320 (operator reads admin creds from kagenti-system)
• Related: fix: move Keycloak admin secret to kagenti-system and remove client-registration sidecar references kagenti#1422 (Helm chart changes for Operator should read keycloak-admin-secret from kagenti-system, not agent namespaces #320)

Files Changed

• kagenti-operator/cmd/main.go (+18 lines)
• internal/keycloak/admin.go (+42 lines)
• internal/controller/clientregistration_controller.go (+90 lines)
• charts/kagenti-operator/values.yaml (+13 lines)
• charts/kagenti-operator/templates/manager/manager.yaml (+46 lines)
• charts/kagenti-operator/templates/manager/configmap-spiffe-helper.yaml (+24 lines, new)

Total: ~233 lines added

Assisted-By: Claude Code (Anthropic AI) noreply@anthropic.com

[Alan-Cha]

✅ Update: SPIFFE Authentication Implementation Complete

Early status update — see final PR description for current state.

Key Discovery: DCR Endpoint Not Required

After investigation, we determined that Keycloak Admin API with JWT-SVID authentication is the correct approach, NOT the DCR endpoint. The DCR endpoint has permission limitations that prevent proper client management.

Implementation Summary

Changed Approach:

• ❌ DCR endpoint (insufficient permissions for client updates)
• ✅ Admin API with JWT-SVID authentication (full client lifecycle support)

Core Changes:

1. Replaced UseDCR flag with UseSpiffeAuth

   • UseSpiffeAuth: Enable JWT-SVID authentication
   • JWTSVIDPath: Path to JWT-SVID file (default: /opt/jwt_svid.token)
   • OperatorClientID: Operator SPIFFE ID

2. Added JWTSVIDGrantToken() to keycloak.Admin

   • Authenticates operator using JWT-SVID
   • Returns access token with manage-clients role (NOT full admin)
   • Uses client-assertion-type:jwt-spiffe

3. Dual authentication path in reconciler

   • UseSpiffeAuth=true: Read JWT-SVID from file → authenticate with JWT-SVID
   • UseSpiffeAuth=false: Use admin credentials (legacy)

E2E Test Results ✅

Test Step	Status	Details
Operator Pod	✅	Running with spiffe-helper sidecar (standalone container, 2/2)
Operator Client Bootstrap	✅	Created with federated-jwt authenticator, manage-clients role
Operator JWT-SVID Auth	✅	HTTP 200, received access token
Agent Client Creation	✅	Operator created client via Admin API (HTTP 201)

Architecture

Before:

Operator → admin credentials (keycloak-admin-secret) → Admin API → Create clients

After:

Operator → JWT-SVID (SPIFFE identity, read from file written by spiffe-helper sidecar) → Admin API → Create clients

Security Benefits

• ✅ No admin credentials in any runtime namespace
• ✅ Operator identity tied to Kubernetes ServiceAccount
• ✅ JWT-SVIDs rotate automatically (short-lived tokens)
• ✅ Reduced blast radius: manage-clients role instead of full admin
• ✅ Audit trail: Specific SPIFFE ID in Keycloak logs

Files Modified (final)

• kagenti-operator/cmd/main.go — SPIFFE flag wiring
• internal/keycloak/admin.go — JWTSVIDGrantToken() method
• internal/controller/clientregistration_controller.go — dual auth path
• charts/kagenti-operator/values.yaml — spiffe: values block
• charts/kagenti-operator/templates/manager/manager.yaml — spiffe-helper sidecar
• charts/kagenti-operator/templates/manager/configmap-spiffe-helper.yaml — new

---

Implementation Status: ✅ Complete and E2E tested (multiple fresh-cluster runs)

[Alan-Cha]

✅ Implementation Complete - Rebased on origin/main

Intermediate status update. Final commits differ from those listed here — see PR commit history for current state.

Key Changes

1. JWT-SVID Authentication Method (admin.go)

Added JWTSVIDGrantToken() method that authenticates the operator using:

• grant_type=client_credentials
• client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-spiffe
• client_assertion=<JWT-SVID>

Returns access token with manage-clients role (not full admin).

2. Controller Integration (clientregistration_controller.go)

Added SPIFFE auth fields to reconciler and dual authentication path:

• UseSpiffeAuth=true: Read JWT-SVID from file, authenticate with JWT-SVID
• UseSpiffeAuth=false: Use admin credentials (legacy, default)

Both paths use the same Admin API for client registration.

Architecture

Before:

Operator → admin credentials → Admin API → Create clients

After:

Operator → JWT-SVID (from spiffe-helper sidecar file) → Admin API → Create clients

[Alan-Cha]

Part of #410

[Alan-Cha]

Summary

This PR implements JWT-SVID (SPIFFE) authentication for the Kagenti operator, eliminating the need for admin credentials when registering OAuth clients. The implementation is well-structured with proper backward compatibility, but has 3 critical security issues that must be fixed before merge.

Areas reviewed: Go code (authentication, controller), Security, Error handling, Backward compatibility, Commit conventions

Commits: 2 commits, all signed-off ✅

CI status: All checks passing ✅

Recommended Action: Fix 3 must-fix security issues before merge

---

Critical Issues

🔴 Security Issues (Must Fix)

1. Path traversal vulnerability - JWTSVIDPath is not validated before file read
2. JWT-SVID token exposure risk - Bearer token could leak in logs/errors
3. Silent failure masks misconfiguration - File read errors requeue without visibility

See inline comments for details and recommended fixes.

---

Positive Observations

✅ Backward compatibility preserved (defaults to admin credentials)
✅ Clean dual-path authentication design
✅ All commits properly signed-off
✅ CI passing (including E2E tests)
✅ No external dependencies added

[Alan-Cha]

[SUGGESTION] Add basic JWT format validation

Validate the JWT-SVID has the expected structure before sending to Keycloak. This provides faster feedback for malformed tokens.

Fix:

import "bytes"

// Basic JWT format check (header.payload.signature)
if bytes.Count(jwtSVID, []byte{'.'}) != 2 {
    logger.Error(nil, "invalid JWT-SVID format", "path", jwtSVIDPath)
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

[Alan-Cha]

Rebased on latest main (f68d4b2) to pick up PR #423 which fixes the skill discovery E2E test failure.

The skill discovery test failure was unrelated to SPIFFE authentication changes - it was caused by recent skill discovery work in PR #388 and fixed by PR #423.

Branch is now up to date and all tests should pass.

[Alan-Cha]

Two findings — both suggestions/nits, no blocking issues. Security-sensitive parts are well-handled: JWT-SVID is never logged, error responses are truncated, path traversal is guarded, and the jwt-spiffe assertion type is correct (intentionally changed from jwt-bearer which failed in testing).

Areas reviewed: Go, Helm/K8s
Commits: 8 commits, all signed-off ✅. First 3 commit subjects lack a scope (feat: instead of feat(controller):/feat(keycloak):), inconsistent with the later commits in this PR — worth fixing before merge.
CI: DCO passes ✅

Verdict: APPROVE (posted as comment — GitHub does not allow self-approval)

---

charts/kagenti-operator/templates/manager/configmap-spiffe-helper.yaml line 21 — suggestion:
The jwt_audience fallback printf "%s/realms/%s" .Values.keycloak.publicUrl .Values.keycloak.realm will produce /realms/kagenti if keycloak.publicUrl is not injected by the parent chart (its default in values.yaml is ""). Standalone deployments that omit spiffe.operatorAuth.jwtAudience will get a broken JWT audience and silent auth failures. Consider requiring explicit configuration: {{ required "spiffe.operatorAuth.jwtAudience must be set when operatorAuth.enabled=true" .Values.spiffe.operatorAuth.jwtAudience }}.

kagenti-operator/cmd/main.go ~line 573 — nit:
The startup log "SPIFFE ID authentication enabled" includes spireSocket (the verified-fetch socket path) but the operator reads the JWT-SVID from a file, not the socket. The more useful debug fields would be jwtSVIDPath and operatorClientID.

[Alan-Cha]

Superseded by #473 — clean 2-commit version with accurate descriptions.