Skip to content

test: enforce restricted PSS for CI user namespace #3487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danish9039 wants to merge 24 commits into
base: master
Choose a base branch
from
Open

test: enforce restricted PSS for CI user namespace #3487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c62cccd
test: enforce PSS restricted for CI user namespace
abdullahpathan22 Apr 9, 2026
86c3b32
ci: update workflow triggers for PSS test coverage
abdullahpathan22 Apr 20, 2026
7893dba
ci: complete restricted PSS trigger coverage
danish9039 May 26, 2026
d3762d4
test: make CI workloads restricted-compatible
danish9039 May 27, 2026
eb452f0
test: patch generated workloads for restricted PSS
danish9039 May 27, 2026
9f0aed4
test: fix restricted PSS CI follow-ups
danish9039 May 27, 2026
fdd7b93
test: fix restricted generated workload checks
danish9039 May 27, 2026
08df5a9
test: configure generated workload security contexts
danish9039 May 27, 2026
548da1e
test: wait for patched katib controller rollout
danish9039 May 27, 2026
6dcf65e
test: render katib controller security flag
danish9039 May 27, 2026
4d6d720
test: enable katib security context config
danish9039 May 27, 2026
66cffdb
test: patch katib suggestion security context
danish9039 May 27, 2026
3de2525
test: avoid kserve auth policy race
danish9039 May 27, 2026
c09bd7b
test: add raw kserve security context
danish9039 May 27, 2026
3bc4d17
test: fail fast in dashboard debug
danish9039 May 28, 2026
9139206
test: harden restricted PSS follow-ups
danish9039 May 29, 2026
6e70923
test: fail fast in debug dumps
danish9039 May 30, 2026
f7ce770
test: move restricted pss fixes into overlays
danish9039 Jun 4, 2026
5b6fef7
test: scope restricted pss changes to ci
danish9039 Jul 4, 2026
1b63367
test: cover restricted pss pipeline gaps
danish9039 Jul 5, 2026
d1e104b
test: drop restricted PSS overlay for the removed Training Operator
danish9039 Aug 4, 2026
53e2f5e
pipeline: scope restricted PSS defaults to continuous integration
danish9039 Aug 4, 2026
e31ff6d
test: format pipeline diagnostics with black
danish9039 Aug 4, 2026
8c4d0d4
test: drop kserve storage initializer live patch
danish9039 Aug 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/dex_oauth2-proxy_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ on:
- common/cert-manager/**
- common/oauth2-proxy/**
- common/istio*/**
- experimental/security/PSS/*
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh
- common/dex/base/**
- tests/istio*
- tests/dex_login_test.py
Expand Down
42 changes: 40 additions & 2 deletions .github/workflows/istio_validation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ on:
- tests/ambient*
- tests/multi_tenancy_install.sh
- tests/profile_controller_install.sh
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh
- applications/dashboard/upstream/profile-controller/**
- .github/workflows/istio_validation.yaml
- common/istio/**
Expand Down Expand Up @@ -253,8 +255,44 @@ jobs:
- name: Test basic connectivity
run: |
kubectl expose deployment test-application --port=80 --target-port=8080 -n $KF_PROFILE
kubectl run test-client --image=busybox --rm -i --restart=Never -n $KF_PROFILE -- \
wget -qO- --timeout=10 test-application.$KF_PROFILE.svc.cluster.local
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: test-client
namespace: ${KF_PROFILE}
spec:
restartPolicy: Never
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: test-client
image: busybox:1.36.1
command:
- wget
- -qO-
- --timeout=10
- test-application.${KF_PROFILE}.svc.cluster.local
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
EOF
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/test-client -n $KF_PROFILE --timeout=120s || (
kubectl logs pod/test-client -n $KF_PROFILE
kubectl describe pod/test-client -n $KF_PROFILE
exit 1
)
kubectl logs pod/test-client -n $KF_PROFILE
kubectl delete pod/test-client -n $KF_PROFILE --ignore-not-found

- name: Apply Pod Security Standards Restricted levels
if: matrix.istio-mode == 'cni'
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/katib_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ on:
pull_request:
paths:
- tests/install_KinD_create_KinD_cluster_install_kustomize.sh
- tests/katib_install.sh
- tests/katib*
- .github/workflows/katib_test.yaml
- applications/katib/upstream/**
- common/istio*/**
Expand All @@ -12,7 +12,9 @@ on:
- tests/profile_controller_install.sh
- applications/dashboard/upstream/profile-controller/**
- common/cert-manager/**
- experimental/security/PSS/*
- applications/katib/overlays/**
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh

permissions:
contents: read
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/kserve_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ on:
- tests/istio*
- tests/multi_tenancy_install.sh
- tests/profile_controller_install.sh
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh
- applications/dashboard/upstream/profile-controller/**
- common/knative/**
- tests/knative_serving_install.sh
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/pipeline_run_from_notebook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ on:
- common/cert-manager/**
- common/oauth2-proxy/**
- common/istio*/**
- tests/pipeline*
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh
- common/kubeflow-namespace/**

permissions:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/pipeline_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ on:
- common/cert-manager/**
- common/oauth2-proxy/**
- common/istio*/**
- tests/pipeline_v1_test.py
- tests/pipeline_v2_test.py
- experimental/security/PSS/*
- tests/pipeline*
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh

permissions:
contents: read
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/trainer_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ on:
- common/cert-manager/**
- common/oauth2-proxy/**
- common/istio*/**
- experimental/security/PSS/*
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh

permissions:
contents: read
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/volumes_web_application_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ on:
- tests/oauth2-proxy_install.sh
- tests/multi_tenancy_install.sh
- tests/profile_controller_install.sh
- tests/kubeflow_profile_install.sh
- applications/dashboard/upstream/profile-controller/**
- tests/volumes_web_application_test.sh

Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/workspaces_pipeline_run_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ on:
- tests/dashboard_install.sh
- tests/multi_tenancy_install.sh
- tests/pipelines_install.sh
- tests/kubeflow_profile_install.sh
- tests/PSS_enable.sh
- tests/workspaces_install.sh
- tests/workspaces_pipeline_run_test.sh
- tests/workspacekind.test.yaml
Expand Down
159 changes: 159 additions & 0 deletions applications/katib/overlays/security/katib-config.yaml
View file
Open in desktop

@juliusvonkohout juliusvonkohout Jun 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you upstream this or make sure to only patch the securitycontext, but not the image for example ?

@danish9039 danish9039 Jul 4, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split to kubeflow/katib#2683; the intended delta is only securityContext/injection config, not image changes.

@danish9039 danish9039 Aug 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the securityContext is intended, but it cannot be expressed as a partial patch: katib-config.yaml is one opaque string inside the katib-config ConfigMap, so kustomize can only replace the whole file. The overlay is a temporary full copy until kubeflow/katib#2683, which adds the same suggestion and metrics-collector contexts to the default configuration upstream, is released and synchronized. At that point this overlay is deleted. The image tags in the copy are identical to the synchronized upstream ones (v0.19.0).

Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
---
apiVersion: config.kubeflow.org/v1beta1
kind: KatibConfig
init:
controller:
webhookPort: 8443
trialResources:
- TrainJob.v1alpha1.trainer.kubeflow.org
- Job.v1.batch
- TFJob.v1.kubeflow.org
- PyTorchJob.v1.kubeflow.org
- MPIJob.v1.kubeflow.org
- XGBoostJob.v1.kubeflow.org
injectSecurityContext: true
runtime:
metricsCollectors:
- kind: StdOut
image: ghcr.io/kubeflow/katib/file-metrics-collector:v0.19.0
- kind: File
image: ghcr.io/kubeflow/katib/file-metrics-collector:v0.19.0
- kind: TensorFlowEvent
image: ghcr.io/kubeflow/katib/tfevent-metrics-collector:v0.19.0
resources:
limits:
memory: 1Gi
suggestions:
- algorithmName: random
image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: tpe
image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: grid
image: ghcr.io/kubeflow/katib/suggestion-optuna:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: hyperband
image: ghcr.io/kubeflow/katib/suggestion-hyperband:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: bayesianoptimization
image: ghcr.io/kubeflow/katib/suggestion-skopt:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: cmaes
image: ghcr.io/kubeflow/katib/suggestion-goptuna:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: sobol
image: ghcr.io/kubeflow/katib/suggestion-goptuna:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: multivariate-tpe
image: ghcr.io/kubeflow/katib/suggestion-optuna:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: enas
image: ghcr.io/kubeflow/katib/suggestion-enas:v0.19.0
resources:
limits:
memory: 400Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: darts
image: ghcr.io/kubeflow/katib/suggestion-darts:v0.19.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- algorithmName: pbt
image: ghcr.io/kubeflow/katib/suggestion-pbt:v0.19.0
persistentVolumeClaimSpec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
earlyStoppings:
- algorithmName: medianstop
image: ghcr.io/kubeflow/katib/earlystopping-medianstop:v0.19.0
15 changes: 15 additions & 0 deletions applications/katib/overlays/security/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kubeflow

resources:
- ../../upstream/installs/katib-with-kubeflow

generatorOptions:
disableNameSuffixHash: true

configMapGenerator:
- name: katib-config
behavior: replace
files:
- katib-config.yaml
18 changes: 18 additions & 0 deletions applications/pipeline/restricted-pss/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Continuous-integration only overlay.
#
# It layers the restricted Pod Security Standards identity defaults on top of the regular
# Pipelines installation, so that the GitHub Actions user namespace can enforce
# `pod-security.kubernetes.io/enforce=restricted`.
#
# The regular installation in `../overlays` deliberately leaves
# `defaultSecurityContextRunAsUser`, `defaultSecurityContextRunAsGroup` and
# `defaultSecurityContextRunAsNonRoot` unset, so that operators keep the upstream
# Kubeflow Pipelines defaults and are not silently given a fixed identity.
resources:
- ../overlays

patches:
- path: pipeline-install-config-restricted-pss.yaml
8 changes: 8 additions & 0 deletions applications/pipeline/restricted-pss/pipeline-install-config-restricted-pss.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: pipeline-install-config
data:
defaultSecurityContextRunAsUser: "1000"
defaultSecurityContextRunAsGroup: "0"
defaultSecurityContextRunAsNonRoot: "true"
13 changes: 13 additions & 0 deletions applications/trainer/overlays/runtimes-restricted/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../upstream/overlays/runtimes

patches:
- path: patches/torch-distributed-restricted-pss.yaml
target:
group: trainer.kubeflow.org
version: v1alpha1
kind: ClusterTrainingRuntime
name: torch-distributed
21 changes: 21 additions & 0 deletions ...ations/trainer/overlays/runtimes-restricted/patches/torch-distributed-restricted-pss.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
- op: add
path: /spec/template/spec/replicatedJobs/0/template/spec/template/spec/securityContext
value:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
- op: add
path: /spec/template/spec/replicatedJobs/0/template/spec/template/spec/containers/0/workingDir
value: /tmp
- op: add
path: /spec/template/spec/replicatedJobs/0/template/spec/template/spec/containers/0/securityContext
value:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
Loading
Loading