chore(spark): align e2e harness with upstream Helm chart defaults

[adibmbrk]

What this PR does / why we need it:

Makes the Spark E2E harness mirror a default spark-operator Helm install so it catches upstream drift instead of masking it:

• Remove the vendored SparkConnect CRD; use the chart's own hook.upgradeCrd=true pre-install hook instead.
• Remove the hand-rolled RBAC (ensure_sparkconnect_rbac) — it granted permissions beyond what the default chart install provides. Examples now run as subprocesses against the CI runner's kubeconfig instead of in-cluster Jobs, which no longer need bespoke ServiceAccount permissions.
• Drop in-cluster (Job-based) example execution entirely: removes hack/Dockerfile.spark-e2e-runner, test/e2e/spark/run_in_cluster.py, the connect_existing_session.py example, and the SPARK_E2E_RUN_IN_CLUSTER/SPARK_E2E_RUNNER_IMAGE env vars/CI steps. This mode existed to let Job pods (running as the default SA) connect via an in-cluster URL; removing the default-SA RBAC (per KEP-107: Improve Spark E2E Robustness #611) breaks that path, so rather than provision a new dedicated SA for it, in-cluster execution is removed and all examples now run as subprocesses against the ambient kubeconfig.
• Extend the cluster watcher to poll SparkApplication in addition to SparkConnect so batch-job test failures get the same diagnostics as session tests.
• Bump the K8s version matrix to 1.32.11, 1.33.7, 1.34.3, 1.35.0 and the SPARK_OPERATOR_VERSION fallback to 2.5.0.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #611

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[adibmbrk]

/assign @tariq-hasan

[tariq-hasan]

/ok-to-test
/retest

[tariq-hasan]

Hi @adibmbrk! Thanks for raising the PR. I have added a few comments.

[tariq-hasan]

We should not remove run_in_cluster.py, Dockerfile.spark-e2e-runner and connect_existing_session.py as in-cluster testing is important.

[tariq-hasan]

Let's repurpose this workflow file to follow the approach taken by the Trainer repo.

sdk/.github/workflows/test-e2e.yaml

Lines 1 to 82 in 3febcd7

	name: E2E Test
	on: pull_request
	concurrency:
	group: ${{ github.workflow }}-${{ github.ref }}
	cancel-in-progress: true
	jobs:
	e2e-test:
	name: E2E Test
	runs-on: oracle-vm-16cpu-64gb-x86-64
	defaults:
	run:
	working-directory: . # Execute make from the root of the SDK repo
	strategy:
	fail-fast: false
	matrix:
	kubernetes-version: ["1.32.3", "1.33.1", "1.34.0", "1.35.0"]
	trainer-ref: ["master"]
	steps:
	- name: Checkout Kubeflow SDK repository
	uses: actions/checkout@v6
	- name: Checkout Kubeflow Trainer repository
	uses: actions/checkout@v6
	with:
	repository: kubeflow/trainer
	ref: ${{ matrix.trainer-ref }}
	path: trainer
	- name: Setup Python
	uses: actions/setup-python@v7
	with:
	python-version: 3.11
	- name: Install Python dependencies
	run: |
	echo "Installing Kubeflow SDK from source with the Notebook dependencies"
	make install-dev extras=docker groups="--no-default-groups --group notebooks"
	# Trainer's hack/e2e-run-notebook.sh calls `papermill` from PATH
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	- name: Setup cluster
	run: |
	cd ./trainer
	make test-e2e-setup-cluster \
	K8S_VERSION=${{ matrix.kubernetes-version }}
	- name: Run e2e test for example Notebooks
	run: |
	mkdir -p artifacts/notebooks # Create the output directory
	cd ./trainer
	# Execute make commands, passing notebook paths and output locations
	make test-e2e-notebook \
	NOTEBOOK_INPUT=./examples/pytorch/image-classification/mnist.ipynb \
	NOTEBOOK_OUTPUT=../artifacts/notebooks/${{ matrix.kubernetes-version }}_mnist.ipynb \
	PAPERMILL_TIMEOUT=900
	make test-e2e-notebook \
	NOTEBOOK_INPUT=./examples/pytorch/question-answering/fine-tune-distilbert.ipynb \
	NOTEBOOK_OUTPUT=../artifacts/notebooks/${{ matrix.kubernetes-version }}_fine-tune-distilbert.ipynb \
	PAPERMILL_TIMEOUT=900
	make test-e2e-notebook \
	NOTEBOOK_INPUT=./examples/local/local-container-mnist.ipynb \
	NOTEBOOK_OUTPUT=../artifacts/notebooks/${{ matrix.kubernetes-version }}_local-container-mnist.ipynb \
	PAPERMILL_TIMEOUT=900
	make test-e2e-notebook \
	NOTEBOOK_INPUT=./examples/local/local-training-mnist.ipynb \
	NOTEBOOK_OUTPUT=../artifacts/notebooks/${{ matrix.kubernetes-version }}_local-training-mnist.ipynb \
	PAPERMILL_TIMEOUT=900
	- name: Upload Artifacts to GitHub
	uses: actions/upload-artifact@v6
	if: always() # Ensure artifacts are uploaded even if previous steps fail
	with:
	name: ${{ matrix.kubernetes-version }}
	path: ./artifacts/notebooks/* # Path relative to the workspace root
	retention-days: 1 #

I don't think we'd want a hand-crafted cluster creation setup for Spark SDK - we should delegate this process to the corresponding upstream spark operator project. The Makefile already supports cluster lifecycle management so let's centralize and reuse that logic instead of making the SDK own setup for a control plane it doesn't ship - given that this cluster setup is already part of the spark operator e2e and we'd want to ensure reproducibility on both the client-side and server-side layers as well as ensure consistency between the client-side components.

https://github.com/kubeflow/spark-operator/blob/cfc9cbaa571330df17140f1bc8e6560e566f33e2/Makefile#L332-L359