chore(docs): update e2e README with new batch job examples by Goku2099 · Pull Request #760 · kubeflow/sdk

Goku2099 · 2026-08-14T06:20:11Z

Description:

Updates test/e2e/spark/README.md to reflect the batch job examples added under KEP-107 (batch_job_lifecycle.py, batch_func_job_lifecycle.py, batch_failed_job.py, batch_job_options.py, spark_job.py).

Adds descriptions for all interactive session and batch job examples under Test Files
Documents the batch job RBAC/ServiceAccount prerequisite
Adds troubleshooting entries for batch job RBAC errors and jobs stuck waiting on status
Expands Quick Validation to cover all example scripts, not just spark_connect_simple.py

No test logic changes — docs only.

google-oss-prow · 2026-08-14T06:20:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Goku2099 · 2026-08-14T06:22:03Z

cc @tariq-hasan @Shekharrajak

Goku2099 · 2026-08-29T04:38:37Z

Hi @andreyvelich, could you please take a look at this PR when you get a chance? Thanks!

tariq-hasan

Hi @Goku2099! I have added a few comments.

tariq-hasan · 2026-08-29T00:29:56Z

+4. For batch job examples: a `spark-operator-spark` ServiceAccount in the target namespace with the required `SparkApplication` RBAC permissions.
+


We should remove this portion as kubeflow/spark-operator#3049 has been marked completed.

Suggested change

4. For batch job examples: a `spark-operator-spark` ServiceAccount in the target namespace with the required `SparkApplication` RBAC permissions.

tariq-hasan · 2026-08-29T00:30:30Z

+### Batch job tests fail with RBAC or permission errors
+
+**Cause:** Batch job submission needs a `spark-operator-spark` ServiceAccount with `SparkApplication` RBAC permissions in the target namespace — this is separate from the interactive session prerequisites.
+
+**Solution:** Verify the ServiceAccount and role bindings exist in the test namespace:
+```bash
+kubectl get serviceaccount spark-operator-spark -n spark-test
+kubectl get rolebinding -n spark-test
+```
+


The same applies for this portion.

Suggested change

### Batch job tests fail with RBAC or permission errors

**Cause:** Batch job submission needs a `spark-operator-spark` ServiceAccount with `SparkApplication` RBAC permissions in the target namespace — this is separate from the interactive session prerequisites.

**Solution:** Verify the ServiceAccount and role bindings exist in the test namespace:

```bash

kubectl get serviceaccount spark-operator-spark -n spark-test

kubectl get rolebinding -n spark-test

```

tariq-hasan · 2026-08-30T03:08:22Z

+**Interactive Sessions**
 - `test_spark_connect_simple_example` - Validates spark_connect_simple.py runs without errors
 - `test_spark_advanced_options_example` - Validates spark_advanced_options.py runs without errors
 - `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)


Please remove this test as it not does exist in test/e2e/spark/test_spark_examples.py.

Suggested change

- `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)

tariq-hasan · 2026-08-30T03:09:27Z

 - `test_spark_connect_simple_example` - Validates spark_connect_simple.py runs without errors
 - `test_spark_advanced_options_example` - Validates spark_advanced_options.py runs without errors
 - `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)
+- `connect_existing_session.py` - Connects to an existing Spark Connect session via `base_url` (not confirmed to have an automated test — likely requires a running session to connect to, similar to demo_existing_sparkconnect.py)


Suggested change

- `connect_existing_session.py` - Connects to an existing Spark Connect session via `base_url` (not confirmed to have an automated test — likely requires a running session to connect to, similar to demo_existing_sparkconnect.py)

- `test_connect_existing_session_example` - Validates connect_existing_session.py (SKIPPED unless `SPARK_E2E_RUN_IN_CLUSTER=1`; requires in-cluster execution)

tariq-hasan · 2026-08-30T03:10:04Z

 - `test_spark_advanced_options_example` - Validates spark_advanced_options.py runs without errors
 - `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)
+- `connect_existing_session.py` - Connects to an existing Spark Connect session via `base_url` (not confirmed to have an automated test — likely requires a running session to connect to, similar to demo_existing_sparkconnect.py)
+- `test_connect_url.py` - Tests URL-based connection to Spark Connect (not confirmed to have an automated test)


This should be removed as well.

Suggested change

- `test_connect_url.py` - Tests URL-based connection to Spark Connect (not confirmed to have an automated test)

tariq-hasan · 2026-08-30T03:11:28Z

Suggested change

### **test_spark_examples.py** (7 tests)

tariq-hasan · 2026-08-30T03:12:34Z

 - `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)
+- `connect_existing_session.py` - Connects to an existing Spark Connect session via `base_url` (not confirmed to have an automated test — likely requires a running session to connect to, similar to demo_existing_sparkconnect.py)
+- `test_connect_url.py` - Tests URL-based connection to Spark Connect (not confirmed to have an automated test)
+**Batch Jobs**


We should add a line before starting the batch job section to ensure the header renders properly.

Suggested change

**Batch Jobs**

**Batch Jobs**

tariq-hasan · 2026-08-30T03:24:34Z

 ### Specific Test
 ```bash
 uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_spark_connect_simple_example -v
+uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_batch_job_lifecycle_example -v
 ```


One example is sufficient I believe since we just want to illustrate how to run a specific test.

Suggested change

### Specific Test

```bash

uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_spark_connect_simple_example -v

uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_batch_job_lifecycle_example -v

```

### Specific Test

```bash

uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_spark_connect_simple_example -v

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

Goku2099 · 2026-08-30T13:48:56Z

Hi @tariq-hasan, I’ve addressed the review comments and pushed the updates. Could you please take another look when you get a chance?

tariq-hasan · 2026-08-31T03:04:46Z

Thanks!
/lgtm

Goku2099 · 2026-08-31T10:52:37Z

cc @andreyvelich

google-oss-prow Bot requested review from Electronic-Waste, astefanutti and szaher August 14, 2026 06:20

google-oss-prow Bot added the size/M label Aug 14, 2026

tariq-hasan reviewed Aug 30, 2026

View reviewed changes

Goku2099 added 7 commits August 30, 2026 19:05

Addressed feedback

e6e0495

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

Addressed feedback

67c0c2a

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

nit fix

baa1bd6

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

nit fix

277bfda

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

docs(spark): update e2e README with new batch job examples

1879405

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

precommit

0897ac1

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

addressed review comments

32612ed

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

Goku2099 force-pushed the e2e-readme branch from d43dfd2 to 32612ed Compare August 30, 2026 13:45

Restore unchanged Spark docs

fd76203

Signed-off-by: Sameer_yadav <159073326+Goku2099@users.noreply.github.com>

google-oss-prow Bot added size/S and removed size/M labels Aug 30, 2026

google-oss-prow Bot assigned tariq-hasan Aug 31, 2026

google-oss-prow Bot added the lgtm label Aug 31, 2026

Shekharrajak approved these changes Aug 31, 2026

View reviewed changes

google-oss-prow Bot assigned Shekharrajak Aug 31, 2026

		4. For batch job examples: a `spark-operator-spark` ServiceAccount in the target namespace with the required `SparkApplication` RBAC permissions.

	- `connect_existing_session.py` - Connects to an existing Spark Connect session via `base_url` (not confirmed to have an automated test — likely requires a running session to connect to, similar to demo_existing_sparkconnect.py)
	- `test_connect_existing_session_example` - Validates connect_existing_session.py (SKIPPED unless `SPARK_E2E_RUN_IN_CLUSTER=1`; requires in-cluster execution)

Uh oh!

Conversation

Goku2099 commented Aug 14, 2026

Description:

Uh oh!

google-oss-prow Bot commented Aug 14, 2026

Uh oh!

Goku2099 commented Aug 14, 2026

Uh oh!

Goku2099 commented Aug 29, 2026

Uh oh!

tariq-hasan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Goku2099 commented Aug 30, 2026

Uh oh!

tariq-hasan commented Aug 31, 2026

Uh oh!

Goku2099 commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants