Skip to content

chore(docs): update e2e README with new batch job examples #760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Goku2099 wants to merge 8 commits into
base: main
Choose a base branch
from
+24 −3
Open

chore(docs): update e2e README with new batch job examples #760

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e6e0495
Addressed feedback
Goku2099 Aug 7, 2026
67c0c2a
Addressed feedback
Goku2099 Aug 7, 2026
baa1bd6
nit fix
Goku2099 Aug 7, 2026
277bfda
nit fix
Goku2099 Aug 7, 2026
1879405
docs(spark): update e2e README with new batch job examples
Goku2099 Aug 14, 2026
0897ac1
precommit
Goku2099 Aug 14, 2026
32612ed
addressed review comments
Goku2099 Aug 30, 2026
fd76203
Restore unchanged Spark docs
Goku2099 Aug 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions test/e2e/spark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,22 @@ End-to-end tests that validate Spark examples execute correctly with Kubernetes

## Test Files

### **test_spark_examples.py** (3 tests)
### **test_spark_examples.py** (7 tests)

Validates that Spark example scripts execute successfully:

**Interactive Sessions**
- `test_spark_connect_simple_example` - Validates spark_connect_simple.py runs without errors
- `test_spark_advanced_options_example` - Validates spark_advanced_options.py runs without errors
- `test_demo_existing_sparkconnect_example` - Validates demo_existing_sparkconnect.py structure (SKIPPED - requires manual port-forward)
- `test_connect_existing_session_example` - Validates connect_existing_session.py (SKIPPED unless `SPARK_E2E_RUN_IN_CLUSTER=1`; requires in-cluster execution)

**Batch Jobs**

@tariq-hasan tariq-hasan Aug 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a line before starting the batch job section to ensure the header renders properly.

Suggested change
**Batch Jobs**
**Batch Jobs**

- `test_batch_job_lifecycle_example` - Validates batch_job_lifecycle.py: submits a `FileJob` (spark_job.py as the remote `file_source`), waits for `COMPLETED`, then exercises `get_job`, `list_jobs` (including a status filter), `get_job_logs`, and `delete_job`
- `test_batch_func_job_lifecycle_example` - Validates batch_func_job_lifecycle.py: submits a `FuncJob` (a Python function run as the Spark app), waits for completion, then exercises the same lifecycle APIs as above
- `test_batch_failed_job_example` - Validates batch_failed_job.py: submits a `FileJob` expected to fail, waits for `FAILED` status, and verifies `get_job`, `get_job_logs`, and `delete_job` still work against a failed job
- `test_batch_job_options_example` - Validates batch_job_options.py: submits a `FileJob` with `Name`, `Labels`, `Annotations`, `NodeSelector`, and `Toleration` options, then verifies those options were applied to the underlying `SparkApplication` CR before deleting the job

`spark_job.py` is not a standalone example — it's the simple Spark application used as the remote `file_source` for the batch job examples (`batch_job_lifecycle.py`, `batch_job_options.py`).

## Prerequisites

Expand Down Expand Up @@ -40,7 +49,18 @@ uv run pytest test/e2e/spark/test_spark_examples.py::TestSparkExamples::test_spa

### Quick Validation (No pytest)
```bash
# Interactive sessions
python3 examples/spark/spark_connect_simple.py
python3 examples/spark/spark_advanced_options.py
python3 examples/spark/demo_existing_sparkconnect.py # requires manual port-forward
python3 examples/spark/connect_existing_session.py # requires an existing Spark Connect session
python3 examples/spark/test_connect_url.py

# Batch job lifecycle
python3 examples/spark/batch_job_lifecycle.py
python3 examples/spark/batch_func_job_lifecycle.py
python3 examples/spark/batch_failed_job.py
python3 examples/spark/batch_job_options.py
```

## Test Configuration
Expand Down Expand Up @@ -98,7 +118,8 @@ E2E tests are integrated into GitHub Actions and run automatically on pull reque
- Python version: 3.11

**Tests:**
- Validates Spark examples execute successfully
- Validates Spark interactive session examples execute successfully
- Validates Spark batch job examples execute successfully
- Creates Kind cluster with Spark Operator
- Runs example validation tests
- Collects logs on failure
Expand Down
Loading