feat(spark)!: delegate SparkApplication driver service account to the Spark Operator - #776
feat(spark)!: delegate SparkApplication driver service account to the Spark Operator#776adibmbrk wants to merge 10 commits into
Conversation
…tion driver The Spark Operator now provides a fallback service account mechanism for the SparkApplication driver spec (kubeflow/spark-operator#3092), so the SDK no longer needs to hardcode `spark-operator-spark` as the default. This keeps the SDK decoupled from operator cluster-admin concerns while still allowing callers to override the service account via `Driver.service_account`. Signed-off-by: adibmbrk <adibmbrk@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
🎉 Welcome to the Kubeflow SDK! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
|
/ok-to-test |
|
/retitle feat(spark)!: delegate SparkApplication driver service account to the Spark Operator |
google-oss-prow
Bot
changed the title
tariq-hasan
left a comment
tariq-hasan
left a comment
There was a problem hiding this comment.
Hi @adibmbrk! I have added a few comments.
|
/assign |
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
Co-authored-by: Tariq Hasan <mmtariquehsn@gmail.com> Signed-off-by: Adib Mubarak <121048412+adibmbrk@users.noreply.github.com>
|
/retest |
get_spark_job_driver_spec previously ignored driver.service_account entirely, silently dropping any custom value set on Driver. Wire it through to SparkV1beta2DriverSpec, falling back to unset when no driver or service account is provided. Signed-off-by: adibmbrk <adibmbrk@gmail.com>
|
Hi @tariq-hasan @Goku2099! Thank you for the review. I have addressed all your comments. Let me know if further changes are needed. |
This reverts commit a739415. Signed-off-by: adibmbrk <adibmbrk@gmail.com>
|
Thanks! |
|
@andreyvelich I am wondering how we should move this PR forward. This change depends on kubeflow/spark-operator#3049 which has been completed recently. As there is an upstream dependency that has not been released yet I am thinking if we'd need a release note for the breaking change that would basically only work for the next operator release - v2.6.0. |
What this PR does / why we need it:
Removes the hardcoded
spark-operator-sparkdefault service account for theSparkApplicationdriver spec. The Spark Operator now supports a fallback service account mechanism for the driver (kubeflow/spark-operator#3092, merged), so the SDK no longer needs to set this default itself.get_spark_job_driver_specnow leavesservice_accountunset unless a caller explicitly provides one viaDriver.service_account, keeping the SDK decoupled from operator cluster-admin concerns.Which issue(s) this PR fixes:
Fixes #616
Checklist: