Replace DaskJob pod with Kubernetes Job

[Matt711]

Closes #695. This PR replaces the Pod created from a DaskJob CRD with a Job. This allows us to use some of the existing functionality of K8s Jobs.

[bstadlbauer]

Hey @Matt711!
Would you mind if I try whether things still work with the Flyte dask plugin (this thing) once you're ready?
It makes some assumptions around the job here, in particular the job-runner pod name, etc. So would be nice if this wouldn't be breaking the plugin

cc @jacobtomlinson

[Matt711]

Hey @Matt711! Would you mind if I try whether things still work with the Flyte dask plugin (this thing) once you're ready? It makes some assumptions around the job here, in particular the job-runner pod name, etc. So would be nice if this wouldn't be breaking the plugin

cc @jacobtomlinson

Yeah, no problem!

[jacobtomlinson]

I think we probably want our spec to replace this, rather than the "spec" within "template".

That does mean we need to modify our CRDs to ref a JobSpec instead. That is a breaking change to our CRDs so we might need to think about how to bump the version there.

dask-kubernetes/dask_kubernetes/operator/customresources/daskjob.yaml

Line 65 in 2aa43ed

$ref: 'python://k8s_crd_resolver/schemata/k8s-1.21.1.json#/definitions/io.k8s.api.core.v1.PodSpec'