Fix K8s executor pod_override stringified without cncf provider#67895
Open
ephraimbuddy wants to merge 1 commit into
Open
Fix K8s executor pod_override stringified without cncf provider#67895ephraimbuddy wants to merge 1 commit into
ephraimbuddy wants to merge 1 commit into
Conversation
Airflow 3 deployments using the Kubernetes executor with the `kubernetes` package installed but without `apache-airflow-providers-cncf-kubernetes` had their `executor_config` `pod_override` V1Pod silently converted to a string during Dag serialization. The stringified pod was persisted to `serialized_dag` and `task_instance.executor_config`, so affected tasks stayed in `queued` and never ran. `_has_kubernetes()` hardcoded an import of the cncf provider's `PodGenerator` for V1Pod ser/deser. When only an alternative executor package is installed, that import failed, `_has_kubernetes()` cached False, and serialization fell back to `str(pod)`. Airflow 2 had an in-core `PodGenerator` fallback for this case that was removed ahead of the 3.0 release. Serialize and deserialize V1Pod objects directly through `kubernetes.client.ApiClient`, which is all `PodGenerator.serialize_pod` and `deserialize_model_dict` wrap anyway. V1Pod ser/deser now works with the cncf provider, an alternative executor package, or the bare `kubernetes` package installed. The backcompat unpickle path in `ensure_pod_is_valid_after_unpickling` is decoupled the same way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Airflow 3 deployments that run the Kubernetes executor with the
kubernetespackage installed but without
apache-airflow-providers-cncf-kuberneteshad their
executor_configpod_overrideV1Podsilently converted to astring during Dag serialization. The stringified pod was persisted to
serialized_dag._dataandtask_instance.executor_config, so affected tasksstayed in
queuedand never ran. (Airflow 2 handled this case via an in-corePodGeneratorfallback that was removed ahead of the 3.0 release.)Root cause
_has_kubernetes()inserialized_objects.pyhardcoded an import of the cncfprovider's
PodGeneratorforV1Podser/deser. When that provider isn'tinstalled the import fails,
_has_kubernetes()cachesFalse, theV1Podserialization branch short-circuits, and
default_serialization()falls backto
str(var).Fix
Serialize/deserialize
V1Podobjects directly throughkubernetes.client.ApiClient— which is allPodGenerator.serialize_podandPodGenerator.deserialize_model_dictwrap anyway — so airflow-core no longerdepends on any specific Kubernetes-executor provider.
V1Podser/deser nowworks with the cncf provider, an alternative executor package, or just the bare
kubernetespackage installed. The backcompat unpickle path inensure_pod_is_valid_after_unpicklingis decoupled the same way.Tests
Added a regression test that makes
apache-airflow-providers-cncf-kubernetesunimportable and asserts
V1Podstill round-trips throughBaseSerialization. The test blocks the...pod_generatorsubmodule directly(not just the parent package) so it can't be fooled by import ordering when the
module is already cached by a sibling test. Existing K8s serialization tests
(
test_serialize_v1pod_attempts_import_before_serializing, theexecutor_config_podfixtures,TestExecutorConfigType) continue to pass —serialization output is byte-identical to the previous cncf-based path.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines