Skip to content

Fix K8s executor pod_override stringified without cncf provider#67895

Open
ephraimbuddy wants to merge 1 commit into
apache:mainfrom
astronomer:afl-32-decouple-v1pod-serialization
Open

Fix K8s executor pod_override stringified without cncf provider#67895
ephraimbuddy wants to merge 1 commit into
apache:mainfrom
astronomer:afl-32-decouple-v1pod-serialization

Conversation

@ephraimbuddy
Copy link
Copy Markdown
Contributor

Airflow 3 deployments that run the Kubernetes executor with the kubernetes
package installed but without apache-airflow-providers-cncf-kubernetes
had their executor_config pod_override V1Pod silently converted to a
string during Dag serialization. The stringified pod was persisted to
serialized_dag._data and task_instance.executor_config, so affected tasks
stayed in queued and never ran. (Airflow 2 handled this case via an in-core
PodGenerator fallback that was removed ahead of the 3.0 release.)

Root cause

_has_kubernetes() in serialized_objects.py hardcoded an import of the cncf
provider's PodGenerator for V1Pod ser/deser. When that provider isn't
installed the import fails, _has_kubernetes() caches False, the V1Pod
serialization branch short-circuits, and default_serialization() falls back
to str(var).

Fix

Serialize/deserialize V1Pod objects directly through
kubernetes.client.ApiClient — which is all PodGenerator.serialize_pod and
PodGenerator.deserialize_model_dict wrap anyway — so airflow-core no longer
depends on any specific Kubernetes-executor provider. V1Pod ser/deser now
works with the cncf provider, an alternative executor package, or just the bare
kubernetes package installed. The backcompat unpickle path in
ensure_pod_is_valid_after_unpickling is decoupled the same way.

Tests

Added a regression test that makes apache-airflow-providers-cncf-kubernetes
unimportable and asserts V1Pod still round-trips through
BaseSerialization. The test blocks the ...pod_generator submodule directly
(not just the parent package) so it can't be fooled by import ordering when the
module is already cached by a sibling test. Existing K8s serialization tests
(test_serialize_v1pod_attempts_import_before_serializing, the
executor_config_pod fixtures, TestExecutorConfigType) continue to pass —
serialization output is byte-identical to the previous cncf-based path.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

Airflow 3 deployments using the Kubernetes executor with the `kubernetes`
package installed but without `apache-airflow-providers-cncf-kubernetes`
had their `executor_config` `pod_override` V1Pod silently converted to a
string during Dag serialization. The stringified pod was persisted to
`serialized_dag` and `task_instance.executor_config`, so affected tasks
stayed in `queued` and never ran.

`_has_kubernetes()` hardcoded an import of the cncf provider's
`PodGenerator` for V1Pod ser/deser. When only an alternative executor
package is installed, that import failed, `_has_kubernetes()` cached
False, and serialization fell back to `str(pod)`. Airflow 2 had an
in-core `PodGenerator` fallback for this case that was removed ahead of
the 3.0 release.

Serialize and deserialize V1Pod objects directly through
`kubernetes.client.ApiClient`, which is all `PodGenerator.serialize_pod`
and `deserialize_model_dict` wrap anyway. V1Pod ser/deser now works with
the cncf provider, an alternative executor package, or the bare
`kubernetes` package installed. The backcompat unpickle path in
`ensure_pod_is_valid_after_unpickling` is decoupled the same way.
@ephraimbuddy ephraimbuddy added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label Jun 2, 2026
@ephraimbuddy ephraimbuddy added this to the Airflow 3.2.3 milestone Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:DAG-processing backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant