Repository navigation
[BUG] partition_on_multi_column_values rejects batch identifiers whose value is 0, "" or False #12239
Description
What is wrong
PandasDataPartitioner.partition_on_multi_column_values and SparkDataPartitioner.partition_on_multi_column_values decide whether a batch identifier was supplied by testing the value for truthiness:
value = batch_identifiers.get(column_name)
if not value:
raise ValueError(...)An identifier that is present and legitimate — 0, "", False, 0.0 — is therefore reported as missing, and the batch is never read. The message itself contradicts the situation:
ValueError: In order for PandasExecution to `_partition_on_multi_column_values`, all values in
column_names must also exist in batch_identifiers. m was not found in batch_identifiers.
m was found — its value is 0.
Neither neighbouring implementation behaves this way. partition_on_column_value (pandas_data_partitioner.py:168) indexes the dict with no truthiness gate, and SqlAlchemyDataPartitioner.partition_on_multi_column_values (sqlalchemy_data_partitioner.py:317-328) emits column == value for every identifier it is given. So identical partitioner_kwargs partition correctly on a SQL backend and raise on pandas and Spark.
Reproduction
measured on develop (8bfb2b2), Python 3.11.15, pandas 3.0.5; the Spark case on pyspark 3.5.5 with JVM 11:
import pandas as pd
from great_expectations.core.batch_spec import RuntimeDataBatchSpec
from great_expectations.execution_engine import PandasExecutionEngine
df = pd.DataFrame({"y": [2020, 2020, 2019, 0, 0], "m": [1, 0, 0, 0, 2]})
PandasExecutionEngine().get_batch_data(
RuntimeDataBatchSpec(
batch_data=df,
partitioner_method="_partition_on_multi_column_values",
partitioner_kwargs={"column_names": ["m"], "batch_identifiers": {"m": 0}},
)
)Before the fix:
multi-column {"m": 0} -> ValueError: ... m was not found in batch_identifiers.
single-column {"m": 0} -> 3 rows (same file, no gate)
multi-column {"code": ""} -> ValueError / single-column -> 2 rows
multi-column {"flag": False} -> ValueError / single-column -> 2 rows
The Spark partitioner produces the same three ValueErrors on the same data.
The SQLAlchemy partitioner, run against a real (in-memory sqlite) table with the same identifiers, has no such gate:
sqlalchemy identifiers={'y': 2020, 'm': 1} -> rows=1 | WHERE y = ? AND m = ?
sqlalchemy identifiers={'y': 2020, 'm': 0} -> rows=2 | WHERE y = ? AND m = ?
sqlalchemy identifiers={'m': 0} -> rows=4 | WHERE m = ?
sqlalchemy identifiers={'y': 0} -> rows=1 | WHERE y = ?
Expected behaviour
A column name that has an entry in batch_identifiers partitions on that entry's value, whatever it is; only a column name with no entry should raise. Zero-indexed and empty-string partitions are ordinary data — a boolean flag, a code = "" "unknown" bucket, a counter whose first bucket is 0 — and a partitioner that names a column present in the identifiers should not call it absent.
Suggested fix
Test for the key instead of the value (if column_name not in batch_identifiers:) in both engines, which keeps the guard's stated purpose — a column name with no matching identifier — and removes the engine divergence.
Activity
joshua-stauffer commentedon Sep 29, 2026
Hi @feiiiiii5 - thanks for the bug report, but this one isn't possible to reach via the public api. It's internal logic that was deprecated with the v1 release. We'll hold off on making any changes here for now; we may remove or refactor this code in the future. If you encountered this via the public api while validating data with GX, feel free to reopen with the public api repo.