Skip to content

[BUG] partition_on_multi_column_values rejects batch identifiers whose value is 0, "" or False #12239

Closed

Description

@feiiiiii5

What is wrong

PandasDataPartitioner.partition_on_multi_column_values and SparkDataPartitioner.partition_on_multi_column_values decide whether a batch identifier was supplied by testing the value for truthiness:

value = batch_identifiers.get(column_name)
if not value:
    raise ValueError(...)

An identifier that is present and legitimate — 0, "", False, 0.0 — is therefore reported as missing, and the batch is never read. The message itself contradicts the situation:

ValueError: In order for PandasExecution to `_partition_on_multi_column_values`, all values in
column_names must also exist in batch_identifiers. m was not found in batch_identifiers.

m was found — its value is 0.

Neither neighbouring implementation behaves this way. partition_on_column_value (pandas_data_partitioner.py:168) indexes the dict with no truthiness gate, and SqlAlchemyDataPartitioner.partition_on_multi_column_values (sqlalchemy_data_partitioner.py:317-328) emits column == value for every identifier it is given. So identical partitioner_kwargs partition correctly on a SQL backend and raise on pandas and Spark.

Reproduction

measured on develop (8bfb2b2), Python 3.11.15, pandas 3.0.5; the Spark case on pyspark 3.5.5 with JVM 11:

import pandas as pd
from great_expectations.core.batch_spec import RuntimeDataBatchSpec
from great_expectations.execution_engine import PandasExecutionEngine

df = pd.DataFrame({"y": [2020, 2020, 2019, 0, 0], "m": [1, 0, 0, 0, 2]})
PandasExecutionEngine().get_batch_data(
    RuntimeDataBatchSpec(
        batch_data=df,
        partitioner_method="_partition_on_multi_column_values",
        partitioner_kwargs={"column_names": ["m"], "batch_identifiers": {"m": 0}},
    )
)

Before the fix:

multi-column  {"m": 0}   -> ValueError: ... m was not found in batch_identifiers.
single-column {"m": 0}   -> 3 rows          (same file, no gate)
multi-column  {"code": ""}  -> ValueError   / single-column -> 2 rows
multi-column  {"flag": False} -> ValueError / single-column -> 2 rows

The Spark partitioner produces the same three ValueErrors on the same data.

The SQLAlchemy partitioner, run against a real (in-memory sqlite) table with the same identifiers, has no such gate:

sqlalchemy identifiers={'y': 2020, 'm': 1} -> rows=1 | WHERE y = ? AND m = ?
sqlalchemy identifiers={'y': 2020, 'm': 0} -> rows=2 | WHERE y = ? AND m = ?
sqlalchemy identifiers={'m': 0}            -> rows=4 | WHERE m = ?
sqlalchemy identifiers={'y': 0}            -> rows=1 | WHERE y = ?

Expected behaviour

A column name that has an entry in batch_identifiers partitions on that entry's value, whatever it is; only a column name with no entry should raise. Zero-indexed and empty-string partitions are ordinary data — a boolean flag, a code = "" "unknown" bucket, a counter whose first bucket is 0 — and a partitioner that names a column present in the identifiers should not call it absent.

Suggested fix

Test for the key instead of the value (if column_name not in batch_identifiers:) in both engines, which keeps the guard's stated purpose — a column name with no matching identifier — and removes the engine divergence.

Activity

  1. joshua-stauffer commented on Sep 29, 2026

    @joshua-stauffer
    Collaborator

    Hi @feiiiiii5 - thanks for the bug report, but this one isn't possible to reach via the public api. It's internal logic that was deprecated with the v1 release. We'll hold off on making any changes here for now; we may remove or refactor this code in the future. If you encountered this via the public api while validating data with GX, feel free to reopen with the public api repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions