Skip to content

expect_column_pair_values_a_to_be_greater_than_b returns opposite verdicts on pandas vs sql engines for null pairs (default ignore_row_if path) #12264

Closed as duplicate

Description

@ptimizeroracle

describe the bug

expect_column_pair_values_a_to_be_greater_than_b gives opposite verdicts on pandas vs sql execution engines for the same data when nulls are involved: single-null pairs are counted as unexpected values on pandas and silently dropped on sqlite (and presumably other sqlalchemy engines, since it falls out of sql null comparison semantics).

reproducer (gx 1.23.0, current main):

import pandas as pd, sqlite3, great_expectations as gx, tempfile, pathlib
from great_expectations.expectations.core import ExpectColumnPairValuesAToBeGreaterThanB

df = pd.DataFrame({"a": [1, 5, None, 7], "b": [2, 3, 4, None]})
tmp = pathlib.Path(tempfile.mkdtemp()) / "ge.db"
con = sqlite3.connect(tmp)
df.to_sql("t", con, index=False)
con.close()

ctx = gx.get_context()
ds = ctx.data_sources.add_sqlite(name="ds", connection_string=f"sqlite:///{tmp}")
asset = ds.add_table_asset(name="t", table_name="t")
sbatch = asset.get_batch(batch_request=asset.build_batch_request())
pbatch = (
    ctx.data_sources.add_pandas(name="pds")
    .add_dataframe_asset(name="df")
    .add_batch_definition_whole_dataframe("w")
    .get_batch(batch_parameters={"dataframe": df})
)

e = ExpectColumnPairValuesAToBeGreaterThanB(column_A="a", column_B="b", or_equal=True, mostly=0.5)
pbatch.validate(e).success   # False, unexpected_count=3 (single-null rows counted)
sbatch.validate(e).success   # True,  unexpected_count=1 (single-null rows dropped)

the same divergence without mostly: partial_unexpected_list is [(1.0, 2.0), (nan, 4.0), (7.0, nan)] on pandas vs [(1.0, 2.0)] on sqlite.

with mostly=0.5 the two engines return opposite success values for identical inputs, so a checkpoint that passes in one environment fails in the other purely from null handling.

ignore_row_if defaults to both_values_are_missing in the docs, so pandas appears to follow the documented default while the sql implementation behaves as if it were either_value_is_missing. related closed issues (#9748) cover explicit ignore_row_if values misbehaving on pandas, not the default-path cross-engine split.

expected behavior

the default null policy should be the same across execution engines: either count single-null pairs as unexpected everywhere (pandas behavior, matches the documented default) or drop them everywhere. happy to look at the sqlalchemy map provider if maintainers confirm which semantics are intended.

Activity

  1. joshua-stauffer commented on Sep 29, 2026

    @joshua-stauffer
    Collaborator

    thanks for the bug report @ptimizeroracle! this was previously reported in #12252, so I'm closing it as duplicate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions