Add row tracking preservation tests for MERGE, UPDATE and DELETE on OSS Delta [databricks]

[jtwynne]

Fixes #15906.

Description

The only tests that check the GPU DML commands preserve row tracking ran on Databricks 17.3. The OSS UPDATE and DELETE commands preserve it through UpdateCommand.preserveRowTrackingColumns, and the OSS MERGE command inherits ClassicMergeExecutor.writeAllChanges, which does the same, but nothing exercised them (#15906 first claimed a defect there; it was wrong, the coverage gap is what remains).

• supports_delta_lake_row_tracking() in spark_session.py: Databricks 17.3 and later, or OSS Delta Lake 3.3, which the plugin pairs with Spark 3.5.x and later.
• test_delta_update_preserves_row_tracking and test_delta_delete_preserves_row_tracking (renamed from _db173) are gated on it instead of Databricks 17.3.
• test_delta_merge_preserves_row_tracking: a matched update and an insert into a row-tracked target, run on both engines through assert_gpu_and_cpu_writes_are_equal_collect. For a Delta test the helper runs the GPU side under assert_rapids_delta_write, which asserts the GPU Delta write in the plans it captures, and only the GPU merge command writes that way, so a fallback to the CPU merge command fails the test. The helper then reads both tables back on the CPU with _metadata.row_id and _metadata.row_commit_version and compares them engine to engine (both tables start from the same single file, so the rows that existed before carry the same ids and versions on both engines; the inserted row's id is masked because the file layout decides it and the join-based GPU merge lays out files differently from the CPU). Per engine, the rows that existed before keep their row id, copied rows keep their commit version, the updated row gets the new one, and the inserted row gets a fresh id. The merge's result row is compared between the engines; the commit logs are not, for the same layout reason. The NOT MATCHED BY SOURCE clause is left out because it runs on the GPU only with Delta 4.1 and Databricks 17.3; Support NOT MATCHED BY SOURCE in the GPU MERGE command on Databricks 17.3 [databricks] #15884 has the 17.3 test with that clause.

The capability predicate includes Databricks 17.3, where UPDATE and DELETE preserve row tracking on the GPU and the two tests pass. The 17.3 GPU MERGE on main regenerates row ids (#15884 fixes that and carries the 17.3 merge test), so test_delta_merge_preserves_row_tracking is skipped on Databricks with that reason until #15884 lands.

Tests

On an Azure Standard_NC16as_T4_v3 (one T4, driver 610.57.04), plugin built from this branch, run_pyspark_from_build.sh --delta_lake -m delta_lake -k preserves_row_tracking with the Delta packages, extension and catalog confs from jenkins/spark-tests.sh:

• Apache Spark 4.0.0, Scala 2.13, Delta Lake 4.0.1 (build version 400): 3 passed (test_delta_merge_preserves_row_tracking, test_delta_update_preserves_row_tracking, test_delta_delete_preserves_row_tracking), 0 failed, no WAS LEAKED lines.
• Apache Spark 3.5.6, Scala 2.12, Delta Lake 3.3.0 (build version 356): the same 3 passed, 0 failed, no WAS LEAKED lines.
• Apache Spark 4.1.1, Scala 2.13, Delta Lake 4.1.0 (build version 411): the same 3 passed, 0 failed, no WAS LEAKED lines.
• Re-run after the merge test moved to the standard helper: 3 passed, 0 failed, no WAS LEAKED lines on each of Spark 4.0.0 / Delta 4.0.1, 3.5.6 / 3.3.0 and 4.1.1 / 4.1.0 (same T4 box).
• Databricks 17.3 LTS ML (Standard_NC16as_T4_v3, run_pyspark_from_build.sh --runtime_env=databricks): test_delta_update_preserves_row_tracking and test_delta_delete_preserves_row_tracking passed, test_delta_merge_preserves_row_tracking skipped with the Support NOT MATCHED BY SOURCE in the GPU MERGE command on Databricks 17.3 [databricks] #15884 reason, no WAS LEAKED lines.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR broadens Delta row-tracking preservation coverage across supported OSS Delta Lake and Databricks environments.

• Adds a capability predicate for runtimes that expose Delta row-tracking metadata.
• Extends UPDATE and DELETE preservation tests beyond Databricks 17.3.
• Adds GPU-versus-CPU MERGE coverage for preserved row IDs and commit versions.
• Uses the standard Delta write assertion path and skips the known-broken Databricks MERGE case pending its dedicated fix.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported test-framework and Databricks coverage concerns are resolved, and no new actionable defects remain.

The MERGE test now uses the standard GPU-verified Delta write helper, UPDATE and DELETE retain equivalent verification, and unsupported or known-broken runtime combinations are explicitly gated.

Important Files Changed

Filename	Overview
integration_tests/src/main/python/spark_session.py	Adds a capability predicate covering Databricks 17.3+ and OSS Spark configurations paired with Delta Lake 3.3+.
integration_tests/src/main/python/delta_lake_merge_test.py	Adds standardized CPU/GPU MERGE validation for row ID and commit-version preservation, with the known Databricks case skipped.
integration_tests/src/main/python/delta_lake_update_test.py	Broadens the existing UPDATE row-tracking test using the shared runtime capability predicate.
integration_tests/src/main/python/delta_lake_delete_test.py	Broadens the existing DELETE row-tracking test using the shared runtime capability predicate.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Detect row-tracking support] --> B{Supported runtime?}
    B -- No --> C[Skip preservation tests]
    B -- Yes --> D[Create matching CPU and GPU row-tracked tables]
    D --> E[Run UPDATE or DELETE or MERGE]
    E --> F[Verify GPU Delta execution]
    F --> G[Read row metadata on CPU]
    G --> H[Compare CPU and GPU results]
    H --> I[Assert existing IDs persist and versions change correctly]

Loading

_{Reviews (7): Last reviewed commit: "Merge branch 'main' into oss-merge-row-t..." | Re-trigger Greptile}