Adding support for DML writes to DV enabled tables

[sdrp713]

Fixes #12577.

Description

This PR adds GPU acceleration for Apache Delta Lake DELETE, UPDATE, and MERGE commands on deletion vector enabled tables.

With deletion vectors, existing Parquet rows are logically removed by recording their physical row indexes in a DV. Updated or inserted rows are written to new Parquet files. This PR accelerates the data intensive parts of that workflow on the GPU while retaining Delta’s existing transaction and deletion vector semantics.

Implementation

The implementation accelerates:

• Scanning candidate Parquet files and applying existing deletion vectors.
• Producing physical row indexes for rows in their original source files.
• Evaluating DELETE, UPDATE, and MERGE predicates.
• Performing the joins and aggregations required by MERGE.
• Grouping modified physical row indexes by source file.
• Writing updated and inserted rows to new Parquet files.

Collision free dictionary IDs are used in GPU aggregations instead of carrying full file paths or relying on hashes. This reduces GPU memory and shuffle overhead while preserving an exact mapping back to each Delta AddFile.

Candidate files that already have deletion vectors are also supported. The GPU scan returns physical row indexes after applying the existing DV, and newly modified positions are combined with the existing deleted positions before the updated DV is written.

Sparse RoaringBitmap construction, DV serialization, Delta action construction, and optimistic transaction commit remain on the CPU. These are relatively smaller operations compared to the larger scans, joins, aggregations, and Parquet writes that remain GPU accelerated.

Performance

Performance was measured on the Spark2A A100 cluster using Spark 3.5.7 and Delta Lake 3.3.1.

The benchmark used approximately 365 days of the SF3000 NDS store_sales dataset. Deterministic predicates modified approximately 10% of the target rows. Each command was measured using five independent CPU runs and five independent GPU runs, with zero warmup iterations and cluster reuse.

Command	CPU mean	GPU mean	Speedup
DELETE	16.449 s	13.887 s	1.184×
UPDATE	15.276 s	7.637 s	2.000×
MERGE	72.611 s	42.159 s	1.722×

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR adds GPU acceleration for DELETE, UPDATE, and MERGE against Delta tables using deletion vectors while retaining Delta’s transaction and DV serialization paths on the CPU.

• Preserves physical Parquet row indexes through GPU scans and schema evolution.
• Adds GPU grouping of modified row positions and CPU construction of updated deletion-vector actions.
• Introduces version-specific Delta 3.3, 4.0, and 4.1 shims for DV-enabled DML.
• Expands CPU/GPU parity coverage for deletion-vector DELETE, UPDATE, and MERGE workflows.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase.scala	Extends Delta Parquet scans to generate explicit physical row indexes and preserve internal metadata columns across mapped schemas.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase2.scala	Preserves cuDF-provided physical row indexes through chunked and non-chunked schema evolution.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/rapids/GpuDeletionVectorBitmapGenerator.scala	Adds GPU-assisted grouping of touched physical row indexes for subsequent CPU deletion-vector construction.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/rapids/GpuDeleteCommandBase.scala	Enables the deletion-vector execution path for accelerated Delta DELETE operations.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/rapids/GpuUpdateCommandBase.scala	Enables accelerated UPDATE processing while producing touched-file deletion-vector actions.
delta-lake/common/src/main/delta-40x-41x/scala/org/apache/spark/sql/delta/rapids/GpuMergeIntoCommand.scala	Integrates deletion-vector processing into the newer Delta MERGE execution path.
delta-lake/delta-33x/src/main/scala/org/apache/spark/sql/delta/rapids/delta33x/GpuMergeIntoCommand.scala	Adapts Delta 3.3 MERGE execution to emit deletion-vector updates and rewritten rows.
integration_tests/src/main/python/delta_lake_utils.py	Expands shared Delta test utilities for creating, inspecting, and validating deletion-vector-enabled tables.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Delta DELETE / UPDATE / MERGE] --> B[GPU candidate-file scan]
  B --> C[Apply existing deletion vectors]
  C --> D[Preserve physical row indexes]
  D --> E[GPU predicate, join, and aggregation work]
  E --> F[Group modified indexes by source file]
  F --> G[CPU bitmap and DV serialization]
  G --> H[Delta action construction]
  H --> I[Optimistic transaction commit]

Loading

_{Reviews (3): Last reviewed commit: "Fix build errors" | Re-trigger Greptile}