Move independent Iceberg helpers to root-safe module [reduced-it] by gerashegalov · Pull Request #15841 · NVIDIA/cudf-spark

gerashegalov · 2026-08-31T21:57:36Z

Related to #15821.

Description

Move a self-contained set of Apache Iceberg helpers from the versioned
iceberg/common source module into the root-safe iceberg-common module:

IcebergDeletionVector;
reader-option and native-row-index utilities;
ThreadConf and SingleFile; and
position-delete field IDs.

Replace the internal GpuIcebergParquetReader and
GpuPositionDeleteFileWriter companion helpers with dedicated root-safe
owners. Fold the implementation-dependent GpuDeleteFilter2 helpers into
GpuDeleteFileInfo, where they remain in the versioned source module.
Together, these changes remove six Iceberg entries from the special
root-layout allowlist.

The catalog and table wrappers remain version-specific because their Spark
catalog interfaces are not source-compatible across all supported versions.
There is no user-facing configuration or behavior change. Algorithms and data
flow are unchanged; only source ownership, final class placement, and internal
companion names change.

Validation on the prepared up-merged branch:

Clean multi-shim dist assembly for the highest patch release in each
supported line:
- Scala 2.12: Spark 3.3.4, 3.4.4, and 3.5.9.
- Scala 2.13: Spark 3.5.9, 4.0.4, 4.1.3, and 4.2.0.
The package-time Iceberg audit passed with all 10 package-private caller
classes at the JAR root across two Scala 2.12 and five Scala 2.13 Iceberg
runtime worlds.
Inspected both assembled distribution JARs: every promoted helper is present
only at the conventional root, with no copy in spark-shared or a selected
Spark-specific directory. The three retired companion classes are absent.
Spark 3.5.9/Scala 2.12 focused Iceberg suites: 28 tests passed and 11
runtime-inapplicable row-lineage cases were canceled.
Spark 4.1.3/Scala 2.13 with Iceberg 1.11: 5 applicable tests passed.
Spark 4.2.0/Scala 2.13 compiled successfully; that line currently selects the
Iceberg stub.
git diff --check passed.

Pre-merge CI provides the exhaustive shim matrix and the
AppClassLoader/extraClassPath Iceberg smoke coverage.

Performance testing is not required. The change adds no per-batch or hot-path
work. Consolidating the delete-filter metadata into GpuDeleteFileInfo may
initialize those small immutable values earlier, which is a negligible
one-time cost.

This pull request was prepared with AI assistance and reviewed by the author
before publication. It also received an independent read-only code review.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(GpuDeleteFilterSuite, GpuPostProcessorSuite,
IcebergDeletionVectorSuite, and RapidsSparkTableSuite.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

greptile-apps · 2026-08-31T22:13:48Z

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, packaging, security, or repository-rule issue remains.

Summary

Promotes deletion-vector, Parquet reader-option, native-row-index, thread-configuration, and position-delete helpers.
Consolidates delete-filter metadata and utilities under GpuDeleteFileInfo.
Removes obsolete companion classes and their special root-layout allowlist entries.
Updates focused post-processor tests to use the new helper owner.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["iceberg/common<br/>version-specific sources"] -->|move independent helpers| B["iceberg-common<br/>root-safe module"]
  B --> C["Distribution JAR root"]
  A -->|call renamed helper owners| B
  D["Special root-layout allowlist"] -->|remove obsolete entries| C

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-08-31T22:26:03Z

build

gerashegalov · 2026-08-31T22:55:07Z

build

NvTimLiu

LGTM from CICD point of view

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

res-life · 2026-09-02T09:25:52Z

      new JHashMap[Integer, Any](),
      expectedSchema,
-      GpuIcebergParquetReader.withNativeRowIndex(shadedSchema),
+      GpuIcebergParquetReaderUtils.withNativeRowIndex(shadedSchema),


[P1] Update the remaining withNativeRowIndex callers

This PR removes object GpuIcebergParquetReader, but lines 1181 and 1218 in this file still call GpuIcebergParquetReader.withNativeRowIndex(...). A clean Spark 3.5 test compile therefore fails with not found: value GpuIcebergParquetReader. Please change those two callers to GpuIcebergParquetReaderUtils.withNativeRowIndex(...) as well.

@res-life updated in 230ec0b

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

res-life

LGTM

gerashegalov · 2026-09-04T14:55:54Z

build

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-09-09T05:36:25Z

build

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-09-09T16:54:33Z

build

Move independent Iceberg helpers to root-safe module

3725ab5

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov mentioned this pull request Aug 31, 2026

[WIP] Unshim Iceberg reader and delete helpers [fast-ut][reduced-it] #15826

Closed

8 tasks

gerashegalov self-assigned this Aug 31, 2026

gerashegalov marked this pull request as ready for review August 31, 2026 22:11

gerashegalov requested a review from a team as a code owner August 31, 2026 22:11

Keep Iceberg catalog wrappers runtime-specific

3ef9f1d

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov requested a review from a team August 31, 2026 23:27

NvTimLiu previously approved these changes Sep 1, 2026

View reviewed changes

NvTimLiu requested a review from a team September 1, 2026 02:56

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

999babe

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

gerashegalov dismissed NvTimLiu’s stale review via 999babe September 1, 2026 04:54

res-life reviewed Sep 2, 2026

View reviewed changes

gerashegalov added 2 commits September 2, 2026 10:50

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

e0a82cd

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Update row-lineage post-processor tests

230ec0b

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

res-life previously approved these changes Sep 4, 2026

View reviewed changes

gerashegalov changed the title ~~Move independent Iceberg helpers to root-safe module [fast-ut][reduced-it]~~ Move independent Iceberg helpers to root-safe module [reduced-it] Sep 4, 2026

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

d3f34b8

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

gerashegalov dismissed res-life’s stale review via d3f34b8 September 4, 2026 19:35

gerashegalov requested review from NvTimLiu and res-life September 4, 2026 21:13

gerashegalov mentioned this pull request Sep 5, 2026

[EPIC] Minimize version-specific production classes #15821

Open

20 tasks

firestarman previously approved these changes Sep 7, 2026

View reviewed changes

gerashegalov added 2 commits September 8, 2026 21:55

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

ed0e8d9

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Fix Iceberg reader utility test after up-merge

f131577

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

3611f1c

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov dismissed firestarman’s stale review via 3611f1c September 9, 2026 14:58

gerashegalov requested a review from firestarman September 9, 2026 16:54

sameerz added the task Work required that improves the product but is not user facing label Sep 9, 2026

gerashegalov mentioned this pull request Sep 10, 2026

Finalize Iceberg root and shim class placement [reduced-it] #15944

Draft

10 tasks

Navigation Menu

Uh oh!

Move independent Iceberg helpers to root-safe module [reduced-it] - #15841

Move independent Iceberg helpers to root-safe module [reduced-it]#15841
gerashegalov wants to merge 9 commits into
NVIDIA:mainfrom
gerashegalov:codex/unshim-iceberg-independent-helpers

gerashegalov commented Aug 31, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading

Summary

Diagram

Uh oh!

gerashegalov commented Aug 31, 2026

Uh oh!

gerashegalov commented Aug 31, 2026

Uh oh!

NvTimLiu left a comment

Uh oh!

res-life Sep 2, 2026

Uh oh!

gerashegalov Sep 2, 2026

Uh oh!

res-life left a comment

Uh oh!

gerashegalov commented Sep 4, 2026

Uh oh!

gerashegalov commented Sep 9, 2026

Uh oh!

gerashegalov commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

gerashegalov commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

gerashegalov commented Aug 31, 2026

Uh oh!

gerashegalov commented Aug 31, 2026

Uh oh!

NvTimLiu left a comment

Choose a reason for hiding this comment

Uh oh!

res-life Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

res-life left a comment

Choose a reason for hiding this comment

Uh oh!

gerashegalov commented Sep 4, 2026

Uh oh!

gerashegalov commented Sep 9, 2026

Uh oh!

gerashegalov commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

gerashegalov commented Aug 31, 2026 •

edited

Loading

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading