Use returned cuDF Parquet footer for Iceberg writes [fast-ut]

[liurenjie1024]

Related to NVIDIA/cudf#18886.

Description

Iceberg GPU writes currently reopen each Parquet output file after closing it to obtain footer metadata for file metrics and split offsets. This causes an unnecessary read from the output filesystem, which is especially costly for remote storage.

This change uses the new cuDF ParquetTableWriter.closeAndGetFooter() API to return the metadata-only Parquet footer during writer finalization. GpuParquetWriter exposes the returned host buffer to the Iceberg appender, which parses it through Iceberg's shaded Parquet reader without reopening the output file.

The shared columnar writer close path preserves the existing finalization sequence and closes the returned buffer if output finalization fails.

Validation:

• Spark 4.1.1 / Scala 2.13 build: all 19 modules succeeded.
• Spark 4.1.1 with Iceberg 1.11.0 local Hadoop catalog: 432 passed, 19 skipped, 6 xfailed, and 3 xpassed.
• The existing Iceberg integration suite exercises append, overwrite, merge, schema and partition evolution, metrics, and split-offset paths.

This change and PR description were prepared with AI assistance from OpenAI Codex.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Iceberg integration suite with Spark 4.1.1 and Iceberg 1.11.0.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

The PR avoids reopening completed Iceberg Parquet files by obtaining footer metadata directly from the cuDF writer.

• Adds a shared close path that returns an owned resource while preserving output finalization.
• Exposes the cuDF Parquet footer buffer through GpuParquetWriter.
• Parses the returned buffer with Iceberg’s shaded Parquet reader for metrics and split offsets.
• Removes the now-unused Iceberg footer-reopen helper and appender FileIO dependency.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
iceberg/common/src/main/java/com/nvidia/spark/rapids/fileio/iceberg/IcebergFileIO.java	Removes the specialized input-file helper that was used only to reopen newly written files for footer metadata.
iceberg/common/src/main/scala/com/nvidia/spark/rapids/iceberg/parquet/GpuIcebergParquetAppender.scala	Replaces post-close filesystem footer reads with parsing of the host buffer returned during writer finalization.
iceberg/common/src/main/scala/org/apache/iceberg/spark/source/GpuSparkFileWriterFactory.scala	Stops passing a redundant Iceberg FileIO instance into the Parquet appender.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/ColumnarOutputWriter.scala	Factors writer shutdown into preparation and output-finalization stages and adds a resource-safe close-and-return helper.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetFileFormat.scala	Narrows the writer type to ParquetTableWriter and exposes its returned footer buffer to Iceberg.

Sequence Diagram

sequenceDiagram
  participant Iceberg as Iceberg writer
  participant Appender as GpuIcebergParquetAppender
  participant Writer as GpuParquetWriter
  participant cuDF as cuDF ParquetTableWriter
  participant Output as Output stream
  participant Reader as Shaded ParquetFileReader

  Iceberg->>Appender: close()
  Appender->>Writer: closeAndGetFooter()
  Writer->>cuDF: closeAndGetFooter()
  cuDF-->>Writer: HostMemoryBuffer
  Writer->>Output: flush buffered data and close
  Writer-->>Appender: footer buffer
  Appender->>Reader: open(HMBInputFile(buffer))
  Reader-->>Appender: ParquetMetadata
  Appender->>Appender: close footer buffer
  Iceberg->>Appender: metrics() / splitOffsets()
  Appender-->>Iceberg: footer-derived metadata

Loading

_{Reviews (5): Last reviewed commit: "Remove unused Iceberg input file helper" | Re-trigger Greptile}

[liurenjie1024]

build

[res-life]

Please clean IcebergFileIO.newIcebergInputFile if it's dead now.

[liurenjie1024]

Removed IcebergFileIO.newIcebergInputFile; the in-memory footer path left it with no callers. Spark 4.1.1/Scala 2.13 build and scalastyle pass.

[res-life]

LGTM

[liurenjie1024]

build

[gerashegalov]

LGTM

[gerashegalov]

build