[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks] by wjxiz1992 · Pull Request #15835 · NVIDIA/cudf-spark

wjxiz1992 · 2026-08-31T08:45:28Z

JaCoCo production line coverage: independently measured, not additive across incompatible shim bytecode tuples (sql-plugin +56, Spark 3.3/Scala 2.12; sql-plugin +45, Spark 4.0/Scala 2.13; measured head ad6d7a5a; current head 844a13f7 is a no-conflict main merge with the 19-file PR diff unchanged)

Fixes #15511.

Description

GPU file scans can surface a raw FileNotFoundException after a cached data file is removed. Before Spark 4, this omits Spark's recovery guidance. On Spark 4.x, it also loses structured-error parity with CPU execution: the GPU path should preserve Spark's error condition and the exact missing PATH parameter.

This change handles both generations directly in the existing fix:

Spark 3.x retains the V1 REFRESH TABLE/recreate guidance and the V2 recreate-only guidance.
Spark 4.x uses Spark's structured missing-file error, preserving the CPU error condition and exact missing path for both V1 and V2 scans.
Direct and direct ExecutionException-wrapped missing-file failures are converted; unrelated exceptions and deeper wrapper chains are left unchanged.
Missing paths are attached where the individual file is known, including coalescing and multithreaded reader boundaries, then preserved through GpuDataSourceRDD and V2 reader factories.

The three inherited Spark tests excluded by #15511 are re-enabled, and focused RAPIDS coverage exercises the relevant paths:

RapidsMetadataCacheV1Suite: SPARK-16336,SPARK-27961 Suggest fixing FileNotFoundException maps to Spark MetadataCacheSuite.scala lines 41-63.
RapidsMetadataCacheV1Suite: SPARK-16337 temporary view refresh maps to Spark MetadataCacheSuite.scala lines 72-99.
RapidsMetadataCacheV2Suite: SPARK-16336,SPARK-27961 Suggest fixing FileNotFoundException maps to Spark MetadataCacheSuite.scala lines 41-63.
Spark 3.x missing ORC file includes recovery guidance - COALESCING/MULTITHREADED verifies the V1/V2-specific guidance.
Spark 4.x MissingFileStructuredErrorSuite covers V1 and V2 with both COALESCING and MULTITHREADED, comparing the CPU/GPU error condition and missing-path parameter and verifying GPU scan-plan evidence.
FileSystemBytesReadTrackerSuite: GPU datasource RDD enriches next() missing-file failures - direct V2/wrapped V1 directly covers reader failures and the metrics finally path.

Test	Tier	Action	Maven result	Status
V1 SPARK-16336/SPARK-27961 missing-file guidance	T1	Fix production exception guidance and remove exclusion	PASSED	RECOVERED
V1 SPARK-16337 temporary view refresh	T1	Add V1 refresh guidance and remove exclusion	PASSED	RECOVERED
V2 SPARK-16336/SPARK-27961 missing-file guidance	T1	Add V2 recreate guidance and remove exclusion	PASSED	RECOVERED
Spark 4.x V1/V2 structured-error parity	T1	Preserve Spark condition and exact missing path	PASSED	COVERED

Original Spark source: https://github.com/apache/spark/blob/f74867bddfbcdd4d08076db36851e88b15e66556/sql/core/src/test/scala/org/apache/spark/sql/MetadataCacheSuite.scala#L41-L99

Local validation:

Spark 3.3 focused suites: Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0; All tests passed; BUILD SUCCESS.
Spark 4.0 focused structured-error suite, ANSI off/on: each run reported Tests: succeeded 10, failed 0, canceled 0, ignored 0, pending 0; BUILD SUCCESS.
Shim coverage check: No Origin.context leaks in shared source; shim signature coverage consistent across peers.
Spark 3.4 sql-plugin compile: BUILD SUCCESS.
Spark 4.2 sql-plugin compile: BUILD SUCCESS.
Spark 4.0 Databricks 17.3 validation could not reach affected-source compilation because spark-parent_2.13:4.0.0-databricks-173 is unavailable locally.
Spark 5.0 validation could not reach final affected-source compilation because rapids-4-spark-private_2.13:spark500:26.10.0-SNAPSHOT is unavailable locally.
JaCoCo production fix-line intersection against the current PR diff: sql-plugin +56 of 140 added production lines on Spark 3.3/Scala 2.12, and sql-plugin +45 of 140 on Spark 4.0/Scala 2.13. These measurements use incompatible classfiles and are intentionally reported independently rather than summed. The final copyright-only commit is bytecode-neutral.

Performance impact: a local reader-wrapper microbenchmark measured a 0.705 ns/row baseline median and 0.711 ns/row wrapped median, a +0.006 ns/row (+0.81%) delta. The successful path adds one delegating wrapper per partition and try/catch boundaries around next/get, with no additional filesystem access, metadata reads, or GPU operations.

AI assistance: The change and PR description were prepared with Codex assistance and reviewed by the author before submission.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Allen Xu <allxu@nvidia.com>

greptile-apps · 2026-08-31T09:18:16Z

Greptile Summary

This PR preserves missing-file path context across asynchronous GPU readers and converts missing-file failures into Spark-version-appropriate recovery guidance or structured errors.

Adds path-carrying missing-file exceptions at ORC, Parquet, Avro, coalescing, and multithreaded reader boundaries.
Adds Spark 3 recovery-hint and Spark 4 structured-error shims.
Extends V1/V2 coverage and re-enables inherited metadata-cache tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuFileNotFoundException.scala	Introduces a private FileNotFoundException subtype that carries the owning path while preserving the original failure as its cause.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/io/async/AsyncRunners.scala	Adds composable failure transformers around asynchronous runner bodies while retaining post-hook execution in finally.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuMultiFileReader.scala	Attaches the current file path to direct missing-file failures before they cross asynchronous execution boundaries.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala	Converts direct and one-level wrapped missing-file failures using the active input partition while preserving metrics and reader cleanup.
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/MissingFileErrorShim.scala	Adds Spark 3 V1/V2-specific missing-file recovery guidance while avoiding duplicate enrichment.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/MissingFileErrorShim.scala	Wraps Spark 4 readers and delegates known paths to Spark's structured missing-file error construction.
tests/src/test/spark400/scala/com/nvidia/spark/rapids/MissingFileStructuredErrorSuite.scala	Verifies CPU/GPU condition and missing-path parity for Spark 4 V1 and V2 coalescing and multithreaded scans.

Sequence Diagram

sequenceDiagram
  participant Scan as GPU file scan
  participant Async as Async reader
  participant RDD as GpuDataSourceRDD
  participant Shim as MissingFileErrorShim
  participant User as Spark caller
  Scan->>Async: Read planned file
  Async-->>Async: FileNotFoundException
  Async-->>RDD: Path-carrying failure
  RDD->>Shim: Convert(path, original error, V1/V2 hint)
  alt Spark 3.x
    Shim-->>User: FileNotFoundException with recovery guidance
  else Spark 4.x
    Shim-->>User: FAILED_READ_FILE.FILE_NOT_EXIST with PATH
  end

_{Reviews (8): Last reviewed commit: "Fix GpuDataSourceRDD import order" | Re-trigger Greptile}

Copilot

Pull request overview

This pull request restores Spark-equivalent missing-file recovery guidance for GPU ORC scans when cached data files are deleted, aligning GPU error messaging with Spark’s V1/V2 expectations and re-enabling previously excluded upstream MetadataCache tests.

Changes:

Add recovery-hint enrichment for FileNotFoundException (including ExecutionException-wrapped cases) in GpuDataSourceRDD, with a switch to include the V1 REFRESH TABLE hint.
Plumb includeRefreshHint = true from V1 file-source scans (GpuFileSourceScanExec) into GpuDataSourceRDD.
Re-enable the previously excluded Spark 3.3 MetadataCache suites and add RAPIDS-focused tests across COALESCING/MULTITHREADED ORC readers.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Re-enables the previously excluded MetadataCache tests now that GPU guidance is expected to match Spark.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsMetadataCacheSuite.scala	Adds RAPIDS-specific tests asserting V1 vs V2 recovery-hint content across ORC reader modes.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileSourceScanExec.scala	Passes `includeRefreshHint = true` for V1 file-source scans when constructing `GpuDataSourceRDD`.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala	Enriches missing-file exceptions with Spark-like recovery guidance, configurable for V1 vs V2 hint text.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

wjxiz1992 · 2026-09-01T03:16:16Z

+          }
      }

      override def next(): Object = {


Performance impact: successful next() calls retain the existing iterator work and metrics-finally path; the added handlers allocate only when a missing-file exception is thrown. Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-08-31T10:46:23Z

build

thirtiseven · 2026-09-01T05:21:51Z

+          "You can explicitly invalidate the cache in Spark by " +
+          "recreating the Dataset/DataFrame involved."
+      }
+      val enrichedException = new FileNotFoundException(s"$message\n$recoveryHint")


Could we make this translation version-aware before merging? This source is shared by every supported Spark build, but starting with Spark 4.0 the CPU file readers translate missing files through FileDataSourceV2.attachFilePath into a SparkException with condition FAILED_READ_FILE.FILE_NOT_EXIST and a path parameter. MetadataCacheSuite now asserts that structured error.

This branch always returns a plain FileNotFoundException with the Spark 3.x message, while the only end-to-end coverage is spark330 and the shared unit test explicitly expects that raw type. As a result, Spark 4.x GPU scans would still diverge from CPU behavior. I think this needs a shimmed translation with the owning file path, plus representative 4.x V1/V2 coverage.

As it's a SparkUT fix, I'm fine with doing it in a follow-up.

Agreed. I’ll keep this SparkUT-focused PR scoped to Spark 3.x recovery guidance and handle Spark 4.x structured-error parity separately.

ok, please file an issue to track if so.

Performance impact: compile-time-only API tightening; runtime policy values and exception-path behavior are unchanged. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven

LGTM

wjxiz1992 · 2026-09-01T07:42:45Z

build

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-01T09:47:08Z

build

wjxiz1992 · 2026-09-01T10:00:20Z

build

thirtiseven · 2026-09-01T10:06:10Z

+
+  Seq(("V1", true, "orc"), ("V2", false, "")).foreach {
+    case (sourceName, useV1, v1Sources) =>
+      Seq(RapidsReaderType.COALESCING, RapidsReaderType.MULTITHREADED).foreach { readerType =>


Could we add a V1 PERFILE case here and route that path through MissingFileErrorShim? On 400db173, GpuFileSourceScanExec sends PERFILE reads through Spark321PlusDBShims.getFileScanRDD, which returns the copied GpuFileScanRDD. That RDD still produces a plain Spark 3-style FileNotFoundException, so DBR 17.3 V1 PERFILE scans will not report FAILED_READ_FILE.FILE_NOT_EXIST.

thirtiseven · 2026-09-01T10:06:12Z

+            val runner = getBatchRunner(tc, file, outLocal, blocks, offset, batchContext)
+            runner.addFailureTransformer {
+              case error: FileNotFoundException =>
+                GpuFileNotFoundException(file.toString, error)


file is a Hadoop Path here. For a local URI, Path.toString normalizes file:///tmp/x to file:/tmp/x, while Spark's structured error and the new test use the full URI. If the failure surfaces during the asynchronous block copy, the GPU path parameter can therefore differ from CPU. Could we carry the original URI or use file.toUri.toString, and cover this failure point?

thirtiseven · 2026-09-01T10:06:14Z

+      createReader: => PartitionReader[T]): PartitionReader[T] = {
+    val reader = withStructuredMissingFile(partition)(createReader)
+    new PartitionReader[T] {
+      override def next(): Boolean = withStructuredMissingFile(partition)(reader.next())


This is now on the successful Spark 4/5 read path: every next() and get() goes through this wrapper and the call-by-name helper, rather than adding work only when an exception occurs. That makes the current “cold exception path / Performance: Not required” justification incomplete. Could we avoid the call-by-name helper on these hot calls and update the performance rationale with focused evidence, or provide focused performance validation?

Signed-off-by: Allen Xu <allxu@nvidia.com>

Resolve the GpuDataSourceRDD conflict by preserving the missing-file conversion while adopting the upstream reader lifecycle and custom metrics integration. Performance impact: the merge adds only exception-path conversion around hasNext/next and one input-partition reference update per reader; the successful row/batch path retains upstream ReaderIterator metric and close behavior. Signed-off-by: Allen Xu <allxu@nvidia.com>

Signed-off-by: Allen Xu <allxu@nvidia.com>

Fix GPU missing-file recovery guidance

07c1035

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 added the bug Something isn't working label Aug 31, 2026

wjxiz1992 requested review from gerashegalov and thirtiseven August 31, 2026 08:45

wjxiz1992 self-assigned this Aug 31, 2026

wjxiz1992 changed the title ~~[WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance~~ [WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance [databricks] Aug 31, 2026

wjxiz1992 changed the title ~~[WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks] Aug 31, 2026

wjxiz1992 added the test Only impacts tests label Aug 31, 2026

wjxiz1992 marked this pull request as ready for review August 31, 2026 09:15

Copilot AI lite review requested due to automatic review settings August 31, 2026 09:15

Copilot started reviewing on behalf of wjxiz1992 August 31, 2026 09:16 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Handle missing files thrown from datasource next

234f8dc

Performance impact: successful next() calls retain the existing iterator work and metrics-finally path; the added handlers allocate only when a missing-file exception is thrown. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala Outdated

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 1, 2026

Require explicit missing-file recovery policy

98650a4

Performance impact: compile-time-only API tightening; runtime policy values and exception-path behavior are unchanged. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven previously approved these changes Sep 1, 2026

View reviewed changes

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks] Sep 1, 2026

Fix Spark 4 missing-file structured errors

c1e1a06

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 dismissed thirtiseven’s stale review via c1e1a06 September 1, 2026 09:29

Fix AsyncRunners copyright year

ad6d7a5

Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven reviewed Sep 1, 2026

View reviewed changes

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 3, 2026

Merge branch 'main' into codex/fix-15511-orc-missing-file-guidance

844a13f

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks] Sep 3, 2026

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 3, 2026

wjxiz1992 added 2 commits September 4, 2026 08:04

Fix GpuDataSourceRDD import order

43d7940

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks] Sep 4, 2026

Uh oh!

Conversation

wjxiz1992 commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Aug 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thirtiseven left a comment

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wjxiz1992 commented Aug 31, 2026 •

edited

Loading

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading