Skip to content

[AutoSparkUT] Recover ORC legacy date read coverage [fast-ut] [databricks] - #15898

Merged
wjxiz1992 merged 1 commit into
NVIDIA:mainfrom
wjxiz1992:codex/recover-15470-orc-date
Sep 7, 2026
Merged

[AutoSparkUT] Recover ORC legacy date read coverage [fast-ut] [databricks]#15898
wjxiz1992 merged 1 commit into
NVIDIA:mainfrom
wjxiz1992:codex/recover-15470-orc-date

Conversation

@wjxiz1992

Copy link
Copy Markdown
Collaborator

JaCoCo production line coverage: not fully measurable locally — the available 26.10 nightly tuple is Scala 2.12/shim 350 (d2f4fe24, b19), while RapidsOrcSourceV1Suite and RapidsOrcSourceV2Suite are shim-330-only and execute 0 tests under shim 350; measured modules: none; N/A: sql-plugin (no compatible shim-330 nightly baseline).

Contributes to #15470.

Description

#15470 tracked a GPU/CPU correctness mismatch when reading a Spark 2.4 legacy ORC DATE: GPU returned 1200-01-08 while Spark CPU returned 1200-01-01. PR #15496 fixed the production ORC calendar-rebasing path, but the inherited Spark test remained excluded from both migrated Spark 3.3 suites.

This change removes the two stale exclusions from RapidsOrcSourceV1Suite and RapidsOrcSourceV2Suite. The original Spark test semantics and expected value are unchanged:

Focused GPU validation after rebasing onto current main (e313d91fdbc22039c360ef69da662df0c5b26f05):

Tests: succeeded 54, failed 0, canceled 0, ignored 14, pending 0
BUILD SUCCESS

Both recovered V1/V2 cases completed successfully. No production code or expected values were changed.

AI assistance: The change and PR description were prepared with Codex assistance and reviewed by the author.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Signed-off-by: Allen Xu <allxu@nvidia.com>
Copilot AI lite review requested due to automatic review settings September 4, 2026 03:07
@wjxiz1992 wjxiz1992 added the test Only impacts tests label Sep 4, 2026
@wjxiz1992 wjxiz1992 self-assigned this Sep 4, 2026
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This test-only change restores inherited Spark coverage for reading Spark 2.4 legacy ORC dates through both Spark 3.3 V1 and V2 source suites.

  • Removes the stale SPARK-31238 exclusion from RapidsOrcSourceV1Suite.
  • Removes the same exclusion from RapidsOrcSourceV2Suite.
  • Leaves production code and expected test values unchanged.

Confidence Score: 5/5

The PR appears safe to merge because it only restores two previously excluded compatibility tests and introduces no identified correctness or security failure.

The exclusion removals cause the existing Spark 3.3 V1 and V2 suites to execute the legacy ORC date compatibility case, with no production behavior or expected result changes.

Important Files Changed

Filename Overview
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala Re-enables the inherited SPARK-31238 legacy ORC date compatibility test for both V1 and V2 suites; no actionable defect was found.

Reviews (1): Last reviewed commit: "test: recover ORC legacy date read cover..." | Re-trigger Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is limited to removing two stale test exclusions in shim-330 settings and does not introduce production behavior changes.

Pull request overview

This PR restores unit test coverage for the legacy Spark 2.4 ORC DATE compatibility case (SPARK-31238) in the Spark 3.3 (shim-330) test suites by removing stale exclusions now that the underlying GPU/CPU correctness mismatch tracked in #15470 has been addressed elsewhere.

Changes:

  • Removed the SPARK-31238 exclusion from RapidsOrcSourceV1Suite test settings.
  • Removed the SPARK-31238 exclusion from RapidsOrcSourceV2Suite test settings.
File summaries
File Description
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala Drops the SPARK-31238 exclusion entries so the legacy ORC DATE compatibility test runs again under Rapids ORC V1/V2 suites.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@wjxiz1992 wjxiz1992 changed the title [AutoSparkUT] Recover ORC legacy date read coverage [fast-ut] [AutoSparkUT] Recover ORC legacy date read coverage [databricks] [fast-ut] Sep 4, 2026
@wjxiz1992 wjxiz1992 changed the title [AutoSparkUT] Recover ORC legacy date read coverage [databricks] [fast-ut] [AutoSparkUT] Recover ORC legacy date read coverage [fast-ut] [databricks] Sep 4, 2026
@wjxiz1992

Copy link
Copy Markdown
Collaborator Author

build

@wjxiz1992
wjxiz1992 merged commit 3d0d35a into NVIDIA:main Sep 7, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Only impacts tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants