[AutoSparkUT] Recover ORC legacy date read coverage [fast-ut] [databricks] - #15898
Conversation
Signed-off-by: Allen Xu <allxu@nvidia.com>
Greptile SummaryThis test-only change restores inherited Spark coverage for reading Spark 2.4 legacy ORC dates through both Spark 3.3 V1 and V2 source suites.
Confidence Score: 5/5The PR appears safe to merge because it only restores two previously excluded compatibility tests and introduces no identified correctness or security failure. The exclusion removals cause the existing Spark 3.3 V1 and V2 suites to execute the legacy ORC date compatibility case, with no production behavior or expected result changes. Important Files Changed
Reviews (1): Last reviewed commit: "test: recover ORC legacy date read cover..." | Re-trigger Greptile |
There was a problem hiding this comment.
🟢 Approval recommended
The change is limited to removing two stale test exclusions in shim-330 settings and does not introduce production behavior changes.
Pull request overview
This PR restores unit test coverage for the legacy Spark 2.4 ORC DATE compatibility case (SPARK-31238) in the Spark 3.3 (shim-330) test suites by removing stale exclusions now that the underlying GPU/CPU correctness mismatch tracked in #15470 has been addressed elsewhere.
Changes:
- Removed the SPARK-31238 exclusion from
RapidsOrcSourceV1Suitetest settings. - Removed the SPARK-31238 exclusion from
RapidsOrcSourceV2Suitetest settings.
File summaries
| File | Description |
|---|---|
| tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala | Drops the SPARK-31238 exclusion entries so the legacy ORC DATE compatibility test runs again under Rapids ORC V1/V2 suites. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
build |
JaCoCo production line coverage: not fully measurable locally — the available 26.10 nightly tuple is Scala 2.12/shim 350 (
d2f4fe24, b19), whileRapidsOrcSourceV1SuiteandRapidsOrcSourceV2Suiteare shim-330-only and execute 0 tests under shim 350; measured modules: none; N/A:sql-plugin(no compatible shim-330 nightly baseline).Contributes to #15470.
Description
#15470 tracked a GPU/CPU correctness mismatch when reading a Spark 2.4 legacy ORC DATE: GPU returned
1200-01-08while Spark CPU returned1200-01-01. PR #15496 fixed the production ORC calendar-rebasing path, but the inherited Spark test remained excluded from both migrated Spark 3.3 suites.This change removes the two stale exclusions from
RapidsOrcSourceV1SuiteandRapidsOrcSourceV2Suite. The original Spark test semantics and expected value are unchanged:SPARK-31238: compatibility with Spark 2.4 in reading datesOrcSourceSuite.scalalines 488-494Focused GPU validation after rebasing onto current
main(e313d91fdbc22039c360ef69da662df0c5b26f05):Both recovered V1/V2 cases completed successfully. No production code or expected values were changed.
AI assistance: The change and PR description were prepared with Codex assistance and reviewed by the author.
Checklists
Documentation
Testing
(Please provide the names of the existing tests in the PR description.)
Performance