[FEA] Support yyyyMMdd under CORRECTED policy [databricks]

[gerashegalov]

Fixes #15588.

Description

The yyyyMMdd format currently falls back to CPU under the CORRECTED time parser policy unless incompatible date formats are enabled. The customer workload described in #15588 was 14.9% faster when the existing GPU path was forced.

This change:

• certifies yyyyMMdd for the fused JNI parser under CORRECTED when spark.rapids.sql.hasExtendedYearValues=false;
• preserves CPU fallback under EXCEPTION so Spark can detect CORRECTED/LEGACY disagreements such as 2024101;
• preserves CPU fallback when extended years may be present, avoiding silent nulls for signed values such as +123450101 and -00010101;
• verifies that date_format(timestamp, 'yyyyMMdd') remains CPU/GPU compatible;
• adds coverage for valid values, strict invalid-input rejection, ANSI behavior, EXCEPTION fallback and disagreement handling, the Spark 4 default CORRECTED policy, and signed extended years.

Local verification:

• mvn -T 64 package -pl sql-plugin -am -DskipTests
• mvn -T 64 package -pl dist,integration_tests -am -DskipTests -Dbuildver=357
• Spark 3.5.7 focused yyyyMMdd GPU integration tests on an NVIDIA Quadro RTX 6000: 24 passed, 1 Spark-4-only test skipped
• Python syntax validation
• git diff --check

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[gerashegalov]

build

[gerashegalov]

build

[greptile-apps]

Greptile Summary

The PR enables GPU handling of packed yyyyMMdd dates under the CORRECTED parser policy while preserving CPU fallback for extended years and EXCEPTION-policy disagreements.

• Splits CORRECTED and EXCEPTION compatible-format sets.
• Applies the existing extended-year guard to packed-date string parsing.
• Adds CPU/GPU parity, ANSI, policy-default, disagreement, and fallback coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the reviewed changes.

No blocking failure remains.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/DateUtils.scala	Selects parser-policy-specific compatibility sets during GPU support tagging.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/datetimeExpressions.scala	Certifies yyyyMMdd for CORRECTED parsing, retains EXCEPTION fallback, and applies the extended-year support guard.
integration_tests/src/main/python/date_time_test.py	Adds packed-date correctness, ANSI, Spark 4 default-policy, disagreement, and extended-year fallback tests.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[yyyyMMdd string expression] --> B{Parser policy}
  B -->|CORRECTED| C{Extended years enabled?}
  C -->|No| D[GPU fused parser]
  C -->|Yes| E[CPU fallback]
  B -->|EXCEPTION| E
  D --> F[Spark-compatible result or ANSI error]
  E --> F

Loading

_{Reviews (2): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[rishic3]

This list claims to be for both CORRECTED and EXCEPTION, and then there is a separate list for LEGACY, but I think that is misleading. EXCEPTION is supposed to:

1. try the CORRECTED parser (returning on success)
2. if it fails try the LEGACY parser
3. if the LEGACY parser succeeds throw SparkUpgradeException

Per step 2 that means being supported under EXCEPTION also requires LEGACY to produce valid success/failure semantics. E.g. AI came up with the combination of 2024101 to yyyyMMdd under EXCEPTION, which should fail (Spark LEGACY accepts it) but we succeed (the JNI throws on it in LEGACY). I think we need three compatibility lists: CORRECTED, EXCEPTION, and LEGACY, where atm yyyyMMdd is not EXCEPTION compatible.

[gerashegalov]

Good catch. Fixed in 18bd197 by separating EXCEPTION_COMPATIBLE_FORMATS from CORRECTED_COMPATIBLE_FORMATS and selecting the policy-specific set during both tagging and execution. yyyyMMdd now falls back to CPU under EXCEPTION, so Spark preserves the CORRECTED/LEGACY disagreement behavior. I added coverage that asserts GetTimestamp fallback for a normal yyyyMMdd value and the expected error for 2024101. The focused Spark 3.5.7 GPU run passed.

[rishic3]

On a separate note this list is also used by date_format, not just parsing. I don't know if we've verified that the reverse direction date_format(timestamp, 'yyyyMMdd') matches Spark.

[gerashegalov]

Verified and covered in 18bd197. yyyyMMdd is now included in the direct date_format parity matrix for both Date and Timestamp inputs, including the runtime-fallback and timezone-rule cases. The focused Spark 3.5.7 GPU run selected 25 yyyyMMdd cases: 24 passed and the Spark-4-only case skipped, with no failures or errors.

[gerashegalov]

build

[gerashegalov]

build