You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Under spark.sql.legacy.timeParserPolicy=EXCEPTION, Spark compares CORRECTED and LEGACY parsing when corrected parsing fails. The fused GPU timestamp parser does not currently preserve that disagreement for the packed MMyyyy format.
For input 12024 with format MMyyyy, Spark CPU throws SparkUpgradeException because CORRECTED parsing fails while LEGACY parses a value. The GPU fused parser returns null instead. Until this is fixed, MMyyyy must remain excluded from the explicit EXCEPTION-policy acceleration allowlist.
Compare CPU execution with GPU execution using the fused timestamp parser.
Expected behavior
The GPU path should reproduce Spark EXCEPTION-policy behavior and throw when CORRECTED and LEGACY parsing disagree. Once the JNI parser exposes that behavior, add MMyyyy back to EXCEPTION_COMPATIBLE_FORMATS with coverage for both ordinary valid values and disagreement inputs.
Environment details (please complete the following information)
Activity