[FEA] Add Iceberg fast test matrix [fast-ut]

[liurenjie1024]

Fixes #15875.

Description

Expand Iceberg integration-test coverage across the supported Spark and Iceberg version matrix without running the complete test suite for every combination.

This change:

• Adds an authoritative YAML compatibility matrix at iceberg/iceberg-versions.yml, based on Apache Iceberg's Spark patch baselines.
• Records upstream-compatible but currently unsupported combinations with explicit reasons.
• Adds an object-oriented, validated Python matrix reader used by nightly and premerge test scripts.
• Uses the matrix to select every supported Iceberg version for local Hadoop, REST, and S3 Tables catalog runs.
• Runs local Hadoop-catalog combinations with ICEBERG_TEST_FAST_RUN=1 so the expanded matrix remains practical.
• Preserves validation and caller control for explicitly requested Iceberg versions.
• Applies fast-mode skipping only to tests already excluded from remote catalogs to reduce runtime, retaining local-only coverage.

AI assistance disclosure: This change and pull request description were prepared with Codex and reviewed by the author before submission and update.

Testing performed:

• Validated the checked-in Iceberg compatibility matrix against every Spark shim declared in pom.xml.
• Exercised supported, unsupported, and explicitly requested version queries through the object model and CLI.
• Compiled all 10 changed Python sources.
• Validated 79 fast-mode test skips and 10 catalog-only skips through AST inspection.
• Checked shell syntax for the modified Jenkins scripts.
• Verified the final diff with git diff --check.

Full GPU integration tests were not run locally because the required built artifacts and GPU test environment were unavailable.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

The PR adds a validated Spark-to-Iceberg compatibility matrix and uses it to expand CI coverage while reducing local Hadoop-catalog runtime.

• Adds authoritative compatibility metadata and a Python matrix reader.
• Iterates supported Iceberg releases in premerge and catalog test jobs.
• Introduces an Iceberg fast-test mode and applies it to selected local-only cases.
• Documents the matrix, fast mode, and new PyYAML dependency.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
iceberg/iceberg-versions.yml	Defines supported and explicitly unsupported Spark/Iceberg combinations, including upstream patch baselines and support reasons.
jenkins/get_iceberg_versions.py	Loads and validates the compatibility matrix against Maven Spark shim properties and exposes supported-version selection through a CLI.
jenkins/spark-tests.sh	Iterates matrix-selected Iceberg versions for local, REST, and S3 Tables catalog jobs and enables fast mode only for local Hadoop runs.
jenkins/spark-premerge-build.sh	Uses matrix-selected Iceberg versions for premerge version-detection tests.
integration_tests/src/main/python/conftest.py	Adds the environment-backed fast-run predicate consumed by Iceberg test decorators.
integration_tests/src/main/python/iceberg/iceberg_append_test.py	Extends selected local-only skip conditions to the explicitly enabled fast-test mode.
integration_tests/src/main/python/iceberg/iceberg_ctas_test.py	Extends selected CTAS test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_delete_test.py	Extends selected delete-path test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_merge_test.py	Extends selected merge-path test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_overwrite_dynamic_test.py	Extends selected dynamic-overwrite test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_overwrite_static_test.py	Extends selected static-overwrite test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_rtas_test.py	Extends selected RTAS test skips to fast mode.
integration_tests/src/main/python/iceberg/iceberg_update_test.py	Extends selected update-path test skips to fast mode.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Matrix[iceberg-versions.yml] --> Reader[get_iceberg_versions.py]
  Pom[pom.xml Spark shims] --> Reader
  Reader --> Validation{Matrix valid?}
  Validation -- No --> Fail[Fail CI selection]
  Validation -- Yes --> Versions[Supported Iceberg versions]
  Versions --> Premerge[Premerge version detection]
  Versions --> Local[Local Hadoop catalog tests]
  Versions --> REST[REST catalog tests]
  Versions --> S3[S3 Tables tests]
  Local --> Fast[ICEBERG_TEST_FAST_RUN=1]
  Fast --> Reduced[Reduced local test suite]

Loading

_{Reviews (3): Last reviewed commit: "Clarify Iceberg matrix derivation" | Re-trigger Greptile}