Spark 4: add bundle regression tests

[bowsii]

Fixes #4664

This change adds Spark 4 bundle regression tests following the JUnit-based approach introduced for Spark 3 in #4588.

Changes

• Add a new sparkBundleTest suite for Spark 4 integration testing

• Add BundleJarSanityIT to validate:

  • bundle JAR loading
  • package resolution through Spark/Ivy
  • catalog initialization
  • namespace operations
  • table creation and querying
  • cleanup behavior

• Add the required ServiceLoader configuration for PolarisServerManager

• Wire the new test suite into check and intTest

• Configure Spark 4-specific test dependencies and runtime classpath

Testing

Executed the Spark 4 bundle regression test suite locally and verified successful execution.

Checklist

• 🛡️ Don't disclose security issues! (contact security@apache.org)
• 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
• 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
• 💡 Added comments for complex logic
• 🧾 Updated CHANGELOG.md (if needed)
• 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

[MonkeyCanCode]

thanks for the PR @bowsii . I am wondering if we should do the same refactor by moving common code to common module.

[bowsii]

Thanks for the suggestion.

I've updated the PR to move the shared bundle regression test logic into plugins/spark/common and wired both the Spark 3.5 and Spark 4.0 integration modules to use the common implementation.

The common test logic and service configuration now live in the common module, while the version-specific modules only provide their own configuration and dependencies.

Could you please take another look when you have a chance?

[MonkeyCanCode]

Thanks for the suggestion.

I've updated the PR to move the shared bundle regression test logic into plugins/spark/common and wired both the Spark 3.5 and Spark 4.0 integration modules to use the common implementation.

The common test logic and service configuration now live in the common module, while the version-specific modules only provide their own configuration and dependencies.

Could you please take another look when you have a chance?

Hello @bowsii,

Thanks for the update. There are couple more open items for #4588 which I think we may need #4588 to be closed first before perform this one (and use git mv to keep the file history as well).

Thanks,
Yong Zheng

[bowsii]

Thanks for the clarification.

That makes sense. I'll keep this PR focused on adding the Spark 4 regression tests and revert the common-module refactor for now. Once #4588 is fully completed, we can revisit moving the shared test code into the common module and use git mv to preserve file history.

Thanks for the review.