From 50d83734571423ae1a7478b2f09c152cd7fbc29a Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Tue, 7 Jul 2026 14:56:45 +0800 Subject: [PATCH 1/6] Add protobuf integration-test dependency infrastructure (plugin-0) (#14885) Part of #14069. First slice carved out of #14354 following the plugin-side split plan in the issue. ### Description #### Problem `from_protobuf` is provided by Spark's optional `spark-protobuf` module, which is not normally present on the integration-test classpath. Subsequent GPU-support PRs need a stable CPU fallback baseline without introducing an unshaded protobuf runtime dependency. #### Changes - Copy `spark-protobuf_${scala.binary.version}` into the integration-test dependencies for Spark 3.4+, for both Scala 2.12 and 2.13 builds. Spark 3.3 profiles skip this artifact. - Add `INCLUDE_SPARK_PROTOBUF_JAR` handling to `run_pyspark_from_build.sh`. Protobuf tests are enabled by default when exactly one matching jar is found: - one jar: add it to `ALL_JARS`; - no jar: disable the tests, warning when inclusion was explicitly requested; - multiple jars: warn and disable the tests to avoid classpath-dependent API selection; - `INCLUDE_SPARK_PROTOBUF_JAR=false`: disable the tests without a warning. - Add a driver-local `local_tmp_path` fixture. The path-based protobuf API reads its descriptor with `java.io.File`, so a Hadoop-backed temporary path is not suitable on distributed filesystems. - Add two CPU-fallback smoke tests: - `test_from_protobuf_smoke_path_api` for Spark 3.4+; - `test_from_protobuf_smoke_binary_descriptor_api` for Spark 3.5+, skipped when the PySpark API does not expose `binaryDescriptorSet`. The tests use static `FileDescriptorSet` bytes and hand-encoded messages. This avoids depending on whichever unshaded protobuf runtime the Spark driver provides. The path-based test writes those bytes to `local_tmp_path`; the Spark 3.5+ test passes the same bytes directly. No GPU implementation is added. Both tests use `assert_gpu_fallback_collect("ProtobufDataToCatalyst")` to compare CPU and GPU-mode results while verifying CPU fallback. #### Testing - Spark 3.5.2 targeted integration run: - `test_from_protobuf_smoke_path_api` - `test_from_protobuf_smoke_binary_descriptor_api` - Result: `2 passed` - Spark 3.4.1 standalone compatibility probe: the path-based API successfully read the static descriptor; the binary descriptor API is not available in Spark 3.4. - Multiple matching `spark-protobuf` jars: warning emitted, jars excluded, and protobuf tests skipped. - `bash -n`, `shellcheck -S error`, and `git diff --check` passed. ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [x] Added or modified tests to cover new code paths - [ ] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required --------- Signed-off-by: Haoyang Li Co-authored-by: Claude Opus 4.7 (1M context) --- integration_tests/pom.xml | 19 +++ integration_tests/run_pyspark_from_build.sh | 26 +++- integration_tests/src/main/python/conftest.py | 12 ++ .../src/main/python/protobuf_test.py | 117 ++++++++++++++++++ pom.xml | 7 ++ scala2.13/integration_tests/pom.xml | 19 +++ scala2.13/pom.xml | 7 ++ 7 files changed, 205 insertions(+), 2 deletions(-) create mode 100644 integration_tests/src/main/python/protobuf_test.py diff --git a/integration_tests/pom.xml b/integration_tests/pom.xml index b4eb325c97f..36c26707dd0 100644 --- a/integration_tests/pom.xml +++ b/integration_tests/pom.xml @@ -132,6 +132,7 @@ parquet-hadoop*.jar spark-avro*.jar + spark-protobuf*.jar @@ -166,6 +167,24 @@ + + copy-spark-protobuf + package + + copy + + + ${spark.protobuf.skipCopy} + true + + + org.apache.spark + spark-protobuf_${scala.binary.version} + ${spark.version} + + + + diff --git a/integration_tests/run_pyspark_from_build.sh b/integration_tests/run_pyspark_from_build.sh index 45781c93ea2..3a5b595779f 100755 --- a/integration_tests/run_pyspark_from_build.sh +++ b/integration_tests/run_pyspark_from_build.sh @@ -48,6 +48,9 @@ # To run all tests, including Avro tests: # INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh # +# To run tests WITHOUT Protobuf tests (protobuf is included by default): +# INCLUDE_SPARK_PROTOBUF_JAR=false ./run_pyspark_from_build.sh +# # To run a specific test: # TEST=my_test ./run_pyspark_from_build.sh # @@ -102,6 +105,7 @@ else # support alternate local jars NOT building from the source code if [ -d "$LOCAL_JAR_PATH" ]; then AVRO_JARS=$(echo "$LOCAL_JAR_PATH"/spark-avro*.jar) + PROTOBUF_JARS=$(echo "$LOCAL_JAR_PATH"/spark-protobuf*.jar) PLUGIN_JAR=$(echo "$LOCAL_JAR_PATH"/rapids-4-spark_*.jar) if [ -f $(echo $LOCAL_JAR_PATH/parquet-hadoop*.jar) ]; then export INCLUDE_PARQUET_HADOOP_TEST_JAR=true @@ -118,6 +122,7 @@ else else [[ "$SCALA_VERSION" != "2.12" ]] && TARGET_DIR=${TARGET_DIR/integration_tests/scala$SCALA_VERSION\/integration_tests} AVRO_JARS=$(echo "$TARGET_DIR"/dependency/spark-avro*.jar) + PROTOBUF_JARS=$(echo "$TARGET_DIR"/dependency/spark-protobuf*.jar) PARQUET_HADOOP_TESTS=$(echo "$TARGET_DIR"/dependency/parquet-hadoop*.jar) # remove the log4j.properties file so it doesn't conflict with ours, ignore errors # if it isn't present or already removed @@ -143,9 +148,26 @@ else AVRO_JARS="" fi - # ALL_JARS includes dist.jar integration-test.jar avro.jar parquet.jar if they exist + INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED=$(echo "${INCLUDE_SPARK_PROTOBUF_JAR}" | tr '[:upper:]' '[:lower:]') + PROTOBUF_JAR_COUNT=$(readlink -e $PROTOBUF_JARS 2>/dev/null | wc -l) + if [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" != "false" \ + && "$PROTOBUF_JAR_COUNT" -eq 1 ]]; + then + export INCLUDE_SPARK_PROTOBUF_JAR=true + else + if [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" != "false" \ + && "$PROTOBUF_JAR_COUNT" -gt 1 ]]; then + >&2 echo "WARNING: Multiple spark-protobuf jars were found (matched: $PROTOBUF_JARS); disabling protobuf tests." + elif [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" == "true" ]]; then + >&2 echo "WARNING: INCLUDE_SPARK_PROTOBUF_JAR=true was requested but a spark-protobuf jar was not found (searched: $PROTOBUF_JARS); disabling protobuf tests." + fi + export INCLUDE_SPARK_PROTOBUF_JAR=false + PROTOBUF_JARS="" + fi + + # ALL_JARS includes dist.jar integration-test.jar avro.jar parquet.jar protobuf.jar if they exist # Remove non-existing paths and canonicalize the paths including get rid of links and `..` - ALL_JARS=$(readlink -e $PLUGIN_JAR $TEST_JARS $AVRO_JARS $PARQUET_HADOOP_TESTS || true) + ALL_JARS=$(readlink -e $PLUGIN_JAR $TEST_JARS $AVRO_JARS $PARQUET_HADOOP_TESTS $PROTOBUF_JARS || true) # `:` separated jars ALL_JARS="${ALL_JARS//$'\n'/:}" diff --git a/integration_tests/src/main/python/conftest.py b/integration_tests/src/main/python/conftest.py index 0c929af6fd4..2f4df539836 100644 --- a/integration_tests/src/main/python/conftest.py +++ b/integration_tests/src/main/python/conftest.py @@ -16,6 +16,8 @@ import os import pytest import random +import shutil +import tempfile import warnings # TODO redo _spark stuff using fixtures @@ -715,6 +717,16 @@ def spark_tmp_path(request): if not debug: fs.delete(path) +# Driver-local counterpart to spark_tmp_path; spark_tmp_path lives in the +# default Hadoop FS, which is not local on distributed setups. +@pytest.fixture +def local_tmp_path(request): + debug = request.config.getoption('debug_tmp_path') + ret = tempfile.mkdtemp(prefix='pyspark_tests_') + yield ret + if not debug: + shutil.rmtree(ret, ignore_errors=True) + class TmpTableFactory: def __init__(self, base_id): self.base_id = base_id diff --git a/integration_tests/src/main/python/protobuf_test.py b/integration_tests/src/main/python/protobuf_test.py new file mode 100644 index 00000000000..41ec2833729 --- /dev/null +++ b/integration_tests/src/main/python/protobuf_test.py @@ -0,0 +1,117 @@ +# Copyright (c) 2026, NVIDIA CORPORATION. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect +import os + +import pytest + +from asserts import assert_gpu_fallback_collect +from marks import allow_non_gpu +from spark_session import is_before_spark_340 +import pyspark.sql.functions as f + +if os.environ.get('INCLUDE_SPARK_PROTOBUF_JAR', 'true').lower() == 'false': + pytestmark = pytest.mark.skip(reason="INCLUDE_SPARK_PROTOBUF_JAR is disabled") +else: + pytestmark = pytest.mark.skipif( + is_before_spark_340(), reason="from_protobuf is Spark 3.4.0+") + + +def _try_import_from_protobuf(): + try: + from pyspark.sql.protobuf.functions import from_protobuf + return from_protobuf + except Exception: + return None + + +@pytest.fixture(scope="module") +def from_protobuf_fn(): + fn = _try_import_from_protobuf() + if fn is None: + pytest.skip("from_protobuf not available") + return fn + + +def _encode_varint(value): + out = bytearray() + value &= 0xFFFFFFFFFFFFFFFF + while True: + bits = value & 0x7F + value >>= 7 + if value: + out.append(bits | 0x80) + else: + out.append(bits) + return bytes(out) + + +def _encode_simple_message(i32_value, s_value): + buf = bytearray() + buf += _encode_varint((1 << 3) | 0) # field 1, VARINT + buf += _encode_varint(i32_value) + s_bytes = s_value.encode("utf-8") + buf += _encode_varint((2 << 3) | 2) # field 2, LENGTH-DELIMITED + buf += _encode_varint(len(s_bytes)) + buf += s_bytes + return bytes(buf) + + +# Avoid depending on whichever unshaded protobuf runtime the Spark driver provides. +_simple_desc_bytes = bytes.fromhex( + "0a360a0c73696d706c652e70726f746f12047465737422200a0653696d706c65" + "120b0a0369333218012001280512090a0173180220012809") + + +@pytest.fixture +def simple_desc(local_tmp_path): + # from_protobuf reads descFilePath via java.io.File on the driver. + desc_path = local_tmp_path + "/simple.desc" + with open(desc_path, "wb") as fp: + fp.write(_simple_desc_bytes) + return desc_path, _simple_desc_bytes + + +_smoke_rows = [(1, "a"), (-2, "bb"), (0, ""), (12345, "hello")] + + +def _make_smoke_df(spark): + encoded = [(_encode_simple_message(i, s),) for (i, s) in _smoke_rows] + return spark.createDataFrame(encoded, ["bin"]) + + +@allow_non_gpu("ProjectExec", "ProtobufDataToCatalyst") +def test_from_protobuf_smoke_path_api(simple_desc, from_protobuf_fn): + desc_path, _ = simple_desc + + def run(spark): + return _make_smoke_df(spark).select( + from_protobuf_fn(f.col("bin"), "test.Simple", desc_path).alias("d")) + + assert_gpu_fallback_collect(run, "ProtobufDataToCatalyst") + + +@allow_non_gpu("ProjectExec", "ProtobufDataToCatalyst") +def test_from_protobuf_smoke_binary_descriptor_api(simple_desc, from_protobuf_fn): + if "binaryDescriptorSet" not in inspect.signature(from_protobuf_fn).parameters: + pytest.skip("binaryDescriptorSet kwarg is Spark 3.5+ only") + _, desc_bytes = simple_desc + + def run(spark): + return _make_smoke_df(spark).select( + from_protobuf_fn(f.col("bin"), "test.Simple", + binaryDescriptorSet=bytearray(desc_bytes)).alias("d")) + + assert_gpu_fallback_collect(run, "ProtobufDataToCatalyst") diff --git a/pom.xml b/pom.xml index 068d0d8dc65..5d1263eb1ab 100644 --- a/pom.xml +++ b/pom.xml @@ -98,6 +98,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true @@ -130,6 +131,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -154,6 +156,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -178,6 +181,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -202,6 +206,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -1050,6 +1055,8 @@ false false 330 + + false 1.8 8 ${java.major.version} diff --git a/scala2.13/integration_tests/pom.xml b/scala2.13/integration_tests/pom.xml index 92e30332bdf..f1503135194 100644 --- a/scala2.13/integration_tests/pom.xml +++ b/scala2.13/integration_tests/pom.xml @@ -132,6 +132,7 @@ parquet-hadoop*.jar spark-avro*.jar + spark-protobuf*.jar @@ -166,6 +167,24 @@ + + copy-spark-protobuf + package + + copy + + + ${spark.protobuf.skipCopy} + true + + + org.apache.spark + spark-protobuf_${scala.binary.version} + ${spark.version} + + + + diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml index ede14695b1f..b780b725aa9 100644 --- a/scala2.13/pom.xml +++ b/scala2.13/pom.xml @@ -98,6 +98,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true @@ -130,6 +131,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -154,6 +156,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -178,6 +181,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -202,6 +206,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x + true delta-lake/delta-21x @@ -1050,6 +1055,8 @@ false false 330 + + false 1.8 8 ${java.major.version} From 9b5c992064abccfb93efb1d13f9b17dc7b94f7a3 Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Fri, 28 Aug 2026 15:20:24 +0800 Subject: [PATCH 2/6] Enable protobuf smoke tests on Databricks runtimes Signed-off-by: Haoyang Li --- integration_tests/README.md | 12 +++++++ integration_tests/run_pyspark_from_build.sh | 34 ++++++++++++++++--- .../src/main/python/protobuf_test.py | 10 ++---- .../src/main/python/spark_session.py | 7 ++++ scala2.13/shim-deps/pom.xml | 3 ++ shim-deps/pom.xml | 3 ++ 6 files changed, 58 insertions(+), 11 deletions(-) diff --git a/integration_tests/README.md b/integration_tests/README.md index 35fd21cdcfc..7bd64d4aaf4 100644 --- a/integration_tests/README.md +++ b/integration_tests/README.md @@ -299,6 +299,18 @@ You do need to have access to a compatible GPU with the needed CUDA drivers. The `--runtime_env` is used to specify the environment you are running the tests in. Valid values are `databricks`,`emr`,`dataproc`,`dataproc_serverless` and `apache`. This is generally used when certain environments have different behavior, and the tests don't have a good way to auto-detect the environment yet. +#### Protobuf tests on Databricks + +`INCLUDE_SPARK_PROTOBUF_JAR` controls external `spark-protobuf` jar injection; it does not control +protobuf test eligibility. Apache Spark runs inject a matching external jar only when exactly one is +found and this variable is not set to `false`. Databricks runs use the runtime-bundled protobuf +implementation instead, so `run_pyspark_from_build.sh --runtime_env=databricks` does not inject a +matching jar from either the build dependencies or `LOCAL_JAR_PATH`, even if the variable is +explicitly set to `true`. + +The smoke tests detect the bundled runtime independently and use a static descriptor set, so they do +not depend on Spark's private, runtime-specific shaded protobuf classes. + ### timezone The RAPIDS plugin currently only supports the UTC time zone. Spark uses the default system time zone unless explicitly set otherwise. diff --git a/integration_tests/run_pyspark_from_build.sh b/integration_tests/run_pyspark_from_build.sh index 3a5b595779f..365c693dc04 100755 --- a/integration_tests/run_pyspark_from_build.sh +++ b/integration_tests/run_pyspark_from_build.sh @@ -29,6 +29,7 @@ # - SPARK_HOME: Path to your Apache Spark installation. # - SKIP_TESTS: If set to true, skips running the Python integration tests. # - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests. +# - INCLUDE_SPARK_PROTOBUF_JAR: Controls external spark-protobuf jar injection, not test selection. # - TEST: Specifies a specific test to run. # - TEST_TAGS: Allows filtering tests based on tags. # - TEST_TYPE: Specifies the type of tests to run. @@ -48,7 +49,7 @@ # To run all tests, including Avro tests: # INCLUDE_SPARK_AVRO_JAR=true ./run_pyspark_from_build.sh # -# To run tests WITHOUT Protobuf tests (protobuf is included by default): +# To run without injecting an external spark-protobuf jar: # INCLUDE_SPARK_PROTOBUF_JAR=false ./run_pyspark_from_build.sh # # To run a specific test: @@ -63,6 +64,28 @@ set -ex +is_databricks_runtime_arg() { + local expect_runtime_env_value=false + local runtime_env="" + local arg + for arg in "$@"; do + if [[ "$expect_runtime_env_value" == "true" ]]; then + runtime_env="$arg" + expect_runtime_env_value=false + continue + fi + case "$arg" in + --runtime_env) + expect_runtime_env_value=true + ;; + --runtime_env=*) + runtime_env="${arg#*=}" + ;; + esac + done + [[ "${runtime_env,,}" == "databricks" ]] +} + SCRIPTPATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )" cd "$SCRIPTPATH" @@ -150,16 +173,19 @@ else INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED=$(echo "${INCLUDE_SPARK_PROTOBUF_JAR}" | tr '[:upper:]' '[:lower:]') PROTOBUF_JAR_COUNT=$(readlink -e $PROTOBUF_JARS 2>/dev/null | wc -l) - if [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" != "false" \ + if is_databricks_runtime_arg "$@"; then + export INCLUDE_SPARK_PROTOBUF_JAR=false + PROTOBUF_JARS="" + elif [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" != "false" \ && "$PROTOBUF_JAR_COUNT" -eq 1 ]]; then export INCLUDE_SPARK_PROTOBUF_JAR=true else if [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" != "false" \ && "$PROTOBUF_JAR_COUNT" -gt 1 ]]; then - >&2 echo "WARNING: Multiple spark-protobuf jars were found (matched: $PROTOBUF_JARS); disabling protobuf tests." + >&2 echo "WARNING: Multiple spark-protobuf jars were found (matched: $PROTOBUF_JARS); not injecting spark-protobuf." elif [[ "$INCLUDE_SPARK_PROTOBUF_JAR_REQUESTED" == "true" ]]; then - >&2 echo "WARNING: INCLUDE_SPARK_PROTOBUF_JAR=true was requested but a spark-protobuf jar was not found (searched: $PROTOBUF_JARS); disabling protobuf tests." + >&2 echo "WARNING: INCLUDE_SPARK_PROTOBUF_JAR=true was requested but a spark-protobuf jar was not found (searched: $PROTOBUF_JARS)." fi export INCLUDE_SPARK_PROTOBUF_JAR=false PROTOBUF_JARS="" diff --git a/integration_tests/src/main/python/protobuf_test.py b/integration_tests/src/main/python/protobuf_test.py index 41ec2833729..c525433e3a4 100644 --- a/integration_tests/src/main/python/protobuf_test.py +++ b/integration_tests/src/main/python/protobuf_test.py @@ -13,20 +13,16 @@ # limitations under the License. import inspect -import os import pytest from asserts import assert_gpu_fallback_collect from marks import allow_non_gpu -from spark_session import is_before_spark_340 +from spark_session import is_spark_protobuf_available import pyspark.sql.functions as f -if os.environ.get('INCLUDE_SPARK_PROTOBUF_JAR', 'true').lower() == 'false': - pytestmark = pytest.mark.skip(reason="INCLUDE_SPARK_PROTOBUF_JAR is disabled") -else: - pytestmark = pytest.mark.skipif( - is_before_spark_340(), reason="from_protobuf is Spark 3.4.0+") +pytestmark = pytest.mark.skipif( + not is_spark_protobuf_available(), reason="from_protobuf is unavailable") def _try_import_from_protobuf(): diff --git a/integration_tests/src/main/python/spark_session.py b/integration_tests/src/main/python/spark_session.py index 513fc6d455d..dc881b20280 100644 --- a/integration_tests/src/main/python/spark_session.py +++ b/integration_tests/src/main/python/spark_session.py @@ -337,6 +337,13 @@ def is_databricks143(): def is_databricks173_or_later(): return is_databricks_version_or_later(17, 3) +def is_spark_protobuf_available(): + if is_databricks_runtime(): + return is_databricks122_or_later() + return (is_spark_340_or_later() and + os.environ.get('INCLUDE_SPARK_PROTOBUF_JAR', 'true').lower() != 'false') + + def supports_delta_lake_deletion_vectors(): """Whether the current Delta Lake runtime provides the deletion-vector feature.""" if is_databricks_runtime(): diff --git a/scala2.13/shim-deps/pom.xml b/scala2.13/shim-deps/pom.xml index f546feeddcd..6fa43674b08 100644 --- a/scala2.13/shim-deps/pom.xml +++ b/scala2.13/shim-deps/pom.xml @@ -210,6 +210,9 @@ databricks + + true + com.nvidia diff --git a/shim-deps/pom.xml b/shim-deps/pom.xml index cc6f4da33cd..7b4da550db7 100644 --- a/shim-deps/pom.xml +++ b/shim-deps/pom.xml @@ -210,6 +210,9 @@ databricks + + true + com.nvidia From 14fbf8ffc63a5ab389715b53ba345ab2c896056c Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Fri, 28 Aug 2026 15:55:11 +0800 Subject: [PATCH 3/6] Do not hide protobuf import failures Signed-off-by: Haoyang Li --- integration_tests/src/main/python/protobuf_test.py | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/integration_tests/src/main/python/protobuf_test.py b/integration_tests/src/main/python/protobuf_test.py index c525433e3a4..6238144a875 100644 --- a/integration_tests/src/main/python/protobuf_test.py +++ b/integration_tests/src/main/python/protobuf_test.py @@ -25,20 +25,10 @@ not is_spark_protobuf_available(), reason="from_protobuf is unavailable") -def _try_import_from_protobuf(): - try: - from pyspark.sql.protobuf.functions import from_protobuf - return from_protobuf - except Exception: - return None - - @pytest.fixture(scope="module") def from_protobuf_fn(): - fn = _try_import_from_protobuf() - if fn is None: - pytest.skip("from_protobuf not available") - return fn + from pyspark.sql.protobuf.functions import from_protobuf + return from_protobuf def _encode_varint(value): From 9433cd6ae29d22c312c895e3e2ceb4e687649b69 Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Mon, 31 Aug 2026 11:46:28 +0800 Subject: [PATCH 4/6] Install Python protobuf in integration test images Signed-off-by: Haoyang Li --- integration_tests/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/integration_tests/requirements.txt b/integration_tests/requirements.txt index 1bd53040f56..aeb856d9295 100644 --- a/integration_tests/requirements.txt +++ b/integration_tests/requirements.txt @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. pytest +protobuf sre_yield pandas pyarrow == 17.0.0 ; python_version == '3.8' From 807efa4ed2bc3de2c6475a3d96896847025f60c7 Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Mon, 31 Aug 2026 12:57:20 +0800 Subject: [PATCH 5/6] Clean up spark-protobuf copy gating Signed-off-by: Haoyang Li --- integration_tests/pom.xml | 2 +- pom.xml | 12 ++++++------ scala2.13/integration_tests/pom.xml | 2 +- scala2.13/pom.xml | 12 ++++++------ scala2.13/shim-deps/pom.xml | 8 +++++++- shim-deps/pom.xml | 8 +++++++- 6 files changed, 28 insertions(+), 16 deletions(-) diff --git a/integration_tests/pom.xml b/integration_tests/pom.xml index 36c26707dd0..dbf9184905d 100644 --- a/integration_tests/pom.xml +++ b/integration_tests/pom.xml @@ -174,7 +174,7 @@ copy - ${spark.protobuf.skipCopy} + ${spark.protobuf.copy.skip} true diff --git a/pom.xml b/pom.xml index 5d1263eb1ab..9a9eaea7f82 100644 --- a/pom.xml +++ b/pom.xml @@ -98,7 +98,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true @@ -131,7 +131,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -156,7 +156,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -181,7 +181,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -206,7 +206,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -1056,7 +1056,7 @@ false 330 - false + false 1.8 8 ${java.major.version} diff --git a/scala2.13/integration_tests/pom.xml b/scala2.13/integration_tests/pom.xml index f1503135194..83541b90297 100644 --- a/scala2.13/integration_tests/pom.xml +++ b/scala2.13/integration_tests/pom.xml @@ -174,7 +174,7 @@ copy - ${spark.protobuf.skipCopy} + ${spark.protobuf.copy.skip} true diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml index b780b725aa9..bdd5d5a03f7 100644 --- a/scala2.13/pom.xml +++ b/scala2.13/pom.xml @@ -98,7 +98,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true @@ -131,7 +131,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -156,7 +156,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -181,7 +181,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -206,7 +206,7 @@ rapids-4-spark-delta-21x rapids-4-spark-delta-22x rapids-4-spark-delta-23x - true + true delta-lake/delta-21x @@ -1056,7 +1056,7 @@ false 330 - false + false 1.8 8 ${java.major.version} diff --git a/scala2.13/shim-deps/pom.xml b/scala2.13/shim-deps/pom.xml index 6fa43674b08..f3be33766a4 100644 --- a/scala2.13/shim-deps/pom.xml +++ b/scala2.13/shim-deps/pom.xml @@ -38,6 +38,9 @@ 330cdh + + true + com.nvidia @@ -56,6 +59,9 @@ 332cdh + + true + com.nvidia @@ -211,7 +217,7 @@ - true + true diff --git a/shim-deps/pom.xml b/shim-deps/pom.xml index 7b4da550db7..e1e293eb8bf 100644 --- a/shim-deps/pom.xml +++ b/shim-deps/pom.xml @@ -38,6 +38,9 @@ 330cdh + + true + com.nvidia @@ -56,6 +59,9 @@ 332cdh + + true + com.nvidia @@ -211,7 +217,7 @@ - true + true From 0054f47548134be48ab5b9329a4843f5dfc41d92 Mon Sep 17 00:00:00 2001 From: Haoyang Li Date: Mon, 31 Aug 2026 15:26:35 +0800 Subject: [PATCH 6/6] Clarify protobuf test eligibility Signed-off-by: Haoyang Li --- integration_tests/README.md | 14 ++++++++------ integration_tests/run_pyspark_from_build.sh | 3 ++- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/integration_tests/README.md b/integration_tests/README.md index 7bd64d4aaf4..068466d1c0a 100644 --- a/integration_tests/README.md +++ b/integration_tests/README.md @@ -86,6 +86,8 @@ For manual installation, you need to setup your environment: - pytest : A framework that makes it easy to write small, readable tests, and can scale to support complex functional testing for applications and libraries (requires Python 3.6+). +- protobuf + : Provides Protocol Buffers APIs for protobuf integration-test fixtures. - sre_yield : Provides a set of APIs to generate string data from a regular expression. - pandas @@ -301,12 +303,12 @@ when certain environments have different behavior, and the tests don't have a go #### Protobuf tests on Databricks -`INCLUDE_SPARK_PROTOBUF_JAR` controls external `spark-protobuf` jar injection; it does not control -protobuf test eligibility. Apache Spark runs inject a matching external jar only when exactly one is -found and this variable is not set to `false`. Databricks runs use the runtime-bundled protobuf -implementation instead, so `run_pyspark_from_build.sh --runtime_env=databricks` does not inject a -matching jar from either the build dependencies or `LOCAL_JAR_PATH`, even if the variable is -explicitly set to `true`. +On Databricks, `INCLUDE_SPARK_PROTOBUF_JAR` controls only external `spark-protobuf` jar injection; it +does not control protobuf test eligibility. Apache Spark runs require a matching external jar and +skip the protobuf tests when this variable is set to `false`. Databricks runs use the runtime-bundled +protobuf implementation instead, so `run_pyspark_from_build.sh --runtime_env=databricks` does not +inject a matching jar from either the build dependencies or `LOCAL_JAR_PATH`, even if the variable +is explicitly set to `true`. The smoke tests detect the bundled runtime independently and use a static descriptor set, so they do not depend on Spark's private, runtime-specific shaded protobuf classes. diff --git a/integration_tests/run_pyspark_from_build.sh b/integration_tests/run_pyspark_from_build.sh index 365c693dc04..23559a080cf 100755 --- a/integration_tests/run_pyspark_from_build.sh +++ b/integration_tests/run_pyspark_from_build.sh @@ -29,7 +29,8 @@ # - SPARK_HOME: Path to your Apache Spark installation. # - SKIP_TESTS: If set to true, skips running the Python integration tests. # - INCLUDE_SPARK_AVRO_JAR: If set to true, includes Avro tests. -# - INCLUDE_SPARK_PROTOBUF_JAR: Controls external spark-protobuf jar injection, not test selection. +# - INCLUDE_SPARK_PROTOBUF_JAR: Controls external spark-protobuf jar injection; setting it to +# false also disables protobuf tests on Apache Spark. # - TEST: Specifies a specific test to run. # - TEST_TAGS: Allows filtering tests based on tags. # - TEST_TYPE: Specifies the type of tests to run.