Skip to content

Add advisory audit for NVIDIA API deprecations [reduced-it] #15867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gerashegalov wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add advisory audit for NVIDIA API deprecations [reduced-it] #15867

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cad0db9
Add advisory NVIDIA deprecation audit
gerashegalov Sep 1, 2026
4c28634
Fail optional audit when deprecations are found
gerashegalov Sep 1, 2026
910bd0d
Run deprecation checks with Jython
gerashegalov Sep 1, 2026
46edaea
Launch Jython from the plugin classpath
gerashegalov Sep 1, 2026
7781efe
Keep deprecation audit on root POM
gerashegalov Sep 1, 2026
c4ea71c
Merge branch 'main' into codex/nvidia-deprecation-audit
gerashegalov Sep 2, 2026
3f860c3
Merge branch 'main' into codex/nvidia-deprecation-audit
gerashegalov Sep 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/workflows/mvn-verify-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ env:
-Drapids.secondaryCacheDir=$HOME/.m2/repository/.sbt/1.0/zinc/org.scala-sbt

permissions:
actions: read
contents: read

jobs:
Expand Down Expand Up @@ -430,3 +431,31 @@ jobs:
fi
}
done

nvidia-deprecation-audit:
name: NVIDIA deprecation audit (optional)
if: ${{ always() }}
needs:
- package-tests
- package-tests-scala213
- verify-213-modules
- verify-all-212-modules
- install-modules
runs-on: ubuntu-latest
steps:
- uses: NVIDIA/spark-rapids-common/checkout@main

- name: Collect compiler deprecations from matrix logs
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
mvn --batch-mode -N antrun:run@nvidia-deprecation-audit \
-Ddeprecation.audit.rawReport="$RUNNER_TEMP/nvidia-deprecation-audit.json"

- name: Upload deprecation report
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: nvidia-deprecation-audit
path: ${{ runner.temp }}/nvidia-deprecation-audit.json
if-no-files-found: ignore
18 changes: 18 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,24 @@ or similarly
./build/buildall --rebuild-dist-only --option="-Ddist.jar.compress=false -Drapids.jni.unpack.skip"
```

### Cross-repository API deprecations

API replacements in cuDF Java, cudf-spark-jni, and cudf-spark-private must allow cudf-spark time to
consume a published artifact before the old entry point becomes deprecated. Introduce the
replacement first while the old method remains supported and delegates to the same implementation.
After the updated snapshot is available, migrate cudf-spark callers in a separate change. Add the
deprecation annotation only after known callers have migrated, and retain the compatibility entry
point for at least one more release before removal.

Scala deprecations originating in NVIDIA-owned `ai.rapids.cudf`, `com.nvidia.spark.rapids`, and
`org.apache.spark.sql.rapids` APIs are reported as compiler information instead of fatal warnings
during this migration window. The same applies to cudf-spark-private's
`org.apache.spark.sql.execution.aggregate.PartialAggUtils` bridge; other APIs in Apache Spark
namespaces are not exempt. Deprecations from other dependencies remain build errors. The
optional NVIDIA deprecation audit in pull requests collects these diagnostics across the Maven
build matrix. Findings or incomplete log collection fail the audit check so contributors inspect
the result, but the audit is not a required build check; build-job results remain authoritative.

## Code contributions

### Source code layout
Expand Down
55 changes: 54 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1152,6 +1152,8 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<jython.version>2.7.3</jython.version>
<deprecation.audit.rawReport>${project.build.directory}/nvidia-deprecation-audit.json</deprecation.audit.rawReport>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1618,7 +1620,7 @@ This will force full Scala code rebuild in downstream modules.
<dependency>
<groupId>org.python</groupId>
<artifactId>jython-standalone</artifactId>
<version>2.7.3</version>
<version>${jython.version}</version>
</dependency>
<dependency>
<groupId>net.sourceforge.pmd</groupId>
Expand Down Expand Up @@ -1742,6 +1744,18 @@ This will force full Scala code rebuild in downstream modules.
<arg>-Wconf:cat=unused-privates:e</arg>
<arg>-Wunused:imports,locals,patvars,privates</arg>
--><!-- #endif scala-2.13 -->
<!-- Wconf arguments are prepended, so keep these last to take
precedence over the general Scala 2.13 deprecation rule.
NVIDIA-owned APIs are updated in lockstep with this project, but
their replacement artifacts are not available before deprecation
annotations land. Keep these diagnostics visible without letting
fatal-warning policies break the update window. cudf-spark-private
does not have a single artifact-specific namespace, so cover its
NVIDIA-owned package roots and its one Spark-package bridge. -->
<arg>-Wconf:cat=deprecation&amp;origin=ai\.rapids\.cudf\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=com\.nvidia\.spark\.rapids\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=org\.apache\.spark\.sql\.rapids\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=org\.apache\.spark\.sql\.execution\.aggregate\.PartialAggUtils([.$].*|$):iv</arg>
</args>
<addJavacArgs>${scala.javac.args}</addJavacArgs>
<!-- #if scala-2.13 --><!--
Expand Down Expand Up @@ -1992,6 +2006,34 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Encode option hyphens so this block remains a valid generated XML comment. -->
<!-- #if scala-2.12 -->
<execution>
<id>nvidia-deprecation-audit</id>
<phase>none</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_deprecation_audit.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/deprecation_audit.py"/>
<arg value="&#45;&#45;repo-root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${deprecation.audit.rawReport}"/>
</java>
</target>
</configuration>
</execution>
<!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2039,6 +2081,17 @@ This will force full Scala code rebuild in downstream modules.
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-B"/>
<arg value="scripts/check_deprecation_policy.py"/>
<arg value="--pom"/>
<arg value="${project.basedir}/pom.xml"/>
<arg value="--maven-repo"/>
<arg value="${settings.localRepository}"/>
</java>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
Expand Down
55 changes: 54 additions & 1 deletion scala2.13/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1152,6 +1152,8 @@
<scala.plugin.version>4.9.10</scala.plugin.version>
<maven.install.plugin.version>3.1.1</maven.install.plugin.version>
<maven.jar.plugin.version>3.3.0</maven.jar.plugin.version>
<jython.version>2.7.3</jython.version>
<deprecation.audit.rawReport>${project.build.directory}/nvidia-deprecation-audit.json</deprecation.audit.rawReport>
<scalatest-maven-plugin.version>2.0.2</scalatest-maven-plugin.version>
<guava.cdh.version>30.0-jre</guava.cdh.version>
<arrow.cdh.version>2.0.0</arrow.cdh.version>
Expand Down Expand Up @@ -1618,7 +1620,7 @@ This will force full Scala code rebuild in downstream modules.
<dependency>
<groupId>org.python</groupId>
<artifactId>jython-standalone</artifactId>
<version>2.7.3</version>
<version>${jython.version}</version>
</dependency>
<dependency>
<groupId>net.sourceforge.pmd</groupId>
Expand Down Expand Up @@ -1742,6 +1744,18 @@ This will force full Scala code rebuild in downstream modules.
<arg>-Wconf:cat=unused-privates:e</arg>
<arg>-Wunused:imports,locals,patvars,privates</arg>
<!-- #endif scala-2.13 -->
<!-- Wconf arguments are prepended, so keep these last to take
precedence over the general Scala 2.13 deprecation rule.
NVIDIA-owned APIs are updated in lockstep with this project, but
their replacement artifacts are not available before deprecation
annotations land. Keep these diagnostics visible without letting
fatal-warning policies break the update window. cudf-spark-private
does not have a single artifact-specific namespace, so cover its
NVIDIA-owned package roots and its one Spark-package bridge. -->
<arg>-Wconf:cat=deprecation&amp;origin=ai\.rapids\.cudf\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=com\.nvidia\.spark\.rapids\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=org\.apache\.spark\.sql\.rapids\..*:iv</arg>
<arg>-Wconf:cat=deprecation&amp;origin=org\.apache\.spark\.sql\.execution\.aggregate\.PartialAggUtils([.$].*|$):iv</arg>
</args>
<addJavacArgs>${scala.javac.args}</addJavacArgs>
<!-- #if scala-2.13 -->
Expand Down Expand Up @@ -1992,6 +2006,34 @@ This will force full Scala code rebuild in downstream modules.
<!-- parent-pom only executions -->
<inherited>false</inherited>
<executions>
<!-- Encode option hyphens so this block remains a valid generated XML comment. -->
<!-- #if scala-2.12 --><!--
<execution>
<id>nvidia-deprecation-audit</id>
<phase>none</phase>
<goals><goal>run</goal></goals>
<configuration>
<target>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/tests/test_deprecation_audit.py"/>
</java>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true" dir="${project.basedir}">
<arg value="-B"/>
<arg value="scripts/deprecation_audit.py"/>
<arg value="&#45;&#45;repo-root"/>
<arg value="${project.basedir}"/>
<arg value="&#45;&#45;raw-report"/>
<arg value="${deprecation.audit.rawReport}"/>
</java>
</target>
</configuration>
</execution>
--><!-- #endif scala-2.12 -->
<execution>
<!--
This is an alternative implementation of the scalastyle check invocation,
Expand Down Expand Up @@ -2039,6 +2081,17 @@ This will force full Scala code rebuild in downstream modules.
<arg value="-p"/>
<arg value="test_check_with_resource_nesting.py"/>
</exec>
<java classname="org.python.util.jython"
classpathref="maven.plugin.classpath"
fork="true" failonerror="true"
dir="${spark.rapids.source.basedir}">
<arg value="-B"/>
<arg value="scripts/check_deprecation_policy.py"/>
<arg value="--pom"/>
<arg value="${project.basedir}/pom.xml"/>
<arg value="--maven-repo"/>
<arg value="${settings.localRepository}"/>
</java>
<!-- Existing nesting exceptions are tracked by
https://github.com/NVIDIA/cudf-spark/issues/11713. -->
<exec executable="python3" failonerror="true"
Expand Down
Loading
Loading