feat: Automate reviewed SynapseML-to-Fabric releases

[Rana Singh (ranadeepsingh)]

What

Turns the SynapseML-to-Fabric release into a reviewed, fail-closed flow driven
by one release matrix. A release engineer provides the OSS version, optional
Internal super-patch, selected targets, and any immutable-package rebuild
counters; the tooling derives the tags, Maven coordinates, pip/UPack versions,
pipeline commands, and BBC-VHD values without retyping incompatible naming
conventions.

This complements #2540: that PR owns derivative tags and Spark release-branch
PRs; this PR owns the reviewed version bump, exact merged-commit tag, release
matrix, explicit Maven publication commands, artifact proof, GitHub Release,
and BBC-VHD edit.

Supported release matrix

For an OSS 1.1.4 / Internal patch 0 release:

Branch	Runtime family	OSS Maven	Internal Maven
master	Spark 3.5 / Scala 2.12 / Python 3.11	synapseml_2.12:1.1.4	synapseml-internal_2.12:1.1.4.0
spark4.0	Spark 4.0 / Scala 2.13 / Python 3.12	synapseml_2.13:1.1.4-spark4.0	synapseml-internal_2.13:1.1.4.0-spark4.0
spark4.1	Spark 4.1 / Scala 2.13 / Python 3.13	synapseml_2.13:1.1.4-spark4.1	synapseml-internal_2.13:1.1.4.0-spark4.1

The generated text plan includes six copy-ready Maven tag-build commands:
pipeline 17563 for all three public coordinates and pipeline 18453 for all
three Internal coordinates. Pipeline 35879 publishes pip and UPack packages;
it does not publish Maven, so the six Maven builds are explicit,
independently verified gates.

Operator flow

# Full Spark 3.5 / 4.0 / 4.1 plan
python scripts/release/release_matrix.py \
  --version 1.1.4 --internal-patch 0

# Machine-readable handoff to SynapseML-Internal
python scripts/release/release_matrix.py \
  --version 1.1.4 --internal-patch 0 --json > release-plan.json

# Nonzero Internal hotfix without republishing immutable OSS artifacts
python scripts/release/release_matrix.py \
  --version 1.1.3 --internal-patch 1 --scope internal-only

1. Release Prepare runs from master, applies the context-anchored version
   bump, rebuilds/version-snapshots docs, opens the reviewed PR, and dispatches
   branch validation.
2. When that exact PR merges, the workflow verifies the versioned docs, tags
   the recorded merge commit, and dispatches feat: automate release tag creation and spark branch rebase (Release Guide steps 1.4-1.5) #2540's derivative-tag and Spark
   branch orchestration.
3. Prepare the Internal branches from the same JSON plan in order: master,
   spark4.0 from master, then spark4.1 from spark4.0.
4. Create the matrix-provided tags at the reviewed merge commits and run all
   emitted pipeline 17563 and 18453 commands.
5. After all six Maven coordinates exist, run the emitted pipeline 35879
   command. OSS and Internal rebuild counters are passed through its real
   SYNAPSEML_PATCH_VERSION and SYNAPSEML_INTERNAL_PATCH_VERSION queue
   variables; targets needing different counters are intentionally split into
   separate plans.
6. Run verify_release.py, then publish the GitHub Release, update BBC-VHD,
   and complete White-Glove and train monitoring.

Fail-closed verification

verify_release.py now requires:

• every public and Internal release tag, with GitHub annotated tags peeled and
  ADO refs resolved to commits;
• one consistent tag commit per target;
• POM and main JAR files for all seven public Maven modules;
• the synapseml-core tests JAR;
• the Internal Maven POM and JAR for each selected Spark line;
• every expected public/Internal pip and UPack package.

Network, authentication, API-shape, and missing-Azure-CLI failures remain
errors rather than missing-artifact lookalikes. The GitHub Release gate uses
--skip ado,internal, so it proves the complete OSS publication without
requiring Internal credentials.

Internal follow-up

SynapseML-Internal PR 2269983
consumes this matrix directly. It makes the queued ADO tag authoritative for
the build version, removes the broad v* auto-publication trigger, validates
the selected Spark/Scala family, updates the two public-version pins
transactionally, and prints the exact tags, Maven coordinate, and pipeline
18453 command.

Release-guide mapping

Fabric release guide	Automated here	Intentionally human-gated
Step 1 - OSS	Version PR, docs snapshot, exact merge tag, derivative orchestration, public Maven commands and proof, GitHub Release	ESRP approval and pipeline authorization
Step 2 - Internal	Shared matrix, Internal tags/Maven coordinates, complete artifact verification	Internal PR review/merge and ESRP approval
Step 3 - pip/UPack	Pipeline 35879 parameters, rebuild variables, post-publish proof	Authorized queue and approval decisions
Step 4 - BBC-VHD	Deterministic setup.sh and version.txt update	BBC-VHD PR, CI triage, White-Glove approval
Step 5 - Release train	Documented handoff	Train selection, rollout monitoring, Fabric confirmation

No ADO PAT or approval credential is stored in GitHub. Cross-system approval
boundaries remain explicit.

Evidence

• Public targeted release/workflow tests: 125 passed, 26 skipped.
• Pinned Black 22.3.0 check for the CI-reported file: clean; affected pipeline
  tests: 48 passed, 26 skipped.
• Previous-primary-tag replay passes against the live tag history, including
  the abandoned v1.1.2 gap and numeric v1.0.9 -> v1.0.10 ordering.
• Workflow/pipeline YAML parsing and git diff --check: clean.
• Internal helper tests: 8 passed; Black, flake8, YAML parsing, and diff
  checks are clean.
• Internal tag-driven SBT version resolution under JDK 11 produced the exact
  requested version, and the shared change applies cleanly to the Spark 4.0
  and 4.1 port branches.
• Final-head SynapseML-Internal validation build
  232910541
  succeeded and its PR build policy is approved.
• Expanded live v1.1.3 replay: 56 of 57 expected rows exist. The sole
  missing row is
  com.microsoft.azure:synapseml-internal_2.13:1.1.3.0-spark4.0, a historical
  Maven hole that the previous process silently accepted and this verifier now
  blocks. The OSS-only replay is complete.

Compatibility

• No public JVM/Python API, serialized model, dependency, or package layout
  changes.
• Existing manual release commands remain possible; these workflows add a
  reviewed default path.
• Existing/conflicting tags, stale or cross-target commits, malformed
  versions, mismatched runtime families, duplicate pins, partial API
  responses, absent artifacts, and accidental BBC-VHD reruns fail with
  actionable errors.

Sources

• SynapseML Fabric Release Guide v2
• SynapseML-Publish-Official pipeline 35879
• Public Maven pipeline 17563
• Internal Maven pipeline 18453
• Historical public v1.1.3 release
• Historical BBC-VHD release PR 1805064
• Derivative-tag automation feat: automate release tag creation and spark branch rebase (Release Guide steps 1.4-1.5) #2540

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds release-engineering automation to make SynapseML-to-Fabric releases more deterministic and reviewable, including a GitHub Actions “prepare → tag merged commit → publish notes” flow plus supporting Python tooling to derive/verify release identifiers and safely bump BBC-VHD pins.

Changes:

• Introduces a release matrix generator (release_matrix.py) and an end-to-end verifier (verify_release.py) with accompanying tests.
• Adds bump_bbcvhd.py (plus tests) to safely update BBC-VHD component pins while preserving line endings and rolling back on failure.
• Adds/updates GitHub workflows for release prepare/tagging and manual, artifact-gated release notes; enables workflow_dispatch for PR validation so release PRs created via GITHUB_TOKEN can be validated.

Show a summary per file

File	Description
scripts/test_bump_version.py	Extends bump-version tests for denylisted repo-relative paths; normalizes path keys to POSIX.
scripts/bump-version.py	Improves Windows console encoding robustness; adds path-based denylist support; normalizes path handling via as_posix().
scripts/release/release_matrix.py	Adds a single-source-of-truth release matrix for tags and artifact versions + CLI rendering.
scripts/release/verify_release.py	Adds live verification of GitHub/ADO tags and Maven/PyPI/UPack/Azure Artifacts presence.
scripts/release/bump_bbcvhd.py	Adds deterministic, rollback-safe BBC-VHD component pin updater with CRLF/LF preservation.
scripts/release/README.md	Documents how to use the new release tooling and where it fits in the guide.
scripts/release/test_release_matrix.py	Adds regression/contract tests for the release matrix derivations and CLI validation.
scripts/release/test_verify_release.py	Adds unit tests for verifier networking/error handling and run plan behavior.
scripts/release/test_bump_bbcvhd.py	Adds tests for BBC-VHD bump idempotency, rollback, and newline preservation.
scripts/release/test_release_workflows.py	Adds tests asserting key workflow contract properties (manual gating, dispatch behavior).
scripts/release/test_prev_tag.sh	Adds a repo-taglist regression script for “previous primary tag” selection logic.
.github/workflows/release-prepare.yml	Adds a reviewed release PR generator and a merged-commit tagger that dispatches downstream orchestration.
.github/workflows/release-notes.yml	Adds a manual, artifact-gated GitHub Release publisher with explicit previous-tag selection.
.github/workflows/pr-validation.yml	Enables workflow_dispatch so validations can be dispatched for bot-opened release PR branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

scripts/release/verify_release.py:260

• --skip internal does not currently skip the internal PyPI feed checks (the synapseml_internal package); it only skips internal git tag checks. This makes --skip internal behave inconsistently compared with --skip public.

    def pip(self, package: str, version: str) -> str:
        if "pip" in self.skip or "ado" in self.skip:
            return SKIPPED
        # Azure Artifacts normalises pypi names: synapseml_internal -> synapseml-internal
        return (
            OK
            if version
            in self._feed_versions("Synapse-Conda", "pypi", package.replace("_", "-"))
            else MISSING
        )

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:75

• On Windows, subprocess.run(..., shell=True) expects a command string, but this code passes a list. That combination can fail (or behave differently) on win32, which would break the fallback ADO auth path even when az is installed. Build a platform-appropriate command (string for shell=True, list otherwise).

    out = subprocess.run(
        [
            "az",
            "account",
            "get-access-token",
            "--resource",
            ADO_RESOURCE,
            "--query",
            "accessToken",
            "-o",
            "tsv",
        ],
        capture_output=True,
        text=True,
        shell=(sys.platform == "win32"),
    )

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed the current-head suppressed Windows auth finding in 6e80f27.

• Windows now gives subprocess.run(..., shell=True) a command string; non-Windows retains an argument list with shell=False.
• Parameterized regression coverage locks both command shapes.
• A token-less Windows fallback replay authenticated through �z and found all 12 selected v1.1.3 master artifacts; the complete release-tooling suite is 279 passed.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:313

• The Maven publication gate here verifies only synapseml-core_{scala} (via public_maven()), but the workflow-generated release notes advertise com.microsoft.azure:synapseml_2.12:${VERSION} (see .github/workflows/release-notes.yml:137) and the docs use the synapseml_2.12 coordinate (website/doctest.py:25). This can let verify_release.py report COMPLETE even if the primary install coordinate is missing (or vice versa). Consider aligning the verification to check the same Maven artifact(s) that users are instructed to consume (e.g., verify both synapseml_{scala} and synapseml-core_{scala}, or switch to the canonical one).

            "maven",
            tp.key,
            f"synapseml-core_{tp.scala}",
            tp.oss_maven_version,
            c.public_maven(tp.scala, tp.oss_maven_version),

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 16/16 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

scripts/release/verify_release.py:124

• _url_exists() performs a full GET for Maven .jar/.pom existence checks. Even without reading the body, this can trigger large downloads (dozens of JARs per run) and make verify_release.py slow/flaky against the CDN. Prefer HEAD and fall back to GET only if the server rejects HEAD (405/501).

def _url_exists(url: str, headers: Dict[str, str]) -> bool:
    req = urllib.request.Request(url, headers=headers)
    try:
        with urllib.request.urlopen(req, timeout=60):
            return True

• Files reviewed: 15/15 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 0 new
• Review effort level: Lite