Skip to content
Draft
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions .github/instructions/ado-pipeline.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,22 @@ Every ADO pipeline in this repo is split into **two YAML files**:

`ob_*` job-scope variables and `LinuxContainerImage` still appear here (ADO requires them at job scope), but they are set from `${{ parameters.* }}`, so the raw author never has to know which OneBranch convention they satisfy.

File-pairing convention: a wrapper at `.github/workflows/ado/<name>.yml` pairs with a raw stages template at `.github/workflows/ado/templates/<stem>-stages.yml`. **Multiple wrappers MAY share a single raw stages template** — that is in fact a primary motivation for the split: define several ADO pipelines (e.g. a DEV NonOfficial wrapper and a PROD Official wrapper, or per-environment variants) that all run the same stages/jobs/steps but differ in OneBranch variant, `featureFlags`, service connection, variable group, container image, etc. When wrappers share a raw template, name them so the relationship is obvious (e.g. `sources-upload-dev.yml` + `sources-upload-prod.yml` both pointing at `templates/sources-upload-stages.yml`). The variant choice cannot be hoisted into a shared sub-template because ADO requires `extends:` at the root of the entry pipeline — that is exactly why each wrapper exists.
File-pairing convention: a wrapper at `.github/workflows/ado/<name>.yml` pairs with a raw stages template at `.github/workflows/ado/templates/<stem>-stages.yml`. **Multiple wrappers MAY share a single raw stages template** — that is in fact a primary motivation for the split: define several ADO pipelines (e.g. a DEV NonOfficial wrapper and a PROD Official wrapper, or per-environment variants) that all run the same stages/jobs/steps but differ in OneBranch variant, `featureFlags`, service connection, variable group, container image, etc. When wrappers share a raw template, name them so the relationship is obvious (e.g. `package-build-dev.yml` + `package-build-prod.yml` both pointing at `templates/package-build-stages.yml`). The variant choice cannot be hoisted into a shared sub-template because ADO requires `extends:` at the root of the entry pipeline — that is exactly why each wrapper exists.

See [.github/workflows/ado/sources-upload.yml](.github/workflows/ado/sources-upload.yml) and [.github/workflows/ado/templates/sources-upload-stages.yml](.github/workflows/ado/templates/sources-upload-stages.yml) for the canonical example.
See [.github/workflows/ado/package-build.yml](.github/workflows/ado/package-build.yml) and [.github/workflows/ado/templates/package-build-stages.yml](.github/workflows/ado/templates/package-build-stages.yml) for the canonical example.

Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/<name>.yml` and are spliced into a job's `steps:` via `- template: steps/<name>.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See [.github/workflows/ado/templates/steps/common-steps.yml](.github/workflows/ado/templates/steps/common-steps.yml), shared by the `package-build` and `sources-upload` pipelines.
Shared **step sub-templates** live under `.github/workflows/ado/templates/steps/<name>.yml` and are spliced into a job's `steps:` via `- template: steps/<name>.yml` (path relative to the including stages template). Use these to share step sequences across stages templates that differ only in a trailing pipeline-specific step. Splicing as **steps** (not a separate job/stage) keeps job-scoped pipeline variables and on-disk files flowing to the steps that follow — a separate job would force output variables + artifact upload/download. The including job must define any job-scope variables the shared steps reference (e.g. `ob_outputDirectory`). See the granular templates under [.github/workflows/ado/templates/steps/](.github/workflows/ado/templates/steps/) (e.g. `ensure-full-history.yml`, `install-deps.yml`, `prepare-change-set.yml`), shared by the `package-build` (via `common-steps.yml`) and `pr-check-ct` pipelines.

**Parameterize the names of pipeline variables a template sets.** If a step template sets a job variable (`##vso[task.setvariable variable=<name>...]`), expose `<name>` through a parameter (with the conventional name as the default) so a caller composing several templates can rename it to avoid collisions. Use a consistent suffix scheme:

- **Same-job variables** (default `##vso[task.setvariable]`, no `isoutput`): parameter `<var_name>Var` (e.g. `sourceCommitVar`, `changedComponentsFileVar`).
- **Output variables** (`isoutput=true`, visible in other jobs/stages): parameter `<var_name>OutputVar` for the variable name, AND parameter `<var_name>OutputVarTask` for the setting task's `name:` (a cross-job output variable is referenced as `<taskName>.<var>`, so the task name is part of the contract).

Reading another template's variable follows the same shape: take a `<var_name>Var` parameter and reference it as `$(${{ parameters.<var_name>Var }})` in `env:` (e.g. `prepare-change-set.yml`'s `fromCommitVar`/`toCommitVar`).

**Pass pipeline parameters and variables into scripts through `env:`.** Do not inline `$(someVariable)` or `${{ parameters.X }}` inside an `inlineScript`/`script` body — map them in the step's `env:` block and reference the environment variable in the script. This keeps the script readable, avoids quoting/injection pitfalls, and makes the data flow explicit. This also covers values used to build a `##vso[...]` logging command: route the parameter through `env:` and reference `$ENV_VAR` in the `echo` rather than substituting `${{ parameters.X }}` inline.

**Surface runtime-consumed parameters as `variables` when the template owns a `variables` section.** When a template defines its own `variables` section (i.e. it declares stages/jobs), convert the input parameters it consumes at runtime into entries in that `variables` section (`- name: foo` / `value: ${{ parameters.foo }}`) and reference them as `$(foo)`. A parameter value may be a runtime `$[ <expression> ]` (or `$(macro)`) expression, which is only evaluated inside the `variables` section — referencing `${{ parameters.foo }}` directly elsewhere would emit the literal, unevaluated string. Parameters used only in compile-time structural positions (e.g. `pool.type`, `timeoutInMinutes`, task `name:`/`azureSubscription:`) stay as parameters.

## OneBranch templates (MANDATORY — wrapper only)

Expand Down Expand Up @@ -239,6 +250,10 @@ Use these as a starting point for a new pipeline. Two files: a wrapper (NonOffic
trigger: none
pr: none

# Variables extracted from this group: ApiAudience, ApiBaseUrl.
variables:
- group: <variable-group-name>

resources:
repositories:
- repository: templates
Expand All @@ -261,7 +276,8 @@ extends:
containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0
poolType: linux
serviceConnection: <service-connection-name>
variableGroup: <variable-group-name>
apiAudience: $(ApiAudience)
apiBaseUrl: $(ApiBaseUrl)
timeoutInMinutes: <int> # explicit, conservative
```

Expand All @@ -280,7 +296,9 @@ parameters:
default: linux
- name: serviceConnection
type: string
- name: variableGroup
- name: apiAudience
type: string
- name: apiBaseUrl
type: string
- name: timeoutInMinutes
type: number
Expand All @@ -293,13 +311,18 @@ stages:
pool:
type: ${{ parameters.poolType }}
variables:
- group: ${{ parameters.variableGroup }} # audience URI, base URL, etc.
- name: ob_outputDirectory
value: ${{ parameters.outputDirectory }}
- name: ob_artifactBaseName
value: ${{ parameters.artifactBaseName }}
- name: LinuxContainerImage
value: ${{ parameters.containerImage }}
# Runtime-consumed params surfaced as variables (see the variable
# rules above); reference as $(apiAudience) / $(apiBaseUrl).
- name: apiAudience
value: ${{ parameters.apiAudience }}
- name: apiBaseUrl
value: ${{ parameters.apiBaseUrl }}
steps:
- task: PipAuthenticate@1
displayName: "Authenticate pip"
Expand All @@ -318,8 +341,8 @@ stages:
--api-audience "$API_AUDIENCE" \
--api-base-url "$API_BASE_URL"
env:
API_AUDIENCE: $(ApiAudience)
API_BASE_URL: $(ApiBaseUrl)
API_AUDIENCE: $(apiAudience)
API_BASE_URL: $(apiBaseUrl)
```

Replace every `<...>` placeholder.
Expand Down
14 changes: 9 additions & 5 deletions .github/workflows/ado/package-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,14 @@
# .github/workflows/ado/templates/package-build-stages.yml
#
# Authenticates via Workload Identity Federation (OIDC) and calls the Control
# Tower APIs to run the v1 post-merge delta build (stopgap until source upload
# is reworked):
# Tower APIs to run the v1 post-merge delta build:
# 1. Resolve the (target, source) commit range for this push from the
# previous CI build (ADO Builds API).
# 2. Submit official package builds for the components that changed across
# that range, via Workload Identity Federation (OIDC) to Control Tower.
#
# The setup + change-detection + validation steps are shared with the
# source-upload pipeline via templates/steps/common-steps.yml.
# The setup + change-detection + validation steps are composed from the shared
# templates/steps/* step templates via templates/steps/common-steps.yml.
#
# Helper scripts live under:
# - scripts/ci/control-tower/ - (Control Tower-specific)
Expand Down Expand Up @@ -44,6 +43,10 @@ trigger: none

pr: none

# Variables extracted from this group: ApiAudience, ApiBaseAFDUrl.
variables:
- group: ControlTower-PRCheck

resources:
repositories:
- repository: templates
Expand Down Expand Up @@ -79,7 +82,8 @@ extends:
containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0
poolType: linux
serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV
variableGroup: ControlTower-PRCheck
apiAudience: $(ApiAudience)
apiBaseAFDUrl: $(ApiBaseAFDUrl)
# Control Tower package target for the 4.0 branch.
packageTarget: azl4
# Must exceed the script's --poll-timeout-seconds (default 600s = 10m)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
# Microsoft Corporation
#
# Wrapper pipeline — passed to ADO as the entry point for the PR package-build
# check. It submits a *scratch* Control Tower build of the components a pull
# request changes, WAITS for that build to finish, and fails the check if the
# build fails (or is rejected). The build runs in Control Tower's own sandbox —
# NO PR-controlled code is built on the CI agent (only read-only change
# detection runs there).
# Wrapper pipeline — passed to ADO as the entry point for the PR Control Tower
# check. It calls Control Tower 'prcheck' (which uploads the changed
# components' missing lookaside sources) and then submits a *scratch* Control
# Tower build of those components, WAITS for that build to finish, and fails
# the check if the build fails (or is rejected). The build runs in Control
# Tower's own sandbox — NO PR-controlled code is built on the CI agent (only
# read-only change detection runs there).
#
# This file owns all OneBranch-specific wiring (governed templates repo,
# NonOfficial variant, featureFlags) and delegates the actual stages/jobs/steps
# to the raw stages template at:
# .github/workflows/ado/templates/pr-package-build-stages.yml
# .github/workflows/ado/templates/pr-check-ct-stages.yml
#
# WHY SCRATCH + REVIEWER-GATED: building unmerged PR code is only safe because
# (a) it is a *scratch* build (throwaway, never persisted to a production
Expand Down Expand Up @@ -56,6 +57,10 @@ trigger: none

pr: none

# Variables extracted from this group: ApiAudience, ApiBaseAFDUrl.
variables:
- group: ControlTower-PRCheck

resources:
repositories:
- repository: templates
Expand Down Expand Up @@ -84,14 +89,15 @@ extends:
enabled: false

stages:
- template: /.github/workflows/ado/templates/pr-package-build-stages.yml@self
- template: /.github/workflows/ado/templates/pr-check-ct-stages.yml@self
parameters:
outputDirectory: $(Build.ArtifactStagingDirectory)/output
artifactBaseName: prpackagebuild
artifactBaseName: prcheckct
containerImage: mcr.microsoft.com/onebranch/azurelinux/build:3.0
poolType: linux
serviceConnection: CT-Endpoints-Access-ServiceConnection-DEV
variableGroup: ControlTower-PRCheck
apiAudience: $(ApiAudience)
apiBaseAFDUrl: $(ApiBaseAFDUrl)
# Control Tower package target for the 4.0 branch.
packageTarget: azl4
# This check WAITS for the Control Tower build to finish (pass/fail),
Expand Down
82 changes: 0 additions & 82 deletions .github/workflows/ado/sources-upload.yml

This file was deleted.

32 changes: 21 additions & 11 deletions .github/workflows/ado/templates/package-build-stages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
# OneBranch-coupled knobs as parameters. The wrapper at
# .github/workflows/ado/package-build.yml supplies concrete values.
#
# The setup + change-detection + validation steps are shared with the
# source-upload pipeline via templates/steps/common-steps.yml; this template
# appends only the package-build-specific Control Tower call.
# The setup + change-detection + validation steps come from the shared
# templates/steps/common-steps.yml (composed from templates/steps/*); this
# template appends only the package-build-specific Control Tower call.

parameters:
- name: outputDirectory
Expand All @@ -21,7 +21,11 @@ parameters:
default: linux
- name: serviceConnection
type: string
- name: variableGroup
# Control Tower API audience + base URL, passed by the wrapper (which owns the
# variable group / env-specific values). The template stays group-agnostic.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove comment - template has no knowledge of the origin (variable group at the moment) of the parameters. This comment can easily go stale. Remove such assumptions of caller's intentions from all templates and scripts you've touched in this PR.

- name: apiAudience
type: string
- name: apiBaseAFDUrl
type: string
- name: timeoutInMinutes
type: number
Expand All @@ -47,13 +51,20 @@ stages:
pool:
type: ${{ parameters.poolType }}
variables:
- group: ${{ parameters.variableGroup }}
- name: ob_outputDirectory
value: ${{ parameters.outputDirectory }}
- name: ob_artifactBaseName
value: ${{ parameters.artifactBaseName }}
- name: LinuxContainerImage
value: ${{ parameters.containerImage }}
# Runtime-consumed params surfaced as variables so a caller may pass
# runtime $[ ] / $(macro) expressions (evaluated here, not inline).

@PawelWMS PawelWMS Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove comment, don't explain the calling convention in every YAML. Apply globally to all changes in this PR. If this approach is missing from AI instruction - update them.

- name: apiAudience
value: ${{ parameters.apiAudience }}
- name: apiBaseAFDUrl
value: ${{ parameters.apiBaseAFDUrl }}
- name: packageTarget
value: ${{ parameters.packageTarget }}
steps:
# Shared setup + change detection + validation (PipAuthenticate,
# install deps, determine commit range, verify locks, prepare change
Expand All @@ -73,17 +84,16 @@ stages:
python3 scripts/ci/control-tower/run_package_build.py \
--api-audience "$API_AUDIENCE" \
--api-base-url "$API_BASE_URL" \
--build-reason "$CT_BUILD_REASON" \
--build-reason "$BUILD_REASON" \
--changed-components-file "$CHANGED_COMPONENTS_FILE" \
--package-target "${{ parameters.packageTarget }}" \
--package-target "$PACKAGE_TARGET" \
--official-build \
--commit-sha "$SOURCE_COMMIT" \
--repo-uri "$UPSTREAM_REPO_URL"
env:
API_AUDIENCE: $(ApiAudience)
API_BASE_URL: $(ApiBaseAFDUrl)
# Non-reserved name: an `env:` override of the reserved BUILD_REASON var is silently ignored by the agent.
CT_BUILD_REASON: $(Build.Reason)
API_AUDIENCE: $(apiAudience)
API_BASE_URL: $(apiBaseAFDUrl)
PACKAGE_TARGET: $(packageTarget)
CHANGED_COMPONENTS_FILE: $(changedComponentsFile)
SOURCE_COMMIT: $(sourceCommit)
UPSTREAM_REPO_URL: $(Build.Repository.Uri)
Loading
Loading