Reduce GitHub CI wall time by removing unnecessary dependency waits #21015

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

Full GitHub CI runs spend substantial time waiting for macOS runners and unnecessarily serial build dependencies. Native bundle deduplication is addressed separately by #21012, but there are further opportunities to shorten the time to CI completion without reducing coverage.

The following measurements use first attempts, separating job queue time (created_at to started_at) from execution time (started_at to completed_at). Wall time spans the first job creation through final results; it excludes gaps between attempts.

The full PR sample passed and its selection artifact explicitly records ALL. Both main samples and the post-change PR's first attempt had failures. The post-change sample has selected coverage, not the full matrix; these runs are not a controlled before/after comparison.

macOS queueing can dominate the critical path. In the successful 86-minute full PR, the macOS x64 native build waited 21.7 minutes, then executed for 24.5 minutes. Its downstream native Dashboard validator subsequently waited 35.7 minutes to execute about 1.1 minutes of work. Starter validation and Homebrew also waited approximately 20 and 19 minutes, respectively.

In both main samples, downstream macOS Templates jobs waited about nine minutes after becoming ready, then executed for approximately seven minutes. The timestamps establish queueing, but do not identify the runner quota or capacity constraint.

Package-only tests wait for native CLI compilation they do not consume. Their workflow dependencies include the entire platform CLI archive producer, although these tests download built NuGets and RID-specific Dashboard/DCP packages rather than the CLI archive. Even with #21012 applied, Build CLI packages took 3.2 minutes on Linux x64, 5.3 minutes on Windows x64, and 5.8 minutes on macOS ARM64.

Extension E2E shards have unnecessary cross-platform dependencies. All Linux and Windows shards wait for both native CLI builds. In the post-change sample, Linux CLI artifacts were ready at 9.7 minutes and the VSIX producer finished at 11.5 minutes, but Windows CLI completion at 16.2 minutes delayed feed verification and the start of every shard.

Sharding repeats significant preparation. Across the two main samples, building projects in 205 no-package test jobs consumed approximately 284–288 aggregate runner-minutes. Across 44 extension E2E shards, dependency installation consumed approximately 28–31 runner-minutes and test compilation another 20–21. These totals are compute cost, not equivalent wall-time savings.

Describe the solution you'd like

Priority: reduce total CI wall time, not aggregate compute consumption. The first two implementation tracks are an earlier platform-package handoff and OS-specific extension E2E dependencies, building on #21012. Develop and validate these in parallel, then measure their combined effect on the final CI completion time with equivalent test selection.

Runner-minute reductions are secondary and do not establish success. Preserve coverage, artifact availability, and blocking failure behavior. macOS job consolidation, shared test preparation, and changes to PR dogfooding availability remain separate follow-ups rather than expanding these first experiments.

First controlled comparison: package handoff is not a wall-time win

The unchanged post-deduplication baseline
took 31m10s; the combined package-handoff and extension-dependency candidate
took 34m35s, measured from first job creation through final results.
Both used the same full-selection dispatch caller and fixed version inputs.
Both completed all 44 extension shards and all six native CLI producers.

The handoff releases platform packages earlier but adds cold CLI workers.
For example, macOS ARM64's CLI package step rose from 340s to 496s;
Windows' rose from 298s to 366s, with additional setup and cleanup costs.
Artifact download itself was only 12–17s per split worker. This comparison
does not isolate runner variability, but it does not meet the end-to-end
acceptance criterion.

Do not advance the package split as a demonstrated improvement.
An extension-only iteration
keeps the original single-worker native builds and tests the dependency
change without that penalty. The separate
native artifact verification
passed on Linux x64, Windows x64 and macOS ARM64: every file in the transferred
prepared payload matched actual native CLI extraction, and the Dashboard
NuGet tools matched both the standalone ZIP and embedded Dashboard.
That proves byte parity, not a speed improvement.

Neither comparison is green: baseline failures were Linux CLI TrayWatch and
a Windows extension-host startup failure. The combined run also exposed seven
extension unit checks still reading the old inline matrix/preflight locations;
those checks have been updated, with all 114 tests in the two affected suites
passing locally in a real VS Code host. Other candidate failures were in
unchanged Hosting, CLI and DevTunnels tests; they are not being fixed or excluded
as part of this experiment.

The independent extension scheduling contribution is draft #21017.
It targets current main without either native-build experiment. Exact-head
local validation passed 567 infrastructure tests, one diagnostics test, and
114 extension tests in a real VS Code host. Its
actual PR CI
completed in 41m33s at its exact head. The Windows extension unit job,
all 44 extension E2E jobs, all 98 CLI E2E jobs, all six Starter validators,
and all five language SDK validators succeeded. The run is red because of
three leaf failures in unchanged CLI and DevTunnels code; final gates
correctly propagated those failures. Advisory browser-debugger failures
remain advisory, not passing tests. Wall-time savings remain unproven.

The isolated extension-only timing run is now complete: no total wall-time
win yet. Its first attempt took 34m13s, versus the unchanged baseline's
31m10s. Linux extension E2E started at 9m11s instead of 14m33s, a 5m22s
earlier start, but the Windows browser-debugger shard remained last.
That shard took 17m11s versus 14m30s in the baseline, with two serial
roughly five-minute debugger failures. Runner and runtime variability
prevent treating this as an isolated causal speed comparison. All 44 E2E
shards still executed; advisory test failures are not passing tests.

The isolated run's macOS x64 Dashboard validator failed while listing
artifacts, before executing validation:

Failed to ListArtifacts: Unable to make request: ETIMEDOUT

Compact final-results aggregation has a verified, bounded improvement.
The baseline downloaded 253 matching diagnostic artifacts totaling 2,221 MiB
just to collect TRX files; the combined candidate downloaded 2,071 MiB.
Those three serial download steps took 60s and 47s, respectively.
The completed full PR run downloaded 335 diagnostic artifacts totaling
2,528 MiB; these downloads took 147s, including 118s for Windows artifacts.
Its final aggregation job executed for 3m26s after all test work completed.

Compact-results validation
completed on the same native-deduplicated timing base:

Do not attribute the whole 5m07s reduction to this change. The Windows
browser shard also started 38s earlier and ran 3m46s faster. The directly
measured aggregation-tail reduction is 42s. Extra compact uploads took a
median of 1s per test job, p95 2s, maximum 3s; the final critical shard
was extension E2E and did not acquire an extra upload.

The artifact-only replay
passed. On one runner, downloading the same source run's full diagnostics
took 52s versus 9s for compact results. Transfer size fell from 2,186 MiB
to 26 MiB. All 396 TRX files had identical relative paths and bytes across
Linux, Windows, and macOS; both combined summaries were identical.
Every original diagnostic artifact and all 44 extension E2E shards remain.

The timing run is still red: Windows Hosting, YARP, and CLI jobs failed in
test execution before the new upload, and the final failure gate propagated
those failures. The successful replay proves artifact and summary parity,
not that the full test suite passed.

Independent draft #21018 contains only the compact-results change.
It is based on main, without the earlier native or extension scheduling
experiments. Exact contribution validation passed 296 focused infrastructure
tests and actionlint. Its
exact-head PR CI
is running. The experimental-base local suite separately passed 298 tests.

Dispatch includes macOS Templates coverage, but not PR-only CLI E2E or Starter
validation, and not branch-gated polyglot lanes. A full dispatch is therefore
not proof of complete PR coverage. In addition, macOS package tests are
disabled on PRs, making a PR macOS package split even less compelling.

Use this issue to track measured critical-path improvements:

1. Do not advance the measured RID-package split. The cold-worker penalty outweighed the earlier package handoff in the completed experiment above. Revisit only with evidence that the extra worker cost can be eliminated.
2. Reduce macOS queue re-entry and repeated preparation. Evaluate combining native Dashboard smoke and starter validation per architecture, preserving Intel x64 and ARM64 execution. Investigate runner capacity separately; adding shards indiscriminately could worsen queueing.
3. Split extension E2E dependencies by OS and run feed preflight alongside artifact production. Linux shards should not wait for the Windows CLI. Keep same-run CLI, NuGet, and VSIX inputs and existing failure handling.
4. Evaluate build-once/test-many preparation where repeated project or test compilation is material. Preserve test isolation and platform-specific dependencies; do not assume cross-platform node_modules reuse is safe.

For each experiment, compare queue time, execution time, total runner-minutes, and the final critical path against a baseline with equivalent selection. Validate unchanged artifact outputs and test coverage. Keep changes only when end-to-end savings exceed added handoff/setup costs.

Additional context

The longest extension shard includes a known timeout, not just useful test execution. In the post-change sample, Windows browser-debugger took 12.5 minutes, including 8.4 minutes in the test step. Its log reported:

Timed out after 299982ms waiting for Blazor WASM root session for resource 'standalone'.

That failure is tracked by #20117. The shard's completed test failure is explicitly advisory, so a successful job conclusion does not mean every test passed. Setup and cleanup failures remain blocking. Fixing this five-minute timeout would save only about two minutes overall in this sample because macOS starter validation would become the next blocker. Moving advisory work outside required completion would be a separate policy change.

There is also a selective-PR artifact policy question. In run 38621384945, selected tests finished by 4.7 minutes, but CI took 29.7 minutes while producing six native CLI archives with no selected automated consumers. Skipping those producers could help narrow PRs and reduce shared runner pressure, but would change all-platform PR dogfooding availability. Do not remove them without an explicit decision about that behavior.

Source entry points:

• Workflow dependencies and final aggregation: .github/workflows/tests.yml
• Per-shard setup and feed verification: .github/workflows/extension-e2e-tests.yml
• Native bundle deduplication: #21012