Reduce total CI duration

[elle-j]

Description

Improves overall time spent for our CI checks to complete. For cache hits, total time decreases from ~70min to ~29min.

Overview

gantt
    title CI on a PR — BEFORE ~70m vs NOW ~29m
    dateFormat HH:mm
    axisFormat %M

    section BEFORE ~70m
    [release] build-wasm 20m                      :r1, 00:00, 20m
    [release] build-x86_64-apple-darwin 17m       :r2, 00:00, 17m
    [release] build-x86_64-pc-windows-msvc 11m    :r3, 00:00, 11m
    [release] build-x86_64-unknown-linux-musl 10m :r4, 00:00, 10m
    [release] build-aarch64-apple-darwin 8m       :r5, 00:00, 8m
    [release] create-macos-universal 14s          :r6, 00:17, 1m
    [test-wasm] own duplicate wasm build 19m      :w1, 00:00, 19m
    [test-wasm] test matrix 3m                    :w2, 00:19, 3m
    [test] required 12m                                            :crit, t1, 00:00, 12m
    [differential-tests] build matrix waits 8m for release         :crit, t2, 00:12, 8m
    [differential-tests] macos-x86_64 compile and hash 44m         :crit, t3, 00:20, 44m
    [differential-tests] windows-x86_64 compile and hash 26m       :t4, 00:20, 26m
    [differential-tests] linux-x86_64 compile and hash 20m         :t5, 00:20, 20m
    [differential-tests] macos-arm64 compile and hash 15m          :t6, 00:20, 15m
    [differential-tests] run-e2e-tests parallel 43m                :t7, 00:22, 43m
    [differential-tests] compare-hashes 5m                         :crit, t8, 01:04, 5m
    [hardhat] use-hardhat chained 1m                               :crit, t9, 01:09, 1m

    section NOW ~29m
    [release] build-windows 3m              :nr1, 00:00, 3m
    [release] build-x86_64-apple-darwin 7m  :nr2, 00:00, 7m
    [release] build-linux 2m                :nr3, 00:00, 2m
    [release] build-aarch64-apple-darwin 2m :nr4, 00:00, 2m
    [release] create-macos-universal 23s    :nr5, 00:07, 1m
    [release] build-wasm cache hit ~11s     :nr6, 00:15, 1m
    [test-wasm] shared wasm build 15m       :nw1, 00:00, 15m
    [test-wasm] test matrix 3m              :nw2, 00:15, 3m
    [test] required 6m                                              :crit, nt1, 00:00, 6m
    [test] check-changes 2s parallel                                :nt2, 00:00, 1m
    [differential-tests] per-target cache hits ~1m 30s              :crit, nt3, 00:06, 2m
    [differential-tests] windows-x86_64 compile and hash 19m        :crit, nt4, 00:08, 19m
    [differential-tests] linux-x86_64 compile and hash 16m          :nt5, 00:08, 16m
    [differential-tests] macos-arm64 compile and hash 13m           :nt6, 00:08, 13m
    [differential-tests] run-e2e-tests 16m                          :nt7, 00:08, 16m
    [differential-tests] compare-hashes 2m                          :crit, nt8, 00:27, 2m
    [hardhat] use-hardhat parallel 1m                               :nt9, 00:07, 1m

Loading

Measurements

Metric	Before	Now	Improvement
PR CI wall-clock	70m 15s	28m 47s	−41m 28s (−59%)
test job (required check)	12m 10s	6m 17s	−5m 53s (faster LLVM download)
compile-and-hash build phase	~7m 30s (waiting on release builds)	~30s (builds already available) to ~1m 30s (if waiting for macos)	−6-7m (per-target build split + faster LLVM download)
run-e2e-tests build phase	~7m 30s (waiting on release builds)	~30s (linux-x86_64 already available)	−7m (per-target build split)
Longest compile-and-hash job	44m 05s (macos-x86_64)	19m 08s (windows-x86_64)	−24m 57s (macos-x86_64 excluded on PRs)
compile-and-hash matrix start time	20m 13s	7m 54s	−12m 19s (per-target build split + faster LLVM download)
run-e2e-tests start time	21m 50s	7m 36s	−14m 14s (per-target build split + faster LLVM download)
run-e2e-tests duration	43m 28s	16m 22s	−27m 06s (concurrency bump + flakiness fix)

Changes

Change	Before	Now
Per-target build sub-workflows	Getting a reusable build depended on all builds completing. Even if e.g. a macOS build is cached, the waiting workflow with macOS runners could be queued for a long time just to get the status and cache of the other workflow, since macOS runners are more scarce than e.g. Ubuntu.	Split umbrella reusable-build.yml into reusable-build-{linux,macos,windows,wasm}.yml. Consumers can wait on a specific target and start earlier instead of blocking on the whole matrix. E.g. E2E and the compile-and-hash matrix start earlier (gate only on the targets needed).
Bottleneck macOS Intel only running on main	macOS Intel runners are 2-3× slower than macOS ARM for the contract compilation tests. Ran on each PR, being the bottleneck.	This build is now excluded from the matrix for PR runs, and only included for pushes to main.
E2E tests use higher concurrency	run-e2e-tests duration: ~43min.	Uses higher number of concurrent tasks, made possible by retester flakiness fix. run-e2e-tests duration: ~19min.
Faster LLVM downloads	Building resolc took ~8-13min (~16 min on macOS intel).	get-llvm/action.yml uses more efficient version detection. Building resolc now takes ~2min (~5-7 min on macOS intel).
Wasm tests use cached artifacts	test-wasm.yml ran its own duplicate build.	test-wasm.yml uses the reusable Wasm build.
Skip unnecessary polkadot-sdk package	For E2E tests, polkadot-omni-node was unconditionally downloaded (~3-4min).	Only pallet-revive-eth-rpc and revive-dev-node are downloaded from the cache.
Cache report-processor	Only retester was cached.	report-processor is cached for the compilations and export hashes jobs.
Hardhat unblocked from differential tests	Started after the differential tests succeeded in order to reuse the Linux build.	Starts as soon as the required test job succeeds, using the reusable Linux build.
Conditionally skip workflows	Some workflows conditionally skipped for md-only changes. Tooling-only changes also ran e.g. test-wasm.yml and differential-tests.yml.	Most workflows are skipped for doc-only updates. Tooling-only updates trigger only the respective tooling test (Hardhat), apart from the required test job. Differential tests are skipped for js-only updates.
Miscellaneous	Used actions running on soon-deprecated Node 20. Difficult to debug Playwright test-wasm failures.	Upgraded actions that used Node 20. Added Playwright debug traces uploaded on test-wasm failures.

Resolved Issues

Closes #533

[xermicus]

Nice! Looks good overall.

[xermicus]

I'd prefer to use our Dockerfile. Currently a two stage build yielding an alpine image including solc, so maybe we want to split the Dockerfile up.

Or what would be even better, split this action up into build-resolc-native and build-resolc-musl. build-musl just tries to build the Dockerfile and sanity check by using the built image to build a contract. This would build the native and musl linux builds in parallel and also ensure the Dockerfile works on CI.

[elle-j]

Good idea on testing the Dockerfile! Regarding native vs musl linux, I think we should still build the musl here though, because that's the binary we distribute so we'd be more confident if that's the build we're using for differential tests etc.

I looked at ways we could refactor the Dockerfile such that it can still be used as it is today (no semantic change), or optionally skip e.g. the LLVM stage if a prebuilt is passed. This way the Dockerfile will always be used for building the linux musl build and can also use the published LLVM.

The full Dockerfile can be tested via a separate workflow (running in parallel). Since building LLVM will take a while, we could e.g. run that on pushes to main only.

(If we go with the Dockerfile refactor, it may be better to add it as a follow-up PR to keep that change more isolated, and keep the preexisting build logic for this PR.)

[xermicus]

Not a big fan of such dependencies adding supply chain risks. How much more complicated would it be to implement the same logic without this?