Skip to content

Don't let a TestOps outage crash the whole report - #819

Merged
epszaw merged 1 commit into
mainfrom
testops-launch-start-resilience
Jul 29, 2026
Merged

Don't let a TestOps outage crash the whole report#819
epszaw merged 1 commit into
mainfrom
testops-launch-start-resilience

Conversation

@todti

@todti todti commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Found while live-testing the upload retry/pacing work (#818) against a local TestOps instance: pointing the plugin at an unreachable endpoint crashed the entire report generation, not just the TestOps upload — other plugins (awesome, csv, dashboard, ...) failed too with plugin error: state is empty.

Root cause: createLaunch, startUpload, and createSession had no error handling at all. If TestOps is unreachable exactly when the launch/session is being created, the exception propagates uncaught out of the plugin's start()/update()/done() lifecycle hooks, which corrupts the shared plugin-state loop for every other plugin in the same run.

Changes

  • Track whether the launch was actually started (#launchStarted); update()/done() skip their work entirely (with a verbose log) if it wasn't.
  • createLaunch + startUpload (in #startUpload()) and createSession (in #upload()) are now wrapped in try/catch: failures are logged and the plugin gracefully no-ops for that stage instead of throwing.
  • stopUpload in done() gets the same treatment for symmetry.

Test plan

  • yarn vitest run in packages/plugin-testops — all tests pass (updated 9 update/done tests that called the lifecycle hooks directly without a preceding start(), which is unrealistic given the new guard)
  • tsc --noEmit — clean
  • oxlint — no new warnings
  • Verified live with a direct repro against an unreachable endpoint: start() now resolves normally and logs Failed to create TestOps launch: ... instead of throwing

createLaunch/startUpload/createSession had no error handling at all: if
TestOps is unreachable exactly when the launch is created, the exception
propagates uncaught out of the plugin's start()/update()/done() lifecycle
hooks. That crashes report generation for every other plugin too (awesome,
csv, dashboard, ...), not just the TestOps upload.

Track whether the launch was actually started; skip update()/done() work
entirely if it wasn't, and catch+log failures in launch/session creation
and CI upload stop instead of letting them bubble up.
@github-actions

Copy link
Copy Markdown

Allure Report Summary

Name Duration Stats New Flaky Retry Report
Allure 3 Report 38m 0s Passed tests 9647   Skipped tests 15   Unknown tests 27 0 0 33 View
My Dashboard 38m 0s Passed tests 9647   Skipped tests 15   Unknown tests 27 0 0 33 View

@github-actions

Copy link
Copy Markdown

Allure perf metrics

Generated at: 2026-07-24T09:11:04.096Z

Phase Count Total Avg Min Max
restoreState.total 1 2477.5 ms 2477.5 ms 2477.5 ms 2477.5 ms
restoreState.dump 3 2476.9 ms 825.6 ms 735.9 ms 881.8 ms
restoreState.attachments 3 1867.3 ms 622.4 ms 558.8 ms 664.6 ms
restoreState.storeRestore 3 180.8 ms 60.3 ms 54.9 ms 65.7 ms
generate.total 1 7747.9 ms 7747.9 ms 7747.9 ms 7747.9 ms
generate.plugins.done 1 7354.3 ms 7354.3 ms 7354.3 ms 7354.3 ms
publish.upload.total 1 121451.3 ms 121451.3 ms 121451.3 ms 121451.3 ms
summary.generate 2 14.9 ms 7.5 ms 6.5 ms 8.4 ms
generate.plugin.done.agent 1 0.2 ms 0.2 ms 0.2 ms 0.2 ms
generate.plugin.done.awesome 1 6260.1 ms 6260.1 ms 6260.1 ms 6260.1 ms
generate.plugin.done.dashboard 1 1088.3 ms 1088.3 ms 1088.3 ms 1088.3 ms
generate.plugin.done.log 1 4.8 ms 4.8 ms 4.8 ms 4.8 ms
generate.plugin.done.testops 1 0.3 ms 0.3 ms 0.3 ms 0.3 ms
publish.upload.plugin.awesome 1 100350.8 ms 100350.8 ms 100350.8 ms 100350.8 ms
publish.upload.plugin.dashboard 1 14097.6 ms 14097.6 ms 14097.6 ms 14097.6 ms

Artifacts: allure-perf-metrics

@epszaw
epszaw merged commit 64d7ef5 into main Jul 29, 2026
14 checks passed
@epszaw
epszaw deleted the testops-launch-start-resilience branch July 29, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants