fix(e2e): stop wp-env warm-up JSON parse flakes#540
Open
mokagio wants to merge 2 commits into
Open
Conversation
E2E runs failed intermittently when the wp-env REST endpoints returned a non-JSON body during warm-up — a PHP notice printed into the response (`<br /><b>Warning...`) or a transient `500` — and `response.json()` threw. Because editor settings are cached in module scope only on success, a single early hiccup poisoned the first fetch and failed every early spec while later ones passed: the flaky signature. Two complementary fixes: - Set `WP_DEBUG_DISPLAY: false` so PHP notices never print into REST bodies. `WP_DEBUG_LOG` stays on, so notices still land in `debug.log`. - Route the settings and assets fetches through a `fetchJson` helper that reads the body as text, retries transient `5xx`/parse failures, and surfaces a body snippet in the final error instead of an opaque `Unexpected token '<'`. --- Generated with the help of Claude Code, https://claude.com/claude-code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
XCFramework BuildThis PR's XCFramework is available for testing. Add the following to your .package(url: "https://github.com/wordpress-mobile/GutenbergKit", branch: "pr-build/540")Built from 4fed003 |
CI surfaced the real warm-up failure the earlier retry only papered over: the Playground runtime keeps installing plugins (Blueprint steps) after it starts serving requests. During those installs WordPress drops a `.maintenance` file and returns a `503`, and there is a race where the file is removed mid-request and PHP fatals — both corrupt the first editor REST call. Gate provisioning in `wp-env-setup.sh` on the settings endpoint returning valid JSON, so the suite never starts against a half-provisioned instance. Raise the in-test fetch retry budget (5s -> 30s) as a backstop for any mid-run blip. --- Generated with the help of Claude Code, https://claude.com/claude-code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 6, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Improves stability of the Web E2E suite during wp-env warm-up by preventing PHP notices from corrupting REST responses and by making the fixtures/settings fetch more resilient to transient failures.
Changes:
- Adds a
fetchJsonhelper ine2e/wp-env-fixtures.jsthat retries transient HTTP/parse failures and surfaces a helpful body snippet on final failure. - Extends
bin/wp-env-setup.shto wait until the editor settings endpoint returns valid JSON before proceeding to run tests. - Updates
.wp-env.jsonto disableWP_DEBUG_DISPLAYso PHP notices don’t leak into REST response bodies.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
e2e/wp-env-fixtures.js |
Routes editor settings/assets fetches through a retrying JSON fetch helper to reduce warm-up flakes. |
bin/wp-env-setup.sh |
Adds a readiness gate for the editor settings REST endpoint to avoid early warm-up failures. |
.wp-env.json |
Disables debug display to keep REST responses clean while preserving debug logging. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@claude review this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I run into E2E failures from time to time in this repo over the past few weeks. The last time this happened, in the build for PR #533 which I was reviewing, I decided to task an agent to investigate with
/goal E2E green consistently. This is the result.The approach makes sense to me. While one should avoid polling in tests, I think it's acceptable in the context of E2E . But, I'm not familiar withe the Playground /
wp-envsystem that's in use for the tests, so I can't really judge.I'm sharing this in the hope it helps getting the E2E back to stable, even if just as a starting point.
AI-generated details below.
What?
Makes the Web E2E job resilient to wp-env warm-up hiccups, which were failing builds intermittently (e.g. build 2401).
Why?
Every failure in the flaky runs traced to
wp-env-fixtures.js, whereresponse.json()threw on a non-JSON body returned by the editor-settings/assets REST endpoints during warm-up — either a PHP notice printed into the response (<br /><b>Warning...) or a transient500. Editor settings are cached in module scope, but only on success, so a single early hiccup poisoned the first fetch and failed every early spec while later ones passed — the classic flaky signature (7 failed / 41 passed with all failures ineditor-load,editor-title,editor-error).How?
Two complementary layers:
.wp-env.json: setWP_DEBUG_DISPLAY: falseso PHP notices never leak into REST bodies.WP_DEBUG_LOGstays on, so they still land indebug.log.e2e/wp-env-fixtures.js: route both fetches through afetchJsonhelper that reads the body as text, retries transient5xx/parse failures, and includes a body snippet in the final error instead of an opaqueUnexpected token '<'.Testing Instructions
This is a draft to exercise CI — the goal is a green (and repeatably green) Web E2E job.
Locally:
make wp-env-startthenmake test-e2e. Before this change the settings-fetch failures reproduce on a cold instance; after, the suite passes (verified: 47/49 passed, the 2 remaining being local-only media-upload timeouts underfullyParallelworkers, which pass at CI'sworkers: 1).