Reduce race condition in extra worker BMH application - #1947
Conversation
Wait for all worker Machines to have a BareMetalHost assigned before applying extra worker manifests. This gives slow-registering workers (e.g. Redfish, ~17 min) time to claim their BMH slot, reducing the chance of machine-api grabbing the extra worker for the primary MachineSet. The wait is non-fatal (default 120 retries * 10s = 20 min): if not all workers register within the timeout (e.g. permanently-unreachable worker BMH in some dualstack environments), the extra workers are applied anyway to preserve existing fallback behavior. The retry count is configurable via EXTRA_WORKER_WAIT_RETRIES. Assisted-By: Claude Opus 4.6
|
/lgtm |
|
/retest ah great, the same unrelated test failed here and in another job in a different patch |
|
/retest |
|
/retest more unrelated failures |
|
/retest |
2 similar comments
|
/retest |
|
/retest |
| break | ||
| fi | ||
| echo "Waiting: ${ASSIGNED}/${NUM_WORKERS} worker machines have a host assigned" | ||
| sleep 10 |
There was a problem hiding this comment.
This is 20 minutes of waiting by default, are you sure the environments with unreachable workers won't timeout anyway?
There was a problem hiding this comment.
the timeout is a non fatal wait, and 20 minutes should be ok in the overall time budget
we can evaluate after some runs if we want to reduce it to 15 or even 10 minute, but I would start with this
|
/retest |
|
/retest-required |
|
/retest |
|
/retest-required |
|
/retest |
1 similar comment
|
/retest |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dtantsur The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/retest |
a0f4896
into
openshift-metal3:master
3 participants
Wait for all worker Machines to have a BareMetalHost assigned before applying extra worker manifests. This gives slow-registering workers (e.g. Redfish, ~17 min) time to claim their BMH slot, reducing the chance of machine-api grabbing the extra worker for the primary MachineSet.
The wait is non-fatal (default 120 retries * 10s = 20 min): if not all workers register within the timeout (e.g. permanently-unreachable worker BMH in some dualstack environments), the extra workers are applied anyway to preserve existing fallback behavior. The retry count is configurable via EXTRA_WORKER_WAIT_RETRIES.
Assisted-By: Claude Opus 4.6