Skip to content

Reduce race condition in extra worker BMH application #1947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Aug 20, 2026
+24 −18
Merged

Reduce race condition in extra worker BMH application #1947

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 24 additions & 18 deletions 06_create_cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,25 +92,31 @@ fi

if [[ -n "${APPLY_EXTRA_WORKERS:-}" ]]; then
if [[ ${NUM_EXTRA_WORKERS} -ne 0 && -s "${OCP_DIR}/extra_host_manifests.yaml" ]]; then
# Wait for all worker Machines to have a BMH assigned before applying
# extra workers, to prevent the machine-api from claiming extra worker
# hosts for the primary MachineSet (race when regular workers are slow
# to register via Redfish).
# Give regular worker Machines time to claim their BMHs before
# introducing extra workers. This reduces the chance of machine-api
# grabbing an extra worker host for the primary MachineSet when a
# regular worker is slow to register (e.g. Redfish). If they don't
# all register within the timeout we proceed anyway — some
# environments have permanently-unreachable worker BMHs and rely on
# the extra worker filling the gap.
EXTRA_WORKER_WAIT_RETRIES=${EXTRA_WORKER_WAIT_RETRIES:-120}
echo "Waiting for all ${NUM_WORKERS} worker machines to have a host assigned..."
timeout 30m bash -c '
while true; do
ASSIGNED=$(oc get machines -n openshift-machine-api \
-l machine.openshift.io/cluster-api-machine-role=worker \
-o json 2>/dev/null | \
jq "[.items[].metadata.annotations // {} | .[\"metal3.io/BareMetalHost\"] // empty] | length")
if [[ "${ASSIGNED}" -ge "'"${NUM_WORKERS}"'" ]]; then
echo "All '"${NUM_WORKERS}"' worker machines have a host assigned"
break
fi
echo "Waiting: ${ASSIGNED}/'"${NUM_WORKERS}"' worker machines have a host assigned"
sleep 10
done
'
ASSIGNED=0
for ((i = 0; i < EXTRA_WORKER_WAIT_RETRIES; i++)); do
ASSIGNED=$(oc get machines -n openshift-machine-api \
-l machine.openshift.io/cluster-api-machine-role=worker \
-o json 2>/dev/null | \
jq "[.items[].metadata.annotations // {} | .[\"metal3.io/BareMetalHost\"] // empty] | length")
if [[ "${ASSIGNED}" -ge "${NUM_WORKERS}" ]]; then
echo "All ${NUM_WORKERS} worker machines have a host assigned"
break
fi
echo "Waiting: ${ASSIGNED}/${NUM_WORKERS} worker machines have a host assigned"
sleep 10

@dtantsur dtantsur Aug 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is 20 minutes of waiting by default, are you sure the environments with unreachable workers won't timeout anyway?

@elfosardo elfosardo Aug 20, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the timeout is a non fatal wait, and 20 minutes should be ok in the overall time budget
we can evaluate after some runs if we want to reduce it to 15 or even 10 minute, but I would start with this

done
if [[ "${ASSIGNED}" -lt "${NUM_WORKERS}" ]]; then
echo "WARNING: only ${ASSIGNED}/${NUM_WORKERS} worker machines have a host assigned, proceeding with extra workers anyway"
fi

oc apply -f "${OCP_DIR}/extra_host_manifests.yaml"

Expand Down