rollout

Example: a staged rollout with a DeploymentPolicy

An interruptionBudget caps how many nodes run at once. A DeploymentPolicy goes further: it splits the fleet into compartments, each with its own batch strategy and its own tolerance for failure, so one rollout can be cautious with some nodes and quick with others.

This example sorts six workers into three groups:

Group	Nodes	Rolls out	Stops when
canary	2	one at a time	a batch fails, so the second node never starts
production	3	1 node, then 2, growing by one per good batch	two batches in a row fail, with a node still waiting
default	1	up to two at a time	(it has a single node, so there is nothing left to stop)

Nodes that match no compartment fall into the policy's default.

Prerequisites

At least six worker nodes; the walkthrough uses exactly six and ignores any others. To make a throwaway cluster:

kind create cluster --name nodewright-examples --config examples/rollout/kind-config.yaml

and install NodeWright.

Run it

Opt every worker in, then sort them into compartments:

WORKERS=($(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name | cut -d/ -f2 | head -6))

for n in "${WORKERS[@]}"; do
  kubectl label node "$n" nodewright.nvidia.com/example=rollout
done
for n in "${WORKERS[@]:0:2}"; do
  kubectl label node "$n" nodewright.nvidia.com/example-tier=canary
done
for n in "${WORKERS[@]:2:3}"; do
  kubectl label node "$n" nodewright.nvidia.com/example-tier=production
done
# the last one carries no tier label and falls into `default`

Create the policy before the NodeWright, because the webhook rejects a reference to a policy that does not exist:

kubectl apply -f examples/rollout/deployment-policy.yaml
kubectl apply -f examples/rollout/nodewright.yaml

Watch the nodes move through in batches (each takes about 20 seconds):

kubectl get nodewright rollout-demo -o jsonpath='{.status.nodeStatus}' ; echo
watch -n 5 "kubectl get nodewright rollout-demo -o jsonpath='{.status.compartmentStatuses}'"

status.compartmentStatuses has one entry per compartment, including its batch state. Compartments run independently, so canary and production move at the same time.

See a compartment stop

The package fails on any node that has /etc/nodewright-example/rollout/fail. Which canary node goes first is not something to rely on, so make both canary nodes fail. From a debug pod on each canary node, run chroot /host sh -c 'mkdir -p /etc/nodewright-example/rollout && touch /etc/nodewright-example/rollout/fail'. A completed package is not re-run on its own, so reset the NodeWright to send it through again:

kubectl nodewright reset rollout-demo --confirm

The first canary node retries, then reads erroring. Its batch failed with 1 of 2 canary nodes processed, which is 50%. That is below safetyLimit: 100, so failureThreshold: 1 is enforced and the canary compartment stops: the second canary node never starts. The production and default compartments keep going. Stopping applies to the compartment that failed, not the whole NodeWright.

A stopped compartment stays stopped until its batch state is reset. After removing the marker files from both canary nodes, reset clears the failed node and the batch state together, and the rollout starts over:

kubectl nodewright reset rollout-demo --confirm

Two things that surprise people

• Once a compartment has a strategy, its budget no longer caps how many nodes run at once. The strategy sets the batch size. To cap concurrency, use fixed with initialBatch: N. The budget still breaks ties for a node that matches several compartments.
• A compartment cannot stop on its last batch. Stopping needs progress, counting failed nodes as processed, to be below safetyLimit. When the final batch fails, progress reaches 100% and there is nothing left to hold back. That is why the canary group has two nodes: with one, a failure could never stop anything.
• safetyLimit defaults to 50. Past half of a compartment, the rollout commits to finishing and stops enforcing failureThreshold. These manifests set it to 100 so a failure stops a rollout at any point.

Clean up

kubectl delete -f examples/rollout/nodewright.yaml
kubectl delete -f examples/rollout/deployment-policy.yaml
for n in "${WORKERS[@]}"; do
  kubectl label node "$n" nodewright.nvidia.com/example- nodewright.nvidia.com/example-tier-
done
# if you planted the failure marker, remove it from the canary nodes (debug pod):
#   chroot /host rm -rf /etc/nodewright-example/rollout
kind delete cluster --name nodewright-examples   # if you made one

Related

• Deployment Policy and Compartments: strategies, thresholds, overlapping selectors, batch state
• Strict Ordering: SKYHOOK_NODE_ORDER, the position of each node in a rollout