Skip to content

fix(kueue): harden upgrade tests — drop version-specific labels and assert Workload cleanup - #2269

Merged
jstourac merged 2 commits into
opendatahub-io:mainfrom
jstourac:kueueMain
Aug 26, 2026
Merged

fix(kueue): harden upgrade tests — drop version-specific labels and assert Workload cleanup#2269
jstourac merged 2 commits into
opendatahub-io:mainfrom
jstourac:kueueMain

Conversation

@jstourac

Copy link
Copy Markdown
Member

Summary

  • Remove assertions for kueue.x-k8s.io/cluster-queue-name and kueue.x-k8s.io/local-queue-name pod labels. These are observability aids introduced in upstream Kueue v0.17.0 (RH Kueue Operator >= 1.4) and are not present on v1.3.x environments. The functional assertions (kueue.x-k8s.io/managed, queue-name, Workload admission) remain.
  • Assert that Kueue garbage-collects Workload objects after notebook deletion, failing with a clear message if cleanup doesn't happen within 60s. Without this, orphaned Workloads hold resource-in-use finalizers on ClusterQueue/ResourceFlavor, blocking teardown.

This was split out from #2243 and as such is a partial forward of the #2142.

Related Issues

  • Fixes:
  • JIRA:

Please review and indicate how it has been tested

  • Locally
  • Jenkins

Additional Requirements

  • If this PR introduces a new test image, did you create a PR to mirror it in disconnected environment?
  • If this PR introduces new marker(s)/adds a new component, was relevant ticket created to update relevant Jenkins job?

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: ce7cf281-66a8-4508-af14-463daef51aec


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

The following are automatically added/executed:

  • PR size label.
  • Run pre-commit
  • Run tox
  • Add PR author as the PR assignee
  • Build image based on the PR

Available user actions:

  • To mark a PR as WIP, add /wip in a comment. To remove it from the PR comment /wip cancel to the PR.
  • To block merging of a PR, add /hold in a comment. To un-block merging of PR comment /hold cancel.
  • To mark a PR as approved, add /lgtm in a comment. To remove, add /lgtm cancel.
    lgtm label removed on each new commit push.
  • To mark PR as verified comment /verified to the PR, to un-verify comment /verified cancel to the PR.
    verified label removed on each new commit push.
  • To Cherry-pick a merged PR /cherry-pick <target_branch_name> to the PR. If <target_branch_name> is valid,
    and the current PR is merged, a cherry-picked PR would be created and linked to the current PR.
  • To build and push image to quay, add /build-push-pr-image in a comment. This would create an image with tag
    pr-<pr_number> to quay repository. This image tag, however would be deleted on PR merge or close action.
Supported labels

{'/lgtm', '/wip', '/build-push-pr-image', '/cherry-pick', '/hold', '/verified'}

@jstourac
jstourac enabled auto-merge (rebase) August 25, 2026 13:47

@hmtosi hmtosi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

jstourac and others added 2 commits August 25, 2026 22:20
The labels kueue.x-k8s.io/cluster-queue-name and
kueue.x-k8s.io/local-queue-name are observability aids introduced in
upstream Kueue v0.17.0 (RH Kueue Operator >= 1.4). They do not affect
workload admission, pod scheduling, or any functional behavior.

Asserting their presence caused the entire kueue upgrade test suite to
fail on environments running RH Kueue Operator v1.3.x (upstream
v0.16.5), cascading into 9+ test errors/failures despite Kueue
functioning correctly.

The remaining assertions (kueue.x-k8s.io/managed, queue-name) along
with workload admission checks are sufficient to validate Kueue
integration across all supported operator versions.

Signed-off-by: Jan Stourac <jstourac@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
After deleting a notebook, wait for Kueue to garbage-collect the
associated Workload objects. Fail with a clear message if Kueue doesn't
clean them up within 60 seconds — this indicates a potential issue with
Kueue's reconciliation of StatefulSet-backed workloads.

Without this, orphaned Workloads hold kueue.x-k8s.io/resource-in-use
finalizers on the ClusterQueue and ResourceFlavor, blocking their
deletion and causing teardown timeouts.

Signed-off-by: Jan Stourac <jstourac@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@jstourac
jstourac merged commit c9aba6a into opendatahub-io:main Aug 26, 2026
12 checks passed
@jstourac
jstourac deleted the kueueMain branch August 26, 2026 08:03
@github-actions

Copy link
Copy Markdown

Status of building tag latest: success.
Status of pushing tag latest to image registry: success.
Image manifest label: success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants