Skip to content

fix(scheduler): make NUMA sort opt-in for binpack/spread policies#1874

Open
mesutoezdil wants to merge 1 commit into
Project-HAMi:masterfrom
mesutoezdil:fix/numa-ignore-for-binpack-spread
Open

fix(scheduler): make NUMA sort opt-in for binpack/spread policies#1874
mesutoezdil wants to merge 1 commit into
Project-HAMi:masterfrom
mesutoezdil:fix/numa-ignore-for-binpack-spread

Conversation

@mesutoezdil

@mesutoezdil mesutoezdil commented May 15, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?
/kind bug

When spread or binpack policy is selected, Less() in pkg/scheduler/policy/gpu_policy.go uses NUMA as the primary sort key.
This overrides the policy intent: a busy NUMA node gets picked over an idle one on the other NUMA node.

Fix: add NumaIgnore bool to DeviceUsageList.
When the pod annotation hami.io/topology-aware-scoring: "false" is set, Less() sorts by Score only.
Default is unchanged (NUMA-first, backward compatible).

Which issue(s) this PR fixes: Fixes #1806

Does this PR introduce a user-facing change?
Yes. Users can now set hami.io/topology-aware-scoring: "false" on a pod to make spread/binpack sort by Score without NUMA grouping.

@hami-robot hami-robot Bot added kind/bug Something isn't working dco-signoff: yes labels May 15, 2026
@hami-robot hami-robot Bot requested review from archlitchi and wawa0210 May 15, 2026 11:39
@hami-robot

hami-robot Bot commented May 15, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mesutoezdil
Once this PR has been reviewed and has the lgtm label, please assign wawa0210 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the hami.io/topology-aware-scoring annotation, allowing users to disable NUMA-aware GPU scheduling. It also enhances error handling by ensuring that failures in decoding pod devices are logged and handled gracefully without stopping the monitoring or scheduling processes. Corresponding unit tests were added for these changes. Feedback was provided to optimize performance in the scheduler by moving loop-invariant calculations for GPU policy and NUMA settings outside of the node iteration loop.

Comment thread pkg/scheduler/scheduler.go Outdated
@codecov

codecov Bot commented May 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 58.53% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/scheduler/policy/gpu_policy.go 100.00% <100.00%> (ø)
pkg/scheduler/scheduler.go 53.07% <100.00%> (-0.20%) ⬇️
pkg/util/types.go 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mesutoezdil mesutoezdil force-pushed the fix/numa-ignore-for-binpack-spread branch from f84e65a to 0477f77 Compare May 15, 2026 11:48
@mesutoezdil

Copy link
Copy Markdown
Contributor Author

Fixed: moved userGPUPolicy and numaIgnore outside the loop.

@wawa0210

wawa0210 commented Jun 4, 2026

Copy link
Copy Markdown
Member

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


any news @archlitchi

@mesutoezdil mesutoezdil force-pushed the fix/numa-ignore-for-binpack-spread branch from 206023b to ccde489 Compare June 4, 2026 13:14
@mesutoezdil mesutoezdil force-pushed the fix/numa-ignore-for-binpack-spread branch from ccde489 to eedf4f3 Compare June 14, 2026 15:47
@mesutoezdil mesutoezdil force-pushed the fix/numa-ignore-for-binpack-spread branch from eedf4f3 to cd18c31 Compare June 14, 2026 15:50
Add NumaIgnore bool to DeviceUsageList. When the pod annotation
hami.io/topology-aware-scoring is set to "false", Less() sorts by
Score only so spread and binpack are not overridden by NUMA grouping.
Default behavior is unchanged (NUMA-first, backward compatible).

Fixes Project-HAMi#1806

Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gpu scheduler policy sort bug

2 participants