fix(scheduler): make NUMA sort opt-in for binpack/spread policies#1874
fix(scheduler): make NUMA sort opt-in for binpack/spread policies#1874mesutoezdil wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mesutoezdil The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Code Review
This pull request introduces the hami.io/topology-aware-scoring annotation, allowing users to disable NUMA-aware GPU scheduling. It also enhances error handling by ensuring that failures in decoding pod devices are logged and handled gracefully without stopping the monitoring or scheduling processes. Corresponding unit tests were added for these changes. Feedback was provided to optimize performance in the scheduler by moving loop-invariant calculations for GPU policy and NUMA settings outside of the node iteration loop.
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
f84e65a to
0477f77
Compare
|
Fixed: moved userGPUPolicy and numaIgnore outside the loop. |
0477f77 to
b4d12ae
Compare
deae042 to
206023b
Compare
any news @archlitchi |
206023b to
ccde489
Compare
ccde489 to
eedf4f3
Compare
eedf4f3 to
cd18c31
Compare
cd18c31 to
dddef02
Compare
dddef02 to
1163746
Compare
Add NumaIgnore bool to DeviceUsageList. When the pod annotation hami.io/topology-aware-scoring is set to "false", Less() sorts by Score only so spread and binpack are not overridden by NUMA grouping. Default behavior is unchanged (NUMA-first, backward compatible). Fixes Project-HAMi#1806 Signed-off-by: mesutoezdil <mesudozdil@gmail.com>
1163746 to
4c3d2dd
Compare
What type of PR is this?
/kind bug
When spread or binpack policy is selected, Less() in pkg/scheduler/policy/gpu_policy.go uses NUMA as the primary sort key.
This overrides the policy intent: a busy NUMA node gets picked over an idle one on the other NUMA node.
Fix: add NumaIgnore bool to DeviceUsageList.
When the pod annotation hami.io/topology-aware-scoring: "false" is set, Less() sorts by Score only.
Default is unchanged (NUMA-first, backward compatible).
Which issue(s) this PR fixes: Fixes #1806
Does this PR introduce a user-facing change?
Yes. Users can now set hami.io/topology-aware-scoring: "false" on a pod to make spread/binpack sort by Score without NUMA grouping.