Skip to content

fix(quota): enforce explicit zero limit in ResourceQuota#1904

Open
iasthc wants to merge 3 commits into
Project-HAMi:masterfrom
iasthc:fix-quota
Open

fix(quota): enforce explicit zero limit in ResourceQuota#1904
iasthc wants to merge 3 commits into
Project-HAMi:masterfrom
iasthc:fix-quota

Conversation

@iasthc

@iasthc iasthc commented May 25, 2026

Copy link
Copy Markdown

What type of PR is this?

/kind bug

What this PR does / why we need it:

QuotaManager used Limit != 0 to decide whether a quota was configured. As a result, setting limits.nvidia.com/gpumem: "0" (or gpucores: "0") was silently treated as "no limit" and every pod was allowed through, instead of blocking GPU usage in the namespace as the docs describe.

This PR adds a LimitSet flag on Quota to distinguish an explicitly-set zero from an auto-initialized entry. FitQuota now gates on LimitSet instead of Limit != 0, so an explicit zero is honored as a hard block.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

  • AddQuota sets LimitSet = true; DelQuota clears it.
  • Entries auto-created by AddUsage keep LimitSet = false, so behavior when no quota is configured is unchanged.
  • Existing NVIDIA-only / nvidia.com/gpu == 1 limitations in the webhook are untouched.
  • New tests: TestFitQuotaZeroLimitEnforced, TestAddQuotaExplicitZero. Existing fixtures updated.

Does this PR introduce a user-facing change?:

Honor an explicit zero in limits.nvidia.com/gpumem / limits.nvidia.com/gpucores ResourceQuota as a hard block instead of silently treating it as "no limit".

@hami-robot hami-robot Bot added kind/bug Something isn't working dco-signoff: no labels May 25, 2026
@hami-robot hami-robot Bot requested a review from FouoF May 25, 2026 14:16
@hami-robot

hami-robot Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: iasthc
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot hami-robot Bot requested a review from wawa0210 May 25, 2026 14:16
@hami-robot

hami-robot Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Welcome @iasthc! It looks like this is your first PR to Project-HAMi/HAMi 🎉

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a LimitSet boolean to the Quota struct to distinguish between an unconfigured limit and an explicit zero limit, ensuring that zero-value quotas are correctly enforced. The logic in FitQuota, AddQuota, and DelQuota has been updated to utilize this flag, and new test cases verify the enforcement of zero limits. Feedback indicates that the GetResourceQuota function needs to be updated to include the LimitSet field when copying objects to prevent data loss for API consumers.

Comment thread pkg/device/quota.go
@hami-robot hami-robot Bot added size/L and removed size/M labels May 25, 2026
@iasthc

iasthc commented May 25, 2026

Copy link
Copy Markdown
Author

Hi @DSFans2014, this fixes a bug where limits.nvidia.com/gpumem: "0" in ResourceQuota was silently ignored. Would appreciate your review when you have time, thanks!

@codecov

codecov Bot commented May 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 58.36% <100.00%> (+0.21%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/device/quota.go 75.16% <100.00%> (+12.49%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@iasthc iasthc requested a deployment to nvidia May 26, 2026 10:40 — with GitHub Actions Abandoned
Comment thread pkg/device/quota.go
Limit int64
Used int64
Limit int64
LimitSet bool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can set the default value of limit to MaxInt64 to avoid introducing a new variable. The information from ResourceQuota will be used in metrics as below.

for ns, val := range sher.GetQuotaManager().GetResourceQuota() {
for quotaname, q := range *val {
ch <- prometheus.MustNewConstMetric(
quotaUsedDesc,
prometheus.GaugeValue,
float64(q.Used),
ns, quotaname, fmt.Sprint(q.Limit),
)
if legacy {
ch <- prometheus.MustNewConstMetric(
legacyQuotaUsed,
prometheus.GaugeValue,
float64(q.Used),
ns, quotaname, fmt.Sprint(q.Limit),
)
}
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I think LimitSet is still the safer choice here:

  1. FitQuota would overflow. pkg/device/quota.go#L75-L78 multiplies Limit by memoryFactor. With Limit == MaxInt64 and memoryFactor > 1 this overflows to a negative number, and legitimate requests get rejected. Guarding against it requires an if Limit == MaxInt64 check — another sentinel flag in disguise.

  2. Metric label. Fair point that the rendering needs work, but limit="9223372036854775807" isn't friendlier than "0". I've pushed a follow-up that uses LimitSet to render "unlimited". This case is real today: AddUsage creates LimitSet=false entries for namespaces with usage but no ResourceQuota, and fix: rollback quota usage when filter annotation patch fails #1898 / fix(quota): roll back usage when Filter evicts a stale pod entry #1905 exercise those paths more often now.

  3. Sentinel semantics would spread. AddUsage and DelQuota would both need to use MaxInt64, and every reader has to remember 0 and MaxInt64 carry special meaning. A bool is one byte and explicit.

The metrics commit is on top of the quota fixes, rebased onto current master — PTAL.

iasthc added 3 commits May 28, 2026 16:50
Signed-off-by: Paul Tsai <paul_tsai@phison.com>
Signed-off-by: Paul Tsai <paul_tsai@phison.com>
…uota

Signed-off-by: Paul Tsai <paul_tsai@phison.com>

@mesutoezdil mesutoezdil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 more thing worth flagging: the metrics label changes from a numeric string to "unlimited" for entries where no quota is configured (LimitSet=false). before this pr those entries showed limit="0" in prometheus. after this pr they show limit="unlimited". any existing dashboard or alert filtering on limit="0" for the unset case will silently break. this is technically a more correct label, but it is a label value change and it affects cardinality. worth adding a note in the pr description or release notes so operators know to update their dashboards.

Comment thread cmd/scheduler/metrics.go
)
for ns, val := range sher.GetQuotaManager().GetResourceQuota() {
for quotaname, q := range *val {
limitLabel := "unlimited"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this changes the prometheus label value from "0" to "unlimited" for namespaces that have usage but no configured quota (LimitSet=false). existing dashboards or alerts that filter on limit="0" for the no-quota case will break silently. the new label is semantically correct, but the change should be called out explicitly for operators.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants