[deckhouse] add queue head info metric and critical label for severity-tiered alerting by diyliv · Pull Request #20476 · deckhouse/deckhouse

diyliv · 2026-06-04T13:43:11Z

Description

Changes in addon-operator:

New metric deckhouse_tasks_queue_head_info with labels queue, module, task_type, hook — shows what task is at the head of each non-empty queue.
New label critical on deckhouse_mm_module_info — value from BasicModule.GetCritical().

Why do we need it, and what problem does it solve?

deckhouse_tasks_queue_length exposes only queue — no visibility into which task/module/hook is at the head of a hung queue. Severity is always "7" regardless of module criticality. deckhouse_mm_module_info lacks a critical label.

This change enables severity-tiered alerting based on module criticality: critical module queues (e.g., cni-cilium) get severity 4, non-critical (e.g., console) get severity 6, and global/parallel task hangs get severity 4. Operators get immediate signal about what is stuck and how important it is.

Example

`critical` label on `deckhouse_mm_module_info`

Critical modules (26 total):

deckhouse_mm_module_info{critical="true",enabled="true",module="chrony"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="cloud-provider-dvp"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="cni-cilium"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="control-plane-manager"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="deckhouse"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="kube-dns"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="local-path-provisioner"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="node-manager"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="registry"} 1
deckhouse_mm_module_info{critical="true",enabled="true",module="registry-packages-proxy"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cloud-provider-aws"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cloud-provider-azure"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cloud-provider-gcp"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cloud-provider-openstack"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cloud-provider-vsphere"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="cni-flannel"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="kube-proxy"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="terraform-manager"} 1
deckhouse_mm_module_info{critical="true",enabled="false",module="user-authn"} 1

Non-critical enabled modules (29 total):

deckhouse_mm_module_info{critical="false",enabled="true",module="admission-policy-engine"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="cert-manager"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="console"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="documentation"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="ingress-nginx"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="monitoring-deckhouse"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="monitoring-kubernetes"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="prometheus"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="upmeter"} 1
deckhouse_mm_module_info{critical="false",enabled="true",module="user-authz"} 1

Verified on dev cluster: 26 modules with critical="true", 41 with critical="false".

`deckhouse_tasks_queue_head_info` — startup phase (non-empty queues)

Captured during a deckhouse pod restart. All queue head changes were correctly reflected as the startup progressed:

Global hook task (module=""):

deckhouse_tasks_queue_head_info{hook="discovery/kubernetes_version.go",module="",queue="main",task_type="GlobalHookEnableKubernetesBindings"} 1

Parallel module run (ParallelModuleRun normalised to module=""):

deckhouse_tasks_queue_head_info{hook="",module="",queue="main",task_type="ParallelModuleRun"} 1

Module run tasks (module=):

deckhouse_tasks_queue_head_info{hook="",module="control-plane-manager",queue="parallel_queue_0",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="node-manager",queue="parallel_queue_1",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="admission-policy-engine",queue="parallel_queue_9",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="cert-manager",queue="parallel_queue_7",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="console",queue="parallel_queue_0",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="ingress-nginx",queue="parallel_queue_5",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="monitoring-custom",queue="parallel_queue_5",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="upmeter",queue="parallel_queue_2",task_type="ModuleRun"} 1
deckhouse_tasks_queue_head_info{hook="",module="user-authz",queue="parallel_queue_14",task_type="ModuleRun"} 1

Module hook tasks (hook=):

deckhouse_tasks_queue_head_info{hook="300-prometheus/hooks/disk_metrics.go",module="prometheus",queue="/modules/prometheus/disk_metrics",task_type="ModuleHookRun"} 1
deckhouse_tasks_queue_head_info{hook="402-ingress-nginx/hooks/safe_daemonset_update.go",module="ingress-nginx",queue="/modules/ingress-nginx/safe_daemonset_update",task_type="ModuleHookRun"} 1

`deckhouse_tasks_queue_head_info` — steady state (all queues empty)

# No series produced — all queues are empty
deckhouse_tasks_queue_head_info  →  (no results)

Verified on dev cluster: metric correctly produces series only for non-empty queues, and they expire within one scrape interval after the queue drains.

Recording rule `d8:deckhouse_queue_is_hung:with_head`

During a running ConvergeModules task on the main queue:

d8:deckhouse_queue_is_hung:with_head{queue="main",module="",task_type="ConvergeModules",hook=""} 1

The recording rule correctly joins deckhouse_tasks_queue_length with deckhouse_tasks_queue_head_info on the queue label and uses min_over_time(..., __SCRAPE_INTERVAL_X_3__) to avoid flapping on short-lived queue spikes.

Checklist

Unit tests pass
Alert rules in deckhouse (separate PR).
Documentation updated.
Deployed to dev cluster, critical label and tasks_queue_head_info verified.

Changelog entries

section: deckhouse
type: feature
summary: Rework D8DeckhouseQueueIsHung alert with queue head info and severity based on module criticality.

Copilot

Pull request overview

This PR updates Deckhouse monitoring to support severity-tiered alerting for hung task queues by enriching queue-hang signals with “queue head” context and module criticality, and bumps the addon-operator dependency to a commit that provides the required metrics/labels.

Changes:

Added a recording rule d8:deckhouse_queue_is_hung:with_head that joins queue length with the new deckhouse_tasks_queue_head_info metric.
Split the existing hung-queue alert into D8DeckhouseQueueIsHungCritical, D8DeckhouseQueueIsHung (non-critical), and D8DeckhouseQueueIsHungGlobal with updated severities and annotations.
Updated Deckhouse alert catalog docs and Go module dependencies (addon-operator and go-openapi/swag/* entries).

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

File	Description
modules/340-monitoring-deckhouse/monitoring/prometheus-rules/deckhouse.yaml	Adds the recording rule with head info and introduces new critical/non-critical/global queue hang alerts.
docs/documentation/_data/deckhouse-alerts.yml	Updates alert catalog entries to reflect the new/renamed alerts and severities.
go.mod	Bumps `github.com/flant/addon-operator` and adjusts `go-openapi/swag/*` requirements.
go.sum	Updates checksums for the new `addon-operator` version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

maxinsky · 2026-06-29T18:41:58Z

+      plk_labels_as_annotations: "module,task_type,hook"
+      summary: Critical module queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.
+      description: |
+        Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.


Suggested change

Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

The critical module queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Module: `{{ $labels.module }}`

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

```bash

d8 k -n d8-system logs -f -l app=deckhouse

```

maxinsky · 2026-06-29T18:43:51Z

-      plk_labels_as_annotations: "instance,pod"
-      summary: The `{{ $labels.queue }}` Deckhouse queue is stuck with {{ $value }} pending task(s).
+      plk_labels_as_annotations: "module,task_type,hook"
+      summary: Queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.


Suggested change

summary: Queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.

summary: Module queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.

maxinsky · 2026-06-29T18:44:14Z

+      plk_labels_as_annotations: "module,task_type,hook"
+      summary: Queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.
+      description: |
+        Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.


Suggested change

Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

The module queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Module: `{{ $labels.module }}`

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

```bash

d8 k -n d8-system logs -f -l app=deckhouse

```

maxinsky · 2026-06-29T18:46:07Z

+        Global/Parallel queue head stayed for more than 10 minutes. task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

        To investigate the issue, check the logs by running the following command:


Suggested change

Global/Parallel queue head stayed for more than 10 minutes. task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

To investigate the issue, check the logs by running the following command:

The Global/Parallel queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

maxinsky · 2026-06-29T18:47:47Z

+        Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.
+      summary: |
+        Queue {{ $labels.module }} (type {{ $labels.task_type }}, hook {{ $labels.hook }}) is hung.


Suggested change

Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

summary: |

Queue {{ $labels.module }} (type {{ $labels.task_type }}, hook {{ $labels.hook }}) is hung.

The module queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Module: `{{ $labels.module }}`

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

```bash

d8 k -n d8-system logs -f -l app=deckhouse

```

summary: |

Module queue {{ $labels.module }} (type {{ $labels.task_type }}, hook {{ $labels.hook }}) is hung.

maxinsky · 2026-06-29T18:48:46Z

+      module: monitoring-deckhouse
+      edition: ce
+      description: |
+        Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.


Suggested change

Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

The critical module queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Module: `{{ $labels.module }}`

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

```bash

d8 k -n d8-system logs -f -l app=deckhouse

```

maxinsky · 2026-06-29T18:50:21Z

+        Global/Parallel queue head stayed for more than 10 minutes. task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

        To investigate the issue, check the logs by running the following command:


Suggested change

Global/Parallel queue head stayed for more than 10 minutes. task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

To investigate the issue, check the logs by running the following command:

The Global/Parallel queue head has remained unchanged for more than 10 minutes.

Affected queue:

- Task type: `{{ $labels.task_type }}`

- Hook: `{{ $labels.hook }}`

To investigate the issue, check the logs of the `deckhouse` controller pods:

Signed-off-by: diyliv <onlogn081@gmail.com>

github-actions Bot assigned diyliv Jun 4, 2026

github-actions Bot added type/dependencies Pull requests that update a dependency file area/core Pull requests that update core modules security/gitleaks/success Gitleaks PR diff scan passed labels Jun 4, 2026

diyliv marked this pull request as draft June 4, 2026 13:45

diyliv changed the title ~~[addon-operator] rework D8DeckhouseQueueIsHung alert with queue head info~~ [addon-operator] added new metrics Jun 4, 2026

diyliv changed the title ~~[addon-operator] added new metrics~~ deckhouse] add queue head info metric and critical label for severity-tiered alerting Jun 5, 2026

diyliv changed the title ~~deckhouse] add queue head info metric and critical label for severity-tiered alerting~~ [deckhouse] add queue head info metric and critical label for severity-tiered alerting Jun 5, 2026

ldmonster requested a review from Copilot June 5, 2026 11:45

Copilot started reviewing on behalf of ldmonster June 5, 2026 11:45 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

github-actions Bot removed security/antivirus/success Dr.Web and Kaspersky antivirus scans passed security/rootless/success Default-user (rootless) validation passed security/cve/failed Trivy CVE scan on PR images failed labels Jun 6, 2026

github-actions Bot added the go Pull requests that update Go code label Jun 23, 2026

diyliv marked this pull request as ready for review June 23, 2026 08:20

diyliv requested review from Zhbert, lazovskiy, ldmonster, maksimmadzhuga and vladimirGuryanov as code owners June 23, 2026 08:20

z9r5 requested review from maxinsky and z9r5 and removed request for Zhbert June 29, 2026 12:51

maxinsky reviewed Jun 29, 2026

View reviewed changes

diyliv marked this pull request as draft July 3, 2026 08:58

diyliv force-pushed the feature/converge_modules branch from 8910416 to 24d2e90 Compare July 4, 2026 17:58

github-actions Bot added area/cluster-and-infrastructure Pull requests that update infra modules and removed area/monitoring Pull requests that update monitoring modules type/dependencies Pull requests that update a dependency file labels Jul 4, 2026

diyliv force-pushed the feature/converge_modules branch from e053f01 to aebb8f4 Compare July 4, 2026 18:15

fix queue alerts

bc30520

Signed-off-by: diyliv <onlogn081@gmail.com>

diyliv force-pushed the feature/converge_modules branch from 5e4b0e6 to bc30520 Compare July 5, 2026 12:25

github-actions Bot removed the go Pull requests that update Go code label Jul 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[deckhouse] add queue head info metric and critical label for severity-tiered alerting #20476

[deckhouse] add queue head info metric and critical label for severity-tiered alerting #20476
diyliv wants to merge 1 commit into
mainfrom
feature/converge_modules

diyliv commented Jun 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

maxinsky Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-        Queue head stayed for more than 10 minutes. Module=`{{ $labels.module }}`, task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.
+        The critical module queue head has remained unchanged for more than 10 minutes.
+        Affected queue:
+        - Module: `{{ $labels.module }}`
+        - Task type: `{{ $labels.task_type }}`
+        - Hook: `{{ $labels.hook }}`
+        To investigate the issue, check the logs of the `deckhouse` controller pods:
+        ```bash
+        d8 k -n d8-system logs -f -l app=deckhouse
+        ```

	summary: Queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.
	summary: Module queue `{{ $labels.module }}` (type `{{ $labels.task_type }}`, hook `{{ $labels.hook }}`) is hung.

		Global/Parallel queue head stayed for more than 10 minutes. task=`{{ $labels.task_type }}`, hook=`{{ $labels.hook }}`.

		To investigate the issue, check the logs by running the following command:

Uh oh!

Conversation

diyliv commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Why do we need it, and what problem does it solve?

Example

critical label on deckhouse_mm_module_info

deckhouse_tasks_queue_head_info — startup phase (non-empty queues)

deckhouse_tasks_queue_head_info — steady state (all queues empty)

Recording rule d8:deckhouse_queue_is_hung:with_head

Checklist

Changelog entries

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

maxinsky Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diyliv commented Jun 4, 2026 •

edited

Loading

`critical` label on `deckhouse_mm_module_info`

`deckhouse_tasks_queue_head_info` — startup phase (non-empty queues)

`deckhouse_tasks_queue_head_info` — steady state (all queues empty)

Recording rule `d8:deckhouse_queue_is_hung:with_head`