Skip to content

fix: Align backup alert coverage with MCC - #273

Open
philippemnoel wants to merge 1 commit into
devfrom
fix/alert-parity-backups
Open

philippemnoel wants to merge 1 commit into
devfrom
fix/alert-parity-backups

Conversation

@philippemnoel

Copy link
Copy Markdown
Member

What

Use kube-state-metrics Cluster backup-status timestamps for stale backups, including plugin backups. Add CNPGBackupFailed for the latest failed backup (including no successful history), and CNPGScheduledBackupStalled for a schedule overdue by more than one hour. Both new warnings require 15 minutes; stale backup remains critical after 26 hours plus 15 minutes.

Extend the supplied custom-resource metrics configuration with ScheduledBackup's next schedule timestamp. Document that operators must apply this configuration and grant kube-state-metrics list/watch permissions on both Clusters and ScheduledBackups alongside the chart upgrade. A cluster with neither successful-backup history nor a reported failure remains outside the stale/failed rules, as documented.

Tests

  • Helm lint with monitoring and PrometheusRule enabled passed.
  • Existing Prometheus-rule rendering checks and alert exclusions passed.
  • Helm-rendered rule regression tests passed with vmalert-tool v1.148.0: Failure after success, recovery after failure, first failed backup, stale plugin backups, success through another backup method, overdue/future schedules, and no backup history.
  • Added a Chainsaw test directory, automatically discovered by the existing CI workflow, to run the rendered rule fixtures.
  • git diff --check passed.

Kubernetes integration remains for CI. No deployment was performed. Chart versions are generated by the publishing workflow.

@philippemnoel
philippemnoel changed the base branch from main to dev September 26, 2026 01:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant