feat: add fault aggressive-scaledown-kubernetes-horizontal-pod-autoscaler

CONTRIBUTORS.md

@@ -42,3 +42,4 @@
 - Daby Sow
 - Nicholas C. M. Fuller
 - Ruchir Puri
+- Hugo Villarreal

documentation/library/faults/README.md

@@ -7,6 +7,7 @@ A fault is a solvable issue injected into an environment to create an incident.
 
 | Name | Platform |
 | --- | --- |
+| [Aggressive Scaledown Kubernetes Horizontal Pod Autoscaler](./aggressive-scaledown-kubernetes-horizontal-pod-autoscaler.md) | Kubernetes |
 | [Cordoned Kubernetes Worker Node](./cordoned-kubernetes-worker-node.md) | Kubernetes |
 | [Crashing Kubernetes Workload Init Container](./crashing-kubernetes-workload-init-container.md) | Kubernetes |
 | [Deleted Kubernetes Service](./deleted-kubernetes-service.md) | Kubernetes |

documentation/library/faults/aggressive-scaledown-kubernetes-horizontal-pod-autoscaler.md

@@ -0,0 +1,32 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Aggressive Scaledown Kubernetes Horizontal Pod Autoscaler
+
+| [Source](../../../scenarios/sre/library/indexes/faults/aggressive-scaledown-kubernetes-horizontal-pod-autoscaler.json) | [Schema](../../../schemas/json/faults/aggressive-scaledown-kubernetes-horizontal-pod-autoscaler.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_aggressive_scaledown_kubernetes_horizontal_pod_autoscaler.yaml) | Kubernetes | Deployment, Performance |
+
+This fault injects a configuration into a horizontal pod autoscaler that causes it to aggressively scale down workloads, keeping replica counts at a minimum even under high demand.
+
+## Expectation
+
+The faulted autoscaler will rapidly scale down the workload to its minimum replica count. Under load, the reduced number of pods will be unable to handle incoming traffic, causing request errors and elevated latency.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+#### Golden Signal Alerts
+
+- HighRequestErrorRate
+- HighRequestLatency
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/workloads/autoscaling/
+- https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
+- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/horizontal-pod-autoscaler-v2/

documentation/library/scenarios/README.md

@@ -47,3 +47,4 @@ _For statistics on the distrubution spread of scenarios, please see [here](./sta
 | [102](./sre/102.md) | sre | medium |
 | [105](./sre/105.md) | sre | medium |
 | [114](./sre/114.md) | sre | low |
+| [115](./sre/115.md) | sre | medium |

documentation/library/scenarios/sre/115.md

@@ -0,0 +1,50 @@
+> [!NOTE]
+> This file is autogenerated. Please do not edit.
+
+# Scenario 115
+
+| [Source](../../../../scenarios/sre/library/indexes/scenarios/115.json) | Medium Complexity | Kubernetes | Deployment, Performance |
+
+## Description
+
+This scenario simulates an OpenTelemetry Demo `frontend` service degradation caused by an HPA misconfigured to aggressively scale down pods under high demand.
+
+## Applications
+
+- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md)
+
+## Faults
+
+- [Aggressive Scaledown Kubernetes Horizontal Pod Autoscaler](../../../../scenarios/sre/docs/faults.md#Aggressive-Scaledown-Kubernetes-Horizontal-Pod-Autoscaler)
+- [OpenTelemetry Demo Feature Flag](../../../../scenarios/sre/docs/faults.md#OpenTelemetry-Demo-Feature-Flag)
+
+## Solution Guides
+
+Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario.
+
+A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance.
+
+A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem.
+
+### Solution 1
+
+#### Variant 1
+
+1. Manually edit the manifest and remove the aggressive scale-down behavior block, restoring the metrics thresholds to appropriate values.
+```shell
+kubectl -n otel-demo edit horizontalpodautoscaler frontend
+```
+
+### Solution 2
+
+#### Variant 1
+
+1. Disable the feature flag (loadGeneratorFloodHomepage) by manually editing the contents of the ConfigMap.
+```shell
+kubectl -n otel-demo edit configmap flagd-config
+```
+
+2. Restart all of the Deployment workloads.
+```shell
+kubectl -n otel-demo rollout restart deployment
+```

documentation/library/scenarios/statistics.md

@@ -7,16 +7,16 @@
 
 | BookInfo | OpenTelemetry Demo |
 | --- | --- |
-| 5 | 35 |
+| 5 | 36 |
 
 ## Category Distribution
 
 | FinOps | SRE |
 | --- | --- |
-| 2 | 38 |
+| 2 | 39 |
 
 ## Complexity Distribution
 
 | Low | Medium | High |
 | --- | --- | --- |
-| 17 | 22 | 1 |
+| 17 | 23 | 1 |

scenarios/sre/library/indexes/faults/31.json

@@ -0,0 +1,87 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/fault.json",
+    "alerts": {
+        "application": [
+            "KubePodNotReady"
+        ],
+        "goldenSignal": [
+            "HighRequestErrorRate",
+            "HighRequestLatency"
+        ]
+    },
+    "arguments": {
+        "jsonSchema": {
+            "properties": {
+                "kubernetesObject": {
+                    "properties": {
+                        "apiVersion": {
+                          "enum": [
+                            "autoscaling/v2"
+                          ],
+                          "type": "string"
+                        },
+                        "kind": {
+                          "enum": [
+                            "HorizontalPodAutoscaler"
+                          ],
+                          "type": "string"
+                        },
+                        "metadata": {
+                            "properties": {
+                                "name": {
+                                    "type": "string"
+                                },
+                                "namespace": {
+                                    "type": "string"
+                                }
+                            },
+                            "required": [
+                                "name",
+                                "namespace"
+                            ],
+                            "type": "object"
+                        }
+                    },
+                    "required": [
+                        "apiVersion",
+                        "kind",
+                        "metadata"
+                    ],
+                    "type": "object"
+                }
+            },
+            "required": [
+                "kubernetesObject"
+            ],
+            "type": "object"
+        }
+    },
+    "description": "This fault injects a configuration into a horizontal pod autoscaler that causes it to aggressively scale down workloads, keeping replica counts at a minimum even under high demand.",
+    "expectation": "The faulted autoscaler will rapidly scale down the workload to its minimum replica count. Under load, the reduced number of pods will be unable to handle incoming traffic, causing request errors and elevated latency.",
+    "id": "aggressive-scaledown-kubernetes-horizontal-pod-autoscaler",
+    "index": 31,
+    "name": "Aggressive Scaledown Kubernetes Horizontal Pod Autoscaler",
+    "platform": "Kubernetes",
+    "resources": [
+        "https://kubernetes.io/docs/concepts/workloads/autoscaling/",
+        "https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/",
+        "https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/",
+        "https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/horizontal-pod-autoscaler-v2/"
+    ],
+    "solutions": {
+        "templates": [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n {{ args.kubernetesObject.metadata.namespace }} edit horizontalpodautoscaler {{ args.kubernetesObject.metadata.name }}",
+                        "text": "Manually edit the manifest and remove or correct the scale-down behavior block and restore the metrics thresholds to appropriate values."
+                    }
+                ]
+            }
+        ]
+    },
+    "tags": [
+        "Deployment",
+        "Performance"
+    ]
+}

scenarios/sre/library/indexes/scenarios/115.json

@@ -0,0 +1,126 @@
+{
+    "$schema": "https://raw.githubusercontent.com/itbench-hub/ITBench/refs/heads/main/schemas/library/index/scenario.json",
+    "alerts": [
+        "KubePodNotReady",
+        "HighRequestErrorRate",
+        "HighRequestLatency"
+    ],
+    "category": "sre",
+    "complexity": "medium",
+    "description": "This scenario simulates an OpenTelemetry Demo `frontend` service degradation caused by an HPA misconfigured to aggressively scale down pods under high demand.",
+    "disruptions": [
+        {
+            "injections": [
+                {
+                    "args": {
+                        "kubernetesObject": {
+                            "apiVersion": "autoscaling/v2",
+                            "kind": "HorizontalPodAutoscaler",
+                            "metadata": {
+                                "name": "frontend",
+                                "namespace": "otel-demo"
+                            }
+                        }
+                    },
+                    "id": "aggressive-scaledown-kubernetes-horizontal-pod-autoscaler"
+                },
+                {
+                    "args": {
+                        "flag": {
+                            "name": "loadGeneratorFloodHomepage",
+                            "state": "on"
+                        },
+                        "kubernetesObject": {
+                            "apiVersion": "v1",
+                            "kind": "ConfigMap",
+                            "metadata": {
+                                "name": "flagd-config",
+                                "namespace": "otel-demo"
+                            }
+                        }
+                    },
+                    "id": "opentelemetry-demo-feature-flag"
+                }
+            ],
+            "waitFor": {
+                "postInjection": [
+                    {
+                        "args": {
+                            "kubernetesObject": {
+                                "apiVersion": "apps/v1",
+                                "kind": "Deployment",
+                                "metadata": {
+                                    "name": "flagd",
+                                    "namespace": "otel-demo"
+                                }
+                            }
+                        },
+                        "id": "restart-kubernetes-workload"
+                    },
+                    {
+                        "args": {
+                            "kubernetesObject": {
+                                "apiVersion": "apps/v1",
+                                "kind": "Deployment",
+                                "metadata": {
+                                    "name": "load-generator",
+                                    "namespace": "otel-demo"
+                                }
+                            }
+                        },
+                        "id": "restart-kubernetes-workload"
+                    }
+                ]
+            }
+        }
+    ],
+    "environment": {
+        "applications": [
+            {
+                "args": {
+                    "autoscaling": {
+                        "kubernetes": {
+                            "hpa": true
+                        }
+                    }
+                },
+                "id": "opentelemetry-demo"
+            }
+        ]
+    },
+    "id": 115,
+    "index": 115,
+    "platforms": [
+        "Kubernetes"
+    ],
+    "solutions": [
+        [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n otel-demo edit horizontalpodautoscaler frontend",
+                        "text": "Manually edit the manifest and remove the aggressive scale-down behavior block, restoring the metrics thresholds to appropriate values."
+                    }
+                ]
+            }
+        ],
+        [
+            {
+                "steps": [
+                    {
+                        "command": "kubectl -n otel-demo edit configmap flagd-config",
+                        "text": "Disable the feature flag (loadGeneratorFloodHomepage) by manually editing the contents of the ConfigMap."
+                    },
+                    {
+                        "command": "kubectl -n otel-demo rollout restart deployment",
+                        "text": "Restart all of the Deployment workloads."
+                    }
+                ]
+            }
+        ]
+    ],
+    "tags": [
+        "Deployment",
+        "Performance"
+    ]
+}

scenarios/sre/project/roles/awx/meta/argument_specs.yaml

@@ -130,6 +130,7 @@ argument_specs:
                   - 102
                   - 105
                   - 114
+                  - 115
                 required: true
                 type: int
             required: true

scenarios/sre/project/roles/awx/vars/main/scenario_fault_count.yaml

@@ -40,3 +40,4 @@ awx_scenario_fault_count:
   102: 1
   105: 1
   114: 1
+  115: 2

scenarios/sre/project/roles/faults/meta/argument_specs.yaml

@@ -33,6 +33,7 @@ argument_specs:
               - invalid-kubernetes-workload-container-command
               - kubernetes-api-server-request-surge
               - misconfigured-kubernetes-horizontal-pod-autoscaler
+              - aggressive-scaledown-kubernetes-horizontal-pod-autoscaler
               - misconfigured-kubernetes-workload-container-readiness-probe
               - modified-kubernetes-workload-container-environment-variable
               - modified-target-port-kubernetes-service

scenarios/sre/project/roles/faults/molecule/inject_aggressive_scaledown_kubernetes_horizontal_pod_autoscaler/converge.yml

@@ -0,0 +1,20 @@
+---
+- name: Run fault injection test
+  hosts:
+    - localhost
+  tasks:
+    - name: Import faults role
+      ansible.builtin.import_role:
+        name: faults
+      vars:
+        faults_cluster:
+          kubeconfig: "{{ cluster.kubeconfig }}"
+        faults_injection_tasks:
+          - id: aggressive-scaledown-kubernetes-horizontal-pod-autoscaler
+            args:
+              kubernetesObject:
+                apiVersion: autoscaling/v2
+                kind: HorizontalPodAutoscaler
+                metadata:
+                  name: test-workload-hpa
+                  namespace: aggressive-scaledown-hpa-test