feat: add 9 Konflux Web RCA incident scenarios (SPAI-414)

documentation/library/faults/README.md

@@ -4,10 +4,16 @@ A fault is a solvable issue injected into an environment to create an incident.
 
 | Name | Platform |
 | --- | --- |
+| [Breaking Configuration Change](./breaking-configuration-change.md) | Kubernetes |
+| [Cluster Availability Loss](./cluster-availability-loss.md) | Kubernetes |
 | [Cordoned Kubernetes Worker Node](./cordoned-kubernetes-worker-node.md) | Kubernetes |
+| [Corrupted Kubernetes Secret Credentials](./corrupted-kubernetes-secret-credentials.md) | Kubernetes |
+| [Crashing Kubernetes Controller Workload](./crashing-kubernetes-controller-workload.md) | Kubernetes |
 | [Crashing Kubernetes Workload Init Container](./crashing-kubernetes-workload-init-container.md) | Kubernetes |
 | [Deleted Kubernetes Service](./deleted-kubernetes-service.md) | Kubernetes |
 | [Disabled Istio Ambient Mode Kubernetes Namespace](./disabled-istio-ambient-mode-kubernetes-namespace.md) | Kubernetes |
+| [Exhausted Etcd Database Storage](./exhausted-etcd-database-storage.md) | Kubernetes |
+| [Failed Release Pipeline Service](./failed-release-pipeline-service.md) | Kubernetes |
 | [Failing Name Resolution Kubernetes Workload DNS Policy](./failing-name-resolution-kubernetes-workload-dns-policy.md) | Kubernetes |
 | [Hanging Kubernetes Workload Init Container](./hanging-kubernetes-workload-init-container.md) | Kubernetes |
 | [Ingress Port Blocking Network Policy](./ingress-port-blocking-network-policy.md) | Kubernetes |
@@ -20,13 +26,16 @@ A fault is a solvable issue injected into an environment to create an incident.
 | [Misconfigured Kubernetes Workload Container Readiness Probe](./misconfigured-kubernetes-workload-container-readiness-probe.md) | Kubernetes |
 | [Modified Kubernetes Workload Container Environment Variable](./modified-kubernetes-workload-container-environment-variable.md) | Kubernetes |
 | [Modified Target Port Kubernetes Service](./modified-target-port-kubernetes-service.md) | Kubernetes |
+| [Monitoring Probe Failure](./monitoring-probe-failure.md) | Kubernetes |
+| [Node Resource Exhaustion](./node-resource-exhaustion.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Container Image](./nonexistent-kubernetes-workload-container-image.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Node](./nonexistent-kubernetes-workload-node.md) | Kubernetes |
 | [Nonexistent Kubernetes Workload Persistent Volume Claim](./nonexistent-kubernetes-workload-persistent-volume-claim.md) | Kubernetes |
 | [OpenTelemetry Demo Feature Flag](./opentelemetry-demo-feature-flag.md) | Kubernetes |
 | [Priority Kubernetes Workload Priority Preemption](./priority-kubernetes-workload-priority-preemption.md) | Kubernetes |
 | [Scaled To Zero Kubernetes Workload](./scaled-to-zero-kubernetes-workload.md) | Kubernetes |
 | [Scheduled Chaos Mesh Experiment](./scheduled-chaos-mesh-experiment.md) | Kubernetes |
+| [Stalled Pipeline Controller](./stalled-pipeline-controller.md) | Kubernetes |
 | [Strict Mutual TLS Istio Service Mesh Enforcement](./strict-mutual-tls-istio-service-mesh-enforcement.md) | Kubernetes |
 | [Traffic Denying Istio Gateway Authorization Policy](./traffic-denying-istio-gateway-authorization-policy.md) | Kubernetes |
 | [Unassigned Kubernetes Workload Container Resource Limits](./unassigned-kubernetes-workload-container-resource-limits.md) | Kubernetes |

documentation/library/faults/breaking-configuration-change.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Breaking Configuration Change
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/35.yaml) | [Schema](../../../schemas/json/faults/breaking-configuration-change.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_breaking_configuration_change.yaml) | Kubernetes | Code, Deployment |
+
+This fault modifies a workload's environment variable to inject a breaking configuration change, simulating real-world incidents where policy updates, ArgoCD sync issues, or operator index corruption cause cascading failures across dependent services.
+
+## Expectation
+
+The faulted workload will fail to function correctly due to the misconfigured environment variable. Dependent services will experience errors as the broken configuration propagates through the service chain.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Golden Signal Alerts
+
+- HighRequestErrorRate
+
+## Resources
+
+- https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
+- https://kubernetes.io/docs/concepts/configuration/configmap/
+- https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/

documentation/library/faults/cluster-availability-loss.md

@@ -0,0 +1,31 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Cluster Availability Loss
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/34.yaml) | [Schema](../../../schemas/json/faults/cluster-availability-loss.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_cluster_availability_loss.yaml) | Kubernetes | Deployment, Performance |
+
+This fault cordons all worker nodes and then evicts the target workload pods, simulating a cluster-wide availability loss where the control plane becomes unable to schedule or maintain workloads. This reproduces real-world incidents where clusters become completely inaccessible or unresponsive.
+
+## Expectation
+
+The target workload pods will be evicted and unable to reschedule due to all nodes being cordoned. Services depending on the workload will experience errors. The cluster will appear degraded with multiple pods in Pending state.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+#### Golden Signal Alerts
+
+- HighRequestErrorRate
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/architecture/nodes/
+- https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
+- https://kubernetes.io/docs/reference/kubectl/generated/kubectl_cordon/
+- https://kubernetes.io/docs/tasks/debug/debug-cluster/

documentation/library/faults/corrupted-kubernetes-secret-credentials.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Corrupted Kubernetes Secret Credentials
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/36.yaml) | [Schema](../../../schemas/json/faults/corrupted-kubernetes-secret-credentials.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_corrupted_kubernetes_secret_credentials.yaml) | Kubernetes | Authentication, Deployment |
+
+This fault corrupts a Kubernetes Secret by replacing its data with invalid credentials, simulating real-world incidents where secret rotation failures, registry authentication issues, or Vault access problems cause service outages.
+
+## Expectation
+
+Workloads that depend on the corrupted Secret will fail to authenticate with external services (registries, databases, APIs). Pods may crash or enter error states when they attempt to use the invalid credentials.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Golden Signal Alerts
+
+- HighRequestErrorRate
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/configuration/secret/
+- https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/
+- https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/secret-v1/
+- https://kubernetes.io/docs/tasks/debug/debug-application/

documentation/library/faults/crashing-kubernetes-controller-workload.md

@@ -0,0 +1,28 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Crashing Kubernetes Controller Workload
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/31.yaml) | [Schema](../../../schemas/json/faults/crashing-kubernetes-controller-workload.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_crashing_kubernetes_controller_workload.yaml) | Kubernetes | Deployment, Performance |
+
+This fault modifies a controller or operator workload's container command to an invalid value, causing the pod to enter a CrashLoopBackOff state. This simulates real-world incidents where controller pods crash due to configuration errors, incompatible upgrades, or corrupted state, leading to service degradation across the cluster.
+
+## Expectation
+
+The faulted controller pod(s) will enter CrashLoopBackOff state. Dependent services will experience degradation as the controller is unable to reconcile resources.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodCrashLooping
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
+- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
+- https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/

documentation/library/faults/exhausted-etcd-database-storage.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Exhausted Etcd Database Storage
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/32.yaml) | [Schema](../../../schemas/json/faults/exhausted-etcd-database-storage.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_exhausted_etcd_database_storage.yaml) | Kubernetes | Performance |
+
+This fault fills the etcd database storage by creating a large number of ConfigMaps, pushing etcd usage past its quota threshold. This simulates real-world incidents where etcd storage exhaustion causes the API server to reject writes, leading to cluster-wide degradation or complete outage.
+
+## Expectation
+
+The etcd database will approach or exceed its storage quota. The Kubernetes API server will begin rejecting mutating requests. New pod scheduling and resource creation will fail.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
+- https://etcd.io/docs/v3.5/op-guide/maintenance/
+- https://kubernetes.io/docs/concepts/overview/components/#etcd
+- https://kubernetes.io/docs/tasks/debug/debug-cluster/

documentation/library/faults/failed-release-pipeline-service.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Failed Release Pipeline Service
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/38.yaml) | [Schema](../../../schemas/json/faults/failed-release-pipeline-service.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_failed_release_pipeline_service.yaml) | Kubernetes | Code, Deployment |
+
+This fault modifies a release pipeline service's container to use an invalid image tag, simulating real-world incidents where release signing failures, broken trusted task lists, or release stage blockages prevent software from being released to production.
+
+## Expectation
+
+The release pipeline service will fail to start due to the invalid image. Dependent release workflows will stall as the service becomes unavailable.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Golden Signal Alerts
+
+- HighRequestErrorRate
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
+- https://kubernetes.io/docs/concepts/containers/images/
+- https://kubernetes.io/docs/tasks/debug/debug-application/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/

documentation/library/faults/monitoring-probe-failure.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Monitoring Probe Failure
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/39.yaml) | [Schema](../../../schemas/json/faults/monitoring-probe-failure.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_monitoring_probe_failure.yaml) | Kubernetes | Deployment, Performance |
+
+This fault scales down a monitoring or probe deployment to zero replicas, simulating real-world incidents where multiple monitoring probes go down simultaneously, masking real issues and creating blind spots in observability.
+
+## Expectation
+
+The monitoring probe workload will be unavailable. Alerts that depend on the probe data will stop firing, creating a false sense of health while actual issues may go undetected.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
+- https://kubernetes.io/docs/tasks/debug/debug-application/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/deployment-v1/
+- https://kubernetes.io/docs/concepts/cluster-administration/monitoring/

documentation/library/faults/node-resource-exhaustion.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Node Resource Exhaustion
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/37.yaml) | [Schema](../../../schemas/json/faults/node-resource-exhaustion.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_node_resource_exhaustion.yaml) | Kubernetes | Deployment, Performance |
+
+This fault injects excessive resource requests into a workload, causing nodes to run out of allocatable resources. This simulates real-world incidents where node-level resource exhaustion, compute capacity shortages, or cluster autoscaler failures lead to pods stuck in Pending state and service degradation.
+
+## Expectation
+
+The faulted workload pods will consume excessive resources, causing other pods on the same node to be evicted or fail to schedule. The node may become NotReady if resource pressure is severe enough.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+- https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/
+- https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
+- https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

documentation/library/faults/stalled-pipeline-controller.md

@@ -0,0 +1,27 @@
+>[!NOTE]
+>This file is autogenerated. Please do not edit.
+
+# Stalled Pipeline Controller
+
+| [Source](../../../scenarios/sre/project/roles/documentation/files/library/faults/indexes/33.yaml) | [Schema](../../../schemas/json/faults/stalled-pipeline-controller.json) | [Implementation](../../../scenarios/sre/project/roles/faults/tasks/inject_stalled_pipeline_controller.yaml) | Kubernetes | Deployment, Performance |
+
+This fault scales down a pipeline controller deployment to zero replicas, simulating a controller failure that prevents new pipeline runs from being processed. This reproduces real-world incidents where Tekton or similar CI/CD controllers become unavailable, causing pipeline runs to stall in a pending state.
+
+## Expectation
+
+New pipeline runs will not be processed and will remain in a pending or queued state. Existing running pipelines may complete but no new work will be picked up by the controller.
+
+### Alerts
+
+The following alerts are expected to fire when this fault is injected.
+
+#### Application Alerts
+
+- KubePodNotReady
+
+## Resources
+
+- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
+- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/deployment-v1/
+- https://tekton.dev/docs/pipelines/
+- https://kubernetes.io/docs/tasks/debug/debug-application/

documentation/library/scenarios/README.md

@@ -45,23 +45,32 @@
 | [102](./sre/102.md) | sre | medium |
 | [105](./sre/105.md) | sre | medium |
 | [114](./sre/114.md) | sre | low |
+| [115](./sre/115.md) | sre | medium |
+| [116](./sre/116.md) | sre | high |
+| [117](./sre/117.md) | sre | medium |
+| [118](./sre/118.md) | sre | high |
+| [119](./sre/119.md) | sre | medium |
+| [120](./sre/120.md) | sre | medium |
+| [121](./sre/121.md) | sre | medium |
+| [122](./sre/122.md) | sre | medium |
+| [123](./sre/123.md) | sre | low |
 
 ## Statistics
 
 ### Application Distribution
 
 | BookInfo | OpenTelemetry Demo |
 | --- | --- |
-| 5 | 35 |
+| 5 | 44 |
 
 ### Category Distribution
 
 | FinOps | SRE |
 | --- | --- |
-| 2 | 38 |
+| 2 | 47 |
 
 ### Complexity Distribution
 
 | Low | Medium | High |
 | --- | --- | --- |
-| 17 | 22 | 1 |
+| 18 | 28 | 3 |

documentation/library/scenarios/sre/115.md

@@ -0,0 +1,49 @@
+> [!NOTE]
+> This file is autogenerated. Please do not edit.
+
+# Scenario 115
+
+| [Source](../../../../scenarios/sre/project/roles/documentation/files/library/scenarios/indexes/115.yaml) | Medium Complexity | Kubernetes | Deployment, Performance |
+
+## Description
+
+This scenario simulates OpenTelemetry Demo's `recommendation` service controller crashing due to a corrupted container command, causing the pod to enter CrashLoopBackOff and degrading upstream services that depend on it.
+
+## Applications
+
+- [OpenTelemetry Demo](../../applications/opentelemetry-demo.md)
+
+## Faults
+
+- [Crashing Kubernetes Controller Workload](../../../../scenarios/sre/docs/faults.md#Crashing-Kubernetes-Controller-Workload)
+
+## Solution Guides
+
+Below is a noncomprehensive list of step by step guides which can lead to remediating the one or more problems in the scenario.
+
+A **solution** represents the process of fixing an issue created by a fault. To remediate the issues in the scenarios, follow the solutions in order of appearance.
+
+A solution can have multiple **variants**. A variant is a different process of deriving the solution. If multiple variants are presented, pick only one to solve the problem.
+
+### Solution 1
+
+#### Variant 1
+
+1. Revert the last change done to the controller manifest.
+```shell
+kubectl -n otel-demo rollout undo deployment/recommendation
+```
+
+#### Variant 2
+
+1. Manually edit the manifest and replace the container command with the correct value.
+```shell
+kubectl -n otel-demo edit deployment recommendation
+```
+
+#### Variant 3
+
+1. Check the previous container logs to understand the crash reason.
+```shell
+kubectl -n otel-demo logs -l app=recommendation -c recommendation --previous
+```