Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/deployment/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Keep is highly configurable through environment variables. This allows you to cu
| **CONSUMER** | Enables or disables the consumer | No | "true" | "true" or "false" |
| **KEEP_VERSION** | Specifies the Keep version | No | "unknown" | Valid version string |
| **KEEP_API_URL** | Specifies the Keep API URL | No | Constructed from HOST and PORT | Valid URL |
| **KEEP_STORE_RAW_ALERTS** | Enables storing of raw alerts | No | "false" | "true" or "false" |
| **KEEP_STORE_RAW_ALERTS** | When `true`: stores full provider webhooks in `alertraw` (one row per webhook) and enables **correlation rule incident enrichments** (`{{ raw.* }}` templates). Raw data for enrichments is passed in-memory on `AlertDto.keep_raw_payload` during ingestion. UI incident-enrichments section is shown only when this flag is on. | No | "false" | "true" or "false" |
| **TENANT_CONFIGURATION_RELOAD_TIME** | Time in minutes to reload tenant configurations | No | 5 | Positive integer |
| **KEEP_LIVE_DEMO_MODE** | Keep will simulate incoming alerts and other activity | No | "false" | "true" or "false" |

Expand Down
118 changes: 118 additions & 0 deletions docs/overview/correlation-rules.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,124 @@ When an incident contains multiple alerts:

Service Issue on host1,host2

## Incident Enrichments

Correlation rules can attach **key/value metadata** to incidents when they are **first created**. Configure this in the Correlation rule editor under **Incident enrichments**.

Each row has:
- **Key** — field name on the incident (e.g. `threshold`, `monitor`, `team`)
- **Value** — static text or a template using `{{ ... }}` syntax

Enrichments appear on the incident detail page and are stored in the incident enrichment record.

### Requirements

Incident enrichments from correlation rules require:

```bash
KEEP_STORE_RAW_ALERTS=true
```

Set this in your `.env` or deployment environment and restart the Keep API.

When `KEEP_STORE_RAW_ALERTS` is **disabled**, correlation rules still create incidents, but **no incident enrichments** are applied.

### When enrichments run

| Scenario | Enrichments |
|----------|-------------|
| **New incident** created by the rule | Templates are rendered and saved |
| **Existing incident** reused (same rule fingerprint) | Not re-run — first alert's enrichments are kept |

Enrichments are resolved from the **triggering alert** at incident creation time, using:

1. **Formatted alert** (`alert.*` / top-level fields on `AlertDto`)
2. **In-memory raw webhook** (`keep_raw_payload` on `AlertDto` during ingestion — not saved on the `alert` row)

When `KEEP_STORE_RAW_ALERTS=true`, the full provider webhook is also written once per webhook to the `alertraw` table (for debugging/archive). Incident enrichments read raw fields from **in-memory** `keep_raw_payload`, not from a DB lookup.

### Template syntax

Use `{{ variable }}` placeholders. Whitespace inside braces is trimmed automatically.

#### Alert fields (formatted `AlertDto`)

Reference normalized alert fields stored in `alert.event`:

```
{{ alert.labels.monitor }}
{{ alert.name }}
{{ alert.severity }}
{{ labels.monitor }} # same as alert.labels.monitor when on AlertDto
```

#### Raw webhook fields (`KEEP_STORE_RAW_ALERTS=true`)

Reference the provider's original webhook payload (full webhook stored in `alertraw`, passed in-memory during ingestion):

```
{{ raw.alerts[0].condition }}
{{ raw.alerts[0].labels.monitor }}
{{ raw.alerts[0].data[0].model.conditions[0].evaluator.params[0] }}
```

Dot index works too: `{{ raw.alerts.0.data.0.model... }}`

- Prefix with `raw.` to read **only** from the raw webhook
- Without `raw.`, Keep tries the formatted alert first, then falls back to raw
- Bracket notation `alerts[0]` and dot notation `alerts.0` both work

For Grafana (and similar providers), the stored raw payload is the **full webhook**. Fields like `data`, `condition`, and `labels` live under `alerts[index]` — **not** at the top level.

| Wrong (returns `N/A`) | Correct |
|-----------------------|---------|
| `{{ raw.data[0].model... }}` | `{{ raw.alerts[0].data[0].model... }}` |
| `{{ raw.condition }}` | `{{ raw.alerts[0].condition }}` |
| `{{ raw.labels.monitor }}` | `{{ raw.alerts[0].labels.monitor }}` or `{{ alert.labels.monitor }}` |

For multi-alert webhooks, use the index of the alert you care about (`alerts[0]`, `alerts[1]`, …). Prefer `{{ alert.* }}` when Grafana already maps the field onto the normalized alert.

### Grafana example

**Webhook payload (simplified):**

```json
{
"title": "NetworkLatencyIsHigh",
"alerts": [{
"condition": "C",
"labels": { "monitor": "router1" },
"data": [{
"model": {
"conditions": [{
"evaluator": { "params": [100], "type": "gt" }
}]
}
}]
}]
}
```

**Correlation rule — Incident enrichments:**

| Key | Value |
|-----|-------|
| `monitor` | `{{ alert.labels.monitor }}` |
| `condition` | `{{ raw.alerts[0].condition }}` |
| `threshold` | `{{ raw.alerts[0].data[0].model.conditions[0].evaluator.params[0] }}` |

**Result on incident:** `monitor=router1`, `condition=C`, `threshold=100`

### Troubleshooting

| Symptom | Likely cause |
|---------|----------------|
| Enrichment value is `N/A` | Wrong template path — for Grafana use `raw.alerts[0].…`, not `raw.data[0].…` |
| No enrichments on incident | `KEEP_STORE_RAW_ALERTS` is not `true`, or API was not restarted after changing `.env` |
| `raw.*` returns `N/A` | Flag off, API not restarted, or no `keep_raw_payload` on alert — send a **new** alert after enabling |
| Incident enrichments UI hidden | `KEEP_STORE_RAW_ALERTS` is not `true` on the API (UI reads `store_raw_alerts_enabled`) |
| Wrong value on reused incident | Enrichments only run at **create**; delete/recreate incident or wait for expiry |

## Examples
- **Metric-based alerts**: Construct a rule to pinpoint alerts associated with specific metrics, such as high CPU usage on servers. This can be achieved by grouping alerts that share a common attribute, like a 'CPU usage' tag, ensuring you quickly identify and address performance issues.
- **Feature-related alerts**: Establish rules to create incident by specific features or services. For instance, you can start incident based on a 'service' or 'URL' tag. This approach is particularly useful for tracking and managing alerts related to distinct functionalities or components within your application.
Expand Down
72 changes: 70 additions & 2 deletions keep-ui/app/(keep)/rules/CorrelationSidebar/CorrelationForm.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,18 @@ import {
Text,
TextInput,
} from "@tremor/react";
import { Controller, get, useFormContext } from "react-hook-form";
import {
Controller,
get,
useFieldArray,
useFormContext,
} from "react-hook-form";
import { AlertDto } from "@/entities/alerts/model";
import { QuestionMarkCircleIcon } from "@heroicons/react/24/outline";
import {
PlusIcon,
QuestionMarkCircleIcon,
TrashIcon,
} from "@heroicons/react/24/outline";
import React from "react";
import { CorrelationFormType } from "./types";
import { useTenantConfiguration } from "@/utils/hooks/useTenantConfiguration";
Expand All @@ -33,6 +42,14 @@ export const CorrelationForm = ({
watch,
formState: { errors, isSubmitted },
} = useFormContext<CorrelationFormType>();
const {
fields: incidentEnrichmentFields,
append: appendIncidentEnrichment,
remove: removeIncidentEnrichment,
} = useFieldArray({
control,
name: "incidentEnrichments",
});

const { data: tenantConfiguration } = useTenantConfiguration();
const { data: users = [] } = useUsers();
Expand Down Expand Up @@ -352,6 +369,57 @@ export const CorrelationForm = ({
<Text>Created incidents require manual approve</Text>
</label>
</div>
{tenantConfiguration?.["store_raw_alerts_enabled"] && (
<fieldset className="space-y-2">
<div className="flex items-center justify-between">
<label className="text-tremor-default font-medium text-tremor-content-strong flex items-center">
Incident enrichments
<Button
className="cursor-default ml-2"
type="button"
tooltip="Add metadata to incidents when this rule creates a new incident. Templates: {{ alert.labels.monitor }}, {{ raw.alerts[0].data[0].model.conditions[0].evaluator.params[0] }}. Grafana: use raw.alerts[0].… not raw.data[0].…. Not re-applied when reusing an existing incident."
icon={QuestionMarkCircleIcon}
size="xs"
variant="light"
color="slate"
/>
</label>
<Button
type="button"
size="xs"
variant="light"
color="orange"
icon={PlusIcon}
onClick={() => appendIncidentEnrichment({ key: "", value: "" })}
>
Add
</Button>
</div>
{incidentEnrichmentFields.map((field, index) => (
<div key={field.id} className="grid grid-cols-[1fr_1fr_auto] gap-2">
<TextInput
type="text"
placeholder="owner"
{...register(`incidentEnrichments.${index}.key` as const)}
/>
<TextInput
type="text"
placeholder="{{ alert.labels.owner }}"
{...register(`incidentEnrichments.${index}.value` as const)}
/>
<Button
type="button"
size="xs"
variant="light"
color="red"
icon={TrashIcon}
tooltip="Remove enrichment"
onClick={() => removeIncidentEnrichment(index)}
/>
</div>
))}
</fieldset>
)}
{tenantConfiguration?.["multi_level_enabled"] && (
<div className="flex items-center space-x-2">
<Controller
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import { useRouter, useSearchParams } from "next/navigation";
import { AlertsFoundBadge } from "./AlertsFoundBadge";
import { useApi } from "@/shared/lib/hooks/useApi";
import { useConfig } from "@/utils/hooks/useConfig";
import { useTenantConfiguration } from "@/utils/hooks/useTenantConfiguration";
import { showErrorToast } from "@/shared/ui";
import { CorrelationFormType } from "./types";
import { TIMEFRAME_UNITS_TO_SECONDS } from "./timeframe-constants";
Expand All @@ -29,6 +30,9 @@ export const CorrelationSidebarBody = ({
}: CorrelationSidebarBodyProps) => {
const api = useApi();
const { data: config } = useConfig();
const { data: tenantConfiguration } = useTenantConfiguration();
const isStoreRawAlertsEnabled =
!!tenantConfiguration?.["store_raw_alerts_enabled"];

const methods = useForm<CorrelationFormType>({
defaultValues: defaultValue,
Expand Down Expand Up @@ -79,8 +83,22 @@ export const CorrelationSidebarBody = ({
multiLevelPropertyName,
threshold,
assignee,
incidentEnrichments,
} = correlationFormData;

const incidentEnrichmentsObject = isStoreRawAlertsEnabled
? incidentEnrichments.reduce(
(acc, enrichment) => {
const key = enrichment.key.trim();
if (key) {
acc[key] = enrichment.value;
}
return acc;
},
{} as Record<string, string>
)
: {};

const body = {
sqlQuery: formatQuery(query, "parameterized_named"),
groupDescription: description,
Expand All @@ -98,6 +116,7 @@ export const CorrelationSidebarBody = ({
multiLevelPropertyName,
threshold,
assignee,
incidentEnrichments: incidentEnrichmentsObject,
};

try {
Expand Down
8 changes: 8 additions & 0 deletions keep-ui/app/(keep)/rules/CorrelationSidebar/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export const DEFAULT_CORRELATION_FORM_VALUES: CorrelationFormType = {
multiLevelPropertyName: "",
threshold: 1,
assignee: undefined,
incidentEnrichments: [],
query: {
combinator: "or",
rules: [
Expand Down Expand Up @@ -63,6 +64,12 @@ export const CorrelationSidebar = ({
);

const timeunit = selectedRule.timeunit ?? "seconds";
const incidentEnrichments = Object.entries(
selectedRule.incident_enrichments || {}
).map(([key, value]) => ({
key,
value: String(value),
}));

return {
name: selectedRule.name,
Expand All @@ -83,6 +90,7 @@ export const CorrelationSidebar = ({
multiLevelPropertyName: selectedRule.multi_level_property_name || "",
threshold: selectedRule.threshold || 1,
assignee: selectedRule.assignee,
incidentEnrichments,
};
}

Expand Down
1 change: 1 addition & 0 deletions keep-ui/app/(keep)/rules/CorrelationSidebar/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ export type CorrelationFormType = {
multiLevelPropertyName?: string;
threshold: number;
assignee?: string;
incidentEnrichments: { key: string; value: string }[];
};
1 change: 1 addition & 0 deletions keep-ui/utils/hooks/useRules.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export type Rule = {
multi_level_property_name: string | null;
threshold: number;
assignee: string | undefined;
incident_enrichments: Record<string, string>;
};

export const useRules = (options?: SWRConfiguration) => {
Expand Down
1 change: 1 addition & 0 deletions keep/api/consts.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
OPENAI_MODEL_NAME = os.environ.get("OPENAI_MODEL_NAME", "gpt-4o-2024-08-06")

KEEP_CORRELATION_ENABLED = os.environ.get("KEEP_CORRELATION_ENABLED", "true") == "true"
KEEP_STORE_RAW_ALERTS = os.environ.get("KEEP_STORE_RAW_ALERTS", "false") == "true"

FINGERPRINT_PAYLOAD_LIMIT = 100

Expand Down
Loading
Loading