diff --git a/src/content/docs/docs/alerts/alert-groups.mdx b/src/content/docs/docs/alerts/alert-groups.mdx index c2e1efad..7f985c0e 100644 --- a/src/content/docs/docs/alerts/alert-groups.mdx +++ b/src/content/docs/docs/alerts/alert-groups.mdx @@ -28,11 +28,11 @@ Columns: **Group** (title), **State** (Open / Resolved), **Instances** (the firi Expand any row to preview that group's alerts inline: alert name, state, severity, received time, and labels, without leaving the list. The group title links straight to the full detail page. -Filters across the top: +Filter the list by: - **State:** All / Open / Resolved toggle. Defaults to **Open**. - **Routing rule:** dropdown filtered to rules in the workspace. -- **Label matchers:** chip input where you type `key=value`. Press Enter to add a chip. +- **Label matchers:** filter by `key=value`. Severity isn't a filter on the list. It lives on the per-instance alerts inside a group, not on the group itself. @@ -40,16 +40,16 @@ Severity isn't a filter on the list. It lives on the per-instance alerts inside A group carries: -- **Title:** set by the first alert that opened the group, and **immutable** afterward, even if a higher-severity alert joins later. A lock icon next to the title marks it. +- **Title:** set by the first alert that opened the group, and **immutable** afterward, even if a higher-severity alert joins later. - **State:** `Open` while at least one underlying instance is firing; `Resolved` once everything quiets. A firing group whose instances are **all silenced** stays `Open` but carries a **Muted** badge. Muting gates notifications; it doesn't resolve the group. - **Labels:** the labels that define the group, read from the alerts that joined it (the group-by keys, in Static mode). -- **Alerts and instances:** how many distinct alerts and firing instances the group holds so far, shown as *"N alerts · N firing instances"*. +- **Alerts and instances:** how many distinct alerts and firing instances the group holds so far. - **Routing rule:** the rule that opened the group. - **Group ID:** the durable identifier you can share with teammates or paste into the assistant. - **Correlation reason:** for an Auto-correlated group, why these alerts were grouped. See [Why these were grouped](#why-these-were-grouped). - **Investigation:** a root-cause investigation, when one is attached. Start it manually with **Run RCA**, or automatically with [Auto-RCA](../auto-rca/); see [Run RCA and View RCA](#run-rca-and-view-rca). -Severity isn't a single group-level field. Each underlying instance carries its own severity, which you read on the **Firing instances** panel and the **Alerts** tab. +Severity isn't a single group-level field. Each underlying instance carries its own severity. ## Group detail @@ -59,16 +59,16 @@ Click any row to open the group detail page. ### Header -- **Title** with a lock icon: it's set by the first alert and can't be changed. +- **Title:** set by the first alert and can't be changed. - **Run RCA / View RCA:** starts or opens a root-cause investigation for the group. See [Run RCA and View RCA](#run-rca-and-view-rca). - **Ask KloudMate Assistant:** opens the assistant chat panel with the group's labels, state, alert and instance counts, and routing rule pre-loaded into the prompt, so you can start investigating without retyping context. -- **Silence this group:** opens the silence creator pre-filled with the group's labels as matchers and bound to the group via `auto_expire_group_id`. +- **Silence this group:** opens the silence creator pre-filled with the group's labels as matchers, bound to the group so the silence expires when it resolves. ### Firing instances -The **Firing instances** panel is the largest section on the page. It's a table of the unique alert instances that joined this group, deduplicated by the per-instance labels the grouping engine uses. +The **Firing instances** panel is a table of the unique alert instances that joined this group, deduplicated by the per-instance labels the grouping engine uses. -- **Common labels bar** at the top: labels shared by every instance, so the per-row labels column only shows what varies. +- **Common labels:** labels shared by every instance, so the per-row labels column only shows what varies. - **Columns:** - **State:** the instance's current state: **Firing**, **No Data**, **Error**, or **Resolved**. Silenced instances are flagged as muted, so you can see at a glance which ones are firing but suppressed. - **Alert:** the alert rule name. Links to the rule when the alert is KloudMate-native (carries an `alarm_id` label). @@ -80,12 +80,12 @@ Use this panel to see what's firing inside the group at a glance. For example, s ### Run RCA and View RCA -The group header carries a single root-cause button, and its label reflects whether an investigation is attached yet: +The root-cause button's label depends on whether an investigation is attached: - **Run RCA:** shown when no investigation is attached. It starts a root-cause investigation for the group on demand. -- **View RCA:** shown once an investigation is attached. It opens a side drawer with the root-cause summary and a **View full investigation** link. +- **View RCA:** shown once an investigation is attached. It opens the root-cause summary, which links to the full investigation. -This is the manual trigger. To investigate every group a rule opens automatically instead, enable [Auto-RCA](../auto-rca/) on the routing rule; an Auto-RCA result surfaces through the same **View RCA** drawer. +This is the manual trigger. To investigate every group a rule opens automatically instead, enable [Auto-RCA](../auto-rca/) on the routing rule; the result appears under the same **View RCA**. ![View RCA drawer with the root-cause summary and a View full investigation link](./images/alert-group-rca-drawer.png) @@ -93,24 +93,24 @@ This is the manual trigger. To investigate every group a rule opens automaticall - **Alerts:** every underlying alert as a row. Columns: Alert (linked to the rule when available), State, Severity, Received at, Labels. Use this when you need the raw event-by-event stream rather than the instance-level rollup. - **Blast radius:** what the group's firing resources connect to. See [Blast radius](#blast-radius). -- **Audit:** a vertical timeline of everything that happened to the group: opened, alerts appended, silence applied, notification dispatched, resolved. Auto-correlated groups also log correlation events here: **"Group opened by AI"** and **"Alert correlated into group"**. +- **Audit:** a timeline of everything that happened to the group: opened, alerts appended, silence applied, notification dispatched, resolved. For Auto-correlated groups, it also logs when the engine opened the group and when it correlated each alert. - **Notifications:** per-channel dispatch outcomes (ok / failed / suppressed-by-silence), with deep links to where the notification landed: Slack thread URL, Jira ticket, KloudMate Incidents incident, and so on. ### Blast radius The **Blast radius** tab resolves the group's firing alerts to the resources they name, then graphs what each one depends on, so you can see how far the incident reaches. -Each affected resource sits at the center of its own graph, with neighbors branching off along labeled edges: **runs on**, **talks to**, **calls**. Neighbors of the same kind collapse into one node (*116 Pods · runs on*) that you click to list them. +Each affected resource anchors its own graph, its neighbors connected by labeled edges: **runs on**, **talks to**, **calls**. Neighbors of the same kind collapse into one node (for example, *116 Pods · runs on*) that you click to list them. -The same graph appears on a single alert's **Blast radius** tab and on a resource's **Dependencies** tab in Cloud Inventory. An empty graph means KloudMate hasn't mapped a connection yet, not that the lookup failed. When nothing is firing or the alerts don't match a known resource, the tab says so instead of drawing one. +The same graph appears on a single alert's **Blast radius** tab and on a resource's **Dependencies** tab in Cloud Inventory. An empty graph means KloudMate hasn't mapped a connection yet, not that the lookup failed. When nothing is firing or the alerts don't match a known resource, there's no graph to draw. ![The Blast radius tab: a row of affected resource chips above a graph of each resource and what it runs on](./images/alert-group-blast-radius.png) ### Why these were grouped -A **Why these were grouped** section sits at the bottom of the detail page, with the reason in plain English, the shared labels, and supporting detail. +**Why these were grouped** gives the reason in plain English, the shared labels, and supporting detail. -The chip and section only appear when the engine actually correlated more than one alert. A lone alert the engine evaluated and left on its own shows no chip, which is expected (see [Cold start](#cold-start)). +This reason appears only when the engine correlated more than one alert. A lone alert the engine left on its own has none, which is expected (see [Cold start](#cold-start)). ### Cold start @@ -118,7 +118,7 @@ A new workspace has little co-occurrence history for the engine to learn from, s ### Feedback -Under the reason, a **Was this grouping correct?** control takes feedback with **Correct** and **Wrong**. +You can rate the grouping **Correct** or **Wrong**. Choosing **Wrong** lets you point at the specific alert that doesn't belong, or mark the whole group as wrong. That feedback trains the correlation engine: a **Wrong** verdict makes it less likely to pair those alerts again, and **Correct** reinforces the grouping. diff --git a/src/content/docs/docs/alerts/annotations-and-severity.mdx b/src/content/docs/docs/alerts/annotations-and-severity.mdx index b72e5eab..08ee3614 100644 --- a/src/content/docs/docs/alerts/annotations-and-severity.mdx +++ b/src/content/docs/docs/alerts/annotations-and-severity.mdx @@ -14,22 +14,22 @@ Annotation values support **Liquid templating**, so you can pull live data from ## Where to find annotations -Annotations live in the **Responder context** section of the alert rule editor (inside the **Add alarm details** step). The pre-defined annotation keys render as labeled form fields: +Annotations live in the **Responder context** section of the alert rule editor. The pre-defined annotation keys are: - **Severity:** free-form severity string, for example `sev1`, `critical`, or `p1`. Plain text or a Liquid template. -- **Summary:** multiline text, shown as the alert headline in notifications. Supports templates. +- **Summary:** the alert headline in notifications. Supports templates. - **Dashboard:** pick a workspace dashboard to link from the notification. - **Panel:** when a dashboard is picked, narrows the link to a specific panel. - **Playbook URL:** link to the runbook responders should follow. Supports templates. -Any other key is a **custom annotation**. Custom annotations live under a collapsible **Custom annotations** accordion below the fixed fields. Click **Add Annotation** to create a new key-value row. +Any other key is a **custom annotation**. Click **Add Annotation** to add one. ## Add a custom annotation -1. **Open the rule editor.** Scroll to **Responder context** and expand the **Custom annotations** accordion. +1. **Open the rule editor.** Scroll to **Responder context** and expand **Custom annotations**. -2. **Click Add Annotation.** A new key-value row appears. +2. **Click Add Annotation.** 3. **Provide a key.** Use any string, for example `service_owner`, `region`, or `runbook_section`. @@ -71,7 +71,7 @@ You can interpolate live alert data into any annotation value with the `{{ }}` s - `{{ state.values.A }}`, `{{ state.values.B }}`: raw outputs of each query or expression node (by node ID). - `{{ state.current_state }}`: `Alerting`, `Normal`, and so on. - `{{ state.resolved }}`: boolean, useful in `{% if state.resolved %}…{% endif %}` branches. -- `{{ rule.name }}`, `{{ rule.id }}`: the alert rule's name and id. Prefer these over `labels.alarm_id` / `labels.alarm_name`; those internal keys are stripped before render. +- `{{ rule.name }}`, `{{ rule.id }}`: the alert rule's name and id. Prefer these over `labels.alarm_id` / `labels.alarm_name`, which aren't available at render time. ### Example: contextual summary @@ -84,13 +84,13 @@ When the alert fires for `web-server-01` at 95%, the notification reads: *Host w ## Testing a template -Every templated value field has a **Test template** icon to its right. Click it to open a side drawer: +Click **Test template** on any templated value field to preview how it renders: -- The drawer pre-loads the Monaco editor with your current template. -- A **Sample context** panel lets you tweak the labels / state / rule JSON the template renders against. -- Click **Test** to see the rendered output below. Parse errors surface inline so you can fix them before saving. +- It opens with your current template loaded. +- Edit the sample labels, state, and rule JSON the template renders against. +- Click **Test** to preview the rendered output. Any parse errors are shown so you can fix them before saving. -The drawer is opt-in; KloudMate doesn't validate templates on save. If a template can't render at notification time, KloudMate falls back to the raw template string rather than dropping the notification. +Testing is optional; KloudMate doesn't validate templates on save. If a template can't render at notification time, KloudMate falls back to the raw template string rather than dropping the notification. ### Failure mode diff --git a/src/content/docs/docs/alerts/auto-rca.mdx b/src/content/docs/docs/alerts/auto-rca.mdx index b45621d7..552abb9e 100644 --- a/src/content/docs/docs/alerts/auto-rca.mdx +++ b/src/content/docs/docs/alerts/auto-rca.mdx @@ -7,15 +7,15 @@ sidebar: Auto-RCA hands an [Alert Group](../alert-groups/) to KloudMate's AI investigator the moment it opens, so by the time you open the group there's already a root-cause summary waiting. You enable it per [Routing Rule](../routing-rules/); every group the rule produces gets investigated. -Auto-RCA is the automatic trigger: it runs on every group a rule opens, without anyone asking. You can also start an investigation manually on any group with the **Run RCA** button. Once an investigation attaches, the button becomes **View RCA** and opens it in a side drawer. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). +Auto-RCA is the automatic trigger: it runs on every group a rule opens, without anyone asking. You can also start an investigation manually on any group with **Run RCA**, and open a finished one with **View RCA**. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). ## How it works -When a routing rule with Auto-RCA enabled opens a group, KloudMate schedules an investigation after the configurable **Auto-RCA delay** (default: 5 minutes). The delay gives related alerts time to fold into the group so the investigation sees the full picture, not only the first alert. +When a routing rule with Auto-RCA enabled opens a group, KloudMate schedules an investigation after the configurable **Auto-RCA delay** (default: 5 minutes). The delay gives related alerts time to fold into the group, so the investigation covers them all, not only the first alert. When the investigation completes, its summary attaches to the group: -- On the group detail page, the header's RCA button becomes **View RCA**. Opening it shows the root-cause summary and a **View full investigation** link. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). +- You can open the summary from the group with **View RCA**, which also links to the full investigation. See [Run RCA and View RCA](../alert-groups/#run-rca-and-view-rca). - The notifications dispatched from the group include the investigation summary, where the channel format supports it: a Slack thread reply, a KloudMate Incidents comment, and so on. :::note[One investigation per group] @@ -40,6 +40,6 @@ Auto-RCA works best on rules that group related alerts: a spike after a deploy, ## Related -- [Routing Rules](../routing-rules/): where the Auto-RCA toggle lives. -- [Alert Groups](../alert-groups/): where the investigation surfaces. +- [Routing Rules](../routing-rules/): enable Auto-RCA per rule. +- [Alert Groups](../alert-groups/): how an investigation attaches to a group. - [KloudMate Assistant](../../kloudmate-assistant/): the broader investigator surface. diff --git a/src/content/docs/docs/alerts/create-alerts.mdx b/src/content/docs/docs/alerts/create-alerts.mdx index 32cedab7..bcc9aa5e 100644 --- a/src/content/docs/docs/alerts/create-alerts.mdx +++ b/src/content/docs/docs/alerts/create-alerts.mdx @@ -13,7 +13,7 @@ Open the **Alerts** section from the left navigation. ![Alerts list](./images/setting-up-kloudmate-alarms-1.png) -The **Alerts** screen lists your alert rules with their current state, name, and description. A summary at the top shows how many rules you have, and how many are currently **Firing** or **Pending**. You can group rules into [Folders](../folders/), which appear as collapsible header rows. +The **Alerts** screen lists your alert rules with their current state, name, and description. You can group rules into [Folders](../folders/). From the **more options (⋯)** icon on any rule, you can: @@ -26,13 +26,13 @@ From the **more options (⋯)** icon on any rule, you can: ![Alert row actions](./images/setting-up-kloudmate-alarms-1.jpeg) -**Pause Notifications** creates a temporary [silence](../silences/) scoped to this alert: the rule keeps evaluating, but KloudMate suppresses its notifications until the silence expires. While it's paused, the rule shows a **Silenced** badge on the alerts list and its detail page; view or end the pause from **Alerts → Silences**. Add matchers to limit it to specific instances. +**Pause Notifications** creates a temporary [silence](../silences/) scoped to this alert: the rule keeps evaluating, but KloudMate suppresses its notifications until the silence expires. While it's paused, the rule is marked **Silenced**; view or end the pause from **Alerts → Silences**. Add matchers to limit it to specific instances. For the key concepts, see the [Alerts Overview](../). ## Creating a new alert -Click the **Create Alert** button at the top right of the **Alerts** screen. A dialog appears with three ways to create an alert: +Click **Create Alert**. A dialog appears with three ways to create an alert: ![Create Alert dialog](./images/create-alarm-dialog.png) @@ -72,9 +72,9 @@ To build a fully custom alert, select **From Scratch** in the **Create Alert** d This opens the alert creation form, where you choose a data source, configure the metric or query to monitor, and define the alert condition on a single page. -Create more queries and expressions with the **Add Query** and **Add Expression** buttons. KloudMate assigns each one a letter, such as **A**, **B**, or **C**, and you can duplicate any block with the copy icon at its top right. +Create more queries and expressions with the **Add Query** and **Add Expression** buttons. KloudMate assigns each one a letter, such as **A**, **B**, or **C**, and you can duplicate any block with its copy icon. -To reach advanced options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode** at the top of the form. +To reach advanced options such as **Math expressions**, **Reduce**, and **Condition expressions**, click **Advanced mode**. ![Advanced mode options](./images/advanced-mode-options.png) @@ -152,7 +152,7 @@ To avoid the **NoData** issue when a single alert uses multiple queries, use the ### 2. Configure evaluation settings -This step opens with a **Folder** dropdown. Select an existing folder, or type a new name to create one inline, to organize the rule and inherit shared defaults. The folder's `interval_seconds`, `no_data_state`, and `eval_error_state` flow into the rule as defaults you can override per field. See [Folders](../folders/). +Pick a **Folder** to organize the rule and inherit shared defaults: select an existing folder, or type a new name to create one inline. The folder's **Evaluate every**, **No data state**, and **Eval error state** flow into the rule as defaults you can override per field. See [Folders](../folders/). ![Evaluation settings](./images/setting-up-kloudmate-alarms-4.png) @@ -175,7 +175,7 @@ For the full lifecycle, and how **Pending duration**, **Recovery period**, and t Click **Preview alerts** to run the query immediately and check the result. :::tip -If the rule belongs to a folder with shared defaults, **Evaluate every**, **Alert state if No data**, and **Alert state if Error** show placeholder values inherited from the folder. Leave them blank to keep inheriting, or type a value to override. A small **Reset to folder default** link appears under any field you've overridden. +If the rule belongs to a folder with shared defaults, **Evaluate every**, **Alert state if No data**, and **Alert state if Error** show placeholder values inherited from the folder. Leave them blank to keep inheriting, or type a value to override. **Reset to folder default** returns an overridden field to inheriting. ::: ### 3. Add alarm details @@ -187,7 +187,7 @@ Set the alert's name and description, then fill in the responder context: - **Alert name:** a name for the alert. - **Description:** a short description of the alert's purpose. -**Responder context** is a labeled section that holds the annotations responders see when the notification lands. The hint above it reads, for example, *"Help on-call responders understand the alarm and act quickly."* +**Responder context** holds the annotations responders see when the notification lands. - **Severity:** free-form severity (for example, `sev1`, `critical`, or `p1`). Supports Liquid templates, so severity can depend on the firing value. - **Summary:** multiline message included in notifications. Supports templates. @@ -195,7 +195,7 @@ Set the alert's name and description, then fill in the responder context: - **Panel:** when you pick a dashboard, narrows the link to a specific panel. - **Playbook URL:** optional runbook URL. -**Custom annotations** is a collapsed accordion at the bottom. Open it to add your own key-value pairs (for example, `service_owner` or `region`). Values support Liquid templates. See [Annotations & Severity](../annotations-and-severity/). +Open **Custom annotations** to add your own key-value pairs (for example, `service_owner` or `region`). Values support Liquid templates. See [Annotations & Severity](../annotations-and-severity/). ### 4. Notifications diff --git a/src/content/docs/docs/alerts/expressions.mdx b/src/content/docs/docs/alerts/expressions.mdx index 62912109..8ed9a827 100644 --- a/src/content/docs/docs/alerts/expressions.mdx +++ b/src/content/docs/docs/alerts/expressions.mdx @@ -11,7 +11,7 @@ Each query or expression has a unique letter (A, B, C, and so on). In an express ## Expression operations -Alert expressions support three operations: +Alert expressions support these operations: - Math expression - Reduce @@ -124,7 +124,7 @@ A condition expression takes the time series or numbers returned by queries or e ![image](./images/expressions-7.png) -The reduction function is applied to the query or expression to produce a single value, which is then compared against the threshold using the condition you chose. When the input is a collection of time series or numbers, the reduction function is applied to each element individually, each reduced output is evaluated against the condition, and the outputs are combined with AND. +KloudMate reduces the query or expression to a single value, then compares it against the threshold using the condition you chose. When the input is a collection of time series or numbers, it reduces each element individually, evaluates each reduced output against the condition, and combines the results with AND. Add more conditions with the **Add condition** button, and choose how they combine with the **AND** or **OR** operators. @@ -134,7 +134,7 @@ Nodes represent the individual queries or expressions used to build alerts and d ![image](./images/expressions-8.png) -Nodes serve as reference points, so you can identify and evaluate specific queries or expressions during analysis. +You reference a node by its letter in other queries and expressions. ![image](./images/expressions-9.png) diff --git a/src/content/docs/docs/alerts/folders.mdx b/src/content/docs/docs/alerts/folders.mdx index b9d2c3cb..8228b658 100644 --- a/src/content/docs/docs/alerts/folders.mdx +++ b/src/content/docs/docs/alerts/folders.mdx @@ -5,9 +5,9 @@ sidebar: order: 4 --- -Folders organize related alert rules and let member rules inherit shared evaluation defaults. They bring the dashboards-style grouping pattern to alerts: opt into a folder view, see your rules collapsed under their owning folder, and let one set of defaults propagate to whichever rules need them. +Folders organize related alert rules and let member rules inherit shared evaluation defaults, the same grouping pattern dashboards use. -On the **Alerts** list, the layout toggle in the toolbar switches between a flat **List view** and a grouped **Folder view**. In folder view, each folder renders as a collapsible header row with its member rules underneath; rules without a folder land in a pinned **Ungrouped** section. +On the **Alerts** list, switch between a flat **List view** and a grouped **Folder view**. In Folder view, rules are grouped under their folder; rules without one appear under **Ungrouped**. ![Alerts list in folder view](./images/alerts-folder-view.png) @@ -19,8 +19,8 @@ You create folders inline from the alert rule editor's **Folder** picker; there' 1. Open an alert rule (create a new one or edit an existing one). 2. In the **Configure evaluation settings** section, click the **Folder** dropdown. -3. Type a new folder name. A `+ Create ""` option appears at the bottom of the dropdown. Click it. -4. KloudMate creates the folder and assigns the current rule to it. A toast confirms the creation. +3. Type a new folder name, then select the `+ Create ""` option that appears. +4. KloudMate creates the folder and assigns the current rule to it. 5. To set folder-level defaults (interval, no-data state, eval-error state), switch to **Folder view** on the Alerts list, open the folder's kebab menu, and select **Edit folder**. ## Edit folder defaults @@ -37,7 +37,7 @@ In Folder view, each folder header row has a kebab menu with **Edit folder** and | **No data state** | Default state when queries return no data. Options: `Normal`, `Alerting`, `NoData`. | | **Eval error state** | Default state when evaluation errors out. Options: `Normal`, `Alerting`, `Error`. | -A note above these reads: *"Member alerts inherit these defaults unless they set their own value. Leave blank for organizational-only folders."* +Member alerts inherit these defaults unless they set their own value. Leave a folder's fields blank to make it organizational-only. ## Move rules into a folder @@ -46,34 +46,35 @@ Move rules from the rule editor, or in bulk from the list: - **From the rule editor:** open the rule and pick the folder from the **Folder** dropdown in the **Configure evaluation settings** section. - **Bulk-move from the list:** select rules with the checkboxes (a toolbar appears with the selection count). Click **Move to folder…**, select a destination from the radio list (existing folders or **Ungrouped**), and confirm. -Moving a rule does **not** rewrite its `interval_seconds`, `no_data_state`, or `eval_error_state` columns. Whatever the rule had before the move stays. The folder defaults only apply where the rule's field is blank. +Moving a rule does **not** rewrite its own **Evaluate every**, **No data state**, or **Eval error state** values. Whatever the rule had before the move stays. The folder defaults only apply where the rule's field is blank. ## Inheritance and overrides -A rule's eval-config fields follow a clear three-state model: +A rule's evaluation fields resolve like this: -| Rule field value | What the editor shows | What runs | -|---|---|---| -| Blank (inheriting) | Folder default as placeholder + `Inheriting from folder ` helper line | The folder default | -| Filled (overriding) | The rule's stored value + `Overrides folder default` helper line + **Reset to folder default** link | The rule's value | -| No folder selected | Plain input, no helper text | The rule's value (or system default if blank) | +| Rule field | What runs | +|---|---| +| Blank, inheriting from its folder | The folder's default | +| Set, overriding its folder | The rule's own value | +| Blank, with no folder | The system default | +| Set, with no folder | The rule's own value | -Clicking **Reset to folder default** clears the field and flips the helper text back to inheriting. Saving with the field blank persists `NULL` for that column, so when the folder default later changes, the rule picks up the new value automatically. +**Reset to folder default** clears an overridden field so it inherits again. A blank field keeps inheriting, so when the folder default later changes, the rule picks up the new value automatically. :::tip -To detach a rule from its folder entirely, change the **Folder** dropdown to the ungrouped option. The eval-config columns keep their stored values; the rule stops inheriting. +To detach a rule from its folder entirely, set its **Folder** to the ungrouped option. The rule keeps its stored evaluation values and stops inheriting. ::: ## Delete a folder -From the folder kebab menu, pick **Delete folder**. The confirmation explains that member alerts will become ungrouped, with their stored eval-config values intact. KloudMate detaches the members first and then deletes the folder row. Rules survive the deletion. +From the folder's kebab menu, select **Delete folder**. Member alerts become ungrouped, keeping their stored evaluation values. KloudMate detaches the members first, then deletes the folder; the rules survive. ## Folder labels in alert groups A rule's folder identity flows into the alert's labels as `alarm_rule_folder_id` and `alarm_rule_folder_name`. That means: - [Routing Rules](../routing-rules/) can match on folder identity (for example, `alarm_rule_folder_name in [Kubernetes, Networking]`). -- The folder name shows as a label chip on the [Alert Group](../alert-groups/) detail header automatically. +- The folder name appears automatically as a label on the [Alert Group](../alert-groups/). ## Related diff --git a/src/content/docs/docs/alerts/instance-absence-detection.mdx b/src/content/docs/docs/alerts/instance-absence-detection.mdx index 51b184d0..d458d68e 100644 --- a/src/content/docs/docs/alerts/instance-absence-detection.mdx +++ b/src/content/docs/docs/alerts/instance-absence-detection.mdx @@ -8,9 +8,9 @@ sidebar: import { LinkCard, CardGrid } from '@astrojs/starlight/components'; -A threshold alert can only judge the datapoints that arrive. It has nothing to say when the datapoints stop. **Alert when an instance stops reporting** covers that case: turn it on for a rule and KloudMate remembers every instance the rule has seen, then alerts when any of them goes silent. +A threshold alert only evaluates the datapoints that arrive. When the datapoints stop, it has nothing to evaluate. **Alert when an instance stops reporting** covers that case: turn it on for a rule and KloudMate remembers every instance the rule has seen, then alerts when any of them goes silent. -An **instance** is a unique series identified by its distinct labels. A heartbeat metric grouped by `host_name` produces one instance per host; group by `serviceName` and `pod_name` and you get one per pod. Absence detection tracks each instance individually: it has its own state, its own history, its own row on the **Instances** tab, and its own dismissal. +An **instance** is a unique series identified by its distinct labels. A heartbeat metric grouped by `host_name` produces one instance per host; group by `serviceName` and `pod_name` and you get one per pod. Absence detection tracks each instance individually: it has its own state, its own history, and its own dismissal. The setting is off by default and opt-in per rule, so existing rules don't change until you enable it. It works with any datasource, whether the data comes from the KloudMate agent or not. @@ -32,9 +32,9 @@ Routing itself works as usual, since each instance carries its own labels. 3. Optionally set **Auto-close after**: how long a silent instance stays tracked, and firing, before it closes on its own, which happens only while other instances are still reporting. Leave it blank to inherit the folder default, or the system default of 24 hours. It accepts durations from `5m` to `72h`, for example `30m` or `48h`. 4. Save the rule. -Turning the toggle on also sets **Alert state if No data** to **Firing**. Both are needed for absence detection, so the form does it in one step. Turning the toggle off leaves the No-data state alone, in case you still want the rule to fire when the whole query returns nothing. +Turning the toggle on also sets **Alert state if No data** to **Firing**, since absence detection needs both. Turning the toggle off leaves the No-data state alone, in case you still want the rule to fire when the whole query returns nothing. -Both settings inherit from [folders](../folders/) exactly like the other evaluation defaults: the rule's own value wins, a blank field falls back to the folder default, and the system default applies last. The folder edit dialog carries the matching fields, **Instance stops reporting** (**Alert** / **Don't alert**) and **Auto-close after**. +Both settings inherit from [folders](../folders/) exactly like the other evaluation defaults: the rule's own value wins, a blank field falls back to the folder default, and the system default applies last. Folders expose the same two settings, **Instance stops reporting** (**Alert** / **Don't alert**) and **Auto-close after**. ## Make the query window longer than the reporting interval @@ -67,15 +67,13 @@ That's about eight minutes after the last datapoint: the 5-minute window plus th ## The Instances tab -The alert detail page's **Instances** tab shows every tracked instance with a **Last seen** column: a relative time, with the absolute timestamp on hover. It records when the instance last appeared in query results, so with a lookback window it can read up to one window later than the instance's final datapoint. While an instance is firing for absence, its row shows the reason `instance stopped reporting; last seen `. +The **Instances** tab lists every tracked instance and when each was **Last seen**. Last seen records when the instance last appeared in query results, so with a lookback window it can read up to one window later than the instance's final datapoint. While an instance is firing for absence, its reason is `instance stopped reporting; last seen `. ### Dismiss an instance that was terminated on purpose -Autoscalers scale in and hosts get decommissioned. When an instance went away on purpose, close its alert with one click instead of waiting out the auto-close window. Admins see a **Dismiss** action on any instance that stopped reporting; the confirmation reads: +Autoscalers scale in and hosts get decommissioned. When an instance went away on purpose, close its alert instead of waiting out the auto-close window. Admins can **Dismiss** any instance that stopped reporting. -> Stop tracking this instance? Its alert closes and it won't fire again unless it reports again. Use this when the instance was terminated on purpose. - -Dismissal forgets the instance; it isn't a mute. If the instance ever reports again, tracking resumes automatically, so a dismissal can't permanently hide a live host. The instance's alert closes right away, and the alert group and any linked ticket close within one evaluation interval. Dismissing an instance that's already gone shows "Already dismissed." and changes nothing. +Dismissal forgets the instance; it isn't a mute. If the instance ever reports again, tracking resumes automatically, so a dismissal can't permanently hide a live host. The instance's alert closes right away, and the alert group and any linked ticket close within one evaluation interval. Dismissing an instance that's already been dismissed changes nothing. If the instance is coming back later, after planned maintenance or a reboot, use a [silence](../silences/) or [maintenance window](../maintenance-windows/) instead. Both apply to absence alerts like any other: the instance keeps its state, notifications are withheld, and it recovers on its own when it reports again. diff --git a/src/content/docs/docs/alerts/maintenance-windows.mdx b/src/content/docs/docs/alerts/maintenance-windows.mdx index db028c23..2ddd30af 100644 --- a/src/content/docs/docs/alerts/maintenance-windows.mdx +++ b/src/content/docs/docs/alerts/maintenance-windows.mdx @@ -5,7 +5,7 @@ sidebar: order: 9 --- -A Maintenance Window is a scheduled period during which matching alerts keep evaluating but their notifications are suppressed. State changes are still recorded in alert history (marked with `silenced_at` / `silenced_by_*` so you can audit what happened), but no notification fires. Like a [Silence](../silences/), a maintenance window is a **notification gate**: it mutes notifications for the matching alerts but never changes their state or resolves them. +A Maintenance Window is a scheduled period during which matching alerts keep evaluating but their notifications are suppressed. State changes are still recorded in alert history, so you can audit what happened, but no notification fires. Like a [Silence](../silences/), a maintenance window is a **notification gate**: it mutes notifications for the matching alerts but never changes their state or resolves them. Use a maintenance window when you have a calendar-driven quiet period: a planned deployment, a recurring batch job window, or a vendor outage you've been told to expect. For one-off, ad-hoc suppression in the moment, use [Silences](../silences/) instead. @@ -17,7 +17,7 @@ Use a maintenance window when you have a calendar-driven quiet period: a planned ## Overview page -The overview page lists every maintenance window with these details: +Each maintenance window shows: - **Name:** name of the maintenance window. - **Type:** whether the window is **One Time** or **Recurring**. @@ -26,8 +26,6 @@ The overview page lists every maintenance window with these details: - **Schedule:** start and end time for the maintenance window. - **Status:** whether the maintenance window is active or paused. -Each row also has an **Actions** menu for further row-level controls. - ![Maintenance window overview](./images/maintenance-window-1.png) ## Creating a maintenance window diff --git a/src/content/docs/docs/alerts/routing-rules.mdx b/src/content/docs/docs/alerts/routing-rules.mdx index e50d3cd1..da4a2d79 100644 --- a/src/content/docs/docs/alerts/routing-rules.mdx +++ b/src/content/docs/docs/alerts/routing-rules.mdx @@ -17,15 +17,15 @@ Routing rules replace the older **Notification Policies** flow. If you previousl ## Where to find them -Open **Alerts → Routing rules** in the left navigation. Rules are evaluated in priority order (lowest number wins). The first rule an alert matches decides which channels notify and how often, and whether the alert notifies on its own or correlates with others into one [Alert Group](../alert-groups/). +Open **Alerts → Routing rules**. Rules are evaluated in priority order (lowest number wins). The first rule an alert matches decides which channels notify and how often, and whether the alert notifies on its own or correlates with others into one [Alert Group](../alert-groups/). ![Routing Rules list](./images/routing-rules-list.png) -The list shows every routing rule in the workspace, sorted by priority (lower wins). The **default passthrough rule** sits at the bottom. It matches everything that didn't already match a higher-priority rule, and it can't be deleted. +Rules are sorted by priority, lowest number first. The **default passthrough rule** matches everything no higher-priority rule caught, and it can't be deleted. ## Anatomy of a rule -A routing rule has these parts: +A routing rule has these fields: | Field | What it does | |---|---| @@ -46,12 +46,12 @@ Each matcher is one label condition: a label key, an operator, and a value. An a The available operators: -| Operator | Wire format | Meaning | -|---|---|---| -| Equals | `equals` | Label value matches the string exactly. | -| Not equals | `not_equals` | Label value differs from the string (also matches alerts that don't carry the label at all). | -| Matches regex | `matches_regex` | Label value matches the regular expression. | -| Doesn't match regex | `not_matches_regex` | Label value doesn't match the regex (also matches alerts without the label). | +| Operator | Meaning | +|---|---| +| Equals | Label value matches the string exactly. | +| Not equals | Label value differs from the string (also matches alerts that don't carry the label at all). | +| Matches regex | Label value matches the regular expression. | +| Doesn't match regex | Label value doesn't match the regex (also matches alerts without the label). | A matcher value is always a single string. To match any of several values, use a regex alternation. For example, `service` **Matches regex** `api|web` matches either service. (There's no `in`/`not_in` operator; the regex form replaces it.) @@ -62,11 +62,9 @@ As you add or edit matchers, the editor shows a live **Affected Alert Instances* Each rule has an **Alert grouping** control, and it defaults to **Static**. - **Static** groups matching alerts by the label keys you choose, the deterministic behavior described under [Group-by keys](#group-by-keys) below. Alerts that share those key values fold into one group. -- **Auto (AI)** hands grouping to the correlation engine. It correlates related alerts from this rule into a single incident on its own, and learns which alerts tend to fire together over time. In this mode the Group-by keys picker is hidden, so you don't pick grouping keys at all. - -In-product, the Auto option reads: *"AI and ML automatically correlate related alerts from this rule into a single incident, learning which alerts tend to fire together over time. No grouping keys needed."* +- **Auto (AI)** hands grouping to the correlation engine. It correlates related alerts from this rule into a single incident on its own, and learns which alerts tend to fire together over time. In this mode you don't choose grouping keys at all. -Leave a rule on **Static** when you want predictable grouping along keys you control. Switch it to **Auto (AI)** when related alerts don't share a clean label to group on, or when you'd rather let the engine find the connections. Either mode produces the same [Alert Group](../alert-groups/); that page shows what the grouped incident looks like and how an Auto-correlated group explains itself. +Leave a rule on **Static** when you want predictable grouping along keys you control. Switch it to **Auto (AI)** when related alerts don't share a clean label to group on, or when you'd rather let the engine find the connections. Either mode produces the same [Alert Group](../alert-groups/). For how each mode decides what belongs together, including why one rule firing on many hosts stays a single Auto group, see [How Alert Grouping Works](../how-alert-grouping-works/). @@ -79,7 +77,7 @@ Group-by keys apply in **Static** mode. They decide **whether** matched alerts c - **Leave it empty, or set only `alarm_id`,** and the rule doesn't correlate anything: each alert rule notifies on its own, one notification per rule. These notifications surface through the normal alerts UI, not the [Alert Groups](../alert-groups/) page. - **Add a real label key** and matching alerts that share that key's value fold into a single alert group. The editor suggests common keys (`service`, `env`, `host`, `region`, `team`, `issue_id`, `check_id`, `alarm_rule_folder_id`), and you can type any other label key. -The editor enforces two constraints: +The editor enforces these constraints: - If you add a real dimension alongside `alarm_id`, KloudMate drops the `alarm_id` automatically. Pinning each group to a single alert rule would defeat the correlation. - `severity` can't be a grouping dimension. It's a per-instance property fixed when the group opens, not an axis to group on. @@ -88,7 +86,7 @@ You can combine keys (for example, `service` + `env`) to scope each group tightl ### Notification timing -The **Notification timing** section of the rule editor decides when the first notification goes out and how often the rule follows up. Enter each setting as duration text, such as `30s`, `5m`, or `4h`. +**Notification timing** decides when the first notification goes out and how often the rule follows up. Enter each setting as duration text, such as `30s`, `5m`, or `4h`. | Setting | Default | What it controls | |---|---|---| @@ -114,7 +112,7 @@ A reminder is delivered with `"event": "repeat"` in the payload. Everything else ## The default passthrough rule -Every workspace has a default passthrough rule pinned to the bottom of the list. It matches everything that didn't match a higher-priority rule. You can edit its destination channels and notification timing, but you can't delete it or change its matchers. +Every workspace has a default passthrough rule. It matches everything that didn't match a higher-priority rule. You can edit its destination channels and notification timing, but you can't delete it or change its matchers. For a clean "everything else goes to Slack" path, point the default passthrough at your fallback Slack channel. @@ -135,7 +133,7 @@ For a clean "everything else goes to Slack" path, point the default passthrough 7. **Optionally enable Auto-RCA.** Toggle Auto-RCA on and set the delay (default 5 minutes after group open). See [Auto-RCA](../auto-rca/). -8. **Save.** The rule appears in the list at its assigned priority. +8. **Save.** ## Row actions @@ -150,7 +148,7 @@ The **Enabled** toggle disables a rule without deleting it. Use it to stage a ru ## Suggest rules -The list header has a **Suggest rules** button (visible to admins only). It opens a drawer with up to five AI-suggested rules based on your workspace's recent alert traffic. +Admins can open **Suggest rules** to get up to five AI-suggested rules based on your workspace's recent alert traffic. ![Suggest rules drawer](./images/routing-rules-suggest.png) diff --git a/src/content/docs/docs/alerts/silences.mdx b/src/content/docs/docs/alerts/silences.mdx index 3462be87..6e1c4197 100644 --- a/src/content/docs/docs/alerts/silences.mdx +++ b/src/content/docs/docs/alerts/silences.mdx @@ -11,11 +11,11 @@ Silences cap at 30 days. The cap keeps a silence from being forgotten and quietl ## Where to find silences -Open **Alerts → Silences** in the left navigation. Silences are ad-hoc; for recurring quiet windows, use [Maintenance Windows](../maintenance-windows/) instead. +Open **Alerts → Silences**. Silences are ad-hoc; for recurring quiet windows, use [Maintenance Windows](../maintenance-windows/) instead. ![Silences list](./images/silences-list.png) -The list has tabs with live counts: **Active (N)** (default), **Expired (N)**, and **All (N)**. Columns: **Matchers** (chips, with an overflow tooltip), **Expires** (a countdown such as *"In 4h"* or *"Expired 12m ago"*), **Scope** (a `Standalone` or `Group-bound` chip), **Reason**, and **Created by**. +Silences are grouped into **Active**, **Expired**, and **All** tabs. ## Anatomy of a silence @@ -24,7 +24,7 @@ The list has tabs with live counts: **Active (N)** (default), **Expired (N)**, a | **Matchers** | Label conditions that pick which alerts the silence applies to. Same operator set as [Routing Rules](../routing-rules/#matchers): Equals, Not equals, Matches regex, Doesn't match regex. | | **Expires at** | Date and time the silence stops. Capped at 30 days from creation. | | **Reason** | Free-form text; recommended so others know why the silence exists. | -| **Scope** | Either **Group-bound** (auto-expires when a specific [Alert Group](../alert-groups/) resolves) or **Standalone** (lives until its `expires_at`). | +| **Scope** | Either **Group-bound** (auto-expires when a specific [Alert Group](../alert-groups/) resolves) or **Standalone** (lives until it expires). | ## Create a silence @@ -34,23 +34,22 @@ Create a silence from the Silences page, an alert group, or an alert's Pause Not 1. Open **Alerts → Silences** and click **New silence**. 2. Add one or more matcher rows. For each, pick a label key, an operator, and a value. -3. Pick an **Expires at** date and time. The form caps duration at 30 days and disables submit otherwise. +3. Pick an **Expires at** date and time, up to 30 days out. 4. Add a **Reason** describing why this silence exists. -5. Click **Save**. The silence appears in the **Active** tab. +5. Click **Save**. ### From an Alert Group On any [Alert Group](../alert-groups/) detail page, click **Silence this group**. The silence creator opens pre-filled with: - The group's labels populated as matchers. -- A hidden `auto_expire_group_id` set to the current group, so the silence stops automatically when the group resolves. -- A banner above the form that reads something like "This silence will auto-expire when group `` resolves." +- A binding to the group, so the silence stops automatically when the group resolves. -Save it from here. The silence shows up as **Group-bound** on the list. +Save it; this creates a **Group-bound** silence. ### From an alert's Pause Notifications -On an alert rule's **more options (⋯) → Pause Notifications**, KloudMate creates a silence scoped to that alert (an `alarm_id` matcher) with an expiry: the alert keeps evaluating, and only its notifications are suppressed. While paused, the rule is marked **Silenced** on the alerts list and its detail page, and the silence appears here in the **Active** tab, where you view or end it. Add matchers to narrow the pause to specific instances. +On an alert rule's **more options (⋯) → Pause Notifications**, KloudMate creates a silence scoped to that alert (an `alarm_id` matcher) with an expiry: the alert keeps evaluating, and only its notifications are suppressed. While paused, the rule is marked **Silenced**; view or end the pause from **Alerts → Silences**. Add matchers to narrow the pause to specific instances. ## Silence vs. maintenance window @@ -59,15 +58,15 @@ Both features are a **notification gate, not a state change**: the alert still e - **Silences** are ad-hoc, label-matcher driven, capped at 30 days, and can be auto-bound to a specific alert group so they expire when the group resolves. - **Maintenance Windows** are scheduled: one-time (RFC3339 start/end) or recurring (`DAILY` / `WEEKLY` / `MONTHLY` with a timezone-aware time-of-day). -Both use the same four label-matcher operators (**Equals**, **Not equals**, **Matches regex**, **Doesn't match regex**), matched against alert rule labels and alert instance labels. +Both use the same label-matcher operators (**Equals**, **Not equals**, **Matches regex**, **Doesn't match regex**), matched against alert rule labels and alert instance labels. -Both leave the alert evaluating. Suppressed state transitions are marked with `silenced_at` and `silenced_by_*` columns on the alert state history, so you can always tell after the fact whether a quiet stretch was a real recovery or a suppressed notification. +Both leave the alert evaluating. Suppressed state transitions are recorded in the alert's history, so you can always tell after the fact whether a quiet stretch was a real recovery or a suppressed notification. Pick a silence for a reactive, one-off suppression you'll re-evaluate within 30 days. Pick a [Maintenance Window](../maintenance-windows/) for planned downtime, recurring quiet hours, or anything that needs a calendar schedule. ## Viewing and ending a silence -Each row's kebab menu has **View** and **End silence**. Ending an active silence removes suppression for matching alerts immediately; new firings notify as normal. +From a row's kebab menu, select **End silence** (or **View**). Ending an active silence removes suppression for matching alerts immediately; new firings notify as normal. ## Related diff --git a/src/content/docs/docs/alerts/smart-alerts.mdx b/src/content/docs/docs/alerts/smart-alerts.mdx index 210c270f..a28b5ee1 100644 --- a/src/content/docs/docs/alerts/smart-alerts.mdx +++ b/src/content/docs/docs/alerts/smart-alerts.mdx @@ -25,7 +25,7 @@ Smart Alerts rules are ordinary alert rules. They move through the same [lifecyc ## Detection methods -There are 3 different detector methods: +Each detector uses one of these methods: | Method | Fires when | Best for | |---|---|---| @@ -37,7 +37,7 @@ There are 3 different detector methods: An anomaly detector learns what normal looks like for each entity from its own recent history, then fires when a reading lands well outside that range. There's no threshold to set. The one knob is **Sensitivity**: **High** catches small deviations, **Low** reacts only to large ones, and **Medium** (the default) suits most metrics. -A brand-new entity has no history to learn from yet. Until enough builds up, it runs on a provisional range and does not fire; its reason reads `collecting baseline`. That's deliberate. A young or flat series would otherwise score an ordinary warm-up ramp as a wild anomaly, and pod names churn on every deploy. Genuine saturation is still caught while a baseline forms, by the [Threshold](#threshold) nets below. +A brand-new entity has no history to learn from yet. Until enough builds up, it runs on a provisional range and does not fire; its reason reads `collecting baseline`. That's deliberate. A young or flat series would otherwise score an ordinary warm-up ramp as a wild anomaly, and pod names churn on every deploy. The [Threshold](#threshold) detectors below still catch genuine saturation while a baseline forms. ### Forecast @@ -47,7 +47,7 @@ Forecasting holds until a series has enough history to tell a real trend from a ### Threshold -A threshold detector is a plain ceiling for a state that's wrong at any scale. It needs no history and fires the moment the value crosses the line: a disk at least 95% full, a container at 95% of its memory limit (OOM risk), a Kubernetes node not Ready, or Kafka partitions under their minimum in-sync replicas. Each threshold ships with a sensible default and is overridable per workspace. +A threshold detector is a plain ceiling for a state that's wrong at any scale. It needs no history and fires the moment the value crosses the line: a disk at least 95% full, a container at 95% of its memory limit (OOM risk), a Kubernetes node not Ready, or Kafka partitions under their minimum in-sync replicas. Each threshold ships with a sensible default that you can override per workspace. ## Browse and enable detectors @@ -99,15 +99,15 @@ Smart Alerts won't fire on a single spike. A breach has to hold across several e Adjust a detector in place from the Smart Alerts page: **Sensitivity** for an anomaly detector, the **horizon** for a forecast, or the **threshold** value for a threshold detector. KloudMate applies the change and keeps managing the rule. -Edit the rule's query or condition in the alert builder and it becomes yours. The rule detaches from Smart Alerts, which stops updating it, and its detail page notes: +Changing how often the rule runs keeps it managed too. Edit **Evaluate every**, **Pending duration**, or **Recovery period** on the rule and KloudMate keeps those values instead of resetting them to the detector's defaults. -> Managed by Smart Alerts. Editing the query or condition converts it to a custom alert that you own. +Editing the rule's **query**, **threshold**, or **condition** takes it over. The rule detaches from Smart Alerts, is labeled **Customized**, and leaves the **Smart Alerts** folder. From then on it behaves like any alert you wrote by hand: KloudMate stops sending it template updates and no longer manages its lifecycle. Its anomaly or forecast detection keeps running; taking a rule over doesn't turn detection off. -To stop a managed alert without taking it over, turn its detector off on the Smart Alerts page. Deleting one directly is blocked, since Smart Alerts would recreate it. KloudMate disables it instead and points you to the detector toggle to remove it for good. +To quiet a managed rule without taking it over, pause it from its row menu with **Pause Evaluation**. The pause holds: saving another Smart Alert won't resume it. To remove a managed rule for good, turn its detector off on the Smart Alerts page. You can't delete a managed rule directly, since Smart Alerts would recreate it. KloudMate disables it instead and points you to the detector toggle. ## Plan availability -Anomaly and forecast detection is a paid-plan capability. You can browse the detector catalog on any plan, and detectors light up as your data arrives, but turning on automatic monitoring, along with adding anomaly or forecast conditions to an alert you build by hand, needs a plan with anomaly detection. Plain threshold alerting isn't gated: you can always [create a threshold alert](../create-alerts/) yourself. When the capability is off, the page shows an upgrade prompt with a **View plans** link. +Anomaly and forecast detection needs a paid plan, both for turning on automatic monitoring and for adding an anomaly or forecast condition to an alert you build by hand. You can browse the detector catalog on any plan, and plain threshold alerting works everywhere: you can always [create a threshold alert](../create-alerts/) yourself. ## Related diff --git a/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx b/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx index 5736728f..50d1b1c3 100644 --- a/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx +++ b/src/content/docs/docs/kloudmate-agent/baseline/host-metrics-and-logs.mdx @@ -1,6 +1,6 @@ --- title: "Host metrics and logs" -description: "What the KloudMate agent collects automatically after install: host metrics on Linux, Windows, Kubernetes, and Docker, plus the system, container, or pod logs each platform exposes — with no configuration." +description: "What the KloudMate agent collects automatically after install: host metrics on Linux, Windows, Kubernetes, and Docker, plus container and pod logs on Docker and Kubernetes. No configuration needed." sidebar: order: 1 --- @@ -23,9 +23,9 @@ What the agent collects depends on the deployment mode, because each platform ex </TabItem> <TabItem label="Windows"> -- **Host metrics:** CPU, memory, disk, and network usage, collected from Windows performance counters. -- **Windows Event Log:** the System, Application, and Security channels. -- **Internet Information Services (IIS) metrics** and **Windows service status**, where those roles are present. +- **Host metrics:** CPU, memory, disk, and network usage. +- **Logs:** none are collected from the host automatically, including the Windows Event Log. Add application log files or Event Log channels with [log monitoring](../../log-monitoring/). +- **Application traces:** none are collected until you turn on tracing for a service, the same as on Linux. Internet Information Services (IIS) traffic is traced through Event Tracing for Windows (ETW) once you enable it for an application pool. See [Windows platform notes](../../platform-notes/windows/#etw-monitoring). </TabItem> <TabItem label="Kubernetes"> diff --git a/src/content/docs/docs/kloudmate-agent/installation/before-you-start.mdx b/src/content/docs/docs/kloudmate-agent/installation/before-you-start.mdx index 9533ca5a..8b7e69e7 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/before-you-start.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/before-you-start.mdx @@ -35,7 +35,7 @@ The agent only makes **outbound** connections to KloudMate. It needs no inbound | Environment | To install | To run and send data | |---|---|---| | **Linux (VM or host)** | root, through `sudo` | Runs as root. eBPF monitoring needs kernel **4.14+**, the `CAP_SYS_ADMIN` or `CAP_BPF` capability, and `debugfs` mounted. Without these, the host still gets metrics and logs. Instrumenting a service writes a systemd config file and restarts that one service. | -| **Windows** | Administrator (Server 2016 or newer) | Runs as the **Local System** account, which it needs to read IIS and WMI. | +| **Windows** | Administrator (Server 2016 or newer) | Runs as the **Local System** account, which it needs to read IIS configuration and query the Service Control Manager. | | **Docker** | Permission to run a container with host mounts | Mount the Docker socket (`/var/run/docker.sock`) for container discovery and PHP tracing. eBPF needs a **privileged** container with host access; a plain container gets container metrics and logs only. | | **Kubernetes** | Kubernetes **1.24+**, Helm **3+**, and cert-manager; permission to create the namespace, cluster RBAC, and CRDs | The node DaemonSet runs **privileged** (`runAsUser: 0`, `CAP_SYS_ADMIN`, `hostPID`, and `hostNetwork` for network flows) with host-path mounts. On clusters with Pod Security restrictions, allow this in the agent's namespace. The chart installs the RBAC. | | **Amazon ECS** | AWS permission to register task-definition revisions and update the services you instrument | Reading task definitions uses `ecs:DescribeTaskDefinition`. A Secrets Manager API key needs `secretsmanager:GetSecretValue` on the task execution role; a custom log group needs `logs:CreateLogGroup`. eBPF needs privileged tasks, or set `KM_EBPF_ENABLED=false`. | diff --git a/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx b/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx index 18c5b2c6..ed2c12b6 100644 --- a/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx +++ b/src/content/docs/docs/kloudmate-agent/installation/windows-agent.mdx @@ -1,6 +1,6 @@ --- title: "Windows" -description: "Install the KloudMate agent on Windows Server to collect host metrics, Windows Event Logs, and application logs." +description: "Install the KloudMate agent on Windows Server to collect host metrics, application logs, and IIS traces." sidebar: order: 6 --- @@ -18,10 +18,9 @@ If your outbound traffic is restricted, see [Before you start](../before-you-sta Once installed, the Windows Agent collects: - **Host metrics:** CPU, memory, disk, and network usage -- **Service monitoring:** Status of critical Windows services (optional, if configured) -- **Windows Event Logs:** System, Application, and Security logs -- **Application logs:** Logs from installed applications (if configured) -- **Alerts:** Configure thresholds for host metrics or service status to notify on critical events +- **Logs:** log files and Windows Event Log channels that you configure with [log monitoring](../../log-monitoring/). Nothing is collected until you add a source +- **IIS traces:** server spans and request-duration metrics through Event Tracing for Windows (ETW), for each application pool you turn tracing on for +- **Alerts:** Configure thresholds on host metrics to notify on critical events - **Dashboards:** Pre-built and custom dashboards to visualize Windows host metrics and logs This allows you to monitor Windows hosts, troubleshoot issues, and correlate logs with metrics for performance insights. @@ -33,7 +32,7 @@ View all collected data in these KloudMate sections: **1. Infrastructure Monitoring** - Monitor Windows host metrics (CPU, memory, disk, network) -- Track service status and host availability +- Track host availability - Monitor uptime and performance trends [Infrastructure Monitoring](../../../infrastructure/) @@ -47,7 +46,7 @@ View all collected data in these KloudMate sections: **3. Log Explorer** -- Access Windows Event Logs (System, Application, Security) +- Read the log files and Windows Event Log channels you configured with log monitoring - Search and filter logs by source, severity, or timestamp - Correlate logs with host metrics to troubleshoot performance issues @@ -87,28 +86,28 @@ The service installs and starts the same as the interactive install. Replace `km ## **Log Integration** -By default, the KloudMate Agent collects basic system logs (the Windows Event Log). +The agent doesn't collect any logs from the host on its own, and that includes the Windows Event Log. You choose what it reads. To collect your own application log **files**, use log monitoring. In the agent's **Logs** tab, point the agent at a folder and file-name pattern, then choose how to parse each line: plain text, JSON, or a regular expression. Multiline grouping, for stack traces or pretty-printed JSON, works on top of any format. You can preview the parsed records before you save. See [Log monitoring](../../log-monitoring/). -Log monitoring can also collect **additional Windows Event Log channels** and read log-source files placed in `C:\ProgramData\kmagent\logs.d`. See [Windows Event Log channels](../../log-monitoring/#windows-event-log-channels) and [Local file sources](../../log-monitoring/#local-file-sources). +Log monitoring also collects **Windows Event Log channels**, such as `System` and `Application`, and reads log-source files placed in `C:\ProgramData\kmagent\logs.d`. See [Windows Event Log channels](../../log-monitoring/#windows-event-log-channels) and [Local file sources](../../log-monitoring/#local-file-sources). For a hand-written `filelog` receiver in manual mode, see [Collect File Logs with KloudMate Agent](../../../logs/collect-file-logs/). ## Configuration and next checks -The agent installs as a Windows service running as Local System. It needs that privilege level to read Internet Information Services (IIS) configuration and Windows Management Instrumentation (WMI). New agents start in managed mode. See the [configuration model](../../concepts/config-model/). +The agent installs as a Windows service running as Local System. It needs that privilege level to read Internet Information Services (IIS) configuration and to query the Service Control Manager. New agents start in managed mode. See the [configuration model](../../concepts/config-model/). -- Host metrics and the Windows Event Log start flowing automatically. See [Host metrics and logs](../../baseline/host-metrics-and-logs/). +- Host metrics start flowing automatically. Logs and traces don't, so turn on the ones you want. See [Host metrics and logs](../../baseline/host-metrics-and-logs/). - eBPF does not apply on Windows. The agent gives you equivalent monitoring through Event Tracing for Windows (ETW). See [Windows platform notes](../../platform-notes/windows/). - To add distributed tracing to .NET applications, see [.NET application APM](../../auto-instrumentation/dotnet/). ## **Next Steps** -- Open **Infrastructure Monitoring** to see host metrics and service status +- Open **Infrastructure Monitoring** to see host metrics - Go to **Dashboards** to visualize host performance and logs -- Use **Log Explorer** to investigate Windows Event Logs and application logs -- Configure **alerts** for host or service thresholds +- Set up [log monitoring](../../log-monitoring/), then use **Log Explorer** to investigate the logs it collects +- Configure **alerts** for host metric thresholds ## Uninstall Agent diff --git a/src/content/docs/docs/kloudmate-agent/log-monitoring.mdx b/src/content/docs/docs/kloudmate-agent/log-monitoring.mdx index 95058239..7997b818 100644 --- a/src/content/docs/docs/kloudmate-agent/log-monitoring.mdx +++ b/src/content/docs/docs/kloudmate-agent/log-monitoring.mdx @@ -141,7 +141,7 @@ On Windows, a log source can collect **Windows Event Log channels** instead of f include: [ "Microsoft-Windows-PowerShell/Operational" ] ``` -Only new events are collected, with no historical backfill. The System, Application, and Security logs are already collected by default (see [Host metrics and logs](../baseline/host-metrics-and-logs/)), so use `eventlog` sources for other channels. Event Log sources are Windows-only. +Only new events are collected, with no historical backfill. No channel is collected until you add a source for it, so add `System`, `Application`, and `Security` the same way if you want them. Event Log sources are Windows-only. ## Manual configuration diff --git a/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx b/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx index 3e2b2b5b..245b37e1 100644 --- a/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx +++ b/src/content/docs/docs/kloudmate-agent/platform-notes/windows.mdx @@ -1,6 +1,6 @@ --- title: "Windows platform notes" -description: "How the KloudMate agent runs on Windows: discovery through IIS and WMI, ETW monitoring, injection mechanisms, and privileges." +description: "How the KloudMate agent runs on Windows: discovery through IIS and the Service Control Manager, ETW monitoring, injection mechanisms, and privileges." sidebar: order: 2 --- @@ -11,21 +11,40 @@ How the agent runs on Windows. For installation, see the [Windows installation g ## A different mechanism, the same model -Windows has no `LD_PRELOAD` and no production eBPF, so the agent uses Windows-native mechanisms. It keeps the same discover, instrument, and report model as the other platforms. Instead of systemd config files, the agent uses the Windows registry, Windows Management Instrumentation (WMI), and Internet Information Services (IIS) configuration. +Windows has no `LD_PRELOAD` and no production eBPF, so the agent uses Windows-native mechanisms. It keeps the same discover, instrument, and report model as the other platforms. Instead of systemd config files, the agent uses the Windows registry, the Service Control Manager, and Internet Information Services (IIS) configuration. ## Discovery To build the Discovered Services list, the agent looks at: -- **IIS** sites and application pools, including each pool's managed runtime version and state. -- **Windows services**, through the Service Control Manager. -- **Listening ports**, through WMI, so it can map applications to ports. +- **IIS** sites and application pools, including each pool's managed runtime version and state. These appear with a kind of `iis`. +- **Windows services**, through the Service Control Manager. These appear with a kind of `windows_service`. +- **Listening ports**, so it can map applications to ports. -Discovered services appear with a kind of `iis` or `windows_service`. See [Discovery](../../concepts/discovery/). +See [Discovery](../../concepts/discovery/). + +### Which Windows services are listed + +The agent lists a Windows service when it recognizes what the service runs. It identifies a service in two ways. + +First it matches the executable name from the service's binary path against the runtimes and databases it supports: + +- **Runtimes:** `java.exe`, `javaw.exe`, `node.exe`, `python.exe`, `pythonw.exe`, and `dotnet.exe`. +- **Databases:** `sqlservr.exe`, `postgres.exe`, and `mysqld.exe`. + +If the name says nothing, the agent reads the executable's PE headers and checks for a CLR header. This is how it finds a **self-contained .NET service**, published as its own executable and named after the application, such as `MyCompany.Worker.exe`. Nothing in that name identifies it as .NET, but the file itself does. The agent reads the file and never runs it. + +It also tells .NET Framework and .NET apart, by looking for a `runtimeconfig.json` file next to the executable. The two use different profilers, so the distinction decides how the service is instrumented. + +A service that matches neither check is left out, which keeps the several hundred built-in Windows services off the list. The agent also skips executables in Windows-owned directories such as `C:\Windows\System32`, since none of them is an application workload. + +IIS application pools are discovered separately, through `applicationHost.config`, and are not subject to any of this. ## ETW monitoring -Because Windows has no eBPF, the agent gives you equivalent zero-code monitoring through a custom Event Tracing for Windows (ETW) receiver. The receiver taps the HTTP.sys provider to emit server spans (at the request-path level) and Rate, Errors, and Duration (RED) metrics for all IIS traffic, with no injection. This covers workloads that cannot use the .NET profiler, including legacy .NET Framework 3.5 pools and non-.NET applications on IIS. The receiver reads events only and makes no changes to your applications. See [.NET](../../auto-instrumentation/dotnet/) for how the CLR profiler and ETW monitoring work together. +Because Windows has no eBPF, the agent gives you equivalent zero-code monitoring through a custom Event Tracing for Windows (ETW) receiver. The receiver taps the HTTP.sys provider to emit server spans (at the request-path level) and Rate, Errors, and Duration (RED) metrics for IIS traffic, with no injection. This covers workloads that cannot use the .NET profiler, including legacy .NET Framework 3.5 pools and non-.NET applications on IIS. The receiver reads events only and makes no changes to your applications. See [.NET](../../auto-instrumentation/dotnet/) for how the CLR profiler and ETW monitoring work together. + +ETW is opt-in, exactly like eBPF on Linux. The agent collects an application pool only after you turn tracing on for it, and it collects nothing at all while no pool is enabled. Turning a pool off stops both its spans and its RED metrics. ## Injection mechanisms @@ -38,18 +57,22 @@ See the [Application APM](../../auto-instrumentation/overview/) language pages f The agent runs as a Windows service under the **Local System** account, which gives it the access it needs to: -- Read IIS configuration and query the Service Control Manager and WMI. +- Read IIS configuration, query the Service Control Manager, and read the system TCP table to map ports to processes. - Write the registry and `applicationHost.config` entries for instrumentation, and recycle pools or restart services. - Register as an ETW consumer, which requires elevation. ## Automatic collection -Alongside application instrumentation, the agent automatically collects Windows host metrics and logs: performance counters, the Windows Event Log, IIS metrics, and Windows service status. See [Host metrics and logs](../../baseline/host-metrics-and-logs/). +The agent automatically collects Windows host metrics: CPU, memory, disk, and network usage. + +That is all it collects on its own. Traces, including the ETW monitoring described above, need you to turn tracing on for a service first. + +No logs are collected from the host automatically. The agent doesn't read the Windows Event Log unless you ask it to: to send a channel such as `System` or `Application`, add an `eventlog` source in [log monitoring](../../log-monitoring/#windows-event-log-channels). See [Host metrics and logs](../../baseline/host-metrics-and-logs/). ## Next steps <CardGrid> - <LinkCard title="Install on Windows" href="../../installation/windows-agent/" description="MSI installation and setup." /> + <LinkCard title="Install on Windows" href="../../installation/windows-agent/" description="Installer download and setup." /> <LinkCard title=".NET application APM" href="../../auto-instrumentation/dotnet/" description="Instrument .NET on IIS and Windows services." /> <LinkCard title="Host metrics and logs" href="../../baseline/host-metrics-and-logs/" description="Metrics and logs collected automatically on Windows." /> </CardGrid>