diff --git a/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md b/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md index 7ec9311999..ff7a1840be 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md +++ b/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md @@ -20,6 +20,8 @@ Monitor the health of materialized views in the following ways: ::: moniker-end * Monitor the `IsHealthy` property by using [`.show materialized-view`](materialized-view-show-command.md#show-materialized-views). +* Inspect extent, hot-cache, size, and effective policy details by using [`.show materialized-view details`](materialized-view-show-details-command.md). + * Check for failures by using [`.show materialized-view failures`](materialized-view-show-failures-command.md#show-materialized-view-failures). > [!NOTE] @@ -28,101 +30,105 @@ Monitor the health of materialized views in the following ways: ## Troubleshooting unhealthy materialized views -If the `MaterializedViewAge` metric constantly increases, and the `MaterializedViewHealth` metric shows that the view is unhealthy, follow these recommendations to identify the root cause: +If the `MaterializedViewAgeSeconds` metric constantly increases and the `MaterializedViewHealth` metric shows that the view is unhealthy, use the following symptoms to identify the root cause. :::moniker range="azure-data-explorer" -* Check the number of materialized views on the cluster, and the current capacity for materialized views: +### Not enough materialization concurrency - ```kusto - .show capacity - | where Resource == "MaterializedView" - | project Resource, Total, Consumed - ``` +Use [`.show capacity`](../show-capacity-command.md) to compare the number of materialized views that can run concurrently with the number currently running: - **Output** +```kusto +.show capacity +| where Resource == "MaterializedView" +| project Resource, Total, Consumed +``` - |Resource|Total|Consumed| - |---|---|---| - |MaterializedView|1|0| +| Resource | Total | Consumed | +|---|---|---| +| MaterializedView | 1 | 0 | - * The number of materialized views that can run concurrently depends on the capacity shown in the `Total` column. The `Consumed` column shows the number of materialized views currently running. If concurrency is limiting materialization, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). +`Total` is the current concurrency limit, and `Consumed` is the number of materialized views currently running. If the limit is delaying materialization, increase the materialization concurrency only after evaluating the effect on other workloads. For more information, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). ::: moniker-end -* Check if there are failures during the materialization process by using [.show materialized-view failures](materialized-view-show-failures-command.md#show-materialized-view-failures). - * If the error is permanent, the system automatically disables the materialized view. To check if it's disabled, use the [.show materialized-view](materialized-view-show-command.md) command and see if the value in the `IsEnabled` column is `false`. Then check the [Journal](../journal.md) for the disabled event by using the [.show journal](../journal.md#show-journal) command. - An example of a permanent failure is a source table schema change that makes it incompatible with the materialized view. For more information, see [.create materialized-view command](materialized-view-create.md#supported-properties). - * If the failure is transient, the system automatically retries the operation. However, the failure can delay the materialization and increase the age of the materialized view. This type of failure occurs, for example, when hitting memory limits or with a query time-out. See the following recommendations for more ways to troubleshoot transient failures. - -* Analyze the materialization process by using the [.show commands-and-queries](../commands-and-queries.md) command. Replace *Databasename* and *ViewName* to filter for a specific view: - - ```kusto - .show commands-and-queries - | where Database == "DatabaseName" and ClientActivityId startswith "DN.MaterializedViews;ViewName;" - ``` - - * Check the memory consumption in the `MemoryPeak` column to identify operations that failed because they reached memory limits, such as [runaway queries](../../concepts/runaway-queries.md). For remediation, see [Increase the materialization memory limit](materialized-views-optimization.md#increase-the-materialization-memory-limit). - - * Check if the materialization process is hitting cold cache. The following example shows cache statistics over the past day for the materialized view, `ViewName`: - - ```kusto - .show commands-and-queries - | where ClientActivityId startswith "DN.MaterializedViews;ViewName" - | where StartedOn > ago(1d) - | extend HotCacheHits = tolong(CacheStatistics.Shards.Hot.HitBytes), - HotCacheMisses = tolong(CacheStatistics.Shards.Hot.MissBytes), - HotCacheRetrieved = tolong(CacheStatistics.Shards.Hot.RetrieveBytes), - ColdCacheHits = tolong(CacheStatistics.Shards.Cold.HitBytes), - ColdCacheMisses = tolong(CacheStatistics.Shards.Cold.MissBytes), - ColdCacheRetrieved = tolong(CacheStatistics.Shards.Cold.RetrieveBytes) - | summarize HotCacheHits = format_bytes(sum(HotCacheHits)), - HotCacheMisses = format_bytes(sum(HotCacheMisses)), - HotCacheRetrieved = format_bytes(sum(HotCacheRetrieved)), - ColdCacheHits =format_bytes(sum(ColdCacheHits)), - ColdCacheMisses = format_bytes(sum(ColdCacheMisses)), - ColdCacheRetrieved = format_bytes(sum(ColdCacheRetrieved)) - ``` - - **Output** - - |HotCacheHits|HotCacheMisses|HotCacheRetrieved|ColdCacheHits|ColdCacheMisses|ColdCacheRetrieved| - |---|---|---|---|---|---| - |26 GB|0 Bytes|0 Bytes|1 GB|0 Bytes|866 MB| - - * If the view isn't fully in the hot cache, materialization can experience disk misses, significantly slowing down the process. - - * If cache misses are slowing materialization, see [Adjust caching policies](materialized-views-optimization.md#adjust-caching-policies). - * Check if the materialization is scanning old records by checking `ScannedExtentsStatistics` with the [.show queries](../show-queries-command.md) command. If the number of scanned extents is high and `MinDataScannedTime` is old, the cycle scans all or most of the *materialized* part to find intersections with the *delta*. For ways to reduce the amount of data scanned, see [Optimize materialized views](materialized-views-optimization.md). -:::moniker range="azure-data-explorer" +### Materialization failures -* Check whether there's enough ingestion capacity by verifying if either the [`MaterializedViewResult` metric](#materializedviewresult-metric) or [IngestionUtilization metric](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) show `InsufficientCapacity` values. These values indicate that available ingestion capacity is limiting materialization. For remediation, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). -::: moniker-end -:::moniker range="microsoft-fabric" +Use [`.show materialized-view failures`](materialized-view-show-failures-command.md#show-materialized-view-failures) to inspect failures. -* Check whether there's enough ingestion capacity by verifying if the [`MaterializedViewResult` metric](#materializedviewresult-metric) shows `InsufficientCapacity` values. These values indicate that available ingestion capacity is limiting materialization. For remediation, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). -::: moniker-end +* For a permanent error, the system automatically disables the materialized view. Use [`.show materialized-view`](materialized-view-show-command.md) to check whether `IsEnabled` is `false`, and use [`.show journal`](../journal.md#show-journal) to find the disabled event. A source table schema change that makes the table incompatible with the materialized view is an example of a permanent error. For more information, see [.create materialized-view](materialized-view-create.md#supported-properties). +* For a transient error, the system automatically retries the operation. Repeated memory-limit or query-timeout failures delay materialization and increase the materialized view age. For ways to reduce transient failures, see [Optimize materialized views](materialized-views-optimization.md). -* If the materialized view is still unhealthy, the service might not have sufficient capacity or resources to materialize all data on time. See [Optimize materialized views](materialized-views-optimization.md) for remediation options. +### Materialization exceeds the memory limit -## MaterializedViewResult metric +Use [`.show commands-and-queries`](../commands-and-queries.md) to inspect `MemoryPeak`. Replace `DatabaseName` and `ViewName` with your values: + +```kusto +.show commands-and-queries +| where Database == "DatabaseName" and ClientActivityId startswith "DN.MaterializedViews;ViewName;" +| project StartedOn, LastUpdatedOn, Duration, State, FailureReason, + TotalCpu, MemoryPeak +``` + +A failure that reaches the memory limit can appear as a [runaway query](../../concepts/runaway-queries.md). Raise the materialization memory limit or reduce the memory required by each cycle. For more information, see [Increase the materialization memory limit](materialized-views-optimization.md#increase-the-materialization-memory-limit). + +### Materialization is hitting cold cache + +Use the cache statistics from [`.show commands-and-queries`](../commands-and-queries.md) to determine whether materialization is retrieving data from the cold cache. The following query summarizes cache activity over the past day for `ViewName`: -The `MaterializedViewResult` metric provides information about the result of a materialization cycle. Use it to identify problems in the materialized view health status. The metric includes the `Database`, `MaterializedViewName`, and a `Result` dimension. +```kusto +.show commands-and-queries +| where ClientActivityId startswith "DN.MaterializedViews;ViewName" +| where StartedOn > ago(1d) +| extend HotCacheHits = tolong(CacheStatistics.Shards.Hot.HitBytes), + HotCacheMisses = tolong(CacheStatistics.Shards.Hot.MissBytes), + HotCacheRetrieved = tolong(CacheStatistics.Shards.Hot.RetrieveBytes), + ColdCacheHits = tolong(CacheStatistics.Shards.Cold.HitBytes), + ColdCacheMisses = tolong(CacheStatistics.Shards.Cold.MissBytes), + ColdCacheRetrieved = tolong(CacheStatistics.Shards.Cold.RetrieveBytes) +| summarize HotCacheHits = format_bytes(sum(HotCacheHits)), + HotCacheMisses = format_bytes(sum(HotCacheMisses)), + HotCacheRetrieved = format_bytes(sum(HotCacheRetrieved)), + ColdCacheHits = format_bytes(sum(ColdCacheHits)), + ColdCacheMisses = format_bytes(sum(ColdCacheMisses)), + ColdCacheRetrieved = format_bytes(sum(ColdCacheRetrieved)) +``` + +| HotCacheHits | HotCacheMisses | HotCacheRetrieved | ColdCacheHits | ColdCacheMisses | ColdCacheRetrieved | +|---|---|---|---|---|---| +| 26 GB | 0 Bytes | 0 Bytes | 1 GB | 0 Bytes | 866 MB | + +Cold-cache hits or retrieved bytes indicate that materialization is reading data outside the hot cache, which can significantly slow the process. Extend the caching policies to cover the data that materialization scans. For more information, see [Adjust caching policies](materialized-views-optimization.md#adjust-caching-policies). -The `Result` dimension can have one of the following values: +### Materialization scans old records -* **Success**: The materialization completed successfully. +Inspect `ScannedExtentsStatistics` with [`.show queries`](../show-queries-command.md). A high number of scanned extents and an old `MinDataScannedTime` indicate that the cycle is scanning all or most of the materialized part to find intersections with the delta. Reduce the scan by using an appropriate datetime group-by key, lookback period, or caching policy. For more information, see [Optimize materialized views](materialized-views-optimization.md). -* **SourceTableNotFound**: The source table of the materialized view was dropped, so the materialized view is automatically disabled. +### Not enough ingestion capacity -* **SourceTableSchemaChange**: The schema of the source table changed in a way that isn't compatible with the materialized view definition. Since the materialized view query no longer matches the materialized view schema, the materialized view is automatically disabled. :::moniker range="azure-data-explorer" -* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of [ingestion capacity](../capacity-policy.md#ingestion-capacity). Insufficient capacity failures can be transient, but recurring failures indicate that available ingestion capacity is limiting materialization. +Check whether the [`MaterializedViewResult` metric](#materializedviewresult-metric) or [IngestionUtilization metric](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) has an `InsufficientCapacity` value. ::: moniker-end :::moniker range="microsoft-fabric" -* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of ingestion capacity. Insufficient capacity failures can be transient, but recurring failures indicate that available ingestion capacity is limiting materialization. +Check whether the [`MaterializedViewResult` metric](#materializedviewresult-metric) has an `InsufficientCapacity` value. ::: moniker-end -* **InsufficientResources:** The database doesn't have sufficient resources (memory) to materialize the materialized view. Insufficient resource errors can be transient, but recurring failures indicate that the database lacks sufficient memory for materialization. For remediation, see [Optimize materialized views](materialized-views-optimization.md). +Recurring `InsufficientCapacity` values indicate that available ingestion capacity is limiting materialization. Increase the resources available for materialization. For more information, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). + +### Materialized view remains unhealthy + +If the preceding diagnostics don't identify a specific cause, the service might not have sufficient capacity or resources to materialize all data on time. Increase available resources or split a memory-intensive view only after applying less disruptive optimizations. For more information, see [Optimize materialized views](materialized-views-optimization.md). + +## MaterializedViewResult metric + +The `MaterializedViewResult` metric provides the result of each materialization cycle. Use it to identify problems with materialized view health. The metric includes the `Database`, `MaterializedViewName`, and `Result` dimensions. + +| Value | Meaning | Action | +|---|---|---| +| `Success` | The materialization cycle completed successfully. | None. | +| `SourceTableNotFound` | The source table was dropped, so the materialized view is automatically disabled. | Restore the source table and [enable the materialized view](materialized-view-enable-disable.md). | +| `SourceTableSchemaChange` | The source table schema is incompatible with the materialized view definition, so the view is automatically disabled. | Make the source schema and materialized view query compatible, and then [enable the materialized view](materialized-view-enable-disable.md). | +| `InsufficientCapacity` | Available ingestion capacity is limiting materialization. The failure can be transient, but recurring values indicate a capacity issue. For Azure Data Explorer, see [Ingestion capacity](../capacity-policy.md#ingestion-capacity). | [Increase available resources](materialized-views-optimization.md#increase-available-resources). | +| `InsufficientResources` | Materialization exceeded the memory limit allowed for a single operation. | [Increase the materialization memory limit](materialized-views-optimization.md#increase-the-materialization-memory-limit) or reduce the memory required by each cycle. | ## Materialized views in follower databases @@ -133,16 +139,6 @@ You can define materialized views in [follower databases](materialized-views-lim ::: moniker-end * The [.show materialized-view failures command](materialized-view-show-failures-command.md) only works in the leader database. -## Track resource consumption - -**Materialized views resource consumption:** Use the [`.show commands-and-queries`](../commands-and-queries.md) command to track the resources the materialized views materialization process consumes. To filter the records for a specific view, use the following query and replace `DatabaseName` and `ViewName` with your values: - -```kusto -.show commands-and-queries -| where Database == "DatabaseName" and ClientActivityId startswith "DN.MaterializedViews;ViewName;" -| project StartedOn, LastUpdatedOn, Duration, State, FailureReason, - TotalCpu, MemoryPeak, CacheStatistics, ScannedExtentsStatistics -``` ## Related content diff --git a/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md b/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md index d59e4d9fbd..2626b302fd 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md +++ b/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md @@ -16,9 +16,9 @@ Use the following optimizations after you [monitor the materialized view](materi ## Increase the materialization memory limit -By default, the `$materialized-views` workload group limits the memory peak of each materialization operation to 15 GB per node. If materialization fails because it reaches this limit, increase `MaxMemoryPerQueryPerNode` in the workload group: +By default, the `$materialized-views` workload group limits the memory peak of each materialization operation to 15 GB per node, or 50% of the node's total physical memory, whichever is lower. If materialization fails because it reaches this limit, increase `MaxMemoryPerQueryPerNode` in the workload group: -```kusto +````kusto .alter-merge workload_group ['$materialized-views'] ``` { "RequestLimitsPolicy": { @@ -28,12 +28,16 @@ By default, the `$materialized-views` workload group limits the memory peak of e } } ``` +```` + +The preceding example increases the limit to 32 GB per node. Monitor other workloads after increasing the limit. For more information, see [Materialized views workload group](../workload-groups.md#materialized-views-workload-group). -The preceding example increases the limit to 32 GB per node. `MaxMemoryPerQueryPerNode` can't exceed 50% of the memory available on each node. Monitor other workloads after increasing the limit. For more information, see [Materialized views workload group](../workload-groups.md#materialized-views-workload-group). +> [!NOTE] +> `MaxMemoryPerQueryPerNode` can't exceed 50% of the total memory available on each node. A command that specifies a higher value fails validation. ## Adjust caching policies -Materialization can slow down when it scans data that isn't in the hot cache. Set a caching policy that covers the period the materialization process is expected to scan. For commands, see [Caching policy management commands](../show-table-cache-policy-command.md). +Materialization can slow down when it scans data that isn't in the hot cache. Set a caching policy that covers the period the materialization process is expected to scan. For more information, see [Caching policy](../cache-policy.md), [.alter materialized-view policy caching](../alter-materialized-view-cache-policy-command.md), and [.alter table policy caching](../alter-table-cache-policy-command.md). The materialized view's caching policy applies only to its materialized part. The source table supplies the delta and also participates in queries, so configure the source table's caching policy for the required period as well. For more information, see [Retention and caching policy](materialized-view-policies.md#retention-and-caching-policy). @@ -41,6 +45,9 @@ The materialized view's caching policy applies only to its materialized part. Th Materialized views that include a `datetime` column as a group-by key can reduce the amount of materialized data scanned during each cycle. Add a datetime group-by key only when it doesn't change the aggregation semantics and the value is immutable for each unique entity. +> [!NOTE] +> You can't change the group-by keys of an existing materialized view. To apply this optimization, create a new materialized view. + For example, if each `EventId` always has the same `Timestamp` value, change: ```kusto @@ -54,16 +61,22 @@ SourceTable | summarize take_any(*) by EventId, Timestamp ``` > [!TIP] -> Late-arriving data in a datetime group-by key can negatively affect materialization performance. If late-arriving records are expected, configure the caching policies to cover the oldest expected values. If they aren't expected, filter out those records or normalize their timestamp values in the materialized view query. +> Late-arriving data in a datetime group-by key can negatively affect materialization performance. For example, if a view groups by `bin(Timestamp, 1d)` and receives a record with a `Timestamp` from six months ago, the materialization process might need to scan the previous six months of the materialized part. If that period is in the cold cache, cache misses further slow materialization. +> +> If late-arriving records are expected, configure the materialized view and source table caching policies to cover the oldest expected values. If they aren't expected, filter out those records or normalize their timestamp values to the current time in the materialized view query. ### Add an immutable creation-time group-by key When most updates apply to recently created entities, add an immutable creation-time column to the group-by keys. The materialization process uses the first datetime group-by key, calculates its minimum value over the delta, and uses that value as the lower boundary of the join with the materialized part. +This change requires a new materialized view because you can't change the group-by keys of an existing view. + This optimization isn't the `lookback` property. A group-by key changes the aggregation granularity, while an incorrectly configured `lookback` can produce duplicate records. Only the first datetime group-by key is used for this optimization, so the order of datetime group-by keys determines which one applies. If the selected column contains a null value in the delta, the optimization is silently skipped for the entire materialization cycle. Don't use this optimization when the creation time isn't included in every update or when adding it changes the aggregation semantics. +Late-arriving records with old creation-time values can also reduce or eliminate the benefit for a cycle. A single old value moves the join's lower boundary backward, potentially causing the cycle to scan a much larger portion of the materialized part without producing an error. + For example, consider ticket bookings that can be changed for up to two years, although most changes occur within two months of booking. If every update includes an immutable `Booking_CreationTimestamp`, change: ```kusto @@ -88,6 +101,8 @@ Set the lookback long enough to include all expected duplicates or updates. A lo Queries are optimized when they filter by a materialized view group-by key. If queries frequently filter by a column that's immutable for each unique entity, include that column in the group-by keys. +You can't change the group-by keys of an existing materialized view. To apply this optimization, create a new materialized view. + For example, if a `ResourceId` always belongs to the same `SubscriptionId`, define the materialized view as: ```kusto @@ -132,7 +147,7 @@ Partitioning keeps a single materialized view and can avoid splitting the data a ## Increase available resources ::: moniker range="azure-data-explorer" -If the cluster doesn't have enough resources to keep materialized views healthy, increase the minimum instance count. [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) doesn't consider materialized view health when making scaling decisions. +If the cluster doesn't have enough resources to keep materialized views healthy, increase the minimum instance count. [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) doesn't consider materialized view health when making scaling decisions. If the `MaterializedViewResult` metric reports `InsufficientCapacity`, scale out the cluster (preferred) or adjust the [ingestion capacity policy](../capacity-policy.md#ingestion-capacity). ::: moniker-end ::: moniker range="microsoft-fabric" @@ -141,12 +156,28 @@ If the Eventhouse doesn't have enough resources to keep materialized views healt When multiple materialized views need to run concurrently, ensure that the materialized views capacity policy allows sufficient concurrency. Increase `ClusterMinimumConcurrentOperations` only after evaluating the effect on other workloads. For more information, see [Materialized views capacity policy](../capacity-policy.md#materialized-views-capacity-policy). +::: moniker range="azure-data-explorer" +For example, the following command sets the minimum number of concurrent materialization operations to three: + +```kusto +.alter-merge cluster policy capacity '{ "MaterializedViewsCapacity": { "ClusterMinimumConcurrentOperations": 3 } }' +``` +::: moniker-end + ## Split into multiple materialized views Splitting can help when a single materialized view's materialization cycle needs too much memory and the preceding optimizations aren't sufficient. The split must reduce the number of groups and the amount of state each materialized view handles. Don't divide the KQL into processing stages. While splitting might increase CPU usage, it reduces the memory peak in materialization cycles. Ensure that `ClusterMinimumConcurrentOperations` allows the split views to run concurrently. Otherwise, the views can run serially and lose the benefit of the split. +For either splitting method, ensure that: + +* The partition expression is deterministic and remains unchanged for the life of the views. +* The partition key is part of the group-by keys. +* Distribution is reasonably balanced. + +Avoid random splitting or splitting on a value that can change for the same logical group. + ### Horizontally shard by a stable key Suppose the original materialized view is: @@ -160,13 +191,13 @@ Suppose the original materialized view is: } ``` -Split tenants deterministically across four materialized views: +Split tenants deterministically across four materialized views by using [`hash_xxhash64()`](../../query/hash-xxhash64-function.md): ```kusto .create materialized-view UsageMV_0 on table Events { Events - | where hash(TenantId, 4) == 0 + | where hash_xxhash64(TenantId, 4) == 0 | summarize EventCount = count(), TotalBytes = sum(Bytes) by TenantId, Day = bin(Timestamp, 1d) } @@ -181,7 +212,7 @@ Create equivalent materialized views for buckets 1, 2, and 3, and expose them th } ``` -Because each group belongs to exactly one materialized view, no final reaggregation is required. +Because each group belongs to exactly one materialized view, no final reaggregation is required. The hash function and bucket count must remain fixed for the life of the views. Changing either can assign an existing group to a different view, so you must recreate and backfill all the shard views. ### Split by a natural business partition @@ -197,14 +228,7 @@ This method is usually clearer than hashing: } ``` -Create corresponding views for the other regions. This method works best when: - -* The partition expression is deterministic. -* The partition key is part of the group-by keys. -* Distribution is reasonably balanced. -* Queries commonly filter by the partition key. - -Avoid random splitting or splitting on a value that can change for the same logical group. +Create corresponding views for the other regions. This method works best when queries commonly filter by the partition key. ## Optimize queries