From 62307a64584c946012cfac1b9b78dc98f9928f3c Mon Sep 17 00:00:00 2001 From: Denise Schlesinger Date: Tue, 11 Aug 2026 14:01:32 +0300 Subject: [PATCH] Doc denisa changes (#7575) * test Added introductory content to the add-cluster-connection article. * mv changes * links * mv changes * corrections * suggested changes applied * chnages * sagiv asks * changes by Shiri * Update data-explorer/kusto/management/materialized-views/materialized-views-optimization.md Co-authored-by: learn-build-service-prod-10[bot] <274431553+learn-build-service-prod-10[bot]@users.noreply.github.com> --------- Co-authored-by: learn-build-service-prod-10[bot] <274431553+learn-build-service-prod-10[bot]@users.noreply.github.com> --- .../materialized-view-alter-lookback.md | 2 + .../materialized-view-alter.md | 3 +- .../materialized-view-create-or-alter.md | 3 +- .../materialized-view-create.md | 89 +------ .../materialized-view-overview.md | 7 +- .../materialized-view-purge.md | 3 +- .../materialized-view-use-cases.md | 1 + .../materialized-views-limitations.md | 4 +- .../materialized-views-monitoring.md | 92 +++----- .../materialized-views-optimization.md | 220 ++++++++++++++++++ data-explorer/kusto/management/toc.yml | 2 + 11 files changed, 270 insertions(+), 156 deletions(-) create mode 100644 data-explorer/kusto/management/materialized-views/materialized-views-optimization.md diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-alter-lookback.md b/data-explorer/kusto/management/materialized-views/materialized-view-alter-lookback.md index 02bb98ec08..10d48e21a6 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-alter-lookback.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-alter-lookback.md @@ -69,5 +69,7 @@ The following example removes the lookback period of the materialized view, `MyV * [Materialized views lookback period](materialized-view-create.md#lookback-period) * [Materialized views](materialized-view-overview.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [.create materialized-view](materialized-view-create.md) * [.alter materialized-view](materialized-view-alter.md) + diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-alter.md b/data-explorer/kusto/management/materialized-views/materialized-view-alter.md index c6a220eab8..2f4c27ab7f 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-alter.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-alter.md @@ -83,5 +83,6 @@ The following command modifies the query definition of materialized view MyView: * [Materialized views](materialized-view-overview.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [.create materialized-view](materialized-view-create.md) -* [.create-or-alter materialized-view](materialized-view-create-or-alter.md) \ No newline at end of file +* [.create-or-alter materialized-view](materialized-view-create-or-alter.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-create-or-alter.md b/data-explorer/kusto/management/materialized-views/materialized-view-create-or-alter.md index b434049f93..5cc373720c 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-create-or-alter.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-create-or-alter.md @@ -88,5 +88,6 @@ For more information, see the [Query parameter](materialized-view-create.md#quer * [Materialized views](materialized-view-overview.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [.create materialized-view](materialized-view-create.md) -* [.alter materialized-view](materialized-view-alter.md) \ No newline at end of file +* [.alter materialized-view](materialized-view-alter.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-create.md b/data-explorer/kusto/management/materialized-views/materialized-view-create.md index 37def73c9a..ba1703ff4b 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-create.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-create.md @@ -284,92 +284,6 @@ The following aggregation functions are supported: * [`percentile`, `percentiles`](../../query/percentiles-aggregation-function.md) * [`tdigest`](../../query/tdigest-aggregation-function.md) -### Performance tips - -* **Use a datetime group-by key**: Materialized views that have a `datetime` column as one of their group-by keys is more efficient than those that don't. The reason is that some optimizations can be applied only when there's a datetime group-by key. If adding a datetime group-by key doesn't change the semantics of your aggregation, we recommend that you add it. You can do this only if the `datetime` column is *immutable* for each unique entity. - - For example, in the following aggregation: - - ```kusto - SourceTable | summarize take_any(*) by EventId - ``` - - If `EventId` always has the same `Timestamp` value, and therefore adding `Timestamp` doesn't change the semantics of the aggregation, it's better to define the view as: - - ```kusto - SourceTable | summarize take_any(*) by EventId, Timestamp - ``` - - > [!TIP] - > Late-arriving data in a datetime group-by key can have a negative impact on the materialized view's performance. For example, assume that a materialized view uses `bin(Timestamp, 1d)` as one of its group-by keys, and newly ingested records to the source table have old `Timestamp` values. These records might negatively affect the materialized view. - > - > If you expect late arriving records ingested to the source table, adjust the caching policy of the materialized view accordingly. For example, if records with Timestamp of six months ago are expected to be ingested to the source table, the materialization process needs to scan the materialized view for the previous six months. If this period is in cold cache, materialization experiences cache misses which have a negative impact on the performance of the view. - > - > If such late arriving records aren't expected, we recommend that in the materialized view query. Either filter these records out or normalize their timestamp values to the current time. - -* **Define a lookback period**: If applicable to your scenario, adding a `lookback` property can significantly improve query performance. For details, see [Lookback period](#lookback-period). - -* **Add columns frequently used for filtering as group-by keys**: Materialized view queries are optimized when they're filtered by one of the materialized view's group-by keys. If you know that your query pattern will often filter by a column that's immutable according to a unique entity in the materialized view, include it in the materialized view's group-by keys. - - For example, a materialized view exposes `arg_max` by a `ResourceId` value that is often filtered by `SubscriptionId`. Assuming that a `ResourceId` value always belongs to the same `SubscriptionId` value, define the materialized view query as: - - ```kusto - .create materialized-view ArgMaxResourceId on table FactResources - { - FactResources | summarize arg_max(Timestamp, *) by SubscriptionId, ResourceId - } - ``` - - The preceding definition is preferable over the following: - - ```kusto - .create materialized-view ArgMaxResourceId on table FactResources - { - FactResources | summarize arg_max(Timestamp, *) by ResourceId - } - ``` - -* **Use update policies where appropriate**: The materialized view can include transformations, normalizations, and lookups in dimension tables. However, we recommend that you move these operations to an [update policy](../update-policy.md). Leave only the aggregation for the materialized view. - - For example, it's better to define the following update policy: - - ```kusto - .alter-merge table Target policy update - @'[{"IsEnabled": true, - "Source": "SourceTable", - "Query": - "SourceTable - | extend ResourceId = strcat('subscriptions/', toupper(SubscriptionId), '/', resourceId)", - | lookup DimResources on ResourceId - | mv-expand Events - "IsTransactional": false}]' - ``` - - And define the following materialized view: - - ```kusto - .create materialized-view Usage on table Events - { - Target - | summarize count() by ResourceId - } - ``` - - The alternative, of including the update policy as part of the materialized view query, might perform worse and therefore not recommended: - - ```kusto - .create materialized-view Usage on table SourceTable - { - SourceTable - | extend ResourceId = strcat('subscriptions/', toupper(SubscriptionId), '/', resourceId) - | lookup DimResources on ResourceId - | mv-expand Events - | summarize count() by ResourceId - } - ``` - -> [!TIP] -> If you require the best query time performance, but you can tolerate some data latency, use the [materialized_view() function](../../query/materialized-view-function.md). ### Backfill a materialized view @@ -555,6 +469,7 @@ If the cancellation isn't finished within 10 minutes, `CancellationState` indica * [Materialized views](materialized-view-overview.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [.alter materialized-view](materialized-view-alter.md) * [.drop materialized-view](materialized-view-drop.md) -* [.show materialized-view(s)](materialized-view-show-command.md) \ No newline at end of file +* [.show materialized-view(s)](materialized-view-show-command.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-overview.md b/data-explorer/kusto/management/materialized-views/materialized-view-overview.md index ea630baa42..133c4728b0 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-overview.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-overview.md @@ -106,7 +106,7 @@ There are 2 ways to query a materialized view: When querying the entire view, the materialized part is combined with the `delta` during query time. This includes aggregating the `delta` and joining it with the materialized part. -* Querying the entire view performs better if the query includes filters on the group by keys of the materialized view query. See more tips about how to create your materialized view, based on your query pattern, in the [`.create materialized-view` performance tips](materialized-view-create.md#performance-tips) section. +* Querying the entire view performs better if the query includes filters on the group by keys of the materialized view query. See more tips about how to create your materialized view, based on your query pattern, in [Materialized views optimization](materialized-views-optimization.md). * The query optimizer chooses summarize/join strategies that are expected to improve query performance. For example, the decision on whether to [shuffle](../../query/shuffle-query.md) the query is based on number of records in `delta` part. The following [client request properties](../../api/rest/request-properties.md) provide some control over the optimizations applied. You can test these properties with your materialized view queries and evaluate their impact on queries performance. |Client request property name|Type|Description| @@ -157,7 +157,7 @@ When querying the materialized part of the view, the return value depends on the The main contributors that can impact a materialized view health are: :::moniker range="azure-data-explorer" -* **Cluster resources:** Like any other process running on the cluster, materialized views consume resources (CPU, memory) from the cluster. If the cluster is overloaded, adding materialized views to it may cause a degradation in the cluster's performance. Monitor your cluster's health using [cluster health metrics](/azure/data-explorer/using-metrics#cluster-metrics). [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) currently doesn't take materialized views health under consideration as part of autoscale rules. +* **Cluster resources:** Like any other process running on the cluster, materialized views consume resources (CPU, memory) from the cluster. If the cluster is overloaded, adding materialized views to it may cause a degradation in the cluster's performance. Monitor your cluster's health using [cluster health metrics](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters). [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) currently doesn't take materialized views health under consideration as part of autoscale rules. * The [materialization process](#how-materialized-views-work) is limited by the amount of memory and CPU it can consume. These limits are defined, and can be changed, in the [materialized views workload group](../workload-groups.md#materialized-views-workload-group). ::: moniker-end @@ -169,7 +169,7 @@ The main contributors that can impact a materialized view health are: * **Number of materialized views in cluster:** The above considerations apply to each individual materialized view defined in the cluster. Each view consumes its own resources, and many views compete with each other on available resources. While there are no hard-coded limits to the number of materialized views in a cluster, the cluster may not be able to handle all materialized views, when there are many defined. The [capacity policy](../capacity-policy.md#materialized-views-capacity-policy) can be adjusted if there is more than a single materialized view in the cluster. Increase the value of `ClusterMinimumConcurrentOperations` in the policy to run more materialized views concurrently. ::: moniker-end -* **Materialized view definition**: The materialized view definition must be defined according to query best practices for best query performance. For more information, see [create command performance tips](materialized-view-create.md#performance-tips). +* **Materialized view definition**: The materialized view definition must be defined according to query best practices for best query performance. For more information, see [create command performance tips](materialized-views-optimization.md). ## Materialized view over materialized view @@ -183,6 +183,7 @@ A materialized view can be created over another materialized view if the source * [Materialized views policies](materialized-view-policies.md) * [Materialized views limitations and known issues](materialized-views-limitations.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [Monitor materialized views](materialized-views-monitoring.md) * [`.create materialized view`](materialized-view-create.md) * [`.alter materialized-view`](materialized-view-alter.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-purge.md b/data-explorer/kusto/management/materialized-views/materialized-view-purge.md index 5ea54b9712..0595ad9042 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-purge.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-purge.md @@ -45,4 +45,5 @@ MV | where avg_Duration > 1h ## Related content -* [Materialized views](materialized-view-overview.md) \ No newline at end of file +* [Materialized views](materialized-view-overview.md) +* [Materialized views optimization](materialized-views-optimization.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-view-use-cases.md b/data-explorer/kusto/management/materialized-views/materialized-view-use-cases.md index 9ff9c8a65c..3f1709828b 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-view-use-cases.md +++ b/data-explorer/kusto/management/materialized-views/materialized-view-use-cases.md @@ -139,6 +139,7 @@ Materialized views and update policies work differently and serve different use * [Materialized views policies](materialized-view-policies.md) * [Materialized views limitations and known issues](materialized-views-limitations.md) * [Monitor materialized views](materialized-views-monitoring.md) +* [Materialized views optimization](materialized-views-optimization.md) * [`.create materialized view`](materialized-view-create.md) * [`.alter materialized-view`](materialized-view-alter.md) * [`{.disable | .enable} materialized-view`](materialized-view-enable-disable.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-views-limitations.md b/data-explorer/kusto/management/materialized-views/materialized-views-limitations.md index d08c0a4c4d..0943e560c3 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-views-limitations.md +++ b/data-explorer/kusto/management/materialized-views/materialized-views-limitations.md @@ -23,7 +23,6 @@ ms.date: 06/03/2026 * The source table of a materialized view can't be a table with a [restricted view access policy](../restricted-view-access-policy.md). * A materialized view can't be created on top of another materialized view, unless the first materialized view is of type `take_any(*)` aggregation. See [materialized view over materialized view](materialized-view-overview.md#materialized-view-over-materialized-view). * Materialized views can't be defined over [external tables](../../query/schema-entities/external-tables.md). -* Materialized views cannot be queried with [cursors](../../query/database-cursor.md) > [!WARNING] > @@ -58,11 +57,12 @@ ms.date: 06/03/2026 ## Other -* [Cursor functions](../database-cursor.md#cursor-functions) can't be used on top of materialized views. +* [Cursor functions](../../query/database-cursor.md#cursor-functions) can't be used on top of materialized views. * Continuous export from a materialized view isn't supported. ## Related content * [Materialized views](materialized-view-overview.md) * [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) * [Monitor materialized views](materialized-views-monitoring.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md b/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md index 506052acb0..7ec9311999 100644 --- a/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md +++ b/data-explorer/kusto/management/materialized-views/materialized-views-monitoring.md @@ -2,29 +2,29 @@ title: Monitor materialized views description: This article describes how to monitor materialized views. ms.reviewer: yifats -ms.topic: reference +ms.topic: how-to ms.date: 02/04/2025 --- # Monitor materialized views > [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)] -Monitor the materialized view's health in the following ways: +Monitor the health of materialized views in the following ways: ::: moniker range="azure-data-explorer" -* Monitor [materialized views metrics](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) in the [Azure portal](https://portal.azure.com/) with [Azure Monitor](/azure/data-explorer/monitor-data-explorer-reference#metrics). Use the materialized view age metric, `MaterializedViewAgeSeconds`, as the primary metric to monitor the freshness of the view. +* Monitor [materialized views metrics](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) in the [Azure portal](https://portal.azure.com/) by using [Azure Monitor](/azure/data-explorer/monitor-data-explorer-reference#metrics). Use the materialized view age metric, `MaterializedViewAgeSeconds`, as the primary metric to monitor the freshness of the view. ::: moniker-end :::moniker range="microsoft-fabric" -* Monitor [materialized view metrics](/fabric/real-time-intelligence/monitor-metrics#metric-specific-dimension-column) in your Microsoft Fabric workspace. Use the materialized view age metric, `MaterializedViewAgeSeconds` as the primary metric to monitor the freshness of the view. For more information, see [Enable monitoring in your workspace](/fabric/get-started/enable-workspace-monitoring). +* Monitor [materialized view metrics](/fabric/real-time-intelligence/monitor-metrics#metric-specific-dimension-column) in your Microsoft Fabric workspace. Use the materialized view age metric, `MaterializedViewAgeSeconds`, as the primary metric to monitor the freshness of the view. For more information, see [Enable monitoring in your workspace](/fabric/get-started/enable-workspace-monitoring). ::: moniker-end -* Monitor the `IsHealthy` property using [`.show materialized-view`](materialized-view-show-command.md#show-materialized-views). +* Monitor the `IsHealthy` property by using [`.show materialized-view`](materialized-view-show-command.md#show-materialized-views). -* Check for failures using [`.show materialized-view failures`](materialized-view-show-failures-command.md#show-materialized-view-failures). +* Check for failures by using [`.show materialized-view failures`](materialized-view-show-failures-command.md#show-materialized-view-failures). > [!NOTE] > -> Materialization never skips any data, even if there are constant failures. The view is always guaranteed to return the most up-to-date snapshot of the query, based on all records in the source table. Constant failures significantly degrade query performance, but don't cause incorrect results in view queries. +> Materialization never skips any data, even if there are constant failures. The view always returns the most up-to-date snapshot of the query, based on all records in the source table. Constant failures significantly degrade query performance, but don't cause incorrect results in view queries. ## Troubleshooting unhealthy materialized views @@ -46,42 +46,22 @@ If the `MaterializedViewAge` metric constantly increases, and the `MaterializedV |---|---|---| |MaterializedView|1|0| - * The number of materialized views that can run concurrently depends on the capacity shown in the `Total` column, while the `Consumed` column shows the number of materialized views currently running. You can use the [Materialized views capacity policy](../capacity-policy.md#materialized-views-capacity-policy) to specify the minimum and maximum number of concurrent operations, overriding the system's default concurrency level. The system determines the current concurrency, shown in `Total`, based on the cluster's available resources. The following example overrides the system's decision and changes the minimum concurrent operations from one to three: - - ```kusto - .alter-merge cluster policy capacity '{ "MaterializedViewsCapacity": { "ClusterMinimumConcurrentOperations": 3 } }' - ``` - - * If you explicitly change this policy, monitor the cluster's health and ensure that other workloads aren't affected by this change. + * The number of materialized views that can run concurrently depends on the capacity shown in the `Total` column. The `Consumed` column shows the number of materialized views currently running. If concurrency is limiting materialization, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). ::: moniker-end -* Check if there are failures during the materialization process using [.show materialized-view failures](materialized-view-show-failures-command.md#show-materialized-view-failures). - * If the error is permanent, the system automatically disables the materialized view. To check if it's disabled, use the [.show materialized-view](materialized-view-show-command.md) command and see if the value in the `IsEnabled` column is `false`. Then check the [Journal](../journal.md) for the disabled event with the [.show journal](../journal.md#show-journal) command. +* Check if there are failures during the materialization process by using [.show materialized-view failures](materialized-view-show-failures-command.md#show-materialized-view-failures). + * If the error is permanent, the system automatically disables the materialized view. To check if it's disabled, use the [.show materialized-view](materialized-view-show-command.md) command and see if the value in the `IsEnabled` column is `false`. Then check the [Journal](../journal.md) for the disabled event by using the [.show journal](../journal.md#show-journal) command. An example of a permanent failure is a source table schema change that makes it incompatible with the materialized view. For more information, see [.create materialized-view command](materialized-view-create.md#supported-properties). * If the failure is transient, the system automatically retries the operation. However, the failure can delay the materialization and increase the age of the materialized view. This type of failure occurs, for example, when hitting memory limits or with a query time-out. See the following recommendations for more ways to troubleshoot transient failures. -* Analyze the materialization process using the [.show commands-and-queries](../commands-and-queries.md) command. Replace *Databasename* and *ViewName* to filter for a specific view: +* Analyze the materialization process by using the [.show commands-and-queries](../commands-and-queries.md) command. Replace *Databasename* and *ViewName* to filter for a specific view: ```kusto .show commands-and-queries | where Database == "DatabaseName" and ClientActivityId startswith "DN.MaterializedViews;ViewName;" ``` - * Check the memory consumption in the `MemoryPeak` column to identify any operations that failed due to hitting memory limits, such as, [runaway queries](../../concepts/runaway-queries.md). By default, the materialization process is limited to a 15-GB memory peak per node. If the queries or commands executed during the materialization process exceed this value, the materialization fails due to memory limits. To increase the memory peak per node, alter the [$materialized-views workload group](../workload-groups.md#materialized-views-workload-group). The following example alters the materialized views workload group to use a maximum of 64-GB memory peak per node during materialization: - - ```kusto - .alter-merge workload_group ['$materialized-views'] ``` - { - "RequestLimitsPolicy": { - "MaxMemoryPerQueryPerNode": { - "Value": 68718888960 - } - } - } - ``` - - > [!NOTE] - > `MaxMemoryPerQueryPerNode` can't exceed 50% of the total memory available on each node. + * Check the memory consumption in the `MemoryPeak` column to identify operations that failed because they reached memory limits, such as [runaway queries](../../concepts/runaway-queries.md). For remediation, see [Increase the materialization memory limit](materialized-views-optimization.md#increase-the-materialization-memory-limit). * Check if the materialization process is hitting cold cache. The following example shows cache statistics over the past day for the materialized view, `ViewName`: @@ -109,73 +89,63 @@ If the `MaterializedViewAge` metric constantly increases, and the `MaterializedV |---|---|---|---|---|---| |26 GB|0 Bytes|0 Bytes|1 GB|0 Bytes|866 MB| - * If the view isn’t fully in the hot cache, materialization can experience disk misses, significantly slowing down the process. + * If the view isn't fully in the hot cache, materialization can experience disk misses, significantly slowing down the process. - * Increasing the caching policy for the materialized view helps avoid cache misses. For more information, see [hot and cold cache and caching policy](../cache-policy.md) and [.alter materialized-view policy caching command](../alter-materialized-view-cache-policy-command.md). - * Check if the materialization is scanning old records by checking the `ScannedExtentsStatistics` with the [.show queries](../show-queries-command.md) command. If the number of scanned extents is high and the `MinDataScannedTime` is old, the materialization cycle needs to scan all, or most, of the *materialized* part of the view. The scan is needed to find intersections with the *delta*. For more information about the *delta* and the *materialized* part, see [How materialized views work](materialized-view-overview.md#how-materialized-views-work). The following recommendations provide ways to reduce the amount of data scanned in materialized cycles by minimizing the intersection with the *delta*. - -* If the materialization cycle scans a large amount of data, potentially including cold cache, consider making the following changes to the materialized view definition: - * Include a `datetime` group-by key in the view definition. This can significantly reduce the amount of data scanned, **as long as there is no late arriving data in this column**. For more information, see [Performance tips](materialized-view-create.md#performance-tips). You need to create a new materialized view since updates to group-by keys aren't supported. - * Use a `lookback` as part of the view definition. For more information, see [.create materialized view supported properties](materialized-view-create.md#supported-properties). + * If cache misses are slowing materialization, see [Adjust caching policies](materialized-views-optimization.md#adjust-caching-policies). + * Check if the materialization is scanning old records by checking `ScannedExtentsStatistics` with the [.show queries](../show-queries-command.md) command. If the number of scanned extents is high and `MinDataScannedTime` is old, the cycle scans all or most of the *materialized* part to find intersections with the *delta*. For ways to reduce the amount of data scanned, see [Optimize materialized views](materialized-views-optimization.md). :::moniker range="azure-data-explorer" -* Check whether there's enough ingestion capacity by verifying if either the [`MaterializedViewResult` metric](#materializedviewresult-metric) or [IngestionUtilization metric](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) show `InsufficientCapacity` values. You can increase ingestion capacity by scaling the available resources (preferred) or by altering the [ingestion capacity policy](../capacity-policy.md#ingestion-capacity). +* Check whether there's enough ingestion capacity by verifying if either the [`MaterializedViewResult` metric](#materializedviewresult-metric) or [IngestionUtilization metric](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) show `InsufficientCapacity` values. These values indicate that available ingestion capacity is limiting materialization. For remediation, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). ::: moniker-end :::moniker range="microsoft-fabric" -* Check whether there's enough ingestion capacity by verifying if the [`MaterializedViewResult` metric](#materializedviewresult-metric) shows `InsufficientCapacity` values. You can increase ingestion capacity by scaling the available resources. +* Check whether there's enough ingestion capacity by verifying if the [`MaterializedViewResult` metric](#materializedviewresult-metric) shows `InsufficientCapacity` values. These values indicate that available ingestion capacity is limiting materialization. For remediation, see [Increase available resources](materialized-views-optimization.md#increase-available-resources). ::: moniker-end -* If the materialized view is still unhealthy, then the service doesn't have sufficient capacity or resources to materialize all the data on time. Consider the following options: - :::moniker range="azure-data-explorer" - * Scale out the cluster by increasing the minimum instance count. [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) doesn't take materialized views into consideration and doesn't scale out the cluster automatically if materialized views are unhealthy. You need to set the minimum instance count to provide the cluster with more resources to accommodate materialized views. - ::: moniker-end - :::moniker range="microsoft-fabric" - * Scale out the Eventhouse to provide it with more resources to accommodate materialized views. For more information, see [Enable minimum consumption](/fabric/real-time-intelligence/manage-monitor-eventhouse#enable-minimum-consumption). - ::: moniker-end - * Divide the materialized view into several smaller views, each covering a subset of the data. For instance, you can split them based on a high cardinality key from the materialized view's group-by keys. All views are based on the same source table, and each view filters by `SourceTable | where hash(key, number_of_views) == i`, where `i` is part of the set `{0,1,…,number_of_views-1}`. Then, you can define a [stored function](../../query/schema-entities/stored-functions.md) that [unions](../../query/union-operator.md) all the smaller materialized views. Use this function in queries to access the combined data. - - While splitting the view might increase CPU usage, it reduces the memory peak in materialization cycles. Reducing the memory peak can help if the single view is failing due to memory limits. +* If the materialized view is still unhealthy, the service might not have sufficient capacity or resources to materialize all data on time. See [Optimize materialized views](materialized-views-optimization.md) for remediation options. ## MaterializedViewResult metric -The `MaterializedViewResult` metric provides information about the result of a materialization cycle and can be used to identify issues in the materialized view health status. The metric includes the `Database` and `MaterializedViewName` and a `Result` dimension. +The `MaterializedViewResult` metric provides information about the result of a materialization cycle. Use it to identify problems in the materialized view health status. The metric includes the `Database`, `MaterializedViewName`, and a `Result` dimension. The `Result` dimension can have one of the following values: * **Success**: The materialization completed successfully. -* **SourceTableNotFound**: The source table of the materialized view was dropped, so the materialized view is disabled automatically. +* **SourceTableNotFound**: The source table of the materialized view was dropped, so the materialized view is automatically disabled. -* **SourceTableSchemaChange**: The schema of the source table changed in a way that isn’t compatible with the materialized view definition. Since the materialized view query no longer matches the materialized view schema, the materialized view is disabled automatically. +* **SourceTableSchemaChange**: The schema of the source table changed in a way that isn't compatible with the materialized view definition. Since the materialized view query no longer matches the materialized view schema, the materialized view is automatically disabled. :::moniker range="azure-data-explorer" -* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of [ingestion capacity](../capacity-policy.md#ingestion-capacity). While insufficient capacity failures can be transient, if they reoccur often, try scaling out the instance or increasing the relevant capacity in the policy. +* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of [ingestion capacity](../capacity-policy.md#ingestion-capacity). Insufficient capacity failures can be transient, but recurring failures indicate that available ingestion capacity is limiting materialization. ::: moniker-end :::moniker range="microsoft-fabric" -* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of ingestion capacity. While insufficient capacity failures can be transient, if they reoccur often, try scaling out the instance or increasing capacity. For more information, see [Plan your capacity size](/fabric/enterprise/plan-capacity). +* **InsufficientCapacity**: The instance doesn't have sufficient capacity to materialize the materialized view, due to a lack of ingestion capacity. Insufficient capacity failures can be transient, but recurring failures indicate that available ingestion capacity is limiting materialization. ::: moniker-end -* **InsufficientResources:** The database doesn't have sufficient resources (CPU/memory) to materialize the materialized view. While insufficient resource errors might be transient, if they reoccur often, try scaling up or scaling out. For more ideas, see [Troubleshooting unhealthy materialized views](#troubleshooting-unhealthy-materialized-views). +* **InsufficientResources:** The database doesn't have sufficient resources (memory) to materialize the materialized view. Insufficient resource errors can be transient, but recurring failures indicate that the database lacks sufficient memory for materialization. For remediation, see [Optimize materialized views](materialized-views-optimization.md). ## Materialized views in follower databases -Materialized views can be defined in [follower databases](materialized-views-limitations.md#follower-databases). However, the monitoring of these materialized views should be based on the leader database, where the materialized view is defined. Specifically: +You can define materialized views in [follower databases](materialized-views-limitations.md#follower-databases). However, you should monitor these materialized views from the leader database, where you define the materialized view. Specifically: ::: moniker range="azure-data-explorer" -* [Metrics](/azure/data-explorer/using-metrics#materialized-view-metrics) related to materialized view execution (`MaterializedViewResult`, `MaterializedViewExtentsRebuild`) are only present in the leader database. Metrics related to monitoring (`MaterializedViewAgeSeconds`, `MaterializedViewHealth`, `MaterializedViewRecordsInDelta`) also appear in the follower databases. +* [Metrics](/azure/data-explorer/monitor-data-explorer-reference#supported-metrics-for-microsoftkustoclusters) related to materialized view execution (`MaterializedViewResult`, `MaterializedViewExtentsRebuild`) are only present in the leader database. Metrics related to monitoring (`MaterializedViewAgeSeconds`, `MaterializedViewHealth`, `MaterializedViewRecordsInDelta`) also appear in the follower databases. ::: moniker-end * The [.show materialized-view failures command](materialized-view-show-failures-command.md) only works in the leader database. ## Track resource consumption -**Materialized views resource consumption:** the resources consumed by the materialized views materialization process can be tracked using the [`.show commands-and-queries`](../commands-and-queries.md) command. Filter the records for a specific view using the following (replace `DatabaseName` and `ViewName`): +**Materialized views resource consumption:** Use the [`.show commands-and-queries`](../commands-and-queries.md) command to track the resources the materialized views materialization process consumes. To filter the records for a specific view, use the following query and replace `DatabaseName` and `ViewName` with your values: ```kusto .show commands-and-queries | where Database == "DatabaseName" and ClientActivityId startswith "DN.MaterializedViews;ViewName;" +| project StartedOn, LastUpdatedOn, Duration, State, FailureReason, + TotalCpu, MemoryPeak, CacheStatistics, ScannedExtentsStatistics ``` ## Related content * [Materialized views](materialized-view-overview.md) -* [Materialized views use cases](materialized-view-use-cases.md) \ No newline at end of file +* [Materialized views use cases](materialized-view-use-cases.md) +* [Materialized views optimization](materialized-views-optimization.md) diff --git a/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md b/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md new file mode 100644 index 0000000000..d59e4d9fbd --- /dev/null +++ b/data-explorer/kusto/management/materialized-views/materialized-views-optimization.md @@ -0,0 +1,220 @@ +--- +title: Optimize materialized views +description: Learn how to optimize materialized view materialization and query performance. +ms.reviewer: denises +ms.topic: how-to +ms.date: 08/11/2026 +--- + +# Optimize materialized views + +> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)] + +A materialized view consists of a materialized part and newly ingested source records that aren't yet materialized, called the *delta*. Materialization performance depends largely on the overlap between the delta and the materialized part. Query performance also depends on combining these parts when you query the entire view. For more information, see [How materialized views work](materialized-view-overview.md#how-materialized-views-work). + +Use the following optimizations after you [monitor the materialized view](materialized-views-monitoring.md) and identify the cause of the performance issue. + +## Increase the materialization memory limit + +By default, the `$materialized-views` workload group limits the memory peak of each materialization operation to 15 GB per node. If materialization fails because it reaches this limit, increase `MaxMemoryPerQueryPerNode` in the workload group: + +```kusto +.alter-merge workload_group ['$materialized-views'] ``` +{ + "RequestLimitsPolicy": { + "MaxMemoryPerQueryPerNode": { + "Value": 34359738368 + } + } +} +``` + +The preceding example increases the limit to 32 GB per node. `MaxMemoryPerQueryPerNode` can't exceed 50% of the memory available on each node. Monitor other workloads after increasing the limit. For more information, see [Materialized views workload group](../workload-groups.md#materialized-views-workload-group). + +## Adjust caching policies + +Materialization can slow down when it scans data that isn't in the hot cache. Set a caching policy that covers the period the materialization process is expected to scan. For commands, see [Caching policy management commands](../show-table-cache-policy-command.md). + +The materialized view's caching policy applies only to its materialized part. The source table supplies the delta and also participates in queries, so configure the source table's caching policy for the required period as well. For more information, see [Retention and caching policy](materialized-view-policies.md#retention-and-caching-policy). + +## Use a datetime group-by key + +Materialized views that include a `datetime` column as a group-by key can reduce the amount of materialized data scanned during each cycle. Add a datetime group-by key only when it doesn't change the aggregation semantics and the value is immutable for each unique entity. + +For example, if each `EventId` always has the same `Timestamp` value, change: + +```kusto +SourceTable | summarize take_any(*) by EventId +``` + +to: + +```kusto +SourceTable | summarize take_any(*) by EventId, Timestamp +``` + +> [!TIP] +> Late-arriving data in a datetime group-by key can negatively affect materialization performance. If late-arriving records are expected, configure the caching policies to cover the oldest expected values. If they aren't expected, filter out those records or normalize their timestamp values in the materialized view query. + +### Add an immutable creation-time group-by key + +When most updates apply to recently created entities, add an immutable creation-time column to the group-by keys. The materialization process uses the first datetime group-by key, calculates its minimum value over the delta, and uses that value as the lower boundary of the join with the materialized part. + +This optimization isn't the `lookback` property. A group-by key changes the aggregation granularity, while an incorrectly configured `lookback` can produce duplicate records. + +Only the first datetime group-by key is used for this optimization, so the order of datetime group-by keys determines which one applies. If the selected column contains a null value in the delta, the optimization is silently skipped for the entire materialization cycle. Don't use this optimization when the creation time isn't included in every update or when adding it changes the aggregation semantics. + +For example, consider ticket bookings that can be changed for up to two years, although most changes occur within two months of booking. If every update includes an immutable `Booking_CreationTimestamp`, change: + +```kusto +summarize arg_max(Timestamp, *) by BookingId +``` + +to: + +```kusto +summarize arg_max(Timestamp, *) by BookingId, Booking_CreationTimestamp +``` + +The materialization process can then use the earliest `Booking_CreationTimestamp` value in the delta as the lower boundary for the join instead of scanning the entire two-year period. + +## Define a lookback period + +The `lookback` property limits how much of the materialized part each materialization cycle scans. It improves materialization performance, not query performance. + +Set the lookback long enough to include all expected duplicates or updates. A lookback that's too short can produce duplicate records. For configuration details and limitations, see [Lookback period](materialized-view-create.md#lookback-period). + +## Add frequently filtered columns as group-by keys + +Queries are optimized when they filter by a materialized view group-by key. If queries frequently filter by a column that's immutable for each unique entity, include that column in the group-by keys. + +For example, if a `ResourceId` always belongs to the same `SubscriptionId`, define the materialized view as: + +```kusto +.create materialized-view ArgMaxResourceId on table FactResources +{ + FactResources + | summarize arg_max(Timestamp, *) by SubscriptionId, ResourceId +} +``` + +This definition is preferable to grouping only by `ResourceId` when queries commonly filter by `SubscriptionId`. + +## Move nonaggregation work out of the view + +If the query only needs a lookup against a dimension table, use the `dimensionTables` property. For more information, see [Query parameter](materialized-view-create.md#query-parameter). + +For other transformations and normalizations, use an [update policy](../update-policy.md) to prepare the data in a target table and leave only the aggregation in the materialized view. For example, define an update policy: + +```kusto +.alter-merge table Target policy update +@'[{"IsEnabled":true,"Source":"SourceTable","Query":"SourceTable | extend NormalizedResourceId = toupper(ResourceId)","IsTransactional":false,"PropagateIngestionProperties":false}]' +``` + +Then define the materialized view over the prepared table: + +```kusto +.create materialized-view Usage on table Target +{ + Target + | summarize count() by NormalizedResourceId +} +``` + +Including the transformation directly in the materialized view query requires it to run during each materialization cycle and might perform worse. + +## Apply a partitioning policy + +Consider a [partitioning policy](materialized-view-policies.md#partitioning-policy) when most queries filter by one of the materialized view's group-by keys. This optimization is common for multitenant data where a group-by key identifies the tenant. + +Partitioning keeps a single materialized view and can avoid splitting the data across multiple views. However, it increases the number of extents and creates more work for the materialization process. + +## Increase available resources + +::: moniker range="azure-data-explorer" +If the cluster doesn't have enough resources to keep materialized views healthy, increase the minimum instance count. [Optimized autoscale](/azure/data-explorer/manage-cluster-horizontal-scaling#optimized-autoscale-recommended-option) doesn't consider materialized view health when making scaling decisions. +::: moniker-end + +::: moniker range="microsoft-fabric" +If the Eventhouse doesn't have enough resources to keep materialized views healthy, enable minimum consumption to provide more resources. For more information, see [Enable minimum consumption](/fabric/real-time-intelligence/manage-monitor-eventhouse#enable-minimum-consumption). +::: moniker-end + +When multiple materialized views need to run concurrently, ensure that the materialized views capacity policy allows sufficient concurrency. Increase `ClusterMinimumConcurrentOperations` only after evaluating the effect on other workloads. For more information, see [Materialized views capacity policy](../capacity-policy.md#materialized-views-capacity-policy). + +## Split into multiple materialized views + +Splitting can help when a single materialized view's materialization cycle needs too much memory and the preceding optimizations aren't sufficient. The split must reduce the number of groups and the amount of state each materialized view handles. Don't divide the KQL into processing stages. + +While splitting might increase CPU usage, it reduces the memory peak in materialization cycles. Ensure that `ClusterMinimumConcurrentOperations` allows the split views to run concurrently. Otherwise, the views can run serially and lose the benefit of the split. + +### Horizontally shard by a stable key + +Suppose the original materialized view is: + +```kusto +.create materialized-view UsageMV on table Events +{ + Events + | summarize EventCount = count(), TotalBytes = sum(Bytes) + by TenantId, Day = bin(Timestamp, 1d) +} +``` + +Split tenants deterministically across four materialized views: + +```kusto +.create materialized-view UsageMV_0 on table Events +{ + Events + | where hash(TenantId, 4) == 0 + | summarize EventCount = count(), TotalBytes = sum(Bytes) + by TenantId, Day = bin(Timestamp, 1d) +} +``` + +Create equivalent materialized views for buckets 1, 2, and 3, and expose them through a function: + +```kusto +.create-or-alter function Usage() +{ + union UsageMV_0, UsageMV_1, UsageMV_2, UsageMV_3 +} +``` + +Because each group belongs to exactly one materialized view, no final reaggregation is required. + +### Split by a natural business partition + +This method is usually clearer than hashing: + +```kusto +.create materialized-view Usage_EU on table Events +{ + Events + | where Region == "EU" + | summarize count(), sum(Bytes) + by Region, TenantId, Day = bin(Timestamp, 1d) +} +``` + +Create corresponding views for the other regions. This method works best when: + +* The partition expression is deterministic. +* The partition key is part of the group-by keys. +* Distribution is reasonably balanced. +* Queries commonly filter by the partition key. + +Avoid random splitting or splitting on a value that can change for the same logical group. + +## Optimize queries + +If you can tolerate some data latency, use the [`materialized_view()` function](../../query/materialized-view-function.md) to query only the materialized part. This approach avoids combining the materialized part with the delta at query time. + +When querying the entire view, filter by group-by keys when possible. You can also test the `materialized_view_shuffle` client request property to control the shuffle strategy for the summarize and join operations. Specify keys to get behavior similar to `hint.shufflekey`, or omit the keys to get behavior similar to `hint.strategy=shuffle`. For details and examples, see [Materialized view query optimizer](materialized-view-overview.md#materialized-view-query-optimizer). + +## Related content + +* [Materialized views](materialized-view-overview.md) +* [Materialized views use cases](materialized-view-use-cases.md) +* [Monitor materialized views](materialized-views-monitoring.md) +* [Materialized views policies](materialized-view-policies.md) diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml index 46526f1783..d1217cbc04 100644 --- a/data-explorer/kusto/management/toc.yml +++ b/data-explorer/kusto/management/toc.yml @@ -172,6 +172,8 @@ items: href: materialized-views/materialized-view-policies.md - name: Materialized views use cases href: materialized-views/materialized-view-use-cases.md + - name: Materialized views optimization + href: materialized-views/materialized-views-optimization.md - name: Materialized views data purge href: materialized-views/materialized-view-purge.md - name: Monitor materialized views