Name of the kubernetes cluster that will host this DSS instance (should generally describe the DSS instance being hosted)
Example: dss-che-1
map(map(string)))Kubernetes CPU and memory requests and limits of the core-service containers. +Leave empty to not set any.
+Example: { requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }
{}string)Size of the CockroachDB storage engine cache, passed to the --cache flag of cockroach start.
+Either a percentage of the memory available to the container or an absolute size.
Example: 25% or 2GiB
"25%"string)A string that specifies a CRDB cluster name. This is used together to ensure that all newly created @@ -148,6 +160,18 @@ From v.17, the recommended CockroachDB version is v24.1.3.
string)This variable has been renamed to locality and is left to warn users about migration.
""string)Maximum memory CockroachDB can use for SQL queries, passed to the --max-sql-memory flag of cockroach start.
+Either a percentage of the memory available to the container or an absolute size.
Example: 25% or 2GiB
"25%"map(map(string)))Kubernetes CPU and memory requests and limits of the CockroachDB containers. +Leave empty to not set any.
+Example: { requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }
{}number)Maximum number of open connections to the datastore.
diff --git a/deploy/infrastructure/modules/terraform-aws-dss/main.tf b/deploy/infrastructure/modules/terraform-aws-dss/main.tf index 23d577b1b..7da8e0d98 100644 --- a/deploy/infrastructure/modules/terraform-aws-dss/main.tf +++ b/deploy/infrastructure/modules/terraform-aws-dss/main.tf @@ -25,6 +25,10 @@ module "terraform-commons-dss" { app_hostname = var.app_hostname crdb_image_tag = var.crdb_image_tag crdb_cluster_name = var.crdb_cluster_name + crdb_cache = var.crdb_cache + crdb_max_sql_memory = var.crdb_max_sql_memory + crdb_resources = var.crdb_resources + core_service_resources = var.core_service_resources db_hostname_suffix = var.db_hostname_suffix datastore_type = var.datastore_type datastore_max_open_conns = var.datastore_max_open_conns diff --git a/deploy/infrastructure/modules/terraform-aws-dss/variables.gen.tf b/deploy/infrastructure/modules/terraform-aws-dss/variables.gen.tf index c20e043de..aab65c002 100644 --- a/deploy/infrastructure/modules/terraform-aws-dss/variables.gen.tf +++ b/deploy/infrastructure/modules/terraform-aws-dss/variables.gen.tf @@ -411,6 +411,68 @@ variable "crdb_cluster_name" { } +variable "crdb_cache" { + type = string + description = <<-EOT + Size of the CockroachDB storage engine cache, passed to the `--cache` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_max_sql_memory" { + type = string + description = <<-EOT + Maximum memory CockroachDB can use for SQL queries, passed to the `--max-sql-memory` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the CockroachDB containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.crdb_resources) : contains(["requests", "limits"], k)]) + error_message = "crdb_resources keys must be `requests` or `limits`." + } +} + + +variable "core_service_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the core-service containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.core_service_resources) : contains(["requests", "limits"], k)]) + error_message = "core_service_resources keys must be `requests` or `limits`." + } +} + + variable "locality" { type = string description = <<-EOT diff --git a/deploy/infrastructure/modules/terraform-google-dss/TFVARS.gen.md b/deploy/infrastructure/modules/terraform-google-dss/TFVARS.gen.md index e757c4e67..de72ba59a 100644 --- a/deploy/infrastructure/modules/terraform-google-dss/TFVARS.gen.md +++ b/deploy/infrastructure/modules/terraform-google-dss/TFVARS.gen.md @@ -76,6 +76,18 @@ Example:Name of the kubernetes cluster that will host this DSS instance (should generally describe the DSS instance being hosted)
Example: dss-che-1
map(map(string)))Kubernetes CPU and memory requests and limits of the core-service containers. +Leave empty to not set any.
+Example: { requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }
{}string)Size of the CockroachDB storage engine cache, passed to the --cache flag of cockroach start.
+Either a percentage of the memory available to the container or an absolute size.
Example: 25% or 2GiB
"25%"string)A string that specifies a CRDB cluster name. This is used together to ensure that all newly created @@ -105,6 +117,18 @@ From v.17, the recommended CockroachDB version is v24.1.3.
string)This variable has been renamed to locality and is left to warn users about migration.
""string)Maximum memory CockroachDB can use for SQL queries, passed to the --max-sql-memory flag of cockroach start.
+Either a percentage of the memory available to the container or an absolute size.
Example: 25% or 2GiB
"25%"map(map(string)))Kubernetes CPU and memory requests and limits of the CockroachDB containers. +Leave empty to not set any.
+Example: { requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }
{}number)Maximum number of open connections to the datastore.
diff --git a/deploy/infrastructure/modules/terraform-google-dss/main.tf b/deploy/infrastructure/modules/terraform-google-dss/main.tf index 8cf9088db..937581e06 100644 --- a/deploy/infrastructure/modules/terraform-google-dss/main.tf +++ b/deploy/infrastructure/modules/terraform-google-dss/main.tf @@ -25,6 +25,10 @@ module "terraform-commons-dss" { app_hostname = var.app_hostname crdb_image_tag = var.crdb_image_tag crdb_cluster_name = var.crdb_cluster_name + crdb_cache = var.crdb_cache + crdb_max_sql_memory = var.crdb_max_sql_memory + crdb_resources = var.crdb_resources + core_service_resources = var.core_service_resources db_hostname_suffix = var.db_hostname_suffix datastore_type = var.datastore_type datastore_max_open_conns = var.datastore_max_open_conns diff --git a/deploy/infrastructure/modules/terraform-google-dss/variables.gen.tf b/deploy/infrastructure/modules/terraform-google-dss/variables.gen.tf index ad306b72d..e3a46e319 100644 --- a/deploy/infrastructure/modules/terraform-google-dss/variables.gen.tf +++ b/deploy/infrastructure/modules/terraform-google-dss/variables.gen.tf @@ -400,6 +400,68 @@ variable "crdb_cluster_name" { } +variable "crdb_cache" { + type = string + description = <<-EOT + Size of the CockroachDB storage engine cache, passed to the `--cache` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_max_sql_memory" { + type = string + description = <<-EOT + Maximum memory CockroachDB can use for SQL queries, passed to the `--max-sql-memory` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the CockroachDB containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.crdb_resources) : contains(["requests", "limits"], k)]) + error_message = "crdb_resources keys must be `requests` or `limits`." + } +} + + +variable "core_service_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the core-service containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.core_service_resources) : contains(["requests", "limits"], k)]) + error_message = "core_service_resources keys must be `requests` or `limits`." + } +} + + variable "locality" { type = string description = <<-EOT diff --git a/deploy/infrastructure/utils/definitions/core_service_resources.tf b/deploy/infrastructure/utils/definitions/core_service_resources.tf new file mode 100644 index 000000000..ea9679cd8 --- /dev/null +++ b/deploy/infrastructure/utils/definitions/core_service_resources.tf @@ -0,0 +1,16 @@ +variable "core_service_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the core-service containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.core_service_resources) : contains(["requests", "limits"], k)]) + error_message = "core_service_resources keys must be `requests` or `limits`." + } +} diff --git a/deploy/infrastructure/utils/definitions/crdb_cache.tf b/deploy/infrastructure/utils/definitions/crdb_cache.tf new file mode 100644 index 000000000..6a733f8c4 --- /dev/null +++ b/deploy/infrastructure/utils/definitions/crdb_cache.tf @@ -0,0 +1,11 @@ +variable "crdb_cache" { + type = string + description = <<-EOT + Size of the CockroachDB storage engine cache, passed to the `--cache` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} diff --git a/deploy/infrastructure/utils/definitions/crdb_max_sql_memory.tf b/deploy/infrastructure/utils/definitions/crdb_max_sql_memory.tf new file mode 100644 index 000000000..64f04bc5c --- /dev/null +++ b/deploy/infrastructure/utils/definitions/crdb_max_sql_memory.tf @@ -0,0 +1,11 @@ +variable "crdb_max_sql_memory" { + type = string + description = <<-EOT + Maximum memory CockroachDB can use for SQL queries, passed to the `--max-sql-memory` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} diff --git a/deploy/infrastructure/utils/definitions/crdb_resources.tf b/deploy/infrastructure/utils/definitions/crdb_resources.tf new file mode 100644 index 000000000..fd2360d02 --- /dev/null +++ b/deploy/infrastructure/utils/definitions/crdb_resources.tf @@ -0,0 +1,16 @@ +variable "crdb_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the CockroachDB containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.crdb_resources) : contains(["requests", "limits"], k)]) + error_message = "crdb_resources keys must be `requests` or `limits`." + } +} diff --git a/deploy/infrastructure/utils/variables.py b/deploy/infrastructure/utils/variables.py index be8457e76..07357048e 100755 --- a/deploy/infrastructure/utils/variables.py +++ b/deploy/infrastructure/utils/variables.py @@ -48,6 +48,10 @@ "desired_aux_db_version", "crdb_image_tag", "crdb_cluster_name", + "crdb_cache", + "crdb_max_sql_memory", + "crdb_resources", + "core_service_resources", "locality", "datastore_max_open_conns", "crdb_external_nodes", diff --git a/deploy/operations/ci/aws-1/variables.gen.tf b/deploy/operations/ci/aws-1/variables.gen.tf index c20e043de..aab65c002 100644 --- a/deploy/operations/ci/aws-1/variables.gen.tf +++ b/deploy/operations/ci/aws-1/variables.gen.tf @@ -411,6 +411,68 @@ variable "crdb_cluster_name" { } +variable "crdb_cache" { + type = string + description = <<-EOT + Size of the CockroachDB storage engine cache, passed to the `--cache` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_max_sql_memory" { + type = string + description = <<-EOT + Maximum memory CockroachDB can use for SQL queries, passed to the `--max-sql-memory` flag of `cockroach start`. + Either a percentage of the memory available to the container or an absolute size. + + Example: `25%` or `2GiB` + EOT + + default = "25%" +} + + +variable "crdb_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the CockroachDB containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.crdb_resources) : contains(["requests", "limits"], k)]) + error_message = "crdb_resources keys must be `requests` or `limits`." + } +} + + +variable "core_service_resources" { + type = map(map(string)) + description = <<-EOT + Kubernetes CPU and memory requests and limits of the core-service containers. + Leave empty to not set any. + + Example: `{ requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }` + EOT + + default = {} + + validation { + condition = alltrue([for k in keys(var.core_service_resources) : contains(["requests", "limits"], k)]) + error_message = "core_service_resources keys must be `requests` or `limits`." + } +} + + variable "locality" { type = string description = <<-EOT diff --git a/deploy/services/helm-charts/dss/templates/dss-core-service.yaml b/deploy/services/helm-charts/dss/templates/dss-core-service.yaml index 9902a46ff..78a400908 100644 --- a/deploy/services/helm-charts/dss/templates/dss-core-service.yaml +++ b/deploy/services/helm-charts/dss/templates/dss-core-service.yaml @@ -92,6 +92,10 @@ spec: {{- end }} command: - core-service + {{- with $dss.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} image: {{ $dss.image }} imagePullPolicy: {{ $dss.imagePullPolicy | default "Always" }} name: core-service diff --git a/deploy/services/helm-charts/dss/values.schema.json b/deploy/services/helm-charts/dss/values.schema.json index 5e9fa6401..9841372d8 100644 --- a/deploy/services/helm-charts/dss/values.schema.json +++ b/deploy/services/helm-charts/dss/values.schema.json @@ -242,6 +242,10 @@ "type": "string", "description": "Image of the DSS. Please note that the usage of the `latest` tag is discouraged to prevent accidental upgrades in case of restart. Example: `docker.io/interuss/dss:v0.15.0`. Official image releases: https://hub.docker.com/r/interuss/dss/tags" }, + "resources": { + "type": "object", + "description": "Kubernetes CPU and memory requests and limits of the core-service containers. Example: `{requests: {cpu: '0.5', memory: 2Gi}, limits: {memory: 2Gi}}`" + }, "conf": { "type": "object", "properties": { diff --git a/deploy/services/helm-charts/dss/values.yaml b/deploy/services/helm-charts/dss/values.yaml index e9f72b9ad..0b2c7f924 100644 --- a/deploy/services/helm-charts/dss/values.yaml +++ b/deploy/services/helm-charts/dss/values.yaml @@ -1,6 +1,7 @@ # Default configuration dss: + resources: {} conf: enableScd: true evict: diff --git a/deploy/services/tanka/cockroachdb.libsonnet b/deploy/services/tanka/cockroachdb.libsonnet index c43f257cb..db87333ca 100644 --- a/deploy/services/tanka/cockroachdb.libsonnet +++ b/deploy/services/tanka/cockroachdb.libsonnet @@ -55,6 +55,7 @@ local volumes = import 'volumes.libsonnet'; value: 'kubernetes-multiregion', }, ], + resources: metadata.cockroach.resources, livenessProbe: { httpGet: { path: '/health', @@ -90,8 +91,8 @@ local volumes = import 'volumes.libsonnet'; locality: 'zone=' + metadata.locality, 'locality-advertise-addr': 'zone=' + metadata.locality + '@$(hostname -f)', 'http-addr': '0.0.0.0', - cache: '25%', - 'max-sql-memory': '25%', + cache: metadata.cockroach.cache, + 'max-sql-memory': metadata.cockroach.maxSqlMemory, } + if metadata.cockroach.clusterName != "" then { 'cluster-name': metadata.cockroach.clusterName, } else {}, diff --git a/deploy/services/tanka/core-service.libsonnet b/deploy/services/tanka/core-service.libsonnet index fd3bf56a5..820e488ad 100644 --- a/deploy/services/tanka/core-service.libsonnet +++ b/deploy/services/tanka/core-service.libsonnet @@ -118,6 +118,7 @@ local awsLoadBalancer(metadata) = base.AWSLoadBalancerWithManagedCert(metadata, }, ] else [], volumeMounts: volumes.all(metadata).backendMounts, + resources: metadata.backend.resources, command: ['core-service'], args_:: { addr: ':' + metadata.backend.port, diff --git a/deploy/services/tanka/metadata_base.libsonnet b/deploy/services/tanka/metadata_base.libsonnet index 4122b2423..775f05d60 100644 --- a/deploy/services/tanka/metadata_base.libsonnet +++ b/deploy/services/tanka/metadata_base.libsonnet @@ -25,6 +25,9 @@ JoinExisting: [], storageClass: 'standard', clusterName: '', + cache: '25%', // Value of the --cache flag of `cockroach start` + maxSqlMemory: '25%', // Value of the --max-sql-memory flag of `cockroach start` + resources: {}, // Kubernetes container resources, e.g. { requests: { cpu: '3.5', memory: '14Gi' }, limits: { cpu: '3.5', memory: '14Gi' } } }, yugabyte: { image: error 'must specify yugabyte db image', @@ -67,6 +70,7 @@ dumpRequests: false, certName: if $.cloud_provider == "aws" then error 'must specify certName for AWS cloud provider', # Only used by AWS sslPolicy: '', # SSL Policy Name. Only used by Google Cloud. + resources: {}, // Kubernetes container resources, e.g. { requests: { cpu: '0.5', memory: '2Gi' }, limits: { memory: '2Gi' } } }, alert: { enable: false, diff --git a/docs/operations/performance.md b/docs/operations/performance.md index 78a58b73c..2c31882b4 100644 --- a/docs/operations/performance.md +++ b/docs/operations/performance.md @@ -6,6 +6,57 @@ Overtime, old entries should be removed to ensure the system is not overloaded p See the detailed [section about cleanup](cleanup.md). +## CPU and memory allocation + +By default, the deployment tooling does not set any Kubernetes CPU or memory requests or limits on the CockroachDB and core-service containers. +Kubernetes may then place them on nodes without enough free memory and kill them when the node runs out of memory (see issue [#1731](https://github.com/interuss/dss/issues/1731)). + +The following settings are available: + +* CPU and memory requests and limits of the CockroachDB and core-service containers. +* CockroachDB `--cache` and `--max-sql-memory` flags (both `25%` by default). See [CockroachDB recommendations](https://www.cockroachlabs.com/docs/stable/recommended-production-settings#cache-and-sql-memory-size) before changing them. + +To set them: + +1. Set the values according to the deployment tool: + + === "Terraform" + Set the following variables (see `TFVARS.gen.md` for details): + + * `crdb_resources`, e.g. `{ requests = { cpu = "3.5", memory = "14Gi" }, limits = { cpu = "3.5", memory = "14Gi" } }` + * `crdb_cache` and `crdb_max_sql_memory`, e.g. `"25%"` + * `core_service_resources`, e.g. `{ requests = { cpu = "0.5", memory = "2Gi" }, limits = { memory = "2Gi" } }` + + Then run `terraform apply` to regenerate the Tanka and Helm configuration. + + === "Tanka" + Set the following fields of the metadata in your `main.jsonnet`: + + ```jsonnet + cockroach+: { + resources: { requests: { cpu: '3.5', memory: '14Gi' }, limits: { cpu: '3.5', memory: '14Gi' } }, + cache: '25%', + maxSqlMemory: '25%', + }, + backend+: { + resources: { requests: { cpu: '0.5', memory: '2Gi' }, limits: { memory: '2Gi' } }, + }, + ``` + + === "Helm" + Set the following values: + + ```yaml + cockroachdb: + conf: + cache: 25% + max-sql-memory: 25% + statefulset: + resources: {requests: {cpu: "3.5", memory: 14Gi}, limits: {cpu: "3.5", memory: 14Gi}} + dss: + resources: {requests: {cpu: "0.5", memory: 2Gi}, limits: {memory: 2Gi}} + ``` + ## The SCD global lock option !!! danger