diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index 7eb02ac8..61d49c21 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -89,7 +89,8 @@ gcloud auth login && gcloud auth application-default login gcloud config set project gcloud services enable container.googleapis.com sqladmin.googleapis.com servicenetworking.googleapis.com \ redis.googleapis.com pubsub.googleapis.com secretmanager.googleapis.com dns.googleapis.com \ - artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com + artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com \ + cloudresourcemanager.googleapis.com gcloud container clusters get-credentials --location # add --internal-ip for a private endpoint kubectl auth can-i '*' '*' --all-namespaces gcloud container clusters describe --location \ @@ -1259,6 +1260,21 @@ gcloud container clusters create agent-relay --location $R --num-nodes 1 \ --enable-shielded-nodes --enable-dataplane-v2 ``` +**`Operation denied by org policy ... default Compute Engine service +account`?** Your organization does not allow nodes to run as the default +service account. Create one for the nodes and pass it: + +```sh +gcloud iam service-accounts create agent-relay-nodes +gcloud projects add-iam-policy-binding $P --role roles/container.defaultNodeServiceAccount \ + --member "serviceAccount:agent-relay-nodes@$P.iam.gserviceaccount.com" +``` + +then add `--service-account agent-relay-nodes@$P.iam.gserviceaccount.com` to +the `clusters create` command, and in step 4 set +`gke_node_service_account = "agent-relay-nodes@.iam.gserviceaccount.com"` +so the image mirror grants it read access. + A regional cluster with `--num-nodes 1` gets one node in each of the region's three zones. Then continue at step 1 with `` = `agent-relay` and `` = `agent-relay-vpc`, and grant the node service account in diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 05cad492..c9055d05 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -78,9 +78,13 @@ when you run them. edge_security_policy = { create = true } ``` - Run `terraform apply`. `terraform output edge` prints its name - (`-edge`). This policy has the OWASP rules in **preview** (they log - and do not block) and a limit of 60 sign-in requests a minute per client. + Run `terraform plan` first and read `rule_counts` in the `edge` output: + one policy holds at most 20 rules with an expression, and the plan tells + you what this policy needs (the defaults need 3). Then `terraform apply`, + and continue only if it finishes without an error. `terraform output edge` + prints the policy's name (`-edge`). This policy has the OWASP rules + in **preview** (they log and do not block) and a limit of 60 sign-in + requests a minute per client. 2. Name it in `values.yaml`, and bind the dashboard Service to its BackendConfig (`-dashboard`): @@ -108,6 +112,27 @@ gcloud compute backend-services list --format='table(name,securityPolicy.basenam Every BackendConfig shows the policy, and the count equals the number of backend services of the Ingress (6 for the default install). +That list can be complete several minutes before every backend enforces the +policy (4 minutes when measured). If your policy refuses some requests (an +allowlist, Cloudflare, client rules), confirm with a request that it must +refuse, **on each backend**, before you rely on it. Each line below reaches a +different backend service: + +```sh +IP=$(gcloud compute addresses describe $STATIC_IP_NAME --global --format='value(address)') +for url in "$DOMAIN/" "api.$DOMAIN/readyz" "cast.$DOMAIN/health" \ + "history.$DOMAIN/healthz" "history.$DOMAIN/v1/delivery/limits" "history.$DOMAIN/"; do + h=${url%%/*} + curl -sk -o /dev/null -w "$url %{http_code}\n" --resolve "$h:443:$IP" "https://$url" +done +``` + +The six are the dashboard, the API, Relaycast, the history reader, the +history upload service and the deny service. From an address the policy does +not admit, every line must end in `403`. Repeat until they all do. Warnings +that mention `Translate` in `kubectl describe ingress` while the policy +attaches are transient. + **Turn the OWASP rules on.** Session transcripts and messages contain code, SQL and shell text, which is what those rules look for. Use the install normally for a week, then look for requests the rules would have blocked: @@ -183,8 +208,8 @@ Cloudflare can reach it. the address it received the request from, and its own. Measured on a live GKE install, the header is `, ` and the right value is 2. Cloudflare adds one more, so behind Cloudflare the header is - `, , ` and the value is 3. The - Cloudflare case is expected, not yet measured; check it as below. + `, , ` and the value is 3 (also + measured). Check yours as below. ```yaml relay-control: @@ -218,9 +243,10 @@ Cloudflare can reach it. ```hcl edge_security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + connecting_asns = [13335] # Cloudflare's network; see below + rate_limits = [] client_ip_rules = [ { name = "vendors", hosts = ["history."], path_prefixes = ["/v1/delivery", "/v1/turns", "/v1/auth/token/refresh"], cidrs = ["", ""], rate_limit = { count = 600, interval_sec = 60 } }, { name = "sign-in", path_prefixes = ["/login", "/api/v1/auth", "/cloud/api/v1/auth"], cidrs = [""], rate_limit = { count = 60, interval_sec = 60 } }, @@ -229,14 +255,59 @@ Cloudflare can reach it. } ``` + `connecting_asns` matters here. A request that one of these rules admits + is not checked against Cloudflare's address ranges afterwards, so without + it anyone who learned the secret header could connect straight to the + load balancer and claim to be any client. With it, a connection that does + not come from Cloudflare's network is refused before any of these rules. + Terraform refuses `client_ip_rules` without it. + + After applying it, wait until it is enforced before you rely on it: a new + rule took almost 4 minutes to reach every backend when measured, and until + then a direct request with the header was still admitted. Repeat the loop + from section 2 from a machine that is not behind Cloudflare, adding the + secret header **and** a client address your staff rule admits: + + ```sh + -H "x-origin-auth: " -H "CF-Connecting-IP: " + ``` + + until all six lines end in `403`. Both headers matter: without the + claimed address the client rules themselves answer 403, which would look + like success while the bypass is still open. + + One gap this cannot close: another Cloudflare customer who learned your + secret header could send requests through their own Cloudflare zone, which + arrive from the same network. Keep the header value secret, rotate it if + it may have leaked (item 3), and treat it like a password. + The first rule that matches a request's path decides it: its listed clients are admitted, everyone else gets 403. The last rule closes every other path. A request that a rule admits is not checked by any later rule, so each rule carries its own `rate_limit` (per client), and the sign-in limit is a rule of its own ahead of the closing one. The module's - README (`terraform/modules/gcp/edge-security`) explains the limits. A - vendor that publishes hundreds of ranges needs a higher Cloud Armor quota, - or that list enforced as a Cloudflare WAF custom rule instead. + README (`terraform/modules/gcp/edge-security`) explains the limits. + + **A long vendor list does not fit.** One Cloud Armor policy holds at most + 20 rules with an expression, and Google does not raise that limit. After + the secret header, the two WAF rules and the sign-in and staff rules, + about 13 are left: roughly 100 vendor ranges. Terraform refuses a plan + that needs more and says how many it counted (`rule_counts` in the `edge` + output shows the same). For a vendor that publishes more, merge adjacent + ranges where you accept the wider block, or enforce that vendor's list as + a Cloudflare WAF custom rule and keep only your own ranges here. + + The project also has a limit of 20 such rules across all its policies + (`SECURITY_POLICY_CEVAL_RULES`). That one can be raised, and a request was + granted in seconds when measured. If another policy in the project already + uses some, the apply fails part-way with + `Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally`. + + **If an apply fails, do not use the policy yet.** It then holds an + arbitrary part of its rules, for example the rule that denies the sign-in + paths without the rule that admits you. Fix the cause and run + `terraform apply` again until it exits without an error. Only then set + `gkeIngress.securityPolicy`. **Done when:** diff --git a/terraform/examples/gcp/README.md b/terraform/examples/gcp/README.md index 7e6addad..434024c8 100644 --- a/terraform/examples/gcp/README.md +++ b/terraform/examples/gcp/README.md @@ -101,6 +101,8 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | desktop\_feed\_bucket\_name | Globally unique bucket for relay-control's signed Desktop feed (SELFHOST\_DESKTOP\_FEED\_GCS\_BUCKET). Empty disables it. | `string` | `""` | no | | desktop\_feed\_has\_signing\_keys | Whether relay-control has the feed's file signer (relay-control.desktopFeed.signingKeys in your values). true: chart\_values and desktop\_feed carry the SELFHOST\_DESKTOP\_FEED\_STORAGE env. Set false to create the bucket and service account before the signing keys exist: the env is left out, because relay-control exits when storage is set without a signer. Once the keys are in place set it back to true, apply (outputs change only on apply), and save chart\_values again. | `bool` | `true` | no | | desktop\_feed\_service\_account\_id | account\_id of the Google service account relay-control uses for the Desktop feed bucket (6-30 characters). | `string` | `"agent-relay-control"` | no | +| edge\_max\_advanced\_rules | Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max\_advanced\_rules). Google fixes it at 20; a plan that needs more is refused. | `number` | `20` | no | +| edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule. | `number` | `10` | no | | edge\_origin\_secret\_header | Shared-secret request header a fronting proxy adds; requests without one of the accepted values (1-3, for rotation) are denied (modules/gcp/edge-security origin\_secret\_header). Stored in state. | ```object({ name = string values = list(string) })``` | `null` | no | | edge\_security\_policy | Cloud Armor policy for every GKE Ingress backend (modules/gcp/edge-security security\_policy: create, waf, rate\_limits, source\_ip\_allowlist, cloudflare, client\_ip\_header, client\_ip\_rules). Off unless create = true. It is exported as gkeIngress.securityPolicy in output.edge, not in chart\_values, because the chart then also needs the dashboard Service's backend-config annotation. | `any` | `{}` | no | | edge\_ssl\_policy | Compute SSL policy for the GKE Ingress HTTPS listener, exported as gkeIngress.sslPolicy in chart\_values. create = false leaves Google's default policy (TLS 1.0 and later). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | @@ -137,7 +139,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | ---- | ----------- | | chart\_values | Complete non-secret managed-service overlay for values-gcp.yaml plus values-gcp-native.yaml. Save with: terraform output -json chart\_values > managed-values.json. | | desktop\_feed | Values for relay-control's signed Desktop feed on GCS (null unless desktop\_feed\_bucket\_name is set). Put env into relay-control.extraEnv and annotations into relay-control.serviceAccount.annotations; signing keys go through relay-control.desktopFeed.signingKeys. env is empty with desktop\_feed\_has\_signing\_keys = false: relay-control exits when the storage env is set without a signer. | -| edge | Edge policy names (edge.tf). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. | +| edge | Edge policy names (edge.tf). rule\_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY\_POLICY\_CEVAL\_RULES, default 20; total: SECURITY\_POLICY\_RULES, default 100). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. | | functions | Pipeline function, topics and secret (null unless enable\_functions). | | image\_mirror\_root | MIRROR\_ROOT for install guide step 5 (null unless image\_mirror\_repository is set). | | pipeline | Native Sessions pipeline values. The Redis CA is public certificate material, not a credential. | diff --git a/terraform/examples/gcp/edge.tf b/terraform/examples/gcp/edge.tf index 62ea6956..3fa7bd11 100644 --- a/terraform/examples/gcp/edge.tf +++ b/terraform/examples/gcp/edge.tf @@ -18,4 +18,6 @@ module "edge_security" { ssl_policy = var.edge_ssl_policy security_policy = var.edge_security_policy origin_secret_header = var.edge_origin_secret_header + max_subexpressions = var.edge_max_subexpressions + max_advanced_rules = var.edge_max_advanced_rules } diff --git a/terraform/examples/gcp/main.tf b/terraform/examples/gcp/main.tf index 9a9c4329..29547161 100644 --- a/terraform/examples/gcp/main.tf +++ b/terraform/examples/gcp/main.tf @@ -84,6 +84,7 @@ module "postgres" { # that identity to the private address inside this VPC so every service can use # sslmode=verify-full without a proxy or public IP. resource "google_dns_managed_zone" "cloud_sql" { + labels = var.labels project = var.project_id name = "${var.name}-cloud-sql" dns_name = module.postgres.dns_name diff --git a/terraform/examples/gcp/outputs.tf b/terraform/examples/gcp/outputs.tf index 45eb9c08..ddccf916 100644 --- a/terraform/examples/gcp/outputs.tf +++ b/terraform/examples/gcp/outputs.tf @@ -113,11 +113,12 @@ output "chart_values" { } output "edge" { - description = "Edge policy names (edge.tf). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2." + description = "Edge policy names (edge.tf). rule_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY_POLICY_CEVAL_RULES, default 20; total: SECURITY_POLICY_RULES, default 100). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2." value = { ssl_policy = module.edge_security.ssl_policy_name security_policy = module.edge_security.security_policy_name client_ip_header = module.edge_security.client_ip_header + rule_counts = module.edge_security.rule_counts } } diff --git a/terraform/examples/gcp/tests/native.tftest.hcl b/terraform/examples/gcp/tests/native.tftest.hcl index cf51fe67..fc38d889 100644 --- a/terraform/examples/gcp/tests/native.tftest.hcl +++ b/terraform/examples/gcp/tests/native.tftest.hcl @@ -256,4 +256,23 @@ run "edge_cloud_armor_behind_cloudflare" { condition = output.edge.security_policy == "agent-relay-edge" && output.edge.client_ip_header == "CF-Connecting-IP" error_message = "The Cloud Armor policy and the client-address header must be reported." } + # Default of 10 subexpressions: secret header, two WAF rules, and two sign-in + # throttles for Cloudflare's 15 ranges (nine per rule). + assert { + condition = output.edge.rule_counts.advanced == 5 && output.edge.rule_counts.total == 8 + error_message = "Behind Cloudflare the default policy needs 5 advanced rules and 8 in all." + } +} + +run "edge_rule_counts_are_reported" { + command = apply + variables { + edge_security_policy = { create = true } + } + + # Two WAF rules and the sign-in throttle have expressions; the default rule is basic. + assert { + condition = output.edge.rule_counts.advanced == 3 && output.edge.rule_counts.total == 4 + error_message = "The default Cloud Armor policy needs 3 advanced rules and 4 rules in all." + } } diff --git a/terraform/examples/gcp/variables.tf b/terraform/examples/gcp/variables.tf index 7dfb12ec..be781219 100644 --- a/terraform/examples/gcp/variables.tf +++ b/terraform/examples/gcp/variables.tf @@ -259,3 +259,15 @@ variable "edge_origin_secret_header" { default = null sensitive = true } + +variable "edge_max_subexpressions" { + description = "Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule." + type = number + default = 10 +} + +variable "edge_max_advanced_rules" { + description = "Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max_advanced_rules). Google fixes it at 20; a plan that needs more is refused." + type = number + default = 20 +} diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index 87f6eee7..2fd03282 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -18,15 +18,16 @@ Rules are evaluated in priority order and the first match decides. A rule that a | Priority | Rule | Action | | --- | --- | --- | | 100 | `origin_secret_header`: the shared-secret header is missing or wrong | deny 403 | +| 150 | `connecting_asns`, `connecting_address_groups`: the connection does not come from the fronting proxy | deny 403 | | 1000+ | `waf`: OWASP preconfigured rules (SQLi, XSS, LFI, RFI, RCE, scanner detection, protocol attack, session fixation, CVE canary) at sensitivity 1 | deny 403, **preview by default** | -| 2000+ | `client_ip_rules`: on the listed paths, admit the listed client addresses (with the rule's own rate limit) and deny everyone else | allow or throttle, then deny 403 | -| 3000+ | `rate_limits`: per-client throttle on path regexes (default: sign-in and password paths, 60 requests a minute) | 429 beyond the limit | -| 9000+ | the allowlist of connecting addresses (`source_ip_allowlist`, Cloudflare's ranges) | allow | +| 10000+ | `client_ip_rules`: on the listed paths, admit the listed client addresses (with the rule's own rate limit) and deny everyone else | allow or throttle, then deny 403 | +| 20000+ | `rate_limits`: per-client throttle on path regexes (default: sign-in and password paths, 60 requests a minute) | 429 beyond the limit | +| 30000+ | the allowlist of connecting addresses (`source_ip_allowlist`, Cloudflare's ranges) | allow | | last | default | allow, or **deny 403 when an allowlist is set** | **The WAF rules start in preview.** Session transcripts, messages and flows carry code, SQL and shell text, which is what the OWASP signatures look for. Preview logs each match without blocking. Read the load balancer's request log (`jsonPayload.previewSecurityPolicy`) for a week of normal use, then set `waf.preview = false`, or remove the rule names that only match your own traffic from `waf.rules`. -A throttle rule admits the requests under its limit, so when an allowlist is set each throttle rule also requires an allowlisted connecting address. Cloud Armor expressions take five subexpressions, which is why an allowlist appears there in groups of four. +A throttle rule admits the requests under its limit, so when an allowlist is set each throttle rule also requires an allowlisted connecting address. Cloud Armor expressions take ten subexpressions, which is why an allowlist appears there in groups of nine. ### Only some addresses may connect @@ -54,14 +55,15 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } - The client address is read from `CF-Connecting-IP` (Cloud Armor `origin.user_ip`), so rate limits count per client and not per Cloudflare address. - `client_ip_rules` decide which clients may reach which paths, by the client address in `CF-Connecting-IP`. They trust that header, so they require `origin_secret_header`. See the next section. -- There is now one more proxy in front of the application, so `relay-control.auth.trustedProxies` goes up by one: 3 behind Cloudflare (2 was measured for the Google load balancer alone; 3 is expected and not yet measured). `docs/gcp-install/3-EDGE-HARDENING.md` shows how to check it. With the wrong number, sign-in rate limits and lockout are shared by every user. +- There is now one more proxy in front of the application, so `relay-control.auth.trustedProxies` goes up by one: 3 behind Cloudflare, 2 for the Google load balancer alone (both measured on a live install). `docs/gcp-install/3-EDGE-HARDENING.md` shows how to check it. With the wrong number, sign-in rate limits and lockout are shared by every user. ### Which clients may reach which paths `client_ip_rules` are allowlists of client addresses, each scoped to path prefixes and optionally to hosts. They are evaluated in list order, and **the first rule whose paths (and hosts) match a request decides it**: admitted if the client address is in `cidrs`, denied 403 otherwise. A request that matches no rule falls through to the rules below. -A Cloud Armor rule that admits a request ends evaluation. So a request admitted by a client rule meets no later rule: not a later client rule, not `rate_limits`, not the allowlist of connecting addresses (the secret header rule, which runs first, has already proved it came through the proxy). Two consequences, both enforced by the module: +A Cloud Armor rule that admits a request ends evaluation. So a request admitted by a client rule meets no later rule: not a later client rule, not `rate_limits`, and **not the allowlist of connecting addresses**. Three consequences, all enforced by the module: +- **Connections from outside the proxy must be denied first.** Otherwise the secret header is the only guard: a caller that holds it could connect straight to the load balancer and put any client address it likes in the header. (Measured on a live policy: such a caller was admitted as a staff address and as a vendor.) The proxy's ranges do not fit one negated expression, so name its network instead: `connecting_asns = [13335]` for Cloudflare, or `connecting_address_groups` with a Cloud Armor address group of its ranges (Enterprise). `secret_header_only = true` accepts the header as the sole guard. Measured on a live policy with `connecting_asns = [13335]`: 31 of 31 direct requests carrying the secret were denied by this rule, and none of 668 requests through Cloudflare were (two Cloudflare locations sampled). A new rule took almost 4 minutes to be enforced on every backend. What remains: another customer of the same proxy who holds your secret arrives from the same network, so the secret still has to stay secret. - **Every rule states its own limit**: `rate_limit = { count, interval_sec }` (per client address, 429 beyond it) or `unthrottled = true`. - **A rule for every path (`["/"]` without `hosts`) makes `rate_limits` unreachable**, so set `rate_limits = []` and express the sign-in limit as a narrower rule ahead of it, as below. @@ -69,9 +71,10 @@ This example lets two vendors' published ranges reach only the session upload pa ```hcl security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + connecting_asns = [13335] # Cloudflare's network: nothing else may connect + rate_limits = [] client_ip_rules = [ { name = "vendor-uploads" # session uploads, on the history host @@ -104,17 +107,57 @@ security_policy = { origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } ``` +- A path prefix ends at a path segment: `/mcp` covers `/mcp` and `/mcp/tools`, not `/mcpfoo`. A trailing slash is ignored, so `/.well-known/` also covers `/.well-known`. With `hosts`, every listed host is paired with every listed path; use two rules when two hosts need different paths. - Put narrower rules first. Repeat your own ranges (for example your VPN's egress range) in a vendor rule: your users' desktop uploads and MCP clients use those paths too, and a rule denies everyone it does not list. - Without the closing `["/"]` rule, paths that no rule names stay open to every client that comes through the proxy (and `rate_limits` still apply to them). -- Each rule costs one advanced rule per four client ranges (three with `hosts`) plus one. A vendor that publishes hundreds of ranges will exceed the project's Cloud Armor quota for advanced rules; ask Google for a higher quota, or enforce that list at the proxy instead (Cloudflare WAF custom rules see the same client address and have no such limit). The limit per rule here is 294 ranges. +- Each rule costs one advanced rule per nine client ranges (eight with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. - No rule in this module can contain `0.0.0.0/0` or `::/0`: validation refuses them in every list. The only "all addresses" match is Cloud Armor's mandatory default rule, which is deny whenever an allowlist is set. +## Quotas: check before you apply + +Rules with an expression ("advanced" rules: everything here except the allowlist and the default rule) meet two limits. Both were measured on a live project. + +| Limit | Value | Can it be raised? | +| --- | --- | --- | +| Per **policy**: `SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY` | 20 | **No.** It is fixed; a request is refused | +| Per **project**: `SECURITY_POLICY_CEVAL_RULES` | 20 | Yes. A request to 100 was granted automatically in seconds (`gcloud beta quotas preferences create`, needs `cloudquotas.googleapis.com`). A new project may be treated differently | +| Per project, all rules: `SECURITY_POLICY_RULES` | 100 | Yes | + +**So one policy holds at most 20 advanced rules, whatever the project allows.** The module refuses a plan that needs more (`max_advanced_rules`), before anything is created. The `rule_counts` output shows the need: + +| Configuration | Advanced rules | Rules in all | +| --- | --- | --- | +| `create = true` (defaults) | 3 | 4 | +| `cloudflare.enabled` with `origin_secret_header` | 5 | 8 | +| the above, with `rate_limits = []` and client rules: 96 host-scoped vendor ranges, a sign-in rule, a closing staff rule | 20 | 23 | +| the same with 293 vendor ranges | 45: **refused** | | + +The `rule_counts` output is what the configuration needs, not what exists: after a failed apply it shows the planned numbers. The connecting check (`connecting_asns`) is itself one rule. + +With the secret header, two WAF rules and sign-in and staff rules, about 13 rules are left for a vendor list: roughly **100 client ranges** (8 per rule with `hosts`, 9 without). A vendor that publishes more than that does not fit in this policy. The options: + +- **merge ranges**: published lists often contain adjacent blocks that one wider CIDR covers; a wider block also admits the addresses in between, so this is your decision to make; +- **enforce the long list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address and has no such limit) and keep only your own short ranges in `client_ip_rules`; +- Cloud Armor Enterprise address groups, which this module does not support. + +If the project quota (not the per-policy limit) is the one you hit, the apply fails part-way with: + +``` +Error waiting to create SecurityPolicyRule: ... Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally. +``` + +`terraform plan` does not check the project quota. It matters when other policies in the project also use advanced rules. + +**A failed apply leaves a partial policy. Do not attach it.** Terraform creates rules in parallel, so an arbitrary subset exists afterwards: in the measured case a deny rule for the sign-in paths without the rule that admits its clients, and 14 of 37 vendor rules. Attached like that, sign-in and most vendors get 403. Set `gkeIngress.securityPolicy` only after an apply has exited 0. The default rule does not count against `SECURITY_POLICY_RULES`. + +One rule expression holds 10 subexpressions (`max_subexpressions`; the API accepted 10 and refused 11 when measured, although Google's rules language reference still says 5). A rule spends one on the path, one on the host when `hosts` is set, and the rest on client ranges: 8 or 9 ranges per rule. A list is limited to 2,500 ranges and 999 rules. + +Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` with "Capture Groups are not allowed". The module writes `(?:a|b)` in everything it generates and refuses a capture group in `rate_limits[*].path_regex`. + ## Chart side `gkeIngress.securityPolicy` puts the policy on the BackendConfig of every backend Service. The dashboard Service (and Relayfile with `gkeIngress.hosts.files`) needs its `cloud.google.com/backend-config` annotation; `helm template` fails with the exact value until it is set. See the chart README, "GKE edge hardening". -Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defaults use three, and seven with `cloudflare.enabled`. - ### Requirements @@ -136,6 +179,7 @@ Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defa | [google_compute_security_policy.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy) | resource | | [google_compute_security_policy_rule.allowlist](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.client_ip](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | +| [google_compute_security_policy_rule.connecting](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.default](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.origin_secret](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.rate_limit](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | @@ -149,8 +193,10 @@ Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defa | name | Name prefix: the SSL policy is -tls and the Cloud Armor policy -edge. | `string` | n/a | yes | | project\_id | Project of the GKE cluster. GKE Ingress resolves gkeIngress.sslPolicy and gkeIngress.securityPolicy by name in this project. | `string` | n/a | yes | | labels | Labels for every resource here that takes labels (the Cloud Armor policy; Compute SSL policies have none in the API). managed-by=terraform is always added. | `map(string)` | `{}` | no | +| max\_advanced\_rules | Rules with an expression that one Cloud Armor policy may hold. Google fixes this at 20 per policy (quota SECURITY\_POLICY\_ADVANCED\_RULES\_PER\_SECURITY\_POLICY, not adjustable), so a plan that needs more is refused here instead of failing half-way through the apply. Change it only if Google has given your project a different limit. | `number` | `20` | no | +| max\_subexpressions | Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule. | `number` | `10` | no | | origin\_secret\_header | Optional shared-secret request header a fronting proxy adds on every request (for Cloudflare: a request header transform rule). Requests without the exact value are denied 403 before any other rule, which proves the request came through the proxy and makes its client-address header trustworthy. values lists every accepted value (1 to 3), so a rotation has no outage: add the new value here and apply, switch the proxy to it, then remove the old value and apply. The values are stored in Terraform state and readable in the Cloud Armor policy by anyone with compute.securityPolicies.get: treat both as secret stores. | ```object({ name = string values = list(string) })``` | `null` | no | -| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(login|(cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | +| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - connecting\_asns, connecting\_address\_groups: where the fronting proxy connects from, as autonomous system numbers (Cloudflare: 13335) or names of existing Cloud Armor address groups holding its ranges (Cloud Armor Enterprise). A connection from anywhere else is denied 403 right after the secret-header check. client\_ip\_rules require one of them: a request those rules admit is not checked against the allowlist of connecting addresses, so otherwise a caller that holds the secret header could skip the proxy and choose its own client address. secret\_header\_only = true accepts that instead. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) connecting_asns = optional(list(number), []) connecting_address_groups = optional(list(string), []) secret_header_only = optional(bool, false) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | | ssl\_policy | Compute SSL policy for the load balancer's HTTPS listener (chart value gkeIngress.sslPolicy). MODERN with TLS 1.2 serves current browsers, the desktop app and CLIs; RESTRICTED narrows the cipher list further (TLS 1.2 minimum is required by it). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | ### Outputs @@ -159,6 +205,7 @@ Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defa | ---- | ----------- | | chart\_values | The gkeIngress keys to merge into the agent-relay values. With securityPolicy, also set the backend-config annotation the chart names on the relay-webapp (and relayfile) Service. | | client\_ip\_header | Header Cloud Armor reads the client address from (null: the connecting address). When set there is one more proxy in front of the load balancer: relay-control.auth.trustedProxies is one more than for the Google load balancer alone (2), so 3 (docs/gcp-install/3-EDGE-HARDENING.md). | +| rule\_counts | Rules the Cloud Armor policy needs, to compare with the project's quotas before applying: advanced counts rules with an expression (quota SECURITY\_POLICY\_CEVAL\_RULES, default 20 per project), total is every rule including the default (quota SECURITY\_POLICY\_RULES, default 100 per project). Zero without a policy. | | security\_policy\_name | Chart value gkeIngress.securityPolicy. Null when security\_policy.create is false. | | ssl\_policy\_name | Chart value gkeIngress.sslPolicy. Null when ssl\_policy.create is false. | diff --git a/terraform/modules/gcp/edge-security/main.tf b/terraform/modules/gcp/edge-security/main.tf index 3d885bc1..24d8e126 100644 --- a/terraform/modules/gcp/edge-security/main.tf +++ b/terraform/modules/gcp/edge-security/main.tf @@ -18,7 +18,7 @@ locals { # Basic rules take 10 ranges each; expressions take five subexpressions, so # an allowlist conjoined with a path match is cut into groups of four. allow_rules = chunklist(local.allowlist, 10) - allow_groups = chunklist(local.allowlist, 4) + allow_groups = chunklist(local.allowlist, var.max_subexpressions - 1) client_ip_header = local.sp.client_ip_header != null ? local.sp.client_ip_header : (local.sp.cloudflare.enabled ? "CF-Connecting-IP" : null) @@ -31,7 +31,7 @@ locals { # itself require an allowed connecting address. rate_rules = merge([for i, r in local.sp.rate_limits : { for j, group in(local.locked ? local.allow_groups : [[]]) : "${r.name}-${j}" => { - priority = 3000 + i * 100 + j + priority = 20000 + i * 1000 + j limit = r expression = join(" && ", compact(["request.path.matches('${replace(r.path_regex, "\\", "\\\\")}')", length(group) > 0 ? "(${join(" || ", [for c in group : "inIpRange(origin.ip, '${c}')"])})" : ""])) } @@ -42,27 +42,38 @@ locals { # and everyone else on those paths is denied. A Cloud Armor rule that admits # ends evaluation, so admitted requests meet no later rule: that is why each # rule states its own limit and why the secret header rule must come first. - client_groups = { for r in local.sp.client_ip_rules : r.name => chunklist(r.cidrs, length(r.hosts) > 0 ? 3 : 4) } + client_groups = { for r in local.sp.client_ip_rules : r.name => chunklist(r.cidrs, var.max_subexpressions - (length(r.hosts) > 0 ? 2 : 1)) } client_rules = merge([for i, r in local.sp.client_ip_rules : merge({ for j, group in local.client_groups[r.name] : "${r.name}-allow-${j}" => { - priority = 2000 + i * 100 + j + priority = 10000 + i * 1000 + j action = r.rate_limit == null ? "allow" : "throttle" rate = { count = try(r.rate_limit.count, 0), interval_sec = try(r.rate_limit.interval_sec, 0) } description = "${r.name}: listed client addresses on ${join(", ", concat(r.hosts, r.path_prefixes))}" expression = "${local.path_match[r.name]} && (${join(" || ", [for c in group : "inIpRange(origin.user_ip, '${c}')"])})" } }, { "${r.name}-deny" = { - priority = 2000 + i * 100 + 99 + priority = 10000 + i * 1000 + 999 action = "deny(403)" rate = { count = 0, interval_sec = 0 } description = "${r.name}: any other client on ${join(", ", concat(r.hosts, r.path_prefixes))}" expression = local.path_match[r.name] } })]...) - # The path (one subexpression) and, with hosts, the Host header (a second), - # which leaves three or four subexpressions for client ranges. + connecting_guard = length(local.sp.connecting_asns) + length(local.sp.connecting_address_groups) > 0 + + # What the policy costs against the project's Cloud Armor quotas. + rule_counts = { + advanced = (local.secret_header ? 1 : 0) + (local.connecting_guard ? 1 : 0) + length(local.waf_groups) + length(local.client_rules) + length(local.rate_rules) + basic = length(local.allow_rules) + 1 + } + # The path (one subexpression) and, with hosts, the Host header (a second); + # the rest of max_subexpressions holds client ranges. Groups are + # non-capturing: Cloud Armor rejects a regular expression with a capture + # group ("Capture Groups are not allowed"). path_match = { for r in local.sp.client_ip_rules : r.name => join(" && ", compact([ - length(r.hosts) > 0 ? "request.headers['host'].lower().matches('^(${join("|", [for h in r.hosts : replace(h, ".", "\\\\.")])})(:[0-9]+)?$')" : "", - "request.path.matches('^(${join("|", [for p in r.path_prefixes : replace(p, ".", "\\\\.")])})')", + length(r.hosts) > 0 ? "request.headers['host'].lower().matches('^(?:${join("|", [for h in r.hosts : replace(h, ".", "\\\\.")])})(?::[0-9]+)?$')" : "", + # A prefix ends at a path segment: /mcp matches /mcp and /mcp/x, not + # /mcpfoo. "/" is every path. + contains(r.path_prefixes, "/") ? "request.path.matches('^/')" : "request.path.matches('^(?:${join("|", [for p in r.path_prefixes : replace(trimsuffix(p, "/"), ".", "\\\\.")])})(?:/|$)')", ])) } } @@ -102,6 +113,18 @@ resource "google_compute_security_policy" "this" { condition = length(local.sp.client_ip_rules) == 0 || (local.secret_header && local.client_ip_header != null) error_message = "client_ip_rules trust a client-address header, so they need origin_secret_header (proof the request came through the proxy) and client_ip_header or cloudflare.enabled." } + precondition { + condition = local.rule_counts.advanced <= var.max_advanced_rules + error_message = "This policy needs ${local.rule_counts.advanced} rules with an expression, and one Cloud Armor policy holds at most ${var.max_advanced_rules} (quota SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY, which Google does not raise). Nothing has been created. Shorten client_ip_rules (each rule holds ${var.max_subexpressions - 1} ranges, ${var.max_subexpressions - 2} with hosts), merge adjacent ranges, or enforce the long list at the fronting proxy; see the module README, Quotas." + } + precondition { + condition = alltrue([for r in local.sp.client_ip_rules : length(local.client_groups[r.name]) <= 999]) && length(local.allow_groups) <= 999 + error_message = "A list needs more than 999 rules at this max_subexpressions. Shorten it, or enforce it at the fronting proxy." + } + precondition { + condition = length(local.sp.client_ip_rules) == 0 || local.connecting_guard || local.sp.secret_header_only + error_message = "client_ip_rules admit requests before the allowlist of connecting addresses is checked, so a caller that holds the secret header could skip the proxy and choose its own client address. Set connecting_asns (Cloudflare: [13335]) or connecting_address_groups so connections from outside the proxy are denied first, or set secret_header_only = true to accept that the secret header is the only guard." + } precondition { condition = alltrue([for r in local.sp.client_ip_rules : r.rate_limit != null || r.unthrottled]) error_message = "Each client_ip_rules entry admits its clients, and admitted requests skip rate_limits. Give the rule its own rate_limit, or set unthrottled = true to accept that it has none." @@ -132,6 +155,29 @@ resource "google_compute_security_policy_rule" "origin_secret" { } } +# Requests admitted by a client rule never reach the allowlist of connecting +# addresses further down, so without this rule the shared-secret header alone +# decides whether a caller that skipped the proxy gets in (and such a caller +# chooses its own client address). The proxy's ranges do not fit one negated +# expression, but its network number does, and so does an address group: deny +# every connection that comes from none of them. +resource "google_compute_security_policy_rule" "connecting" { + count = local.armor && local.connecting_guard ? 1 : 0 + project = var.project_id + security_policy = google_compute_security_policy.this[0].name + priority = 150 + action = "deny(403)" + description = "Connections from outside the fronting proxy (${join(", ", concat([for a in local.sp.connecting_asns : "AS ${a}"], local.sp.connecting_address_groups))})" + match { + expr { + expression = join(" && ", concat( + [for a in local.sp.connecting_asns : "origin.asn != ${a}"], + [for g in local.sp.connecting_address_groups : "!evaluateAddressGroup('${g}', origin.ip)"], + )) + } + } +} + resource "google_compute_security_policy_rule" "waf" { for_each = local.armor ? { for i, group in local.waf_groups : tostring(i) => group } : {} project = var.project_id @@ -198,7 +244,7 @@ resource "google_compute_security_policy_rule" "allowlist" { for_each = local.armor ? { for i, group in local.allow_rules : tostring(i) => group } : {} project = var.project_id security_policy = google_compute_security_policy.this[0].name - priority = 9000 + tonumber(each.key) + priority = 30000 + tonumber(each.key) action = "allow" description = "Allowed connecting addresses (${local.sp.cloudflare.enabled ? "Cloudflare and " : ""}source_ip_allowlist)" match { diff --git a/terraform/modules/gcp/edge-security/outputs.tf b/terraform/modules/gcp/edge-security/outputs.tf index d58c3d27..af6f760a 100644 --- a/terraform/modules/gcp/edge-security/outputs.tf +++ b/terraform/modules/gcp/edge-security/outputs.tf @@ -20,3 +20,11 @@ output "client_ip_header" { description = "Header Cloud Armor reads the client address from (null: the connecting address). When set there is one more proxy in front of the load balancer: relay-control.auth.trustedProxies is one more than for the Google load balancer alone (2), so 3 (docs/gcp-install/3-EDGE-HARDENING.md)." value = local.armor ? local.client_ip_header : null } + +output "rule_counts" { + description = "Rules the Cloud Armor policy needs, to compare with the project's quotas before applying: advanced counts rules with an expression (quota SECURITY_POLICY_CEVAL_RULES, default 20 per project), total is every rule including the default (quota SECURITY_POLICY_RULES, default 100 per project). Zero without a policy." + value = local.armor ? { + advanced = local.rule_counts.advanced + total = local.rule_counts.advanced + local.rule_counts.basic + } : { advanced = 0, total = 0 } +} diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index ad09f92e..c1c16a8b 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -3,6 +3,12 @@ mock_provider "google" {} variables { project_id = "proj" name = "relay" + # Runs that build long lists to test rule splitting lift the per-policy + # limit; the limit itself has its own runs at the end. + max_advanced_rules = 1000 + # Most runs pin 5 subexpressions per rule so that small lists exercise the + # splitting into several rules; the module's default (10) has its own runs. + max_subexpressions = 5 } run "defaults_create_only_the_tls_policy" { @@ -70,7 +76,7 @@ run "waf_and_rate_limit_defaults" { error_message = "The default rules include cve-canary (Log4Shell)." } assert { - condition = keys(google_compute_security_policy_rule.rate_limit) == ["sign-in-0"] && google_compute_security_policy_rule.rate_limit["sign-in-0"].action == "throttle" && google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(login|(cloud/)?api/v1/auth/)')" + condition = keys(google_compute_security_policy_rule.rate_limit) == ["sign-in-0"] && google_compute_security_policy_rule.rate_limit["sign-in-0"].action == "throttle" && google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(?:login|(?:cloud/)?api/v1/auth/)')" error_message = "One throttle rule on the sign-in paths." } assert { @@ -116,8 +122,9 @@ run "cloudflare_origin_lockdown" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } + create = true + cloudflare = { enabled = true } + secret_header_only = true client_ip_rules = [{ name = "uploads" path_prefixes = ["/v1/delivery", "/v1/turns"] @@ -146,7 +153,7 @@ run "cloudflare_origin_lockdown" { error_message = "Rate limits stay inside the allowlist and count per client address." } assert { - condition = google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(login|(cloud/)?api/v1/auth/)') && (inIpRange(origin.ip, '173.245.48.0/20') || inIpRange(origin.ip, '103.21.244.0/22') || inIpRange(origin.ip, '103.22.200.0/22') || inIpRange(origin.ip, '103.31.4.0/22'))" + condition = google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(?:login|(?:cloud/)?api/v1/auth/)') && (inIpRange(origin.ip, '173.245.48.0/20') || inIpRange(origin.ip, '103.21.244.0/22') || inIpRange(origin.ip, '103.22.200.0/22') || inIpRange(origin.ip, '103.31.4.0/22'))" error_message = "Five subexpressions at most: the path and four ranges." } assert { @@ -154,7 +161,7 @@ run "cloudflare_origin_lockdown" { error_message = "Five client ranges: two allow rules and the path deny." } assert { - condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(/v1/delivery|/v1/turns)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(/v1/delivery|/v1/turns)')" + condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)(?:/|$)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)(?:/|$)')" error_message = "Client rules match the path and origin.user_ip, then deny the path." } # Order: secret header, WAF, client rules, rate limits, allowlist, default. @@ -168,9 +175,10 @@ run "client_rules_need_the_secret_header" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] } } expect_failures = [google_compute_security_policy.this] @@ -231,9 +239,10 @@ run "vendor_paths_and_a_closed_remainder" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [ { name = "vendors" @@ -264,7 +273,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "Every rule admits its clients (four ranges per rule, three with hosts) and then denies its paths." } assert { - condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(history\\\\.relay\\\\.example\\\\.com)(:[0-9]+)?$') && request.path.matches('^(/v1/delivery|/v1/turns|/v1/auth/token/refresh)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" + condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/v1/turns|/v1/auth/token/refresh)(?:/|$)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" error_message = "Host, path and the client ranges, within five subexpressions." } # A short vendor list ahead of a staff-only "/" rule: the vendor is admitted @@ -280,7 +289,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "A one-range rule with a rate_limit throttles per client address." } assert { - condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^(/)')" + condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^/')" error_message = "An unthrottled rule admits with allow; the closing rule denies every remaining path." } assert { @@ -294,9 +303,10 @@ run "client_rule_must_choose_a_limit" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] } origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } } @@ -308,9 +318,10 @@ run "catch_all_client_rule_refuses_dead_rate_limits" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] } origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } } @@ -333,3 +344,364 @@ run "rate_limit_regex_with_an_escape" { error_message = "The backslash is doubled inside the quoted expression string." } } + +# The policy reports what it costs against the project's rule quotas, and a +# higher max_subexpressions packs a long client list into fewer rules. +run "rule_counts_and_wider_expressions" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(24) : "198.51.${i}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + # 24 ranges at 3 per rule = 8 admitting rules + 1 deny; + secret header + 2 WAF = 12 advanced; + 2 allowlist + default = 15. + assert { + condition = length(google_compute_security_policy_rule.client_ip) == 9 && output.rule_counts.advanced == 12 && output.rule_counts.total == 15 + error_message = "rule_counts must equal the rules created." + } + assert { + condition = output.rule_counts.total == length(google_compute_security_policy_rule.client_ip) + length(google_compute_security_policy_rule.waf) + length(google_compute_security_policy_rule.origin_secret) + length(google_compute_security_policy_rule.rate_limit) + length(google_compute_security_policy_rule.allowlist) + length(google_compute_security_policy_rule.default) + error_message = "total counts every rule resource." + } +} + +run "ten_subexpressions_need_a_third_of_the_rules" { + command = apply + variables { + max_subexpressions = 10 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(24) : "198.51.${i}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + # 8 ranges per rule (10 - host - path): 3 admitting rules + 1 deny. + assert { + condition = length(google_compute_security_policy_rule.client_ip) == 4 && output.rule_counts.advanced == 7 && length(regexall("inIpRange", google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression)) == 8 + error_message = "With 10 subexpressions a host-scoped rule carries 8 ranges." + } +} + +run "policy_off_reports_zero_rules" { + command = apply + assert { + condition = output.rule_counts.advanced == 0 && output.rule_counts.total == 0 + error_message = "No policy, no rules." + } +} + +# A capture group in a user regex is refused at plan: Cloud Armor rejects it at apply. +run "capture_group_in_path_regex_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "auth", path_regex = "^/(login|setup)", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} + +run "non_capturing_and_escaped_parentheses_accepted" { + command = apply + variables { + security_policy = { create = true, waf = { enabled = false }, rate_limits = [{ name = "auth", path_regex = "^/(?:login|setup)\\(x\\)", count = 10, interval_sec = 60 }] } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 1 + error_message = "(?:...) groups and escaped parentheses are allowed." + } +} + +# No expression the module builds contains a capture group. +run "generated_expressions_have_no_capture_groups" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [ + { name = "vendors", hosts = ["history.relay.example.com", "api.relay.example.com"], path_prefixes = ["/v1/delivery", "/.well-known"], cidrs = ["192.0.2.0/24"], unthrottled = true }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = alltrue([for e in concat( + [for r in google_compute_security_policy_rule.client_ip : r.match[0].expr[0].expression], + [for r in google_compute_security_policy_rule.rate_limit : r.match[0].expr[0].expression], + ) : length(regexall("matches\\('[^']*\\((?:[^?]|$)", e)) == 0]) + error_message = "Every group inside a matches() pattern must be non-capturing." + } + assert { + condition = google_compute_security_policy_rule.client_ip["vendors-deny"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com|api\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/\\\\.well-known)(?:/|$)')" + error_message = "Host and path patterns in the form the API accepted." + } +} + +# The default: 10 subexpressions. Cloudflare's 15 ranges need two throttle rules, not four. +run "default_is_ten_subexpressions" { + command = apply + variables { + max_subexpressions = 10 + security_policy = { create = true, cloudflare = { enabled = true } } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 2 && length(regexall("inIpRange", google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression)) == 9 && output.rule_counts.advanced == 5 + error_message = "Path plus nine ranges per throttle rule." + } +} + +# A list longer than the old cap (294 ranges, 99 rules per list): 300 ranges +# at three per rule is 100 admitting rules. Its rules must stay in one band, +# in order, ahead of the next client rule. +run "long_list_keeps_its_band_and_order" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true + client_ip_rules = [ + { + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(300) : "10.${floor(i / 256)}.${i % 256}.0/24"] + unthrottled = true + }, + { + name = "staff" + path_prefixes = ["/"] + cidrs = ["203.0.113.0/24"] + unthrottled = true + }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + assert { + condition = length([for k, r in google_compute_security_policy_rule.client_ip : k if startswith(k, "vendors-allow-")]) == 100 + error_message = "300 ranges at three per rule are 100 admitting rules." + } + assert { + condition = length(distinct([for r in google_compute_security_policy_rule.client_ip : r.priority])) == length(google_compute_security_policy_rule.client_ip) + error_message = "No two client rules share a priority." + } + assert { + condition = max([for k, r in google_compute_security_policy_rule.client_ip : r.priority if startswith(k, "vendors-allow-")]...) < google_compute_security_policy_rule.client_ip["vendors-deny"].priority && google_compute_security_policy_rule.client_ip["vendors-deny"].priority < google_compute_security_policy_rule.client_ip["staff-allow-0"].priority && google_compute_security_policy_rule.client_ip["staff-allow-0"].priority < google_compute_security_policy_rule.client_ip["staff-deny"].priority + error_message = "Every admitting rule precedes its deny, and the deny precedes the next client rule." + } + assert { + condition = max([for r in google_compute_security_policy_rule.client_ip : r.priority]...) < min([for r in google_compute_security_policy_rule.allowlist : r.priority]...) && output.rule_counts.advanced == 106 + error_message = "Client rules stay ahead of the allowlist, and the count is reported (100 + 2 denies + 1 staff allow + secret header + 2 WAF)." + } +} + +# Patterns without a capture group that contain other parentheses are accepted. +run "flags_and_bracketed_parentheses_accepted" { + command = apply + variables { + security_policy = { + create = true + waf = { enabled = false } + rate_limits = [ + { name = "flag", path_regex = "(?i)^/login", count = 10, interval_sec = 60 }, + { name = "class", path_regex = "^/file[(]old[)]$", count = 10, interval_sec = 60 }, + { name = "escaped", path_regex = "^/a\\(b\\)(?:c|d)", count = 10, interval_sec = 60 }, + ] + } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 3 + error_message = "Flags, parentheses in a character class and escaped parentheses are not capture groups." + } +} + +run "named_capture_group_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "named", path_regex = "^/(?Pv1|v2)/", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} + +run "capture_group_after_a_class_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "mixed", path_regex = "^/[(]x[)](a|b)", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} + +# One policy holds 20 rules with an expression, and Google does not raise it. +# A plan that needs more is refused before anything is created. +run "more_than_twenty_advanced_rules_refused_at_plan" { + command = plan + variables { + max_subexpressions = 10 + max_advanced_rules = 20 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(293) : "10.${floor(i / 256)}.${i % 256}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + expect_failures = [google_compute_security_policy.this] +} + +# The shape the live proof applied cleanly: exactly 20. +run "twenty_advanced_rules_fit" { + command = apply + variables { + max_subexpressions = 10 + max_advanced_rules = 20 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true + client_ip_rules = [ + { name = "vendors", hosts = ["history.relay.example.com"], path_prefixes = ["/v1/delivery"], cidrs = [for i in range(96) : "10.0.${i}.0/24"], unthrottled = true }, + { name = "sign-in", path_prefixes = ["/login"], cidrs = ["203.0.113.0/24"], rate_limit = { count = 60, interval_sec = 60 } }, + { name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + # 96 ranges at 8 = 12 admitting rules + deny; sign-in 2; staff 2; secret header; 2 WAF. + assert { + condition = output.rule_counts.advanced == 20 && output.rule_counts.total == 23 + error_message = "96 host-scoped vendor ranges with sign-in and staff rules need exactly 20 advanced rules." + } +} + +# A prefix stops at a path segment, and a trailing slash in the input does not double it. +run "path_prefix_ends_at_a_segment" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "mcp", path_prefixes = ["/mcp", "/cloud/mcp/"], cidrs = ["192.0.2.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.client_ip["mcp-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/mcp|/cloud/mcp)(?:/|$)')" + error_message = "/mcp must match /mcp and /mcp/x but not /mcpfoo." + } + # The same pattern, checked against sample paths. + assert { + condition = google_compute_security_policy_rule.client_ip["mcp-deny"].action == "deny(403)" && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcpfoo")) == 0 && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcp/tools")) == 1 && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcp")) == 1 + error_message = "The pattern has the segment boundary." + } +} + +# connecting_asns: a connection from outside the proxy's network is denied +# before any rule can admit it, so the secret header alone is not enough. +run "connecting_asns_deny_other_networks_first" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + connecting_asns = [13335, 209242] + rate_limits = [] + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.connecting[0].action == "deny(403)" && google_compute_security_policy_rule.connecting[0].match[0].expr[0].expression == "origin.asn != 13335 && origin.asn != 209242" + error_message = "Deny when the connection comes from none of the listed networks." + } + assert { + condition = google_compute_security_policy_rule.origin_secret[0].priority < google_compute_security_policy_rule.connecting[0].priority && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.client_ip : r.priority]...) && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.waf : r.priority]...) + error_message = "The network check runs before the WAF and before every rule that admits." + } + assert { + condition = output.rule_counts.advanced == 6 + error_message = "Secret header, network check, two WAF rules, staff allow and deny." + } +} + +run "no_connecting_asn_rule_by_default" { + command = apply + variables { + security_policy = { create = true, cloudflare = { enabled = true } } + } + assert { + condition = length(google_compute_security_policy_rule.connecting) == 0 + error_message = "connecting_asns is opt-in." + } +} + +# client_ip_rules admit before the connecting-address allowlist, so the plan is +# refused until connections from outside the proxy are denied first, or the +# operator accepts the secret header as the only guard. +run "client_rule_needs_a_connecting_check" { + command = plan + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + rate_limits = [] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + expect_failures = [google_compute_security_policy.this] +} + +# The same guard with an address group of the proxy's ranges (Cloud Armor Enterprise). +run "connecting_address_group_denies_other_sources_first" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + connecting_address_groups = ["relay-cloudflare"] + rate_limits = [] + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.connecting[0].match[0].expr[0].expression == "!evaluateAddressGroup('relay-cloudflare', origin.ip)" && google_compute_security_policy_rule.connecting[0].action == "deny(403)" && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.client_ip : r.priority]...) + error_message = "A connection whose address is outside the proxy's address group is denied before any client rule." + } +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index 63175805..a3417853 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -46,6 +46,7 @@ variable "security_policy" { - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client_ip_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin_secret_header. - client_ip_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user_ip) is in cidrs, denied 403 otherwise. A last rule with path_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin_secret_header. An admitted request meets no later rule, including rate_limits, so every rule states its own limit: rate_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate_limits = [] and express sign-in limits as an earlier, narrower rule. + - connecting_asns, connecting_address_groups: where the fronting proxy connects from, as autonomous system numbers (Cloudflare: 13335) or names of existing Cloud Armor address groups holding its ranges (Cloud Armor Enterprise). A connection from anywhere else is denied 403 right after the secret-header check. client_ip_rules require one of them: a request those rules admit is not checked against the allowlist of connecting addresses, so otherwise a caller that holds the secret header could skip the proxy and choose its own client address. secret_header_only = true accepts that instead. - adaptive_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). EOT type = object({ @@ -67,7 +68,7 @@ variable "security_policy" { ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" - path_regex = "^/(login|(cloud/)?api/v1/auth/)" + path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) @@ -87,7 +88,10 @@ variable "security_policy" { "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) - client_ip_header = optional(string) + client_ip_header = optional(string) + connecting_asns = optional(list(number), []) + connecting_address_groups = optional(list(string), []) + secret_header_only = optional(bool, false) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) @@ -111,21 +115,28 @@ variable "security_policy" { ) : can(cidrhost(c, 0)) && !contains(["0.0.0.0/0", "::/0"], c)]) error_message = "Every allowlist entry must be a CIDR range, and never 0.0.0.0/0 or ::/0." } + # Capture groups: after dropping escaped characters and character classes, + # a "(" that is not followed by "?" opens one, and so does a named group + # ((?P or (?). Flags such as (?i) and (?:...) are fine. validation { condition = alltrue([for r in var.security_policy.rate_limits : - can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && length(r.path_regex) <= 512 && + can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && length(regexall("\\((?:[^?]|$)|\\(\\?P?<[A-Za-z_]", replace(replace(r.path_regex, "/\\\\./", ""), "/\\[[^\\]]*\\]/", ""))) == 0 && length(r.path_regex) <= 512 && r.count >= 1 && contains([10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700, 3600], r.interval_sec) && r.ban_duration_sec >= 0 ]) && length(var.security_policy.rate_limits) <= 9 && length(distinct(var.security_policy.rate_limits[*].name)) == length(var.security_policy.rate_limits) - error_message = "At most 9 rate_limits with unique lowercase names, a path_regex without single quotes, count >= 1 and an interval_sec Cloud Armor accepts (10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700 or 3600)." + error_message = "At most 9 rate_limits with unique lowercase names, a path_regex without single quotes or capture groups (Cloud Armor rejects them: write (?:a|b), not (a|b)), count >= 1 and an interval_sec Cloud Armor accepts (10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700 or 3600)." } validation { condition = alltrue([for r in var.security_policy.client_ip_rules : - can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && length(r.cidrs) >= 1 && length(r.cidrs) <= 294 && + can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && length(r.cidrs) >= 1 && length(r.cidrs) <= 2500 && length(r.path_prefixes) >= 1 && alltrue([for p in r.path_prefixes : can(regex("^/[A-Za-z0-9/._~-]*$", p))]) && alltrue([for h in r.hosts : can(regex("^[a-z0-9]([a-z0-9.-]*[a-z0-9])?$", h))]) && (r.rate_limit == null ? true : r.rate_limit.count >= 1 && contains([10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700, 3600], r.rate_limit.interval_sec)) ]) && length(var.security_policy.client_ip_rules) <= 9 && length(distinct(var.security_policy.client_ip_rules[*].name)) == length(var.security_policy.client_ip_rules) - error_message = "At most 9 client_ip_rules with unique lowercase names, 1-294 cidrs, path_prefixes that start with / and hold only unreserved URL characters, lowercase DNS hosts, and a rate_limit with count >= 1 and an interval_sec Cloud Armor accepts." + error_message = "At most 9 client_ip_rules with unique lowercase names, 1-2500 cidrs, path_prefixes that start with / and hold only unreserved URL characters, lowercase DNS hosts, and a rate_limit with count >= 1 and an interval_sec Cloud Armor accepts." + } + validation { + condition = length(var.security_policy.connecting_asns) + length(var.security_policy.connecting_address_groups) <= 8 && alltrue([for a in var.security_policy.connecting_asns : a >= 1 && a <= 4294967295 && floor(a) == a]) && alltrue([for g in var.security_policy.connecting_address_groups : can(regex("^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$", g))]) + error_message = "connecting_asns are autonomous system numbers (whole numbers from 1 to 4294967295) and connecting_address_groups are address group names (not full paths); at most 8 in all." } validation { condition = var.security_policy.client_ip_header == null || can(regex("^[A-Za-z][A-Za-z0-9-]{0,63}$", var.security_policy.client_ip_header)) @@ -147,3 +158,23 @@ variable "origin_secret_header" { error_message = "origin_secret_header.name is a lowercase header name; values holds 1-3 distinct values of 32-256 characters from A-Z a-z 0-9 _ - (for example: openssl rand -hex 32)." } } + +variable "max_subexpressions" { + description = "Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule." + type = number + default = 10 + validation { + condition = contains([5, 6, 7, 8, 9, 10], var.max_subexpressions) + error_message = "max_subexpressions is 5 to 10." + } +} + +variable "max_advanced_rules" { + description = "Rules with an expression that one Cloud Armor policy may hold. Google fixes this at 20 per policy (quota SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY, not adjustable), so a plan that needs more is refused here instead of failing half-way through the apply. Change it only if Google has given your project a different limit." + type = number + default = 20 + validation { + condition = var.max_advanced_rules >= 1 && floor(var.max_advanced_rules) == var.max_advanced_rules + error_message = "max_advanced_rules is a positive whole number." + } +}