From c8555a8bfd6f847411ae96954d783b6f1a1fa715 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 11:34:46 +0100 Subject: [PATCH 01/10] fix(edge-security): report Cloud Armor rule counts, document the quotas, lift the 294-range limit Found by the live GCP proof at plan time, before any apply: a client rule with a vendor's published ranges (about 290) plans about 105 rules with an expression and 109 rules in all, against Google's defaults of 20 (SECURITY_POLICY_CEVAL_RULES) and 100 (SECURITY_POLICY_RULES) per project. The apply would fail part-way with a quota error, and nothing said so beforehand. - New output rule_counts (advanced, total), also in examples/gcp output.edge, so terraform plan shows what the policy needs. - Module README "Quotas: check before you apply" and the GKE edge guide: request the two quotas first, or enforce a long list at the fronting proxy. - max_subexpressions (default 5, unchanged behaviour): how many ranges fit in one rule. Google's quota page lists 10, which would need about a third as many rules; not proven, so opt-in. - A rule list was capped at 294 ranges by the priority spacing, and the vendor list was at 292. Rule priorities now use bands of 1000 (client rules 10000+, rate limits 20000+, allowlist 30000+): up to 999 rules and 2500 ranges per list. Relative order is unchanged. An existing policy's rules are replaced on the next apply. - The guide's API list gains cloudresourcemanager.googleapis.com, which the Terraform root needs for data.google_project (plan otherwise fails 403). Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/0-START-HERE.md | 3 +- docs/gcp-install/3-EDGE-HARDENING.md | 19 ++++-- terraform/examples/gcp/README.md | 3 +- terraform/examples/gcp/edge.tf | 1 + terraform/examples/gcp/outputs.tf | 3 +- .../examples/gcp/tests/native.tftest.hcl | 13 ++++ terraform/examples/gcp/variables.tf | 6 ++ terraform/modules/gcp/edge-security/README.md | 32 +++++++-- terraform/modules/gcp/edge-security/main.tf | 21 ++++-- .../modules/gcp/edge-security/outputs.tf | 8 +++ .../tests/edge_security.tftest.hcl | 65 +++++++++++++++++++ .../modules/gcp/edge-security/variables.tf | 14 +++- 12 files changed, 167 insertions(+), 21 deletions(-) diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index dc666c69..58c6c76b 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -74,7 +74,8 @@ gcloud auth login && gcloud auth application-default login gcloud config set project gcloud services enable container.googleapis.com sqladmin.googleapis.com servicenetworking.googleapis.com \ redis.googleapis.com pubsub.googleapis.com secretmanager.googleapis.com dns.googleapis.com \ - artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com + artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com \ + cloudresourcemanager.googleapis.com gcloud container clusters get-credentials --location # add --internal-ip for a private endpoint kubectl auth can-i '*' '*' --all-namespaces gcloud container clusters describe --location \ diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 05cad492..353fea0d 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -78,7 +78,10 @@ when you run them. edge_security_policy = { create = true } ``` - Run `terraform apply`. `terraform output edge` prints its name + Run `terraform plan` first and read `rule_counts` in the `edge` output: a + new project allows 20 rules with an expression and 100 rules in all, and + the plan tells you what this policy needs (the defaults need 3 and 4). + Then `terraform apply`. `terraform output edge` prints the policy's name (`-edge`). This policy has the OWASP rules in **preview** (they log and do not block) and a limit of 60 sign-in requests a minute per client. @@ -234,9 +237,17 @@ Cloudflare can reach it. other path. A request that a rule admits is not checked by any later rule, so each rule carries its own `rate_limit` (per client), and the sign-in limit is a rule of its own ahead of the closing one. The module's - README (`terraform/modules/gcp/edge-security`) explains the limits. A - vendor that publishes hundreds of ranges needs a higher Cloud Armor quota, - or that list enforced as a Cloudflare WAF custom rule instead. + README (`terraform/modules/gcp/edge-security`) explains the limits. + + **Check the quota before you apply a vendor list.** Each rule takes three + or four ranges, so a vendor that publishes about 290 ranges needs about + 105 rules with an expression, against a default of 20 per project (and 100 + rules in all). `terraform plan` shows the numbers in the `edge` output's + `rule_counts`. Request `SECURITY_POLICY_CEVAL_RULES` and + `SECURITY_POLICY_RULES` increases for the Compute Engine API first (allow + headroom for the vendor adding ranges), or enforce that list as a + Cloudflare WAF custom rule and keep only your own ranges here. Without + either, the apply fails part-way with a quota error. **Done when:** diff --git a/terraform/examples/gcp/README.md b/terraform/examples/gcp/README.md index 4f315038..6eff78c5 100644 --- a/terraform/examples/gcp/README.md +++ b/terraform/examples/gcp/README.md @@ -100,6 +100,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | deletion\_protection | Prevent deletion of Cloud SQL, Memorystore and Pub/Sub data resources. Disable only for a disposable proof environment. | `bool` | `true` | no | | desktop\_feed\_bucket\_name | Globally unique bucket for relay-control's signed Desktop feed (SELFHOST\_DESKTOP\_FEED\_GCS\_BUCKET). Empty disables it. | `string` | `""` | no | | desktop\_feed\_service\_account\_id | account\_id of the Google service account relay-control uses for the Desktop feed bucket (6-30 characters). | `string` | `"agent-relay-control"` | no | +| edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 5 is the documented limit; a higher value packs long client lists into fewer rules once an apply in your project has accepted it. | `number` | `5` | no | | edge\_origin\_secret\_header | Shared-secret request header a fronting proxy adds; requests without one of the accepted values (1-3, for rotation) are denied (modules/gcp/edge-security origin\_secret\_header). Stored in state. | ```object({ name = string values = list(string) })``` | `null` | no | | edge\_security\_policy | Cloud Armor policy for every GKE Ingress backend (modules/gcp/edge-security security\_policy: create, waf, rate\_limits, source\_ip\_allowlist, cloudflare, client\_ip\_header, client\_ip\_rules). Off unless create = true. It is exported as gkeIngress.securityPolicy in output.edge, not in chart\_values, because the chart then also needs the dashboard Service's backend-config annotation. | `any` | `{}` | no | | edge\_ssl\_policy | Compute SSL policy for the GKE Ingress HTTPS listener, exported as gkeIngress.sslPolicy in chart\_values. create = false leaves Google's default policy (TLS 1.0 and later). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | @@ -136,7 +137,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | ---- | ----------- | | chart\_values | Complete non-secret managed-service overlay for values-gcp.yaml plus values-gcp-native.yaml. Save with: terraform output -json chart\_values > managed-values.json. | | desktop\_feed | Values for relay-control's signed Desktop feed on GCS (null unless desktop\_feed\_bucket\_name is set). Put env into relay-control.extraEnv and annotations into relay-control.serviceAccount.annotations; signing keys go through relay-control.desktopFeed.signingKeys. | -| edge | Edge policy names (edge.tf). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. | +| edge | Edge policy names (edge.tf). rule\_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY\_POLICY\_CEVAL\_RULES, default 20; total: SECURITY\_POLICY\_RULES, default 100). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. | | functions | Pipeline function, topics and secret (null unless enable\_functions). | | image\_mirror\_root | MIRROR\_ROOT for install guide step 5 (null unless image\_mirror\_repository is set). | | pipeline | Native Sessions pipeline values. The Redis CA is public certificate material, not a credential. | diff --git a/terraform/examples/gcp/edge.tf b/terraform/examples/gcp/edge.tf index 62ea6956..e10e2222 100644 --- a/terraform/examples/gcp/edge.tf +++ b/terraform/examples/gcp/edge.tf @@ -18,4 +18,5 @@ module "edge_security" { ssl_policy = var.edge_ssl_policy security_policy = var.edge_security_policy origin_secret_header = var.edge_origin_secret_header + max_subexpressions = var.edge_max_subexpressions } diff --git a/terraform/examples/gcp/outputs.tf b/terraform/examples/gcp/outputs.tf index e2390480..22a7885e 100644 --- a/terraform/examples/gcp/outputs.tf +++ b/terraform/examples/gcp/outputs.tf @@ -113,11 +113,12 @@ output "chart_values" { } output "edge" { - description = "Edge policy names (edge.tf). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2." + description = "Edge policy names (edge.tf). rule_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY_POLICY_CEVAL_RULES, default 20; total: SECURITY_POLICY_RULES, default 100). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2." value = { ssl_policy = module.edge_security.ssl_policy_name security_policy = module.edge_security.security_policy_name client_ip_header = module.edge_security.client_ip_header + rule_counts = module.edge_security.rule_counts } } diff --git a/terraform/examples/gcp/tests/native.tftest.hcl b/terraform/examples/gcp/tests/native.tftest.hcl index 557936a8..e03db5ed 100644 --- a/terraform/examples/gcp/tests/native.tftest.hcl +++ b/terraform/examples/gcp/tests/native.tftest.hcl @@ -237,3 +237,16 @@ run "edge_cloud_armor_behind_cloudflare" { error_message = "The Cloud Armor policy and the client-address header must be reported." } } + +run "edge_rule_counts_are_reported" { + command = apply + variables { + edge_security_policy = { create = true } + } + + # Two WAF rules and the sign-in throttle have expressions; the default rule is basic. + assert { + condition = output.edge.rule_counts.advanced == 3 && output.edge.rule_counts.total == 4 + error_message = "The default Cloud Armor policy needs 3 advanced rules and 4 rules in all." + } +} diff --git a/terraform/examples/gcp/variables.tf b/terraform/examples/gcp/variables.tf index d871148e..18784162 100644 --- a/terraform/examples/gcp/variables.tf +++ b/terraform/examples/gcp/variables.tf @@ -253,3 +253,9 @@ variable "edge_origin_secret_header" { default = null sensitive = true } + +variable "edge_max_subexpressions" { + description = "Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max_subexpressions). 5 is the documented limit; a higher value packs long client lists into fewer rules once an apply in your project has accepted it." + type = number + default = 5 +} diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index 87f6eee7..e0597b8a 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -19,9 +19,9 @@ Rules are evaluated in priority order and the first match decides. A rule that a | --- | --- | --- | | 100 | `origin_secret_header`: the shared-secret header is missing or wrong | deny 403 | | 1000+ | `waf`: OWASP preconfigured rules (SQLi, XSS, LFI, RFI, RCE, scanner detection, protocol attack, session fixation, CVE canary) at sensitivity 1 | deny 403, **preview by default** | -| 2000+ | `client_ip_rules`: on the listed paths, admit the listed client addresses (with the rule's own rate limit) and deny everyone else | allow or throttle, then deny 403 | -| 3000+ | `rate_limits`: per-client throttle on path regexes (default: sign-in and password paths, 60 requests a minute) | 429 beyond the limit | -| 9000+ | the allowlist of connecting addresses (`source_ip_allowlist`, Cloudflare's ranges) | allow | +| 10000+ | `client_ip_rules`: on the listed paths, admit the listed client addresses (with the rule's own rate limit) and deny everyone else | allow or throttle, then deny 403 | +| 20000+ | `rate_limits`: per-client throttle on path regexes (default: sign-in and password paths, 60 requests a minute) | 429 beyond the limit | +| 30000+ | the allowlist of connecting addresses (`source_ip_allowlist`, Cloudflare's ranges) | allow | | last | default | allow, or **deny 403 when an allowlist is set** | **The WAF rules start in preview.** Session transcripts, messages and flows carry code, SQL and shell text, which is what the OWASP signatures look for. Preview logs each match without blocking. Read the load balancer's request log (`jsonPayload.previewSecurityPolicy`) for a week of normal use, then set `waf.preview = false`, or remove the rule names that only match your own traffic from `waf.rules`. @@ -106,15 +106,33 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } - Put narrower rules first. Repeat your own ranges (for example your VPN's egress range) in a vendor rule: your users' desktop uploads and MCP clients use those paths too, and a rule denies everyone it does not list. - Without the closing `["/"]` rule, paths that no rule names stay open to every client that comes through the proxy (and `rate_limits` still apply to them). -- Each rule costs one advanced rule per four client ranges (three with `hosts`) plus one. A vendor that publishes hundreds of ranges will exceed the project's Cloud Armor quota for advanced rules; ask Google for a higher quota, or enforce that list at the proxy instead (Cloudflare WAF custom rules see the same client address and have no such limit). The limit per rule here is 294 ranges. +- Each rule costs one advanced rule per four client ranges (three with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. - No rule in this module can contain `0.0.0.0/0` or `::/0`: validation refuses them in every list. The only "all addresses" match is Cloud Armor's mandatory default rule, which is deny whenever an allowlist is set. +## Quotas: check before you apply + +A new project allows **20 rules with an expression** (`SECURITY_POLICY_CEVAL_RULES`) and **100 security policy rules** in all (`SECURITY_POLICY_RULES`), across every policy in the project. Google bills each rule monthly. + +The `rule_counts` output says what this policy needs, and `terraform plan` shows it before anything is created: + +| Configuration | Advanced rules | Rules in all | +| --- | --- | --- | +| `create = true` (defaults) | 3 | 4 | +| `cloudflare.enabled` with `origin_secret_header` | 7 | 10 | +| plus a host-scoped client rule with 290 vendor ranges | about 105 | about 108 | + +A long vendor list therefore fails on both quotas in a fresh project. Either: + +- **request the quotas first** (Console, IAM and Admin, Quotas: `SECURITY_POLICY_CEVAL_RULES` and `SECURITY_POLICY_RULES` for the Compute Engine API, with the numbers from `rule_counts` plus headroom for the vendor adding ranges), and apply once Google grants them; or +- **enforce that list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address) and keep only your own short ranges in `client_ip_rules`; or +- **use wider expressions**: `max_subexpressions` (default 5, the limit in Google's rules language reference) sets how many ranges fit in one rule. Google's quota page lists 10, which would cut a long list to about a third as many rules. Raise it only after an apply in your project has accepted a rule of that size; it has not been proven here. + +A list is limited to 2,500 ranges and 999 rules. + ## Chart side `gkeIngress.securityPolicy` puts the policy on the BackendConfig of every backend Service. The dashboard Service (and Relayfile with `gkeIngress.hosts.files`) needs its `cloud.google.com/backend-config` annotation; `helm template` fails with the exact value until it is set. See the chart README, "GKE edge hardening". -Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defaults use three, and seven with `cloudflare.enabled`. - ### Requirements @@ -149,6 +167,7 @@ Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defa | name | Name prefix: the SSL policy is -tls and the Cloud Armor policy -edge. | `string` | n/a | yes | | project\_id | Project of the GKE cluster. GKE Ingress resolves gkeIngress.sslPolicy and gkeIngress.securityPolicy by name in this project. | `string` | n/a | yes | | labels | Labels for every resource here that takes labels (the Cloud Armor policy; Compute SSL policies have none in the API). managed-by=terraform is always added. | `map(string)` | `{}` | no | +| max\_subexpressions | Subexpressions Cloud Armor accepts in one rule expression. The rules language reference says 5; Google's quota page lists 10. A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so a higher value needs fewer rules for the same list: with 10, about a third as many. Raise it only after an apply in your project has accepted a rule of that size. | `number` | `5` | no | | origin\_secret\_header | Optional shared-secret request header a fronting proxy adds on every request (for Cloudflare: a request header transform rule). Requests without the exact value are denied 403 before any other rule, which proves the request came through the proxy and makes its client-address header trustworthy. values lists every accepted value (1 to 3), so a rotation has no outage: add the new value here and apply, switch the proxy to it, then remove the old value and apply. The values are stored in Terraform state and readable in the Cloud Armor policy by anyone with compute.securityPolicies.get: treat both as secret stores. | ```object({ name = string values = list(string) })``` | `null` | no | | security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(login|(cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | | ssl\_policy | Compute SSL policy for the load balancer's HTTPS listener (chart value gkeIngress.sslPolicy). MODERN with TLS 1.2 serves current browsers, the desktop app and CLIs; RESTRICTED narrows the cipher list further (TLS 1.2 minimum is required by it). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | @@ -159,6 +178,7 @@ Advanced rules (`expr`) count against the project's Cloud Armor quotas: the defa | ---- | ----------- | | chart\_values | The gkeIngress keys to merge into the agent-relay values. With securityPolicy, also set the backend-config annotation the chart names on the relay-webapp (and relayfile) Service. | | client\_ip\_header | Header Cloud Armor reads the client address from (null: the connecting address). When set there is one more proxy in front of the load balancer: relay-control.auth.trustedProxies is one more than for the Google load balancer alone (2), so 3 (docs/gcp-install/3-EDGE-HARDENING.md). | +| rule\_counts | Rules the Cloud Armor policy needs, to compare with the project's quotas before applying: advanced counts rules with an expression (quota SECURITY\_POLICY\_CEVAL\_RULES, default 20 per project), total is every rule including the default (quota SECURITY\_POLICY\_RULES, default 100 per project). Zero without a policy. | | security\_policy\_name | Chart value gkeIngress.securityPolicy. Null when security\_policy.create is false. | | ssl\_policy\_name | Chart value gkeIngress.sslPolicy. Null when ssl\_policy.create is false. | diff --git a/terraform/modules/gcp/edge-security/main.tf b/terraform/modules/gcp/edge-security/main.tf index 3d885bc1..f54222ca 100644 --- a/terraform/modules/gcp/edge-security/main.tf +++ b/terraform/modules/gcp/edge-security/main.tf @@ -18,7 +18,7 @@ locals { # Basic rules take 10 ranges each; expressions take five subexpressions, so # an allowlist conjoined with a path match is cut into groups of four. allow_rules = chunklist(local.allowlist, 10) - allow_groups = chunklist(local.allowlist, 4) + allow_groups = chunklist(local.allowlist, var.max_subexpressions - 1) client_ip_header = local.sp.client_ip_header != null ? local.sp.client_ip_header : (local.sp.cloudflare.enabled ? "CF-Connecting-IP" : null) @@ -31,7 +31,7 @@ locals { # itself require an allowed connecting address. rate_rules = merge([for i, r in local.sp.rate_limits : { for j, group in(local.locked ? local.allow_groups : [[]]) : "${r.name}-${j}" => { - priority = 3000 + i * 100 + j + priority = 20000 + i * 1000 + j limit = r expression = join(" && ", compact(["request.path.matches('${replace(r.path_regex, "\\", "\\\\")}')", length(group) > 0 ? "(${join(" || ", [for c in group : "inIpRange(origin.ip, '${c}')"])})" : ""])) } @@ -42,22 +42,27 @@ locals { # and everyone else on those paths is denied. A Cloud Armor rule that admits # ends evaluation, so admitted requests meet no later rule: that is why each # rule states its own limit and why the secret header rule must come first. - client_groups = { for r in local.sp.client_ip_rules : r.name => chunklist(r.cidrs, length(r.hosts) > 0 ? 3 : 4) } + client_groups = { for r in local.sp.client_ip_rules : r.name => chunklist(r.cidrs, var.max_subexpressions - (length(r.hosts) > 0 ? 2 : 1)) } client_rules = merge([for i, r in local.sp.client_ip_rules : merge({ for j, group in local.client_groups[r.name] : "${r.name}-allow-${j}" => { - priority = 2000 + i * 100 + j + priority = 10000 + i * 1000 + j action = r.rate_limit == null ? "allow" : "throttle" rate = { count = try(r.rate_limit.count, 0), interval_sec = try(r.rate_limit.interval_sec, 0) } description = "${r.name}: listed client addresses on ${join(", ", concat(r.hosts, r.path_prefixes))}" expression = "${local.path_match[r.name]} && (${join(" || ", [for c in group : "inIpRange(origin.user_ip, '${c}')"])})" } }, { "${r.name}-deny" = { - priority = 2000 + i * 100 + 99 + priority = 10000 + i * 1000 + 999 action = "deny(403)" rate = { count = 0, interval_sec = 0 } description = "${r.name}: any other client on ${join(", ", concat(r.hosts, r.path_prefixes))}" expression = local.path_match[r.name] } })]...) + # What the policy costs against the project's Cloud Armor quotas. + rule_counts = { + advanced = (local.secret_header ? 1 : 0) + length(local.waf_groups) + length(local.client_rules) + length(local.rate_rules) + basic = length(local.allow_rules) + 1 + } # The path (one subexpression) and, with hosts, the Host header (a second), # which leaves three or four subexpressions for client ranges. path_match = { for r in local.sp.client_ip_rules : r.name => join(" && ", compact([ @@ -102,6 +107,10 @@ resource "google_compute_security_policy" "this" { condition = length(local.sp.client_ip_rules) == 0 || (local.secret_header && local.client_ip_header != null) error_message = "client_ip_rules trust a client-address header, so they need origin_secret_header (proof the request came through the proxy) and client_ip_header or cloudflare.enabled." } + precondition { + condition = alltrue([for r in local.sp.client_ip_rules : length(local.client_groups[r.name]) <= 999]) && length(local.allow_groups) <= 999 + error_message = "A list needs more than 999 rules at this max_subexpressions. Shorten it, or enforce it at the fronting proxy." + } precondition { condition = alltrue([for r in local.sp.client_ip_rules : r.rate_limit != null || r.unthrottled]) error_message = "Each client_ip_rules entry admits its clients, and admitted requests skip rate_limits. Give the rule its own rate_limit, or set unthrottled = true to accept that it has none." @@ -198,7 +207,7 @@ resource "google_compute_security_policy_rule" "allowlist" { for_each = local.armor ? { for i, group in local.allow_rules : tostring(i) => group } : {} project = var.project_id security_policy = google_compute_security_policy.this[0].name - priority = 9000 + tonumber(each.key) + priority = 30000 + tonumber(each.key) action = "allow" description = "Allowed connecting addresses (${local.sp.cloudflare.enabled ? "Cloudflare and " : ""}source_ip_allowlist)" match { diff --git a/terraform/modules/gcp/edge-security/outputs.tf b/terraform/modules/gcp/edge-security/outputs.tf index d58c3d27..af6f760a 100644 --- a/terraform/modules/gcp/edge-security/outputs.tf +++ b/terraform/modules/gcp/edge-security/outputs.tf @@ -20,3 +20,11 @@ output "client_ip_header" { description = "Header Cloud Armor reads the client address from (null: the connecting address). When set there is one more proxy in front of the load balancer: relay-control.auth.trustedProxies is one more than for the Google load balancer alone (2), so 3 (docs/gcp-install/3-EDGE-HARDENING.md)." value = local.armor ? local.client_ip_header : null } + +output "rule_counts" { + description = "Rules the Cloud Armor policy needs, to compare with the project's quotas before applying: advanced counts rules with an expression (quota SECURITY_POLICY_CEVAL_RULES, default 20 per project), total is every rule including the default (quota SECURITY_POLICY_RULES, default 100 per project). Zero without a policy." + value = local.armor ? { + advanced = local.rule_counts.advanced + total = local.rule_counts.advanced + local.rule_counts.basic + } : { advanced = 0, total = 0 } +} diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index ad09f92e..2dedb88f 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -333,3 +333,68 @@ run "rate_limit_regex_with_an_escape" { error_message = "The backslash is doubled inside the quoted expression string." } } + +# The policy reports what it costs against the project's rule quotas, and a +# higher max_subexpressions packs a long client list into fewer rules. +run "rule_counts_and_wider_expressions" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(24) : "198.51.${i}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + # 24 ranges at 3 per rule = 8 admitting rules + 1 deny; + secret header + 2 WAF = 12 advanced; + 2 allowlist + default = 15. + assert { + condition = length(google_compute_security_policy_rule.client_ip) == 9 && output.rule_counts.advanced == 12 && output.rule_counts.total == 15 + error_message = "rule_counts must equal the rules created." + } + assert { + condition = output.rule_counts.total == length(google_compute_security_policy_rule.client_ip) + length(google_compute_security_policy_rule.waf) + length(google_compute_security_policy_rule.origin_secret) + length(google_compute_security_policy_rule.rate_limit) + length(google_compute_security_policy_rule.allowlist) + length(google_compute_security_policy_rule.default) + error_message = "total counts every rule resource." + } +} + +run "ten_subexpressions_need_a_third_of_the_rules" { + command = apply + variables { + max_subexpressions = 10 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(24) : "198.51.${i}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + # 8 ranges per rule (10 - host - path): 3 admitting rules + 1 deny. + assert { + condition = length(google_compute_security_policy_rule.client_ip) == 4 && output.rule_counts.advanced == 7 && length(regexall("inIpRange", google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression)) == 8 + error_message = "With 10 subexpressions a host-scoped rule carries 8 ranges." + } +} + +run "policy_off_reports_zero_rules" { + command = apply + assert { + condition = output.rule_counts.advanced == 0 && output.rule_counts.total == 0 + error_message = "No policy, no rules." + } +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index 63175805..abb91aa2 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -120,12 +120,12 @@ variable "security_policy" { } validation { condition = alltrue([for r in var.security_policy.client_ip_rules : - can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && length(r.cidrs) >= 1 && length(r.cidrs) <= 294 && + can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && length(r.cidrs) >= 1 && length(r.cidrs) <= 2500 && length(r.path_prefixes) >= 1 && alltrue([for p in r.path_prefixes : can(regex("^/[A-Za-z0-9/._~-]*$", p))]) && alltrue([for h in r.hosts : can(regex("^[a-z0-9]([a-z0-9.-]*[a-z0-9])?$", h))]) && (r.rate_limit == null ? true : r.rate_limit.count >= 1 && contains([10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700, 3600], r.rate_limit.interval_sec)) ]) && length(var.security_policy.client_ip_rules) <= 9 && length(distinct(var.security_policy.client_ip_rules[*].name)) == length(var.security_policy.client_ip_rules) - error_message = "At most 9 client_ip_rules with unique lowercase names, 1-294 cidrs, path_prefixes that start with / and hold only unreserved URL characters, lowercase DNS hosts, and a rate_limit with count >= 1 and an interval_sec Cloud Armor accepts." + error_message = "At most 9 client_ip_rules with unique lowercase names, 1-2500 cidrs, path_prefixes that start with / and hold only unreserved URL characters, lowercase DNS hosts, and a rate_limit with count >= 1 and an interval_sec Cloud Armor accepts." } validation { condition = var.security_policy.client_ip_header == null || can(regex("^[A-Za-z][A-Za-z0-9-]{0,63}$", var.security_policy.client_ip_header)) @@ -147,3 +147,13 @@ variable "origin_secret_header" { error_message = "origin_secret_header.name is a lowercase header name; values holds 1-3 distinct values of 32-256 characters from A-Z a-z 0-9 _ - (for example: openssl rand -hex 32)." } } + +variable "max_subexpressions" { + description = "Subexpressions Cloud Armor accepts in one rule expression. The rules language reference says 5; Google's quota page lists 10. A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so a higher value needs fewer rules for the same list: with 10, about a third as many. Raise it only after an apply in your project has accepted a rule of that size." + type = number + default = 5 + validation { + condition = contains([5, 6, 7, 8, 9, 10], var.max_subexpressions) + error_message = "max_subexpressions is 5 to 10." + } +} From b39e76ab75b2f3a241cfc10ef68eb16be80de3f8 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 11:42:57 +0100 Subject: [PATCH 02/10] fix(edge-security): non-capturing regex groups (Cloud Armor rejects capture groups); 10 subexpressions Measured against the Cloud Armor API by the live GCP proof: - BLOCKER in the merged module: every client_ip_rules expression used request.path.matches('^(a|b)') and a host pattern with groups. Cloud Armor refuses them: "Invalid regular expression ... Capture Groups are not allowed", so each client allow and deny rule would fail at apply. All generated patterns now use (?:...), the form the API accepted. The default sign-in rate-limit regex had the same problem and is fixed, and a capture group in rate_limits[*].path_regex is refused at plan. A test asserts no generated expression contains one. - The API accepts 10 subexpressions per expression and refuses 11. max_subexpressions now defaults to 10: nine client ranges per rule, eight with hosts. A 290-range vendor list needs about 45 advanced rules instead of about 105; still above the default quota of 20, so the quota request stays a documented prerequisite. - Guide, empty-project path: what to do when the organization forbids the default Compute Engine service account for nodes. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/0-START-HERE.md | 15 ++++ docs/gcp-install/3-EDGE-HARDENING.md | 15 ++-- terraform/examples/gcp/README.md | 2 +- .../examples/gcp/tests/native.tftest.hcl | 6 ++ terraform/examples/gcp/variables.tf | 4 +- terraform/modules/gcp/edge-security/README.md | 23 +++--- terraform/modules/gcp/edge-security/main.tf | 10 ++- .../tests/edge_security.tftest.hcl | 73 +++++++++++++++++-- .../modules/gcp/edge-security/variables.tf | 10 +-- 9 files changed, 122 insertions(+), 36 deletions(-) diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index 58c6c76b..dc4d9430 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -1158,6 +1158,21 @@ gcloud container clusters create agent-relay --location $R --num-nodes 1 \ --enable-shielded-nodes --enable-dataplane-v2 ``` +**`Operation denied by org policy ... default Compute Engine service +account`?** Your organization does not allow nodes to run as the default +service account. Create one for the nodes and pass it: + +```sh +gcloud iam service-accounts create agent-relay-nodes +gcloud projects add-iam-policy-binding $P --role roles/container.defaultNodeServiceAccount \ + --member "serviceAccount:agent-relay-nodes@$P.iam.gserviceaccount.com" +``` + +then add `--service-account agent-relay-nodes@$P.iam.gserviceaccount.com` to +the `clusters create` command, and in step 4 set +`gke_node_service_account = "agent-relay-nodes@.iam.gserviceaccount.com"` +so the image mirror grants it read access. + A regional cluster with `--num-nodes 1` gets one node in each of the region's three zones. Then continue at step 1 with `` = `agent-relay` and `` = `agent-relay-vpc`, and grant the node service account in diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 353fea0d..28f42450 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -239,15 +239,14 @@ Cloudflare can reach it. sign-in limit is a rule of its own ahead of the closing one. The module's README (`terraform/modules/gcp/edge-security`) explains the limits. - **Check the quota before you apply a vendor list.** Each rule takes three - or four ranges, so a vendor that publishes about 290 ranges needs about - 105 rules with an expression, against a default of 20 per project (and 100 + **Check the quota before you apply a vendor list.** Each rule takes eight + or nine ranges, so a vendor that publishes about 290 ranges needs about + 45 rules with an expression, against a default of 20 per project (and 100 rules in all). `terraform plan` shows the numbers in the `edge` output's - `rule_counts`. Request `SECURITY_POLICY_CEVAL_RULES` and - `SECURITY_POLICY_RULES` increases for the Compute Engine API first (allow - headroom for the vendor adding ranges), or enforce that list as a - Cloudflare WAF custom rule and keep only your own ranges here. Without - either, the apply fails part-way with a quota error. + `rule_counts`. Request a `SECURITY_POLICY_CEVAL_RULES` increase for the + Compute Engine API first (allow headroom for the vendor adding ranges), or + enforce that list as a Cloudflare WAF custom rule and keep only your own + ranges here. Without either, the apply fails part-way with a quota error. **Done when:** diff --git a/terraform/examples/gcp/README.md b/terraform/examples/gcp/README.md index 6eff78c5..6314280a 100644 --- a/terraform/examples/gcp/README.md +++ b/terraform/examples/gcp/README.md @@ -100,7 +100,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | deletion\_protection | Prevent deletion of Cloud SQL, Memorystore and Pub/Sub data resources. Disable only for a disposable proof environment. | `bool` | `true` | no | | desktop\_feed\_bucket\_name | Globally unique bucket for relay-control's signed Desktop feed (SELFHOST\_DESKTOP\_FEED\_GCS\_BUCKET). Empty disables it. | `string` | `""` | no | | desktop\_feed\_service\_account\_id | account\_id of the Google service account relay-control uses for the Desktop feed bucket (6-30 characters). | `string` | `"agent-relay-control"` | no | -| edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 5 is the documented limit; a higher value packs long client lists into fewer rules once an apply in your project has accepted it. | `number` | `5` | no | +| edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule. | `number` | `10` | no | | edge\_origin\_secret\_header | Shared-secret request header a fronting proxy adds; requests without one of the accepted values (1-3, for rotation) are denied (modules/gcp/edge-security origin\_secret\_header). Stored in state. | ```object({ name = string values = list(string) })``` | `null` | no | | edge\_security\_policy | Cloud Armor policy for every GKE Ingress backend (modules/gcp/edge-security security\_policy: create, waf, rate\_limits, source\_ip\_allowlist, cloudflare, client\_ip\_header, client\_ip\_rules). Off unless create = true. It is exported as gkeIngress.securityPolicy in output.edge, not in chart\_values, because the chart then also needs the dashboard Service's backend-config annotation. | `any` | `{}` | no | | edge\_ssl\_policy | Compute SSL policy for the GKE Ingress HTTPS listener, exported as gkeIngress.sslPolicy in chart\_values. create = false leaves Google's default policy (TLS 1.0 and later). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | diff --git a/terraform/examples/gcp/tests/native.tftest.hcl b/terraform/examples/gcp/tests/native.tftest.hcl index e03db5ed..f715919f 100644 --- a/terraform/examples/gcp/tests/native.tftest.hcl +++ b/terraform/examples/gcp/tests/native.tftest.hcl @@ -236,6 +236,12 @@ run "edge_cloud_armor_behind_cloudflare" { condition = output.edge.security_policy == "agent-relay-edge" && output.edge.client_ip_header == "CF-Connecting-IP" error_message = "The Cloud Armor policy and the client-address header must be reported." } + # Default of 10 subexpressions: secret header, two WAF rules, and two sign-in + # throttles for Cloudflare's 15 ranges (nine per rule). + assert { + condition = output.edge.rule_counts.advanced == 5 && output.edge.rule_counts.total == 8 + error_message = "Behind Cloudflare the default policy needs 5 advanced rules and 8 in all." + } } run "edge_rule_counts_are_reported" { diff --git a/terraform/examples/gcp/variables.tf b/terraform/examples/gcp/variables.tf index 18784162..e3390f98 100644 --- a/terraform/examples/gcp/variables.tf +++ b/terraform/examples/gcp/variables.tf @@ -255,7 +255,7 @@ variable "edge_origin_secret_header" { } variable "edge_max_subexpressions" { - description = "Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max_subexpressions). 5 is the documented limit; a higher value packs long client lists into fewer rules once an apply in your project has accepted it." + description = "Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule." type = number - default = 5 + default = 10 } diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index e0597b8a..f8fb907b 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -26,7 +26,7 @@ Rules are evaluated in priority order and the first match decides. A rule that a **The WAF rules start in preview.** Session transcripts, messages and flows carry code, SQL and shell text, which is what the OWASP signatures look for. Preview logs each match without blocking. Read the load balancer's request log (`jsonPayload.previewSecurityPolicy`) for a week of normal use, then set `waf.preview = false`, or remove the rule names that only match your own traffic from `waf.rules`. -A throttle rule admits the requests under its limit, so when an allowlist is set each throttle rule also requires an allowlisted connecting address. Cloud Armor expressions take five subexpressions, which is why an allowlist appears there in groups of four. +A throttle rule admits the requests under its limit, so when an allowlist is set each throttle rule also requires an allowlisted connecting address. Cloud Armor expressions take ten subexpressions, which is why an allowlist appears there in groups of nine. ### Only some addresses may connect @@ -106,7 +106,7 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } - Put narrower rules first. Repeat your own ranges (for example your VPN's egress range) in a vendor rule: your users' desktop uploads and MCP clients use those paths too, and a rule denies everyone it does not list. - Without the closing `["/"]` rule, paths that no rule names stay open to every client that comes through the proxy (and `rate_limits` still apply to them). -- Each rule costs one advanced rule per four client ranges (three with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. +- Each rule costs one advanced rule per nine client ranges (eight with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. - No rule in this module can contain `0.0.0.0/0` or `::/0`: validation refuses them in every list. The only "all addresses" match is Cloud Armor's mandatory default rule, which is deny whenever an allowlist is set. ## Quotas: check before you apply @@ -118,16 +118,17 @@ The `rule_counts` output says what this policy needs, and `terraform plan` shows | Configuration | Advanced rules | Rules in all | | --- | --- | --- | | `create = true` (defaults) | 3 | 4 | -| `cloudflare.enabled` with `origin_secret_header` | 7 | 10 | -| plus a host-scoped client rule with 290 vendor ranges | about 105 | about 108 | +| `cloudflare.enabled` with `origin_secret_header` | 5 | 8 | +| plus a host-scoped client rule with 290 vendor ranges | about 45 | about 48 | -A long vendor list therefore fails on both quotas in a fresh project. Either: +A long vendor list therefore fails on the first quota in a fresh project. Either: -- **request the quotas first** (Console, IAM and Admin, Quotas: `SECURITY_POLICY_CEVAL_RULES` and `SECURITY_POLICY_RULES` for the Compute Engine API, with the numbers from `rule_counts` plus headroom for the vendor adding ranges), and apply once Google grants them; or -- **enforce that list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address) and keep only your own short ranges in `client_ip_rules`; or -- **use wider expressions**: `max_subexpressions` (default 5, the limit in Google's rules language reference) sets how many ranges fit in one rule. Google's quota page lists 10, which would cut a long list to about a third as many rules. Raise it only after an apply in your project has accepted a rule of that size; it has not been proven here. +- **request the quota first** (Console, IAM and Admin, Quotas: `SECURITY_POLICY_CEVAL_RULES` for the Compute Engine API, and `SECURITY_POLICY_RULES` if `total` nears 100, with the numbers from `rule_counts` plus headroom for the vendor adding ranges), and apply once Google grants it; or +- **enforce that list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address) and keep only your own short ranges in `client_ip_rules`. -A list is limited to 2,500 ranges and 999 rules. +One rule expression holds 10 subexpressions (`max_subexpressions`; the API accepted 10 and refused 11 when measured, although Google's rules language reference still says 5). A rule spends one on the path, one on the host when `hosts` is set, and the rest on client ranges: 8 or 9 ranges per rule. A list is limited to 2,500 ranges and 999 rules. + +Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` with "Capture Groups are not allowed". The module writes `(?:a|b)` in everything it generates and refuses a capture group in `rate_limits[*].path_regex`. ## Chart side @@ -167,9 +168,9 @@ A list is limited to 2,500 ranges and 999 rules. | name | Name prefix: the SSL policy is -tls and the Cloud Armor policy -edge. | `string` | n/a | yes | | project\_id | Project of the GKE cluster. GKE Ingress resolves gkeIngress.sslPolicy and gkeIngress.securityPolicy by name in this project. | `string` | n/a | yes | | labels | Labels for every resource here that takes labels (the Cloud Armor policy; Compute SSL policies have none in the API). managed-by=terraform is always added. | `map(string)` | `{}` | no | -| max\_subexpressions | Subexpressions Cloud Armor accepts in one rule expression. The rules language reference says 5; Google's quota page lists 10. A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so a higher value needs fewer rules for the same list: with 10, about a third as many. Raise it only after an apply in your project has accepted a rule of that size. | `number` | `5` | no | +| max\_subexpressions | Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule. | `number` | `10` | no | | origin\_secret\_header | Optional shared-secret request header a fronting proxy adds on every request (for Cloudflare: a request header transform rule). Requests without the exact value are denied 403 before any other rule, which proves the request came through the proxy and makes its client-address header trustworthy. values lists every accepted value (1 to 3), so a rotation has no outage: add the new value here and apply, switch the proxy to it, then remove the old value and apply. The values are stored in Terraform state and readable in the Cloud Armor policy by anyone with compute.securityPolicies.get: treat both as secret stores. | ```object({ name = string values = list(string) })``` | `null` | no | -| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(login|(cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | +| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | | ssl\_policy | Compute SSL policy for the load balancer's HTTPS listener (chart value gkeIngress.sslPolicy). MODERN with TLS 1.2 serves current browsers, the desktop app and CLIs; RESTRICTED narrows the cipher list further (TLS 1.2 minimum is required by it). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | ### Outputs diff --git a/terraform/modules/gcp/edge-security/main.tf b/terraform/modules/gcp/edge-security/main.tf index f54222ca..66d5e87d 100644 --- a/terraform/modules/gcp/edge-security/main.tf +++ b/terraform/modules/gcp/edge-security/main.tf @@ -63,11 +63,13 @@ locals { advanced = (local.secret_header ? 1 : 0) + length(local.waf_groups) + length(local.client_rules) + length(local.rate_rules) basic = length(local.allow_rules) + 1 } - # The path (one subexpression) and, with hosts, the Host header (a second), - # which leaves three or four subexpressions for client ranges. + # The path (one subexpression) and, with hosts, the Host header (a second); + # the rest of max_subexpressions holds client ranges. Groups are + # non-capturing: Cloud Armor rejects a regular expression with a capture + # group ("Capture Groups are not allowed"). path_match = { for r in local.sp.client_ip_rules : r.name => join(" && ", compact([ - length(r.hosts) > 0 ? "request.headers['host'].lower().matches('^(${join("|", [for h in r.hosts : replace(h, ".", "\\\\.")])})(:[0-9]+)?$')" : "", - "request.path.matches('^(${join("|", [for p in r.path_prefixes : replace(p, ".", "\\\\.")])})')", + length(r.hosts) > 0 ? "request.headers['host'].lower().matches('^(?:${join("|", [for h in r.hosts : replace(h, ".", "\\\\.")])})(?::[0-9]+)?$')" : "", + "request.path.matches('^(?:${join("|", [for p in r.path_prefixes : replace(p, ".", "\\\\.")])})')", ])) } } diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index 2dedb88f..2614d89e 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -3,6 +3,9 @@ mock_provider "google" {} variables { project_id = "proj" name = "relay" + # Most runs pin 5 subexpressions per rule so that small lists exercise the + # splitting into several rules; the module's default (10) has its own runs. + max_subexpressions = 5 } run "defaults_create_only_the_tls_policy" { @@ -70,7 +73,7 @@ run "waf_and_rate_limit_defaults" { error_message = "The default rules include cve-canary (Log4Shell)." } assert { - condition = keys(google_compute_security_policy_rule.rate_limit) == ["sign-in-0"] && google_compute_security_policy_rule.rate_limit["sign-in-0"].action == "throttle" && google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(login|(cloud/)?api/v1/auth/)')" + condition = keys(google_compute_security_policy_rule.rate_limit) == ["sign-in-0"] && google_compute_security_policy_rule.rate_limit["sign-in-0"].action == "throttle" && google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(?:login|(?:cloud/)?api/v1/auth/)')" error_message = "One throttle rule on the sign-in paths." } assert { @@ -146,7 +149,7 @@ run "cloudflare_origin_lockdown" { error_message = "Rate limits stay inside the allowlist and count per client address." } assert { - condition = google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(login|(cloud/)?api/v1/auth/)') && (inIpRange(origin.ip, '173.245.48.0/20') || inIpRange(origin.ip, '103.21.244.0/22') || inIpRange(origin.ip, '103.22.200.0/22') || inIpRange(origin.ip, '103.31.4.0/22'))" + condition = google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression == "request.path.matches('^/(?:login|(?:cloud/)?api/v1/auth/)') && (inIpRange(origin.ip, '173.245.48.0/20') || inIpRange(origin.ip, '103.21.244.0/22') || inIpRange(origin.ip, '103.22.200.0/22') || inIpRange(origin.ip, '103.31.4.0/22'))" error_message = "Five subexpressions at most: the path and four ranges." } assert { @@ -154,7 +157,7 @@ run "cloudflare_origin_lockdown" { error_message = "Five client ranges: two allow rules and the path deny." } assert { - condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(/v1/delivery|/v1/turns)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(/v1/delivery|/v1/turns)')" + condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)')" error_message = "Client rules match the path and origin.user_ip, then deny the path." } # Order: secret header, WAF, client rules, rate limits, allowlist, default. @@ -264,7 +267,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "Every rule admits its clients (four ranges per rule, three with hosts) and then denies its paths." } assert { - condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(history\\\\.relay\\\\.example\\\\.com)(:[0-9]+)?$') && request.path.matches('^(/v1/delivery|/v1/turns|/v1/auth/token/refresh)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" + condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/v1/turns|/v1/auth/token/refresh)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" error_message = "Host, path and the client ranges, within five subexpressions." } # A short vendor list ahead of a staff-only "/" rule: the vendor is admitted @@ -280,7 +283,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "A one-range rule with a rate_limit throttles per client address." } assert { - condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^(/)')" + condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/)')" error_message = "An unthrottled rule admits with allow; the closing rule denies every remaining path." } assert { @@ -398,3 +401,63 @@ run "policy_off_reports_zero_rules" { error_message = "No policy, no rules." } } + +# A capture group in a user regex is refused at plan: Cloud Armor rejects it at apply. +run "capture_group_in_path_regex_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "auth", path_regex = "^/(login|setup)", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} + +run "non_capturing_and_escaped_parentheses_accepted" { + command = apply + variables { + security_policy = { create = true, waf = { enabled = false }, rate_limits = [{ name = "auth", path_regex = "^/(?:login|setup)\\(x\\)", count = 10, interval_sec = 60 }] } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 1 + error_message = "(?:...) groups and escaped parentheses are allowed." + } +} + +# No expression the module builds contains a capture group. +run "generated_expressions_have_no_capture_groups" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + client_ip_rules = [ + { name = "vendors", hosts = ["history.relay.example.com", "api.relay.example.com"], path_prefixes = ["/v1/delivery", "/.well-known"], cidrs = ["192.0.2.0/24"], unthrottled = true }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = alltrue([for e in concat( + [for r in google_compute_security_policy_rule.client_ip : r.match[0].expr[0].expression], + [for r in google_compute_security_policy_rule.rate_limit : r.match[0].expr[0].expression], + ) : length(regexall("matches\\('[^']*\\((?:[^?]|$)", e)) == 0]) + error_message = "Every group inside a matches() pattern must be non-capturing." + } + assert { + condition = google_compute_security_policy_rule.client_ip["vendors-deny"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com|api\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/\\\\.well-known)')" + error_message = "Host and path patterns in the form the API accepted." + } +} + +# The default: 10 subexpressions. Cloudflare's 15 ranges need two throttle rules, not four. +run "default_is_ten_subexpressions" { + command = apply + variables { + max_subexpressions = 10 + security_policy = { create = true, cloudflare = { enabled = true } } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 2 && length(regexall("inIpRange", google_compute_security_policy_rule.rate_limit["sign-in-0"].match[0].expr[0].expression)) == 9 && output.rule_counts.advanced == 5 + error_message = "Path plus nine ranges per throttle rule." + } +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index abb91aa2..84ab384a 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -67,7 +67,7 @@ variable "security_policy" { ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" - path_regex = "^/(login|(cloud/)?api/v1/auth/)" + path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) @@ -113,10 +113,10 @@ variable "security_policy" { } validation { condition = alltrue([for r in var.security_policy.rate_limits : - can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && length(r.path_regex) <= 512 && + can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && !strcontains(replace(replace(r.path_regex, "(?:", ""), "\\(", ""), "(") && length(r.path_regex) <= 512 && r.count >= 1 && contains([10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700, 3600], r.interval_sec) && r.ban_duration_sec >= 0 ]) && length(var.security_policy.rate_limits) <= 9 && length(distinct(var.security_policy.rate_limits[*].name)) == length(var.security_policy.rate_limits) - error_message = "At most 9 rate_limits with unique lowercase names, a path_regex without single quotes, count >= 1 and an interval_sec Cloud Armor accepts (10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700 or 3600)." + error_message = "At most 9 rate_limits with unique lowercase names, a path_regex without single quotes or capture groups (Cloud Armor rejects them: write (?:a|b), not (a|b)), count >= 1 and an interval_sec Cloud Armor accepts (10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700 or 3600)." } validation { condition = alltrue([for r in var.security_policy.client_ip_rules : @@ -149,9 +149,9 @@ variable "origin_secret_header" { } variable "max_subexpressions" { - description = "Subexpressions Cloud Armor accepts in one rule expression. The rules language reference says 5; Google's quota page lists 10. A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so a higher value needs fewer rules for the same list: with 10, about a third as many. Raise it only after an apply in your project has accepted a rule of that size." + description = "Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule." type = number - default = 5 + default = 10 validation { condition = contains([5, 6, 7, 8, 9, 10], var.max_subexpressions) error_message = "max_subexpressions is 5 to 10." From 9e22f5aa6c958ad406ce313fa408671d6f97605e Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 11:44:16 +0100 Subject: [PATCH 03/10] test(edge-security): a 300-range list keeps its priority band and order Review feedback on #487: the list fixtures had 24 ranges and would still pass with the old 294-range cap or 100-wide priority spacing. Adds a host-scoped list of 300 ranges (100 admitting rules) followed by another client rule, asserting unique priorities, every allow before its deny, the deny before the next rule, and the reported rule count. Co-Authored-By: Claude Fable 5.2 --- .../tests/edge_security.tftest.hcl | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index 2614d89e..89111792 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -461,3 +461,50 @@ run "default_is_ten_subexpressions" { error_message = "Path plus nine ranges per throttle rule." } } + +# A list longer than the old cap (294 ranges, 99 rules per list): 300 ranges +# at three per rule is 100 admitting rules. Its rules must stay in one band, +# in order, ahead of the next client rule. +run "long_list_keeps_its_band_and_order" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + client_ip_rules = [ + { + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(300) : "10.${floor(i / 256)}.${i % 256}.0/24"] + unthrottled = true + }, + { + name = "staff" + path_prefixes = ["/"] + cidrs = ["203.0.113.0/24"] + unthrottled = true + }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + + assert { + condition = length([for k, r in google_compute_security_policy_rule.client_ip : k if startswith(k, "vendors-allow-")]) == 100 + error_message = "300 ranges at three per rule are 100 admitting rules." + } + assert { + condition = length(distinct([for r in google_compute_security_policy_rule.client_ip : r.priority])) == length(google_compute_security_policy_rule.client_ip) + error_message = "No two client rules share a priority." + } + assert { + condition = max([for k, r in google_compute_security_policy_rule.client_ip : r.priority if startswith(k, "vendors-allow-")]...) < google_compute_security_policy_rule.client_ip["vendors-deny"].priority && google_compute_security_policy_rule.client_ip["vendors-deny"].priority < google_compute_security_policy_rule.client_ip["staff-allow-0"].priority && google_compute_security_policy_rule.client_ip["staff-allow-0"].priority < google_compute_security_policy_rule.client_ip["staff-deny"].priority + error_message = "Every admitting rule precedes its deny, and the deny precedes the next client rule." + } + assert { + condition = max([for r in google_compute_security_policy_rule.client_ip : r.priority]...) < min([for r in google_compute_security_policy_rule.allowlist : r.priority]...) && output.rule_counts.advanced == 106 + error_message = "Client rules stay ahead of the allowlist, and the count is reported (100 + 2 denies + 1 staff allow + secret header + 2 WAF)." + } +} From 41e9acbe0ccffa1ec0e7957d700107fb47d2e7e8 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 12:00:24 +0100 Subject: [PATCH 04/10] fix(edge-security): refuse only real capture groups in path_regex Review feedback on #487: the check refused any remaining opening parenthesis, so valid patterns such as (?i)^/login and ^/file[(]old[)]$ failed plan. It now drops escaped characters and character classes first and refuses only a group that captures: "(" not followed by "?", or a named group. Tests cover flags, bracketed and escaped parentheses, a named group and a capture group after a class. Co-Authored-By: Claude Fable 5.2 --- .../tests/edge_security.tftest.hcl | 36 +++++++++++++++++++ .../modules/gcp/edge-security/variables.tf | 5 ++- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index 89111792..9bf114e7 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -508,3 +508,39 @@ run "long_list_keeps_its_band_and_order" { error_message = "Client rules stay ahead of the allowlist, and the count is reported (100 + 2 denies + 1 staff allow + secret header + 2 WAF)." } } + +# Patterns without a capture group that contain other parentheses are accepted. +run "flags_and_bracketed_parentheses_accepted" { + command = apply + variables { + security_policy = { + create = true + waf = { enabled = false } + rate_limits = [ + { name = "flag", path_regex = "(?i)^/login", count = 10, interval_sec = 60 }, + { name = "class", path_regex = "^/file[(]old[)]$", count = 10, interval_sec = 60 }, + { name = "escaped", path_regex = "^/a\\(b\\)(?:c|d)", count = 10, interval_sec = 60 }, + ] + } + } + assert { + condition = length(google_compute_security_policy_rule.rate_limit) == 3 + error_message = "Flags, parentheses in a character class and escaped parentheses are not capture groups." + } +} + +run "named_capture_group_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "named", path_regex = "^/(?Pv1|v2)/", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} + +run "capture_group_after_a_class_refused" { + command = plan + variables { + security_policy = { create = true, rate_limits = [{ name = "mixed", path_regex = "^/[(]x[)](a|b)", count = 10, interval_sec = 60 }] } + } + expect_failures = [var.security_policy] +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index 84ab384a..073d87d6 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -111,9 +111,12 @@ variable "security_policy" { ) : can(cidrhost(c, 0)) && !contains(["0.0.0.0/0", "::/0"], c)]) error_message = "Every allowlist entry must be a CIDR range, and never 0.0.0.0/0 or ::/0." } + # Capture groups: after dropping escaped characters and character classes, + # a "(" that is not followed by "?" opens one, and so does a named group + # ((?P or (?). Flags such as (?i) and (?:...) are fine. validation { condition = alltrue([for r in var.security_policy.rate_limits : - can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && !strcontains(replace(replace(r.path_regex, "(?:", ""), "\\(", ""), "(") && length(r.path_regex) <= 512 && + can(regex("^[a-z0-9]([-a-z0-9]{0,30}[a-z0-9])?$", r.name)) && !strcontains(r.path_regex, "'") && length(regexall("\\((?:[^?]|$)|\\(\\?P?<[A-Za-z_]", replace(replace(r.path_regex, "/\\\\./", ""), "/\\[[^\\]]*\\]/", ""))) == 0 && length(r.path_regex) <= 512 && r.count >= 1 && contains([10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700, 3600], r.interval_sec) && r.ban_duration_sec >= 0 ]) && length(var.security_policy.rate_limits) <= 9 && length(distinct(var.security_policy.rate_limits[*].name)) == length(var.security_policy.rate_limits) error_message = "At most 9 rate_limits with unique lowercase names, a path_regex without single quotes or capture groups (Cloud Armor rejects them: write (?:a|b), not (a|b)), count >= 1 and an interval_sec Cloud Armor accepts (10, 30, 60, 120, 180, 240, 300, 600, 900, 1200, 1800, 2700 or 3600)." From 43a2e12406c5f6d8dfb48b445dce4f0b8b53431c Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 12:08:29 +0100 Subject: [PATCH 05/10] docs(edge-security): the quota error text, and never attach a policy after a failed apply Measured by the live proof at b39e76ab: plan gives no quota warning; apply fails with Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded (limit 20) and leaves an arbitrary subset of rules, including a sign-in deny without its allow. The README and guide now quote the error and say to attach the policy only after an apply that exits cleanly. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 13 ++++++++++--- terraform/modules/gcp/edge-security/README.md | 8 ++++++++ 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 28f42450..4fbcaabf 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -81,8 +81,8 @@ when you run them. Run `terraform plan` first and read `rule_counts` in the `edge` output: a new project allows 20 rules with an expression and 100 rules in all, and the plan tells you what this policy needs (the defaults need 3 and 4). - Then `terraform apply`. `terraform output edge` prints the policy's name - (`-edge`). This policy has the OWASP rules in **preview** (they log + Then `terraform apply`, and continue only if it finishes without an + error. `terraform output edge` prints the policy's name (`-edge`). This policy has the OWASP rules in **preview** (they log and do not block) and a limit of 60 sign-in requests a minute per client. 2. Name it in `values.yaml`, and bind the dashboard Service to its @@ -246,7 +246,14 @@ Cloudflare can reach it. `rule_counts`. Request a `SECURITY_POLICY_CEVAL_RULES` increase for the Compute Engine API first (allow headroom for the vendor adding ranges), or enforce that list as a Cloudflare WAF custom rule and keep only your own - ranges here. Without either, the apply fails part-way with a quota error. + ranges here. Without either, the apply fails part-way with + `Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally`. + + **If an apply fails, do not use the policy yet.** It then holds an + arbitrary part of its rules, for example the rule that denies the sign-in + paths without the rule that admits you. Raise the quota or shorten the + list, and run `terraform apply` again until it exits without an error. + Only then set `gkeIngress.securityPolicy`. **Done when:** diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index f8fb907b..a9d7ed5c 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -126,6 +126,14 @@ A long vendor list therefore fails on the first quota in a fresh project. Either - **request the quota first** (Console, IAM and Admin, Quotas: `SECURITY_POLICY_CEVAL_RULES` for the Compute Engine API, and `SECURITY_POLICY_RULES` if `total` nears 100, with the numbers from `rule_counts` plus headroom for the vendor adding ranges), and apply once Google grants it; or - **enforce that list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address) and keep only your own short ranges in `client_ip_rules`. +`terraform plan` does not check the quota; the limit is met only at apply, which then fails with: + +``` +Error waiting to create SecurityPolicyRule: ... Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally. +``` + +**A failed apply leaves a partial policy. Do not attach it.** Terraform creates rules in parallel, so an arbitrary subset exists afterwards: in the measured case a deny rule for the sign-in paths without the rule that admits its clients, and 14 of 37 vendor rules. Attached like that, sign-in and most vendors get 403. Set `gkeIngress.securityPolicy` only after an apply has exited 0; after a failure, raise the quota or shorten the list and apply again until it does. (Measured on a new project: the same policy with the list cut to fit 20 advanced rules applied cleanly. The default rule does not count against `SECURITY_POLICY_RULES`.) + One rule expression holds 10 subexpressions (`max_subexpressions`; the API accepted 10 and refused 11 when measured, although Google's rules language reference still says 5). A rule spends one on the path, one on the host when `hosts` is set, and the rest on client ranges: 8 or 9 ranges per rule. A list is limited to 2,500 ranges and 999 rules. Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` with "Capture Groups are not allowed". The module writes `(?:a|b)` in everything it generates and refuses a capture group in `rate_limits[*].path_regex`. From 5218f59dcc0d9e50d2f7a32ec43411fa0299730d Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 12:31:35 +0100 Subject: [PATCH 06/10] fix(edge-security): refuse plans above the fixed limit of 20 advanced rules per policy Measured by the live proof: after the project quota was raised to 100, the full policy still failed, 0 of 25 rules created, with Quota 'SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY' exceeded. Limit: 20. That quota is fixed (a request to raise it is refused), so one policy holds at most 20 rules with an expression whatever the project allows. - A precondition (max_advanced_rules, default 20) refuses such a plan before anything is created, with the count and what to do. - README and guide: the two limits, which one can be raised (the project one, granted automatically in seconds when measured), and that about 100 vendor ranges is what fits beside the secret header, WAF, sign-in and staff rules. A longer list has to be merged into wider ranges or enforced at the proxy. - Tests: 293 host-scoped ranges are refused at plan; the 96-range shape the proof applied cleanly needs exactly 20. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 40 ++++++++------ terraform/examples/gcp/README.md | 1 + terraform/examples/gcp/edge.tf | 1 + terraform/examples/gcp/variables.tf | 6 +++ terraform/modules/gcp/edge-security/README.md | 27 +++++++--- terraform/modules/gcp/edge-security/main.tf | 4 ++ .../tests/edge_security.tftest.hcl | 52 +++++++++++++++++++ .../modules/gcp/edge-security/variables.tf | 10 ++++ 8 files changed, 116 insertions(+), 25 deletions(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 4fbcaabf..4a98c319 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -78,12 +78,13 @@ when you run them. edge_security_policy = { create = true } ``` - Run `terraform plan` first and read `rule_counts` in the `edge` output: a - new project allows 20 rules with an expression and 100 rules in all, and - the plan tells you what this policy needs (the defaults need 3 and 4). - Then `terraform apply`, and continue only if it finishes without an - error. `terraform output edge` prints the policy's name (`-edge`). This policy has the OWASP rules in **preview** (they log - and do not block) and a limit of 60 sign-in requests a minute per client. + Run `terraform plan` first and read `rule_counts` in the `edge` output: + one policy holds at most 20 rules with an expression, and the plan tells + you what this policy needs (the defaults need 3). Then `terraform apply`, + and continue only if it finishes without an error. `terraform output edge` + prints the policy's name (`-edge`). This policy has the OWASP rules + in **preview** (they log and do not block) and a limit of 60 sign-in + requests a minute per client. 2. Name it in `values.yaml`, and bind the dashboard Service to its BackendConfig (`-dashboard`): @@ -239,21 +240,26 @@ Cloudflare can reach it. sign-in limit is a rule of its own ahead of the closing one. The module's README (`terraform/modules/gcp/edge-security`) explains the limits. - **Check the quota before you apply a vendor list.** Each rule takes eight - or nine ranges, so a vendor that publishes about 290 ranges needs about - 45 rules with an expression, against a default of 20 per project (and 100 - rules in all). `terraform plan` shows the numbers in the `edge` output's - `rule_counts`. Request a `SECURITY_POLICY_CEVAL_RULES` increase for the - Compute Engine API first (allow headroom for the vendor adding ranges), or - enforce that list as a Cloudflare WAF custom rule and keep only your own - ranges here. Without either, the apply fails part-way with + **A long vendor list does not fit.** One Cloud Armor policy holds at most + 20 rules with an expression, and Google does not raise that limit. After + the secret header, the two WAF rules and the sign-in and staff rules, + about 13 are left: roughly 100 vendor ranges. Terraform refuses a plan + that needs more and says how many it counted (`rule_counts` in the `edge` + output shows the same). For a vendor that publishes more, merge adjacent + ranges where you accept the wider block, or enforce that vendor's list as + a Cloudflare WAF custom rule and keep only your own ranges here. + + The project also has a limit of 20 such rules across all its policies + (`SECURITY_POLICY_CEVAL_RULES`). That one can be raised, and a request was + granted in seconds when measured. If another policy in the project already + uses some, the apply fails part-way with `Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally`. **If an apply fails, do not use the policy yet.** It then holds an arbitrary part of its rules, for example the rule that denies the sign-in - paths without the rule that admits you. Raise the quota or shorten the - list, and run `terraform apply` again until it exits without an error. - Only then set `gkeIngress.securityPolicy`. + paths without the rule that admits you. Fix the cause and run + `terraform apply` again until it exits without an error. Only then set + `gkeIngress.securityPolicy`. **Done when:** diff --git a/terraform/examples/gcp/README.md b/terraform/examples/gcp/README.md index 6314280a..facbedc1 100644 --- a/terraform/examples/gcp/README.md +++ b/terraform/examples/gcp/README.md @@ -100,6 +100,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only). | deletion\_protection | Prevent deletion of Cloud SQL, Memorystore and Pub/Sub data resources. Disable only for a disposable proof environment. | `bool` | `true` | no | | desktop\_feed\_bucket\_name | Globally unique bucket for relay-control's signed Desktop feed (SELFHOST\_DESKTOP\_FEED\_GCS\_BUCKET). Empty disables it. | `string` | `""` | no | | desktop\_feed\_service\_account\_id | account\_id of the Google service account relay-control uses for the Desktop feed bucket (6-30 characters). | `string` | `"agent-relay-control"` | no | +| edge\_max\_advanced\_rules | Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max\_advanced\_rules). Google fixes it at 20; a plan that needs more is refused. | `number` | `20` | no | | edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule. | `number` | `10` | no | | edge\_origin\_secret\_header | Shared-secret request header a fronting proxy adds; requests without one of the accepted values (1-3, for rotation) are denied (modules/gcp/edge-security origin\_secret\_header). Stored in state. | ```object({ name = string values = list(string) })``` | `null` | no | | edge\_security\_policy | Cloud Armor policy for every GKE Ingress backend (modules/gcp/edge-security security\_policy: create, waf, rate\_limits, source\_ip\_allowlist, cloudflare, client\_ip\_header, client\_ip\_rules). Off unless create = true. It is exported as gkeIngress.securityPolicy in output.edge, not in chart\_values, because the chart then also needs the dashboard Service's backend-config annotation. | `any` | `{}` | no | diff --git a/terraform/examples/gcp/edge.tf b/terraform/examples/gcp/edge.tf index e10e2222..3fa7bd11 100644 --- a/terraform/examples/gcp/edge.tf +++ b/terraform/examples/gcp/edge.tf @@ -19,4 +19,5 @@ module "edge_security" { security_policy = var.edge_security_policy origin_secret_header = var.edge_origin_secret_header max_subexpressions = var.edge_max_subexpressions + max_advanced_rules = var.edge_max_advanced_rules } diff --git a/terraform/examples/gcp/variables.tf b/terraform/examples/gcp/variables.tf index e3390f98..e9397e8d 100644 --- a/terraform/examples/gcp/variables.tf +++ b/terraform/examples/gcp/variables.tf @@ -259,3 +259,9 @@ variable "edge_max_subexpressions" { type = number default = 10 } + +variable "edge_max_advanced_rules" { + description = "Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max_advanced_rules). Google fixes it at 20; a plan that needs more is refused." + type = number + default = 20 +} diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index a9d7ed5c..47e60fff 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -111,28 +111,38 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } ## Quotas: check before you apply -A new project allows **20 rules with an expression** (`SECURITY_POLICY_CEVAL_RULES`) and **100 security policy rules** in all (`SECURITY_POLICY_RULES`), across every policy in the project. Google bills each rule monthly. +Rules with an expression ("advanced" rules: everything here except the allowlist and the default rule) meet two limits. Both were measured on a live project. -The `rule_counts` output says what this policy needs, and `terraform plan` shows it before anything is created: +| Limit | Value | Can it be raised? | +| --- | --- | --- | +| Per **policy**: `SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY` | 20 | **No.** It is fixed; a request is refused | +| Per **project**: `SECURITY_POLICY_CEVAL_RULES` | 20 | Yes. A request to 100 was granted automatically in seconds (`gcloud beta quotas preferences create`, needs `cloudquotas.googleapis.com`). A new project may be treated differently | +| Per project, all rules: `SECURITY_POLICY_RULES` | 100 | Yes | + +**So one policy holds at most 20 advanced rules, whatever the project allows.** The module refuses a plan that needs more (`max_advanced_rules`), before anything is created. The `rule_counts` output shows the need: | Configuration | Advanced rules | Rules in all | | --- | --- | --- | | `create = true` (defaults) | 3 | 4 | | `cloudflare.enabled` with `origin_secret_header` | 5 | 8 | -| plus a host-scoped client rule with 290 vendor ranges | about 45 | about 48 | +| the above, with `rate_limits = []` and client rules: 96 host-scoped vendor ranges, a sign-in rule, a closing staff rule | 20 | 23 | +| the same with 293 vendor ranges | 45: **refused** | | -A long vendor list therefore fails on the first quota in a fresh project. Either: +With the secret header, two WAF rules and sign-in and staff rules, about 13 rules are left for a vendor list: roughly **100 client ranges** (8 per rule with `hosts`, 9 without). A vendor that publishes more than that does not fit in this policy. The options: -- **request the quota first** (Console, IAM and Admin, Quotas: `SECURITY_POLICY_CEVAL_RULES` for the Compute Engine API, and `SECURITY_POLICY_RULES` if `total` nears 100, with the numbers from `rule_counts` plus headroom for the vendor adding ranges), and apply once Google grants it; or -- **enforce that list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address) and keep only your own short ranges in `client_ip_rules`. +- **merge ranges**: published lists often contain adjacent blocks that one wider CIDR covers; a wider block also admits the addresses in between, so this is your decision to make; +- **enforce the long list at the fronting proxy** (a Cloudflare WAF custom rule sees the same client address and has no such limit) and keep only your own short ranges in `client_ip_rules`; +- Cloud Armor Enterprise address groups, which this module does not support. -`terraform plan` does not check the quota; the limit is met only at apply, which then fails with: +If the project quota (not the per-policy limit) is the one you hit, the apply fails part-way with: ``` Error waiting to create SecurityPolicyRule: ... Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally. ``` -**A failed apply leaves a partial policy. Do not attach it.** Terraform creates rules in parallel, so an arbitrary subset exists afterwards: in the measured case a deny rule for the sign-in paths without the rule that admits its clients, and 14 of 37 vendor rules. Attached like that, sign-in and most vendors get 403. Set `gkeIngress.securityPolicy` only after an apply has exited 0; after a failure, raise the quota or shorten the list and apply again until it does. (Measured on a new project: the same policy with the list cut to fit 20 advanced rules applied cleanly. The default rule does not count against `SECURITY_POLICY_RULES`.) +`terraform plan` does not check the project quota. It matters when other policies in the project also use advanced rules. + +**A failed apply leaves a partial policy. Do not attach it.** Terraform creates rules in parallel, so an arbitrary subset exists afterwards: in the measured case a deny rule for the sign-in paths without the rule that admits its clients, and 14 of 37 vendor rules. Attached like that, sign-in and most vendors get 403. Set `gkeIngress.securityPolicy` only after an apply has exited 0. The default rule does not count against `SECURITY_POLICY_RULES`. One rule expression holds 10 subexpressions (`max_subexpressions`; the API accepted 10 and refused 11 when measured, although Google's rules language reference still says 5). A rule spends one on the path, one on the host when `hosts` is set, and the rest on client ranges: 8 or 9 ranges per rule. A list is limited to 2,500 ranges and 999 rules. @@ -176,6 +186,7 @@ Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` | name | Name prefix: the SSL policy is -tls and the Cloud Armor policy -edge. | `string` | n/a | yes | | project\_id | Project of the GKE cluster. GKE Ingress resolves gkeIngress.sslPolicy and gkeIngress.securityPolicy by name in this project. | `string` | n/a | yes | | labels | Labels for every resource here that takes labels (the Cloud Armor policy; Compute SSL policies have none in the API). managed-by=terraform is always added. | `map(string)` | `{}` | no | +| max\_advanced\_rules | Rules with an expression that one Cloud Armor policy may hold. Google fixes this at 20 per policy (quota SECURITY\_POLICY\_ADVANCED\_RULES\_PER\_SECURITY\_POLICY, not adjustable), so a plan that needs more is refused here instead of failing half-way through the apply. Change it only if Google has given your project a different limit. | `number` | `20` | no | | max\_subexpressions | Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule. | `number` | `10` | no | | origin\_secret\_header | Optional shared-secret request header a fronting proxy adds on every request (for Cloudflare: a request header transform rule). Requests without the exact value are denied 403 before any other rule, which proves the request came through the proxy and makes its client-address header trustworthy. values lists every accepted value (1 to 3), so a rotation has no outage: add the new value here and apply, switch the proxy to it, then remove the old value and apply. The values are stored in Terraform state and readable in the Cloud Armor policy by anyone with compute.securityPolicies.get: treat both as secret stores. | ```object({ name = string values = list(string) })``` | `null` | no | | security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | diff --git a/terraform/modules/gcp/edge-security/main.tf b/terraform/modules/gcp/edge-security/main.tf index 66d5e87d..c231dfac 100644 --- a/terraform/modules/gcp/edge-security/main.tf +++ b/terraform/modules/gcp/edge-security/main.tf @@ -109,6 +109,10 @@ resource "google_compute_security_policy" "this" { condition = length(local.sp.client_ip_rules) == 0 || (local.secret_header && local.client_ip_header != null) error_message = "client_ip_rules trust a client-address header, so they need origin_secret_header (proof the request came through the proxy) and client_ip_header or cloudflare.enabled." } + precondition { + condition = local.rule_counts.advanced <= var.max_advanced_rules + error_message = "This policy needs ${local.rule_counts.advanced} rules with an expression, and one Cloud Armor policy holds at most ${var.max_advanced_rules} (quota SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY, which Google does not raise). Nothing has been created. Shorten client_ip_rules (each rule holds ${var.max_subexpressions - 1} ranges, ${var.max_subexpressions - 2} with hosts), merge adjacent ranges, or enforce the long list at the fronting proxy; see the module README, Quotas." + } precondition { condition = alltrue([for r in local.sp.client_ip_rules : length(local.client_groups[r.name]) <= 999]) && length(local.allow_groups) <= 999 error_message = "A list needs more than 999 rules at this max_subexpressions. Shorten it, or enforce it at the fronting proxy." diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index 9bf114e7..07aa96a1 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -3,6 +3,9 @@ mock_provider "google" {} variables { project_id = "proj" name = "relay" + # Runs that build long lists to test rule splitting lift the per-policy + # limit; the limit itself has its own runs at the end. + max_advanced_rules = 1000 # Most runs pin 5 subexpressions per rule so that small lists exercise the # splitting into several rules; the module's default (10) has its own runs. max_subexpressions = 5 @@ -544,3 +547,52 @@ run "capture_group_after_a_class_refused" { } expect_failures = [var.security_policy] } + +# One policy holds 20 rules with an expression, and Google does not raise it. +# A plan that needs more is refused before anything is created. +run "more_than_twenty_advanced_rules_refused_at_plan" { + command = plan + variables { + max_subexpressions = 10 + max_advanced_rules = 20 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + client_ip_rules = [{ + name = "vendors" + hosts = ["history.relay.example.com"] + path_prefixes = ["/v1/delivery"] + cidrs = [for i in range(293) : "10.${floor(i / 256)}.${i % 256}.0/24"] + unthrottled = true + }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + expect_failures = [google_compute_security_policy.this] +} + +# The shape the live proof applied cleanly: exactly 20. +run "twenty_advanced_rules_fit" { + command = apply + variables { + max_subexpressions = 10 + max_advanced_rules = 20 + security_policy = { + create = true + cloudflare = { enabled = true } + rate_limits = [] + client_ip_rules = [ + { name = "vendors", hosts = ["history.relay.example.com"], path_prefixes = ["/v1/delivery"], cidrs = [for i in range(96) : "10.0.${i}.0/24"], unthrottled = true }, + { name = "sign-in", path_prefixes = ["/login"], cidrs = ["203.0.113.0/24"], rate_limit = { count = 60, interval_sec = 60 } }, + { name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }, + ] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + # 96 ranges at 8 = 12 admitting rules + deny; sign-in 2; staff 2; secret header; 2 WAF. + assert { + condition = output.rule_counts.advanced == 20 && output.rule_counts.total == 23 + error_message = "96 host-scoped vendor ranges with sign-in and staff rules need exactly 20 advanced rules." + } +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index 073d87d6..8df7ff80 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -160,3 +160,13 @@ variable "max_subexpressions" { error_message = "max_subexpressions is 5 to 10." } } + +variable "max_advanced_rules" { + description = "Rules with an expression that one Cloud Armor policy may hold. Google fixes this at 20 per policy (quota SECURITY_POLICY_ADVANCED_RULES_PER_SECURITY_POLICY, not adjustable), so a plan that needs more is refused here instead of failing half-way through the apply. Change it only if Google has given your project a different limit." + type = number + default = 20 + validation { + condition = var.max_advanced_rules >= 1 && floor(var.max_advanced_rules) == var.max_advanced_rules + error_message = "max_advanced_rules is a positive whole number." + } +} From 60afa9ae7ee899fd07c324b1d6882f01e6ff9746 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 13:16:01 +0100 Subject: [PATCH 07/10] fix(edge-security): deny connections from outside the proxy before client rules; path prefixes end at a segment Findings from the live proof with the policy attached (S3-d and smaller ones): - SECURITY: a request admitted by a client rule never reaches the allowlist of connecting addresses, so the shared-secret header was the only guard. A caller holding the secret, connecting straight to the load balancer, was admitted and chose its own client address through CF-Connecting-IP (as a staff address, and as a vendor on a vendor path). New rule at priority 150 denies a connection that comes from none of connecting_asns (Cloudflare: 13335) or connecting_address_groups (an address group of the proxy's ranges, Cloud Armor Enterprise); the proxy's 15 ranges do not fit one negated expression. client_ip_rules are now refused without one of them, unless secret_header_only = true accepts the header as the sole guard. - A path prefix had no segment boundary (/mcp also matched /mcpfoo). Patterns now end in (?:/|$); "/" stays every path. - Guide: listing the policy on all backend services passed about 4 minutes before every backend enforced it; the check now sends a request per host. trustedProxies 3 behind Cloudflare is measured, not expected. - examples/gcp: labels on the private Cloud SQL DNS zone. Proven live before this commit: the capture-group fix, 10 subexpressions, throttle keyed per CF-Connecting-IP, the sign-in limit, WAF preview logging, vendor path scoping. NOT yet proven: origin.asn in a deny rule and the address-group form of the guard. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 34 +++- terraform/examples/gcp/main.tf | 1 + terraform/modules/gcp/edge-security/README.md | 17 +- terraform/modules/gcp/edge-security/main.tf | 35 +++- .../tests/edge_security.tftest.hcl | 179 ++++++++++++++---- .../modules/gcp/edge-security/variables.tf | 10 +- 6 files changed, 227 insertions(+), 49 deletions(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 4a98c319..39e2a4a7 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -112,6 +112,22 @@ gcloud compute backend-services list --format='table(name,securityPolicy.basenam Every BackendConfig shows the policy, and the count equals the number of backend services of the Ingress (6 for the default install). +That list can be complete several minutes before every backend enforces the +policy (4 minutes when measured). If your policy refuses some requests (an +allowlist, Cloudflare, client rules), confirm with a request that it must +refuse, **on each host**, before you rely on it: + +```sh +IP=$(gcloud compute addresses describe $STATIC_IP_NAME --global --format='value(address)') +for h in $DOMAIN api.$DOMAIN cast.$DOMAIN history.$DOMAIN; do + curl -sk -o /dev/null -w "$h %{http_code}\n" --resolve "$h:443:$IP" "https://$h/" +done +``` + +From an address the policy does not admit, every line must end in `403`. +Repeat until they all do. Warnings that mention `Translate` in +`kubectl describe ingress` while the policy attaches are transient. + **Turn the OWASP rules on.** Session transcripts and messages contain code, SQL and shell text, which is what those rules look for. Use the install normally for a week, then look for requests the rules would have blocked: @@ -187,8 +203,8 @@ Cloudflare can reach it. the address it received the request from, and its own. Measured on a live GKE install, the header is `, ` and the right value is 2. Cloudflare adds one more, so behind Cloudflare the header is - `, , ` and the value is 3. The - Cloudflare case is expected, not yet measured; check it as below. + `, , ` and the value is 3 (also + measured). Check yours as below. ```yaml relay-control: @@ -222,9 +238,10 @@ Cloudflare can reach it. ```hcl edge_security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + connecting_asns = [13335] # Cloudflare's network; see below + rate_limits = [] client_ip_rules = [ { name = "vendors", hosts = ["history."], path_prefixes = ["/v1/delivery", "/v1/turns", "/v1/auth/token/refresh"], cidrs = ["", ""], rate_limit = { count = 600, interval_sec = 60 } }, { name = "sign-in", path_prefixes = ["/login", "/api/v1/auth", "/cloud/api/v1/auth"], cidrs = [""], rate_limit = { count = 60, interval_sec = 60 } }, @@ -233,6 +250,13 @@ Cloudflare can reach it. } ``` + `connecting_asns` matters here. A request that one of these rules admits + is not checked against Cloudflare's address ranges afterwards, so without + it anyone who learned the secret header could connect straight to the + load balancer and claim to be any client. With it, a connection that does + not come from Cloudflare's network is refused before any of these rules. + Terraform refuses `client_ip_rules` without it. + The first rule that matches a request's path decides it: its listed clients are admitted, everyone else gets 403. The last rule closes every other path. A request that a rule admits is not checked by any later diff --git a/terraform/examples/gcp/main.tf b/terraform/examples/gcp/main.tf index 9a9c4329..29547161 100644 --- a/terraform/examples/gcp/main.tf +++ b/terraform/examples/gcp/main.tf @@ -84,6 +84,7 @@ module "postgres" { # that identity to the private address inside this VPC so every service can use # sslmode=verify-full without a proxy or public IP. resource "google_dns_managed_zone" "cloud_sql" { + labels = var.labels project = var.project_id name = "${var.name}-cloud-sql" dns_name = module.postgres.dns_name diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index 47e60fff..06204313 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -18,6 +18,7 @@ Rules are evaluated in priority order and the first match decides. A rule that a | Priority | Rule | Action | | --- | --- | --- | | 100 | `origin_secret_header`: the shared-secret header is missing or wrong | deny 403 | +| 150 | `connecting_asns`, `connecting_address_groups`: the connection does not come from the fronting proxy | deny 403 | | 1000+ | `waf`: OWASP preconfigured rules (SQLi, XSS, LFI, RFI, RCE, scanner detection, protocol attack, session fixation, CVE canary) at sensitivity 1 | deny 403, **preview by default** | | 10000+ | `client_ip_rules`: on the listed paths, admit the listed client addresses (with the rule's own rate limit) and deny everyone else | allow or throttle, then deny 403 | | 20000+ | `rate_limits`: per-client throttle on path regexes (default: sign-in and password paths, 60 requests a minute) | 429 beyond the limit | @@ -54,14 +55,15 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } - The client address is read from `CF-Connecting-IP` (Cloud Armor `origin.user_ip`), so rate limits count per client and not per Cloudflare address. - `client_ip_rules` decide which clients may reach which paths, by the client address in `CF-Connecting-IP`. They trust that header, so they require `origin_secret_header`. See the next section. -- There is now one more proxy in front of the application, so `relay-control.auth.trustedProxies` goes up by one: 3 behind Cloudflare (2 was measured for the Google load balancer alone; 3 is expected and not yet measured). `docs/gcp-install/3-EDGE-HARDENING.md` shows how to check it. With the wrong number, sign-in rate limits and lockout are shared by every user. +- There is now one more proxy in front of the application, so `relay-control.auth.trustedProxies` goes up by one: 3 behind Cloudflare, 2 for the Google load balancer alone (both measured on a live install). `docs/gcp-install/3-EDGE-HARDENING.md` shows how to check it. With the wrong number, sign-in rate limits and lockout are shared by every user. ### Which clients may reach which paths `client_ip_rules` are allowlists of client addresses, each scoped to path prefixes and optionally to hosts. They are evaluated in list order, and **the first rule whose paths (and hosts) match a request decides it**: admitted if the client address is in `cidrs`, denied 403 otherwise. A request that matches no rule falls through to the rules below. -A Cloud Armor rule that admits a request ends evaluation. So a request admitted by a client rule meets no later rule: not a later client rule, not `rate_limits`, not the allowlist of connecting addresses (the secret header rule, which runs first, has already proved it came through the proxy). Two consequences, both enforced by the module: +A Cloud Armor rule that admits a request ends evaluation. So a request admitted by a client rule meets no later rule: not a later client rule, not `rate_limits`, and **not the allowlist of connecting addresses**. Three consequences, all enforced by the module: +- **Connections from outside the proxy must be denied first.** Otherwise the secret header is the only guard: a caller that holds it could connect straight to the load balancer and put any client address it likes in the header. (Measured on a live policy: such a caller was admitted as a staff address and as a vendor.) The proxy's ranges do not fit one negated expression, so name its network instead: `connecting_asns = [13335]` for Cloudflare, or `connecting_address_groups` with a Cloud Armor address group of its ranges (Enterprise). `secret_header_only = true` accepts the header as the sole guard. - **Every rule states its own limit**: `rate_limit = { count, interval_sec }` (per client address, 429 beyond it) or `unthrottled = true`. - **A rule for every path (`["/"]` without `hosts`) makes `rate_limits` unreachable**, so set `rate_limits = []` and express the sign-in limit as a narrower rule ahead of it, as below. @@ -69,9 +71,10 @@ This example lets two vendors' published ranges reach only the session upload pa ```hcl security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + connecting_asns = [13335] # Cloudflare's network: nothing else may connect + rate_limits = [] client_ip_rules = [ { name = "vendor-uploads" # session uploads, on the history host @@ -104,6 +107,7 @@ security_policy = { origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } ``` +- A path prefix ends at a path segment: `/mcp` covers `/mcp` and `/mcp/tools`, not `/mcpfoo`. With `hosts`, every listed host is paired with every listed path; use two rules when two hosts need different paths. - Put narrower rules first. Repeat your own ranges (for example your VPN's egress range) in a vendor rule: your users' desktop uploads and MCP clients use those paths too, and a rule denies everyone it does not list. - Without the closing `["/"]` rule, paths that no rule names stay open to every client that comes through the proxy (and `rate_limits` still apply to them). - Each rule costs one advanced rule per nine client ranges (eight with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. @@ -173,6 +177,7 @@ Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` | [google_compute_security_policy.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy) | resource | | [google_compute_security_policy_rule.allowlist](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.client_ip](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | +| [google_compute_security_policy_rule.connecting](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.default](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.origin_secret](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | | [google_compute_security_policy_rule.rate_limit](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy_rule) | resource | @@ -189,7 +194,7 @@ Regular expressions must not contain capture groups: Cloud Armor refuses `(a|b)` | max\_advanced\_rules | Rules with an expression that one Cloud Armor policy may hold. Google fixes this at 20 per policy (quota SECURITY\_POLICY\_ADVANCED\_RULES\_PER\_SECURITY\_POLICY, not adjustable), so a plan that needs more is refused here instead of failing half-way through the apply. Change it only if Google has given your project a different limit. | `number` | `20` | no | | max\_subexpressions | Subexpressions in one rule expression. Cloud Armor accepts 10 and refuses 11 (measured against the API; the rules language reference still says 5). A rule that matches a path (and host) and a list of client ranges uses one (two) for the match and the rest for ranges, so fewer means more rules for the same list. Lower it only if your project's API refuses a rule. | `number` | `10` | no | | origin\_secret\_header | Optional shared-secret request header a fronting proxy adds on every request (for Cloudflare: a request header transform rule). Requests without the exact value are denied 403 before any other rule, which proves the request came through the proxy and makes its client-address header trustworthy. values lists every accepted value (1 to 3), so a rotation has no outage: add the new value here and apply, switch the proxy to it, then remove the old value and apply. The values are stored in Terraform state and readable in the Cloud Armor policy by anyone with compute.securityPolicies.get: treat both as secret stores. | ```object({ name = string values = list(string) })``` | `null` | no | -| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | +| security\_policy | Cloud Armor backend security policy (chart value gkeIngress.securityPolicy), off unless create = true. - waf: preconfigured OWASP rules (deny 403) at the given sensitivity (1 = fewest false positives). preview = true (the default) only logs matches: session transcripts and messages carry code, SQL and shell text, so read the logs before enforcing. - rate\_limits: per-client throttle (429) on the listed path regexes; the default covers password sign-in, setup-key and login paths. - source\_ip\_allowlist: when non-empty (or cloudflare.enabled), the default rule becomes deny 403 and only these connecting addresses are admitted: a pure source-IP allowlist. - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client\_ip\_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user\_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin\_secret\_header. - client\_ip\_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path\_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user\_ip) is in cidrs, denied 403 otherwise. A last rule with path\_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin\_secret\_header. An admitted request meets no later rule, including rate\_limits, so every rule states its own limit: rate\_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate\_limits = [] and express sign-in limits as an earlier, narrower rule. - connecting\_asns, connecting\_address\_groups: where the fronting proxy connects from, as autonomous system numbers (Cloudflare: 13335) or names of existing Cloud Armor address groups holding its ranges (Cloud Armor Enterprise). A connection from anywhere else is denied 403 right after the secret-header check. client\_ip\_rules require one of them: a request those rules admit is not checked against the allowlist of connecting addresses, so otherwise a caller that holds the secret header could skip the proxy and choose its own client address. secret\_header\_only = true accepts that instead. - adaptive\_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). | ```object({ create = optional(bool, false) waf = optional(object({ enabled = optional(bool, true) preview = optional(bool, true) sensitivity = optional(number, 1) rules = optional(list(string), [ "sqli-v33-stable", "xss-v33-stable", "lfi-v33-stable", "rfi-v33-stable", "rce-v33-stable", "scannerdetection-v33-stable", "protocolattack-v33-stable", "sessionfixation-v33-stable", "cve-canary", ]) }), {}) rate_limits = optional(list(object({ name = string path_regex = string count = number interval_sec = number ban_duration_sec = optional(number, 0) })), [{ name = "sign-in" path_regex = "^/(?:login|(?:cloud/)?api/v1/auth/)" count = 60 interval_sec = 60 }]) source_ip_allowlist = optional(list(string), []) cloudflare = optional(object({ enabled = optional(bool, false) include_ipv6 = optional(bool, false) # https://www.cloudflare.com/ips/ as read on 2026-11-18. Cloudflare # announces changes; compare the list with that page before each apply. ipv4_cidrs = optional(list(string), [ "173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/13", "104.24.0.0/14", "172.64.0.0/13", "131.0.72.0/22", ]) ipv6_cidrs = optional(list(string), [ "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) client_ip_header = optional(string) connecting_asns = optional(list(number), []) connecting_address_groups = optional(list(string), []) secret_header_only = optional(bool, false) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) hosts = optional(list(string), []) cidrs = list(string) rate_limit = optional(object({ count = number, interval_sec = number })) unthrottled = optional(bool, false) })), []) adaptive_protection = optional(bool, false) })``` | `{}` | no | | ssl\_policy | Compute SSL policy for the load balancer's HTTPS listener (chart value gkeIngress.sslPolicy). MODERN with TLS 1.2 serves current browsers, the desktop app and CLIs; RESTRICTED narrows the cipher list further (TLS 1.2 minimum is required by it). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no | ### Outputs diff --git a/terraform/modules/gcp/edge-security/main.tf b/terraform/modules/gcp/edge-security/main.tf index c231dfac..24d8e126 100644 --- a/terraform/modules/gcp/edge-security/main.tf +++ b/terraform/modules/gcp/edge-security/main.tf @@ -58,9 +58,11 @@ locals { description = "${r.name}: any other client on ${join(", ", concat(r.hosts, r.path_prefixes))}" expression = local.path_match[r.name] } })]...) + connecting_guard = length(local.sp.connecting_asns) + length(local.sp.connecting_address_groups) > 0 + # What the policy costs against the project's Cloud Armor quotas. rule_counts = { - advanced = (local.secret_header ? 1 : 0) + length(local.waf_groups) + length(local.client_rules) + length(local.rate_rules) + advanced = (local.secret_header ? 1 : 0) + (local.connecting_guard ? 1 : 0) + length(local.waf_groups) + length(local.client_rules) + length(local.rate_rules) basic = length(local.allow_rules) + 1 } # The path (one subexpression) and, with hosts, the Host header (a second); @@ -69,7 +71,9 @@ locals { # group ("Capture Groups are not allowed"). path_match = { for r in local.sp.client_ip_rules : r.name => join(" && ", compact([ length(r.hosts) > 0 ? "request.headers['host'].lower().matches('^(?:${join("|", [for h in r.hosts : replace(h, ".", "\\\\.")])})(?::[0-9]+)?$')" : "", - "request.path.matches('^(?:${join("|", [for p in r.path_prefixes : replace(p, ".", "\\\\.")])})')", + # A prefix ends at a path segment: /mcp matches /mcp and /mcp/x, not + # /mcpfoo. "/" is every path. + contains(r.path_prefixes, "/") ? "request.path.matches('^/')" : "request.path.matches('^(?:${join("|", [for p in r.path_prefixes : replace(trimsuffix(p, "/"), ".", "\\\\.")])})(?:/|$)')", ])) } } @@ -117,6 +121,10 @@ resource "google_compute_security_policy" "this" { condition = alltrue([for r in local.sp.client_ip_rules : length(local.client_groups[r.name]) <= 999]) && length(local.allow_groups) <= 999 error_message = "A list needs more than 999 rules at this max_subexpressions. Shorten it, or enforce it at the fronting proxy." } + precondition { + condition = length(local.sp.client_ip_rules) == 0 || local.connecting_guard || local.sp.secret_header_only + error_message = "client_ip_rules admit requests before the allowlist of connecting addresses is checked, so a caller that holds the secret header could skip the proxy and choose its own client address. Set connecting_asns (Cloudflare: [13335]) or connecting_address_groups so connections from outside the proxy are denied first, or set secret_header_only = true to accept that the secret header is the only guard." + } precondition { condition = alltrue([for r in local.sp.client_ip_rules : r.rate_limit != null || r.unthrottled]) error_message = "Each client_ip_rules entry admits its clients, and admitted requests skip rate_limits. Give the rule its own rate_limit, or set unthrottled = true to accept that it has none." @@ -147,6 +155,29 @@ resource "google_compute_security_policy_rule" "origin_secret" { } } +# Requests admitted by a client rule never reach the allowlist of connecting +# addresses further down, so without this rule the shared-secret header alone +# decides whether a caller that skipped the proxy gets in (and such a caller +# chooses its own client address). The proxy's ranges do not fit one negated +# expression, but its network number does, and so does an address group: deny +# every connection that comes from none of them. +resource "google_compute_security_policy_rule" "connecting" { + count = local.armor && local.connecting_guard ? 1 : 0 + project = var.project_id + security_policy = google_compute_security_policy.this[0].name + priority = 150 + action = "deny(403)" + description = "Connections from outside the fronting proxy (${join(", ", concat([for a in local.sp.connecting_asns : "AS ${a}"], local.sp.connecting_address_groups))})" + match { + expr { + expression = join(" && ", concat( + [for a in local.sp.connecting_asns : "origin.asn != ${a}"], + [for g in local.sp.connecting_address_groups : "!evaluateAddressGroup('${g}', origin.ip)"], + )) + } + } +} + resource "google_compute_security_policy_rule" "waf" { for_each = local.armor ? { for i, group in local.waf_groups : tostring(i) => group } : {} project = var.project_id diff --git a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl index 07aa96a1..c1c16a8b 100644 --- a/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl +++ b/terraform/modules/gcp/edge-security/tests/edge_security.tftest.hcl @@ -122,8 +122,9 @@ run "cloudflare_origin_lockdown" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } + create = true + cloudflare = { enabled = true } + secret_header_only = true client_ip_rules = [{ name = "uploads" path_prefixes = ["/v1/delivery", "/v1/turns"] @@ -160,7 +161,7 @@ run "cloudflare_origin_lockdown" { error_message = "Five client ranges: two allow rules and the path deny." } assert { - condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)')" + condition = google_compute_security_policy_rule.client_ip["uploads-allow-1"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)(?:/|$)') && (inIpRange(origin.user_ip, '198.18.0.0/15'))" && google_compute_security_policy_rule.client_ip["uploads-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["uploads-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/v1/delivery|/v1/turns)(?:/|$)')" error_message = "Client rules match the path and origin.user_ip, then deny the path." } # Order: secret header, WAF, client rules, rate limits, allowlist, default. @@ -174,9 +175,10 @@ run "client_rules_need_the_secret_header" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] } } expect_failures = [google_compute_security_policy.this] @@ -237,9 +239,10 @@ run "vendor_paths_and_a_closed_remainder" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [ { name = "vendors" @@ -270,7 +273,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "Every rule admits its clients (four ranges per rule, three with hosts) and then denies its paths." } assert { - condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/v1/turns|/v1/auth/token/refresh)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" + condition = google_compute_security_policy_rule.client_ip["vendors-allow-0"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/v1/turns|/v1/auth/token/refresh)(?:/|$)') && (inIpRange(origin.user_ip, '192.0.2.0/24') || inIpRange(origin.user_ip, '198.51.100.0/24'))" error_message = "Host, path and the client ranges, within five subexpressions." } # A short vendor list ahead of a staff-only "/" rule: the vendor is admitted @@ -286,7 +289,7 @@ run "vendor_paths_and_a_closed_remainder" { error_message = "A one-range rule with a rate_limit throttles per client address." } assert { - condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/)')" + condition = google_compute_security_policy_rule.client_ip["staff-allow-0"].action == "allow" && length(google_compute_security_policy_rule.client_ip["staff-allow-0"].rate_limit_options) == 0 && google_compute_security_policy_rule.client_ip["staff-deny"].action == "deny(403)" && google_compute_security_policy_rule.client_ip["staff-deny"].match[0].expr[0].expression == "request.path.matches('^/')" error_message = "An unthrottled rule admits with allow; the closing rule denies every remaining path." } assert { @@ -300,9 +303,10 @@ run "client_rule_must_choose_a_limit" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "uploads", path_prefixes = ["/v1/delivery"], cidrs = ["192.0.2.0/24"] }] } origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } } @@ -314,9 +318,10 @@ run "catch_all_client_rule_refuses_dead_rate_limits" { command = plan variables { security_policy = { - create = true - cloudflare = { enabled = true } - client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] } origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } } @@ -346,9 +351,10 @@ run "rule_counts_and_wider_expressions" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [{ name = "vendors" hosts = ["history.relay.example.com"] @@ -376,9 +382,10 @@ run "ten_subexpressions_need_a_third_of_the_rules" { variables { max_subexpressions = 10 security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [{ name = "vendors" hosts = ["history.relay.example.com"] @@ -430,8 +437,9 @@ run "generated_expressions_have_no_capture_groups" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } + create = true + cloudflare = { enabled = true } + secret_header_only = true client_ip_rules = [ { name = "vendors", hosts = ["history.relay.example.com", "api.relay.example.com"], path_prefixes = ["/v1/delivery", "/.well-known"], cidrs = ["192.0.2.0/24"], unthrottled = true }, ] @@ -446,7 +454,7 @@ run "generated_expressions_have_no_capture_groups" { error_message = "Every group inside a matches() pattern must be non-capturing." } assert { - condition = google_compute_security_policy_rule.client_ip["vendors-deny"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com|api\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/\\\\.well-known)')" + condition = google_compute_security_policy_rule.client_ip["vendors-deny"].match[0].expr[0].expression == "request.headers['host'].lower().matches('^(?:history\\\\.relay\\\\.example\\\\.com|api\\\\.relay\\\\.example\\\\.com)(?::[0-9]+)?$') && request.path.matches('^(?:/v1/delivery|/\\\\.well-known)(?:/|$)')" error_message = "Host and path patterns in the form the API accepted." } } @@ -472,9 +480,10 @@ run "long_list_keeps_its_band_and_order" { command = apply variables { security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [ { name = "vendors" @@ -556,9 +565,10 @@ run "more_than_twenty_advanced_rules_refused_at_plan" { max_subexpressions = 10 max_advanced_rules = 20 security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [{ name = "vendors" hosts = ["history.relay.example.com"] @@ -579,9 +589,10 @@ run "twenty_advanced_rules_fit" { max_subexpressions = 10 max_advanced_rules = 20 security_policy = { - create = true - cloudflare = { enabled = true } - rate_limits = [] + create = true + cloudflare = { enabled = true } + rate_limits = [] + secret_header_only = true client_ip_rules = [ { name = "vendors", hosts = ["history.relay.example.com"], path_prefixes = ["/v1/delivery"], cidrs = [for i in range(96) : "10.0.${i}.0/24"], unthrottled = true }, { name = "sign-in", path_prefixes = ["/login"], cidrs = ["203.0.113.0/24"], rate_limit = { count = 60, interval_sec = 60 } }, @@ -596,3 +607,101 @@ run "twenty_advanced_rules_fit" { error_message = "96 host-scoped vendor ranges with sign-in and staff rules need exactly 20 advanced rules." } } + +# A prefix stops at a path segment, and a trailing slash in the input does not double it. +run "path_prefix_ends_at_a_segment" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + secret_header_only = true + client_ip_rules = [{ name = "mcp", path_prefixes = ["/mcp", "/cloud/mcp/"], cidrs = ["192.0.2.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.client_ip["mcp-deny"].match[0].expr[0].expression == "request.path.matches('^(?:/mcp|/cloud/mcp)(?:/|$)')" + error_message = "/mcp must match /mcp and /mcp/x but not /mcpfoo." + } + # The same pattern, checked against sample paths. + assert { + condition = google_compute_security_policy_rule.client_ip["mcp-deny"].action == "deny(403)" && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcpfoo")) == 0 && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcp/tools")) == 1 && length(regexall("^(?:/mcp|/cloud/mcp)(?:/|$)", "/mcp")) == 1 + error_message = "The pattern has the segment boundary." + } +} + +# connecting_asns: a connection from outside the proxy's network is denied +# before any rule can admit it, so the secret header alone is not enough. +run "connecting_asns_deny_other_networks_first" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + connecting_asns = [13335, 209242] + rate_limits = [] + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.connecting[0].action == "deny(403)" && google_compute_security_policy_rule.connecting[0].match[0].expr[0].expression == "origin.asn != 13335 && origin.asn != 209242" + error_message = "Deny when the connection comes from none of the listed networks." + } + assert { + condition = google_compute_security_policy_rule.origin_secret[0].priority < google_compute_security_policy_rule.connecting[0].priority && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.client_ip : r.priority]...) && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.waf : r.priority]...) + error_message = "The network check runs before the WAF and before every rule that admits." + } + assert { + condition = output.rule_counts.advanced == 6 + error_message = "Secret header, network check, two WAF rules, staff allow and deny." + } +} + +run "no_connecting_asn_rule_by_default" { + command = apply + variables { + security_policy = { create = true, cloudflare = { enabled = true } } + } + assert { + condition = length(google_compute_security_policy_rule.connecting) == 0 + error_message = "connecting_asns is opt-in." + } +} + +# client_ip_rules admit before the connecting-address allowlist, so the plan is +# refused until connections from outside the proxy are denied first, or the +# operator accepts the secret header as the only guard. +run "client_rule_needs_a_connecting_check" { + command = plan + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + rate_limits = [] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + expect_failures = [google_compute_security_policy.this] +} + +# The same guard with an address group of the proxy's ranges (Cloud Armor Enterprise). +run "connecting_address_group_denies_other_sources_first" { + command = apply + variables { + security_policy = { + create = true + cloudflare = { enabled = true } + connecting_address_groups = ["relay-cloudflare"] + rate_limits = [] + client_ip_rules = [{ name = "staff", path_prefixes = ["/"], cidrs = ["203.0.113.0/24"], unthrottled = true }] + } + origin_secret_header = { name = "x-origin-auth", values = ["0123456789abcdef0123456789abcdef"] } + } + assert { + condition = google_compute_security_policy_rule.connecting[0].match[0].expr[0].expression == "!evaluateAddressGroup('relay-cloudflare', origin.ip)" && google_compute_security_policy_rule.connecting[0].action == "deny(403)" && google_compute_security_policy_rule.connecting[0].priority < min([for r in google_compute_security_policy_rule.client_ip : r.priority]...) + error_message = "A connection whose address is outside the proxy's address group is denied before any client rule." + } +} diff --git a/terraform/modules/gcp/edge-security/variables.tf b/terraform/modules/gcp/edge-security/variables.tf index 8df7ff80..a3417853 100644 --- a/terraform/modules/gcp/edge-security/variables.tf +++ b/terraform/modules/gcp/edge-security/variables.tf @@ -46,6 +46,7 @@ variable "security_policy" { - cloudflare: admit Cloudflare's published ranges as the connecting addresses (origin lockdown) and read the client address from CF-Connecting-IP. - client_ip_header: header an upstream proxy sets with the real client address (Cloud Armor origin.user_ip). Only trustworthy when nothing but that proxy can reach the load balancer: combine with an allowlist of the proxy's ranges, or origin_secret_header. - client_ip_rules: per-client allowlists scoped to paths (and optionally hosts), evaluated in list order. The first rule whose path_prefixes (and hosts, when given) match a request decides it: admitted if the client address (origin.user_ip) is in cidrs, denied 403 otherwise. A last rule with path_prefixes = ["/"] therefore closes every other path to everyone but its cidrs. Requires origin_secret_header. An admitted request meets no later rule, including rate_limits, so every rule states its own limit: rate_limit (requests per interval per client address), or unthrottled = true. With a rule for "/" set rate_limits = [] and express sign-in limits as an earlier, narrower rule. + - connecting_asns, connecting_address_groups: where the fronting proxy connects from, as autonomous system numbers (Cloudflare: 13335) or names of existing Cloud Armor address groups holding its ranges (Cloud Armor Enterprise). A connection from anywhere else is denied 403 right after the secret-header check. client_ip_rules require one of them: a request those rules admit is not checked against the allowlist of connecting addresses, so otherwise a caller that holds the secret header could skip the proxy and choose its own client address. secret_header_only = true accepts that instead. - adaptive_protection: Layer 7 DDoS detection (alerts; automatic deployment needs Cloud Armor Enterprise). EOT type = object({ @@ -87,7 +88,10 @@ variable "security_policy" { "2a06:98c0::/29", "2c0f:f248::/32", ]) }), {}) - client_ip_header = optional(string) + client_ip_header = optional(string) + connecting_asns = optional(list(number), []) + connecting_address_groups = optional(list(string), []) + secret_header_only = optional(bool, false) client_ip_rules = optional(list(object({ name = string path_prefixes = list(string) @@ -130,6 +134,10 @@ variable "security_policy" { ]) && length(var.security_policy.client_ip_rules) <= 9 && length(distinct(var.security_policy.client_ip_rules[*].name)) == length(var.security_policy.client_ip_rules) error_message = "At most 9 client_ip_rules with unique lowercase names, 1-2500 cidrs, path_prefixes that start with / and hold only unreserved URL characters, lowercase DNS hosts, and a rate_limit with count >= 1 and an interval_sec Cloud Armor accepts." } + validation { + condition = length(var.security_policy.connecting_asns) + length(var.security_policy.connecting_address_groups) <= 8 && alltrue([for a in var.security_policy.connecting_asns : a >= 1 && a <= 4294967295 && floor(a) == a]) && alltrue([for g in var.security_policy.connecting_address_groups : can(regex("^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$", g))]) + error_message = "connecting_asns are autonomous system numbers (whole numbers from 1 to 4294967295) and connecting_address_groups are address group names (not full paths); at most 8 in all." + } validation { condition = var.security_policy.client_ip_header == null || can(regex("^[A-Za-z][A-Za-z0-9-]{0,63}$", var.security_policy.client_ip_header)) error_message = "client_ip_header is an HTTP header name, for example CF-Connecting-IP or True-Client-IP." From 442e8c80576ccd7a4e5ac2cdf6222054221d60d0 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 13:33:15 +0100 Subject: [PATCH 08/10] docs(edge-security): probe every backend after attaching the policy; two README notes Review feedback on #487: the per-host loop only requested / on the history host, which reaches the deny service, so the history reader and upload backends were never probed. The check now sends one request to each of the six backend services. README: a trailing slash on a path prefix is ignored, the connecting check counts as a rule, and rule_counts shows the plan, not what exists after a failed apply. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 17 +++++++++++------ terraform/modules/gcp/edge-security/README.md | 4 +++- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 39e2a4a7..9e861468 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -115,18 +115,23 @@ backend services of the Ingress (6 for the default install). That list can be complete several minutes before every backend enforces the policy (4 minutes when measured). If your policy refuses some requests (an allowlist, Cloudflare, client rules), confirm with a request that it must -refuse, **on each host**, before you rely on it: +refuse, **on each backend**, before you rely on it. Each line below reaches a +different backend service: ```sh IP=$(gcloud compute addresses describe $STATIC_IP_NAME --global --format='value(address)') -for h in $DOMAIN api.$DOMAIN cast.$DOMAIN history.$DOMAIN; do - curl -sk -o /dev/null -w "$h %{http_code}\n" --resolve "$h:443:$IP" "https://$h/" +for url in "$DOMAIN/" "api.$DOMAIN/readyz" "cast.$DOMAIN/health" \ + "history.$DOMAIN/healthz" "history.$DOMAIN/v1/delivery/limits" "history.$DOMAIN/"; do + h=${url%%/*} + curl -sk -o /dev/null -w "$url %{http_code}\n" --resolve "$h:443:$IP" "https://$url" done ``` -From an address the policy does not admit, every line must end in `403`. -Repeat until they all do. Warnings that mention `Translate` in -`kubectl describe ingress` while the policy attaches are transient. +The six are the dashboard, the API, Relaycast, the history reader, the +history upload service and the deny service. From an address the policy does +not admit, every line must end in `403`. Repeat until they all do. Warnings +that mention `Translate` in `kubectl describe ingress` while the policy +attaches are transient. **Turn the OWASP rules on.** Session transcripts and messages contain code, SQL and shell text, which is what those rules look for. Use the install diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index 06204313..6dcc5ea8 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -107,7 +107,7 @@ security_policy = { origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } ``` -- A path prefix ends at a path segment: `/mcp` covers `/mcp` and `/mcp/tools`, not `/mcpfoo`. With `hosts`, every listed host is paired with every listed path; use two rules when two hosts need different paths. +- A path prefix ends at a path segment: `/mcp` covers `/mcp` and `/mcp/tools`, not `/mcpfoo`. A trailing slash is ignored, so `/.well-known/` also covers `/.well-known`. With `hosts`, every listed host is paired with every listed path; use two rules when two hosts need different paths. - Put narrower rules first. Repeat your own ranges (for example your VPN's egress range) in a vendor rule: your users' desktop uploads and MCP clients use those paths too, and a rule denies everyone it does not list. - Without the closing `["/"]` rule, paths that no rule names stay open to every client that comes through the proxy (and `rate_limits` still apply to them). - Each rule costs one advanced rule per nine client ranges (eight with `hosts`) plus one. See "Quotas" below before you list a vendor's published ranges. @@ -132,6 +132,8 @@ Rules with an expression ("advanced" rules: everything here except the allowlist | the above, with `rate_limits = []` and client rules: 96 host-scoped vendor ranges, a sign-in rule, a closing staff rule | 20 | 23 | | the same with 293 vendor ranges | 45: **refused** | | +The `rule_counts` output is what the configuration needs, not what exists: after a failed apply it shows the planned numbers. The connecting check (`connecting_asns`) is itself one rule. + With the secret header, two WAF rules and sign-in and staff rules, about 13 rules are left for a vendor list: roughly **100 client ranges** (8 per rule with `hosts`, 9 without). A vendor that publishes more than that does not fit in this policy. The options: - **merge ranges**: published lists often contain adjacent blocks that one wider CIDR covers; a wider block also admits the addresses in between, so this is your decision to make; From f4d7b54f066d9ea3afbc5bd997f1bc5915983652 Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 13:48:58 +0100 Subject: [PATCH 09/10] docs(edge-security): the connecting check is proven live; enforcement lag and the remaining gap Measured on the proof stack at 60afa9ae with connecting_asns = [13335]: 31 of 31 direct requests carrying the secret were denied by rule 150, and none of 668 requests through Cloudflare were. The guide now says to poll a direct request with the header until every backend refuses it (a new rule took almost 4 minutes), and names the gap that remains: another customer of the same proxy who holds the secret. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 12 ++++++++++++ terraform/modules/gcp/edge-security/README.md | 2 +- 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index 9e861468..f4a2da0e 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -262,6 +262,18 @@ Cloudflare can reach it. not come from Cloudflare's network is refused before any of these rules. Terraform refuses `client_ip_rules` without it. + After applying it, wait until it is enforced before you rely on it: a new + rule took almost 4 minutes to reach every backend when measured, and until + then a direct request with the header was still admitted. Repeat the loop + from section 2 with the header added + (`-H "x-origin-auth: "`), from a machine that is not behind + Cloudflare, until all six lines end in `403`. + + One gap this cannot close: another Cloudflare customer who learned your + secret header could send requests through their own Cloudflare zone, which + arrive from the same network. Keep the header value secret, rotate it if + it may have leaked (item 3), and treat it like a password. + The first rule that matches a request's path decides it: its listed clients are admitted, everyone else gets 403. The last rule closes every other path. A request that a rule admits is not checked by any later diff --git a/terraform/modules/gcp/edge-security/README.md b/terraform/modules/gcp/edge-security/README.md index 6dcc5ea8..2fd03282 100644 --- a/terraform/modules/gcp/edge-security/README.md +++ b/terraform/modules/gcp/edge-security/README.md @@ -63,7 +63,7 @@ origin_secret_header = { name = "x-origin-auth", values = [var.origin_secret] } A Cloud Armor rule that admits a request ends evaluation. So a request admitted by a client rule meets no later rule: not a later client rule, not `rate_limits`, and **not the allowlist of connecting addresses**. Three consequences, all enforced by the module: -- **Connections from outside the proxy must be denied first.** Otherwise the secret header is the only guard: a caller that holds it could connect straight to the load balancer and put any client address it likes in the header. (Measured on a live policy: such a caller was admitted as a staff address and as a vendor.) The proxy's ranges do not fit one negated expression, so name its network instead: `connecting_asns = [13335]` for Cloudflare, or `connecting_address_groups` with a Cloud Armor address group of its ranges (Enterprise). `secret_header_only = true` accepts the header as the sole guard. +- **Connections from outside the proxy must be denied first.** Otherwise the secret header is the only guard: a caller that holds it could connect straight to the load balancer and put any client address it likes in the header. (Measured on a live policy: such a caller was admitted as a staff address and as a vendor.) The proxy's ranges do not fit one negated expression, so name its network instead: `connecting_asns = [13335]` for Cloudflare, or `connecting_address_groups` with a Cloud Armor address group of its ranges (Enterprise). `secret_header_only = true` accepts the header as the sole guard. Measured on a live policy with `connecting_asns = [13335]`: 31 of 31 direct requests carrying the secret were denied by this rule, and none of 668 requests through Cloudflare were (two Cloudflare locations sampled). A new rule took almost 4 minutes to be enforced on every backend. What remains: another customer of the same proxy who holds your secret arrives from the same network, so the secret still has to stay secret. - **Every rule states its own limit**: `rate_limit = { count, interval_sec }` (per client address, 429 beyond it) or `unthrottled = true`. - **A rule for every path (`["/"]` without `hosts`) makes `rate_limits` unreachable**, so set `rate_limits = []` and express the sign-in limit as a narrower rule ahead of it, as below. From 73b6764271366424f04a9699a030504cdcd3d36d Mon Sep 17 00:00:00 2001 From: Miya Date: Thu, 19 Nov 2026 14:12:28 +0100 Subject: [PATCH 10/10] docs(edge-security): the connecting-check probe must claim an admitted client address Review feedback on #487: with the secret header alone, a test machine outside the client ranges gets 403 from the client rules themselves, so the probe passed before the connecting check was enforced. It now also sends CF-Connecting-IP with an address the staff rule admits, which is the bypass the check closes. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/3-EDGE-HARDENING.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/gcp-install/3-EDGE-HARDENING.md b/docs/gcp-install/3-EDGE-HARDENING.md index f4a2da0e..c9055d05 100644 --- a/docs/gcp-install/3-EDGE-HARDENING.md +++ b/docs/gcp-install/3-EDGE-HARDENING.md @@ -265,9 +265,16 @@ Cloudflare can reach it. After applying it, wait until it is enforced before you rely on it: a new rule took almost 4 minutes to reach every backend when measured, and until then a direct request with the header was still admitted. Repeat the loop - from section 2 with the header added - (`-H "x-origin-auth: "`), from a machine that is not behind - Cloudflare, until all six lines end in `403`. + from section 2 from a machine that is not behind Cloudflare, adding the + secret header **and** a client address your staff rule admits: + + ```sh + -H "x-origin-auth: " -H "CF-Connecting-IP: " + ``` + + until all six lines end in `403`. Both headers matter: without the + claimed address the client rules themselves answer 403, which would look + like success while the bypass is still open. One gap this cannot close: another Cloudflare customer who learned your secret header could send requests through their own Cloudflare zone, which