Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion docs/gcp-install/0-START-HERE.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ gcloud auth login && gcloud auth application-default login
gcloud config set project <project id>
gcloud services enable container.googleapis.com sqladmin.googleapis.com servicenetworking.googleapis.com \
redis.googleapis.com pubsub.googleapis.com secretmanager.googleapis.com dns.googleapis.com \
artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com
artifactregistry.googleapis.com compute.googleapis.com iamcredentials.googleapis.com \
cloudresourcemanager.googleapis.com
gcloud container clusters get-credentials <cluster> --location <region> # add --internal-ip for a private endpoint
kubectl auth can-i '*' '*' --all-namespaces
gcloud container clusters describe <cluster> --location <region> \
Expand Down Expand Up @@ -1259,6 +1260,21 @@ gcloud container clusters create agent-relay --location $R --num-nodes 1 \
--enable-shielded-nodes --enable-dataplane-v2
```

**`Operation denied by org policy ... default Compute Engine service
account`?** Your organization does not allow nodes to run as the default
service account. Create one for the nodes and pass it:

```sh
gcloud iam service-accounts create agent-relay-nodes
gcloud projects add-iam-policy-binding $P --role roles/container.defaultNodeServiceAccount \
--member "serviceAccount:agent-relay-nodes@$P.iam.gserviceaccount.com"
```

then add `--service-account agent-relay-nodes@$P.iam.gserviceaccount.com` to
the `clusters create` command, and in step 4 set
`gke_node_service_account = "agent-relay-nodes@<project id>.iam.gserviceaccount.com"`
so the image mirror grants it read access.

A regional cluster with `--num-nodes 1` gets one node in each of the region's
three zones. Then continue at step 1 with `<cluster>` = `agent-relay` and
`<cluster VPC>` = `agent-relay-vpc`, and grant the node service account in
Expand Down
93 changes: 82 additions & 11 deletions docs/gcp-install/3-EDGE-HARDENING.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,13 @@ when you run them.
edge_security_policy = { create = true }
```

Run `terraform apply`. `terraform output edge` prints its name
(`<name>-edge`). This policy has the OWASP rules in **preview** (they log
and do not block) and a limit of 60 sign-in requests a minute per client.
Run `terraform plan` first and read `rule_counts` in the `edge` output:
one policy holds at most 20 rules with an expression, and the plan tells
you what this policy needs (the defaults need 3). Then `terraform apply`,
and continue only if it finishes without an error. `terraform output edge`
prints the policy's name (`<name>-edge`). This policy has the OWASP rules
in **preview** (they log and do not block) and a limit of 60 sign-in
requests a minute per client.

2. Name it in `values.yaml`, and bind the dashboard Service to its
BackendConfig (`<gkeIngress.name>-dashboard`):
Expand Down Expand Up @@ -108,6 +112,27 @@ gcloud compute backend-services list --format='table(name,securityPolicy.basenam
Every BackendConfig shows the policy, and the count equals the number of
backend services of the Ingress (6 for the default install).

That list can be complete several minutes before every backend enforces the
policy (4 minutes when measured). If your policy refuses some requests (an
allowlist, Cloudflare, client rules), confirm with a request that it must
refuse, **on each backend**, before you rely on it. Each line below reaches a
different backend service:

```sh
IP=$(gcloud compute addresses describe $STATIC_IP_NAME --global --format='value(address)')
for url in "$DOMAIN/" "api.$DOMAIN/readyz" "cast.$DOMAIN/health" \
"history.$DOMAIN/healthz" "history.$DOMAIN/v1/delivery/limits" "history.$DOMAIN/"; do
h=${url%%/*}
curl -sk -o /dev/null -w "$url %{http_code}\n" --resolve "$h:443:$IP" "https://$url"
done
```

The six are the dashboard, the API, Relaycast, the history reader, the
history upload service and the deny service. From an address the policy does
not admit, every line must end in `403`. Repeat until they all do. Warnings
that mention `Translate` in `kubectl describe ingress` while the policy
attaches are transient.

**Turn the OWASP rules on.** Session transcripts and messages contain code,
SQL and shell text, which is what those rules look for. Use the install
normally for a week, then look for requests the rules would have blocked:
Expand Down Expand Up @@ -183,8 +208,8 @@ Cloudflare can reach it.
the address it received the request from, and its own. Measured on a live
GKE install, the header is `<client>, <load balancer>` and the right value
is 2. Cloudflare adds one more, so behind Cloudflare the header is
`<client>, <Cloudflare>, <load balancer>` and the value is 3. The
Cloudflare case is expected, not yet measured; check it as below.
`<client>, <Cloudflare>, <load balancer>` and the value is 3 (also
measured). Check yours as below.

```yaml
relay-control:
Expand Down Expand Up @@ -218,9 +243,10 @@ Cloudflare can reach it.

```hcl
edge_security_policy = {
create = true
cloudflare = { enabled = true }
rate_limits = []
create = true
cloudflare = { enabled = true }
connecting_asns = [13335] # Cloudflare's network; see below
rate_limits = []
client_ip_rules = [
{ name = "vendors", hosts = ["history.<domain>"], path_prefixes = ["/v1/delivery", "/v1/turns", "/v1/auth/token/refresh"], cidrs = ["<vendor range>", "<your range>"], rate_limit = { count = 600, interval_sec = 60 } },
{ name = "sign-in", path_prefixes = ["/login", "/api/v1/auth", "/cloud/api/v1/auth"], cidrs = ["<your range>"], rate_limit = { count = 60, interval_sec = 60 } },
Expand All @@ -229,14 +255,59 @@ Cloudflare can reach it.
}
```

`connecting_asns` matters here. A request that one of these rules admits
is not checked against Cloudflare's address ranges afterwards, so without
it anyone who learned the secret header could connect straight to the
load balancer and claim to be any client. With it, a connection that does
not come from Cloudflare's network is refused before any of these rules.
Terraform refuses `client_ip_rules` without it.

After applying it, wait until it is enforced before you rely on it: a new
rule took almost 4 minutes to reach every backend when measured, and until
then a direct request with the header was still admitted. Repeat the loop
from section 2 from a machine that is not behind Cloudflare, adding the
secret header **and** a client address your staff rule admits:

```sh
-H "x-origin-auth: <value>" -H "CF-Connecting-IP: <an address in your range>"
```

until all six lines end in `403`. Both headers matter: without the
claimed address the client rules themselves answer 403, which would look
like success while the bypass is still open.

One gap this cannot close: another Cloudflare customer who learned your
secret header could send requests through their own Cloudflare zone, which
arrive from the same network. Keep the header value secret, rotate it if
it may have leaked (item 3), and treat it like a password.

The first rule that matches a request's path decides it: its listed
clients are admitted, everyone else gets 403. The last rule closes every
other path. A request that a rule admits is not checked by any later
rule, so each rule carries its own `rate_limit` (per client), and the
sign-in limit is a rule of its own ahead of the closing one. The module's
README (`terraform/modules/gcp/edge-security`) explains the limits. A
vendor that publishes hundreds of ranges needs a higher Cloud Armor quota,
or that list enforced as a Cloudflare WAF custom rule instead.
README (`terraform/modules/gcp/edge-security`) explains the limits.

**A long vendor list does not fit.** One Cloud Armor policy holds at most
20 rules with an expression, and Google does not raise that limit. After
the secret header, the two WAF rules and the sign-in and staff rules,
about 13 are left: roughly 100 vendor ranges. Terraform refuses a plan
that needs more and says how many it counted (`rule_counts` in the `edge`
output shows the same). For a vendor that publishes more, merge adjacent
ranges where you accept the wider block, or enforce that vendor's list as
a Cloudflare WAF custom rule and keep only your own ranges here.

The project also has a limit of 20 such rules across all its policies
(`SECURITY_POLICY_CEVAL_RULES`). That one can be raised, and a request was
granted in seconds when measured. If another policy in the project already
uses some, the apply fails part-way with
`Quota 'SECURITY_POLICY_CEVAL_RULES' exceeded. Limit: 20.0 globally`.

**If an apply fails, do not use the policy yet.** It then holds an
arbitrary part of its rules, for example the rule that denies the sign-in
paths without the rule that admits you. Fix the cause and run
`terraform apply` again until it exits without an error. Only then set
`gkeIngress.securityPolicy`.

**Done when:**

Expand Down
4 changes: 3 additions & 1 deletion terraform/examples/gcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only).
| desktop\_feed\_bucket\_name | Globally unique bucket for relay-control's signed Desktop feed (SELFHOST\_DESKTOP\_FEED\_GCS\_BUCKET). Empty disables it. | `string` | `""` | no |
| desktop\_feed\_has\_signing\_keys | Whether relay-control has the feed's file signer (relay-control.desktopFeed.signingKeys in your values). true: chart\_values and desktop\_feed carry the SELFHOST\_DESKTOP\_FEED\_STORAGE env. Set false to create the bucket and service account before the signing keys exist: the env is left out, because relay-control exits when storage is set without a signer. Once the keys are in place set it back to true, apply (outputs change only on apply), and save chart\_values again. | `bool` | `true` | no |
| desktop\_feed\_service\_account\_id | account\_id of the Google service account relay-control uses for the Desktop feed bucket (6-30 characters). | `string` | `"agent-relay-control"` | no |
| edge\_max\_advanced\_rules | Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max\_advanced\_rules). Google fixes it at 20; a plan that needs more is refused. | `number` | `20` | no |
| edge\_max\_subexpressions | Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max\_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule. | `number` | `10` | no |
| edge\_origin\_secret\_header | Shared-secret request header a fronting proxy adds; requests without one of the accepted values (1-3, for rotation) are denied (modules/gcp/edge-security origin\_secret\_header). Stored in state. | ```object({ name = string values = list(string) })``` | `null` | no |
| edge\_security\_policy | Cloud Armor policy for every GKE Ingress backend (modules/gcp/edge-security security\_policy: create, waf, rate\_limits, source\_ip\_allowlist, cloudflare, client\_ip\_header, client\_ip\_rules). Off unless create = true. It is exported as gkeIngress.securityPolicy in output.edge, not in chart\_values, because the chart then also needs the dashboard Service's backend-config annotation. | `any` | `{}` | no |
| edge\_ssl\_policy | Compute SSL policy for the GKE Ingress HTTPS listener, exported as gkeIngress.sslPolicy in chart\_values. create = false leaves Google's default policy (TLS 1.0 and later). | ```object({ create = optional(bool, true) profile = optional(string, "MODERN") min_tls_version = optional(string, "TLS_1_2") })``` | `{}` | no |
Expand Down Expand Up @@ -137,7 +139,7 @@ Terraform or OpenTofu >= 1.11 (the Cloud SQL admin password is write-only).
| ---- | ----------- |
| chart\_values | Complete non-secret managed-service overlay for values-gcp.yaml plus values-gcp-native.yaml. Save with: terraform output -json chart\_values > managed-values.json. |
| desktop\_feed | Values for relay-control's signed Desktop feed on GCS (null unless desktop\_feed\_bucket\_name is set). Put env into relay-control.extraEnv and annotations into relay-control.serviceAccount.annotations; signing keys go through relay-control.desktopFeed.signingKeys. env is empty with desktop\_feed\_has\_signing\_keys = false: relay-control exits when the storage env is set without a signer. |
| edge | Edge policy names (edge.tf). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. |
| edge | Edge policy names (edge.tf). rule\_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY\_POLICY\_CEVAL\_RULES, default 20; total: SECURITY\_POLICY\_RULES, default 100). security\_policy is null unless edge\_security\_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client\_ip\_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2. |
| functions | Pipeline function, topics and secret (null unless enable\_functions). |
| image\_mirror\_root | MIRROR\_ROOT for install guide step 5 (null unless image\_mirror\_repository is set). |
| pipeline | Native Sessions pipeline values. The Redis CA is public certificate material, not a credential. |
Expand Down
2 changes: 2 additions & 0 deletions terraform/examples/gcp/edge.tf
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,6 @@ module "edge_security" {
ssl_policy = var.edge_ssl_policy
security_policy = var.edge_security_policy
origin_secret_header = var.edge_origin_secret_header
max_subexpressions = var.edge_max_subexpressions
max_advanced_rules = var.edge_max_advanced_rules
}
1 change: 1 addition & 0 deletions terraform/examples/gcp/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ module "postgres" {
# that identity to the private address inside this VPC so every service can use
# sslmode=verify-full without a proxy or public IP.
resource "google_dns_managed_zone" "cloud_sql" {
labels = var.labels
project = var.project_id
name = "${var.name}-cloud-sql"
dns_name = module.postgres.dns_name
Expand Down
3 changes: 2 additions & 1 deletion terraform/examples/gcp/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,12 @@ output "chart_values" {
}

output "edge" {
description = "Edge policy names (edge.tf). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2."
description = "Edge policy names (edge.tf). rule_counts is what the Cloud Armor policy needs against the project's quotas (advanced: SECURITY_POLICY_CEVAL_RULES, default 20; total: SECURITY_POLICY_RULES, default 100). security_policy is null unless edge_security_policy.create is set; put it in gkeIngress.securityPolicy together with the backend-config annotation the chart asks for. client_ip_header set means a proxy fronts the load balancer: relay-control.auth.trustedProxies is 3 instead of 2."
value = {
ssl_policy = module.edge_security.ssl_policy_name
security_policy = module.edge_security.security_policy_name
client_ip_header = module.edge_security.client_ip_header
rule_counts = module.edge_security.rule_counts
}
}

Expand Down
19 changes: 19 additions & 0 deletions terraform/examples/gcp/tests/native.tftest.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -256,4 +256,23 @@ run "edge_cloud_armor_behind_cloudflare" {
condition = output.edge.security_policy == "agent-relay-edge" && output.edge.client_ip_header == "CF-Connecting-IP"
error_message = "The Cloud Armor policy and the client-address header must be reported."
}
# Default of 10 subexpressions: secret header, two WAF rules, and two sign-in
# throttles for Cloudflare's 15 ranges (nine per rule).
assert {
condition = output.edge.rule_counts.advanced == 5 && output.edge.rule_counts.total == 8
error_message = "Behind Cloudflare the default policy needs 5 advanced rules and 8 in all."
}
}

run "edge_rule_counts_are_reported" {
command = apply
variables {
edge_security_policy = { create = true }
}

# Two WAF rules and the sign-in throttle have expressions; the default rule is basic.
assert {
condition = output.edge.rule_counts.advanced == 3 && output.edge.rule_counts.total == 4
error_message = "The default Cloud Armor policy needs 3 advanced rules and 4 rules in all."
}
}
12 changes: 12 additions & 0 deletions terraform/examples/gcp/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -259,3 +259,15 @@ variable "edge_origin_secret_header" {
default = null
sensitive = true
}

variable "edge_max_subexpressions" {
description = "Subexpressions per Cloud Armor rule expression (modules/gcp/edge-security max_subexpressions). 10 is what the API accepts; lower it only if your project refuses a rule."
type = number
default = 10
}

variable "edge_max_advanced_rules" {
description = "Rules with an expression one Cloud Armor policy may hold (modules/gcp/edge-security max_advanced_rules). Google fixes it at 20; a plan that needs more is refused."
type = number
default = 20
}
Loading
Loading