From 1c39ccdd3c8fec9b0a433166efdf6115135053d3 Mon Sep 17 00:00:00 2001 From: Miya Date: Wed, 18 Nov 2026 23:31:14 +0100 Subject: [PATCH 1/2] fix(gcp-install): trustedProxies is 2 behind the GKE Ingress, not 1 Measured on a live GKE install (read-only): the Google external Application Load Balancer appends two X-Forwarded-For entries, ", ". relay-control takes the Nth entry from the right, so the guide's trustedProxies: 1 selected the load balancer's address for every client: all users shared one per-address sign-in rate limit and one password lockout. - docs/gcp-install/values.yaml: trustedProxies: 2, with the reason. - docs/gcp-install/0-START-HERE.md: a section for installs made from an earlier copy of the guide: how to read the header from the relay-webapp access log, the value to set, and a "Done when" check. - relay-control and agent-relay READMEs: count the entries a proxy appends, not the proxies. Docs and the guide's values template only; no chart template or default changes, so no chart version bump. Co-Authored-By: Claude Fable 5.2 --- charts/agent-relay/README.md | 3 ++- charts/relay-control/README.md | 7 ++++++- docs/gcp-install/0-START-HERE.md | 36 ++++++++++++++++++++++++++++++++ docs/gcp-install/values.yaml | 7 +++++-- 4 files changed, 49 insertions(+), 4 deletions(-) diff --git a/charts/agent-relay/README.md b/charts/agent-relay/README.md index 294c8f6a..746d02ee 100644 --- a/charts/agent-relay/README.md +++ b/charts/agent-relay/README.md @@ -155,7 +155,8 @@ image from relaycontrol #13 or later. so the bootstrap admin can sign in once with the setup key (24 hours, single use) and set a password, then hand out temporary passwords to members. `relay-control.auth.trustedProxies` (default `1` with password sign-in) - must equal the number of proxies in front of the API. Both need a + must equal the number of `X-Forwarded-For` entries the proxies in front of + the API append: `2` behind the GKE Ingress, whose load balancer appends two. Both need a relay-control image from source `bd1c9a03` or later. See the [relay-control chart](https://github.com/AgentWorkforce/infra/blob/main/charts/relay-control/README.md#first-admin-without-smtp-or-sso-the-setup-key). diff --git a/charts/relay-control/README.md b/charts/relay-control/README.md index b0e8507d..72c6e9f5 100644 --- a/charts/relay-control/README.md +++ b/charts/relay-control/README.md @@ -177,7 +177,12 @@ address, so the API must know how many proxies append to `X-Forwarded-For`. `1` with password sign-in, which is right only when this chart's ingress (or the single load balancer in front of it) is the one proxy and direct access to the pod and Service is restricted, for example with `networkPolicy`. Set the -real hop count for any other chain, or `0` when clients connect directly. The +real hop count for any other chain, or `0` when clients connect directly. +Count the `X-Forwarded-For` entries the chain appends, not the proxies: a +Google Cloud external Application Load Balancer (GKE Ingress) appends two (the +client's address, then its own), so the value there is `2`; with `1` every +user shares the load balancer's address, and with it one rate limit and +lockout. The API refuses to start with password sign-in and no value. `SELFHOST_SIGNIN_METHODS`, `SELFHOST_TRUSTED_PROXIES` and `SELFHOST_SETUP_KEY_FILE` are managed values: move any `extraEnv` entries for them to `auth.password`, `auth.trustedProxies` diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index 9090a2d0..c7eca5e5 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -847,6 +847,42 @@ reads (`/turns`, replay, events) are fixed in 2026.11.6 and verified on a GKE install. If you uploaded sessions on 2026.11.1 to 2026.11.5, run "Re-stamp sessions uploaded before 2026.11.6" below once after upgrading. +## Set `trustedProxies` to 2 (installs made from an earlier copy of this guide) + +Earlier copies of `values.yaml` had `relay-control.auth.trustedProxies: 1`. +The Google load balancer adds two entries to `X-Forwarded-For` (the client's +address, then its own), so with 1 the API takes the load balancer's address +as every client's address. Sign-in then works, but all users share one +per-address rate limit and one password lockout: a few wrong passwords from +anyone can lock everyone out for a while. + +Check what your pods receive: + +```sh +kubectl -n $NAMESPACE logs deploy/relay-webapp -c dashboard --tail 5 +``` + +The last quoted field of each line is `X-Forwarded-For`. With the load +balancer alone it is `, `: two entries. +In `values.yaml` set: + +```yaml +relay-control: + auth: + trustedProxies: 2 +``` + +and run step 11's `helm upgrade` again. With a proxy or CDN in front of the +load balancer the header has one more entry; use that count. + +**Done when:** + +```sh +kubectl -n $NAMESPACE get deploy relay-control -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="SELFHOST_TRUSTED_PROXIES")].value}{"\n"}' +``` + +prints `2`. + ## Re-stamp sessions uploaded before 2026.11.6 (upgrades only) Releases 2026.11.1 to 2026.11.5 recorded sessions uploaded to GCS as if they diff --git a/docs/gcp-install/values.yaml b/docs/gcp-install/values.yaml index 67ed95e0..05029a7a 100644 --- a/docs/gcp-install/values.yaml +++ b/docs/gcp-install/values.yaml @@ -273,8 +273,11 @@ relay-control: auth: password: enabled: true - # One hop: the Google load balancer. Rate limits depend on this being right. - trustedProxies: 1 + # The Google load balancer appends two X-Forwarded-For entries: the + # client's address, then its own. 2 selects the client; 1 would select the + # load balancer's address for every user, so all users would share one + # sign-in limit and lockout. Add one more for each proxy or CDN in front. + trustedProxies: 2 selectedFeatures: [usage, launch] publicUrl: https://api.__DOMAIN__ dashboardUrl: https://__DOMAIN__/ From 8b8a6e2da0a4cad0e9abdac955468bd561f5be18 Mon Sep 17 00:00:00 2001 From: Miya Date: Wed, 18 Nov 2026 23:38:23 +0100 Subject: [PATCH 2/2] fix(gcp-install): raise trustedProxies for a CDN only once direct requests are refused Review feedback on #479: with a count one higher for a fronting proxy, a caller who connects to the load balancer directly can put any address in X-Forwarded-For and have the API trust it. The guide and the values template now say to raise the count only after the load balancer refuses requests that skip the proxy. Co-Authored-By: Claude Fable 5.2 --- docs/gcp-install/0-START-HERE.md | 11 +++++++++-- docs/gcp-install/values.yaml | 4 +++- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index c7eca5e5..a642194e 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -872,8 +872,15 @@ relay-control: trustedProxies: 2 ``` -and run step 11's `helm upgrade` again. With a proxy or CDN in front of the -load balancer the header has one more entry; use that count. +and run step 11's `helm upgrade` again. + +**A proxy or CDN in front of the load balancer** adds one more entry, so the +count is one higher. Raise it only once the load balancer refuses requests +that do not come through the proxy (for example with a Cloud Armor policy +that admits only the proxy's addresses). Until then keep 2: a caller who +connects to the load balancer directly can put any address in +`X-Forwarded-For`, and with the higher count the API would take that forged +address as the client's. **Done when:** diff --git a/docs/gcp-install/values.yaml b/docs/gcp-install/values.yaml index 05029a7a..2db306c3 100644 --- a/docs/gcp-install/values.yaml +++ b/docs/gcp-install/values.yaml @@ -276,7 +276,9 @@ relay-control: # The Google load balancer appends two X-Forwarded-For entries: the # client's address, then its own. 2 selects the client; 1 would select the # load balancer's address for every user, so all users would share one - # sign-in limit and lockout. Add one more for each proxy or CDN in front. + # sign-in limit and lockout. With a proxy or CDN in front, add one, but + # only after the load balancer refuses requests that skip the proxy; + # otherwise a direct caller can forge the address the API trusts. trustedProxies: 2 selectedFeatures: [usage, launch] publicUrl: https://api.__DOMAIN__