diff --git a/charts/agent-relay/README.md b/charts/agent-relay/README.md index 294c8f6a..746d02ee 100644 --- a/charts/agent-relay/README.md +++ b/charts/agent-relay/README.md @@ -155,7 +155,8 @@ image from relaycontrol #13 or later. so the bootstrap admin can sign in once with the setup key (24 hours, single use) and set a password, then hand out temporary passwords to members. `relay-control.auth.trustedProxies` (default `1` with password sign-in) - must equal the number of proxies in front of the API. Both need a + must equal the number of `X-Forwarded-For` entries the proxies in front of + the API append: `2` behind the GKE Ingress, whose load balancer appends two. Both need a relay-control image from source `bd1c9a03` or later. See the [relay-control chart](https://github.com/AgentWorkforce/infra/blob/main/charts/relay-control/README.md#first-admin-without-smtp-or-sso-the-setup-key). diff --git a/charts/relay-control/README.md b/charts/relay-control/README.md index b0e8507d..72c6e9f5 100644 --- a/charts/relay-control/README.md +++ b/charts/relay-control/README.md @@ -177,7 +177,12 @@ address, so the API must know how many proxies append to `X-Forwarded-For`. `1` with password sign-in, which is right only when this chart's ingress (or the single load balancer in front of it) is the one proxy and direct access to the pod and Service is restricted, for example with `networkPolicy`. Set the -real hop count for any other chain, or `0` when clients connect directly. The +real hop count for any other chain, or `0` when clients connect directly. +Count the `X-Forwarded-For` entries the chain appends, not the proxies: a +Google Cloud external Application Load Balancer (GKE Ingress) appends two (the +client's address, then its own), so the value there is `2`; with `1` every +user shares the load balancer's address, and with it one rate limit and +lockout. The API refuses to start with password sign-in and no value. `SELFHOST_SIGNIN_METHODS`, `SELFHOST_TRUSTED_PROXIES` and `SELFHOST_SETUP_KEY_FILE` are managed values: move any `extraEnv` entries for them to `auth.password`, `auth.trustedProxies` diff --git a/docs/gcp-install/0-START-HERE.md b/docs/gcp-install/0-START-HERE.md index 9090a2d0..a642194e 100644 --- a/docs/gcp-install/0-START-HERE.md +++ b/docs/gcp-install/0-START-HERE.md @@ -847,6 +847,49 @@ reads (`/turns`, replay, events) are fixed in 2026.11.6 and verified on a GKE install. If you uploaded sessions on 2026.11.1 to 2026.11.5, run "Re-stamp sessions uploaded before 2026.11.6" below once after upgrading. +## Set `trustedProxies` to 2 (installs made from an earlier copy of this guide) + +Earlier copies of `values.yaml` had `relay-control.auth.trustedProxies: 1`. +The Google load balancer adds two entries to `X-Forwarded-For` (the client's +address, then its own), so with 1 the API takes the load balancer's address +as every client's address. Sign-in then works, but all users share one +per-address rate limit and one password lockout: a few wrong passwords from +anyone can lock everyone out for a while. + +Check what your pods receive: + +```sh +kubectl -n $NAMESPACE logs deploy/relay-webapp -c dashboard --tail 5 +``` + +The last quoted field of each line is `X-Forwarded-For`. With the load +balancer alone it is `, `: two entries. +In `values.yaml` set: + +```yaml +relay-control: + auth: + trustedProxies: 2 +``` + +and run step 11's `helm upgrade` again. + +**A proxy or CDN in front of the load balancer** adds one more entry, so the +count is one higher. Raise it only once the load balancer refuses requests +that do not come through the proxy (for example with a Cloud Armor policy +that admits only the proxy's addresses). Until then keep 2: a caller who +connects to the load balancer directly can put any address in +`X-Forwarded-For`, and with the higher count the API would take that forged +address as the client's. + +**Done when:** + +```sh +kubectl -n $NAMESPACE get deploy relay-control -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="SELFHOST_TRUSTED_PROXIES")].value}{"\n"}' +``` + +prints `2`. + ## Re-stamp sessions uploaded before 2026.11.6 (upgrades only) Releases 2026.11.1 to 2026.11.5 recorded sessions uploaded to GCS as if they diff --git a/docs/gcp-install/values.yaml b/docs/gcp-install/values.yaml index 67ed95e0..2db306c3 100644 --- a/docs/gcp-install/values.yaml +++ b/docs/gcp-install/values.yaml @@ -273,8 +273,13 @@ relay-control: auth: password: enabled: true - # One hop: the Google load balancer. Rate limits depend on this being right. - trustedProxies: 1 + # The Google load balancer appends two X-Forwarded-For entries: the + # client's address, then its own. 2 selects the client; 1 would select the + # load balancer's address for every user, so all users would share one + # sign-in limit and lockout. With a proxy or CDN in front, add one, but + # only after the load balancer refuses requests that skip the proxy; + # otherwise a direct caller can forge the address the API trusts. + trustedProxies: 2 selectedFeatures: [usage, launch] publicUrl: https://api.__DOMAIN__ dashboardUrl: https://__DOMAIN__/