Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ Obtain these from the operator; do not discover or guess cloud identifiers:
- release and namespace, normally `agent-relay`;
- names of the global static IP, managed certificate, FrontendConfig, GCS
bucket, Kubernetes service accounts, and existing Kubernetes Secret;
- name of the Compute SSL policy (TLS 1.2 minimum) for `gkeIngress.sslPolicy`,
and, when the customer wants them, the Cloud Armor policy
(`gkeIngress.securityPolicy`) or a TLS Secret (`gkeIngress.tlsSecretName`);
see `docs/gcp-install/3-EDGE-HARDENING.md`;
- Artifact Registry repositories and immutable `sha256:` image digests;
- a customer-specific, non-secret Helm overlay exported as
`CUSTOMER_VALUES_FILE`;
Expand Down
2 changes: 1 addition & 1 deletion charts/agent-relay/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: agent-relay
description: Self-hosted Agent Relay messaging and sessions suite
type: application
version: 0.7.57
version: 0.7.59
appVersion: "0.1.0"
kubeVersion: ">=1.28.0-0"
annotations:
Expand Down
43 changes: 43 additions & 0 deletions charts/agent-relay/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ before installing. The required value overrides are:
| `global.imageRegistry` | Artifact Registry hostname for the selected region, such as `us-central1-docker.pkg.dev`. |
| `global.tenant.orgId` | Stable customer organization ID. Do not change it after data has been written. |
| `gkeIngress.name`, `staticIpName`, `managedCertificateName`, `frontendConfigName` | Customer-selected names. `staticIpName` must name a reserved global external IPv4 address; the chart creates the other resources. |
| `gkeIngress.sslPolicy` | Recommended. Name of a Compute SSL policy with a TLS 1.2 minimum; see [GKE edge hardening](#gke-edge-hardening). |
| `gkeIngress.hosts.{root,api,cast,history}` | Four customer-controlled public DNS names whose A records point at the reserved address. |
| `relaycast.image.{repository,digest}` | Customer-accessible Relaycast image mirror and its real immutable digest. |
| `sessions-pipeline.worker.image.{repository,digest}` and `sessions-pipeline.alleyoop.image.{repository,digest}` | Repository path under `global.imageRegistry` and the same immutable Sessions services digest. |
Expand Down Expand Up @@ -252,6 +253,48 @@ For a base domain `relay.example.com`:
Leaf ingresses are disabled and each ClusterIP backend uses a GKE standalone
NEG. Sensitive Relaycast paths still route to an endpoint-less deny Service.

## GKE edge hardening

Three optional `gkeIngress` values, each empty by default. Each names
something the chart cannot create, which must exist before the upgrade;
[`terraform/modules/gcp/edge-security`](https://github.com/AgentWorkforce/infra/blob/main/terraform/modules/gcp/edge-security)
creates the two policies. The operator steps and checks are in
[`docs/gcp-install/3-EDGE-HARDENING.md`](https://github.com/AgentWorkforce/infra/blob/main/docs/gcp-install/3-EDGE-HARDENING.md).

| Value | Effect |
| --- | --- |
| `gkeIngress.sslPolicy` | Compute SSL policy for the HTTPS listener (`FrontendConfig.spec.sslPolicy`). Without it Google's default policy applies, which still accepts TLS 1.0 and 1.1. Use a policy with `min_tls_version = TLS_1_2`. |
| `gkeIngress.securityPolicy` | Cloud Armor backend security policy, set as `spec.securityPolicy.name` on the BackendConfig of **every** backend behind the Ingress. |
| `gkeIngress.tlsSecretName` | A `kubernetes.io/tls` Secret served instead of the ManagedCertificate (which is then not rendered). For an edge whose DNS points at a proxy or CDN, where Google cannot issue a managed certificate. The certificate must cover every host. |

The chart leaves `sslPolicy` empty by default because a name that does not
exist in the project stops the Ingress from syncing. `terraform/examples/gcp`
creates the policy and exports it in `chart_values`, so an install made from
the GKE guide has it set.

With `securityPolicy`, the api, cast, history, actors and relaygate
BackendConfigs gain the policy, and the chart renders policy-only BackendConfigs
for the backends that had none: `<gkeIngress.name>-dashboard`,
`<gkeIngress.name>-edge-deny` and, with `gkeIngress.hosts.files`,
`<gkeIngress.name>-files`. The dashboard and Relayfile Services belong to child
charts, so bind them in values (the render fails with the exact value
otherwise):

```yaml
gkeIngress:
sslPolicy: agent-relay-tls
securityPolicy: agent-relay-edge
relay-webapp:
service:
annotations:
cloud.google.com/neg: '{"ingress": true}'
cloud.google.com/backend-config: '{"default": "agent-relay-dashboard"}'
```

Behind a proxy that fronts the load balancer (Cloudflare), also raise
`relay-control.auth.trustedProxies` by one (the Google load balancer alone
needs 2; the guide shows how to check).

## Customer-neutral EKS with RDS native History

`values-eks-rds-native.yaml` is the reusable EKS overlay for the supported
Expand Down
92 changes: 91 additions & 1 deletion charts/agent-relay/templates/gke-ingress.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,25 @@
{{- include "agent-relay.validate" . -}}
{{- if dig "enabled" false (.Values.gkeIngress | default dict) }}
{{- $gke := .Values.gkeIngress | default dict -}}
{{- /* Edge hardening, all optional and off by default (each names a Google
Cloud resource this chart cannot create; terraform/modules/gcp/edge-security
creates them):
sslPolicy Compute SSL policy for the load balancer's HTTPS listener
(FrontendConfig spec.sslPolicy). Without it Google's default
policy applies, which still accepts TLS 1.0.
securityPolicy Cloud Armor backend security policy, set on the
BackendConfig of every backend Service behind the Ingress.
tlsSecretName kubernetes.io/tls Secret served instead of the
ManagedCertificate, for an edge whose public DNS points at a
proxy in front of the load balancer (a Google-managed
certificate cannot be issued there). */ -}}
{{- $computeName := "^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$" -}}
{{- $sslPolicy := dig "sslPolicy" "" $gke | toString -}}
{{- if and $sslPolicy (not (regexMatch $computeName $sslPolicy)) }}{{ fail "gkeIngress.sslPolicy must be the name of a Compute Engine SSL policy in the cluster's project (lowercase letters, digits and hyphens, at most 63 characters), not a URL" }}{{ end -}}
{{- $securityPolicy := dig "securityPolicy" "" $gke | toString -}}
{{- if and $securityPolicy (not (regexMatch $computeName $securityPolicy)) }}{{ fail "gkeIngress.securityPolicy must be the name of a Cloud Armor security policy in the cluster's project (lowercase letters, digits and hyphens, at most 63 characters), not a URL" }}{{ end -}}
{{- $tlsSecretName := dig "tlsSecretName" "" $gke | toString -}}
{{- if and $tlsSecretName (not (regexMatch "^[a-z0-9]([-a-z0-9.]{0,251}[a-z0-9])?$" $tlsSecretName)) }}{{ fail "gkeIngress.tlsSecretName must be the name of a kubernetes.io/tls Secret in the release namespace" }}{{ end -}}
{{- $components := .Values.components | default dict -}}
{{- $relaycast := .Values.relaycast | default dict -}}
{{- $cloudApi := index .Values "relay-control" | default dict -}}
Expand Down Expand Up @@ -59,7 +78,8 @@ shape (hosts.files without components.relayfile fails further down). */ -}}
{{- $nativeReadEdge := printf "%s-history-read" ($gkeName | trunc 50 | trimSuffix "-") -}}
{{- $nativeIngestEdge := printf "%s-history-ingest" ($gkeName | trunc 48 | trimSuffix "-") -}}
{{- $staticIpName := required "gkeIngress.staticIpName is required when gkeIngress.enabled=true" (dig "staticIpName" "" $gke) -}}
{{- $managedCertificateName := required "gkeIngress.managedCertificateName is required when gkeIngress.enabled=true" (dig "managedCertificateName" "" $gke) -}}
{{- $managedCertificateName := dig "managedCertificateName" "" $gke -}}
{{- if not (or $tlsSecretName $managedCertificateName) }}{{ fail "gkeIngress.managedCertificateName is required when gkeIngress.enabled=true (or set gkeIngress.tlsSecretName to serve a certificate from a Secret)" }}{{ end -}}
{{- $frontendConfigName := required "gkeIngress.frontendConfigName is required when gkeIngress.enabled=true" (dig "frontendConfigName" "" $gke) -}}
{{- $hostRoot := dig "hosts" "root" "" $gke -}}
{{- $hostApi := dig "hosts" "api" "" $gke -}}
Expand Down Expand Up @@ -172,6 +192,38 @@ derived from gkeIngress.name. The name is truncated to 59 characters before
{{- $haveAnnotation := dig "service" "annotations" "cloud.google.com/backend-config" "" $cloudApi -}}
{{- if ne (toString $haveAnnotation | nospace) (nospace $wantBackendConfig) }}{{ fail (printf "gkeIngress requires relay-control.service.backendConfig=%s (<gkeIngress.name>-api) so the api backend gets the BackendConfig timeout" $apiBackendConfig) }}{{ end -}}
{{- end }}
{{- /* Cloud Armor attaches per backend service, through the BackendConfig
its Service names. The dashboard, Relayfile and edge-deny Services have no
BackendConfig otherwise, so with gkeIngress.securityPolicy each gets one that
carries only the policy. The edge-deny Service is rendered here and binds its
own; the dashboard and Relayfile Services belong to child charts, whose
values a parent cannot template, so their backend-config annotation is
checked against the derived name (the render fails with the value to set). */ -}}
{{- $dashboardBackendConfig := printf "%s-dashboard" ($gkeName | trunc 53 | trimSuffix "-") -}}
{{- $filesBackendConfig := printf "%s-files" ($gkeName | trunc 57 | trimSuffix "-") -}}
{{- $denyBackendConfig := printf "%s-edge-deny" ($gkeName | trunc 53 | trimSuffix "-") -}}
{{- if $securityPolicy }}
{{- $armorOnly := dict $dashboardBackendConfig (list "relay-webapp" $dashboard) -}}
{{- if $hostFiles }}{{ $_ := set $armorOnly $filesBackendConfig (list "relayfile" $relayfile) }}{{ end -}}
{{- range $bc, $child := $armorOnly }}
{{- $have := dig "service" "annotations" "cloud.google.com/backend-config" "" (index $child 1) | toString | nospace -}}
{{- if ne $have (printf "{\"default\":\"%s\"}" $bc) }}{{ fail (printf "gkeIngress.securityPolicy requires %s.service.annotations[cloud.google.com/backend-config]='{\"default\": \"%s\"}' so the Cloud Armor policy also covers that backend" (index $child 0) $bc) }}{{ end -}}
{{- end }}
{{- range $bc := concat (list $dashboardBackendConfig $denyBackendConfig) (ternary (list $filesBackendConfig) (list) (not (empty $hostFiles))) }}
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: {{ $bc }}
namespace: {{ $.Release.Namespace | quote }}
labels:
app.kubernetes.io/name: agent-relay
app.kubernetes.io/instance: {{ $.Release.Name }}
spec:
securityPolicy:
name: {{ $securityPolicy | quote }}
---
{{- end }}
{{- end }}
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
Expand All @@ -181,6 +233,10 @@ metadata:
app.kubernetes.io/name: agent-relay
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
timeoutSec: {{ $apiTimeout }}
---
{{- /* Relaycast WebSockets (/v1/ws, /v1/node/ws) through the cast backend are
Expand Down Expand Up @@ -208,6 +264,10 @@ metadata:
app.kubernetes.io/name: agent-relay
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
timeoutSec: {{ $castTimeout }}
---
{{- $gateEdge := printf "%s-gate" ($gkeName | trunc 58 | trimSuffix "-") -}}
Expand All @@ -226,6 +286,10 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: gate-edge
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
timeoutSec: {{ $castTimeout }}
healthCheck:
type: HTTP
Expand Down Expand Up @@ -253,6 +317,7 @@ spec:
- { name: http, port: 8080, targetPort: http, protocol: TCP }
---
{{- end }}
{{- if not $tlsSecretName }}
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
Expand All @@ -273,6 +338,7 @@ spec:
- {{ . | quote }}
{{- end }}
---
{{- end }}
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
Expand All @@ -282,6 +348,9 @@ metadata:
app.kubernetes.io/name: agent-relay
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- with $sslPolicy }}
sslPolicy: {{ . | quote }}
{{- end }}
redirectToHttps:
enabled: true
responseCodeName: MOVED_PERMANENTLY_DEFAULT
Expand All @@ -300,6 +369,9 @@ metadata:
app.kubernetes.io/component: edge-deny
annotations:
cloud.google.com/neg: '{"ingress": true}'
{{- if $securityPolicy }}
cloud.google.com/backend-config: {{ printf "{\"default\": \"%s\"}" $denyBackendConfig | squote }}
{{- end }}
spec:
type: ClusterIP
ports:
Expand All @@ -323,6 +395,10 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: actors-edge
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
timeoutSec: {{ $actorsTimeout | int }}
healthCheck:
type: HTTP
Expand Down Expand Up @@ -362,6 +438,10 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: native-history-read-edge
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
healthCheck:
type: HTTP
requestPath: /healthz
Expand All @@ -377,6 +457,10 @@ metadata:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: native-history-ingest-edge
spec:
{{- with $securityPolicy }}
securityPolicy:
name: {{ . | quote }}
{{- end }}
healthCheck:
type: HTTP
requestPath: /healthz
Expand Down Expand Up @@ -435,9 +519,15 @@ metadata:
annotations:
kubernetes.io/ingress.class: gce
kubernetes.io/ingress.global-static-ip-name: {{ $staticIpName | quote }}
{{- if not $tlsSecretName }}
networking.gke.io/managed-certificates: {{ $managedCertificateName | quote }}
{{- end }}
networking.gke.io/v1beta1.FrontendConfig: {{ $frontendConfigName | quote }}
spec:
{{- if $tlsSecretName }}
tls:
- secretName: {{ $tlsSecretName | quote }}
{{- end }}
rules:
- host: {{ $hostRoot | quote }}
http:
Expand Down
14 changes: 14 additions & 0 deletions charts/agent-relay/values-gcp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,18 @@ gkeIngress:
staticIpName: example-agent-relay-ingress
managedCertificateName: example-agent-relay
frontendConfigName: example-agent-relay
# RECOMMENDED: the name of a Compute SSL policy with min TLS 1.2 (create it
# first: docs/gcp-install step 7 or terraform/modules/gcp/edge-security).
# Left empty, the load balancer uses Google's default policy, which still
# accepts TLS 1.0 and 1.1.
# sslPolicy: example-agent-relay-tls
# Optional: a Cloud Armor security policy for every backend (README: GKE
# edge hardening). Also set the dashboard Service's backend-config
# annotation, relay-webapp.service.annotations below.
# securityPolicy: example-agent-relay-edge
# Optional: serve a certificate from a kubernetes.io/tls Secret instead of
# the ManagedCertificate (an edge behind a proxy or CDN).
# tlsSecretName: example-agent-relay-tls
hosts:
# REQUIRED OVERRIDE: four public DNS names controlled by the customer.
root: dashboard.placeholder.company
Expand Down Expand Up @@ -495,6 +507,8 @@ relay-webapp:
service:
annotations:
cloud.google.com/neg: '{"ingress": true}'
# Required with gkeIngress.securityPolicy: <gkeIngress.name>-dashboard.
# cloud.google.com/backend-config: '{"default": "example-agent-relay-dashboard"}'
ingress:
enabled: false
networkPolicy:
Expand Down
19 changes: 19 additions & 0 deletions charts/agent-relay/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,25 @@ gkeIngress:
staticIpName: agent-relay-ingress
managedCertificateName: agent-relay
frontendConfigName: agent-relay
# Optional edge hardening. Each names a Google Cloud resource in the
# cluster's project that must exist before the upgrade (the chart cannot
# create it; terraform/modules/gcp/edge-security does). README: GKE edge
# hardening.
#
# Compute SSL policy for the HTTPS listener. Empty leaves Google's default
# policy, which still accepts TLS 1.0 and 1.1; set a policy with
# min TLS 1.2 (profile MODERN or RESTRICTED).
sslPolicy: ""
# Cloud Armor backend security policy, applied to every backend Service
# behind the Ingress. Needs the backend-config annotation on the
# relay-webapp Service (and relayfile with hosts.files); the render fails
# with the exact value when it is missing.
securityPolicy: ""
# kubernetes.io/tls Secret served instead of the ManagedCertificate (which
# is then not rendered). For an edge whose DNS points at a proxy or CDN in
# front of the load balancer, where Google cannot issue a managed
# certificate. The certificate must cover every host below.
tlsSecretName: ""
hosts:
root: ""
api: ""
Expand Down
Loading
Loading