Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/agent-relay/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: agent-relay
description: Self-hosted Agent Relay messaging and sessions suite
type: application
version: 0.7.59
version: 0.7.60
appVersion: "0.1.0"
kubeVersion: ">=1.28.0-0"
annotations:
Expand Down
43 changes: 43 additions & 0 deletions charts/agent-relay/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,49 @@ Behind a proxy that fronts the load balancer (Cloudflare), also raise
`relay-control.auth.trustedProxies` by one (the Google load balancer alone
needs 2; the guide shows how to check).

## Internal-only GKE edge

For an install that must not be reachable from the internet, set
`gkeIngress.mode: internal` (start from `values-gcp-internal.yaml`). GKE then
creates a regional **internal** Application Load Balancer: a private address,
HTTPS only, reachable from the VPC and from the networks routed to it (VPN,
Interconnect). It is the GKE counterpart of an internal-scheme ALB on EKS.
Hosts, routes, backends and timeouts are the same as on the external edge.

| | External (default) | Internal |
| --- | --- | --- |
| Ingress class | `gce` | `gce-internal` |
| `gkeIngress.staticIpName` | a global external address | a **regional internal** address |
| Certificate | ManagedCertificate, or `tlsSecretName` | `tlsSecretName`, or `internal.preSharedCerts` (regional self-managed certificates). A Google-managed certificate is not possible |
| HTTP | redirected to HTTPS | closed (`allow-http: "false"`) |
| `sslPolicy`, `securityPolicy` | optional | refused: GKE Ingress supports neither FrontendConfig nor Cloud Armor on an internal load balancer |
| Sources the pods see | Google front ends | the VPC's **proxy-only subnet** |

Before the install, the VPC needs a proxy-only subnet in the cluster's region,
a reserved regional internal address, and a firewall rule from that subnet to
the nodes;
[`terraform/modules/gcp/internal-edge`](https://github.com/AgentWorkforce/infra/blob/main/terraform/modules/gcp/internal-edge)
creates the three. Then:

- `gkeIngress.internal.proxyOnlySubnetCidr` is the subnet's range, and every
backend's `networkPolicy` must allow it next to the two health-check ranges.
The render fails, naming the backend, when one does not.
`values-gcp-internal.yaml` restates all five lists.
- DNS: A records for every host, pointing at the private address, in a zone
your VPN clients resolve (a private zone, or your corporate DNS). The public
URLs in the values stay `https://<host>`.
- Certificate: with `tlsSecretName`, cert-manager with a DNS-01 challenge
issues and renews a publicly trusted certificate without the hosts being
reachable from the internet. If the DNS zone is in another Google Cloud
project, grant cert-manager's Workload Identity `roles/dns.admin` on that
zone's project. A private CA works if every desktop, CLI and agent trusts it.
- Agents and desktops outside the VPN cannot upload sessions or connect. Plan
for that before choosing this mode.
- Pass the overlay on every `helm upgrade`, or merge it into your site values.
An upgrade without it renders the external edge again.

Rendered and tested in CI; not yet proven on a live cluster.

## Customer-neutral EKS with RDS native History

`values-eks-rds-native.yaml` is the reusable EKS overlay for the supported
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ spec:
{{- if dig "hosts" "actors" "" (.Values.gkeIngress | default dict) }}
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
{{- if eq (dig "mode" "external" (.Values.gkeIngress | default dict) | toString) "internal" }}
# The internal load balancer's proxies (gkeIngress.internal.proxyOnlySubnetCidr).
- ipBlock: { cidr: {{ dig "internal" "proxyOnlySubnetCidr" "" (.Values.gkeIngress | default dict) | quote }} }
{{- end }}
{{- end }}
{{- range dig "extraIngressFrom" (list) $policy }}
- {{- toYaml . | nindent 10 }}
Expand Down
56 changes: 53 additions & 3 deletions charts/agent-relay/templates/gke-ingress.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ creates them):
{{- if and $securityPolicy (not (regexMatch $computeName $securityPolicy)) }}{{ fail "gkeIngress.securityPolicy must be the name of a Cloud Armor security policy in the cluster's project (lowercase letters, digits and hyphens, at most 63 characters), not a URL" }}{{ end -}}
{{- $tlsSecretName := dig "tlsSecretName" "" $gke | toString -}}
{{- if and $tlsSecretName (not (regexMatch "^[a-z0-9]([-a-z0-9.]{0,251}[a-z0-9])?$" $tlsSecretName)) }}{{ fail "gkeIngress.tlsSecretName must be the name of a kubernetes.io/tls Secret in the release namespace" }}{{ end -}}
{{- $gkeInternal := eq (dig "mode" "external" $gke | toString) "internal" -}}
{{- $gkeProxyCidr := dig "internal" "proxyOnlySubnetCidr" "" $gke | toString -}}
{{- if $gkeInternal }}{{- if not (regexMatch "^(10(\\.(25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])){3}|172\\.(1[6-9]|2[0-9]|3[01])(\\.(25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])){2}|192\\.168(\\.(25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])){2})/(8|9|1[0-9]|2[0-6])$" $gkeProxyCidr) }}{{ fail "gkeIngress.mode=internal requires gkeIngress.internal.proxyOnlySubnetCidr: the RFC 1918 range (/26 or larger) of the VPC's proxy-only subnet (purpose REGIONAL_MANAGED_PROXY) in the cluster's region; the load balancer's proxies connect to the pods from it" }}{{ end -}}{{ end -}}
{{- $components := .Values.components | default dict -}}
{{- $relaycast := .Values.relaycast | default dict -}}
{{- $cloudApi := index .Values "relay-control" | default dict -}}
Expand Down Expand Up @@ -53,10 +56,13 @@ shape (hosts.files without components.relayfile fails further down). */ -}}
{{- range $name, $peers := ternary (dict) (dict "sessionReadApi.networkPolicy.publicFrom" (dig "publicFrom" (list) $readerPolicy) "ingestEdge.networkPolicy.ingressFrom" (dig "ingressFrom" (list) $ingestPolicy)) (not (empty $gateOnly)) }}
{{- $hasGfePrimary := false -}}
{{- $hasGfeSecondary := false -}}
{{- $hasProxyOnly := false -}}
{{- range $peer := $peers }}
{{- if and (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) "35.191.0.0/16") (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasGfePrimary = true }}{{ end -}}
{{- if and (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) "130.211.0.0/22") (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasGfeSecondary = true }}{{ end -}}
{{- if and $gkeInternal (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) $gkeProxyCidr) (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasProxyOnly = true }}{{ end -}}
{{- end }}
{{- if and $gkeInternal (not $hasProxyOnly) }}{{ fail (printf "gkeIngress.mode=internal requires sessions-pipeline.%s to allow the proxy-only subnet %s (gkeIngress.internal.proxyOnlySubnetCidr) without exclusions" $name $gkeProxyCidr) }}{{ end -}}
{{- if not $hasGfePrimary }}{{ fail (printf "native gkeIngress requires sessions-pipeline.%s to allow GKE range 35.191.0.0/16 without exclusions" $name) }}{{ end -}}
{{- if not $hasGfeSecondary }}{{ fail (printf "native gkeIngress requires sessions-pipeline.%s to allow GKE range 130.211.0.0/22 without exclusions" $name) }}{{ end -}}
{{- end }}
Expand All @@ -65,12 +71,15 @@ shape (hosts.files without components.relayfile fails further down). */ -}}
{{- $hasSameNamespace := false -}}
{{- $hasGfePrimary := false -}}
{{- $hasGfeSecondary := false -}}
{{- $hasProxyOnly := false -}}
{{- range $peer := dig "networkPolicy" "ingressFrom" (list) $backendValues }}
{{- if and $gkeInternal (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) $gkeProxyCidr) (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasProxyOnly = true }}{{ end -}}
{{- if and (eq (len $peer) 1) (hasKey $peer "podSelector") (empty (get $peer "podSelector")) }}{{ $hasSameNamespace = true }}{{ end -}}
{{- if and (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) "35.191.0.0/16") (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasGfePrimary = true }}{{ end -}}
{{- if and (eq (len $peer) 1) (eq (dig "ipBlock" "cidr" "" $peer) "130.211.0.0/22") (empty (dig "ipBlock" "except" (list) $peer)) }}{{ $hasGfeSecondary = true }}{{ end -}}
{{- end }}
{{- if not $hasSameNamespace }}{{ fail (printf "gkeIngress requires %s.networkPolicy.ingressFrom to retain same-namespace pod access" $name) }}{{ end -}}
{{- if and $gkeInternal (not $hasProxyOnly) }}{{ fail (printf "gkeIngress.mode=internal requires %s.networkPolicy.ingressFrom to allow the proxy-only subnet %s (gkeIngress.internal.proxyOnlySubnetCidr) without exclusions" $name $gkeProxyCidr) }}{{ end -}}
{{- if not $hasGfePrimary }}{{ fail (printf "gkeIngress requires %s.networkPolicy.ingressFrom to allow GKE health checker/proxy range 35.191.0.0/16 without exclusions" $name) }}{{ end -}}
{{- if not $hasGfeSecondary }}{{ fail (printf "gkeIngress requires %s.networkPolicy.ingressFrom to allow GKE health checker/proxy range 130.211.0.0/22 without exclusions" $name) }}{{ end -}}
{{- end }}
Expand All @@ -79,8 +88,36 @@ shape (hosts.files without components.relayfile fails further down). */ -}}
{{- $nativeIngestEdge := printf "%s-history-ingest" ($gkeName | trunc 48 | trimSuffix "-") -}}
{{- $staticIpName := required "gkeIngress.staticIpName is required when gkeIngress.enabled=true" (dig "staticIpName" "" $gke) -}}
{{- $managedCertificateName := dig "managedCertificateName" "" $gke -}}
{{- if not (or $tlsSecretName $managedCertificateName) }}{{ fail "gkeIngress.managedCertificateName is required when gkeIngress.enabled=true (or set gkeIngress.tlsSecretName to serve a certificate from a Secret)" }}{{ end -}}
{{- $frontendConfigName := required "gkeIngress.frontendConfigName is required when gkeIngress.enabled=true" (dig "frontendConfigName" "" $gke) -}}
{{- /* mode external (default): a global external Application Load Balancer
(Ingress class gce), a global static address, a ManagedCertificate or TLS
Secret, and a FrontendConfig. mode internal: a regional internal Application
Load Balancer (class gce-internal) that answers only inside the VPC and the
networks routed to it (VPN, Interconnect). GKE Ingress supports neither
ManagedCertificate nor FrontendConfig nor Cloud Armor on an internal load
balancer, so the certificate is a TLS Secret or pre-shared regional
certificates, port 80 is closed instead of redirected, and sslPolicy and
securityPolicy are refused. Its proxies connect to the pods from the VPC's
proxy-only subnet, which every backend NetworkPolicy must admit. */ -}}
{{- $mode := dig "mode" "external" $gke | toString -}}
{{- if not (has $mode (list "external" "internal")) }}{{ fail (printf "gkeIngress.mode must be external or internal, got %q" $mode) }}{{ end -}}
{{- $isInternal := eq $mode "internal" -}}
{{- $internal := dig "internal" (dict) $gke -}}
{{- if not (kindIs "map" $internal) }}{{ fail "gkeIngress.internal must be a map" }}{{ end -}}
{{- $preSharedCerts := dig "preSharedCerts" (list) $internal -}}
{{- if not (kindIs "slice" $preSharedCerts) }}{{ fail "gkeIngress.internal.preSharedCerts must be a list of regional SSL certificate names" }}{{ end -}}
{{- $proxyCidr := dig "proxyOnlySubnetCidr" "" $internal | toString -}}
{{- if $isInternal }}
{{- if and (empty $preSharedCerts) (not $tlsSecretName) }}{{ fail "gkeIngress.mode=internal requires a certificate: gkeIngress.tlsSecretName (a kubernetes.io/tls Secret) or gkeIngress.internal.preSharedCerts (regional SSL certificate names); ManagedCertificate is not supported on an internal load balancer" }}{{ end -}}
{{- if and $preSharedCerts $tlsSecretName }}{{ fail "set only one of gkeIngress.tlsSecretName and gkeIngress.internal.preSharedCerts" }}{{ end -}}
{{- range $cert := $preSharedCerts }}
{{- if not (and (kindIs "string" $cert) (regexMatch $computeName $cert)) }}{{ fail (printf "gkeIngress.internal.preSharedCerts: %v is not a Compute Engine SSL certificate name" $cert) }}{{ end -}}
{{- end }}
{{- if $securityPolicy }}{{ fail "gkeIngress.securityPolicy cannot be used with gkeIngress.mode=internal: GKE Ingress does not attach Cloud Armor to an internal load balancer. Restrict who can reach it with VPC firewall rules and the routes of your VPN or Interconnect" }}{{ end -}}
{{- if $sslPolicy }}{{ fail "gkeIngress.sslPolicy cannot be used with gkeIngress.mode=internal: it is applied through a FrontendConfig, which GKE supports on external Ingresses only" }}{{ end -}}
{{- else if or $preSharedCerts $proxyCidr }}{{ fail "gkeIngress.internal.* is set but gkeIngress.mode is external; set gkeIngress.mode=internal or clear gkeIngress.internal" }}{{ end -}}
{{- if not (or $isInternal $tlsSecretName $managedCertificateName) }}{{ fail "gkeIngress.managedCertificateName is required when gkeIngress.enabled=true (or set gkeIngress.tlsSecretName to serve a certificate from a Secret)" }}{{ end -}}
{{- $frontendConfigName := dig "frontendConfigName" "" $gke -}}
{{- if not (or $isInternal $frontendConfigName) }}{{ fail "gkeIngress.frontendConfigName is required when gkeIngress.enabled=true" }}{{ end -}}
{{- $hostRoot := dig "hosts" "root" "" $gke -}}
{{- $hostApi := dig "hosts" "api" "" $gke -}}
{{- $hostCast := dig "hosts" "cast" "" $gke -}}
Expand Down Expand Up @@ -317,7 +354,7 @@ spec:
- { name: http, port: 8080, targetPort: http, protocol: TCP }
---
{{- end }}
{{- if not $tlsSecretName }}
{{- if not (or $tlsSecretName $isInternal) }}
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
Expand All @@ -339,6 +376,7 @@ spec:
{{- end }}
---
{{- end }}
{{- if not $isInternal }}
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
Expand All @@ -355,6 +393,7 @@ spec:
enabled: true
responseCodeName: MOVED_PERMANENTLY_DEFAULT
---
{{- end }}
# Deliberately has no selector. Sensitive Relaycast paths, and /internal on
# the api host, are routed here so the edge returns an error instead of
# forwarding them to the public backend.
Expand Down Expand Up @@ -517,12 +556,23 @@ metadata:
app.kubernetes.io/name: agent-relay
app.kubernetes.io/instance: {{ .Release.Name }}
annotations:
{{- if $isInternal }}
# Regional internal Application Load Balancer: reachable only from the
# VPC and the networks routed to it. HTTPS only; port 80 is not opened.
kubernetes.io/ingress.class: gce-internal
kubernetes.io/ingress.regional-static-ip-name: {{ $staticIpName | quote }}
kubernetes.io/ingress.allow-http: "false"
{{- if $preSharedCerts }}
ingress.gcp.kubernetes.io/pre-shared-cert: {{ join "," $preSharedCerts | quote }}
{{- end }}
{{- else }}
kubernetes.io/ingress.class: gce
kubernetes.io/ingress.global-static-ip-name: {{ $staticIpName | quote }}
{{- if not $tlsSecretName }}
networking.gke.io/managed-certificates: {{ $managedCertificateName | quote }}
{{- end }}
networking.gke.io/v1beta1.FrontendConfig: {{ $frontendConfigName | quote }}
{{- end }}
spec:
{{- if $tlsSecretName }}
tls:
Expand Down
78 changes: 78 additions & 0 deletions charts/agent-relay/values-gcp-internal.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Internal-only GKE edge: apply after values-gcp.yaml (and your site values),
# on EVERY helm upgrade of the release, or merge it into your site values.
# Helm keeps nothing from the previous command: an upgrade without these
# settings switches the install back to a public load balancer.
#
# helm upgrade --install ... -f values-gcp.yaml -f site.yaml -f values-gcp-internal.yaml
#
# The Ingress becomes a regional internal Application Load Balancer (class
# gce-internal). It has a private address and answers only inside the VPC and
# the networks routed to it (VPN, Interconnect); nothing is reachable from
# the internet. README: Internal-only GKE edge.
#
# Before the install, in the cluster's VPC and region (the Terraform module
# terraform/modules/gcp/internal-edge creates all three):
# - a proxy-only subnet (purpose REGIONAL_MANAGED_PROXY);
# - a reserved regional internal address, named in gkeIngress.staticIpName;
# - a firewall rule that lets the proxy-only subnet reach the nodes.
gkeIngress:
mode: internal
# GKE cannot apply an SSL policy or Cloud Armor to an internal load
# balancer, and the chart refuses them in this mode. Cleared here so a value
# from an earlier file (for example managed-values.json) does not stop the
# install.
sslPolicy: ""
securityPolicy: ""
# REQUIRED OVERRIDE: the name of the reserved regional INTERNAL address (not
# the global address of an external install).
staticIpName: example-agent-relay-internal
# The certificate. A Google-managed certificate cannot be used on an
# internal load balancer. Either a kubernetes.io/tls Secret:
# REQUIRED OVERRIDE (or use internal.preSharedCerts below instead).
tlsSecretName: example-agent-relay-tls
internal:
# REQUIRED OVERRIDE: the proxy-only subnet's range. The load balancer's
# proxies connect to the pods from it, so it is also in every list below.
proxyOnlySubnetCidr: 10.129.0.0/23
# ...or regional self-managed SSL certificates
# (gcloud compute ssl-certificates create NAME --region REGION ...);
# then leave tlsSecretName empty.
preSharedCerts: []

# Helm replaces lists, so each backend's allowed sources are restated here
# with the proxy-only subnet added. Keep the two Google health-check ranges.
# REQUIRED OVERRIDE: replace 10.129.0.0/23 in all five lists with your range.
relaycast:
networkPolicy:
ingressFrom:
- podSelector: {}
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
- ipBlock: { cidr: 10.129.0.0/23 }
relay-control:
networkPolicy:
ingressFrom:
- podSelector: {}
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
- ipBlock: { cidr: 10.129.0.0/23 }
relay-webapp:
networkPolicy:
ingressFrom:
- podSelector: {}
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
- ipBlock: { cidr: 10.129.0.0/23 }
sessions-pipeline:
sessionReadApi:
networkPolicy:
publicFrom:
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
- ipBlock: { cidr: 10.129.0.0/23 }
ingestEdge:
networkPolicy:
ingressFrom:
- ipBlock: { cidr: 35.191.0.0/16 }
- ipBlock: { cidr: 130.211.0.0/22 }
- ipBlock: { cidr: 10.129.0.0/23 }
16 changes: 16 additions & 0 deletions charts/agent-relay/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,22 @@ gkeIngress:
# front of the load balancer, where Google cannot issue a managed
# certificate. The certificate must cover every host below.
tlsSecretName: ""
# external (default): a global external Application Load Balancer.
# internal: a regional internal one (Ingress class gce-internal) that
# answers only inside the VPC and the networks routed to it. Start from
# values-gcp-internal.yaml; README: Internal-only GKE edge. In internal
# mode staticIpName names a regional internal address, the certificate is
# tlsSecretName or internal.preSharedCerts, managedCertificateName and
# frontendConfigName are unused, and sslPolicy and securityPolicy are
# refused (GKE supports neither on an internal Ingress).
mode: external
internal:
# Range of the VPC's proxy-only subnet (purpose REGIONAL_MANAGED_PROXY) in
# the cluster's region. Required in internal mode, and every backend's
# networkPolicy must allow it.
proxyOnlySubnetCidr: ""
# Regional self-managed SSL certificate names, instead of tlsSecretName.
preSharedCerts: []
hosts:
root: ""
api: ""
Expand Down
Loading
Loading