Harden the appscode.com serving path against intermittent 5xx - #1328
Merged
Conversation
A customer reported recurring 503/504/520/522/500 and premature connection closures against /api/v1/dbaas/billing/reports/namespaces. Investigation of the production cluster found three defects in the serving path. platform-api probes: liveness and readiness declared only httpGet, so the Kubernetes defaults applied (timeoutSeconds 1, failureThreshold 3). Since /api/v1/version is served by the same process as every other request, it stalls whenever the process does -- observed at 39.8s while the pod was otherwise serving traffic -- and the kubelet killed the container 30s later. All replicas share the same upstreams, so they stall and get killed together. Liveness now fires only after ~2m of a genuinely wedged process; readiness keeps a tight period so it drains traffic first. Envoy data plane: replicas defaults to 1 with no disruption budget, making the gateway a single point of failure for the whole domain. Add envoyPDB and a hostname topology spread, both gated on replicas > 1 so single-replica installs are unaffected. Deployments that need redundancy set envoy.replicas. Route timeouts: 2m exceeds Cloudflare's 100s origin timeout, so a slow backend surfaced as a Cloudflare 520/524 with no matching entry in our logs. Lowered to 55s across the ace routes. Also add a BackendTrafficPolicy with retries limited to triggers where the backend provably never saw the request, keeping non-idempotent POSTs such as /api/v1/register safe, plus passive health checking to eject a failing replica faster than readiness can. Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
ArnobKumarSaha
added a commit
that referenced
this pull request
Sep 1, 2026
Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Customer saw recurring 5xx (503/504/522/520 + SSL-closed) on
appscode.com/api/v1/dbaas/billing/reports/namespaces. The handler is fine (112ms–1.84s, 0 5xx in 48h of Envoy logs); the failures come from the serving path.charts/platform-api/templates/statefulset.yaml— liveness/readiness had onlyhttpGet, so k8s defaults gave a 30s kill window./api/v1/versionis served by the same process as everything else, so it stalls when the process does:ace-platform-api-2logged 40s/api/v1/versionresponses, thenexitCode: 1— no panic, no OOM. All replicas shareace-db/ace-nats, so they stalled and got killed together. Liveness now needs ~2m (timeout 10,period 20,failureThreshold 6); readiness keeps a tight period with a 5s timeout so it sheds traffic first.startupProbegets a 5s timeout too.charts/service-gateway—envoy.replicaswas commented out, so envoy-gateway runs 1 replica with no PDB. Every appscode.com request transits that one pod; any reschedule is a domain outage (the customer's 522s stop exactly when the current envoy pod was created, Aug 6). AddedenvoyPDB.maxUnavailable: 1and a hostname topology spread (ScheduleAnyway), both gated onreplicas > 1so single-replica and DaemonSet installs render unchanged. Default stays 1 — prod should setservice-gateway.envoy.replicas: 3.charts/ace/templates/gateway/route-main.yaml—request/backendRequestwere2min 5 places, above Cloudflare's 100s origin timeout, so anything in the 100–120s band surfaced as a Cloudflare 520/524 with nothing in our logs. Lowered to55s.New
BackendTrafficPolicy— without one, a single connection failure during a pod restart was returned to Cloudflare as a 503 even with healthy replicas left. Retries are limited toconnect-failure,refused-stream,reset-before-request— all three guarantee the backend never processed the request, soPOST /api/v1/registerand/api/v1/license/issuestay safe without per-rule scoping. Passive health checking ejects a failing replica faster than readiness.Not in this PR
ab61c162); prod runsplatform-api-v2026.6.19withresources: {}, henceBestEffortpods evicted first. Chart upgrade fixes it.gotenbergsidecar still has noresources— follow-up.nats: no responders availablein the billing processor/aggregator is application-side; tracked separately.Verification
helm templateplatform-api: probes render with intended thresholds.helm templateservice-gateway:replicas=1→ no PDB/spread;replicas=3→ both; DaemonSet unchanged.kubectl apply --dry-run=serveraccepts the newBackendTrafficPolicy,HTTPRoute,EnvoyProxy/GatewayClass.helm lintpasses on both charts.