Recover a GatewayClass wedged in Terminating - #5292
Open
electricjesus wants to merge 1 commit into
Open
Conversation
Deleting the GatewayAPI CR marks the GatewayClass for deletion through the ownerRef we set on it. Envoy Gateway's gateway-exists finalizer then blocks that delete while any Gateway still uses the class, so it never completes. Envoy Gateway keeps managing a class in that state but skips its parametersRef, because it only resolves the ref when DeletionTimestamp is nil. Gateway Deployments fall back to the upstream Envoy image with no imagePullSecrets, and the class is still reported Accepted=True, so nothing surfaces the fault. Re-creating the CR restores the EnvoyProxy but cannot clear a DeletionTimestamp. Watch GatewayClass, clear the finalizers on a terminating one so the delete can finish, and report it on TigeraStatus rather than healing silently.
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a Calico Enterprise Gateway API edge case where a GatewayClass can become stuck in Terminating (blocked by Envoy Gateway’s finalizer), causing Envoy Gateway to skip parametersRef resolution and potentially roll gateway Deployments back to an upstream Envoy image without expected pull secrets.
Changes:
- Adds a lazy watch on
GatewayClassresources so the controller reconciles when a class becomes wedged inTerminating. - Introduces reconcile logic to “unwedge” terminating GatewayClasses by clearing finalizers and surfacing the repair via TigeraStatus.
- Adds a unit test covering the wedged-Terminating repair path and its status reporting.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/controller/gatewayapi/gatewayapi_controller.go | Adds GatewayClass watch wiring and implements the “unwedge terminating GatewayClass” repair logic surfaced through status. |
| pkg/controller/gatewayapi/gatewayapi_controller_test.go | Adds a unit test validating that a terminating GatewayClass is released and the repair is reported via SetDegraded. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+689
to
+702
| blocking := 0 | ||
| for j := range gateways { | ||
| if string(gateways[j].Spec.GatewayClassName) == name { | ||
| blocking++ | ||
| } | ||
| } | ||
|
|
||
| reqLogger.Info("GatewayClass is stuck terminating; clearing finalizers so it can be re-created", | ||
| "gatewayClass", name, "deletionTimestamp", gc.DeletionTimestamp, | ||
| "finalizers", gc.Finalizers, "blockingGateways", blocking) | ||
|
|
||
| patched := gc.DeepCopy() | ||
| patched.Finalizers = nil | ||
| if err := r.client.Patch(ctx, patched, client.MergeFrom(gc)); err != nil { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Deleting the
GatewayAPICR marks theGatewayClassfor deletion, since we set a controller ownerRef on it. Envoy Gateway'sgateway-exists-finalizerblocks that delete while any Gateway still uses the class, so it never finishes and nothing repairs it.Envoy Gateway keeps managing a terminating class but skips its
parametersRef, because it only resolves the ref whenDeletionTimestampis nil (internal/provider/kubernetes/controller.go:366, EG v1.8.3). Gateway deployments fall back to the upstreamdocker.io/envoyproxy/envoyimage with noimagePullSecrets, and the class still reportsAccepted=True. Re-creating the CR restores theEnvoyProxybut can't clear aDeletionTimestamp, so the cluster stays wrong until someone clears the finalizer by hand.The gateway pod carries on serving and tigerastatus stays green, so this shows up later as a pod that won't start after a restart or drain.
This watches
GatewayClass, clears the finalizers on a terminating one, and reports the repair on tigerastatus. The next reconcile re-creates the class and Envoy Gateway resolvesparametersRefnormally.Clearing another controller's finalizer is deliberate. The wedge only exists because our ownerRef marked a class that Envoy Gateway won't release, and there's no other way out of it. Gateways lose their managed class for a few seconds while the class is re-created, so the proxy deployment is torn down and re-provisioned.
Test plan
go build,go vet,gofmtandgo test ./pkg/controller/gatewayapi/...all pass.New unit test covers a class with
DeletionTimestampset, and asserts both the release and the tigerastatus message. It fails when the repair is short-circuited.Checked by hand on a GCP kubeadm cluster running Enterprise master. Waiting, restarting envoy-gateway, annotating the Gateway and deleting the deployment all left it wedged. Clearing the finalizer and reconciling fixed it in 6 seconds.
Not covered: no e2e, and the upstream skip in EG is unchanged since v0.5.0 and still on EG main, so this only handles our side of it.
Release Note
For PR author
make gen-filesmake gen-versions