feat(destroy-addons): graceful teardown for both providers + Crossplane deletion-ordering (consolidates #641, fixes #593/#34/#35/#36)

[allamand]

What

Graceful, ordered teardown for the redeploy issues (#593, #34/#35/#36), now consolidating #641 so #641 can be closed. Covers both cluster providers plus the Crossplane composition.

kind-kro-ack hub:destroy-addons (original #830, now upgraded)

Ordered reverse-of-provision teardown so on-hub ACK/KRO/Crossplane controllers delete their managed AWS resources while still alive:

1. app workloads (non-default/platform projects) → 2. spoke clusters → 3. hub addons (reverse sync-wave, skip infra) → 4. infra stack LAST (crossplane/kro/ack-*) → AWS sweep net (prefix-scoped IAM roles/policies, ALBs scoped by elbv2.k8s.aws/cluster tag, delivery sources).
   Now adopts fix(destroy-addons): dynamic addon discovery and composition deletion ordering #641's technique: dynamic AppSet discovery (kubectl, not static registry files) + ownerReference-based child-app detection + skip finalizer/deletion waits when no children (no more per-addon 30s/120s timeouts).

kind-crossplane hub:destroy-addons (ported from #641)

Same rewrite: dynamic discovery + sync-wave sort, ownerReference child detection, agent-platform chain phase, 2>&1 for error visibility, no ignore_error.

Crossplane composition — deletion ordering (from #641)

Adds 6 Usage resources (natgw-uses-eip, route-uses-igw, route-uses-natgw, rta-uses-rt, cluster-uses-subnets, cluster-uses-vpc) to abstractions/crossplane/platform-cluster so composed AWS resources delete in dependency order (avoids orphaned EIP/NAT/VPC). + DELETION-ORDERING.md.

Relationship to #641

Supersedes #641 (which targeted feature/agent-platform with a divergent resource-groups/ layout). Its changes are re-applied here adapted to the rc3 structure (abstractions/crossplane/platform-cluster). #641's core.yaml external-dns probes are already present on rc3; its abstractions.yaml exclude is N/A to the rc3 layout — both intentionally omitted.

Validation

• Both Taskfiles parse as valid YAML; kro-ack teardown verified read-only against the kro-c1 hub earlier.
• Composition: 6 Usage blocks parse cleanly and are placed before public-rt (network) and addon-vpc-cni (cluster).
• ⚠️ Not E2E-tested — destructive teardown; needs a full provision→destroy cycle on a reused account to confirm Grafana changes for using same realm and new users config #34/Enabled Java and nginx monitoring dashboards #35/check if spot role exists before creation #36.

Targets release/v0.3.0-rc3. Fixes #593, #34, #35, #36.

[allamand]

Thanks for the thorough review — addressed the blocker in a9527e6.

🔴 #1 IAM sweep blast radius (fixed): the destroy-time sweep no longer matches roles/policies by name alone. A role/policy is now swept only if it BOTH starts with the prefix AND carries an ACK ownership tag (services.k8s.aws/*), and never if its name matches an infra/bootstrap/CFN denylist:
-team-stack-, SharedRole, -cluster-mgmt-, -cluster-role$, -cluster-node-role$, -crossplane-provider, Crossplane.*ProviderRole, -argo-rollouts.
So the IDE execution role, cluster/provider roles, and the #831 argo-rollouts roles are protected; only ACK-created orphans (the #34 EntityAlreadyExists culprits) are removed.

🟢 cosmetics (fixed): renumbered the duplicate step 8 (fleet-secrets → 9, leftover → 10); added a keep-in-sync note to INFRA_STACK.

Deferred (follow-ups, not blocking) — agree they're worth doing:

• Bump golang.org/x/crypto from 0.13.0 to 0.17.0 in /applications/golang #2 strip-finalizers fallback can orphan DynamoDB/ECR/EFS; extend the sweep (with the same ACK-tag scoping) or lengthen the graceful wait.
• Feature/Platform Engineering Backstage and CrossPlane templates. #3 align agent-platform handling between providers (crossplane preserves it; kro-ack only SKIPs it).
• Fix for Postgres credentials for CrossPlane MR to work fine. #4 centralize the INFRA_STACK / step-8 list / SKIP list (commented for now).
• .NET sample application source code #5 pagination — note AWS CLI v2 auto-paginates list-roles/list-policies by default (no --max-items), so all pages are already scanned; will confirm.
• final "remaining AWS resources" report instead of silent best-effort.

Still not E2E-tested (destructive teardown) — a full provision→destroy on a reused account is the real validation for #34/#35/#36.

[allamand]

Code review — graceful teardown (kind-kro-ack / kind-crossplane) + Crossplane Usage deletion-ordering

Read all 4 files and validated 3 points against Team 10's live hub (Crossplane v2.2.1).

Verdict

Solid, well-documented, structured and defensive (best-effort / idempotent). The teardown logic and the Usage approach are correct. One point to fix before merge (IAM sweep blast radius — now fixed, see below), a few medium robustness items, and minor nits.

✅ Strengths (verified)

• Teardown order is correct: stop regeneration (bootstrap / cluster-addons appsets) → workloads → spokes → hub addons → infra stack LAST → AWS sweep. Consistent with keeping the kro/ACK/Crossplane controllers alive to delete their AWS resources.
• Usage resources are the right fix for the EIP/NATGW and VPC/Subnet delete races (Crossplane has no native delete ordering). matchControllerRef: true correctly scopes to the PlatformCluster instance. DELETION-ORDERING.md is excellent.
• Version compat validated: hub on Crossplane v2.2.1, CRD usages.protection.crossplane.io present → the protection.crossplane.io/v1beta1 Usages will create. Note they are also created at provision time, so this was a create-path risk too — cleared here.
• apiVersions consistent (classic trap, checked): the Usage of/by match the real MRs — EC2 v1beta1, EKS Cluster v1beta2. So the ordering actually engages (a mismatch would silently disable the protection).
• Live addon discovery (kubectl, not the static registry) + ownerReference-based child detection → robust.

🔴 Blocker — FIXED (a9527e6)

The prefix-based IAM sweep was too broad. Deleting every ${PREFIX}-* role/policy by name matched, on the live hub: the IDE execution role *-team-stack-*PEEKSSharedRole* (CFN-managed — deleting it breaks the CFN stack delete), the infra/bootstrap roles (*-cluster-mgmt-*, *-cluster-role, *-cluster-node-role, *-crossplane-provider*, *Crossplane*ProviderRole), and the *-argo-rollouts roles from #831.
Fix: a role/policy is now swept only if it both starts with the prefix and carries an ACK ownership tag (services.k8s.aws/*), and never if its name matches an explicit infra/bootstrap/CFN denylist.

🟠 Medium (recommended follow-ups, not blocking)

2. Strip-finalizers fallback can orphan AWS resources. On timeout the workloads/leftover steps do finalizers:[] + delete --wait=false; if ArgoCD hasn't cascaded, the kro/ACK CRs (AppmodService, CICDPipeline, DynamoDB, ECR…) aren't deleted → ACK won't clean the AWS side. The final sweep only covers ALB + IAM + delivery-sources (not DynamoDB/ECR/EFS). → lengthen the graceful wait before stripping, or extend the sweep.
3. agent-platform handled differently between providers. kind-crossplane preserves it (preserveResourcesOnDeletion:true); kind-kro-ack only puts it in SKIP (never deleted). Intentional? If it owns AWS resources, the kro-ack path orphans them. Clarify/align.
4. INFRA_STACK diverges from the hardcoded ordered list (maintainability): the SKIP list (step 7), the explicit reverse-order list (step 8), and INFRA_STACK are three places to keep in sync. Added a note; consider centralizing.
5. Pagination: confirm aws iam list-roles/list-policies cover all pages (AWS CLI v2 auto-paginates by default when --max-items is unset).

🟢 Nits

• Duplicate # 8. in kind-kro-ack — fixed (fleet-secrets → 9, leftover → 10).
• Spoke wait loop grep -E 'clusters-kro-|clusters-.*spoke' assumes "spoke" in the app name — fine for peeks-spoke-*, fragile in general.
• ignore_error: true everywhere → destroy always succeeds even on a partial teardown (residual AWS cost hidden). Consider a final "remaining resources" report (elbv2/iam/eks/amp list).
• The ALB sweep doesn't clean orphaned target groups / SGs (minor).

TL;DR

Mergeable now that the IAM sweep is scoped (the only real risk). Points 2–5 are recommended robustness improvements; the rest is cosmetic. The Usage approach and teardown order are correct and compatible with the deployed Crossplane v2.2.1.

[allamand]

Applied the reviewer's consistency recommendation (a) in the latest commit: the kind-crossplane destroy sweep now carries the same DENY_RE denylist as the kind-kro-ack sweep, so it skips the infra/bootstrap/CFN/provider/cluster roles and the #831 -argo-rollouts roles (they share the cluster-name prefix but must survive teardown / a reused-account redeploy). Kept the broad "catch crossplane orphans by prefix" intent — just added the guard. Mediums #2–#5 remain open follow-ups.

[allamand]

Follow-ups from the review addressed:

• Feature/Platform Engineering Backstage and CrossPlane templates. #3 agent-platform (fixed): mirrored the kind-crossplane agent-platform phase into kind-kro-ack — the AppSet is stopped (preserveResourcesOnDeletion) before its child apps are orphaned. Without it, agent-platform sat only in the step-7 SKIP, so its AppSet would survive and regenerate the workloads deleted in step 5. No-op on rc3 (agent-platform not present) but keeps the providers symmetric and correct if it is added.
• Bump golang.org/x/crypto from 0.13.0 to 0.17.0 in /applications/golang #2 orphaned AWS CRs (partial, safe): added a prefix-scoped + ACK-tag-guarded ECR sweep (repos are named ${PREFIX}/<app>[/cache]). DynamoDB tables are intentionally NOT swept — their names are arbitrary (tableName) with no deployment-scoping prefix/tag, so a broad tag-only delete would repeat the Bump golang.org/x/net from 0.15.0 to 0.17.0 in /applications/golang #1 blast radius; they rely on graceful CR deletion. EFS isn't created by the app RGDs.
• Fix for Postgres credentials for CrossPlane MR to work fine. #4 INFRA_STACK sync: step-7 SKIP already derives from {{.INFRA_STACK}}; only the step-8 explicit reverse-order list must stay separate (ordering can't be derived from the set). Left the keep-in-sync comment rather than a forced refactor.
• .NET sample application source code #5 pagination: non-issue — AWS CLI v2 auto-paginates iam list-roles/list-policies by default (no --max-items), so --query filters the complete set. No change.

Remaining known trade-off: the strip-finalizers fallback can still orphan a stuck DynamoDB table on a hard timeout (by design, to not hang teardown) — surfaced, not swept.