Skip to content

feat: hub self-adoption — pivot management from ephemeral kind to hub EKS #775

Description

@allamand

Problem

Currently the bootstrap kind cluster must remain permanently alive to host KRO/ACK controllers that manage peeks-hub and spoke clusters. This causes several issues:

  • Credential expiry: The kind cluster uses EC2 instance role STS tokens (~1h) via a aws-credentials secret. After expiry, ACK controllers lose access → PIAs stuck, EksCluster stuck IN_PROGRESS, manual task credentials:refresh required.
  • Persistent EC2 dependency: The kind cluster runs inside the IDE EC2. If the IDE is restarted or SSM session expires, the kind cluster is lost and the entire management plane is gone.
  • Non-idiomatic: Using a local kind cluster as the long-running control plane for production EKS clusters is fragile.

Proposed Solution: Hub Self-Adoption (CAPI Pivot Pattern)

Inspired by Cluster API pivot, the idea is to move the management of peeks-hub from the ephemeral kind cluster to the hub EKS cluster itself.

Phases

Phase 1 (today): Kind bootstrap → creates peeks-hub

kind (ephemeral) ──KRO/ACK──→ creates peeks-hub EKS
                                         │
                                         ▼
                                  ArgoCD/KRO/ACK (EKS Capabilities)

Phase 2 (proposed): hub:pivot → peeks-hub self-manages

peeks-hub ──KRO/ACK (in-cluster)──→ adopts its own EksCluster resource
peeks-hub ──KRO/ACK──→ manages spoke clusters
kind → destroyed (no longer needed)

Implementation Steps

  1. Install KRO + ACK on peeks-hub (as EKS Capabilities — already done today)
  2. Export EksCluster manifest from kind — serialize the current EksCluster/peeks-hub resource
  3. Apply on peeks-hub context — with adopt: true or by setting the correct owner references so ACK adopts the existing AWS resources instead of recreating them
  4. Delete from kind — remove the EksCluster resource from kind (ACK will not delete the AWS resources if adopted properly)
  5. Pivot spoke management — move spoke EksClusters to peeks-hub context
  6. Destroy kindkind delete cluster

Benefits

  • ✅ No credential expiry (EKS PodIdentity, never temporary tokens)
  • ✅ No kind cluster to maintain after bootstrap
  • task destroy becomes cleaner (no kind dependency)
  • ✅ Better resilience — hub EKS is the durable control plane
  • ✅ Consistent with fleet model (hub manages spokes)

Challenges

  • ACK adopt-or-create semantics for existing resources
  • Circular dependency during bootstrap (need kind to create hub, then pivot)
  • Crossplane on hub must be re-pointed to hub context for spoke management
  • EksCluster RGD may need adoptionPolicy support

References

Acceptance Criteria

  • task hub:pivot moves management from kind to peeks-hub
  • kind delete cluster succeeds after pivot without breaking peeks-hub or spokes
  • No credentials:refresh needed after pivot
  • task install re-run after pivot still works (idempotent)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions