bug: S3Bucket RGD creates conflicting IAMRoleSelectors when multiple instances exist in same namespace

[allamand]

Summary

When multiple S3Bucket Kro instances are created in the same namespace, each instance creates its own IAMRoleSelector scoped to the entire namespace and s3.services.k8s.aws group. This causes a conflict: two selectors with identical scope compete for the same ACK resources, causing the ACK S3 controller to enter an infinite reconcile loop (reading the resource every ~10 seconds) without ever creating the S3 bucket.

Environment

• EKS Capabilities: ACK v46.137.1-eks-1, Kro v0.9.2-eks-3
• Affected RGD: s3bucket.kro.run
• File: gitops/addons/charts/kro/resource-groups/manifests/rg-s3-bucket.yaml

Symptoms

• Second (and any subsequent) S3Bucket instance stays IN_PROGRESS forever
• The underlying ACK Bucket resource has no status conditions at all (empty status: {})
• No CreateBucket call is ever made to the S3 API (confirmed via CloudTrail — zero events)
• The ACK controller polls the Bucket resource every ~10s indefinitely but never reconciles it
• Kro reports: waiting for node "s3bucket": failed to evaluate readyWhen expression

Root Cause

The IAMRoleSelector template in the RGD uses a namespace-wide scope:

spec:
  namespaceSelector:
    names:
      - ${schema.metadata.namespace}   # e.g. "default"
  resourceTypeSelector:
    - group: s3.services.k8s.aws
      kind: ""
      version: ""

This means every instance creates a selector that claims ALL s3.services.k8s.aws resources in the namespace. When a second instance is created:

• Instance 1 → creates IAMRoleSelector covering all S3 resources in default
• Instance 2 → creates a second IAMRoleSelector with the same scope

ACK resolves the first selector for all matching resources (both Bucket objects), and the second selector creates an ambiguous conflict. The second Bucket resource is then permanently stuck — the controller reads it every 10s but never writes a status or calls the AWS S3 API.

Evidence (live cluster)

Two conflicting selectors observed on a real cluster deployment:

Selector	Created	Status
peeks-s3-kro-754909878517-84607729b8-s3	first	✅ ACK reconciled, bucket exists
peeks-s3-kro-754909878517-ae34de8103-s3	6h later	❌ ACK stuck in read loop

• 16,642 GET calls from the ACK capability role on the broken Bucket over 45 hours
• Zero PATCH/UPDATE/status writes on the broken Bucket in the same period (audit log confirmed)
• No CreateBucket CloudTrail event for the broken bucket

Proposed Fix

Option A — Scope the IAMRoleSelector to the specific resource instance (preferred)

Add kind, version, and a labelSelector matching the Kro-owned label so each instance's selector only covers its own Bucket:

  - id: iamroleselector
    template:
      apiVersion: services.k8s.aws/v1alpha1
      kind: IAMRoleSelector
      metadata:
        name: ${schema.metadata.name}-s3
      spec:
        arn: arn:aws:iam::${schema.spec.accountId}:role/peeks-cluster-mgmt-s3
        namespaceSelector:
          names:
            - ${schema.metadata.namespace}
        resourceTypeSelector:
          - group: s3.services.k8s.aws
            kind: Bucket
            version: v1alpha1
        labelSelector:
          matchLabels:
            kro.run/instance-name: ${schema.metadata.name}

Note: Requires verifying ACK IAMRoleSelector supports labelSelector alongside resourceTypeSelector. If not, filling in kind and version at minimum reduces the blast radius.

Option B — Single shared IAMRoleSelector as a platform-level resource (cleaner architecture)

Remove the IAMRoleSelector from the S3Bucket RGD entirely. Deploy a single, shared IAMRoleSelector for s3.services.k8s.aws in each tenant namespace as a platform-managed resource (e.g., via the platform-manifests ArgoCD ApplicationSet). This mirrors the existing pattern used for s3-peeks-spoke-staging.

Benefits:

• No per-instance selectors → no conflicts regardless of how many S3Bucket instances are created
• Platform team manages IAM bindings centrally
• Backstage template no longer needs to know or pass the IAM role ARN

The IAMRoleSelector block would be removed from rg-s3-bucket.yaml entirely.

Immediate Workaround (for existing broken deployments)

Delete the duplicate IAMRoleSelector created by the later instance. Since both selectors point to the same IAM role ARN, the first one already covers all resources:

kubectl delete iamroleselector <second-instance-name>-s3

⚠️ The selector has kro.run/owned: "true" — Kro will recreate it on the next reconcile. This workaround is only effective after the RGD is patched to use Option A or B above.

Related Files

• gitops/addons/charts/kro/resource-groups/manifests/rg-s3-bucket.yaml
• platform/backstage/templates/s3-bucket-ack-kro/skeleton/manifests/kro-s3-bucket.yaml