feat(pod-identity): add Argo Rollouts IAM role for AMP query (IRSA, #42)

[allamand]

Problem

The Argo Rollouts controller needs to query Amazon Managed Prometheus (AMP) to run the metrics analysis gate in Phase 30.5. The SA annotation in gitops/addons/registry/gitops.yaml already points to arn:aws:iam::<account>:role/<clusterName>-argo-rollouts, but that role was never created and the OIDC provider was never registered.

Root cause: IRSA required, Pod Identity won't work

prometheus/common/sigv4 v0.1.0 (used by Argo Rollouts for AMP SigV4 auth) uses aws-sdk-go v1.38.35 which supports AWS_WEB_IDENTITY_TOKEN_FILE (IRSA) but not AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (Pod Identity token file).

Upstream tracking: argoproj/argo-rollouts#4536
Upstream fix PR (migrate to aws-sdk-go-v2): argoproj/argo-rollouts#5010

Complete fix (all 5 pieces)

1. IAM Role + Policy (ack-pod-identity chart)

New argo-rollouts identity with skipPIA: true and AMP query policy. skipPIA prevents creating a PodIdentityAssociation (which would trigger Pod Identity injection instead of IRSA).

2. IRSA trust policy in role.yaml (new)

When skipPIA: true AND aws.oidcIssuer is set, the role gets an IRSA trust policy (sts:AssumeRoleWithWebIdentity with OIDC condition) instead of the default Pod Identity trust (pods.eks.amazonaws.com).

3. oidcIssuer injected into chart values (core.yaml)

The ack_pod_identities ApplicationSet now injects aws.oidcIssuer from the cluster secret annotation eks_oidc_issuer (written by the EksCluster RGD at cluster-creation time).

4. OIDC provider registration at bootstrap (kind-kro-ack Taskfile)

New task spokes:register-oidc-providers — discovers spoke clusters from fleet/members, reads each cluster's OIDC issuer, and registers it as an IAM OpenID Connect provider if not already present. This is the missing piece: EKS creates the OIDC endpoint automatically but does NOT register the provider in IAM, which is required for IRSA to work.

Called from workshop/Taskfile.yaml after wait-for-spokes (clusters are ACTIVE at that point), only when PROVIDER=kind-kro-ack.

5. Mirror in crossplane-pod-identity (drift test requirement)

Same skipPIA flag and oidcIssuer support mirrored to the Crossplane chart.

Files changed

File	Change
ack-pod-identity/values.yaml	argo-rollouts identity: skipPIA: true, AMP policy
ack-pod-identity/templates/role.yaml	IRSA trust when skipPIA=true + oidcIssuer set
ack-pod-identity/templates/pod-identity-association.yaml	Skip PIA when skipPIA: true
crossplane-pod-identity/values.yaml	Mirror
crossplane-pod-identity/templates/pod-identity-association.yaml	Mirror
overlays/environments/dev/pod-identities/values.yaml	Enable on dev spokes
overlays/environments/prod/pod-identities/values.yaml	Enable on prod spokes
gitops/addons/registry/core.yaml	Inject aws.oidcIssuer from cluster secret
cluster-providers/kind-kro-ack/Taskfile.yaml	New task spokes:register-oidc-providers
workshop/Taskfile.yaml	Call OIDC registration after wait-for-spokes

Future state

Once argoproj/argo-rollouts#5010 is merged and released (aws-sdk-go-v2, Pod Identity support):

• Remove skipPIA: true → use Pod Identity normally
• Remove spokes:register-oidc-providers call from workshop Taskfile
• The OIDC provider registration becomes optional

Validation

Tested on kro-c1 (586794472760), spoke cluster peeks-spoke-dev:

1. OIDC provider registered manually (simulates the new bootstrap task) ✅
2. ACK creates IAM Role with IRSA trust policy ✅
3. No PodIdentityAssociation (skipPIA) → IRSA injection via AWS_WEB_IDENTITY_TOKEN_FILE ✅
4. Manual AnalysisRun: phase=Successful, value=[1,1,1...] ✅

Fixes Issue #42 in platform-engineering-on-eks tracker.

[allamand]

Live test results + Root cause analysis

✅ What works

• ACK creates IAM Role peeks-spoke-dev-argo-rollouts + PodIdentityAssociation ✅
• Pod restart → AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials injected ✅
• AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token present ✅

❌ What doesn't work yet (upstream bug)

prometheus/common/sigv4 v0.1.0 (used by Argo Rollouts all versions including v1.10) depends on aws-sdk-go v1.38.35 (April 2021). This version only supports AWS_CONTAINER_AUTHORIZATION_TOKEN (env var with token value) — it does not support AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (path to token file, introduced in SDK v1.44+).

EKS Pod Identity injects the file path variant, not the env var. So the SDK v1.38.35 credential chain fails with NoCredentialProviders.

NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Fix options

Option A (upstream fix) — Update prometheus/common/sigv4 to use a newer aws-sdk-go version that reads the token from the file. Needs a PR to https://github.com/prometheus/common and then updating argo-rollouts to use the newer version.

Option B (Helm workaround) — Add an initContainer or sidecar to copy the token file content into AWS_CONTAINER_AUTHORIZATION_TOKEN env var. Can be done via the Helm chart extraEnv + extraVolumes.

Option C (use IRSA annotation instead of PIA) — The SA already has the IRSA annotation (eks.amazonaws.com/role-arn). If the OIDC provider for the spoke cluster is registered, IRSA injects a WEBIDENTITY_TOKEN_FILE which aws-sdk-go v1.38+ does support. We created the OIDC provider manually (kro-c1) and it worked in the previous test run where IRSA was used.

Recommendation

This PR is still valuable — it creates the IAM Role + PIA correctly. But a companion change is needed to make the credentials actually work:

• Either implement Option B (Helm values override to expose token as env var)
• Or implement Option C (OIDC provider + IRSA trust policy, relying on the existing SA annotation)

Option C is the path of least resistance given the SA annotation already uses IRSA format.

[allamand]

✅ Live test VALIDATED on kro-c1

Summary

After resolving two additional issues, the metrics gate works end-to-end:

AnalysisRun result:

phase=Successful  value=[1,1,1,1,1,1,1,1,0,1...]
✅ AMP SigV4 via IRSA FULLY VALIDATED!

Issue 1 — Pod Identity vs IRSA (aws-sdk-go v1 incompatibility)

prometheus/common/sigv4 v0.1.0 uses aws-sdk-go v1.38.35 which does not support AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE. EKS Pod Identity injects the token as a file path; the old SDK only handles AWS_CONTAINER_AUTHORIZATION_TOKEN (token value as env var).

Fix: Remove the PodIdentityAssociation for argo-rollouts. The SA already has the eks.amazonaws.com/role-arn IRSA annotation — with the OIDC provider registered and PIA absent, EKS injects AWS_WEB_IDENTITY_TOKEN_FILE instead, which the SDK v1.38 does support.

PR action: This chart creates the IAM role via ACK and a PodIdentityAssociation. The PIA needs to be removed from the ACK template — we want IRSA injection, not Pod Identity injection. Update pod-identity-association.yaml to skip creating the PIA for argo-rollouts (or add a skipPIA flag).

Issue 2 — AMP URL must NOT include /api/v1/query (Bug #38 extension)

The ExternalSecret amp-workspace-secrets-rust-microservice-dev syncs from peeks/platform/amp and the amp-workspace value was set to the full query URL:
https://aps-workspaces.../api/v1/query

The Prometheus client appends /api/v1/query to the address, producing:
https://aps-workspaces.../api/v1/query/api/v1/query → HTTP 404

Fix: Store only the workspace base URL:
https://aps-workspaces.us-west-2.amazonaws.com/workspaces/${ws-id}

This needs to be fixed in the bootstrap process that creates peeks/platform/amp in Secrets Manager.

Next steps for this PR

1. Modify ack-pod-identity/templates/pod-identity-association.yaml to support a skipPIA flag (or remove PIA for argo-rollouts)
2. Document that the OIDC provider must be pre-registered for spoke clusters
3. Update issues tracker with Bug Fixing Guest from Teams for backstage templates #38 extension (URL format)