0-init.sh timeout: la phase IAM Identity Center dépasse le WaitCondition CloudFormation (v0.2.17)

[allamand]

Summary

Workshop Studio deployments on v0.2.17 fail consistently with IdeInitScriptWaitCondition timed out (Received 0 conditions when expecting 1). The CloudFormation stack ends up in CREATE_FAILED after ~2h20mn of deployment.

Two distinct bugs are involved: one is the root cause of the timeout, the other surfaces during manual re-runs.

---

Bug #1 (root cause) — Keycloak PostSync hook never fires → SAML descriptor unavailable

What happens

configure_identity_center.py polls the Keycloak SAML metadata endpoint:

https://<domain>/keycloak/realms/platform/protocol/saml/descriptor

This endpoint only becomes available after the realm configuration Job (triggered by the PostSync hook of the keycloak-peeks-hub ArgoCD app) has completed. That Job creates the platform realm, users, OIDC clients, and the SAML client.

Root cause: ArgoCD race condition → PostSync hook never triggered

During the initial bootstrap, the ArgoCD app keycloak-peeks-hub gets stuck in a revision conflict loop:

health=Healthy  sync=Unknown  operation=Running   (repeating indefinitely)

This conflict occurs when the BootstrapStack CodeBuild pushes the Keycloak app while an ArgoCD sync is already in progress (race condition with ApplicationSets generating multiple apps simultaneously). ArgoCD never completes the sync cleanly → the PostSync hook is never triggered → the realm config Job never runs → the platform realm and SAML client do not exist → the descriptor returns 404.

Evidence (deployment timeline from 2026-08-26, CloudWatch SSM logs)

Time (UTC)	Event
23:01:24	keycloak-0 starts
23:02:26	keycloak-0 Ready
23:03:10	keycloak-1 Ready
23:05:03	0-init.sh enters IDC phase — waits for keycloak-peeks-hub Healthy
23:05:03–23:10:46	Loop: sync=Unknown operation=Running patched ×6, never resolved
23:10:46	0-init.sh declares "✓ Keycloak healthy" and launches configure_identity_center.py
23:10:50–23:42:17	Attempt 1/3: polls SAML for 30 min → TimeoutError
23:43:18–23:58:31	Attempt 2/3: same loop, killed by WaitCondition expiry (3600s)
23:58:31	CloudFormation: WaitCondition timed out. Received 0 conditions when expecting 1

The keycloak-clients secret (produced by the realm config Job) was created at 2026-08-27T09:23:39Z (during a manual re-run the next morning) — proving it did not exist during the initial bootstrap.

Retry budget structurally incompatible with the wrapper timeout

Worst case IDC = 3 attempts × 30 min SAML wait ≈ 98 min
WaitCondition / SSM timeout                     = 60 min

Even if the SAML descriptor eventually became available, the second attempt would be killed before it could succeed.

---

Bug #2 — ModuleNotFoundError: No module named 'boto3' on manual re-runs

Symptom

When re-running 0-init.sh outside the original CFN context (sudo su - ec2-user):

[INFO] Running configure_identity_center.py (attempt 1/3)...
Traceback (most recent call last):
  File ".../configure_identity_center.py", line 31, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

(× 3 attempts — IDC users are never created)

Root cause: two different python3 binaries depending on shell type

Context	python3 resolved	boto3
sudo su - ec2-user (login shell — CFN context)	~/.local/share/mise/installs/python/3.12.14/bin/python3	✅ present
AWS-RunShellScript / bash -lc	/usr/bin/python3 (system, Python 3.9.25)	❌ absent

mise activates its Python via profile hooks (~/.bashrc, ~/.profile). These hooks are loaded by sudo su - (true login shell) but not reliably by other SSM invocation forms. The resolved python3 is then the system Python 3.9 which does not have boto3.

Impact: makes manual re-runs unusable to recover a failed deployment — exactly the scenario where they are most needed.

---

Files affected

File	Lines	Bug
platform/infra/terraform/scripts/0-init.sh	~110–175	#1 and #2
platform/infra/terraform/scripts/configure_identity_center.py	~588–603	#1

---

Proposed fixes

Fix #1a — Wait for the right signal (application signal, not ArgoCD health)

Replace the health=Healthy ArgoCD app check with a wait on the keycloak-clients secret (created by the PostSync realm config Job):

# In configure_idc_and_argocd_token() in 0-init.sh
# BEFORE launching configure_identity_center.py

print_status "INFO" "Waiting for Keycloak realm config Job to complete (keycloak-clients secret)..."
kubectl wait secret/keycloak-clients -n keycloak \
  --for=jsonpath='{.metadata.name}'=keycloak-clients \
  --timeout=600s || {
    print_status "WARNING" "Keycloak config Job did not complete in time, skipping IDC configuration"
    return 0
  }

Fix #1b — Reduce the SAML timeout in configure_identity_center.py

# Line ~590 of configure_identity_center.py
# Before:
deadline = time.time() + 1800   # 30 minutes — way too long

# After:
deadline = time.time() + 300    # 5 minutes — sufficient if the realm exists

An available SAML descriptor responds in milliseconds. Beyond 5 minutes, the realm simply does not exist — retrying will not help.

Fix #1c — Align idc_max_retries with the available time budget

# In 0-init.sh
# With deadline=300s, total budget = 3 × 5 min = 15 min → fits within the WaitCondition
local idc_max_retries=3
local idc_retry_wait=30   # reduced from 60s

Fix #2 — Ensure python3 resolves to the mise python in all contexts

# Add at the top of configure_idc_and_argocd_token() in 0-init.sh,
# before any python3 call
export PATH="${HOME}/.local/share/mise/shims:${HOME}/.local/bin:${PATH}"

Optional fix — Explicit failure signal to CloudFormation WaitCondition

Currently 0-init.sh always sends cfn-signal -e 0 (success) even on IDC failure. Sending an explicit failure signal would allow CloudFormation to roll back cleanly rather than waiting for the timeout:

# In the PEEKSSetupIDE SSM document (CDK team-stack.ts)
# Replace:
/opt/aws/bin/cfn-signal -e 0 '{{WaitConditionHandle}}'
# With:
/opt/aws/bin/cfn-signal -e $exit_code '{{WaitConditionHandle}}'