makeitworkcloud · xnoto · Aug 28, 2026 · Aug 28, 2026
@@ -2,6 +2,14 @@
 
 GitOps manifests for the k3s cluster behind [makeitwork.cloud](https://makeitwork.cloud/). ArgoCD reconciles this repo using KSOPS for inline secret decryption.
 
+## Contributor and operator guides
+
+- [Add a workload](docs/adding-a-workload.md)
+- [Roll out or roll back a chart workload](docs/rollout-and-rollback.md)
+- [Configure kubectl access](docs/kubeconfig.example.yaml)
+- [Author and publish a chart](https://github.com/makeitworkcloud/charts/blob/main/docs/adding-a-chart.md)
+- [Configure post-publish GitOps PR automation](https://github.com/makeitworkcloud/charts/blob/main/docs/gitops-update-automation.md)
+
 ## Layout
 
 ```
@@ -174,12 +182,14 @@ The age public key is committed in `.sops.yaml`. The matching private key is loa
 
 ## CI/CD
 
-The repository uses `.github/workflows/test.yml` and `.github/workflows/sync.yml`:
-
-1. **test** (`ubuntu-latest`) — runs pre-commit (yamllint, kube-linter, conventional-commit, etc.)
-2. **sync** (`arc-tf` runner, `main` only) — after tests pass, patches each App-of-Apps root (`bootstrap-secrets`, `gitops-operators`, `gitops-workloads`) to initiate an ArgoCD sync at the tested SHA
-
-The in-cluster ARC runner uses its ServiceAccount token to talk to the API directly. The sync workflow initiates reconciliation but does not wait for it to finish. Afterward, confirm each affected Application reports the target revision, `Synced`, and `Healthy`. Use `workflow_dispatch` on `sync.yml` to retry the selected ref when necessary.
+Pull requests and `main` run `.github/workflows/test.yml`, which executes the
+hooks in `.pre-commit-config.yaml`: YAML and repository hygiene checks, secret
+scanning, and kube-linter. After `main` passes, `.github/workflows/sync.yml`
+uses the in-cluster runner to request a sync of `bootstrap-secrets`,
+`gitops-operators`, and `gitops-workloads` at the tested SHA. It initiates
+reconciliation but does not wait for final health. Follow
+[Rollout and rollback](docs/rollout-and-rollback.md) for the publication, review,
+verification, failure, and rollback procedure.
 
 ## Resource Sizing
 

@@ -0,0 +1,183 @@
+# Adding a workload
+
+Use this procedure to add a chart-backed workload without bypassing the
+App-of-Apps boundary. The chart may be new and unpublished while the cluster
+change is prepared, but the Application must pin a published, immutable chart
+version before it can merge.
+
+## 1. Define ownership
+
+Decide which repository owns each resource before writing manifests:
+
+- The chart owns the reusable workload release: Deployments or StatefulSets,
+  non-secret configuration, probes, and chart values.
+- This repository owns cluster-specific integration under `workloads/<name>`:
+  encrypted Secrets, namespace metadata, storage, stable Services, and
+  operator-specific resources such as `TunnelBinding`.
+- Give every Kubernetes object one owner. A second Argo CD source is rendered
+  alongside the chart; it does not patch chart output automatically.
+
+Keep chart authoring and publication details in the charts repository. Follow
+its [adding a chart
+guide](https://github.com/makeitworkcloud/charts/blob/main/docs/adding-a-chart.md)
+to publish an immutable OCI version. GitOps updater automation is a separate,
+explicit opt-in; follow the charts repository's [automation
+guide](https://github.com/makeitworkcloud/charts/blob/main/docs/gitops-update-automation.md)
+when the workload needs it. Do not reproduce updater credential or GitHub App
+rotation procedures here.
+
+## 2. Create the cluster overlay
+
+Create `workloads/<name>/kustomization.yaml` only when the chart needs
+cluster-owned resources. Register each resource in that Kustomization. Typical
+files include:
+
+```text
+workloads/<name>/
+  kustomization.yaml
+  namespace.yaml
+  persistent-volume-claim.yaml
+  service.yaml
+  tunnel-binding.yaml
+  ksops-<name>-secrets.yaml
+  <name>-secret.yaml
+```
+
+Include only the files the workload needs:
+
+- Use an explicit `Namespace` when labels, annotations, or other namespace
+  metadata must be Git-owned. Otherwise `CreateNamespace=true` can create the
+  destination namespace.
+- Keep a Service here only when the chart does not own the required stable
+  Service. Its selector and ports must match the chart workload.
+- Keep a PVC here when storage is cluster-owned. Configure the chart to use the
+  existing claim, and review storage-class, reclaim, upgrade, rollback, and
+  prune behavior before relying on it for persistent data.
+- Add a `TunnelBinding` only for externally reachable workloads. It must be in
+  the Service namespace, and `subjects[].name` must exactly match that Service.
+  The binding owns workload tunnel DNS; do not duplicate its CNAME in
+  `tfroot-cloudflare`.
+
+For secrets, create SOPS-encrypted Secret manifests using the approved
+secret-editing process and list them in a KSOPS generator:
+
+```yaml
+generators:
+  - ksops-<name>-secrets.yaml
+```
+
+Follow `.sops.yaml` selective-encryption rules. Never commit plaintext, print
+decrypted values into review output, or decrypt secrets merely to run routine
+validation.
+
+## 3. Create the child Application
+
+Create `workloads/apps/<name>-app.yaml`. Use placeholders while preparing a
+chart that has not been published, then replace `<published-chart-version>`
+with the immutable OCI chart version before merge.
+
+```yaml
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: <name>
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  project: default
+  sources:
+    - chart: <chart-name>
+      repoURL: <oci-registry>/<organization>/charts
+      targetRevision: <published-chart-version>
+    - repoURL: https://github.com/<organization>/<cluster-repository>.git
+      path: workloads/<name>
+      targetRevision: main
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: <namespace>
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - SkipDryRunOnMissingResource=true
+    retry:
+      limit: 5
+      backoff:
+        duration: 30s
+        maxDuration: 5m
+        factor: 2
+```
+
+Remove the second source when no cluster overlay is needed. Keep `project`,
+destination server, sync policy, and retry behavior aligned with neighboring
+Applications unless an approved workload requirement says otherwise. Set the
+destination namespace explicitly for namespaced charts. Use
+`SkipDryRunOnMissingResource=true` only when custom resources require it; it is
+not a substitute for making required CRDs available.
+
+The annotation orders the child Application resource within the
+`gitops-workloads` sync. It does not order resources inside the child or across
+independent root Applications. Choose a different wave only for a demonstrated
+dependency among resources managed by `gitops-workloads`.
+
+## 4. Handle CRDs and ordering
+
+Operator-owned CRDs and controllers belong under `operators/`, not in the
+workload overlay. If the workload creates a custom resource whose CRD might not
+exist when child Applications are created, add the CRD name to
+`workloads/apps/wait-for-crds.yaml`. The `PreSync` job must observe the CRD as
+`Established` before the root creates workload resources.
+
+Do not use sync waves to imply ordering across independent Argo CD
+Applications. Add a wait condition when a real cross-Application dependency
+exists. A chart that owns its own CRDs should keep their installation and
+resource ordering within that chart unless the CRDs are promoted to an
+operator-owned boundary.
+
+## 5. Register the Application
+
+Add only the child Application manifest to
+`workloads/apps/kustomization.yaml`:
+
+```yaml
+resources:
+  - <name>-app.yaml
+```
+
+Do not add `workloads/<name>` to that file. The child Application's repository
+source owns the overlay.
+
+No bootstrap change is needed for a normal workload addition because
+`bootstrap/workloads-app.yaml` already reconciles `workloads/apps`. Change
+bootstrap only when the App-of-Apps source, ownership boundary, destination, or
+bootstrap-applied resources themselves must change. A new child Application,
+namespace, chart version, or workload overlay alone is not a bootstrap change.
+
+## 6. Validate and review
+
+Run the repository checks before opening the pull request:
+
+```bash
+pre-commit run --all-files
+git diff --check
+```
+
+The `test` workflow runs the hooks configured in `.pre-commit-config.yaml`,
+including YAML syntax and repository hygiene checks, secret scanning, and
+kube-linter. It does not prove that Argo CD can pull an unpublished chart or
+that the workload will become Healthy. Confirm in review that:
+
+- Every `kustomization.yaml` reference exists and only the child Application
+  was added to `workloads/apps/kustomization.yaml`.
+- The chart version is published and pinned, not a mutable tag or range.
+- Chart and cluster resources do not claim the same Kubernetes object.
+- CRD gates, namespace, Service, PVC, `TunnelBinding`, and encrypted Secret
+  ownership are explicit.
+- The generated GitOps pull request changes desired state only. It does not
+  deploy or sync Argo CD directly.
+
+After merge, follow [Rollout and rollback](rollout-and-rollback.md).
@@ -0,0 +1,129 @@
+# Rollout and rollback
+
+This guide covers chart-backed workloads whose immutable OCI version is pinned
+in `workloads/apps/<name>-app.yaml`.
+
+## Delivery flow
+
+1. Merge the chart change only after chart CI passes. Publication creates the
+   immutable OCI artifact; publication alone does not change cluster desired
+   state.
+2. If the chart is explicitly configured for post-publish automation, the
+   charts updater opens or updates a generated GitOps pull request that pins
+   the child Application to the published `targetRevision`. Otherwise, open a
+   focused GitOps pull request manually after publication. Neither kind of pull
+   request deploys or syncs Argo CD.
+3. Review the version, diff scope, ownership boundaries, and repository checks.
+   Merge only after the target version exists in the registry and cluster CI
+   passes.
+4. The merged commit runs the `test` workflow on `main`. On success,
+   `.github/workflows/sync.yml` requests a sync of the App-of-Apps roots at the
+   tested commit SHA.
+5. `gitops-workloads` reconciles the child Application definition. The child's
+   automated policy then reconciles the pinned chart and any repository overlay.
+
+The sync workflow returns after submitting the root sync requests. It does not
+wait for root or child Applications to reach `Synced` or `Healthy`; verification
+is a separate operator step.
+
+## Verify a rollout
+
+Use approved Argo CD and Kubernetes access. Replace every placeholder rather
+than treating an existing workload's names as defaults.
+
+1. Confirm the merged child Application specifies the intended immutable chart
+   version. For a multi-source Application, inspect the chart source directly:
+
+   ```bash
+   kubectl -n argocd get application <application> \
+     -o jsonpath='{range .spec.sources[*]}{.chart}{"\t"}{.targetRevision}{"\n"}{end}'
+   ```
+
+2. Confirm both the root and child have reconciled, and inspect any reported
+   conditions or failed operation:
+
+   ```bash
+   kubectl -n argocd get application gitops-workloads <application>
+   kubectl -n argocd describe application <application>
+   kubectl -n argocd get application gitops-workloads \
+     -o jsonpath='{.status.sync.revision}{"\n"}'
+   kubectl -n argocd get application <application> \
+     -o jsonpath='{.status.sync.revision}{"\n"}{.status.sync.revisions}{"\n"}'
+   ```
+
+   The affected root and child must report the expected revision, `Synced`, and
+   `Healthy`. For multi-source children, Argo CD may report multiple source
+   revisions; verify the chart version and repository revision separately.
+
+3. Inspect the child Application's resource tree in Argo CD. Confirm all
+   expected resources are present, no unexpected resources are being pruned,
+   and no resource remains Progressing, Degraded, Missing, or OutOfSync.
+
+4. Verify the controller rollout and resulting pods:
+
+   ```bash
+   kubectl -n <namespace> rollout status deployment/<deployment> --timeout=5m
+   kubectl -n <namespace> get pods
+   ```
+
+   Use the applicable rollout command for a StatefulSet, DaemonSet, or other
+   controller instead of assuming every chart creates a Deployment.
+
+5. If reconciliation or startup is not clean, inspect events and bounded logs
+   without printing credentials:
+
+   ```bash
+   kubectl -n <namespace> get events --sort-by=.lastTimestamp
+   kubectl -n <namespace> logs deployment/<deployment> --all-containers --tail=100
+   ```
+
+6. Run a protocol-appropriate functional check against `<endpoint>` from an
+   approved network location. Validate authentication and a representative
+   request without placing credentials in shell history or logs.
+
+## Handle a failed rollout
+
+Locate the failing boundary before changing desired state:
+
+- Publication failure: confirm the immutable version exists in the registry.
+- Updater failure: inspect the charts workflow and whether the generated pull
+  request has the expected single-version change.
+- Cluster CI failure: fix the pull request; do not bypass required checks.
+- Root failure: inspect `gitops-workloads`, including the `wait-for-crds`
+  `PreSync` job and Application conditions.
+- Child failure: inspect its source revisions, operation state, resource tree,
+  Kubernetes events, rollout state, and logs.
+- Functional failure: preserve evidence and determine whether configuration,
+  compatibility, data migration, networking, or the chart caused the failure.
+
+A manual sync may retry the desired Git revision, but it does not create a new
+desired state or replace review. Live patches are diagnostic or emergency-only:
+automated self-heal can revert drift, prune can remove resources absent from
+Git, and root reconciliation can restore the child Application specification.
+Any emergency live change must be captured in a reviewed Git change or removed
+after diagnosis so Git again matches the cluster.
+
+## Roll back
+
+Rollback is a reviewed desired-state change, not a registry retag or live
+Application patch.
+
+1. Identify the previous known-good immutable chart version from Git history
+   and confirm that artifact still exists. Review chart compatibility with
+   current Secrets, configuration, CRDs, and persisted data. A chart rollback
+   cannot undo an incompatible data migration.
+2. Open a focused pull request changing the chart source's `targetRevision` in
+   `workloads/apps/<name>-app.yaml` to that previous version. Include the failed
+   version, reason for rollback, and verification plan.
+3. Run and review the normal repository checks. Merge the rollback pull request
+   through the protected branch; do not bypass review because the change is
+   urgent.
+4. After the `test` workflow succeeds, confirm the sync workflow submitted the
+   tested SHA. Then wait for `gitops-workloads` to update the child Application
+   and for the child to reconcile the previous chart.
+
+Repeat the complete rollout verification after rollback: pinned target version,
+root and child revisions, `Synced`, `Healthy`, resource tree, controller rollout,
+pods, events, bounded logs, persistent-data behavior, and the functional
+endpoint. Record any emergency action and follow-up fix in the canonical Git
+history.
@@ -2,6 +2,11 @@
 
 This directory owns workload Applications and their manifests. Use `workloads/apps` for reconciled application entry points; preserve application boundaries and Argo ordering.
 
+Use [Adding a workload](../docs/adding-a-workload.md) for the canonical
+Application and overlay procedure, and [Rollout and
+rollback](../docs/rollout-and-rollback.md) for chart delivery, verification,
+failure handling, and rollback. Keep this file focused on workload policy.
+
 External services use Cloudflare Tunnel resources rather than an in-cluster ingress controller. A workload TunnelBinding must reference an existing Service in the same namespace. Workload tunnel DNS is operator-owned; do not add its CNAME to `tfroot-cloudflare`.
 
 For OpenCode, chart configuration is owned by `charts/opencode-server`. When a new version is published, its post-publish workflow opens or updates a GitOps PR that changes only the consuming Application's pinned chart version. The PR remains subject to this repository's CI and normal merge review; it does not sync Argo CD or deploy directly.