diff --git a/.terraform-docs.yml b/.terraform-docs.yml index 25bebc1..6b249bd 100644 --- a/.terraform-docs.yml +++ b/.terraform-docs.yml @@ -6,6 +6,10 @@ output: template: | {{ .Content }} + + ## Operator guides + + - [Chart updater GitHub App runbook](docs/chart-updater-github-app.md) settings: diff --git a/AGENTS.md b/AGENTS.md index 9f612a0..8798490 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,4 +4,6 @@ OpenTofu root for Make IT Work Cloud GitHub organization infrastructure. This root centrally manages organization policy, repository protections, Dependabot callers, and distributed Actions configuration. Before adding a centrally managed path, verify that no target repository owns that same path; preserve the `_dependabot-notify.yml` collision boundary in `shared-workflows`. +`README.md` is generated from `.terraform-docs.yml`; regenerate it instead of editing generated sections by hand. Use `docs/chart-updater-github-app.md` for chart updater GitHub App, installation, and key operations. + Use GitHub MCP and PR CI plans as validation authority. `main` is an environment-gated apply path; use scoped branches and PRs, never direct pushes. Do not run OpenTofu, secret-decryption, curl, import, state, or apply commands from this server. Never expose encrypted secret content, credentials, state, or sensitive plans. diff --git a/README.md b/README.md index 336396c..3792988 100644 --- a/README.md +++ b/README.md @@ -43,4 +43,8 @@ No modules. ## Outputs No outputs. + +## Operator guides + +- [Chart updater GitHub App runbook](docs/chart-updater-github-app.md) diff --git a/docs/chart-updater-github-app.md b/docs/chart-updater-github-app.md new file mode 100644 index 0000000..d0e51eb --- /dev/null +++ b/docs/chart-updater-github-app.md @@ -0,0 +1,240 @@ +# Chart Updater GitHub App Runbook + +This is the canonical operator runbook for the GitHub App used by the chart +updater. Read identifiers, repository names, and secret names from the linked +source configuration instead of copying them into other documentation. + +## Purpose and current consumers + +The [charts workflow](https://github.com/makeitworkcloud/charts/blob/main/.github/workflows/helm.yml) +publishes an immutable `opencode-server` chart release and uses a short-lived +GitHub App installation token to update the +[GitOps chart reference](https://github.com/makeitworkcloud/kustomize-cluster/blob/main/workloads/apps/opencode-app.yaml). +The workflow opens a pull request; it does not sync Argo CD or deploy the +release. + +The current source repository, destination repository, App ID, Actions secret +name, and token repository request are defined in the charts workflow and in +the [`local.secrets` mapping](../main.tf). The Actions secret is materialized by +[`github_actions_secret.secrets`](../gh-secrets.tf). + +Chart updater operation does not depend on AWS Secrets Manager. The private key +used by the workflow is stored in this repository's SOPS-backed secret file and +distributed directly to the configured GitHub Actions repository secret. An +old AWS Secrets Manager copy is legacy and unreferenced. It is not part of +normal key rotation or recovery. Deleting that legacy copy requires a separate +approved cleanup after the agreed rollback window. + +## Ownership boundaries + +| Concern | Owner and source of truth | +| --- | --- | +| GitHub App creation, permissions, keys, and organization installation | Organization owners manage these manually in GitHub. This root does not Terraform-manage the App or its installation. | +| Encrypted private key and Actions secret recipients | This root owns the SOPS-backed value and recipient list in [`main.tf`](../main.tf), using the encryption policy in [`.sops.yaml`](../.sops.yaml). | +| Actions secret distribution | [`gh-secrets.tf`](../gh-secrets.tf) writes the configured repository secret. It cannot create or configure a GitHub App. | +| Chart publication and updater token request | The [charts workflow](https://github.com/makeitworkcloud/charts/blob/main/.github/workflows/helm.yml) owns the App ID reference, requested repositories, requested token permissions, and pull-request automation. | +| Desired deployment state and rollout | `kustomize-cluster` owns the GitOps reference. Follow its [rollout and rollback guide](https://github.com/makeitworkcloud/kustomize-cluster/blob/main/docs/rollout-and-rollback.md). | + +## Keep the four scopes distinct + +These scopes can overlap, but one never implies another: + +| Scope | Where it is configured | What it controls | +| --- | --- | --- | +| Terraform-managed repositories | `local.github_repositories` in [`main.tf`](../main.tf) | Repositories, protections, and other organization configuration managed by this root. It grants no GitHub App access. | +| Repositories receiving the Actions secret | The chart updater entry in `local.secrets` in [`main.tf`](../main.tf) | Source repositories that receive the App private key as an Actions secret. It grants no installation access by itself. | +| App installation repositories | GitHub organization installation settings, with **Only select repositories** | Repositories on which the App installation is authorized. This is manual configuration and is not represented in Terraform state. | +| Workflow-requested token repositories | The `repositories` input in the [charts workflow](https://github.com/makeitworkcloud/charts/blob/main/.github/workflows/helm.yml) | The subset placed on the installation token created for one workflow run. Every requested repository must also be selected in the App installation. | + +The source workflow repository needs the Actions secret. It does not need to be +selected in the App installation unless a workflow token also requests access +to that repository. Verify the live installation scope in GitHub rather than +inferring it from this root. + +## Initial App creation and installation + +App creation and installation are one-time manual organization-owner actions. +Use an approved change record and a secure workstation. + +1. Create an organization-owned GitHub App in GitHub's developer settings. Use + the approved display name and disable webhooks unless a separately approved + consumer requires them. +2. Set repository permissions to **Contents: Read and write** and **Pull + requests: Read and write**. Do not grant organization, account, or additional + repository permissions without a reviewed requirement. +3. Record the resulting App ID in the source workflow's `app-id` input. Treat + that source reference as canonical; do not duplicate the numeric ID here. +4. Generate a private key. Keep the downloaded key out of the repository, + terminal output, tickets, chat, and CI logs. +5. Install the App on the organization using **Only select repositories**. + Select every repository requested by the source workflow and no others. Do + not select a source repository merely because it receives the Actions + secret. +6. Put the private key into the existing SOPS-backed field using the key + replacement procedure below, then distribute it through the reviewed + Terraform workflow. +7. Remove the downloaded plaintext file from the workstation after the + encrypted edit is saved and its presence is no longer needed. + +## Replace or rotate the SOPS-backed key + +For a planned rotation, keep the old GitHub App key active until the new key +has passed smoke verification. GitHub can accept both keys during the overlap. + +Before editing, install `sops` and use approved AWS credentials that can use the +KMS recipient configured in [`.sops.yaml`](../.sops.yaml). Do not copy that +recipient into documentation, tickets, or command output. + +1. Generate a new private key from the GitHub App settings. Locate the App by + the `app-id` reference in the source workflow. +2. From a workstation with approved SOPS access, open + `secrets/secrets.yaml` with `sops secrets/secrets.yaml`. +3. Replace only the existing chart updater field referenced by + [`main.tf`](../main.tf). Preserve its YAML type and formatting. Save and + close the SOPS editor without printing, piping, logging, or pasting the + plaintext key. +4. Confirm the file remains SOPS-encrypted and that the diff contains only the + expected encrypted field and SOPS metadata changes. Do not attach the diff + to external systems. +5. Remove the temporary plaintext download. Do not use a command-line argument, + shell history, Terraform variable, or unencrypted intermediate file to pass + the key. +6. Open a focused pull request and use the reviewed plan/apply flow below. +7. Verify secret metadata, then complete a real release smoke test before + deleting the old App key. + +## Reviewed plan and apply + +Use the repository's [OpenTofu caller](../.github/workflows/opentofu.yml); do not +run an unreviewed local apply. + +1. Open a pull request containing only the approved SOPS-backed change and any + intentional recipient-list change. +2. Require the test and plan checks to pass. Review the plan comment and confirm + it updates only the expected `github_actions_secret` instances. A key-only + rotation must not create repositories or claim to modify the GitHub App or + its installation. +3. Stop if a plan, log, or comment exposes a secret value or unexpected + sensitive data. Remove the exposure and rotate the affected key before + continuing. +4. Merge only after review. The push to `main` invokes the environment-gated + apply path. Confirm the apply job completed successfully before verification. + +## Verify secret metadata without retrieving the secret + +GitHub Actions secret values cannot be read back. Verify only the metadata: + +1. Read the source repository and Actions secret name from + [`main.tf`](../main.tf). +2. In the source repository, open **Settings > Secrets and variables > Actions** + and confirm the expected secret exists and its update time follows the + successful apply. +3. If using the GitHub API, call only the repository Actions secret metadata + endpoint and select `name`, `created_at`, and `updated_at`. Do not call any + external secret store or attempt to reconstruct the value. + +Example with values taken from source configuration at execution time: + +```sh +gh api "repos///actions/secrets/" \ + --jq '{name, created_at, updated_at}' +``` + +## Smoke verification with a real chart release + +Coordinate rotation with the next real `opencode-server` change. Do not reuse +an existing chart version or manufacture a mutable test release; published OCI +chart versions are immutable. + +1. Follow the charts repository process to make the real change and increment + `opencode-server/Chart.yaml` to a new version. +2. Confirm the charts pull request passes repository hygiene, Helm validation, + rendering, and packaging checks. +3. Merge the chart pull request and confirm the `main` workflow publishes that + exact version to GHCR. +4. Confirm the scoped GitHub App token step succeeds and the post-publish job + creates or updates a pull request in the configured destination repository. +5. Verify the generated pull request is authored through the App, changes only + the intended GitOps chart reference, and pins `targetRevision` to the newly + published immutable version. +6. Treat the generated pull request as the credential smoke test. Review and + roll it out separately using the `kustomize-cluster` [rollout and rollback + guide](https://github.com/makeitworkcloud/kustomize-cluster/blob/main/docs/rollout-and-rollback.md). + The updater must not sync Argo CD or deploy directly. +7. Delete the old App key only after the new key has passed this smoke test and + the rollback window is closed. + +## Add a new source repository + +Adding a source broadens private-key distribution and requires explicit review. + +1. If this root should manage the repository itself, add it to + `local.github_repositories` and add its exact required check-run names to the + status-check map in [`main.tf`](../main.tf). Do not add it merely to deliver a + secret if ownership belongs elsewhere. +2. Add the source repository to the chart updater secret's `repositories` list + in [`main.tf`](../main.tf). +3. Add or update the source workflow to use the canonical Actions secret and + App ID references. Request explicit `owner`, `repositories`, Contents, and + Pull requests scopes when creating the installation token. +4. Add only the workflow-requested destination repositories to the manual App + installation. The new source itself is not automatically an installation + repository. +5. Complete the reviewed plan/apply, metadata verification, and real-release + smoke test. + +## Add a new destination repository + +1. Decide separately whether this root should Terraform-manage the repository. + If so, update the managed repository and required-check maps in + [`main.tf`](../main.tf). +2. Add the destination to the App's organization installation using **Only + select repositories**. This is a manual organization-owner action. +3. Add the destination to the source workflow's `repositories` token request + and update the workflow logic to make only the approved changes there. +4. Confirm the existing App-level Contents and Pull requests permissions are + sufficient. Any permission increase requires separate review and manual + acceptance on the organization installation. +5. If a new source also needs the private key, add it through the source + procedure. A destination does not need the Actions secret merely because it + receives commits or pull requests. +6. Validate with a real immutable release and inspect the generated pull + request before rollout. + +## Emergency revocation + +If a private key may be exposed, revoke access before restoring automation: + +1. Delete the affected private key in the GitHub App settings immediately. If + the scope of exposure is unclear, suspend or uninstall the organization App + installation to invalidate installation-token creation broadly. +2. Disable the affected updater workflow if repeated failures or unwanted + writes are possible. Deleting the repository Actions secret is only a + temporary containment measure because the next Terraform apply will restore + the configured secret. +3. Record the revocation time, affected repositories, workflow runs, and any + App-authored commits or pull requests. Review the organization audit log. +4. Generate a replacement key, update SOPS, and use the reviewed plan/apply + path. Do not use the legacy AWS Secrets Manager copy as a fallback. +5. Restore the selected-repository installation or workflow only after its + scope is reviewed, then perform metadata and real-release smoke verification. +6. Revoke any remaining superseded key after recovery is confirmed. + +## Audit and rollback + +Retain the approved issue or change record, encrypted Git diff, pull-request +reviews, plan/apply checks, GitHub organization audit events, secret metadata +timestamps, release workflow run, and generated GitOps pull request. Never +attach a private key, decrypted SOPS content, Actions secret value, or sensitive +plan output to the audit record. + +For a planned rotation, the active old key is the rollback path. If the new key +fails before the old key is revoked, revert the encrypted secret-file change in +a new reviewed pull request and let the environment-gated apply redistribute +the prior value. Restore manual App permissions or selected repositories in +GitHub if those changed. If the old key was already revoked, it cannot be +restored; generate another key and fix forward. + +The legacy AWS Secrets Manager copy is unreferenced and outside this rollback +path. Its eventual deletion must be a separate approved cleanup after operators +agree that no rollback window remains.