Skip to content

Converge PostgreSQL databases owned by a declared role (S-006-5, 6, 7, 12, 13) - #139

Merged
shipmill-romamo[bot] merged 3 commits into
mainfrom
feat/122-owned-postgresql-converge
Oct 27, 2026
Merged

shipmill-romamo[bot] merged 3 commits into
mainfrom
feat/122-owned-postgresql-converge

Conversation

@shipmill-romamo

@shipmill-romamo shipmill-romamo Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements S-006's converge of a database owned by a declared role (docs/specs/006-standalone-postgresql.md), criteria S-006-5, S-006-6, S-006-7, S-006-12 and S-006-13. Network access over TLS (S-006-8 to S-006-10) is #123, so the spec stays approved.

  • cloudfall_postgresql creates the owner login role along with the application roles, sets its password on every converge (tasks/owner_password.yml), and gives it the database: CREATE DATABASE ... OWNER, plus ALTER DATABASE ... OWNER TO when an existing database belongs to someone else. The password reaches the host as a 0600 file owned by postgres. psql reads that file through \set ... \cat`, after SET password_encryption = 'scram-sha-256', then runs ALTER ROLE ... PASSWORD :'cloudfall_password', so the value is in no task argument, no stdin, and no command line. Every task that sees it is no_log, and an always` removes the file. A converge that has no staged password for an owner fails before any change
  • The role's argument spec makes application optional and takes passwordRef. It still refuses remoteAccess, which Open a PostgreSQL service to declared sources over TLS (S-006-8, 9, 10) #123 converges
  • How the value gets to Ansible: the controller decrypts each passwordRef (SopsSecretProvider.fetch_value, the secrets render layout) and writes it to a 0600 file in a 0700 private temp dir outside the project (staged_password_files). It passes only the paths, as --extra-vars '{"cloudfall_services_password_files": {service: {owner: path}}}', and removes the dir when the run ends, pass or fail. This is the pattern data copy already uses for its source URL file: a controller-side file whose path crosses the command line, never its content
  • All four paths that run services.yml go through it, each with a --secrets-dir (default secrets): the converge_services MCP tool (mcp serve --secrets-dir already existed), the services step of cloudfall migrate (new flag), operator approve/operator run (new flag on OperatorEngineArgs; engine_executor now takes the provider), and cloudfall-engine playbook run services.yml (new flag; reads the owned databases from the rendered inventory, decrypts through ctx.run sops. An inventory that isn't a rendered Cloudfall one, or declares no owned database, stages nothing and runs exactly as on main)
  • A missing file, missing key, or empty value raises PRECONDITION naming the expected path and key before any playbook starts. migrate checks before its first step, so the baseline doesn't run either
  • Answers name the reference, the key, and the value's SHA-256 in a new owned_roles list: the converge_services tool (new ServicesConverged answer, high_entropy=False), playbook run (Played.owned_roles, empty for other playbooks), and migrate's services step detail (ownedRoles, only when an owner exists). All of these are additive
  • Operator answer: operator approve's answer is the proposal receipt, so its spec.outcome gains an optional ownedRoles list (same shape: service, owner, passwordRef, key, sha256) once a converge-services run sets an owner password. operator run adds ownedRoles to each executed proposal's entry. operator-proposal.schema.json gets the optional property only, so every existing receipt still validates. The executor now returns the OwnedRoleRecords it set (internal; engine_executor is only called from app.py)
  • Backups, the restore drill and data copy needed no change. They already select by name and restore with --role <owner>; tests now prove it for an owned database
  • docs/secrets-guide.md gains "Database owner passwords"

Fixes #122
Part of #81

Test plan

  • engine/tests/test_examples_render_unchanged.py (S-006-12): renders config/examples through cloudfall-engine inventory render and the cloudfall.conf, nftables.conf, backup and restore-check templates, and compares them byte for byte with goldens under engine/tests/golden/examples/. The goldens were captured from origin/main before any change on this branch
  • engine/tests/test_postgresql_owner_role.py:
    • Templar replays of the role's loops and conditions: the owner gets a login role and a password include, the owned database is created with its owner and re-owned when it drifts, and a missing staged password fails the assert (S-006-5)
    • the rendered SQL on stdin (S-006-5)
    • no_log on every task that sees the password (S-006-6)
    • the password tasks run for real under ansible-playbook -vvv on localhost with a stand-in psql. The value reaches the stand-in via a 0600 file, the file is gone afterwards, and the value is in neither the Ansible output nor psql's stdin or argv (S-006-6)
    • the backup and drill scripts for billing (S-006-13)
    • the data-copy selection and receipt tasks replayed on localhost: restore role billing and receipt spec.owner: billing (S-006-13)
  • engine/tests/test_services_passwords.py: cloudfall-engine playbook run services with a missing file or key answers PRECONDITION with path and key, and Ansible never starts (S-006-7). With a plain INI inventory, or JSON of another shape, playbook run services.yml exits 0 as on main, with no secrets read and no argument error. A real run against a local host with a stand-in sops on PATH (--tags that match no task) answers owned_roles with the hash; the value is in neither stdout nor stderr at -vvv, and the staged file is gone (S-006-6)
  • sdk/tests/test_owned_postgresql_converge.py: converge_services with a recording step runner that reads the staged file mid-run (value, 0600, dir 0700, outside the project, removed afterwards), with the value in no argv, environment, project file, rendered inventory or answer (S-006-6). Missing file, missing key and empty value run no step (S-006-7). The operator executor fails with no step. approve() answers and receipts ownedRoles with the hash, the receipt round-trips through ProposalStore.load, and the value appears in neither the answer, the receipt, nor any project file. operator run's autonomy report names each executed proposal's roles. cloudfall migrate --yes answers PRECONDITION before any step and writes no plan. The migrate services step records ownedRoles and keeps the value out of the plan file. The converge_services MCP tool, over its real stdio transport, answers PRECONDITION naming path and key
  • specs.py coverage --spec 006 lists tests for S-006-5, 6, 7, 12 and 13
  • Ran locally: uv run ruff check ., uv run mypy, uv run cloudfall config validate --project config/examples, cloudfall-engine inventory render, the services.yml and data.yml syntax checks, ansible-lint engine/ansible (with engine/ansible/ansible.cfg), and engine/tests plus the SDK tests of touched modules (migrate, data migration, operator, secrets, agent server, lifecycle, owned PostgreSQL, commands, app, arguments). The full suite is left to CI
  • Not verified on a live Debian host with PostgreSQL: that pg_roles shows billing with rolcanlogin and a SCRAM-SHA-256 hash; that psql's backquote \set reads the staged file as postgres and ALTER ROLE stores it; the ALTER DATABASE ... OWNER TO path on a real cluster; pg_dump of billing, the drill passing, and pg_restore --role billing against a real source; and a real sops decrypt

Plan

  • Serves: quality bar
  • Files: config/schemas/v1/operator-proposal.schema.json, engine/ansible/roles/cloudfall_postgresql/{defaults,meta,tasks/main.yml,tasks/owner_password.yml}, engine/ansible/playbooks/services.yml, engine/src/cloudfall_engine/cli.py, sdk/src/cloudfall/{secrets,lifecycle,migrate,operator,agent_tools,app}.py, docs/secrets-guide.md, changelog.d/122-owned-postgresql-converge.md, tests and engine/tests/golden/examples
  • Order: capture S-006-12 goldens from main first, then secret resolution and staging in secrets.py, converge_services in lifecycle, the four callers, the role tasks and argument spec, and tests per criterion (see the spec's Behaviour)
  • Proof: S-006-5, S-006-6, S-006-7, S-006-12, S-006-13 (specs.py coverage --spec 006; S-006-8 to S-006-10 stay with Open a PostgreSQL service to declared sources over TLS (S-006-8, 9, 10) #123)

🤖 Generated with Claude Code

services.yml now creates the owner login role, sets its password from
passwordRef as SCRAM-SHA-256 on every converge, and gives it the
database. Every path that runs services.yml (the converge_services MCP
tool, cloudfall migrate, operator approve and run, and
cloudfall-engine playbook run services.yml) reads the password from
--secrets-dir and stages it in a 0600 file in a private temp dir
outside the project, passing only the path to Ansible, as data copy
does with its source URL file. The role copies it to the host as 0600,
psql reads it there, and every task that sees it is no_log. Answers
name the reference, the key, and the value's SHA-256. A missing file
or key fails with PRECONDITION naming the path and key before any
playbook runs.

Backups, the restore drill, and data copy already select by name and
restore as the owner; tests now prove it for an owned database. The
example project renders byte-identical to goldens captured from main.

Fixes #122
Part of #81

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shipmill-romamo Bot and others added 2 commits October 27, 2026 11:55
Spec 006 lists the operator's converge-services operation among the
paths that run services.yml, and its answer must name the reference,
the key, and the value's SHA-256. operator approve answers with the
proposal receipt, so its outcome gains an optional ownedRoles list;
operator run adds it to each executed proposal's entry. The schema
change is one optional property, so every existing receipt validates.

Part of #122

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloudfall-engine playbook run services.yml refused an inventory that
wasn't a rendered Cloudfall one with an argument error, changing the
command for its current users. Such an inventory, or one declaring no
owned database, now stages no passwords and runs as on main; only an
inventory that declares owned databases reads the secrets, and a
missing file or key there still stops with PRECONDITION.

Part of #122

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

@romamo Decided without waiting: the owner password reaches Ansible as the path of a 0600 file staged outside the project for the run (the "own role variable" of spec 006 is that path, never the value), and converge refuses an empty password value like a missing key, and one role owning databases with two different passwordRefs (secrets_owner_conflict).

Why: spec 006 says the value reaches the host in a 0600 file removed after the run and never appears in Ansible output (S-006-6); passing the path keeps Ansible from holding the value at all, as data copy already does for its source URL.
Not taken: passing the value itself as an extra var with no_log (Ansible holds the secret in memory and in any debug dump), and accepting an empty or conflicting password (a role that can't log in, or one whose password flips each converge).

Reply with a comment starting with veto before 2026-10-28 10:45 UTC to reverse it, adding what you want instead if you like. A later veto still counts and gets a follow-up.

@shipmill-romamo

Copy link
Copy Markdown
Contributor Author

@romamo Decided without waiting: --secrets-dir (default secrets) is added to cloudfall migrate, the operator commands and cloudfall-engine playbook run; the answers of playbook run, converge_services, migrate's services step, operator approve and operator run gain owned_roles/ownedRoles (reference, key, SHA-256, never the value), and the proposal receipt gains one optional property for it.

Why: spec 006 names --secrets-dir on each path that runs services.yml, and S-006-6 asks each answer to name the reference, key and hash. An inventory with no owned database runs exactly as before, any inventory shape included.
Not taken: leaving the operator path without the hash, which would miss S-006-6 there.

Reply with a comment starting with veto before 2026-10-28 10:45 UTC to reverse it, adding what you want instead if you like. A later veto still counts and gets a follow-up.

@shipmill-romamo shipmill-romamo Bot added the decided An agent decided at notify; the maintainer may veto label Oct 27, 2026
@shipmill-romamo
shipmill-romamo Bot merged commit 9f1c03c into main Oct 27, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the feat/122-owned-postgresql-converge branch October 27, 2026 10:16
shipmill-romamo Bot added a commit that referenced this pull request Oct 27, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shipmill-romamo Bot added a commit that referenced this pull request Oct 27, 2026
* Run Component build steps in artifact build

artifact build packaged the git tree only, so a Component could not ship a
built frontend such as a gitignored web/dist. It now checks every
spec.buildTools entry on PATH at its minimum version before cloning, runs
each spec.build step in the clean checkout through env -i with only PATH,
HOME, LANG, TMPDIR and CI=true, streams step output to stderr so stdout
stays one JSON envelope, and packages the tracked files plus every
spec.artifact.include path. A failed or timed-out step or a missing include
writes no archive. The Artifact metadata gains an optional spec.build with
the steps and tool versions. A Component without build is packaged exactly
as before.

Spec 002 is now built: every criterion has its test and its verification.

Fixes #115

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bound artifact build by its declared step timeouts; refuse deleted tracked files

A fixed 1800 s command limit cut long builds short with treaty's TIMEOUT,
and the SDK/MCP path capped the engine call at 3600 s, so a step never
reached its own timeoutSeconds. artifact build now declares no fixed limit
and applies its own: 1800 s for the clone plus every declared step timeout
(ComponentInventory.artifact_timeout_seconds), shared by every child, so
a step's own limit is the one that fires (artifact_build_step_failed). An
explicit --timeout still replaces it. The SDK's artifact step carries no
cap of its own and leaves the limit to the engine.

A step that deletes a tracked file broke S-002-3 by leaving it out of the
archive. The build now refuses it with artifact_tracked_file_missing,
naming the path, before any archive is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep artifact build --timeout 0 meaning no limit

Treaty's --timeout reaches a handler only as the effective timeout, the
same whether the flag was omitted or 0, so the computed limit silently
replaced an explicit 0. The command now takes --timeout as a field of its
own, which treaty allows in place of its framework flag, parsed with
Timeout.parse: omitted applies the clone time plus every declared step
timeout, 0 disables the limit, and any other value replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refuse an artifact.include path that leaves the checkout through a symlink

A tracked symlink above an include path (web/link -> /outside, include
web/link/secrets) let the build package controller files outside the
checkout. The include path itself may still be a symlink, archived as one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sort the engine CLI's type-only imports after the rebase onto #139

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: shipmill-romamo[bot] <338557544+shipmill-romamo[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

decided An agent decided at notify; the maintainer may veto

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Converge owned PostgreSQL databases with their role and password (S-006-5, 6, 7, 12, 13)

0 participants