Open a PostgreSQL service to declared sources over TLS (S-006-8, 10)

changelog.d/123-postgresql-remote-access.md

@@ -0,0 +1,3 @@
+### Added
+
+- A PostgreSQL service with `spec.postgresql.remoteAccess` opens to its declared sources over TLS: `services.yml` sets `listen_addresses = '127.0.0.1,<listenAddress>'` and `ssl = on` with the Debian `ssl-cert` snakeoil pair (restarting the cluster), and writes one `hostssl <database> <owner> <cidr> scram-sha-256` line per owned database and source between Cloudfall markers in `pg_hba.conf`, reloading only after `pg_hba_file_rules` reports no error; `baseline.yml` adds one `ip saddr`/`ip6 saddr <cidr> tcp dport <port> accept` rule per source, rendered as a `cloudfall_firewall.allowedInbound` entry with a new optional `sources` list. Removing `remoteAccess` resets the listener and removes the `pg_hba.conf` block and the firewall rule; databases named by `application` stay socket-only (#123)

docs/specs/006-standalone-postgresql.md

@@ -142,3 +142,9 @@ Touches `config/schemas/v1/service.schema.json`, `sdk/src/cloudfall/validation.p
 - cloudfall-dev/cloudfall#123: S-006-8, S-006-9, S-006-10
 
 ## Verification
+
+`remoteAccess` has not run against a live Debian host yet: the files were rendered from a validated project with the role templates, the role's decisions were evaluated with Ansible's templating, and the `pg_hba.conf` tasks ran under `ansible-playbook` on the controller against a copy of Debian's file with a stand-in for `psql`. PostgreSQL's real parse of the file, the restart onto the new listener, TLS with the snakeoil pair, and `nft` loading the rules are unexercised.
+
+- S-006-8: `test_s006_8_remote_access_renders_listener_tls_hba_and_firewall` and `test_s006_8_the_firewall_entry_carries_its_sources` (`engine/tests/test_postgresql_remote_access.py`) rendered `listen_addresses = '127.0.0.1,10.0.0.2'`, `ssl = on`, the block line `hostssl billing billing 10.0.0.0/24 scram-sha-256` (none for the application database `crm`), and `ip saddr 10.0.0.0/24 tcp dport 5432 accept` as the only rule for port 5432; `test_s006_8_s006_10_the_hba_block_opens_then_closes_leaving_the_rest` wrote exactly that line between the Cloudfall markers, reloaded once, and left Debian's lines unchanged
+- S-006-9: not run; it needs the live proving run on two hosts
+- S-006-10: `test_s006_10_without_remote_access_the_service_renders_closed` rendered `listen_addresses = '127.0.0.1'`, no rule for port 5432, and the block state `absent`; `test_s006_8_s006_10_the_hba_block_opens_then_closes_leaving_the_rest` then removed the block and the file read byte-identical to Debian's

engine/README.md

@@ -115,7 +115,9 @@ The firewall role renders `/etc/nftables.conf` from the rendered
 SSH port), validates it with `nft --check` before installation, and refuses
 any ruleset that does not allow the control connection's SSH port. The
 managed table replaces the complete ruleset; hosts whose profile declares no
-firewall are left unmanaged.
+firewall are left unmanaged. An entry with a `sources` list (a PostgreSQL
+service's `remoteAccess`) renders one `ip saddr`/`ip6 saddr` rule per source
+and never counts as the SSH port.
 
 The access role installs active environment-scoped `SshPublicKey` resources
 for root, validates the complete OpenSSH configuration, and only then
@@ -159,6 +161,13 @@ of every database restores into a scratch database before dropping it:
 sudo -u postgres /usr/local/sbin/cloudfall-postgresql-restore-check
 ```
 
+A service that declares `postgresql.remoteAccess` also listens on its
+`listenAddress` with `ssl = on` (the `ssl-cert` snakeoil pair); a change to
+the listener restarts the cluster. Its `pg_hba.conf` gains a block between
+Cloudfall markers with one `hostssl` SCRAM line per owned database and source,
+applied with `pg_reload_conf()` only after `pg_hba_file_rules` reports no
+error; a service without `remoteAccess` has the block removed.
+
 ## Build and deploy component releases
 
 `cloudfall-engine artifact build` clones a component's declared repository at

engine/ansible/roles/cloudfall_firewall/meta/argument_specs.yml

@@ -28,6 +28,12 @@ argument_specs:
                 required: true
               description:
                 type: str
+              sources:
+                type: list
+                elements: str
+                description: >-
+                  CIDRs the rule is restricted to, one rule per source; an
+                  entry without sources accepts every address
       cloudfall_firewall_configuration_path:
         type: path
         default: /etc/nftables.conf

engine/ansible/roles/cloudfall_firewall/tasks/main.yml

@@ -10,12 +10,35 @@
       Cloudfall firewall management requires Debian, systemd, and a declared
       default-deny ruleset with at least one allowed inbound rule
 
+- name: Refuse a source-restricted rule that opens to every address
+  ansible.builtin.assert:
+    that:
+      - >-
+        cloudfall_firewall_ruleset.allowedInbound
+        | selectattr('sources', 'defined')
+        | map(attribute='sources')
+        | select('equalto', [])
+        | list
+        | length == 0
+      - >-
+        cloudfall_firewall_ruleset.allowedInbound
+        | selectattr('sources', 'defined')
+        | map(attribute='sources')
+        | flatten
+        | select('in', ['0.0.0.0/0', '::/0'])
+        | list
+        | length == 0
+    fail_msg: >-
+      A firewall rule restricted to sources needs at least one, and none may
+      be 0.0.0.0/0 or ::/0
+
 - name: Refuse a ruleset that would block the control connection
   ansible.builtin.assert:
     that:
       - >-
         ansible_port | default(22) | int in (
           cloudfall_firewall_ruleset.allowedInbound
+          | rejectattr('sources', 'defined')
           | selectattr('protocol', 'equalto', 'tcp')
           | map(attribute='port')
           | map('int')

engine/ansible/roles/cloudfall_firewall/templates/nftables.conf.j2

@@ -13,7 +13,15 @@ table inet cloudfall {
         ip protocol icmp accept
         meta l4proto ipv6-icmp accept
 {% for rule in cloudfall_firewall_ruleset.allowedInbound %}
+{% if rule.sources is defined %}
+{% for source in rule.sources %}
 {% if rule.description is defined %}
+        {{ 'ip6' if ':' in source else 'ip' }} saddr {{ source }} {{ rule.protocol }} dport {{ rule.port }} accept comment "{{ rule.description }}"
+{% else %}
+        {{ 'ip6' if ':' in source else 'ip' }} saddr {{ source }} {{ rule.protocol }} dport {{ rule.port }} accept
+{% endif %}
+{% endfor %}
+{% elif rule.description is defined %}
         {{ rule.protocol }} dport {{ rule.port }} accept comment "{{ rule.description }}"
 {% else %}
         {{ rule.protocol }} dport {{ rule.port }} accept

engine/ansible/roles/cloudfall_postgresql/defaults/main.yml

@@ -6,3 +6,6 @@ cloudfall_postgresql_repository_key_path: /etc/apt/keyrings/postgresql.asc
 cloudfall_postgresql_repository_url: https://apt.postgresql.org/pub/repos/apt
 cloudfall_postgresql_password_files: {}
 cloudfall_postgresql_password_path: /var/lib/postgresql/.cloudfall-owner-password
+cloudfall_postgresql_ssl_cert_file: /etc/ssl/certs/ssl-cert-snakeoil.pem
+cloudfall_postgresql_ssl_key_file: /etc/ssl/private/ssl-cert-snakeoil.key
+cloudfall_postgresql_hba_marker: "# {mark} CLOUDFALL MANAGED REMOTE ACCESS"

engine/ansible/roles/cloudfall_postgresql/meta/argument_specs.yml

@@ -67,6 +67,16 @@ argument_specs:
                       key:
                         type: str
                         required: true
+              remoteAccess:
+                type: dict
+                options:
+                  listenAddress:
+                    type: str
+                    required: true
+                  allowedSources:
+                    type: list
+                    elements: str
+                    required: true
           backup:
             type: dict
             required: true
@@ -98,6 +108,18 @@ argument_specs:
         type: path
         default: /var/lib/postgresql/.cloudfall-owner-password
         description: Host path an owner role's password is staged at, 0600
+      cloudfall_postgresql_ssl_cert_file:
+        type: path
+        default: /etc/ssl/certs/ssl-cert-snakeoil.pem
+        description: Server certificate of remoteAccess, the ssl-cert snakeoil
+      cloudfall_postgresql_ssl_key_file:
+        type: path
+        default: /etc/ssl/private/ssl-cert-snakeoil.key
+        description: Server key of remoteAccess, the ssl-cert snakeoil
+      cloudfall_postgresql_hba_marker:
+        type: str
+        default: "# {mark} CLOUDFALL MANAGED REMOTE ACCESS"
+        description: Markers of the remoteAccess block in pg_hba.conf
       cloudfall_postgresql_metrics_role:
         type: str
         default: alloy

engine/ansible/roles/cloudfall_postgresql/tasks/main.yml

@@ -12,6 +12,58 @@
       PostgreSQL deployment requires its declared Debian server and a
       loopback-only bind address
 
+- name: Refuse remote access that is not narrowed to declared sources
+  ansible.builtin.assert:
+    that:
+      - cloudfall_postgresql_remote_access.listenAddress | length > 0
+      - >-
+        cloudfall_postgresql_remote_access.listenAddress
+        not in ['*', '0.0.0.0', '::', 'localhost']
+      - "',' not in cloudfall_postgresql_remote_access.listenAddress"
+      - "\"'\" not in cloudfall_postgresql_remote_access.listenAddress"
+      - cloudfall_postgresql_remote_access.allowedSources | length > 0
+      - >-
+        cloudfall_postgresql_remote_access.allowedSources
+        | select('in', ['0.0.0.0/0', '::/0'])
+        | list
+        | length == 0
+      - >-
+        cloudfall_postgresql_service.postgresql.databases
+        | selectattr('passwordRef', 'defined')
+        | list
+        | length > 0
+      # pg_hba.conf reads these bare words as keywords, not names: a database
+      # named all would open every database, an owner named all every role.
+      - >-
+        cloudfall_postgresql_service.postgresql.databases
+        | selectattr('passwordRef', 'defined')
+        | map(attribute='name')
+        | select('in', cloudfall_postgresql_hba_keywords)
+        | list
+        | length == 0
+      - >-
+        cloudfall_postgresql_service.postgresql.databases
+        | selectattr('passwordRef', 'defined')
+        | map(attribute='owner')
+        | select('in', cloudfall_postgresql_hba_keywords)
+        | list
+        | length == 0
+    fail_msg: >-
+      remoteAccess of service {{ cloudfall_postgresql_service.id }} needs one
+      listen address that is not a wildcard, at least one source that is not
+      0.0.0.0/0 or ::/0, and a database with an owner, and no owned database
+      or owner named {{ cloudfall_postgresql_hba_keywords | join(', ') }}
+  vars:
+    cloudfall_postgresql_remote_access: >-
+      {{ cloudfall_postgresql_service.postgresql.remoteAccess }}
+    cloudfall_postgresql_hba_keywords:
+      - all
+      - sameuser
+      - samerole
+      - samegroup
+      - replication
+  when: cloudfall_postgresql_service.postgresql.remoteAccess is defined
+
 - name: Require a staged password for every owner role
   ansible.builtin.assert:
     that:
@@ -91,6 +143,26 @@
     cache_valid_time: 3600
     policy_rc_d: 101
 
+- name: Install the snakeoil certificate pair for remote access
+  ansible.builtin.apt:
+    name: ssl-cert
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+  when: cloudfall_postgresql_service.postgresql.remoteAccess is defined
+
+- name: Let PostgreSQL read the snakeoil key
+  ansible.builtin.user:
+    name: postgres
+    groups:
+      - ssl-cert
+    append: true
+  # The server reads its key at start, so a new group needs a restart.
+  notify: Restart PostgreSQL
+  when: cloudfall_postgresql_service.postgresql.remoteAccess is defined
+
+# listen_addresses and ssl only change at server start: a changed file
+# restarts the cluster through the handler, flushed before readiness.
 - name: Install the managed cluster configuration
   ansible.builtin.template:
     src: cloudfall.conf.j2
@@ -132,6 +204,94 @@
   until: cloudfall_postgresql_readiness.rc == 0
   when: not ansible_check_mode
 
+# The block is written with remoteAccess and removed without it, so an
+# earlier opening never lingers; lines outside the markers stay untouched.
+- name: Converge the remote access block of pg_hba.conf
+  vars:
+    cloudfall_postgresql_hba_path: >-
+      /etc/postgresql/{{
+        cloudfall_postgresql_service.postgresql.majorVersion
+      }}/main/pg_hba.conf
+  block:
+    - name: Write or remove the Cloudfall block in pg_hba.conf
+      ansible.builtin.blockinfile:
+        path: "{{ cloudfall_postgresql_hba_path }}"
+        marker: "{{ cloudfall_postgresql_hba_marker }}"
+        block: >-
+          {{
+            lookup('ansible.builtin.template', 'pg_hba-remote-access.j2')
+            if cloudfall_postgresql_service.postgresql.remoteAccess is defined
+            else ''
+          }}
+        state: >-
+          {{
+            'present'
+            if cloudfall_postgresql_service.postgresql.remoteAccess is defined
+            else 'absent'
+          }}
+      register: cloudfall_postgresql_hba
+
+    # pg_hba_file_rules parses the file on disk; reload only a file it reads.
+    - name: Refuse a pg_hba.conf PostgreSQL cannot read
+      ansible.builtin.command:
+        argv:
+          - /usr/bin/psql
+          - --port
+          - "{{ cloudfall_postgresql_service.bind.port }}"
+          - --tuples-only
+          - --no-align
+          - --command
+          - >-
+            SELECT line_number || ': ' || error
+            FROM pg_hba_file_rules WHERE error IS NOT NULL
+      become: true
+      become_user: postgres
+      register: cloudfall_postgresql_hba_errors
+      changed_when: false
+      failed_when: >-
+        cloudfall_postgresql_hba_errors.rc != 0
+        or cloudfall_postgresql_hba_errors.stdout_lines | length > 0
+      when:
+        - not ansible_check_mode
+        - cloudfall_postgresql_hba is changed
+
+    - name: Reload PostgreSQL to apply pg_hba.conf
+      ansible.builtin.command:
+        argv:
+          - /usr/bin/psql
+          - --port
+          - "{{ cloudfall_postgresql_service.bind.port }}"
+          - --tuples-only
+          - --no-align
+          - --command
+          - SELECT pg_reload_conf()
+      become: true
+      become_user: postgres
+      register: cloudfall_postgresql_hba_reload
+      changed_when: true
+      failed_when: >-
+        cloudfall_postgresql_hba_reload.rc != 0
+        or cloudfall_postgresql_hba_reload.stdout | trim != 't'
+      when:
+        - not ansible_check_mode
+        - cloudfall_postgresql_hba is changed
+  rescue:
+    - name: Close remote access when pg_hba.conf could not be applied
+      ansible.builtin.blockinfile:
+        path: "{{ cloudfall_postgresql_hba_path }}"
+        marker: "{{ cloudfall_postgresql_hba_marker }}"
+        state: absent
+
+    - name: Stop on a pg_hba.conf that was not applied
+      ansible.builtin.fail:
+        msg: >-
+          pg_hba.conf of service {{ cloudfall_postgresql_service.id }} was not
+          reloaded and its Cloudfall block was removed:
+          {{
+            cloudfall_postgresql_hba_errors.stdout_lines | default([])
+            | join('; ')
+          }}
+
 - name: Read existing PostgreSQL roles
   ansible.builtin.command:
     argv:

engine/ansible/roles/cloudfall_postgresql/templates/cloudfall.conf.j2

@@ -1,3 +1,10 @@
 # Managed by Cloudfall. Local changes will be overwritten.
+{% if cloudfall_postgresql_service.postgresql.remoteAccess is defined %}
+listen_addresses = '{{ cloudfall_postgresql_service.bind.address }},{{ cloudfall_postgresql_service.postgresql.remoteAccess.listenAddress }}'
+ssl = on
+ssl_cert_file = '{{ cloudfall_postgresql_ssl_cert_file }}'
+ssl_key_file = '{{ cloudfall_postgresql_ssl_key_file }}'
+{% else %}
 listen_addresses = '{{ cloudfall_postgresql_service.bind.address }}'
+{% endif %}
 port = {{ cloudfall_postgresql_service.bind.port }}

engine/ansible/roles/cloudfall_postgresql/templates/pg_hba-remote-access.j2

@@ -0,0 +1,5 @@
+{% for database in cloudfall_postgresql_service.postgresql.databases if database.passwordRef is defined %}
+{% for source in cloudfall_postgresql_service.postgresql.remoteAccess.allowedSources %}
+hostssl {{ database.name }} {{ database.owner }} {{ source }} scram-sha-256
+{% endfor %}
+{% endfor %}