Render Domain routes and serve static files from the release (S-003-2..7, S-003-13..15)

changelog.d/162-domain-routes-render.md

@@ -0,0 +1,3 @@
+### Added
+
+- `cloudfall domains apply` renders a Domain's `spec.routes`: a proxy route becomes a `location` that proxies to its `upstream` with the full request path, and a static route serves `<installRoot>/current/<directory>` of its component, answering `spaFallback` for an unknown path when it declares one and 404 when it doesn't. Without a `/` route, a path no route matches answers 404, not Nginx's default page. Locations render longest path first, whatever the declared order, where the single proxy location rendered before (the port 80 server without active TLS, the 443 server with it); the port 80 redirect and the ACME challenge location stay. The site role grants the Nginx worker (`www-data`) traverse-only access (POSIX ACL `user:www-data:--x`, installing `acl`) from the application root to the install root and its `releases/`, so it reads the release files but never `shared/env/<component>.env`, and a deploy or `cloudfall rollback` switches what a static route serves without re-running the domains playbook or reloading Nginx. The domains playbook's early check now refuses only a Domain with neither `proxy.upstream` nor `routes`. A Domain with only `proxy.upstream` renders a byte-identical vhost (#162)

docs/specs/003-domain-routes.md

@@ -1,6 +1,6 @@
 # S-003: Domain routes
 
-status: approved
+status: built
 serves: G-3
 
 ## Problem
@@ -135,3 +135,21 @@ Touches `config/schemas/v1/domain.schema.json`, `sdk/src/cloudfall/inventory.py`
 - cloudfall-dev/cloudfall#162: S-003-2, S-003-3, S-003-4, S-003-5, S-003-6, S-003-7, S-003-13, S-003-14, S-003-15
 
 ## Verification
+
+No criterion has run against a real Nginx or a live Debian host yet. Validation ran through `cloudfall config validate`; the vhost was rendered by Ansible's own `template` module on localhost from the engine inventory of the example project with routes; what a request answers was worked out by a small model of Nginx's longest-prefix match, `alias`, `root` and `try_files` over a release tree under the test's temporary directory; the ACL tasks ran on localhost with stand-ins for `getfacl` and `setfacl`. `nginx -t`, real requests, a real POSIX ACL and the `www-data` user are unexercised.
+
+- S-003-1: `test_s003_1_proxy_and_spa_static_routes_pass_config_validate` ran `cloudfall config validate` on the example Domain with a `/api/` proxy route and a `/` static route with `spaFallback: index.html`, exit 0; `test_s003_1_a_domain_with_only_an_upstream_serializes_as_before`, no `routes` key, passed
+- S-003-2: `test_s003_2_routes_render_where_the_single_proxy_rendered`, TLS on and off on 1.22.1 and 1.26.3: balanced braces, every directive ending in `;`, the routes in the 80 server without TLS and in the 443 server with it, each version's HTTP/2 spelling, passed; `nginx -t` was not run
+- S-003-3: `test_s003_3_unknown_paths_fall_back_and_assets_are_served`: `/` renders `alias <installRoot>/current/web/dist/;` and `try_files $uri /index.html;`, and the model answered `/some/unknown/path` with `index.html`'s bytes and `/assets/app.js` with that file's, passed; no real GET
+- S-003-4: `test_s003_4_api_requests_reach_the_upstream_with_their_full_path`: `/api/` renders `proxy_pass http://127.0.0.1:8310;` with no URI part, so the model passes `/api/health` unchanged, passed; no real upstream saw the request
+- S-003-5: `test_s003_5_a_static_route_without_a_fallback_answers_404`: a `/docs/` route without `spaFallback` renders no `try_files`, and the model answered an existing file 200 and a missing one 404, passed; no real GET
+- S-003-6: `test_s003_6_a_rollback_serves_the_previous_release_without_nginx` ran `rollback.yml`'s own symlink switch on localhost and the model served the previous release's `index.html` for `/`; the vhost names `current`, never a release id, and the rollback tasks never touch Nginx, passed; no real `cloudfall rollback` or Nginx
+- S-003-7: `test_s003_7_an_upstream_only_domain_renders_byte_identical` compared Ansible's render against `engine/tests/golden/nginx-site/`, captured from the template on origin/main before the change, for TLS on and off on 1.22.1 and 1.26.3, byte-identical
+- S-003-8: `test_s003_8_two_routes_for_one_path_are_refused`, exit `CONFIG_INVALID` with `domain_route_path_duplicate`, passed
+- S-003-9: `test_s003_9_a_domain_with_upstream_and_routes_is_refused` (`domain_route_source_invalid`) and `test_s003_9_a_domain_with_neither_upstream_nor_routes_still_passes`, passed
+- S-003-10: `test_s003_10_a_route_on_the_acme_challenge_is_refused` (`domain_route_path_reserved`) and `test_s003_10_an_invalid_route_path_is_refused` (a static path without `/`, a space, `;`, `{`: `domain_route_path_invalid`), passed
+- S-003-11: `test_s003_11_a_static_route_naming_a_missing_component_is_refused` (`resource_reference_missing`) and `test_s003_11_a_static_component_off_the_proxy_server_is_refused` (`domain_route_component_not_on_proxy`), passed
+- S-003-12: `test_s003_12_a_static_directory_or_fallback_outside_the_release_is_refused` (`..`, absolute, `.`) and `test_s003_12_route_value_objects_match_the_schema`, passed
+- S-003-13: `test_s003_13_locations_render_longest_path_first_whatever_the_order`, TLS on and off on both versions: `/api/v2/`, `/docs/`, `/api/`, `/` in that order, and the same routes declared in reverse rendered identical bytes, passed
+- S-003-14: `test_s003_14_the_worker_may_only_traverse_to_the_release` ran the role's collect and grant tasks on localhost with stand-in ACL tools: `user:www-data:--x` on `/srv/apps/crm`, `/srv/apps/crm/backend` and its `releases/` only, nothing on a second run, the env file left `root` `0640` and no user or group task in the role; `test_s003_14_an_upstream_only_domain_grants_nothing`, passed. A real `www-data` reading or failing to read a file was not checked
+- S-003-15: `test_s003_15_the_acme_challenge_still_renders_on_a_domain_with_routes`, TLS on and off: the 80 server keeps `location /.well-known/acme-challenge/ { root /var/lib/letsencrypt; }`, and the model served a token from the webroot and 404 for a missing one, passed; no real certbot challenge

engine/README.md

@@ -258,9 +258,20 @@ task domains:deploy
 task domains:deploy ISSUE_CERTIFICATES=true
 ```
 
-`proxy.upstream` is optional in the schema, for a Domain the fleet routes with
-its own proxy, but the site role needs it. Before it changes any server, the
-playbook fails and names every Domain of the run that has no `upstream`.
+`proxy.upstream` and `routes` are optional in the schema, for a Domain the
+fleet routes with its own proxy, but the site role needs one of them. Before it
+changes any server, the playbook fails and names every Domain of the run that
+has neither.
+
+A Domain with `routes` renders one `location` per route, longest path first.
+A proxy route passes the full request path to its `upstream`; a static route
+serves `<installRoot>/current/<directory>` of its component, so a deploy or a
+rollback switches what it serves with no Nginx reload, and answers its
+`spaFallback` for an unknown path. Without a `/` route, a path no route
+matches answers 404. The role installs `acl` and grants the
+Nginx worker (`www-data`) traverse-only access (`user:www-data:--x`) from the
+application root to the install root and its `releases/`; the worker joins no
+group, so `shared/env/<component>.env` stays unreadable to it.
 
 The site role installs Nginx and Certbot, serves the ACME challenge webroot,
 and converges in two phases: until a certificate exists the route serves

engine/ansible/playbooks/domains.yml

@@ -1,23 +1,23 @@
 ---
-# `upstream` is optional in the schema, since a fleet may route a Domain with
-# its own proxy, but cloudfall_nginx_site cannot render one without it. The
-# check runs across every host of the run before the serial play below, so a
-# refused run changes no server.
+# `upstream` and `routes` are optional in the schema, since a fleet may route a
+# Domain with its own proxy, but cloudfall_nginx_site cannot render one with
+# neither. The check runs across every host of the run before the serial play
+# below, so a refused run changes no server.
 - name: Refuse Domains the site role cannot render
   hosts: cloudfall_servers
   gather_facts: false
   any_errors_fatal: true
   tasks:
-    - name: Require an upstream on every Domain this run renders
+    - name: Require an upstream or routes on every Domain this run renders
       ansible.builtin.assert:
         that:
           - cloudfall_domains_without_upstream | length == 0
         fail_msg: >-
           {{ 'Domain' if cloudfall_domains_without_upstream | length == 1
           else 'Domains' }} {{ cloudfall_domains_without_upstream | join(', ') }}
           {{ 'has' if cloudfall_domains_without_upstream | length == 1
-          else 'have' }} no proxy.upstream, which cloudfall_nginx_site needs to
-          render {{ 'it' if cloudfall_domains_without_upstream | length == 1
+          else 'have' }} no proxy.upstream or routes, which cloudfall_nginx_site
+          needs to render {{ 'it' if cloudfall_domains_without_upstream | length == 1
           else 'them' }}; no server was changed
         quiet: true
       vars:
@@ -39,6 +39,7 @@
               else cloudfall_domains_in_run
             )
             | rejectattr('proxy.upstream', 'defined')
+            | rejectattr('routes', 'defined')
             | map(attribute='id')
             | unique
             | sort

engine/ansible/roles/cloudfall_nginx_site/defaults/main.yml

@@ -2,3 +2,5 @@
 cloudfall_nginx_site_issue_certificate: false
 cloudfall_nginx_site_acme_webroot: /var/lib/letsencrypt
 cloudfall_nginx_site_receipt_directory: ""
+# Debian's Nginx runs its workers as www-data.
+cloudfall_nginx_site_worker_user: www-data

engine/ansible/roles/cloudfall_nginx_site/meta/argument_specs.yml

@@ -39,6 +39,44 @@ argument_specs:
                   port:
                     type: int
                     required: true
+          routes:
+            type: list
+            elements: dict
+            options:
+              path:
+                type: str
+                required: true
+              upstream:
+                type: dict
+                options:
+                  address:
+                    type: str
+                    required: true
+                  port:
+                    type: int
+                    required: true
+              static:
+                type: dict
+                options:
+                  component:
+                    type: str
+                    required: true
+                  directory:
+                    type: str
+                    required: true
+                  spaFallback:
+                    type: str
+                  servedDirectory:
+                    type: path
+                    required: true
+                  traverseDirectories:
+                    type: list
+                    elements: path
+                    required: true
+            mutually_exclusive:
+              - [upstream, static]
+            required_one_of:
+              - [upstream, static]
           origin:
             type: dict
             required: true
@@ -67,3 +105,6 @@ argument_specs:
       cloudfall_nginx_site_receipt_directory:
         type: str
         default: ""
+      cloudfall_nginx_site_worker_user:
+        type: str
+        default: www-data

engine/ansible/roles/cloudfall_nginx_site/tasks/main.yml

@@ -5,11 +5,13 @@
       - ansible_facts["distribution"] == "Debian"
       - ansible_facts["service_mgr"] == "systemd"
       - cloudfall_nginx_site_domain.proxy.server == cloudfall_server_id
-      - cloudfall_nginx_site_domain.proxy.upstream is defined
+      - >-
+        cloudfall_nginx_site_domain.proxy.upstream is defined
+        or cloudfall_nginx_site_domain.routes is defined
       - cloudfall_nginx_site_domain.tls.mode in ["disabled", "required"]
     fail_msg: >-
       Domain deployment requires its declared Debian proxy server and a
-      declared upstream endpoint
+      declared upstream endpoint or routes
 
 - name: Install Nginx and Certbot
   ansible.builtin.apt:
@@ -20,6 +22,22 @@
     update_cache: true
     cache_valid_time: 3600
 
+- name: Collect the directories on the static routes' paths
+  ansible.builtin.set_fact:
+    cloudfall_nginx_site_traverse_directories: >-
+      {{
+        cloudfall_nginx_site_domain.routes | default([])
+        | selectattr('static', 'defined')
+        | map(attribute='static.traverseDirectories')
+        | flatten(levels=1)
+        | unique
+        | list
+      }}
+
+- name: Let the Nginx worker reach the static routes' releases
+  ansible.builtin.include_tasks: static_access.yml
+  when: cloudfall_nginx_site_traverse_directories | length > 0
+
 # The TLS server block spells HTTP/2 the way the installed Nginx parses it:
 # `http2 on;` exists only from 1.25.1, and Debian 12 ships 1.22.1. Nginx
 # prints its version on stderr. The probe also runs in check mode, since the

engine/ansible/roles/cloudfall_nginx_site/tasks/static_access.yml

@@ -0,0 +1,50 @@
+---
+# Bootstrap creates the application root, the install root and `releases/`
+# 0750 for the application user, so the Nginx worker cannot reach a static
+# route's release. It gets traverse-only access (`x`, never `r`) along that
+# path by a POSIX ACL entry for its own user; it never joins the
+# application's group, which can read `shared/env/<component>.env`. The
+# release directories below `releases/` keep their archived modes, and
+# `current` is followed on every request, so a deploy or a rollback needs no
+# second grant.
+- name: Install the POSIX ACL tools
+  ansible.builtin.apt:
+    name: acl
+    state: present
+    update_cache: true
+    cache_valid_time: 3600
+
+- name: Read the ACL of each directory on a static route's path
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/getfacl
+      - --absolute-names
+      - --omit-header
+      - "{{ item }}"
+  loop: "{{ cloudfall_nginx_site_traverse_directories }}"
+  register: cloudfall_nginx_site_traverse_acls
+  changed_when: false
+  # In check mode the ACL tools may not be installed yet.
+  failed_when:
+    - cloudfall_nginx_site_traverse_acls.rc != 0
+    - not ansible_check_mode
+  check_mode: false
+
+- name: Let the Nginx worker traverse each directory on a static route's path
+  ansible.builtin.command:
+    argv:
+      - /usr/bin/setfacl
+      - --modify
+      - "user:{{ cloudfall_nginx_site_worker_user }}:--x"
+      - "{{ item }}"
+  loop: >-
+    {{
+      cloudfall_nginx_site_traverse_acls.results
+      | rejectattr(
+          'stdout', 'search',
+          '(?m)^user:' ~ cloudfall_nginx_site_worker_user ~ ':..x'
+        )
+      | map(attribute='item')
+      | list
+    }}
+  changed_when: true

engine/ansible/roles/cloudfall_nginx_site/templates/domain-site.conf.j2

@@ -3,13 +3,12 @@
     [cloudfall_nginx_site_domain.primaryName]
     + cloudfall_nginx_site_domain.aliases
   ) | join(' ') %}
-{% set upstream_address = cloudfall_nginx_site_domain.proxy.upstream.address %}
-{% set upstream_host = '[' ~ upstream_address ~ ']'
-    if ':' in upstream_address else upstream_address %}
-{% macro proxy_location() %}
-    location / {
+{% macro proxy_location(path, upstream) %}
+{% set upstream_host = '[' ~ upstream.address ~ ']'
+    if ':' in upstream.address else upstream.address %}
+    location {{ path }} {
         proxy_http_version 1.1;
-        proxy_pass http://{{ upstream_host }}:{{ cloudfall_nginx_site_domain.proxy.upstream.port }};
+        proxy_pass http://{{ upstream_host }}:{{ upstream.port }};
         proxy_pass_request_headers on;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
@@ -19,6 +18,58 @@
         proxy_set_header X-Forwarded-Port $server_port;
     }
 {% endmacro %}
+{#
+  A static route serves the component's current release, so a deploy or a
+  rollback that moves `current` changes what it serves without a reload. Its
+  path and the alias both end in `/`, which keeps the alias from reaching
+  outside the directory.
+#}
+{% macro static_location(path, static) %}
+    location {{ path }} {
+        alias {{ static.servedDirectory }}/;
+{% if static.spaFallback is defined %}
+        try_files $uri {{ path }}{{ static.spaFallback }};
+{% endif %}
+    }
+{% endmacro %}
+{#
+  Routes render longest path first, then by path, so the vhost does not
+  depend on the declared order; Nginx matches the longest prefix either way.
+  A Domain with only proxy.upstream renders its one `location /` as before.
+#}
+{% macro locations() %}
+{% if cloudfall_nginx_site_domain.routes is defined %}
+{% set routes = cloudfall_nginx_site_domain.routes | sort(attribute='path') %}
+{% set ordered = namespace(routes=[]) %}
+{% for length in routes | map(attribute='path') | map('length') | unique | sort(reverse=true) %}
+{% for route in routes if route.path | length == length %}
+{% set ordered.routes = ordered.routes + [route] %}
+{% endfor %}
+{% endfor %}
+{% for route in ordered.routes %}
+{% if not loop.first %}
+
+{% endif %}
+{% if route.static is defined %}
+{{ static_location(route.path, route.static) -}}
+{% else %}
+{{ proxy_location(route.path, route.upstream) -}}
+{% endif %}
+{% endfor %}
+{#
+  Without a `/` route, a path no route matches would fall to Nginx's default
+  root (Debian's welcome page, answering 200); it answers 404 instead.
+#}
+{% if routes | selectattr('path', 'equalto', '/') | list | length == 0 %}
+
+    location / {
+        return 404;
+    }
+{% endif %}
+{% else %}
+{{ proxy_location('/', cloudfall_nginx_site_domain.proxy.upstream) -}}
+{% endif %}
+{% endmacro %}
 server {
     listen 80;
     listen [::]:80;
@@ -33,7 +84,7 @@ server {
         return 301 https://$host$request_uri;
     }
 {% else %}
-{{ proxy_location() -}}
+{{ locations() -}}
 {% endif %}
 }
 {% if cloudfall_nginx_site_tls_active %}
@@ -54,6 +105,6 @@ server {
     ssl_protocols TLSv1.2 TLSv1.3;
     ssl_session_tickets off;
 
-{{ proxy_location() -}}
+{{ locations() -}}
 }
 {% endif %}