diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b45f3a3..aaa62a6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -569,9 +569,25 @@ jobs: test -s before/sha256 sudo sha256sum -c before/sha256 - # 3. modes and ownership in the config tree unchanged + # 3. modes and ownership in the config tree unchanged, with one expected addition + # (since 7.3.8, ADR-016): update-all/reconcile create an EMPTY noauth.domains next + # to every instance's blocklists//blocked.domains, with that file's mode and + # owner. It is expected only where the previous release had none; any other + # difference (also next to the admin's own /etc/linuxmuster-squid/blocked.domains) + # still fails. sudo find /etc/linuxmuster-squid -printf '%m %U:%G %p\n' | sort > after/etc-modes - diff -u before/etc-modes after/etc-modes + cp before/etc-modes before/etc-modes.expected + while read -r mode owner path; do + noauth_file="${path%/blocked.domains}/noauth.domains" + if awk -v p="$noauth_file" '$3 == p { found = 1 } END { exit !found }' before/etc-modes; then + continue + fi + echo "$mode $owner $noauth_file" >> before/etc-modes.expected + sudo test -f "$noauth_file" || { echo "missing: $noauth_file"; exit 1; } + if sudo test -s "$noauth_file"; then echo "not empty: $noauth_file"; exit 1; fi + done < <(awk '$3 ~ /^\/etc\/linuxmuster-squid\/blocklists\/[^\/]+\/blocked\.domains$/' before/etc-modes) + sort -o before/etc-modes.expected before/etc-modes.expected + diff -u before/etc-modes.expected after/etc-modes # 4. the instance record is written by the software, not the admin, and the # postinst rewrites `image:` on purpose (update-all). Every key the admin diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 295dd10..be05e2f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -480,9 +480,25 @@ jobs: test -s before/sha256 sudo sha256sum -c before/sha256 - # 3. modes and ownership in the config tree unchanged + # 3. modes and ownership in the config tree unchanged, with one expected addition + # (since 7.3.8, ADR-016): update-all/reconcile create an EMPTY noauth.domains next + # to every instance's blocklists//blocked.domains, with that file's mode and + # owner. It is expected only where the previous release had none; any other + # difference (also next to the admin's own /etc/linuxmuster-squid/blocked.domains) + # still fails. sudo find /etc/linuxmuster-squid -printf '%m %U:%G %p\n' | sort > after/etc-modes - diff -u before/etc-modes after/etc-modes + cp before/etc-modes before/etc-modes.expected + while read -r mode owner path; do + noauth_file="${path%/blocked.domains}/noauth.domains" + if awk -v p="$noauth_file" '$3 == p { found = 1 } END { exit !found }' before/etc-modes; then + continue + fi + echo "$mode $owner $noauth_file" >> before/etc-modes.expected + sudo test -f "$noauth_file" || { echo "missing: $noauth_file"; exit 1; } + if sudo test -s "$noauth_file"; then echo "not empty: $noauth_file"; exit 1; fi + done < <(awk '$3 ~ /^\/etc\/linuxmuster-squid\/blocklists\/[^\/]+\/blocked\.domains$/' before/etc-modes) + sort -o before/etc-modes.expected before/etc-modes.expected + diff -u before/etc-modes.expected after/etc-modes # 4. the instance record is written by the software, not the admin, and the # postinst rewrites `image:` on purpose (update-all). Every key the admin diff --git a/README.md b/README.md index 2fd3d3e..89c64f9 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,11 @@ the same AD/Kerberos infrastructure the school already operates. authentication via `negotiate_kerberos_auth`, authorization via `ext_kerberos_ldap_group_acl` against the instance's AD group. HTTPS is **not decrypted** (SNI peek/splice or CONNECT `dstdomain`). Per-instance domain - blocklist: HTTP → 403, HTTPS → the TLS handshake is cut (no block page without decryption). + blocklist: HTTP → 403, HTTPS → certificate warning in the browser (squid's 403 inside TLS + with an untrusted throwaway certificate; the blocked server is never contacted, allowed traffic + is never decrypted). + Optional per-instance list of domains reachable without proxy login from the school subnets + (e.g. Microsoft 365; the blocklist still wins). - **Control Plane:** hardened systemd service with **REST API** (FastAPI) and **CLI** (Typer, thin client) — creates instances, configures policies, and **updates digest-pinned with health-check auto-rollback**. Shipped as a @@ -76,8 +80,9 @@ sudo lmnsquid create --school all --role students --ad-group role-student \ --realm EXAMPLE.ORG --visible-hostname proxy.example.org \ --keytab-secret proxy.keytab --http-port 3129 --school-subnets 10.0.0.0/8 ``` -> `REALM=EXAMPLE.ORG bash /usr/share/linuxmuster-squid/scripts/discover-ad-facts.sh` (on the DC) -> prints the global role groups, the internet groups per school and the two `create` commands. +> `bash /usr/share/linuxmuster-squid/scripts/discover-ad-facts.sh` (on the DC) reads the realm +> there and prints the global role groups, the internet groups per school and the `create` +> commands. > `--image` defaults to the maintained, digest-pinned image. **3. Everyday commands:** @@ -90,9 +95,11 @@ lmnsquid update all-teachers # -> maintained default image (health lmnsquid update-all # every instance -> default image lmnsquid rollback all-teachers # to the last known-good lmnsquid reconcile # re-apply stored state (restore / after a crash) -lmnsquid blocklist all-students add example.org # block a domain + subdomains (HTTP 403, HTTPS TLS abort) +lmnsquid blocklist all-students add example.org # block a domain + subdomains (HTTP 403, HTTPS cert warning) lmnsquid blocklist all-students reload # apply without restart; list / remove likewise -lmnsquid rm all-students # container + cache/log volumes + blocklist (--keep-logs keeps the logs) +lmnsquid noauth all-students add sharepoint.com # reachable WITHOUT proxy login from the school subnets +lmnsquid noauth all-students reload # (exam mode and Internetsperre do not apply: docs/operations.md) +lmnsquid rm all-students # container + cache/log volumes + both lists (--keep-logs keeps the logs) lmnsquid version ``` diff --git a/RELEASE.md b/RELEASE.md index 1e143cb..d444a72 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -64,6 +64,21 @@ does not yet exist — every `lmnsquid create --image ghcr.io/…@sha256: reload` answers 409 and the lists still +reach squid unfiltered. The E2E service `squid-old` and the integration test +`test_real_old_image_ignores_the_noauth_list` deliberately keep testing the image 7.3.7 pins +(`…@sha256:d13625b5…`, a constant in `scripts/tests/e2e_kerberos.sh` and +`controlplane/tests/test_docker_integration.py`), i.e. what instances run until the upgrade +replaces their containers; they do not follow `DEFAULT_IMAGE` and need no change for the pin. + Code + docs + a `debian/changelog` bullet in the same commit (Conventional Commits) → push → CI green (`ci.yml`: fast tier, container build, install smoke, upgrade smoke) → Kevin sets the release version in the changelog head and tags `v` → `release.yml` builds, smokes and diff --git a/controlplane/lmnsquid/api.py b/controlplane/lmnsquid/api.py index b910e51..19b23a2 100644 --- a/controlplane/lmnsquid/api.py +++ b/controlplane/lmnsquid/api.py @@ -13,10 +13,17 @@ from fastapi import Depends, FastAPI, HTTPException, Request, status from fastapi.responses import JSONResponse -from .blocklist import Blocklist +from .blocklist import Blocklist, noauth_list from .config import Settings from .docker_service import DockerService -from .models import DEFAULT_IMAGE, BlocklistEntry, Instance, InstancePatch, UpdateRequest +from .models import ( + DEFAULT_IMAGE, + BlocklistEntry, + Instance, + InstancePatch, + NoauthEntry, + UpdateRequest, +) from .reconciler import Reconciler from .security import make_verify_token from .store import Store @@ -38,6 +45,7 @@ def create_app( """Build the FastAPI app wiring routes to the store, reconciler and docker service.""" verify = make_verify_token(settings) blocklist = Blocklist(settings.blocklists_dir) + noauth = noauth_list(settings.blocklists_dir) app = FastAPI(title="linuxmuster-squid control plane") @app.exception_handler(DockerException) @@ -218,6 +226,68 @@ def reload_blocklist(name: str) -> dict[str, Any]: status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(exc) ) from exc + # ------------------------------------------- domains without proxy auth (ADR-016) + @app.get("/v1/instances/{name}/noauth", dependencies=auth) + async def get_noauth(name: str) -> dict[str, Any]: + _require(name) + return {"name": name, "domains": noauth.read(name)} + + @app.post("/v1/instances/{name}/noauth", dependencies=auth) + async def add_noauth_domain(name: str, entry: NoauthEntry) -> dict[str, Any]: + """Let ``entry.domain`` and its subdomains through without proxy auth (after ``reload``).""" + _require(name) + domains = noauth.add(name, entry.domain) + audit.info("noauth add name=%s domain=%s", name, entry.domain) + return {"name": name, "domains": domains} + + @app.delete("/v1/instances/{name}/noauth/{domain}", dependencies=auth) + async def remove_noauth_domain(name: str, domain: str) -> dict[str, Any]: + _require(name) + try: + domains = noauth.remove(name, domain) + except ValueError as exc: + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(exc) + ) from exc + except KeyError as exc: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"{domain!r} is not on the noauth list of {name!r}", + ) from exc + audit.info("noauth remove name=%s domain=%s", name, domain) + return {"name": name, "domains": domains} + + @app.post("/v1/instances/{name}/noauth/reload", dependencies=auth) + def reload_noauth(name: str) -> dict[str, Any]: + """Re-read the lists (lists-sync.sh, ``squid -k reconfigure``), refused if the image + has no noauth list. + + An image from before ADR-016 has no noauth ACL in its squid.conf and never reads the + list: answer 409 instead of a silent "reloaded". (A ``noauth.domains`` that appeared + after the container started is read by this reload.) + """ + _require(name) + audit.info("noauth reload name=%s", name) + try: + if not docker.noauth_active(name): + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=( + f"the data-plane image of {name!r} does not support the noauth list, " + "so the list has no effect: it needs a linuxmuster-squid release whose " + "pinned data-plane image supports it; installing that release moves the " + "instances onto that image (update-all), and `lmnsquid update " + f"{name}` moves this one if it stayed on its old image" + ), + ) + return docker.reload(name) + except LookupError as exc: + raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(exc)) from exc + except RuntimeError as exc: + raise HTTPException( + status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(exc) + ) from exc + # ----------------------------------------------------------- lifecycle ops @app.post("/v1/instances/{name}/start", dependencies=auth) async def start_instance(name: str) -> dict[str, Any]: diff --git a/controlplane/lmnsquid/blocklist.py b/controlplane/lmnsquid/blocklist.py index 6d4c081..74e9da6 100644 --- a/controlplane/lmnsquid/blocklist.py +++ b/controlplane/lmnsquid/blocklist.py @@ -1,13 +1,15 @@ # SPDX-FileCopyrightText: Kevin Stenzel # SPDX-License-Identifier: GPL-3.0-or-later -"""Per-instance domain blocklist: a file on the host, mounted read-only into the container. - -Every instance owns ``//blocked.domains``. The DIRECTORY is -bind-mounted read-only at ``/etc/squid/lists`` -- the path the rendered ``squid.conf`` -already reads its ``dstdomain`` / ``ssl::server_name`` ACL from -- so the image needs no -change and the read-only rootfs stays read-only. The directory rather than the file is -mounted on purpose: a single-file bind mount pins the inode, so an atomic replace on the -host (an editor, ``blocklist-refresh.sh``, this module) would never reach the container. +"""Per-instance domain lists: files on the host, mounted read-only into the container. + +Every instance owns the directory ``//`` with two lists: +``blocked.domains`` (the blocklist) and ``noauth.domains`` (domains reachable without proxy +auth, ADR-016). The DIRECTORY is bind-mounted read-only at ``/etc/squid/lists`` -- the path +the rendered ``squid.conf`` reads its ``dstdomain`` / ``ssl::server_name`` ACLs from -- so the +read-only rootfs stays read-only, and the second list needs no mount of its own (the container +spec, and with it every running container, stays as it was). The directory rather than the +file is mounted on purpose: a single-file bind mount pins the inode, so an atomic replace on +the host (an editor, ``blocklist-refresh.sh``, this module) would never reach the container. """ from __future__ import annotations @@ -16,9 +18,11 @@ import re import shutil import tempfile +from collections.abc import Callable, Iterable from pathlib import Path FILE_NAME = "blocked.domains" +NOAUTH_FILE_NAME = "noauth.domains" CONTAINER_DIR = "/etc/squid/lists" # squid.conf: acl blocked_domains dstdomain "/etc/squid/lists/blocked.domains" # Instance name = - (models._NAME_RE per part): the directory name under root. _INSTANCE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9-]{1,62}$") @@ -26,6 +30,7 @@ # One DNS name (labels of letters/digits/hyphens), after IDNA encoding and lower-casing. _LABEL = r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?" _DOMAIN_RE = re.compile(rf"^{_LABEL}(?:\.{_LABEL})*$") +_INET_ATON_PART = re.compile(r"^(?:0x[0-9a-f]*|[0-9]+)$") def normalize_domain(value: str) -> str: @@ -47,11 +52,70 @@ def normalize_domain(value: str) -> str: return f".{name}" +def normalize_noauth_domain(value: str) -> str: + """:func:`normalize_domain`, but only a domain below a TLD: at least two labels, the last + one not numeric. + + An entry opens access without authentication, so it must name a domain: an IP literal + (``10.0.0.1``, ``127.1``, ``0x7f000001``, ``2887516181``) would put an address on the list + (squid matches IP-literal URLs against the list text) and a single label (``com``) a whole + top-level domain. The blocklist accepts both on purpose: a wrong entry there only blocks. + """ + entry = normalize_domain(value) + labels = entry[1:].split(".") + if len(labels) < 2: + raise ValueError(f"{value!r} is not a domain below a top-level domain") + # Every label an inet_aton number (decimal, octal, 0x-hex): an IPv4 address in some + # notation (1.2.3.0x4, 0x7f.0x1). Same rule as image/lists-sync.sh. + if labels[-1].isdigit() or all(_INET_ATON_PART.match(label) for label in labels): + raise ValueError(f"{value!r} looks like an IP address, not a domain") + return entry + + +def uncovered(entries: Iterable[str]) -> list[str]: + """Return ``entries`` without the ones another entry already covers, sorted. + + squid 6 refuses to load a ``dstdomain``/``ssl::server_name`` list (``FATAL: Bungled``) + when a name covered by a listed parent domain comes first (``.a.example.org`` before + ``.example.org``, or ``example.org`` next to ``.example.org``), so a list is never + written that way. ``.example.org`` covers itself, ``example.org`` and every + ``*.example.org``; entries are compared lower-cased without trailing dots. The image's + ``lists-sync.sh`` applies the same rule to whatever the mounted file holds. + """ + unique: list[str] = [] + for raw in entries: + entry = raw.strip().lower().rstrip(".") + if entry and entry != "." and entry not in unique: + unique.append(entry) + dotted = {e for e in unique if e.startswith(".")} + kept: list[str] = [] + for entry in unique: + rest = entry[1:] if entry.startswith(".") else entry + covered = not entry.startswith(".") and f".{entry}" in dotted + while not covered and "." in rest: + rest = rest.split(".", 1)[1] + covered = f".{rest}" in dotted + if not covered: + kept.append(entry) + return sorted(kept) + + class Blocklist: - """Read and modify the ``blocked.domains`` files under ``root`` (one dir per instance).""" + """Read and modify one list file (``blocked.domains`` by default) under ``root``. + + One directory per instance; ``noauth_list(root)`` handles the noauth list in the same + directories. + """ - def __init__(self, root: str) -> None: + def __init__( + self, + root: str, + file_name: str = FILE_NAME, + normalize: Callable[[str], str] = normalize_domain, + ) -> None: self.root = Path(root) + self.file_name = file_name + self.normalize = normalize def dir_for(self, name: str) -> Path: """Host directory mounted into the instance ``name`` (rejects anything but a name).""" @@ -60,7 +124,7 @@ def dir_for(self, name: str) -> Path: return self.root / name def file_for(self, name: str) -> Path: - return self.dir_for(name) / FILE_NAME + return self.dir_for(name) / self.file_name def ensure(self, name: str) -> Path: """Create the directory and an empty list if absent; return the directory. @@ -70,7 +134,7 @@ def ensure(self, name: str) -> Path: """ directory = self.dir_for(name) directory.mkdir(mode=0o755, parents=True, exist_ok=True) - file = directory / FILE_NAME + file = directory / self.file_name if not file.exists(): self._write(file, []) return directory @@ -88,26 +152,31 @@ def read(self, name: str) -> list[str]: return entries def add(self, name: str, domain: str) -> list[str]: - """Add ``domain`` (idempotent); return the new list.""" - entry = normalize_domain(domain) + """Add ``domain`` (idempotent); return the new list. + + A name below a listed domain changes nothing (the parent already covers it); a + parent replaces the names below it (:func:`uncovered`). + """ + entry = self.normalize(domain) self.ensure(name) - entries = sorted(set(self.read(name)) | {entry}) + entries = uncovered([*self.read(name), entry]) self._write(self.file_for(name), entries) return entries def remove(self, name: str, domain: str) -> list[str]: """Remove ``domain`` (with or without the leading dot); ``KeyError`` if absent.""" entry = normalize_domain(domain) - current = self.read(name) + current = [e.lower().rstrip(".") for e in self.read(name)] # A hand-edited file may carry the bare form; both mean the same target. remaining = [e for e in current if e not in (entry, entry[1:])] if len(remaining) == len(current): raise KeyError(entry) + remaining = uncovered(remaining) self._write(self.file_for(name), remaining) return remaining def delete(self, name: str) -> None: - """Remove the instance's directory (called when the instance is removed).""" + """Remove the instance's directory with every list in it (instance removed).""" directory = self.dir_for(name) if directory.is_dir(): shutil.rmtree(directory) @@ -117,7 +186,7 @@ def _write(file: Path, entries: list[str]) -> None: # Atomic replace inside the directory: squid (on reload) or an editor never sees a # half-written list, and a root-owned file left behind by blocklist-refresh.sh # does not block the control plane (only the directory must be writable). - fd, tmp = tempfile.mkstemp(dir=file.parent, prefix=f".{FILE_NAME}.", suffix=".tmp") + fd, tmp = tempfile.mkstemp(dir=file.parent, prefix=f".{file.name}.", suffix=".tmp") try: with os.fdopen(fd, "w", encoding="utf-8") as fh: fh.write("".join(f"{entry}\n" for entry in entries)) @@ -127,3 +196,8 @@ def _write(file: Path, entries: list[str]) -> None: if os.path.exists(tmp): os.unlink(tmp) raise + + +def noauth_list(root: str) -> Blocklist: + """The noauth list (ADR-016) of the instances under ``root``, with its stricter names.""" + return Blocklist(root, NOAUTH_FILE_NAME, normalize_noauth_domain) diff --git a/controlplane/lmnsquid/cli.py b/controlplane/lmnsquid/cli.py index cd16405..d8308d7 100644 --- a/controlplane/lmnsquid/cli.py +++ b/controlplane/lmnsquid/cli.py @@ -122,7 +122,7 @@ def rm( "default removes it together with the cache volume", ), ) -> None: - """Remove an instance: container, cache + log volumes, blocklist and definition.""" + """Remove an instance: container, cache + log volumes, both lists and definition.""" with _get_client() as c: _emit( c.delete(f"/v1/instances/{name}", params={"keep_logs": "true"} if keep_logs else None) @@ -130,7 +130,8 @@ def rm( blocklist_app = typer.Typer( - help="Per-instance domain blocklist (HTTP -> 403; HTTPS is cut at the TLS handshake). " + help="Per-instance domain blocklist (HTTP -> 403; HTTPS -> certificate warning in the " + "browser, the blocked server is never contacted). " "Changes take effect after `reload`.", no_args_is_help=True, ) @@ -171,6 +172,52 @@ def blocklist_reload(ctx: typer.Context) -> None: _emit(c.post(f"/v1/instances/{ctx.obj}/blocklist/reload")) +noauth_app = typer.Typer( + help="Per-instance domains reachable WITHOUT proxy authentication, from --school-subnets " + "only (e.g. Microsoft 365). The blocklist still wins; HTTPS only when the TLS SNI is on " + "the list too. WARNING: anyone in the school subnets reaches these domains without a " + "ticket, so exam accounts reach them in exam mode (OneDrive, Teams, Outlook ...) and the " + "Internetsperre does not apply to them. Add the organisation's own domains only, never a " + "public suffix such as co.uk or github.io. Changes take effect after `reload`.", + no_args_is_help=True, +) +app.add_typer(noauth_app, name="noauth") + + +@noauth_app.callback() +def _noauth(ctx: typer.Context, name: str = typer.Argument(..., help="instance name")) -> None: + """Manage the noauth list of one instance: list | add | remove | reload.""" + ctx.obj = name + + +@noauth_app.command("list") +def noauth_list(ctx: typer.Context) -> None: + """Show the domains reachable without proxy authentication.""" + with _get_client() as c: + _emit(c.get(f"/v1/instances/{ctx.obj}/noauth")) + + +@noauth_app.command("add") +def noauth_add(ctx: typer.Context, domain: str) -> None: + """Let a domain and all its subdomains through without authentication (then `reload`).""" + with _get_client() as c: + _emit(c.post(f"/v1/instances/{ctx.obj}/noauth", json={"domain": domain})) + + +@noauth_app.command("remove") +def noauth_remove(ctx: typer.Context, domain: str) -> None: + """Require authentication for a domain again (then `reload`).""" + with _get_client() as c: + _emit(c.delete(f"/v1/instances/{ctx.obj}/noauth/{quote(domain, safe='')}")) + + +@noauth_app.command("reload") +def noauth_reload(ctx: typer.Context) -> None: + """Apply the list without a restart; fails if the instance's image ignores the list.""" + with _get_client() as c: + _emit(c.post(f"/v1/instances/{ctx.obj}/noauth/reload")) + + @app.command() def start(name: str) -> None: """Start the instance container.""" diff --git a/controlplane/lmnsquid/docker_service.py b/controlplane/lmnsquid/docker_service.py index f934d15..86cc0cc 100644 --- a/controlplane/lmnsquid/docker_service.py +++ b/controlplane/lmnsquid/docker_service.py @@ -16,7 +16,7 @@ from docker.models.containers import Container from docker.types import LogConfig -from .blocklist import CONTAINER_DIR, Blocklist +from .blocklist import CONTAINER_DIR, Blocklist, noauth_list from .models import Instance logger = logging.getLogger("lmnsquid.docker") @@ -24,6 +24,20 @@ # The config entrypoint.sh renders and starts squid with (`squid -k` must read the same # file to find pid_filename). _SQUID_CONF = "/run/lmnsquid/squid.conf" +# Reload: images with lists-sync.sh read the lists from /run/lmnsquid/lists, which the script +# rebuilds from the mounted ones (without the entries squid refuses to load); older images read +# the mounted files directly. Its messages (ignored entries) are printed for the API response +# and appended to /run/lmnsquid/lists-sync.log, which the entrypoint mirrors into the container +# log (`lmnsquid logs`). Fixed text, nothing from a request goes into it. +_RELOAD = ( + "set -e; if [ -x /usr/local/bin/lists-sync.sh ]; then " + "/usr/local/bin/lists-sync.sh 2>/run/lmnsquid/lists-sync.last " + "|| { cat /run/lmnsquid/lists-sync.last; exit 1; }; " + "cat /run/lmnsquid/lists-sync.last; " + "cat /run/lmnsquid/lists-sync.last >> /run/lmnsquid/lists-sync.log; fi; " + f"exec squid -k reconfigure -f {_SQUID_CONF}" +) +_NOTICE_PREFIX = "linuxmuster-squid: " # Container label carrying the fingerprint of everything the container was created # from; a container whose label matches the desired spec is left alone by # ensure_running (reconcile is a no-op for it), one without or with another @@ -55,6 +69,7 @@ def __init__( self.log_max_size: str = log_max_size self.log_max_file: int = log_max_file self.blocklist = Blocklist(blocklists_dir) + self.noauth = noauth_list(blocklists_dir) self.health_timeout = health_timeout self.poll_interval = poll_interval self.client: docker.DockerClient = ( @@ -183,9 +198,12 @@ def ensure_running(self, inst: Instance) -> dict[str, Any]: spec = self.spec_for(inst) digest = self.fingerprint(spec) - # Blocklist: created empty if absent (create, reconcile, update all pass through + # Lists: created empty if absent (create, reconcile, update all pass through # here, so instances from before 7.3.1 get the mount on their next replacement). + # noauth.domains too, so the file an admin edits is there from the start (a missing + # one is an empty list to lists-sync.sh, which reads it again on every reload). self.blocklist.ensure(inst.name) + self.noauth.ensure(inst.name) cname = inst.container_name existing = self._get(inst.name) @@ -323,19 +341,8 @@ def remove(self, name: str, keep_logs: bool = False) -> None: logger.warning("volume %s not removed: %s", volume, exc) self.blocklist.delete(name) - def reload(self, name: str) -> dict[str, Any]: - """Make the running squid re-read squid.conf and its ACL list files (the blocklist). - - Runs ``squid -k reconfigure`` inside the container: squid parses the config and - signals its running copy (SIGHUP). No restart -- the cache and open client - connections survive, the auth/group helpers are restarted. Deliberately NOT - ``container.kill(signal="SIGHUP")``: Docker records every kill(), whatever the - signal, as a manual stop, and an ``unless-stopped`` container is then no longer - started after a reboot (seen in the lab). Like ``access_logs`` this needs - ``docker exec`` (not available behind the socket proxy with ``EXEC: 0``). - Raises ``LookupError`` when there is no running container, ``RuntimeError`` - when the exec is refused or squid does. - """ + def _running(self, name: str) -> Container: + """The running container of ``name``; ``LookupError`` when there is none.""" container = self._get(name) if container is None: raise LookupError(f"instance {name!r} has no container") @@ -343,22 +350,62 @@ def reload(self, name: str) -> dict[str, Any]: state: dict[str, Any] = container.attrs.get("State", {}) or {} if not state.get("Running", False): raise LookupError(f"instance {name!r} is not running") + return container + + def _exec(self, container: Container, cmd: list[str]) -> tuple[int, str]: try: - exit_code, output = container.exec_run( - ["squid", "-k", "reconfigure", "-f", _SQUID_CONF] - ) + exit_code, output = container.exec_run(cmd) except APIError as exc: # The daemon is alive; exec itself was refused (socket proxy with EXEC: 0). raise RuntimeError( - f"docker exec refused ({exc.explanation or exc}); blocklist reload needs " + f"docker exec refused ({exc.explanation or exc}); list reload needs " "EXEC at the docker socket proxy (ADR-012/ADR-014)" ) from exc text = ( output.decode("utf-8", errors="replace") if isinstance(output, bytes) else str(output) ) + return exit_code, text + + def noauth_active(self, name: str) -> bool: + """Whether the running squid has the noauth ACL (ADR-016) in its configuration. + + Not so for an image from before ADR-016 (its template has no such ACL); a reload + cannot change that, only a container of a newer image can. Same preconditions and + errors as :meth:`reload`. + """ + container = self._running(name) + exit_code, text = self._exec(container, ["grep", "-qF", "acl noauth_domains ", _SQUID_CONF]) + if exit_code not in (0, 1): + raise RuntimeError(f"cannot read {_SQUID_CONF} (exit {exit_code}): {text.strip()}") + return exit_code == 0 + + def reload(self, name: str) -> dict[str, Any]: + """Make the running squid re-read squid.conf and its ACL list files (both lists). + + Runs ``lists-sync.sh`` (images that have it: rebuilds the lists squid reads, without + entries squid would refuse) and then ``squid -k reconfigure`` inside the container. If + squid still refuses the configuration, the running squid keeps the previous one and a + ``RuntimeError`` carries squid's message. squid -k reconfigure parses the config and + signals its running copy (SIGHUP). No restart -- the cache and open client + connections survive, the auth/group helpers are restarted. Deliberately NOT + ``container.kill(signal="SIGHUP")``: Docker records every kill(), whatever the + signal, as a manual stop, and an ``unless-stopped`` container is then no longer + started after a reboot (seen in the lab). Like ``access_logs`` this needs + ``docker exec`` (not available behind the socket proxy with ``EXEC: 0``). + Raises ``LookupError`` when there is no running container, ``RuntimeError`` + when the exec is refused or squid does. ``notices`` are the messages of + ``lists-sync.sh`` (entries it ignored), also written to the container log. + """ + container = self._running(name) + exit_code, text = self._exec(container, ["sh", "-c", _RELOAD]) if exit_code != 0: - raise RuntimeError(f"squid -k reconfigure exited {exit_code}: {text.strip()}") - return {"name": name, "reloaded": True} + raise RuntimeError(f"list reload exited {exit_code}: {text.strip()}") + notices = [ + line[len(_NOTICE_PREFIX) :] + for line in text.splitlines() + if line.startswith(_NOTICE_PREFIX) + ] + return {"name": name, "reloaded": True, "notices": notices} # -- introspection ----------------------------------------------------- diff --git a/controlplane/lmnsquid/models.py b/controlplane/lmnsquid/models.py index c0ea4c5..9bccc6a 100644 --- a/controlplane/lmnsquid/models.py +++ b/controlplane/lmnsquid/models.py @@ -15,7 +15,7 @@ from pydantic import BaseModel, computed_field, field_validator -from .blocklist import normalize_domain +from .blocklist import normalize_domain, normalize_noauth_domain # school/role -> name -> filename + container/volume name: no '/', '..' (case allowed). _NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9-]{0,30}$") @@ -212,3 +212,15 @@ class BlocklistEntry(BaseModel): def _v_domain(cls, v: str) -> str: # Normalized here so an invalid name is a 422 at the boundary, not a 500. return normalize_domain(v) + + +class NoauthEntry(BaseModel): + """Body for ``POST /v1/instances/{name}/noauth``: one domain below a top-level domain.""" + + domain: str + + @field_validator("domain") + @classmethod + def _v_domain(cls, v: str) -> str: + # No IP literals, no single labels (ADR-016): a 422 at the boundary. + return normalize_noauth_domain(v) diff --git a/controlplane/tests/conftest.py b/controlplane/tests/conftest.py index 0566f87..9c37da8 100644 --- a/controlplane/tests/conftest.py +++ b/controlplane/tests/conftest.py @@ -15,7 +15,7 @@ from starlette.testclient import TestClient from lmnsquid.api import create_app -from lmnsquid.blocklist import Blocklist +from lmnsquid.blocklist import Blocklist, noauth_list from lmnsquid.config import Settings from lmnsquid.models import Instance from lmnsquid.reconciler import Reconciler @@ -42,6 +42,7 @@ def __init__( self.docker_host = docker_host self.secrets_dir = secrets_dir self.blocklist = Blocklist(blocklists_dir) + self.noauth = noauth_list(blocklists_dir) self.containers: dict[str, dict[str, Any]] = {} self.volumes: set[str] = set() # Test-observability hooks. @@ -70,14 +71,17 @@ def ensure_running(self, inst: Instance) -> dict[str, Any]: # Mirror the real service: the replacement is created first, so a failure # leaves the previous container exactly as it was and raises. raise RuntimeError("simulated pull failure") - # Mirror the real service: the blocklist is created (empty) and mounted, and the + # Mirror the real service: both lists are created (empty) and mounted, and the # cache/log volumes come into existence with the container. self.blocklist.ensure(inst.name) + self.noauth.ensure(inst.name) self.volumes.update({f"lmnsquid-cache-{inst.name}", f"lmnsquid-logs-{inst.name}"}) self.containers[inst.name] = { "running": True, "image": inst.image, "health": "unhealthy" if "bad" in inst.image else "healthy", + # An image tagged "old" stands for one from before ADR-016 (ignores noauth). + "noauth_active": "old" not in inst.image, "env": self.env_for(inst), "logs": f"started {inst.container_name}\n", "access_logs": ( @@ -113,12 +117,18 @@ def remove(self, name: str, keep_logs: bool = False) -> None: self.volumes.discard(f"lmnsquid-logs-{name}") self.blocklist.delete(name) + def noauth_active(self, name: str) -> bool: + container = self.containers.get(name) + if container is None or not container["running"]: + raise LookupError(f"instance {name!r} is not running") + return bool(container["noauth_active"]) + def reload(self, name: str) -> dict[str, Any]: container = self.containers.get(name) if container is None or not container["running"]: raise LookupError(f"instance {name!r} is not running") self.reloaded.append(name) - return {"name": name, "reloaded": True} + return {"name": name, "reloaded": True, "notices": []} def status(self, name: str) -> dict[str, Any]: container = self.containers.get(name) diff --git a/controlplane/tests/test_blocklist.py b/controlplane/tests/test_blocklist.py index 5148330..1cedcc2 100644 --- a/controlplane/tests/test_blocklist.py +++ b/controlplane/tests/test_blocklist.py @@ -12,7 +12,7 @@ from typer.testing import CliRunner from lmnsquid import cli -from lmnsquid.blocklist import Blocklist, normalize_domain +from lmnsquid.blocklist import Blocklist, normalize_domain, uncovered from lmnsquid.config import Settings runner = CliRunner() @@ -68,6 +68,41 @@ def test_blocklist_file_lifecycle(tmp_path: Path) -> None: assert bl.read("s-teachers") == [] # absent file reads as empty +@pytest.mark.parametrize( + ("entries", "expected"), + [ + # squid 6 refuses these orders ("FATAL: Bungled"): the covered name goes + ([".graph.microsoft.com", ".microsoft.com"], [".microsoft.com"]), + (["microsoft.com", ".microsoft.com"], [".microsoft.com"]), + ([".microsoft.com", "a.b.microsoft.com", "MICROSOFT.COM."], [".microsoft.com"]), + # duplicates (also by case and trailing dot) collapse + ([".a.org", ".A.org", ".a.org."], [".a.org"]), + # a bare name covers nothing, a sibling or a mere suffix is not covered + (["a.org", "x.a.org"], ["a.org", "x.a.org"]), + ([".a.org", ".ba.org", ".org.example"], [".a.org", ".ba.org", ".org.example"]), + (["", " ", ".", ".b.org"], [".b.org"]), + ], +) +def test_uncovered(entries: list[str], expected: list[str]) -> None: + assert uncovered(entries) == expected + + +def test_add_never_writes_a_list_squid_cannot_load(tmp_path: Path) -> None: + bl = Blocklist(str(tmp_path)) + file = tmp_path / "s-teachers" / "blocked.domains" + assert bl.add("s-teachers", "graph.microsoft.com") == [".graph.microsoft.com"] + # a parent replaces the names below it ... + assert bl.add("s-teachers", "microsoft.com") == [".microsoft.com"] + # ... and a name below a listed parent changes nothing + assert bl.add("s-teachers", "login.microsoft.com") == [".microsoft.com"] + assert file.read_text() == ".microsoft.com\n" + # a hand-edited file with a covered name first is cleaned on the next write + file.write_text(".a.example.org\n.example.org\nexample.org\n") + assert bl.add("s-teachers", "other.org") == [".example.org", ".other.org"] + file.write_text(".a.example.org\n.example.org\n.x.org\n") + assert bl.remove("s-teachers", "x.org") == [".example.org"] + + def test_blocklist_rejects_unsafe_name(tmp_path: Path) -> None: bl = Blocklist(str(tmp_path)) for bad in ("../x", "a/b", "", ".", "a", "a.b"): @@ -114,7 +149,11 @@ def test_blocklist_endpoints( # reload -> squid -k reconfigure run in the running container resp = client.post(f"/v1/instances/{name}/blocklist/reload", headers=auth_headers) - assert resp.status_code == 200 and resp.json() == {"name": name, "reloaded": True} + assert resp.status_code == 200 and resp.json() == { + "name": name, + "reloaded": True, + "notices": [], + } assert docker.reloaded == [name] # stopped container -> 409, not a silent "ok" diff --git a/controlplane/tests/test_docker_integration.py b/controlplane/tests/test_docker_integration.py index 7d128e4..083bb74 100644 --- a/controlplane/tests/test_docker_integration.py +++ b/controlplane/tests/test_docker_integration.py @@ -20,6 +20,13 @@ ) IMAGE = os.environ.get("LMNSQUID_IT_IMAGE", "linuxmuster-squid:dev") +# The data-plane image 7.3.7 pins, the last one without ADR-016: a constant, not DEFAULT_IMAGE, +# which moves to an image that reads the noauth list before 7.3.8 is tagged. +OLD_IMAGE = os.environ.get( + "LMNSQUID_IT_OLD_IMAGE", + "ghcr.io/faircomp/linuxmuster-squid" + "@sha256:d13625b51a99d62f14c37d18453092bbde651cb34df080dd49f3937f7f5f46ef", +) def test_real_container_lifecycle(tmp_path: Path) -> None: @@ -32,7 +39,8 @@ def test_real_container_lifecycle(tmp_path: Path) -> None: # exercised by the compose E2E, not here — this proves docker-py lifecycle. (secrets / "it.keytab").write_bytes(b"dummy-keytab-not-valid") - svc = DockerService(secrets_dir=str(secrets)) + lists = tmp_path / "lists" + svc = DockerService(secrets_dir=str(secrets), blocklists_dir=str(lists)) inst = Instance( school="it", role="teachers", @@ -56,6 +64,37 @@ def test_real_container_lifecycle(tmp_path: Path) -> None: time.sleep(5) assert svc.status(inst.name)["running"] is True + # ADR-016: noauth.domains exists before the start, so this image reads it, and a + # reload picks up an entry added afterwards. + assert (lists / inst.name / "noauth.domains").read_text() == "" + assert svc.noauth_active(inst.name) is True + svc.noauth.add(inst.name, "office.com") + assert svc.reload(inst.name) == {"name": inst.name, "reloaded": True, "notices": []} + + # Hand-edited lists that squid 6 refuses as such (a covered name before its parent, + # the bare apex): lists-sync.sh drops those entries on reload and at the next start. + (lists / inst.name / "blocked.domains").write_text( + ".a.example.org\nexample.org\n.example.org\n" + ) + (lists / inst.name / "noauth.domains").write_text( + ".graph.microsoft.com\n.microsoft.com\nmicrosoft.com\n" + ) + reloaded = svc.reload(inst.name) + assert reloaded["reloaded"] is True + # what was ignored is in the response and in the container log (`lmnsquid logs`) + assert any("already covered" in n for n in reloaded["notices"]), reloaded + time.sleep(2) + assert "blocked.domains: 2 entries ignored, already covered" in svc.logs(inst.name, tail=50) + svc.restart(inst.name) + deadline = time.monotonic() + 60 + health = None + while time.monotonic() < deadline: + health = svc.status(inst.name)["health"] + if health in ("healthy", "unhealthy"): + break + time.sleep(2) + assert svc.status(inst.name)["running"] is True and health == "healthy" + assert svc.stop(inst.name)["running"] is False assert svc.start(inst.name)["running"] is True finally: @@ -78,7 +117,7 @@ def test_real_update_auto_rollback(tmp_path: Path) -> None: (secrets / "it.keytab").write_bytes(b"dummy-keytab-not-valid") store = Store(str(tmp_path / "instances")) - docker = DockerService(secrets_dir=str(secrets)) + docker = DockerService(secrets_dir=str(secrets), blocklists_dir=str(tmp_path / "lists")) reconciler = Reconciler(store, docker) updater = Updater(store, docker, reconciler, health_timeout=45.0, poll_interval=2.0) @@ -106,3 +145,39 @@ def test_real_update_auto_rollback(tmp_path: Path) -> None: assert store.get(inst.name).image == IMAGE # type: ignore[union-attr] finally: docker.remove(inst.name) + + +def test_real_old_image_ignores_the_noauth_list(tmp_path: Path) -> None: + """An image from before ADR-016 (what installed instances run until the upgrade replaces + their containers) next to a list directory with noauth.domains: it starts healthy, and the + noauth reload is refused instead of reporting a reload that changes nothing.""" + from lmnsquid.docker_service import DockerService + from lmnsquid.models import Instance + + old_image = OLD_IMAGE + secrets = tmp_path / "secrets" + secrets.mkdir() + (secrets / "it.keytab").write_bytes(b"dummy-keytab-not-valid") + lists = tmp_path / "lists" + svc = DockerService(secrets_dir=str(secrets), blocklists_dir=str(lists)) + inst = Instance( + school="ito", + role="teachers", + ad_group="teachers", + realm="EXAMPLE.INTERNAL", + visible_hostname="squid-ito.example.internal", + http_port=3197, + school_subnets="0.0.0.0/0", + keytab_secret="it.keytab", + cache_size_mb=100, + image=old_image, + ) + svc.remove(inst.name) + svc.noauth.add(inst.name, "office.com") # a list that is there before the container + try: + status = svc.ensure_running(inst) # health-gated: raises if it does not come up + assert status["running"] is True and status["health"] == "healthy" + assert svc.noauth_active(inst.name) is False + assert svc.reload(inst.name)["reloaded"] is True # the blocklist reload still works + finally: + svc.remove(inst.name) diff --git a/controlplane/tests/test_docker_service.py b/controlplane/tests/test_docker_service.py index c1a29c1..e6b311f 100644 --- a/controlplane/tests/test_docker_service.py +++ b/controlplane/tests/test_docker_service.py @@ -12,7 +12,7 @@ import pytest from docker.errors import APIError, NotFound -from lmnsquid.blocklist import Blocklist +from lmnsquid.blocklist import Blocklist, noauth_list from lmnsquid.docker_service import SPEC_LABEL, DockerService from lmnsquid.models import Instance @@ -22,6 +22,7 @@ def _service(tmp_path: Path) -> DockerService: ds.client = MagicMock() ds.secrets_dir = str(tmp_path / "secrets") ds.blocklist = Blocklist(str(tmp_path / "blocklists")) + ds.noauth = noauth_list(str(tmp_path / "blocklists")) ds.container_bind_ip = "0.0.0.0" ds.log_max_size = "20m" ds.log_max_file = 5 @@ -71,6 +72,89 @@ def test_fingerprint_is_stable_and_sensitive(tmp_path: Path) -> None: spec = ds.spec_for(_inst()) blocklist_dir = str(tmp_path / "blocklists" / "s-teachers") assert spec["volumes"][blocklist_dir]["bind"] == "/etc/squid/lists" + # The noauth list lives in the same mounted directory: no mount of its own, so the + # spec (and with it every container an upgrade would otherwise replace) is unchanged. + assert len(spec["volumes"]) == 4 + assert not any("noauth" in str(k) or "noauth" in str(v) for k, v in spec.items()) + + +def test_ensure_running_creates_both_lists_before_the_container(tmp_path: Path) -> None: + ds = _service(tmp_path) + _registry(ds, {}) + seen: list[bool] = [] + lists = tmp_path / "blocklists" / "s-teachers" + + def create(*_a: Any, **_k: Any) -> MagicMock: + # The entrypoint reads noauth.domains only when it is there at container start. + seen.append((lists / "noauth.domains").is_file() and (lists / "blocked.domains").is_file()) + return _container("lmnsquid-s-teachers.new") + + ds.client.containers.create.side_effect = create + ds.ensure_running(_inst()) + assert seen == [True] + assert (lists / "noauth.domains").read_text() == "" + + +@pytest.mark.parametrize(("grep_exit", "expected"), [(0, True), (1, False)]) +def test_noauth_active_reads_the_rendered_config( + tmp_path: Path, grep_exit: int, expected: bool +) -> None: + ds = _service(tmp_path) + c = _container("lmnsquid-s-teachers") + c.exec_run.return_value = (grep_exit, b"") + _registry(ds, {"lmnsquid-s-teachers": c}) + assert ds.noauth_active("s-teachers") is expected + cmd = c.exec_run.call_args.args[0] + assert cmd == ["grep", "-qF", "acl noauth_domains ", "/run/lmnsquid/squid.conf"] + + +def test_reload_syncs_the_lists_then_reconfigures(tmp_path: Path) -> None: + ds = _service(tmp_path) + c = _container("lmnsquid-s-teachers") + c.exec_run.return_value = (0, b"") + _registry(ds, {"lmnsquid-s-teachers": c}) + c.exec_run.return_value = ( + 0, + b"linuxmuster-squid: noauth.domains: ignored, not a domain name: com\n" + b"squid: some other line\n", + ) + assert ds.reload("s-teachers") == { + "name": "s-teachers", + "reloaded": True, + "notices": ["noauth.domains: ignored, not a domain name: com"], + } + cmd = c.exec_run.call_args.args[0] + assert cmd[:2] == ["sh", "-c"] + # lists-sync.sh first (only where the image has it), then squid -k reconfigure. + assert "[ -x /usr/local/bin/lists-sync.sh ]" in cmd[2] + assert cmd[2].index("lists-sync.sh 2>") < cmd[2].index("squid -k reconfigure") + assert ">> /run/lmnsquid/lists-sync.log" in cmd[2] + assert cmd[2].endswith("exec squid -k reconfigure -f /run/lmnsquid/squid.conf") + + c.exec_run.return_value = (1, b"FATAL: Bungled /run/lmnsquid/squid.conf line 59") + with pytest.raises(RuntimeError, match="Bungled"): + ds.reload("s-teachers") + + +def test_noauth_active_errors(tmp_path: Path) -> None: + ds = _service(tmp_path) + stopped = _container("lmnsquid-s-teachers", running=False) + _registry(ds, {"lmnsquid-s-teachers": stopped}) + with pytest.raises(LookupError): + ds.noauth_active("s-teachers") + _registry(ds, {}) + with pytest.raises(LookupError): + ds.noauth_active("s-teachers") + broken = _container("lmnsquid-s-teachers") + broken.exec_run.return_value = (2, b"grep: /run/lmnsquid/squid.conf: No such file") + _registry(ds, {"lmnsquid-s-teachers": broken}) + with pytest.raises(RuntimeError, match="cannot read"): + ds.noauth_active("s-teachers") + refused = _container("lmnsquid-s-teachers") + refused.exec_run.side_effect = APIError("exec disabled") + _registry(ds, {"lmnsquid-s-teachers": refused}) + with pytest.raises(RuntimeError, match="EXEC"): + ds.noauth_active("s-teachers") def test_spec_rejects_a_keytab_outside_secrets_dir(tmp_path: Path) -> None: diff --git a/controlplane/tests/test_lists_sync.py b/controlplane/tests/test_lists_sync.py new file mode 100644 index 0000000..1278f97 --- /dev/null +++ b/controlplane/tests/test_lists_sync.py @@ -0,0 +1,112 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# SPDX-License-Identifier: GPL-3.0-or-later +"""image/lists-sync.sh as it is (sh + awk + sort), next to the Python rules it must match. + +The control plane refuses a noauth entry with :func:`normalize_noauth_domain`; the container +ignores a hand-written one with lists-sync.sh. Both must agree, so every case here goes +through both. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + +from lmnsquid.blocklist import normalize_noauth_domain, uncovered + +SCRIPT = Path(__file__).resolve().parents[2] / "image" / "lists-sync.sh" + +pytestmark = pytest.mark.skipif( + not all(shutil.which(tool) for tool in ("sh", "awk", "sort", "sed")), + reason="needs sh, awk, sort and sed", +) + + +def _sync(tmp_path: Path, blocked: bytes = b"", noauth: bytes | None = None) -> tuple[Path, str]: + src, dst = tmp_path / "src", tmp_path / "dst" + src.mkdir(exist_ok=True) + (src / "blocked.domains").write_bytes(blocked) + if noauth is not None: + (src / "noauth.domains").write_bytes(noauth) + env = {**os.environ, "LISTS_SYNC_SRC": str(src), "LISTS_SYNC_DST": str(dst)} + done = subprocess.run( + ["sh", str(SCRIPT)], env=env, capture_output=True, text=True, check=True, timeout=60 + ) + return dst, done.stderr + + +def _python_accepts(name: str) -> bool: + try: + normalize_noauth_domain(name) + except ValueError: + return False + return True + + +@pytest.mark.parametrize( + "name", + [ + # IPv4 in inet_aton notations: dotted, short, decimal, octal, hex, mixed + "172.28.0.20", + "127.1", + "2887516181", + "0177.0.0.1", + "0x7f000001", + "0x7f.0x1", + "1.2.3.0x4", + "0x0a.0x00.0x00.0x01", + "0xac.0x1c.0x0.0x15", + "172.28.0.0x15", + # single labels, IPv6, not a name + "com", + ".com", + "localhost", + "::1", + "fe80::1", + "*.example.org", + "x_y.example.org", + # names that must pass: hex-looking or numeric labels next to a real label + "office.com", + ".sharepoint.com", + "0xcafe.com", + "1und1.de", + "10.0.0.example", + "x.co.uk", + "xn--bcher-kva.de", + ], +) +def test_python_and_lists_sync_agree(tmp_path: Path, name: str) -> None: + dst, stderr = _sync(tmp_path, noauth=f"{name}\n".encode()) + kept = (dst / "noauth.domains").read_text().split() + assert bool(kept) is _python_accepts(name), (name, kept, stderr) + if not kept: + assert f"ignored, not a domain name: {name.lower()}" in stderr + + +def test_covered_entries_like_python(tmp_path: Path) -> None: + raw = [".a.example.org", "example.org", ".example.org", "B.Example.Org.", ".other.org"] + dst, stderr = _sync(tmp_path, blocked="\n".join(raw).encode() + b"\n") + assert sorted((dst / "blocked.domains").read_text().split()) == uncovered(raw) + assert "already covered by a listed parent domain" in stderr + + +def test_missing_noauth_file_is_an_empty_list(tmp_path: Path) -> None: + dst, stderr = _sync(tmp_path, blocked=b".example.org\n") + assert (dst / "noauth.domains").read_text() == "" + assert "domains without proxy auth: none" in stderr + + +def test_crlf_line_endings_in_both_lists(tmp_path: Path) -> None: + # A list saved by a Windows editor: every entry must survive, without the \r. + dst, stderr = _sync( + tmp_path, + blocked=b".blocked.example.org\r\n.a.blocked.example.org\r\n", + noauth=b"# Microsoft 365\r\n.office.com\r\n.sharepoint.com # comment\r\n", + ) + assert (dst / "blocked.domains").read_text() == ".blocked.example.org\n" + assert (dst / "noauth.domains").read_text().split() == [".office.com", ".sharepoint.com"] + assert "not a domain name" not in stderr diff --git a/controlplane/tests/test_noauth.py b/controlplane/tests/test_noauth.py new file mode 100644 index 0000000..a38a396 --- /dev/null +++ b/controlplane/tests/test_noauth.py @@ -0,0 +1,288 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# SPDX-License-Identifier: GPL-3.0-or-later +"""Domains without proxy auth (ADR-016): list file, API endpoints, CLI, and `rm`.""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +import pytest +from starlette.testclient import TestClient +from typer.testing import CliRunner + +from lmnsquid import cli +from lmnsquid.blocklist import Blocklist, noauth_list, normalize_noauth_domain +from lmnsquid.config import Settings + +runner = CliRunner() +NAME = "default-school-teachers" + + +def _create(client: TestClient, auth: dict[str, str], data: dict[str, Any]) -> str: + assert client.post("/v1/instances", json=data, headers=auth).status_code == 201 + return NAME + + +# ------------------------------------------------------------------ helper + + +@pytest.mark.parametrize( + ("raw", "expected"), + [ + ("office.com", [".office.com"]), + (" Outlook.Office.COM. ", [".outlook.office.com"]), + (".sharepoint.com", [".sharepoint.com"]), + ("bücher.de", [".xn--bcher-kva.de"]), + ], +) +def test_noauth_add_normalizes(tmp_path: Path, raw: str, expected: list[str]) -> None: + na = noauth_list(str(tmp_path)) + assert na.add("s-teachers", raw) == expected + assert (tmp_path / "s-teachers" / "noauth.domains").read_text() == f"{expected[0]}\n" + + +@pytest.mark.parametrize( + "bad", + [ + "", + ".", + "*.office.com", + "office.com/x", + "a b.com", + "-x.com", + "x_y.com", + "a" * 64, + # IP literals in every form squid would see as a host, and single labels (a TLD) + "172.28.0.20", + "10.0.0.1", + "127.1", + "0x7f000001", + "0x7f.0x1", + "1.2.3.0x4", + "0x0a.0x00.0x00.0x01", + "0177.0.0.1", + "2887516181", + "::1", + "[::1]", + "::ffff:172.28.0.21", + "com", + ".com", + "intranet", + ], +) +def test_noauth_add_rejects_invalid_names(tmp_path: Path, bad: str) -> None: + na = noauth_list(str(tmp_path)) + with pytest.raises(ValueError): + na.add("s-teachers", bad) + assert not (tmp_path / "s-teachers" / "noauth.domains").exists() + + +@pytest.mark.parametrize("bad", ["../x", "a/b", "..", "", "a", "a.b", "/etc"]) +def test_noauth_rejects_unsafe_instance_name(tmp_path: Path, bad: str) -> None: + na = noauth_list(str(tmp_path / "lists")) + with pytest.raises(ValueError): + na.add(bad, "office.com") + assert not (tmp_path / "x").exists() and not (tmp_path / "lists").exists() + + +def test_both_lists_share_the_directory(tmp_path: Path) -> None: + root = str(tmp_path) + bl, na = Blocklist(root), noauth_list(root) + bl.add("s-teachers", "blocked.example") + na.add("s-teachers", "office.com") + directory = tmp_path / "s-teachers" + assert sorted(p.name for p in directory.iterdir()) == ["blocked.domains", "noauth.domains"] + assert bl.read("s-teachers") == [".blocked.example"] + assert na.read("s-teachers") == [".office.com"] + assert (directory / "noauth.domains").stat().st_mode & 0o777 == 0o644 + na.remove("s-teachers", "office.com") + assert bl.read("s-teachers") == [".blocked.example"] # the other list is untouched + na.delete("s-teachers") # rm: the directory goes with both lists + assert not directory.exists() + + +def test_noauth_read_and_add_with_crlf(tmp_path: Path) -> None: + file = tmp_path / "s-teachers" / "noauth.domains" + file.parent.mkdir() + file.write_bytes(b".office.com\r\n.sharepoint.com\r\n") + na = noauth_list(str(tmp_path)) + assert na.read("s-teachers") == [".office.com", ".sharepoint.com"] + assert na.add("s-teachers", "teams.microsoft.com") == [ + ".office.com", + ".sharepoint.com", + ".teams.microsoft.com", + ] + assert b"\r" not in file.read_bytes() + + +def test_noauth_names_vs_blocklist_names(tmp_path: Path) -> None: + # a domain below a TLD with letters in the last label passes, also IDNA and digits inside + assert normalize_noauth_domain("1und1.de") == ".1und1.de" + assert normalize_noauth_domain("x.co.uk") == ".x.co.uk" + # the blocklist keeps accepting a TLD and an address on purpose: a wrong entry there + # only blocks more + bl = Blocklist(str(tmp_path)) + assert bl.add("s-teachers", "zip") == [".zip"] + assert bl.add("s-teachers", "10.0.0.1") == [".10.0.0.1", ".zip"] + + +def test_noauth_remove_accepts_what_add_refuses(tmp_path: Path) -> None: + # a hand-written "com" or address can still be taken out with the CLI + file = tmp_path / "s-teachers" / "noauth.domains" + file.parent.mkdir() + file.write_text(".com\n10.0.0.1\n.office.com\n") + na = noauth_list(str(tmp_path)) + assert na.remove("s-teachers", "com") == [".office.com", "10.0.0.1"] + assert na.remove("s-teachers", "10.0.0.1") == [".office.com"] + + +# --------------------------------------------------------------------- API + + +def test_noauth_requires_token(client: TestClient) -> None: + assert client.get(f"/v1/instances/{NAME}/noauth").status_code == 401 + assert client.post(f"/v1/instances/{NAME}/noauth", json={"domain": "x.org"}).status_code == 401 + assert client.delete(f"/v1/instances/{NAME}/noauth/x.org").status_code == 401 + assert client.post(f"/v1/instances/{NAME}/noauth/reload").status_code == 401 + bad = {"Authorization": "Bearer nope"} + assert client.get(f"/v1/instances/{NAME}/noauth", headers=bad).status_code == 403 + + +def test_noauth_unknown_or_unsafe_instance( + client: TestClient, auth_headers: dict[str, str] +) -> None: + assert client.get("/v1/instances/x-y/noauth", headers=auth_headers).status_code == 404 + resp = client.post("/v1/instances/x-y/noauth", json={"domain": "x.org"}, headers=auth_headers) + assert resp.status_code == 404 + assert client.post("/v1/instances/x-y/noauth/reload", headers=auth_headers).status_code == 404 + # a traversal attempt never reaches the file system + resp = client.get("/v1/instances/..%2F..%2Fetc/noauth", headers=auth_headers) + assert resp.status_code in (404, 422) + + +def test_noauth_endpoints( + client: TestClient, + auth_headers: dict[str, str], + instance_data: dict[str, Any], + docker: Any, + settings: Settings, +) -> None: + name = _create(client, auth_headers, instance_data) + file = Path(settings.blocklists_dir) / name / "noauth.domains" + assert file.is_file() and file.read_text() == "" # created empty with the instance + resp = client.get(f"/v1/instances/{name}/noauth", headers=auth_headers) + assert resp.status_code == 200 and resp.json() == {"name": name, "domains": []} + + resp = client.post( + f"/v1/instances/{name}/noauth", json={"domain": "Office.com"}, headers=auth_headers + ) + assert resp.status_code == 200 and resp.json()["domains"] == [".office.com"] + assert file.read_text() == ".office.com\n" + # the blocklist is a separate file in the same directory + assert client.get(f"/v1/instances/{name}/blocklist", headers=auth_headers).json() == { + "name": name, + "domains": [], + } + + for bad in ("not a host", "172.28.0.20", "com", "127.1", "::1"): + resp = client.post( + f"/v1/instances/{name}/noauth", json={"domain": bad}, headers=auth_headers + ) + assert resp.status_code == 422, bad + assert file.read_text() == ".office.com\n" + + resp = client.post(f"/v1/instances/{name}/noauth/reload", headers=auth_headers) + assert resp.status_code == 200 and resp.json() == { + "name": name, + "reloaded": True, + "notices": [], + } + assert docker.reloaded == [name] + + resp = client.delete(f"/v1/instances/{name}/noauth/office.com", headers=auth_headers) + assert resp.status_code == 200 and resp.json()["domains"] == [] + resp = client.delete(f"/v1/instances/{name}/noauth/office.com", headers=auth_headers) + assert resp.status_code == 404 + resp = client.delete(f"/v1/instances/{name}/noauth/bad%20host", headers=auth_headers) + assert resp.status_code == 422 + + client.post(f"/v1/instances/{name}/stop", headers=auth_headers) + resp = client.post(f"/v1/instances/{name}/noauth/reload", headers=auth_headers) + assert resp.status_code == 409 + + +def test_noauth_reload_refuses_an_image_that_ignores_the_list( + client: TestClient, auth_headers: dict[str, str], instance_data: dict[str, Any], docker: Any +) -> None: + data = {**instance_data, "image": "ghcr.io/faircomp/linuxmuster-squid:old"} + name = _create(client, auth_headers, data) + client.post(f"/v1/instances/{name}/noauth", json={"domain": "office.com"}, headers=auth_headers) + resp = client.post(f"/v1/instances/{name}/noauth/reload", headers=auth_headers) + assert resp.status_code == 409 + detail = resp.json()["detail"] + assert "does not support the noauth list" in detail and "release" in detail + # `lmnsquid update` only after the release (before it, it pulls the same old image) + assert "installing that release" in detail and f"lmnsquid update {name}" in detail + assert docker.reloaded == [] # no "reloaded" that changes nothing + # the blocklist reload is unaffected + resp = client.post(f"/v1/instances/{name}/blocklist/reload", headers=auth_headers) + assert resp.status_code == 200 + + +def test_rm_removes_the_noauth_list( + client: TestClient, + auth_headers: dict[str, str], + instance_data: dict[str, Any], + settings: Settings, +) -> None: + name = _create(client, auth_headers, instance_data) + client.post(f"/v1/instances/{name}/noauth", json={"domain": "office.com"}, headers=auth_headers) + assert client.delete(f"/v1/instances/{name}", headers=auth_headers).status_code == 204 + assert not (Path(settings.blocklists_dir) / name).exists() + + +# --------------------------------------------------------------------- CLI + + +@pytest.fixture +def patch_client(monkeypatch: pytest.MonkeyPatch, app: Any, token: str) -> None: + def factory() -> TestClient: + tc = TestClient(app) + tc.headers.update({"Authorization": f"Bearer {token}"}) + return tc + + monkeypatch.setattr(cli, "_get_client", factory) + + +def test_cli_noauth(patch_client: None, instance_data: dict[str, Any], docker: Any) -> None: + args = ["create", "--school", instance_data["school"], "--role", instance_data["role"]] + for key in ("ad_group", "realm", "visible_hostname", "keytab_secret"): + args += [f"--{key.replace('_', '-')}", instance_data[key]] + assert runner.invoke(cli.app, args).exit_code == 0 + + r = runner.invoke(cli.app, ["noauth", NAME, "add", "office.com"]) + assert r.exit_code == 0 and '".office.com"' in r.output, r.output + r = runner.invoke(cli.app, ["noauth", NAME, "list"]) + assert r.exit_code == 0 and '".office.com"' in r.output + r = runner.invoke(cli.app, ["noauth", NAME, "reload"]) + assert r.exit_code == 0 and '"reloaded": true' in r.output + r = runner.invoke(cli.app, ["noauth", NAME, "remove", "office.com"]) + assert r.exit_code == 0 and '".office.com"' not in r.output + assert runner.invoke(cli.app, ["noauth", NAME, "remove", "office.com"]).exit_code == 1 + assert runner.invoke(cli.app, ["noauth", NAME, "add", "bad host"]).exit_code == 1 + assert runner.invoke(cli.app, ["noauth", NAME, "add", "10.0.0.1"]).exit_code == 1 + assert runner.invoke(cli.app, ["noauth", NAME, "add", "com"]).exit_code == 1 + + +def test_cli_noauth_help_warns() -> None: + # Rich styles and wraps the help after the environment: on GitHub Actions (GITHUB_ACTIONS, + # FORCE_COLOR) it inserts ANSI codes inside words, and the width follows COLUMNS. So: no + # colour variables, a wide terminal, and still strip escape codes and line breaks. + plain = {"GITHUB_ACTIONS": None, "FORCE_COLOR": None, "NO_COLOR": "1", "TERM": "dumb"} + r = runner.invoke(cli.app, ["noauth", "--help"], env={**plain, "COLUMNS": "200"}) + assert r.exit_code == 0 + text = " ".join(re.sub(r"\x1b\[[0-9;?]*[A-Za-z]", "", r.output).split()) + for needle in ("exam mode", "Internetsperre", "school-subnets", "blocklist still wins"): + assert needle in text, needle diff --git a/debian/changelog b/debian/changelog index 0a095d3..b3c356b 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,64 @@ +linuxmuster-squid (7.3.8) lmn73; urgency=medium + + * New per-instance list of domains reachable without proxy login, for + services that break behind proxy authentication (Microsoft 365): + "lmnsquid noauth list|add |remove |reload" + (GET/POST/DELETE /v1/instances/{name}/noauth, POST .../noauth/reload), + stored in /etc/linuxmuster-squid/blocklists//noauth.domains next to + the blocklist. A listed domain and its subdomains are reachable without a + ticket from --school-subnets only; the blocklist still wins; HTTPS passes + only when the TLS SNI is on the list too (another SNI or none cuts the + handshake); IP addresses in any notation (also 0x7f.0x1 or 1.2.3.0x4) + and single labels (a whole TLD) are refused as entries, and an + IP-literal URL never matches. Empty by default, so + nothing changes until you add a domain. Warning: exam accounts reach listed + domains during an exam, the Internetsperre does not apply to them, and + the access log shows no user for them. A recommended Microsoft 365 list + with its source is in docs/operations.md. The list needs a data-plane + image that knows it: until the instances run one, "noauth reload" answers + 409 and the list has no effect. Existing instances are not restarted for + it. + * provision-keytab.sh refuses a proxy name that a computer account already + answers for. Samba treats host/ as an alias of HTTP/, so a computer + account PROXY$ (the proxy host in devices.csv with a role that gets an + account) holds HTTP/ too; samba-tool only logged the + collision and the script exported a keytab that could not work. On every + run, also when the SPN is already on the service account, it now looks + for HTTP/, host/ and host/ on other accounts and + stops with FATAL and a hint (devices.csv role iponly) before it changes or + exports anything. It also stops when it cannot check (no ldbsearch, no + read access to sam.ldb, or ldbsearch failing, with its message) and + refuses an argument that is not a fully qualified host name. + * discover-ad-facts.sh reads the realm on the DC (smb.conf realm, via + testparm) instead of trusting REALM: a REALM given in the environment is + only compared, and a mismatch prints a WARNING (a short REALM such as + BS-WIZ for BS-WIZ.LLAN used to end up in the printed commands). The + commands now carry proxy. instead of the placeholder + proxy. (PROXY_FQDN overrides it), and a third template creates one + proxy for everybody in the group internet, which per sophomorix.ini also + holds parents, staff and the school admins. + * A blocklist with a subdomain sorted before its parent domain + (".a.example.org" before ".example.org", or "example.org" next to it) + no longer takes the proxy down. squid 6 refuses such a list ("FATAL: + Bungled"): "blocklist reload" failed and the next start of the container + failed. Category lists from blocklist-refresh.sh and hand edits produced + it easily. "lmnsquid blocklist|noauth add" now keeps a listed parent and + drops what it covers (adding a name below a listed domain changes + nothing), blocklist-refresh.sh drops covered entries, and the new + data-plane image builds the lists squid reads without them at every + start and reload, naming them in "lmnsquid logs " and in the + reload output ("notices"). + * Documentation corrected: HTTPS to a blocked domain was described as cut + at the TLS handshake. squid answers a CONNECT it denies itself: "200", + then TLS with the instance's self-signed throwaway certificate and its + 403 page inside; the blocked server is never contacted and browsers + show a certificate warning. Only a name blocked in the SNI behind an + allowed CONNECT host is cut during the handshake. Traffic to allowed + sites is spliced and never decrypted, as before; the behaviour itself + is unchanged (README, install.md, operations.md, ADR-002). + + -- Kevin Stenzel Fri, 30 Oct 2026 14:58:39 +0100 + linuxmuster-squid (7.3.7) lmn73; urgency=medium * Upgrades respect a service the admin switched off. Up to 7.3.6 every diff --git a/deploy/e2e/client/Dockerfile b/deploy/e2e/client/Dockerfile index 08e983f..cac3640 100644 --- a/deploy/e2e/client/Dockerfile +++ b/deploy/e2e/client/Dockerfile @@ -6,7 +6,7 @@ FROM ubuntu:24.04 ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update \ - && apt-get install -y --no-install-recommends krb5-user curl ca-certificates \ + && apt-get install -y --no-install-recommends krb5-user curl ca-certificates openssl \ && rm -rf /var/lib/apt/lists/* COPY krb5.conf /etc/krb5.conf COPY assert.sh /assert.sh diff --git a/deploy/e2e/client/assert.sh b/deploy/e2e/client/assert.sh index f4d427b..86de25a 100755 --- a/deploy/e2e/client/assert.sh +++ b/deploy/e2e/client/assert.sh @@ -64,6 +64,32 @@ check_via() { if [ "$code" = "$exp" ]; then echo " [PASS] $desc ($code)"; else echo " [FAIL] $desc: expected $exp, got $code"; fail=1; fi } +# HTTPS through a proxy: " " must equal +# (e.g. "200 200" = tunnel up and TLS done; "200 000" = tunnel up, TLS cut; "403 000"). +check_tunnel() { + local desc="$1" user="$2" proxy="$3" url="$4" exp="$5" got + kdestroy -q 2>/dev/null || true + if [ -n "$user" ]; then + printf '%s' "$PW" | kinit "$user@$REALM" >/dev/null 2>&1 || { echo " [FAIL] $desc: kinit $user"; fail=1; return; } + fi + got=$(curl -sk -o /dev/null -w '%{http_connect} %{http_code}' --max-time 15 \ + --proxy "$proxy" --proxy-negotiate -U : "$url" 2>/dev/null) + if [ "$got" = "$exp" ]; then echo " [PASS] $desc ($got)"; else echo " [FAIL] $desc: expected '$exp', got '$got'"; fail=1; fi +} + +# A TLS handshake through the proxy WITHOUT a ticket, with a chosen SNI (or none: "-"): +# = ok (server certificate received) or cut (no certificate: terminated). +check_tls_sni() { + local desc="$1" proxy="$2" target="$3" sni="$4" exp="$5" got=cut + local args=(-proxy "$proxy" -connect "$target") + if [ "$sni" = "-" ]; then args+=(-noservername); else args+=(-servername "$sni"); fi + kdestroy -q 2>/dev/null || true + if timeout 20 openssl s_client "${args[@]}" /dev/null | grep -q 'BEGIN CERTIFICATE'; then + got=ok + fi + if [ "$got" = "$exp" ]; then echo " [PASS] $desc ($got)"; else echo " [FAIL] $desc: expected $exp, got $got"; fail=1; fi +} + echo "== Kerberos E2E assertions ==" check "Teacher allowed" teacher1 "$ALLOWED" 200 check "Student denied" student1 "$ALLOWED" 403 @@ -81,6 +107,46 @@ check_via "Internet-gate: teacher in 'internet' -> ok" inetok "$PROXY3" check_via "Internet-gate OR: teacher in 'schule2-internet' -> ok" inetor "$PROXY3" "$ALLOWED" 200 check_via "Internet-gate: teacher in NO internet group -> deny" inetno "$PROXY3" "$ALLOWED" 403 +# Domains without proxy auth (ADR-016). squid-noauth lists secure.example.internal and +# both.example.internal (the latter also blocked); school_net = this client (10.228.0.30). +NA="http://squid4.example.internal:3128" +NA_HOST="squid4.example.internal:3128" +echo "-- noauth (client $(hostname -I 2>/dev/null | tr -d ' '), school_net 10.228.0.30/32) --" +check_via "noauth: listed HTTP, no ticket -> 200" "" "$NA" "http://secure.example.internal/" 200 +check_via "noauth: listed subdomain, no ticket -> 200" "" "$NA" "http://www.secure.example.internal/" 200 +check_via "noauth: not listed, no ticket -> 407" "" "$NA" "$ALLOWED" 407 +# squid-noauth knows 10.228.0.21 -> secure.example.internal (extra_hosts = its reverse +# cache): without dstdomain -n this IP-literal URL would match the list (200, measured). +check_via "noauth: IP literal of a listed host -> 407" "" "$NA" "http://10.228.0.21/" 407 +check_via "noauth: IP written into the list -> 407" "" "$NA" "http://10.228.0.20/" 407 +check_via "noauth: listed AND blocked HTTP, no ticket -> 403" "" "$NA" "http://both.example.internal/" 403 +check_tunnel "noauth: listed HTTPS, no ticket -> tunnel + TLS" "" "$NA" "https://secure.example.internal/" "200 200" +check_tunnel "noauth: not listed HTTPS, no ticket -> 407" "" "$NA" "https://secure2.example.internal/" "407 000" +# A denied CONNECT on an ssl-bump port: squid answers the CONNECT and delivers its 403 page +# inside TLS with its throwaway certificate (browsers reject it: TLS error); curl -k shows it. +check_tunnel "noauth: listed AND blocked HTTPS -> denied (403)" "" "$NA" "https://both.example.internal/" "200 403" +check_tls_sni "noauth: CONNECT listed, SNI listed -> handshake" "$NA_HOST" secure.example.internal:443 secure.example.internal ok +check_tls_sni "noauth: CONNECT listed, foreign SNI -> cut" "$NA_HOST" secure.example.internal:443 origin.example.internal cut +check_tls_sni "noauth: CONNECT listed, blocked SNI -> cut" "$NA_HOST" secure.example.internal:443 blocked.example.com cut +check_tls_sni "noauth: CONNECT listed, no SNI -> cut" "$NA_HOST" secure.example.internal:443 - cut +# With a ticket the instance behaves as before (the 200/403/407 matrix of install.md §6). +check_via "noauth instance: teacher -> 200" teacher1 "$NA" "$ALLOWED" 200 +check_via "noauth instance: student -> 403" student1 "$NA" "$ALLOWED" 403 +check_via "noauth instance: blocked (teacher) -> 403" teacher1 "$NA" "http://blocked.example.com/" 403 +check_via "noauth instance: blocked subdomain (teacher) -> 403" teacher1 "$NA" "http://a.blocked.example.com/" 403 +check_via "noauth instance: listed+blocked (teacher) -> 403" teacher1 "$NA" "http://both.example.internal/" 403 +check_via "noauth instance: listed (teacher) -> 200" teacher1 "$NA" "http://secure.example.internal/" 200 +check_tunnel "noauth instance: HTTPS not listed (teacher) -> 200" teacher1 "$NA" "https://secure2.example.internal/" "200 200" +# An empty list (squid: the image's empty file) and a directory without the file (squid2, +# as an older control plane mounts it) change nothing: +check_via "empty list: listed elsewhere, no ticket -> 407" "" "$PROXY" "http://secure.example.internal/" 407 +check_via "no list file: no ticket -> 407" "" "$PROXY2" "http://secure.example.internal/" 407 +# The image from before ADR-016 (7.3.7's pin) with a directory that holds noauth.domains: +PROXY5="http://squid5.example.internal:3128" +check_via "old image + noauth.domains: no ticket -> 407" "" "$PROXY5" "http://secure.example.internal/" 407 +check_via "old image + noauth.domains: teacher -> 200" teacher1 "$PROXY5" "http://secure.example.internal/" 200 +check_via "old image + noauth.domains: student -> 403" student1 "$PROXY5" "$ALLOWED" 403 + echo if [ "$fail" -eq 0 ]; then echo "E2E: ALL ASSERTIONS OK"; else echo "E2E: FAILED"; fi exit "$fail" diff --git a/deploy/e2e/client/krb5.conf b/deploy/e2e/client/krb5.conf index 5c8feb0..183314f 100644 --- a/deploy/e2e/client/krb5.conf +++ b/deploy/e2e/client/krb5.conf @@ -14,7 +14,7 @@ [realms] EXAMPLE.INTERNAL = { - kdc = 172.28.0.2 + kdc = 10.228.0.2 } [domain_realm] diff --git a/deploy/e2e/docker-compose.yml b/deploy/e2e/docker-compose.yml index 609f05d..62e1311 100644 --- a/deploy/e2e/docker-compose.yml +++ b/deploy/e2e/docker-compose.yml @@ -3,7 +3,8 @@ # SPDX-License-Identifier: GPL-3.0-or-later # Kerberos E2E stack for linuxmuster-squid (heavy tier / crabbox). -# Proves: teacher->200, student->403, blocked domain->403, no ticket->407. +# Proves: teacher->200, student->403, blocked domain->403, no ticket->407, and the +# domains without proxy auth (ADR-016) incl. their negative cases. # Orchestrated by scripts/tests/e2e_kerberos.sh (brings services up in order, # creates fixtures and runs the assertions). Realm/domain: EXAMPLE.INTERNAL. name: lmnsquid-e2e @@ -13,7 +14,7 @@ networks: driver: bridge ipam: config: - - subnet: 172.28.0.0/24 + - subnet: 10.228.0.0/24 volumes: shared: {} # DC exports the Keytab here, Squid reads it (ro) @@ -28,11 +29,11 @@ services: DOMAIN: "EXAMPLE.INTERNAL" DOMAINPASS: "Passw0rd!" NOCOMPLEXITY: "true" - HOSTIP: "172.28.0.2" + HOSTIP: "10.228.0.2" INSECURELDAP: "true" networks: e2e: - ipv4_address: 172.28.0.2 + ipv4_address: 10.228.0.2 volumes: - shared:/shared @@ -50,10 +51,10 @@ services: SCHOOL_SUBNETS: "0.0.0.0/0" # Subnet scope is tested in P3 KEYTAB: "/shared/squid.keytab" CACHE_SIZE_MB: "200" - dns: ["172.28.0.2"] # AD-DNS: resolves SRV/_ldap._tcp + origin A-record + dns: ["10.228.0.2"] # AD-DNS: resolves SRV/_ldap._tcp + origin A-record networks: e2e: - ipv4_address: 172.28.0.10 + ipv4_address: 10.228.0.10 volumes: - shared:/shared:ro - ./blocked.domains:/etc/squid/lists/blocked.domains:ro @@ -77,13 +78,14 @@ services: SCHOOL_SUBNETS: "0.0.0.0/0" KEYTAB: "/shared/squid.keytab" CACHE_SIZE_MB: "200" - dns: ["172.28.0.2"] + dns: ["10.228.0.2"] networks: e2e: - ipv4_address: 172.28.0.11 + ipv4_address: 10.228.0.11 volumes: - shared:/shared:ro - - ./blocked.domains:/etc/squid/lists/blocked.domains:ro + # The list directory of a control plane without ADR-016 (no noauth.domains). + - ./lists-legacy:/etc/squid/lists:ro read_only: true tmpfs: ["/run", "/tmp", "/var/log/squid", "/var/spool/squid"] cap_drop: ["ALL"] @@ -106,10 +108,10 @@ services: SCHOOL_SUBNETS: "0.0.0.0/0" KEYTAB: "/shared/squid.keytab" CACHE_SIZE_MB: "200" - dns: ["172.28.0.2"] + dns: ["10.228.0.2"] networks: e2e: - ipv4_address: 172.28.0.12 + ipv4_address: 10.228.0.12 volumes: - shared:/shared:ro - ./blocked.domains:/etc/squid/lists/blocked.domains:ro @@ -119,13 +121,73 @@ services: cap_add: ["SETUID", "SETGID", "DAC_OVERRIDE", "CHOWN"] security_opt: ["no-new-privileges:true"] + # Domains without proxy auth (ADR-016): the list directory mounted like the control plane + # does it; school_net is the test client alone, so a request from any other container is + # "outside the school subnets". + squid-noauth: + image: linuxmuster-squid:dev + hostname: squid4.example.internal + depends_on: [samba-dc, squid] + environment: + INSTANCE: "e2e-noauth" + VISIBLE_HOSTNAME: "squid4.example.internal" + REALM: "EXAMPLE.INTERNAL" + AD_GROUP: "teachers" + SCHOOL_SUBNETS: "10.228.0.30/32" + KEYTAB: "/shared/squid.keytab" + CACHE_SIZE_MB: "200" + dns: ["10.228.0.2"] + # squid loads /etc/hosts into its reverse cache too: the IP-literal test needs the + # address to map back to a listed name (proves dstdomain -n). + extra_hosts: ["secure.example.internal:10.228.0.21"] + networks: + e2e: + ipv4_address: 10.228.0.13 + volumes: + - shared:/shared:ro + - ./lists-noauth:/etc/squid/lists:ro + read_only: true + tmpfs: ["/run", "/tmp", "/var/log/squid", "/var/spool/squid"] + cap_drop: ["ALL"] + cap_add: ["SETUID", "SETGID", "DAC_OVERRIDE", "CHOWN"] + security_opt: ["no-new-privileges:true"] + + # An image from before ADR-016 (the one 7.3.7 pins; set by e2e_kerberos.sh) with a list + # directory that holds noauth.domains: what an instance runs until the upgrade replaces its + # container. It must ignore the list and keep asking for a ticket. + squid-old: + # (default only for the other smokes that start the DC from this file) + image: ${LMNSQUID_OLD_IMAGE:-linuxmuster-squid:dev} + hostname: squid5.example.internal + depends_on: [samba-dc, squid] + environment: + INSTANCE: "e2e-old" + VISIBLE_HOSTNAME: "squid5.example.internal" + REALM: "EXAMPLE.INTERNAL" + AD_GROUP: "teachers" + SCHOOL_SUBNETS: "0.0.0.0/0" + KEYTAB: "/shared/squid.keytab" + CACHE_SIZE_MB: "200" + dns: ["10.228.0.2"] + networks: + e2e: + ipv4_address: 10.228.0.14 + volumes: + - shared:/shared:ro + - ./lists-old:/etc/squid/lists:ro + read_only: true + tmpfs: ["/run", "/tmp", "/var/log/squid", "/var/spool/squid"] + cap_drop: ["ALL"] + cap_add: ["SETUID", "SETGID", "DAC_OVERRIDE", "CHOWN"] + security_opt: ["no-new-privileges:true"] + # Allowed target (returns 200). origin: image: nginx:alpine hostname: origin.example.internal networks: e2e: - ipv4_address: 172.28.0.20 + ipv4_address: 10.228.0.20 # Allowed HTTPS target (self-signed) — for the splice test. origin-https: @@ -133,13 +195,13 @@ services: hostname: secure.example.internal networks: e2e: - ipv4_address: 172.28.0.21 + ipv4_address: 10.228.0.21 # Kerberos client that runs the four assertions (ENTRYPOINT = /assert.sh). test-client: build: ./client depends_on: [squid] - dns: ["172.28.0.2"] + dns: ["10.228.0.2"] networks: e2e: - ipv4_address: 172.28.0.30 + ipv4_address: 10.228.0.30 diff --git a/deploy/e2e/lists-legacy/blocked.domains b/deploy/e2e/lists-legacy/blocked.domains new file mode 100644 index 0000000..1966405 --- /dev/null +++ b/deploy/e2e/lists-legacy/blocked.domains @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# +# SPDX-License-Identifier: GPL-3.0-or-later + +# A list directory as a control plane without ADR-016 mounts it: blocked.domains only, +# no noauth.domains. The image must start and behave as before (no domain without auth). +blocked.example.com diff --git a/deploy/e2e/lists-noauth/blocked.domains b/deploy/e2e/lists-noauth/blocked.domains new file mode 100644 index 0000000..29c758e --- /dev/null +++ b/deploy/e2e/lists-noauth/blocked.domains @@ -0,0 +1,11 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# +# SPDX-License-Identifier: GPL-3.0-or-later + +# Blocklist of the squid-noauth E2E instance (the directory is mounted at /etc/squid/lists, +# as the control plane does). both.example.internal is on both lists: the blocklist wins. +# .a.blocked.example.com and the bare apex: entries squid 6 refuses next to their parent. +.a.blocked.example.com +blocked.example.com +.blocked.example.com +.both.example.internal diff --git a/deploy/e2e/lists-noauth/noauth.domains b/deploy/e2e/lists-noauth/noauth.domains new file mode 100644 index 0000000..55afb6a --- /dev/null +++ b/deploy/e2e/lists-noauth/noauth.domains @@ -0,0 +1,15 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# +# SPDX-License-Identifier: GPL-3.0-or-later + +# Domains reachable without proxy auth for the squid-noauth E2E instance (ADR-016). +# Hand-edited on purpose: a subdomain sorted before its parent and the bare apex next to it +# would stop squid 6 ("FATAL: Bungled"); lists-sync.sh in the image drops them. +.a.secure.example.internal +secure.example.internal +.secure.example.internal +# Also hand-written, and ignored by lists-sync.sh for this list: an address (origin) and a +# single label that would open every *.internal name. +10.228.0.20 +internal +.both.example.internal diff --git a/deploy/e2e/lists-old/blocked.domains b/deploy/e2e/lists-old/blocked.domains new file mode 100644 index 0000000..555ff1c --- /dev/null +++ b/deploy/e2e/lists-old/blocked.domains @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# +# SPDX-License-Identifier: GPL-3.0-or-later + +# Lists of the squid-old E2E instance (an image from before ADR-016, which reads the +# mounted files directly, so no entry here may overlap another). +.blocked.example.com diff --git a/deploy/e2e/lists-old/noauth.domains b/deploy/e2e/lists-old/noauth.domains new file mode 100644 index 0000000..f8ddbcb --- /dev/null +++ b/deploy/e2e/lists-old/noauth.domains @@ -0,0 +1,6 @@ +# SPDX-FileCopyrightText: Kevin Stenzel +# +# SPDX-License-Identifier: GPL-3.0-or-later + +# The image of squid-old predates ADR-016 and must ignore this list. +.secure.example.internal diff --git a/docs/architecture.md b/docs/architecture.md index e5f4b86..59baf59 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -63,17 +63,36 @@ Status document. Keep it up to date with every substantive change (see - **School scope:** `acl school_net src ${SCHOOL_SUBNETS}` (a school is identified by its subnet — no marker "on the wire"). - **HTTPS filtering without decryption:** preferably SNI **peek + splice** - (squid-openssl; throwaway CA only for the peek step, **never** distributed to - clients), fallback CONNECT `dstdomain` (no SSL needed). A blocked HTTPS name is - **terminated at the TLS handshake** (`ssl_bump terminate`): the client sees a - connection error, not a 403 page — there is no error page without decryption. + (squid-openssl; throwaway certificate only to enable the bump machinery, **never** + distributed to clients), fallback CONNECT `dstdomain` (no SSL needed). Allowed traffic is + spliced, never decrypted. A CONNECT that `http_access` denies (e.g. a blocked host) is + answered by squid itself: `200`, then TLS with the untrusted throwaway certificate and its 403 + page inside (squid's error delivery on an ssl-bump port; the target is never contacted; the + browser shows a certificate warning). A blocked SNI behind an allowed CONNECT host is + **terminated during the handshake** (`ssl_bump terminate`), without any page (ADR-002, + clarification). - **Blocklist per instance:** `//blocked.domains` on the host (`/etc/linuxmuster-squid/blocklists/…`), the directory bind-mounted read-only at `/etc/squid/lists` — the path the template reads for `dstdomain` and `ssl::server_name`. Managed via `lmnsquid blocklist add|remove|list|reload` (the API endpoints below); `reload` = `squid -k reconfigure` inside the container - (docker exec), no restart. Entries are `.example.org` = domain + subdomains. UT Capitole/Toulouse + (docker exec), no restart. Entries are `.example.org` = domain + subdomains. Squid reads + copies under `/run/lmnsquid/lists/` that `lists-sync.sh` builds from the mounted files at + start and before every reload (its messages go to the container log and into the reload + answer as `notices`), without entries a listed parent domain covers (squid 6 stops + with `FATAL: Bungled` when such an entry precedes its parent); the control plane never writes + them in the first place. UT Capitole/Toulouse category lists via `blocklist-refresh.sh` (host cron) into the same file. +- **Domains without proxy authentication per instance (ADR-016, optional):** + `//noauth.domains` in the same mounted directory (no extra mount, the + container spec is unchanged), managed via `lmnsquid noauth add|remove|list|reload`. + A listed domain is reachable from `school_net` without a ticket; the blocklist wins; a + CONNECT on this path must carry a listed SNI (no SNI → terminated); the list ACL is + `dstdomain -n` (no reverse DNS). Empty by default (= the old behaviour). A missing + `noauth.domains` (older control plane) is an empty list; + `noauth reload` refuses (409) when the running container does not read the list (an image + from before ADR-016). Price: no identity for these requests (exam mode, Internetsperre; + threat model T16). - **Hardening:** `cache_effective_user proxy`; Keytab as a secret (tmpfs, readable by `proxy`); `KRB5CCNAME=FILE:` (the kernel-keyring ccache fails when unprivileged); `read_only` rootfs + tmpfs; `cap_drop: ALL` (+ minimal SETUID/SETGID/DAC_OVERRIDE); @@ -84,9 +103,10 @@ Status document. Keep it up to date with every substantive change (see - **REST-API (FastAPI/uvicorn):** CRUD over instance definitions + lifecycle actions (`start/stop/restart/update/rollback/status/logs`) + the - per-instance blocklist (`GET/POST/DELETE …/blocklist`, `POST …/blocklist/reload`). - `DELETE /v1/instances/{name}` removes container, cache + log volumes and the - blocklist (`?keep_logs=true` keeps the log volume). + per-instance blocklist (`GET/POST/DELETE …/blocklist`, `POST …/blocklist/reload`) and + noauth list (`GET/POST/DELETE …/noauth`, `POST …/noauth/reload`). + `DELETE /v1/instances/{name}` removes container, cache + log volumes and both + lists (`?keep_logs=true` keeps the log volume). `HTTPBearer(auto_error=False)` + `hmac.compare_digest`; app-wide dependency; bind **127.0.0.1 only** (in-app TLS is NOT implemented; off-host only via an operator-owned TLS reverse proxy — on a non-loopback bind `main.py` warns, because the @@ -119,5 +139,10 @@ Status document. Keep it up to date with every substantive change (see 2. Squid `407 Proxy-Authenticate: Negotiate` → client sends a Kerberos ticket (SSO). 3. `negotiate_kerberos_auth` checks against the Keytab → username (`%LOGIN`). 4. `ext_kerberos_ldap_group_acl` checks group membership (ticket/LDAP, SRV). -5. `http_access` (first-match-wins): `deny !authenticated` → `deny blocked` → - `allow authenticated role_group school_net` → `deny all`. Result 200 / 403 / 407. +5. `http_access` (first-match-wins): the port/CONNECT/IP-literal denies → + `deny noauth+blocked` → `allow noauth school_net` (no ticket asked; empty list = never) → + `deny !authenticated` → `deny blocked` → `allow authenticated role_group school_net` → + `deny all`. Result 200 / 403 / 407. +6. HTTPS (CONNECT, no decryption): `ssl_bump` peeks at the ClientHello (step 1) and at step 2 + terminates a blocked SNI, and for a CONNECT to a noauth domain an SNI that is not on the + noauth list or a missing SNI; everything else is spliced. diff --git a/docs/decisions.md b/docs/decisions.md index 15d1688..5d9e72b 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -30,6 +30,22 @@ requires explicit mode. **Source:** Squid wiki Features/Authentication. CONNECT `dstdomain`; **no** SSL-bump MITM. The peek CA is never distributed to clients. **Why:** privacy-friendly, no client CA rollout, no breaking of cert pinning; SSL-bumped traffic could no longer carry Kerberos identity. +**Clarification (2026-11-02, measured in the E2E and the lab; Kevin: keep the behaviour):** +traffic to a real target is never decrypted: an allowed CONNECT is spliced. Two things still +happen at the TLS layer. (1) `ssl_bump terminate` closes the connection during the handshake, +without a certificate or a page, for a blocked SNI behind an allowed CONNECT host and, on the +noauth path (ADR-016), for a foreign or missing SNI. (2) A CONNECT that `http_access` denies with +403 (blocked CONNECT host, not in the role or internet group, CONNECT to an IP literal or a port +other than 443, outside the school subnets) is answered by squid itself: on an ssl-bump port it +confirms the CONNECT (`200`), completes TLS with the client using the instance's self-signed +throwaway certificate (`bump.pem`, `generate-host-certificates=off`) and sends its 403 page +inside that session. squid.conf does not describe this; the squid 6 source does +(`client_side_request.cc`: "we will attempt to bump the client to serve the error"). The target is +never contacted. Browsers reject the certificate and show a certificate warning, no block page; a +client that accepts it anyway (`curl -k`, a user clicking through the warning) sends its request +line and headers for that denied name to the proxy, which logs the URL. A 407 is never answered +this way (squid does not bump during authentication). The certificate is presented, never +distributed or trusted. ### ADR-003 — One instance per (school × role) **Status:** Accepted (user decision). **Decision:** separate Squid containers @@ -144,8 +160,18 @@ inode); reconfigure keeps the cache and client connections. **Not** `docker kill although squid is PID 1 and would reconfigure: Docker records every kill(), whatever the signal, as a manual stop, and an `unless-stopped` container then stays down after a reboot (seen in the lab). Price: like `access-logs`, `reload` needs `docker exec` (ADR-012). -**Limit (documented):** a blocked HTTPS name is terminated at the TLS handshake, there is -no 403 page without decryption (ADR-002). +**Limit (documented):** HTTPS to a blocked name never shows a usable block page: a blocked +CONNECT host gets squid's 403 inside TLS with an untrusted certificate (browser: certificate +warning), a blocked SNI behind an allowed CONNECT host is cut during the handshake (ADR-002, +clarification). +**Amendment (2026-10-30, with ADR-016):** squid 6 refuses to load a `dstdomain` / +`ssl::server_name` list in which a name covered by a listed parent domain comes first +(`.a.example.org` before `.example.org`, or `example.org` next to it: `FATAL: Bungled`; a +`reload` fails, the next container start fails), which category lists and hand edits produce +easily. Squid therefore reads copies under `/run/lmnsquid/lists/` built by the image's +`lists-sync.sh` at start and before every `reload`, without covered entries (logged); the +control plane (`add` keeps the parent and drops what it covers) and `blocklist-refresh.sh` +never write them. ### ADR-015 — Build inputs pinned: hashed Python locks, digests, action SHAs, draft-first releases **Status:** Accepted (2026-09-23, stage A "supply chain" of the linuxmusterDEV archive plan). @@ -205,6 +231,43 @@ rejects `--python-platform`, so the lock is compiled on linux/x86_64 (CI, Renova with `--python-version=3.12`; environment markers then evaluate as on the target. **Price:** dependency updates need a merged PR; the lock is regenerated, never hand-edited. +### ADR-016 — Domains without proxy authentication, per instance, beside the blocklist +**Status:** Proposed (2026-10-30, Kevin's request: Microsoft 365 through the proxy, without a +login, no client-side bypass). **Decision:** each instance may list domains in +`//noauth.domains`, managed like the blocklist (`lmnsquid noauth +list|add|remove|reload`, entries `.example.org` = domain + subdomains; only names below a +top-level domain: IP literals in any form (IPv6; IPv4 in every `inet_aton` notation, i.e. a +name whose labels are all decimal, octal or `0x` numbers) and single labels are refused, and +ignored in the +container when hand-written). A request to a listed +domain from `school_net` is allowed **without** proxy authentication; the rules, in order: +the existing denies (Safe_ports, CONNECT only to 443, no IP literals on CONNECT) stay first; +`deny noauth_domains blocked_domains` (the blocklist wins, 403); `allow noauth_domains +school_net`; then the unchanged chain (`deny !authenticated`, …). The list ACL is `dstdomain +-n`: no reverse DNS for IP-literal URLs, otherwise whoever controls a PTR record could put an +address on the list (measured: without `-n`, `http:///` with a PTR to a listed name got +200 without a ticket). A CONNECT that came through this way may not hand its tunnel to another +name: at `SslBump2` it is terminated when the SNI is not on the list (`ssl::server_name +--client-requested`) or when the ClientHello carries no SNI (`external_acl_type … %ssl::>sni` +with a four-line helper, because `ssl::server_name` falls back to the CONNECT host without +SNI). Still no decryption (peek, splice, terminate only; ADR-002). The empty list is the +default and there is no built-in content; a recommended Microsoft 365 list lives in +`docs/operations.md` with its source. The file sits in the directory already mounted at +`/etc/squid/lists` (ADR-014), so the container spec does not change: an upgrade replaces no +container for it. +**Why:** Microsoft lists proxy authentication among the techniques that break Microsoft 365 +connections and asks to exempt its required endpoints; a client-side bypass (proxy exceptions ++ firewall rules) was not wanted, the traffic should still pass the proxy's domain filter. +**Price (threat model T16):** a listed domain is reachable by everyone in the school subnets +without identity: exam accounts reach it in exam mode, the Internetsperre does not apply to it, +and the access log shows `-` instead of a user for it. **Rollout:** the template is in the +data-plane image; `DEFAULT_IMAGE` must point at an image built with it before the release +that ships the control plane is tagged (RELEASE.md). Until then the list is +ignored and `lmnsquid noauth reload` answers 409 instead of reporting a reload that +changes nothing (it checks the rendered `squid.conf` in the container). An image with the +template next to a directory without `noauth.domains` (an older control plane) uses an empty +list until the file appears and the next reload reads it. + --- ## Site facts to be verified (P0, enter with source/date) diff --git a/docs/deployment-gpo.md b/docs/deployment-gpo.md index 698faf6..4098a77 100644 --- a/docs/deployment-gpo.md +++ b/docs/deployment-gpo.md @@ -78,7 +78,9 @@ Log per client (browser, codes): - [ ] **Visitor** (teacher/student from **another school**) → the **local** proxy works (global role group). - [ ] **Internetsperre:** remove a user from their `internet` group → within ~10 s → **403**. - [ ] Blocked domain (`lmnsquid blocklist add ` + `reload`): HTTP → **403** page; - HTTPS → browser connection/TLS error (no block page — the proxy does not decrypt). + HTTPS → browser certificate warning (the proxy serves its 403 inside TLS with an + untrusted throwaway certificate; the blocked server is never contacted), or a connection + error when only the SNI is blocked. No usable block page (ADR-002, clarification). - [ ] `Squid access.log` shows the Kerberos username + ACL verdict. Document the result (client/browser/codes) here or in the ticket. Only then is diff --git a/docs/install.md b/docs/install.md index cb335c9..f60ab6e 100644 --- a/docs/install.md +++ b/docs/install.md @@ -25,13 +25,32 @@ groups are prefixed, e.g. `-internet`). All commands as root. - `apt-get install -y docker.io krb5-user curl` (`krb5-user` only for the checks below). - **DNS:** the proxy FQDN needs an A record (and PTR). The linuxmuster DHCP does *not* register dynamic leases in the Samba DNS: add the proxy host as a static device in - `devices.csv` (import creates A/PTR), or on the DC - `samba-tool dns add proxy A -U `. + `devices.csv` with the **role `iponly`** (9th column) and run `linuxmuster-import-devices`, + or on the DC `samba-tool dns add proxy A -U `. + **Not with another role:** the default (`classroom-studentcomputer`), `server` and the other + roles with a computer account give the proxy a computer account `PROXY$` with + `host/`, and Samba treats `host/` as an alias of `HTTP/` (`sPNMappings`): that + account then holds the proxy's `HTTP/` name too, Kerberos SSO breaks, and + `provision-keytab.sh` (step 3) stops with a `FATAL`. sophomorix creates the A/PTR records for + every device with a fixed IP, a computer account only for roles with `COMPUTER_ACCOUNT = + TRUE` (`iponly`: `FALSE`; `sophomorix.ini`, `sophomorix-device`). Check on the DC: + `samba-tool computer show proxy` → `Unable to find computer "proxy$"`, and + `host ` → the proxy's address. A proxy that already got an account: set the role + to `iponly`, import again (sophomorix removes the account), then run step 3. - **Time:** NTP in sync with the DC (`timedatectl`; Kerberos tolerates < 5 min). - **`/etc/hosts` on the proxy host:** the proxy FQDN must resolve to the host's LAN IP *on the host itself*. cloud-init writes `127.0.1.1 `; with that, requests from the host reach the container via Docker's userland proxy as `172.17.0.1`, outside `--school-subnets`, and get 403. Put ` ` there. + **cloud-init hosts:** check `grep -rs manage_etc_hosts /etc/cloud/ /var/lib/cloud/instance/`. + With `manage_etc_hosts: true` cloud-init rewrites `/etc/hosts` from a template on **every + boot**, so a hand edit is gone after the next reboot (seen in the lab). Change the template + instead: on Ubuntu 24.04 `/etc/cloud/templates/hosts.debian.tmpl`, line + `127.0.1.1 {{fqdn}} {{hostname}}` → ` {{fqdn}} {{hostname}}`. With + `manage_etc_hosts: localhost` it appends `127.0.1.1 ` on every boot; set it to `false` + where it is defined (the instance's user-data, or a file under `/etc/cloud/cloud.cfg.d/`) and + keep your line in `/etc/hosts`. (cloud-init docs, module "Update Etc Hosts"; the template name + checked with cloud-init 26.2 on Ubuntu 24.04.) - The proxy host does **not** join the domain (ADR-009); it only needs the keytab from step 3. ## 2. Install the package @@ -87,13 +106,19 @@ and `lmnsquid restart ` ([`keytab-and-dns.md`](keytab-and-dns.md)). ## 4. Which groups? (on the DC) ```bash -REALM= bash /usr/share/linuxmuster-squid/scripts/discover-ad-facts.sh +bash /usr/share/linuxmuster-squid/scripts/discover-ad-facts.sh ``` -prints the **global role groups** `role-teacher` / `role-student` / `role-staff` (each spans -every school of the domain) for `--ad-group`, the **internet groups** (`internet` for the -default school, `-internet` for the others) for `--internet-group`, and two ready-made -`create` commands. Use the global role groups: a visitor from another school is then accepted at +reads the realm on the DC (`realm` in its `smb.conf`, the Kerberos realm; a `REALM=…` given in +the environment is only compared with it and a mismatch is reported as `WARNING`), and prints +the **global role groups** `role-teacher` / `role-student` / `role-staff` (each spans every +school of the domain) for `--ad-group`, the **internet groups** (`internet` for the default +school, `-internet` for the others) for `--internet-group`, and ready-made `create` +commands with `--realm --visible-hostname proxy.` filled in +(`PROXY_FQDN=` for another host name): template A, a teacher and a student proxy, and +template B, one proxy for everybody in `internet` (no teacher/student separation; per +`sophomorix.ini` the group holds teachers, students, parents, staff, extra students/course +members and the school admins of the default school). Use the global role groups: a visitor from another school is then accepted at the local proxy while teacher/student separation still holds. Do **not** use the per-school `teachers`/`students` groups unless schools must filter differently, and never a class group (`5a-students` looks like a role group but is not). @@ -151,17 +176,45 @@ lmnsquid blocklist all-students reload # squid re-reads the list (n lmnsquid blocklist all-students remove example.org && lmnsquid blocklist all-students reload ``` -HTTP to a blocked domain → **403**. HTTPS to a blocked domain is **cut at the TLS handshake** -(the proxy peeks at the SNI and terminates, it never decrypts): the browser shows a -connection/TLS error, not a block page. Details, the file path and UT-Capitole category +HTTP to a blocked domain → **403**. HTTPS to a blocked domain → the browser shows a +**certificate warning**, no block page: the proxy answers the CONNECT itself with its 403 page +inside TLS with an untrusted throwaway certificate; the blocked server is never contacted, and +traffic to allowed sites is never decrypted (it is spliced). Details, the file path and +UT-Capitole category lists: [`operations.md`](operations.md#blocklist-per-instance). -## 8. Clients +## 8. Domains without proxy authentication (optional) + +Some services break behind proxy authentication; Microsoft names it among the techniques that +cause Microsoft 365 connection problems. Instead of a bypass on the clients, an instance can let +listed domains through **without a ticket**, still through the proxy (domain filter, no +decryption): + +```bash +lmnsquid noauth all-students add sharepoint.com # -> [".sharepoint.com"]: domain + subdomains +lmnsquid noauth all-students list +lmnsquid noauth all-students reload # squid re-reads the lists (no restart) +``` + +> **Warning:** a listed domain is open to **everyone in `--school-subnets`** without identity: +> exam accounts reach it during an **exam** (OneDrive, Teams, Outlook …), the **Internetsperre** +> does not apply to it, and the access log shows `-` instead of a user. Keep `--school-subnets` +> tight (the default `0.0.0.0/0` means everyone who reaches the port) and the list short. + +The blocklist still wins (HTTP 403; HTTPS certificate warning as in step 7). HTTPS passes only +when the TLS SNI is on the list as well; a different SNI or none cuts the handshake. Check from a client in `` without +a ticket: `kdestroy; req $P:3129 https:///` gives 200, +`req $P:3129 http://example.com/` still 407. The recommended Microsoft 365 list with its source +and the details: [`operations.md`](operations.md#domains-without-proxy-authentication-per-instance). +`reload` answers 409 while the instance still runs a data-plane image that does not support the +list; a package release whose pinned image supports it moves the instances onto it. + +## 9. Clients Assign the proxy per role by GPO (silent Kerberos SSO) and force the proxy at the firewall: [`deployment-gpo.md`](deployment-gpo.md). Clients must use the **FQDN**, never the IP. -## 9. Updates +## 10. Updates ```bash apt-get install -y ./linuxmuster-squid__amd64.deb # restarts the service, then update-all (*) @@ -182,10 +235,10 @@ Verify with `docker inspect -f '{{range .Mounts}}{{.Destination}} {{end}}' lmnsq `journalctl -t linuxmuster-squid` and print a `WARNING:` in the apt output naming any instance that could not be brought up; the apt transaction itself never fails over them. -## 10. Removing +## 11. Removing ```bash -lmnsquid rm all-students # container, cache + log volumes, blocklist, definition +lmnsquid rm all-students # container, cache + log volumes, both lists, definition lmnsquid rm all-students --keep-logs # same, but keeps the access-log volume lmnsquid-logs-all-students ``` diff --git a/docs/keytab-and-dns.md b/docs/keytab-and-dns.md index 4d72cdf..09eb400 100644 --- a/docs/keytab-and-dns.md +++ b/docs/keytab-and-dns.md @@ -70,10 +70,26 @@ file; the control plane does **not** provision it itself (least privilege). NTLM fallback). `provision-keytab.sh` checks this beforehand (`ldbsearch`) and is idempotent; to check manually: `setspn -X` (Windows) or `ldbsearch -H .../sam.ldb '(servicePrincipalName=HTTP/)' sAMAccountName` (Samba). +- **No computer account with the proxy's name** — `host/` is an alias of `HTTP/` (the + `sPNMappings` attribute on `CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,…` + lists `http` under `host=`, in AD and in Samba). A computer account `PROXY$` holds + `host/proxy` and `host/proxy.` and with them, for the KDC, `HTTP/proxy.`. + `samba-tool spn add HTTP/proxy. svc-squid` then only logs + `check_spn_alias_collision: … when 'host/proxy.' is on 'CN=PROXY,…'` and adds the SPN + anyway (seen in a real installation, 2026-10-30). Typical cause: the proxy host in + `devices.csv` with a role that gets a computer account; use the role **`iponly`** (DNS + records, no account; [`install.md`](install.md) step 1). `provision-keytab.sh` looks for + `HTTP/`, `host/` and `host/` on any other account on every run (also + when the SPN is already on the service account) and stops with `FATAL`, exit ≠ 0, before it + changes or exports anything. Without `ldbsearch` (package `ldb-tools`, a dependency of + linuxmuster-base7), without read access to `sam.ldb` or when `ldbsearch` fails it stops as + well, with the reason. ## DNS & time requirements (hard) -- **A record** for every proxy FQDN (`visible_hostname`). +- **A record** for every proxy FQDN (`visible_hostname`). From `devices.csv`: role + **`iponly`** (A/PTR without a computer account), never a role with a computer account (see + "No computer account with the proxy's name" above). - **Forward + reverse DNS** OR `rdns = false` — the image already generates `/etc/krb5.conf` with `rdns=false` + `/etc/ldap/ldap.conf` `SASL_NOCANON on`, so that the `ldap/` SPN is formed from the literal DC name (not via PTR). diff --git a/docs/operations.md b/docs/operations.md index 49a2cfb..4434687 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -38,7 +38,7 @@ lmnsquid list lmnsquid status default-school-teachers lmnsquid stop|start|restart default-school-teachers lmnsquid logs default-school-teachers --tail 100 -lmnsquid rm default-school-teachers # + cache/log volumes, blocklist, .prev; --keep-logs keeps the log volume +lmnsquid rm default-school-teachers # + cache/log volumes, both lists, .prev; --keep-logs keeps the log volume ``` The keytab must already be present as secret `` in `secrets_dir` (`/etc/linuxmuster-squid/secrets`) — see `keytab-and-dns.md`. @@ -53,7 +53,8 @@ The keytab must already be present as secret `` in `secrets_dir` msg-internet`) so it covers **visitors** too and works with global `role-teacher`/`role-student` proxies — a user passes if in **any** listed group. Omit to enforce the role group only. - **`lmnsquid rm `** removes the container, the cache volume, the **log volume** - (`lmnsquid-logs-` = access-log history, personal data), the blocklist directory and + (`lmnsquid-logs-` = access-log history, personal data), the list directory (blocklist + and noauth list) and the definition (+ its `.prev` rollback note). `--keep-logs` (API: `?keep_logs=true`) keeps the log volume, e.g. for a retention obligation; delete it later with `docker volume rm lmnsquid-logs-`. The keytab in `secrets/` stays (operator-managed). @@ -73,26 +74,161 @@ lmnsquid blocklist default-school-students reload # squid re-reads - `add`/`remove` write the file (sorted, one `.domain` per line; comments are not preserved) and take effect on the next **`reload`** — batch your changes, then reload once. - `reload` runs `squid -k reconfigure` inside the container (docker exec, like + A name below a listed domain is already covered: adding it changes nothing, and adding a + parent removes the names below it (squid 6 refuses to load a list in which a covered name, + e.g. `.a.example.org` or `example.org`, comes before `.example.org`: `FATAL: Bungled`). + `reload` runs `lists-sync.sh` and `squid -k reconfigure` inside the container (docker exec, like `access-logs` — not available behind the socket proxy with `EXEC: 0`); the auth/group helpers restart, the cache stays. API: `GET/POST /v1/instances/{name}/blocklist`, `DELETE …/blocklist/{domain}`, `POST …/blocklist/reload`. -- **What the user sees:** HTTP → **403** (Squid error page, also for teachers). HTTPS → the - proxy peeks at the SNI and **terminates the TLS handshake**: the browser shows a - connection/TLS error, *not* a block page (there is none without decryption, ADR-002). - A blocked name shows up as `TCP_DENIED/403 … GET http://…` resp. `TCP_DENIED/200 0 CONNECT …:443` - in the access log. +- **What the user sees:** HTTP → **403** (Squid error page, also for teachers). HTTPS → a + **certificate warning**, no block page: squid refuses the CONNECT to the blocked host by + answering it itself, with `200` and then TLS with the instance's untrusted throwaway + certificate and its 403 page inside; the target is never contacted. Someone who clicks + through the warning sees the 403 page, and the proxy sees the path and headers of that + request (it never sees traffic of an allowed site, that is spliced). A blocked name only in + the SNI (behind an allowed CONNECT host) is cut during the TLS handshake: connection error. + ADR-002 (clarification) has the details. In the access log a blocked name shows up as + `TCP_DENIED/403 … GET http://…`, resp. `TCP_DENIED/200 0 CONNECT :443`, followed by + `NONE_NONE/403 … GET https:///…` when the client accepted the certificate + (`NONE_NONE/200 0 CONNECT …` followed by `TCP_TUNNEL/200` is an allowed, spliced connection). - **Hand-editing / category lists:** the file may be edited directly (one domain per line, - leading dot = with subdomains; `#` comments) followed by `lmnsquid blocklist reload`. + leading dot = with subdomains; `#` comments; Windows line endings are fine) followed by + `lmnsquid blocklist reload`. + Squid does not read the mounted file itself: at every start and reload the image's + `lists-sync.sh` builds the list it reads (`/run/lmnsquid/lists/`) and leaves out every entry + a listed parent domain already covers, naming them in `lmnsquid logs ` + (`… entries ignored, already covered by a listed parent domain: …`) and, for a reload, also + in its answer (`"notices": [...]`); so a hand-edited list + cannot stop the proxy. `blocklist-refresh.sh` drops those entries itself. (The image pinned + by 7.3.7 and older reads the mounted file directly: with it such a list makes `reload` fail + and the next start of the container fail.) UT-Capitole category lists: `INSTANCE= BLOCK_CATEGORIES="adult malware phishing" bash /usr/share/linuxmuster-squid/scripts/blocklist-refresh.sh` from a host cron **as user `lmnsquid`** (fail-closed size floor), then `reload`. The refresh **replaces the whole file** — entries added with `lmnsquid blocklist add` are lost, so keep manually curated instances out of the refresh or re-add them afterwards. The directory must stay owned by `lmnsquid` so the API can keep writing (it replaces the file atomically). +- **Large category lists (known limit):** `lists-sync.sh` reads the whole list at every start + and reload (measured in the cold verification of 2026-10-30: 2 million entries ≈ 25 s and ≈ 290 MiB), and squid then parses it + (≈ 55 s for the same list, twice at a start). With lists that size a start can exceed the + 90 s health gate of `update`/`reconcile` and roll back. Keep category selections moderate + and check `lmnsquid status ` after enabling a large one. - **Upgraded from 7.3.0?** Containers created before 7.3.1 run without the mount until recreated once; the postinst runs `lmnsquid reconcile` for that (see Updates). +## Domains without proxy authentication (per instance) + +Each instance can let a list of domains through **without proxy authentication** (ADR-016), +`/etc/linuxmuster-squid/blocklists//noauth.domains` (created empty with the instance), +seen by the container at `/etc/squid/lists/noauth.domains` — the same mounted directory as the +blocklist. Empty = every request needs a ticket, as before. + +``` +lmnsquid noauth all-students add sharepoint.com # -> [".sharepoint.com"]: the domain and all subdomains +lmnsquid noauth all-students list +lmnsquid noauth all-students remove sharepoint.com +lmnsquid noauth all-students reload # squid re-reads both lists: no restart +``` + +> **⚠️ Warning — read before adding anything:** +> - **Exam mode:** exam accounts (and every other device in the school subnets) reach a listed +> domain without a ticket during an exam, e.g. OneDrive, Teams and Outlook with the +> Microsoft 365 list below, although the proxy refuses them everything else. +> - **Internetsperre:** does not apply to listed domains (without identity there is no group +> check); a user removed from the `internet` group still reaches them. +> - **No user in the log:** these requests appear as `-` in the access log. +> - **Only from `--school-subnets`:** from anywhere else a ticket is still required. With the +> default `0.0.0.0/0` that is everyone who can reach the proxy port, so set the real client +> subnets (`lmnsquid edit --school-subnets …`) before using the list. + +- **Rules:** the blocklist wins (a domain on both lists gets **403**; HTTPS is cut). The + existing rules stay in front (Safe_ports, CONNECT only to 443, no IP literals on CONNECT); an + IP-literal URL never matches the list (`dstdomain -n`, no reverse DNS; addresses cannot be + put on it, see below). HTTPS through the + list passes only when the TLS **SNI** is on the list too; a different SNI or a ClientHello + without SNI is **terminated** (no domain fronting into other names; no decryption, ADR-002). + With a ticket nothing changes (200 / 403 / 407 as before). +- `add`/`remove` write the file (same format and rules as the blocklist, one `.domain` per + line, `#` comments allowed when hand-edited) and take effect on the next **`reload`**. + Stricter names than the blocklist: only a domain below a top-level domain, so `add` refuses + IP addresses in any form (IPv6, and every IPv4 notation `inet_aton` reads: `10.0.0.1`, + `127.1`, `0x7f000001`, `0x7f.0x1`, `1.2.3.0x4`; refused is any name whose labels are all + decimal, octal or `0x` numbers) and single labels such as + `com` (they would open an address or a whole TLD); hand-written lines of that kind are + ignored by the container and named in `lmnsquid logs ` and in the answer of + `lmnsquid noauth reload` (`"notices"`). The blocklist accepts both on + purpose: blocking a TLD (`zip`) is a legitimate choice and a wrong entry there only blocks + more. API: + `GET/POST /v1/instances/{name}/noauth`, `DELETE …/noauth/{domain}`, `POST …/noauth/reload`. +- **`reload` answers 409** when the instance's data-plane image does not support the list + (an image from before the list). That is a release matter, not an admin step: a + linuxmuster-squid release whose pinned image supports the list moves the instances onto it + when the package is upgraded (`update-all` in the postinst); `lmnsquid update ` only + helps once such a release is installed (then it moves an instance that stayed on its old + image, e.g. after a rollback). The 409 text says the same. Without that check a reload would report success and + change nothing. + `blocklist reload` is not affected. A `noauth.domains` created after the container started + (an older control plane) is picked up by the next reload. +- **What `lmnsquid rm` removes:** the directory with both lists. +- **Known limits:** + - Names with two labels pass the name check, including public suffixes such as `co.uk`, + `com.de` or `github.io` (anyone can publish under them): adding one opens a whole + registration space without authentication. There is no public-suffix list built in; add + only the organisation's own domains (`sharepoint.com`, not `com.de`). + - `noauth list` shows the file as it is: after a hand edit it can contain lines the + container ignores (an address, a single label, a line with a trailing comment). What squid + really uses is what `reload` and `lmnsquid logs` do not report as ignored. + - Control characters in a hand-written entry appear unescaped in `docker logs` (the reload + answer escapes them), and `/run/lmnsquid/lists-sync.log` (tmpfs) grows by the messages of + each reload until the container restarts (a few hundred bytes per reload). + +### Recommended list: Microsoft 365 + +Microsoft asks to exempt the **required** Microsoft 365 endpoints from proxy authentication and +inspection and does not support allowing only some of them ("Routing connections through +network infrastructure applying its own authentication such as proxy authentication" is listed +among the configurations "known to cause connectivity issues"; Microsoft 365 network +connectivity principles, learn.microsoft.com, updated 2026-08-31). Source of the entries: +the Microsoft 365 endpoints web service, `https://endpoints.office.com/endpoints/worldwide` +(service areas Common, Exchange, SharePoint, Skype/Teams; version `2026103000`, fetched +2026-10-30): every **required** item of category **Optimize** or **Allow** with TCP ports +80/443 (ids 1, 9, 12, 31, 46, 56, 64). `*.x` and `x` are both one entry `x` here (the list +always covers the domain and its subdomains, which is slightly wider than `*.x`). Names under +an entry already listed are left out (`ccs.` and `device.login.microsoftonline.com` under +`login.microsoftonline.com`, `api.passwordreset.microsoftonline.com` under +`passwordreset.microsoftonline.com`): they are covered anyway, `add` would not store them, and +squid 6 refuses a list in which such a name comes before its parent. + +``` +for d in outlook.cloud.microsoft outlook.office.com outlook.office365.com protection.outlook.com \ + lync.com teams.cloud.microsoft teams.microsoft.com sharepoint.com \ + officeapps.live.com online.office.com office.live.com \ + auth.microsoft.com msftidentity.com account.activedirectory.windowsazure.com \ + accounts.accesscontrol.windows.net adminwebservice.microsoftonline.com \ + autologon.microsoftazuread-sso.com clientconfig.microsoftonline-p.net \ + companymanager.microsoftonline.com graph.microsoft.com \ + graph.windows.net login-us.microsoftonline.com login.microsoft.com \ + login.microsoftonline-p.com login.microsoftonline.com login.windows.net \ + logincert.microsoftonline.com loginex.microsoftonline.com nexus.microsoftonline-p.com \ + passwordreset.microsoftonline.com protection.office.com security.microsoft.com \ + compliance.microsoft.com defender.microsoft.com purview.microsoft.com; do + lmnsquid noauth all-students add "$d" >/dev/null +done +lmnsquid noauth all-students list && lmnsquid noauth all-students reload +``` + +- **Not on the list, on purpose:** mail protocols and SMTP (id 10, required: TCP 25; id 2 is + not even required: TCP 143, 587, 993, 995 — none of it HTTP proxy traffic) and Teams media + (id 11: UDP 3478–3481, no URLs; + a forward proxy cannot carry UDP — allow it at the firewall if Teams calls are needed). + Endpoints of category **Default** keep going through the proxy with authentication. +- **The list changes** (Microsoft recommends updating it automatically). Compare from time to + time with `curl -s 'https://endpoints.office.com/version/worldwide?clientrequestid='` + and the endpoints call above; for a client that still fails, look for `TCP_DENIED/407` lines + of Microsoft names in `lmnsquid logs ` before adding anything: every entry widens what + is reachable without identity. + ## Updates (digest-pinned, health auto-rollback) ``` @@ -235,7 +371,7 @@ Access logs show **who visited/was blocked from which site** = personal data ## Backup - `/etc/linuxmuster-squid/config.yml` (API token!), `/etc/linuxmuster-squid/secrets/` (keytabs), - `/etc/linuxmuster-squid/blocklists/` (per-instance blocklists), + `/etc/linuxmuster-squid/blocklists/` (per-instance blocklists and noauth lists), - `instances_dir` (`/var/lib/linuxmuster-squid/instances/*.yaml` — git-versioned = change log; the postinst creates and configures the repo as its owner `lmnsquid`, with git's background maintenance off; every create/edit/update/rm is one commit: diff --git a/docs/references.md b/docs/references.md index 564a16a..23ad187 100644 --- a/docs/references.md +++ b/docs/references.md @@ -106,3 +106,42 @@ research workflows on **2026-07-02** (confidence in parentheses). — https://github.com/containrrr/watchtower · https://docs.renovatebot.com/docker/ - `.deb` reproducible via dh-virtualenv (no pip-in-postinst). (high) — https://dh-virtualenv.readthedocs.io/en/latest/ + +## Domains without proxy authentication (ADR-016, checked 2026-10-30) + +- `dstdomain`: "a reverse lookup is tried if a IP based URL is used and no match is found"; + `-n` disables lookups (mismatch instead). `ssl::server_name`: no DNS lookups; without SNI the + CONNECT target is used instead; `--client-requested` = the SNI. `at_step SslBump2` = after the + ClientHello. Quoted `"file"` = one entry per line. (high) + — https://www.squid-cache.org/Doc/config/acl/ +- `external_acl_type`: FORMAT takes logformat codes, unavailable values expand to `-`, `%DATA` + (the ACL arguments) is `-` without any; replies `OK`/`ERR`/`BH`. `%ssl::>sni` = "SSL client + SNI sent to Squid". (high) — https://www.squid-cache.org/Doc/config/external_acl_type/ · + https://www.squid-cache.org/Doc/config/logformat/ +- `ssl_bump` takes fast and slow ACLs; first match wins; actions peek/splice/terminate. (high) + — https://www.squid-cache.org/Doc/config/ssl_bump/ +- Measured with squid-openssl 6.14 (the image): the helper line is ` `, i.e. `- -` + without SNI; a missing ACL file is an `ERROR` plus an empty ACL, not fatal; without `-n` an + IP-literal URL whose address maps back to a listed name (squid's reverse cache) matched + `dstdomain`; a denied CONNECT on an ssl-bump port gets `200` and squid's 403 page inside TLS + with the throwaway certificate. (measured, E2E) +- Microsoft 365: proxy authentication is among the configurations "known to cause + connectivity issues"; exempt the full set of required endpoints, selective allow-listing is + not supported. (high) — + https://learn.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-network-connectivity-principles + (updated 2026-08-31) · endpoints web service https://endpoints.office.com/endpoints/worldwide + (version 2026103000). + +## Data-plane HTTPS errors and cloud-init (checked 2026-11-02) + +- A CONNECT denied by `http_access` on an ssl-bump port is answered by bumping the client + connection to serve the error page (not described in squid.conf / the SslBump wiki pages; squid + 6 source `src/client_side_request.cc`: "If it applies, we will attempt to bump the client to + serve the error", "Do not bump during authentication"). Measured: `200`, then the 403 page inside + TLS with the static `tls-cert`; the target is not contacted. (high, source + measurement) + — https://github.com/squid-cache/squid/blob/v6/src/client_side_request.cc +- cloud-init `update_etc_hosts`: runs every boot; `manage_etc_hosts: true` re-renders `/etc/hosts` + from the template ("To make modifications persistent across a reboot, you must modify" the + template), `localhost` appends a `127.0.1.1` entry every boot, `false` (default) leaves the file + alone. Ubuntu 24.04 (cloud-init 26.2) uses `/etc/cloud/templates/hosts.debian.tmpl` with + `127.0.1.1 {{fqdn}} {{hostname}}`. (high) — https://docs.cloud-init.io/en/latest/reference/modules.html diff --git a/docs/test-strategy.md b/docs/test-strategy.md index cf9938c..73d5937 100644 --- a/docs/test-strategy.md +++ b/docs/test-strategy.md @@ -78,18 +78,25 @@ from the caller's PATH. ## Heavy tier — Kerberos E2E (Docker) docker-compose stack: `samba-dc` + `squid` + `origin` + `test-client` (details in -the `/test` skill). **Core proof** (P1), assertions on `%{http_code}`: +the `/test` skill), on the fixed network `10.228.0.0/24`: outside Docker's default address +pools (`172.17–31.0.0/16`, `192.168.0.0/16`), which other compose projects on the same host take +one after the other, so the stack does not fail with "Pool overlaps". `run.sh e2e` (and `all`) also runs the admin-script smokes against the +same Samba DC image: `provision_keytab_smoke.sh` (SPN/alias conflicts, idempotence, fail-closed +cases) and `discover_ad_facts_smoke.sh` (groups, realm from the DC, templates). Not in GitHub +CI: the heavy tier stays out of it (`ci.yml` header), and a new job would be a new required +check of the `main` ruleset. **Core proof** (P1), assertions on `%{http_code}`: | Case | Action | Expectation | |---|---|---| | Teacher allowed | `kinit teacher1`; via teacher instance | **200** | | Student denied (authN ok, authZ fail) | `kinit student1`; via teacher instance | **403** (not 407) | | Blocked domain (HTTP) | Teacher → entry from `blocked.domains` | **403** | -| Blocked domain (HTTPS) | Teacher → CONNECT to a blocked name | **not 200**: TLS handshake terminated (curl exit ≠ 0), no 403 page | +| Blocked domain (HTTPS) | Teacher → CONNECT to a blocked name | **not 200** behind the tunnel (`curl -k`): squid answers the CONNECT itself and serves its 403 inside TLS with the throwaway certificate (the noauth case asserts `200 403`; the certificate itself is not asserted, it was checked by hand in the lab) | | No ticket | `kdestroy`; request | **407** | The 403-vs-407 split is the actual proof (authenticated-but-unauthorized -vs. not-authenticated). The compose stack bind-mounts `deploy/e2e/blocked.domains`; the +vs. not-authenticated). The compose stack bind-mounts `deploy/e2e/blocked.domains` (and the +directories `deploy/e2e/lists-noauth/`, `deploy/e2e/lists-legacy/` for the noauth cases); the **product path** (`lmnsquid blocklist add … && … reload` on a managed instance → 403) is proven against a real domain in the lab (7.3.1 campaign fix) and is part of the acceptance list in `deployment-gpo.md`. @@ -97,7 +104,8 @@ acceptance list in `deployment-gpo.md`. ## Negative/security catalog (grows per phase; part of the DoD) - **P1:** no ticket→407; student→403; SPN mismatch/FQDN-as-IP → no 200. -- **P2:** blocked HTTPS domain (SNI/CONNECT)→TLS handshake terminated, never 200, +- **P2:** blocked HTTPS domain: CONNECT host blocked → squid's 403 inside TLS (untrusted + throwaway certificate), blocked SNI → TLS handshake terminated; never 200, **without** client CA; allowed→200; behavior on missing SNI defined. - **P3:** teacher of school A via school-B instance→403; prefixed group names take effect; subnet scope takes effect. @@ -118,6 +126,40 @@ acceptance list in `deployment-gpo.md`. back to `lmnsquid`, history kept; a service the admin disabled (`systemctl disable --now`) before the upgrade from the release stays disabled and stopped across two upgrades in a row, with the "instances not updated" line in the apt output (CI upgrade-smoke). +- **Domains without proxy auth (ADR-016)** (E2E, instance `squid-noauth` with the list + directory mounted like the control plane, `school_net` = the test client only): listed + domain and subdomain without ticket → 200, HTTPS → tunnel + TLS; unlisted → 407 (HTTP and + CONNECT); listed **and** blocked → 403 (HTTP; HTTPS: squid's 403 inside TLS); CONNECT to a + listed name with a foreign SNI, a blocked SNI or **no** SNI → TLS cut (openssl s_client); + IP-literal URL whose address maps back to a listed name → 407 (`dstdomain -n`); request from + outside `school_net` → 407 (HTTP and CONNECT, squidclient from another container); with a + ticket the 200/403/407 matrix is unchanged on that instance. Empty list (`squid`) and a list + directory without `noauth.domains` (`squid2`, as an older control plane mounts it) → 407 as + before, the latter logs a notice. The image from before ADR-016 (`squid-old`, the digest + 7.3.7 pins, a constant that does not follow `DEFAULT_IMAGE`) next to a directory with + `noauth.domains`: unchanged + (407 without ticket, 200/403 with). Counter-runs: without `-n` the IP-literal case and + without the SNI rule the foreign-SNI case fail. `cp-docker-it` (real DockerService): both + list files exist before the container starts, the new image reads the noauth list and + reloads, the image from before ADR-016 is healthy with it and `noauth_active` is false. Unit: name + normalization (IDNA, leading dot, invalid names; IP literals as dotted, short, hex, dotted hex + (`0x7f.0x1`, `1.2.3.0x4`), octal, decimal + and IPv6 forms and single labels refused for noauth, accepted for the blocklist; E2E: such + hand-written lines ignored and logged, `http:///` → 407), unsafe instance names, API 401/403 without + or with a wrong token, 404/422, reload 409 when the container ignores the list or is stopped, + CLI and `--help` warnings. +- **Python and the image agree:** `controlplane/tests/test_lists_sync.py` runs + `image/lists-sync.sh` itself (sh + awk) and `normalize_noauth_domain` on the same cases (every + IPv4 notation, IPv6, single labels, valid names with numeric or hex-looking labels) and + requires the same verdict; the covered-entry rule likewise against `uncovered()`. +- **Lists squid 6 refuses** (`FATAL: Bungled`: a name covered by a listed parent before it, + or the bare apex next to it), both lists: unit (`uncovered`, `add` keeps the parent and + drops what it covers, a hand-edited file is cleaned on the next write), blocklist smoke + (`blocklist-refresh.sh` drops `a.bad3.example` under `bad3.example`), E2E (`squid-noauth` + mounts such hand-edited lists: starts, logs the ignored entries, reloads, comes back after a + restart, blocked subdomain → 403), cp-docker-it (real DockerService: such lists written after + the start → reload ok → restart → healthy; counter-check with an image whose `lists-sync.sh` + only copies: the reload fails with `Bungled`). - **P10:** keytab perms; manager ACL not reachable externally; API bind ≠ 0.0.0.0; bypass/traversal; DC outage does not stall (ttl/grace). diff --git a/docs/threat-model.md b/docs/threat-model.md index 7ae519f..78a34dd 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -24,18 +24,19 @@ Risks + countermeasures. Grows per roadmap phase; the negative tests for it live | T1 | **Keytab theft** | Secret (tmpfs, `:ro`), `0600` host, readable only by `proxy`, isolated per instance, never in env/log; Compose file secrets not encrypted at rest → harden the host | perms test; no keytab in `docker inspect`/logs | | T2 | **Proxy bypass** (non-inline) | OPNsense blocks direct 80/443 egress (force-proxy); client subnets = `SCHOOL_SUBNETS` | E2E + firewall review | | T3 | **Authorization bypass** (student on the teacher proxy, wrong group assignment) | group ACL enforced server-side; GPO groups == ACL groups; prefix rule correct | E2E: student→403; multischool matrix | -| T4 | **HTTPS privacy** | no SSL-bump MITM; only SNI-splice/CONNECT; peek CA never distributed | `squid.conf` review; no `bump` verdict | +| T4 | **HTTPS privacy** | no SSL-bump MITM of traffic to a real target: allowed CONNECTs are spliced (SNI peek, CONNECT `dstdomain`); the peek certificate is never distributed or trusted. A CONNECT denied by `http_access` gets squid's own 403 inside TLS with that untrusted throwaway certificate (squid's error delivery on an ssl-bump port; the target is never contacted); a client that clicks through the certificate warning reveals the path and headers of that denied request to the proxy (ADR-002, clarification) | `squid.conf` review (`ssl_bump` only peek/splice/terminate); E2E: denied HTTPS → not 200 behind the tunnel (`curl -k`; the noauth case asserts `200 403`), allowed → spliced; the throwaway certificate itself is not asserted, only checked by hand in the lab (2026-10-30) | | T5 | **Control-plane RCE = host root** (Docker socket) | API **only `127.0.0.1`** + token (`compare_digest`), hardened systemd service, audit log. The socket proxy reduces the endpoint surface, **but does NOT downgrade below host root** (`container create` with host bind is inherently root-equivalent) → the real answer is **rootless Docker**. In-app TLS not implemented; off-host only via an operator-provided TLS reverse proxy (`main.py` warns on non-loopback bind) | API 401/403 tests; bind check ≠ 0.0.0.0 | | T6 | **Auth all-or-nothing / SSO outage** | auth never globally disableable by default; fallback Kerberos→Basic/LDAP (no NTLM); monitor SSO health | E2E: no ticket→407 | | T7 | **DNS/SPN misconfig** (silent Kerberos fail) | fwd+rev DNS resp. `rdns=false`; FQDN==SPN==`VISIBLE_HOSTNAME`; NTP<5min; AES enctypes | E2E canonicalization; `klist -k` | -| T8 | **Blocklist rot / tampering** (supply chain) | per-instance list on the host (`/etc/linuxmuster-squid/blocklists//blocked.domains`, mounted read-only into the container, writable only by root/`lmnsquid`), maintained via `lmnsquid blocklist` + `reload`; category lists via `blocklist-refresh.sh` from a host cron with a fail-closed size floor (a truncated download never replaces the list); make outage visible. **HTTPS limit:** a blocked HTTPS name is cut at the TLS handshake (peek + terminate, no decryption) — the user sees a connection error, no block page; HTTP gets 403 | blocklist smoke; lab proof 7.3.1 (HTTP 403, HTTPS TLS abort) | +| T8 | **Blocklist rot / tampering** (supply chain) | per-instance list on the host (`/etc/linuxmuster-squid/blocklists//blocked.domains`, mounted read-only into the container, writable only by root/`lmnsquid`), maintained via `lmnsquid blocklist` + `reload`; category lists via `blocklist-refresh.sh` from a host cron with a fail-closed size floor (a truncated download never replaces the list); make outage visible. A list squid 6 cannot load (a covered name before its parent: `FATAL: Bungled`) would take the instance down at the next start: squid reads a copy without such entries (`lists-sync.sh`), and neither the API nor the refresh writes them. **HTTPS limit:** no usable block page for HTTPS: a blocked CONNECT host gets squid's 403 inside TLS with an untrusted throwaway certificate (browser: certificate warning; the target is never contacted), a blocked SNI behind an allowed CONNECT host is cut during the handshake (`ssl_bump terminate`); HTTP gets 403 | blocklist smoke; lab proof 7.3.1 (HTTP 403, HTTPS no block page), re-measured 2026-10-30: blocked CONNECT host → `200`, then 403 inside TLS (E2E: `200 403`), with the throwaway certificate (checked by hand in the lab, not asserted) | | T9 | **Unattended bad update → dead school** | digest pin, human-merge (no automerge; bumps by hand while Renovate is disabled), health-gated update + auto-rollback, known-good persisted | E2E: bad image → rollback | | T10 | **DC/LDAP outage stalls every request** | external ACL `ttl/negative_ttl/grace` + tune helper concurrency | load test P10 | | T11 | **Container escape/privilege** | `read_only` rootfs + tmpfs, `cap_drop: ALL` (minimal caps), non-root `proxy`, manager ACL localhost | hardening review P10 | -| T12 | **Exam mode** | `-exam` in no teachers/students group → ACL denies; lmn7 disables the proxy in exam mode | docs + optional test | +| T12 | **Exam mode** | `-exam` in no teachers/students group → ACL denies; lmn7 disables the proxy in exam mode. **Not** for domains on the noauth list (T16) | docs + optional test | | T13 | **Access logs = personal data** (browsing behavior, GDPR) | retention = `log_retention_days` (documented deletion period, default 30), gzip-rotated; access logging disableable per instance (`access_log_enabled:false`); log access only via API token, queries into the audit log; no secrets in the log; `lmnsquid rm` deletes the log volume with the instance (`--keep-logs` only on request) | retention/rotation smoke; `access_log none` render test; rm test | | T14 | **SNI filter bypass: ECH / QUIC / DoH** | **ECH** (Encrypted Client Hello) encrypts the SNI → splice filter blind; **QUIC/HTTP3** runs over **UDP 443**, bypassing the TCP forward proxy; **DoH** circumvents the DNS view. Limit of name-based filtering — the proxy alone does not close this. **Mitigate at the network edge:** OPNsense **block UDP 443** (forces TCP/443 through the proxy), block known DoH resolvers + `use-application-dns.net` (canary disables Firefox DoH); observe ECH adoption. For real completeness you would need SSL interception (a deliberate non-goal, ADR-002). | firewall review at the site; documented limit | | T15 | **Poisoned build input** (supply chain of the `.deb`, installed as root) — a compromised or silently changed PyPI release, build image or GitHub Action lands in the package | Python deps (pip included) only from `controlplane/requirements.lock` with SHA-256 hashes (`--require-hashes --no-deps --only-binary :all:`, wheels only), nothing younger than 7 days picked (`--exclude-newer=P7D`), own wheel built without build isolation from a hashed setuptools; build container, data-plane base image and actions pinned by digest/commit SHA; changes only via human-merged PRs (raised by hand while Renovate is disabled); releases published draft-first (immutable releases) with the asset hash checked against the build (ADR-015) | `lock-deps.sh --gate` first in the fast tier AND in every build, before anything from a lock is installed or run (no venv `bin/` on PATH; `/usr/bin/python3 -I`; every entry point restarts itself under `env -i` through `/bin/bash -p` with an allowlist, `packaging/clean-env.sh`: fixed PATH and `LC_ALL`, the caller's `HOME`, `TMPDIR`, proxies, CA bundles (`SSL_CERT_FILE`, `SSL_CERT_DIR`, `REQUESTS_CA_BUNDLE`, `PIP_CERT`) and the repository's own switches, nothing else; not undone, because it ran before the restart: the caller's first bash for `bash