diff --git a/README.md b/README.md index 378e722f..17b6edab 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ documents for different readers. at all. Every pinned digest is also scanned weekly for fixable HIGH and CRITICAL CVEs, and each image with findings has an open issue until a scan finds it clean. -- **Documented decisions and runbooks.** 115 ADRs covering what was chosen +- **Documented decisions and runbooks.** 116 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 53 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. Every critical alert links to @@ -235,7 +235,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md changelog.md │ ├── diagrams/ # the network diagram (SVG) and its predecessors -│ ├── adr/ # 115 ADRs — architecture decision records +│ ├── adr/ # 116 ADRs — architecture decision records │ ├── exercises/ # purple-team exercise records — see its README │ └── runbooks/ # 53 runbooks; successor-handover.md is the front door └── Makefile # make help diff --git a/docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md b/docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md index 4215b289..52be08cb 100644 --- a/docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md +++ b/docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md @@ -2,6 +2,13 @@ **Status:** Accepted · 2026-10 +> [!NOTE] +> Amended by [ADR-0116](0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md): +> the provider gains an alias for `gandalf`, read from `phoenix.env`, and the +> range's attack VM is declared in this tree with a script that finishes and +> proves it after every apply. The text here is left as written, per ADR-0001 +> and ADR-0115. + ## Context [ADR-0074](0074-build-the-lab-templates-with-packer-from-phoenix.md) makes diff --git a/docs/adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md b/docs/adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md new file mode 100644 index 00000000..aaa2dccd --- /dev/null +++ b/docs/adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md @@ -0,0 +1,109 @@ +# ADR-0116: Build the range's attack VM with OpenTofu, and prove it with a script + +**Status:** Accepted · 2026-11 · amends [ADR-0076](0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md) + +## Context + +[ADR-0076](0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md) +put "#421's attack VM and targets" in OpenTofu's scope, and expected the range +to be "a second node's guests in this tree rather than a second tree". When +the range was built on `gandalf` +([ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), +[ADR-0106](0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)), +its attack VM, `ragnarok`, was made by `scripts/make-attack-vm.sh` instead, +outside the pipeline. The 2026-11-11 audit found that, and the owner chose to +bring it into `tofu/` +([#1443](https://github.com/Gerrrt/HomeLab/issues/1443)). + +Two things stood in the way. + +- **The tree had one provider, read from the environment.** + `PROXMOX_VE_ENDPOINT` and `PROXMOX_VE_API_TOKEN` in `phoenix.env` name + `Saruman`. `gandalf.env` names `gandalf`, but in the curl scripts' own + variables (`PROXMOX_URL`, `PROXMOX_TOKEN_ID`, `PROXMOX_TOKEN_SECRET`). +- **`ragnarok` is more than a clone.** Its isolation is in three parts: + - `net1` on `vmbr1` with no gateway; + - forwarding off, persistently, in `99-no-forward.conf`; + - a proof after every make that forwarding is off and nothing NATs. + + That proof runs over SSH, against a host key read through the guest agent + ([ADR-0085](0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md)). + +## Decision + +**One tree and one state, with an aliased provider for `gandalf`.** +`provider "proxmox" { alias = "gandalf" }` takes its endpoint and token from +`TF_VAR_gandalf_endpoint` and `TF_VAR_gandalf_api_token`, two new lines in +`phoenix.env`. Their names collide with nothing there, so every runbook step +that sources `phoenix.env` before a plan works unchanged. `gandalf.env`, which +overrides the curl scripts' `PROXMOX_*` names, is not needed for a plan. The +token is then in two mode-600 files on `phoenix` rather than one. A provider's +variables are not written to state. TLS is verified against `gandalf`'s CA, which `phoenix` +already trusts. + +**OpenTofu makes the VM.** `ragnarok` is `module.range_guest` in +`tofu/guests.tf`: the same guest module as every other lab guest, called with +the `gandalf` provider. It is a full clone of 902 on `range` with: + +- `net0` on `vmbr0`, DHCP, its MAC pinned; +- `net1` on `vmbr1` at `172.30.30.10/24` with no gateway, through the module's + new `extra_networks`; +- both firewall flags off; +- cloud-init user `kali` with `phoenix`'s key. + +It is in no pool, because the token on `gandalf` holds no pool grant and this +keeps it so. + +**A script finishes and proves it, after every apply.** +`scripts/make-attack-vm.sh` becomes `scripts/finish-attack-vm.sh`. It no +longer clones or configures, and refuses to run if 130 does not exist. It: + +- reads the config back, including `net1`'s address and the absent gateway; +- starts the guest if it is stopped; +- writes `99-no-forward.conf` over the host-key-checked SSH; +- runs §8's third check. + +The sysctl file stays in the script rather than in cloud-init. A cloud-init +snippet would have tofu upload it to `gandalf` over SSH, a second credential +path into the range host for one file. + +**The range is still rebuilt, not repaired.** ADR-0017's rule is unchanged. +What changes is the tool: a change to `ragnarok` is a destroy and apply of +130, then `finish-attack-vm.sh`. + +This amends ADR-0076 in two places. Its single, environment-read provider +gains an alias. Its expectation that the range is "a second node's guests in +this tree" is carried out, for the attack VM. + +## Alternatives considered + +- **A second tree for the range,** with its own state and its own passphrase + to escrow, and the provider's environment variables from `gandalf.env`. A + `Saruman` plan would never need `gandalf` powered on. Declined by the owner + for one state and one plan, at the cost below. +- **All of it in cloud-init,** the sysctl file included, with the script + reduced to the proof. Declined for the credential it needs, as above. +- **Leave `ragnarok` script-built,** and supersede that part of ADR-0076's + scope. Declined: the guest the range's isolation depends on would be the + one guest the pipeline cannot rebuild. + +## Consequences + +- **Every plan reads `gandalf`.** `gandalf` must be up, even for a plan that + touches only `Saruman`. `gandalf` is off between sessions (ADR-0017), so a `Saruman`-only + change waits for it, or is planned with `-target`. +- **The script's §8 proof still runs after every make.** An apply that is not + followed by `finish-attack-vm.sh` leaves a VM whose forwarding has not been + turned off. `build-the-playground.md` §5 says so, in that order. +- **The `/vms/130` grant goes with every destroy.** It was already the case + with the script, and the script still checks for it first. +- **The first apply replaces the script-built `ragnarok`.** Importing a guest + that tofu did not clone plans a replace, because `clone` cannot be read back + ([`build-the-lab-domain.md`](../runbooks/build-the-lab-domain.md)). The + range is disposable, so the runbook destroys 130 and applies, and does not + import. +- **Reopened by:** + - the targets of #421 joining the tree, which would make the range a larger + share of every plan; + - `gandalf` gaining the snippet storage and SSH access that a cloud-init + sysctl file would need, for some other reason. diff --git a/docs/architecture.md b/docs/architecture.md index bbf8a1da..9856f4d3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -186,7 +186,7 @@ hole from the monitoring VLAN into the monitored one. | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, speedtest-tracker, docker-socket-proxy, Alloy | | `Saruman` (10.0.30.20) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | -| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). The attack VM `ragnarok` (VMID 130), made 2026-11-09 by [`scripts/make-attack-vm.sh`](../scripts/make-attack-vm.sh) from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. Three targets, VMIDs 100–102 on `vmbr1` only, made 2026-11-10; the list is [`build-the-playground.md`](runbooks/build-the-playground.md) §6's. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) | +| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). The attack VM `ragnarok` (VMID 130), declared in [`tofu/`](../tofu/README.md) on `gandalf`'s aliased provider and finished by [`scripts/finish-attack-vm.sh`](../scripts/finish-attack-vm.sh) ([ADR-0116](adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)), from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. Three targets, VMIDs 100–102 on `vmbr1` only, made 2026-11-10; the list is [`build-the-playground.md`](runbooks/build-the-playground.md) §6's. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy and its Docker socket proxy, and Caddy as the ingest proxy in front of the first two ([#834](https://github.com/Gerrrt/HomeLab/issues/834)) — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `fenrir` (10.0.30.90) | 🟢 30 | [`stacks/sensor`](../stacks/sensor) | Zeek on a `tc` mirror of `Saruman`'s lab bridge, so that the domain's east-west traffic, which crosses no router and so never reaches Suricata on `morpheus`, is observed at all ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)). A guest on `Saruman`, VMID 190, with a second NIC alone on the port-less `vmbr1` that receives only the mirror's copies. Its logs go to `alexander`, never to VLAN 99; whether the mirror is delivering is answered on the hypervisor by `homelab_zeek_mirror_active`. Built 2026-09-30 by [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md), and proved by rebooting `Saruman` with the mirror's timer disabled: the gauge fell to 0 and `ZeekMirrorInactive` fired. Alloy agent (Docker) | diff --git a/docs/runbooks/build-the-lab-templates.md b/docs/runbooks/build-the-lab-templates.md index b286917a..d642656a 100644 --- a/docs/runbooks/build-the-lab-templates.md +++ b/docs/runbooks/build-the-lab-templates.md @@ -532,7 +532,7 @@ and 910 with this section and §5b to clear it. | `PhoenixBuilder` | `/storage/range` | the template's disks, on the range pool ([ADR-0096](../adr/0096-fit-ifrits-range-disk-as-a-sata-ssd-in-the-tinys-bay.md)) | | `PhoenixBuilder` | `/storage/local` | the installer ISO | | `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr0` | the build NIC | - | `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr1` | `ragnarok`'s range NIC, which [`make-attack-vm.sh`](../../scripts/make-attack-vm.sh) attaches ([`build-the-playground.md`](build-the-playground.md) §5). Kept 2026-11-09 | + | `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr1` | `ragnarok`'s range NIC, which `tofu/` attaches ([ADR-0116](../adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)) ([`build-the-playground.md`](build-the-playground.md) §5). Kept 2026-11-09 | | `PVEAuditor` | `/nodes/gandalf` | read-only. Lower case, as `pvesh get /nodes` prints it | Nothing is granted on `local-lvm`, because the range's disks stay off the diff --git a/docs/runbooks/build-the-playground.md b/docs/runbooks/build-the-playground.md index b0b6ce33..b3d11e6d 100644 --- a/docs/runbooks/build-the-playground.md +++ b/docs/runbooks/build-the-playground.md @@ -588,51 +588,84 @@ has never fired is a rule nobody knows can. One VM, and it is the only guest with a leg on both bridges. -> **Made on `gandalf` 2026-11-09** ([#421](https://github.com/Gerrrt/HomeLab/issues/421)): -> **`ragnarok`, VMID 130**, a full clone of `tpl-kali` (902). 902 was built -> on `gandalf` from `phoenix` the same night -> ([`build-the-lab-templates.md`](build-the-lab-templates.md) §5, -> [#790](https://github.com/Gerrrt/HomeLab/issues/790)). Everything below is -> one command on `phoenix`: -> -> ```bash -> set -a; . ~/.config/proxmox/gandalf.env; set +a -> scripts/make-attack-vm.sh -> ``` -> -> Once, on `gandalf` as root, and again after any destroy of 130, which -> drops the VMID's ACLs (the script checks before it clones): -> -> ```bash -> pveum role add PhoenixHostKeys --privs "VM.GuestAgent.FileRead" -> pveum acl modify /vms/130 --users phoenix@pve --roles PhoenixBuilder,PhoenixHostKeys -> ``` -> -> The script reads `ragnarok`'s ed25519 host key through the guest agent -> once cloud-init has finished, and its SSH accepts that key and no other, -> as [ADR-0085](../adr/0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md) -> does for the lab domain: the check below crosses VLAN 30, and a PASS -> from whatever answered for the lease would prove nothing. The grant is on -> 130 alone, not on `/vms`, where FileRead would read any range guest. A -> grant on `/vms/130` replaces the inherited `PhoenixBuilder` there, so it -> names both roles. -> -> [`make-attack-vm.sh`](../../scripts/make-attack-vm.sh) clones onto `range` and sets -> 4 cores and 8 GiB. ADR-0017's "roughly half" of 32 GB counted BloodHound, -> which [ADR-0081](../adr/0081-run-bloodhound-ce-on-a-saruman-guest.md) moved -> to `eden`. It gives `net0` on `vmbr0` DHCP and `net1` on `vmbr1` -> `172.30.30.10/24` with no gateway, both with the guest firewall flag off. It -> reads the config back, then writes `99-no-forward.conf` over SSH and runs -> §8's third check. It refuses to run if VMID 130 exists, because the range is -> rebuilt, not repaired. On its first run, `ragnarok` took `10.0.30.111`. Both -> forwarding sysctls read `0`, neither table had NAT, and the one default -> route was via `10.0.30.1` on `eth0`. `eth1` held `172.30.30.10/24` and -> nothing else. Its cloud-init user is `kali`, with `phoenix`'s key. That -> first run predates the host-key check and the stricter NAT reads. Once the -> grant above was made, the script from the key read on was run against -> `ragnarok` the same night and passed. An impostor key was refused, once -> the script turned off SSH connection sharing (`ControlPath=none`): -> without that, a reused connection skips the check. +**Made by OpenTofu, finished and proved by a script** +([ADR-0116](../adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)). +`ragnarok`, VMID 130, is `module.range_guest` in +[`tofu/guests.tf`](../../tofu/guests.tf): a full clone of `tpl-kali` (902) +on `range`, 4 cores and 8 GiB, `net0` on `vmbr0` with DHCP and its MAC +pinned, `net1` on `vmbr1` at `172.30.30.10/24` with **no gateway**, both +firewall flags off, cloud-init user `kali` with `phoenix`'s key, and in no +pool. ADR-0017's "roughly half" of 32 GB counted BloodHound, which +[ADR-0081](../adr/0081-run-bloodhound-ce-on-a-saruman-guest.md) moved to +`eden`. + +**Once, on `gandalf` as root, and again after any destroy of 130,** which +drops the VMID's ACLs: + +```bash +pveum role add PhoenixHostKeys --privs "VM.GuestAgent.FileRead" # once; it exists after the first time +pveum acl modify /vms/130 --users phoenix@pve --roles PhoenixBuilder,PhoenixHostKeys +``` + +The grant is on 130 alone, not on `/vms`, where FileRead would read any range +guest. A grant on `/vms/130` replaces the inherited `PhoenixBuilder` there, so +it names both roles. The ACL can be set while 130 does not exist. + +**Once, on `phoenix`,** give `phoenix.env` the two lines the aliased provider +reads. They go there, not in `gandalf.env`, so that every step that sources +`phoenix.env` before a plan keeps working. Then put `ragnarok`'s MAC in +`tofu/identities.auto.tfvars` (any free `BC:24:11` address), then escrow it +([`provision-lab-guests.md`](provision-lab-guests.md) §3): + +```bash +( umask 077; set -a; . ~/.config/proxmox/gandalf.env; set +a + printf 'TF_VAR_gandalf_endpoint=https://10.0.30.30:8006/\nTF_VAR_gandalf_api_token=%s=%s\n' \ + "${PROXMOX_TOKEN_ID}" "${PROXMOX_TOKEN_SECRET}" >> ~/.config/proxmox/phoenix.env ) +``` + +**Make it, then finish it.** Every plan reads `gandalf` as well as `Saruman`, +so have `gandalf` up. The script reads `gandalf.env`, so source it after +`phoenix.env`: + +```bash +cd ~/code/Gerrrt/HomeLab +umask 077; set -a; . ~/.config/proxmox/phoenix.env; . ~/.config/proxmox/gandalf.env; set +a +tofu -chdir=tofu plan -target='module.range_guest["ragnarok"]' -out=ragnarok.tfplan +tofu -chdir=tofu apply ragnarok.tfplan && rm tofu/ragnarok.tfplan +scripts/finish-attack-vm.sh +``` + +**The apply is not the end.** Until +[`finish-attack-vm.sh`](../../scripts/finish-attack-vm.sh) has run, the guest +has not had forwarding turned off. The script: + +- reads the config back (`net1` on `vmbr1` at `.10/24` with no gateway, `net0` + on `vmbr0`); +- starts the guest if it is stopped; +- reads its ed25519 host key through the guest agent once cloud-init has + finished, and its SSH accepts that key and no other, as + [ADR-0085](../adr/0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md) + does for the lab domain (the check crosses VLAN 30, and a PASS from + whatever answered for the lease would prove nothing); +- writes `99-no-forward.conf` and runs §8's third check. + +The range is rebuilt, not repaired: a change to `ragnarok` is a destroy and +apply of 130, then the script. + +**The script-built `ragnarok` is replaced, not imported.** It was made by +`make-attack-vm.sh` on 2026-11-09, outside the pipeline. Importing it would +plan a replace anyway, because `clone` cannot be read back +([`build-the-lab-domain.md`](build-the-lab-domain.md)), and the range is +disposable. So, once, as root on `gandalf`: `qm stop 130` and +`qm destroy 130 --purge`. Then grant `/vms/130` again as above, and make it as +above. + +> **As run on 2026-11-09, by the earlier script.** `ragnarok` took +> `10.0.30.111`. Both forwarding sysctls read `0`, neither table had NAT, and +> the one default route was via `10.0.30.1` on `eth0`. `eth1` held +> `172.30.30.10/24` and nothing else. An impostor host key was refused, once +> the script turned off SSH connection sharing (`ControlPath=none`): without +> that, a reused connection skips the check. - **Its disk on `range`**, the SATA SSD's pool from §1, like every guest here. - **`net0` on `vmbr0`** — ImaginationLAN, DHCP from the pool. Nothing points at diff --git a/docs/runbooks/provision-lab-guests.md b/docs/runbooks/provision-lab-guests.md index d8067001..b4fcd9bb 100644 --- a/docs/runbooks/provision-lab-guests.md +++ b/docs/runbooks/provision-lab-guests.md @@ -97,6 +97,14 @@ else fi ``` +**Two more lines, for `gandalf`.** Since +[ADR-0116](../adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md), +the tree also reads `gandalf` through an aliased provider, from +`TF_VAR_gandalf_endpoint` and `TF_VAR_gandalf_api_token`. They go in this file +too, so a plan needs `phoenix.env` alone, and every plan needs `gandalf` up. +[`build-the-playground.md` §5](build-the-playground.md#5-the-attack-vm) +writes them. + `passphrase length 44` and `API 200`. Then `rm` the `.bak-445` copy, which holds the token. A full root makes the copy fail. The first run hit exactly that, and the build's §2 now grows the volume (`build-the-jumpbox.md`). A diff --git a/packer/kali.pkr.hcl b/packer/kali.pkr.hcl index 20044d38..daf80865 100644 --- a/packer/kali.pkr.hcl +++ b/packer/kali.pkr.hcl @@ -5,7 +5,8 @@ # BUILT on gandalf 2026-11-09, twice, each build smoke-tested (#790). It goes # through gandalf's own API with gandalf's own token (gandalf.env, not # phoenix.env), and its disk on `range`: build-the-lab-templates.md §5. The -# attack VM, ragnarok, is cloned from it by scripts/make-attack-vm.sh. +# attack VM, ragnarok, is cloned from it by tofu/ and finished by +# scripts/finish-attack-vm.sh (ADR-0116). # # The one source here that uses Packer's HTTP server. Debian's installer reads # a preseed from the install medium or from a URL, not from a second disc, so diff --git a/scripts/README.md b/scripts/README.md index aa72c35b..9b391f28 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -160,7 +160,7 @@ They need the running estate, so CI cannot run them. Timers do. | `export-dashboards.sh`, `export_dashboards.py` | `dashboards-export` | Folds UI edits in the running Grafana back over the committed JSON | | `capture-screenshots.sh` | `screenshots` | Renders the dashboards into [`docs/images/`](../docs/images/README.md) | | `packer-smoke.sh` | — | Clones a Packer template, boots it, checks it and destroys it ([`packer/`](../packer/README.md)) | -| `make-attack-vm.sh` | — | Makes the range's attack VM, `ragnarok`, on `gandalf` from 902, and proves it neither forwards nor NATs, over SSH checked against a host key read through the guest agent ([`build-the-playground.md`](../docs/runbooks/build-the-playground.md) §5) | +| `finish-attack-vm.sh` | — | Finishes the range's attack VM, `ragnarok`, after `tofu/` makes it on `gandalf` from 902 ([ADR-0116](../docs/adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)): writes `99-no-forward.conf` and proves it neither forwards nor NATs, over SSH checked against a host key read through the guest agent ([`build-the-playground.md`](../docs/runbooks/build-the-playground.md) §5) | | `stage-agent-msis.sh` | — | Stages the Wazuh and Velociraptor MSIs the lab pipeline installs, and the two Debian packages `garuda` installs, into `${SOC_DATA_DIR}/agent-msi` on `odin`, which the SOC stack's Caddy serves. The vendor packages are checked against their pins; the Velociraptor `.deb` is built by the server and its hash printed ([#1068](https://github.com/Gerrrt/HomeLab/issues/1068), [ADR-0092](../docs/adr/0092-enrol-garuda-in-odins-soc-with-debian-packages-staged-on-odin.md)) | | `lab-known-hosts.sh` | — | Reads the lab domain's SSH host keys through the guest agent into `ansible/.known_hosts`, after any rebuild ([ADR-0085](../docs/adr/0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md)) | | `gen_population.py` | — | The lab domain's users, into `ansible/population/` | diff --git a/scripts/make-attack-vm.sh b/scripts/finish-attack-vm.sh similarity index 76% rename from scripts/make-attack-vm.sh rename to scripts/finish-attack-vm.sh index 50be44f2..68e13221 100755 --- a/scripts/make-attack-vm.sh +++ b/scripts/finish-attack-vm.sh @@ -1,21 +1,24 @@ #!/usr/bin/env bash # -# Make the range's attack VM, ragnarok, on gandalf from tpl-kali (902), and -# prove it does not forward (#421, build-the-playground.md §5). +# Finish the range's attack VM, ragnarok, after tofu has made it on gandalf, +# and prove it does not forward (#421, build-the-playground.md §5, ADR-0116). # -# WHY A SCRIPT. The range is rebuilt, not repaired (ADR-0017), so the one guest -# on it worth configuring carefully is one command to make again. The -# isolation it depends on is in what this sets, and §8's third check runs at -# the end of every make rather than when someone remembers to. -# -# WHAT IT MAKES. A full clone of 902 at VMID 130, its disk on `range`: -# net0 vmbr0, ImaginationLAN, DHCP from the pool. No reservation: nothing -# points at it (§5). +# WHAT TOFU MAKES (tofu/guests.tf, module.range_guest). A full clone of 902 +# (tpl-kali) at VMID 130 on `range`, with: +# net0 vmbr0, ImaginationLAN, DHCP from the pool. Its MAC is pinned, but +# nothing points at it, so morpheus holds no reservation (§5). # net1 vmbr1, the range bridge, 172.30.30.10/24 and NO gateway. -# Both with the guest firewall flag off (§4's last step). Cloud-init user -# `kali`, phoenix's key. Then, over SSH: /etc/sysctl.d/99-no-forward.conf, +# Both with the guest firewall flag off. Cloud-init user `kali`, phoenix's key. +# +# WHAT THIS DOES, AFTER EVERY APPLY. It reads the config back, because the one +# thing a mistake there would break is the isolation. It starts the guest if it +# is stopped. Then, over SSH: /etc/sysctl.d/99-no-forward.conf, # `sysctl --system`, and §8 check three: both forwarding sysctls 0 and no NAT -# in nft or legacy iptables. +# in nft or legacy iptables. Writing the sysctl file stays here rather than in +# cloud-init, because a cloud-init snippet would need tofu to upload it to +# gandalf over SSH, a second credential path for one file (ADR-0116). The +# range is still rebuilt, not repaired (ADR-0017): a change is a tofu +# destroy and apply of 130, then this. # # HOST KEY. The SSH that reports §8 check three crosses VLAN 30, where ARP # spoofing is expected (docs/security.md), so the key ragnarok presents is @@ -29,11 +32,11 @@ # (build-the-lab-templates.md §5), curl, jq and ssh-keygen, and the token # holding VM.GuestAgent.FileRead on /vms/130 alone (PhoenixHostKeys, # build-the-playground.md §5). Destroying the VM drops that ACL, so it is -# checked before anything is cloned. +# checked first. # # Usage: # set -a; . ~/.config/proxmox/gandalf.env; set +a -# scripts/make-attack-vm.sh +# scripts/finish-attack-vm.sh set -euo pipefail die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } @@ -41,7 +44,6 @@ info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } ok() { printf '\033[0;32m PASS\033[0m %s\n' "$*"; } NODE=gandalf -TEMPLATE=902 VMID=130 NAME=ragnarok RANGE_ADDR=172.30.30.10/24 @@ -75,48 +77,30 @@ wait_task() { die "task ${upid} still running after 30 minutes" } -api GET "/nodes/${NODE}/qemu/${TEMPLATE}/config" | jq -e '.data.template == 1' >/dev/null \ - || die "VM ${TEMPLATE} on ${NODE} is not a template: build tpl-kali first" -if api GET "/nodes/${NODE}/qemu/${VMID}/status/current" >/dev/null 2>&1; then - die "VMID ${VMID} already exists on ${NODE}. The range is rebuilt, not repaired: destroy it first" -fi +api GET "/nodes/${NODE}/qemu/${VMID}/status/current" >/dev/null 2>&1 \ + || die "VMID ${VMID} does not exist on ${NODE}: tofu makes it (build-the-playground.md §5), then this runs" # The host-key read needs FileRead on this VMID, and a destroy removes the -# VMID's ACLs. Better to stop here than after a ten-minute clone. +# VMID's ACLs, so a fresh apply of 130 needs the grant again. Stop here, not +# after waiting ten minutes for cloud-init. api GET /access/permissions -G --data-urlencode "path=/vms/${VMID}" \ | jq -e --arg p "/vms/${VMID}" '.data[$p]["VM.GuestAgent.FileRead"] == 1' >/dev/null \ || die "${PROXMOX_TOKEN_ID} lacks VM.GuestAgent.FileRead on /vms/${VMID}. On ${NODE} as root: pveum acl modify /vms/${VMID} --users phoenix@pve --roles PhoenixBuilder,PhoenixHostKeys (build-the-playground.md §5)" -info "full clone ${TEMPLATE} -> ${VMID} (${NAME}) on range" -wait_task "$(api POST "/nodes/${NODE}/qemu/${TEMPLATE}/clone" \ - --data-urlencode "newid=${VMID}" --data-urlencode "name=${NAME}" \ - --data-urlencode "full=1" --data-urlencode "storage=range" | jq -r '.data')" -ok "cloned" - -# sshkeys is URL-encoded inside the form value, as packer-smoke.sh does. -keys="$(jq -rn --arg k "$(cat "${SSH_KEY}.pub")" '$k | @uri')" -api PUT "/nodes/${NODE}/qemu/${VMID}/config" \ - --data-urlencode "cores=4" --data-urlencode "memory=8192" \ - --data-urlencode "net0=virtio,bridge=vmbr0,firewall=0" \ - --data-urlencode "net1=virtio,bridge=vmbr1,firewall=0" \ - --data-urlencode "ipconfig0=ip=dhcp" \ - --data-urlencode "ipconfig1=ip=${RANGE_ADDR}" \ - --data-urlencode "ciuser=kali" --data-urlencode "sshkeys=${keys}" \ - --data-urlencode "tags=range;attack" \ - --data-urlencode "description=The range's attack VM (#421, build-the-playground.md §5). Made by scripts/make-attack-vm.sh." \ - >/dev/null -ok "net0 vmbr0 DHCP, net1 vmbr1 ${RANGE_ADDR} no gateway, firewall flags off" - -# The one thing a typo here would break is the isolation, so read it back. +# The one thing a mistake in tofu/guests.tf would break is the isolation, so +# read it back before anything else. cfg="$(api GET "/nodes/${NODE}/qemu/${VMID}/config")" jq -e '(.data.net1 | test("bridge=vmbr1")) and (.data.net0 | test("bridge=vmbr0")) + and ((.data.ipconfig1 // "") | test("ip=172\\.30\\.30\\.10/24")) and ((.data.ipconfig1 // "") | test("gw") | not)' <<<"${cfg}" >/dev/null \ || die "the config did not read back as written: $(jq -c '.data | {net0,net1,ipconfig0,ipconfig1}' <<<"${cfg}")" ok "config reads back" -info "starting" -wait_task "$(api POST "/nodes/${NODE}/qemu/${VMID}/status/start" | jq -r '.data')" +if [[ "$(api GET "/nodes/${NODE}/qemu/${VMID}/status/current" | jq -r '.data.status')" != running ]]; then + info "starting" + wait_task "$(api POST "/nodes/${NODE}/qemu/${VMID}/status/start" | jq -r '.data')" +fi info "waiting for the guest agent to report the VLAN 30 lease" addr="" diff --git a/tofu/README.md b/tofu/README.md index 618259f0..eda54e92 100644 --- a/tofu/README.md +++ b/tofu/README.md @@ -32,7 +32,10 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan VMs (191–198, [ADR-0090]), in the `dotfiles` pool, #921's always-on analyst workstation `garuda` (162, [ADR-0091]), in the `analyst` pool, #451's BloodHound guest `eden` (141, [ADR-0081]), in the `bloodhound` pool, - and the proof guest, 998, only under `-var proof=true`. + and the proof guest, 998, only under `-var proof=true`. On `gandalf`, + through the provider's `gandalf` alias, it holds the range's attack VM + `ragnarok` (130, [ADR-0116]), in no pool, which + `scripts/finish-attack-vm.sh` finishes and proves after every apply. - `identities.auto.tfvars` holds each guest's MAC and SMBIOS UUID. It exists only on `phoenix`, is gitignored, and is escrowed in `secrets/tofu.sops.yaml`; `identities.auto.tfvars.example` shows its shape. @@ -65,5 +68,6 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan [ADR-0090]: ../docs/adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md [ADR-0081]: ../docs/adr/0081-run-bloodhound-ce-on-a-saruman-guest.md +[ADR-0116]: ../docs/adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md [ADR-0076]: ../docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md [runbook]: ../docs/runbooks/provision-lab-guests.md diff --git a/tofu/guests.tf b/tofu/guests.tf index 8b40542d..f9c8bb97 100644 --- a/tofu/guests.tf +++ b/tofu/guests.tf @@ -240,6 +240,59 @@ module "guest" { smbios_uuid = each.value.smbios_uuid } +# The range, on gandalf (ADR-0017, ADR-0116): #421's attack VM. A separate +# module call, because a module instance has one provider and this one is +# gandalf's alias. The same module, so the clone, the EFI disk and the agent +# are what every guest here has. +# +# ragnarok is the one guest with a leg on both bridges. net0 on vmbr0 takes a +# DHCP lease from the pool: nothing points at it, so morpheus holds no +# reservation, but the MAC is pinned like every other guest's. net1 on vmbr1 +# is 172.30.30.10/24 with NO gateway. Forwarding off and no NAT are not this +# tree's to make: scripts/finish-attack-vm.sh writes 99-no-forward.conf and +# proves both after every apply, over SSH checked against a host key read +# through the guest agent (ADR-0085). No pool: phoenix's token on gandalf +# holds no pool grant, and this keeps it so. Secure Boot off, as tpl-kali's +# is. `kali`, the user the template and the script have always used. +locals { + range_guests = { + for name, g in { + ragnarok = { vm_id = 130, template = 902, cores = 4, memory_mib = 8192, disk_gib = 64 } + } : + name => merge(g, { + mac_address = var.guest_identities[name].mac_address + smbios_uuid = var.guest_identities[name].smbios_uuid + tags = ["attack", "range"] + extra_networks = [{ bridge = "vmbr1", ipv4_address = "172.30.30.10/24" }] + }) + } +} + +module "range_guest" { + source = "./modules/guest" + for_each = local.range_guests + providers = { proxmox = proxmox.gandalf } + + name = each.key + node = "gandalf" + datastore = "range" + vm_id = each.value.vm_id + template = each.value.template + cores = each.value.cores + memory_mib = each.value.memory_mib + disk_gib = each.value.disk_gib + linux = true + on_boot = false + secure_boot = false + tags = each.value.tags + username = "kali" + password = null + ssh_keys = [trimspace(file(pathexpand(var.ssh_public_key_file)))] + mac_address = each.value.mac_address + smbios_uuid = each.value.smbios_uuid + extra_networks = each.value.extra_networks +} + output "startup_orders" { description = "What root on Saruman sets after an apply, one `qm set` per guest that boots with the host." value = { diff --git a/tofu/identities.auto.tfvars.example b/tofu/identities.auto.tfvars.example index 55dbfc38..0632b33b 100644 --- a/tofu/identities.auto.tfvars.example +++ b/tofu/identities.auto.tfvars.example @@ -24,4 +24,5 @@ guest_identities = { dot-windows = { mac_address = "BC:24:11:00:00:C6" } garuda = { mac_address = "BC:24:11:00:00:A2" } eden = { mac_address = "BC:24:11:00:00:8D" } + ragnarok = { mac_address = "BC:24:11:00:00:82" } } diff --git a/tofu/modules/guest/main.tf b/tofu/modules/guest/main.tf index 324ef4af..31461e28 100644 --- a/tofu/modules/guest/main.tf +++ b/tofu/modules/guest/main.tf @@ -118,6 +118,18 @@ resource "proxmox_virtual_environment_vm" "this" { # networking (packer/variables.pkr.hcl says the same). } + # net1 and on. The range's leg must have no gateway: the attack VM is the one + # dual-homed guest, and a second default route is a second way out + # (ADR-0017). ip_config below gives these addresses with gateway unset. + dynamic "network_device" { + for_each = var.extra_networks + content { + bridge = network_device.value.bridge + model = "virtio" + firewall = false + } + } + agent { enabled = true } @@ -144,6 +156,14 @@ resource "proxmox_virtual_environment_vm" "this" { gateway = var.ipv4 == null ? null : var.ipv4.gateway } } + dynamic "ip_config" { + for_each = var.extra_networks + content { + ipv4 { + address = ip_config.value.ipv4_address + } + } + } dynamic "dns" { for_each = length(var.dns_servers) > 0 ? [1] : [] content { diff --git a/tofu/modules/guest/variables.tf b/tofu/modules/guest/variables.tf index ad9dc098..45009533 100644 --- a/tofu/modules/guest/variables.tf +++ b/tofu/modules/guest/variables.tf @@ -29,7 +29,9 @@ variable "node" { } variable "pool_id" { - type = string + type = string + default = null + description = "The pool on the guest's node. Null puts it in none: ragnarok, on gandalf, where phoenix's token holds no pool grant (ADR-0116)." } variable "datastore" { @@ -179,3 +181,12 @@ variable "dns_servers" { default = [] description = "cloud-init nameservers. Empty leaves Proxmox to fill them from the host, as before." } + +variable "extra_networks" { + type = list(object({ + bridge = string + ipv4_address = string + })) + default = [] + description = "NICs after net0, as net1 and on, each with a static address and NO gateway, the firewall flag off, and a MAC Proxmox chooses. ragnarok's range leg is the one so far: vmbr1, 172.30.30.10/24 (ADR-0017, ADR-0116)." +} diff --git a/tofu/providers.tf b/tofu/providers.tf index 7231242f..3414e180 100644 --- a/tofu/providers.tf +++ b/tofu/providers.tf @@ -8,3 +8,19 @@ provider "proxmox" { insecure = false } + +# gandalf, the range host (ADR-0017, ADR-0116). A second node in the same tree +# and the same state, as ADR-0076 foresaw, through an alias rather than the +# environment: PROXMOX_VE_* already names Saruman. The endpoint and the token +# come from TF_VAR_gandalf_endpoint and TF_VAR_gandalf_api_token in +# phoenix.env, beside the Saruman lines, so every tofu step that sources +# phoenix.env works unchanged (build-the-playground.md §5). A variable a +# provider reads is not written to state either. TLS is verified +# against gandalf's CA, which phoenix trusts since build-the-lab-templates.md +# §5. +provider "proxmox" { + alias = "gandalf" + endpoint = var.gandalf_endpoint + api_token = var.gandalf_api_token + insecure = false +} diff --git a/tofu/variables.tf b/tofu/variables.tf index dcc5f803..1c3b451a 100644 --- a/tofu/variables.tf +++ b/tofu/variables.tf @@ -46,3 +46,17 @@ variable "guest_identities" { error_message = "Each of the six domain guests needs its smbios_uuid in guest_identities, as tofu/identities.auto.tfvars.example shows." } } + +# gandalf's API, for the aliased provider (providers.tf, ADR-0116). No default: +# every plan reads gandalf as well as Saruman, so the two lines must be in +# phoenix.env and gandalf must be up. That cost was chosen over a second tree (#1443). +variable "gandalf_endpoint" { + type = string + description = "https://10.0.30.30:8006/, from TF_VAR_gandalf_endpoint in phoenix.env." +} + +variable "gandalf_api_token" { + type = string + sensitive = true + description = "phoenix@pve!builder= on gandalf, from TF_VAR_gandalf_api_token in phoenix.env." +}