Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ documents for different readers.
at all. Every pinned digest is also scanned weekly for fixable HIGH and
CRITICAL CVEs, and each image with findings has an open issue until a scan
finds it clean.
- **Documented decisions and runbooks.** 115 ADRs covering what was chosen
- **Documented decisions and runbooks.** 116 ADRs covering what was chosen
and what was rejected — including the costs accepted knowingly; 53
runbooks for the operations that are easy to get wrong at 1am, one of which
is the handover page a successor reads first. Every critical alert links to
Expand Down Expand Up @@ -235,7 +235,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology
│ ├── architecture.md network.md hardware.md
│ ├── observability.md security.md roadmap.md changelog.md
│ ├── diagrams/ # the network diagram (SVG) and its predecessors
│ ├── adr/ # 115 ADRs — architecture decision records
│ ├── adr/ # 116 ADRs — architecture decision records
│ ├── exercises/ # purple-team exercise records — see its README
│ └── runbooks/ # 53 runbooks; successor-handover.md is the front door
└── Makefile # make help
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

**Status:** Accepted · 2026-10

> [!NOTE]
> Amended by [ADR-0116](0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md):
> the provider gains an alias for `gandalf`, read from `phoenix.env`, and the
> range's attack VM is declared in this tree with a script that finishes and
> proves it after every apply. The text here is left as written, per ADR-0001
> and ADR-0115.

## Context

[ADR-0074](0074-build-the-lab-templates-with-packer-from-phoenix.md) makes
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# ADR-0116: Build the range's attack VM with OpenTofu, and prove it with a script

**Status:** Accepted · 2026-11 · amends [ADR-0076](0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md)

## Context

[ADR-0076](0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md)
put "#421's attack VM and targets" in OpenTofu's scope, and expected the range
to be "a second node's guests in this tree rather than a second tree". When
the range was built on `gandalf`
([ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md),
[ADR-0106](0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)),
its attack VM, `ragnarok`, was made by `scripts/make-attack-vm.sh` instead,
outside the pipeline. The 2026-11-11 audit found that, and the owner chose to
bring it into `tofu/`
([#1443](https://github.com/Gerrrt/HomeLab/issues/1443)).

Two things stood in the way.

- **The tree had one provider, read from the environment.**
`PROXMOX_VE_ENDPOINT` and `PROXMOX_VE_API_TOKEN` in `phoenix.env` name
`Saruman`. `gandalf.env` names `gandalf`, but in the curl scripts' own
variables (`PROXMOX_URL`, `PROXMOX_TOKEN_ID`, `PROXMOX_TOKEN_SECRET`).
- **`ragnarok` is more than a clone.** Its isolation is in three parts:
- `net1` on `vmbr1` with no gateway;
- forwarding off, persistently, in `99-no-forward.conf`;
- a proof after every make that forwarding is off and nothing NATs.

That proof runs over SSH, against a host key read through the guest agent
([ADR-0085](0085-check-the-lab-domains-host-keys-against-keys-read-through-the-guest-agent.md)).

## Decision

**One tree and one state, with an aliased provider for `gandalf`.**
`provider "proxmox" { alias = "gandalf" }` takes its endpoint and token from
`TF_VAR_gandalf_endpoint` and `TF_VAR_gandalf_api_token`, two new lines in
`phoenix.env`. Their names collide with nothing there, so every runbook step
that sources `phoenix.env` before a plan works unchanged. `gandalf.env`, which
overrides the curl scripts' `PROXMOX_*` names, is not needed for a plan. The
token is then in two mode-600 files on `phoenix` rather than one. A provider's
variables are not written to state. TLS is verified against `gandalf`'s CA, which `phoenix`
already trusts.

**OpenTofu makes the VM.** `ragnarok` is `module.range_guest` in
`tofu/guests.tf`: the same guest module as every other lab guest, called with
the `gandalf` provider. It is a full clone of 902 on `range` with:

- `net0` on `vmbr0`, DHCP, its MAC pinned;
- `net1` on `vmbr1` at `172.30.30.10/24` with no gateway, through the module's
new `extra_networks`;
- both firewall flags off;
- cloud-init user `kali` with `phoenix`'s key.

It is in no pool, because the token on `gandalf` holds no pool grant and this
keeps it so.

**A script finishes and proves it, after every apply.**
`scripts/make-attack-vm.sh` becomes `scripts/finish-attack-vm.sh`. It no
longer clones or configures, and refuses to run if 130 does not exist. It:

- reads the config back, including `net1`'s address and the absent gateway;
- starts the guest if it is stopped;
- writes `99-no-forward.conf` over the host-key-checked SSH;
- runs §8's third check.

The sysctl file stays in the script rather than in cloud-init. A cloud-init
snippet would have tofu upload it to `gandalf` over SSH, a second credential
path into the range host for one file.

**The range is still rebuilt, not repaired.** ADR-0017's rule is unchanged.
What changes is the tool: a change to `ragnarok` is a destroy and apply of
130, then `finish-attack-vm.sh`.

This amends ADR-0076 in two places. Its single, environment-read provider
gains an alias. Its expectation that the range is "a second node's guests in
this tree" is carried out, for the attack VM.

## Alternatives considered

- **A second tree for the range,** with its own state and its own passphrase
to escrow, and the provider's environment variables from `gandalf.env`. A
`Saruman` plan would never need `gandalf` powered on. Declined by the owner
for one state and one plan, at the cost below.
- **All of it in cloud-init,** the sysctl file included, with the script
reduced to the proof. Declined for the credential it needs, as above.
- **Leave `ragnarok` script-built,** and supersede that part of ADR-0076's
scope. Declined: the guest the range's isolation depends on would be the
one guest the pipeline cannot rebuild.

## Consequences

- **Every plan reads `gandalf`.** `gandalf` must be up, even for a plan that
touches only `Saruman`. `gandalf` is off between sessions (ADR-0017), so a `Saruman`-only
change waits for it, or is planned with `-target`.
- **The script's §8 proof still runs after every make.** An apply that is not
followed by `finish-attack-vm.sh` leaves a VM whose forwarding has not been
turned off. `build-the-playground.md` §5 says so, in that order.
- **The `/vms/130` grant goes with every destroy.** It was already the case
with the script, and the script still checks for it first.
- **The first apply replaces the script-built `ragnarok`.** Importing a guest
that tofu did not clone plans a replace, because `clone` cannot be read back
([`build-the-lab-domain.md`](../runbooks/build-the-lab-domain.md)). The
range is disposable, so the runbook destroys 130 and applies, and does not
import.
- **Reopened by:**
- the targets of #421 joining the tree, which would make the range a larger
share of every plan;
- `gandalf` gaining the snippet storage and SSH access that a cloud-init
sysctl file would need, for some other reason.
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ hole from the monitoring VLAN into the monitored one.
| --- | --- | --- | --- |
| `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, speedtest-tracker, docker-socket-proxy, Alloy |
| `Saruman` (10.0.30.20) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). The attack VM `ragnarok` (VMID 130), made 2026-11-09 by [`scripts/make-attack-vm.sh`](../scripts/make-attack-vm.sh) from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. Three targets, VMIDs 100–102 on `vmbr1` only, made 2026-11-10; the list is [`build-the-playground.md`](runbooks/build-the-playground.md) §6's. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) |
| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). The attack VM `ragnarok` (VMID 130), declared in [`tofu/`](../tofu/README.md) on `gandalf`'s aliased provider and finished by [`scripts/finish-attack-vm.sh`](../scripts/finish-attack-vm.sh) ([ADR-0116](adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)), from template 902 and dual-homed: a DHCP lease on VLAN 30, and `172.30.30.10` on `vmbr1`. Three targets, VMIDs 100–102 on `vmbr1` only, made 2026-11-10; the list is [`build-the-playground.md`](runbooks/build-the-playground.md) §6's. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) |
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy and its Docker socket proxy, and Caddy as the ingest proxy in front of the first two ([#834](https://github.com/Gerrrt/HomeLab/issues/834)) — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `fenrir` (10.0.30.90) | 🟢 30 | [`stacks/sensor`](../stacks/sensor) | Zeek on a `tc` mirror of `Saruman`'s lab bridge, so that the domain's east-west traffic, which crosses no router and so never reaches Suricata on `morpheus`, is observed at all ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)). A guest on `Saruman`, VMID 190, with a second NIC alone on the port-less `vmbr1` that receives only the mirror's copies. Its logs go to `alexander`, never to VLAN 99; whether the mirror is delivering is answered on the hypervisor by `homelab_zeek_mirror_active`. Built 2026-09-30 by [`build-the-sensor-guest.md`](runbooks/build-the-sensor-guest.md), and proved by rebooting `Saruman` with the mirror's timer disabled: the gauge fell to 0 and `ZeekMirrorInactive` fired. Alloy agent (Docker) |
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/build-the-lab-templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -532,7 +532,7 @@ and 910 with this section and §5b to clear it.
| `PhoenixBuilder` | `/storage/range` | the template's disks, on the range pool ([ADR-0096](../adr/0096-fit-ifrits-range-disk-as-a-sata-ssd-in-the-tinys-bay.md)) |
| `PhoenixBuilder` | `/storage/local` | the installer ISO |
| `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr0` | the build NIC |
| `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr1` | `ragnarok`'s range NIC, which [`make-attack-vm.sh`](../../scripts/make-attack-vm.sh) attaches ([`build-the-playground.md`](build-the-playground.md) §5). Kept 2026-11-09 |
| `PhoenixBuilder` | `/sdn/zones/localnetwork/vmbr1` | `ragnarok`'s range NIC, which `tofu/` attaches ([ADR-0116](../adr/0116-build-the-range-attack-vm-with-opentofu-and-prove-it-with-a-script.md)) ([`build-the-playground.md`](build-the-playground.md) §5). Kept 2026-11-09 |
| `PVEAuditor` | `/nodes/gandalf` | read-only. Lower case, as `pvesh get /nodes` prints it |

Nothing is granted on `local-lvm`, because the range's disks stay off the
Expand Down
Loading
Loading