diff --git a/docs/runbooks/build-the-bloodhound-guest.md b/docs/runbooks/build-the-bloodhound-guest.md index 09bbe9df..e585b42d 100644 --- a/docs/runbooks/build-the-bloodhound-guest.md +++ b/docs/runbooks/build-the-bloodhound-guest.md @@ -6,7 +6,9 @@ **Time:** about an evening. §1–§3 take under an hour, because the OS comes from a template rather than an installer. -**You will need:** root on `Saruman`, the Ubuntu template `901` +**You will need:** `phoenix` with the toolchain +([`provision-lab-guests.md`](provision-lab-guests.md)), root on `Saruman`, the +Ubuntu template `901` ([ADR-0074](../adr/0074-build-the-lab-templates-with-packer-from-phoenix.md)), a shell on `prometheus` for §4's certificate, the Mac on Hicks to carry it to `eden`, and `alexander`. @@ -44,14 +46,14 @@ same, this runbook points there and does not keep a second copy that drifts. ## 1. Create the VM, from the template -On `Saruman`. The guest is a **full clone of template `901`**, Packer's Ubuntu +The guest is a **full clone of template `901`**, Packer's Ubuntu 26.04 ([ADR-0074](../adr/0074-build-the-lab-templates-with-packer-from-phoenix.md)), not an ISO install. A clone needs no console, and it starts from an image that has already passed `scripts/packer-smoke.sh`. It also inherits the template's shape, which differs from `alexander`'s and `odin`'s: q35 and OVMF, with an EFI disk, and the guest agent already installed. -Check the room first, against both numbers ADR-0081 states. `pvesm status` +Check the room first, on `Saruman`, against both numbers ADR-0081 states. `pvesm status` gives what is written, and the `lvs` sum gives what is allocated: ```bash @@ -64,43 +66,87 @@ On 2026-10-06 that was 28.7% written and 920 GiB allocated of 876. This guest adds 32 to the allocation. If the written figure is past half, stop and ask why before adding anything. -Clone onto `local-lvm`. That puts the OS disk, the EFI disk and the cloud-init -drive on the HDD mirror, as ADR-0081 decides: +The guest is declared in [`tofu/`](../../tofu/README.md) (`guests.tf`, +`bloodhound`), and made from `phoenix` like every pipeline guest +([`provision-lab-guests.md`](provision-lab-guests.md)). The declaration is §0's +table: + +- the OS disk, the EFI disk and the cloud-init drive on `local-lvm`, the HDD + mirror, without the template's `ssd=1`, as ADR-0081 decides; +- a 32 GiB data disk on `large_data`, as `scsi1`; +- `10.0.30.41/24` with the gateway and nameserver `10.0.30.1`; +- `firewall=0` on `net0`, its MAC pinned; +- tags `bloodhound` and `on-demand`, `onboot` off, and the `bloodhound` pool. + +The search domain is not set, so Proxmox fills it from the host, +`Isengard.Lab`. That is harmless, but it was not chosen. + +Three things have to be in place first: + +1. **The pool grant**, as root on `Saruman`, once: the `/pool/bloodhound` line + in [`provision-lab-guests.md` §2](provision-lab-guests.md#2-what-the-token-is-missing-on-saruman). +2. **`eden`'s MAC in `tofu/identities.auto.tfvars`** on `phoenix`, under + `eden`, then escrowed with + [`provision-lab-guests.md` §3](provision-lab-guests.md)'s command. A first + build takes any free `BC:24:11` address; a rebuild keeps the one it had. +3. **`garnet`'s key on `phoenix`.** The guest's user is `garnet`, and it takes + two keys: `phoenix`'s, which every pipeline guest has, and the one in + `garnet`'s `authorized_keys` on `alexander`. Read that one on `Saruman`, + and put it on `phoenix` as `~/.config/proxmox/eden-garnet.pub` + (`var.eden_operator_key_file`): + + ```bash + qm guest exec 140 -- cat /home/garnet/.ssh/authorized_keys \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["out-data"],end="")' \ + | grep ssh-ed25519 + ``` + +**If `eden` exists already** (VMID 141 is in `qm list` on `Saruman`), stop +here and go to *Once: bring the hand-built `eden` into the pipeline* below. +The commands next are for a first build only: an apply with 141 taken fails +on the clone, after it has made the pool. + +For a first build, on `phoenix`: ```bash -qm clone 901 141 --name eden --full 1 --storage local-lvm +cd ~/code/Gerrrt/HomeLab +umask 077; set -a; . ~/.config/proxmox/phoenix.env; set +a +tofu -chdir=tofu plan -target='module.guest["eden"]' -out=eden.tfplan +tofu -chdir=tofu apply eden.tfplan && rm tofu/eden.tfplan ``` -Then size it, add the data disk on the SSDs, and give cloud-init the user, the -key and the address. The key is the one already in `garnet`'s -`authorized_keys` on `alexander`: +The plan should add the guest and the `bloodhound` pool, and nothing else. +Proxmox warns that the sum of thin volumes exceeds the pool. That is the +allocation ADR-0081 accepts, and the warning is expected. + +### Once: bring the hand-built `eden` into the pipeline + +`eden` was first built by hand, with `qm clone 901 141` and `qm set` +(2026-10-07), so the state does not know it. Import it once, as the domain's +six were ([`build-the-lab-domain.md`](build-the-lab-domain.md#rebuild-from-the-pipeline)): ```bash -qm guest exec 140 -- cat /home/garnet/.ssh/authorized_keys \ - | python3 -c 'import json,sys;print(json.load(sys.stdin)["out-data"],end="")' \ - | grep ssh-ed25519 > /root/eden.pub +tofu -chdir=tofu import 'module.guest["eden"].proxmox_virtual_environment_vm.this' Saruman/141 +tofu -chdir=tofu plan -target='module.guest["eden"]' ``` +The plan should **replace** `eden` and **create** the `bloodhound` pool, and +nothing else. A guest built by hand has no `clone` to read back, so this is the +one plan where a replace is expected. Anything else is a stop. + +The replace destroys both disks, the graph and the guest's age key with them. +ADR-0081 accepts that: the graph is one collection run from rebuilt, and the +key is replaced by issuing new values. So take it out of the stack first, then +let the pipeline make it again: + ```bash -qm set 141 \ - --cores 4 --memory 8192 --balloon 0 --onboot 0 --tags on-demand \ - --net0 virtio,bridge=vmbr0,firewall=0 \ - --scsi0 local-lvm:vm-141-disk-1,cache=none,discard=on,iothread=1,replicate=0 \ - --scsi1 large_data:32,discard=on,iothread=1,ssd=1 \ - --ciuser garnet --sshkeys /root/eden.pub \ - --ipconfig0 ip=10.0.30.41/24,gw=10.0.30.1 --nameserver 10.0.30.1 +tofu -chdir=tofu destroy -target='module.guest["eden"]' +tofu -chdir=tofu plan -target='module.guest["eden"]' -out=eden.tfplan +tofu -chdir=tofu apply eden.tfplan && rm tofu/eden.tfplan ``` -- **`--net0` is restated** to get `firewall=0`, as on every guest here, and a - fresh `bc:24:11` MAC. -- **`--scsi0` is restated** to drop the template's `ssd=1`. That disk is on the - HDD mirror now, and the flag would tell the guest otherwise. -- **The search domain** is not set, so Proxmox fills it from the host, - `Isengard.Lab`. It is harmless, but it was not chosen. Add - `--searchdomain` if that matters. - -`qm set` warns that the sum of thin volumes exceeds the pool. That is the -allocation ADR-0081 accepts, and the warning is expected. +Then continue from §2. At §3, the guest is new and so is its key: follow §11 +for the `.sops.yaml` rule and the secrets file. ## 2. First boot, and the data disk @@ -372,8 +418,10 @@ On the same day, in one pull request with §5's lab changes: ## 10. Take it out -1. `qm shutdown 141` then `qm destroy 141 --purge`. `--purge` removes it from - every backup job and HA group that names it. None should. +1. Remove `eden` from `tofu/guests.tf` and its line from + `identities.auto.tfvars` (and the escrow). Then on `phoenix`, + `tofu -chdir=tofu apply`. The plan should destroy the guest and the + `bloodhound` pool, and nothing else. 2. Remove the `bloodhound` rule from `.sops.yaml`, then `secrets/bloodhound.sops.yaml`. On `alexander`, remove `INGEST_TOKEN_EDEN` from the five places §5 added it. diff --git a/docs/runbooks/build-the-lab-domain.md b/docs/runbooks/build-the-lab-domain.md index 676d58a7..4ce4a4f5 100644 --- a/docs/runbooks/build-the-lab-domain.md +++ b/docs/runbooks/build-the-lab-domain.md @@ -400,11 +400,16 @@ skips step 4. `lab-domain` pool (an import brings in guests, not their pool), and do nothing else. That is the one plan where a replace is expected: `clone` cannot be read back from a running guest. Anything else is a stop. -5. **Destroy.** Only the guests. On a later rebuild, that leaves the pool - and its grant in place: +5. **Destroy.** Only the six, named one by one. On a later rebuild, that + leaves the pool and its grant in place. **Never `-target=module.guest`**: + that is every guest on `Saruman` in the tree, `garuda`, `eden` and the + dotfiles VMs included, and it would take `garuda`'s case notes and `eden`'s + graph with the domain. ADR-0081 puts BloodHound off the domain so that it + survives this step. ```bash - tofu -chdir=tofu destroy -target=module.guest + SIX=(); for g in bahamut leviathan titan ramuh carbuncle siren; do SIX+=(-target="module.guest[\"$g\"]"); done + tofu -chdir=tofu destroy "${SIX[@]}" ``` Check on `Saruman` that `qm list` has no 150–155. @@ -412,10 +417,12 @@ skips step 4. name, so that the rebuilt guests' enrolment under the same names is not refused as a duplicate. This comes after the destroy, so a run that stops before it leaves the old guests still reporting. -7. **Apply.** This also creates the pool, the first time. +7. **Apply,** targeted at the six as step 5 was. This also creates the pool, + the first time. Targeted, so that a guest declared but not yet made or + imported elsewhere in the tree is not created here as a side effect. ```bash - tofu -chdir=tofu plan -out=next.tfplan + tofu -chdir=tofu plan "${SIX[@]}" -out=next.tfplan tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan ``` diff --git a/docs/runbooks/provision-lab-guests.md b/docs/runbooks/provision-lab-guests.md index 63355ff5..d8067001 100644 --- a/docs/runbooks/provision-lab-guests.md +++ b/docs/runbooks/provision-lab-guests.md @@ -115,6 +115,7 @@ pveum acl modify /pool/proof --users phoenix@pve --roles PhoenixBuilder pveum acl modify /pool/lab-domain --users phoenix@pve --roles PhoenixBuilder pveum acl modify /pool/dotfiles --users phoenix@pve --roles PhoenixBuilder pveum acl modify /pool/analyst --users phoenix@pve --roles PhoenixBuilder +pveum acl modify /pool/bloodhound --users phoenix@pve --roles PhoenixBuilder pveum acl list | grep phoenix ``` @@ -132,9 +133,10 @@ Each later pool gets its own `acl modify` line here and a row below: | Path | Role | For | | --- | --- | --- | | `/pool/proof` | `PhoenixBuilder` | §4's proof guest. It exists only while the proof runs, and §4's teardown deletes this grant with it. Re-run it before any later `-var proof=true` | -| `/pool/lab-domain` | `PhoenixBuilder` | ADR-0029's six. A rebuild destroys with `-target=module.guest`, so the pool and this grant outlive the guests. An untargeted `tofu destroy` takes both, and this line must be run again before the next apply | +| `/pool/lab-domain` | `PhoenixBuilder` | ADR-0029's six. A rebuild destroys with a `-target` per guest, never `-target=module.guest`, which would take every Saruman guest in the tree with them. The pool and this grant outlive the six. An untargeted `tofu destroy` takes both, and this line must be run again before the next apply | | `/pool/dotfiles` | `PhoenixBuilder` | [ADR-0090](../adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles VMs, 191–198. The same rule as `lab-domain`: rebuild one with `-target`, or run this line again | | `/pool/analyst` | `PhoenixBuilder` | [ADR-0091](../adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)'s `garuda`, 162. The pool is its own resource, so a targeted destroy of the guest leaves the pool and this grant. Teardown destroys both targets (build-the-analyst-workstation.md §13), and then this line must be run again before a rebuild | +| `/pool/bloodhound` | `PhoenixBuilder` | [ADR-0081](../adr/0081-run-bloodhound-ce-on-a-saruman-guest.md)'s `eden`, 141, brought into the pipeline by [#1443](https://github.com/Gerrrt/HomeLab/issues/1443). Its own pool, so a targeted destroy of the guest leaves the pool and this grant | If an apply fails with `Permission check failed (/…, Some.Privilege)`, add that privilege to the role and record it here. This is ADR-0043's rule again. diff --git a/tofu/README.md b/tofu/README.md index ea71e6a2..618259f0 100644 --- a/tofu/README.md +++ b/tofu/README.md @@ -30,8 +30,9 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan lives exactly as long as a guest in it does. It holds ADR-0029's six domain guests (150–155), in the `lab-domain` pool, #920's on-demand dotfiles VMs (191–198, [ADR-0090]), in the `dotfiles` pool, #921's always-on - analyst workstation `garuda` (162, [ADR-0091]), in the `analyst` pool, and - the proof guest, 998, only under `-var proof=true`. + analyst workstation `garuda` (162, [ADR-0091]), in the `analyst` pool, + #451's BloodHound guest `eden` (141, [ADR-0081]), in the `bloodhound` pool, + and the proof guest, 998, only under `-var proof=true`. - `identities.auto.tfvars` holds each guest's MAC and SMBIOS UUID. It exists only on `phoenix`, is gitignored, and is escrowed in `secrets/tofu.sops.yaml`; `identities.auto.tfvars.example` shows its shape. @@ -54,7 +55,8 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan reservations are keyed by MAC, and the endpoints' activation by the UUID. A rebuild is [`build-the-lab-domain.md`'s](../docs/runbooks/build-the-lab-domain.md#rebuild-from-the-pipeline), - and it destroys with `-target=module.guest` so the pool's grant survives. + and it destroys and applies with a `-target` per domain guest, so the + pool's grant survives and no other guest in `module.guest` goes with them. - **CI proves that the tree parses and that the encryption holds. `phoenix` proves that it applies.** `scripts/lint.sh` runs `fmt -check` and `validate`, and `scripts/self-tests.sh` runs the encryption proof. Both use @@ -62,5 +64,6 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan finished once the runbook's §4 has passed against a real apply. [ADR-0090]: ../docs/adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md +[ADR-0081]: ../docs/adr/0081-run-bloodhound-ce-on-a-saruman-guest.md [ADR-0076]: ../docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md [runbook]: ../docs/runbooks/provision-lab-guests.md diff --git a/tofu/guests.tf b/tofu/guests.tf index 573e0168..8b40542d 100644 --- a/tofu/guests.tf +++ b/tofu/guests.tf @@ -156,7 +156,39 @@ locals { }) } - guests = merge(local.lab_domain, local.dotfiles, local.analyst, local.proof_guests) + # #451's eden (141), BloodHound CE, ADR-0081's shape as + # build-the-bloodhound-guest.md §0 tabulates it: the OS disk on local-lvm, + # the HDD mirror, without the SSD flag; Postgres on a 32 GiB data disk on + # large_data; a static .41, because the stack's certificate and the + # analyst's browser name it; off between sessions. Built by hand first, and + # brought here by #1443: imported once, then rebuilt from the pipeline, + # which ADR-0081 allows because the graph is derived data and eden is not + # backed up. `garnet`, not `operator`: the user alexander's operator logs in + # as, with that key beside phoenix's (var.eden_operator_key_file). + bloodhound = { + for name, g in { + eden = { vm_id = 141, template = 901, cores = 4, memory_mib = 8192, disk_gib = 32 } + } : + name => merge(g, { + pool = "bloodhound" + mac_address = var.guest_identities[name].mac_address + smbios_uuid = var.guest_identities[name].smbios_uuid + startup = null + linux = true + on_boot = false + tags = ["bloodhound", "on-demand"] + password = null + username = "garnet" + datastore = "local-lvm" + os_disk_ssd = false + data_disks = [{ datastore = "large_data", size_gib = 32 }] + ipv4 = { address = "10.0.30.41/24", gateway = "10.0.30.1" } + dns_servers = ["10.0.30.1"] + extra_keys = [trimspace(file(pathexpand(var.eden_operator_key_file)))] + }) + } + + guests = merge(local.lab_domain, local.dotfiles, local.analyst, local.bloodhound, local.proof_guests) # A pool exists exactly while it groups a guest: it is derived from the # guests, not listed beside them, so the last guest leaving a pool takes the @@ -196,7 +228,13 @@ module "guest" { secure_boot = lookup(each.value, "secure_boot", true) # Absent means Proxmox's default. dot-windows runs VirtIO GPU (#1108). vga = lookup(each.value, "vga", null) - ssh_keys = each.value.linux ? [trimspace(file(pathexpand(var.ssh_public_key_file)))] : [] + ssh_keys = each.value.linux ? concat([trimspace(file(pathexpand(var.ssh_public_key_file)))], lookup(each.value, "extra_keys", [])) : [] + # eden's shape (#1443). Absent means what every other guest has. + datastore = lookup(each.value, "datastore", "large_data") + os_disk_ssd = lookup(each.value, "os_disk_ssd", true) + data_disks = lookup(each.value, "data_disks", []) + ipv4 = lookup(each.value, "ipv4", null) + dns_servers = lookup(each.value, "dns_servers", []) mac_address = each.value.mac_address smbios_uuid = each.value.smbios_uuid diff --git a/tofu/identities.auto.tfvars.example b/tofu/identities.auto.tfvars.example index 390faca5..55dbfc38 100644 --- a/tofu/identities.auto.tfvars.example +++ b/tofu/identities.auto.tfvars.example @@ -23,4 +23,5 @@ guest_identities = { dot-nixos = { mac_address = "BC:24:11:00:00:C5" } dot-windows = { mac_address = "BC:24:11:00:00:C6" } garuda = { mac_address = "BC:24:11:00:00:A2" } + eden = { mac_address = "BC:24:11:00:00:8D" } } diff --git a/tofu/modules/guest/main.tf b/tofu/modules/guest/main.tf index ac384ba8..324ef4af 100644 --- a/tofu/modules/guest/main.tf +++ b/tofu/modules/guest/main.tf @@ -87,7 +87,24 @@ resource "proxmox_virtual_environment_vm" "this" { size = var.disk_gib discard = "on" iothread = true - ssd = true + ssd = var.os_disk_ssd + } + + # A data disk is created empty: the guest formats and mounts it (eden's is + # build-the-bloodhound-guest.md §2). It is part of the VM, so a destroy or + # a replace takes it, which ADR-0081 accepts for eden: its graph is derived + # data, one collection run from rebuilt. + dynamic "disk" { + for_each = var.data_disks + content { + interface = "scsi${disk.key + 1}" + datastore_id = disk.value.datastore + size = disk.value.size_gib + file_format = "raw" + discard = "on" + iothread = true + ssd = true + } } network_device { @@ -123,7 +140,14 @@ resource "proxmox_virtual_environment_vm" "this" { datastore_id = var.datastore ip_config { ipv4 { - address = "dhcp" + address = var.ipv4 == null ? "dhcp" : var.ipv4.address + gateway = var.ipv4 == null ? null : var.ipv4.gateway + } + } + dynamic "dns" { + for_each = length(var.dns_servers) > 0 ? [1] : [] + content { + servers = var.dns_servers } } user_account { diff --git a/tofu/modules/guest/variables.tf b/tofu/modules/guest/variables.tf index 01fc189f..ad9dc098 100644 --- a/tofu/modules/guest/variables.tf +++ b/tofu/modules/guest/variables.tf @@ -145,3 +145,37 @@ variable "password" { default = null sensitive = true } + +# The variables below exist for eden (#1443), the first guest whose hand-built +# shape the module did not already describe. Each defaults to what every other +# guest has, so their plans are unchanged. + +variable "os_disk_ssd" { + type = bool + default = true + description = "Tell the guest scsi0 is an SSD. True on large_data. False for an OS disk on local-lvm, the HDD mirror (eden, ADR-0081), where the flag would say otherwise." +} + +variable "data_disks" { + type = list(object({ + datastore = string + size_gib = number + })) + default = [] + description = "Disks after scsi0, as scsi1, scsi2 and on, each with discard, iothread and ssd. eden's Postgres disk is the one so far (ADR-0081: 32 GiB on large_data)." +} + +variable "ipv4" { + type = object({ + address = string + gateway = string + }) + default = null + description = "A static address in CIDR form and its gateway, written to cloud-init. Null is DHCP from morpheus's reservation, as every other Linux guest has." +} + +variable "dns_servers" { + type = list(string) + default = [] + description = "cloud-init nameservers. Empty leaves Proxmox to fill them from the host, as before." +} diff --git a/tofu/variables.tf b/tofu/variables.tf index f201c6aa..dcc5f803 100644 --- a/tofu/variables.tf +++ b/tofu/variables.tf @@ -10,6 +10,12 @@ variable "ssh_public_key_file" { description = "phoenix's key, the one ADR-0043 says the toolchain puts on every guest it builds. Same default as packer/." } +variable "eden_operator_key_file" { + type = string + default = "~/.config/proxmox/eden-garnet.pub" + description = "garnet's public key from alexander, which eden's operator logs in with beside phoenix's own (build-the-bloodhound-guest.md §1). A public key, but kept beside the credentials so the checkout stays the same on every host." +} + # Each guest's MAC and, for the six domain guests, the SMBIOS UUID the # hand-built one had. They are not in this repository: docs/security.md's # "What this repository deliberately does not publish" withholds full MACs,