Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 79 additions & 31 deletions docs/runbooks/build-the-bloodhound-guest.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
**Time:** about an evening. §1–§3 take under an hour, because the OS comes
from a template rather than an installer.

**You will need:** root on `Saruman`, the Ubuntu template `901`
**You will need:** `phoenix` with the toolchain
([`provision-lab-guests.md`](provision-lab-guests.md)), root on `Saruman`, the
Ubuntu template `901`
([ADR-0074](../adr/0074-build-the-lab-templates-with-packer-from-phoenix.md)),
a shell on `prometheus` for §4's certificate, the Mac on Hicks to carry it to
`eden`, and `alexander`.
Expand Down Expand Up @@ -44,14 +46,14 @@ same, this runbook points there and does not keep a second copy that drifts.

## 1. Create the VM, from the template

On `Saruman`. The guest is a **full clone of template `901`**, Packer's Ubuntu
The guest is a **full clone of template `901`**, Packer's Ubuntu
26.04 ([ADR-0074](../adr/0074-build-the-lab-templates-with-packer-from-phoenix.md)),
not an ISO install. A clone needs no console, and it starts from an image that
has already passed `scripts/packer-smoke.sh`. It also inherits the template's
shape, which differs from `alexander`'s and `odin`'s: q35 and OVMF, with an EFI
disk, and the guest agent already installed.

Check the room first, against both numbers ADR-0081 states. `pvesm status`
Check the room first, on `Saruman`, against both numbers ADR-0081 states. `pvesm status`
gives what is written, and the `lvs` sum gives what is allocated:

```bash
Expand All @@ -64,43 +66,87 @@ On 2026-10-06 that was 28.7% written and 920 GiB allocated of 876. This guest
adds 32 to the allocation. If the written figure is past half, stop and ask
why before adding anything.

Clone onto `local-lvm`. That puts the OS disk, the EFI disk and the cloud-init
drive on the HDD mirror, as ADR-0081 decides:
The guest is declared in [`tofu/`](../../tofu/README.md) (`guests.tf`,
`bloodhound`), and made from `phoenix` like every pipeline guest
([`provision-lab-guests.md`](provision-lab-guests.md)). The declaration is §0's
table:

- the OS disk, the EFI disk and the cloud-init drive on `local-lvm`, the HDD
mirror, without the template's `ssd=1`, as ADR-0081 decides;
- a 32 GiB data disk on `large_data`, as `scsi1`;
- `10.0.30.41/24` with the gateway and nameserver `10.0.30.1`;
- `firewall=0` on `net0`, its MAC pinned;
- tags `bloodhound` and `on-demand`, `onboot` off, and the `bloodhound` pool.

The search domain is not set, so Proxmox fills it from the host,
`Isengard.Lab`. That is harmless, but it was not chosen.

Three things have to be in place first:

1. **The pool grant**, as root on `Saruman`, once: the `/pool/bloodhound` line
in [`provision-lab-guests.md` §2](provision-lab-guests.md#2-what-the-token-is-missing-on-saruman).
2. **`eden`'s MAC in `tofu/identities.auto.tfvars`** on `phoenix`, under
`eden`, then escrowed with
[`provision-lab-guests.md` §3](provision-lab-guests.md)'s command. A first
build takes any free `BC:24:11` address; a rebuild keeps the one it had.
3. **`garnet`'s key on `phoenix`.** The guest's user is `garnet`, and it takes
two keys: `phoenix`'s, which every pipeline guest has, and the one in
`garnet`'s `authorized_keys` on `alexander`. Read that one on `Saruman`,
and put it on `phoenix` as `~/.config/proxmox/eden-garnet.pub`
(`var.eden_operator_key_file`):

```bash
qm guest exec 140 -- cat /home/garnet/.ssh/authorized_keys \
| python3 -c 'import json,sys;print(json.load(sys.stdin)["out-data"],end="")' \
| grep ssh-ed25519
```

**If `eden` exists already** (VMID 141 is in `qm list` on `Saruman`), stop
here and go to *Once: bring the hand-built `eden` into the pipeline* below.
The commands next are for a first build only: an apply with 141 taken fails
on the clone, after it has made the pool.

For a first build, on `phoenix`:

```bash
qm clone 901 141 --name eden --full 1 --storage local-lvm
cd ~/code/Gerrrt/HomeLab
umask 077; set -a; . ~/.config/proxmox/phoenix.env; set +a
tofu -chdir=tofu plan -target='module.guest["eden"]' -out=eden.tfplan
tofu -chdir=tofu apply eden.tfplan && rm tofu/eden.tfplan
```

Then size it, add the data disk on the SSDs, and give cloud-init the user, the
key and the address. The key is the one already in `garnet`'s
`authorized_keys` on `alexander`:
The plan should add the guest and the `bloodhound` pool, and nothing else.
Proxmox warns that the sum of thin volumes exceeds the pool. That is the
allocation ADR-0081 accepts, and the warning is expected.

### Once: bring the hand-built `eden` into the pipeline

`eden` was first built by hand, with `qm clone 901 141` and `qm set`
(2026-10-07), so the state does not know it. Import it once, as the domain's
six were ([`build-the-lab-domain.md`](build-the-lab-domain.md#rebuild-from-the-pipeline)):

```bash
qm guest exec 140 -- cat /home/garnet/.ssh/authorized_keys \
| python3 -c 'import json,sys;print(json.load(sys.stdin)["out-data"],end="")' \
| grep ssh-ed25519 > /root/eden.pub
tofu -chdir=tofu import 'module.guest["eden"].proxmox_virtual_environment_vm.this' Saruman/141
tofu -chdir=tofu plan -target='module.guest["eden"]'
```

The plan should **replace** `eden` and **create** the `bloodhound` pool, and
nothing else. A guest built by hand has no `clone` to read back, so this is the
one plan where a replace is expected. Anything else is a stop.

The replace destroys both disks, the graph and the guest's age key with them.
ADR-0081 accepts that: the graph is one collection run from rebuilt, and the
key is replaced by issuing new values. So take it out of the stack first, then
let the pipeline make it again:

```bash
qm set 141 \
--cores 4 --memory 8192 --balloon 0 --onboot 0 --tags on-demand \
--net0 virtio,bridge=vmbr0,firewall=0 \
--scsi0 local-lvm:vm-141-disk-1,cache=none,discard=on,iothread=1,replicate=0 \
--scsi1 large_data:32,discard=on,iothread=1,ssd=1 \
--ciuser garnet --sshkeys /root/eden.pub \
--ipconfig0 ip=10.0.30.41/24,gw=10.0.30.1 --nameserver 10.0.30.1
tofu -chdir=tofu destroy -target='module.guest["eden"]'
tofu -chdir=tofu plan -target='module.guest["eden"]' -out=eden.tfplan
tofu -chdir=tofu apply eden.tfplan && rm tofu/eden.tfplan
```

- **`--net0` is restated** to get `firewall=0`, as on every guest here, and a
fresh `bc:24:11` MAC.
- **`--scsi0` is restated** to drop the template's `ssd=1`. That disk is on the
HDD mirror now, and the flag would tell the guest otherwise.
- **The search domain** is not set, so Proxmox fills it from the host,
`Isengard.Lab`. It is harmless, but it was not chosen. Add
`--searchdomain` if that matters.

`qm set` warns that the sum of thin volumes exceeds the pool. That is the
allocation ADR-0081 accepts, and the warning is expected.
Then continue from §2. At §3, the guest is new and so is its key: follow §11
for the `.sops.yaml` rule and the secrets file.

## 2. First boot, and the data disk

Expand Down Expand Up @@ -372,8 +418,10 @@ On the same day, in one pull request with §5's lab changes:

## 10. Take it out

1. `qm shutdown 141` then `qm destroy 141 --purge`. `--purge` removes it from
every backup job and HA group that names it. None should.
1. Remove `eden` from `tofu/guests.tf` and its line from
`identities.auto.tfvars` (and the escrow). Then on `phoenix`,
`tofu -chdir=tofu apply`. The plan should destroy the guest and the
`bloodhound` pool, and nothing else.
2. Remove the `bloodhound` rule from `.sops.yaml`, then
`secrets/bloodhound.sops.yaml`. On `alexander`, remove `INGEST_TOKEN_EDEN`
from the five places §5 added it.
Expand Down
17 changes: 12 additions & 5 deletions docs/runbooks/build-the-lab-domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,22 +400,29 @@ skips step 4.
`lab-domain` pool (an import brings in guests, not their pool), and do
nothing else. That is the one plan where a replace is expected:
`clone` cannot be read back from a running guest. Anything else is a stop.
5. **Destroy.** Only the guests. On a later rebuild, that leaves the pool
and its grant in place:
5. **Destroy.** Only the six, named one by one. On a later rebuild, that
leaves the pool and its grant in place. **Never `-target=module.guest`**:
that is every guest on `Saruman` in the tree, `garuda`, `eden` and the
dotfiles VMs included, and it would take `garuda`'s case notes and `eden`'s
graph with the domain. ADR-0081 puts BloodHound off the domain so that it
survives this step.

```bash
tofu -chdir=tofu destroy -target=module.guest
SIX=(); for g in bahamut leviathan titan ramuh carbuncle siren; do SIX+=(-target="module.guest[\"$g\"]"); done
tofu -chdir=tofu destroy "${SIX[@]}"
```

Check on `Saruman` that `qm list` has no 150–155.
6. **The old Wazuh registrations go.** On `odin`, remove the six agents by
name, so that the rebuilt guests' enrolment under the same names is not
refused as a duplicate. This comes after the destroy, so a run that stops
before it leaves the old guests still reporting.
7. **Apply.** This also creates the pool, the first time.
7. **Apply,** targeted at the six as step 5 was. This also creates the pool,
the first time. Targeted, so that a guest declared but not yet made or
imported elsewhere in the tree is not created here as a side effect.

```bash
tofu -chdir=tofu plan -out=next.tfplan
tofu -chdir=tofu plan "${SIX[@]}" -out=next.tfplan
tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan
```

Expand Down
4 changes: 3 additions & 1 deletion docs/runbooks/provision-lab-guests.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ pveum acl modify /pool/proof --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /pool/lab-domain --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /pool/dotfiles --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /pool/analyst --users phoenix@pve --roles PhoenixBuilder
pveum acl modify /pool/bloodhound --users phoenix@pve --roles PhoenixBuilder
pveum acl list | grep phoenix
```

Expand All @@ -132,9 +133,10 @@ Each later pool gets its own `acl modify` line here and a row below:
| Path | Role | For |
| --- | --- | --- |
| `/pool/proof` | `PhoenixBuilder` | §4's proof guest. It exists only while the proof runs, and §4's teardown deletes this grant with it. Re-run it before any later `-var proof=true` |
| `/pool/lab-domain` | `PhoenixBuilder` | ADR-0029's six. A rebuild destroys with `-target=module.guest`, so the pool and this grant outlive the guests. An untargeted `tofu destroy` takes both, and this line must be run again before the next apply |
| `/pool/lab-domain` | `PhoenixBuilder` | ADR-0029's six. A rebuild destroys with a `-target` per guest, never `-target=module.guest`, which would take every Saruman guest in the tree with them. The pool and this grant outlive the six. An untargeted `tofu destroy` takes both, and this line must be run again before the next apply |
| `/pool/dotfiles` | `PhoenixBuilder` | [ADR-0090](../adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles VMs, 191–198. The same rule as `lab-domain`: rebuild one with `-target`, or run this line again |
| `/pool/analyst` | `PhoenixBuilder` | [ADR-0091](../adr/0091-put-a-kali-purple-analyst-workstation-on-saruman.md)'s `garuda`, 162. The pool is its own resource, so a targeted destroy of the guest leaves the pool and this grant. Teardown destroys both targets (build-the-analyst-workstation.md §13), and then this line must be run again before a rebuild |
| `/pool/bloodhound` | `PhoenixBuilder` | [ADR-0081](../adr/0081-run-bloodhound-ce-on-a-saruman-guest.md)'s `eden`, 141, brought into the pipeline by [#1443](https://github.com/Gerrrt/HomeLab/issues/1443). Its own pool, so a targeted destroy of the guest leaves the pool and this grant |

If an apply fails with `Permission check failed (/…, Some.Privilege)`, add that
privilege to the role and record it here. This is ADR-0043's rule again.
Expand Down
9 changes: 6 additions & 3 deletions tofu/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,9 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan
lives exactly as long as a guest in it does. It holds ADR-0029's six domain
guests (150–155), in the `lab-domain` pool, #920's on-demand dotfiles
VMs (191–198, [ADR-0090]), in the `dotfiles` pool, #921's always-on
analyst workstation `garuda` (162, [ADR-0091]), in the `analyst` pool, and
the proof guest, 998, only under `-var proof=true`.
analyst workstation `garuda` (162, [ADR-0091]), in the `analyst` pool,
#451's BloodHound guest `eden` (141, [ADR-0081]), in the `bloodhound` pool,
and the proof guest, 998, only under `-var proof=true`.
- `identities.auto.tfvars` holds each guest's MAC and SMBIOS UUID. It exists
only on `phoenix`, is gitignored, and is escrowed in
`secrets/tofu.sops.yaml`; `identities.auto.tfvars.example` shows its shape.
Expand All @@ -54,13 +55,15 @@ tofu -chdir=tofu apply next.tfplan && rm tofu/next.tfplan
reservations are keyed by MAC, and the endpoints' activation by the UUID.
A rebuild is
[`build-the-lab-domain.md`'s](../docs/runbooks/build-the-lab-domain.md#rebuild-from-the-pipeline),
and it destroys with `-target=module.guest` so the pool's grant survives.
and it destroys and applies with a `-target` per domain guest, so the
pool's grant survives and no other guest in `module.guest` goes with them.
- **CI proves that the tree parses and that the encryption holds. `phoenix`
proves that it applies.** `scripts/lint.sh` runs `fmt -check` and
`validate`, and `scripts/self-tests.sh` runs the encryption proof. Both use
the image `stacks/observability/compose.yaml` pins. A change here is only
finished once the runbook's §4 has passed against a real apply.

[ADR-0090]: ../docs/adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md
[ADR-0081]: ../docs/adr/0081-run-bloodhound-ce-on-a-saruman-guest.md
[ADR-0076]: ../docs/adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md
[runbook]: ../docs/runbooks/provision-lab-guests.md
42 changes: 40 additions & 2 deletions tofu/guests.tf
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,39 @@ locals {
})
}

guests = merge(local.lab_domain, local.dotfiles, local.analyst, local.proof_guests)
# #451's eden (141), BloodHound CE, ADR-0081's shape as
# build-the-bloodhound-guest.md §0 tabulates it: the OS disk on local-lvm,
# the HDD mirror, without the SSD flag; Postgres on a 32 GiB data disk on
# large_data; a static .41, because the stack's certificate and the
# analyst's browser name it; off between sessions. Built by hand first, and
# brought here by #1443: imported once, then rebuilt from the pipeline,
# which ADR-0081 allows because the graph is derived data and eden is not
# backed up. `garnet`, not `operator`: the user alexander's operator logs in
# as, with that key beside phoenix's (var.eden_operator_key_file).
bloodhound = {
for name, g in {
eden = { vm_id = 141, template = 901, cores = 4, memory_mib = 8192, disk_gib = 32 }
} :
name => merge(g, {
pool = "bloodhound"
mac_address = var.guest_identities[name].mac_address
smbios_uuid = var.guest_identities[name].smbios_uuid
startup = null
linux = true
on_boot = false
tags = ["bloodhound", "on-demand"]
password = null
username = "garnet"
datastore = "local-lvm"
os_disk_ssd = false
data_disks = [{ datastore = "large_data", size_gib = 32 }]
ipv4 = { address = "10.0.30.41/24", gateway = "10.0.30.1" }
dns_servers = ["10.0.30.1"]
extra_keys = [trimspace(file(pathexpand(var.eden_operator_key_file)))]
})
}

guests = merge(local.lab_domain, local.dotfiles, local.analyst, local.bloodhound, local.proof_guests)
Comment thread
Gerrrt marked this conversation as resolved.
Resolved

# A pool exists exactly while it groups a guest: it is derived from the
# guests, not listed beside them, so the last guest leaving a pool takes the
Expand Down Expand Up @@ -196,7 +228,13 @@ module "guest" {
secure_boot = lookup(each.value, "secure_boot", true)
# Absent means Proxmox's default. dot-windows runs VirtIO GPU (#1108).
vga = lookup(each.value, "vga", null)
ssh_keys = each.value.linux ? [trimspace(file(pathexpand(var.ssh_public_key_file)))] : []
ssh_keys = each.value.linux ? concat([trimspace(file(pathexpand(var.ssh_public_key_file)))], lookup(each.value, "extra_keys", [])) : []
# eden's shape (#1443). Absent means what every other guest has.
datastore = lookup(each.value, "datastore", "large_data")
os_disk_ssd = lookup(each.value, "os_disk_ssd", true)
data_disks = lookup(each.value, "data_disks", [])
ipv4 = lookup(each.value, "ipv4", null)
dns_servers = lookup(each.value, "dns_servers", [])

mac_address = each.value.mac_address
smbios_uuid = each.value.smbios_uuid
Expand Down
1 change: 1 addition & 0 deletions tofu/identities.auto.tfvars.example
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,5 @@ guest_identities = {
dot-nixos = { mac_address = "BC:24:11:00:00:C5" }
dot-windows = { mac_address = "BC:24:11:00:00:C6" }
garuda = { mac_address = "BC:24:11:00:00:A2" }
eden = { mac_address = "BC:24:11:00:00:8D" }
}
28 changes: 26 additions & 2 deletions tofu/modules/guest/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,24 @@ resource "proxmox_virtual_environment_vm" "this" {
size = var.disk_gib
discard = "on"
iothread = true
ssd = true
ssd = var.os_disk_ssd
}

# A data disk is created empty: the guest formats and mounts it (eden's is
# build-the-bloodhound-guest.md §2). It is part of the VM, so a destroy or
# a replace takes it, which ADR-0081 accepts for eden: its graph is derived
# data, one collection run from rebuilt.
dynamic "disk" {
for_each = var.data_disks
content {
interface = "scsi${disk.key + 1}"
datastore_id = disk.value.datastore
size = disk.value.size_gib
file_format = "raw"
discard = "on"
iothread = true
ssd = true
}
}

network_device {
Expand Down Expand Up @@ -123,7 +140,14 @@ resource "proxmox_virtual_environment_vm" "this" {
datastore_id = var.datastore
ip_config {
ipv4 {
address = "dhcp"
address = var.ipv4 == null ? "dhcp" : var.ipv4.address
gateway = var.ipv4 == null ? null : var.ipv4.gateway
}
}
dynamic "dns" {
for_each = length(var.dns_servers) > 0 ? [1] : []
content {
servers = var.dns_servers
}
}
user_account {
Expand Down
34 changes: 34 additions & 0 deletions tofu/modules/guest/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -145,3 +145,37 @@ variable "password" {
default = null
sensitive = true
}

# The variables below exist for eden (#1443), the first guest whose hand-built
# shape the module did not already describe. Each defaults to what every other
# guest has, so their plans are unchanged.

variable "os_disk_ssd" {
type = bool
default = true
description = "Tell the guest scsi0 is an SSD. True on large_data. False for an OS disk on local-lvm, the HDD mirror (eden, ADR-0081), where the flag would say otherwise."
}

variable "data_disks" {
type = list(object({
datastore = string
size_gib = number
}))
default = []
description = "Disks after scsi0, as scsi1, scsi2 and on, each with discard, iothread and ssd. eden's Postgres disk is the one so far (ADR-0081: 32 GiB on large_data)."
}

variable "ipv4" {
type = object({
address = string
gateway = string
})
default = null
description = "A static address in CIDR form and its gateway, written to cloud-init. Null is DHCP from morpheus's reservation, as every other Linux guest has."
}

variable "dns_servers" {
type = list(string)
default = []
description = "cloud-init nameservers. Empty leaves Proxmox to fill them from the host, as before."
}
6 changes: 6 additions & 0 deletions tofu/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ variable "ssh_public_key_file" {
description = "phoenix's key, the one ADR-0043 says the toolchain puts on every guest it builds. Same default as packer/."
}

variable "eden_operator_key_file" {
type = string
default = "~/.config/proxmox/eden-garnet.pub"
description = "garnet's public key from alexander, which eden's operator logs in with beside phoenix's own (build-the-bloodhound-guest.md §1). A public key, but kept beside the credentials so the checkout stays the same on every host."
}

# Each guest's MAC and, for the six domain guests, the SMBIOS UUID the
# hand-built one had. They are not in this repository: docs/security.md's
# "What this repository deliberately does not publish" withholds full MACs,
Expand Down
Loading