virtkit ships a GitLab custom executor that runs each CI job in a throwaway microVM: a fresh guest per job, destroyed when the job ends, with its own kernel and no shared state between jobs. Concurrent jobs each get their own VM. This guide covers configuring the runner, selecting the job image, sizing the guest, keeping the host inside its memory, controlling egress, and attaching services.
For the host-side setup mechanics (runner wiring, state layout, exit codes) see
also the driver README; the full
config reference is config.example.toml.
Point the runner's custom-executor stages at vk in
/etc/gitlab-runner/config.toml:
[[runners]]
[runners.custom]
config_exec = "/usr/local/bin/vk"
config_args = ["gitlab", "config"]
prepare_exec = "/usr/local/bin/vk"
prepare_args = ["gitlab", "prepare"]
run_exec = "/usr/local/bin/vk"
run_args = ["gitlab", "run"]
cleanup_exec = "/usr/local/bin/vk"
cleanup_args = ["gitlab", "cleanup"]The executor reads its host configuration from the usual virtkit config file
($VIRTKIT_CONFIG or /etc/virtkit/config.toml). Everything below that a job
controls is set through .gitlab-ci.yml job variables.
Job VMs mask VMX/SVM, so a job cannot boot microVMs of its own. A runner that hosts jobs which build or test virtkit itself grants them that:
[executor.vm]
nested = trueIt applies to the job VM whatever MICROVM_IMAGE names. In a compose: fleet it
also unlocks the per-service marker, so the nesting guest can be a sibling rather
than the primary:
services:
builder:
image: virtkit/build
x-virtkit: { nested: true }Without the grant that marker is refused when the fleet loads rather than cleared — a fleet that asked for a nesting builder must not look like it got one. Nesting reaches host KVM's nested paths, so the grant is the host admin's to make and there is deliberately no job variable for it.
The grant is per host and unconditional: every job VM this runner boots gets VMX/SVM, asked for or not. Set it on a runner dedicated to trusted pipelines, not one that also runs fork merge requests.
The host must be loaded with kvm_intel.nested=1 / kvm_amd.nested=1;
vk check --feature gitlab reports when it is not, and prepare refuses the
job rather than booting a guest that cannot nest — on either backend. The
setting itself is the libkrun backend's: under VIRTKIT_VMM=cloud-hypervisor
nothing masks VMX/SVM, so a job nests whenever the host allows it, setting or
no setting.
A host-side checkout — which dockerfile: and compose: job images need — clones the
sources on the runner instead of in the guest. Point checkout_dir at the runner's builds
tmpfs when the state disk is too slow:
[executor]
host_checkout = true
checkout_dir = "/builds"
checkout_cache_idle_secs = 1800A tree is reused by concurrent slot and project, so a second job on the same slot fetches
rather than re-clones. Before each checkout, and during vk gc, trees unused for longer than
this are removed — which is what keeps a tmpfs checkout_dir from filling with the
repositories of jobs that have moved on. Unset, checkouts inherit image_cache_idle_secs
(30 minutes by default). A tree a prepare or a running guest is using is never removed.
Under an explicit checkout_dir, virtkit keeps its trees in
<checkout_dir>/vk/<slot>/<project>, so the sweep walks only its own — a sibling tree from
another GitLab executor is outside the root it reads, and only trees virtkit created are
eligible within it. Each tree also records which directory it is, rechecked immediately before
removal, so bookkeeping left behind by a tree removed from outside cannot stand in for whatever
appears at that path next. The configured root must still be writable only by the runner
user: anyone who can replace the private subtree can interfere with the runner's checkouts.
Releases up to 0.31.0 put their checkouts directly under an explicit checkout_dir, which
leaves them outside the new root, so an upgrade neither migrates nor reclaims them: remove the
old <checkout_dir>/<slot> directories once no runner from before the upgrade is using them.
The first job per slot and project after the upgrade clones instead of fetching, since its tree
has moved. Under the default <state_dir>/checkouts root the trees keep their paths and are
still fetched into, but carry no identity record until the same slot and project is checked out
again, which is when they become reclaimable.
Job variables are read once per job, at the job level. GitLab passes them to the
executor prefixed with CUSTOM_ENV_; set them under a job's (or the pipeline's)
variables:. An empty string is treated exactly like an unset variable — it
does not override the host default.
| Variable | Effect |
|---|---|
MICROVM_IMAGE |
Guest image (prefix-based source; see below). Unset → local/default. |
MICROVM_CPUS |
vCPU count, clamped to the host [executor.vm] max_cpus ceiling. |
MICROVM_MEM |
Guest RAM as <n>G, clamped to [executor.vm] max_mem. |
MICROVM_USER |
User to run the job as inside the guest. |
MICROVM_EGRESS_ALLOW_IP / _ALLOW_NAME / _AUDIT |
Narrow the run-phase egress cap (see Egress). |
MICROVM_BUILD_EGRESS_ALLOW_IP / _ALLOW_NAME / _AUDIT |
Narrow the build-phase egress cap. |
MICROVM_USAGE_REPORT |
End this job's trace with what every job of its project has been using (see Sizing a project). |
MICROVM_IMAGE is prefix-based — the prefix names the source (when unset, the
job's plain GitLab image: is read the same way):
- unset →
local/default(the baked default bundle); local/<name>— a bundle directory under[local] dir;virtkit/<name>[:tag|@sha256:…]— a bundle in the[registry]repo;docker/<name>[:tag|@sha256:…]— an OCI image from the[docker]repo, booted directly (embedded kernel + agent). A reference[docker]does not route — and every one of them when that section is absent — is offered first to the vk-registry this runner already uses ([build] cache_registry/[registry]), which relays and caches the upstreams it is configured for, and pulled from the registry it names otherwise;dockerfile:<path>[?context=<dir>&buildcontext=NAME=DIR&arg=NAME=VALUE][#<stage>]— a git-defined image: virtkit builds the Dockerfile from the job's own checkout (eachRUNin a microVM, no Docker involved) and boots the result, so the job image lives in the repo instead of a registry.<path>is relative to the repo root; the build context defaults to the Dockerfile's directory (?context=.for the repo root),?arg=supplies a--build-arg(repeatable),?buildcontext=NAME=DIR(repeatable) declares an extra repo-root-relative directory aCOPY --from=NAMEorRUN --mount=…,from=NAMEmay read, and#<stage>selects a stage. Every path stays inside the checkout. Built images are cached and shared across jobs and runners. Requires[executor] host_checkout;compose:<file>#<primary>— a whole fleet from a compose file in the checkout: boots<primary>as the job VM and the other services (built or pulled the same way) as siblings on the job network. Samehost_checkoutrequirement. A service sizes its own guest with anx-virtkit: { cpus:, mem: }marker (default 2 vCPUs / 1G), clamped to the host[executor.vm] max_cpus/max_memceilings like the job's ownMICROVM_CPUS/MICROVM_MEM; the primary keeps following those job variables.x-virtkit: { nested: true }needs the runner to have set[executor.vm] nestedand is refused when the fleet loads otherwise: nesting reaches host KVM, so it is the runner's decision rather than a job's, the same way the executor never hands a job the guest PMU. A service's persistent state —x-virtkit: { persist_root: true },overlay,persistvolumes (see the README's Volumes and persistent state) — lives under.virtkit/beside the compose file, inside the checkout that every job starts by cleaning: it lasts for the job, across a service being stopped and started within it, and never from one job to the next.
my-job:
variables:
MICROVM_IMAGE: virtkit/myimage # :tag (default latest) or @sha256:…
test-in-repo-image:
variables:
MICROVM_IMAGE: dockerfile:ci/Dockerfile#test # built from this checkout, then bootedEach phase reports what it cost the runner. A job that builds its own image
(image: dockerfile:… / compose:…) gets the build figures as the build ends, next to
its timing breakdown — a job whose image is already built runs no build and reports none:
virtkit: build resource usage: cpu 8m12s, peak memory 3.2 GiB (largest process 1.2 GiB), read 1.8 GiB, written 4.1 GiB, sent 12 MiB, received 2.7 GiB
and the run figures come at the very end of the trace:
virtkit: job resource usage: cpu 2m14s, peak memory 1.6 GiB, overlay 1.2 GiB of 16.0 GiB, read 3.4 GiB, written 812 MiB, sent 3 MiB, received 941 MiB
cpu is all the CPU time the phase burned on the host, the guests' own execution
included — for the build, the stage guests plus vk's own work assembling and caching the
image; for the run, the job's microVM plus the host processes around it (the switch, the
virtio-fs daemons, any service VM). Against the phase's own duration it gives the
parallelism it reached: a ceiling for sizing one guest's vCPUs, since the total also
carries the host helpers and every sibling VM.
When the guest kernel's OOM killer terminates a process, a line identifies each victim before the figures above:
virtkit: guest OOM: the kernel killed cc1plus (pid 1234, 1.9 GiB RSS) at +48s (raise MICROVM_MEM)
The size is the victim's reclaimed anonymous RSS, and +48s is its guest uptime at death.
A step that exits with signal 9 and no message often indicates an OOM kill; raise
MICROVM_MEM. No line means either that no process was killed or that the guest was too old
or unresponsive to report it.
peak memory is the most the phase held at one time, not the memory left at the end: a
guest hands freed RAM straight back to the host, so its live figure says little about the
peak it passed through. A build reports its largest single process as well, because the
two size different knobs:
- the total is what the host had to have free while the phase ran — for a build,
several stage guests at once, so lower it with
[build] jobs; for a job, the run VM and its helpers together, i.e.MICROVM_MEMplus the VMM's own overhead. - the largest process — a stage guest, or vk itself — is what
[build] memhas to cover for one stage.
The two are measured as their processes allow, and err in opposite directions. A build's guests are gone by the time it reports, so both its memory figures are sampled while it runs — a spike shorter than a tenth of a second can be missed, and longer than that on a host where reading the process tree is expensive, since the sampler backs off from its own cost. Each sweep totals and ranks the same reads, so the total always covers the largest process reported beside it. A job's VM is read in one pass instead, summing the high-water mark of every process still running at the end: marks that never coincided both count, while a helper that peaked and exited earlier contributes its CPU time but no memory. Read either figure as a ceiling — pages the VMM shares with its virtio-fs daemons count in each — and for a job with services it spans every service VM, not just the one the stages run in.
Either line is omitted rather than guessed at when the figures cannot be had: the run ones for a job whose guest died and took the supervisor with it, the build ones for a build that shared the host process with another (both would be charged for each other's guests) or that the host could spare no sampler thread for.
overlay is how full the job filled its writable layer, against what that layer held.
With [executor] checkout_overlay (the default) a job builds on an overlay above its checkout
whose upper layer is a tmpfs inside the VM, so every write under CI_PROJECT_DIR — the build
tree, a package cache pointed there, an unpacked archive — is guest RAM, capped at
[executor] checkout_overlay_size (80% of the VM's memory by default). That cap is a wall a job
can hit: it fails with ENOSPC while every disk on the host sits empty, and the written
figure beside it says the job wrote nothing at all, because none of those pages ever reached a
block device. Read the pair as the room the job had left — 4.2 GiB of 16.0 GiB has plenty,
15.9 GiB of 16.0 GiB is a job about to fail on space — and raise MICROVM_MEM when a build
tree needs more, since the capacity follows it.
Because that checkout is read-only for the job's life, the guest caches its paths, attributes,
misses and directory listings for the whole job: git status or a build tool's dependency check
walks the tree against the host once, and every later pass is answered from the guest's own
cache. With checkout_overlay = false the share is read-write and keeps close-to-open
consistency (attributes re-fetched after 5 s).
The figure is the high-water mark, not what the layer held at the end: a job that unpacks an archive and deletes it would otherwise read as having needed nothing. It comes from the guest, which is the only place a tmpfs can be seen from, so a job with no overlaid checkout reports none — as does one on a guest whose agent is older than the figure.
[executor] checkout_overlay_size is where the two failures either side of that wall are traded
off, and the mark above is what to size it from:
- too low and a job dies for want of a partition on a VM that had the memory for it — the build tree hits the cap while gigabytes of guest RAM sit unused;
- too near 100% and the layer starves the job's own processes instead. Compilers spike late (linking), so the failure lands as an OOM kill of whatever the kernel picks — far harder to read than a filesystem reporting itself full.
It costs nothing below the cap: tmpfs pages are allocated on use, so a job that never fills the
layer is unaffected by how large it was allowed to grow. Nor does raising it commit host memory
— [executor.schedule] admission reserves MICROVM_MEM either way, so the host is already sized for a
job that uses all of its VM.
The default is 80% rather than the kernel's own 50% tmpfs default. That default exists to keep
an unevictable tmpfs from starving the long-lived services of a general-purpose machine; a
one-shot job guest has no such services, and the tree it is there to build is the legitimate
main consumer of its memory. Set "50%" to restore the kernel's behaviour, or an absolute
"12G" where every job on the runner is the same shape.
read and written are what the phase cost the runner's storage, not what its programs
asked for: read is what was actually fetched from the block layer, so a guest re-reading a
file it has already pulled into its own page cache costs nothing here, and written is what
was handed over to be written out. Two consequences are worth expecting rather than
discovering:
- work that never reaches a disk is not counted, because there is nothing to count: a
[executor] checkout_dirpointed at the runner's builds tmpfs, or a guest write absorbed by its overlay, is RAM, and a tmpfs page never reaches the block layer; - a job re-run on a warm host reads far less than the same job on a cold one, which is the cache doing its job rather than the measurement wavering.
sent and received are what the phase's guests moved in and out of the host, counted
by the switch that forwards it — payload from the guests' side, so the framing around it and
any retransmit under it are the host's traffic rather than the job's. What the guests send
each other is switched between them without ever being forwarded, so it is not in the figure.
Each phase counts its own: what a dockerfile: build's RUN steps fetched is on the build
line, the packages the job's steps installed and the artifacts they uploaded on the job line.
Only what a guest pulled is in either: vk resolves and pulls a base image itself, on the
host and outside the switch, so those layers are in neither figure. A job
on net.mode = "tap" has no switch in its path, so it reports no network at all rather than
zero.
A measured zero is printed — read 0 B says this job touched no disk, which is a fact
about the job. The clause is left off only where the figure could not be taken at all: a
kernel built without CONFIG_TASK_IO_ACCOUNTING, or a network nothing counted. The history
keeps the two apart as well, so a host that cannot measure never leaves a job remembered as
having moved nothing, and vk check says whether this kernel accounts block I/O:
ok usage block I/O accounted, process tree from the kernel's child lists
The figures above are one number per phase. For the shape of a job over time — which step
saturated the CPUs, when memory climbed, what process was running while nothing else moved —
each job's guest also records itself: every 10 seconds it samples its own /proc and appends
a sample to a log the host keeps after the VM is gone.
<state_dir>/atop/2026-08-11/42137-acme-web-test_unit/atop.log
Every job's trace ends with the account of it, in a section GitLab's web UI shows folded — one line to open when a job is worth looking into, and out of the way when it is not:
▶ what the job's guest did (vk atop 42137)
Opened, it holds exactly what --summary prints below. A raw trace (the artifact, curl of the
trace API, a runner log) has no UI to fold it, so it shows the same lines unfolded between the
section markers — which is what a reader piping a trace through grep wants anyway. A job whose
guest recorded nothing gets no section at all rather than an empty one.
On the runner, vk atop finds one again from a job id, or from any part of a recorded
job's name — the newest run answering — printing just the path, so it composes with whatever
reads it. Nothing matching means an empty stdout and a non-zero exit:
vk atop 42137 # that run
less $(vk atop test_unit) # the last run of this jobThe name to give is the one in the directory, which is the job's name with anything outside
[A-Za-z0-9._-] replaced — a test:unit job is recorded as test_unit. A job id matches only
the id a directory name leads with, so 42 never answers for job 42137.
--summary reads the log for you instead, accounting the whole job — what its guest did with
its processors and memory, what it moved, where it was held up, and which of its processes the
time went to:
$ vk atop 42137 --summary
virtkit: 42137-acme-web-test_unit — what its guest did:
recorded 2026/08/12 07:21:49 → 07:21:59 UTC (10s), 11 samples at 1s, the first
covering the guest's boot (counted in the totals, not the rates)
guest runner on 2 cpus, 988 MiB memory, no swap
cpu 1.7s of cpu time — 0.2s user, 0.9s system, 0.6s stolen by the host; 0.1s
waiting for disk; 16% busy at peak, 5% on average, cpu 1 busiest at 22%
of its own time
load 0.00 at peak, 0.00 / 0.00 / 0.00 at the end (1m / 5m / 15m); 285 context
switches a second at peak
memory 63 MiB held at peak (6% of the VM), 146 MiB of cache at peak
pressure cpu 182ms, 0.9% at 07:21:51; io 50ms, 0.3% at 07:21:55
disk 2 MiB read, 92 KiB written — vda busiest, 96ms busy
network eth0 received 42 B, sent 714 B
cpu over time ▁▁▁▁▁▁▁▁▁▂ (peak 16%)
memory over time ▁▁▁▁▁▁▁▁▁▁ (peak 6%)
what ran — the 10 of 61 that used the most cpu
command pid cpu peak rss read written
cc1plus ×1184 (2 failed) - 6m12s 184 MiB 1 GiB 96 MiB
/init tsi_hijack 1 0.3s 11 MiB 2 MiB 20 KiB
…
(Wrapped here to fit the page — the real output is one line per label.)
The commands a job ran over and over are one row each, with the number of runs and how many of
them failed: a build that forks a thousand compilers is a thousand processes no sweep ever saw,
and one row of cc1plus ×1184 says what a thousand rows of one run each would bury. A process
the sampler did watch keeps its own row, however many namesakes came and went around it.
--view walks the recording a sample at a time in a full-screen panel — the guest's
processors (each core its own bar), memory and swap, pressure, disks and interfaces, then the
processes ordered by what they were using at that moment:
42137-acme-web-test_unit — sample 11/11 at 07:21:59 +1s
cpu 16% [██ ] user 3% sys 10% wait 0% steal 3% 0.2s of cpu time
cores 0:[▊ ] 10% 1:[█▊ ] 22%
mem 6% [▊ ] 63 MiB of 988 MiB cache 146 MiB no swap
psi cpu 0.4% mem 0.0% mem-full 0.0% io 0.0% io-full 0.0%
disk vda read 0 B/s write 4 KiB/s busy 4ms
net eth0 in 0 B/s out 42 B/s 0 tcp connections, 0 resent
pid st >cpu memory disk command
65 S 0.1s 1 MiB 8 KiB sh
700 E 0.0s 512 KiB 1 KiB true
1 S 0.0s 11 MiB 0 B /init tsi_hijack
…
←/→ step home/end jump c/m/d sort by cpu a whole job / filter q quit
The st column is the state each task was in, E marking one that ended during the sample.
←/→ step through the samples and Home/End jump to either end; c, m and d order
the processes by processor time, memory or disk; a swaps each process's activity in this
sample for its whole-job totals; / filters the table to the commands matching what you type;
q leaves. The first sample is marked, because its counters cover the guest's boot rather than
an interval. The filter matches on ASCII.
--follow is the same panel over a job that is still running: samples appear as the guest
commits them, and while the view sits on the last one it moves with them. Stepping back holds
it still — the header says [paused] — until End, or stepping forward onto the last sample,
takes the live tail up again. Both need a terminal on stdin and stdout; without one (a pipe, or
TERM=dumb) they say so and point at --summary. A process's disk figures read - where the
guest's kernel accounted none, as in the summary.
Totals cover every sample, including the first — a job's VM boot is part of what the job cost.
A figure computed over an interval — the cpu percentages, the rate of context switching, the
sparklines — leaves that first sample out, because it covers however long the guest had been up
rather than one interval, and averaging a boot into the picture flattens everything after it. A
figure that was simply the reading at a moment — the load, the memory held, a pressure average
— comes from every sample, the first included, so a peak the guest reached while booting is
still reported as one. pressure is the line to read when a job is slow for no
visible reason: it is time the guest spent waiting for a processor, for memory or for its
disk, which no total of what it used can show. A figure the kernel could not measure prints as
-, never as a zero.
--json writes the samples themselves, one object per line, for anything that would rather
compute than read. The units are the log's own: pages, with their pagesize beside them;
ticks, with their hertz beside them; 512-byte sectors; and KiB for a process's resident size
(rsize_kib). A counter the guest's kernel does not have is null rather than a zero — as is
a scale whose record a sample did not carry, so check hertz and pagesize before dividing by
one. A log with no complete sample writes nothing and exits 0, where --summary reports that
there is nothing to account yet:
vk atop 42137 --json | jq -c '{t: .epoch, user_sys: (.cpu.user + .cpu.system)}'
vk atop 42137 --json | jq '.procs | max_by(.rsize_kib) | {name, rsize_kib}'The format is the text atop -P prints, pinned to the field order of atop 2.8.1 (what Debian
12 ships), so anything that already reads that — a parser, or plain grep/awk — reads these
logs. Each line is one record: a label, this guest's name, the epoch, the date and time, the
seconds the sample covers, then the label's own fields. A SEP line closes each sample, and a
RESET line opens the first one, whose counters cover the guest's whole boot. Counter labels
carry per-interval differences; size labels carry the value as it stood.
The system labels are CPU, cpu (per processor), CPL, MEM, SWP, PAG, PSI, DSK and
NET; every process gets a PRG, PRC, PRM and PRD line, and so does every task that
ended during the interval — those carry state E, the status they exited with and how long
they lived. So the busiest samples of a job
are one sort away:
awk '$1 == "CPU" { print $5, $9 + $10 }' atop.log | sort -k2 -n | tail # time, busy ticks
grep '^PRM ' atop.log | sort -k12 -n | tail # largest processes (column 12 only while no
# process name holds a space — see below)Worth knowing before reading a log:
- it is the job's own VM, not the fleet: a
compose:job's service VMs are not sampled; - a process that starts and exits between two samples is still recorded — the guest's kernel
reports every task as it dies, and those land in the sample covering the exit with state
E, their whole life on them. Two things follow: a dead process has no command line left to read, so it appears under the name the kernel keeps (cc1plus, not the arguments it was given), and a guest exiting tasks faster than the sampler can read them loses some — the log says so on the guest console, and it takes extreme churn; - a thread exiting is not recorded on its own: its time is charged to the process it belongs to, which is what the process's own record carries;
- there are no per-process network figures (
PRN), which real atop needs a kernel module for. What the job moved in and out of the host is already on its resource line above; - a disk that moved nothing in a sample gets no
DSKline in it: the guest kernel carries sixteen ramdisks and eight loop devices that never see a sector; - a field this guest cannot source carries the value atop itself prints when it has no answer — CPU frequency, cgroup limits and the proportional set size are the ones to ignore rather than read;
- a process's name and command line are parenthesised and may hold spaces, so column numbers
shift on the
PR*labels:awk/sort -kare safe on the system labels, and aPRG/PRMline has to be split on its parentheses first; - the timestamps and the date directory are UTC.
Recording is not free, and it is on by default. Each job's guest gets a read-write virtio-fs
share of its own archive directory — the one directory it can write, and it can write anything
and any amount into it until the job ends — and it boots with psi=1, which its own scheduler
pays for in exchange for the PSI label. atop = false gives up the recording and the share;
the guest still boots with psi=1, since idle page-cache trimming asks for the same figures —
only turning that off as well ([executor.vm] reclaim or [executor.vm] balloon) gives the scheduler it back.
Set [executor] atop_interval_secs for a finer or coarser resolution, and atop = false to
record nothing:
[executor]
atop = true # default
atop_interval_secs = 10 # default; at least 1
atop_retention_days = 14 # default; 0 keeps only today's jobsThe archive is kept for atop_retention_days and then reclaimed: each day past the window is
dropped whole by the first job recorded the following day — once a day, not once a job — so the
directory stays bounded on a runner nobody visits. The window is counted in whole UTC days
beside today's, so 14 leaves fifteen date directories. Only a directory whose name is a date
in that YYYY-MM-DD form is ever dropped: a file, a symlink, or anything else an operator
leaves in the archive is never touched, so a log worth keeping can simply be renamed.
Two things to know before setting it. A day goes even if a guest is still recording into it, so
a job that outlives the window has its log unlinked while it is still writing — the guest keeps
appending to a file nobody can open again. With 0 that is any job running past midnight, and
with a short window any job outliving it. And the reclamation belongs to the recording:
atop = false stops both, so an archive already on disk stays at the size it reached and has to
be removed by hand.
The phases are sized independently, and by different people: the run VM by the job, the build by the host.
| Run phase | Build phase | |
|---|---|---|
| vCPUs | [executor.vm] cpus, per job MICROVM_CPUS (capped by [executor.vm] max_cpus) |
[build] cpus, per stage guest — unset = the host's CPUs shared out among the stages running at once, between 2 and 4 each |
| RAM | [executor.vm] mem, per job MICROVM_MEM (capped by [executor.vm] max_mem) |
[build] mem, per stage guest — unset = 4G; a stage overrides it with # vk: mem=… or --stage-mem |
| Concurrency | one VM per job (the runner's own concurrent) |
[build] jobs stages at once — unset = as many of this build's stages, smallest first, as fit in 80% of host MemTotal, capped at 16 |
Build sizing is host configuration only — there are no MICROVM_* equivalents, because
built images are cached and shared across jobs and runners, so no single job owns the
build guest it happens to trigger. Raising [build] mem lowers the derived jobs count
unless you also set it: per-stage headroom and stage concurrency trade against each other.
jobs is a ceiling, not a promise. It is derived from the memory the host has, so that
a build's width does not depend on the minute it started — but a host also carries job
VMs, other builds and whatever else it runs, so a stage additionally reserves its guest
RAM against what is free before it boots, and waits when the host is full. The build
announces both, and a stage parked on memory says so in the trace:
virtkit: build: up to 6 stage(s) at once (from host memory), each cpus=12, mem=4G
virtkit: build: host memory 31241 MiB, 15180 MiB in use elsewhere, 3124 MiB held back — 12937 MiB free for stage guests now
With stages sized individually the ceiling is not a division: it is as many of this build's stages as fit, smallest first, in the same 80% — and never more stages than the build has. The stages that differ are named on the line, and the gate's own reading is in MiB, since a build of one 24G stage and four 2G ones has no single number of stages its free memory is worth:
virtkit: build: up to 4 stage(s) at once (from host memory), each cpus=12, mem=4G; sized apart: compile mem=8G cpus=16
One stage often needs far more than the rest — the compile, the link — and sizing every
stage for it wastes both memory and concurrency. A # vk: comment above a stage's FROM
sizes that stage alone:
# vk: mem=8G cpus=16
FROM rust:1.90 AS compilemem and cpus are the keys; either may be omitted, and a stage without the comment takes
[build] mem / [build] cpus as before. vk build --stage-mem compile=8G --stage-cpus compile=16 says the same thing for one run without editing the file — the flags win over the
comment, field by field, and name a stage by its AS name, or stage<N> by position without
one. In a build of several units (--compose, or several --target) a name sizes that stage
in every unit declaring it. A flag naming a stage no unit declares fails the build, as a hint
that cannot mean anything does. It rides a comment so the Dockerfile still builds with docker build, and so the size stays out of the instruction stream cache keys are computed from: a
stage built at 2G and the same stage built at 8G produce the same artifact under the same key,
so tuning one throws no cache away. A hint larger than the host can give one stage is held to
what the host has, with a line in the build log — the 24G a stage wants on the build server
must not make it unbuildable on a laptop. Anything the hint cannot mean — an unknown key, an
unreadable size, a # vk: line that precedes no FROM — fails the build rather than being
ignored.
Each stage reports its peak guest memory when it finishes and in the final timing breakdown,
alongside the guest size. The guest measures MemTotal - MemAvailable, excluding reclaimable
page cache that a host-side VMM measurement would include. The figure is read from each of a
stage's guests as it goes, so a stage killed out of memory prints no completion line, and the
final breakdown carries only what its earlier guests had reported — a lower bound on what it
needed, or nothing at all if it died in its first.
Back to the gate: a build's own guests are charged at their declared size from the moment they
are admitted, never at what they have faulted in so far — a guest that booted seconds ago has
touched almost none of its RAM, and a gate that believed MemAvailable would wave the next
stages in against memory the first one is about to take. Everything else on the host is
measured afresh each time a stage is about to be admitted, so a build narrows when a job VM
starts beside it and widens again when one ends. A stage with nothing else of this build's
live is always admitted, whatever the host looks like: a build with no way to make room should
be slow, not stuck. Stages that do wait are admitted oldest first, so a large stage is not
overtaken indefinitely by the smaller ones queued behind it.
The measurement is of memory actually in use, so it lags a job VM that [executor.schedule] mem_budget has just granted but which has not faulted its RAM in yet — the same trap the
gate avoids for the build's own guests, one level out. Where jobs and builds share a host,
that is a reason to bound both up front rather than to rely on the gate alone. Set
[build] no_mem_gate to switch the gate off entirely and let jobs bound the build by
itself.
A guest holds every file page it ever read until something in it needs the memory, so a job that untarred a source tree and built it goes on holding that tree long after the stage that read it — duplicated in the host's own page cache, or, on a busy runner, in its swap. Pages a guest frees already come back through the balloon's free-page reporting; the file cache does not.
[executor.vm] reclaim makes it come back too:
[executor.vm]
reclaim = "auto" # the defaultauto evicts, whenever the guest is not under memory pressure, the file cache it has not
touched for a minute or two — by age, through the kernel's multi-gen LRU, so what a build is
still re-reading stays and what it read once goes. A size ("512M", "2G") or a share of the
guest's RAM ("5%") keeps that much cache as a fixed floor instead, and "off" leaves the
guest alone. Compose services follow the job VM's setting unless they declare their own
x-virtkit.reclaim.
It rides on the balloon, so [executor.vm] balloon = false stops the job VM trimming: without
free-page reporting the trimmed pages never reach the host, and the job would lose its cache
for nothing. Compose services attach a balloon of their own whatever [executor.vm] balloon says, so
they go on trimming under the setting they inherited.
Job virtio-fs shares — the checkout (host_checkout), tools tree and [executor.share] — use
DAX windows to read the host page cache directly. Previously, each guest cached its own
copy: ten concurrent job VMs reading a tools tree held eleven copies on the host.
[executor.vm] dax sets the window size:
[executor.vm]
dax = "8G" # per share; the default: files of 1M and more through the window
# dax = "8G:always" # every file through the window
# dax = "8G:inode=64K" # another size floorThe window reserves guest address space, not memory, and costs nothing until mapped.
"off" disables it. Each mapping costs the host an mmap and the guest an EPT invalidation
per 2 MiB range whatever the file's size, which a source tree's small files never repay:
by default only regular files of 1M and more are mapped (dax=inode; the host marks them),
the rest read through the guest page cache as without DAX. Each guest supports 64G of
windows — eight at the default size, with further shares served without DAX. Guests with
more than 63.25G of RAM have no room for windows and receive none.
Mappings are 4 KiB-granular, so the benefit is avoiding a tools tree's memory cost per
VM, not per-fault latency. Compose services inherit the job VM's
setting unless they declare x-virtkit.dax. DAX requires the built-in VMM;
cloud-hypervisor has no DAX path and serves shares the ordinary way whatever this says.
gitlab-runner decides how many jobs to take with concurrent, a count that knows nothing
about what those jobs boot. Past the host's RAM the OOM killer arbitrates — it takes a
VMM, and that job dies mid-stage with no explanation in its trace.
Set a memory budget and a job instead claims the guest RAM it is about to boot, and waits when the host is full:
[executor.schedule]
mem_budget = "50%" # total guest RAM admitted at once; exact "48G" also works
wait_timeout_secs = 600 # then the job gives upA percentage is resolved from this host's MemTotal, so one config scales across runners of
different sizes. It is rounded up to the whole-GiB unit job sizes come in — a runner's
MemTotal always reads somewhat under its nominal size, and rounding down there would cost a
whole job's worth of budget — and any percentage is capped at the whole GiB the host actually
reports, so 100% cannot round past the machine. A
host whose /proc/meminfo cannot be read fails a percentage budget rather than guessing one.
The budget stays a guest RAM ceiling either way, not a host one: leave enough of the host outside it for the VMMs, a tmpfs-backed checkout, and anything else the box runs. Unset, admission is disabled (the default).
A job that waits says so in its trace, and says so again when it gets in:
virtkit: waiting for 8192 MiB of the host's 49152 MiB memory budget (45056 MiB reserved, 1 job(s) asked first)
virtkit: admitted after waiting 34s for memory
Claims are held for the job's whole life and released at cleanup — or by the kernel, if
the job dies, so a crash frees its memory rather than leaking the budget. Jobs are
admitted oldest request first, so a large job cannot be starved by a stream of small
ones; the cost is that a small job may queue behind a large one it would have fit
alongside. Several runner processes on one host share the ledger as long as they share a
state_dir and run as the same user.
A job that gives up waiting exits a system failure, which GitLab retries only for jobs that ask for it:
heavy-job:
retry:
max: 2
when: runner_system_failureA MICROVM_MEM above the whole budget is clamped to it, the same way [executor.vm] max_mem clamps
one: a job asking for more than the host can ever admit would otherwise fail every attempt.
Keep [executor.vm] mem itself at or under the budget, though — a default no job can fit in makes
every job on the host fail admission.
By default the budget counts what a job declares (MICROVM_MEM, or [executor.vm] mem), which
is nearly always far more than it uses — so a runner gated this way runs fewer jobs than it
could.
Turn on from_history and a job is instead admitted against its own recent peaks — the
same figures its trace reports:
[executor.schedule]
mem_budget = "50%"
from_history = truevirtkit: reserving 531 MiB from what this job has been using (it declares 2048 MiB)
Runs are remembered per project and job name, under
<state_dir>/history/<project id>-<slug>/<job name>-<digest>, so test:unit is predicted
from test:unit, not from the heavy build beside it. The project's own id scopes it, since
two projects can share a slug; the digest of the job's name as written keeps two names that
reduce to the same filename — test:unit and test/unit — from sharing a history.
Every job records its run, whether or not this setting is on, so turning it on has something to work from. Each file keeps at most its thousand most recent runs, but there is one file per job name a project has ever run and nothing prunes them, so the directory grows slowly with the job names a host has seen. It holds no state anything else depends on: delete it, or any file in it, whenever you like — the jobs it covered fall back to their declared size for one run and start again.
A reservation is the largest run of the last 14 days plus 25% — the largest, because a job that peaks one run in five needs room for that run — never below 512 MiB and never above what the job declares. A job with no history yet falls back to its declared size, as does a job whose name changed.
A job's runs also count only while its memory ceiling is the one they ran under. Raise
MICROVM_MEM from 4G to 16G and the job is treated as unknown again for one run: what it
reached pressed against 4 GiB says nothing about what it reaches given 16, and predicting
the new job from the old one is how a host ends up overcommitted the moment someone widens a
job that needed it. Lowering the ceiling starts it again for the same reason, and putting a
ceiling back finds the earlier runs still there — they are set aside, not discarded, for as long
as the file's thousand-run cap has not pushed them out behind the newer ceiling's runs.
The window is days rather than a count of runs, because what changes a job's appetite — a dependency, a fixture, the code — changes on calendar time, while the same number of runs spans half an hour on a busy merge queue and most of a year on a release job. A job too quiet to have anything inside the window keeps its last five runs however old they are, so a monthly release job is still estimated from what it did rather than from what it asks for. A spike therefore stops being believed by ageing out, not by being outvoted.
Every job trace says where it stands, whether or not the host reserves this way — reading it is how you decide to:
virtkit: job resource usage: cpu 2m14s, peak memory 1.6 GiB, overlay 1.2 GiB of 16.0 GiB, read 3.4 GiB, written 812 MiB, sent 3 MiB, received 941 MiB
virtkit: most this job has used lately: memory 2.1 GiB, overlay 1.9 GiB of 16.0 GiB, read 12.0 GiB, written 3.1 GiB, sent 40 MiB, received 4.2 GiB over 37 runs; the next run reserves 2.6 GiB
Only the memory is reserved against; the writable layer and the traffic ride along because a job that fills its overlay or pulls 4 GiB in and out of the host every run is a fact about the host worth knowing. Each figure is its own maximum over the window, so they need not all come from the same run.
The run count is what the estimate rests on: the runs of the last 14 days, or the last five
however old for a job too quiet to have that many. The ; the next run reserves … clause
appears only where the host really does reserve that way.
The VM is still given the memory it declares; only the reservation shrinks. That is the point: guest RAM is faulted in on use and handed back when freed, so a generous declared size costs nothing at runtime, while an honest reservation is what lets the host run more than a handful of jobs.
The trade is real, though: a job that suddenly needs much more than it ever has — a new dependency, a bigger fixture — is admitted against the old figure, and the host can be overcommitted for that one run. The 25% headroom and the fortnight window absorb drift, not a step change. Leave it off until a few pipelines have been measured.
One job's trace says what that job needs. vk gitlab usage says what a whole project does —
every job this host remembers, heaviest first, with what its next run would reserve and what
the lot would reserve if they all ran at once, which is the figure [executor.schedule] mem_budget has
to cover:
$ vk gitlab usage acme
virtkit: 42-acme — what its jobs have been using lately:
job memory overlay ceiling reserves runs read written sent received
build 5.9 GiB 3.9 GiB / 4.0 GiB 8.0 GiB 7.3 GiB 24 3.4 GiB 812 MiB 3 MiB 941 MiB
test_unit 500 MiB - 2.0 GiB 625 MiB 37 - - 2 MiB 88 MiB
virtkit: 2 jobs; all at once they would reserve 7.9 GiB, against a budget of 16.0 GiBThe argument is any part of a project's <id>-<slug> directory name, so the slug alone will
do; without one it reports every project on the host. reserves is what each job's next run
would claim: its declared size, or — with [executor.schedule] from_history on, as above — what its
history says it needs. overlay is the writable layer against its capacity, the one column
holding a figure a job can fail against rather than merely be sized by: build above has
150 MiB of room left. A - is a figure no run could measure
— an unaudited kernel for the disk columns, a net.mode = "tap" job for the network ones, a
job with no overlaid checkout for overlay — which is not the same as a job that moved nothing.
A job can ask for its own project's report and get it in its trace, for an operator with the GitLab UI but no shell on the runner:
project-sizing:
when: manual
variables:
MICROVM_USAGE_REPORT: "1" # 1/true/yes/on
script:
- echo sizingIt reports that job's project and no other, whatever else this host has run — matched
exactly, so a project whose directory name merely contains another's is not confused with
it. Which project a job belongs to is taken from the runner's own account of the job, not
from the CI_* variables beside it, so it is not something a job can name for itself — and on
a runner old enough not to write that account, the report is refused rather than answered from
what the job claims.
On a multi-socket runner, guest RAM that is scattered across the sockets costs the job for its whole life: each page lands on whichever node the thread that first touched it happened to be running on, and the scheduler then moves the vCPU threads away from it. A VM placed on one node — its vCPUs pinned there and its memory preferring that node — pays none of that.
Placement is on by default and does nothing on a single-node host, which is most of them. It
is host-wide, not a runner setting: the same key places vk run guests, build stage guests
and compose services on the machine.
[numa]
mode = "auto" # the default; "off" or "interleave" insteadauto puts each VM on the node with the most room left that it fits on, and interleaves one
that fits nowhere — more RAM than the node has room for, or more vCPUs than the node has
CPUs — across every node instead, which at least spreads it evenly rather than piling it onto
whichever node booted it. The node is a preference for memory and not a cap: a guest that
grows past what its node has free spills onto another node instead of driving that node into
reclaim and an out-of-memory kill, so a placement the host outgrows costs locality and nothing
more. Each VM says which it got:
virtkit: NUMA: node 1 (cpus 24-47) of 2
virtkit: NUMA: interleaved across 2 nodes (the VM does not fit one node)
With mem_budget set, placement uses the admission ledger. Each job's node is recorded
beside its claim, so later jobs count what others were granted, including RAM they have
yet to fault in. Without a budget, jobs use live per-node free memory plus this process's
outstanding allocations, like other VMs. Live memory figures lag newly admitted VMs;
setting a budget accounts for those grants across runners on a busy host.
off places nothing and leaves it to the kernel. interleave spreads every VM across all
nodes without choosing one, for a host whose jobs are all larger than a node, or one where
something outside virtkit is taking memory the ledger cannot see.
Placement is best-effort: it is applied to the VMM process before it starts, and a host that refuses it — a cpuset that excludes the chosen node, a kernel built without NUMA — gets a line in the log and a VM that boots unplaced, never a failed job.
To measure what it is worth on a given host, run the same job (or the same vk build) twice,
once with mode = "off" and once with mode = "auto", and compare the wall time the trace
reports. The kernel's own counters say whether the difference is the placement: while the VM
runs, numastat -p <vmm> breaks that VMM's resident memory down per node. -p takes a
process-name pattern as well as a pid, which is the easier handle on a job: under libkrun the
job's VMM is named vk:<hostname> ([executor.vm] hostname), under cloud-hypervisor it is
cloud-hypervisor. numastat with no arguments counts the allocations that did not land on
the node that asked, numa_miss and other_node. A placed VM concentrates its pages on one
node and stops growing other_node; an interleaved one splits them evenly. Check the
virtkit: NUMA: line first when a run measures the same either way: a VM wider than a node is
interleaved in both, and interleaving is not what the comparison is testing.
A waiting job has already been assigned by GitLab: it holds a concurrent slot and its own
timeout runs while it waits. Admission keeps a host from overcommitting; it does not keep
work out of the runner. For that, see below.
Nor does it cover everything a job boots. The claim is taken at the start of prepare, so a
job that builds its own image holds its full guest RAM across that build — and the build's
own stage guests ([build] mem × [build] jobs) are outside the budget entirely. A host
has to leave room for both. An auto [build] jobs does not close that gap on its own: it is
a fixed share of the whole machine, not of what is left of it. A stage does also wait for
the memory it is about to use (see Sizing the two phases), so a
build narrows while jobs hold the host — but that waiting measures memory already faulted
in, not what the job ledger has granted, so it lags a job VM that has only just been
admitted. Set [build] jobs by hand to bound the build up front as well.
concurrent decides how many jobs gitlab-runner accepts. Sized by hand it is a guess
that has to hold for the worst pipeline, so it is usually set low — and a job it turns away
stays pending in GitLab, holding no slot, running no timeout, free to land on another
runner. That is the one thing admission cannot do for a job already assigned.
gitlab-runner re-reads config.toml when it changes, so concurrent can follow the host.
Two pieces do that, split along the privilege line:
| Runs as | Does | |
|---|---|---|
vk tune |
the runner user | Reads the ledger, works out how many jobs fit, writes that one number to <state_dir>/schedule/desired-concurrency |
vk-runnerctl |
root | Reads that number, clamps it into a range it configures, edits concurrent, puts the file back atomically |
The split is the point: config.toml is root's, and granting vk the right to write it
would grant root outright — vk run boots VMs with arbitrary mounts. vk-runnerctl instead
takes no arguments and no paths; everything it touches is named in its own root-owned
config. The worst an attacker who owns the runner user can do is ask for a concurrency an
administrator already allowed.
# /etc/virtkit/runnerctl.toml — root-owned, 0644
runner_config = "/etc/gitlab-runner/config.toml"
desired_file = "/var/lib/virtkit/schedule/desired-concurrency"
min = 1
max = 12 # never exceeded, whatever is requested
cooldown_secs = 60 # shortest interval between two writes
stale_secs = 300 # a request older than this is treated as gone
# reload_command = ["systemctl", "kill", "-s", "HUP", "gitlab-runner"]Run the measuring half from a user timer every half minute (vk tune), and the privileged
half either from a root timer — nothing is granted to anyone — or from the runner user
through a sudoers rule that allows no arguments:
gitlab-runner ALL=(root) NOPASSWD: /usr/local/lib/vk/vk-runnerctl ""
What it does with the file is deliberately narrow: it rewrites the single concurrent line
and proves the result differs from the original at that key alone before installing it, so
comments, key order and registration tokens survive untouched, and the original is kept once
as config.toml.vk-orig. Lowering the number never disturbs a running job — it only stops
new ones being taken.
The number itself is "the jobs running now, plus what both the budget and the host still have room for", at the size a job on this host typically reserves:
virtkit: runner concurrency 2 (6144 of 8192 MiB committed by 1 job(s), typical job 2048 MiB, 23040 of 32768 MiB host memory available)
It falls the moment the host fills and climbs back one step at a time, because a job that
has just started has not yet reached its real size. Alongside the ledger, MemAvailable
limits how many typical jobs fit after keeping 15% of physical RAM free. A tmpfs-backed
repository checkout therefore lowers the offered slots by its actual allocated size, as does
an unrelated service on the box, without a guessed per-repository reserve; reclaimable page
cache does not, since MemAvailable already counts it as available. Below that 15% floor the
runner takes nothing beyond what it is already running, whatever the ledger says. One slot is
always offered even then, since concurrent = 0 is not a setting gitlab-runner has and a
runner that accepted nothing at all would never pick up again. And if vk tune stops writing,
the runner is not left throttled: vk-runnerctl walks concurrent back up to max a step at
a time.
A host whose /proc/meminfo cannot be read reports host memory unreadable and schedules on
the budget alone.
Getting the number wrong is cheap on purpose. It decides what the runner accepts, never what is committed: too high and the extra jobs queue at the admission gate exactly as before, too low and the host idles until the next run. The gate is the guarantee; this is the throughput.
A job's networking is governed by the host [egress] configuration. virtkit runs
one userspace switch per job; the booted job guest and every service
VM share that switch, each with its own address. By default they all share the run
policy, but a service may narrow itself further with its own allowlist (see
per-service egress) — the switch enforces the policy per
source VM, so one VM cannot use another's.
There are two independent phases, each with its own host cap:
[egress]— the run phase: the booted job guest and its service VMs.[egress.build]— the build phase: theRUNsteps of a git-defined image (image: dockerfile:…) or a composebuild:service. Absent by default, meaning the build runs unrestricted, likedocker build.
Each phase has two dimensions — allow_ip (direct, non-DNS egress) and
allow_name (DNS suffixes) — set in the host config:
[egress]
# DNS suffixes, dot-anchored: "corp.example.com" also matches *.corp.example.com
allow_name = ["corp.example.com", "crates.io", "debian.org"]
# Direct-IP egress as CIDRs, optionally port-scoped as CIDR:port (else any port).
# Needed only for a destination a job reaches by literal IP (no DNS name).
allow_ip = ["10.10.140.49/32"]The list semantics are the load-bearing part:
- an absent list is unrestricted for that dimension;
- an explicit empty list (
allow_name = []) denies that dimension; - a phase is unrestricted only when both its lists are absent;
- consequently, once either dimension is present the phase is a restricted
allowlist and the other, omitted dimension is deny-all. For example, with
allow_nameset butallow_ipabsent, all direct-IP egress is denied — a job reaching a host by literal IP needs that IP added toallow_ip.
A job may narrow its egress to a subset of the host cap, never widen it. Each
variable is a list — one or more entries separated by spaces, commas, tabs, or
newlines (empty entries are ignored). A # begins an end-of-line comment, so a
block-scalar list can document each entry inline:
least-privilege-job:
variables:
# Any of these separators work; mix freely.
MICROVM_EGRESS_ALLOW_NAME: "crates.io debian.org github.com"
# equivalently: "crates.io,debian.org,github.com"
MICROVM_EGRESS_ALLOW_IP: "10.0.0.0/8 192.168.5.10/32"The value is always a string, never a YAML sequence — a GitLab variable value
cannot be a list. Use the separators above, or a block scalar (|) to keep a long
list readable one-per-line (newlines are valid separators):
variables:
MICROVM_EGRESS_ALLOW_NAME: | # a string, split on newlines — not a YAML list
crates.io # Rust registry
static.crates.io # crate downloads
index.crates.io # sparse index
debian.org # apt mirrors (also covers deb./security.) variables:
MICROVM_EGRESS_ALLOW_NAME: # ✗ invalid — a variable value cannot be a sequence
- crates.io
- debian.org(A genuine list is only used in the host [egress] config — allow_name = ["crates.io", "debian.org"] — not in a job variable.)
You can also reference other CI/CD variables — GitLab expands $VAR / ${VAR} in
the value before the executor sees it, so a shared list defined once expands into
the list virtkit parses:
variables:
COMMON_EGRESS: "crates.io debian.org github.com" # define once (group/project/pipeline)
rust-job:
variables:
# Reference the shared list, and add a job-specific host.
MICROVM_EGRESS_ALLOW_NAME: "$COMMON_EGRESS internal.corp.example.com"- Against an unconstrained (absent) cap dimension, the variable defines the list freely.
- Against a restricted cap, every requested entry must fall within it
(a suffix of some
allow_name, or a subset of someallow_ipCIDR). A request outside the cap fails the job with a job-visible error. - Against a deny-all (empty, or omitted-while-restricted) dimension, a job can add nothing.
The same applies to the build phase via MICROVM_BUILD_EGRESS_ALLOW_IP /
_ALLOW_NAME.
A services: entry can carry its own MICROVM_EGRESS_ALLOW_IP /
MICROVM_EGRESS_ALLOW_NAME in its variables: to get an egress allowlist
distinct from the primary and from other services. The rules are the same as a
job-level request — it narrows the host [egress] cap, never widens it — with one
addition: for a service, a present-but-empty value denies that dimension (e.g.
MICROVM_EGRESS_ALLOW_NAME: "" gives the service no external DNS egress). A service
that sets neither variable shares the run policy.
A per-service value is the same kind of list (space/comma/newline-separated)
and supports the same $VAR references:
variables:
MIRROR_EGRESS: "registry.corp.example.com mirror.corp.example.com"
job-with-services:
services:
- name: postgres:16
alias: db
variables:
MICROVM_EGRESS_ALLOW_NAME: "" # db gets no external egress
- name: registry-proxy:latest
alias: proxy
variables:
# A list, via a shared variable plus an extra host.
MICROVM_EGRESS_ALLOW_NAME: "$MIRROR_EGRESS cache.corp.example.com"
- name: fetcher:latest
alias: fetch
variables:
MICROVM_EGRESS_ALLOW_NAME: "crates.io,pypi.org,files.pythonhosted.org"The switch enforces each policy against the flow's source VM — authenticated by the per-VM socket it arrives on, not by any address the guest writes into its packets — so a service cannot borrow another VM's (looser) policy. Requests are validated at job prepare, so a per-service value outside the host cap fails the job with a visible error before anything boots.
Every job trace names what that job is known to reach out to, just before what it cost:
virtkit: names this job has contacted under unrestricted egress: deb.debian.org, github.com
This is the standing list, not this run's: each run adds what its guests resolved in the
run phase to what the job has resolved before, so a nightly step's host is on it even in
a pipeline that did not run one. Write an allow_name from it and the job keeps working — bar
any entry showing ?, which is a name the guest sent outside the characters a hostname is
spelled with, kept only to show that it happened. A
dockerfile: build's own resolutions are not on it — those are the build phase's, and
[egress.build] is a separate cap with its own audit.
The list belongs to the policy, not to the job. Narrowing an allowlist starts it again
and the line then names the policy the list was gathered under
(under egress policy a379a6f61c4b8e02) — what a job reached while it could reach anything
says nothing about the same job once an allowlist is in force, and those names would
otherwise linger as if it still needed them. It is kept per job under <state_dir>/sites/,
keyed exactly as the run history is, and capped: past a couple of hundred names the line
says how many more it is not showing, because a list that long is no longer an allowlist
anyone would paste.
Audit mode adds the counts for one run: every resolution that run made, plus the external IPs it dialed without a resolution behind them, printed where the job can see it (at the end of the job trace for the run phase, after the build for the build phase). Turn it on when the standing list above is not enough — to see how often a name is reached, or to catch a step talking to a literal IP.
[egress]
audit = true # host-wide, for every job's run phase
[egress.build]
audit = true # for the build phaseA job can enable it for itself even when the host default is off:
discover-egress-job:
variables:
MICROVM_EGRESS_AUDIT: "1" # 1/true/yes/on
MICROVM_BUILD_EGRESS_AUDIT: "1"Outside CI, the same controls are CLI flags on vk run / vk build:
--audit-egress audits the booted guest and --build-audit-egress a build's
RUN steps (mirroring the --net / --build-net phase split).
CI services: run as sibling microVMs on the per-job switch, each resolvable
by its alias over the switch's DNS. They need the per-job LAN, so the job must
set [net] mode = "switch"; vk pulls each service image host-side (registry
credentials never enter a guest) into the same digest-keyed cache the job's
image: uses — an OCI ref (mysql:8, redis:7) under <state_dir>/docker, a
virtkit/ bundle under <state_dir>/registry.
integration-test:
services:
- name: mysql:8
alias: db
- name: redis:7
alias: cache
script:
- ./run-tests.sh # reaches the DB at host `db`Each service is a real VM with its own address on the shared switch, so a service
can carry its own egress allowlist in its variables: — see
per-service egress for the semantics and an example. A
service that sets no egress variable shares the job's run policy.