virtkit boots OCI images as rootless Linux microVMs and builds Dockerfiles without Docker. Each guest runs its own kernel; the host side is a single static binary with an embedded VMM, guest kernel, and agent.
vk runs as an ordinary user and needs read/write access to /dev/kvm. There is no
host daemon in the local workflow and no requirement for tap devices, bridges, firewall
rules, or CAP_NET_ADMIN.
Typical uses are:
- running an OCI image as a disposable development or test machine;
- building Dockerfiles without a Docker daemon, with each
RUNisolated in a microVM; - running compose-style service fleets on a private guest network;
- isolating GitLab custom-executor jobs in fresh VMs; and
- producing raw disks, VMDKs, OVAs, and bootable ISOs from Dockerfile-driven builds.
virtkit is Linux- and KVM-specific. Release artifacts are built for x86-64 Linux
(x86_64-unknown-linux-musl); this is not a cross-platform Docker Desktop replacement.
- An x86-64 Linux host with KVM enabled.
- Read/write access to
/dev/kvmfor the user runningvk. - Network access when pulling images or using guest egress.
- Docker or an existing
vkbinary only when building virtkit itself from source.
curl -fsSL https://github.com/virtkit-dev/virtkit/releases/latest/download/install.sh | shThis installs the latest vk into $XDG_BIN_HOME (else ~/.local/bin) and verifies the
checksum published beside it. BINDIR selects another directory and VIRTKIT_VERSION another
release. From a checkout with .virtkit/toolchain.lock, it installs the pinned release.
vk update updates an installed vk to a later release.
vk toolchain lock pins the team's release in .virtkit/toolchain.lock, with each
artifact's checksum and download URLs. vk toolchain install populates a per-version
cache without changing the vk on PATH. vk toolchain export provides those paths and
checksums to scripts and image builds; vk toolchain status reports what is pinned and
installed.
Run the host preflight before debugging a failed boot:
vk checkIt checks KVM access, the selected VMM backend, the guest kernel and agent, and the
host-side requirements for configured features. Scripts can require a release with
vk check --min-version 0.45, which exits non-zero on an older vk.
A fresh WSL2 distro cannot boot microVMs: nested virtualization is off, and nothing loads
KVM or opens /dev/kvm to the user. vk check says which of those still holds and prints
the steps that remain. vk check --fix asks first, then takes the ones it can — the two
files below and the commands, leaving you the WSL restart and the new login — and reports
the checks again so what is left is visible. It needs a terminal to ask on.
In full, starting on the Windows side:
# %UserProfile%\.wslconfig — needs Windows 11, or the Store WSL on a recent Windows 10.
[wsl2]
nestedVirtualization=trueThat key applies only when WSL itself restarts: run wsl --shutdown from Windows and reopen
the distro, after which /proc/cpuinfo carries vmx or svm. Then, in the distro:
sudo modprobe kvm_intel # kvm_amd on an AMD host
sudo groupadd kvm # only if the distro has no kvm group
sudo usermod -aG kvm "$USER" # log in again before it applies
sudo chown root:kvm /dev/kvm
sudo chmod 660 /dev/kvmNeither the module nor the device's ownership survives the next wsl --shutdown. An
/etc/wsl.conf boot command redoes both on every start of the distro:
[boot]
command = modprobe kvm_intel; chown root:kvm /dev/kvm; chmod 660 /dev/kvm# Pull Alpine and open an interactive shell in a fresh microVM.
vk run alpine:latest --shell
# Run one command and return its exit status.
vk run debian:trixie-slim -- cat /etc/os-release
# Compile the current checkout in a disposable VM. /work maps this directory,
# so target/ remains on the host after the VM exits.
vk run rust:1-alpine --workdir . --net --cpus host --mem 4G -- \
cargo build --release
# Build the final Dockerfile stage, boot it, and run the test entrypoint.
vk run -f Dockerfile --net -- ./run-tests.shImage conversions and Dockerfile build results are content-addressed and reused on
later runs. Use vk help <command> for full command documentation; short -h output is
kept intentionally compact.
vk run IMAGE pulls the image, converts its root filesystem to a bootable ext4 disk,
starts a microVM, and executes either the requested command or a shell. By default the
guest uses virtkit's embedded kernel and runs vk-agent as PID 1.
Use --net to allow guest egress. Networking is implemented by a userspace switch in
the vk process, and outbound traffic leaves through ordinary host sockets. The host
does not need a bridge, tap device, or firewall changes.
For images intended to boot as full machines, --kernel image --init image loads the
image's /boot/vmlinuz, modules, and init system. --init entrypoint instead runs the
image's ENTRYPOINT and CMD as PID 1. A kernel supplied with --kernel <path> is also
supported.
--nested exposes KVM to the guest so it can run microVMs of its own. The host must have
KVM nesting enabled (kvm_intel.nested=1 or kvm_amd.nested=1), which the flag checks
before pulling or building anything. Treat this as a grant for trusted guests: nested
virtualization reaches the host kernel's KVM paths.
vk build evaluates Dockerfiles directly; it does not call Docker. Each RUN executes
in a microVM, instruction snapshots are cached, and independent stages may build in
parallel. vk run -f Dockerfile is the convenient build-and-run form.
A stage can declare resources without making the Dockerfile incompatible with Docker:
# vk: mem=8G cpus=16
FROM rust:1-alpine AS build
RUN cargo build --releasePer-run overrides take precedence:
vk build --stage-mem build=12G --stage-cpus build=8 -f Dockerfile --out rootfs.ext4The resource hint is a comment and does not enter the instruction cache key. When a stage asks for more memory than the host can allocate to one guest, virtkit clamps the request and emits a warning explaining the effective size and OOM risk.
Builds report each stage's peak guest memory as the stage finishes and in a block under the final timing breakdown:
Stage memory (peak demand / guest size)
[build] 3.1 GiB of 8.0 GiB
[runtime] 412 MiB of 4.0 GiB
The guest measures MemTotal - MemAvailable, excluding reclaimable page cache so the
result reflects demand rather than every page touched. The per-stage line is printed as a
stage finishes, so a stage that fails has none; the final block still lists whatever its
guests had reported by then.
vk run --compose compose.yml boots services as separate VMs on a shared network. Each
service resolves by name. Services can run alongside a primary command, as the primary
with --primary, or as a standalone fleet until interrupted.
examples/compose.yaml exercises everything below in one
annotated file; it is parsed by the test suite, so it always matches what vk accepts.
A compose file is services: and nothing else (a deprecated version: is accepted and
ignored). vk parses it strictly: an unknown key anywhere is an error, so a docker volumes:
or networks: section is refused rather than skipped, and named volumes are not supported —
bind a path. A service uses image: or build:, plus any of environment, env_file,
command, entrypoint, user, hostname, depends_on, volumes, profiles and the
x-virtkit marker. Every host path is relative to the compose file.
image: names an OCI image, pulled and converted host-side and shared across runs.
build: is a directory (build: ./app) or a mapping:
services:
app:
build:
context: ./app
dockerfile: [Dockerfile, Dockerfile.dev] # several files merge into one stage namespace
target: runtime # any stage across them
args:
UID: "$VK_UID" # bare and braced both interpolate; keeps a
GID: "${VK_GID}" # shared tree's ownership coherent
additional_contexts:
assets: ./assets # `COPY --from=assets`; local directories onlyBuilt images are content-addressed and reused; a build: sibling is built on its first
start (vk service up streams the build), the --primary service up front.
environment: (map or list) upserts over the image's own environment; env_file: is a
path, a list of paths, or {path, required: false} entries, layered beneath it.
entrypoint: replaces the image's entrypoint and drops its command; command: alone
replaces only the command; user: replaces the user. hostname: (a DNS label) defaults to
the service name and is what other services resolve.
Values interpolate $VAR, ${VAR} and ${VAR:-default} from the environment over a
sibling .env; $$ is a literal $. An unset variable with no default fails the load
— an empty image tag or bind path is always a bug — and the other docker modifiers (:?,
:+, ${VAR-default}) are rejected. Five reserved names come from the run itself, so a
committed file needs no host paths or ids: ${VK_WORKSPACE} (--workspace, else the
cwd), ${VK_STATE_DIR} (--state-dir, else the run's scratch), ${VK_SELF} (the running
vk, to hand a guest its own copy), ${VK_UID} and ${VK_GID}. A variable holding several
newline-separated binds expands into several volumes: entries.
depends_on (a list, or a map whose only accepted condition: is service_started)
orders starts; there is no readiness wait, so retry a first connection. Services with
profiles: stay declared but down unless a profile is activated (--profile NAME,
repeatable) or an enabled service depends on them; vk service up NAME starts one anyway.
virtkit-specific settings live under x-virtkit. Services default to 2 vCPUs and 1 GiB of
memory; set cpus and mem when a service needs a different guest size, and nested: true
for a service that runs microVMs of its own (the host must allow nesting):
services:
database:
image: postgres:17
x-virtkit:
cpus: 2
mem: 2G
builder:
image: local/builder
x-virtkit:
nested: true
volumes:
- ${VK_SELF}:/usr/local/bin/vk:ro # the host's own vk, for nested builds--service-cpus NAME=N and --service-mem NAME=SIZE override those values for one run.
init chooses PID 1 — default (the vk agent), image (the image's own /sbin/init) or
entrypoint (its ENTRYPOINT+CMD) — and kernel the kernel: default (the pinned guest
kernel), image (the image's own kernel and modules) or a kernel file path. Together they
boot a systemd or otherwise self-booting image as a service. persist_root keeps the root
filesystem across restarts (see Volumes and persistent state).
Guests give idle memory back. Pages one frees return to the host through the balloon, and a
guest not under memory pressure also evicts the file cache it has not touched for a minute or
two (by age, through the kernel's multi-gen LRU) and hands those pages back, so a dev VM that
read a few gigabytes while building stops holding them once it idles, while what a running
build keeps re-reading stays cached. That is reclaim: auto; off keeps everything, a size
(512M) or share (5%) keeps that much as a fixed floor. vk run --reclaim sets it for the
primary and any service without its own, [executor.vm] reclaim for the GitLab executor. It needs the
agent as PID 1, so an init of image or entrypoint opts out; vk build stage guests are
left alone as well, since trimming would move the peak-memory mark they are measured by.
Directory virtio-fs shares — --workdir, -v directory binds and compose volumes — use
DAX windows to read the host page cache directly. The host maps shared files into guest
address space, avoiding a second copy through the filesystem protocol. Several VMs cache a
shared tree once; host-side content edits are visible without a re-read, and reopening a
file no longer re-reads it. Mappings are 4 KiB-granular, so the win is in memory rather than
in per-fault latency.
The window reserves address space, not memory, and costs nothing until mapped. It defaults
to 8G per share; vk run --dax, a service's x-virtkit.dax and the executor's [executor.vm] dax
resize it or turn it off. Each mapping costs the host an mmap and the guest an EPT
invalidation per 2 MiB range whatever the file's size, so by default only regular files of
1M and more go through the window (dax=inode; the host marks them) and smaller files read
through the guest's page cache as without DAX; <size>:always maps every file, and
<size>:inode=<min> moves the floor. Each guest supports 64G of windows — eight at the
default size, with further shares served without DAX. Guests with more than 63.25G of RAM
have no room for windows and receive none. DAX requires the built-in VMM. Under
VIRTKIT_VMM=cloud-hypervisor, and for single-file binds and vk build stage guests,
shares are served the ordinary way.
A guest gets one interface, eth0, by default. nics gives it more — eth1 upward, each
with its own address on the same LAN — for an appliance that assigns services to separate
interfaces:
services:
appliance:
image: local/appliance
x-virtkit:
nics: 3--service-nics NAME=N overrides that for one run, like --service-cpus/--service-mem.
vk run --nics N does the same for the primary VM (it needs --net, which --compose
implies). eth0 keeps the default route and stays the address a service name resolves to;
the extra interfaces are addressed but given no route, so egress leaves through eth0
unless the guest routes it elsewhere. Every interface is a real port on the LAN: each has
its own MAC, answers ARP, and can carry its own listening services — which is what an
appliance separating admin from user traffic needs. Up to 8 per guest;
vk check --feature nics reports whether a vk supports them.
A service starts from a clean copy of its image every time: its root filesystem is a
throwaway layer over the image, and volumes: bind host paths into the guest. A bind is
host:guest[:mode], with the host path relative to the compose file:
| mode | the guest sees | writes go |
|---|---|---|
rw (default), ro |
the host directory or file, live | to the host (rw), or are refused (ro) |
overlay |
the host tree, read-only underneath | to guest RAM — fast, gone at reboot |
overlay,persist[,size=SIZE] |
the host tree, read-only underneath | to a disk that survives restarts (see below) |
disk[,size=SIZE] |
a private filesystem of its own | to that disk; the host path is its image |
socket |
a unix socket at the guest path | each connection is relayed to the host socket |
overlay is for build trees and checkouts: reads come from the host, every write lands
in guest memory and never touches the host tree. Add persist to keep those writes on
disk instead. A read-only share (ro, overlay) takes immutable when the host will not
change the tree while the service runs: the guest then keeps what it read for its whole
life, so a pass over the tree (git status, a build's dependency check) asks the host once
rather than on every pass, and the share carries no extended attributes. disk is for data that needs real filesystem semantics (ownership, sockets,
device nodes) a shared directory cannot offer — a database's data directory, say. socket
forwards a host service's unix socket and is implied when the host path is one — for
example, /var/run/docker.sock:/var/run/docker.sock lets the guest drive the host's Docker.
Only bytes cross over vsock, so the guest never learns the host path, but it receives
everything the socket grants: a Docker socket is host-root-equivalent, so grant it
deliberately. Shares take ,optional to skip a bind whose source is absent; size= (10G,
512M) sets a new disk's capacity and is ignored once it exists.
Set persist_root when the whole root must persist — an appliance whose state is not
confined to a few directories:
services:
appliance:
build: ./appliance
x-virtkit:
persist_root: true # / survives restart and down/up
volumes:
- ./config:/etc/appliance:ro
database:
image: postgres:17
volumes:
- ./pgdata.qcow2:/var/lib/postgresql:disk,size=20G
builder:
image: local/builder
volumes:
- ./src:/workspace:overlay # scratch: reads from the host, writes in RAM
- ./cache:/root/.cache:overlay,persist # keeps its writes across restartsPersistent state — a persist_root root, an overlay,persist layer — survives an in-guest
reboot, vk service down/up, and stopping and later restarting the run. When the run pins a
durable state directory — vk dev, or vk run --state-dir — it is kept there, out of the
workspace and so out of the guest's view; otherwise it lives under .virtkit/ beside the
compose file (add that to .gitignore). Either way it is reset when what it
was built on changes: a new image rebuilds a persistent root from scratch, and a new image
or a changed host tree discards a persistent overlay's writes. A disk volume persists
unconditionally; delete its file to start over. All of this applies alike to a service
booted with --primary and to its siblings; ad-hoc -v binds on vk run accept every
mode but overlay,persist, which has no compose file to keep its state beside.
vk run --compose FILE with an image or -f boots that as the primary VM and the services
beside it, torn down when the run exits. --primary NAME boots a compose service as the
primary instead — docker compose run semantics: its image is the rootfs, its config the
command's environment, with no trailing command its entrypoint and command run, and only
its depends_on chain boots alongside. With neither, --compose alone is compose up: the
services only, held until Ctrl-C.
Inside the primary guest, /run/vk/services/<name>/{state,ctl,log} exposes service
state, control, and logs through ordinary files. vk service up|down|reboot|status is the
corresponding command interface: up starts a declared (or profiled-down) service, building
it on first use; down powers it off; reboot restarts its guest in place on the same
disks; status reports one or all. vk run --ssh enables SSH access for development VMs,
including VS Code Remote-SSH workflows. In a CI fleet a service's environment: may also
carry its own egress allowlist — see Per-service egress.
vk dev boots a project's development environment from .virtkit/config.toml.
The same configuration drives the shell, editor, services, hooks and host integration.
vk dev init # write a first config from devcontainer.json, a compose file,
# a Dockerfile or a stock image; validate an existing one
vk dev shell # boot if needed, then a login shell as the config's user
vk dev exec -- cargo test # a command in the environment, exit status reproduced
vk dev code # VS Code over Remote-SSH, extensions and settings reconciled
vk dev service up runner # a profiled compose service, built on first use
vk dev endpoints # the stable host addresses its ports are published on
vk dev task pre-commit -- "$@" # a project command under its declared execution policy
vk dev status # what is running and whether its configuration matches
vk dev plan # inspect the resolved configuration without running it
vk dev doctor # check the host and configuration requirements
vk dev refresh # rebuild and restart into the current config
vk dev stop
vk dev list # every environment this host keeps state for, from anywhere
vk dev gc --all-stale # review/remove stale state; asks on a terminalConfiguration, local overrides, compose services, endpoints, persistent storage, editor setup, lifecycle hooks and isolated tasks are covered in the project development guide. It includes a complete workflow and a service layout with on-demand test appliances.
A vk run --state-dir DIR boots a VM that outlives a single command: its sockets and
console log live in DIR, and the run is recorded in a host-side registry. vk list reads
that registry, dropping entries whose run has died, and is what vk exec, vk status,
vk logs, vk stop and vk reboot resolve a directory against — vk logs also reads the
console of a VM the registry has already dropped, since DIR keeps it. A run without
--state-dir is not listed; pair --state-dir with --detach for a background VM.
vk run --state-dir "$PWD/.vk" --detach --ssh -f Dockerfile --target dev
vk listPID UPTIME MEM NAME SERVICES PROJECT PUBLISHED
41230 2h14m 1.2G/8G app/Dockerfile:dev - ~/app 127.0.0.1:8443->localhost:443
41877 35m 5.9G/16G shop db, redis, web, +4 ~/shop 127.0.0.1:5432->127.0.0.1:5432@db
MEM is what the VM is costing the host now over the size it booted with: the resident
memory of its whole process tree — the guest, its service VMs, the switch, the virtiofsds
and the forwards — counted proportionally so a page several of them map is charged once,
over the --mem token as the run recorded it. When proportional usage is unavailable,
resident usage is used instead and may count shared pages more than once. The total includes
service VMs and helpers, so it is not the primary guest's memory utilization.
Either half is - on its own when unknown,
and the cell is a bare - when neither is. NAME is the built Dockerfile with its target
stage, the compose primary, or the image ref.
SERVICES lists the compose services running beside the primary, or every declared one when
the VM cannot be asked; - for none. Past three names, the rest are counted (+4).
PROJECT is the run's --workspace, then its --workdir, then its launch directory, with
$HOME shown as ~. PUBLISHED is every port vk publish ensure holds open on the host
for the VM, as listen->to, with @NAME when a compose sibling rather than the primary
dials the target and (unconfirmed) when the publisher's liveness could not be checked;
- when nothing is published. --wide (-w) names every service, prints the project
directory in full and adds EXEC ADDRESS, recorded as given (a relative --state-dir lists a
relative path). --json and --field already report every field, so neither takes --wide.
An optional pid or directory scopes the list. A pid selects one VM, as with vk stop.
A directory selects VMs whose project is that directory or below it, or whose state dir
matches it exactly. If no VM matches, the message names the target:
no running vk VM with pid 99999, or no running vk VM under <dir>.
When a selector names exactly one VM, vk list prints its full record instead of a table
row: every field --json carries, then each compose service as name, state, LAN address and
exec address, and each published port as name, listen->to (with @NAME when a compose
sibling dials) and the publisher's pid. When liveness is unconfirmed, (unconfirmed)
marks the pid; in the table it marks the address. Nothing is folded in a full record,
so --wide has no effect.
vk list ~/work # every VM under a tree (a table when several match)
vk list . # VMs for this tree or below; one VM prints in full
vk list 41877 # one VM, by pid
vk list /home/me/app/.vk # one VM, by its state dirNAME shop
PID 41877
UPTIME 35m (since 2026/09/04 08:12:03 UTC)
PROJECT /home/me/shop
STATE DIR /home/me/shop/.vk
EXEC ADDRESS vsock-auto:///home/me/shop/.vk/vsock.sock:4444
SSH 127.0.0.1:2222
GUEST IP 10.0.0.2
VMM libkrun (pid 41902)
CPUS 4
MEM 8G
MEM USED 5.9 GiB
NESTED no
ATOP LOG -
SERVICES db running 10.0.0.3 vsock-auto:///home/me/shop/.vk/svc-db/vsock.sock:4444
redis running 10.0.0.4 vsock-auto:///home/me/shop/.vk/svc-redis/vsock.sock:4444
web running 10.0.0.5 vsock-auto:///home/me/shop/.vk/svc-web/vsock.sock:4444
worker running 10.0.0.6 vsock-auto:///home/me/shop/.vk/svc-worker/vsock.sock:4444
mailer running 10.0.0.7 vsock-auto:///home/me/shop/.vk/svc-mailer/vsock.sock:4444
queue running 10.0.0.8 vsock-auto:///home/me/shop/.vk/svc-queue/vsock.sock:4444
minio running 10.0.0.9 vsock-auto:///home/me/shop/.vk/svc-minio/vsock.sock:4444
search stopped 10.0.0.10 vsock-auto:///home/me/shop/.vk/svc-search/vsock.sock:4444
PUBLISHED pg 127.0.0.1:5432->127.0.0.1:5432@db pid 42011
--json gives an array of objects, one per VM, with pid, label, project_dir,
exec_addr, state_dir, vmm, vmm_pid, cpus, mem, mem_used_bytes (what the VM's
process tree holds on the host now, in bytes — null when it could not be read),
nested, guest_ip (the eth0 address on a --net LAN), ssh_addr, atop_log,
created_secs, uptime_secs, services (every declared compose service with its
name, exec_addr, state and LAN ip), and published (each publisher's name,
listen, to and pid, plus via when a compose sibling dials — the one
vk publish ensure --via named — and "unconfirmed": true when its liveness could not
be checked). --field picks fields without jq, one --field per field: one line per VM
and tab-separated in flag order, or with --json objects holding only those fields; a
dotted path reaches into nested values, and a key a record omits reads null:
vk list . --field pid # the pid to hand to vk stop
vk list . --field guest_ip # the VM's address on the --net LAN
vk list . --field label --field services.0.ip
vk list . --field published.0.listen # where the first published port listens
vk list --json | jq '.[] | select(.vmm == "cloud-hypervisor")'--stale adds a column to the table, and a STALE row to a full record, reading yes,
no, or - when unknown (as for an image boot). It says whether a fresh vk run would
rebuild the VM's image because the Dockerfile, build context or base image of the VM or of
one of its build: services changed since it booted. It resolves base image digests over
the network, so it is opt-in; --json then carries stale too.
The GitLab custom executor creates a fresh microVM for each job and destroys it during
cleanup. Job images from .gitlab-ci.yml are converted on demand. A repository can also
use image: dockerfile:… to build its job image directly from a checked-in Dockerfile.
The executor supports concurrent jobs, service VMs, per-phase egress policy, resource
accounting, and optional host-load throttling through vk-runnerctl. Configuration and
the security model are covered in the GitLab CI guide.
vk build --disk lets a Dockerfile stage partition and install a bootloader into a
caller-owned raw disk. That disk can then be packaged natively, without qemu-img,
ovftool, or xorriso on the host:
vk export vmdk disk.raw appliance.vmdk
vk export ova disk.raw appliance.ova
vk export iso staged-root installer.iso \
--bios-boot boot/grub/eltorito.img \
--efi-boot boot/grub/efi.img \
--hybrid-mbr isohdpfx.binVMDK output uses VMware's streamOptimized format; OVA output is ready for ESXi/vCenter import. ISO output can include BIOS and UEFI boot images, plus an optional hybrid MBR for USB media. See the appliance guide for the expected disk and staged-tree layouts.
Each guest has its own kernel. The default local path is rootless and daemonless, but the isolation boundary still depends on KVM and the selected VMM. Guest networking is userspace-only on the host. Nested KVM should be reserved for trusted workloads.
vk-runnerctl is the one optional component designed to run as root. It accepts no
caller-controlled arguments or paths; it only applies the configured GitLab runner
concurrency range. Local image execution and building do not require it.
Builds, vk run, and CI jobs report host CPU time, peak memory, disk traffic, and guest
network traffic when they finish. Concurrent builds and compose fleets also identify the
largest host process, which helps distinguish an oversized guest from excessive
concurrency.
Treat the CPU, memory, and disk figures as ceilings: they include vk and its host
helpers alongside the guests. Network figures cover guest traffic alone because
host-side image pulls do not cross the userspace switch that counts it.
CI jobs also report how much of the guest's writable layer they filled. That layer can
hit ENOSPC while the host still has free disk space.
The guest agent records processes killed by the guest kernel's OOM killer. The host reports
them alongside memory figures in build-stage completion output, CI traces, completed
vk run output, and vk status for a live VM. Each line identifies the victim, the
anonymous RSS reclaimed, its guest uptime at death, and the setting that raises the memory
limit. This can explain a command that exited with signal 9.
virtkit: guest OOM: the kernel killed cc1plus (pid 1234, 1.9 GiB RSS) at +48s (raise --mem)
By default, each CI guest records an atop -P-compatible sample every 10 seconds,
covering CPU, memory, disk, network, and process activity. vk atop follows a running
recording or inspects one retained after the job VM has been removed. See the
resource-usage documentation for accounting
boundaries and interpretation.
Local image conversion and build caches require no server. vk-registry is optional and
is useful when several runners need a shared OCI store, pull-through cache, or build-once
coordination. Its lease and heartbeat protocol prevents runners from independently
building the same content while a healthy peer is already doing so.
Use vk registry push|pull|inspect for guest bundles and vk registry status|gc for a
local store. The central server and storage model are documented in
vk-registry/DESIGN.md.
| Binary | Purpose |
|---|---|
vk |
Host CLI, VMM, image builder, userspace network, compose runner, and GitLab executor. It embeds the default guest kernel and vk-agent. |
vk-agent |
Guest PID 1 and command server. It configures mounts, networking, hostname, shared directories, optional SSH, and host-driven execution over vsock. |
vk-registry |
Optional OCI-distribution server with a pull-through cache, shared build cache, and build-once locking. |
vk-runnerctl |
Optional root-side helper that adjusts GitLab runner concurrency within an administrator-configured range. |
The default VMM is the embedded libkrun. An
external Cloud Hypervisor binary can be selected with
the vmm configuration key or VIRTKIT_VMM=cloud-hypervisor.
The host converts OCI layers to native ext4 images in userspace. Build inputs identify cached disks and instruction snapshots. Guests communicate with the host over vsock for command execution, service control, and CI lifecycle operations. Guest Ethernet frames are handled by the in-process userspace switch, which provides ARP, DHCP, DNS, and TCP/UDP egress through host sockets.
Release binaries are static musl PIE executables. The Rust toolchain, build image, its packages, guest kernel, and vendored libkrun source are pinned so release artifacts can be rebuilt byte-for-byte — see Build from source.
| Command | Use it for |
|---|---|
vk run |
Boot an image, Dockerfile target, or compose fleet; run a command or shell. |
vk build |
Build Dockerfile stages into a bootable ext4 image or caller-owned disk. |
vk exec |
Run a command in an existing guest and return the command's exit status. |
vk list |
List running --state-dir VMs and their compose services; scope by pid or directory, -w for the full table, --json/--field for scripts. |
vk stop |
Stop a VM selected by pid or project directory, or stop all registered VMs. |
vk reboot |
Reboot a running VM in place through its guest, or power-cycle it with --force. |
vk status |
Probe a guest agent, or report whether its root image is stale. |
vk logs |
Show a VM's console log, telling kernel, agent and guest output apart; --level warn, --agent, --service NAME, -f. |
vk atop |
Follow or inspect guest resource recordings. |
vk check |
Validate KVM, VMM, embedded assets, configured host features, and an optional minimum vk version. |
vk gc |
Reclaim unused image bases, CI checkouts, and image-cache chunks. |
vk update |
Check for or install a digest-verified GitHub release. |
vk dev ... |
Boot, enter, refresh and stop a project's development environment from .virtkit/config.toml; its services, endpoints, storage, tasks and editor. vk dev list|gc covers every environment on the host. |
vk toolchain lock|install|export|status |
Pin a project's virtkit release and install its artifacts from the lock. |
vk service up|down|reboot|status |
Control compose services from the primary guest. |
vk registry ... |
Publish, fetch, inspect, report on, or sweep OCI stores. |
vk gitlab ... |
Implement the GitLab custom-executor lifecycle. |
vk export ... |
Package raw disks or staged files as VMDK, OVA, or ISO artifacts. |
Advanced commands are listed by vk help-all. They include the stdio/vsock connector,
network forwarding processes, SSH agent proxy, path inspection, OCI/ext4 conversion
tools, image fingerprinting, and the bundled Cloud Hypervisor virtiofsd.
vk uses the first configuration source that applies:
--config <path>$VIRTKIT_CONFIG$XDG_CONFIG_HOME/virtkit/config.toml, or~/.config/virtkit/config.toml/etc/virtkit/config.toml
An explicit path that does not exist is an error. Standard user and system paths are
optional. With no configuration file, local run and build workflows use defaults.
vk config # print the effective TOML and its source
vk config --example # print the annotated reference configuration
vk config --path # print the resolved configuration path
vk paths # print resolved state, cache, and registry pathsvk-driver/config.example.toml documents every key.
The small environment-variable surface is:
| Variable | Effect |
|---|---|
VIRTKIT_CONFIG |
Select a configuration file. |
VIRTKIT_VMM |
Override the VMM backend; currently useful for cloud-hypervisor. |
VIRTKIT_DEBUG=1 |
Enable verbose VMM and guest logging. |
VK_SWITCH_LOG=debug |
Set the userspace network switch's log level in switch.log (default: warn). |
VIRTKIT_TIMING=1 |
Print per-phase build and boot timing. |
VIRTKIT_PROGRESS=plain |
Use line-oriented build progress suitable for CI logs. |
VIRTKIT_NO_TITLE |
Disable terminal-title updates without disabling the dashboard. |
Two configuration values can point at the same content-addressed store:
[build] cache_registrystores build stages and base snapshots inbuild-cache.[registry] repostores named bootable guest bundles.
Each accepts either a registry endpoint or a local absolute path/file:// URL. Pointing
both local settings at one directory enables chunk deduplication across build cache and
guest bundles. cache_registry = "none" disables future build-cache writes. Use
vk registry status and vk registry gc to inspect and reclaim a local store.
vk-registry has a separate configuration namespace. Its store-oriented subcommands
recognize VK_REGISTRY_CONFIG, VK_REGISTRY_ROOT, and
VK_REGISTRY_ADMIN_SOCKET as documented in its design file and command help.
Build the pinned guest kernel first, then the static binaries:
./build-kernel.sh # dist/vmlinux
./build.sh # dist/{vk,vk-agent,vk-registry,vk-runnerctl,...}The scripts use a vk found on PATH to build inside a microVM; otherwise they use
Docker. Pass --docker to force Docker. build.sh --use-virtkit=<dist> selects a
specific existing virtkit build.
vk normally embeds dist/vmlinux, so build.sh refuses to proceed when the kernel is
missing. --no-kernel produces a non-shippable binary that requires --kernel at
runtime. The kernel changes less often than the Rust binaries, so one kernel build can be
reused across normal edit/build cycles.
For iteration, use the repository's development commands:
./dev.sh check -p vk-core # one crate, while iterating
./dev.sh test -p vk-core --lib dockerignore::tests # one module's tests
./dev.sh fmt --check && ./dev.sh check && ./dev.sh clippy && ./dev.sh test # before committing
./build.sh --fast # only when a runnable debug vk is neededRelease builds produce byte-for-byte reproducible artifacts, and every input that fixes those bytes is pinned to something that stays fetchable:
- the build image (
.devcontainer/Dockerfile) is the officialnixos/niximage by digest, and its toolchain — Rust, a musl cross gcc for the vendored C, mold, and the kernel build tools — comes from.devcontainer/nix/flake.nixlocked to exact nixpkgs commits inflake.lock. Nix runs only inside the image while it is built; no host needs Nix, and nothing is pushed to any registry; - the guest kernel is a pinned vanilla release, checked against its published sha256 (with a signed-tag fallback), and libkrun is vendored;
dist/build-info.txtanddist/kernel-build-info.txtrecord the commit, base image digest, locked nixpkgs revision, and the sha256 of every artifact, with the exact command that rebuilds and verifies them.
./build.sh --bootstrap-check is the proof: it performs the Docker build, rebuilds a clean
copy of the tree from scratch in a microVM booted by the vk it just produced, and fails
unless the binaries are identical. Releases run this check in CI, so a published binary
has been reproduced independently before it is published. --fast uses the unoptimized
development profile and is not a release artifact.
vk-core/ shared host/guest protocol and runtime helpers
vk-driver/ host driver, builder, VMM, networking, compose, and GitLab executor
vk-agent/ guest PID 1 and exec server
vk-registry/ optional central OCI store and distribution server
vk-runnerctl/ optional root-side GitLab concurrency helper
vk-selfupdate/ shared self-update implementation for vk and vk-registry
vk-fs/ filesystem objects created private and published whole
third_party/ vendored libkrun and local patches
.devcontainer/ pinned build image (nixos/nix base + nix/flake.nix and flake.lock toolchain)
kernel/ pinned guest-kernel configuration and build inputs
docs/ operational guides
examples/ annotated compose file exercising every compose feature
tests/ end-to-end scripts run against a built vk; release-e2e.sh gates a release
build.sh reproducible binary build
build-kernel.sh reproducible guest-kernel build
dev.sh check/fmt/lint/test environment in a reusable development VM
Copyright © Vincent Vanackere and WALLIX. See LICENSE and NOTICE.