virtio-blk: blockdev-mirroring for live storage migration

[Coffeeri]

TLDR
Add blockdev-mirroring as live storage migration for the virtio-blk device. Without stopping the guest, the operator starts a mirror of a disk onto a destination file, waits for it to reach an in-sync state, then switches the VM to the destination. The whole flow runs over four REST endpoints.

Motivation

Some operators serve VM disk images from e.g. NFS shares mounted on the host. A share can fill up, and the operator then needs to migrate one disk image to another share with free space, without stopping the VM. Because the shares are mounted on the host, the VMM has filesystem-level access to both the source and the destination file.

We implement this in VMM rather than in a separate process. The mirror has to coordinate with VMM-level state: while it runs, the VMM must reject operations that would disturb the disk (see Design), and only the VMM can gate those.
A vhost-user-blk process could run the mirror itself and keep the swap transparent to the VMM, but it would still depend on the VMM for that gating, so we keep the mirror and the gating in one binary, which also keeps the libvirt integration simple. The vhost-user control interface is also too restricted to carry the start, progress, complete, and cancel commands and the mirror's state.

Design

• CopyWorker: a background thread copies the source to the destination in 512 KiB blocks. All-zero blocks are punched as holes so sparse images stay sparse. Note: the granularity is somewhat arbitrarily chosen and needs to be discussed, especially regarding a finer granularity for hole punching.
• MirroringAsyncIo: each virtqueue worker's AsyncIo backend is swapped for one that forwards reads to the source and every mutating op to both disks, and waits for both completions before acknowledging the write to the guest. A destination error degrades that queue to source-only and fails the mirror, so the guest never reads corrupted data.
• RangeLockManager: exclusive per-range locks shared between the copy worker and the guest writes, so the background copy and a concurrent guest write never race on the same range and the destination stays consistent.

The mirror's phase is shared between the copy worker and the per-queue backends:

stateDiagram-v2
    direction LR
    [*] --> running: start
    running --> ready: copy done
    ready --> completing: complete
    completing --> completed: switched
    completed --> [*]
    running --> failed: I/O error
    ready --> failed: I/O error
    running --> cancelling: cancel
    ready --> cancelling: cancel
    failed --> cancelling: cancel
    cancelling --> [*]

Loading

A mirror can be cancelled at any time before completion, which reverts every virtqueue worker to the source and keeps the VM on the source disk. After completion it cannot be undone: by then some virtqueue workers may have switched to the destination and written there only, so there is no consistent state to roll back to without losing acknowledged writes.
A mirror that fails on a destination I/O error stays in failed, keeps the VM on the source disk, and must be cancelled to clear it.

While a mirror is active, the VMM rejects operations that would disturb the disk or the mirror's state:

• snapshotting the VM
• live-migrating the VM
• resizing the mirrored disk
• removing (hot-unplugging) the device
• rebooting (vm.reboot), shutting down, or deleting the VM

Pausing the VM is allowed during an active mirror, but starting, completing, or cancelling a mirror is rejected while the device is paused (MirrorDevicePaused). An orderly guest reboot or shutdown resets the virtio-blk device, which cancels the mirror and reverts the queues to the source disk. The guest then restarts or powers off, with the mirror dropped rather than completed.

This approach is analogous to QEMU's blockdev-mirror with sync=full and copy-mode=write-blocking: a full background copy plus synchronous propagation of every guest write to the destination. Unlike QEMU it keeps no dirty bitmap or convergence loop, because with every write already current on the destination a single linear pass reaches a consistent state and a deterministic in-sync point. The trade-off is added write latency, which is fine for storage rebalancing.

API

All endpoints are PUT on the VMM API socket.

• vm.disk-mirror-start - begin mirroring disk id onto destination_path.
• vm.disk-mirror-status - report the current phase and copy progress.
• vm.disk-mirror-complete - switch the VM to the destination (accepted only from ready).
• vm.disk-mirror-cancel - abort and keep the VM on the source.

See docs/disk_mirroring.md for the operator workflow, failure handling, and full design.

Missing / TODO

• ch-remote subcommands for the disk-mirror endpoints (currently API-only). will be introduced in follow up PR
• Expose destination handling on vm.disk-mirror-start (create a new disk vs. reuse an existing one), defaulting to requiring an existing destination.
• Verify compatibility with block devices using direct I/O.
• Verify that qcow2 backing files do not hinder the current implementation. (backing files are flattened at destination)
• Add libvirt NixOS tests.

[phip1611]

I just finished my first review round. Given the nature and size of the PR, it is still fairly coarse-grained at this point.

First of all, thank you for your patience and for working on this over the last couple of weeks. IIRC, you have already successfully tested this with our customer in their infrastructure.

At this early stage, my main concerns are:

• Is this the simplest and/or least invasive design?
• All the locking primitives make me a little anxious. We really need to avoid deadlocks.

I like that you already thought about cancellation and about preventing changes to the VM while a disk is being migrated. This is good!

That being said, I am not entirely sure how to continue here given the complexity. I think some LLM-assisted reviews could help identify the design space and check the implementation. Combined with a presentation to the team and an open discussion, this should help us move this forward.

Thanks!

[phip1611]

this sounds a little weird to me. I thought that the new location is some host-visible NFS volume and in the end the file will be in /mnt/nfs_share2 instead of /mnt/nfs_share1. So I'd expect the backend (workers) just uses the new file location? I feel like virtqueues is the wrong termonilogy here or I am missing something

[Coffeeri]

Each virtqueue worker has its own AsyncIo backend. The Block device instructs each worker to switch to a different backend, in this case, one pointing to the destination image disk path. So the switch happens per queue.

Are you missing the word “workers” here?

[scholzp]

Some remarks. I'm not done yet and need more time. So expect more feedback in the next days.
What I've seen so far looks quite good! :)