Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,11 @@ On Apple Silicon macOS, downloads and imports use Android 15 Google Play `arm64-
- The group with the largest free share (free logical processors ÷ the group's logical processors) is chosen, the bigger L3 on a tie; a single instance gets the big-L3 group.
- When no group has room for the instance's vCPUs, it is not pinned.
- "Free" means after the vCPUs of every qemu on the machine, including those of other root folders and those started by AUTO-MAS. Those processes are only read, never changed.
- While the guest does many small disk writes (downloads, for example), it runs one busy loop per vCPU (`mas-idlepoll.sh`) so the vCPUs do not halt, which makes downloads faster.
- Many small disk writes (downloads, for example) are made faster on Windows by two extra QEMU arguments at boot. What "written" means stays the same: a request completes once the data is in the host file.
- `-global virtio-blk-pci.ioeventfd=off`: the vCPU thread handles disk requests itself, with fewer thread hand-offs.
- `-acpitable`: the guest gets an idle-state table (`modavd-cst.aml` in the instance folder, generated for its CPU count). While it waits for a disk request it busy-waits briefly (200 µs at most) before halting, and halts as usual otherwise.
- Boot with the environment variable `MODAVD_SYNC_IO=off` to leave both arguments out.
- Without the table (`MODAVD_SYNC_IO=off`, an instance whose CPU count cannot be read, or a table path that does not fit on QEMU's command line), the earlier method is used: while the guest does many small disk writes it runs one busy loop per vCPU (`mas-idlepoll.sh`) so the vCPUs do not halt.
- The number of loops never exceeds what the host has left: logical processors minus the vCPUs of every qemu (this instance's included; counted within its group when pinned).
- So on machines with fewer than twice as many logical processors as the instance has vCPUs, even a single instance gets fewer loops: a default 6-core instance gets 2 on an 8-thread machine and none on 6 threads or fewer, and downloads are slower than with all loops (measured on a 7950X3D, 1–2 loops do about as well as none).
- This is a deliberate trade-off: it leaves processors for qemu's own threads, Windows and other programs.
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,11 @@ Apple Silicon macOS 的下载与导入使用 Android 15 Google Play `arm64-v8a`
- 选空余比例(空余核数 ÷ 这组的核数)最大的一组,一样时选 L3 大的;只开一台时就是 L3 大的那组。
- 各组都放不下本实例的 vCPU 就不绑。
- 空余是扣掉整台机器上所有 qemu 的 vCPU 以后算的,包括别的根目录和 AUTO-MAS 起的。那些进程只读、不改。
- 下载这类小块密集写盘时,客体里每个 vCPU 起一个忙等循环(`mas-idlepoll.sh`),不让 vCPU 停机,下载更快。
- 下载这类小块密集写盘,在 Windows 上靠开机时多给 QEMU 的两个参数加速,「写完」的含义不变(数据进了宿主文件才算完成):
- `-global virtio-blk-pci.ioeventfd=off`:读写请求由 vCPU 线程直接处理,少几次线程交接。
- `-acpitable`:给客体一张空闲状态表(实例目录里的 `modavd-cst.aml`,按核数生成)。客体等读写完成时先忙等一小段(最多 200 µs)再停机,其余时候照常停机。
- 设环境变量 `MODAVD_SYNC_IO=off` 再开机,这两个参数都不带。
- 表没带上时(`MODAVD_SYNC_IO=off`、实例读不出核数,或者表的路径放不进 QEMU 命令行),退回原来的办法:小块密集写盘时客体里每个 vCPU 起一个忙等循环(`mas-idlepoll.sh`),不让 vCPU 停机。
- 循环数不超过宿主的余量:逻辑核数减去所有 qemu 的 vCPU 数(含本实例;绑了核就按那一组算)。
- 所以逻辑核数不到本实例 vCPU 两倍的电脑上,只开一台也会少开:默认 6 核的实例在 8 线程的电脑上只开 2 个,6 线程及以下不开,下载会比开满时慢(在 7950X3D 上实测,1–2 个循环和不开差不多)。
- 这是有意的取舍:给 qemu 自己的线程、Windows 和别的程序留出核。
Expand Down
16 changes: 15 additions & 1 deletion internal/emu/afterboot.go
Original file line number Diff line number Diff line change
Expand Up @@ -563,8 +563,22 @@ func idlepollCap(cores, spare int) int {
return max(0, min(cores, spare))
}

// setupIdlepoll starts the watcher with at most limit busy loops per burst.
// guestPollsItself: the guest booted with the _CST table of syncIOQemuArgs and registered acpi_idle (driver is the
// content of /sys/devices/system/cpu/cpuidle/current_driver, "none" without the table). Its menu governor then
// busy-waits on the CPU whose task waits for I/O, for as long as that pays.
func guestPollsItself(driver string) bool {
return strings.TrimSpace(driver) == "acpi_idle"
}

// setupIdlepoll starts the watcher with at most limit busy loops per burst, unless the guest busy-waits by itself:
// the loops then add nothing and keep every vCPU busy for the whole burst. 11-10, Arknights resource download on a
// 6-vCPU instance with the table: 10.4-11.8 MiB/s and 6.0 host cores for qemu with the loops, 11.3-12.1 MiB/s and
// 3.8 cores without.
func (m *Manager) setupIdlepoll(ctx context.Context, i int, limit int) {
if guestPollsItself(m.shell(ctx, i, "cat /sys/devices/system/cpu/cpuidle/current_driver", 0).Out) {
m.Log.Infof("实例 %d 密集读写加速: 客体等读写时自己忙等(acpi_idle),不起忙等守护", i)
return
}
remote, ok := m.pushGuestScript(ctx, i, guest.IdlePoll)
if !ok {
return
Expand Down
14 changes: 14 additions & 0 deletions internal/emu/afterboot_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,20 @@ func TestIdlepollReport(t *testing.T) {
}
}

func TestGuestPollsItself(t *testing.T) {
// what `cat /sys/devices/system/cpu/cpuidle/current_driver` prints: acpi_idle with the _CST table, none without
for driver, want := range map[string]bool{
"acpi_idle\n": true, "acpi_idle\r\n": true, " acpi_idle ": true,
"none\n": false, "": false, "haltpoll\n": false, "intel_idle\n": false,
"cat: /sys/devices/system/cpu/cpuidle/current_driver: No such file or directory\n": false,
"acpi_idle2\n": false,
} {
if got := guestPollsItself(driver); got != want {
t.Errorf("guestPollsItself(%q) = %v, want %v", driver, got, want)
}
}
}

func TestParseWriteback(t *testing.T) {
if e, w, ok := parseWriteback("200\r\n100\r\n"); !ok || e != 200 || w != 100 {
t.Errorf("good: %d %d %v", e, w, ok)
Expand Down
8 changes: 5 additions & 3 deletions internal/emu/launch.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,10 @@ const adbServerStartTimeout = 20 * time.Second
// #32). Wi-Fi then comes from the emulator's own hostapd and slirp, where it falls back when netsim is unreachable;
// no Bluetooth, which the first boot turns off anyway. Uwb is on in the SDK defaults, only the image keeps it off.
//
// qemuExtra comes last, still inside the -qemu part: the manager's own QEMU arguments (syncIOQemuArgs).
//
// The window is not an instance option: it is decided at boot by whoever starts it.
func buildLaunchArgs(avdName string, console, memoryMB int, headless bool) []string {
func buildLaunchArgs(avdName string, console, memoryMB int, headless bool, qemuExtra ...string) []string {
args := []string{
"-avd", avdName,
"-no-snapshot",
Expand Down Expand Up @@ -75,7 +77,7 @@ func buildLaunchArgs(avdName string, console, memoryMB int, headless bool) []str
if root.CurrentPlatform().MirrorOS != "macos" {
args = append(args, "-device", "virtio-balloon-pci,free-page-reporting=on")
}
return args
return append(args, qemuExtra...)
}

// launchPlan is what prepareLaunch settled before emulator.exe is started.
Expand Down Expand Up @@ -156,7 +158,7 @@ func (m *Manager) launchPlanFor(ctx context.Context, inst *avd.Instance, memoryM
}

headless := !window
args := buildLaunchArgs(inst.Name, port, memoryMB, headless)
args := buildLaunchArgs(inst.Name, port, memoryMB, headless, m.syncIOQemuArgs(inst)...)
m.pruneLogs(i)
logPath, err := emulatorLogPath(m.Root.LogsDir(), inst.Name)
if err != nil {
Expand Down
169 changes: 169 additions & 0 deletions internal/emu/syncio.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
package emu

import (
"bytes"
"encoding/binary"
"fmt"
"os"
"path/filepath"
"strings"

"modavd/internal/avd"
"modavd/internal/root"
)

// Synchronous disk I/O under WHPX (ModAVD/ModAVD#35). A guest fsync on ext4 is three write requests in a row (data,
// journal, commit), each one a round trip to QEMU that the guest waits for. Two QEMU arguments make that round trip
// cheaper without changing what "written" means (a request still completes only once the data is in the host file):
//
// - -global virtio-blk-pci.ioeventfd=off: WHPX has no in-kernel eventfd, QEMU emulates it in user mode, which costs
// a request three main-loop passes and three thread hand-offs. Off, the vCPU thread handles the request itself.
// - -acpitable <SSDT with a _CST per CPU>: the guest has no cpuidle driver and halts whenever it has nothing to run,
// and Hyper-V needs 20-24 us to wake a halted vCPU for every completion. The table declares one C1 entered by
// HLT with an exit latency of cstLatencyUs. The guest's own acpi_idle driver and menu governor then busy-wait
// for up to 2 x cstLatencyUs when a task on that CPU waits for I/O (or the CPU is about to be needed again
// anyway) and halt otherwise.
//
// Measured 11-10 on a 6-core test instance (two writers, 7 KB + fsync each, the Arknights download pattern):
// 15-16 MB/s without, 19 with the first, 19-22 with the second, 33-36 with both; single-threaded 4 KiB + fsync
// 584-638 us -> 319 us. An idle guest spent 0.1 s of 240 CPU-seconds busy-waiting. The Arknights resource download
// itself, three alternating 150 s windows each: 6.9-7.8 -> 10.4-11.8 MiB/s. With the table the busy loops of
// mas-idlepoll.sh are not started any more (setupIdlepoll).
//
// Windows only: the other hosts use another accelerator and were not measured.
const (
// envSyncIO set to "off" starts the emulator without the two arguments (A/B runs, and the way back).
envSyncIO = "MODAVD_SYNC_IO"
// cstFileName is the table in the instance's own folder, rewritten when the CPU count changes.
cstFileName = "modavd-cst.aml"
cstLatencyUs = 100
// QEMU names the processor objects C000..C0FE (hw/acpi/cpu.c); one more would need another name format.
cstMaxCPUs = 255
)

// syncIOQemuArgs is what goes after -qemu for the two changes above. The table is optional: when it cannot be
// written or named on the command line, the instance starts without it.
func (m *Manager) syncIOQemuArgs(inst *avd.Instance) []string {
if !root.CurrentPlatform().Windows() || strings.EqualFold(strings.TrimSpace(os.Getenv(envSyncIO)), "off") {
return nil
}
args := []string{"-global", "virtio-blk-pci.ioeventfd=off"}
cpus, ok := inst.CPU()
if !ok || cpus < 1 || cpus > cstMaxCPUs {
m.Log.Warnf("实例 %d 读不出 CPU 核数(hw.cpu.ncore),这次开机不带空闲状态表", inst.Index)
return args
}
rel, err := writeCstTable(filepath.Dir(m.Root.EmulatorExe()), inst.Dir, cpus)
if err != nil {
m.Log.Warnf("实例 %d 这次开机不带空闲状态表: %v", inst.Index, err)
return args
}
return append(args, "-acpitable", "file="+rel)
}

// writeCstTable puts the table for cpus processors into instDir and returns its path relative to emuDir, the
// emulator's working directory. Relative because QEMU splits file= at ':' and ',' and opens the name with the ANSI
// code page: <root>\sdk\emulator -> ..\..\avd\mas_<i>.avd\modavd-cst.aml is plain ASCII wherever the root is.
func writeCstTable(emuDir, instDir string, cpus int) (string, error) {
path := filepath.Join(instDir, cstFileName)
rel, err := filepath.Rel(emuDir, path)
if err != nil {
return "", fmt.Errorf("%s 和模拟器目录 %s 不在同一个盘上", path, emuDir)
}
for _, c := range rel {
if c >= 0x80 || c < 0x20 || c == ':' || c == ',' || c == '"' {
return "", fmt.Errorf("表的相对路径 %s 里有 QEMU 命令行放不下的字符", rel)
}
}
table := cstTable(cpus, cstLatencyUs)
if old, err := os.ReadFile(path); err == nil && bytes.Equal(old, table) {
return rel, nil
}
if err := os.WriteFile(path, table, 0o644); err != nil {
return "", err
}
return rel, nil
}

// cstTable is an ACPI SSDT that adds to each of QEMU's processor objects \_SB.CPUS.C000 .. (one per CPU)
//
// Name (_CST, Package () { 1, Package () { ResourceTemplate () { Register (FFixedHW, 0, 0, 0) }, 1, latency, 1000 } })
//
// i.e. one C-state of type C1. Bit offset 0 is not the MWAIT encoding, so Linux enters it with HLT ("ACPI HLT").
// A table naming a processor QEMU did not create fails to load in the guest and changes nothing.
func cstTable(cpus, latencyUs int) []byte {
// Generic Register Descriptor (0x82, 12 bytes: FFixedHW, width 0, offset 0, access 0, address 0) + End Tag
res := append([]byte{0x82, 0x0c, 0x00, 0x7f, 0x00, 0x00, 0x00}, make([]byte, 8)...)
res = append(res, 0x79, 0x00)
buffer := amlPackage(0x11, append(amlInteger(len(res)), res...))
state := []byte{0x04}
state = append(state, buffer...)
state = append(state, amlInteger(1)...) // type C1
state = append(state, amlInteger(latencyUs)...) // worst-case exit latency, microseconds
state = append(state, amlInteger(1000)...) // power, milliwatts (unused by Linux)
cst := append([]byte{0x02}, amlInteger(1)...) // two elements: the count, then the one state
cst = append(cst, amlPackage(0x12, state)...)
term := append([]byte{0x08, '_', 'C', 'S', 'T'}, amlPackage(0x12, cst)...)

var body []byte
for i := 0; i < cpus; i++ {
name := fmt.Sprintf("\\/\x03_SB_CPUSC%03X", i) // root, MultiNamePrefix, 3 segments
body = append(body, amlPackage(0x10, append([]byte(name), term...))...)
}
table := make([]byte, 36, 36+len(body))
copy(table[0:], "SSDT")
binary.LittleEndian.PutUint32(table[4:], uint32(36+len(body)))
table[8] = 1 // revision
copy(table[10:], "MODAVD")
copy(table[16:], "CPUCST ")
binary.LittleEndian.PutUint32(table[24:], 1)
copy(table[28:], "MAVD")
binary.LittleEndian.PutUint32(table[32:], 1)
table = append(table, body...)
var sum byte
for _, b := range table {
sum += b
}
table[9] = -sum
return table
}

// amlPackage is op, PkgLength, content. PkgLength counts its own bytes: one byte up to 63, else the low nibble in the
// lead byte (with the number of following bytes in its top two bits) and the rest in up to three more.
func amlPackage(op byte, content []byte) []byte {
out := []byte{op}
for n := 1; n <= 4; n++ {
total := len(content) + n
if n == 1 {
if total <= 0x3f {
out = append(out, byte(total))
break
}
continue
}
if total < 1<<(4+8*(n-1)) {
out = append(out, byte(n-1)<<6|byte(total&0x0f))
for k, rest := 1, total>>4; k < n; k, rest = k+1, rest>>8 {
out = append(out, byte(rest))
}
break
}
}
return append(out, content...)
}

// amlInteger is the shortest AML integer constant for v (0 <= v < 2^32).
func amlInteger(v int) []byte {
switch {
case v == 0:
return []byte{0x00}
case v == 1:
return []byte{0x01}
case v <= 0xff:
return []byte{0x0a, byte(v)}
case v <= 0xffff:
return []byte{0x0b, byte(v), byte(v >> 8)}
default:
return []byte{0x0c, byte(v), byte(v >> 8), byte(v >> 16), byte(v >> 24)}
}
}
Loading
Loading