diff --git a/docs/backends/nvx/nvx.md b/docs/backends/nvx/nvx.md index 9a937fa67..edb5a1346 100644 --- a/docs/backends/nvx/nvx.md +++ b/docs/backends/nvx/nvx.md @@ -114,10 +114,10 @@ characters are rejected. The signed runtime validates the remaining reference syntax and pulls the selected image on the host. Host-side image acquisition does not grant the guest network access. MXC has no separate registry-credentials input. The runtime registers each reference once and keeps -it by its text in the [agent state directory](#agent-state-directory), so later -requests boot it without contacting the registry, and a moving tag such as -`latest` keeps naming the content first pulled. Use a version tag or a digest -(`repository@sha256:…`) to control what a sandbox boots. +it by its text in the [agent state directory](#agent-state-directory). While +it remains registered, later requests boot it without contacting the registry, +and a moving tag such as `latest` keeps naming the content first pulled. Use a +version tag or a digest (`repository@sha256:…`) to control what a sandbox boots. ## Supported operations and policy @@ -272,20 +272,38 @@ runtime keeps its own state. MXC obtains that digest from the pinned MXC makes the current user the directory's owner and gives it an access list for only that user and Local SYSTEM, so the user's elevated and non-elevated requests can share it. Every create-and-run request -reuses it, so an image is pulled once rather than for every request, and -concurrent requests share it. MXC coordinates the agent's stop and deprovision -operations and closes the client after each request; it does not maintain a -persistent container database. - -To pull a reference again or reclaim disk space, delete the directory while no -MicroVM request or platform-owning SDK process is using it; the next request -creates it again and captures new templates. Lifecycle operations retain their -separate, unversioned root, `%LOCALAPPDATA%\Microsoft\MXC\nvx`, so existing -container IDs and records do not move. Removing a bundle-specific directory does -not remove lifecycle containers, registered images, or templates in that root. -An upgrade that changes the runtime file digests cannot start containers -provisioned with the previous files. Deprovision those containers and provision -them again with the selected bundle; do not move or edit their records. +reuses it, and concurrent requests share registered images. MXC coordinates the +agent's stop and deprovision operations and closes the client after each request; +it does not maintain a persistent container database. + +Lifecycle operations retain their separate, unversioned root, +`%LOCALAPPDATA%\Microsoft\MXC\nvx`, so existing container IDs and records do not +move. Each root has a 10 GiB (10,737,418,240 bytes) limit on the combined size of +counted staged images. The limit is not a total disk-usage limit: temporary +conversion files, templates, other state, and older bundle-specific roots are +outside it. Other agent hosts sharing a root can use different limits. + +When a new image needs space, the agent evicts the largest eligible unused +images first. Images used by provisioned, running, or stopped sandboxes are +retained until the sandboxes are deprovisioned. The agent also retains images +protected as host defaults or prefetched paths and images registered by a +caller-owned path. A newly admitted image larger than the limit, or one that +cannot fit without evicting protected or in-use images, fails registration. If +an eviction fails, earlier successful evictions can remain in effect. + +The limit applies during admission; it does not immediately trim an existing +cache above 10 GiB, and reusing a registered image does not trigger a trim. An +evicted reference must be pulled again and cannot be used while its registry is +unreachable. There is no age-based refresh: an intact registered reference can +continue to use its original content indefinitely. + +Removing a bundle-specific directory while no MicroVM request or platform-owning +SDK process is using it removes that root's cache and templates; the next request +creates it again. It does not remove lifecycle containers, registered images, or +templates in the separate lifecycle root. An upgrade that changes the runtime +file digests cannot start containers provisioned with the previous files. +Deprovision those containers and provision them again with the selected bundle; +do not move or edit their records. If deprovisioning fails, MXC reports the failure and the directory's location. The agent's records for that sandbox remain there rather than being deleted diff --git a/docs/development/build-and-test/ci-validation-infrastructure.md b/docs/development/build-and-test/ci-validation-infrastructure.md index 097480a5c..aa8385de2 100644 --- a/docs/development/build-and-test/ci-validation-infrastructure.md +++ b/docs/development/build-and-test/ci-validation-infrastructure.md @@ -349,6 +349,21 @@ capture followed by reuse without recapture in later processes. Concurrent first requests exercise both template capture and image registration; later concurrent requests must still clone the same template with the registry unreachable. +The image-cache capacity scenario uses the signed image tool to create a +large protected image and verifies admission across the actual 10 GiB boundary +in both agent state roots. It checks root independence, concurrent create-and-run +requests, eviction and re-pull, reuse of an existing over-limit cache, dry-run, +subsequent admission, and protection of provisioned, running, and stopped +containers. It also checks oversized-image rejection through the signed agent's +local-image registration. The test needs registry access and at least 30 GiB of +free disk space available to the test process on the volume holding the staged +test executable. It checks that volume before creating its fixtures and fails +with the available and required byte counts when space is insufficient. If the +volume cannot be queried, it reports the directory and error. The check does not +reserve space; concurrent use and temporary files on other volumes can still +cause the test to fail. The limit covers counted staged images, not temporary +files or other agent state. + Explicit 256-MiB and 512-MiB requests exercise create-and-run execution and lifecycle starts. Native diagnostics must report a clone for 256 MiB and a cold boot with a memory-capacity reason for 512 MiB, without an initialization-fallback @@ -404,7 +419,8 @@ and unpack the Rust crate package to check that its tests compile. Actual VM scenarios run only in the scheduled Windows x64 job. Its dispatcher requires completion markers for CLI and SDK snapshots, lifecycle adoption, fallback warnings and lifecycle diagnostics, concurrent image reuse, and all three -interruption phases, as well as the existing execution and policy scenarios. +interruption phases, the image-cache capacity scenario, and the existing +execution and policy scenarios. Missing markers, empty suites, skips, ignored tests, and filtered tests fail the job. diff --git a/scripts/ci/run_backend_validation_tests.ps1 b/scripts/ci/run_backend_validation_tests.ps1 index 42810e891..9b5b1e683 100644 --- a/scripts/ci/run_backend_validation_tests.ps1 +++ b/scripts/ci/run_backend_validation_tests.ps1 @@ -539,6 +539,7 @@ function Invoke-MicroVmTests { 'JSON image and policy' = 'VM_EXECUTED JSON selected-image filesystem read/write/deny and cleanup confirmed clones' 'executor' = 'VM_EXECUTED executor sequential ' 'executor image reuse' = 'VM_EXECUTED executor registered-image reuse offline concurrent first-capture clone isolation' + 'image cache capacity' = 'VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection' 'executor configs' = 'VM_EXECUTED executor configs ' 'executor interruption' = 'VM_EXECUTED executor console interruption capture/startup/execution/owned-vmm/cleanup' 'network rules' = 'VM_EXECUTED reachable controls and network allow/deny/nonmatching-port enforcement confirmed clones' diff --git a/scripts/ci/stage-nvx-validation-tests.ps1 b/scripts/ci/stage-nvx-validation-tests.ps1 index bbd3cc6fa..712dc8c71 100644 --- a/scripts/ci/stage-nvx-validation-tests.ps1 +++ b/scripts/ci/stage-nvx-validation-tests.ps1 @@ -57,6 +57,8 @@ foreach ($name in @( 'windows_x64::signed_bundle_executor_configs', 'windows_x64::signed_bundle_executor_create_and_run', 'windows_x64::signed_bundle_executor_reuses_registered_images', + 'windows_x64::image_cache_fixture_space_check_fails_closed', + 'windows_x64::signed_bundle_image_cache_capacity', 'windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots', 'windows_x64::signed_bundle_snapshot_memory_capacity', 'windows_x64::signed_bundle_executor_console_interrupt', diff --git a/scripts/tests/microvm-validation.test.js b/scripts/tests/microvm-validation.test.js index 9b5d2c51f..c56874578 100644 --- a/scripts/tests/microvm-validation.test.js +++ b/scripts/tests/microvm-validation.test.js @@ -203,6 +203,12 @@ fn main() { if mode != "missing-image-reuse-discovery" { println!("windows_x64::signed_bundle_executor_reuses_registered_images: test"); } + if mode != "missing-image-space-check-discovery" { + println!("windows_x64::image_cache_fixture_space_check_fails_closed: test"); + } + if mode != "missing-image-capacity-discovery" { + println!("windows_x64::signed_bundle_image_cache_capacity: test"); + } if mode != "missing-platform-snapshots-discovery" { println!("windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots: test"); } @@ -293,6 +299,9 @@ fn main() { } else { println!("VM_EXECUTED executor registered-image reuse offline and concurrent"); } + if mode != "missing-image-capacity" { + println!("VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection"); + } if mode != "missing-network" { let suffix = if mode == "unconfirmed-network" { "" } else { " confirmed clones" }; println!("VM_EXECUTED reachable controls and network allow/deny/nonmatching-port enforcement{suffix}"); @@ -696,6 +705,7 @@ test("artifact dispatcher executes prebuilt SDK tests and preserves the log", () assert.match(log, /VM_EXECUTED executor /); assert.match(log, /VM_EXECUTED executor console interruption capture\/startup\/execution\/owned-vmm\/cleanup/); assert.match(log, /VM_EXECUTED executor registered-image reuse offline concurrent first-capture clone isolation/); + assert.match(log, /VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection/); assert.match(log, /VM_EXECUTED reachable controls and network allow\/deny\/nonmatching-port enforcement confirmed clones/); assert.match(log, /VM_EXECUTED UDP datagrams replies delivery counts protocol separation and inclusive ranges confirmed clones/); assert.match(log, /VM_EXECUTED host-loopback ingress TCP\/UDP allow deny ready listeners and reply controls confirmed clones/); @@ -719,6 +729,7 @@ for (const [mode, message] of [ ["missing-configs", /did not execute the executor configs VM scenario/], ["missing-interruption", /did not execute the executor interruption VM scenario/], ["missing-image-reuse", /did not execute the executor image reuse VM scenario/], + ["missing-image-capacity", /did not execute the image cache capacity VM scenario/], ["missing-network", /did not execute the network rules VM scenario/], ["missing-network-udp", /did not execute the UDP network rules VM scenario/], ["missing-host-loopback", /did not execute the host-loopback ingress VM scenario/], @@ -888,6 +899,8 @@ for (const [mode, message] of [ ["missing-sdk-lifecycle-discovery", /did not discover windows_x64::signed_bundle_sdk_json_lifecycle/], ["missing-interruption-discovery", /did not discover windows_x64::signed_bundle_executor_console_interrupt/], ["missing-image-reuse-discovery", /did not discover windows_x64::signed_bundle_executor_reuses_registered_images/], + ["missing-image-space-check-discovery", /did not discover windows_x64::image_cache_fixture_space_check_fails_closed/], + ["missing-image-capacity-discovery", /did not discover windows_x64::signed_bundle_image_cache_capacity/], ["missing-platform-snapshots-discovery", /did not discover[\s\S]*windows_x64::signed_bundle_executor_starts_sandboxes_from_platform_snapshots/], ["missing-snapshot-memory-discovery", /did not discover[\s\S]*windows_x64::signed_bundle_snapshot_memory_capacity/], ["missing-network-discovery", /did not discover[\s\S]*windows_network::signed_bundle_network_rules_enforce_allow_deny_and_port_selectors/], @@ -911,6 +924,8 @@ for (const [mode, message] of [ "missing-sdk-lifecycle-discovery", "missing-interruption-discovery", "missing-image-reuse-discovery", + "missing-image-space-check-discovery", + "missing-image-capacity-discovery", "missing-platform-snapshots-discovery", "missing-snapshot-memory-discovery", "missing-network-discovery", diff --git a/src/mxc-sdk/src/backends/nvx/crate_runtime.rs b/src/mxc-sdk/src/backends/nvx/crate_runtime.rs index 552ffd6c8..756e24772 100644 --- a/src/mxc-sdk/src/backends/nvx/crate_runtime.rs +++ b/src/mxc-sdk/src/backends/nvx/crate_runtime.rs @@ -181,9 +181,12 @@ pub(super) struct NvxClient(Option); // unless a request sets their memory, so the smallest templates fit them. const PLATFORM_SNAPSHOT_MAX_VCPUS: u32 = 1; const PLATFORM_SNAPSHOT_MAX_MEMORY_MIB: u32 = 256; +const IMAGE_CACHE_CAPACITY_BYTES: u64 = 10 * 1024 * 1024 * 1024; fn agent_setup(bundle: &BundledAgent, state_root: &Path) -> SetupConfig { - bundle.setup(state_root).with_cpu_profile(CpuProfile::Host) + let mut setup = bundle.setup(state_root).with_cpu_profile(CpuProfile::Host); + setup.images.max_cache_capacity_bytes = Some(IMAGE_CACHE_CAPACITY_BYTES); + setup } static RUNTIME_DIRECTORY: OnceLock = OnceLock::new(); @@ -1549,11 +1552,20 @@ mod tests { let bundle = BundledAgent::from_dir(r"C:\bundle").unwrap(); let setup = agent_setup(&bundle, Path::new(r"C:\state")); assert_eq!(setup.cpu_profile, CpuProfile::Host); + assert_eq!(setup.images.max_cache_capacity_bytes, Some(10_737_418_240)); + let mut other_image_settings = setup.images.clone(); + other_image_settings.max_cache_capacity_bytes = None; + assert_eq!( + other_image_settings, + aci_edge_sandboxes::agent::ImageSettings::default() + ); let json = serde_json::to_value(&setup).unwrap(); assert_eq!(json["cpuProfile"], "host"); + assert_eq!(json["images"]["maxCacheCapacityBytes"], 10_737_418_240_u64); assert!(json.get("platformSnapshots").is_none(), "{json}"); let setup = setup.with_platform_snapshots(platform_snapshots()); + assert_eq!(setup.images.max_cache_capacity_bytes, Some(10_737_418_240)); assert_eq!( serde_json::to_value(&setup).unwrap()["platformSnapshots"], serde_json::json!({"enabled": true, "maxVcpus": 1, "maxMemoryMib": 256}) diff --git a/src/mxc-sdk/tests/nvx_e2e.rs b/src/mxc-sdk/tests/nvx_e2e.rs index 9c18311e8..748dcd7da 100644 --- a/src/mxc-sdk/tests/nvx_e2e.rs +++ b/src/mxc-sdk/tests/nvx_e2e.rs @@ -220,6 +220,73 @@ mod windows_x64 { const CAPTURE_LIMIT: usize = 64 * 1024 * 1024; const CAPTURE_ERROR: &str = "NVX signed bundled agent: combined stdout/stderr capture limit of 67108864 bytes exceeded"; + const IMAGE_CACHE_FIXTURE_FREE_BYTES: u64 = 30 * 1024 * 1024 * 1024; + + fn validate_image_cache_fixture_space( + directory: &Path, + available: u64, + required: u64, + ) -> Result<(), String> { + if available < required { + return Err(format!( + "image cache fixture directory {} has {available} bytes available; \ + at least {required} bytes are required. Free space on this volume or \ + run the test from a volume with sufficient space", + directory.display() + )); + } + Ok(()) + } + + fn check_image_cache_fixture_space(directory: &Path, required: u64) -> Result<(), String> { + use std::os::windows::ffi::OsStrExt; + use windows::core::PCWSTR; + use windows::Win32::Storage::FileSystem::GetDiskFreeSpaceExW; + + let mut path: Vec = directory.as_os_str().encode_wide().collect(); + if path.last() != Some(&(b'\\' as u16)) { + path.push(b'\\' as u16); + } + path.push(0); + let mut available = 0; + unsafe { GetDiskFreeSpaceExW(PCWSTR(path.as_ptr()), Some(&mut available), None, None) } + .map_err(|error| { + format!( + "cannot determine free space for image cache fixture directory {}: {error}", + directory.display() + ) + })?; + println!( + "image cache fixture directory {}: {available} bytes available; \ + {required} bytes required", + directory.display() + ); + validate_image_cache_fixture_space(directory, available, required) + } + + #[test] + fn image_cache_fixture_space_check_fails_closed() { + let directory = tempfile::tempdir().expect("space-check test directory"); + let required = IMAGE_CACHE_FIXTURE_FREE_BYTES; + assert!(validate_image_cache_fixture_space(directory.path(), required, required).is_ok()); + assert!( + validate_image_cache_fixture_space(directory.path(), required + 1, required).is_ok() + ); + let error = validate_image_cache_fixture_space(directory.path(), required - 1, required) + .unwrap_err(); + assert!( + error.contains(&directory.path().display().to_string()), + "{error}" + ); + assert!(error.contains(&(required - 1).to_string()), "{error}"); + assert!(error.contains(&required.to_string()), "{error}"); + assert!(check_image_cache_fixture_space(directory.path(), 0).is_ok()); + assert!(check_image_cache_fixture_space(directory.path(), u64::MAX).is_err()); + let missing = directory.path().join("missing"); + let error = check_image_cache_fixture_space(&missing, 0).unwrap_err(); + assert!(error.contains(&missing.display().to_string()), "{error}"); + assert!(error.contains("cannot determine free space"), "{error}"); + } #[test] fn native_validation_rejects_requests_without_booting_a_vm() { @@ -1869,6 +1936,457 @@ print("NVX_JSON_IMAGE_AND_POLICY_OK", end="") ); } + #[test] + #[ignore = "VM: requires Windows x64, WHP, signed bundle, registry access and 30 GiB free disk"] + fn signed_bundle_image_cache_capacity() { + use aci_edge_sandboxes::agent::{ + AgentBackend, BundledAgent, ImageDigest, ImageSettings, SandboxDefaults, + }; + use aci_edge_sandboxes::ImageSource; + use std::ffi::OsStr; + + fn container_from_output(output: &Output, local: &Path) -> LifecycleContainer { + LifecycleContainer { + id: serde_json::from_slice::(&output.stdout).unwrap()["result"]["sandboxId"] + .as_str() + .unwrap() + .to_owned(), + local_app_data: Some(local.to_owned()), + cleaned: false, + } + } + + fn deprovision(container: &mut LifecycleContainer, local: &Path) { + let output = executor_output_with_env( + &lifecycle_phase(&container.id, "deprovision"), + false, + false, + &[("LOCALAPPDATA", local.as_os_str())], + ); + assert!(output.status.success(), "{output:?}"); + container.cleaned = true; + } + + const CAPACITY: u64 = 10_737_418_240; + const MIB: u64 = 1024 * 1024; + fn padding_image(bundle: &BundledAgent, directory: &Path, length: u64) -> PathBuf { + let raw = directory.join("padding.raw"); + let vhd = directory.join("padding.vhd"); + std::fs::File::create(&raw) + .unwrap() + .set_len(length - MIB) + .unwrap(); + let output = Command::new(bundle.directory().join(r"bin\direct-images.exe")) + .arg("create-gpt") + .arg("--raw-file") + .arg(&raw) + .arg("--output") + .arg(&vhd) + .arg("--format") + .arg("json") + .output() + .expect("generate capacity fixture"); + assert!(output.status.success(), "{output:?}"); + std::fs::remove_file(raw).unwrap(); + vhd + } + + let bundle = BundledAgent::discover().expect("signed NVX bundle"); + let test_dir = std::env::current_exe() + .expect("test executable") + .parent() + .expect("test executable directory") + .to_path_buf(); + check_image_cache_fixture_space(&test_dir, IMAGE_CACHE_FIXTURE_FREE_BYTES) + .unwrap_or_else(|error| panic!("{error}")); + + for lifecycle in [false, true] { + let fixture = tempfile::tempdir_in(&test_dir).expect("capacity fixture directory"); + let local = fixture.path().join("local"); + std::fs::create_dir(&local).unwrap(); + let environment = [("LOCALAPPDATA", local.as_os_str())]; + let dry_request = if lifecycle { + lifecycle_provision() + } else { + process_only_request("printf cache") + }; + let output = executor_output_with_env(&dry_request, false, true, &environment); + assert!(output.status.success(), "{output:?}"); + let root = if lifecycle { + local.join(r"Microsoft\MXC\nvx") + } else { + agent_state_directory(&local) + }; + + let seed = AgentBackend::new(bundle.config(bundle.setup(&root))) + .expect("open unlimited fixture host"); + let mut images = [ + "mcr.microsoft.com/azurelinux/base/core:3.0", + "mcr.microsoft.com/cbl-mariner/base/core:2.0", + "docker.io/library/python:3.12-alpine", + ] + .map(|reference| { + let image = seed + .register_reference(reference, ImageDigest::Compute) + .expect("register fixture reference"); + (reference, image) + }); + images.sort_by_key(|(_, image)| std::cmp::Reverse(image.length)); + let [(old_ref, old), (new_ref, new), (third_ref, third)] = images; + assert_ne!( + old.id, new.id, + "fixture references {old_ref} and {new_ref} have the same content" + ); + assert_ne!( + third.id, old.id, + "fixture references {third_ref} and {old_ref} have the same content" + ); + assert_ne!( + third.id, new.id, + "fixture references {third_ref} and {new_ref} have the same content" + ); + assert!( + new.length > MIB && old.length > new.length && old.length < CAPACITY, + "fixture sizes must satisfy 1 MiB < {new_ref} ({}) < {old_ref} ({}) < \ + 10 GiB; choose suitable distinct references", + new.length, + old.length + ); + assert!( + third.length > 0 && new.length + third.length < CAPACITY, + "fixture sizes for {new_ref} ({}) and {third_ref} ({}) must fit below 10 GiB; \ + choose suitable distinct references", + new.length, + third.length + ); + assert!( + new.length > third.length, + "fixture images must have distinct sizes: {new_ref} ({}), {third_ref} ({}); \ + choose distinct stable references", + new.length, + third.length + ); + assert!(seed.unregister_image(&new.id).unwrap()); + assert!(seed.unregister_image(&third.id).unwrap()); + + let mut old_request = if lifecycle { + lifecycle_provision() + } else { + process_only_request("printf cache") + }; + old_request["microvm"]["image"] = json!(old_ref); + let output = executor_output_with_env(&old_request, false, false, &environment); + assert!(output.status.success(), "{output:?}"); + let mut old_container = lifecycle.then(|| container_from_output(&output, &local)); + + // Both the initial eviction and the later largest-first check must fit. + let padding_length = + ((CAPACITY - old.length).min(CAPACITY - new.length - third.length) / MIB) * MIB; + assert!(padding_length >= 2 * MIB, "padding must be at least 2 MiB"); + assert!( + padding_length + old.length <= CAPACITY, + "padding plus the larger image must fit" + ); + assert!( + padding_length + old.length + new.length > CAPACITY, + "padding plus both images must exceed capacity" + ); + assert!( + padding_length + new.length + third.length <= CAPACITY, + "the smaller and incoming images must fit with padding" + ); + assert!( + padding_length + old.length + third.length > CAPACITY, + "the larger and incoming images must exceed capacity with padding" + ); + let vhd = padding_image(&bundle, fixture.path(), padding_length); + + let setup = bundle + .setup(&root) + .with_defaults(SandboxDefaults::default().with_image(ImageSource::Path(vhd))) + .with_images(ImageSettings::default().with_max_cache_capacity_bytes(CAPACITY)); + let protected = + AgentBackend::new(bundle.config(setup)).expect("protect capacity fixture"); + let padding = protected.image_id().expect("protected fixture ID"); + let registered = protected.images().unwrap(); + let actual_padding = registered.iter().find(|image| image.id == padding).unwrap(); + assert_eq!(actual_padding.length, padding_length); + + if !lifecycle { + let mut other_root = lifecycle_provision(); + other_root["microvm"]["image"] = json!(new_ref); + let output = executor_output_with_env(&other_root, false, false, &environment); + assert!(output.status.success(), "{output:?}"); + let mut container = container_from_output(&output, &local); + deprovision(&mut container, &local); + assert!(protected + .images() + .unwrap() + .iter() + .any(|image| image.id == old.id)); + } + + let mut new_request = if lifecycle { + lifecycle_provision() + } else { + process_only_request("printf cache") + }; + new_request["microvm"]["image"] = json!(new_ref); + if let Some(container) = &mut old_container { + let dry = executor_output_with_env(&new_request, false, true, &environment); + assert!(dry.status.success(), "{dry:?}"); + assert!(protected + .images() + .unwrap() + .iter() + .any(|image| image.id == old.id)); + for phase in [None, Some("start"), Some("stop")] { + if let Some(phase) = phase { + let output = executor_output_with_env( + &lifecycle_phase(&container.id, phase), + false, + false, + &environment, + ); + assert!(output.status.success(), "{output:?}"); + } + let blocked = + executor_output_with_env(&new_request, false, false, &environment); + assert!(!blocked.status.success(), "{blocked:?}"); + let message = format!( + "{}{}", + String::from_utf8_lossy(&blocked.stdout), + String::from_utf8_lossy(&blocked.stderr) + ); + assert!(message.contains("backend_unavailable"), "{message}"); + assert!(message.contains("image cache"), "{message}"); + assert!(message.contains("unused and evictable"), "{message}"); + assert!(protected + .images() + .unwrap() + .iter() + .any(|image| image.id == old.id)); + } + + deprovision(container, &local); + } + + let output = if lifecycle { + executor_output_with_env(&new_request, false, false, &environment) + } else { + std::thread::scope(|scope| { + let runs = (0..2) + .map(|_| { + scope.spawn(|| { + executor_output_with_env(&new_request, false, false, &environment) + }) + }) + .collect::>(); + let mut outputs = runs + .into_iter() + .map(|run| run.join().expect("concurrent capacity request")); + let first = outputs.next().unwrap(); + for output in outputs { + assert!(output.status.success(), "{output:?}"); + } + first + }) + }; + assert!(output.status.success(), "{output:?}"); + let mut new_container = lifecycle.then(|| container_from_output(&output, &local)); + let registered = protected.images().unwrap(); + assert!(!registered.iter().any(|image| image.id == old.id)); + assert!(registered.iter().any(|image| image.id == padding)); + let current: Vec<_> = registered + .iter() + .filter(|image| image.id != padding) + .collect(); + assert_eq!(current.len(), 1, "{registered:?}"); + assert_eq!( + current[0].length, new.length, + "fixture reference {new_ref} changed size between registrations" + ); + let current_id = current[0].id; + assert_ne!(current_id, padding, "fixture images must be distinct"); + + let offline = [ + environment[0], + ("HTTPS_PROXY", OsStr::new("http://127.0.0.1:9")), + ("HTTP_PROXY", OsStr::new("http://127.0.0.1:9")), + ]; + let output = executor_output_with_env(&old_request, false, false, &offline); + assert!( + !output.status.success(), + "evicted reference must be pulled again" + ); + let failure = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(failure.contains(old_ref), "{failure}"); + if let Some(container) = &mut new_container { + deprovision(container, &local); + } + let output = executor_output_with_env(&old_request, false, false, &environment); + assert!(output.status.success(), "{output:?}"); + let registered = protected.images().unwrap(); + assert!(!registered.iter().any(|image| image.id == current_id)); + let restored: Vec<_> = registered + .iter() + .filter(|image| image.id != padding) + .collect(); + assert_eq!(restored.len(), 1, "{registered:?}"); + assert_eq!( + restored[0].length, old.length, + "fixture reference {old_ref} changed size between registrations" + ); + if lifecycle { + let mut container = container_from_output(&output, &local); + deprovision(&mut container, &local); + } + + // Register the smaller candidate first so oldest-first cannot pass this check. + assert!( + seed.unregister_image(&restored[0].id).unwrap(), + "remove the larger image before arranging registration order" + ); + let empty_candidates = protected.images().unwrap(); + assert_eq!(empty_candidates.len(), 1, "{empty_candidates:?}"); + assert_eq!(empty_candidates[0].id, padding); + let extra = seed + .register_reference(new_ref, ImageDigest::Compute) + .expect("register the older, smaller eviction candidate"); + let smaller_only = protected.images().unwrap(); + assert_eq!(smaller_only.len(), 2, "{smaller_only:?}"); + assert!(smaller_only.iter().any(|image| image.id == padding)); + assert!(smaller_only.iter().any(|image| image.id == extra.id)); + assert_eq!( + extra.length, new.length, + "fixture reference {new_ref} changed size between registrations" + ); + assert_ne!(extra.id, padding, "fixture images must be distinct"); + assert!( + !smaller_only.iter().any(|image| image.id == restored[0].id), + "larger candidate must be absent before its later registration: {smaller_only:?}" + ); + let largest = seed + .register_reference(old_ref, ImageDigest::Compute) + .expect("register the newer, larger eviction candidate"); + assert_eq!( + largest.length, old.length, + "fixture reference {old_ref} changed size between registrations" + ); + assert_ne!(largest.id, extra.id, "eviction candidates must be distinct"); + assert_ne!(largest.id, padding, "fixture images must be distinct"); + + // An older unlimited host can leave this root above the new limit. + let before = protected.images().unwrap(); + assert!(before.iter().map(|image| image.length).sum::() > CAPACITY); + assert!(before.iter().any(|image| image.id == extra.id)); + assert_eq!(before.len(), 3, "{before:?}"); + assert!(before.iter().any(|image| image.id == padding)); + assert!(before.iter().any(|image| image.id == largest.id)); + assert_eq!(extra.length, new.length); + assert!(old.length > extra.length); + let total = before.iter().map(|image| image.length).sum::(); + let after_largest = total - largest.length + third.length; + let after_smaller = total - extra.length + third.length; + assert!( + after_largest <= CAPACITY, + "evicting the largest image alone must make room: {after_largest} > {CAPACITY}; \ + {before:?}" + ); + assert!( + after_smaller > CAPACITY, + "evicting only the smaller image must not make room: {after_smaller} <= \ + {CAPACITY}; {before:?}" + ); + println!( + "image cache largest-first fixture: lifecycle={lifecycle}, capacity={CAPACITY}, \ + padding={padding_length}, older smaller={new_ref}:{}:{}, \ + newer largest={old_ref}:{}:{}, incoming={third_ref}:{}, \ + after-largest={after_largest}, after-smaller={after_smaller}", + extra.id, extra.length, largest.id, largest.length, third.length + ); + let output = executor_output_with_env(&old_request, false, false, &environment); + assert!(output.status.success(), "{output:?}"); + if lifecycle { + let mut container = container_from_output(&output, &local); + deprovision(&mut container, &local); + } + let after = protected.images().unwrap(); + assert_eq!( + before.iter().map(|image| image.id).collect::>(), + after.iter().map(|image| image.id).collect::>(), + "reusing an existing image must not trim an over-limit cache" + ); + let mut third_request = if lifecycle { + lifecycle_provision() + } else { + process_only_request("printf cache") + }; + third_request["microvm"]["image"] = json!(third_ref); + let output = executor_output_with_env(&third_request, false, false, &environment); + assert!(output.status.success(), "{output:?}"); + if lifecycle { + let mut container = container_from_output(&output, &local); + deprovision(&mut container, &local); + } + let after_admission = protected.images().unwrap(); + assert!(after_admission.iter().any(|image| image.id == padding)); + assert!( + after_admission + .iter() + .map(|image| image.length) + .sum::() + <= CAPACITY + ); + assert!( + !after_admission.iter().any(|image| image.id == largest.id), + "admission must evict the largest unused image" + ); + assert!( + after_admission.iter().any(|image| image.id == extra.id), + "admission must retain the smaller unused image" + ); + let admitted: Vec<_> = after_admission + .iter() + .filter(|image| image.id != padding && image.id != extra.id) + .collect(); + assert_eq!(admitted.len(), 1, "{after_admission:?}"); + assert_eq!( + admitted[0].length, third.length, + "fixture reference {third_ref} changed size between registrations" + ); + assert_ne!( + admitted[0].id, largest.id, + "fixture images must be distinct" + ); + protected.close().unwrap(); + seed.close().unwrap(); + } + let fixture = tempfile::tempdir_in(&test_dir).expect("oversized fixture directory"); + let oversized = padding_image(&bundle, fixture.path(), CAPACITY + MIB); + assert_eq!( + std::fs::metadata(&oversized).unwrap().len(), + CAPACITY + MIB + 512 + ); + let setup = bundle + .setup(fixture.path().join("state")) + .with_defaults(SandboxDefaults::default().with_image(ImageSource::Path(oversized))) + .with_images(ImageSettings::default().with_max_cache_capacity_bytes(CAPACITY)); + let error = AgentBackend::new(bundle.config(setup)).unwrap_err(); + assert_eq!( + error.code(), + aci_edge_sandboxes::ErrorCode::PolicyValidation + ); + assert!(error + .to_string() + .contains("more than the image cache's capacity")); + println!("VM_EXECUTED image cache 10 GiB eviction repull and lifecycle protection"); + } + #[test] #[ignore = "VM: requires separately built wxc-exec.exe, WHP and exact signed official bundle"] fn signed_bundle_executor_starts_sandboxes_from_platform_snapshots() {