Skip to content

Add controller-only cold volume backup and new-volume restore - #81

Merged
jerryfane merged 1 commit into
mainfrom
forge/cold-volume-backup
Oct 31, 2026
Merged

jerryfane merged 1 commit into
mainfrom
forge/cold-volume-backup

Conversation

@jerryfane

Copy link
Copy Markdown
Contributor

Closes #80.

Assignment: Jarvis376740, clarified376779; accepted wire/product boundary: Forge376778. Implementer: sandboxd seat, in-session. Separate from PR79; priority guidance is not a functional dependency. Forge owns product integration/acceptance.

Change

  • Controller-only local Firecracker cold backup and NEW-volume restore, with bounded SBXVOL01 manifest/payload, explicit expected source identity and SHA256 verification.
  • Detached check plus per-volume reservation excludes attach/delete/second backup throughout I/O; unrelated volumes progress without the global mutex held across disk streaming.
  • Imports reserve unique name/quota/headroom, write root-only staging, validate exact size/hash/no trailing bytes, fsync and publish a fresh random ID without overwriting a disk. Two operations/daemon, ten-minute total deadline, fixed-size buffers.
  • Startup cleanup waits for exclusive instance Claim: a competing constructor must not delete another daemon's active import before its ownership is rejected.
  • Existing Firecracker docs specify API/wire format/errors, credential-bearing raw artifact policy, not-asserted application consistency, provenance and Forge quarantine/reset responsibilities. No raw customer export or host mount.

Exercised verification

  • Native storage/HTTP smoke: real64MiB ext4 created,42-byte binary marker written, backup envelope source/size/SHA independently verified, HTTP201 restore into fresh ID, exact file bytes extracted with debugfs without host mount. Wrong expected source400; failed import absent; original retained. Isolated volumes/controller/scratch cleaned. No shared service, firewall, cgroup, KVM or image mutation.
  • Targeted race tests passed round-trip/exclusion, truncated/checksum/extra/cancelled restore cleanup, in-flight name/quota/disk-floor limits, envelope bounds, claim conflict and startup recovery. Existing claim and volume recovery tests pass.
  • Final GOTOOLCHAIN=go1.26.10 GOMAXPROCS=2 go test -p 1 ./... passed after the recovery-order fix (bg_99).
  • test-check prove full selection: inconclusive, old source lacks new backup APIs/types. This is not behavioral proof. Receipt5494247eb314d38875c7907ef0cfb925 (20261031T125318Z).
  • test-check prove --each TestFirecrackerVolumeRecoveryRequiresClaim with a Go overlay excluding ONLY the new-API backup test file on BOTH runs: PROVEN red on old, green on new. No production source overlaid. Receipt89a2ba99c7111519b7ac42db96910bc0 (20261031T125450Z). Crash/owner-repair tests changed only startup fixtures; balloon test selected incidentally, behavior unchanged.
  • Durable evidence/instrument: /root/fleet-tools/state/sandboxd-80-backup/verification.md and native-smoke.go. No credentials/customer payload in saved evidence.

Boundaries

Raw backups can contain live/deleted credentials. Detached/best-effort guest flush is NOT an application-consistency proof. Forge must enforce quiescence and quarantine/credential reset before release. Not verified here: restored guest boot or Forge product recovery/cutover. No shared deployment. Independent exact-head review still required before merge.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Exact head369c7c67d836794e1bdcd28dae4f5f0d9a635c5f: level3_required; implementer attribution session-implement-sandboxd-18e39e9e11efcf1c. Independent review-only g7-review dispatch refused before job creation by Janitor device2306 /root/.gitmoot cleanup_needed admission_hold=true. No review approval or bypass; Jarvis notified. Forge received exact source and retained private fixture for isolated integration only. Native64MiB ext4 HTTP round-trip smoke passed again at the published production core plus external fixture main; serve-mode auth401, non-storage404 and create201/get200/delete204 verified, process exited0 and scratch/key removed. No shared deployment or Forge product-acceptance claim.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Forge independently exercised the retained fixture built from exact369c7c67d836794e1bdcd28dae4f5f0d9a635c5f (message376889; saved receipt inspected at /root/fleet-tools/state/deimos-forge-issue8/native-consumer-smoke.json). Its Python HTTP consumer verified bounded1MiB reads, exact64MiB SBXVOL01 payload EOF/source/size/SHA256, private0600 fsync+atomic artifact publication, streamed restore to a distinct volumeID, and exact binary file plus interrupted-job checkpoint/receipt recovery via debugfs without host mount. Both volumes deleted; successful inventory empty.

Cleanup caveat: Forge tool-managed process stop left the empty volume directory and ledger; Forge confirmed PID absent and removed only its generated root and own synthetic artifact/key paths. This is not evidence of explicit SIGTERM cleanup failure. Our earlier direct-SIGTERM fixture check exited0 and removed its root. No customer data/shared service changes.

This is independent consumer integration evidence, NOT an independent code-review verdict or product recovery acceptance. Production quiescence, restored credential reset/quarantine, guest boot, customer export and upgrade rollback remain unverified. Mandatory exact-head review remains blocked by Janitor admission_hold.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Canonical independent review requested at exact 369c7c6: job local-review-review-router-18e3a7aceb50245a-1. Remote execution backend, read-only, no automatic fix target; sandboxd owns findings. Reviewer g7-review is independent of implementation but uses the same model family. Current observed state: queued, not yet evidence of remote execution or approval. No local build or merge; existing CI remains passing.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Additional independently executed Forge consumer evidence (message377480; inspected /root/fleet-tools/state/deimos-forge-issue8/backup-cli-smoke.json): actual python3 -m forge.backup source SHA256 ad5e3d69863ab0fade7854f267aee1c7d80e6dae2b058dd78ffaca6a74084566 exercised the retained native369c7c67 fixture. Verified64MiB backup/new-volume restore, exact binary/checkpoint recovery, repeat restore returns the same consumer-recorded volume, cross-customer restore refusal and implicit application-consistency refusal. Both native volumes deleted; fixture SIGTERM exit0 and scratch absent. Crash-consistent only; no guest boot/customer release, application-consistent backup, credential reset, customer export or image rollback claim. This is consumer execution evidence, not independent native code-review approval. Canonical native review remains pending; reported possible frozen fallback-policy issue to Gitmoot/Jarvis without duplicating dispatch.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Agent: g7-review
Runtime: omp
Job: local-review-review-router-18e3a7aceb50245a-1

Decision: approved
Summary: Reviewed all 11 changed files at exact head 369c7c6 against base b3247af. No actionable correctness findings. Examined transfer reservations, attachment/deletion exclusion, restore capacity accounting, envelope validation, cancellation cleanup, durable publication, claim-gated recovery, and authenticated HTTP handling. Build, vet, affected-package tests, focused race checks, and an independent HTTP smoke check passed. The frozen prior-verdict file contains no entries for PR #81, so there are no prior findings to answer. Real Firecracker/KVM integration tests remained opt-in and skipped; no live-service actions or repository edits were performed.

Tests Run

  • {"command":"/usr/local/go/bin/go version","outcome":"PASS: go1.26.4 linux/amd64. The initial PATH-based invocation could not find go; subsequent checks used the installed toolchain explicitly."}
  • {"command":"/usr/local/go/bin/go build ./...","outcome":"PASS."}
  • {"command":"/usr/local/go/bin/go vet ./...","outcome":"PASS."}
  • {"command":"/usr/local/go/bin/go test -count=1 -v ./internal/vm ./internal/control","outcome":"PASS: vm 17.578s; control 8.872s. Exercised native backup/restore round-trip, attachment and deletion exclusion, truncated/corrupt/extra payload rejection, cancellation cleanup, quota/name reservations, envelope bounds, crash recovery, and claim exclusivity. Opt-in KVM cases skipped."}
  • {"command":"/usr/local/go/bin/go test -race -count=1 -run 'TestFirecracker(ColdBackupRoundTrip|RestoreFailureCleanup|RestoreReservesQuotaAndName|VolumeRecoveryRequiresClaim|VolumeCrashLeftovers|VolumeStartupRepairs)$|TestVolumeBackupEnvelopeBounds$' ./internal/vm","outcome":"PASS: 2.252s; no race reports."}
  • {"command":"/usr/local/go/bin/go test -overlay=/proc/6121/fd/10 -count=1 -run '^TestReviewHTTPTransfers$' -v ./internal/control","outcome":"PASS: 0.181s. Independent memory-only overlay exercised the production HTTP handler over a local HTTP connection with a test storage provider: verified a complete 64 MiB backup and digest, chunked restore returning 201 with fresh identity/token, authentication rejection, invalid body/query/source rejection, and transfer saturation returning 429. No repository files were created or modified; overlay descriptors were closed afterward."}

Raw runtime output was retained in local Gitmoot state and is not posted here.

@jerryfane

Copy link
Copy Markdown
Contributor Author

Independent exact-head review APPROVED: g7-review job local-review-review-router-18e3a7aceb50245a-1 at369c7c67d836794e1bdcd28dae4f5f0d9a635c5f; evidence=executed, findings=0. Implementer was sandboxd seat, not reviewer. Reviewer reports Go1.26.4 build ./..., vet ./..., uncached verbose internal/vm and internal/control tests, focused storage/claim race tests, and an independent memory-overlay HTTP smoke against the production handler all PASS. Smoke covered complete64MiB backup/digest, chunked fresh-ID restore201, auth/input rejection and transfer saturation429. Opt-in live KVM tests skipped; no live-service actions or repository edits. These are reviewer executions, not reruns by the implementing seat. Remote backend and effective OMP runtime recorded; selected model is Astra, actual fallback provider is not separately established by this verdict payload. Current exact head unchanged; L3 classifier rerun and both CI jobs passing. Existing assignment authorizes this seat (merge=self) to merge after the gate; no shared deployment or Forge product acceptance is implied.

@jerryfane
jerryfane merged commit d3fb0a9 into main Oct 31, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Controller-only cold volume backup and restore into a new volume

1 participant