Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/firecracker.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,76 @@ lost there; the volume's journal keeps it consistent. A volume
survives its VM's end, a daemon restart or `kill -9` and that destroy;
only `DELETE /volumes/{id}` removes it.

### Controller-only cold backup and new-volume restore

These operations use the control `X-API-Key`, not a guest token, and are available
only on a local Firecracker worker with volumes enabled. **Raw ext4 backups are
credential-bearing**, including possible deleted blocks. They are not sanitized
customer exports. Store artifacts privately with trusted provenance; SHA-256
detects corruption, not an attacker replacing both manifest and payload.

`POST /volumes/{volumeID}/backup` takes no body/query and returns
`application/vnd.sandboxd.volume-backup`. A detached check and per-volume
reservation atomically exclude guest attachment, deletion and another backup
(409) until hashing and streaming finish or are cancelled. Unrelated volumes
can progress; the driver does not hold its global mutex across bulk I/O.

The binary envelope is:

| Bytes | Content |
|---|---|
| 8 | ASCII `SBXVOL01` |
| 4 | Big-endian unsigned JSON manifest length, 1–4096 |
| manifest length | UTF-8 JSON fields below |
| exactly `sizeBytes` | Raw ext4 data; no trailing bytes |

Manifest fields: `sourceVolumeID`, integer `sizeBytes`, lowercase hex `sha256`
of the disk payload, and `applicationConsistency:"not-asserted"`. Disk size must
be a whole MiB, from 64 MiB through 1 TiB. The server hashes the reserved stable
disk before emitting the manifest. A receiver must verify exact payload length,
source ID and digest before atomically publishing its immutable artifact.
HTTP 200 headers alone do not establish a completed backup.

`POST /volumes/restore?name=<new-name>&sourceVolumeID=<expected-origin-ID>`
accepts that media type and envelope. The explicitly expected source must match
the manifest. It returns 201 with the usual volume identity/token response only
after publishing a complete, synced disk under a **fresh random ID**. Existing
IDs/names are never overwritten; a duplicate/reserved name returns 409.
Unknown versions, malformed metadata, truncated/extra data and checksum mismatch
return 400. An unsupported native backup driver returns 501.

At most two transfers per daemon run concurrently; each has a ten-minute total
deadline including hashing and transfer. Additional HTTP transfers return 429.
Native operation conflicts return 409. Imports reserve name and full logical
capacity before accepting disk bytes; existing budget and disk-floor checks
apply (400 on insufficient capacity). Disk headroom conservatively reserves the
full size of concurrent imports even when their staging files are partly written.
Transfer buffers are fixed-size; no disk-sized in-memory archive is built.

Restore writes root-only staging, validates all bytes, syncs the file, then
publishes record-before-disk without replacing an existing file. Errors/cancel
remove staging; restart removes incomplete import leftovers only after obtaining
the exclusive instance claim. A competing daemon constructor cannot erase an
active restore before its conflicting claim is rejected. A crash after complete
durable publication but before
the response may leave a valid new volume: reconcile by its requested name
instead of retrying under another name. No untrusted filesystem is mounted on
the host.

Detached does **not** mean application-quiesced: the guest flush above is
best-effort. Forge's integration policy stops admission/customer access and
customer services, verifies their exit, and requires a successful explicit sync;
failed/unknown quiescence aborts a normal backup. Forge records that separate
receipt, not a native application-consistency guarantee.

Forge restores into quarantine: no internet, customer route or automatic
service replay; the original workspace stays stopped/revoked during cutover.
Capabilities, client enrollment and guest SSH host identity must be rotated.
Retained service/subscription credentials are cleared through their supported
interfaces and deliberately re-enrolled before release. Deleting local tokens
does not prove provider-side revocation. Customer file export remains a separate
explicit allowlist policy; this raw API makes no secret-free-content promise.

## Memory return (`-fc-balloon`)

A Firecracker VM keeps every page its guest has ever touched: after a
Expand Down
6 changes: 5 additions & 1 deletion internal/control/control.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,8 @@ type Service struct {
// volumes is Config.Volumes: nil when volumes are off.
volumes vm.Volumes
// egress is Config.Egress: nil when egress policies are off.
egress vm.EgressPolicies
egress vm.EgressPolicies
volumeTransferSlots chan struct{}
}

// Open creates the durable ledger before accepting any VM allocations. The
Expand Down Expand Up @@ -204,6 +205,7 @@ func Open(ctx context.Context, path string, driver vm.Driver, cfg Config) (*Serv
s := &Service{ledger: ledger, cfg: cfg, templates: templates, tokenKey: slices.Clone(cfg.TokenSecret), workers: members, byID: make(map[string]*member, len(members)),
busy: make(map[string]chan struct{}), apiHash: sha256.Sum256([]byte(cfg.APIKey)), stop: make(chan struct{}), done: make(chan struct{}),
signers: make(map[string]string), volumes: cfg.Volumes, egress: cfg.Egress}
s.volumeTransferSlots = make(chan struct{}, vm.MaxVolumeTransfers)
rows, err := ledger.Active(ctx)
if err != nil {
_ = ledger.Close()
Expand Down Expand Up @@ -779,6 +781,8 @@ func (s *Service) launch(ctx context.Context, row *store.Row, place placement) e
// The driver alone knows every VM trace holding a volume; the client
// learns why, while the row is held exactly as for any failed start.
switch {
case errors.Is(err, vm.ErrVolumeTransferBusy):
return &refusal{http.StatusConflict, "volume storage operation in progress"}
case errors.Is(err, vm.ErrVolumeInUse):
return &refusal{http.StatusConflict, "volume is attached to another sandbox"}
case errors.Is(err, vm.ErrVolumeNotFound):
Expand Down
111 changes: 111 additions & 0 deletions internal/control/volume_backups.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
package control

import (
"context"
"errors"
"io"
"log"
"mime"
"net/http"
"net/url"
"strings"
"time"

"github.com/gitmoot/sandboxd/internal/vm"
)

// serveVolumeTransfers runs after the normal control API authentication. Raw
// block backups are operator-only credentials, never customer-safe file exports.
func (s *Service) serveVolumeTransfers(w http.ResponseWriter, r *http.Request) bool {
id, backup := strings.CutSuffix(strings.TrimPrefix(r.URL.Path, "/volumes/"), "/backup")
restore := r.URL.Path == "/volumes/restore"
if !restore && (!strings.HasPrefix(r.URL.Path, "/volumes/") || !backup) {
return false
}
if r.Method != http.MethodPost {
s.fail(w, ProfileE2B, http.StatusMethodNotAllowed, "method not allowed")
return true
}
native, ok := s.volumes.(vm.VolumeBackups)
if !ok {
s.fail(w, ProfileE2B, http.StatusNotImplemented, "native volume backups unavailable")
return true
}
if !restore && (!vm.ValidVolumeID(id) || r.ContentLength != 0 || len(r.TransferEncoding) != 0 || r.URL.RawQuery != "") {
s.fail(w, ProfileE2B, http.StatusBadRequest, "backup requires a volume ID and no body or query")
return true
}
select {
case s.volumeTransferSlots <- struct{}{}:
defer func() { <-s.volumeTransferSlots }()
default:
s.fail(w, ProfileE2B, http.StatusTooManyRequests, "too many volume transfers")
return true
}
deadline := time.Now().Add(vm.VolumeTransferTimeout)
ctx, cancel := context.WithDeadline(r.Context(), deadline)
defer cancel()
controller := http.NewResponseController(w)
if err := controller.SetReadDeadline(deadline); err != nil && !errors.Is(err, http.ErrNotSupported) {
s.fail(w, ProfileE2B, 503, "transfer deadlines unavailable")
return true
}
if err := controller.SetWriteDeadline(deadline); err != nil && !errors.Is(err, http.ErrNotSupported) {
s.fail(w, ProfileE2B, 503, "transfer deadlines unavailable")
return true
}
stopped := make(chan struct{})
stop := context.AfterFunc(ctx, func() {
_ = controller.SetReadDeadline(time.Now())
_ = controller.SetWriteDeadline(time.Now())
close(stopped)
})
defer func() {
if !stop() {
<-stopped
}
_ = controller.SetReadDeadline(time.Time{})
_ = controller.SetWriteDeadline(time.Time{})
}()
if restore {
contentType, params, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
query, queryErr := url.ParseQuery(r.URL.RawQuery)
if err != nil || queryErr != nil || contentType != vm.VolumeBackupMediaType || len(params) != 0 || len(query) != 2 || len(query["name"]) != 1 || !vm.ValidVolumeName(query.Get("name")) || len(query["sourceVolumeID"]) != 1 || !vm.ValidVolumeID(query.Get("sourceVolumeID")) {
s.fail(w, ProfileE2B, http.StatusBadRequest, "restore requires backup media type, new name and expected sourceVolumeID")
return true
}
manifest, err := vm.ReadVolumeManifest(r.Body)
if err != nil || manifest.SourceVolumeID != query.Get("sourceVolumeID") {
s.fail(w, ProfileE2B, http.StatusBadRequest, "invalid volume backup manifest")
return true
}
volume, err := native.RestoreVolume(ctx, query.Get("name"), manifest, r.Body)
if err != nil {
s.volumeFailure(w, "restore", err)
return true
}
jsonResponse(w, http.StatusCreated, s.withToken(volume))
return true
}
snapshot, err := native.OpenVolumeBackup(ctx, id)
if err != nil {
s.volumeFailure(w, "backup", err)
return true
}
defer snapshot.Data.Close()
w.Header().Set("Content-Type", vm.VolumeBackupMediaType)
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("X-Content-Type-Options", "nosniff")
if err := vm.WriteVolumeManifest(w, snapshot.Manifest); err != nil {
log.Printf("volume backup manifest transfer failed: %v", err)
panic(http.ErrAbortHandler)
}
// Disk size was checked before headers; copy exactly that many bytes. A
// failed stream aborts the response rather than appending an error to a backup.
count, err := io.CopyBuffer(w, io.LimitReader(snapshot.Data, snapshot.Manifest.SizeBytes), make([]byte, 64<<10))
if err != nil || count != snapshot.Manifest.SizeBytes || ctx.Err() != nil {
log.Printf("volume backup transfer incomplete: bytes=%d error=%v", count, err)
panic(http.ErrAbortHandler)
}
return true
}
7 changes: 7 additions & 0 deletions internal/control/volumes.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ func (s *Service) withToken(volume vm.Volume) volumeAndToken {
// serveVolumes answers E2B's volume API when volumes are configured, and
// reports whether path was a volume route.
func (s *Service) serveVolumes(w http.ResponseWriter, r *http.Request) bool {
if s.serveVolumeTransfers(w, r) {
return true
}
path := r.URL.Path
switch {
case path == "/volumecontent" || strings.HasPrefix(path, "/volumecontent/"):
Expand Down Expand Up @@ -91,6 +94,10 @@ func (s *Service) volumeFailure(w http.ResponseWriter, action string, err error)
s.fail(w, ProfileE2B, http.StatusConflict, "a volume with that name already exists")
case errors.Is(err, vm.ErrVolumeInUse):
s.fail(w, ProfileE2B, http.StatusConflict, "volume is attached to a sandbox: kill the sandbox first")
case errors.Is(err, vm.ErrVolumeTransferBusy):
s.fail(w, ProfileE2B, http.StatusConflict, "volume storage operation in progress")
case errors.Is(err, vm.ErrVolumeBackupInvalid):
s.fail(w, ProfileE2B, http.StatusBadRequest, "invalid volume backup")
case errors.Is(err, vm.ErrVolumeBudget):
// 400 is the only client error the SDKs declare for create.
s.fail(w, ProfileE2B, http.StatusBadRequest, "volume budget exhausted: delete a volume first")
Expand Down
3 changes: 2 additions & 1 deletion internal/vm/firecracker.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,8 @@ type FirecrackerDriver struct {

// volumeMu orders every volume create, delete, attach and release, so
// an attach's in-use check and its record write are one step.
volumeMu sync.Mutex
volumeMu sync.Mutex
volumeTransfers map[string]fcVolumeTransfer
// mkfs is the trusted host mke2fs, set when volumes are on.
mkfs string

Expand Down
Loading
Loading