Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Volumes are off by default. They stay off on the Apple (Mac) driver and on a gat

- `POST /volumes` `{"name"}` creates one: `201` `{"volumeID","name","token"}`. A name is 1–63 letters, digits, `_` or `-`, starting with a letter or digit (else `400`); a taken name is `409`; a volume that would take the sum of all volume sizes over `-fc-volume-budget-mib` is `400` (`volume budget exhausted`). Unknown body fields are `400`.
- `GET /volumes` lists `[{"volumeID","name"}]` by name; `GET /volumes/{id}` (the SDKs' `Volume.connect`, `get_info`) answers `{"volumeID","name","token"}` or `404`. The `token` is an HMAC of the volume ID under the token secret, and grants nothing: the volume content API it would authorize is refused.
- `POST /volumes/{id}/resize` `{"sizeMiB": N}` is a sandboxd operator route outside the E2B API (control API key): on the Firecracker driver it grows a detached volume to N MiB and answers `200 {"volumeID","name","sizeMiB"}`; it never shrinks (`400`), and an attached or busy volume is `409` (see [firecracker.md](firecracker.md#growing-a-detached-volume)).
- `DELETE /volumes/{id}` is `204` and deletes the data; an unknown ID is `404`, and a volume attached to a sandbox is `409` (kill the sandbox first).
- `/volumecontent/…` (the SDKs' volume file methods: `list`, `read_file`, `write_file`, …) answers `501` `the volume content API is not supported: mount the volume in a sandbox (volumeMounts) and use its files there`.
- `POST /v2/sandboxes` with `"volumeMounts":[{"name","path"}]` mounts the named volume at `path` (absolute, clean, not `/`; else `400`), owned by `user`. At most one mount per sandbox (`400` otherwise); absent, `null` or `[]` mounts nothing. An unknown name is `404`; a volume attached to another sandbox is `409` `volume "<name>" is attached to sandbox <id>`, refused before any capacity is reserved (two creates racing for one volume: the driver admits one, the other is `409` too). A volume is attached to at most one sandbox at a time, until that sandbox ends.
Expand Down
39 changes: 39 additions & 0 deletions docs/firecracker.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,45 @@ lost there; the volume's journal keeps it consistent. A volume
survives its VM's end, a daemon restart or `kill -9` and that destroy;
only `DELETE /volumes/{id}` removes it.

### Growing a detached volume

`POST /volumes/{id}/resize` with `{"sizeMiB": N}` (control API key, like the
backup route; sandbox envd tokens cannot reach it) grows a **detached** volume
in place and answers `200 {"volumeID", "name", "sizeMiB"}`. It never shrinks:
a smaller size is `400`, the same size is a no-op `200`. An attached,
backing-up, restoring or already-growing volume is `409`; an unknown ID `404`;
a grow past `-fc-volume-budget-mib` or below the disk floor (the growth, not
the whole size, counts) is `400`. The new size must be between 64 MiB and
1 TiB. Drivers without the capability answer `501`. Use it to raise an
existing volume after `-fc-volume-mib` was raised, which only affects new
volumes.

Steps, with the volume held (no attach, delete or backup) throughout: check
budget and floor, take an exclusive `flock` on the disk file, extend it, `e2fsck -f -p`, `resize2fs`, then
replace the record with one atomic rename. The host uses its own trusted
`e2fsck` and `resize2fs` (root-owned regular files under `/usr/sbin` or
`/sbin`, found when a grow runs; `e2fsprogs` ships them with `mke2fs`). Nothing is mounted. If `e2fsck` cannot
repair the filesystem safely (exit 2 or higher, e.g. `4`) the file goes back to its
old size and the volume is unchanged. If `resize2fs` fails, or the daemon
stops mid-grow, the disk file is larger than its record until the same
request is repeated: the steps are repeatable, and a disk file larger than the
requested size is refused rather than cut. Until then the volume can still
be attached (the guest sees a larger device holding the old filesystem) but a
backup of it is refused. Budget accounting counts a volume at the larger of
its record and its disk file, so an unfinished grow is never free space.

`e2fsck` and `resize2fs` do not die with the daemon. They inherit the locked
disk, so the `flock` outlives a killed daemon until the last tool has exited;
every disk user (attach, delete, backup, grow) checks it and answers busy
(`409`) meanwhile. A restarted daemon therefore cannot attach a disk a tool
is still writing.

Trust: `e2fsck` and `resize2fs` run as root on a filesystem the guest wrote,
which is the one place the host parses guest-controlled metadata (`mke2fs`
only writes a fresh one). They take an image file, not a mount, and run with
the daemon's minimal environment. An operator who does not accept that parsing
should not call the route.

### Controller-only cold backup and new-volume restore

These operations use the control `X-API-Key`, not a guest token, and are available
Expand Down
50 changes: 49 additions & 1 deletion internal/control/volumes.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,17 @@ type newVolumeRequest struct {
Name string `json:"name"`
}

// resizeVolumeRequest asks to grow a detached volume to SizeMiB.
type resizeVolumeRequest struct {
SizeMiB int `json:"sizeMiB"`
}

// resizedVolume is the grow response: the volume with its new size.
type resizedVolume struct {
volumeEntry
SizeMiB int `json:"sizeMiB"`
}

// volumeToken is the token the SDKs require with a volume. It grants
// nothing: the volume content API it would authorize is refused, and a
// volume's data is reached only from a sandbox it is mounted in.
Expand All @@ -47,7 +58,7 @@ func (s *Service) withToken(volume vm.Volume) volumeAndToken {
// serveVolumes answers E2B's volume API when volumes are configured, and
// reports whether path was a volume route.
func (s *Service) serveVolumes(w http.ResponseWriter, r *http.Request) bool {
if s.serveVolumeTransfers(w, r) {
if s.serveVolumeTransfers(w, r) || s.serveVolumeResize(w, r) {
return true
}
path := r.URL.Path
Expand Down Expand Up @@ -96,6 +107,8 @@ func (s *Service) volumeFailure(w http.ResponseWriter, action string, err error)
s.fail(w, ProfileE2B, http.StatusConflict, "volume is attached to a sandbox: kill the sandbox first")
case errors.Is(err, vm.ErrVolumeTransferBusy):
s.fail(w, ProfileE2B, http.StatusConflict, "volume storage operation in progress")
case errors.Is(err, vm.ErrVolumeSize):
s.fail(w, ProfileE2B, http.StatusBadRequest, "invalid volume size: it can only grow, between 64 MiB and 1 TiB")
case errors.Is(err, vm.ErrVolumeBackupInvalid):
s.fail(w, ProfileE2B, http.StatusBadRequest, "invalid volume backup")
case errors.Is(err, vm.ErrVolumeBudget):
Expand Down Expand Up @@ -216,3 +229,38 @@ func (s *Service) volumeMount(ctx context.Context, raw json.RawMessage, hardened
}
return nil, http.StatusNotFound, fmt.Sprintf("volume %q not found", requested.Name)
}

// serveVolumeResize answers POST /volumes/{id}/resize, an operator route
// behind the control API key like backup: it grows a detached volume and never
// shrinks one. Customers, who hold only a sandbox's envd token, cannot reach it.
func (s *Service) serveVolumeResize(w http.ResponseWriter, r *http.Request) bool {
id, ok := strings.CutSuffix(strings.TrimPrefix(r.URL.Path, "/volumes/"), "/resize")
if !ok || !strings.HasPrefix(r.URL.Path, "/volumes/") {
return false
}
if r.Method != http.MethodPost {
s.fail(w, ProfileE2B, http.StatusMethodNotAllowed, "method not allowed")
return true
}
resizer, ok := s.volumes.(vm.VolumeResizer)
if !ok {
s.fail(w, ProfileE2B, http.StatusNotImplemented, "volume grow unavailable")
return true
}
if !vm.ValidVolumeID(id) {
s.fail(w, ProfileE2B, http.StatusNotFound, "volume not found")
return true
}
var request resizeVolumeRequest
if err := decode(w, r, &request); !errors.Is(err, io.EOF) {
s.fail(w, ProfileE2B, http.StatusBadRequest, "invalid volume resize body")
return true
}
volume, err := resizer.ResizeVolume(r.Context(), id, request.SizeMiB)
if err != nil {
s.volumeFailure(w, "resize", err)
return true
}
jsonResponse(w, http.StatusOK, resizedVolume{volumeEntry{volume.ID, volume.Name}, volume.SizeMiB})
return true
}
71 changes: 71 additions & 0 deletions internal/control/volumes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,32 @@ func (f *fakeVolumes) DeleteVolume(_ context.Context, id string) error {
return fmt.Errorf("volume %s: %w", id, vm.ErrVolumeNotFound)
}

// ResizeVolume grows like the driver: only larger, only detached.
func (f *fakeVolumes) ResizeVolume(_ context.Context, id string, sizeMiB int) (vm.Volume, error) {
f.mu.Lock()
defer f.mu.Unlock()
for i := range f.volumes {
volume := &f.volumes[i]
if volume.ID != id {
continue
}
switch {
case volume.AttachedTo != "":
return vm.Volume{}, fmt.Errorf("volume %s: %w", id, vm.ErrVolumeInUse)
case sizeMiB < volume.SizeMiB:
return vm.Volume{}, fmt.Errorf("volume %s: %w", id, vm.ErrVolumeSize)
}
volume.SizeMiB = sizeMiB
return *volume, nil
}
return vm.Volume{}, fmt.Errorf("volume %s: %w", id, vm.ErrVolumeNotFound)
}

// plainVolumes is a store that cannot grow volumes.
type plainVolumes struct{ *fakeVolumes }

func (plainVolumes) ResizeVolume() {}

func (f *fakeVolumes) attach(name, sandbox string) {
f.mu.Lock()
defer f.mu.Unlock()
Expand Down Expand Up @@ -396,3 +422,48 @@ func TestVolumesNeedLocalDriver(t *testing.T) {
t.Fatal("volumes accepted without a local driver")
}
}

func TestVolumeGrowAPI(t *testing.T) {
s, _, volumes := openVolumes(t, 1)
data := createVolume(t, s, "data")
path := "/volumes/" + data.ID + "/resize"

got := request(t, s, http.MethodPost, path, map[string]any{"sizeMiB": 1024})
if want := fmt.Sprintf(`{"volumeID":%q,"name":"data","sizeMiB":1024}`+"\n", data.ID); got.Code != http.StatusOK || got.Body.String() != want {
t.Fatalf("grow: %d %s, want %s", got.Code, got.Body.String(), want)
}
if list, _ := volumes.Volumes(context.Background()); list[0].SizeMiB != 1024 {
t.Fatalf("driver size = %d", list[0].SizeMiB)
}
volumes.attach("data", "sbx-1")
for _, c := range []struct {
label, method, path string
body any
status int
message string
}{
{"attached", http.MethodPost, path, map[string]any{"sizeMiB": 2048}, http.StatusConflict, "attached to a sandbox"},
{"unknown volume", http.MethodPost, "/volumes/vol-ffffffffffffffffffffffffffffffff/resize", map[string]any{"sizeMiB": 2048}, http.StatusNotFound, "volume not found"},
{"malformed id", http.MethodPost, "/volumes/data/resize", map[string]any{"sizeMiB": 2048}, http.StatusNotFound, "volume not found"},
{"wrong method", http.MethodGet, path, nil, http.StatusMethodNotAllowed, "method not allowed"},
{"unknown field", http.MethodPost, path, map[string]any{"sizeMiB": 2048, "force": true}, http.StatusBadRequest, "invalid volume resize body"},
} {
wantFailure(t, c.label, request(t, s, c.method, c.path, c.body), c.status, c.message)
}
volumes.mu.Lock()
volumes.volumes[0].AttachedTo = ""
volumes.mu.Unlock()
wantFailure(t, "shrink", request(t, s, http.MethodPost, path, map[string]any{"sizeMiB": 512}), http.StatusBadRequest, "can only grow")
// A missing size is size 0, which the driver refuses; it is never a no-op.
wantFailure(t, "no body", request(t, s, http.MethodPost, path, nil), http.StatusBadRequest, "can only grow")
}

func TestVolumeGrowNeedsDriverSupport(t *testing.T) {
driver := &specDriver{fakeDriver: &fakeDriver{instances: map[string]vm.Instance{}}}
cfg := mixedConfig(e2bBase, 1)
cfg.Volumes = plainVolumes{&fakeVolumes{}}
s := openAt(t, filepath.Join(t.TempDir(), "ledger.sqlite"), driver, cfg)
created := createVolume(t, s, "data")
wantFailure(t, "unsupported", request(t, s, http.MethodPost, "/volumes/"+created.ID+"/resize", map[string]any{"sizeMiB": 1024}),
http.StatusNotImplemented, "grow unavailable")
}
4 changes: 4 additions & 0 deletions internal/vm/firecracker.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@ type fcHost interface {
// FormatVolume runs the trusted mkfs binary on the volume image file
// path; nothing is mounted on the host.
FormatVolume(ctx context.Context, mkfs, path string) error
// ResizeVolume runs the trusted e2fsck (-f -p) and resize2fs on a
// volume image file already extended to its new size. Nothing is mounted.
// lock is the flocked disk file; the tools inherit it (see volumeLocked).
ResizeVolume(ctx context.Context, e2fsck, resize2fs, path string, lock *os.File) error

ApplyFirewall(ctx context.Context, ruleset string) error
FirewallState(ctx context.Context) (string, error)
Expand Down
38 changes: 26 additions & 12 deletions internal/vm/firecracker_backup.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,14 @@ import (
)

// volumeTransfers is protected by volumeMu. A zero SizeMiB is a backup of an
// existing volume; a positive size reserves a new name and restore capacity.
// existing volume; a positive size reserves a new name and restore capacity;
// GrowTo > 0 is a grow of an existing volume from GrowFrom MiB to GrowTo MiB.
// Reservations are process-local: the daemon's instance/ledger claim excludes
// other writers, and restart removes incomplete .ext4.tmp imports.
type fcVolumeTransfer struct {
Name string
SizeMiB int
Name string
SizeMiB int
GrowFrom, GrowTo int
}

func (d *FirecrackerDriver) reserveVolumeTransfer(id string, transfer fcVolumeTransfer) error {
Expand All @@ -48,18 +50,30 @@ func (d *FirecrackerDriver) releaseVolumeTransfer(id string) {
// reservedVolumeMiB also rejects names reserved by an in-flight restore. Count
// full restore sizes against disk headroom conservatively, even if a stream has
// partially allocated its staging file; a slow import must never overcommit it.
func (d *FirecrackerDriver) reservedVolumeMiB(name string) (int, error) {
reserved := 0
for _, transfer := range d.volumeTransfers {
//
// It returns two figures. budget is what the volume budget must add to the
// volumes' footprints: restore sizes, and the part of a grow the disk file has
// not been extended by yet (the extended part is in the footprint). headroom is
// what the host disk floor must hold back: restore sizes and each grow's whole
// growth until it finishes, because extending the file is sparse and the
// allocation only happens while e2fsck and resize2fs run.
func (d *FirecrackerDriver) reservedVolumeMiB(name string) (budget, headroom int, err error) {
for id, transfer := range d.volumeTransfers {
if transfer.GrowTo > 0 {
budget += max(transfer.GrowTo-max(transfer.GrowFrom, d.diskMiB(id)), 0)
headroom += transfer.GrowTo - transfer.GrowFrom
continue
}
if transfer.SizeMiB == 0 {
continue
}
if transfer.Name == name {
return 0, ErrVolumeExists
return 0, 0, ErrVolumeExists
}
reserved += transfer.SizeMiB
budget += transfer.SizeMiB
headroom += transfer.SizeMiB
}
return reserved, nil
return budget, headroom, nil
}

func (d *FirecrackerDriver) OpenVolumeBackup(ctx context.Context, id string) (VolumeBackup, error) {
Expand Down Expand Up @@ -274,7 +288,7 @@ func (d *FirecrackerDriver) reserveRestore(ctx context.Context, name string, siz
if len(d.volumeTransfers) >= MaxVolumeTransfers {
return fcVolume{}, ErrVolumeTransferBusy
}
reserved, err := d.reservedVolumeMiB(name)
reserved, headroom, err := d.reservedVolumeMiB(name)
if err != nil {
return fcVolume{}, err
}
Expand All @@ -287,7 +301,7 @@ func (d *FirecrackerDriver) reserveRestore(ctx context.Context, name string, siz
if volume.Name == name {
return fcVolume{}, ErrVolumeExists
}
used += volume.SizeMiB
used += d.volumeFootprintMiB(volume)
}
if used+size > d.cfg.VolumeBudgetMiB {
return fcVolume{}, ErrVolumeBudget
Expand All @@ -296,7 +310,7 @@ func (d *FirecrackerDriver) reserveRestore(ctx context.Context, name string, siz
if err != nil {
return fcVolume{}, err
}
if free < uint64(d.cfg.DiskFloorMiB+size+reserved)<<20 {
if free < uint64(d.cfg.DiskFloorMiB+size+headroom)<<20 {
return fcVolume{}, ErrVolumeBudget
}
var random [16]byte
Expand Down
28 changes: 28 additions & 0 deletions internal/vm/firecracker_host.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,21 @@ func newLinuxFCHost(names fcNames) *linuxFCHost {
}

func (h *linuxFCHost) run(ctx context.Context, stdin []byte, name string, args ...string) ([]byte, error) {
return h.runInheriting(ctx, nil, stdin, name, args...)
}

// runInheriting runs a tool that also inherits keep as its fd 3. A flock on
// keep is held by the open file description, which the child shares: it stays
// locked until the tool exits, even if the daemon dies first.
func (h *linuxFCHost) runInheriting(ctx context.Context, keep *os.File, stdin []byte, name string, args ...string) ([]byte, error) {
if h.toolsErr != nil {
return nil, h.toolsErr
}
c := exec.CommandContext(ctx, name, args...)
c.Env = []string{fcHostPath}
if keep != nil {
c.ExtraFiles = []*os.File{keep}
}
if stdin != nil {
c.Stdin = bytes.NewReader(stdin)
}
Expand Down Expand Up @@ -136,6 +146,24 @@ func (h *linuxFCHost) FormatVolume(ctx context.Context, mkfs, path string) error
return err
}

// ResizeVolume checks the volume's filesystem, then grows it to fill the
// image file. e2fsck exit 1 means it repaired errors, which -p only does when
// that is safe; anything else stops the grow before resize2fs, with
// errVolumeFsck so the caller knows the filesystem was not resized. lock is
// the caller's open, flocked file for path; each tool inherits it, so the
// disk stays fenced while a tool runs, even if the daemon dies (the tools do
// not die with it).
func (h *linuxFCHost) ResizeVolume(ctx context.Context, e2fsck, resize2fs, path string, lock *os.File) error {
if _, err := h.runInheriting(ctx, lock, nil, e2fsck, "-f", "-p", path); err != nil {
var exit *exec.ExitError
if !errors.As(err, &exit) || exit.ExitCode() != 1 {
return fmt.Errorf("%w: %w", errVolumeFsck, err)
}
}
_, err := h.runInheriting(ctx, lock, nil, resize2fs, path)
return err
}

// MonitorRuleset runs `nft monitor` in the daemon's (host) namespace. Every
// line it prints is one committed ruleset change; the content does not
// matter, since the caller re-reads the table itself. The subprocess dies
Expand Down
22 changes: 22 additions & 0 deletions internal/vm/firecracker_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,11 @@ type fakeFCHost struct {
// FormatVolume target.
owners map[string]int
formatted []string
// resized records each ResizeVolume call; resizeGate blocks it until
// closed and resizeErr is what it returns.
resized []string
resizeGate chan struct{}
resizeErr error
// tables is each namespace's current table: the one it was created with,
// or the last it was replaced with;
// failReplace fails the next replacements, and readback, when set,
Expand Down Expand Up @@ -154,6 +159,23 @@ func (h *fakeFCHost) FormatVolume(_ context.Context, mkfs, path string) error {
h.formatted = append(h.formatted, mkfs+" "+path)
return nil
}

// ResizeVolume stands in for e2fsck and resize2fs: it records the disk size it
// was handed, optionally blocks on resizeGate, and fails with resizeErr.
func (h *fakeFCHost) ResizeVolume(_ context.Context, e2fsck, resize2fs, path string, _ *os.File) error {
info, err := os.Stat(path)
if err != nil {
return err
}
h.mu.Lock()
h.resized = append(h.resized, fmt.Sprintf("%s %s %s %d", e2fsck, resize2fs, path, info.Size()))
gate, fail := h.resizeGate, h.resizeErr
h.mu.Unlock()
if gate != nil {
<-gate
}
return fail
}
func (h *fakeFCHost) ApplyFirewall(_ context.Context, ruleset string) error {
h.mu.Lock()
defer h.mu.Unlock()
Expand Down
Loading
Loading