From 498416618de73f7ccceb2cfefef7a67e8c00bd2b Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 20:17:55 +0100 Subject: [PATCH 1/8] docs(openspec): propose scheduled start and stop for remote environments --- .../.openspec.yaml | 2 + .../scheduled-remote-start-stop/design.md | 45 +++++++ .../scheduled-remote-start-stop/proposal.md | 29 +++++ .../specs/remote-schedule/spec.md | 110 ++++++++++++++++++ .../scheduled-remote-start-stop/tasks.md | 37 ++++++ 5 files changed, 223 insertions(+) create mode 100644 openspec/changes/scheduled-remote-start-stop/.openspec.yaml create mode 100644 openspec/changes/scheduled-remote-start-stop/design.md create mode 100644 openspec/changes/scheduled-remote-start-stop/proposal.md create mode 100644 openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md create mode 100644 openspec/changes/scheduled-remote-start-stop/tasks.md diff --git a/openspec/changes/scheduled-remote-start-stop/.openspec.yaml b/openspec/changes/scheduled-remote-start-stop/.openspec.yaml new file mode 100644 index 00000000..e3966d7a --- /dev/null +++ b/openspec/changes/scheduled-remote-start-stop/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-05 diff --git a/openspec/changes/scheduled-remote-start-stop/design.md b/openspec/changes/scheduled-remote-start-stop/design.md new file mode 100644 index 00000000..9d90b060 --- /dev/null +++ b/openspec/changes/scheduled-remote-start-stop/design.md @@ -0,0 +1,45 @@ +## Context + +See proposal.md for why. The control plane (`remote/`) already runs one shared set of Lambdas for every environment, finds an environment's resources by name, and exposes each Lambda through a SigV4-signed Function URL. Environment state lives in SSM parameters under `/cloud-vm-llm//` and in tags on the instance. The stop Lambda is already invoked on a five-minute EventBridge rule, and tells that call from a Function URL call by the event's `source`. `spinloop remote keep` is the nearest existing per-environment setting and follows the same client → Function URL → Lambda route. + +## Goals / Non-Goals + +**Goals:** +- Schedules that fire with no user machine running. +- Cron expressions with a time zone, correct across daylight saving. +- One CLI to set, show and clear them, and status that shows what runs next. +- Reuse the existing start and stop code, so a scheduled run cannot behave differently from a manual one. + +**Non-Goals:** +- Running schedules from the CLI or a local daemon. +- Local (non-cloud) nodes. Only remote environments get schedules. +- Adding a single schedule or removing a single one by id. The list is replaced whole. +- Scheduled terminate. A scheduled stop pauses. + +## Decisions + +**EventBridge Scheduler, not an EventBridge rule per schedule.** EventBridge Scheduler takes a cron expression with an IANA time zone and handles daylight saving. Classic rules are UTC only. The cost is that schedules are created at runtime, so the stack creates one schedule group (`spinloop-remote`) and one role that Scheduler assumes to invoke the start and stop Lambdas. + +**A schedule Lambda owns the schedules.** A new Lambda with a Function URL takes `PUT` (replace the list), `GET` (read it) and `DELETE` (clear), for the environment named by `?env=`, like the other Lambdas. It is the only code with `scheduler:*` permission, limited to the group. This keeps the CLI user's policy to "invoke this URL", as for every other command. + +**The list is stored in SSM, the schedules are derived from it.** The list is kept as JSON in `/cloud-vm-llm//schedules`. A replace writes the parameter, then makes the Scheduler schedules in the group named `--` match it: create or update those in the list, delete the rest of that environment's prefix. If the Scheduler calls fail part way, the parameter already holds the intended list and a repeat of the same request converges. Validation (cron, zone, action, env name) runs before anything is written. + +**Cron form.** The CLI and API take the usual five-field cron. Scheduler's own cron has six fields (with year) and a `?` rule for day fields, so the Lambda converts: it adds `*` for the year and replaces the day-of-month or day-of-week `*` with `?` when the other is set. The conversion is one pure function with table tests. Expressions that cannot be converted (for example both day fields set) are rejected, naming the expression. + +**Targets and payload.** Each schedule's target is the start or stop Lambda with a fixed JSON input `{"source":"spinloop.schedule","action":"start|stop","environment":""}`. The start Lambda and stop Lambda each gain a branch for that source, ahead of the Function URL branch. Scheduler invokes Lambda asynchronously, so the 15-minute start is not cut short. The scheduled start calls the same start function as the URL path; the scheduled stop calls the same pause function, after reading the instance's `Retain-Until` tag. + +**Next run for status.** The status path in the schedule Lambda reads the schedules and computes the next firing of each action with a cron library in the Lambda, in the schedule's zone. Status is not read from Scheduler, so it is the same whether or not Scheduler has caught up. + +**Client.** `internal/remote` gains `SetSchedules`, `GetSchedules`, `ClearSchedules` calling a new `schedule_url` from the config, and `Status` gains the next-run fields. A missing `schedule_url` returns the "re-run bootstrap" error. The CLI group is `remote schedule {set,show,clear}`, built the way `keep` is. + +## Risks / Trade-offs + +- Scheduler creates schedules a few seconds after the call returns; a `set` followed at once by a firing minute is not guaranteed to be seen. Accepted. +- A scheduled start for an environment whose weights are not seeded starts a seed (the existing behaviour) and then ends without a ready instance. The log says so; the next firing retries. +- A stop skipped for `Retain-Until` is not retried; the idle sweep ends the instance later as usual. +- Existing deployments need a bootstrap re-run for the new Lambda, URL and role. +- `remote/` is a public repo: the group name and prefixes are fixed strings, nothing account-specific is committed. + +## Open Questions + +None that change what gets built. Assumption recorded: scheduled stop pauses rather than terminates, and skips when `Retain-Until` is in the future. diff --git a/openspec/changes/scheduled-remote-start-stop/proposal.md b/openspec/changes/scheduled-remote-start-stop/proposal.md new file mode 100644 index 00000000..bc136ff2 --- /dev/null +++ b/openspec/changes/scheduled-remote-start-stop/proposal.md @@ -0,0 +1,29 @@ +## Why + +A remote environment only starts when someone asks for it and is stopped by the idle sweep. A team that works office hours has to start it by hand each morning and pay for the idle time before the sweep stops it each evening. Issue 178 asks for cloud nodes to start and stop on one or more cron schedules. + +## What Changes + +- Each remote environment can hold a list of schedules. A schedule is a cron expression, a time zone and an action: `start` or `stop`. +- The schedules are stored and run in the AWS control plane (`remote/`), so they fire with no machine of the user's switched on. EventBridge Scheduler runs them. +- A new schedule Lambda accepts a replacement list of schedules for one environment, returns the current list, and clears it. +- A scheduled start runs the same start as `spinloop remote start`. A scheduled stop pauses the instance (stops it without terminating it), unless the instance has a `Retain-Until` deadline in the future, in which case the stop is skipped. +- New CLI commands: `spinloop remote schedule set`, `show` and `clear`. +- `spinloop remote status` reports the next scheduled start and stop when the environment has schedules. + +## Capabilities + +### New Capabilities + +- `remote-schedule`: storing, running and reporting the cron schedules that start and stop a remote environment. + +### Modified Capabilities + +None. The existing start, stop and keep behaviour is reused unchanged; a scheduled run calls the same code paths. + +## Impact + +- `remote/`: new schedule Lambda and Function URL, an EventBridge Scheduler schedule group and a role the schedules assume, new handling in the start and stop Lambdas for a scheduled invocation, new stack output and `SpinloopRemoteConfig` field (`schedule_url`). +- `internal/remote` and `cmd/spinloop/remote.go`: the client calls, the `schedule` command group, and the status lines. +- `docs/`: command reference, remote guide and the control plane's HTTP API. +- The remote CLI user's IAM policy gains permission to invoke the schedule Lambda. Existing deployments need `spinloop remote bootstrap` re-run to get the feature; the CLI reports a clear error when `schedule_url` is missing. diff --git a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md new file mode 100644 index 00000000..06dce638 --- /dev/null +++ b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md @@ -0,0 +1,110 @@ +## ADDED Requirements + +### Requirement: An environment holds a list of schedules + +Each environment SHALL hold zero or more schedules. A schedule SHALL have an action (`start` or `stop`), a cron expression in the five-field form (minute, hour, day of month, month, day of week) and an IANA time zone. When no time zone is given it SHALL be UTC. Setting schedules for an environment SHALL replace its whole list, so the list after the call is exactly the one sent. An invalid cron expression, an unknown time zone, an unknown action or an invalid environment name SHALL be rejected, and a rejected request SHALL leave the existing schedules unchanged. The schedules SHALL be stored in the control plane, not on any user's machine, so they fire whether or not any user machine is on. + +#### Scenario: Setting office-hours schedules + +- **WHEN** schedules are set for an environment with a `start` at `0 8 * * 1-5` and a `stop` at `0 18 * * 1-5`, in `Europe/London` +- **THEN** the environment's list holds those two schedules and no others + +#### Scenario: Setting replaces the previous list + +- **WHEN** an environment has two schedules and a set request is made with one different schedule +- **THEN** the environment has only that one schedule afterwards, and the two earlier ones no longer fire + +#### Scenario: An invalid expression is rejected + +- **WHEN** a set request contains a cron expression that is not five valid fields +- **THEN** the request is refused, the error names the expression, and the environment's earlier schedules still fire + +#### Scenario: Schedules are per environment + +- **WHEN** schedules are set for one environment +- **THEN** no other environment's schedules change and no other environment's instance is started or stopped + +### Requirement: A schedule fires in its own time zone + +A schedule SHALL fire when its cron expression matches in its time zone, including across daylight saving changes: a schedule at 08:00 in `Europe/London` fires at 08:00 local time on both sides of a clock change. + +#### Scenario: A schedule follows local time + +- **WHEN** a `start` schedule is set at `0 8 * * *` in `Europe/London` +- **THEN** it fires at 07:00 UTC in summer and 08:00 UTC in winter + +### Requirement: A scheduled start brings the environment up + +When a `start` schedule fires, the control plane SHALL start the environment's instance with the same behaviour as an on-demand start, including the weights check and the wait until the model is answering. When the instance is already running, the scheduled start SHALL leave it as it is. When the start cannot find capacity or the weights are absent, the run SHALL record why in the start Lambda's log and SHALL NOT retry until the next firing. + +#### Scenario: A stopped environment is started on schedule + +- **WHEN** a `start` schedule fires and the environment has no running instance +- **THEN** the instance is launched or re-woken, as an on-demand start would do + +#### Scenario: A running environment is left alone + +- **WHEN** a `start` schedule fires and the instance is already running +- **THEN** nothing is launched and the instance keeps running + +### Requirement: A scheduled stop pauses the environment + +When a `stop` schedule fires, the control plane SHALL pause the environment's instance: stop it without terminating it, as `spinloop remote pause` does, so it can be woken again. When the instance carries a `Retain-Until` deadline that has not passed, the scheduled stop SHALL be skipped and the skip recorded in the stop Lambda's log. When there is no running instance, the scheduled stop SHALL do nothing. + +#### Scenario: A running environment is paused on schedule + +- **WHEN** a `stop` schedule fires and the instance is running with no retention deadline +- **THEN** the instance is stopped and not terminated + +#### Scenario: A retained instance is not stopped + +- **WHEN** a `stop` schedule fires and the instance has a `Retain-Until` time in the future +- **THEN** the instance keeps running + +#### Scenario: Nothing to stop + +- **WHEN** a `stop` schedule fires and the environment has no running instance +- **THEN** the run does nothing and reports no error + +### Requirement: The schedule command sets, shows and clears schedules + +`spinloop remote schedule set` SHALL accept one or more `--start CRON` and `--stop CRON` flags and an optional `--timezone ZONE`, and SHALL replace the environment's schedules with them. `spinloop remote schedule show` SHALL print the environment's schedules, one per line, giving the action, expression and time zone, and SHALL print that there are none when the list is empty. `spinloop remote schedule clear` SHALL remove every schedule. The commands SHALL select the environment as the other `remote` subcommands do: `--env ` or the per-user default. `set` with neither `--start` nor `--stop` SHALL fail and say to use `clear` to remove schedules. When the deployment's control plane has no schedule endpoint, the commands SHALL fail with an error naming the fix (re-run `spinloop remote bootstrap`). + +#### Scenario: Setting two schedules from the command line + +- **WHEN** the user runs `spinloop remote schedule set --start "0 8 * * 1-5" --stop "0 18 * * 1-5" --timezone Europe/London` +- **THEN** the environment has those two schedules and the command prints them + +#### Scenario: Showing no schedules + +- **WHEN** the user runs `spinloop remote schedule show` for an environment with none +- **THEN** the output says there are no schedules and the command succeeds + +#### Scenario: Clearing + +- **WHEN** the user runs `spinloop remote schedule clear` +- **THEN** the environment has no schedules and none fires afterwards + +#### Scenario: Set with nothing to set + +- **WHEN** the user runs `spinloop remote schedule set` with no `--start` or `--stop` +- **THEN** the command fails and tells the user to use `clear` to remove schedules + +#### Scenario: An older control plane + +- **WHEN** the deployment's configuration has no schedule endpoint +- **THEN** the command fails with an error telling the user to re-run `spinloop remote bootstrap` + +### Requirement: Status reports the next scheduled runs + +When the environment has schedules, `spinloop remote status` SHALL report the next time a start fires and the next time a stop fires, as absolute times. When it has none, the status SHALL omit those lines. + +#### Scenario: A scheduled environment shows its next runs + +- **WHEN** the user runs `spinloop remote status` for an environment with a start and a stop schedule +- **THEN** the output includes "next start" and "next stop" lines with absolute times + +#### Scenario: An unscheduled environment omits the lines + +- **WHEN** the user runs `spinloop remote status` for an environment with no schedules +- **THEN** the output has no "next start" or "next stop" line diff --git a/openspec/changes/scheduled-remote-start-stop/tasks.md b/openspec/changes/scheduled-remote-start-stop/tasks.md new file mode 100644 index 00000000..c43c4a90 --- /dev/null +++ b/openspec/changes/scheduled-remote-start-stop/tasks.md @@ -0,0 +1,37 @@ +## 1. Control plane: schedule model + +- [ ] 1.1 Add `remote/lambda/shared/schedules.ts`: types, validation (action, zone, env name), five-field cron → Scheduler cron conversion, next-run calculation +- [ ] 1.2 Table tests for conversion and validation, including day-field `?` rules, rejected expressions, and daylight saving next-run + +## 2. Control plane: schedule Lambda + +- [ ] 2.1 Add `remote/lambda/schedule/index.ts`: `PUT` replaces, `GET` reads (with next runs), `DELETE` clears; stores the list in `/cloud-vm-llm//schedules` +- [ ] 2.2 Reconcile the Scheduler schedules in the group to the stored list (create, update, delete by environment prefix) +- [ ] 2.3 Tests with mocked SSM and Scheduler clients: replace, clear, invalid input leaves state unchanged, other environments untouched + +## 3. Control plane: start and stop + +- [ ] 3.1 Start Lambda: handle a `spinloop.schedule` event for `start`, reusing the on-demand start +- [ ] 3.2 Stop Lambda: handle a `spinloop.schedule` event for `stop`, pausing, skipping when `Retain-Until` is in the future +- [ ] 3.3 Tests for the running, stopped, retained and no-instance cases + +## 4. Stack + +- [ ] 4.1 Add the schedule Lambda, its Function URL, the Scheduler group, and the role Scheduler assumes to invoke start and stop +- [ ] 4.2 Grant the schedule Lambda SSM read/write on the schedules parameter and Scheduler permissions limited to the group, plus `iam:PassRole` for the Scheduler role +- [ ] 4.3 Add the stack output and the `schedule_url` field in `SpinloopRemoteConfig`, and let the remote CLI user invoke the URL +- [ ] 4.4 Stack test (assertions on the template) and run `scripts/check-no-cloud-identifiers.sh` + +## 5. CLI client + +- [ ] 5.1 Add `schedule_url` to the remote config, loading and bootstrap output parsing +- [ ] 5.2 Add `SetSchedules`, `GetSchedules`, `ClearSchedules` in `internal/remote`, with the "re-run bootstrap" error when the URL is missing +- [ ] 5.3 Add next start / next stop to the status response and the `remote status` output +- [ ] 5.4 Add `spinloop remote schedule set|show|clear` with `--start`, `--stop`, `--timezone`, `--env`, plus tab completion +- [ ] 5.5 Unit tests for the client calls, the command's argument handling and the status lines + +## 6. Documentation and specs + +- [ ] 6.1 Update `docs/commands`, the remote guide and the control-plane HTTP API page +- [ ] 6.2 Add a note to `docs/maintainer/internals.md` on the cron conversion and the Scheduler/SSM split +- [ ] 6.3 Run `go test ./...`, `go vet ./...`, `gofmt`, and `pnpm test` in `remote/` From ef0c287bb68d009d0204970a9f5216dbea9d2f4c Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 20:28:01 +0100 Subject: [PATCH 2/8] feat(remote): start and stop environments on cron schedules A schedule Lambda keeps each environment's schedule list in SSM and mirrors it into EventBridge Scheduler, which runs the start and stop Lambdas in each schedule's own time zone. A scheduled stop pauses the instance and is skipped while a retention deadline is in force. Adds spinloop remote schedule set, show and clear. --- cmd/spinloop/commands.go | 1 + cmd/spinloop/remote_schedule.go | 207 +++++++++++ cmd/spinloop/remote_schedule_test.go | 204 +++++++++++ cmd/spinloop/viper_test.go | 15 +- docs/env-vars.md | 1 + internal/remote/aws.go | 17 +- internal/remote/aws_test.go | 2 + internal/remote/remote.go | 10 +- internal/remote/schedule.go | 101 ++++++ internal/remote/schedule_test.go | 144 ++++++++ .../scheduled-remote-start-stop/design.md | 4 +- .../scheduled-remote-start-stop/proposal.md | 4 +- .../specs/remote-schedule/spec.md | 15 +- .../scheduled-remote-start-stop/tasks.md | 34 +- remote/lambda/schedule/index.ts | 182 ++++++++++ remote/lambda/shared/schedules.ts | 322 ++++++++++++++++++ remote/lambda/start/index.ts | 29 +- remote/lambda/stop/index.ts | 50 ++- remote/lib/llm-stack.ts | 64 +++- remote/package.json | 1 + remote/pnpm-lock.yaml | 253 ++++++++++++++ remote/test/schedule-api.test.ts | 191 +++++++++++ remote/test/schedule-start.test.ts | 141 ++++++++ remote/test/schedule-stop.test.ts | 115 +++++++ remote/test/schedules.test.ts | 154 +++++++++ remote/test/stack.test.ts | 65 +++- 26 files changed, 2268 insertions(+), 58 deletions(-) create mode 100644 cmd/spinloop/remote_schedule.go create mode 100644 cmd/spinloop/remote_schedule_test.go create mode 100644 internal/remote/schedule.go create mode 100644 internal/remote/schedule_test.go create mode 100644 remote/lambda/schedule/index.ts create mode 100644 remote/lambda/shared/schedules.ts create mode 100644 remote/test/schedule-api.test.ts create mode 100644 remote/test/schedule-start.test.ts create mode 100644 remote/test/schedule-stop.test.ts create mode 100644 remote/test/schedules.test.ts diff --git a/cmd/spinloop/commands.go b/cmd/spinloop/commands.go index b2de2ebb..0d52673f 100644 --- a/cmd/spinloop/commands.go +++ b/cmd/spinloop/commands.go @@ -414,6 +414,7 @@ names a file — falling back to the default environment. Each subcommand's remoteEnvCmd(), remoteListCmd(), remoteKeepCmd(), + remoteScheduleCmd(), ) return remote } diff --git a/cmd/spinloop/remote_schedule.go b/cmd/spinloop/remote_schedule.go new file mode 100644 index 00000000..e0037f91 --- /dev/null +++ b/cmd/spinloop/remote_schedule.go @@ -0,0 +1,207 @@ +package main + +import ( + "context" + "fmt" + "io" + "os" + "time" + + "github.com/spf13/cobra" + "github.com/spinloop-ai/spinloop/internal/remote" +) + +// remoteScheduleCmd is the schedule subcommand parent. A schedule is a cron +// expression that starts or stops one environment; the control plane stores +// the list and runs it, so schedules fire with no machine of the operator's +// switched on. +func remoteScheduleCmd() *cobra.Command { + schedule := &cobra.Command{ + Use: "schedule", + Short: "start and stop an environment on a cron schedule", + Long: `starts and stops an environment on cron schedules that the control plane runs, so +they fire whether or not any of your machines is on. Each schedule is a +five-field cron expression (minute hour day-of-month month day-of-week) in a +time zone, with the action start or stop. A scheduled start does what +spinloop remote start does; a scheduled stop pauses the instance, and is +skipped while the instance is kept (spinloop remote keep).`, + SilenceErrors: true, + SilenceUsage: true, + RunE: groupFallback, + } + schedule.AddCommand( + remoteScheduleSetCmd(), + remoteScheduleShowCmd(), + remoteScheduleClearCmd(), + ) + return schedule +} + +func remoteScheduleSetCmd() *cobra.Command { + var ( + envName string + starts []string + stops []string + timezone string + ) + c := &cobra.Command{ + Use: "set", + Short: "replace the environment's schedules", + Long: `replaces the environment's whole list of schedules with the ones given. Repeat +--start and --stop for more than one. The time zone is an IANA name such as +Europe/London and applies to every schedule given; it defaults to UTC. + + spinloop remote schedule set --start "0 8 * * 1-5" --stop "0 18 * * 1-5" \ + --timezone Europe/London`, + Args: cobra.NoArgs, + SilenceErrors: true, + SilenceUsage: true, + RunE: func(c *cobra.Command, _ []string) error { + resolve(c) + return runRemoteScheduleSet(envName, starts, stops, timezone) + }, + } + fs := c.Flags() + fs.StringVar(&envName, "env", "", envFlagUsage) + fs.StringArrayVar(&starts, "start", nil, "cron expression that starts the environment (repeatable)") + fs.StringArrayVar(&stops, "stop", nil, "cron expression that stops the environment (repeatable)") + fs.StringVar(&timezone, "timezone", "", "IANA time zone for the schedules (default UTC)") + compRegister(c, "env", compEnvs) + c.ValidArgsFunction = noPositionals + return c +} + +func remoteScheduleShowCmd() *cobra.Command { + var envName string + c := &cobra.Command{ + Use: "show", + Short: "list the environment's schedules and their next runs", + Args: cobra.NoArgs, + SilenceErrors: true, + SilenceUsage: true, + RunE: func(c *cobra.Command, _ []string) error { + resolve(c) + return runRemoteScheduleShow(envName) + }, + } + c.Flags().StringVar(&envName, "env", "", envFlagUsage) + compRegister(c, "env", compEnvs) + c.ValidArgsFunction = noPositionals + return c +} + +func remoteScheduleClearCmd() *cobra.Command { + var envName string + c := &cobra.Command{ + Use: "clear", + Short: "remove every schedule of the environment", + Args: cobra.NoArgs, + SilenceErrors: true, + SilenceUsage: true, + RunE: func(c *cobra.Command, _ []string) error { + resolve(c) + return runRemoteScheduleClear(envName) + }, + } + c.Flags().StringVar(&envName, "env", "", envFlagUsage) + compRegister(c, "env", compEnvs) + c.ValidArgsFunction = noPositionals + return c +} + +// scheduleListFor builds the list a set sends: every --start, then every +// --stop, each in the one time zone. +func scheduleListFor(starts, stops []string, timezone string) []remote.Schedule { + list := make([]remote.Schedule, 0, len(starts)+len(stops)) + for _, expr := range starts { + list = append(list, remote.Schedule{Action: "start", Cron: expr, Timezone: timezone}) + } + for _, expr := range stops { + list = append(list, remote.Schedule{Action: "stop", Cron: expr, Timezone: timezone}) + } + return list +} + +// runRemoteScheduleSet is the body of `spinloop remote schedule set`. +func runRemoteScheduleSet(envName string, starts, stops []string, timezone string) error { + if len(starts) == 0 && len(stops) == 0 { + return fmt.Errorf("nothing to set: pass --start and/or --stop (use `spinloop remote schedule clear` to remove schedules)") + } + cfg, err := resolveRemoteConfig(envName, "") + if err != nil { + return err + } + list, err := remote.SetSchedules(context.Background(), cfg, scheduleListFor(starts, stops, timezone)) + if err != nil { + return err + } + printSchedules(os.Stdout, list) + return nil +} + +// runRemoteScheduleShow is the body of `spinloop remote schedule show`. +func runRemoteScheduleShow(envName string) error { + cfg, err := resolveRemoteConfig(envName, "") + if err != nil { + return err + } + list, err := remote.GetSchedules(context.Background(), cfg) + if err != nil { + return err + } + printSchedules(os.Stdout, list) + return nil +} + +// runRemoteScheduleClear is the body of `spinloop remote schedule clear`. +func runRemoteScheduleClear(envName string) error { + cfg, err := resolveRemoteConfig(envName, "") + if err != nil { + return err + } + list, err := remote.ClearSchedules(context.Background(), cfg) + if err != nil { + return err + } + printSchedules(os.Stdout, list) + return nil +} + +// printSchedules writes one line per schedule — action, expression, zone — +// then the next time a start and a stop fire, when there is one. An empty list +// says so. +func printSchedules(w io.Writer, list *remote.ScheduleList) { + if len(list.Schedules) == 0 { + fmt.Fprintln(w, "no schedules") + return + } + for _, s := range list.Schedules { + zone := s.Timezone + if zone == "" { + zone = "UTC" + } + fmt.Fprintf(w, "%-5s %s (%s)\n", s.Action, s.Cron, zone) + } + if next := formatNextRun(list.Next.Start); next != "" { + fmt.Fprintf(w, "next start: %s\n", next) + } + if next := formatNextRun(list.Next.Stop); next != "" { + fmt.Fprintf(w, "next stop: %s\n", next) + } +} + +// formatNextRun renders a next-run time as an RFC 3339 instant in UTC, or "" +// for none. A value that does not parse is returned as sent, so a control +// plane reply in another form is shown rather than hidden. +func formatNextRun(raw string) string { + if raw == "" { + return "" + } + t, err := time.Parse(time.RFC3339, raw) + if err != nil { + return raw + } + return t.UTC().Format(time.RFC3339) +} + +func cmdRemoteSchedule(args []string) error { return execCmd(remoteScheduleCmd(), args) } diff --git a/cmd/spinloop/remote_schedule_test.go b/cmd/spinloop/remote_schedule_test.go new file mode 100644 index 00000000..482f7fa6 --- /dev/null +++ b/cmd/spinloop/remote_schedule_test.go @@ -0,0 +1,204 @@ +package main + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + + "github.com/spinloop-ai/spinloop/internal/remote" +) + +// scheduleFixture serves the schedule Lambda's reply and registers a default +// environment whose schedule URL points at it. It returns what the last +// request carried. +type scheduleRequest struct { + method string + env string + body string +} + +func scheduleFixture(t *testing.T, status int, reply string) *scheduleRequest { + t.Helper() + isolateConfig(t) + stubAWSEnv(t) + got := &scheduleRequest{} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + got.method, got.env, got.body = r.Method, r.URL.Query().Get("env"), string(b) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _, _ = w.Write([]byte(reply)) + })) + t.Cleanup(server.Close) + writeRemoteConfig(t, server.URL) + path := must1(remote.EnvConfigPath("default")) + var cfg remote.Config + if err := json.Unmarshal(must1(os.ReadFile(path)), &cfg); err != nil { + t.Fatal(err) + } + cfg.ScheduleURL = server.URL + cfg.Environment = "default" + if err := os.WriteFile(path, must1(json.Marshal(cfg)), 0o600); err != nil { + t.Fatal(err) + } + return got +} + +const officeReply = `{"environment":"default","schedules":[` + + `{"action":"start","cron":"0 8 * * 1-5","timezone":"Europe/London"},` + + `{"action":"stop","cron":"0 18 * * 1-5","timezone":"Europe/London"}],` + + `"next":{"start":"2026-10-06T07:00:00.000Z","stop":"2026-10-05T17:00:00.000Z"}}` + +func TestRemoteSchedule_SetSendsTheListAndPrintsIt(t *testing.T) { + got := scheduleFixture(t, http.StatusOK, officeReply) + + out := captureStdout(t, func() { + err := cmdRemoteSchedule([]string{ + "set", "--env", "default", + "--start", "0 8 * * 1-5", "--stop", "0 18 * * 1-5", "--timezone", "Europe/London", + }) + if err != nil { + t.Errorf("set: %v", err) + } + }) + + if got.method != http.MethodPut || got.env != "default" { + t.Errorf("unexpected request: %+v", got) + } + var sent struct { + Schedules []remote.Schedule `json:"schedules"` + } + if err := json.Unmarshal([]byte(got.body), &sent); err != nil { + t.Fatal(err) + } + want := []remote.Schedule{ + {Action: "start", Cron: "0 8 * * 1-5", Timezone: "Europe/London"}, + {Action: "stop", Cron: "0 18 * * 1-5", Timezone: "Europe/London"}, + } + if len(sent.Schedules) != 2 || sent.Schedules[0] != want[0] || sent.Schedules[1] != want[1] { + t.Errorf("sent %+v, want %+v", sent.Schedules, want) + } + for _, line := range []string{ + "start 0 8 * * 1-5 (Europe/London)", + "stop 0 18 * * 1-5 (Europe/London)", + "next start: 2026-10-06T07:00:00Z", + "next stop: 2026-10-05T17:00:00Z", + } { + if !strings.Contains(out, line) { + t.Errorf("output missing %q:\n%s", line, out) + } + } +} + +func TestRemoteSchedule_SetAcceptsRepeatedFlags(t *testing.T) { + got := scheduleFixture(t, http.StatusOK, officeReply) + captureStdout(t, func() { + err := cmdRemoteSchedule([]string{ + "set", "--env", "default", "--start", "0 8 * * 1-5", "--start", "0 13 * * 6", + }) + if err != nil { + t.Errorf("set: %v", err) + } + }) + if strings.Count(got.body, `"action":"start"`) != 2 { + t.Errorf("expected two start schedules, got %s", got.body) + } +} + +func TestRemoteSchedule_SetWithNothingPointsAtClear(t *testing.T) { + got := scheduleFixture(t, http.StatusOK, officeReply) + err := cmdRemoteSchedule([]string{"set", "--env", "default"}) + if err == nil || !strings.Contains(err.Error(), "schedule clear") { + t.Errorf("expected an error naming clear, got %v", err) + } + if got.method != "" { + t.Errorf("nothing should be sent, got a %s", got.method) + } +} + +func TestRemoteSchedule_ShowListsSchedules(t *testing.T) { + got := scheduleFixture(t, http.StatusOK, officeReply) + out := captureStdout(t, func() { + if err := cmdRemoteSchedule([]string{"show", "--env", "default"}); err != nil { + t.Errorf("show: %v", err) + } + }) + if got.method != http.MethodGet { + t.Errorf("show should GET, got %s", got.method) + } + if !strings.Contains(out, "next start:") || !strings.Contains(out, "next stop:") { + t.Errorf("show should report the next runs:\n%s", out) + } +} + +func TestRemoteSchedule_ShowSaysWhenThereAreNone(t *testing.T) { + scheduleFixture(t, http.StatusOK, `{"environment":"default","schedules":[],"next":{"start":"","stop":""}}`) + out := captureStdout(t, func() { + if err := cmdRemoteSchedule([]string{"show", "--env", "default"}); err != nil { + t.Errorf("show: %v", err) + } + }) + if strings.TrimSpace(out) != "no schedules" { + t.Errorf("expected %q, got %q", "no schedules", out) + } + if strings.Contains(out, "next start") || strings.Contains(out, "next stop") { + t.Errorf("no next-run lines expected:\n%s", out) + } +} + +func TestRemoteSchedule_OmitsANextRunThatDoesNotExist(t *testing.T) { + scheduleFixture(t, http.StatusOK, `{"environment":"default","schedules":[{"action":"start","cron":"0 8 * * *","timezone":"UTC"}],"next":{"start":"2026-10-06T08:00:00.000Z","stop":""}}`) + out := captureStdout(t, func() { + if err := cmdRemoteSchedule([]string{"show", "--env", "default"}); err != nil { + t.Errorf("show: %v", err) + } + }) + if !strings.Contains(out, "next start:") || strings.Contains(out, "next stop") { + t.Errorf("only the start should have a next run:\n%s", out) + } +} + +func TestRemoteSchedule_ClearDeletes(t *testing.T) { + got := scheduleFixture(t, http.StatusOK, `{"environment":"default","schedules":[],"next":{}}`) + out := captureStdout(t, func() { + if err := cmdRemoteSchedule([]string{"clear", "--env", "default"}); err != nil { + t.Errorf("clear: %v", err) + } + }) + if got.method != http.MethodDelete { + t.Errorf("clear should DELETE, got %s", got.method) + } + if !strings.Contains(out, "no schedules") { + t.Errorf("clear should say there are none:\n%s", out) + } +} + +func TestRemoteSchedule_ControlPlaneRejectionIsTheError(t *testing.T) { + scheduleFixture(t, http.StatusBadRequest, `{"error":"invalid cron expression \"x\": expected five fields"}`) + err := cmdRemoteSchedule([]string{"set", "--env", "default", "--start", "x"}) + if err == nil || !strings.Contains(err.Error(), "expected five fields") { + t.Errorf("expected the control plane's reason, got %v", err) + } +} + +func TestRemoteSchedule_OlderControlPlaneNamesTheFix(t *testing.T) { + isolateConfig(t) + stubAWSEnv(t) + writeRemoteConfig(t, "https://start.example/") + err := cmdRemoteSchedule([]string{"show", "--env", "default"}) + if err == nil || !strings.Contains(err.Error(), "spinloop remote bootstrap") { + t.Errorf("expected an error naming bootstrap, got %v", err) + } +} + +func TestRemoteSchedule_NeedsAnEnvironment(t *testing.T) { + isolateConfig(t) + err := cmdRemoteSchedule([]string{"show"}) + if err == nil || !strings.Contains(err.Error(), "--env") { + t.Errorf("expected the no-environment error, got %v", err) + } +} diff --git a/cmd/spinloop/viper_test.go b/cmd/spinloop/viper_test.go index 2d67fbc5..d9efd7ad 100644 --- a/cmd/spinloop/viper_test.go +++ b/cmd/spinloop/viper_test.go @@ -94,13 +94,14 @@ func TestViperRemoteEnvPrecedence(t *testing.T) { const envValue = "https://env.example/wins" legs := map[string]func(remote.Config) string{ - "SPINLOOP_REMOTE_START_URL": func(c remote.Config) string { return c.StartURL }, - "SPINLOOP_REMOTE_STOP_URL": func(c remote.Config) string { return c.StopURL }, - "SPINLOOP_REMOTE_DEPLOY_URL": func(c remote.Config) string { return c.DeployURL }, - "SPINLOOP_REMOTE_STATS_URL": func(c remote.Config) string { return c.StatsURL }, - "SPINLOOP_REMOTE_ENV_URL": func(c remote.Config) string { return c.EnvURL }, - "SPINLOOP_REMOTE_UPDATE_URL": func(c remote.Config) string { return c.UpdateURL }, - "SPINLOOP_REMOTE_REGION": func(c remote.Config) string { return c.Region }, + "SPINLOOP_REMOTE_START_URL": func(c remote.Config) string { return c.StartURL }, + "SPINLOOP_REMOTE_STOP_URL": func(c remote.Config) string { return c.StopURL }, + "SPINLOOP_REMOTE_DEPLOY_URL": func(c remote.Config) string { return c.DeployURL }, + "SPINLOOP_REMOTE_STATS_URL": func(c remote.Config) string { return c.StatsURL }, + "SPINLOOP_REMOTE_ENV_URL": func(c remote.Config) string { return c.EnvURL }, + "SPINLOOP_REMOTE_UPDATE_URL": func(c remote.Config) string { return c.UpdateURL }, + "SPINLOOP_REMOTE_SCHEDULE_URL": func(c remote.Config) string { return c.ScheduleURL }, + "SPINLOOP_REMOTE_REGION": func(c remote.Config) string { return c.Region }, } // Unset variables fall through to the file. diff --git a/docs/env-vars.md b/docs/env-vars.md index 7cfdd4ce..dec25b28 100644 --- a/docs/env-vars.md +++ b/docs/env-vars.md @@ -30,6 +30,7 @@ from the environment or a `.env` beside the Spinloop — never written into an | `SPINLOOP_REMOTE_STATS_URL` | Override the stats Lambda Function URL. | | `SPINLOOP_REMOTE_ENV_URL` | Override the env Lambda Function URL. | | `SPINLOOP_REMOTE_UPDATE_URL` | Override the update Lambda Function URL (drives `keep`). | +| `SPINLOOP_REMOTE_SCHEDULE_URL` | Override the schedule Lambda Function URL (drives `schedule`). | | `SPINLOOP_REMOTE_REGION` | Override the AWS region (else `AWS_REGION`, else the region in the Function URL host). | | `SPINLOOP_REMOTE_PACKAGE_MANAGER` | Pin the package manager (`pnpm`/`npm`) `spinloop remote bootstrap` and `bake` use. | diff --git a/internal/remote/aws.go b/internal/remote/aws.go index 94103629..d0e2bb72 100644 --- a/internal/remote/aws.go +++ b/internal/remote/aws.go @@ -209,14 +209,15 @@ func DiscoverControlPlane(ctx context.Context, cfg aws.Config, stackName string) func controlPlaneFromOutputs(stackName string, outputs map[string]string) (ControlPlane, error) { layer := ControlPlane{ Config: Config{ - StartURL: outputs["StartUrl"], - StopURL: outputs["StopUrl"], - DeployURL: outputs["DeployUrl"], - StatsURL: outputs["StatsUrl"], - EnvURL: outputs["EnvUrl"], - SeedURL: outputs["SeedUrl"], - UpdateURL: outputs["UpdateUrl"], - Region: outputs["Region"], + StartURL: outputs["StartUrl"], + StopURL: outputs["StopUrl"], + DeployURL: outputs["DeployUrl"], + StatsURL: outputs["StatsUrl"], + EnvURL: outputs["EnvUrl"], + SeedURL: outputs["SeedUrl"], + UpdateURL: outputs["UpdateUrl"], + ScheduleURL: outputs["ScheduleUrl"], + Region: outputs["Region"], }, WeightsBucket: outputs["WeightsBucket"], } diff --git a/internal/remote/aws_test.go b/internal/remote/aws_test.go index 65dc9574..f171af43 100644 --- a/internal/remote/aws_test.go +++ b/internal/remote/aws_test.go @@ -23,6 +23,7 @@ func TestControlPlaneFromOutputs_MapsEveryStackOutput(t *testing.T) { "StatsUrl": "https://stats.example.aws/", "EnvUrl": "https://env.example.aws/", "UpdateUrl": "https://update.example.aws/", + "ScheduleUrl": "https://schedule.example.aws/", "Region": "eu-west-1", "WeightsBucket": "weights-bucket", "VpcId": "vpc-123", // not part of the config @@ -45,6 +46,7 @@ func TestControlPlaneFromOutputs_MapsEveryStackOutput(t *testing.T) { {"StatsURL", layer.Config.StatsURL, outputs["StatsUrl"]}, {"EnvURL", layer.Config.EnvURL, outputs["EnvUrl"]}, {"UpdateURL", layer.Config.UpdateURL, outputs["UpdateUrl"]}, + {"ScheduleURL", layer.Config.ScheduleURL, outputs["ScheduleUrl"]}, {"Region", layer.Config.Region, outputs["Region"]}, {"WeightsBucket", layer.WeightsBucket, outputs["WeightsBucket"]}, } diff --git a/internal/remote/remote.go b/internal/remote/remote.go index 93dc9a49..f5a7cebe 100644 --- a/internal/remote/remote.go +++ b/internal/remote/remote.go @@ -54,7 +54,12 @@ type Config struct { // (currently: set-keep). Optional — configs predating the update Lambda // still work for start/stop/deploy; Keep will fail with a clear message. UpdateURL string `json:"update_url"` - Region string `json:"region"` + // ScheduleURL is the Lambda that sets, reads and clears an environment's + // start/stop schedules. Optional in the same way as UpdateURL: a config + // written before it existed keeps working for every other subcommand, and + // the schedule subcommands fail naming the fix. + ScheduleURL string `json:"schedule_url"` + Region string `json:"region"` // BaseURL is the endpoint's own address (the environment's stable Elastic // IP). It belongs to the deployment rather than to the Spinloop, so it is // written here and `apply` reads it back for a Spinloop that states no @@ -125,6 +130,9 @@ func finishConfig(cfg Config, getenv func(string) string, source string) (Config if v := getenv("SPINLOOP_REMOTE_UPDATE_URL"); v != "" { cfg.UpdateURL = v } + if v := getenv("SPINLOOP_REMOTE_SCHEDULE_URL"); v != "" { + cfg.ScheduleURL = v + } if v := getenv("SPINLOOP_REMOTE_REGION"); v != "" { cfg.Region = v } diff --git a/internal/remote/schedule.go b/internal/remote/schedule.go new file mode 100644 index 00000000..5075c5d7 --- /dev/null +++ b/internal/remote/schedule.go @@ -0,0 +1,101 @@ +package remote + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" +) + +// Schedule is one cron-driven start or stop of an environment, as the +// schedule Lambda stores it. +type Schedule struct { + // Action is "start" or "stop". + Action string `json:"action"` + // Cron is a five-field expression: minute, hour, day of month, month, day of week. + Cron string `json:"cron"` + // Timezone is an IANA zone name; the control plane reads an empty one as UTC. + Timezone string `json:"timezone"` +} + +// ScheduleList is the schedule Lambda's reply: the environment's schedules +// and when each action next fires (RFC 3339, empty when there is none). +type ScheduleList struct { + Environment string `json:"environment"` + Schedules []Schedule `json:"schedules"` + Next struct { + Start string `json:"start"` + Stop string `json:"stop"` + } `json:"next"` +} + +// SetSchedules replaces the environment's schedules with the list given. The +// control plane validates every expression and zone before it changes +// anything, so a rejected list leaves the earlier schedules in place. +func SetSchedules(ctx context.Context, cfg Config, schedules []Schedule) (*ScheduleList, error) { + if schedules == nil { + schedules = []Schedule{} + } + body, err := json.Marshal(map[string][]Schedule{"schedules": schedules}) + if err != nil { + return nil, err + } + return scheduleCall(ctx, cfg, http.MethodPut, body) +} + +// GetSchedules returns the environment's schedules and their next runs. +func GetSchedules(ctx context.Context, cfg Config) (*ScheduleList, error) { + return scheduleCall(ctx, cfg, http.MethodGet, nil) +} + +// ClearSchedules removes every schedule of the environment. +func ClearSchedules(ctx context.Context, cfg Config) (*ScheduleList, error) { + return scheduleCall(ctx, cfg, http.MethodDelete, nil) +} + +// scheduleCall sends one request to the schedule Lambda. It has its own reply +// shape, so it uses send rather than call, and adds the environment itself. +func scheduleCall(ctx context.Context, cfg Config, method string, body []byte) (*ScheduleList, error) { + if cfg.ScheduleURL == "" { + return nil, fmt.Errorf( + "no schedule_url configured: the control plane predates schedules — re-run `spinloop remote bootstrap` (or set SPINLOOP_REMOTE_SCHEDULE_URL)") + } + u, err := url.Parse(cfg.ScheduleURL) + if err != nil { + return nil, err + } + if cfg.Environment != "" { + q := u.Query() + q.Set("env", cfg.Environment) + u.RawQuery = q.Encode() + } + status, respBody, err := send(ctx, cfg, method, u.String(), body) + if err != nil { + return nil, err + } + if status != http.StatusOK { + var reply struct { + Error string `json:"error"` + Message string `json:"message"` + } + _ = json.Unmarshal(respBody, &reply) + detail := reply.Error + if detail == "" { + detail = reply.Message + } + if detail == "" { + detail = string(respBody) + } + hint := "" + if status == http.StatusForbidden { + hint = forbiddenHint(cfg.Region, detail) + } + return nil, fmt.Errorf("schedule returned HTTP %d%s: %s", status, hint, truncate(detail, 200)) + } + out := &ScheduleList{} + if err := json.Unmarshal(respBody, out); err != nil { + return nil, fmt.Errorf("schedule returned an unreadable reply: %s", truncate(string(respBody), 200)) + } + return out, nil +} diff --git a/internal/remote/schedule_test.go b/internal/remote/schedule_test.go new file mode 100644 index 00000000..ca1b3f8c --- /dev/null +++ b/internal/remote/schedule_test.go @@ -0,0 +1,144 @@ +package remote + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// scheduleServer records the request the client sends and replies with reply. +func scheduleServer(t *testing.T, status int, reply string) (*httptest.Server, *http.Request, *[]byte) { + t.Helper() + var got http.Request + var body []byte + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = *r + body, _ = io.ReadAll(r.Body) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _, _ = w.Write([]byte(reply)) + })) + t.Cleanup(server.Close) + return server, &got, &body +} + +func TestSetSchedules(t *testing.T) { + stubAWSEnv(t) + server, got, body := scheduleServer(t, http.StatusOK, + `{"environment":"dev","schedules":[{"action":"start","cron":"0 8 * * 1-5","timezone":"Europe/London"}],"next":{"start":"2026-10-06T07:00:00.000Z","stop":""}}`) + cfg := Config{StartURL: server.URL, StopURL: server.URL, ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + + list, err := SetSchedules(context.Background(), cfg, []Schedule{ + {Action: "start", Cron: "0 8 * * 1-5", Timezone: "Europe/London"}, + }) + if err != nil { + t.Fatal(err) + } + if got.Method != http.MethodPut { + t.Errorf("set should PUT, got %s", got.Method) + } + if got.URL.Query().Get("env") != "dev" { + t.Errorf("set should carry the environment, got %q", got.URL.RawQuery) + } + var sent struct { + Schedules []Schedule `json:"schedules"` + } + if err := json.Unmarshal(*body, &sent); err != nil { + t.Fatal(err) + } + if len(sent.Schedules) != 1 || sent.Schedules[0].Cron != "0 8 * * 1-5" || sent.Schedules[0].Timezone != "Europe/London" { + t.Errorf("unexpected body: %s", *body) + } + if len(list.Schedules) != 1 || list.Next.Start != "2026-10-06T07:00:00.000Z" || list.Next.Stop != "" { + t.Errorf("unexpected reply: %+v", list) + } +} + +func TestSetSchedules_NilSendsAnEmptyList(t *testing.T) { + stubAWSEnv(t) + server, _, body := scheduleServer(t, http.StatusOK, `{"environment":"dev","schedules":[],"next":{}}`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + if _, err := SetSchedules(context.Background(), cfg, nil); err != nil { + t.Fatal(err) + } + if string(*body) != `{"schedules":[]}` { + t.Errorf("a nil list should be sent as an empty one, got %s", *body) + } +} + +func TestGetSchedules(t *testing.T) { + stubAWSEnv(t) + server, got, _ := scheduleServer(t, http.StatusOK, `{"environment":"dev","schedules":[],"next":{"start":"","stop":""}}`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + list, err := GetSchedules(context.Background(), cfg) + if err != nil { + t.Fatal(err) + } + if got.Method != http.MethodGet { + t.Errorf("get should GET, got %s", got.Method) + } + if len(list.Schedules) != 0 { + t.Errorf("expected no schedules, got %+v", list.Schedules) + } +} + +func TestClearSchedules(t *testing.T) { + stubAWSEnv(t) + server, got, _ := scheduleServer(t, http.StatusOK, `{"environment":"dev","schedules":[],"next":{}}`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + if _, err := ClearSchedules(context.Background(), cfg); err != nil { + t.Fatal(err) + } + if got.Method != http.MethodDelete { + t.Errorf("clear should DELETE, got %s", got.Method) + } +} + +func TestSchedules_NoScheduleURL(t *testing.T) { + stubAWSEnv(t) + cfg := Config{StartURL: "https://start/", StopURL: "https://stop/", Region: "eu-west-1"} + for name, call := range map[string]func() error{ + "set": func() error { _, err := SetSchedules(context.Background(), cfg, nil); return err }, + "get": func() error { _, err := GetSchedules(context.Background(), cfg); return err }, + "clear": func() error { _, err := ClearSchedules(context.Background(), cfg); return err }, + } { + err := call() + if err == nil || !strings.Contains(err.Error(), "spinloop remote bootstrap") { + t.Errorf("%s: expected an error naming bootstrap, got %v", name, err) + } + } +} + +func TestSchedules_RejectionCarriesTheReplysDetail(t *testing.T) { + stubAWSEnv(t) + server, _, _ := scheduleServer(t, http.StatusBadRequest, + `{"error":"invalid cron expression \"nope\": expected five fields"}`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + _, err := SetSchedules(context.Background(), cfg, []Schedule{{Action: "start", Cron: "nope"}}) + if err == nil || !strings.Contains(err.Error(), "HTTP 400") || !strings.Contains(err.Error(), "expected five fields") { + t.Errorf("expected the reply's detail, got %v", err) + } +} + +func TestSchedules_ForbiddenSaysWhatToCheck(t *testing.T) { + stubAWSEnv(t) + server, _, _ := scheduleServer(t, http.StatusForbidden, `{"Message":"forbidden"}`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + _, err := GetSchedules(context.Background(), cfg) + if err == nil || !strings.Contains(err.Error(), "lambda:InvokeFunctionUrl") { + t.Errorf("expected the permission hint, got %v", err) + } +} + +func TestSchedules_UnreadableReply(t *testing.T) { + stubAWSEnv(t) + server, _, _ := scheduleServer(t, http.StatusOK, `not json`) + cfg := Config{ScheduleURL: server.URL, Region: "eu-west-1", Environment: "dev"} + if _, err := GetSchedules(context.Background(), cfg); err == nil || !strings.Contains(err.Error(), "unreadable") { + t.Errorf("expected an unreadable-reply error, got %v", err) + } +} diff --git a/openspec/changes/scheduled-remote-start-stop/design.md b/openspec/changes/scheduled-remote-start-stop/design.md index 9d90b060..eead13bc 100644 --- a/openspec/changes/scheduled-remote-start-stop/design.md +++ b/openspec/changes/scheduled-remote-start-stop/design.md @@ -28,9 +28,9 @@ See proposal.md for why. The control plane (`remote/`) already runs one shared s **Targets and payload.** Each schedule's target is the start or stop Lambda with a fixed JSON input `{"source":"spinloop.schedule","action":"start|stop","environment":""}`. The start Lambda and stop Lambda each gain a branch for that source, ahead of the Function URL branch. Scheduler invokes Lambda asynchronously, so the 15-minute start is not cut short. The scheduled start calls the same start function as the URL path; the scheduled stop calls the same pause function, after reading the instance's `Retain-Until` tag. -**Next run for status.** The status path in the schedule Lambda reads the schedules and computes the next firing of each action with a cron library in the Lambda, in the schedule's zone. Status is not read from Scheduler, so it is the same whether or not Scheduler has caught up. +**Next runs.** There is no `remote status` command (`spinloop status` is a fleet-wide table with one row per node), so the next runs are shown by `schedule show` rather than a status line. The schedule Lambda computes the next firing of each action from the stored list, in each schedule's zone, with its own small cron evaluator in `lambda/shared/schedules.ts` (no cron library). The result is not read from Scheduler, so it is the same whether or not Scheduler has caught up. A time the clocks skip is not a firing; a time they repeat fires at the first occurrence. -**Client.** `internal/remote` gains `SetSchedules`, `GetSchedules`, `ClearSchedules` calling a new `schedule_url` from the config, and `Status` gains the next-run fields. A missing `schedule_url` returns the "re-run bootstrap" error. The CLI group is `remote schedule {set,show,clear}`, built the way `keep` is. +**Client.** `internal/remote` gains `SetSchedules`, `GetSchedules`, `ClearSchedules` calling a new `schedule_url` from the config. A missing `schedule_url` returns the "re-run bootstrap" error. The CLI group is `remote schedule {set,show,clear}`, built the way `keep` is. ## Risks / Trade-offs diff --git a/openspec/changes/scheduled-remote-start-stop/proposal.md b/openspec/changes/scheduled-remote-start-stop/proposal.md index bc136ff2..485dd10a 100644 --- a/openspec/changes/scheduled-remote-start-stop/proposal.md +++ b/openspec/changes/scheduled-remote-start-stop/proposal.md @@ -9,7 +9,7 @@ A remote environment only starts when someone asks for it and is stopped by the - A new schedule Lambda accepts a replacement list of schedules for one environment, returns the current list, and clears it. - A scheduled start runs the same start as `spinloop remote start`. A scheduled stop pauses the instance (stops it without terminating it), unless the instance has a `Retain-Until` deadline in the future, in which case the stop is skipped. - New CLI commands: `spinloop remote schedule set`, `show` and `clear`. -- `spinloop remote status` reports the next scheduled start and stop when the environment has schedules. +- `spinloop remote schedule show` (and `set`) report the next scheduled start and stop when the environment has schedules. ## Capabilities @@ -24,6 +24,6 @@ None. The existing start, stop and keep behaviour is reused unchanged; a schedul ## Impact - `remote/`: new schedule Lambda and Function URL, an EventBridge Scheduler schedule group and a role the schedules assume, new handling in the start and stop Lambdas for a scheduled invocation, new stack output and `SpinloopRemoteConfig` field (`schedule_url`). -- `internal/remote` and `cmd/spinloop/remote.go`: the client calls, the `schedule` command group, and the status lines. +- `internal/remote` and `cmd/spinloop/remote.go`: the client calls and the `schedule` command group, which prints the next runs. - `docs/`: command reference, remote guide and the control plane's HTTP API. - The remote CLI user's IAM policy gains permission to invoke the schedule Lambda. Existing deployments need `spinloop remote bootstrap` re-run to get the feature; the CLI reports a clear error when `schedule_url` is missing. diff --git a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md index 06dce638..f3385fac 100644 --- a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md +++ b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md @@ -95,16 +95,21 @@ When a `stop` schedule fires, the control plane SHALL pause the environment's in - **WHEN** the deployment's configuration has no schedule endpoint - **THEN** the command fails with an error telling the user to re-run `spinloop remote bootstrap` -### Requirement: Status reports the next scheduled runs +### Requirement: Show reports the next scheduled runs -When the environment has schedules, `spinloop remote status` SHALL report the next time a start fires and the next time a stop fires, as absolute times. When it has none, the status SHALL omit those lines. +When the environment has schedules, `spinloop remote schedule show` (and `set`, which prints the same listing) SHALL report the next time a start fires and the next time a stop fires, as absolute UTC times, each on its own "next start" or "next stop" line. An action with no schedule, or whose schedule never fires again, SHALL have no line. When the environment has no schedules, the output SHALL say so and have no "next" line. #### Scenario: A scheduled environment shows its next runs -- **WHEN** the user runs `spinloop remote status` for an environment with a start and a stop schedule +- **WHEN** the user runs `spinloop remote schedule show` for an environment with a start and a stop schedule - **THEN** the output includes "next start" and "next stop" lines with absolute times +#### Scenario: An action with no schedule has no line + +- **WHEN** the environment has only a start schedule +- **THEN** the output has a "next start" line and no "next stop" line + #### Scenario: An unscheduled environment omits the lines -- **WHEN** the user runs `spinloop remote status` for an environment with no schedules -- **THEN** the output has no "next start" or "next stop" line +- **WHEN** the user runs `spinloop remote schedule show` for an environment with no schedules +- **THEN** the output says there are no schedules and has no "next start" or "next stop" line diff --git a/openspec/changes/scheduled-remote-start-stop/tasks.md b/openspec/changes/scheduled-remote-start-stop/tasks.md index c43c4a90..65095935 100644 --- a/openspec/changes/scheduled-remote-start-stop/tasks.md +++ b/openspec/changes/scheduled-remote-start-stop/tasks.md @@ -1,34 +1,34 @@ ## 1. Control plane: schedule model -- [ ] 1.1 Add `remote/lambda/shared/schedules.ts`: types, validation (action, zone, env name), five-field cron → Scheduler cron conversion, next-run calculation -- [ ] 1.2 Table tests for conversion and validation, including day-field `?` rules, rejected expressions, and daylight saving next-run +- [x] 1.1 Add `remote/lambda/shared/schedules.ts`: types, validation (action, zone, env name), five-field cron → Scheduler cron conversion, next-run calculation +- [x] 1.2 Table tests for conversion and validation, including day-field `?` rules, rejected expressions, and daylight saving next-run ## 2. Control plane: schedule Lambda -- [ ] 2.1 Add `remote/lambda/schedule/index.ts`: `PUT` replaces, `GET` reads (with next runs), `DELETE` clears; stores the list in `/cloud-vm-llm//schedules` -- [ ] 2.2 Reconcile the Scheduler schedules in the group to the stored list (create, update, delete by environment prefix) -- [ ] 2.3 Tests with mocked SSM and Scheduler clients: replace, clear, invalid input leaves state unchanged, other environments untouched +- [x] 2.1 Add `remote/lambda/schedule/index.ts`: `PUT` replaces, `GET` reads (with next runs), `DELETE` clears; stores the list in `/cloud-vm-llm//schedules` +- [x] 2.2 Reconcile the Scheduler schedules in the group to the stored list (create, update, delete by environment prefix) +- [x] 2.3 Tests with mocked SSM and Scheduler clients: replace, clear, invalid input leaves state unchanged, other environments untouched ## 3. Control plane: start and stop -- [ ] 3.1 Start Lambda: handle a `spinloop.schedule` event for `start`, reusing the on-demand start -- [ ] 3.2 Stop Lambda: handle a `spinloop.schedule` event for `stop`, pausing, skipping when `Retain-Until` is in the future -- [ ] 3.3 Tests for the running, stopped, retained and no-instance cases +- [x] 3.1 Start Lambda: handle a `spinloop.schedule` event for `start`, reusing the on-demand start +- [x] 3.2 Stop Lambda: handle a `spinloop.schedule` event for `stop`, pausing, skipping when `Retain-Until` is in the future +- [x] 3.3 Tests for the running, stopped, retained and no-instance cases ## 4. Stack -- [ ] 4.1 Add the schedule Lambda, its Function URL, the Scheduler group, and the role Scheduler assumes to invoke start and stop -- [ ] 4.2 Grant the schedule Lambda SSM read/write on the schedules parameter and Scheduler permissions limited to the group, plus `iam:PassRole` for the Scheduler role -- [ ] 4.3 Add the stack output and the `schedule_url` field in `SpinloopRemoteConfig`, and let the remote CLI user invoke the URL -- [ ] 4.4 Stack test (assertions on the template) and run `scripts/check-no-cloud-identifiers.sh` +- [x] 4.1 Add the schedule Lambda, its Function URL, the Scheduler group, and the role Scheduler assumes to invoke start and stop +- [x] 4.2 Grant the schedule Lambda SSM read/write on the schedules parameter and Scheduler permissions limited to the group, plus `iam:PassRole` for the Scheduler role +- [x] 4.3 Add the stack output and the `schedule_url` field in `SpinloopRemoteConfig`, and let the remote CLI user invoke the URL +- [x] 4.4 Stack test (assertions on the template) and run `scripts/check-no-cloud-identifiers.sh` ## 5. CLI client -- [ ] 5.1 Add `schedule_url` to the remote config, loading and bootstrap output parsing -- [ ] 5.2 Add `SetSchedules`, `GetSchedules`, `ClearSchedules` in `internal/remote`, with the "re-run bootstrap" error when the URL is missing -- [ ] 5.3 Add next start / next stop to the status response and the `remote status` output -- [ ] 5.4 Add `spinloop remote schedule set|show|clear` with `--start`, `--stop`, `--timezone`, `--env`, plus tab completion -- [ ] 5.5 Unit tests for the client calls, the command's argument handling and the status lines +- [x] 5.1 Add `schedule_url` to the remote config, loading and bootstrap output parsing +- [x] 5.2 Add `SetSchedules`, `GetSchedules`, `ClearSchedules` in `internal/remote`, with the "re-run bootstrap" error when the URL is missing +- [x] 5.3 Print next start / next stop from `schedule show` and `set`, omitting an action with no next run +- [x] 5.4 Add `spinloop remote schedule set|show|clear` with `--start`, `--stop`, `--timezone`, `--env`, plus tab completion +- [x] 5.5 Unit tests for the client calls, the command's argument handling and the status lines ## 6. Documentation and specs diff --git a/remote/lambda/schedule/index.ts b/remote/lambda/schedule/index.ts new file mode 100644 index 00000000..70173244 --- /dev/null +++ b/remote/lambda/schedule/index.ts @@ -0,0 +1,182 @@ +import { + CreateScheduleCommand, + DeleteScheduleCommand, + ListSchedulesCommand, + SchedulerClient, + UpdateScheduleCommand, +} from '@aws-sdk/client-scheduler'; +import { + DeleteParameterCommand, + GetParameterCommand, + PutParameterCommand, + SSMClient, +} from '@aws-sdk/client-ssm'; +import type { LambdaFunctionURLEvent, LambdaFunctionURLResult } from 'aws-lambda'; +import { errorName, requireEnv } from '../shared/aws'; +import { environmentFrom } from '../shared/environments'; +import { jsonResponse } from '../shared/http'; +import { + nextRuns, + SCHEDULE_EVENT_SOURCE, + ScheduleError, + schedulerNamePrefix, + schedulesParam, + toSchedulerCron, + validateSchedules, + type Schedule, + type ScheduledRunEvent, +} from '../shared/schedules'; + +const SCHEDULE_GROUP = requireEnv('SCHEDULE_GROUP'); +const SCHEDULER_ROLE_ARN = requireEnv('SCHEDULER_ROLE_ARN'); +const START_FN_ARN = requireEnv('START_FN_ARN'); +const STOP_FN_ARN = requireEnv('STOP_FN_ARN'); + +const ssm = new SSMClient({}); +const scheduler = new SchedulerClient({}); + +/** + * Function URL for an environment's schedules, named by `?env=`: + * - GET reads the list and when each action next fires. + * - PUT replaces the list with the JSON body `{"schedules": [...]}`. + * - DELETE removes every schedule. + * + * The list is kept in SSM and mirrored into EventBridge Scheduler schedules. + * Input is validated before anything is written, so a rejected request leaves + * the existing schedules as they were. + */ +export async function handler(event: LambdaFunctionURLEvent): Promise { + let env: string; + try { + env = environmentFrom(event.queryStringParameters); + } catch (err) { + return jsonResponse(400, { error: (err as Error).message }); + } + const method = event.requestContext?.http?.method ?? 'GET'; + try { + switch (method) { + case 'GET': + return report(env, await readSchedules(env)); + case 'PUT': + return await replace(env, event); + case 'DELETE': + return await clear(env); + default: + return jsonResponse(405, { error: `unsupported method ${method}; accepted: GET, PUT, DELETE` }); + } + } catch (err) { + if (err instanceof ScheduleError) { + return jsonResponse(400, { error: err.message }); + } + console.log(JSON.stringify({ cmd: 'schedule', method, environment: env, error: errorName(err) })); + throw err; + } +} + +function report(env: string, schedules: Schedule[]): LambdaFunctionURLResult { + return jsonResponse(200, { + environment: env, + schedules, + next: nextRuns(schedules, new Date()), + }); +} + +async function replace(env: string, event: LambdaFunctionURLEvent): Promise { + let body: unknown; + try { + body = JSON.parse(event.isBase64Encoded ? Buffer.from(event.body ?? '', 'base64').toString() : (event.body ?? '')); + } catch { + throw new ScheduleError('the body must be JSON: {"schedules": [...]}'); + } + const schedules = validateSchedules((body as { schedules?: unknown } | null)?.schedules); + // Everything that can be refused is refused above. Past here a failure is the + // AWS side, and the stored list is already the intended one, so repeating the + // request converges. + const expressions = schedules.map((s) => toSchedulerCron(s.cron)); + if (schedules.length === 0) { + return clear(env); + } + await ssm.send( + new PutParameterCommand({ + Name: schedulesParam(env), + Value: JSON.stringify(schedules), + Type: 'String', + Overwrite: true, + }), + ); + await reconcile(env, schedules, expressions); + console.log(JSON.stringify({ cmd: 'schedule-set', environment: env, count: schedules.length })); + return report(env, schedules); +} + +async function clear(env: string): Promise { + try { + await ssm.send(new DeleteParameterCommand({ Name: schedulesParam(env) })); + } catch (err) { + if (errorName(err) !== 'ParameterNotFound') { + throw err; + } + } + await reconcile(env, [], []); + console.log(JSON.stringify({ cmd: 'schedule-clear', environment: env })); + return report(env, []); +} + +async function readSchedules(env: string): Promise { + try { + const out = await ssm.send(new GetParameterCommand({ Name: schedulesParam(env) })); + return validateSchedules(JSON.parse(out.Parameter?.Value ?? '[]')); + } catch (err) { + if (errorName(err) === 'ParameterNotFound') { + return []; + } + throw err; + } +} + +/** Make the environment's Scheduler schedules in the group match the list. */ +async function reconcile(env: string, schedules: Schedule[], expressions: string[]): Promise { + const prefix = schedulerNamePrefix(env); + const existing = new Set(); + let token: string | undefined; + do { + const page = await scheduler.send( + new ListSchedulesCommand({ GroupName: SCHEDULE_GROUP, NamePrefix: prefix, NextToken: token }), + ); + for (const s of page.Schedules ?? []) { + if (s.Name) { + existing.add(s.Name); + } + } + token = page.NextToken; + } while (token); + + const wanted = new Set(); + for (const [i, schedule] of schedules.entries()) { + const name = `${prefix}${i + 1}`; + wanted.add(name); + const input: ScheduledRunEvent = { + source: SCHEDULE_EVENT_SOURCE, + action: schedule.action, + environment: env, + }; + const params = { + Name: name, + GroupName: SCHEDULE_GROUP, + ScheduleExpression: expressions[i], + ScheduleExpressionTimezone: schedule.timezone, + FlexibleTimeWindow: { Mode: 'OFF' as const }, + Target: { + Arn: schedule.action === 'start' ? START_FN_ARN : STOP_FN_ARN, + RoleArn: SCHEDULER_ROLE_ARN, + Input: JSON.stringify(input), + }, + }; + await scheduler.send(existing.has(name) ? new UpdateScheduleCommand(params) : new CreateScheduleCommand(params)); + } + for (const name of existing) { + if (!wanted.has(name)) { + await scheduler.send(new DeleteScheduleCommand({ Name: name, GroupName: SCHEDULE_GROUP })); + } + } +} diff --git a/remote/lambda/shared/schedules.ts b/remote/lambda/shared/schedules.ts new file mode 100644 index 00000000..5f9a4dcd --- /dev/null +++ b/remote/lambda/shared/schedules.ts @@ -0,0 +1,322 @@ +/** + * Schedules: when an environment is started and stopped without anyone asking. + * + * An environment holds a list of schedules, each a five-field cron expression, + * an IANA time zone and an action. The list is stored in an SSM parameter and + * mirrored into EventBridge Scheduler schedules, which fire the start and stop + * Lambdas with a `ScheduledRunEvent`. Everything here is pure (no AWS calls), + * so the Lambda that owns the schedules and the two that run them share it. + */ + +import { createHash } from 'node:crypto'; + +export const SCHEDULE_ACTIONS = ['start', 'stop'] as const; +export type ScheduleAction = (typeof SCHEDULE_ACTIONS)[number]; + +export interface Schedule { + action: ScheduleAction; + /** Five fields: minute, hour, day of month, month, day of week. */ + cron: string; + /** IANA zone name, e.g. `Europe/London`. */ + timezone: string; +} + +/** The `source` of the event EventBridge Scheduler sends to the start and stop Lambdas. */ +export const SCHEDULE_EVENT_SOURCE = 'spinloop.schedule'; + +export interface ScheduledRunEvent { + source: typeof SCHEDULE_EVENT_SOURCE; + action: ScheduleAction; + environment: string; +} + +export function isScheduledRunEvent(event: unknown): event is ScheduledRunEvent { + return ( + typeof event === 'object' && + event !== null && + (event as { source?: unknown }).source === SCHEDULE_EVENT_SOURCE + ); +} + +/** More than this on one environment is almost certainly a mistake, and bounds the Scheduler calls per request. */ +export const MAX_SCHEDULES = 20; + +export const DEFAULT_TIMEZONE = 'UTC'; + +/** Raised for input the caller can fix; the Lambda maps it to a 400. */ +export class ScheduleError extends Error {} + +/** SSM parameter holding an environment's schedule list. */ +export function schedulesParam(env: string): string { + return `/cloud-vm-llm/${env}/schedules`; +} + +/** + * The prefix of the Scheduler schedule names belonging to one environment. + * Schedule names are limited to 64 characters and an environment name can be + * that long on its own, so the prefix is a hash of the name; the environment + * itself travels in each schedule's input. + */ +export function schedulerNamePrefix(env: string): string { + return `e-${createHash('sha256').update(env).digest('hex').slice(0, 16)}-`; +} + +const FIELD_RANGES: ReadonlyArray<{ name: string; min: number; max: number }> = [ + { name: 'minute', min: 0, max: 59 }, + { name: 'hour', min: 0, max: 23 }, + { name: 'day of month', min: 1, max: 31 }, + { name: 'month', min: 1, max: 12 }, + { name: 'day of week', min: 0, max: 7 }, +]; + +const WEEKDAY_NAMES = ['SUN', 'MON', 'TUE', 'WED', 'THU', 'FRI', 'SAT']; + +export interface ParsedCron { + minutes: number[]; + hours: number[]; + daysOfMonth: number[]; + months: number[]; + /** 0 (Sunday) to 6 (Saturday). */ + daysOfWeek: number[]; + /** True when the field was `*` (or an equivalent full range), so it does not restrict. */ + anyDayOfMonth: boolean; + anyDayOfWeek: boolean; + /** Source text of the day-of-week field, for rendering. */ + dayOfWeekField: string; +} + +function invalid(expr: string, why: string): ScheduleError { + return new ScheduleError(`invalid cron expression ${JSON.stringify(expr)}: ${why}`); +} + +function parseField(expr: string, field: string, index: number): number[] { + const { name, min, max } = FIELD_RANGES[index]; + const values = new Set(); + for (const part of field.split(',')) { + const [rangePart, stepPart, extra] = part.split('/'); + if (extra !== undefined || rangePart === '') { + throw invalid(expr, `bad ${name} field ${JSON.stringify(field)}`); + } + let step = 1; + if (stepPart !== undefined) { + if (!/^\d+$/.test(stepPart) || Number(stepPart) < 1) { + throw invalid(expr, `bad step in ${name} field ${JSON.stringify(field)}`); + } + step = Number(stepPart); + } + let lo: number; + let hi: number; + if (rangePart === '*') { + lo = min; + hi = max; + } else { + const m = /^(\d+)(?:-(\d+))?$/.exec(rangePart); + if (!m) { + throw invalid(expr, `bad ${name} field ${JSON.stringify(field)}`); + } + lo = Number(m[1]); + hi = m[2] === undefined ? (stepPart === undefined ? lo : max) : Number(m[2]); + } + if (lo < min || hi > max || lo > hi) { + throw invalid(expr, `${name} must be within ${min}-${max}`); + } + for (let v = lo; v <= hi; v += step) { + values.add(v); + } + } + return [...values].sort((a, b) => a - b); +} + +/** Parse a five-field cron expression. Names (`MON`, `JAN`) are not accepted. */ +export function parseCron(expr: string): ParsedCron { + const fields = expr.trim().split(/\s+/); + if (fields.length !== 5) { + throw invalid(expr, 'expected five fields (minute hour day-of-month month day-of-week)'); + } + const [minutes, hours, daysOfMonth, months, dow] = fields.map((f, i) => parseField(expr, f, i)); + const daysOfWeek = [...new Set(dow.map((d) => d % 7))].sort((a, b) => a - b); + const anyDayOfMonth = daysOfMonth.length === 31; + const anyDayOfWeek = daysOfWeek.length === 7; + if (!anyDayOfMonth && !anyDayOfWeek) { + throw invalid(expr, 'set either the day of month or the day of week, not both'); + } + return { + minutes, + hours, + daysOfMonth, + months, + daysOfWeek, + anyDayOfMonth, + anyDayOfWeek, + dayOfWeekField: fields[4], + }; +} + +/** + * The EventBridge Scheduler form of a five-field expression: six fields (year + * added), `?` in whichever day field is not used, and weekday names, since the + * numbering of weekdays differs from classic cron (Scheduler counts Sunday as 1). + */ +export function toSchedulerCron(expr: string): string { + const parsed = parseCron(expr); + const [minute, hour, dom, month] = expr.trim().split(/\s+/); + const dayOfMonth = parsed.anyDayOfMonth ? (parsed.anyDayOfWeek ? '*' : '?') : dom; + const dayOfWeek = parsed.anyDayOfWeek + ? '?' + : parsed.daysOfWeek.map((d) => WEEKDAY_NAMES[d]).join(','); + return `cron(${minute} ${hour} ${dayOfMonth} ${month} ${dayOfWeek} *)`; +} + +export function isValidTimezone(zone: string): boolean { + try { + new Intl.DateTimeFormat('en-US', { timeZone: zone }); + return true; + } catch { + return false; + } +} + +/** + * Validate an untrusted schedule list, filling the default time zone. Throws a + * `ScheduleError` naming the first problem. + */ +export function validateSchedules(input: unknown): Schedule[] { + if (!Array.isArray(input)) { + throw new ScheduleError('schedules must be a list'); + } + if (input.length > MAX_SCHEDULES) { + throw new ScheduleError(`at most ${MAX_SCHEDULES} schedules per environment`); + } + return input.map((raw, i) => { + if (typeof raw !== 'object' || raw === null) { + throw new ScheduleError(`schedule ${i + 1} must be an object`); + } + const { action, cron, timezone } = raw as Record; + if (!(SCHEDULE_ACTIONS as readonly unknown[]).includes(action)) { + throw new ScheduleError( + `schedule ${i + 1}: unknown action ${JSON.stringify(action)}; accepted: ${SCHEDULE_ACTIONS.join(', ')}`, + ); + } + if (typeof cron !== 'string') { + throw new ScheduleError(`schedule ${i + 1}: cron must be a string`); + } + parseCron(cron); + const zone = timezone === undefined || timezone === '' ? DEFAULT_TIMEZONE : timezone; + if (typeof zone !== 'string' || !isValidTimezone(zone)) { + throw new ScheduleError(`schedule ${i + 1}: unknown time zone ${JSON.stringify(zone)}`); + } + return { action: action as ScheduleAction, cron: cron.trim().split(/\s+/).join(' '), timezone: zone }; + }); +} + +interface WallTime { + year: number; + month: number; + day: number; + hour: number; + minute: number; +} + +function wallTimeIn(utcMs: number, zone: string): WallTime { + const parts = new Intl.DateTimeFormat('en-US', { + timeZone: zone, + hourCycle: 'h23', + year: 'numeric', + month: 'numeric', + day: 'numeric', + hour: 'numeric', + minute: 'numeric', + }).formatToParts(new Date(utcMs)); + const get = (type: string) => Number(parts.find((p) => p.type === type)?.value); + return { + year: get('year'), + month: get('month'), + day: get('day'), + hour: get('hour'), + minute: get('minute'), + }; +} + +function wallAsUtc(w: WallTime): number { + return Date.UTC(w.year, w.month - 1, w.day, w.hour, w.minute); +} + +/** + * The instant a wall-clock time falls at in a zone, or null when the zone's + * clocks skip that time (the hour lost when they go forward). When the clocks + * repeat an hour, the first occurrence is returned. + */ +function instantOf(w: WallTime, zone: string): number | null { + const asUtc = wallAsUtc(w); + // The offset around the wanted time, probed a day either side so a change at + // the wanted time itself cannot hide it. + const candidates = new Set(); + for (const probe of [asUtc - 86_400_000, asUtc, asUtc + 86_400_000]) { + candidates.add(wallAsUtc(wallTimeIn(probe, zone)) - probe); + } + const matches = [...candidates] + .map((offset) => asUtc - offset) + .filter((utc) => wallAsUtc(wallTimeIn(utc, zone)) === asUtc) + .sort((a, b) => a - b); + return matches.length > 0 ? matches[0] : null; +} + +/** + * The next instant after `after` that the expression matches in the zone, or + * null when there is none within the next two years. + */ +export function nextRun(expr: string, zone: string, after: Date): Date | null { + const cron = parseCron(expr); + const start = wallTimeIn(after.getTime(), zone); + const dayMs = 86_400_000; + const firstDay = Date.UTC(start.year, start.month - 1, start.day); + for (let n = 0; n < 732; n++) { + const day = new Date(firstDay + n * dayMs); + const month = day.getUTCMonth() + 1; + if (!cron.months.includes(month)) { + continue; + } + const dayOk = cron.anyDayOfMonth + ? cron.daysOfWeek.includes(day.getUTCDay()) + : cron.daysOfMonth.includes(day.getUTCDate()); + if (!dayOk) { + continue; + } + for (const hour of cron.hours) { + for (const minute of cron.minutes) { + const instant = instantOf( + { + year: day.getUTCFullYear(), + month, + day: day.getUTCDate(), + hour, + minute, + }, + zone, + ); + if (instant !== null && instant > after.getTime()) { + return new Date(instant); + } + } + } + } + return null; +} + +/** The next time each action fires across a list of schedules. */ +export function nextRuns( + schedules: Schedule[], + after: Date, +): { start: string | null; stop: string | null } { + const earliest = (action: ScheduleAction): string | null => { + let best: Date | null = null; + for (const s of schedules.filter((x) => x.action === action)) { + const next = nextRun(s.cron, s.timezone, after); + if (next && (!best || next < best)) { + best = next; + } + } + return best ? best.toISOString() : null; + }; + return { start: earliest('start'), stop: earliest('stop') }; +} diff --git a/remote/lambda/start/index.ts b/remote/lambda/start/index.ts index baf71592..70ae0bb8 100644 --- a/remote/lambda/start/index.ts +++ b/remote/lambda/start/index.ts @@ -38,6 +38,7 @@ import { } from '../shared/environments'; import { DAEMON_STATUS_CMD, parseDaemonStatus } from '../shared/daemon'; import { jsonResponse } from '../shared/http'; +import { isScheduledRunEvent, type ScheduledRunEvent } from '../shared/schedules'; import { acquireWakeLock, releaseWakeLock, wakeLockHeld } from '../shared/wake-lock'; import { weightsPresent } from '../shared/seed'; import { findSeedInstances, seedAlive } from '../shared/seed/discovery'; @@ -133,9 +134,12 @@ function parseRetainUntil(raw: string | undefined): string | null { } export async function handler( - event: LambdaFunctionURLEvent, + event: LambdaFunctionURLEvent | ScheduledRunEvent, context: Context, ): Promise { + if (isScheduledRunEvent(event)) { + return scheduledStart(event, context); + } let env: string; try { env = environmentFrom(event.queryStringParameters); @@ -164,6 +168,29 @@ async function startHoldsLock(env: string): Promise { } } +/** + * A `start` schedule firing: the same wake as an on-demand start, with no + * retention deadline. A running instance is left as it is (the wake reports it + * ready), and the wake takes the environment's start lock like any other, so a + * schedule firing while someone starts the environment by hand launches one + * instance, not two. Nobody is waiting on the reply, so the outcome is logged, + * and a failure to launch is not retried before the next firing. + */ +async function scheduledStart( + event: ScheduledRunEvent, + context: Context, +): Promise { + if (event.action !== 'start') { + console.log(JSON.stringify({ mode: 'scheduled', action: 'ignored', scheduledAction: event.action })); + return jsonResponse(200, { state: 'ignored', environment: event.environment }); + } + const result = await wake(event.environment, context, null); + console.log( + JSON.stringify({ mode: 'scheduled', action: 'start', environment: event.environment, result }), + ); + return result; +} + /** GET — report one environment's state without side effects. */ async function status(env: string): Promise { const eip = await findEnvEip(env); diff --git a/remote/lambda/stop/index.ts b/remote/lambda/stop/index.ts index b3e74782..e35480de 100644 --- a/remote/lambda/stop/index.ts +++ b/remote/lambda/stop/index.ts @@ -19,6 +19,7 @@ import { DAEMON_STATUS_CMD, parseDaemonStatus } from '../shared/daemon'; import { decideIdle, idleFromDaemonStatus, type MetricsResult } from '../shared/idle'; import { jsonResponse } from '../shared/http'; import { SEED_ID_TAG_KEY, SEED_TAG_VALUE } from '../shared/seed/identity'; +import { isScheduledRunEvent, type ScheduledRunEvent } from '../shared/schedules'; import { decideSeedReap } from '../shared/seed/reap'; import { latestStream, writeTerminalRecord } from '../shared/seed/status'; @@ -32,13 +33,23 @@ const SEED_STALL_MINUTES = Number(requireEnv('SEED_STALL_MINUTES')); const STOP_RETENTION_MINUTES = Number(requireEnv('STOP_RETENTION_MINUTES')); -type StopEvent = ScheduledEvent | LambdaFunctionURLEvent; +type StopEvent = ScheduledEvent | LambdaFunctionURLEvent | ScheduledRunEvent; export function isScheduledEvent(event: StopEvent): event is ScheduledEvent { return (event as ScheduledEvent).source === 'aws.events'; } export async function handler(event: StopEvent): Promise { + if (isScheduledRunEvent(event)) { + // A user-defined schedule firing, not the sweep. Only `stop` schedules + // target this Lambda; anything else is ignored rather than acted on. + if (event.action === 'stop') { + await scheduledStop(event.environment); + } else { + console.log(JSON.stringify({ mode: 'scheduled', action: 'ignored', scheduledAction: event.action })); + } + return; + } if (isScheduledEvent(event)) { // Two passes over two disjoint populations, keyed on different tag values // and judged by different signals. A seed runs no spinloop daemon, so it must @@ -100,6 +111,38 @@ async function manualStop(event: LambdaFunctionURLEvent): Promise { + const instance = await findManagedInstance(TAG_KEY, TAG_VALUE, [ + { Name: `tag:${ENV_TAG_KEY}`, Values: [env] }, + ]); + if (!instance || instance.state !== 'running') { + console.log( + JSON.stringify({ mode: 'scheduled', action: 'noop', environment: env, state: instance?.state ?? 'none' }), + ); + return; + } + if (instance.retainUntil && instance.retainUntil.getTime() > Date.now()) { + console.log( + JSON.stringify({ + mode: 'scheduled', + action: 'skip', + reason: 'retained', + environment: env, + instanceId: instance.instanceId, + retainUntil: instance.retainUntil.toISOString(), + }), + ); + return; + } + await pauseInstance(instance, env, false, 'scheduled'); +} + /** * Stop (never terminate) one environment's instance. Tagging before the stop * means a crash in between leaves a stopped instance with its stop time @@ -113,13 +156,14 @@ async function pauseInstance( instance: InstanceInfo, env: string, force: boolean, + mode: 'manual' | 'scheduled' = 'manual', ): Promise { if (instance.state === 'stopped') { if (!instance.stoppedAt) { await tagInstance(instance.instanceId, STOPPED_AT_TAG, new Date().toISOString()); } console.log( - JSON.stringify({ mode: 'manual', action: 'noop', environment: env, instanceId: instance.instanceId }), + JSON.stringify({ mode, action: 'noop', environment: env, instanceId: instance.instanceId }), ); return jsonResponse(200, { state: 'stopped', environment: env }); } @@ -130,7 +174,7 @@ async function pauseInstance( await stopInstance(instance.instanceId); console.log( JSON.stringify({ - mode: 'manual', + mode, action: 'stop', environment: env, instanceId: instance.instanceId, diff --git a/remote/lib/llm-stack.ts b/remote/lib/llm-stack.ts index b7b2cfec..775a3be0 100644 --- a/remote/lib/llm-stack.ts +++ b/remote/lib/llm-stack.ts @@ -9,6 +9,7 @@ import { aws_logs as logs, aws_s3 as s3, aws_s3_assets as s3assets, + aws_scheduler as scheduler, aws_secretsmanager as secretsmanager, } from 'aws-cdk-lib'; import { Construct } from 'constructs'; @@ -693,6 +694,64 @@ export class LlmStack extends cdk.Stack { const updateUrl = updateFn.addFunctionUrl({ authType: lambda.FunctionUrlAuthType.AWS_IAM }); + // Schedules — cron-driven start and stop per environment. The schedule + // Lambda keeps each environment's list in SSM and mirrors it into + // EventBridge Scheduler schedules in this group; Scheduler (not a classic + // rule) because it takes an IANA time zone and follows daylight saving. + const scheduleGroup = new scheduler.CfnScheduleGroup(this, 'ScheduleGroup'); + const scheduleArnPattern = `arn:${cdk.Aws.PARTITION}:scheduler:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:schedule/${scheduleGroup.ref}/*`; + // What a schedule runs as: invoke the start or stop Lambda, nothing else. + const schedulerRole = new iam.Role(this, 'SchedulerRole', { + description: 'Assumed by EventBridge Scheduler to run scheduled starts and stops', + assumedBy: new iam.ServicePrincipal('scheduler.amazonaws.com', { + conditions: { StringEquals: { 'aws:SourceAccount': cdk.Aws.ACCOUNT_ID } }, + }), + }); + startFn.grantInvoke(schedulerRole); + stopFn.grantInvoke(schedulerRole); + + const scheduleFn = new nodejs.NodejsFunction(this, 'ScheduleFn', { + description: 'Sets, reads and clears an environment\'s cron start/stop schedules', + entry: path.join(__dirname, '..', 'lambda', 'schedule', 'index.ts'), + handler: 'handler', + runtime: lambda.Runtime.NODEJS_22_X, + architecture: lambda.Architecture.ARM_64, + timeout: cdk.Duration.seconds(60), + memorySize: 256, + logGroup: lambdaLogGroup('ScheduleFnLogGroup', 'schedule'), + environment: { + SCHEDULE_GROUP: scheduleGroup.ref, + SCHEDULER_ROLE_ARN: schedulerRole.roleArn, + START_FN_ARN: startFn.functionArn, + STOP_FN_ARN: stopFn.functionArn, + }, + }); + scheduleFn.addToRolePolicy( + new iam.PolicyStatement({ + actions: ['ssm:GetParameter', 'ssm:PutParameter', 'ssm:DeleteParameter'], + resources: [envParamArn], + }), + ); + scheduleFn.addToRolePolicy( + new iam.PolicyStatement({ + actions: ['scheduler:CreateSchedule', 'scheduler:UpdateSchedule', 'scheduler:DeleteSchedule'], + resources: [scheduleArnPattern], + }), + ); + // ListSchedules has no resource-level scoping. + scheduleFn.addToRolePolicy( + new iam.PolicyStatement({ actions: ['scheduler:ListSchedules'], resources: ['*'] }), + ); + scheduleFn.addToRolePolicy( + new iam.PolicyStatement({ + actions: ['iam:PassRole'], + resources: [schedulerRole.roleArn], + conditions: { StringEquals: { 'iam:PassedToService': 'scheduler.amazonaws.com' } }, + }), + ); + + const scheduleUrl = scheduleFn.addFunctionUrl({ authType: lambda.FunctionUrlAuthType.AWS_IAM }); + // The human-facing principal behind the CLI's long-lived credential: // `spinloop remote auth --store` creates an access key for this user and // keeps it in the operator's OS keystore, so day-to-day control calls @@ -708,7 +767,7 @@ export class LlmStack extends cdk.Stack { // managed-policy limit is 6,144). The invoke-url grant takes the form // grantInvokeUrl renders — the two actions, the auth-type conditions, the // backing functions' ARNs — merged into one statement per action. - const controlFunctionArns = [startUrl, stopUrl, deployUrl, statsUrl, seedUrl, envUrl, updateUrl].map( + const controlFunctionArns = [startUrl, stopUrl, deployUrl, statsUrl, seedUrl, envUrl, updateUrl, scheduleUrl].map( (url) => url.functionArn, ); const controlLogGroupArns = [ @@ -777,6 +836,7 @@ export class LlmStack extends cdk.Stack { new cdk.CfnOutput(this, 'EnvUrl', { value: envUrl.url }); new cdk.CfnOutput(this, 'SeedUrl', { value: seedUrl.url }); new cdk.CfnOutput(this, 'UpdateUrl', { value: updateUrl.url }); + new cdk.CfnOutput(this, 'ScheduleUrl', { value: scheduleUrl.url }); new cdk.CfnOutput(this, 'Region', { value: this.region }); new cdk.CfnOutput(this, 'WeightsBucket', { value: weightsBucket.bucketName }); new cdk.CfnOutput(this, 'VpcId', { value: vpc.vpcId }); @@ -791,7 +851,7 @@ export class LlmStack extends cdk.Stack { // environment's address is its own EIP, allocated at `spinloop remote // deploy` and returned by it. new cdk.CfnOutput(this, 'SpinloopRemoteConfig', { - value: `{"start_url":"${startUrl.url}","stop_url":"${stopUrl.url}","deploy_url":"${deployUrl.url}","stats_url":"${statsUrl.url}","env_url":"${envUrl.url}","seed_url":"${seedUrl.url}","update_url":"${updateUrl.url}","region":"${this.region}"}`, + value: `{"start_url":"${startUrl.url}","stop_url":"${stopUrl.url}","deploy_url":"${deployUrl.url}","stats_url":"${statsUrl.url}","env_url":"${envUrl.url}","seed_url":"${seedUrl.url}","update_url":"${updateUrl.url}","schedule_url":"${scheduleUrl.url}","region":"${this.region}"}`, }); } } diff --git a/remote/package.json b/remote/package.json index 62f179c7..a465012a 100644 --- a/remote/package.json +++ b/remote/package.json @@ -20,6 +20,7 @@ "@aws-sdk/client-cloudwatch-logs": "^3.1111.0", "@aws-sdk/client-ec2": "^3.700.0", "@aws-sdk/client-s3": "^3.1095.0", + "@aws-sdk/client-scheduler": "^3.1146.0", "@aws-sdk/client-secrets-manager": "^3.700.0", "@aws-sdk/client-ssm": "^3.700.0", "@huggingface/hub": "^2.15.0" diff --git a/remote/pnpm-lock.yaml b/remote/pnpm-lock.yaml index a5370293..7a42c6d5 100644 --- a/remote/pnpm-lock.yaml +++ b/remote/pnpm-lock.yaml @@ -17,6 +17,9 @@ importers: '@aws-sdk/client-s3': specifier: ^3.1095.0 version: 3.1095.0 + '@aws-sdk/client-scheduler': + specifier: ^3.1146.0 + version: 3.1146.0 '@aws-sdk/client-secrets-manager': specifier: ^3.700.0 version: 3.1094.0 @@ -93,6 +96,10 @@ packages: resolution: {integrity: sha512-eeobm3TKci47CTTHIxc5s5UDjLL0gTKfjlcyzKU+NMoeIJ3flKPq51Fhi2nUDiMNbzsY5HAynM3bHAQFHxRTUw==} engines: {node: '>=20.0.0'} + '@aws-sdk/client-scheduler@3.1146.0': + resolution: {integrity: sha512-/KcEsYx9RR40bHley1cukoYW5o+sPV0g3IQg1i+Pr1HypGh0yj2XCQnWvDJldr4F9Rmsu5sCsmOq5h4z9n5b2A==} + engines: {node: '>=20.0.0'} + '@aws-sdk/client-secrets-manager@3.1094.0': resolution: {integrity: sha512-pOPasPHgz0+z6qcbX3gNzlOPuZ3JCgkUFczqdR4dC67g8nr5BCcJZ3La5GUxteCV1S2X98+3Q6mnZ23AtTbsNQ==} engines: {node: '>=20.0.0'} @@ -113,6 +120,10 @@ packages: resolution: {integrity: sha512-7+Kcrkvrk9lM/m7jRhHpT4jCdvzGHsuaSRbF8TdzzkY1mRzp/Ogwf9c7H29k4gGhey0BBWhCWr16+t0J61gwmg==} engines: {node: '>=20.0.0'} + '@aws-sdk/core@3.978.1': + resolution: {integrity: sha512-LbY9aGsEiznDWmUc30Nwv3aIX/+dbwTx8KfS0yOC3NPYMO+O91e6jkT1azf34FwjOndq8/Q+RcVVZz5xnerwdg==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-env@3.972.60': resolution: {integrity: sha512-BAkxdoe7tpDDqCghGpuOeHQRbm/2znVvOQm0AvpQbA2tbfMN46doN4zx65fv85ImP3KADwc2zQPmbrlI9MPfMg==} engines: {node: '>=20.0.0'} @@ -125,6 +136,10 @@ packages: resolution: {integrity: sha512-AreCFzcB4kH2HF9031Ot0jSJr3KXvRg6e8uDeub20JEVdZU3Bv0sTq1plc7VsT3KiqutlzH7l0j50UcCWHUioA==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-env@3.972.72': + resolution: {integrity: sha512-xTKO/FWJPozTIXbozVnVGoNBhaGba8TBcx+KyUjRVeOlXE+dUc7GTR1cLvu0uTdIdmemzaFbqqCshXeZA1fZew==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-http@3.972.62': resolution: {integrity: sha512-g/0fGqKTb9xpKdd9AtpmV5Eo3DFKbnkpA2+w0peISSlu7NfAoWOuYBFxsu+yWBtxU89ka55ezoZBCbFaS8pjYQ==} engines: {node: '>=20.0.0'} @@ -137,10 +152,18 @@ packages: resolution: {integrity: sha512-A8ObcqVmDMnk4F9NozZ7JwmUu9Q4xyBJkmyq1C5U+wNM9ht9J7+EuuyabsLWXZnOoTqFaJuYBYTKf5CTipkEjA==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-http@3.972.74': + resolution: {integrity: sha512-u91E/hT8f4d1xy0Jl7VG4nVKJ3lxbrZkoBTeSVoJdWBiSEUMwMS/9+e0H/aJVQV//Lt5wuzP+E69v4aRSsNTmw==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-ini@3.973.14': resolution: {integrity: sha512-7c+Wti2LsERNWMfm7ySz3/6RPopFW3Nmn7s63Xpcq6R/tRuY5hpvkHA2xVgi5ukJbvok9l0IDtVEvqTtg+X7dw==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-ini@3.973.17': + resolution: {integrity: sha512-ged4KXdBkvIC81bLvNHHuQKdKak/VXhQTR1NWYTTqW0474nlmsxy9O/vlgTIohDDWH3xpBdtVMZRyjb+DnocDA==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-ini@3.973.5': resolution: {integrity: sha512-ylubazcRfq2TVus/qXucSXeC42Qdjp5HQxTu68K/BsdMiZlcSLD1zkpoCgApXZX1Y6YJhtGGs7ZHhO/GuIgBlw==} engines: {node: '>=20.0.0'} @@ -161,6 +184,10 @@ packages: resolution: {integrity: sha512-LVixwOnEJfrrfKHeZjBA8pIMTZjNDq8ak8VpcoWUuCJDrSnBNU8POJksULMgvN089P0MXtQYH2Zs627/MK1K0g==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-login@3.972.79': + resolution: {integrity: sha512-L+Z85anONJd8MaiuraO4wRxATCdEejBZ3K3eymzWI5JPXa9sOS9CkIm72PBKqXKX+Z9p9NGMX5AIMXm0LEflgw==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-node@3.972.71': resolution: {integrity: sha512-HIg7Q2osBzajQwL+1Vkyh2E7Gim3eTNb9RHIsOxDGjW0eZg4oEKtRs5sioCnc73ilhaOm4gX2lHVF8J7+nt2rg==} engines: {node: '>=20.0.0'} @@ -173,6 +200,10 @@ packages: resolution: {integrity: sha512-bE2qh8ww4iClO1jHsBXdOE8FUgzDbdxbyorNjSCoPSkQd51k3jODItuPZfuwcLHZqDXsH+bI4AMHhqtuyR7mSg==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-node@3.972.84': + resolution: {integrity: sha512-oHt854odINVwzwsh+c5x69j0ajm4DbqqqVJ+O1ECsCIZeMDAbzFpXItaqP7UZstJj/ATdTk/KFSH0LaNAgV+kA==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-process@3.972.60': resolution: {integrity: sha512-YIo3f99hM43QdYG8hDzwGemnR/pU95b0kramqSJUTleCqaB7+HwKf7YZFHqvOgTqZTPx/mRmNIqoDRr3U0Z3Tw==} engines: {node: '>=20.0.0'} @@ -185,10 +216,18 @@ packages: resolution: {integrity: sha512-9kpTNdZTrcqXTfhxM7fgl9Z68ek3Fu5oe3Yf+A/pJGibEqpgZxz2tSY7SinmyCIU2PJ+ygY4FPoBBnLpocMtrQ==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-process@3.972.72': + resolution: {integrity: sha512-rLIp2xbMjX/k9/od7APpqq1ZgXXnV0pOL1Th3ZsL8Wu0TRtBsDTVS8iPqcfRFcHakFxPvR04OSTv2ka2qOb/2A==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-sso@3.973.13': resolution: {integrity: sha512-Oc81qauMPzUoTnAS2YKpNwY6sY/LUyQTEeaf6yP197WMxkEBQfcKLR1MFpD7+pNTubXnfkH6gwpji+Gc7iyD2Q==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-sso@3.973.16': + resolution: {integrity: sha512-IGihaJfFZYacJJr/odqILCoK7W/mvrZ7cuK7ECn3sAu4vLC6u0V8bS7mCGbdugJ8Aum2tnvqmx0F2MRFp2rn9g==} + engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-sso@3.973.4': resolution: {integrity: sha512-BPdmL8sSBOCv4ngZ+3LHxyc3CNqDCEK37CHioCk7zGrTMY5sUtkH8q+o6qA80nn6w3/fyBPGNE7OIRlmoOxRQA==} engines: {node: '>=20.0.0'} @@ -209,6 +248,10 @@ packages: resolution: {integrity: sha512-YPN6uoGDgjjjeVFZrcOeCJqmB6zpXoeeNgIjqe+DexJaWqdjVfCCe+VAZwli9Z2h8KhFW8oxkO39emQ1tyz/Mw==} engines: {node: '>=20.0.0'} + '@aws-sdk/credential-provider-web-identity@3.972.78': + resolution: {integrity: sha512-/y9WvNtlcPBGLR0qc1a+9J/xtYZfVczvLUOuXaVWylzttH7ewsxwHtjmiJSolNrVSDorIxHGHMU61CbonRkmwA==} + engines: {node: '>=20.0.0'} + '@aws-sdk/middleware-sdk-ec2@3.972.48': resolution: {integrity: sha512-KIqurwP5A7AwAA2S3cd8y41IbxNFmxtTHi1ZzdEkDRLhreKFaDKk9zIJgclGfsOMaOmao+nb0S/CInOgdwS7Ig==} engines: {node: '>=20.0.0'} @@ -229,6 +272,10 @@ packages: resolution: {integrity: sha512-bit+VpqWNyi3wHxFoTsTliNXimCSL2r2OeDTm7ZrG+YsTZ2D7ofDJ6r/t9PVBn80i6/v0X2h9Tgw6QP2MAKfPw==} engines: {node: '>=20.0.0'} + '@aws-sdk/nested-clients@3.997.46': + resolution: {integrity: sha512-oRxtBcka/JGHGs9l9p9IVajGoTP8vTPmoAzdHGy4Qcy9P5vPnDf6nhIeM/COQNY9k/OahImTRaLkHftoXvfcmQ==} + engines: {node: '>=20.0.0'} + '@aws-sdk/signature-v4-multi-region@3.996.41': resolution: {integrity: sha512-QMUytg+FQMGouc8gHS00KoYih3+N6cqmVI/pQGOIo7Nr7OpQaiXjSYOuL+vsPZ1tymY4LAQ8MYcHJmws5LRxng==} engines: {node: '>=20.0.0'} @@ -241,6 +288,10 @@ packages: resolution: {integrity: sha512-bBuyztukzXq6plzFGHAWiQt0QXo+HL8b8lX5cFTzkez/74PtS1c0qPFCIVuHkyoT+miH2qOjAcm1/yoro2ESPA==} engines: {node: '>=20.0.0'} + '@aws-sdk/signature-v4-multi-region@3.996.47': + resolution: {integrity: sha512-Zk08macMvQTHzQJCLJVkOlviVoqwYMrpXv4lmLN7b7sAbiMoOK7Go0NYdR5UeF+MW8LIbRmwrNy9u/5VvX1U5g==} + engines: {node: '>=20.0.0'} + '@aws-sdk/token-providers@3.1092.0': resolution: {integrity: sha512-hBYUAr6iBLNFcsiWTgtBb0stdSw39VOUq4Sp4A5caCNf66BAZplWN4FleKrVpJx5li2YgdnK2DqoFSMWC642FQ==} engines: {node: '>=20.0.0'} @@ -253,6 +304,10 @@ packages: resolution: {integrity: sha512-JfljgoVtl+s3Qy21n9a7Z48uCQaOXcN74KJ3TEQfPoB293GrXFSt6HSQJF1sTZ8c/5QedEvd3NjJQMO4u9qa5A==} engines: {node: '>=20.0.0'} + '@aws-sdk/token-providers@3.1138.0': + resolution: {integrity: sha512-GpyAr0DD63YOEmYFM6Df+gJuIgC92MMTiBK4FTKfxii5MJ9ge20epR7LyroulscYlG89J+ZB2ivFDPjvfQhzdw==} + engines: {node: '>=20.0.0'} + '@aws-sdk/types@3.974.2': resolution: {integrity: sha512-3W6IUtSxFbH6X7Wb7DzGCV5QiFQsd0g8bOfntpmDxQlzBoKWUMBu/JPQR0DwkE+Hpnxd6db1tXbOwdeHddG6cA==} engines: {node: '>=20.0.0'} @@ -261,6 +316,10 @@ packages: resolution: {integrity: sha512-dSFDNG00MEz0/xl5gxL62giLd1iYyJsTxZ1I1DOj6lC+bbgLB4TRsYClJg3b62dhXT1uATzsTNXPnC+33EJV3A==} engines: {node: '>=20.0.0'} + '@aws-sdk/types@3.974.6': + resolution: {integrity: sha512-v/clNZzZnDxGyvpHMOGpJKVXFAExJzUNAAjaWGdcx8QAcXLGwTaOkw33p5SHAi0YAioK32xB3hWwOekRVfmfKg==} + engines: {node: '>=20.0.0'} + '@aws-sdk/xml-builder@3.972.36': resolution: {integrity: sha512-RdGmS1GLrtaTOLE1ElSluMldNrpk9Emq6uYs8SS8iHlu5xTAmM9rRkM91o48+rIRryBtyO9t+uLYCoMG6jVMVA==} engines: {node: '>=20.0.0'} @@ -273,6 +332,10 @@ packages: resolution: {integrity: sha512-FTti8DS5MMWXNUWiRwXAJeYS+0GHHiMy0+7XOhcwk63ILHmfS2UFy2z/HNpZCSOJJ3P3dnWY6hfYNW3DF0nXUA==} engines: {node: '>=20.0.0'} + '@aws-sdk/xml-builder@3.972.41': + resolution: {integrity: sha512-ctjVSyCMegrWfXlx6VqzSBFI6UqmQ5ZlnfMhdLIiWmhoH8UAQxSCP5N3OpG7X3k4LnS7ou74C4mt20+bfTW2aQ==} + engines: {node: '>=20.0.0'} + '@aws/lambda-invoke-store@0.3.0': resolution: {integrity: sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==} engines: {node: '>=18.0.0'} @@ -795,6 +858,10 @@ packages: resolution: {integrity: sha512-CUGXpnPkVdjUCbix+83sWLW9VFgQOm44MDOx/ihITJMAnOZKvL8YYIc7DR9pP/tZ8CIRvMiON/TucvygqbHO3w==} engines: {node: '>=18.0.0'} + '@smithy/core@3.35.1': + resolution: {integrity: sha512-i4YPS4B6ts7bjn7UwLnGjiZdprOvHvgGobFZsYK3GIY3E5hIqtj0rReU69BcTpGp+fvtraSNXeG1l+jtJvF55w==} + engines: {node: '>=18.0.0'} + '@smithy/credential-provider-imds@4.4.13': resolution: {integrity: sha512-X+2HNZhWi5i3rJsCas0LPf6fTQUaKyJ40zd8aTO/bwpRfpU3biYaqLr7C1WMibL7PVKJalpi1PyybjGPNoHC8Q==} engines: {node: '>=18.0.0'} @@ -811,10 +878,18 @@ packages: resolution: {integrity: sha512-nZyWTmSpJEXl6VtWVMBJve/7x12DZu6sIX1z1a+ZMaHlQQRs9Zpu6NbTe/gmxYXVRpkjxyDYpZ5gx2IM6f/Wkw==} engines: {node: '>=18.0.0'} + '@smithy/fetch-http-handler@5.8.0': + resolution: {integrity: sha512-ycSJu3tFAQ4v04CBB0agqFMVsSQ1iG3yw+SpgxRqKfaURpQD4CZ8Wn0zPMmSnOuTpTh65Vz+EA0rMrw089wvkA==} + engines: {node: '>=18.0.0'} + '@smithy/node-http-handler@4.11.2': resolution: {integrity: sha512-avwAh9HM3h2lcfjvP3zYIZGf+XVgLQ91wOJ2qoFbNpW1UZeZb33aGlhTZvtkANHfcGhJroRY64525OjfgOg30g==} engines: {node: '>=18.0.0'} + '@smithy/node-http-handler@4.12.1': + resolution: {integrity: sha512-ThMkboGeONWXAelq9FvGsuJC4rOi+qyC4/zhUF58xYpxUg5sQKx2VXZYJmtNjr4dSuBJ1HeJXETQILCz3wOHvw==} + engines: {node: '>=18.0.0'} + '@smithy/node-http-handler@4.9.10': resolution: {integrity: sha512-ETQz9v/Z+nTQc6fRWTXxUpxJqwpmzB3Tn3WKAdHwWkeT+m+HE5czs6GNG8vW+4vyxXSls65RVcvOZwk7Q/PS/Q==} engines: {node: '>=18.0.0'} @@ -827,6 +902,10 @@ packages: resolution: {integrity: sha512-P7Ki6px6OOrxVtx8K7nLmyx4SlXUW/uTKDdMG44UHefmPGSRMBKe2v+TM59WdLcpUIrBrnuCsIqiM2MbsZjmhw==} engines: {node: '>=18.0.0'} + '@smithy/signature-v4@5.7.4': + resolution: {integrity: sha512-tHy0K0VtqNd5Y7Y41h0a0Lhh0L1GzC08dTWg0F7vRJWFtTENg7IZikf3wQkanYIRdb7ngoIPMTmqgUi401fEeQ==} + engines: {node: '>=18.0.0'} + '@smithy/types@4.16.1': resolution: {integrity: sha512-0JFs3V2y2M9tKW5na/qxe69Zv+uxLMO7QBbhxF/FHu/Gp2NFZAAL9tWl9PU02xxo07pb3G9FTyjNc6D5uZrJIg==} engines: {node: '>=18.0.0'} @@ -835,6 +914,10 @@ packages: resolution: {integrity: sha512-FOKpVZob9MPTn2znRzGrnsMHv7BOsKVw3XiP/cOyYLDVZ9qKp4nifIiSCuUU/fIj5Vu0UOAxCFr+qRAtG0NUkA==} engines: {node: '>=18.0.0'} + '@smithy/types@4.19.0': + resolution: {integrity: sha512-r7jh49VJxGerfAcTQA6gXcKc+98zOp/tqRwzYjgOE+iSQsP6cEU1hq2QzbuipmP68QtYdY9wKEhiCQZIzHgZ4Q==} + engines: {node: '>=18.0.0'} + '@types/aws-lambda@8.10.162': resolution: {integrity: sha512-Fn658grtLOci1oxi1391vvDWJRKNGWRSqfxRkmN/Iy3c0tQH1USMKEXcPYHLvope+ZgTFocx9FRQJx1muBL6qw==} @@ -1403,6 +1486,17 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/client-scheduler@3.1146.0': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/credential-provider-node': 3.972.84 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/fetch-http-handler': 5.8.0 + '@smithy/node-http-handler': 4.12.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/client-secrets-manager@3.1094.0': dependencies: '@aws-sdk/core': 3.976.0 @@ -1458,6 +1552,17 @@ snapshots: bowser: 2.14.1 tslib: 2.8.1 + '@aws-sdk/core@3.978.1': + dependencies: + '@aws-sdk/types': 3.974.6 + '@aws-sdk/xml-builder': 3.972.41 + '@aws/lambda-invoke-store': 0.3.0 + '@smithy/core': 3.35.1 + '@smithy/signature-v4': 5.7.4 + '@smithy/types': 4.19.0 + bowser: 2.14.1 + tslib: 2.8.1 + '@aws-sdk/credential-provider-env@3.972.60': dependencies: '@aws-sdk/core': 3.976.0 @@ -1482,6 +1587,14 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-env@3.972.72': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-http@3.972.62': dependencies: '@aws-sdk/core': 3.976.0 @@ -1512,6 +1625,16 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-http@3.972.74': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/fetch-http-handler': 5.8.0 + '@smithy/node-http-handler': 4.12.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-ini@3.973.14': dependencies: '@aws-sdk/core': 3.977.8 @@ -1528,6 +1651,22 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-ini@3.973.17': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/credential-provider-env': 3.972.72 + '@aws-sdk/credential-provider-http': 3.972.74 + '@aws-sdk/credential-provider-login': 3.972.79 + '@aws-sdk/credential-provider-process': 3.972.72 + '@aws-sdk/credential-provider-sso': 3.973.16 + '@aws-sdk/credential-provider-web-identity': 3.972.78 + '@aws-sdk/nested-clients': 3.997.46 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/credential-provider-imds': 4.5.2 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-ini@3.973.5': dependencies: '@aws-sdk/core': 3.976.0 @@ -1587,6 +1726,15 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-login@3.972.79': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/nested-clients': 3.997.46 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-node@3.972.71': dependencies: '@aws-sdk/credential-provider-env': 3.972.60 @@ -1629,6 +1777,20 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-node@3.972.84': + dependencies: + '@aws-sdk/credential-provider-env': 3.972.72 + '@aws-sdk/credential-provider-http': 3.972.74 + '@aws-sdk/credential-provider-ini': 3.973.17 + '@aws-sdk/credential-provider-process': 3.972.72 + '@aws-sdk/credential-provider-sso': 3.973.16 + '@aws-sdk/credential-provider-web-identity': 3.972.78 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/credential-provider-imds': 4.5.2 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-process@3.972.60': dependencies: '@aws-sdk/core': 3.976.0 @@ -1653,6 +1815,14 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-process@3.972.72': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-sso@3.973.13': dependencies: '@aws-sdk/core': 3.977.8 @@ -1663,6 +1833,16 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-sso@3.973.16': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/nested-clients': 3.997.46 + '@aws-sdk/token-providers': 3.1138.0 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/credential-provider-sso@3.973.4': dependencies: '@aws-sdk/core': 3.976.0 @@ -1710,6 +1890,15 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/credential-provider-web-identity@3.972.78': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/nested-clients': 3.997.46 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/middleware-sdk-ec2@3.972.48': dependencies: '@aws-sdk/core': 3.976.0 @@ -1761,6 +1950,17 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/nested-clients@3.997.46': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/signature-v4-multi-region': 3.996.47 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/fetch-http-handler': 5.8.0 + '@smithy/node-http-handler': 4.12.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/signature-v4-multi-region@3.996.41': dependencies: '@aws-sdk/types': 3.974.2 @@ -1782,6 +1982,13 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/signature-v4-multi-region@3.996.47': + dependencies: + '@aws-sdk/types': 3.974.6 + '@smithy/signature-v4': 5.7.4 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/token-providers@3.1092.0': dependencies: '@aws-sdk/core': 3.976.0 @@ -1809,6 +2016,15 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/token-providers@3.1138.0': + dependencies: + '@aws-sdk/core': 3.978.1 + '@aws-sdk/nested-clients': 3.997.46 + '@aws-sdk/types': 3.974.6 + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/types@3.974.2': dependencies: '@smithy/types': 4.16.1 @@ -1819,6 +2035,11 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/types@3.974.6': + dependencies: + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws-sdk/xml-builder@3.972.36': dependencies: '@smithy/types': 4.16.1 @@ -1834,6 +2055,11 @@ snapshots: '@smithy/types': 4.16.1 tslib: 2.8.1 + '@aws-sdk/xml-builder@3.972.41': + dependencies: + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@aws/lambda-invoke-store@0.3.0': {} '@babel/helper-string-parser@7.29.7': {} @@ -2136,6 +2362,11 @@ snapshots: '@smithy/types': 4.17.2 tslib: 2.8.1 + '@smithy/core@3.35.1': + dependencies: + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@smithy/credential-provider-imds@4.4.13': dependencies: '@smithy/core': 3.29.8 @@ -2160,12 +2391,24 @@ snapshots: '@smithy/types': 4.17.2 tslib: 2.8.1 + '@smithy/fetch-http-handler@5.8.0': + dependencies: + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@smithy/node-http-handler@4.11.2': dependencies: '@smithy/core': 3.33.2 '@smithy/types': 4.17.2 tslib: 2.8.1 + '@smithy/node-http-handler@4.12.1': + dependencies: + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@smithy/node-http-handler@4.9.10': dependencies: '@smithy/core': 3.29.8 @@ -2184,6 +2427,12 @@ snapshots: '@smithy/types': 4.17.2 tslib: 2.8.1 + '@smithy/signature-v4@5.7.4': + dependencies: + '@smithy/core': 3.35.1 + '@smithy/types': 4.19.0 + tslib: 2.8.1 + '@smithy/types@4.16.1': dependencies: tslib: 2.8.1 @@ -2192,6 +2441,10 @@ snapshots: dependencies: tslib: 2.8.1 + '@smithy/types@4.19.0': + dependencies: + tslib: 2.8.1 + '@types/aws-lambda@8.10.162': {} '@types/chai@5.2.3': diff --git a/remote/test/schedule-api.test.ts b/remote/test/schedule-api.test.ts new file mode 100644 index 00000000..ed563a85 --- /dev/null +++ b/remote/test/schedule-api.test.ts @@ -0,0 +1,191 @@ +import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { LambdaFunctionURLEvent } from 'aws-lambda'; +import { schedulerNamePrefix, schedulesParam } from '../lambda/shared/schedules'; + +// The schedule Lambda: GET/PUT/DELETE on one environment's schedule list, kept +// in SSM and mirrored into EventBridge Scheduler schedules. + +const LAMBDA_ENV = { + SCHEDULE_GROUP: 'test-group', + SCHEDULER_ROLE_ARN: 'arn:aws:iam::000000000000:role/scheduler', + START_FN_ARN: 'arn:aws:lambda:us-east-1:000000000000:function:start', + STOP_FN_ARN: 'arn:aws:lambda:us-east-1:000000000000:function:stop', +}; + +const ssmSend = vi.fn(); +const schedulerSend = vi.fn(); + +vi.mock('@aws-sdk/client-ssm', async (importOriginal) => ({ + ...(await importOriginal()), + SSMClient: class { + send = (cmd: unknown) => ssmSend(cmd); + }, +})); +vi.mock('@aws-sdk/client-scheduler', async (importOriginal) => ({ + ...(await importOriginal()), + SchedulerClient: class { + send = (cmd: unknown) => schedulerSend(cmd); + }, +})); + +let handler: (event: LambdaFunctionURLEvent) => Promise<{ statusCode: number; body: string }>; + +beforeAll(async () => { + Object.assign(process.env, LAMBDA_ENV); + handler = (await import('../lambda/schedule/index')).handler as typeof handler; +}); + +function event(method: string, env: string | undefined, body?: unknown): LambdaFunctionURLEvent { + return { + queryStringParameters: env ? { env } : {}, + requestContext: { http: { method } }, + body: body === undefined ? undefined : JSON.stringify(body), + isBase64Encoded: false, + } as unknown as LambdaFunctionURLEvent; +} + +/** The commands sent to a client, as [constructor name, input]. */ +function sent(fn: typeof ssmSend): Array<[string, Record]> { + return fn.mock.calls.map(([cmd]) => [cmd.constructor.name, cmd.input]); +} + +const OFFICE = [ + { action: 'start', cron: '0 8 * * 1-5', timezone: 'Europe/London' }, + { action: 'stop', cron: '0 18 * * 1-5', timezone: 'Europe/London' }, +]; + +beforeEach(() => { + vi.clearAllMocks(); + ssmSend.mockResolvedValue({}); + schedulerSend.mockImplementation(async (cmd: { constructor: { name: string } }) => + cmd.constructor.name === 'ListSchedulesCommand' ? { Schedules: [] } : {}, + ); +}); + +describe('PUT', () => { + it('stores the list and creates one schedule per entry', async () => { + const res = await handler(event('PUT', 'dev', { schedules: OFFICE })); + expect(res.statusCode).toBe(200); + expect(JSON.parse(res.body).schedules).toEqual(OFFICE); + + const [put] = sent(ssmSend); + expect(put[0]).toBe('PutParameterCommand'); + expect(put[1].Name).toBe(schedulesParam('dev')); + expect(JSON.parse(put[1].Value)).toEqual(OFFICE); + + const creates = sent(schedulerSend).filter(([n]) => n === 'CreateScheduleCommand'); + expect(creates).toHaveLength(2); + const prefix = schedulerNamePrefix('dev'); + expect(creates[0][1]).toMatchObject({ + Name: `${prefix}1`, + GroupName: 'test-group', + ScheduleExpression: 'cron(0 8 ? * MON,TUE,WED,THU,FRI *)', + ScheduleExpressionTimezone: 'Europe/London', + Target: { Arn: LAMBDA_ENV.START_FN_ARN, RoleArn: LAMBDA_ENV.SCHEDULER_ROLE_ARN }, + }); + expect(JSON.parse(creates[0][1].Target.Input)).toEqual({ + source: 'spinloop.schedule', + action: 'start', + environment: 'dev', + }); + expect(creates[1][1].Target.Arn).toBe(LAMBDA_ENV.STOP_FN_ARN); + }); + + it('updates schedules that exist and deletes the ones no longer listed', async () => { + const prefix = schedulerNamePrefix('dev'); + schedulerSend.mockImplementation(async (cmd: { constructor: { name: string } }) => + cmd.constructor.name === 'ListSchedulesCommand' + ? { Schedules: [{ Name: `${prefix}1` }, { Name: `${prefix}2` }, { Name: `${prefix}3` }] } + : {}, + ); + const res = await handler(event('PUT', 'dev', { schedules: [OFFICE[0]] })); + expect(res.statusCode).toBe(200); + + const calls = sent(schedulerSend); + expect(calls.filter(([n]) => n === 'UpdateScheduleCommand').map(([, i]) => i.Name)).toEqual([`${prefix}1`]); + expect(calls.filter(([n]) => n === 'CreateScheduleCommand')).toHaveLength(0); + expect(calls.filter(([n]) => n === 'DeleteScheduleCommand').map(([, i]) => i.Name)).toEqual([ + `${prefix}2`, + `${prefix}3`, + ]); + }); + + it('lists only the named environment’s schedules, so others are untouched', async () => { + await handler(event('PUT', 'dev', { schedules: OFFICE })); + const [list] = sent(schedulerSend); + expect(list[0]).toBe('ListSchedulesCommand'); + expect(list[1].NamePrefix).toBe(schedulerNamePrefix('dev')); + expect(list[1].NamePrefix).not.toBe(schedulerNamePrefix('prod')); + }); + + it('refuses an invalid expression and writes nothing', async () => { + const res = await handler( + event('PUT', 'dev', { schedules: [{ action: 'start', cron: 'every day' }] }), + ); + expect(res.statusCode).toBe(400); + expect(JSON.parse(res.body).error).toContain('"every day"'); + expect(ssmSend).not.toHaveBeenCalled(); + expect(schedulerSend).not.toHaveBeenCalled(); + }); + + it('refuses a body that is not JSON', async () => { + const bad = { ...event('PUT', 'dev'), body: '{nope' } as LambdaFunctionURLEvent; + const res = await handler(bad); + expect(res.statusCode).toBe(400); + expect(ssmSend).not.toHaveBeenCalled(); + }); + + it('treats an empty list as a clear', async () => { + const res = await handler(event('PUT', 'dev', { schedules: [] })); + expect(res.statusCode).toBe(200); + expect(sent(ssmSend)[0][0]).toBe('DeleteParameterCommand'); + }); +}); + +describe('DELETE', () => { + it('removes the stored list and every schedule of the environment', async () => { + const prefix = schedulerNamePrefix('dev'); + schedulerSend.mockImplementation(async (cmd: { constructor: { name: string } }) => + cmd.constructor.name === 'ListSchedulesCommand' ? { Schedules: [{ Name: `${prefix}1` }] } : {}, + ); + const res = await handler(event('DELETE', 'dev')); + expect(res.statusCode).toBe(200); + expect(JSON.parse(res.body).schedules).toEqual([]); + expect(sent(ssmSend)[0][0]).toBe('DeleteParameterCommand'); + expect(sent(schedulerSend).filter(([n]) => n === 'DeleteScheduleCommand')).toHaveLength(1); + }); + + it('succeeds when nothing was stored', async () => { + ssmSend.mockRejectedValue(Object.assign(new Error('missing'), { name: 'ParameterNotFound' })); + const res = await handler(event('DELETE', 'dev')); + expect(res.statusCode).toBe(200); + }); +}); + +describe('GET', () => { + it('returns the stored list and the next run of each action', async () => { + ssmSend.mockResolvedValue({ Parameter: { Value: JSON.stringify(OFFICE) } }); + const res = await handler(event('GET', 'dev')); + expect(res.statusCode).toBe(200); + const body = JSON.parse(res.body); + expect(body.schedules).toEqual(OFFICE); + expect(Date.parse(body.next.start)).toBeGreaterThan(Date.now()); + expect(Date.parse(body.next.stop)).toBeGreaterThan(Date.now()); + }); + + it('returns an empty list with no next runs when nothing is stored', async () => { + ssmSend.mockRejectedValue(Object.assign(new Error('missing'), { name: 'ParameterNotFound' })); + const res = await handler(event('GET', 'dev')); + expect(JSON.parse(res.body)).toMatchObject({ schedules: [], next: { start: null, stop: null } }); + }); +}); + +describe('request checks', () => { + it('requires an environment', async () => { + expect((await handler(event('GET', undefined))).statusCode).toBe(400); + }); + + it('rejects other methods', async () => { + expect((await handler(event('POST', 'dev'))).statusCode).toBe(405); + }); +}); diff --git a/remote/test/schedule-start.test.ts b/remote/test/schedule-start.test.ts new file mode 100644 index 00000000..c4b0f6ab --- /dev/null +++ b/remote/test/schedule-start.test.ts @@ -0,0 +1,141 @@ +import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { Context, LambdaFunctionURLResult } from 'aws-lambda'; +import { DAEMON_STATUS_CMD } from '../lambda/shared/daemon'; +import type { ScheduledRunEvent } from '../lambda/shared/schedules'; + +// A `start` schedule firing reaches the same wake as an on-demand start: a +// stopped instance is re-woken, a running one is left alone. All AWS calls are +// stubbed. + +const LAMBDA_ENV = { + TAG_KEY: 'cloud-vm-llm:managed', + TAG_VALUE: 'true', + ENGINE_PORT: '8000', + AMI_ROLE_TAG_KEY: 'cloud-vm-llm:role', + AMI_ROLE_TAG_VALUE: 'runtime-ami', + AMI_RUNNER_TAG_KEY: 'cloud-vm-llm:runner', + INSTANCE_TYPE: 'g6e.xlarge', + SUBNET_IDS: 'subnet-test', + INSTANCE_PROFILE_ARN: 'arn:aws:iam::0:instance-profile/test', + WEIGHTS_BUCKET: 'test-bucket', + MAX_CONCURRENT_SEEDS: '2', + AWS_REGION: 'us-east-1', + BOOT_LOG_GROUP: '/test/boot', + LLAMACPP_LOG_GROUP: '/test/llamacpp', + VLLM_LOG_GROUP: '/test/vllm', +}; + +const findManagedInstance = vi.fn(); +const getInstance = vi.fn(); +const startEngineDaemon = vi.fn(); +const startInstance = vi.fn(); +const runInstance = vi.fn(); +const tagInstance = vi.fn(); +const isSsmAgentOnline = vi.fn(); +const runShellCommand = vi.fn(); +const readDeployConfig = vi.fn(); +const findEnvEip = vi.fn(); +const findEnvSecurityGroup = vi.fn(); +const readEnvApiKey = vi.fn(); + +vi.mock('../lambda/shared/aws', async (importOriginal) => ({ + ...(await importOriginal()), + findManagedInstance: (...args: unknown[]) => findManagedInstance(...args), + getInstance: (...args: unknown[]) => getInstance(...args), + startEngineDaemon: (...args: unknown[]) => startEngineDaemon(...args), + startInstance: (...args: unknown[]) => startInstance(...args), + runInstance: (...args: unknown[]) => runInstance(...args), + tagInstance: (...args: unknown[]) => tagInstance(...args), + isSsmAgentOnline: (...args: unknown[]) => isSsmAgentOnline(...args), + runShellCommand: (...args: unknown[]) => runShellCommand(...args), + readDeployConfig: (...args: unknown[]) => readDeployConfig(...args), +})); + +vi.mock('../lambda/shared/environments', async (importOriginal) => ({ + ...(await importOriginal()), + findEnvEip: (...args: unknown[]) => findEnvEip(...args), + findEnvSecurityGroup: (...args: unknown[]) => findEnvSecurityGroup(...args), + readEnvApiKey: (...args: unknown[]) => readEnvApiKey(...args), +})); + +vi.mock('../lambda/shared/seed', () => ({ weightsPresent: async () => true })); + +let handler: (event: ScheduledRunEvent, context: Context) => Promise; + +beforeAll(async () => { + Object.assign(process.env, LAMBDA_ENV); + ({ handler } = await import('../lambda/start/index')); +}); + +const context = { getRemainingTimeInMillis: () => 600_000 } as unknown as Context; + +function runEvent(action: 'start' | 'stop'): ScheduledRunEvent { + return { source: 'spinloop.schedule', action, environment: 'dev' }; +} + +function structured(result: LambdaFunctionURLResult): { statusCode: number; body: string } { + return result as { statusCode: number; body: string }; +} + +beforeEach(() => { + vi.clearAllMocks(); + readDeployConfig.mockResolvedValue({ + runner: 'llamacpp', + modelId: 'org/model', + quant: 'Q4_K_M', + weightsPrefix: 'llamacpp/org/model/Q4_K_M', + contextSize: 32768, + servedModelName: 'friendly', + serveArgs: [], + companions: {}, + }); + findEnvEip.mockResolvedValue({ publicIp: '198.51.100.7', allocationId: 'eipalloc-test' }); + findEnvSecurityGroup.mockResolvedValue('sg-test'); + readEnvApiKey.mockResolvedValue('sk-test'); + isSsmAgentOnline.mockResolvedValue(true); + runShellCommand.mockImplementation((_instanceId: string, command: string) => + command === DAEMON_STATUS_CMD + ? Promise.resolve({ status: 'Success', stdout: JSON.stringify({ state: 'stopped' }) }) + : Promise.resolve({ status: 'Success', stdout: '200' }), + ); + startEngineDaemon.mockResolvedValue(true); +}); + +describe('a scheduled start', () => { + it('re-wakes a stopped instance, with no retention deadline', async () => { + findManagedInstance.mockResolvedValue({ instanceId: 'i-off', state: 'stopped' }); + getInstance.mockResolvedValue({ instanceId: 'i-off', state: 'running', launchTime: new Date() }); + + const result = await handler(runEvent('start'), context); + expect(structured(result).statusCode).toBe(200); + expect(JSON.parse(structured(result).body).state).toBe('ready'); + expect(startInstance).toHaveBeenCalledWith('i-off'); + expect(tagInstance).not.toHaveBeenCalledWith('i-off', 'Retain-Until', expect.anything()); + }); + + it('leaves a running instance alone', async () => { + findManagedInstance.mockResolvedValue({ instanceId: 'i-run', state: 'running' }); + getInstance.mockResolvedValue({ instanceId: 'i-run', state: 'running', launchTime: new Date() }); + + const result = await handler(runEvent('start'), context); + expect(structured(result).statusCode).toBe(200); + expect(startInstance).not.toHaveBeenCalled(); + expect(runInstance).not.toHaveBeenCalled(); + }); + + it('reports an unconfigured environment without launching anything', async () => { + readDeployConfig.mockRejectedValue(new Error('unconfigured')); + findManagedInstance.mockResolvedValue(undefined); + + const result = await handler(runEvent('start'), context); + expect(structured(result).statusCode).toBe(503); + expect(runInstance).not.toHaveBeenCalled(); + expect(startInstance).not.toHaveBeenCalled(); + }); + + it('ignores an event whose action is not start', async () => { + const result = await handler(runEvent('stop'), context); + expect(JSON.parse(structured(result).body).state).toBe('ignored'); + expect(findManagedInstance).not.toHaveBeenCalled(); + }); +}); diff --git a/remote/test/schedule-stop.test.ts b/remote/test/schedule-stop.test.ts new file mode 100644 index 00000000..b96f3233 --- /dev/null +++ b/remote/test/schedule-stop.test.ts @@ -0,0 +1,115 @@ +import { beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { ScheduledRunEvent } from '../lambda/shared/schedules'; + +// A `stop` schedule firing pauses the running instance, never terminates it, +// and does nothing when there is no running instance or the instance is still +// under a retention deadline. All AWS calls are stubbed. + +const LAMBDA_ENV = { + TAG_KEY: 'cloud-vm-llm:managed', + TAG_VALUE: 'true', + IDLE_THRESHOLD_MINUTES: '15', + GRACE_PERIOD_MINUTES: '10', + MAX_RUNTIME_MINUTES: '240', + STOP_RETENTION_MINUTES: '60', + MAX_SEED_MINUTES: '60', + SEED_STALL_MINUTES: '10', +}; + +const findManagedInstance = vi.fn(); +const stopEngineDaemon = vi.fn(); +const stopInstance = vi.fn(); +const terminateInstance = vi.fn(); +const tagInstance = vi.fn(); + +vi.mock('../lambda/shared/aws', async (importOriginal) => ({ + ...(await importOriginal()), + findManagedInstance: (...args: unknown[]) => findManagedInstance(...args), + stopEngineDaemon: (...args: unknown[]) => stopEngineDaemon(...args), + stopInstance: (...args: unknown[]) => stopInstance(...args), + terminateInstance: (...args: unknown[]) => terminateInstance(...args), + tagInstance: (...args: unknown[]) => tagInstance(...args), +})); + +let handler: (event: ScheduledRunEvent) => Promise; + +beforeAll(async () => { + Object.assign(process.env, LAMBDA_ENV); + handler = (await import('../lambda/stop/index')).handler as typeof handler; +}); + +function runEvent(action: 'start' | 'stop' = 'stop'): ScheduledRunEvent { + return { source: 'spinloop.schedule', action, environment: 'dev' }; +} + +beforeEach(() => { + vi.clearAllMocks(); + stopEngineDaemon.mockResolvedValue(undefined); +}); + +describe('a scheduled stop', () => { + it('pauses a running instance without terminating it', async () => { + findManagedInstance.mockResolvedValue({ instanceId: 'i-run', state: 'running' }); + + await handler(runEvent()); + + expect(findManagedInstance).toHaveBeenCalledWith('cloud-vm-llm:managed', 'true', [ + { Name: 'tag:cloud-vm-llm:env', Values: ['dev'] }, + ]); + expect(tagInstance).toHaveBeenCalledWith('i-run', 'Stopped-At', expect.any(String)); + expect(stopEngineDaemon).toHaveBeenCalledWith('i-run'); + expect(stopInstance).toHaveBeenCalledWith('i-run'); + expect(terminateInstance).not.toHaveBeenCalled(); + }); + + it('pauses an instance whose retention deadline has passed', async () => { + findManagedInstance.mockResolvedValue({ + instanceId: 'i-run', + state: 'running', + retainUntil: new Date(Date.now() - 60_000), + }); + + await handler(runEvent()); + + expect(stopInstance).toHaveBeenCalledWith('i-run'); + }); + + it('skips an instance that is still retained', async () => { + findManagedInstance.mockResolvedValue({ + instanceId: 'i-run', + state: 'running', + retainUntil: new Date(Date.now() + 3_600_000), + }); + + await handler(runEvent()); + + expect(stopInstance).not.toHaveBeenCalled(); + expect(terminateInstance).not.toHaveBeenCalled(); + expect(tagInstance).not.toHaveBeenCalled(); + }); + + it('does nothing when there is no instance', async () => { + findManagedInstance.mockResolvedValue(undefined); + + await expect(handler(runEvent())).resolves.toBeUndefined(); + + expect(stopInstance).not.toHaveBeenCalled(); + expect(terminateInstance).not.toHaveBeenCalled(); + }); + + it.each(['stopped', 'stopping', 'pending'])('does nothing when the instance is %s', async (state) => { + findManagedInstance.mockResolvedValue({ instanceId: 'i-x', state }); + + await handler(runEvent()); + + expect(stopInstance).not.toHaveBeenCalled(); + expect(terminateInstance).not.toHaveBeenCalled(); + }); + + it('ignores an event whose action is not stop', async () => { + await handler(runEvent('start')); + + expect(findManagedInstance).not.toHaveBeenCalled(); + expect(stopInstance).not.toHaveBeenCalled(); + }); +}); diff --git a/remote/test/schedules.test.ts b/remote/test/schedules.test.ts new file mode 100644 index 00000000..71cd11f4 --- /dev/null +++ b/remote/test/schedules.test.ts @@ -0,0 +1,154 @@ +import { describe, expect, it } from 'vitest'; +import { + isScheduledRunEvent, + nextRun, + nextRuns, + parseCron, + ScheduleError, + schedulerNamePrefix, + toSchedulerCron, + validateSchedules, +} from '../lambda/shared/schedules'; + +describe('toSchedulerCron', () => { + it.each([ + ['0 8 * * 1-5', 'cron(0 8 ? * MON,TUE,WED,THU,FRI *)'], + ['30 18 * * *', 'cron(30 18 * * ? *)'], + ['0 9 1 * *', 'cron(0 9 1 * ? *)'], + ['*/15 * * * *', 'cron(*/15 * * * ? *)'], + ['0 0 * * 0', 'cron(0 0 ? * SUN *)'], + ['0 0 * * 7', 'cron(0 0 ? * SUN *)'], + ['0 8 * 1,6 6,0', 'cron(0 8 ? 1,6 SUN,SAT *)'], + ])('converts %s', (input, want) => { + expect(toSchedulerCron(input)).toBe(want); + }); + + it('rejects a day of month and a day of week together', () => { + expect(() => toSchedulerCron('0 8 1 * 1')).toThrow(/not both/); + }); +}); + +describe('parseCron', () => { + it.each([ + '', + '* * * *', + '* * * * * *', + '60 * * * *', + '* 24 * * *', + '* * 0 * *', + '* * * 13 *', + '* * * * 8', + 'a * * * *', + '*/0 * * * *', + '5-1 * * * *', + '* * * * MON', + ])('rejects %j and names it', (expr) => { + expect(() => parseCron(expr)).toThrow(ScheduleError); + expect(() => parseCron(expr)).toThrow(JSON.stringify(expr)); + }); + + it('expands lists, ranges and steps', () => { + const c = parseCron('0,30 8-10 * 1-3 */2'); + expect(c.minutes).toEqual([0, 30]); + expect(c.hours).toEqual([8, 9, 10]); + expect(c.months).toEqual([1, 2, 3]); + expect(c.daysOfWeek).toEqual([0, 2, 4, 6]); + }); +}); + +describe('validateSchedules', () => { + it('fills the default time zone and normalises spacing', () => { + expect(validateSchedules([{ action: 'start', cron: '0 8 * * 1-5' }])).toEqual([ + { action: 'start', cron: '0 8 * * 1-5', timezone: 'UTC' }, + ]); + }); + + it('accepts an empty list', () => { + expect(validateSchedules([])).toEqual([]); + }); + + it('rejects a non-list, an unknown action, a bad zone and a bad cron', () => { + expect(() => validateSchedules({})).toThrow(/must be a list/); + expect(() => validateSchedules([{ action: 'pause', cron: '0 8 * * *' }])).toThrow(/unknown action/); + expect(() => + validateSchedules([{ action: 'start', cron: '0 8 * * *', timezone: 'Mars/Olympus' }]), + ).toThrow(/unknown time zone/); + expect(() => validateSchedules([{ action: 'start', cron: 'nope' }])).toThrow(/"nope"/); + }); + + it('rejects too many schedules', () => { + const many = Array.from({ length: 21 }, () => ({ action: 'start', cron: '0 8 * * *' })); + expect(() => validateSchedules(many)).toThrow(/at most/); + }); +}); + +describe('nextRun', () => { + it('follows local time across daylight saving', () => { + const summer = nextRun('0 8 * * *', 'Europe/London', new Date('2026-07-01T12:00:00Z')); + const winter = nextRun('0 8 * * *', 'Europe/London', new Date('2026-12-01T12:00:00Z')); + expect(summer?.toISOString()).toBe('2026-07-02T07:00:00.000Z'); + expect(winter?.toISOString()).toBe('2026-12-02T08:00:00.000Z'); + }); + + it('skips weekends for a weekday schedule', () => { + // Friday 2026-10-09 evening; the next 08:00 weekday is Monday. + const next = nextRun('0 8 * * 1-5', 'UTC', new Date('2026-10-09T18:00:00Z')); + expect(next?.toISOString()).toBe('2026-10-12T08:00:00.000Z'); + }); + + it('returns only instants strictly after the reference', () => { + const next = nextRun('0 8 * * *', 'UTC', new Date('2026-10-05T08:00:00Z')); + expect(next?.toISOString()).toBe('2026-10-06T08:00:00.000Z'); + }); + + it('skips a time the clocks jump over', () => { + // 01:30 does not exist in London on 2026-03-29; the next one is the day after. + const next = nextRun('30 1 * * *', 'Europe/London', new Date('2026-03-28T12:00:00Z')); + expect(next?.toISOString()).toBe('2026-03-30T00:30:00.000Z'); + }); + + it('uses the first of a repeated time when the clocks go back', () => { + // 01:30 happens twice in London on 2026-10-25; the first is still BST. + const next = nextRun('30 1 * * *', 'Europe/London', new Date('2026-10-24T12:00:00Z')); + expect(next?.toISOString()).toBe('2026-10-25T00:30:00.000Z'); + }); + + it('handles a day of month', () => { + const next = nextRun('0 9 15 * *', 'UTC', new Date('2026-10-20T00:00:00Z')); + expect(next?.toISOString()).toBe('2026-11-15T09:00:00.000Z'); + }); + + it('returns null for a date that never occurs', () => { + expect(nextRun('0 0 31 2 *', 'UTC', new Date('2026-01-01T00:00:00Z'))).toBeNull(); + }); +}); + +describe('nextRuns', () => { + it('reports the earliest of each action, or null when there is none', () => { + const after = new Date('2026-10-05T10:00:00Z'); + const runs = nextRuns( + [ + { action: 'start', cron: '0 8 * * *', timezone: 'UTC' }, + { action: 'start', cron: '0 12 * * *', timezone: 'UTC' }, + ], + after, + ); + expect(runs).toEqual({ start: '2026-10-05T12:00:00.000Z', stop: null }); + }); +}); + +describe('identifiers', () => { + it('derives a short, stable Scheduler name prefix per environment', () => { + const prefix = schedulerNamePrefix('dev'); + expect(prefix).toBe(schedulerNamePrefix('dev')); + expect(prefix).not.toBe(schedulerNamePrefix('prod')); + expect(`${prefix}19`.length).toBeLessThanOrEqual(64); + expect(schedulerNamePrefix('a'.repeat(64)).length).toBeLessThan(30); + }); + + it('recognises a scheduled run event', () => { + expect(isScheduledRunEvent({ source: 'spinloop.schedule', action: 'start', environment: 'dev' })).toBe(true); + expect(isScheduledRunEvent({ source: 'aws.events' })).toBe(false); + expect(isScheduledRunEvent(null)).toBe(false); + }); +}); diff --git a/remote/test/stack.test.ts b/remote/test/stack.test.ts index 9b7d27b1..1b080a19 100644 --- a/remote/test/stack.test.ts +++ b/remote/test/stack.test.ts @@ -135,10 +135,10 @@ describe('LlmStack (control plane)', () => { expect(groups[0].Properties.SecurityGroupIngress).toBeUndefined(); }); - it('creates the start, stop, deploy, stats, env, seed and update Lambdas with IAM-authenticated function URLs', () => { - template.resourceCountIs('AWS::Lambda::Function', 7); + it('creates the start, stop, deploy, stats, env, seed, update and schedule Lambdas with IAM-authenticated function URLs', () => { + template.resourceCountIs('AWS::Lambda::Function', 8); const urls = template.findResources('AWS::Lambda::Url'); - expect(Object.keys(urls)).toHaveLength(7); + expect(Object.keys(urls)).toHaveLength(8); for (const url of Object.values(urls)) { expect(url.Properties.AuthType).toBe('AWS_IAM'); } @@ -153,7 +153,7 @@ describe('LlmStack (control plane)', () => { expect(cliStatements.length).toBeGreaterThan(0); }); - it('grants the CLI user invoke-url permission on all seven control functions, and nothing else in lambda', () => { + it('grants the CLI user invoke-url permission on all eight control functions, and nothing else in lambda', () => { const actionsOf = (s: Statement): string[] => [s.Action].flat(); // The invoke-url grant takes the form grantInvokeUrl renders — // lambda:InvokeFunctionUrl (AWS_IAM auth condition) plus @@ -163,7 +163,7 @@ describe('LlmStack (control plane)', () => { const urlFunctionIds = new Set( Object.values(urls).map((u) => u.Properties.TargetFunctionArn['Fn::GetAtt'][0]), ); - expect(urlFunctionIds).toHaveLength(7); + expect(urlFunctionIds).toHaveLength(8); // The grant names the functions' ARNs through each URL resource's // FunctionArn attribute; resolve that back to the backing function so the // assertion is on the functions, not the reference path. @@ -329,13 +329,57 @@ describe('LlmStack (control plane)', () => { [s.Action].flat().includes('iam:PassRole'), ); expect(passRole.length).toBeGreaterThan(0); - // A wildcard PassRole would let a caller hand EC2 any role in the account. + // A wildcard PassRole would let a caller hand EC2 (or Scheduler) any role in the account. for (const statement of passRole) { expect(JSON.stringify(statement.Resource)).not.toBe('"*"'); - expect(JSON.stringify(statement.Condition)).toContain('ec2.amazonaws.com'); + expect(JSON.stringify(statement.Condition)).toMatch(/ec2\.amazonaws\.com|scheduler\.amazonaws\.com/); } }); + it('runs schedules from a Scheduler group, as a role that can only invoke the start and stop Lambdas', () => { + template.resourceCountIs('AWS::Scheduler::ScheduleGroup', 1); + const roles = template.findResources('AWS::IAM::Role') as Record; + const [roleId, role] = Object.entries(roles).find(([, r]) => + JSON.stringify(r.Properties.AssumeRolePolicyDocument).includes('scheduler.amazonaws.com'), + )!; + expect(role).toBeDefined(); + const policies = template.findResources('AWS::IAM::Policy') as Record; + const granted = Object.values(policies) + .filter((p) => (p.Properties.Roles as { Ref: string }[]).some((r) => r.Ref === roleId)) + .flatMap((p) => p.Properties.PolicyDocument.Statement as Statement[]); + // Only lambda:InvokeFunction (one statement per target function), and never on a wildcard. + expect(new Set(granted.flatMap((s) => [s.Action].flat()))).toEqual(new Set(['lambda:InvokeFunction'])); + expect(granted.length).toBeGreaterThanOrEqual(2); + for (const s of granted) { + expect(JSON.stringify(s.Resource)).not.toContain('"*"'); + } + }); + + it('gives the schedule Lambda its group, role and target functions, and scopes its Scheduler writes to the group', () => { + const fns = template.findResources('AWS::Lambda::Function') as Record; + const schedule = Object.values(fns).find((f) => + f.Properties.Environment?.Variables?.SCHEDULE_GROUP !== undefined, + ); + expect(schedule).toBeDefined(); + const vars = Object.keys(schedule.Properties.Environment.Variables).sort(); + expect(vars).toEqual(['SCHEDULER_ROLE_ARN', 'SCHEDULE_GROUP', 'START_FN_ARN', 'STOP_FN_ARN'].sort()); + + const statements = allPolicyStatements(template); + const writes = statements.filter((s) => [s.Action].flat().includes('scheduler:CreateSchedule')); + expect(writes).toHaveLength(1); + expect([writes[0].Action].flat().sort()).toEqual([ + 'scheduler:CreateSchedule', + 'scheduler:DeleteSchedule', + 'scheduler:UpdateSchedule', + ]); + expect(JSON.stringify(writes[0].Resource)).not.toBe('"*"'); + expect(JSON.stringify(writes[0].Resource)).toContain(':schedule/'); + }); + + it('keeps the idle sweep as the only classic rule: user schedules live in Scheduler', () => { + template.resourceCountIs('AWS::Events::Rule', 1); + }); + it('schedules the idle sweep every 5 minutes', () => { template.hasResourceProperties('AWS::Events::Rule', { ScheduleExpression: 'rate(5 minutes)' }); }); @@ -523,7 +567,7 @@ describe('LlmStack (control plane)', () => { // The regression this guards: without an explicit log group, Lambda // auto-creates one with no retention policy at all — every invocation of // every control Lambda kept forever. - for (const name of ['start', 'stop', 'deploy', 'seed', 'stats', 'env', 'update']) { + for (const name of ['start', 'stop', 'deploy', 'seed', 'stats', 'env', 'update', 'schedule']) { template.hasResourceProperties('AWS::Logs::LogGroup', { LogGroupName: `/cloud-vm-llm/lambda/${name}`, RetentionInDays: 3, @@ -535,8 +579,8 @@ describe('LlmStack (control plane)', () => { for (const fn of Object.values(fns)) { expect(fn.Properties.LoggingConfig?.LogGroup).toBeDefined(); } - // 4 instance-facing groups (llamacpp, vllm, boot, seed) + 7 Lambda groups. - template.resourceCountIs('AWS::Logs::LogGroup', 11); + // 4 instance-facing groups (llamacpp, vllm, boot, seed) + 8 Lambda groups. + template.resourceCountIs('AWS::Logs::LogGroup', 12); }); it('gives the seed Lambda its own function, cap and concurrency bound', () => { @@ -636,6 +680,7 @@ describe('LlmStack (control plane)', () => { 'DeployUrl', 'StatsUrl', 'EnvUrl', + 'ScheduleUrl', 'WeightsBucket', 'VpcId', 'SeedInstanceProfileArn', From a58ade501465c87b69211156c7eaf50772a2cbd8 Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 20:32:08 +0100 Subject: [PATCH 3/8] test(remote): cover schedule validation edges and failures after the list is stored --- cmd/spinloop/remote_schedule_test.go | 41 ++++++++++++++++++++++++++++ remote/test/schedule-api.test.ts | 35 ++++++++++++++++++++++-- remote/test/schedules.test.ts | 36 ++++++++++++++++++++++++ 3 files changed, 109 insertions(+), 3 deletions(-) diff --git a/cmd/spinloop/remote_schedule_test.go b/cmd/spinloop/remote_schedule_test.go index 482f7fa6..a5df3f70 100644 --- a/cmd/spinloop/remote_schedule_test.go +++ b/cmd/spinloop/remote_schedule_test.go @@ -202,3 +202,44 @@ func TestRemoteSchedule_NeedsAnEnvironment(t *testing.T) { t.Errorf("expected the no-environment error, got %v", err) } } + +func TestRemoteSchedule_ClearAndShowReportControlPlaneFailures(t *testing.T) { + scheduleFixture(t, http.StatusInternalServerError, `{"error":"scheduler unavailable"}`) + for _, verb := range []string{"clear", "show"} { + err := cmdRemoteSchedule([]string{verb, "--env", "default"}) + if err == nil || !strings.Contains(err.Error(), "scheduler unavailable") { + t.Errorf("%s: expected the control plane's reason, got %v", verb, err) + } + } +} + +func TestRemoteSchedule_SetFailsWithoutAnEnvironmentBeforeSendingAnything(t *testing.T) { + isolateConfig(t) + err := cmdRemoteSchedule([]string{"set", "--start", "0 8 * * *"}) + if err == nil || !strings.Contains(err.Error(), "no environment named") { + t.Errorf("expected the no-environment error, got %v", err) + } +} + +func TestPrintSchedules_ShowsUTCWhenTheZoneIsEmptyAndPassesAnOddNextRunThrough(t *testing.T) { + var b strings.Builder + list := &remote.ScheduleList{Schedules: []remote.Schedule{{Action: "start", Cron: "0 8 * * *"}}} + list.Next.Start = "tomorrow-ish" + printSchedules(&b, list) + out := b.String() + if !strings.Contains(out, "start 0 8 * * * (UTC)") { + t.Errorf("an empty zone should read as UTC:\n%s", out) + } + if !strings.Contains(out, "next start: tomorrow-ish") { + t.Errorf("an unparseable next run should be shown as sent:\n%s", out) + } +} + +func TestFormatNextRun_ConvertsOffsetsToUTC(t *testing.T) { + if got := formatNextRun("2026-10-06T09:00:00+02:00"); got != "2026-10-06T07:00:00Z" { + t.Errorf("got %q", got) + } + if got := formatNextRun(""); got != "" { + t.Errorf("an empty value should stay empty, got %q", got) + } +} diff --git a/remote/test/schedule-api.test.ts b/remote/test/schedule-api.test.ts index ed563a85..e8813215 100644 --- a/remote/test/schedule-api.test.ts +++ b/remote/test/schedule-api.test.ts @@ -7,9 +7,9 @@ import { schedulerNamePrefix, schedulesParam } from '../lambda/shared/schedules' const LAMBDA_ENV = { SCHEDULE_GROUP: 'test-group', - SCHEDULER_ROLE_ARN: 'arn:aws:iam::000000000000:role/scheduler', - START_FN_ARN: 'arn:aws:lambda:us-east-1:000000000000:function:start', - STOP_FN_ARN: 'arn:aws:lambda:us-east-1:000000000000:function:stop', + SCHEDULER_ROLE_ARN: 'arn:aws:iam::0:role/scheduler', + START_FN_ARN: 'arn:aws:lambda:us-east-1:0:function:start', + STOP_FN_ARN: 'arn:aws:lambda:us-east-1:0:function:stop', }; const ssmSend = vi.fn(); @@ -180,6 +180,35 @@ describe('GET', () => { }); }); +describe('failures after validation', () => { + it('keeps the stored list and surfaces the AWS error, rather than calling it a bad request', async () => { + schedulerSend.mockRejectedValue(Object.assign(new Error('throttled'), { name: 'ThrottlingException' })); + await expect(handler(event('PUT', 'dev', { schedules: OFFICE }))).rejects.toThrow('throttled'); + // The list was stored before the mirror was attempted, so a repeat converges. + expect(sent(ssmSend)[0][0]).toBe('PutParameterCommand'); + }); + + it('leaves the schedules alone when the stored list cannot be removed', async () => { + ssmSend.mockRejectedValue(Object.assign(new Error('denied'), { name: 'AccessDeniedException' })); + await expect(handler(event('DELETE', 'dev'))).rejects.toThrow('denied'); + expect(schedulerSend).not.toHaveBeenCalled(); + }); + + it('removes schedules listed across several Scheduler pages', async () => { + const prefix = schedulerNamePrefix('dev'); + let page = 0; + schedulerSend.mockImplementation(async (cmd: { constructor: { name: string } }) => { + if (cmd.constructor.name !== 'ListSchedulesCommand') return {}; + page += 1; + return page === 1 + ? { Schedules: [{ Name: `${prefix}1` }], NextToken: 't' } + : { Schedules: [{ Name: `${prefix}2` }] }; + }); + await handler(event('DELETE', 'dev')); + expect(sent(schedulerSend).filter(([n]) => n === 'DeleteScheduleCommand')).toHaveLength(2); + }); +}); + describe('request checks', () => { it('requires an environment', async () => { expect((await handler(event('GET', undefined))).statusCode).toBe(400); diff --git a/remote/test/schedules.test.ts b/remote/test/schedules.test.ts index 71cd11f4..576eee4d 100644 --- a/remote/test/schedules.test.ts +++ b/remote/test/schedules.test.ts @@ -137,6 +137,42 @@ describe('nextRuns', () => { }); }); +describe('validateSchedules, malformed entries', () => { + it('rejects an entry that is not an object and a cron that is not a string', () => { + expect(() => validateSchedules(['0 8 * * *'])).toThrow(/schedule 1 must be an object/); + expect(() => validateSchedules([null])).toThrow(/must be an object/); + expect(() => validateSchedules([{ action: 'start', cron: 8 }])).toThrow(/cron must be a string/); + }); + + it('treats an empty time zone as UTC and rejects a non-string one', () => { + expect(validateSchedules([{ action: 'stop', cron: '0 18 * * *', timezone: '' }])[0].timezone).toBe('UTC'); + expect(() => validateSchedules([{ action: 'stop', cron: '0 18 * * *', timezone: 5 }])).toThrow( + /unknown time zone/, + ); + }); + + it('names the schedule that is wrong, by position', () => { + expect(() => + validateSchedules([ + { action: 'start', cron: '0 8 * * *' }, + { action: 'stop', cron: '0 18 * * *', timezone: 'Nowhere/Land' }, + ]), + ).toThrow(/schedule 2/); + }); +}); + +describe('parseCron, field edges', () => { + it.each(['1,,2 * * * *', '1/2/3 * * * *', '*/x * * * *', '1- * * * *'])('rejects %j', (expr) => { + expect(() => parseCron(expr)).toThrow(ScheduleError); + }); + + it('accepts a step on a start value, and reads the full day-of-month range as unrestricted', () => { + expect(parseCron('5/20 * * * *').minutes).toEqual([5, 25, 45]); + expect(parseCron('0 0 1-31 * 1').anyDayOfMonth).toBe(true); + expect(toSchedulerCron('0 0 1-31 * 1')).toBe('cron(0 0 ? * MON *)'); + }); +}); + describe('identifiers', () => { it('derives a short, stable Scheduler name prefix per environment', () => { const prefix = schedulerNamePrefix('dev'); From 731263de6f1334ec6d4b6fb879652df67606308b Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 20:32:08 +0100 Subject: [PATCH 4/8] docs(remote): document scheduled start and stop --- README.md | 5 ++- docs/commands/remote.md | 41 ++++++++++++++++++- docs/guides/remote.md | 10 +++++ docs/maintainer/internals.md | 4 +- .../scheduled-remote-start-stop/tasks.md | 6 +-- remote/README.md | 8 +++- 6 files changed, 67 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5fbeef99..4fd37d39 100644 --- a/README.md +++ b/README.md @@ -302,7 +302,7 @@ spinloop harness open [] [-H ] [--spinloop[=]] [args...] # launch the harness (a leading Spinloop or alias is # applied first) spinloop completion # tab completion (bash, zsh, powershell) -spinloop remote [path] +spinloop remote [path] # control the remote GPU inference instance # (bootstrap does the once-per-account setup; # bake bakes the runner AMI(s) it launches from; @@ -310,6 +310,7 @@ spinloop remote # what did it say? (readable after it's gone) spinloop remote pause # stop it now; a later start re-wakes it spinloop remote restart # fresh engine, same address: stop it and wake it again spinloop remote keep 4h # prevent the idle sweep from stopping it for 4 hours +spinloop remote schedule # start and stop it on cron schedules (set, show, clear) spinloop remote stop # terminate it now, rather than waiting for the idle timer ``` @@ -95,7 +96,7 @@ spinloop remote stop --env qwen3.6-27b # done for good: terminate it ``` Every command that acts on an endpoint — `start`, `status`, `metrics`, `logs`, -`pause`, `restart`, `keep`, `stop` — selects it with the same `--env ` +`pause`, `restart`, `keep`, `schedule`, `stop` — selects it with the same `--env ` flag; with no flag they use the `default` environment. `start` prints its progress on stderr, and the export lines for `eval` only with `--print-env`, so a plain `start` leaves stdout empty for other uses. @@ -308,6 +309,44 @@ see how long the instance is protected for. It requires a control plane with the update Lambda (bootstrap with a recent version, or re-bootstrap). +## Starting and stopping on a schedule + +```sh +spinloop remote schedule set --env dev \ + --start "0 8 * * 1-5" --stop "0 18 * * 1-5" --timezone Europe/London +spinloop remote schedule show --env dev # the schedules, and when each action next runs +spinloop remote schedule clear --env dev # remove them all +``` + +A schedule is a five-field cron expression (minute, hour, day of month, month, +day of week) with an action, `start` or `stop`. `set` takes one or more +`--start` and `--stop` flags and a single `--timezone` (an IANA name such as +`Europe/London`, default `UTC`) that applies to all of them. It replaces the +environment's whole list, so what you pass is exactly what runs afterwards. +Expressions are checked before anything changes: a bad one is refused, naming +it, and the earlier schedules carry on. Day of month and day of week cannot +both be set in one expression, and names such as `MON` are not accepted; +use numbers (`0` or `7` is Sunday). + +The control plane stores and runs the schedules, so they fire whether or not +any of your machines is on. Times follow the zone's clock, including daylight +saving: `0 8 * * *` in `Europe/London` is 08:00 local all year. + +- A scheduled start does what `start` does, including the weights check. If the + instance is already running it is left alone. If it cannot start (no GPU + capacity, weights not seeded), the reason is in the start Lambda's log and + nothing retries until the next firing. +- A scheduled stop pauses the instance, as `pause` does, so a later start + re-wakes it. If the instance is being kept (`keep`, `start --keep`) the stop + is skipped. +- `show` prints one line per schedule, then `next start` and `next stop` as UTC + times. An action with no schedule has no line. + +It requires a control plane with the schedule Lambda: a deployment that predates +it fails with a message to re-run `spinloop remote bootstrap`. The URL is the +`schedule_url` in the environment's `remote.json`, or +`SPINLOOP_REMOTE_SCHEDULE_URL`. + ## Restarting the engine ```sh diff --git a/docs/guides/remote.md b/docs/guides/remote.md index 9a4ca5f7..842f3dba 100644 --- a/docs/guides/remote.md +++ b/docs/guides/remote.md @@ -72,6 +72,16 @@ spinloop remote keep 4h --env qwen3.6-27b # the idle sweep won't touch it for 4 spinloop remote restart --env qwen3.6-27b # fresh engine, same address ``` +Work office hours? Let the cloud start and stop it for you: + +```sh +spinloop remote schedule set --env qwen3.6-27b \ + --start "0 8 * * 1-5" --stop "0 18 * * 1-5" --timezone Europe/London +``` + +It's up and warm by the time you sit down, and paused when you leave. See +[`spinloop remote`](../commands/remote.md#starting-and-stopping-on-a-schedule). + ## From another machine The environment is registered per user and per machine, so two machines that diff --git a/docs/maintainer/internals.md b/docs/maintainer/internals.md index 65e4b5a8..4816ac4f 100644 --- a/docs/maintainer/internals.md +++ b/docs/maintainer/internals.md @@ -33,7 +33,9 @@ These are mistakes already made here; each was silent rather than loud, which is **A freshly issued access key is not instantly resolvable.** STS lags the `iam:CreateAccessKey` response by seconds — on one account the store's verification needed six attempts before the key resolved, at roughly 8–10 s propagation. The verify probe in `cmd/spinloop/remote_auth.go` (`verifyNewKey`) retries only when the error message contains `InvalidClientTokenId`, on an exponential backoff from 1 s to a 16 s cap (a ~31 s window over six attempts), and the stderr line says which retry is running. Two things to preserve: the match is a string match because the pinned `service/sts` SDK version has no typed error for that code — if the SDK is upgraded, switch to the typed check — and the retry is exclusive to that code. Any other verification failure, a key that resolves to a different account or lacks a permission, fails at once and deletes the key it created; waiting it out would only delay the deletion. `verifyProbeBackoff` is a seam the tests zero. -**An IAM user's inline policies are capped at 2,048 characters in aggregate.** The control plane's seven functions each take a `grantInvokeUrl` pair — two actions, the auth-type conditions, the function's ARN — and with the log-reading, stack-discovery, pricing and self-service statements the document far exceeds that; the first deploy of the `RemoteCliUser` inline policy failed with `ServiceLimitExceeded`, which CDK does not warn about ahead of time. It is now a stack-owned `AWS::IAM::ManagedPolicy` (`RemoteCliPolicy`, 6,144 cap; the deployed document measures ~2 KB, so the grant list has room to grow). Keep it managed rather than re-inlining it, and keep the iam self-service ARN built from the `AWS::Partition`/`AWS::AccountId` pseudo parameters instead of the user's `Arn`: the policy attaches to that user, so referencing the user from inside it is a dependency cycle. +**An IAM user's inline policies are capped at 2,048 characters in aggregate.** The control plane's eight functions each take a `grantInvokeUrl` pair — two actions, the auth-type conditions, the function's ARN — and with the log-reading, stack-discovery, pricing and self-service statements the document far exceeds that; the first deploy of the `RemoteCliUser` inline policy failed with `ServiceLimitExceeded`, which CDK does not warn about ahead of time. It is now a stack-owned `AWS::IAM::ManagedPolicy` (`RemoteCliPolicy`, 6,144 cap; the deployed document measures ~2 KB, so the grant list has room to grow). Keep it managed rather than re-inlining it, and keep the iam self-service ARN built from the `AWS::Partition`/`AWS::AccountId` pseudo parameters instead of the user's `Arn`: the policy attaches to that user, so referencing the user from inside it is a dependency cycle. + +**Schedules live in SSM, and the EventBridge Scheduler schedules are a copy of that list.** The schedule Lambda writes an environment's list to `/cloud-vm-llm//schedules`, then creates, updates and deletes Scheduler schedules in the stack's schedule group until they match it. Everything that can be refused (cron, zone, action) is checked before the write, so a bad request changes nothing; a Scheduler failure after the write leaves the stored list as intended and a repeat of the same request converges. Scheduler schedule names are `e-<16 hex of sha256(env)>-`, not the environment name: names are capped at 64 characters, an environment name can be that long alone, and the prefix has to be exact so one environment's reconcile never lists another's. The environment travels in each schedule's JSON input instead. Users write five-field cron; Scheduler wants six fields, `?` in the unused day field, and its own weekday numbering (Sunday is 1), so `toSchedulerCron` expands weekdays to names (`MON,TUE`) and refuses an expression that sets both day fields, which classic cron would treat as "either" and Scheduler cannot express. "Next run" is computed by `nextRun` in `lambda/shared/schedules.ts`, not read from Scheduler: it walks calendar days in the schedule's zone, skips a wall-clock time the clocks jump over, and takes the first of a time they repeat. The start and stop Lambdas tell a schedule firing from the idle sweep and from a Function URL call by the event's `source` (`spinloop.schedule`, `aws.events`, none); keep the check for the schedule's source ahead of the others. **A start holds a per-environment lock for its whole run, and it is an SSM parameter, not a conditional write.** `wake()` in the start Lambda takes `/cloud-vm-llm//wake-lock` (created with `Overwrite: false`) after the read-only checks and before the weights check, and releases it in a `finally`. It has to cover the whole poll and not only the launch, because the tag lookup that decides whether to launch lags a launch: a second start that got in after the lock was released could still miss the first one's instance. The value is `{owner, expiresAt}` with the expiry set to the invocation's remaining time plus 30 seconds, so a start killed before its `finally` blocks the environment for no longer than it could have run. A refused start replies 503 `starting` with 15 seconds to retry, which the CLI and gateway already retry. Taking over an expired lock is read, re-read, delete, create-if-absent; SSM has no compare-and-set, so two starts taking over the same abandoned lock within milliseconds can still both proceed. That needs a killed start plus a near-simultaneous pair, and a DynamoDB conditional write would remove it. Releasing needs `ssm:DeleteParameter`, granted on `parameter/cloud-vm-llm/*/wake-lock` only. Stops do not take the lock: a lock held for up to 15 minutes would refuse the stop that gets you out of a hung start, and the stop Lambda's 120 second limit could not wait for it. Instead each polling loop in a start checks the instance state and ends the start, releasing the lock, if the instance is stopped, stopping or terminated. The status read reports the lock without taking it: `starting` when the lock is held and the instance is absent, stopped or pending, and `start_in_progress` in every reply. A start waiting for capacity has released its lock between attempts, so it is not shown. diff --git a/openspec/changes/scheduled-remote-start-stop/tasks.md b/openspec/changes/scheduled-remote-start-stop/tasks.md index 65095935..c6cb30b3 100644 --- a/openspec/changes/scheduled-remote-start-stop/tasks.md +++ b/openspec/changes/scheduled-remote-start-stop/tasks.md @@ -32,6 +32,6 @@ ## 6. Documentation and specs -- [ ] 6.1 Update `docs/commands`, the remote guide and the control-plane HTTP API page -- [ ] 6.2 Add a note to `docs/maintainer/internals.md` on the cron conversion and the Scheduler/SSM split -- [ ] 6.3 Run `go test ./...`, `go vet ./...`, `gofmt`, and `pnpm test` in `remote/` +- [x] 6.1 Update `docs/commands`, the remote guide, the README and `remote/README.md` +- [x] 6.2 Add a note to `docs/maintainer/internals.md` on the cron conversion and the Scheduler/SSM split +- [x] 6.3 Run `go test ./...`, `go vet ./...`, `gofmt`, and `pnpm test` in `remote/` diff --git a/remote/README.md b/remote/README.md index 482be8ff..559f30b9 100644 --- a/remote/README.md +++ b/remote/README.md @@ -70,6 +70,8 @@ spinloop remote stop ───────▶ stop Lambda AWS_IAM auth spinloop remote pause ──────▶ (stop, not terminate) │ engine on :8000 ▲ EventBridge rate(5 min) ─────────┘ (idle sweep: stop, then terminate) ▼ +spinloop remote schedule ─▶ schedule Lambda ─▶ SSM list + EventBridge Scheduler (cron, time zone) + └─ fires start / stop Lambda for the env on each schedule coding agent ── OPENAI_BASE_URL=http://:8000/v1 + api key ──▶ direct HTTP ``` @@ -479,7 +481,11 @@ The Lambdas log every decision to CloudWatch. In the **stop** Lambda's log group, each 5-minute tick prints a JSON line — grep `"mode":"idle"` to see why it kept or terminated the instance (e.g. `"decision":"stop","reason":"idle for 32.9 min"`, or `"reason":"retained until …"` when a `Retain-Until` tag is set). -The **start** Lambda logs the launch AZ and each wake phase. +The **start** Lambda logs the launch AZ and each wake phase. A schedule firing +logs `"mode":"scheduled"`: in the start Lambda's group it carries the wake's +result, and in the stop Lambda's group `"action":"stop"`, `"action":"noop"`, or +`"action":"skip","reason":"retained"` when a `Retain-Until` deadline held the +instance up. ## Security notes From 1d6fd0dace5922621e5245b29c7d7b87f86c6295 Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 21:03:32 +0100 Subject: [PATCH 5/8] docs(openspec): state the purpose of the remote-schedule capability --- .../specs/remote-schedule/spec.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md index f3385fac..e855dd28 100644 --- a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md +++ b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md @@ -1,3 +1,10 @@ +## Purpose + +Let a remote environment start and stop itself on cron schedules, so a team that +works set hours has the model up when it sits down and stops paying for it when +it leaves. The control plane stores and runs the schedules, so they fire with no +machine of the operator's switched on; the CLI sets, shows and clears them. + ## ADDED Requirements ### Requirement: An environment holds a list of schedules From 019db9dcb141a4b023f5f95410ecf97d0b331449 Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Mon, 5 Oct 2026 23:18:03 +0100 Subject: [PATCH 6/8] fix(remote): give the schedule Lambda the control plane version --- remote/lib/llm-stack.ts | 1 + remote/test/control-plane-version.test.ts | 2 +- remote/test/stack.test.ts | 4 +++- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/remote/lib/llm-stack.ts b/remote/lib/llm-stack.ts index 775a3be0..0fa2be2d 100644 --- a/remote/lib/llm-stack.ts +++ b/remote/lib/llm-stack.ts @@ -720,6 +720,7 @@ export class LlmStack extends cdk.Stack { memorySize: 256, logGroup: lambdaLogGroup('ScheduleFnLogGroup', 'schedule'), environment: { + CONTROL_PLANE_VERSION: cfg.controlPlaneVersion, SCHEDULE_GROUP: scheduleGroup.ref, SCHEDULER_ROLE_ARN: schedulerRole.roleArn, START_FN_ARN: startFn.functionArn, diff --git a/remote/test/control-plane-version.test.ts b/remote/test/control-plane-version.test.ts index 9e2e6d03..e7932d47 100644 --- a/remote/test/control-plane-version.test.ts +++ b/remote/test/control-plane-version.test.ts @@ -79,7 +79,7 @@ describe('control plane version config', () => { new LlmStack(app, 'test-runtime', { config, env: { region: config.region } }), ); const fns = Object.values(template.findResources('AWS::Lambda::Function')) as any[]; - expect(fns).toHaveLength(7); + expect(fns).toHaveLength(8); for (const fn of fns) { expect(fn.Properties.Environment.Variables.CONTROL_PLANE_VERSION).toBe('1.30.0'); } diff --git a/remote/test/stack.test.ts b/remote/test/stack.test.ts index 1b080a19..17d0b9e4 100644 --- a/remote/test/stack.test.ts +++ b/remote/test/stack.test.ts @@ -362,7 +362,9 @@ describe('LlmStack (control plane)', () => { ); expect(schedule).toBeDefined(); const vars = Object.keys(schedule.Properties.Environment.Variables).sort(); - expect(vars).toEqual(['SCHEDULER_ROLE_ARN', 'SCHEDULE_GROUP', 'START_FN_ARN', 'STOP_FN_ARN'].sort()); + expect(vars).toEqual( + ['CONTROL_PLANE_VERSION', 'SCHEDULER_ROLE_ARN', 'SCHEDULE_GROUP', 'START_FN_ARN', 'STOP_FN_ARN'].sort(), + ); const statements = allPolicyStatements(template); const writes = statements.filter((s) => [s.Action].flat().includes('scheduler:CreateSchedule')); From 4e8fd149b12c59acaec1fc54e62e06f612577790 Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Wed, 7 Oct 2026 11:11:18 +0100 Subject: [PATCH 7/8] fix(remote): take the start lock on a scheduled start and scope the schedule grant A scheduled start goes through the same lock as a client's start, so a schedule firing beside a manual start launches one instance. The schedule Lambda's SSM access is limited to its own parameter, which keeps the start Lambda the only one that can delete the start lock. --- .../scheduled-remote-start-stop/design.md | 2 + .../specs/remote-schedule/spec.md | 5 +++ remote/lib/llm-stack.ts | 6 ++- remote/test/schedule-start.test.ts | 38 ++++++++++++++++ remote/test/stack.test.ts | 45 +++++++++++-------- 5 files changed, 77 insertions(+), 19 deletions(-) diff --git a/openspec/changes/scheduled-remote-start-stop/design.md b/openspec/changes/scheduled-remote-start-stop/design.md index eead13bc..b744d87b 100644 --- a/openspec/changes/scheduled-remote-start-stop/design.md +++ b/openspec/changes/scheduled-remote-start-stop/design.md @@ -28,6 +28,8 @@ See proposal.md for why. The control plane (`remote/`) already runs one shared s **Targets and payload.** Each schedule's target is the start or stop Lambda with a fixed JSON input `{"source":"spinloop.schedule","action":"start|stop","environment":""}`. The start Lambda and stop Lambda each gain a branch for that source, ahead of the Function URL branch. Scheduler invokes Lambda asynchronously, so the 15-minute start is not cut short. The scheduled start calls the same start function as the URL path; the scheduled stop calls the same pause function, after reading the instance's `Retain-Until` tag. +**A scheduled start takes the environment's start lock.** The scheduled start calls the same `wake()` as a client's start, and `wake()` takes the per-environment lock added by the start lock change (`endpoint-lifecycle`). A schedule firing while someone runs `spinloop remote start` therefore cannot launch a second instance. The scheduled run finds the lock held, ends with a retryable reply that is only logged, and the next firing tries again. A scheduled stop does not take the lock; stops are never locked, and a start whose instance is stopped under it ends at once. + **Next runs.** There is no `remote status` command (`spinloop status` is a fleet-wide table with one row per node), so the next runs are shown by `schedule show` rather than a status line. The schedule Lambda computes the next firing of each action from the stored list, in each schedule's zone, with its own small cron evaluator in `lambda/shared/schedules.ts` (no cron library). The result is not read from Scheduler, so it is the same whether or not Scheduler has caught up. A time the clocks skip is not a firing; a time they repeat fires at the first occurrence. **Client.** `internal/remote` gains `SetSchedules`, `GetSchedules`, `ClearSchedules` calling a new `schedule_url` from the config. A missing `schedule_url` returns the "re-run bootstrap" error. The CLI group is `remote schedule {set,show,clear}`, built the way `keep` is. diff --git a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md index e855dd28..7a28f2b8 100644 --- a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md +++ b/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md @@ -54,6 +54,11 @@ When a `start` schedule fires, the control plane SHALL start the environment's i - **WHEN** a `start` schedule fires and the instance is already running - **THEN** nothing is launched and the instance keeps running +#### Scenario: A scheduled start and a manual start launch one instance + +- **WHEN** a `start` schedule fires while a start from a client for the same environment holds its start lock +- **THEN** the scheduled start launches nothing and ends with a retryable reply, and only one instance exists afterwards + ### Requirement: A scheduled stop pauses the environment When a `stop` schedule fires, the control plane SHALL pause the environment's instance: stop it without terminating it, as `spinloop remote pause` does, so it can be woken again. When the instance carries a `Retain-Until` deadline that has not passed, the scheduled stop SHALL be skipped and the skip recorded in the stop Lambda's log. When there is no running instance, the scheduled stop SHALL do nothing. diff --git a/remote/lib/llm-stack.ts b/remote/lib/llm-stack.ts index 0fa2be2d..2cb3ee46 100644 --- a/remote/lib/llm-stack.ts +++ b/remote/lib/llm-stack.ts @@ -730,7 +730,11 @@ export class LlmStack extends cdk.Stack { scheduleFn.addToRolePolicy( new iam.PolicyStatement({ actions: ['ssm:GetParameter', 'ssm:PutParameter', 'ssm:DeleteParameter'], - resources: [envParamArn], + // Its own parameter only: it does not read the deploy-config or touch + // the start lock. + resources: [ + `arn:${cdk.Aws.PARTITION}:ssm:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:parameter/cloud-vm-llm/*/schedules`, + ], }), ); scheduleFn.addToRolePolicy( diff --git a/remote/test/schedule-start.test.ts b/remote/test/schedule-start.test.ts index c4b0f6ab..b72b7854 100644 --- a/remote/test/schedule-start.test.ts +++ b/remote/test/schedule-start.test.ts @@ -60,6 +60,16 @@ vi.mock('../lambda/shared/environments', async (importOriginal) => ({ vi.mock('../lambda/shared/seed', () => ({ weightsPresent: async () => true })); +// A scheduled start takes the environment's start lock like any other start; +// here the lock is a stub so each test can say whether another start holds it. +const acquireWakeLock = vi.fn(); +const releaseWakeLock = vi.fn(); +vi.mock('../lambda/shared/wake-lock', () => ({ + acquireWakeLock: (...args: unknown[]) => acquireWakeLock(...args), + releaseWakeLock: (...args: unknown[]) => releaseWakeLock(...args), + wakeLockHeld: async () => false, +})); + let handler: (event: ScheduledRunEvent, context: Context) => Promise; beforeAll(async () => { @@ -79,6 +89,8 @@ function structured(result: LambdaFunctionURLResult): { statusCode: number; body beforeEach(() => { vi.clearAllMocks(); + acquireWakeLock.mockResolvedValue(true); + releaseWakeLock.mockResolvedValue(undefined); readDeployConfig.mockResolvedValue({ runner: 'llamacpp', modelId: 'org/model', @@ -133,6 +145,32 @@ describe('a scheduled start', () => { expect(startInstance).not.toHaveBeenCalled(); }); + it('takes the environment’s start lock and releases it afterwards', async () => { + findManagedInstance.mockResolvedValue({ instanceId: 'i-run', state: 'running' }); + getInstance.mockResolvedValue({ instanceId: 'i-run', state: 'running', launchTime: new Date() }); + + await handler(runEvent('start'), context); + + expect(acquireWakeLock).toHaveBeenCalledTimes(1); + expect(acquireWakeLock.mock.calls[0][0]).toBe('dev'); + expect(releaseWakeLock).toHaveBeenCalledTimes(1); + expect(releaseWakeLock.mock.calls[0][0]).toBe('dev'); + }); + + it('launches nothing while a manual start holds the lock', async () => { + acquireWakeLock.mockResolvedValue(false); + findManagedInstance.mockResolvedValue(null); + + const result = await handler(runEvent('start'), context); + + expect(structured(result).statusCode).toBe(503); + expect(JSON.parse(structured(result).body).state).toBe('starting'); + expect(findManagedInstance).not.toHaveBeenCalled(); + expect(runInstance).not.toHaveBeenCalled(); + expect(startInstance).not.toHaveBeenCalled(); + expect(releaseWakeLock).not.toHaveBeenCalled(); + }); + it('ignores an event whose action is not start', async () => { const result = await handler(runEvent('stop'), context); expect(JSON.parse(structured(result).body).state).toBe('ignored'); diff --git a/remote/test/stack.test.ts b/remote/test/stack.test.ts index 17d0b9e4..b18597f8 100644 --- a/remote/test/stack.test.ts +++ b/remote/test/stack.test.ts @@ -484,30 +484,39 @@ describe('LlmStack (control plane)', () => { expect(JSON.stringify(terminate!.Condition)).toContain('cloud-vm-llm'); }); - it('lets only the start Lambda delete SSM parameters, and only the start lock', () => { + it('lets only the start and schedule Lambdas delete SSM parameters, each only its own', () => { const policies = template.findResources('AWS::IAM::Policy') as Record; - const withDelete = Object.values(policies).filter((p) => + const fns = template.findResources('AWS::Lambda::Function') as Record; + const deleters = Object.values(policies).filter((p) => (p.Properties.PolicyDocument.Statement as Statement[]).some((s) => [s.Action].flat().includes('ssm:DeleteParameter'), ), ); - expect(withDelete).toHaveLength(1); + expect(deleters).toHaveLength(2); + + // What each may delete, keyed by the Lambda the role belongs to. The Lambdas + // are told apart by settings only they are given: the AMI settings for the + // start Lambda, the schedule group for the schedule Lambda. + const deletable: Record = {}; + for (const policy of deleters) { + const statements = (policy.Properties.PolicyDocument.Statement as Statement[]).filter((s) => + [s.Action].flat().includes('ssm:DeleteParameter'), + ); + expect(statements).toHaveLength(1); + const resources = JSON.stringify(statements[0].Resource); + expect(resources).not.toContain('"*"'); + const roleId = (policy.Properties.Roles as { Ref: string }[])[0].Ref; + const owner = Object.values(fns).find((f) => f.Properties.Role['Fn::GetAtt'][0] === roleId); + const vars = owner?.Properties.Environment.Variables ?? {}; + const who = vars.AMI_ROLE_TAG_KEY ? 'start' : vars.SCHEDULE_GROUP ? 'schedule' : 'other'; + deletable[who] = resources; + } - const statements = (withDelete[0].Properties.PolicyDocument.Statement as Statement[]).filter((s) => - [s.Action].flat().includes('ssm:DeleteParameter'), - ); - expect(statements).toHaveLength(1); - expect([statements[0].Action].flat()).toEqual(['ssm:DeleteParameter']); - const resources = JSON.stringify(statements[0].Resource); - expect(resources).toContain('parameter/cloud-vm-llm/*/wake-lock'); - expect(resources).not.toContain('"*"'); - - // The role it is attached to belongs to the start Lambda, found by the - // AMI settings only that Lambda is given. - const roleId = (withDelete[0].Properties.Roles as { Ref: string }[])[0].Ref; - const fns = template.findResources('AWS::Lambda::Function') as Record; - const owner = Object.values(fns).find((f) => f.Properties.Role['Fn::GetAtt'][0] === roleId); - expect(owner?.Properties.Environment.Variables.AMI_ROLE_TAG_KEY).toBeDefined(); + expect(Object.keys(deletable).sort()).toEqual(['schedule', 'start']); + expect(deletable.start).toContain('parameter/cloud-vm-llm/*/wake-lock'); + expect(deletable.start).not.toContain('schedules'); + expect(deletable.schedule).toContain('parameter/cloud-vm-llm/*/schedules'); + expect(deletable.schedule).not.toContain('wake-lock'); }); it('passes the AMI role tag, weights bucket and subnet list to the start Lambda', () => { From ddb579194c6bf6b94bcb887bfa5394dcde7ef61e Mon Sep 17 00:00:00 2001 From: spinloop-agent Date: Wed, 7 Oct 2026 11:19:41 +0100 Subject: [PATCH 8/8] docs(openspec): archive the scheduled-remote-start-stop change --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/remote-schedule/spec.md | 0 .../tasks.md | 0 openspec/specs/remote-schedule/spec.md | 128 ++++++++++++++++++ 6 files changed, 128 insertions(+) rename openspec/changes/{scheduled-remote-start-stop => archive/2026-10-07-scheduled-remote-start-stop}/.openspec.yaml (100%) rename openspec/changes/{scheduled-remote-start-stop => archive/2026-10-07-scheduled-remote-start-stop}/design.md (100%) rename openspec/changes/{scheduled-remote-start-stop => archive/2026-10-07-scheduled-remote-start-stop}/proposal.md (100%) rename openspec/changes/{scheduled-remote-start-stop => archive/2026-10-07-scheduled-remote-start-stop}/specs/remote-schedule/spec.md (100%) rename openspec/changes/{scheduled-remote-start-stop => archive/2026-10-07-scheduled-remote-start-stop}/tasks.md (100%) create mode 100644 openspec/specs/remote-schedule/spec.md diff --git a/openspec/changes/scheduled-remote-start-stop/.openspec.yaml b/openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/.openspec.yaml similarity index 100% rename from openspec/changes/scheduled-remote-start-stop/.openspec.yaml rename to openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/.openspec.yaml diff --git a/openspec/changes/scheduled-remote-start-stop/design.md b/openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/design.md similarity index 100% rename from openspec/changes/scheduled-remote-start-stop/design.md rename to openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/design.md diff --git a/openspec/changes/scheduled-remote-start-stop/proposal.md b/openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/proposal.md similarity index 100% rename from openspec/changes/scheduled-remote-start-stop/proposal.md rename to openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/proposal.md diff --git a/openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md b/openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/specs/remote-schedule/spec.md similarity index 100% rename from openspec/changes/scheduled-remote-start-stop/specs/remote-schedule/spec.md rename to openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/specs/remote-schedule/spec.md diff --git a/openspec/changes/scheduled-remote-start-stop/tasks.md b/openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/tasks.md similarity index 100% rename from openspec/changes/scheduled-remote-start-stop/tasks.md rename to openspec/changes/archive/2026-10-07-scheduled-remote-start-stop/tasks.md diff --git a/openspec/specs/remote-schedule/spec.md b/openspec/specs/remote-schedule/spec.md new file mode 100644 index 00000000..4f3609a4 --- /dev/null +++ b/openspec/specs/remote-schedule/spec.md @@ -0,0 +1,128 @@ +# remote-schedule Specification + +## Purpose +Let a remote environment start and stop itself on cron schedules, so a team that +works set hours has the model up when it sits down and stops paying for it when +it leaves. The control plane stores and runs the schedules, so they fire with no +machine of the operator's switched on; the CLI sets, shows and clears them. + +## Requirements + +### Requirement: An environment holds a list of schedules + +Each environment SHALL hold zero or more schedules. A schedule SHALL have an action (`start` or `stop`), a cron expression in the five-field form (minute, hour, day of month, month, day of week) and an IANA time zone. When no time zone is given it SHALL be UTC. Setting schedules for an environment SHALL replace its whole list, so the list after the call is exactly the one sent. An invalid cron expression, an unknown time zone, an unknown action or an invalid environment name SHALL be rejected, and a rejected request SHALL leave the existing schedules unchanged. The schedules SHALL be stored in the control plane, not on any user's machine, so they fire whether or not any user machine is on. + +#### Scenario: Setting office-hours schedules + +- **WHEN** schedules are set for an environment with a `start` at `0 8 * * 1-5` and a `stop` at `0 18 * * 1-5`, in `Europe/London` +- **THEN** the environment's list holds those two schedules and no others + +#### Scenario: Setting replaces the previous list + +- **WHEN** an environment has two schedules and a set request is made with one different schedule +- **THEN** the environment has only that one schedule afterwards, and the two earlier ones no longer fire + +#### Scenario: An invalid expression is rejected + +- **WHEN** a set request contains a cron expression that is not five valid fields +- **THEN** the request is refused, the error names the expression, and the environment's earlier schedules still fire + +#### Scenario: Schedules are per environment + +- **WHEN** schedules are set for one environment +- **THEN** no other environment's schedules change and no other environment's instance is started or stopped + +### Requirement: A schedule fires in its own time zone + +A schedule SHALL fire when its cron expression matches in its time zone, including across daylight saving changes: a schedule at 08:00 in `Europe/London` fires at 08:00 local time on both sides of a clock change. + +#### Scenario: A schedule follows local time + +- **WHEN** a `start` schedule is set at `0 8 * * *` in `Europe/London` +- **THEN** it fires at 07:00 UTC in summer and 08:00 UTC in winter + +### Requirement: A scheduled start brings the environment up + +When a `start` schedule fires, the control plane SHALL start the environment's instance with the same behaviour as an on-demand start, including the weights check and the wait until the model is answering. When the instance is already running, the scheduled start SHALL leave it as it is. When the start cannot find capacity or the weights are absent, the run SHALL record why in the start Lambda's log and SHALL NOT retry until the next firing. + +#### Scenario: A stopped environment is started on schedule + +- **WHEN** a `start` schedule fires and the environment has no running instance +- **THEN** the instance is launched or re-woken, as an on-demand start would do + +#### Scenario: A running environment is left alone + +- **WHEN** a `start` schedule fires and the instance is already running +- **THEN** nothing is launched and the instance keeps running + +#### Scenario: A scheduled start and a manual start launch one instance + +- **WHEN** a `start` schedule fires while a start from a client for the same environment holds its start lock +- **THEN** the scheduled start launches nothing and ends with a retryable reply, and only one instance exists afterwards + +### Requirement: A scheduled stop pauses the environment + +When a `stop` schedule fires, the control plane SHALL pause the environment's instance: stop it without terminating it, as `spinloop remote pause` does, so it can be woken again. When the instance carries a `Retain-Until` deadline that has not passed, the scheduled stop SHALL be skipped and the skip recorded in the stop Lambda's log. When there is no running instance, the scheduled stop SHALL do nothing. + +#### Scenario: A running environment is paused on schedule + +- **WHEN** a `stop` schedule fires and the instance is running with no retention deadline +- **THEN** the instance is stopped and not terminated + +#### Scenario: A retained instance is not stopped + +- **WHEN** a `stop` schedule fires and the instance has a `Retain-Until` time in the future +- **THEN** the instance keeps running + +#### Scenario: Nothing to stop + +- **WHEN** a `stop` schedule fires and the environment has no running instance +- **THEN** the run does nothing and reports no error + +### Requirement: The schedule command sets, shows and clears schedules + +`spinloop remote schedule set` SHALL accept one or more `--start CRON` and `--stop CRON` flags and an optional `--timezone ZONE`, and SHALL replace the environment's schedules with them. `spinloop remote schedule show` SHALL print the environment's schedules, one per line, giving the action, expression and time zone, and SHALL print that there are none when the list is empty. `spinloop remote schedule clear` SHALL remove every schedule. The commands SHALL select the environment as the other `remote` subcommands do: `--env ` or the per-user default. `set` with neither `--start` nor `--stop` SHALL fail and say to use `clear` to remove schedules. When the deployment's control plane has no schedule endpoint, the commands SHALL fail with an error naming the fix (re-run `spinloop remote bootstrap`). + +#### Scenario: Setting two schedules from the command line + +- **WHEN** the user runs `spinloop remote schedule set --start "0 8 * * 1-5" --stop "0 18 * * 1-5" --timezone Europe/London` +- **THEN** the environment has those two schedules and the command prints them + +#### Scenario: Showing no schedules + +- **WHEN** the user runs `spinloop remote schedule show` for an environment with none +- **THEN** the output says there are no schedules and the command succeeds + +#### Scenario: Clearing + +- **WHEN** the user runs `spinloop remote schedule clear` +- **THEN** the environment has no schedules and none fires afterwards + +#### Scenario: Set with nothing to set + +- **WHEN** the user runs `spinloop remote schedule set` with no `--start` or `--stop` +- **THEN** the command fails and tells the user to use `clear` to remove schedules + +#### Scenario: An older control plane + +- **WHEN** the deployment's configuration has no schedule endpoint +- **THEN** the command fails with an error telling the user to re-run `spinloop remote bootstrap` + +### Requirement: Show reports the next scheduled runs + +When the environment has schedules, `spinloop remote schedule show` (and `set`, which prints the same listing) SHALL report the next time a start fires and the next time a stop fires, as absolute UTC times, each on its own "next start" or "next stop" line. An action with no schedule, or whose schedule never fires again, SHALL have no line. When the environment has no schedules, the output SHALL say so and have no "next" line. + +#### Scenario: A scheduled environment shows its next runs + +- **WHEN** the user runs `spinloop remote schedule show` for an environment with a start and a stop schedule +- **THEN** the output includes "next start" and "next stop" lines with absolute times + +#### Scenario: An action with no schedule has no line + +- **WHEN** the environment has only a start schedule +- **THEN** the output has a "next start" line and no "next stop" line + +#### Scenario: An unscheduled environment omits the lines + +- **WHEN** the user runs `spinloop remote schedule show` for an environment with no schedules +- **THEN** the output says there are no schedules and has no "next start" or "next stop" line