INTR-626 - Let provisioners place several workflow deployments on one sidecar

[FarinM]

Stacked on #199

Why

The Hub reserved every sidecar for a single allocation. A sidecar credential authenticated exactly one probe or allocation, and every generation change or release closed the socket. A provisioner therefore had to start new capacity for every deployment, even when it already ran a sidecar that could host the work.

Whether work shares a sidecar is the provisioner's decision, not the Hub's. This PR lets a provisioner answer ensure with a sidecar it already runs. To make that safe, the sidecar link, mail, history pushes, and undeploy no longer assume one deployment per sidecar. The Hub also stops trying to recover a lost or missing deployment. A worker that misses its reconnect deadline may still be running, and the Hub cannot restore the files it holds. Starting the deployment again elsewhere could run it twice or lose that state, so the Hub fails the deployment and releases it instead. docs/SIDECAR_PLACEMENT.md has the provisioner contract.

What this adds

• Sidecar reuse in the provisioner contract. EnsureSidecarResult takes an optional sidecarId. The Hub accepts that sidecar only while it still hosts another probe or allocation of the same provisioner binding, counting work that is being released or replaced. Otherwise it rejects the placement with sidecar_reuse_rejected. destroy releases one allocation from the sidecar. The provisioner decides when to stop the sidecar, and it has to track which allocations run on which sidecar durably, because the Hub never re-announces them.

• One connection, many bindings. A sidecar credential now authenticates the sidecar for every binding it currently hosts. Each binding keeps its own generation fence. Releasing an allocation detaches and undeploys only that allocation, without closing the socket. Adopting a probe's sidecar no longer disconnects the probe.

• hello / welcome handshake. register and reconnect are replaced by hello, which reports a bounded inventory of the incarnations the sidecar holds, each stamped with its generation. The sidecar holds back anything the Hub must receive until welcome arrives. Deploy and undeploy replies are matched by request id, with separate agent.deploy.error and agent.undeploy.error frames. Undeploys are ordered by generation, so an undeploy for a superseded generation never touches the newer copy.

• No Hub-side recovery of a running deployment. A lost worker is failed and released, not replaced. A connected sidecar that no longer holds a deployment has that deployment failed, not redeployed. Replacement remains only for provisioning, which has run nothing yet. Initialization now serves only first deploys. A workflow-run seed that would move existing history is refused as a conflict.

• Route rules on a shared sidecar. A reconnecting sidecar gets a deployment's route back only if its first deploy completed and its run has not ended. A copy whose deploy is still uncertain is undeployed. A copy whose run ended on its own stays unrouted until the Hub releases it, so a reconnect does not cut its retention short. A copy of a run the Hub cancelled or failed is undeployed, since a restart that lost its stopped mark may be running that run again.

• Self-terminated deployments. A child that stops or fails to restore is marked stopped in its run record and reported in a deployment.stopped frame. The Hub reconciles its history, then fails its runs and keeps the copy for inspection. A stop the Hub drives also marks the record stopped instead of removing it. Undeploy marks the record tearing-down and removes it last, so a crash in the middle of an undeploy still leaves a record for the next boot to report.

• Isolation between co-hosted deployments.

  • The sidecar's Hub URL, id, and token are no longer passed into workflow children.
  • A child can send mail only as its own deployment.
  • Mail between deployments on the same sidecar, or from a deployment to itself, is relayed through the Hub. Before, it went into a local inbox nothing read, while the Hub recorded it as delivered.

  This does not isolate co-hosted deployments from each other. Every child still runs as the sidecar's OS user - so it's up to the provisioner plugin to properly advertise whether it shares the sidecar.

• Per-address frame lanes. The sidecar used to handle every Hub frame on one serial queue, so one slow deploy could hold up every other deployment on the sidecar, including the heartbeat pong. Frames for one address are still handled in order, but different addresses no longer wait on each other. Pongs and probes run on their own. Sender-key updates and inbound signature checks share a separate ordered lane, so a mail is checked against exactly the keys that arrived before it.

• Long-lived sidecar hygiene. Undeploy now cleans up everything the deployment left on the sidecar: its run repository, grants repository, local conversation copy, push state, and directories a tool left read-only. It runs every teardown step even when one fails, and answers with an error naming the failed steps. Pack transfer ids are random, so a late answer from before a restart cannot settle a new transfer. Shared closure-cache entries stay pinned while callers read them.

• Mail and push reliability on a shared link.

  • The Hub answers every workflow-run push it receives.
  • A disconnected push stays pending for recovery.
  • A first push is resent only for the repository-bootstrap race.
  • When the outage queue is full, the sidecar drops correlation registrations and audit copies first, and fails a send only when nothing else is left to drop.
  • Pending mail is kept when its deployment loses its route.
  • A stale mail retry cannot resend a mail that was already redelivered.
  • When the Hub gives up on dispatched mail, the mail is left to its dispatch row instead of being reported undelivered.

• Capacity limits. A sidecar hosts at most 128 deployments, the most its hello can report. The Hub refuses to place another on a full sidecar, and the sidecar refuses a deploy past the limit. At boot the sidecar restores 8 deployments at a time and restores at most 128. Records past the limit stay unspawned and are reported stopped after every welcome.

• Lifecycle additions.

  • maxDisconnected sits next to maxLifetime in the lifecycle policy, with the same inheritance and caps, a 15m default, and the WORKFLOW_DEFAULT_MAX_DISCONNECTED env var. It replaces the fixed 2-minute window, and a Hub start restarts the clock.
  • A first deploy that keeps failing before its frame is sent is failed 1 minute after its first failure, instead of being retried forever.
  • Runs the Hub fails now record failureCode and failureMessage.

• Removed dead paths. Removed: sync.request, the undeploy's agent-state push (statePushed is gone from agent.undeploy.ack), the Hub's unused session-address routes, and the disconnect queue. The remaining retention setting is renamed to mailHoldTTLMs.

• E2E coverage. The local-process provisioner can share processes between allocations. With E2E_SHARE_SIDECARS=tenant, a spec checks that two deployments run on one process.

Example: a provisioner that reuses sidecars

function startSidecar(req: EnsureSidecarRequest) {
  const sidecar = Bun.spawn(["bun", "run", "apps/sidecar/src/index.ts"], {
    env: {
      ...process.env,
      HUB_WS_URL: req.hubWebSocketUrl,
      SIDECAR_ID: req.sidecarId,
      SIDECAR_TOKEN: req.token,
      SIDECAR_DATA_DIR: `/var/lib/intx/${req.sidecarId}`,
    },
  });
  return { id: req.sidecarId, stop: () => sidecar.kill() };
}

function createSingleSidecarProvisioner(): SidecarProvisioner {
  let sidecar: ReturnType<typeof startSidecar> | null = null;
  const allocations = new Set<string>();

  return {
    id: "single-sidecar",
    apiVersion: 1,
    bindingFingerprint: "single-sidecar:v1",
    capabilities: [],

    async ensure(req) {
      if (sidecar === null) sidecar = startSidecar(req);
      allocations.add(req.allocationId);
      return { kind: "accepted", sidecarId: sidecar.id };
    },

    async destroy(req) {
      allocations.delete(req.allocationId);
      if (allocations.size === 0 && sidecar !== null) {
        sidecar.stop();
        sidecar = null;
      }
      return { kind: "destroyed" };
    },
  };
}

Every ensure answers with the one sidecar this provisioner runs. The first ensure starts it, because the Hub mints a sidecar's identity and token per request. The provisioner stops the sidecar when its last allocation is destroyed, since the Hub only accepts a sidecar that still hosts other work.

API changes

• Tenant and installed-workflow lifecycle policies accept maxDisconnected, and the lifecycle status route returns it.
• Workflow run views include failureCode and failureMessage.
• The allocation status docs describe recovering: the Hub is replacing capacity whose provisioning failed before the deployment first ran.

Database migrations

• 0101_sidecar_reuse drops sidecar_allocation_active_sidecar_idx, which allowed only one active allocation per sidecar. It adds a partial index on workflow_run.sidecar_id, because accepting a reused sidecar deletes the identity the Hub minted, and without the index each delete scans every run.
• 0102_workflow_run_failure_reason adds nullable failure_code and failure_message to workflow_run.
• 0103_sidecar_allocation_max_disconnected adds max_disconnected_ms to sidecar_allocation. Existing rows are backfilled to the 15-minute default, and the column then becomes NOT NULL with a check that it is > 0.
• 0104_sidecar_allocation_first_deploy_failure adds nullable first_deploy_failed_at to sidecar_allocation. It records when a first deploy first failed, so the 1-minute limit counts from the same point across Hub restarts.

Reviewer notes

• The wire protocol between Hub and sidecar changes (hello / welcome, typed deploy and undeploy replies, generation stamps). Hubs and sidecars from before this PR cannot talk to ones from after it.
• The stock sidecar does not isolate co-hosted deployments: every child runs as the sidecar's OS user over its data directory. Sharing is only appropriate for work that is allowed to see each other's state.
• Work placed on an existing sidecar gets the same 2-minute connect window as a new sidecar, even if that sidecar is restarting. A deployment that misses it is failed, which costs little because it has not run yet.
• The first-deploy limit is short on purpose. Failures before the deploy frame is sent, such as a removed catalog offering or a credential that will not decrypt, rarely fix themselves, so a quick failure gets 2 retries 30 seconds apart. A restore that times out after its 120-second acknowledgement window gets 1 retry.
• Most of the logic is in packages/hub-sessions/src/ws/sidecar-handler.ts, packages/hub-agent/src/ws/hub-link.ts, apps/sidecar/src/workflow-host-wiring.ts, and packages/db/src/sidecar-allocation-store.ts. The drizzle snapshots inflate the line count by about 18k lines.
• Reviewing commit by commit is probably easier than reviewing the whole diff. The first four commits remove Hub-side recovery. Next come the multi-binding connection (a4d9c7023) and the reuse contract (5e4d4361f). The isolation, lane, and cleanup fixes follow, and the lifecycle limits are at the top.

[linear]

INTR-626

[alexanderguy]

This is a pull-request review done by an agent, following the repository's pull-request-review flow on 14c89996.

Requesting changes on the four comments. The branch reads clearly as it stands: a provisioner may keep several deployments on a sidecar it already runs, hello stays within the incarnation cap, and a stop that reached disk stays unrouted. CI on run 37492819281 is green. The comments are where a restart or a reuse can still release a copy that should stay, or accept a placement after destroy has stopped the sidecar.

[alexanderguy]

This is a pull-request review done by an agent, on the rebased head. Requesting changes on the comment about a deployment the hello left out.

[alexanderguy]

LGTM

This is a pull-request review done by an agent. The open comment is followed up in INTR-643.