diff --git a/AGENTS.md b/AGENTS.md index be1458e6..082c2fa2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,8 @@ or one of them only where it is used. - Defaults are local and carry no external coordinates: SQLite document store, filesystem object store, an OCI registry at `localhost:5000`, env-var secret store, `LocalRunner`, `local` Docker sandboxes for envs and agents. MongoDB, S3, Cloud Storage, ECR, AWS Secrets Manager, Google Cloud - Secret Manager, Modal and E2B exist as implementations and are selected by config. + Secret Manager, Modal, E2B and Tensorlake exist as implementations and are selected by config + (Tensorlake through the `tensorlake` extra). - The model endpoint is unset until `[model] base_url` / `api_key` (or `LITELLM_BASE_URL` / `LITELLM_API_KEY`) is configured. A loopback URL handed into a `local` container (the model endpoint, an agent, peer or gateway on this machine) becomes `host.docker.internal`. On Linux that @@ -95,7 +96,7 @@ and deserialized through a registry. | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | -| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | +| `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `tensorlake` (the `tensorlake` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store) and `EnvironmentGatewayProvider`, which renders a docker-compose for the gateway and its MCP servers inside the sandbox; `providers/env_state/` holds env-state providers (`local_postgres` built in). | | `a2a_agent/` | The `A2AAgent` entity (`a2a_agent`), its stores and the validator steps. The protocol package provides the agent-side framework. | | `runner/` | The `[runner]` seam: `Runner.submit()` returns `(run_id, instance_id)`; `LocalRunner` is built in. | | `explorer/` | Optional local web UI: `agent-env up`, needs the `explorer` extra, binds loopback `:8234`. | diff --git a/README.md b/README.md index 67aa102e..1132669a 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ Or install it with pip, into a virtualenv: pip install agentenv-framework ``` -To use the SDK in your own project, add it as a dependency with `uv add agentenv-framework`. The explorer, `agent-env up`, needs the `explorer` extra (`uv tool install 'agentenv-framework[explorer]'`, or the same extra with uvx or pip) and an `.agentenv/config.toml` in the current folder or above it; an empty one keeps every local default. +To use the SDK in your own project, add it as a dependency with `uv add agentenv-framework`. The explorer, `agent-env up`, needs the `explorer` extra (`uv tool install 'agentenv-framework[explorer]'`, or the same extra with uvx or pip) and an `.agentenv/config.toml` in the current folder or above it; an empty one keeps every local default. The `tensorlake` sandbox backend needs the `tensorlake` extra (`pip install 'agentenv-framework[tensorlake]'`) and an `api_key` in `[sandbox.providers.tensorlake.config]`. Its sandboxes boot from the public image `agentenv-dind-host-v1`; set `image` there to use another registered Docker-capable image. A plain install runs on local stores and needs no cloud SDK. The cloud store backends are extras: `aws` for S3, Secrets Manager, DynamoDB and ECR (`pip install 'agentenv-framework[aws]'`), and `gcp` for Cloud Storage, Secret Manager, Firestore and Artifact Registry (`pip install 'agentenv-framework[gcp]'`). A config that names a backend without its extra fails, naming the extra to install. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 73f7d264..e0f4b0ff 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -14,7 +14,7 @@ marks, or contributor names to endorse or promote Scale AI, or related products. ## Python dependencies -The runtime dependencies of `agentenv-framework` and its `aws`, `explorer`, `gcp` and `sail` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. +The runtime dependencies of `agentenv-framework` and its `aws`, `explorer`, `gcp`, `sail` and `tensorlake` extras, at the versions resolved in `uv.lock`. Development-only dependencies are not listed. ### a2a-sdk 0.3.26 @@ -716,6 +716,14 @@ The runtime dependencies of `agentenv-framework` and its `aws`, `explorer`, `gcp - Source: - [License text 1](#license-text-1) +### tensorlake 0.5.148 + +- License: Apache-2.0 +- Copyright: Copyright [2025] [Tensorlake] +- Author: Tensorlake Inc. +- Source: +- [License text 1](#license-text-1) + ### tiktoken 0.12.0 - License: MIT @@ -806,6 +814,22 @@ The runtime dependencies of `agentenv-framework` and its `aws`, `explorer`, `gcp - Source: - [License text 4](#license-text-4) +### websocket-client 1.9.2 + +- License: Apache-2.0 +- Copyright: Copyright 2026 engn33r +- Author: liris +- Source: +- [License text 1](#license-text-1) + +### websockets 17.2 + +- License: BSD-3-Clause +- Copyright: Copyright (c) Aymeric Augustin and contributors +- Author: Aymeric Augustin +- Source: +- [License text 71](#license-text-71) + ### yarl 1.24.2 - License: Apache-2.0 @@ -7714,3 +7738,30 @@ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ``` + +### License text 71 + +```text +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + * Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` diff --git a/pyproject.toml b/pyproject.toml index aa7dd5eb..1fda8c5d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -72,8 +72,12 @@ gcp = [ sail = [ "sail~=0.12.8", # Sailbox exec/fs/listener/egress APIs validated against the 0.12 line ] +# The Tensorlake sandbox provider, selected as the built-in `tensorlake` backend. +tensorlake = [ + "tensorlake~=0.5.140", # AsyncSandbox APIs validated against 0.5.140 to 0.5.149 +] dev = [ - "agentenv-framework[aws,explorer,gcp,sail]", # so the AWS, explorer, GCP backend and Sail provider tests run in CI + "agentenv-framework[aws,explorer,gcp,sail,tensorlake]", # so the AWS, explorer, GCP backend, Sail and Tensorlake provider tests run in CI "fastmcp>=3.4,<4", # the standalone-fastmcp mount path of the protocol; <4 keeps mcp 1.x "griffelib==2.3.0", # the plugin API check; exact, since what it reports as a break changes between releases "moto>=5.0.0", diff --git a/src/agent_env/cli/a2a_agent/deploy.py b/src/agent_env/cli/a2a_agent/deploy.py index 5c6ae687..70603161 100644 --- a/src/agent_env/cli/a2a_agent/deploy.py +++ b/src/agent_env/cli/a2a_agent/deploy.py @@ -18,7 +18,7 @@ @click.option("--ttl-seconds", type=click.IntRange(min=MIN_TTL_SECONDS, max=MAX_TTL_SECONDS), default=DEFAULT_TTL_SECONDS, help=f"VM lifetime in seconds (default {DEFAULT_TTL_SECONDS})") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, tensorlake, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].agent_default (else local) when omitted.") def deploy(agent_id: str, agent_version: int | None, env_var_pairs: tuple[str, ...], ttl_seconds: int, sandbox: str): diff --git a/src/agent_env/cli/env/deploy.py b/src/agent_env/cli/env/deploy.py index 770e5e38..470bbf1b 100644 --- a/src/agent_env/cli/env/deploy.py +++ b/src/agent_env/cli/env/deploy.py @@ -19,7 +19,7 @@ @click.option("--gateway-mode", type=click.Choice([m.value for m in GatewayMode], case_sensitive=False), default=GatewayMode.PERFORMANCE.value, help="Gateway mode (performance or consistent)") @click.option("--sandbox", default=None, - help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, local) " + help="Sandbox backend(s): a built-in (modal, modal_vm, e2b, tensorlake, local) " "or a name from [sandbox.providers] in .agentenv/config.toml; comma-separated for a " "fallback chain. Defaults to [sandbox].default (else local) when omitted.") @click.option("--service-db", "service_db_env_id", default=None, diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index bb6dc46f..8d74fe72 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -287,6 +287,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "modal_vm": "agent_env.providers.sandbox_providers.modal_vm_sandbox:ModalVmSandboxProvider", "e2b": "agent_env.providers.sandbox_providers.e2b:E2BSandboxProvider", "sail_vm": "agent_env.providers.sandbox_providers.sail_vm.provider:SailVmSandboxProvider", + "tensorlake": "agent_env.providers.sandbox_providers.tensorlake.provider:TensorlakeSandboxProvider", "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", } diff --git a/src/agent_env/providers/sandbox_providers/tensorlake/README.md b/src/agent_env/providers/sandbox_providers/tensorlake/README.md new file mode 100644 index 00000000..6845fad2 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/tensorlake/README.md @@ -0,0 +1,147 @@ +# `tensorlake` sandbox provider + +Runs agent-env sandboxes on [Tensorlake](https://tensorlake.ai) sandboxes: MicroVMs booted from a +Docker-capable host image. Like `sail_vm`, `modal_vm` and `e2b`, it is a VM provider: agent-env's +docker-in-VM flows run on it unchanged. That covers the gateway's docker-compose, an agent's `docker run`, +image loading and artifact collection. + +## Set up + +### 1. Install the extra + +The Tensorlake SDK is an optional extra: + +```bash +pip install 'agentenv-framework[tensorlake]' # or: uv add 'agentenv-framework[tensorlake]' +``` + +Without it, selecting `tensorlake` fails with a `ConfigError` that names this extra. Nothing else needs it. + +### 2. Get a Tensorlake API key + +Create a key in the Tensorlake dashboard. For local work, export it: + +```bash +export TENSORLAKE_API_KEY=... +``` + +For a shared deployment, put it in your secret store instead, for example as `tensorlake_api_key`, and +reference it with `secret:tensorlake_api_key`. Never put the key itself in `config.toml`. The provider +reads the key only from its config, never from the process environment. + +### 3. Configure + +A complete `.agentenv/config.toml` that runs every environment and agent on Tensorlake, with the local stores: + +```toml +[sandbox] +default = "tensorlake" # environments and sandboxes +agent_default = "tensorlake" # agents + +[sandbox.providers.tensorlake.config] +api_key = "env:TENSORLAKE_API_KEY" # or "secret:tensorlake_api_key" + +[model] +base_url = "https://litellm.example.com" +api_key = "env:LITELLM_API_KEY" +``` + +To keep the local default and use Tensorlake per run instead, add only the +`[sandbox.providers.tensorlake.config]` table and pass `--sandbox tensorlake`. `agent-env config show` prints +the file in effect and masks the key. + +### 4. Run something + +The bundled `hello` task deploys a sandbox, loads a file into it and checks it, with no model needed: + +```bash +agent-env run hello --sandbox tensorlake +``` + +`agent-env -v run …` also logs each sandbox as it starts +(`Tensorlake sandbox started: sandbox_id=… image=agentenv-dind-host-v1 attribution=…`). + +## Configuration reference + +All keys go under `[sandbox.providers.tensorlake.config]`. Unknown keys are refused. + +| Key | Default | Meaning | +|---|---|---| +| `api_key` | required | The Tensorlake API key, as an `env:` or `secret:` reference. | +| `image` | `"agentenv-dind-host-v1"` | A registered Docker-capable Tensorlake image to boot from. | +| `api_url` | `"https://api.tensorlake.ai"` | The Tensorlake API. Every SDK call uses it, whatever `TENSORLAKE_API_URL` says. | + +## The host image + +Sandboxes boot from `agentenv-dind-host-v1` by default, a public image built from `host_image.Dockerfile` +next to this file: `tensorlake/ubuntu-systemd` with Docker Engine and the Compose v2 plugin. Tensorlake +runs `dockerd` only as the systemd `docker.service`, so the provider starts it with `systemctl` and does +not start `dockerd` by hand. + +To use your own image, publish it once and set `image`: + +```bash +tl sbx image create host_image.Dockerfile -n my-docker-host \ + --disk_mb 30720 --builder_disk_mb 24576 --docker_compat +``` + +An image that is not registered fails the create with a `ValueError` that contains this command. The +provider never boots a different image. A changed Dockerfile is +published under a new version, never over the old one. + +## Resources and lifetime + +- **CPU and memory:** Tensorlake hosts run without KVM, so cores are slower than the request suggests. + The provider raises a request to at least 2 CPUs, 4 GiB of memory and 1 GiB of memory per CPU. More + than 8 GiB of memory per CPU is refused. +- **Disk:** at least 30 GiB, the disk the host image is published with, and at most 100 GiB. +- **Changed or ignored requests:** a raised CPU, memory or disk request logs a warning. `boot_mode` has no + Tensorlake equivalent, so a `boot_mode` that is set logs a warning and has no effect. +- **Refused requests:** a request Tensorlake can't meet is refused before anything is created. +- **Lifetime:** the sandbox's `timeout` is Tensorlake's idle timeout. After it, the sandbox terminates. + +## Networking + +- **Ports:** each exposed port gets a public URL, `https://-.sandbox.tensorlake.ai`, + which fills `tunnel_urls`. It has no platform authentication, like Modal's tunnels. A Bearer header + would hand the Tensorlake API key to every workload that calls the URL. +- **Egress:** allow-all, or an allowlist of hostnames, IPv4 addresses and IPv4 CIDRs. A policy with IPv6 + entries is refused with `NetworkPolicyUnsupportedError` before anything is created, and a fallback chain + skips to its next provider. Tensorlake applies the policy outside the VM, so one policy covers `dockerd` and every + container. Tensorlake refuses a policy update with a host that does not resolve. +- **Image downloads:** before it loads images under an allowlist, the provider adds the hosts of the + signed download URLs, then waits up to 60 s for the change to reach the host firewall. +- **Reconnect:** `get_sandbox` restores the ports and the applied egress policy. A policy it can't + represent leaves `network_policy` unknown, and image loading then fails closed. + +## Attribution and cost + +Tensorlake sandboxes have no labels. The provider logs one `agent_env.tensorlake_sandbox_started` event per +sandbox, with `tensorlake_sandbox_id`, the full attribution (including `[sandbox.attribution]` defaults), +`cpus`, `memory_mb` and `disk_mb`. To attribute cost, join Tensorlake's per-sandbox usage on the sandbox id. + +## Cleanup + +- A sandbox that fails to start is deleted. +- A sandbox whose setup fails after it starts (ports, Docker) is terminated. +- If the caller is cancelled while the create request is in flight, the provider deletes the sandbox + that request makes, once it answers. +- `terminate` is idempotent: a sandbox that is already gone is not an error. + +## Things to know + +- **Commands run as `tl-user`:** agent-env's `sudo` prefixes stay, and file uploads land in `/tmp` first, + then a root command joins them at the target path. +- **Lost connections:** when the command stream ends without an exit status, `exec` returns exit code -1. + `exec_script` retries idempotent scripts on -1 only. +- **Uploads:** the ingress proxy refuses request bodies above about 4 MB, so uploads go through the file + API in 512 KiB parts, 8 at a time, and the joined file is checked against its sha256. An image tarball + from an object store that can't sign URLs, such as the local default, is streamed the same way. +- **The model key enters the sandbox:** an agent's `LITELLM_API_KEY` is passed in, as on other providers. + +## Testing + +`tst/integration/providers/sandbox_providers/tensorlake_sandbox_smoke_test.py` runs against a real +Tensorlake account when `[sandbox.providers.tensorlake.config]` resolves. Otherwise it skips with +`agentenv-capability-missing: remote_sandbox`. The unit tests are +`tst/unit/providers/sandbox_providers/tensorlake_sandbox_provider_test.py` and `tensorlake_sandbox_test.py`. diff --git a/src/agent_env/providers/sandbox_providers/tensorlake/__init__.py b/src/agent_env/providers/sandbox_providers/tensorlake/__init__.py new file mode 100644 index 00000000..2613b860 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/tensorlake/__init__.py @@ -0,0 +1 @@ +"""Tensorlake sandbox provider; needs the ``tensorlake`` extra.""" diff --git a/src/agent_env/providers/sandbox_providers/tensorlake/host_image.Dockerfile b/src/agent_env/providers/sandbox_providers/tensorlake/host_image.Dockerfile new file mode 100644 index 00000000..18c0f711 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/tensorlake/host_image.Dockerfile @@ -0,0 +1,15 @@ +# The Docker-capable host image Tensorlake sandboxes boot from, published as agentenv-dind-host-v1. +# Tensorlake runs dockerd only as the systemd docker.service. A changed file is published under a new version. +FROM tensorlake/ubuntu-systemd + +RUN export DEBIAN_FRONTEND=noninteractive \ + && apt-get update \ + && apt-get install -y ca-certificates curl \ + && install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \ + && chmod a+r /etc/apt/keyrings/docker.asc \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" > /etc/apt/sources.list.d/docker.list \ + && apt-get update \ + && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \ + && (systemctl enable docker || true) \ + && rm -rf /var/lib/apt/lists/* diff --git a/src/agent_env/providers/sandbox_providers/tensorlake/provider.py b/src/agent_env/providers/sandbox_providers/tensorlake/provider.py new file mode 100644 index 00000000..087a97c7 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/tensorlake/provider.py @@ -0,0 +1,338 @@ +"""Tensorlake VM sandbox provider. + +Each sandbox is an ephemeral Tensorlake MicroVM booted from a Docker-capable host image, by default +the public ``agentenv-dind-host-v1`` built from ``host_image.Dockerfile``. +The gateway and agent containers run inside it through the inherited ``VmSandbox`` helpers. +""" + +from __future__ import annotations + +import asyncio +import functools +import logging +import math +from collections.abc import Awaitable, Callable +from ipaddress import ip_network +from pathlib import Path +from typing import Any, ClassVar, Self + +from tensorlake.sandbox import AsyncSandbox, AsyncSandboxClient, RemoteAPIError, SandboxNotFoundError + +from agent_env.attribution import PIPELINE_STEP_KEY, RUN_ID_KEY, Attribution +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy +from agent_env.providers.sandbox_providers.sandbox_provider import ( + SandboxProvider, + apply_default_attribution, + refuse_unenforceable_policy, +) +from agent_env.providers.sandbox_providers.tensorlake.sandbox import ( + TensorlakeSandbox, + network_config_fields, + network_policy_from_config, +) + +logger = logging.getLogger(__name__) + +# Tensorlake hosts run without KVM, so cores are slower than the request suggests. +_MIN_CPUS = 2 +_MIN_MEMORY_MB = 4 * 1024 +_MIN_MEMORY_MB_PER_CPU = 1024 +_MAX_MEMORY_MB_PER_CPU = 8 * 1024 +# The disk the host image is published with; a smaller sandbox disk cannot hold it. +_MIN_DISK_MB = 30 * 1024 +_MAX_DISK_MB = 100 * 1024 +# The disk the image builder needs to build the host image. +_BUILDER_DISK_MB = 24 * 1024 +# Passed to every SDK call: the SDK otherwise falls back to TENSORLAKE_API_URL in some calls +# and to the public endpoint in others, so creation and cleanup could hit different services. +DEFAULT_API_URL = "https://api.tensorlake.ai" +DEFAULT_IMAGE = "agentenv-dind-host-v1" +HOST_IMAGE_DOCKERFILE = Path(__file__).with_name("host_image.Dockerfile") +SANDBOX_STARTED_EVENT = "agent_env.tensorlake_sandbox_started" +_CONFIG_KEYS = ("api_key", "image", "api_url") + +_reapers: set[asyncio.Task] = set() + + +class TensorlakeSandboxProvider(SandboxProvider): + """Provision Docker-capable Tensorlake sandboxes. + + ``api_key`` comes from resolved provider config, never from the process environment: + configure it with a ``secret:`` reference in ``[sandbox.providers.tensorlake.config]``. + ``image`` names a registered Docker-capable image and defaults to the public host image. + ``api_url`` defaults to the public Tensorlake API. + """ + + # Exposed ports are served at ``https://-.sandbox.tensorlake.ai``. + EGRESS_HOSTS: ClassVar[tuple[str, ...]] = ("*.sandbox.tensorlake.ai",) + + def __init__( + self, + *, + api_key: str, + image: str = DEFAULT_IMAGE, + api_url: str = DEFAULT_API_URL, + sandbox_cls: Any = AsyncSandbox, + client_cls: Any = AsyncSandboxClient, + ): + if not isinstance(api_key, str) or not api_key.strip(): + raise ValueError("Tensorlake api_key must be a non-empty resolved secret") + if not isinstance(image, str) or not image.strip(): + raise ValueError("Tensorlake image must be a non-empty registered image name") + if not isinstance(api_url, str) or not api_url.strip(): + raise ValueError("Tensorlake api_url must be a non-empty URL") + self._api_key = api_key + self._image = image.strip() + self._api_url = api_url.strip() + # Injectable so unit tests never build an SDK client. + self._sandbox_cls = sandbox_cls + self._client_cls = client_cls + + @classmethod + def from_config(cls, **config: Any) -> Self: + section = "[sandbox.providers.tensorlake.config]" + unknown = set(config) - set(_CONFIG_KEYS) + if unknown: + raise ConfigError(f"{section} has unknown key(s): {sorted(unknown)}") + if not config.get("api_key"): + raise ConfigError(f"{section} requires a non-empty 'api_key'") + for key in _CONFIG_KEYS: + value = config.get(key) + if value is not None and (not isinstance(value, str) or not value.strip()): + raise ConfigError(f"{section} '{key}' must be a non-empty string") + return cls(**config) + + @classmethod + def supports_network_policy(cls, policy: NetworkPolicy) -> bool: + """Tensorlake filters IPv4 addresses, IPv4 CIDRs and hostnames; it has no IPv6 rules.""" + for entry in (*policy.allow_hosts, *policy.allow_cidrs): + try: + if ip_network(entry, strict=False).version == 6: + return False + except ValueError: + continue + return True + + def _client_kwargs(self) -> dict[str, Any]: + return {"api_key": self._api_key, "api_url": self._api_url} + + @staticmethod + def _resources(cpu: float, memory: int, disk_size_gb: float) -> tuple[float, int, int]: + """Raise a request to Tensorlake's floors, and reject memory it cannot give that many CPUs.""" + cpus = max(float(cpu), _MIN_CPUS) + memory_mb = max(int(memory), _MIN_MEMORY_MB, math.ceil(cpus * _MIN_MEMORY_MB_PER_CPU)) + if memory_mb > cpus * _MAX_MEMORY_MB_PER_CPU: + raise ValueError( + f"Tensorlake allows at most {_MAX_MEMORY_MB_PER_CPU} MB of memory per CPU; " + f"requested {memory_mb} MB with {cpus:g} CPUs" + ) + disk_mb = max(math.ceil(float(disk_size_gb) * 1024), _MIN_DISK_MB) + if disk_mb > _MAX_DISK_MB: + raise ValueError(f"Tensorlake allows at most {_MAX_DISK_MB // 1024} GB of disk; requested {disk_size_gb} GB") + if (cpus, memory_mb, disk_mb) != (cpu, memory, float(disk_size_gb) * 1024): + logger.info( + "Raised the Tensorlake sandbox request from cpu=%s, memory=%sMB, disk=%sGB " + "to cpu=%s, memory=%sMB, disk=%sMB", + cpu, memory, disk_size_gb, cpus, memory_mb, disk_mb, + ) + return cpus, memory_mb, disk_mb + + async def create_vm( + self, + *, + image: str | None = None, + boot_mode: str | None = None, + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + exposed_ports: list[int] | None = None, + setup_for_gateway: bool = True, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> TensorlakeSandbox: + """Create an ephemeral Tensorlake sandbox and expose ``exposed_ports`` publicly. + + ``image`` overrides the configured image and must be a registered Docker-capable + Tensorlake image. ``timeout`` becomes Tensorlake's idle timeout, after which the + sandbox terminates. ``boot_mode`` has no Tensorlake equivalent and is ignored. Tensorlake + sandboxes have no labels, so ``attribution`` goes into the started event, keyed by sandbox id. + """ + refuse_unenforceable_policy(self, network_policy) + if boot_mode is not None: + logger.warning("Ignoring boot_mode=%s: Tensorlake sandboxes have no boot modes", boot_mode) + cpus, memory_mb, disk_mb = self._resources(cpu, memory, disk_size_gb) + effective_policy = self.effective_network_policy(network_policy) + image_name = image or self._image + ports = list(dict.fromkeys(exposed_ports or [])) + logger.info( + "Creating Tensorlake sandbox (image=%s, ports=%s, cpus=%s, memory=%sMB, disk=%sMB, timeout=%ss)", + image_name, ports, cpus, memory_mb, disk_mb, timeout, + ) + # Without wait=False the SDK leaves a sandbox queued on timeout and its id is + # unknown if the caller cancels, so a failed or abandoned start could leak it. + resolved_attribution = { + key: str(value) for key, value in apply_default_attribution(dict(attribution or {})).items() if value is not None + } + try: + pending = await self._create_or_reclaim(self._sandbox_cls.create( + image=image_name, + cpus=cpus, + memory_mb=memory_mb, + disk_mb=disk_mb, + timeout_secs=timeout, + **network_config_fields(effective_policy), + **self._client_kwargs(), + wait=False, + )) + except RemoteAPIError as exc: + if exc.status_code == 400 and "not registered" in str(exc): + raise ValueError(self._unregistered_image_message(image_name)) from exc + raise + try: + raw_sandbox = await pending.ready() + except BaseException: + await self._delete_sandbox(pending.sandbox_id) + raise + try: + tunnel_urls: dict[int, str] = {} + if ports: + # Public ports match the other remote backends; a Bearer header would hand + # the Tensorlake API key to every workload that calls the URL. + info = await raw_sandbox.update(exposed_ports=ports, allow_unauthenticated_access=True) + tunnel_urls = self._tunnel_urls(info, ports) + sandbox = TensorlakeSandbox( + raw_sandbox, read_info=self._info_reader(raw_sandbox.sandbox_id), tunnel_urls=tunnel_urls, + network_policy=effective_policy, + ) + logger.info( + "Tensorlake sandbox started: sandbox_id=%s image=%s attribution=%s", + raw_sandbox.sandbox_id, image_name, resolved_attribution, + extra={ + "event": SANDBOX_STARTED_EVENT, + "tensorlake_sandbox_id": raw_sandbox.sandbox_id, + "tensorlake_attribution": resolved_attribution, + PIPELINE_STEP_KEY: resolved_attribution.get(PIPELINE_STEP_KEY), + RUN_ID_KEY: resolved_attribution.get(RUN_ID_KEY), + "cpus": cpus, + "memory_mb": memory_mb, + "disk_mb": disk_mb, + }, + ) + if setup_for_gateway: + await sandbox.setup_vm_for_gateway(ports) + return sandbox + except BaseException: + try: + await raw_sandbox.terminate() + except Exception as cleanup_error: # noqa: BLE001 - cleanup must not mask the create failure + logger.warning( + "Failed to terminate Tensorlake sandbox %s after setup failure: %s", + raw_sandbox.sandbox_id, cleanup_error, + ) + raise + + async def _create_or_reclaim(self, create: Any) -> Any: + """Await a create. If the caller is cancelled first, wait for the create to settle and delete the + sandbox it yields before the cancellation goes on: nothing else holds its id, and ``asyncio.run`` + cancels tasks still pending when it ends. Only a second cancellation leaves that to a callback.""" + task = asyncio.ensure_future(create) + + def delete_orphan(done: asyncio.Future) -> None: + if done.cancelled() or done.exception() is not None: + return + reaper = asyncio.ensure_future(self._delete_sandbox(done.result().sandbox_id)) + _reapers.add(reaper) + reaper.add_done_callback(_reapers.discard) + + try: + return await asyncio.shield(task) + except asyncio.CancelledError: + try: + created = await asyncio.shield(task) + except asyncio.CancelledError: + task.add_done_callback(delete_orphan) + raise + except Exception as exc: # noqa: BLE001 - a create that failed left nothing to delete + logger.debug("Tensorlake create cancelled by the caller then failed: %s", exc) + else: + await self._delete_sandbox(created.sandbox_id) + raise + + def _unregistered_image_message(self, image_name: str) -> str: + return ( + f"Tensorlake image {image_name!r} is not registered at {self._api_url}. Publish it once with:\n" + f" tl sbx image create {HOST_IMAGE_DOCKERFILE} -n {image_name} " + f"--disk_mb {_MIN_DISK_MB} --builder_disk_mb {_BUILDER_DISK_MB} --docker_compat\n" + "or set `image` in [sandbox.providers.tensorlake.config] to a registered Docker-capable image." + ) + + async def _delete_sandbox(self, sandbox_id: str) -> None: + try: + async with self._client_cls.for_cloud(**self._client_kwargs()) as client: + await client.delete(sandbox_id) + except SandboxNotFoundError: + pass + except Exception as cleanup_error: # noqa: BLE001 - cleanup must not mask the create failure + logger.warning("Failed to delete Tensorlake sandbox %s after start failure: %s", sandbox_id, cleanup_error) + + def _info_reader(self, sandbox_id: str) -> Callable[[], Awaitable[Any]]: + return functools.partial(self._live_info, sandbox_id) + + async def _live_info(self, sandbox_id: str) -> Any: + """The sandbox as the API reports it now: ``AsyncSandbox.info`` returns the handle's cached copy, so two + handles of one sandbox would each see only their own updates.""" + async with self._client_cls.for_cloud(**self._client_kwargs()) as client: + return (await client.get(sandbox_id)).value + + @staticmethod + def _tunnel_urls(info: Any, ports: list[int]) -> dict[int, str]: + urls: dict[int, str] = {} + for port in ports: + url = info.url_for_port(port) + if url is None: + raise RuntimeError(f"Tensorlake returned no ingress endpoint for port {port}") + urls[port] = url + return urls + + async def create_sandbox( + self, + *, + image_name: str, + port: int, + env: dict[str, str], + cpu: float = 1.0, + memory: int = 8192, + disk_size_gb: float = 10, + timeout: int = 3600 * 2, + attribution: Attribution | None = None, + network_policy: NetworkPolicy | None = None, + ) -> TensorlakeSandbox: + # A VM backend: the gateway and agent paths start image_name inside the VM afterwards. + del image_name, env + return await self.create_vm( + cpu=cpu, + memory=memory, + disk_size_gb=disk_size_gb, + timeout=timeout, + exposed_ports=[port], + attribution=attribution, + network_policy=network_policy, + ) + + async def get_sandbox(self, sandbox_id: str) -> TensorlakeSandbox: + raw_sandbox = await self._sandbox_cls.connect(sandbox_id, **self._client_kwargs()) + info = await raw_sandbox.info() + # A sandbox that isn't running has no ingress endpoint; teardown still has to reach it to terminate it. + tunnel_urls = { + port: url for port in info.exposed_ports or [] if (url := info.url_for_port(port)) is not None + } + sandbox = TensorlakeSandbox(raw_sandbox, read_info=self._info_reader(sandbox_id), tunnel_urls=tunnel_urls) + try: + sandbox.network_policy = network_policy_from_config(info.network_policy, sandbox_id=sandbox_id) + except RuntimeError as exc: + logger.warning( + "Reconnected to Tensorlake sandbox %s, but image loading will fail closed: %s", sandbox_id, exc, + ) + return sandbox diff --git a/src/agent_env/providers/sandbox_providers/tensorlake/sandbox.py b/src/agent_env/providers/sandbox_providers/tensorlake/sandbox.py new file mode 100644 index 00000000..36467db4 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/tensorlake/sandbox.py @@ -0,0 +1,356 @@ +"""Tensorlake's async sandbox presented through the :class:`VmSandbox` contract. + +Tensorlake applies the egress policy outside the VM, so one policy covers ``dockerd`` and every +container it runs. Public ports go through the Tensorlake ingress proxy, so the guest needs no +firewall rules. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import hashlib +import io +import logging +import shlex +import time +import uuid +import weakref +from collections.abc import AsyncIterator, Awaitable, Callable, Mapping +from ipaddress import ip_network +from types import SimpleNamespace +from typing import TYPE_CHECKING, Any +from urllib.parse import urlparse + +from tensorlake.sandbox import NetworkConfig, SandboxConnectionError, SandboxNotFoundError + +from agent_env.config import get_config +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, VmSandbox +from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM + +if TYPE_CHECKING: + from agent_env.store.object_store import ObjectStore + +logger = logging.getLogger(__name__) + + +def network_config_fields(policy: NetworkPolicy) -> dict[str, Any]: + """``policy`` as Tensorlake ``NetworkConfig`` fields, which ``create`` also takes as keywords. + + A non-empty ``allow_out`` is default-deny; an empty one allows everything, so an empty + allowlist has to turn internet access off instead. + """ + entries = [*policy.allow_hosts, *policy.allow_cidrs] + return { + "allow_internet_access": not policy.restricts_egress or bool(entries), + "allow_out": entries if policy.restricts_egress else [], + "deny_out": [], + } + + +def network_policy_from_config(network: Any, *, sandbox_id: str) -> NetworkPolicy: + """Translate Tensorlake's applied ``NetworkConfig`` back, failing on a policy agent-env did not set. + + Image loading extends the applied policy with download hosts, and extending a misread one + is a security bug, so anything unrecognized raises. + """ + if network is None: + return NetworkPolicy() + allow_out = list(getattr(network, "allow_out", None) or []) + deny_out = list(getattr(network, "deny_out", None) or []) + allow_internet = getattr(network, "allow_internet_access", True) + if deny_out or (allow_out and not allow_internet): + raise RuntimeError( + f"Cannot recover the network policy of Tensorlake sandbox {sandbox_id}: " + f"allow_internet_access={allow_internet}, allow_out={allow_out}, deny_out={deny_out} " + "is not a policy agent-env applies" + ) + if not allow_out: + return NetworkPolicy() if allow_internet else NetworkPolicy(mode=NetworkMode.ALLOWLIST) + hosts: list[str] = [] + cidrs: list[str] = [] + for entry in allow_out: + try: + ip_network(entry, strict=False) + except ValueError: + hosts.append(entry) + else: + cidrs.append(entry) + return NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=tuple(hosts), allow_cidrs=tuple(cidrs)) + + +# One lock per sandbox in each event loop, held to widen its egress policy, so concurrent downloads don't each widen +# a stale copy and drop the other's host. +_policy_locks: weakref.WeakValueDictionary[tuple[asyncio.AbstractEventLoop, str], asyncio.Lock] = ( + weakref.WeakValueDictionary() +) + +# What exec returns when the transport drops without an exit status; exec_script retries only this. +_UNKNOWN_EXIT_CODE = -1 + + +class _BytesReader: + """The small ``StreamReader`` subset consumed by ``Sandbox.exec_with_output``.""" + + def __init__(self, value: str | bytes | None): + self._value = value.encode() if isinstance(value, str) else (value or b"") + + async def read(self) -> bytes: + return self._value + + +class _CompletedProcess: + """Adapt Tensorlake's completed ``CommandResult`` to the common process protocol.""" + + def __init__(self, result: Any): + self.stdout = _BytesReader(getattr(result, "stdout", "")) + self.stderr = _BytesReader(getattr(result, "stderr", "")) + self._exit_code = int(getattr(result, "exit_code", _UNKNOWN_EXIT_CODE)) + + async def wait(self) -> int: + return self._exit_code + + +class TensorlakeSandbox(VmSandbox): + """A Tensorlake MicroVM with a systemd-managed Docker daemon. + + ``sandbox`` is an already-created ``tensorlake.sandbox.AsyncSandbox``; the provider owns + creation, ports and reconnect. ``read_info`` reads the sandbox's info from the API, not the handle's cache. + """ + + type = "tensorlake" + _DOCKER_READINESS_COMMAND_TIMEOUT = 10 + _DOCKER_LOG_TAIL_LINES = 40 + # The ingress proxy rejects request bodies above about 4 MB. + _UPLOAD_CHUNK_BYTES = 512 * 1024 + _UPLOADS_IN_FLIGHT = 8 + # The join re-runs only when the transport drops; it rewrites the whole file, so a retry is safe. + _JOIN_RETRIES = 2 + # A policy change reaches the host firewall in about 5-10 s. + _EGRESS_PROPAGATION_TIMEOUT = 60 + _EGRESS_PROBE_INTERVAL = 2 + _EGRESS_PROBE_TIMEOUT = 5 + + def __init__( + self, + sandbox: Any, + *, + read_info: Callable[[], Awaitable[Any]], + tunnel_urls: Mapping[int, str] | None = None, + network_policy: NetworkPolicy | None = None, + ): + self._sandbox = sandbox + self._read_info = read_info + self.sandbox_id = sandbox.sandbox_id + self.mode = SANDBOX_MODE_VM + self.vnc_url = None + self.tunnel_urls = dict(tunnel_urls or {}) + self.network_policy = network_policy + + async def terminate(self) -> None: + """Delete the sandbox; one that is already gone is not an error.""" + try: + await self._sandbox.terminate() + except SandboxNotFoundError: + logger.info("Tensorlake sandbox %s was already absent during terminate", self.sandbox_id) + + async def exec(self, *command: str) -> _CompletedProcess: + """Run argv directly; commands run as ``tl-user``, so the inherited ``sudo`` prefixes stay.""" + return await self._run_command(*command, timeout=None) + + async def _run_command(self, *command: str, timeout: float | None) -> _CompletedProcess: + try: + result = await self._sandbox.run(command[0], args=list(command[1:]), timeout=timeout) + except SandboxConnectionError as exc: + return _CompletedProcess(SimpleNamespace(exit_code=_UNKNOWN_EXIT_CODE, stdout="", stderr=str(exc))) + return _CompletedProcess(result) + + async def _exec_with_output_bounded(self, *command: str, timeout: float) -> tuple[int, str, str]: + process = await self._run_command(*command, timeout=timeout) + stdout, stderr = await asyncio.gather(process.stdout.read(), process.stderr.read()) + return await process.wait(), stdout.decode(), stderr.decode() + + async def setup_vm_for_gateway(self, exposed_ports: list[int] | None = None) -> None: + """Start Docker and check for Compose v2, which ``deploy_env`` needs; ports need no firewall.""" + await self.wait_for_vm() + exit_code, version, stderr = await self._exec_with_output_bounded( + "sudo", "docker", "compose", "version", timeout=self._DOCKER_READINESS_COMMAND_TIMEOUT, + ) + if exit_code != 0: + detail = (stderr or version).strip() + raise RuntimeError( + "Tensorlake sandbox image must include the Docker Compose v2 plugin required by deploy-env" + + (f": {detail}" if detail else "") + ) + logger.info("Docker Compose ready in Tensorlake sandbox %s: %s", self.sandbox_id, version.strip()) + + async def wait_for_vm(self) -> None: + """Start the systemd ``docker.service`` and poll ``docker info`` until it answers. + + The platform can stop a ``dockerd`` started by hand, so there is no manual fallback. + """ + deadline = time.monotonic() + self._VM_READY_TIMEOUT + try: + exit_code, _, stderr = await self._exec_with_output_bounded( + "sudo", "systemctl", "start", "docker", timeout=self._DOCKER_READINESS_COMMAND_TIMEOUT * 6, + ) + if exit_code != 0: + logger.debug("systemctl start docker exited %s in %s: %s", exit_code, self.sandbox_id, stderr) + except Exception as exc: + logger.debug("systemctl start docker did not complete in %s: %s", self.sandbox_id, exc) + + attempts = 0 + while time.monotonic() < deadline: + attempts += 1 + timeout = max(0.001, min(self._DOCKER_READINESS_COMMAND_TIMEOUT, deadline - time.monotonic())) + try: + exit_code, _, _ = await self._exec_with_output_bounded("sudo", "docker", "info", timeout=timeout) + except Exception as exc: + logger.debug("Docker readiness probe failed in Tensorlake sandbox %s: %s", self.sandbox_id, exc) + exit_code = -1 + if exit_code == 0: + logger.info("Docker ready in Tensorlake sandbox %s after %s polls", self.sandbox_id, attempts) + return + sleep_seconds = min(self._VM_READY_POLL_INTERVAL, max(0.0, deadline - time.monotonic())) + if sleep_seconds: + await asyncio.sleep(sleep_seconds) + + try: + _, log_tail, _ = await self._exec_with_output_bounded( + "sudo", "journalctl", "-u", "docker", "--no-pager", "-n", str(self._DOCKER_LOG_TAIL_LINES), + timeout=self._DOCKER_READINESS_COMMAND_TIMEOUT, + ) + except Exception as exc: + log_tail = f"unavailable ({type(exc).__name__}: {exc})" + raise RuntimeError( + f"Docker not ready in Tensorlake sandbox {self.sandbox_id} after {self._VM_READY_TIMEOUT}s " + f"({attempts} polls); docker.service journal tail:\n{log_tail}" + ) + + async def _write_bytes_to_vm_path(self, data: bytes, vm_path: str) -> None: + await self._upload_parts(_read_parts(io.BytesIO(data), self._UPLOAD_CHUNK_BYTES), vm_path) + + async def _write_unsigned_object(self, object_store: ObjectStore, object_url: str, vm_path: str) -> None: + """Stream an object the store can't sign through the file API, not as base64 over exec.""" + with contextlib.closing(await asyncio.to_thread(object_store.open, object_url)) as source: + await self._upload_parts(_read_parts(source, self._UPLOAD_CHUNK_BYTES), vm_path) + + async def _upload_parts(self, parts: AsyncIterator[bytes], vm_path: str) -> None: + """Upload ``parts`` through the file API, several at once, then join them as root at ``vm_path`` + and check its sha256. + + The parts go to /tmp because ``vm_path`` may be in a root-owned directory. The join and + cleanup loop over the part indexes, so the command stays short for any payload size. + """ + stem = f"/tmp/_tl_upload_{uuid.uuid4().hex[:12]}" + digest = hashlib.sha256() + gate = asyncio.Semaphore(self._UPLOADS_IN_FLIGHT) + count = 0 + + async def upload(path: str, part: bytes) -> None: + try: + await self._sandbox.write_file(path, part) + finally: + gate.release() + + try: + try: + async with asyncio.TaskGroup() as group: + async for part in parts: + await gate.acquire() + digest.update(part) + group.create_task(upload(f"{stem}.{count}", part)) + count += 1 + except ExceptionGroup as failed: + raise failed.exceptions[0] + quoted = shlex.quote(vm_path) + written = await self.exec_script( + f'for i in $(seq 0 {count - 1}); do cat "{stem}.$i" || exit 1; done > {quoted} && sha256sum {quoted}', + max_retries=self._JOIN_RETRIES, + ) + finally: + if count: + try: + await self.exec_script(f'for i in $(seq 0 {count - 1}); do rm -f "{stem}.$i"; done') + except Exception as e: + logger.warning("Best-effort cleanup of %s.* failed (ignored): %s", stem, e) + actual = written.split()[0] if written.split() else "" + if actual != digest.hexdigest(): + raise RuntimeError( + f"{vm_path} in Tensorlake sandbox {self.sandbox_id} does not match the upload: its sha256 is " + f"{actual or 'missing'}, the upload's {digest.hexdigest()}" + ) + + async def apply_network_policy(self, policy: NetworkPolicy) -> None: + """Replace the sandbox's egress policy; Tensorlake rejects the update if a host does not resolve.""" + await self._sandbox.update(network=NetworkConfig(**network_config_fields(policy))) + self.network_policy = policy + + async def _load_tarballs(self, artifacts: list) -> None: + """Load image tarballs after adding their signed-download hosts to a restrictive policy. + + The object store that serves image tarballs is infrastructure, not a workload + destination. A reconnected sandbox whose policy is unknown fails closed. + """ + policy = self.network_policy + if policy is None: + raise RuntimeError( + f"Cannot load Docker images in reconnected Tensorlake sandbox {self.sandbox_id}: " + "its applied network policy is unknown, so signed download hosts cannot be added safely" + ) + if not policy.restricts_egress: + await super()._load_tarballs(artifacts) + return + signed_urls = await self._signed_image_urls(artifacts) + await self._allow_download_hosts(signed_urls) + await self._load_docker_images(artifacts, signed_urls) + + async def _download_object_to_vm(self, object_url: str, vm_path: str) -> None: + """Download an object after adding its signed-download host to a restrictive policy, as image tarballs are. + A reconnected sandbox, whose policy is unknown, downloads as before.""" + policy = self.network_policy + if policy is not None and policy.restricts_egress: + signed = await asyncio.to_thread(get_config().get_object_store_at(object_url).signed_get_url, object_url) + await self._allow_download_hosts([signed]) + await super()._download_object_to_vm(object_url, vm_path) + + async def _allow_download_hosts(self, signed_urls: list[str | None]) -> None: + """Add the hosts of ``signed_urls`` to a restrictive applied policy and wait for them, under the sandbox's + lock, so a concurrent download's host isn't dropped and no download starts before its host is reachable.""" + hosts = {parsed.hostname for url in signed_urls if url and (parsed := urlparse(url)).hostname} + if not hosts: + return + async with _policy_locks.setdefault((asyncio.get_running_loop(), self.sandbox_id), asyncio.Lock()): + # Another wrapper of this sandbox may have widened the policy since this one last read it. + info = await self._read_info() + policy = network_policy_from_config(info.network_policy, sandbox_id=self.sandbox_id) + new_hosts = sorted(hosts - set(policy.allow_hosts)) + if new_hosts: + await self.apply_network_policy(policy.with_hosts(new_hosts)) + await self._wait_for_egress(new_hosts) + + async def _wait_for_egress(self, hosts: list[str]) -> None: + """Wait until each host accepts a connection, since a policy change is not instant.""" + deadline = time.monotonic() + self._EGRESS_PROPAGATION_TIMEOUT + probes = " && ".join( + f"curl -s -o /dev/null --max-time {self._EGRESS_PROBE_TIMEOUT} {shlex.quote(f'https://{host}/')}" + for host in hosts + ) + while (remaining := deadline - time.monotonic()) > 0: + try: + exit_code, _, _ = await self._exec_with_output_bounded("sh", "-c", probes, timeout=remaining) + except Exception as exc: + logger.debug("Egress probe failed in Tensorlake sandbox %s: %s", self.sandbox_id, exc) + exit_code = -1 + if exit_code == 0: + return + await asyncio.sleep(min(self._EGRESS_PROBE_INTERVAL, max(0.0, deadline - time.monotonic()))) + logger.warning( + "Tensorlake sandbox %s still cannot reach %s after %ss; loading images anyway", + self.sandbox_id, hosts, self._EGRESS_PROPAGATION_TIMEOUT, + ) + + +async def _read_parts(source: Any, size: int) -> AsyncIterator[bytes]: + while part := await asyncio.to_thread(source.read, size): + yield part diff --git a/tst/integration/env/gateway/gateway_test.py b/tst/integration/env/gateway/gateway_test.py index dc047b63..a1763333 100644 --- a/tst/integration/env/gateway/gateway_test.py +++ b/tst/integration/env/gateway/gateway_test.py @@ -449,6 +449,11 @@ def agentenv_website_env() -> WebsiteEnv: id="sail_vm", marks=skip_without_remote_sandbox("sail_vm"), ), + pytest.param( + "tensorlake", + id="tensorlake", + marks=skip_without_remote_sandbox("tensorlake"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( @@ -470,6 +475,9 @@ def sandbox_provider(request): elif request.param == "sail_vm": set_sandbox_provider(build_sandbox_provider("sail_vm")) set_env_sandbox_provider(build_sandbox_provider("sail_vm")) + elif request.param == "tensorlake": + set_sandbox_provider(build_sandbox_provider("tensorlake")) + set_env_sandbox_provider(build_sandbox_provider("tensorlake")) try: yield request.param finally: diff --git a/tst/integration/env/task_step/task_steps_test.py b/tst/integration/env/task_step/task_steps_test.py index 25a63d9a..03fd2174 100644 --- a/tst/integration/env/task_step/task_steps_test.py +++ b/tst/integration/env/task_step/task_steps_test.py @@ -279,6 +279,11 @@ def echo_agent() -> A2AAgent: id="sail_vm", marks=skip_without_remote_sandbox("sail_vm"), ), + pytest.param( + "tensorlake", + id="tensorlake", + marks=skip_without_remote_sandbox("tensorlake"), + ), ]) def sandbox_provider(request): from agent_env.providers import ( @@ -305,6 +310,10 @@ def sandbox_provider(request): set_sandbox_provider(build_sandbox_provider("sail_vm")) set_env_sandbox_provider(build_sandbox_provider("sail_vm")) set_agent_sandbox_provider(build_sandbox_provider("sail_vm")) + elif request.param == "tensorlake": + set_sandbox_provider(build_sandbox_provider("tensorlake")) + set_env_sandbox_provider(build_sandbox_provider("tensorlake")) + set_agent_sandbox_provider(build_sandbox_provider("tensorlake")) try: yield request.param finally: diff --git a/tst/integration/providers/sandbox_providers/tensorlake_sandbox_smoke_test.py b/tst/integration/providers/sandbox_providers/tensorlake_sandbox_smoke_test.py new file mode 100644 index 00000000..2109ab51 --- /dev/null +++ b/tst/integration/providers/sandbox_providers/tensorlake_sandbox_smoke_test.py @@ -0,0 +1,98 @@ +"""Tensorlake VM smoke test. + +Runs when the resolved config can build the ``tensorlake`` sandbox provider (the ``tensorlake`` extra +and ``[sandbox.providers.tensorlake]`` with ``api_key = "secret:tensorlake_api_key"``) — the +``remote_sandbox`` capability — and skips with the declared reason otherwise. The test never reads or +prints the API key. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import os + +import httpx +import pytest +import pytest_asyncio + +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider +from tst.util.capabilities import skip_without_remote_sandbox + +_PORT = 8080 + +pytestmark = [ + pytest.mark.integration, + pytest.mark.int_test_slow, + pytest.mark.asyncio, + skip_without_remote_sandbox("tensorlake"), +] + + +@pytest.fixture(scope="module") +def tensorlake_provider(): + """Build the configured provider without exposing its resolved API key.""" + return build_sandbox_provider("tensorlake") + + +@pytest_asyncio.fixture(scope="module") +async def tensorlake_sandbox(tensorlake_provider): + """Provision one Tensorlake VM and always terminate it after the smoke test.""" + sandbox = await tensorlake_provider.create_vm(exposed_ports=[_PORT], timeout=900) + try: + yield sandbox + finally: + await sandbox.terminate() + + +async def test_tensorlake_vm_docker_ports_upload_and_reconnect(tensorlake_provider, tensorlake_sandbox): + """Exercise the host image, nested Docker, a public port end to end, a multi-part upload, and reconnect.""" + assert tensorlake_sandbox.type == "tensorlake" + assert tensorlake_sandbox.mode == "vm" + assert tensorlake_sandbox.network_policy == NetworkPolicy() + + exit_code, version, stderr = await tensorlake_sandbox.exec_with_output( + "sudo", "docker", "info", "--format", "{{.ServerVersion}}" + ) + assert exit_code == 0, stderr + assert version.strip() + + await tensorlake_sandbox.exec_script( + f"docker run -d --name smoke -p {_PORT}:{_PORT} public.ecr.aws/docker/library/python:3.12-slim " + f"python -m http.server {_PORT}" + ) + url = tensorlake_sandbox.tunnel_urls[_PORT] + assert url.startswith("https://") + async with httpx.AsyncClient(timeout=10) as client: + for _ in range(30): + try: + if (await client.get(url)).status_code == 200: + break + except httpx.HTTPError: + pass + await asyncio.sleep(2) + else: + pytest.fail(f"{url} never answered 200") + + payload = os.urandom(1024 * 1024 + 17) + await tensorlake_sandbox.write_host_file(payload, "/opt/smoke/payload.bin") + digest = await tensorlake_sandbox.exec_script("sha256sum /opt/smoke/payload.bin") + assert digest.split()[0] == hashlib.sha256(payload).hexdigest() + + reconnected = await build_sandbox_provider("tensorlake").get_sandbox(tensorlake_sandbox.sandbox_id) + assert reconnected.sandbox_id == tensorlake_sandbox.sandbox_id + assert reconnected.tunnel_urls == tensorlake_sandbox.tunnel_urls + assert reconnected.network_policy == NetworkPolicy() + + +async def test_tensorlake_allowlist_survives_reconnect_and_terminate_is_idempotent(tensorlake_provider): + """A fresh provider reads back the applied allowlist, and a second terminate is not an error.""" + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("pypi.org",), allow_cidrs=("1.1.1.1/32",)) + sandbox = await tensorlake_provider.create_vm(timeout=600, setup_for_gateway=False, network_policy=policy) + try: + reconnected = await build_sandbox_provider("tensorlake").get_sandbox(sandbox.sandbox_id) + assert reconnected.network_policy == sandbox.network_policy + finally: + await sandbox.terminate() + await reconnected.terminate() diff --git a/tst/unit/providers/sandbox_providers/egress_hosts_test.py b/tst/unit/providers/sandbox_providers/egress_hosts_test.py index 7024b69d..81278575 100644 --- a/tst/unit/providers/sandbox_providers/egress_hosts_test.py +++ b/tst/unit/providers/sandbox_providers/egress_hosts_test.py @@ -46,6 +46,7 @@ def test_union_covers_every_registered_platforms_tunnel_hosts(platform_registry) assert _EXTERNAL in hosts # ...and its rewritten value assert "*.modal.host" in hosts # a Modal EGRESS_HOSTS entry assert "*.e2b.app" in hosts # E2B's - ingress URL + assert "*.sandbox.tensorlake.ai" in hosts def test_union_holds_only_platform_hosts(): @@ -95,4 +96,6 @@ def test_the_builtin_floor_is_pinned(builtin_registry): """Every restricted sandbox gets these, so an empty allowlist is not total denial. Pinned rather than counted: a new entry should be argued for in review. Platforms registered from config add theirs on top.""" - assert set(all_sandbox_egress_hosts()) == {"*.modal.host", "*.w.modal.host", "*.e2b.app", "*.sail.box"} + assert set(all_sandbox_egress_hosts()) == { + "*.modal.host", "*.w.modal.host", "*.e2b.app", "*.sail.box", "*.sandbox.tensorlake.ai", + } diff --git a/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py b/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py index be4aab6f..8e406696 100644 --- a/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py +++ b/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py @@ -28,6 +28,7 @@ from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox from agent_env.providers.sandbox_providers.sandbox import port_bindings +from agent_env.providers.sandbox_providers.tensorlake.sandbox import TensorlakeSandbox RESERVED = [AGENT_ENV_GATEWAY_MCP_PORT, DB_WEB_PORT, DB_MCP_PORT] @@ -50,7 +51,7 @@ def test_base_sandbox_host_port_is_identity(): assert Sandbox.host_port(None, 18765) == 18765 -@pytest.mark.parametrize("sandbox_class", [ModalVmSandbox, ModalSandbox, E2BSandbox]) +@pytest.mark.parametrize("sandbox_class", [ModalVmSandbox, ModalSandbox, E2BSandbox, TensorlakeSandbox]) def test_remote_sandboxes_publish_on_every_interface(sandbox_class): """Their tunnels reach a published port from outside the VM, so it stays on every interface.""" assert port_bindings(sandbox_class.host_ips, 18765, 18765) == ["18765:18765"] diff --git a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py index 0b801a53..467b21f5 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py @@ -21,6 +21,7 @@ from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import _BUILTIN_SANDBOX_PROVIDERS +from agent_env.providers.sandbox_providers.tensorlake.provider import TensorlakeSandboxProvider def _modal_provider(cls): @@ -86,11 +87,23 @@ async def _sail() -> Sandbox: return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) +async def _tensorlake() -> Sandbox: + sdk = MagicMock() + pending = MagicMock(sandbox_id="tl-test") + pending.ready = AsyncMock(return_value=MagicMock(sandbox_id="tl-test")) + sdk.create = AsyncMock(return_value=pending) + provider = TensorlakeSandboxProvider( + api_key="test-key", sandbox_cls=sdk, client_cls=MagicMock(), + ) + return await provider.create_vm(exposed_ports=[], setup_for_gateway=False) + + FACTORIES = { "modal": _modal, "modal_vm": _modal_vm, "e2b": _e2b, "sail_vm": _sail, + "tensorlake": _tensorlake, "local": _local, } diff --git a/tst/unit/providers/sandbox_providers/provider_capabilities_test.py b/tst/unit/providers/sandbox_providers/provider_capabilities_test.py index 73326044..33001b84 100644 --- a/tst/unit/providers/sandbox_providers/provider_capabilities_test.py +++ b/tst/unit/providers/sandbox_providers/provider_capabilities_test.py @@ -26,13 +26,15 @@ Accepts, SandboxProvider, ) +from agent_env.providers.sandbox_providers.tensorlake.provider import TensorlakeSandboxProvider +from agent_env.providers.sandbox_providers.tensorlake.sandbox import TensorlakeSandbox SRC = Path(__file__).resolve().parents[4] / "src" / "agent_env" PROVIDERS = SRC / "providers" / "sandbox_providers" LOADABLE, NAME, NAME_OR_CONTEXT = Accepts.LOADABLE, Accepts.NAME, Accepts.NAME_OR_CONTEXT PROVIDER_CLASSES = [LocalSandboxProvider, ModalSandboxProvider, ModalVmSandboxProvider, E2BSandboxProvider, - SailVmSandboxProvider, ChainedSandboxProvider] + SailVmSandboxProvider, TensorlakeSandboxProvider, ChainedSandboxProvider] @pytest.mark.parametrize("cls, on_this_machine, creates_vms, sandbox, container, private_network", [ @@ -41,7 +43,8 @@ (ModalVmSandboxProvider, False, True, LOADABLE, NAME, False), (E2BSandboxProvider, False, True, LOADABLE, NAME, False), (SailVmSandboxProvider, False, True, LOADABLE, NAME, False), -], ids=["local", "modal", "modal_vm", "e2b", "sail_vm"]) + (TensorlakeSandboxProvider, False, True, LOADABLE, NAME, False), +], ids=["local", "modal", "modal_vm", "e2b", "sail_vm", "tensorlake"]) def test_each_provider_declares_where_it_runs_and_the_images_it_runs(cls, on_this_machine, creates_vms, sandbox, container, private_network): assert ((cls.ON_THIS_MACHINE, SANDBOX_MODE_VM in cls.SUPPORTED_SANDBOX_MODES, cls.PRIVATE_NETWORK) @@ -230,7 +233,7 @@ def test_an_image_that_records_no_platform_isnt_checked(): assert image.platform is None and image.load_problem(on_this_machine=False, platform="linux/arm64") is None -@pytest.mark.parametrize("sandbox", [Sandbox, ModalSandbox, ModalVmSandbox, E2BSandbox, SailVmSandbox]) +@pytest.mark.parametrize("sandbox", [Sandbox, ModalSandbox, ModalVmSandbox, E2BSandbox, SailVmSandbox, TensorlakeSandbox]) def test_a_sandbox_runs_linux_amd64_unless_its_provider_sets_another(sandbox): assert sandbox.platform == "linux/amd64" diff --git a/tst/unit/providers/sandbox_providers/tensorlake_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/tensorlake_sandbox_provider_test.py new file mode 100644 index 00000000..788a49d9 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/tensorlake_sandbox_provider_test.py @@ -0,0 +1,453 @@ +"""Offline unit tests for the Tensorlake sandbox provider.""" + +from __future__ import annotations + +import asyncio +import logging +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest +from tensorlake.sandbox import RemoteAPIError, SandboxNotFoundError, SandboxPending + +from agent_env.config.errors import ConfigError +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, NetworkPolicyUnsupportedError +from agent_env.providers.sandbox_providers.tensorlake.provider import ( + HOST_IMAGE_DOCKERFILE, + SANDBOX_STARTED_EVENT, + TensorlakeSandboxProvider, +) +from agent_env.providers.sandbox_providers.tensorlake.sandbox import TensorlakeSandbox + + +def _info(*, ports=(), network=None): + info = MagicMock() + info.exposed_ports = list(ports) + info.network_policy = network + info.url_for_port.side_effect = lambda port: f"https://{port}-tl-test.sandbox.tensorlake.ai" + return info + + +def _raw_sandbox() -> MagicMock: + raw = MagicMock() + raw.sandbox_id = "tl-test" + raw.update = AsyncMock(side_effect=lambda **kwargs: _info(ports=kwargs.get("exposed_ports", ()))) + raw.terminate = AsyncMock() + raw.info = AsyncMock(return_value=_info()) + return raw + + +def _provider(raw: MagicMock | None = None, **kwargs) -> tuple[TensorlakeSandboxProvider, MagicMock]: + sdk = MagicMock() + sdk.pending = MagicMock(sandbox_id="tl-test") + sdk.pending.ready = AsyncMock(return_value=raw or _raw_sandbox()) + sdk.create = AsyncMock(return_value=sdk.pending) + sdk.connect = AsyncMock(return_value=raw or _raw_sandbox()) + sdk.client = MagicMock() + sdk.client.delete = AsyncMock() + client_cls = MagicMock() + client_cls.for_cloud.return_value.__aenter__ = AsyncMock(return_value=sdk.client) + client_cls.for_cloud.return_value.__aexit__ = AsyncMock(return_value=None) + sdk.client_cls = client_cls + provider = TensorlakeSandboxProvider( + api_key="tl-key", sandbox_cls=sdk, client_cls=client_cls, **kwargs, + ) + return provider, sdk + + +@pytest.fixture(autouse=True) +def _no_other_platform_hosts(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr(TensorlakeSandboxProvider, "effective_network_policy", classmethod( + lambda cls, policy: (policy or NetworkPolicy()).with_hosts(cls.EGRESS_HOSTS) + )) + + +def test_from_config_requires_an_api_key(): + with pytest.raises(ConfigError, match="requires a non-empty 'api_key'"): + TensorlakeSandboxProvider.from_config(image="my-image") + + +@pytest.mark.parametrize("key", ["sandbox_cls", "client_cls", "region"]) +def test_from_config_refuses_unknown_keys(key: str): + with pytest.raises(ConfigError, match=rf"unknown key\(s\): \['{key}'\]"): + TensorlakeSandboxProvider.from_config(api_key="tl-key", **{key: "x"}) + + +@pytest.mark.parametrize(("key", "value"), [("image", 1), ("image", " "), ("api_url", ["https://api.example"])]) +def test_from_config_refuses_a_non_string_or_blank_value(key: str, value): + with pytest.raises(ConfigError, match=f"'{key}' must be a non-empty string"): + TensorlakeSandboxProvider.from_config(api_key="tl-key", **{key: value}) + + +def test_from_config_takes_every_known_key(): + provider = TensorlakeSandboxProvider.from_config(api_key="tl-key", image="team-host", api_url="https://api.example") + + assert provider._client_kwargs() == {"api_key": "tl-key", "api_url": "https://api.example"} + assert provider._image == "team-host" + + +@pytest.mark.parametrize("bad_key", ["", " "]) +def test_constructor_rejects_a_blank_api_key(bad_key: str): + with pytest.raises(ValueError, match="api_key"): + TensorlakeSandboxProvider(api_key=bad_key) + + +def test_ipv6_policies_are_unsupported(): + assert TensorlakeSandboxProvider.supports_network_policy( + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example",), allow_cidrs=("10.0.0.0/8",)) + ) + assert not TensorlakeSandboxProvider.supports_network_policy( + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_cidrs=("2001:db8::/32",)) + ) + assert not TensorlakeSandboxProvider.supports_network_policy( + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("::1",)) + ) + + +@pytest.mark.asyncio +async def test_create_vm_refuses_an_ipv6_policy_before_creating_anything(): + provider, sdk = _provider() + + with pytest.raises(NetworkPolicyUnsupportedError): + await provider.create_vm( + network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_cidrs=("2001:db8::/32",)), + setup_for_gateway=False, + ) + + sdk.create.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_create_vm_logs_an_ignored_boot_mode_and_a_raised_disk(caplog: pytest.LogCaptureFixture): + provider, _ = _provider() + + with caplog.at_level(logging.INFO): + await provider.create_vm(boot_mode="uefi", cpu=2, memory=4096, disk_size_gb=10, setup_for_gateway=False) + + assert "Ignoring boot_mode=uefi" in caplog.text + assert "disk=10GB to cpu=2.0, memory=4096MB, disk=30720MB" in caplog.text + + +@pytest.mark.asyncio +async def test_create_vm_raises_resources_to_the_floors_and_exposes_ports_publicly(): + raw = _raw_sandbox() + provider, sdk = _provider(raw) + + sandbox = await provider.create_vm( + cpu=1, memory=2048, disk_size_gb=10, timeout=900, exposed_ports=[8080, 8080, 9000], setup_for_gateway=False, + ) + + assert isinstance(sandbox, TensorlakeSandbox) + sdk.create.assert_awaited_once_with( + image="agentenv-dind-host-v1", + cpus=2.0, + memory_mb=4096, + disk_mb=30 * 1024, + timeout_secs=900, + allow_internet_access=True, + allow_out=[], + deny_out=[], + api_key="tl-key", + api_url="https://api.tensorlake.ai", + wait=False, + ) + raw.update.assert_awaited_once_with(exposed_ports=[8080, 9000], allow_unauthenticated_access=True) + assert sandbox.tunnel_urls == { + 8080: "https://8080-tl-test.sandbox.tensorlake.ai", + 9000: "https://9000-tl-test.sandbox.tensorlake.ai", + } + assert sandbox.network_policy == NetworkPolicy() + + +@pytest.mark.asyncio +async def test_create_vm_keeps_a_request_above_the_floors(): + provider, sdk = _provider(api_url="https://api.example") + + await provider.create_vm(cpu=4, memory=16384, disk_size_gb=50, setup_for_gateway=False) + + kwargs = sdk.create.await_args.kwargs + assert (kwargs["cpus"], kwargs["memory_mb"], kwargs["disk_mb"]) == (4.0, 16384, 50 * 1024) + assert kwargs["api_url"] == "https://api.example" + + +@pytest.mark.asyncio +async def test_creation_and_cleanup_use_one_endpoint_whatever_the_environment(monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv("TENSORLAKE_API_URL", "https://ambient.example") + provider, sdk = _provider() + sdk.pending.ready.side_effect = SandboxPending("tl-test", pending_reason="no_capacity") + + with pytest.raises(SandboxPending): + await provider.create_vm(setup_for_gateway=False) + + assert sdk.create.await_args.kwargs["api_url"] == "https://api.tensorlake.ai" + assert sdk.client_cls.for_cloud.call_args.kwargs["api_url"] == "https://api.tensorlake.ai" + + +@pytest.mark.asyncio +async def test_create_vm_raises_memory_to_one_gb_per_cpu(): + provider, sdk = _provider() + + await provider.create_vm(cpu=6, memory=4096, setup_for_gateway=False) + + assert sdk.create.await_args.kwargs["memory_mb"] == 6 * 1024 + + +@pytest.mark.parametrize( + ("resources", "message"), + [({"cpu": 2, "memory": 32768}, "per CPU"), ({"disk_size_gb": 200}, "GB of disk")], +) +@pytest.mark.asyncio +async def test_create_vm_rejects_resources_tensorlake_cannot_give(resources, message): + provider, sdk = _provider() + + with pytest.raises(ValueError, match=message): + await provider.create_vm(setup_for_gateway=False, **resources) + + sdk.create.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_create_vm_sends_an_allowlist_with_the_ingress_domain(): + provider, sdk = _provider() + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example",), allow_cidrs=("10.0.0.0/8",)) + + sandbox = await provider.create_vm(network_policy=policy, setup_for_gateway=False) + + kwargs = sdk.create.await_args.kwargs + assert kwargs["allow_internet_access"] is True + assert kwargs["allow_out"] == ["api.example", "*.sandbox.tensorlake.ai", "10.0.0.0/8"] + assert sandbox.network_policy == policy.with_hosts(["*.sandbox.tensorlake.ai"]) + + +@pytest.mark.asyncio +async def test_create_vm_uses_a_configured_or_requested_image(): + provider, sdk = _provider(image="team-docker-host") + + await provider.create_vm(setup_for_gateway=False) + assert sdk.create.await_args.kwargs["image"] == "team-docker-host" + + await provider.create_vm(image="request-image", setup_for_gateway=False) + assert sdk.create.await_args.kwargs["image"] == "request-image" + + +@pytest.mark.asyncio +async def test_create_vm_explains_how_to_publish_an_unregistered_image(): + provider, sdk = _provider(api_url="https://api.example") + sdk.create.side_effect = RemoteAPIError( + 400, "Image 'agentenv-dind-host-v1' is not registered in the server. Register it first." + ) + + with pytest.raises(ValueError) as raised: + await provider.create_vm(setup_for_gateway=False) + + message = str(raised.value) + assert "'agentenv-dind-host-v1' is not registered at https://api.example" in message + assert "host_image.Dockerfile -n agentenv-dind-host-v1 --disk_mb 30720" in message + assert isinstance(raised.value.__cause__, RemoteAPIError) + + +@pytest.mark.asyncio +async def test_create_vm_lets_other_api_errors_through(): + provider, sdk = _provider() + error = RemoteAPIError(400, "cpus out of range") + sdk.create.side_effect = error + + with pytest.raises(RemoteAPIError) as raised: + await provider.create_vm(setup_for_gateway=False) + + assert raised.value is error + + +def test_the_host_image_dockerfile_ships_with_the_provider(): + assert HOST_IMAGE_DOCKERFILE.is_file() + assert "docker-compose-plugin" in HOST_IMAGE_DOCKERFILE.read_text() + + +@pytest.mark.asyncio +async def test_create_vm_terminates_the_sandbox_when_setup_fails(monkeypatch: pytest.MonkeyPatch): + raw = _raw_sandbox() + provider, _ = _provider(raw) + monkeypatch.setattr(TensorlakeSandbox, "setup_vm_for_gateway", AsyncMock(side_effect=RuntimeError("no docker"))) + + with pytest.raises(RuntimeError, match="no docker"): + await provider.create_vm(exposed_ports=[8080]) + + raw.terminate.assert_awaited_once_with() + + +@pytest.mark.asyncio +async def test_create_vm_deletes_a_sandbox_that_does_not_start(): + provider, sdk = _provider() + sdk.pending.ready.side_effect = SandboxPending("tl-test", pending_reason="no_capacity") + + with pytest.raises(SandboxPending): + await provider.create_vm(setup_for_gateway=False) + + sdk.client_cls.for_cloud.assert_called_once_with(api_key="tl-key", api_url="https://api.tensorlake.ai") + sdk.client.delete.assert_awaited_once_with("tl-test") + + +@pytest.mark.asyncio +async def test_create_vm_deletes_the_sandbox_when_the_caller_cancels_the_start(): + provider, sdk = _provider() + started = asyncio.Event() + + async def never_ready(): + started.set() + await asyncio.Event().wait() + + sdk.pending.ready.side_effect = never_ready + task = asyncio.create_task(provider.create_vm(setup_for_gateway=False)) + await started.wait() + task.cancel() + + with pytest.raises(asyncio.CancelledError): + await task + sdk.client.delete.assert_awaited_once_with("tl-test") + + +@pytest.mark.asyncio +async def test_a_cancelled_create_vm_deletes_the_sandbox_its_create_yields_before_it_ends(): + provider, sdk = _provider() + started, release = asyncio.Event(), asyncio.Event() + + async def slow_create(**kwargs): + started.set() + await release.wait() + return sdk.pending + + sdk.create.side_effect = slow_create + task = asyncio.create_task(provider.create_vm(setup_for_gateway=False)) + await started.wait() + task.cancel() + await asyncio.sleep(0) + + assert not task.done() + release.set() + with pytest.raises(asyncio.CancelledError): + await task + sdk.client.delete.assert_awaited_once_with("tl-test") + sdk.pending.ready.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_a_create_vm_cancelled_twice_leaves_the_delete_to_a_callback(): + provider, sdk = _provider() + started, release = asyncio.Event(), asyncio.Event() + + async def slow_create(**kwargs): + started.set() + await release.wait() + return sdk.pending + + sdk.create.side_effect = slow_create + task = asyncio.create_task(provider.create_vm(setup_for_gateway=False)) + await started.wait() + task.cancel() + await asyncio.sleep(0) + task.cancel() + + with pytest.raises(asyncio.CancelledError): + await task + sdk.client.delete.assert_not_awaited() + release.set() + for _ in range(5): + await asyncio.sleep(0) + sdk.client.delete.assert_awaited_once_with("tl-test") + + +@pytest.mark.asyncio +async def test_create_vm_logs_the_attribution_in_the_started_event(caplog: pytest.LogCaptureFixture): + caplog.set_level(logging.INFO, logger="agent_env.providers.sandbox_providers.tensorlake.provider") + provider, _ = _provider() + + await provider.create_vm(setup_for_gateway=False, attribution={"run_id": "inst-1", "team": "t", "unset": None}) + + (record,) = [r for r in caplog.records if getattr(r, "event", None) == SANDBOX_STARTED_EVENT] + assert record.tensorlake_sandbox_id == "tl-test" + assert record.tensorlake_attribution == {"run_id": "inst-1", "team": "t"} + assert record.run_id == "inst-1" + assert "tl-key" not in caplog.text + + +@pytest.mark.asyncio +async def test_create_vm_ignores_a_sandbox_already_gone_during_cleanup(): + provider, sdk = _provider() + sdk.pending.ready.side_effect = RuntimeError("failed") + sdk.client.delete.side_effect = SandboxNotFoundError("tl-test") + + with pytest.raises(RuntimeError, match="failed"): + await provider.create_vm(setup_for_gateway=False) + + +@pytest.mark.asyncio +async def test_create_sandbox_is_a_vm_with_one_port(): + raw = _raw_sandbox() + provider, _ = _provider(raw) + provider.create_vm = AsyncMock(return_value="vm") + + assert await provider.create_sandbox(image_name="agent:latest", port=8000, env={"A": "1"}, timeout=60) == "vm" + assert provider.create_vm.await_args.kwargs["exposed_ports"] == [8000] + + +@pytest.mark.asyncio +async def test_get_sandbox_recovers_ports_and_policy(): + raw = _raw_sandbox() + network = SimpleNamespace(allow_internet_access=True, allow_out=["api.example", "10.0.0.0/8"], deny_out=[]) + raw.info = AsyncMock(return_value=_info(ports=[8080], network=network)) + provider, sdk = _provider(raw) + + sandbox = await provider.get_sandbox("tl-test") + + sdk.connect.assert_awaited_once_with("tl-test", api_key="tl-key", api_url="https://api.tensorlake.ai") + assert sandbox.tunnel_urls == {8080: "https://8080-tl-test.sandbox.tensorlake.ai"} + assert sandbox.network_policy == NetworkPolicy( + mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example",), allow_cidrs=("10.0.0.0/8",) + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("make", ["create_vm", "get_sandbox"]) +async def test_the_sandbox_reads_its_info_from_the_api_not_the_handle_cache(make: str): + raw = _raw_sandbox() + provider, sdk = _provider(raw) + live = _info(ports=[8080]) + sdk.client.get = AsyncMock(return_value=SimpleNamespace(value=live)) + + if make == "create_vm": + sandbox = await provider.create_vm(setup_for_gateway=False) + else: + sandbox = await provider.get_sandbox("tl-test") + raw.info.reset_mock() + + assert await sandbox._read_info() is live + sdk.client.get.assert_awaited_once_with("tl-test") + raw.info.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_get_sandbox_without_an_ingress_endpoint_still_reconnects_for_teardown(): + raw = _raw_sandbox() + info = _info(ports=[8080]) + info.url_for_port.side_effect = lambda port: None + raw.info = AsyncMock(return_value=info) + provider, _ = _provider(raw) + + sandbox = await provider.get_sandbox("tl-test") + await sandbox.terminate() + + assert sandbox.tunnel_urls == {} + raw.terminate.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_get_sandbox_with_an_unknown_policy_fails_closed(): + raw = _raw_sandbox() + raw.info = AsyncMock( + return_value=_info(network=SimpleNamespace(allow_internet_access=True, allow_out=[], deny_out=["1.2.3.4"])) + ) + provider, _ = _provider(raw) + + sandbox = await provider.get_sandbox("tl-test") + + assert sandbox.network_policy is None diff --git a/tst/unit/providers/sandbox_providers/tensorlake_sandbox_test.py b/tst/unit/providers/sandbox_providers/tensorlake_sandbox_test.py new file mode 100644 index 00000000..6fd8e116 --- /dev/null +++ b/tst/unit/providers/sandbox_providers/tensorlake_sandbox_test.py @@ -0,0 +1,486 @@ +"""Offline unit tests for the Tensorlake VmSandbox adapter.""" + +from __future__ import annotations + +import asyncio +import hashlib +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest +from tensorlake.sandbox import NetworkConfig, SandboxConnectionError, SandboxNotFoundError + +from agent_env.providers.sandbox_providers.sandbox import NetworkMode, NetworkPolicy, VmSandbox +from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM +from agent_env.providers.sandbox_providers.tensorlake import sandbox as tl_sandbox +from agent_env.providers.sandbox_providers.tensorlake.sandbox import ( + TensorlakeSandbox, + network_config_fields, + network_policy_from_config, +) +from agent_env.store import set_object_store + + +def _artifact(url: str | None = "s3://bucket/image.tar.gz", *, context_only: bool = False) -> SimpleNamespace: + return SimpleNamespace( + tar_gz_object_url=url, context_only=context_only, image_name="img:latest", load_problem=lambda **_: None, + rebuilt_for=lambda *_, **__: False, + ) + + +def _inner(policy: NetworkPolicy | None = None) -> MagicMock: + """A fake SDK sandbox. ``live_info`` reports the network policy ``update`` last applied; ``info`` reports the one + it was created with, as the SDK's cached copy does.""" + applied = SimpleNamespace(network_policy=NetworkConfig(**network_config_fields(policy)) if policy else None) + + async def update(*, network=None, **kwargs): + if network is not None: + applied.network_policy = network + + sandbox = MagicMock() + sandbox.sandbox_id = "tl-test" + sandbox.terminate = AsyncMock() + sandbox.update = AsyncMock(side_effect=update) + sandbox.info = AsyncMock(return_value=SimpleNamespace(network_policy=applied.network_policy)) + sandbox.live_info = AsyncMock(side_effect=lambda: applied) + sandbox.write_file = AsyncMock() + sandbox.run = AsyncMock(return_value=SimpleNamespace(exit_code=0, stdout="hello", stderr="warning")) + return sandbox + + +def _wrap(inner: MagicMock, **kwargs) -> TensorlakeSandbox: + return TensorlakeSandbox(inner, read_info=inner.live_info, **kwargs) + + +def test_vm_identity_and_tunnel_urls(): + sandbox = _wrap(_inner(), tunnel_urls={8080: "https://8080-tl-test.sandbox.tensorlake.ai"}) + + assert isinstance(sandbox, VmSandbox) + assert sandbox.type == "tensorlake" + assert sandbox.mode == SANDBOX_MODE_VM + assert sandbox.sandbox_id == "tl-test" + assert sandbox.vnc_url is None + assert sandbox.tunnel_urls == {8080: "https://8080-tl-test.sandbox.tensorlake.ai"} + + +@pytest.mark.asyncio +async def test_exec_passes_argv_unchanged_and_keeps_sudo(): + inner = _inner() + sandbox = _wrap(inner) + + result = await sandbox.exec_with_output("sudo", "bash", "-c", "docker images && echo 'hello world'") + + assert result == (0, "hello", "warning") + inner.run.assert_awaited_once_with( + "sudo", args=["bash", "-c", "docker images && echo 'hello world'"], timeout=None, + ) + + +@pytest.mark.asyncio +async def test_nonzero_exit_is_a_process_result(): + inner = _inner() + inner.run.return_value = SimpleNamespace(exit_code=7, stdout="out", stderr="bad") + + assert await _wrap(inner).exec_with_output("sudo", "docker", "images") == (7, "out", "bad") + + +@pytest.mark.asyncio +async def test_setup_starts_docker_service_polls_until_ready_and_checks_compose(monkeypatch: pytest.MonkeyPatch): + sandbox = _wrap(_inner()) + sandbox._exec_with_output_bounded = AsyncMock( + side_effect=[(0, "", ""), (1, "", "not ready"), (0, "", ""), (0, "Docker Compose version v2.40.0", "")] + ) + sleep = AsyncMock() + monkeypatch.setattr(tl_sandbox.asyncio, "sleep", sleep) + + await sandbox.setup_vm_for_gateway([18765]) + + commands = [call.args for call in sandbox._exec_with_output_bounded.await_args_list] + assert commands == [ + ("sudo", "systemctl", "start", "docker"), + ("sudo", "docker", "info"), + ("sudo", "docker", "info"), + ("sudo", "docker", "compose", "version"), + ] + assert not any("iptables" in " ".join(command) for command in commands) + sleep.assert_awaited_once_with(sandbox._VM_READY_POLL_INTERVAL) + + +@pytest.mark.asyncio +async def test_setup_rejects_an_image_without_compose_v2(): + sandbox = _wrap(_inner()) + sandbox._exec_with_output_bounded = AsyncMock( + side_effect=[(0, "", ""), (0, "", ""), (1, "", "docker: 'compose' is not a docker command")] + ) + + with pytest.raises(RuntimeError, match="must include the Docker Compose v2 plugin"): + await sandbox.setup_vm_for_gateway([]) + + +@pytest.mark.asyncio +async def test_docker_readiness_timeout_reports_the_service_journal(): + inner = _inner() + sandbox = _wrap(inner) + sandbox._VM_READY_TIMEOUT = 0 + inner.run.return_value = SimpleNamespace(exit_code=0, stdout="dockerd failed to start", stderr="") + + with pytest.raises(RuntimeError, match=r"(?s)Docker not ready.*dockerd failed to start"): + await sandbox.wait_for_vm() + + assert inner.run.await_args.args == ("sudo",) + assert inner.run.await_args.kwargs["args"][:3] == ["journalctl", "-u", "docker"] + + +@pytest.mark.asyncio +async def test_a_failing_probe_counts_as_not_ready(monkeypatch: pytest.MonkeyPatch): + inner = _inner() + sandbox = _wrap(inner) + sandbox._VM_READY_TIMEOUT = 1 + sandbox._DOCKER_READINESS_COMMAND_TIMEOUT = 0.25 + inner.run.side_effect = [ + SimpleNamespace(exit_code=0, stdout="", stderr=""), + TimeoutError("docker info timed out"), + SimpleNamespace(exit_code=0, stdout="", stderr=""), + ] + monkeypatch.setattr(tl_sandbox.asyncio, "sleep", AsyncMock()) + + await sandbox.wait_for_vm() + + assert inner.run.await_args_list[1].kwargs["timeout"] <= 0.25 + + +@pytest.mark.asyncio +async def test_terminate_terminates_the_sdk_sandbox(): + inner = _inner() + + await _wrap(inner).terminate() + + inner.terminate.assert_awaited_once_with() + + +@pytest.mark.asyncio +async def test_terminate_of_a_sandbox_that_is_gone_is_not_an_error(): + inner = _inner() + inner.terminate.side_effect = SandboxNotFoundError("tl-test") + + await _wrap(inner).terminate() + + +@pytest.mark.asyncio +async def test_a_lost_transport_is_exit_minus_one_so_idempotent_scripts_retry(monkeypatch: pytest.MonkeyPatch): + inner = _inner() + inner.run.side_effect = [ + SandboxConnectionError("stream ended without an exit event"), + SimpleNamespace(exit_code=0, stdout="done", stderr=""), + ] + monkeypatch.setattr(tl_sandbox.asyncio, "sleep", AsyncMock()) + sandbox = _wrap(inner) + + assert await sandbox.exec_with_output("true") == (-1, "", "Connection error: stream ended without an exit event") + inner.run.side_effect = [ + SandboxConnectionError("lost"), + SimpleNamespace(exit_code=0, stdout="done", stderr=""), + ] + assert await sandbox.exec_script("echo done", max_retries=1) == "done" + + +def _sha256_reply(data: bytes) -> AsyncMock: + return AsyncMock(return_value=f"{hashlib.sha256(data).hexdigest()} /path\n") + + +@pytest.mark.asyncio +async def test_large_writes_upload_in_proxy_sized_parts_and_join_as_root(): + inner = _inner() + sandbox = _wrap(inner) + sandbox._UPLOAD_CHUNK_BYTES = 4 + sandbox.exec_script = _sha256_reply(b"0123456789") + + await sandbox._write_bytes_to_vm_path(b"0123456789", "/opt/agent/file.bin") + + uploads = sorted(call.args for call in inner.write_file.await_args_list) + assert [content for _, content in uploads] == [b"0123", b"4567", b"89"] + parts = [path for path, _ in uploads] + stem = parts[0].removesuffix(".0") + assert stem.startswith("/tmp/") + assert parts == [f"{stem}.{index}" for index in range(3)] + join, cleanup = (call.args[0] for call in sandbox.exec_script.await_args_list) + assert join == ( + f'for i in $(seq 0 2); do cat "{stem}.$i" || exit 1; done > /opt/agent/file.bin' + " && sha256sum /opt/agent/file.bin" + ) + assert cleanup == f'for i in $(seq 0 2); do rm -f "{stem}.$i"; done' + + +@pytest.mark.asyncio +async def test_join_command_length_does_not_grow_with_the_part_count(): + inner = _inner() + sandbox = _wrap(inner) + sandbox._UPLOAD_CHUNK_BYTES = 1 + sandbox.exec_script = _sha256_reply(b"x" * 10_000) + + await sandbox._write_bytes_to_vm_path(b"x" * 10_000, "/opt/agent/file.bin") + + assert inner.write_file.await_count == 10_000 + assert all(len(call.args[0]) < 200 for call in sandbox.exec_script.await_args_list) + + +@pytest.mark.asyncio +async def test_empty_write_still_creates_the_file(): + inner = _inner() + sandbox = _wrap(inner) + sandbox.exec_script = _sha256_reply(b"") + + await sandbox._write_bytes_to_vm_path(b"", "/tmp/empty") + + inner.write_file.assert_not_awaited() + (join,) = (call.args[0] for call in sandbox.exec_script.await_args_list) + assert join.startswith("for i in $(seq 0 -1); do ") + assert join.endswith("done > /tmp/empty && sha256sum /tmp/empty") + + +@pytest.mark.asyncio +async def test_unsigned_object_streams_through_the_file_api(tmp_path): + payload = bytes(range(256)) * 40 + source = tmp_path / "image.tar.gz" + source.write_bytes(payload) + store = MagicMock() + store.open = MagicMock(side_effect=lambda url: open(source, "rb")) + inner = _inner() + sandbox = _wrap(inner) + sandbox._UPLOAD_CHUNK_BYTES = 1000 + sandbox.exec_script = _sha256_reply(payload) + + await sandbox._write_unsigned_object(store, "file:///store/image.tar.gz", "/tmp/image.tar.gz") + + store.open.assert_called_once_with("file:///store/image.tar.gz") + uploads = sorted(inner.write_file.await_args_list, key=lambda call: int(call.args[0].rsplit(".", 1)[1])) + assert b"".join(call.args[1] for call in uploads) == payload + assert inner.write_file.await_count == 11 + assert all(len(call.args[0]) < 200 for call in sandbox.exec_script.await_args_list) + + +@pytest.mark.asyncio +async def test_parts_upload_concurrently_up_to_the_limit(): + inner = _inner() + in_flight = peak = 0 + + async def write_file(path, content): + nonlocal in_flight, peak + in_flight += 1 + peak = max(peak, in_flight) + await asyncio.sleep(0.05) + in_flight -= 1 + + inner.write_file = AsyncMock(side_effect=write_file) + sandbox = _wrap(inner) + sandbox._UPLOAD_CHUNK_BYTES = 1 + sandbox._UPLOADS_IN_FLIGHT = 3 + sandbox.exec_script = _sha256_reply(b"x" * 9) + + await sandbox._write_bytes_to_vm_path(b"x" * 9, "/tmp/file.bin") + + assert peak == 3 + + +@pytest.mark.asyncio +async def test_a_failed_part_raises_its_error_and_cleans_up(): + inner = _inner() + inner.write_file = AsyncMock(side_effect=[None, SandboxConnectionError("proxy closed"), None, None]) + sandbox = _wrap(inner) + sandbox._UPLOAD_CHUNK_BYTES = 1 + sandbox._UPLOADS_IN_FLIGHT = 1 + sandbox.exec_script = AsyncMock(return_value="") + + with pytest.raises(SandboxConnectionError, match="proxy closed"): + await sandbox._write_bytes_to_vm_path(b"abcd", "/tmp/file.bin") + + (cleanup,) = (call.args[0] for call in sandbox.exec_script.await_args_list) + assert "rm -f" in cleanup + + +@pytest.mark.asyncio +async def test_a_joined_file_with_the_wrong_digest_is_refused(): + inner = _inner() + sandbox = _wrap(inner) + sandbox.exec_script = _sha256_reply(b"other") + + with pytest.raises(RuntimeError, match="does not match the upload"): + await sandbox._write_bytes_to_vm_path(b"data", "/tmp/file.bin") + + +def test_network_config_fields(): + assert network_config_fields(NetworkPolicy()) == {"allow_internet_access": True, "allow_out": [], "deny_out": []} + assert network_config_fields( + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example",), allow_cidrs=("10.0.0.0/8",)) + ) == {"allow_internet_access": True, "allow_out": ["api.example", "10.0.0.0/8"], "deny_out": []} + # An empty allow_out means "allow everything" to Tensorlake, so an empty allowlist turns internet off. + assert network_config_fields(NetworkPolicy(mode=NetworkMode.ALLOWLIST)) == { + "allow_internet_access": False, "allow_out": [], "deny_out": [], + } + + +@pytest.mark.parametrize( + "policy", + [ + NetworkPolicy(), + NetworkPolicy(mode=NetworkMode.ALLOWLIST), + NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example", "*.cdn.example"), allow_cidrs=("8.8.8.8", "10.0.0.0/8")), + ], +) +def test_network_policy_round_trips(policy: NetworkPolicy): + applied = SimpleNamespace(**network_config_fields(policy)) + + assert network_policy_from_config(applied, sandbox_id="tl-test") == policy + + +def test_no_applied_policy_reads_as_allow_all(): + assert network_policy_from_config(None, sandbox_id="tl-test") == NetworkPolicy() + + +@pytest.mark.parametrize( + "applied", + [ + SimpleNamespace(allow_internet_access=True, allow_out=[], deny_out=["10.0.0.0/8"]), + SimpleNamespace(allow_internet_access=False, allow_out=["10.0.0.0/8"], deny_out=[]), + ], +) +def test_a_policy_agent_env_never_applies_is_refused(applied): + with pytest.raises(RuntimeError, match="Cannot recover the network policy"): + network_policy_from_config(applied, sandbox_id="tl-test") + + +@pytest.mark.asyncio +async def test_apply_network_policy_updates_the_sandbox(): + inner = _inner() + sandbox = _wrap(inner) + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("api.example",)) + + await sandbox.apply_network_policy(policy) + + inner.update.assert_awaited_once_with( + network=NetworkConfig(allow_internet_access=True, allow_out=["api.example"], deny_out=[]) + ) + assert sandbox.network_policy == policy + + +@pytest.mark.asyncio +async def test_restricted_image_loading_adds_the_signed_host_and_waits_for_it(monkeypatch: pytest.MonkeyPatch): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("workload.example",), allow_cidrs=("10.0.0.0/8",)) + inner = _inner(policy) + sandbox = _wrap(inner, network_policy=policy) + sandbox.exec_script = AsyncMock(return_value="") + artifact = _artifact() + store = MagicMock() + store.signed_get_url.return_value = "https://downloads.example/path?signature=x" + set_object_store(store) + base_load = AsyncMock() + monkeypatch.setattr(VmSandbox, "_load_docker_images", base_load) + + await sandbox.load_docker_images([artifact]) + + inner.update.assert_awaited_once_with( + network=NetworkConfig( + allow_internet_access=True, + allow_out=["workload.example", "downloads.example", "10.0.0.0/8"], + deny_out=[], + ) + ) + assert sandbox.network_policy == policy.with_hosts(["downloads.example"]) + command, args = inner.run.await_args.args[0], inner.run.await_args.kwargs["args"] + assert command == "sh" and "https://downloads.example/" in args[-1] + base_load.assert_awaited_once_with([artifact], ["https://downloads.example/path?signature=x"]) + + +@pytest.mark.asyncio +async def test_restricted_image_loading_skips_the_update_when_the_host_is_already_allowed(monkeypatch: pytest.MonkeyPatch): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("downloads.example",)) + inner = _inner(policy) + sandbox = _wrap(inner, network_policy=policy) + store = MagicMock() + store.signed_get_url.return_value = "https://downloads.example/image.tar.gz" + set_object_store(store) + monkeypatch.setattr(VmSandbox, "_load_docker_images", AsyncMock()) + + await sandbox.load_docker_images([_artifact("s3://bucket/img")]) + + inner.update.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_restricted_image_loading_pulls_ref_images_and_builds_context_images(monkeypatch: pytest.MonkeyPatch): + sandbox = _wrap(_inner(), network_policy=NetworkPolicy(mode=NetworkMode.ALLOWLIST)) + pull, build = AsyncMock(), AsyncMock() + monkeypatch.setattr(VmSandbox, "pull_images", pull) + monkeypatch.setattr(VmSandbox, "build_images", build) + ref, context = _artifact(None), _artifact(None, context_only=True) + + await sandbox.load_docker_images([ref, context]) + + pull.assert_awaited_once_with(["img:latest"]) + build.assert_awaited_once_with([context]) + + +@pytest.mark.asyncio +async def test_restricted_object_download_adds_the_signed_host(monkeypatch: pytest.MonkeyPatch): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST) + sandbox = _wrap(_inner(policy), network_policy=policy) + sandbox.exec_script = AsyncMock(return_value="") + sandbox._exec_with_output_bounded = AsyncMock(return_value=(0, "", "")) + store = MagicMock() + store.signed_get_url.return_value = "https://downloads.example/context.tar.gz?signature=x" + set_object_store(store) + + await sandbox.load_object_file("s3://bucket/context.tar.gz", "/tmp/context.tar.gz") + + assert sandbox.network_policy.allow_hosts == ("downloads.example",) + assert "context.tar.gz?signature=x" in sandbox.exec_script.await_args.args[0] + + +@pytest.mark.asyncio +async def test_concurrent_downloads_through_two_wrappers_keep_each_others_hosts(monkeypatch: pytest.MonkeyPatch): + policy = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("workload.example",)) + inner = _inner(policy) + first, second = _wrap(inner, network_policy=policy), _wrap(inner, network_policy=policy) + monkeypatch.setattr(TensorlakeSandbox, "_wait_for_egress", AsyncMock()) + + await asyncio.gather( + first._allow_download_hosts(["https://a.example/x"]), + second._allow_download_hosts(["https://b.example/y"]), + ) + + applied = network_policy_from_config((await inner.live_info()).network_policy, sandbox_id="tl-test") + assert set(applied.allow_hosts) == {"workload.example", "a.example", "b.example"} + + +@pytest.mark.asyncio +async def test_unknown_policy_refuses_image_load(): + sandbox = _wrap(_inner(), network_policy=None) + + with pytest.raises(RuntimeError, match="applied network policy is unknown"): + await sandbox.load_docker_images([_artifact()]) + + +@pytest.mark.asyncio +async def test_unreachable_download_host_is_a_warning_not_a_failure( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +): + monkeypatch.setattr(TensorlakeSandbox, "_EGRESS_PROPAGATION_TIMEOUT", 0.05) + monkeypatch.setattr(TensorlakeSandbox, "_EGRESS_PROBE_INTERVAL", 0.01) + sandbox = _wrap(_inner()) + sandbox._exec_with_output_bounded = AsyncMock(return_value=(7, "", "")) + + await sandbox._wait_for_egress(["downloads.example"]) + + assert "still cannot reach" in caplog.text + + +@pytest.mark.asyncio +async def test_egress_wait_bounds_each_probe_by_the_time_left(monkeypatch: pytest.MonkeyPatch): + clock = iter([100.0, 100.0, 158.0, 158.0, 161.0, 161.0]) + monkeypatch.setattr(tl_sandbox, "time", SimpleNamespace(monotonic=lambda: next(clock))) + sandbox = _wrap(_inner()) + sandbox._exec_with_output_bounded = AsyncMock(return_value=(7, "", "")) + + await sandbox._wait_for_egress(["downloads.example"]) + + assert [call.kwargs["timeout"] for call in sandbox._exec_with_output_bounded.await_args_list] == [60.0, 2.0] diff --git a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py index 8761c8a5..18aafecb 100644 --- a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py +++ b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py @@ -18,6 +18,7 @@ SandboxProviderTypeError, build_sandbox_provider, ) +from agent_env.providers.sandbox_providers.tensorlake.provider import TensorlakeSandboxProvider class _RecordingProvider(SandboxProvider): @@ -257,6 +258,33 @@ def test_sail_missing_api_key_is_a_config_error(monkeypatch, tmp_path): build_sandbox_provider("sail_vm") +def test_tensorlake_builtin_receives_interpolated_key_and_image(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.tensorlake.config] + api_key = "env:TL_TEST_API_KEY" + image = "team-docker-host" + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + monkeypatch.setenv("TL_TEST_API_KEY", "resolved-test-key") + + provider = build_sandbox_provider("tensorlake") + + assert isinstance(provider, TensorlakeSandboxProvider) + assert provider._api_key == "resolved-test-key" + assert provider._image == "team-docker-host" + + +def test_tensorlake_missing_api_key_is_a_config_error(monkeypatch, tmp_path): + cfg = _write_config(tmp_path, """ + [sandbox.providers.tensorlake.config] + image = "team-docker-host" + """) + monkeypatch.setenv("AGENT_ENV_CONFIG", str(cfg)) + + with pytest.raises(ConfigError, match="requires a non-empty 'api_key'"): + build_sandbox_provider("tensorlake") + + def test_builtin_config_reaches_chain_members(monkeypatch, tmp_path): cfg = _write_config(tmp_path, """ [sandbox.providers.modal.config] diff --git a/tst/unit/test_capabilities.py b/tst/unit/test_capabilities.py index 3aecc9d8..6710db6f 100644 --- a/tst/unit/test_capabilities.py +++ b/tst/unit/test_capabilities.py @@ -141,9 +141,9 @@ def test_default_agent_other_failures_read_as_present_rather_than_absent(monkeyp assert capabilities.default_a2a_agent_is_registered() is True -def _fake_e2b_builder(monkeypatch, outcome): +def _fake_builder(monkeypatch, provider, outcome): def build(spec): - assert spec == "e2b" + assert spec == provider if isinstance(outcome, Exception): raise outcome return outcome @@ -151,32 +151,23 @@ def build(spec): monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox_provider.build_sandbox_provider", build) -def test_e2b_absent_when_its_config_is_missing(monkeypatch): - _fake_e2b_builder(monkeypatch, ConfigError("[sandbox.providers.e2b.config] requires a non-empty 'base_template'")) - assert capabilities.remote_sandbox_is_available("e2b") is False - assert capabilities.skip_without_remote_sandbox("e2b").kwargs["reason"] == "agentenv-capability-missing: remote_sandbox" +_CONFIGURED_PROVIDERS = pytest.mark.parametrize("provider", ["e2b", "sail_vm", "tensorlake"]) -def test_e2b_present_when_the_provider_builds(monkeypatch): - _fake_e2b_builder(monkeypatch, object()) - assert capabilities.remote_sandbox_is_available("e2b") is True +@_CONFIGURED_PROVIDERS +def test_configured_provider_absent_when_its_config_is_missing(monkeypatch, provider): + _fake_builder(monkeypatch, provider, ConfigError(f"[sandbox.providers.{provider}.config] requires a non-empty 'api_key'")) + assert capabilities.remote_sandbox_is_available(provider) is False + assert capabilities.skip_without_remote_sandbox(provider).kwargs["reason"] == "agentenv-capability-missing: remote_sandbox" -def test_e2b_other_failures_read_as_present_rather_than_absent(monkeypatch): - _fake_e2b_builder(monkeypatch, RuntimeError("api key rejected")) - assert capabilities.remote_sandbox_is_available("e2b") is True +@_CONFIGURED_PROVIDERS +def test_configured_provider_present_when_the_provider_builds(monkeypatch, provider): + _fake_builder(monkeypatch, provider, object()) + assert capabilities.remote_sandbox_is_available(provider) is True -@pytest.mark.parametrize( - ("outcome", "available"), - [(ConfigError("[sandbox.providers.sail_vm.config] requires a non-empty 'api_key'"), False), (object(), True), (RuntimeError("boom"), True)], -) -def test_sail_availability_follows_the_provider_build(monkeypatch, outcome, available): - def build(spec): - assert spec == "sail_vm" - if isinstance(outcome, Exception): - raise outcome - return outcome - - monkeypatch.setattr("agent_env.providers.sandbox_providers.sandbox_provider.build_sandbox_provider", build) - assert capabilities.remote_sandbox_is_available("sail_vm") is available +@_CONFIGURED_PROVIDERS +def test_configured_provider_other_failures_read_as_present_rather_than_absent(monkeypatch, provider): + _fake_builder(monkeypatch, provider, RuntimeError("api key rejected")) + assert capabilities.remote_sandbox_is_available(provider) is True diff --git a/tst/unit/test_optional_extras_stay_optional.py b/tst/unit/test_optional_extras_stay_optional.py index 135b186d..b3b1a8d3 100644 --- a/tst/unit/test_optional_extras_stay_optional.py +++ b/tst/unit/test_optional_extras_stay_optional.py @@ -121,13 +121,23 @@ def _closure(extras: frozenset[str]) -> set[str]: def _only_in(extra: str) -> list[str]: - """Module paths the extra installs and a core install does not.""" - core = _closure(frozenset()) + """Module paths the extra installs and neither a core install nor any other extra does: a + distribution two extras share (grpcio, for gcp and tensorlake) stays for the other's modules.""" + others = frozenset(_optional_requirements()) - {extra} + core = _closure(frozenset()) | set().union(*(_closure(frozenset({other})) for other in others)) provided_by_core = set().union(*(_module_paths(d) for d in core)) extra_only = _closure(frozenset({extra})) - core return sorted(set().union(*(_module_paths(d) for d in extra_only)) - provided_by_core) +def _not_in_core() -> list[str]: + """Module paths some extra installs and a core install does not, including those two extras share.""" + core = _closure(frozenset()) + provided_by_core = set().union(*(_module_paths(d) for d in core)) + optional = set().union(*(_closure(frozenset({extra})) for extra in _optional_requirements())) - core + return sorted(set().union(*(_module_paths(d) for d in optional)) - provided_by_core) + + def _run(script: str, blocked: list[str], *args: str) -> str: done = subprocess.run( [sys.executable, "-c", script, *args], input=json.dumps(blocked), @@ -144,6 +154,10 @@ def test_core_and_every_store_module_import_without_the_extra(extra): assert json.loads(_run(_IMPORT_CORE, blocked, json.dumps(_NOT_CORE))) == [] +def test_core_and_every_store_module_import_without_any_extra(): + assert json.loads(_run(_IMPORT_CORE, _not_in_core(), json.dumps(_NOT_CORE))) == [] + + @pytest.mark.parametrize( ("extra", "impl", "base"), [ @@ -166,8 +180,13 @@ def test_core_and_every_store_module_import_without_the_extra(extra): "agent_env.store.document_store:DocumentStore", ), ("aws", "agent_env.store.document_store:DynamoDbDocumentStore", "agent_env.store.document_store:DocumentStore"), + ( + "tensorlake", + "agent_env.providers.sandbox_providers.tensorlake.provider:TensorlakeSandboxProvider", + "agent_env.providers.sandbox_providers.sandbox_provider:SandboxProvider", + ), ], - ids=["gcs", "secret-manager", "s3", "s3-reexport", "secrets-manager-reexport", "dynamodb", "dynamodb-reexport"], + ids=["gcs", "secret-manager", "s3", "s3-reexport", "secrets-manager-reexport", "dynamodb", "dynamodb-reexport", "tensorlake"], ) def test_an_impl_without_its_extra_names_the_extra_to_install(extra, impl, base): message = _run(_LOAD_IMPL, _only_in(extra), impl, base) diff --git a/tst/util/capabilities.py b/tst/util/capabilities.py index ef86c8f6..3222a1b2 100644 --- a/tst/util/capabilities.py +++ b/tst/util/capabilities.py @@ -19,7 +19,7 @@ #: Only the agent-driving paths need a model endpoint. MODEL_ENDPOINT = "model_endpoint_configured" -#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail_vm``) need credentials the resolved +#: The remote sandbox providers (``modal``, ``modal_vm``, ``e2b``, ``sail_vm``, ``tensorlake``) need credentials the resolved #: config may not carry; the local default never does. REMOTE_SANDBOX = "remote_sandbox" @@ -75,16 +75,17 @@ def skip_without_model_endpoint() -> pytest.MarkDecorator: def remote_sandbox_is_available(provider: str) -> bool: - """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail_vm`` sandbox - provider, credentials included. - - Modal: any failure to resolve the credentials answers False. E2B and Sail: building the provider - resolves ``[sandbox.providers..config]`` and its ``secret:`` references; only a - ``ConfigError`` (absent or incomplete config) answers False, any other failure answers True and - lets the test fail on the real problem, as ``model_endpoint_is_configured`` does.""" - if provider not in ("modal", "modal_vm", "e2b", "sail_vm"): + """Whether the resolved config can build the ``modal`` / ``modal_vm`` / ``e2b`` / ``sail_vm`` / + ``tensorlake`` sandbox provider, credentials included. + + Modal: any failure to resolve the credentials answers False. E2B, Sail and Tensorlake: building + the provider resolves ``[sandbox.providers..config]`` and its ``secret:`` references; only a + ``ConfigError`` (absent or incomplete config, or the provider's extra not installed) answers + False, any other failure answers True and lets the test fail on the real problem, as + ``model_endpoint_is_configured`` does.""" + if provider not in ("modal", "modal_vm", "e2b", "sail_vm", "tensorlake"): raise ValueError(f"unknown remote sandbox provider {provider!r}") - if provider in ("e2b", "sail_vm"): + if provider in ("e2b", "sail_vm", "tensorlake"): from agent_env.config.errors import ConfigError from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider diff --git a/uv.lock b/uv.lock index d1c6ef3b..1d29ad12 100644 --- a/uv.lock +++ b/uv.lock @@ -88,6 +88,7 @@ dev = [ { name = "pytest-xdist" }, { name = "requests" }, { name = "sail" }, + { name = "tensorlake" }, { name = "uvicorn" }, ] explorer = [ @@ -104,11 +105,14 @@ gcp = [ sail = [ { name = "sail" }, ] +tensorlake = [ + { name = "tensorlake" }, +] [package.metadata] requires-dist = [ { name = "a2a-sdk", specifier = "==0.3.26" }, - { name = "agentenv-framework", extras = ["aws", "explorer", "gcp", "sail"], marker = "extra == 'dev'" }, + { name = "agentenv-framework", extras = ["aws", "explorer", "gcp", "sail", "tensorlake"], marker = "extra == 'dev'" }, { name = "agentenv-framework-protocol", editable = "packages/agentenv-protocol" }, { name = "boto3", marker = "extra == 'aws'", specifier = ">=1.35.0" }, { name = "botocore", marker = "extra == 'aws'", specifier = ">=1.35.0" }, @@ -143,10 +147,11 @@ requires-dist = [ { name = "requests", marker = "extra == 'gcp'", specifier = ">=2.31" }, { name = "sail", marker = "extra == 'sail'", specifier = "~=0.12.8" }, { name = "starlette", specifier = ">=0.40" }, + { name = "tensorlake", marker = "extra == 'tensorlake'", specifier = "~=0.5.140" }, { name = "uvicorn", specifier = ">=0.47" }, { name = "uvicorn", marker = "extra == 'explorer'", specifier = ">=0.29" }, ] -provides-extras = ["aws", "explorer", "gcp", "sail", "dev"] +provides-extras = ["aws", "explorer", "gcp", "sail", "tensorlake", "dev"] [[package]] name = "agentenv-framework-protocol" @@ -1528,6 +1533,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, ] +[package.optional-dependencies] +http2 = [ + { name = "h2" }, +] + [[package]] name = "httpx-sse" version = "0.4.3" @@ -3419,6 +3429,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/4e/9c/8408129632f2cbd70018cff4a7eb78cc105b141de233b699feeda15d44c9/synchronicity-0.12.6-py3-none-any.whl", hash = "sha256:bc2bab6dde31f6bd9389912cef573b7fa7a5cb4a120ec786275e3284f06473ae", size = 41971, upload-time = "2026-10-02T15:41:42.672Z" }, ] +[[package]] +name = "tensorlake" +version = "0.5.149" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "grpcio" }, + { name = "httpx", extra = ["http2"] }, + { name = "protobuf" }, + { name = "pydantic" }, + { name = "websocket-client" }, + { name = "websockets" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/e5/5f912f624df4f970d1f65f3021de2c4bff996fe694c8d403703f5e509d39/tensorlake-0.5.149-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:0cb8b1c22b1b886b6957fa65ddc1e3add407c7d5483168a1372290841387ef99", size = 8350835, upload-time = "2026-10-11T04:27:45.991Z" }, + { url = "https://files.pythonhosted.org/packages/40/6d/42786f7c27a66739324e2e9768a7decdc58088b211cb42332022de9d1322/tensorlake-0.5.149-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a9fcf052a1641b88756dbbfd8aceea8183d43af1b28b29a4e7a63232ec687a85", size = 8629932, upload-time = "2026-10-11T04:27:48.97Z" }, + { url = "https://files.pythonhosted.org/packages/9a/07/d1687c3833f3b6866554abc5fc78999a929846d8ea907995dc6e110fce1d/tensorlake-0.5.149-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b64893799fe684fd18f098f1f5cecc7ef3224926c03db7ba3209520f0c10f02f", size = 8948006, upload-time = "2026-10-11T04:27:51.182Z" }, + { url = "https://files.pythonhosted.org/packages/1a/80/04264c98c0da4544045724d634c3219de0c1599a8286b6d0eae1ae7f5c27/tensorlake-0.5.149-cp310-abi3-win_amd64.whl", hash = "sha256:b3cc75bc0f7d1d4a4a0b11a31fcc8274ee9fc5e9ee91a09d9f396ac7012ee8c4", size = 9628190, upload-time = "2026-10-11T04:27:54.207Z" }, +] + [[package]] name = "tiktoken" version = "0.12.0" @@ -3721,6 +3750,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ce/77/7a02b0f05b3ffcdbef9719ce3ee0b508d6a29b58e95299f1580055671db3/wcmatch-11.0.1-py3-none-any.whl", hash = "sha256:fd149ecddb9f0a88ea780017d6dde17c994e494e7f7303d4e3c9d6251f978f4b", size = 43449, upload-time = "2026-08-14T15:20:39.379Z" }, ] +[[package]] +name = "websocket-client" +version = "1.9.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/cb/a5abcc2891249f393827c650c6296660ce40374ac22d99ab9aea41f9d2a2/websocket_client-1.9.2.tar.gz", hash = "sha256:0fcb57545848be86992e128218fd96dd87a6769ffdb1a968dff79632b85604d0", size = 84110, upload-time = "2026-08-31T14:08:40.964Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d5/d2/cc4dc1271e464942db7ee278baae2daa99ee77cb2af744025c04da585a3e/websocket_client-1.9.2-py3-none-any.whl", hash = "sha256:e1a673830a9c7bfa47b1cd3d5e4178f4c9651d80a4eab02c9c23a1c3ec6250ce", size = 95786, upload-time = "2026-08-31T14:08:39.899Z" }, +] + [[package]] name = "websockets" version = "17.2"