Skip to content

Security: bhatti/PlexSpaces

Security

docs/security.md

PlexSpaces Security Guide

Overview

PlexSpaces provides comprehensive security features for production deployments, including:

  • Node-to-Node Authentication: a short-lived node token (a JWT signed with the cluster JWT key) that is sent only over TLS, or to configured seed nodes when plaintext is explicitly allowed
  • User API Authentication: JWT-based authentication for the HTTP and gRPC APIs, with a per-method access policy on gRPC
  • Not enforced: inbound mTLS. The node does not require or verify client certificates (see "Node-to-Node Authentication")
  • Tenant Isolation: Mandatory tenant isolation for all operations
  • Security Validation: Automatic validation that secrets are not hardcoded in config files

Security Architecture

Defense in Depth

PlexSpaces uses multiple layers of security to prevent unauthorized access:

  1. Transport Layer Security: outbound node-to-node TLS (CA-verified peers) enabled by runtime.security.mtls.enable_mtls with paths from the config or PLEXSPACES_MTLS_*, or a TLS-terminating proxy or mesh; the node token is never sent over plaintext unless explicitly allowed
  2. Authentication (JWT): Verifies user identity and extracts tenant context
  3. Authorization (Tenant Isolation): Enforces data access boundaries
  4. Query Filtering: Database-level enforcement of tenant boundaries
  5. Method access policy: every gRPC method is classified (member, tenant admin, system only, node internal, node delegated); unlisted methods are denied
  6. Request Context: Propagates security context through the entire call chain

Authentication vs Authorization

  • Authentication (AuthN): Verifies identity (who you are)
    • Node-to-node: node token (JWT) over TLS
    • User APIs: JWT tokens
  • Authorization (AuthZ): Verifies permissions (what you can do)
    • Tenant isolation enforced at repository/service layer
    • All queries filtered by tenant_id
    • gRPC method policy table (node::grpc_auth): member, tenant-admin and system-only methods

Link, monitor, unlink, demonitor

These operations use the caller’s RequestContext (JWT-derived tenant when auth is enabled, namespace from the request boundary). On each node, ActorRegistry::validate_link_monitor_operand_scope runs only for operands hosted on that node: namespace on ActorId must match RequestContext::namespace; when auth is enabled, the actor must appear under (tenant_id, namespace, actor_id) in the local registry (live sender or registered index entry). link / unlink mirror monitor / demonitor: validate and mutate locals inside the local branch; read ActorService only in remote else branches. Remote-only operands are enforced when the peer handles the forwarded RPC. This is orthogonal to ActorVisibility (ActorRef::tell / ask).

Tenant Isolation

Tenant isolation is mandatory in PlexSpaces. All operations require a tenant_id:

  • RequestContext: Go-style context carries tenant_id and namespace through call chain
  • Repository Pattern: All repository methods require RequestContext
  • Service Layer: All service methods require RequestContext
  • SQL Queries: All queries automatically filter by tenant_id and namespace

Spawning for another tenant: the caller's tenant is authoritative. A spec that names a tenant other than the caller's is refused with gRPC PermissionDenied ("Tenant isolation violation") by ActorServiceImpl::refuse_other_tenant, before the ActorFactory is reached; an empty spec tenant inherits the caller's. SpawnActors (batch) uses the same function and fails the offending request (its result carries success = false and the message; nothing is spawned). The factory repeats the check as defence in depth and runs it before the behavior is created, so a refused spec never instantiates a behavior (possibly a WASM module). Rejected alternatives: leaving it to the factory (a generic Internal error that looks like a server fault), and overwriting the tenant with the caller's (it hid the request). Tests: crates/services/tests/actor_service/actor_service_spawn_tests.rs.

Multi-tenancy Architecture

PlexSpaces implements a two-level isolation model:

  1. Tenant-id (Primary isolation)

    • Source of Truth: the verified JWT (HTTP and gRPC). A client certificate never provides a tenant.
    • Purpose: Tenant-level data isolation
    • When empty: Only allowed when auth is disabled (PLEXSPACES_DISABLE_AUTH=1)
  2. Namespace (scoping inside a tenant, authorized by the token's namespaces claim, see Namespace authorization)

    • Source of Truth: Application (when actor is part of an app) or Actor (direct creation)
    • Authorization: a token reaches only the namespaces its namespaces claim lists (or all of its tenant's with "*"). The auth layers refuse a header or path namespace outside the claim, and the request context refuses any other source of a non-empty namespace (label). The actor, process-group, object-registry, application, key/value, tuple-space, workflow and scheduling services also check the body fields, embedded ActorIds, application_id and the empty namespace; the dashboard service, its HTTP routes and metrics visibility (GetMetrics, /metrics) filter or authorize per namespace too; the blob gRPC and REST servers authorize the namespace too (see "Blob namespace authorization" under Namespace authorization). Namespaces limit what an API caller may do, not what deployed code may do (see Namespaces confine API callers, not deployed code). A token without a namespaces claim reaches no namespace; mint it with one (CLI jwt-create --namespaces, API tokens, OIDC group mapping).
    • Purpose: Allows tenants to create multiple environments (e.g., "production", "staging")
    • Storage: Stored in ActorRef.namespace and Actor.namespace proto field
    • When empty: Allowed (represents default namespace within tenant)
┌─────────────────────────────────────────────────────────────────┐
│                        Tenant A                                  │
│  ┌─────────────────────┐  ┌─────────────────────┐               │
│  │  Namespace: prod    │  │  Namespace: staging │               │
│  │  - App1             │  │  - App1 (test)      │               │
│  │  - App2             │  │  - App2 (test)      │               │
│  └─────────────────────┘  └─────────────────────┘               │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                        Tenant B                                  │
│  ┌─────────────────────┐                                        │
│  │  Namespace: default │                                        │
│  │  - MyApp            │                                        │
│  └─────────────────────┘                                        │
└─────────────────────────────────────────────────────────────────┘

Namespace authorization

  • Claim: namespaces on the JWT. Absent or empty means no namespace access (fail closed); "*" is the explicit "every namespace of the token's tenant"; a tenant admin is implicitly "*" for its own tenant only (an admin of another tenant reaches nothing there; only the system scope, an admin with an empty tenant, spans tenants).
  • Proto bounds: a tenant slug (Tenant.slug) is 1 to 128 characters of [A-Za-z0-9._-], and a namespace entry is "*" or 1 to 128 such characters; a list holds at most 64 entries (buf.validate on User.allowed_namespaces, GetOrCreateByEmail/SetUserGrants/CreateApiToken namespaces, NamespaceList and the middleware namespaces claim). A pattern cannot express the dots-only rule or the reserved names (plexspaces-node, __other__, system, plexspaces), so plexspaces_common::scope_names stays the authority and enforces both. The buf.validate annotations are the contract for clients and generated validators; the node enforces the rules in hand-written checks, not by running the annotations.
  • Tier 1, the auth layers: the gRPC layer (authenticate_user, after the x-namespace syntax check and before any identity header is stamped), the HTTP middleware (x-namespace header and the /api/v1/actors/{namespace}/{type} path) and the blob REST router refuse a namespace outside the claim with PermissionDenied (gRPC) / 403 (HTTP) before any handler runs. The decision uses only the token and the name, so it is identical for an existing and a missing resource, and the message never lists the allowed set. An absent or empty x-namespace names no namespace and passes the layer (the services decide); a second x-namespace value is InvalidArgument on gRPC and refused with 403 on HTTP, and a non-text value is refused.
  • The stamped grant: the layers stamp the trusted header x-allowed-namespaces from the verified claims, exactly * for an admin or a holder of "*", otherwise the comma-joined entries (the entry charset has no comma), nothing for a token without namespaces. It is one of the identity headers: removed from every client request first (so a forged value never survives), stripped on public methods, never forwarded to a peer node, and withheld from actor messages.
  • The layer marker: every request an auth layer processes carries the request extension AuthLayerApplied { auth_enabled } (defined once in plexspaces_common::AuthLayerApplied, because the blob servers do not depend on the actor crate; plexspaces_actor re-exports it next to AuthenticatedNodeCall; AuthLayerApplied::new stays pub because the auth layers insert it in production and the services of other crates stamp it in their tests, which a pub(crate) constructor cannot allow; rejected: a feature gate, which would need a second ungated constructor for the layers): the gRPC layer inserts it on the user, delegated, internal, public-method and authentication-off paths and removes any it finds on an arriving request first; the HTTP-to-gRPC bridges insert it on the in-process request they build, from the same switch the HTTP middleware used. Extensions never cross the wire. request_context_from_grpc_request, request_context_from_grpc_call and authorized_request_context read it together with the node's own switch (the stricter wins): on a node that authenticates, a request without the marker skipped the layer and is not authenticated (Unauthenticated, whatever identity headers it carries); on a node with authentication off nothing needs proving, so an in-process caller needs no marker. A service without a service locator (the scheduler) reads only the marker, through authorized_context_from_metadata(metadata, extensions, labels, fallback, prefix), which reads the marker itself so no caller can forget the rule, and a request without it is Unauthenticated there. DeployApplication decides "API deployment" by the marker (a request the layer processed) and not by the presence of identity headers: an API deployment belongs to the tenant of the verified context (the no-auth tenant on a node without authentication), while a startup or file-copy deploy runs in process without the marker, ignores any identity header it carries and takes its tenant from the operator's app-config.toml. Decision: one marker on the request instead of an auth_enabled constructor parameter per service (each copy could be wired wrongly, and a flag cannot tell a layer-processed request from a bypassing one). Rejected alternative: an identity header (a client could forge it unless every layer strips it). The dashboard's HTTP handlers build every in-process request with one function (dashboard_request) that forwards the identity headers and inserts the marker from the same switch the HTTP middleware used, and the dashboard service builds its caller from the marker like every other service (a request without it is Unauthenticated); the metrics service reads only the marker, as the scheduler does.
  • Tier 2, the request context: RequestContext carries allowed_namespaces (proto field 12). request_context_from_grpc_metadata reads the header once per request (a second value is refused; absent or empty means no namespaces, so a request that skipped the layer fails closed) and RequestContext::from_auth validates the final non-empty namespace, whichever source it came from (header or the client's namespace label), so a label cannot reach a namespace the header could not. RequestContext::authorize_namespace(&str) is the one decision: contexts without authentication (auth_enabled == false, auth-off nodes, tests, the WASM host built at deploy) and the system scope are unrestricted; an admin reaches every namespace of its own tenant including the empty one; a held "*" (a wildcard token, or a delegated node call, which carries it) reaches every namespace including the empty one, like an admin; any other caller needs a non-empty namespace that is listed, so a token without "*" and without the admin flag is refused for an unnamed namespace. One pure rule, plexspaces_common::namespace_grant_covers, serves both JwtClaims::allows_namespace and the context, so claim and context cannot drift; plexspaces_common::namespaces_grant_lists is the same rule as a set for queries that filter by namespaces (None when the grant may name the empty namespace, otherwise the non-empty entries, possibly none). An empty namespace builds a context (many methods name none); a service that must refuse it for a restricted caller says so through authorize_namespace(""). Refusals are RequestContextError::NamespaceNotAllowed, which services map to PermissionDenied with plexspaces_actor::request_context_status(&error, FallbackStatus, prefix) (the enum picks the status of the non-denial failures, InvalidArgument or Unauthenticated, each service keeping its own).
  • Derived contexts keep the grant: work that runs in another namespace than the caller's own (routing a message, stopping an actor, a dashboard action) uses RequestContext::derive_in_namespace(ns): it authorizes ns with authorize_namespace, then returns a copy of the caller's context in ns that keeps auth_enabled and the grant, so the derived work stays restricted. Building the derived context with RequestContext::new_without_auth(tenant, ns) instead would make it unrestricted (authorize_namespace on such a context always succeeds), which is the bypass this method exists to prevent. Request-handling code does not use new_without_auth for work derived from a caller. A canonical actor id named in an HTTP ask/tell/stop path must be in a namespace the caller reaches; the HTTP edge additionally refuses a decoded path segment that contains / (a canonical id smuggled through %2F%2F), and an actor id of another namespace than the request path is refused for stop.
  • Building the context: RequestContext::from_auth(skeleton, auth_enabled, default_tenant_id) takes the identity as a proto RequestContext skeleton (its tenant_id, namespace, user_id, admin, allowed_namespaces and headers are used; request id, timestamp and the internal flag are filled fresh) plus the node's authentication state. A x-namespace header that is present but not readable as text is refused (NamespaceNotAllowed), never replaced by the client's label.
  • Node-to-node: a delegated node call (the origin node already authorized its client) is stamped x-allowed-namespaces: * by the receiving layer after any forged value is removed, because services read metadata and cannot see the layer's AuthenticatedNodeCall proof. Trust model: the tenant of a delegated call is the one the calling node states in x-tenant-id (validated as a tenant name), not one bound by the node token, so any node that holds the cluster key may act in any tenant and any namespace through this stamp; node tokens are cluster-internal credentials. An internal call is the system scope and carries no grant. apply_request_context_to_grpc_metadata never sends the grant.
  • HTTP-to-gRPC bridges (actor routes, deploy, node list, WebSocket frames, dashboard) call the services in process and so skip the gRPC layer. They forward the verified identity (tenant, user id, admin flag, allowed namespaces) and the request's x-namespace with one shared, validating helper, plexspaces_grpc_middleware::jwt::forward_identity (the namespace must be text and a valid namespace name; anything else is not forwarded); a WebSocket session keeps the identity of its upgrade request. Roles and groups are not forwarded. The one builder of those requests, scoped_grpc_request, also inserts the layer marker (above) from the node's authentication switch.
  • Header entries are re-validated when claims become the stamp: an entry that is not a valid namespace name or "*" (for example a,*, which would otherwise smuggle a second entry through the comma join) is dropped, and the reader skips such entries too.
  • gRPC CreateApiToken caps a token at the minter's grant read from the request context (allowed_namespaces), completing the minter cap above.
  • Exposure (what is and is not enforced). Enforced: the header and path checks of tier 1 on every gRPC and HTTP surface; the final non-empty namespace of every context built from request metadata (header or label); the ActorService and the ProcessGroup service in full: every implemented method (GetActor, ListActors, CheckActorExists and StreamMessages are Unimplemented stubs that read no namespace) builds its context through one chokepoint that authorizes the final context namespace, so a caller without "*" or the admin flag is refused an empty namespace (a restricted token must send x-namespace), and every other namespace a request names is authorized against the caller's grant before any side effect: the namespace of SpawnActor, SpawnActors (every request of a batch is authorized before any is spawned, so a refused batch spawns nothing; rejected: per-request failures in the response, because the requests before the refused one would already have spawned), SendMessage, AskReply and DeleteActor, a canonical actor id in identity.name, in actor_type and in DeleteActor.actor_id, every id of GetActorStates, the operands of LinkActor, UnlinkActor, MonitorActor, DemonitorActor and NotifyActorDown (checked before the registry's own operand scope rule, which only looks at local operands), the canonical member id of JoinGroup/LeaveGroup, and the sender_id and reply_to of every user-sent message (SendMessage, AskReply and the messages of the shard-group RPCs, authorized before routing because an actor replies to them with an unrestricted context). A name free of //, :: and @ is a bare name and names no namespace; a name that contains one of them but is not a valid canonical actor id is InvalidArgument (fail closed). The ProcessGroup service rejects a request body namespace that differs from the context namespace (InvalidArgument) and group names that are empty, longer than 255 characters, contain a control or Unicode format character (bidirectional overrides, zero-width characters) or have a :member segment (the substring :member:, or a name ending in :member), because storage keys are tenant:{tenant_id}:group:{name}[:member:{id}] and only such a name can collide with a member key; any other : (channel:{id}, user-session:{id}) is allowed, since deployed guests use it, and refusing every : was rejected as it prevents nothing more. The rule lives in the registry's key builders, so every operation on every entry point (gRPC, the WASM host, the channel backend, the facets) obeys it, delete_group("g:member:A") included; the gRPC service checks it first for a clean InvalidArgument. The REST verb POST /api/v1/process-groups/{group_name}:publish takes the verb after the last : of the last path segment, so it is unambiguous for names with :; DELETE /api/v1/process-groups/{group_name} needs such a name percent-encoded (%3A). The shared helpers (authorize, authorize_embedded, derive, authorized_request_context) live in plexspaces_actor::grpc_request_context. The refusal is PermissionDenied with the one fixed message, identical for an existing and a missing target, and never lists the grant. The process-group registry of the actor crate applies the same rule to its own context and member ids and works on the caller's context instead of an unrestricted copy. Also enforced: HTTP ask, tell and stop of an actor; the dashboard actor stop; WebSocket tell/ask frames (they reach the ActorService through the bridge with the identity of the session's token). Also enforced, with the same fixed refusal and the same guarantee that nothing is touched first:
    • the object registry: Register, Unregister, Lookup, Heartbeat and BatchHeartbeat (the caller context is built once; every entry of a batch is authorized and derived from it before any is applied) authorize the namespace they name (the caller's header, else the body field; a body namespace that differs from the header is authorized too) and the namespace of an object id that parses as a canonical actor id, and refuse an empty namespace for a caller without "*" or admin. Registry ids are arbitrary strings (worker@host) and the registry keys by the context's namespace and never routes by an embedded one, so an id that does not parse names no namespace and is not refused; the strict rule above (authorize_embedded) stays for actor routing, where a lenient parser downstream could route such a name (authorize_if_canonical is the registry's rule). Discover and ListObjectTypes are tenant-wide reads: a named namespace is authorized like a point operation, an empty one is filtered to the namespaces the caller's grant lists, in the storage query itself (DiscoverOptions.namespaces, every backend), so pages, has_more and counts stay exact. The scope is typed: None filters nothing and Some(empty) (a token with no grant) matches no row, answered by every backend with an empty result without querying, so a caller allowed nothing fails closed by construction. A page token carries only a position, so one reused under another grant continues in that grant's own filtered view and shows nothing outside it.
    • the application service: application_id is a namespace (an application's actors, journal state and registrations live in the namespace of that name, and undeploy purges it), so DeployApplication, UndeployApplication and GetApplicationStatus authorize it before anything is read or changed, and the undeploy cleanup runs in a context derived from the caller's (it keeps the grant). DeployApplication scopes the module to the application id, not to the caller's possibly empty namespace. UndeployApplication also stops the running dependents of the application (other application names, hence other namespaces): it authorizes every dependent it would stop (ApplicationManagerImpl::stop_plan, read-only) before anything is stopped and refuses the whole undeploy if one is outside the caller's grant. ListApplications filters the rows to the applications whose name the grant reaches.
    • the key/value, tuple space and workflow services take their scope from the request context alone; their one chokepoint authorizes the context's final namespace, so a caller without "*" or admin is refused an empty namespace there as well. A refused workflow mutation stores no definition and starts no execution.
    • the scheduler (ScheduleActor, GetSchedulingStatus) builds the caller's context from the request (the layer marker above), authorizes the final namespace before it validates the body or stores anything, and keeps the grant in the context it stores under. A scheduling request is keyed by (tenant_id, namespace, request_id): the client-chosen id is unique per tenant and namespace, so another tenant's id is never a duplicate and cannot be probed; a duplicate in the caller's own scope is ALREADY_EXISTS with the fixed text Request <id> already exists and no backend text.
    • the dashboard service (gRPC and HTTP, the same code): the service builds one caller context per request from the metadata and the layer marker, and every view either filters its rows to the namespaces the caller may see or authorizes the one namespace it reads. Tenant-wide lists filter: GetApplications and GetObjects push the allowed set into the registry query (DiscoverOptions.namespaces, so pages and offsets count only visible rows), GetActors skips actors of other namespaces, and the actor counts of GetSummary and GetNodeDashboard count only visible namespaces. A named namespace (the namespace field or query parameter of the application, actor and object lists) is authorized first and is PermissionDenied (HTTP 403), never an empty list. Point views take the namespace the request body names, else the caller's x-namespace, and authorize it before anything is read or forwarded: GetKeyValues, GetTupleSpaces (the shared default space when none is named, authorized like any other), GetBlobs, GetBlobPresignedUrl and GetWorkflows, so a restricted token that names no namespace is refused there like every point operation. An application row is matched by the namespace of its registry registration, which need not equal the application name (the proto field ApplicationInfo.namespace, shown by the dashboard JSON and detail page), and its figures sum only the series labelled with its tenant and namespace. The node-wide shard counters of an actor row are shown in the system scope only. The service links are for tenant administrators only (caller.is_admin(); every caller of a node without authentication holds it): a member gets PermissionDenied (403), also for the list forwarded to a peer, because a link names an outbound endpoint and its policy, which is configuration of the tenant and not data a member works with (rejected: showing members the allow-listed view). They come from the live catalog only (ServiceLinkAccess::list_links, the same visibility rule as ListServiceLinks: the allow-listed view below); there is no fallback to the static configuration, which carries the credential-bearing fields and every tenant's allow-list. Reads forwarded to a peer node travel as delegated calls, which carry the wildcard grant, so the peer would answer a tenant-wide query with the whole tenant: a point read is authorized before it is forwarded, a forwarded object listing asks the peer for one namespace at a time (one Discover per namespace the caller may list, at most the grant cap of 64, following each answer's page token until the page and one look-ahead row are in hand, merged in namespace order which is the registry's order), so its pages and has_next are exact and count only rows the caller may see, and the rows of a tenant-wide metric answer are filtered again on the forwarding node for the caller. Forwarded object rows have the same shape as local ones: no metadata, and no node_id or grpc_address below the system scope (cluster topology). A failed peer call reaches the caller as a fixed message naming the operation (Remote blob list failed, ...); the peer's own message, which can carry addresses or another tenant's names, goes to the log only. Rejected alternative: forwarding the caller's grant to the peer (the delegated call is the one trusted channel between nodes). HTTP handlers map a refusal to 403 and a missing identity to 401, not to a server error (the table is below).
    • A node without authentication serves everything. There is no tenant identity to protect, so the lists and metrics of the dashboard use the system scope (every tenant, every namespace, every series), while the point views (key/value, tuple spaces, blobs and workflows, and the dashboard actor stop) act in the no-auth tenant default, which is the tenant every store of such a node uses for a request that names none; the blob list, the presigned link and the HTTP blob download share one scope function (blob_scope), so they agree. A request that names a tenant is that tenant's administrator. Rejected alternative: failing the point views on such a node (the stores are keyed by an exact tenant, and the node is meant to be usable without credentials).
    • Listings report whether they count. PageResponse.total_size_known is false for a listing that pages with a look-ahead instead of counting (the object registry, the blob store); total_size is then 0 and has_next is still exact, and the dashboard shows "N+ shown" instead of a total.
    • metrics visibility is one rule (MetricsVisibility) shared by MetricsService.GetMetrics, the dashboard metric views and GET /metrics: the whole recorder for the system scope, the caller's tenant for a caller who may name the empty namespace (an admin, a holder of "*"), and for a token whose grant lists namespaces only that tenant's series labelled with one of them plus the tenant's user series (RecordMetric, which carry no namespace label). Any other series of the tenant without a namespace label is hidden, because it may aggregate namespaces the token was not granted; a token with no grant sees only the user series. A namespace (or label_filter["namespace"]) the grant does not list is PermissionDenied, not an empty answer. GetMetrics reads whether the layer authenticates from the request's own marker (the service has no constructor flag): a request without the marker is Unauthenticated, and when the layer did not authenticate (a node without authentication) the caller sees every series, like GET /metrics and the dashboard on such a node; a delegated peer call carries the wildcard and so receives the whole tenant, which the forwarding dashboard filters for its own caller. A token without a tenant has no visibility (/metrics answers 401 for it). The plexspaces_user_ prefix is reserved for the series tenants record through RecordMetric (the restricted views show them to the whole tenant), so no framework series uses it and a test fails if one does (the grant-change counter is plexspaces_auth_grants_changed_total).
    • Not namespace-scoped, with the reason:
      • MetricsService RecordActorActivation, RecordChannelMetrics and RecordMessageRouting are system-only. RecordMetric is Member and tenant-wide: the series carries the tenant label and no namespace label (the label is reserved), so a restricted token can create or overwrite a user series that the other namespaces of its tenant see (and that every token of the tenant reads back, whatever its grant).
      • NodeService is system-only except GetHealth, which reads no tenant data (authorizing a namespace there would only break the health probes of restricted tokens).
      • SystemService and TracingControlService hold node-wide state and no tenant data.
      • ServiceLinkService is node-wide outbound configuration. Adding and removing links is system-only; GetServiceLink and ListServiceLinks are TenantAdmin and no namespace is checked. A link is shared by every tenant, so every reader but the system scope (and a node without authentication) sees only an allow-listed view of it: name, transport, base_url with any userinfo and query stripped, publish_to_registry, policy_template and allowed_tenants reduced to the caller's own tenant (an empty list stays empty, meaning every tenant; a list that names the caller's tenant becomes just that tenant). Every other field (default_headers, api_key_header_name, api_key_env_var, bearer_token_env_var, and any field added later) is hidden by construction: the view is built by an exhaustive destructuring of the stored link, so a new field stops compiling until it is classified, and is never copied by default. Rejected alternative: clearing the credential fields of a copy (a deny-list shows a new sensitive field by default). The same rule serves the gRPC reads and the dashboard (which shows links to tenant administrators only), and the registry holds no secret in base_url: AddServiceLink and the configuration load refuse a userinfo, a query or a fragment in it. A link the caller's tenant may not use reads as absent (NotFound on get, omitted from a list), because an empty list would claim "every tenant" and any other value would be false; for the same reason a deploy that requires a link the node lacks and one that requires a link the tenant may not use get one identical error.
      • WasmRuntimeService reads no tenant or namespace.
      • UserService is tenant-scoped (accounts and tokens); the namespaces a token may hold are capped when it is issued.
    • The blob gRPC and REST servers authorize the request's namespace at one chokepoint (see "Blob namespace authorization" below); the empty namespace is a scope of its own that only a wildcard grant or an admin may name.
    • The generic layer lets the empty namespace through: an absent or empty x-namespace names no namespace and passes tier 1 (many methods name none). A service that must refuse it for a restricted caller does so with authorize_namespace(""); every service listed as enforced above does, and a service not listed does not.
    • CLI: plexspaces invoke sends no x-namespace header, so a token without "*" and without the admin flag is refused by the services above (an empty namespace is not allowed for it).
  • Not provided: a failed-authentication rate limiter.

Namespaces confine API callers, not deployed code

The namespaces claim limits what an API caller may do. It does not limit what deployed code may do: deployed code and WASM guests run with an unrestricted context (auth_enabled == false, which authorize_namespace always accepts):

  • an actor's ActorContext (for example send_reply, which builds its context with RequestContext::new_without_auth(tenant, namespace));
  • the WASM host context (new_without_auth with the application's tenant and default namespace);
  • the process-group registry of the actor crate when authentication is off.

A deployed actor or guest can therefore spawn actors, send messages, join process groups and link or monitor actors in any namespace of its tenant by naming a canonical actor id, whatever the grant of the user whose message it is handling. Deploying code is not a way for a restricted token to widen its reach: DeployApplication and DeployModule are TenantAdmin methods and an admin holds every namespace of its tenant. Treat every namespace of a tenant as one trust domain for deployed code.

The tier-1 and service checks above still apply to every call that arrives over gRPC or HTTP; only the in-process paths listed here skip them.

The request flow, from the wire to the database query:

                   gRPC Request
    │                               │
    ▼                               ▼
JWT Middleware              gRPC auth layer (JWT / node token)
    │                               │
    ├── Extract tenant_id ──────────┤
    │   (from JWT claims)           │
    │                               │
    ▼                               ▼
┌────────────────────────────────────────────────────┐
│               RequestContext                        │
│  - tenant_id: from auth (verified JWT)             │
│  - namespace: from application/actor or param      │
│  - user_id: from JWT claims                        │
│  - request_id: generated (ULID)                    │
└────────────────────────────────────────────────────┘
    │
    ▼
Service Layer → Repository Layer → Database Query
    │                                   │
    └───────────────────────────────────┘
      All queries filter by tenant_id + namespace

Namespace in ActorRef

The ActorRef stores the actor's namespace:

pub struct ActorRef {
    id: ActorId,
    namespace: String,  // Source of truth for namespace
    inner: ActorRefInner,
    // ...
}

// The caller's RequestContext (tenant from auth) is passed to every send; the ActorRef
// refuses a caller from another tenant.
actor_ref.tell(&ctx, message).await?;

Services That Take Namespace Explicitly

For services not tied to a specific actor (locks, registry, keyvalue), namespace is passed as a parameter:

// Lock service - namespace as parameter
lock_manager.acquire_lock(&ctx, options).await?;

// KeyValue service - namespace as parameter  
kv_store.get(&ctx, namespace, key).await?;

// Registry service - namespace as parameter
registry.register(&ctx, namespace, key, value).await?;

Tenant filtering is always applied to these services based on ctx.tenant_id().

Page tokens

Every keyset list pages with page_token / next_page_token. The codec, scope check and size policy live once in plexspaces_common::paging; each storage defines only its own typed key. Users today: the object registry Discover call (the WASM registry host function shares it) and the five workflow storage lists (list_definition_versions, list_definitions, get_step_execution_history, list_executions_by_status, list_stale_executions) with the workflow service's ListDefinitions / ListExecutions / GetStepExecutions and the dashboard's GetWorkflows. A token is an unsigned position marker, v1. and the hex of the last key read; it is neither signed nor secret, and a client can write one for any key. Decision: unsigned, because signing needs a key distributed to every node and would not change what a caller can see. It is safe because it never selects what a caller may see: every query is filtered by the caller's own tenant and namespace, every key carries its tenant and namespace, and a token that names another tenant (outside the system scope) or another namespace is rejected with INVALID_ARGUMENT. A forged token can only move the caller's own sweep to a different position. Do not rely on the encoding to hide key contents. Tokens longer than the storage's bound, with an unknown version or malformed are rejected the same way. Object registry Discover always orders by registry key, so sort_by and descending are rejected with INVALID_ARGUMENT instead of being ignored. Tenant, namespace and object id are limited to 255 bytes at registration, which bounds a token at 1561 characters (MAX_TOKEN_LEN).

Workflow storage tokens carry [tenant, namespace, sort values, id] (the DynamoDB stale scan token holds one such cursor per shard query); every field is escaped (%, NUL), so the bound is 9193 characters (MAX_PAGE_TOKEN_LEN, the max_len of the proto fields; the stale scan, not exposed to clients, has the larger MAX_STALE_TOKEN_LEN). The bound counts bytes: tenant, namespace, definition id and version are limited to 255 bytes. A token of another tenant or namespace is rejected with the storage's InvalidDefinition ("invalid page token: belongs to another tenant" / "... namespace"), which the services answer as INVALID_ARGUMENT, the same as a malformed or too long token, and a token whose key parts do not fit the query (a DynamoDB shard that is not a wanted shard, a step id of another execution, a version token for another definition, a position that is no number or no PostgreSQL timestamp). The cross-tenant recovery scan (list_stale_executions without only_scope) accepts a token of any tenant, as it reads every tenant; with only_scope every cursor must belong to that scope. A token is bound to its SCOPE only, not to the list that issued it: it is validated per query, and one that fits the query only moves the sweep, like a forged one. A control character (NUL included) in ANY field of a workflow storage token, and a byte below # (space, !, ") or an empty value in an id or version field, is a malformed token, refused before a backend sees it because PostgreSQL rejects a NUL in a text parameter; the object-registry tokens keep their wire behaviour (ObjectKey::to_bytes NUL handling). Forged tokens (hostile % and quote characters, other tenants' names, control characters) are conformance scenarios of every storage backend.

Node-to-Node Authentication

What is and is not enforced

  • Enforced: with authentication enabled, every gRPC call between nodes carries a node token (see "gRPC authentication and the node-to-node trust model"). Peers verify it with the cluster JWT key and constrain what it may do.
  • Not enforced: inbound mTLS. The node serves gRPC and HTTP on one plain listener. It does not request or verify client certificates, and a client certificate is never a source of identity or tenant. The mtls block of the security config and the certificate auto-generation helpers (cert_gen) do not turn on inbound TLS. They only run a startup check: with mtls.enable_mtls: true (the default once an mtls block is present) the node exits unless the PLEXSPACES_MTLS_* files exist, or auto_generate_certs: true makes it write a development CA and server certificate into cert_dir (PLEXSPACES_MTLS_CERT_DIR, default /app/certs). Nothing then uses those files unless the config paths or the PLEXSPACES_MTLS_* variables also point at them (outbound TLS, below).
  • mTLS path precedence: the release config first, then the environment, the same rule as the JWT key (plexspaces_common::config_then_env, used by config conversion, the startup check and config_manager::initialize). A non-blank ca_certificate / client_certificate / client_private_key / cert_dir under runtime.security.mtls wins; the PLEXSPACES_MTLS_* variables only fill blanks, so a stale variable cannot replace what a reviewed file names. Rejected alternative: environment first, which let a leftover variable override the file. The startup check and the code that dials peers (build_connection_manager in crates/services/src/service_initialization.rs) resolve the paths through one function (plexspaces_common::security_validator::MtlsFilePaths::resolve), and both return the trimmed value.
  • Outbound TLS: only when runtime.security.mtls.enable_mtls is explicitly true (design decision: certificate defaults such as auto-generated certificates, or leftover PLEXSPACES_MTLS_* variables, never switch a node's transport; rejected alternative: enabling TLS whenever three existing files are found). The CA certificate, certificate and key paths then come from the config, else from PLEXSPACES_MTLS_CA_CERT, PLEXSPACES_MTLS_SERVER_CERT and PLEXSPACES_MTLS_SERVER_KEY; if one of them is missing the node logs a warning and stays on plaintext. With TLS on, the connection manager dials peers over TLS with that client certificate and verifies the peer against the CA (server name plexspaces.local). The scheme follows the transport, not the stored address: peers are advertised and stored as http://host:port, and a connection manager with TLS configured always dials https://host:port (plexspaces_common::dial_endpoint), so it never downgrades to plaintext. Their TLS must be terminated by something that holds a certificate of that CA (a proxy, sidecar or mesh that also enforces client certificates if you want mutual authentication). Whether a channel is TLS is read back from the scheme of the endpoint tonic dials, and that same value decides whether the node token may be attached; the startup log field mtls_outbound reports whether the connection manager actually has TLS (a missing certificate file or a load failure leaves it plaintext), not just whether the environment variable is set. Peer identity comparisons (trust_peer, address equivalence) use canonical_node_address_key, which ignores the scheme. A wire-level test (grpc-middleware/tests/node_transport_auth_test.rs) runs a real TLS handshake against a CA generated in the test and shows the token is attached over TLS, withheld from plaintext destinations, and never sent to a plaintext peer by a TLS-configured manager.
  • Startup rule: with authentication enabled the node refuses to start unless that outbound TLS is configured or allow_insecure_node_transport: true is set in the security config. With the opt-in, the node token is sent over plaintext only to the peers listed in cluster_seed_nodes; any other plaintext destination gets no token and the call fails (UNAUTHENTICATED, counted in plexspaces_node_token_withheld_total). The opt-in logs a warning at startup. Use it for development and isolated test clusters only.
  • Single node is exempt: a node with no cluster_seed_nodes configured does not need TLS or the opt-in to start with authentication enabled. It dials no configured peer, so it has no node token to send. The node token source is still installed and the interceptor stays fail-closed: a call to any peer that is neither reachable over TLS nor a trusted seed (for example a node that later joins and is dialled back, or an application's seed_nodes connected at deploy time) is refused with UNAUTHENTICATED and counted in plexspaces_node_token_withheld_total, never sent without credentials. Give a node seed nodes only together with TLS or the explicit plaintext opt-in.
  • Authentication disabled on a public bind is refused. With disable_auth every caller is served as an administrator of every tenant, so a node whose listen_addr is not a loopback IP (0.0.0.0, [::], a LAN or public address) refuses to start unless allow_unauthenticated_public_bind: true is set in the security config (or PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=1). With the opt-in the node starts and logs a prominent warning (AUTHENTICATION IS DISABLED ...). A loopback bind (127.0.0.1, [::1]) never needs it, and neither does a node with authentication enabled. Use the opt-in for isolated development networks and throwaway containers only. The bundled release.yaml (auth off, 0.0.0.0) sets it with a comment; release-auth.yaml does not.

Certificate management

Production certificates should come from your PKI (cert-manager, Vault, a mesh CA). Example CA and server certificate generation with openssl:

openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 365 -key ca.key -out ca.crt
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -out server.crt
export PLEXSPACES_MTLS_CA_CERT=/certs/ca.crt
export PLEXSPACES_MTLS_SERVER_CERT=/certs/server.crt
export PLEXSPACES_MTLS_SERVER_KEY=/certs/server.key
# and, in the release config: runtime.security.mtls.enable_mtls: true

Certificate rotation is not automated: replace the files and restart the node.

User API Authentication (JWT)

Overview

JWT authentication provides secure access to user-facing APIs:

  • HTTP API: JWT validated at the HTTP gateway; tenant_id comes only from validated JWT claims (never from client-sent headers like x-tenant-id).
  • gRPC: One authentication layer (node::grpc_auth) wraps every gRPC and gRPC-Web service. It validates the Bearer JWT with the same code as HTTP, strips client-supplied identity headers and stamps them from the claims. See "gRPC authentication and the node-to-node trust model" below.

The JWT middleware:

  • Validates JWT signature and expiration
  • Supports ES256 (ECDSA P-256, asymmetric) only; HS256 shared secrets are rejected
  • Extracts tenant_id, roles, groups, and is_admin from claims
  • Sets request metadata from claims only (prevents header injection)
  • Propagates tenant context via RequestContext
  • Exposes a JWKS endpoint (GET /.well-known/jwks.json) for ES256 public key discovery

JWT Claims (HTTP and gRPC)

Standard and custom claims used by PlexSpaces:

Claim Type Required Description
sub string Yes Subject (user ID)
exp number Yes Expiration time (Unix seconds)
iat number Yes Issued at (Unix seconds)
tenant_id string Yes Tenant identifier (required for multi-tenant isolation)
roles array No Role names for RBAC
groups array No Group names
is_admin boolean No Admin privilege flag
namespaces array No Namespaces of the token's tenant it may reach: namespace names or "*" (every namespace of the tenant). Absent or empty means none (fail closed)

Namespaces claim: a JSON array of strings, like roles and groups. Each entry is "*" or a valid namespace name ([A-Za-z0-9._-], at most 128 bytes, not only dots); empty entries are rejected and a list holds at most 64 entries. One validator (plexspaces_common::validate_namespace_entries) is used at every issuance path: jwt-create, HTTP and gRPC API-token creation (CreateApiToken), the user grants (GetOrCreateByEmail, SetUserGrants) and the OIDC config. A tenant admin (is_admin) implicitly reaches every namespace of its own tenant; a restricted token never reaches the empty "no namespace" scope. The claim helper JwtClaims::allows_namespace(request_tenant, ns) holds these rules in one place, including the tenant check: a token of another tenant reaches nothing there, an admin included; the only cross-tenant holder is the system scope (an admin with an empty tenant_id). The auth layers and the request context enforce the claim (see Namespace authorization).

Who may grant what (minter cap): a token minted through the API is capped at its minter. Every requested entry must be allowed by the minter's own namespaces, so "*" needs a minter that holds "*" or is a tenant admin; an over-grant is refused (HTTP 403, gRPC PermissionDenied), never silently narrowed. A token minted for another user is additionally capped by that user's allowed_namespaces (the intersection; a "*" on one side defers to the other). Requesting no namespaces yields a token that reaches none. Over gRPC the minter's grant is the allowed_namespaces of the request context, i.e. the trusted x-allowed-namespaces header stamped from the caller's verified token, so a non-admin member can mint only what its own token holds. API-token claims omit an empty aud array.

Security: When auth is enabled, tenant_id comes from the verified JWT only. Client-provided x-tenant-id is not trusted; the gateway and the gRPC auth layer overwrite it from the validated token. Tenant names are restricted to [A-Za-z0-9._-], at most 128 bytes, and plexspaces-node, __other__, system, plexspaces and (with auth on) default are reserved; names match ignoring ASCII case (SYSTEM and Default are refused like system and default). plexspaces (INTERNAL_PUBLICATION_TENANT) is the tenant under which the node publishes its configured service links to the object registry at startup; the node builds that context itself, so reserving the name does not affect the publication.

Rejected alternatives behind the code comments

  • Namespace authorization guard (node/src/namespace_gate.rs). Rejected: a runtime registry (every handler would have to register itself, the discipline the table checks), substring search of the sources (it accepts a comment, a string, a dropped result and a missing function), a dataflow analysis (large, and it would still need the service tests), and a bare authorize call per handler instead of a chokepoint (a missed handler stays open and the call is duplicated). The policy_gate module uses the same syn technique for the HTTP routes.
  • HTTP policy gate (node/src/http_routes/policy_gate.rs). Handlers keep no idea of their own about who may call (no is_admin test, no inline refusal): every route path is a constant of the gate module and the tests keep a row per route with its gRPC methods. Rejected: a gate in each route module (they drift) and discovering the routes by scanning the router (axum does not list its routes). A SystemOnly row has no user caller, so the HTTP route refuses every JWT user, a tenant admin included; rejected: letting a tenant admin through (the HTTP port would give a platform-operator capability the gRPC port withholds). The gate answers 403 with the policy row's message and GET /api/v1/nodes maps it to its own text in the handler; rejected: a message field on GatedRoute (authorize_caller would have to take the route, not the method).
  • Reserved tenant names. Rejected: a separate registry key space for internal rows (every reader would need to know about it) and a migration that renames an offending stored tenant (the operator decides what a tenant that held a reserved name becomes). The reserved names are rejected at every door where a tenant is claimed or stored from outside; the node's own context for the internal publication is built in process and never passes through those doors.
  • The empty namespace. namespace_grant_covers covers it only for a wildcard or an admin, so a token restricted to named namespaces cannot reach tenant-wide scope by naming nothing. Rejected: treating the empty namespace as "any namespace of the grant" (a request naming none would act in whichever namespace a service defaults to, which a restricted token was never granted).
  • Download headers and service-link URLs. The download name is reduced to its last path segment, cut to 200 characters (MAX_FILE_NAME_CHARS) so the header stays bounded (each non-ASCII character grows to up to 12 encoded bytes), and sent twice: an ASCII filename fallback and a percent-encoded UTF-8 filename*. Rejected: interpolating the name into a quoted string (a quote or CR/LF breaks the header) and each surface carrying its own escaping. A service link is STORED in its canonical form at every entry (configuration load and AddServiceLink), so every reader, the client that dials and a later comparison see the string that was judged. Rejected: storing the text as given and re-validating on each read.
  • Stored reserved tenant rows. The node refuses to start (no renaming migration): what the tenant, its users, tokens and data should become is the operator's decision, a silent rename would orphan data keyed by the old name. Rejected: starting with the row (the reserved name stays claimable through its tokens) and dropping it (data loss).

Reserved tenant names

One function, plexspaces_common::validate_tenant_name(name, auth_enabled), decides what a tenant name may be, and every door that can run with authentication off passes the node's actual authentication mode (tenant creation through SqlTenantRepository::new(pool, auth_enabled), the apps-directory loader, release specs, the HTTP deploy); the doors that read tokens (the gRPC auth layer, the HTTP surfaces) and plexspaces jwt-create apply true, because a node with authentication off reads no tokens. default (the no-auth tenant) is reserved only with authentication on and is a usable tenant on a node with authentication off; the other reserved names are refused in both modes, and all of them match ignoring ASCII case. A reserved name is refused at every door where a tenant is claimed or stored from outside: the claims of the gRPC auth layer and of the HTTP surfaces (the HTTP auth middleware, the WebSocket upgrade and the blob REST server, through http_gateway::claims_with_claimable_tenant; 401), tenant creation, the apps-directory loader (a directory with such a name is skipped), tenant names in release specs (the entry is skipped and logged), plexspaces jwt-create (it refuses to mint such a token), and a startup check: the node REFUSES TO START when a stored tenant row already carries a reserved or invalid name (refuse_reserved_tenant_rows; the message names the row's slug and id). The node's own internal context for the publication never passes through those doors. Decision: refuse to start and let the operator rename or delete the row. Rejected alternatives: a migration that renames the tenant (what the tenant, its users, tokens and data should become is the operator's decision, and a silent rename would orphan data keyed by the old name), starting with the row (the reserved name would stay claimable through its tokens) and deleting it (data loss).

The startup check needs a readable user schema. With auto_migrate off the schema is the operator's; when the tenants table cannot be read, a node with authentication ON refuses to start (tokens and tenants cannot be checked), and a node with authentication OFF skips the user service with a warning and starts (user_schema_usable in crates/node/src/node/grpc_server.rs). A row with a reserved or invalid name stops the node in both modes. The same rule covers a user database that cannot be opened. The error text of a connection string that is not a SQLite URL never contains the string, which may hold credentials.

Dashboard pages

The dashboard HTML pages are public (/dashboard/node/:node_id, /dashboard/metrics/:node_id, /dashboard/application/:name, /dashboard/tenant/:tenant_id) and put the path segment into the page. The segment is escaped where it is substituted, by the place the placeholder sits in: element text gets an HTML escape (&, <, >, ", '), a single-quoted script string (const nodeId = ':node_id';) gets a JS escape in which every character but ASCII letters, digits, -, _ and . is a \u{..} escape (fill_template in crates/dashboard/src/dashboard_handlers/pages.rs). Rejected alternative: refusing ids with punctuation, which would hide the pages of legitimate ids. The tests substitute a script payload into each page.

Creating JWT Tokens (CLI)

Use the PlexSpaces CLI to create tokens for local or integration testing:

# Create a JWT signed with the node's ES256 private key
# (key file from --private-key-file or PLEXSPACES_JWT_PRIVATE_KEY_FILE)
export PLEXSPACES_JWT_PRIVATE_KEY_FILE=./certs/jwt-es256.pem
cargo run -p plexspaces-cli -- jwt-create \
  --tenant-id internal \
  --sub system \
  --roles admin \
  --groups "ops,dev" \
  --is-admin \
  --exp-hours 1

# Or pass the key file explicitly
cargo run -p plexspaces-cli -- jwt-create \
  --tenant-id my-tenant \
  --sub user@example.com \
  --roles "read,write" \
  --namespaces "staging,prod" \
  --private-key-file ./certs/jwt-es256.pem \
  --exp-hours 24

--namespaces takes a comma-separated list of namespace names, or "*" for every namespace of the tenant. Without it the token carries no namespaces and reaches none (the CLI prints a hint on stderr). The CLI signs with the node's key, so no minter cap applies to it.

The CLI prints the token and a usage hint. Use it in HTTP requests as Authorization: Bearer <token>.

Configuration

Configure JWT in your release.yaml (or release-auth.yaml):

runtime:
  security:
    jwt:
      enable_jwt: true
      private_key_pem: ""             # Inline PEM → reads PLEXSPACES_JWT_PRIVATE_KEY env var
      private_key_file: ""            # Path to PEM → reads PLEXSPACES_JWT_PRIVATE_KEY_FILE env var
      auto_generate_key: false        # Only honoured when auth is disabled; with auth on a persistent key is required
      issuer: "plexspaces"            # Not enforced: the validator does not check iss
      jwks_url: ""                    # Not enforced: external JWKS validation is not implemented
      allowed_audiences:              # Not enforced: the validator does not check aud
        - "plexspaces-api"

Only enable_jwt, private_key_pem, private_key_file, and auto_generate_key are read at runtime. (JwtConfig has no HS256 secret, algorithm, token TTL or claim-name fields and SecurityConfig has no api_keys; the protobuf field numbers are contiguous with no reserved entries, so a serialized config must be written with the current numbering.) issuer, jwks_url and allowed_audiences are accepted but not enforced (iss and aud are not checked, and JWKS validation is not implemented). Design decision: the signature already proves the cluster's own key issued a token and access is scoped by tenant_id, not by an audience, so the two checks are reserved for accepting tokens from an external identity provider (which needs JWKS validation); enforcing them now would reject the node's own tokens for no security gain (comment on validate_es256_token in crates/grpc-middleware/src/jwt.rs). The claim names are fixed (tenant_id, sub), token lifetimes are chosen by whoever issues the token (there are no token_ttl, refresh_token_ttl, tenant_id_claim or user_id_claim options in the proto or the YAML, because nothing would read them), and the algorithm is always ES256. auto_generate_key defaults to false and is honoured only when authentication is disabled; with authentication on it is ignored and a missing key stops startup. Tokens are validated only against the node's own ES256 key.

Unknown keys are rejected everywhere. Every section of a release YAML (not just jwt) fails the config load with an "unknown field" error naming the offending key (for example locks_backend). A leftover secret, algorithm, tenant_id_claim, user_id_claim, token_ttl, refresh_token_ttl or disable_auth_for_testing key therefore fails startup instead of looking honoured. The same holds for security.api_keys, security.authz_config (policy_file_path) and service_identity.public_key: they would be parsed and dropped, because authorization is the built-in policy table and API tokens are signed JWTs, so they are rejected rather than kept as a silent no-op (the YAML structs do not carry them; the service_identity block itself is still accepted and its private key is masked in logs, but it is not read at runtime: it is a deliberate placeholder kept for a per-node credentials design (per-node asymmetric keys with a replay cache, then tenant/method-bound tokens), an exception to this rule. Node-to-node calls authenticate with the cluster JWT key and outbound mTLS uses mtls; do not rely on service_identity for anything). The proto has no SecurityConfig.api_keys field and no API-key message. The reason: a security option that is silently ignored is worse than a startup error, because the operator believes it is active. See Installation.

Key Management

ES256 (Asymmetric, Preferred):

PlexSpaces uses ECDSA P-256 (ES256) by default. The private key signs tokens; the corresponding public key verifies them and is exposed via the JWKS endpoint.

Key resolution (one shared rule used by config loading, startup validation, the auth snapshot and JwtKeyPair::from_config, so they cannot disagree):

  1. If either private_key_pem or private_key_file is set in the release config, the environment is not consulted at all.
  2. Otherwise PLEXSPACES_JWT_PRIVATE_KEY (inline PEM) and PLEXSPACES_JWT_PRIVATE_KEY_FILE (path) are read.
  3. Blank (whitespace-only) values count as unset and the file path is trimmed, so a stray space never names a file.
  4. If both a PEM and a file remain, the PEM wins and one warning is logged, by the loader that builds the key (the resolver itself is silent because config conversion, validation and loading all call it during one startup).
  5. A configured key file that does not exist is generated and saved there with mode 0600 only by a node without seed nodes (also with authentication on). With authentication on and cluster_seed_nodes set, a missing key file stops startup with an error naming the path: each node would otherwise generate a different key and reject its peers' tokens. The rule is about the node's own seed list, not about whether a cluster exists: the first node of a cluster (empty cluster_seed_nodes) may generate the key file, and any node with seed nodes, even a list that names only itself, refuses to. Create the key once and distribute it (Kubernetes Secret, shared volume). An ephemeral in-memory key is never used when authentication is enabled.
  6. A key file with a mode that lets the group write it or others touch it (mode & 0o027) logs a warning; 0600 and 0440 (a Secret mounted with defaultMode: 0440 and fsGroup) do not.

Why config first, then environment, and not a per-field merge: a stale PLEXSPACES_JWT_PRIVATE_KEY left in a shell or container image must not silently replace the key file an operator configured, nor pair a config PEM with an environment file path. Why the PEM wins over the file instead of failing the start: from_config has always documented the PEM first, and one source is enough to proceed; the warning makes the conflict visible.

First start on a shared path. Generating the file is race-free: each starter writes a private temp file (named with its pid and a ULID, mode 0600) and hard-links it to the target, which fails if the file already exists; the starter then reads the published file back. Nodes that start at the same moment on a shared volume therefore all end with the same key, whichever node won. A rename would instead let each node overwrite the previous winner's key. The target directory must support hard links (ordinary Linux and macOS filesystems do; some network and FUSE mounts do not: provide the file instead of generating it there). For a cluster, prefer distributing one key (a Kubernetes Secret) over generating it on a shared volume.

Startup. With authentication enabled the node resolves the key once at startup and shares it between session/API-token signing and HTTP/gRPC verification and node tokens. If no persistent key is configured, or PLEXSPACES_JWT_SECRET is set, the node refuses to start with an explicit error. The key is required even when jwt.enable_jwt is false: enable_jwt only controls the early check in validate_security_config (see below), while node-to-node tokens are signed with the same key. Use the same key (file or PEM) on every node of a cluster. Rotating the key invalidates every user and node token; roll all nodes to the new key together.

Generate a key manually:

openssl ecparam -genkey -name prime256v1 -noout -out jwt-es256.pem

plexspaces jwt-create signs with an existing key only: if the file named by --private-key-file (or PLEXSPACES_JWT_PRIVATE_KEY_FILE) does not exist, the command fails instead of generating one. A mistyped path would otherwise yield tokens signed with a key no node trusts.

JWKS Endpoint: GET /.well-known/jwks.json returns the ES256 public key in JWK format. External services can use this to verify PlexSpaces-issued tokens without sharing secrets.

HS256 is not supported. There is no secret configuration key, and PLEXSPACES_JWT_SECRET is rejected at startup whenever authentication is enabled, whether or not enable_jwt is set. The reason: the node signs node-to-node tokens with the ES256 key regardless of enable_jwt, so an operator who still exports the secret expects it to matter, and silently ignoring it would leave them believing HS256 tokens are accepted. An empty or whitespace-only PLEXSPACES_JWT_SECRET (a VAR= placeholder in a compose file) counts as unset.

Validation:

  • Startup validation (validate_security_config) skips everything when authentication is disabled. With authentication on it first rejects a leftover PLEXSPACES_JWT_SECRET (an "HS256 is not supported" error), then, when enable_jwt is true, requires an ES256 private key from private_key_pem / private_key_file or PLEXSPACES_JWT_PRIVATE_KEY / PLEXSPACES_JWT_PRIVATE_KEY_FILE. A jwks_url does not satisfy it.
  • With authentication on and enable_jwt: false the validator asks for no key, but node startup still does (see Startup), so the key requirement holds in every authenticated configuration.
  • If authentication is enabled and no persistent key is available the node does not start
  • If PLEXSPACES_DISABLE_AUTH=1 is set, auth is disabled (testing only)

Multi-Node Clusters: All nodes in a cluster must share the same JWT signing key so tokens issued by one node are valid on all others. The default scripts/server.sh uses a shared path (./certs/jwt-es256.pem) regardless of port. In production, distribute the same PEM file to all nodes (via Kubernetes secrets, Vault, etc.) or set PLEXSPACES_JWT_PRIVATE_KEY from a shared secret store.

The image is not secure by default; the deployment files are. The container image ships with authentication off (PLEXSPACES_DISABLE_AUTH=1, PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=1 in the Dockerfile) for one reason: docker run on a laptop works without a key. That default is labelled as local-development-only in the Dockerfile, so anyone running the bare image, or a hand-written manifest that does not override it, gets an unauthenticated node. The shipped deployment files (docker-compose.yml, k8s/*.yaml) override it; docker-compose.dev.yml keeps it off on purpose and says so:

File Authentication JWT key Other
docker-compose.yml PLEXSPACES_DISABLE_AUTH=0, public-bind opt-in 0 certs/jwt-es256.pem bind-mounted read-only at /app/jwt; create it first with openssl (no CLI command generates the key) named volume for /app/data
k8s/deployment.yaml same, as container env plexspaces-jwt Secret mounted read-only at /app/jwt no seed nodes (pod names are not stable)
k8s/statefulset.yaml same same seed nodes and allow_insecure_node_transport: true in its ConfigMap (see below)

StatefulSet transport: isolated cluster only. The nodes have no inbound TLS, and a node with authentication on and seed nodes refuses to start unless it dials peers over TLS or sets allow_insecure_node_transport. Outbound-only TLS would make every peer call fail (TLS to a plaintext listener), so the manifest uses the opt-in: the node token is sent in plaintext to the seed nodes. Use it only where the pod network is trusted, and put a service mesh or TLS sidecar in front of the nodes for transport security.

The overrides are explicit (0, not just omitted) because the image default would otherwise win. With the key missing the node refuses to start, which is the intended failure. Do not apply k8s/secrets.yaml over a Secret you have already filled in; create plexspaces-jwt with kubectl create secret generic plexspaces-jwt --from-file=jwt-es256.pem (scripts/deploy-k8s.sh checks the Secrets and never applies that file). The mounted key is group-readable (defaultMode: 0440 with fsGroup: 1000) so the non-root user can read it, and the node accepts that mode without a warning. The release config embedded in each manifest is loaded by a test (test_release_yaml_embedded_in_the_k8s_manifests_loads); the manifests are otherwise not exercised against a cluster.

Tenant Extraction (No Client-Sent tenant_id When Auth On)

When authentication is enabled:

  • HTTP: The gateway validates the Authorization: Bearer <token> header and derives tenant_id (and optionally roles, groups, is_admin) from the JWT only. Path parameters or headers like x-tenant-id from the client are not used for tenant identity.
  • gRPC: The gRPC auth layer sets x-tenant-id (and related headers) from the verified token only; handlers read them through request_context_from_grpc_metadata, which never takes the tenant from request labels when auth is enabled.

HTTP routes follow the gRPC method policy. POST /api/v1/applications/deploy and DELETE /api/v1/applications/:application_id need a tenant admin, because the gRPC DeployApplication and UndeployApplication rows are TenantAdmin ("deploying code runs it on the node"). The routes do not keep their own rule: they ask the same policy table (grpc_auth::policy_for and check_user_class) with the claims the HTTP auth middleware verified, so the two entry points cannot drift. Non-admin is 403, no verified claims 401, no JWT key 503. Rejected alternative: a separate is_admin test in the route, which would drift from the table.

The same gate (http_routes::policy_gate::authorize_caller) guards every HTTP route that stands for a gRPC method with a non-Member row, and each route names its method in one table (policy_gate.rs) that the routers and the tests both read:

HTTP route gRPC method Row Who gets through (authentication on)
POST /api/v1/applications/deploy, DELETE /api/v1/applications/:id DeployApplication, UndeployApplication TenantAdmin tenant admin
GET /api/v1/auth/users ListUsers TenantAdmin tenant admin (own tenant only)
GET /api/v1/auth/tenants ListTenants TenantAdmin tenant admin (its own tenant)
POST /api/v1/admin/tracing/enable, POST .../disable, GET .../status EnableTracing, DisableTracing, GetTracingStatus SystemOnly nobody: every JWT user, a tenant admin included, gets 403; only a node with authentication off serves them
GET /api/v1/nodes ListConnectedNodes SystemOnly nobody (same as above)

SystemOnly means "no user token", so a tenant admin is refused on the tracing routes exactly as it is over gRPC; there is no platform-operator role. Rejected alternative: letting a tenant admin toggle tracing over HTTP, which would give the HTTP port a capability the gRPC port withholds. Routes with a Member or Public counterpart (actors, API tokens, login, JWKS, static application files) need only the authentication middleware; GET /metrics has a SystemOnly gRPC counterpart (MetricsService/GetMetrics) but is deliberately different: it serves only the series labelled with the caller's tenant. The dashboard routes live in the dashboard crate and stand for DashboardService, whose rows are all Member. The blob REST routes run on their own server (blob_http_router), authenticate with blob_http_router::authenticate and stand for the blob service's Member rows; they are not in the route table. A test fails if a gated route's declared method is missing from the policy table or is a Member row, or if the gate's outcomes differ from the row. A gated handler reached without the middleware layer (no verified claims) answers 401: a missing layer fails closed, and router-level tests build each gated router with and without the layer. /ws is the exception: its handler does not check claims itself, so a router built without the layer does not answer 401 for it (the test only shows it is not refused as unauthorized). The tests of node::http_assembly run the node's real assembly (all_http_routes inside with_gateway_auth) with authentication on: no token is 401 on a gated route, the dashboard API and /metrics, a member on a TenantAdmin route is 403, and forged x-tenant-id/x-admin headers are replaced by the token's values.

WebSocket upgrade (/ws)

The upgrade is authenticated by the same middleware and the same authenticate_claims as every other route: the token comes from ?token= (browsers cannot set headers on a WebSocket, so it wins over the header) or Authorization: Bearer; a session cookie is not accepted (a cross-site page could otherwise open a session with the user's cookie). A revoked API token is 401, a revocation lookup failure 503 (fail closed), an oversized credential 401. Rejected alternative: the handler validating the token itself, which would repeat revocation and the size bound.

A /ws session ends with its token. The token is verified at the upgrade, so the session carries a cutoff computed from the verified claims. At the token's exp (the deadline is an absolute timer instant fixed in the upgrade handler, from the node clock, so the handshake time does not delay it and clock steps after the upgrade do not move it; an exp already passed ends the session at once, one further than 30 years away is capped at 30 years; a frame handler or revocation lookup that is still running at exp, for example blocked on a full channel or a hung database, is dropped, so neither can keep the session open past it) the node sends a WebSocket close frame with code 1008 (policy violation) and reason token expired. A token with a jti (an API token) is also rechecked against the revocation list every 60 seconds (WS_REVOCATION_RECHECK, first check 60 s after the upgrade); a revoked token ends the session with code 1008 and reason token revoked. A revocation is therefore seen within 60 s plus the 5 s TTL of the revocation cache, less when it was made through the same node. A recheck that fails (database error) or cannot run (the node has no revocation list) keeps the session open and increments plexspaces_ws_revocation_recheck_errors_total: the upgrade already failed closed on the same lookup, and one database blip must not end every session of the node. Tokens without a jti (session tokens) are cut off at exp only. With authentication off there is no cutoff. Rejected alternatives: cutting off at exp only (a revoked API token, valid for months, would keep its session), and revalidating the token on every frame (a lookup per message). Cost: one cached lookup per jti session per minute (100,000 such sessions are 100,000 lookups a minute against a 5 s cache). Every session end is counted in plexspaces_ws_sessions_closed_total{reason}. A client should reconnect with a fresh token on close code 1008.

?token= puts a bearer token in the URL. URLs are logged by proxies, load balancers and access logs and can leak through referrers. Use wss://, issue short-lived tokens for WebSocket sessions, and scrub the token query parameter in proxy and access logs.

Deploy archives and actor headers

Deploy zips are bounded by inflated bytes. The 200 MiB total decompressed cap (equal to the upload limit) of POST /api/v1/applications/deploy is charged with the bytes actually inflated, because a zip header can declare a few bytes for an entry that inflates to hundreds of MiB and the node keeps every buffer. Each read stops one byte past the remaining budget; the declared size is only an early exit, and each buffer is reserved from the declared size but never beyond the remaining budget plus one byte, so memory held is the cap plus allocator slack. Unused entries (look-alikes of app-config.toml, files outside static/) are not inflated at all, and every entry name, used or not, is checked for path traversal. Rejected alternative: trusting the declared sizes. A zip with more than 10,000 entries is refused with 400 before any entry is read. The unpack runs on spawn_blocking, never on an async worker, and at most 4 deploys run past the upload at once; the slot is held until the deploy ends (the unpacked files stay in memory through registration and the deploy call) and a deploy that finds no free slot gets 503 with Retry-After immediately instead of queueing behind long inflates. Worst case for the deploy path is about 1.6 GiB (4 x input zip up to 200 MiB plus unpacked files up to 200 MiB); uploads in flight are not counted by the slot, each is bounded by the 200 MiB body limit, and the multipart read may briefly hold a second copy when the received buffer is shared. The entry-count check happens right after the archive is opened, so parsing a hostile central directory is not capped by it; the body limit and the slot bound that cost. Rejected alternative: unbounded concurrent unpacks.

Actors never receive credentials. The actor service copies the request headers into the actor message (ActorServiceImpl::build_message_metadata, shared by ask and tell, for the HTTP bridge and for direct gRPC AskReply/SendMessage), except authorization, proxy-authorization, cookie, set-cookie, the identity headers the gateway stamps from verified claims (x-tenant-id, x-user-id, x-admin, x-user-roles, x-user-groups, x-node-caller) and x-plexspaces-call-kind, compared case-insensitively. An actor can log, store or echo what it receives, so a bearer token or session cookie must not be among it, and it learns its tenant from the request context, never by re-reading a token or header. Rejected alternatives: an allow-list of forwarded headers, which would silently break actors that read application headers such as content-type or x-request-id; stripping only in the HTTP gateway, which would let a gRPC caller make an actor see a forged x-admin.

Namespace authorization guard. crates/node/src/namespace_gate.rs declares, for every gRPC method of the policy table, where its namespace is authorized: a named function (Authorized), a node-wide or system-only method with no namespace (Exempt, with a reason), a list that filters rows below the handler (RowFilter, not followed by the guard and covered by the service tests), an unimplemented method (Stub(file), whose handler must do nothing but return Status::unimplemented, checked), or a method whose handler does not authorize the namespace it uses (None). The tests parse the sources with syn (a dev-dependency of the node crate only): the table equals grpc_auth::classified_paths() and, through the proto descriptor, every method of every mounted service has a row (a method added to a proto without one fails); every Authorized function is found as exactly one non-test item (a bare name such as get, put, list, read or count that matches two items fails closed) and contains a call of a shared rule or of its service chokepoint whose result is used. The shared rules are matched by call form: the free functions (authorize, derive, authorize_embedded, authorized_request_context, listable_namespaces, authorized_blob_context, ...) only as calls of a path and the RequestContext rules (authorize_namespace, derive_in_namespace, can_see_namespace) only as method calls, so an unrelated x.authorize(..) does not count. This check is a smoke check, not a proof: it shows that a call with the name of a shared rule is made and used, not that it receives the request's namespace; the behaviour is proven by the service tests (caller x own/other/empty). A result is used when it is the operand of ?, returned, matched, or bound to a name that is read later (a comment, a string, check(..); and let _ = check(..); fail); every chokepoint itself uses a shared rule; every row marked system-only is a SystemOnly policy row (no Member or TenantAdmin method is marked so); the None and RowFilter sets equal pinned lists, so a method that gains or loses its authorization changes a list in the same commit; and every generated server constructed in the node crate (new, from_arc, with_interceptor, also inside add_service!) is a pinned mounted service with rows, and a mount the scan cannot read fails the test (the Firecracker service must stay unmounted). The request messages of Exempt and None rows that a user token can reach (not the system-only, stub and unmounted rows) are walked in the proto descriptor set (plexspaces_proto::FILE_DESCRIPTOR_SET, written by buf build in make proto-buf): any field with namespace in its name (namespace, namespaces, allowed_namespaces), a tenant_id* field (tenant_id_filter), application_id, *actor_id*, labels, receiver_id or target anywhere in such a request must be listed in SCOPE_FIELD_EXEMPTIONS with a reason (SetUserGrants and CreateApiToken carry the grant being stored or minted, ListUsers a tenant filter honoured for the system scope only). crates/proto/tests/descriptor_tests.rs compares the descriptor with the generated Rust by message names and service method names only (field names are not compared): a message or method of the descriptor that is not a generated struct or trait method fails, so a descriptor.bin that missed a regeneration of a new message or method is noticed, while a field added to an existing message is not; make proto-buf is therefore the only supported way to regenerate (see the proto crate README). The HTTP routes of the node's main router are declared in http_routes/policy_gate.rs: production code keeps only the path (and, for a gated route, the gRPC method) in GatedRoute and OpenRoute, and the unit tests keep a table with one row per route (what it stands for and its HttpNs: Grpc(rows), RowFilter or Exempt(reason)). A test parses the node sources and fails when a path registered with .route(policy_gate::X.path, ..) is not in GATED_ROUTES or OPEN_ROUTES (a route registered with a literal path is not seen), another fails when a route has no row or a row has no route, and every route whose namespace is authorized by gRPC rows (HttpNs::Grpc) must name Authorized rows (and, for a gated route, its own method). Decisions: parse, do not match text (rejected: substring search accepts a comment or a dropped result); a runtime registry was rejected (it is the same discipline the table checks). The only method without namespace authorization, pinned, is WasmRuntimeService/DeployModule: the module cache is content-addressed (the SHA-256 of the module bytes) and node-wide, holds code and no tenant data, and its request has no namespace; whether a tenant admin may add code to the node-wide cache is decided with the deployed-code pass.

Blob namespace authorization. The blob gRPC and REST servers authorize the request's namespace at one chokepoint (plexspaces_blob::server::context::authorized_blob_context) with the shared rule RequestContextExt::authorize_namespace: a request without the AuthLayerApplied marker is Unauthenticated (gRPC) or 401 (REST) whatever headers it carries; a missing or empty tenant is the same; the empty namespace (no x-namespace) is a scope of its own that only a wildcard grant or an admin may name, so a restricted token that names none, a namespace outside the grant, a second x-allowed-namespaces value or a non-text x-namespace is PermissionDenied (gRPC) or 403 (REST); with authentication off (as the marker says) the context is unrestricted and the grant and admin headers are not read. The node's blob HTTP middleware removes a planted marker and inserts the real one on both paths. Decision: the servers read the marker and the grant from the request; there is no auth_enabled constructor flag. Rejected alternatives: a flag per server (each copy could be wired wrongly) and a dependency of the blob crate on the actor crate (the marker and the ALLOWED_NAMESPACES_HEADER constant live in plexspaces_common instead). All six blob RPCs are Authorized rows of the guard.

Application registrations

Where a node or an application is reached is decided by the node, never by a tenant. The gRPC ObjectRegistry.Register and the WASM host register call the one predicate plexspaces_actor::object_registrations::admit_tenant_registration, with authentication on or off: a NODE row is refused (PermissionDenied / invalid-input); an APPLICATION row that names another node (by node id or gRPC address) is refused the same way; every row that passes is stored with the receiving node's own id and address, so an address that is merely equivalent to the node's (a path, a query, another scheme) is never stored in the caller's spelling, and an empty node id or address is filled the same way (the empty value is the caller's to leave out). A tenant-written address would redirect traffic, and the credentials that travel with it, to a host of the tenant's choice. The gRPC door skips the predicate only for the system scope (the nodes register themselves there); the WASM host applies it to every guest. Where each door gets "this node" from: the gRPC door from the node configuration (NodeConfig.id and advertised_grpc_address), the WASM host from the same node configuration, whose id and advertised address are carried by WasmConfig (node_id, node_grpc_address) into HostFunctions at instantiation, also for pooled instances. Without a node identity (no node configuration) both doors fail closed: the gRPC door answers FailedPrecondition and the WASM host refuses the row. The node table readers (NodeRegistry: list, lookup, the address-duplicate scan of a registration, the database sync, the heartbeat refresh and the namespace lookup; the node-id resolution of the service locator and the gRPC transport client, which return an address to dial; the heartbeat election ring and stale scan of the node; and the scheduler's CapacityTracker, list and single lookup) accept only rows of the system tenant (the empty tenant), through the one predicate plexspaces_actor::object_registrations::is_system_node_row, so a NODE row that reached the registry under a tenant is never read as a node, elected as a recorder, recorded against, or counted as capacity. Rejected alternatives: two predicates, one per door (the doors could disagree, for example on a node with authentication off), and overwriting both fields of an APPLICATION row without the check (it hides a malformed request instead of refusing it). The dashboard never dials the address of an application row (see Services).

Thin node registration

A /ws client proposes a name, never an identity. The node builds the id thin:{tenant}:{client id} from the tenant of the verified claims (a ULID stands in for an empty client id), and the ack (assigned_node_id) carries it, so a client learns its id from the ack. The node ignores the client's node_address (the stored address is always ws://{id}, and it is never dialed) and removes the client's cluster capability, which would choose the namespace the row is stored under. The session registry is keyed by the same id, so two tenants cannot collide on a client id.

Three rules enforce it. (1) NodeRegistry::register_node, unregister_node and send_heartbeat require that the context owns the node id: the system scope (is_system_admin) owns every id; a tenant owns only ids with its own thin:{tenant}: prefix, and a tenant registers thin nodes only. (2) No full node takes an id that starts with thin:, so a thin id never names a full node. (3) The object registry (scope_registration) refuses a NODE row unless the context is the system scope; NodeRegistry always persists through the system context, so the cluster's own nodes are unaffected. The by-id readers lookup_node_in_object_registry and refresh_node_heartbeat_in_registry also accept only system-tenant rows. The NodeService membership RPCs (RegisterNodes, UnregisterNode, SendHeartbeat, DisconnectNodes) run in the system scope, as ConnectNodes does: the method policy admits them to the system scope only (SystemOnly), and a node without authentication does not apply the policy, so any caller reaches them there, as it reaches every other method.

Attacks closed. A tenant member registered over /ws with node_address set to a host of its choice: the dashboard and the actor-service channel dial the address of a node row with delegated metadata, so the row redirected that traffic. A tenant member registered the id of a real node: the row was overwritten and, when the session closed, unregister_node removed the real node and marked every tenant's objects on it dead. With authentication off the tenant comes from the x-tenant-id header; it is validated with validate_tenant_name(name, false) (an empty header is allowed and gives thin::{client}), and an invalid value is a 400 at the upgrade, so a tenant name cannot contain : and forge another tenant's id prefix.

Validation of the client id. The client part is checked with plexspaces_common::validate_node_client_id, the scope-name rule ([A-Za-z0-9._-], at most 128 bytes, not only dots; empty means a ULID). :, @, control characters and NUL are outside it. A bad id gets a WsNodeRegisterAck with success=false and an error_message (invalid node_id: ..., never echoing the id), then a close frame (1008); no session or node row exists afterwards.

Atomic registration and ordering. WsRegistry::try_register claims the id with one DashMap entry operation, so of several concurrent upgrades with the same id exactly one is acked and the rest receive the duplicate error (code 9). Each session has a session_id (a ULID) and WsRegistry::unregister(node_id, session_id) removes only the entry of that session. One task per session registers the node in the NodeRegistry, waits for the session to end and unregisters it, so the two never run out of order; close_session cancels the session's pending asks, awaits that task, and only then removes the session from the WsRegistry, so a reconnect with the same id cannot register before the old unregistration finished and cannot be unregistered by it. Nothing but the closing session removes its own WsRegistry entry. The stale-session reaper (scan_and_reap_stale) only signals the session (SessionLifecycle, a watch channel on WsSession); the session loop then returns CloseReason::Stale, the handler sends close frame 1008 ("stale session") and runs the ordered close above. A reconnect therefore cannot be unregistered by the old session's close. Reconnects: WsSession::user_id holds the authenticated sub (none with authentication off); WsRegistry::register_or_replace lets the SAME user replace a live session (signal Replaced, close frame 1008 "session replaced", wait up to 30 seconds until the old session has unregistered, then register), refuses a different user and, with authentication off, always refuses. Bounds: each NodeRegistry register or unregister call of a thin session takes at most 10 seconds and ThinMembership::end at most 25 (an abort after that), each timeout counted in plexspaces_ws_thin_node_registry_timeouts_total{operation} and logged as a warning, and a ThinMembership dropped before it reached its wait aborts its task, so a stuck registry cannot hold a session or its id forever. Rejected alternatives: the reaper removing the entry (a reconnect could then be unregistered by the old close), a CancellationToken (not a dependency of the node crate; a watch channel carries the reason as well), and letting any user of the tenant replace a session (a second user could take over the first one's asks).

Delivery to a thin node. WsActorTransportClient (the one transport every actor tell and ask to a remote node id passes) compares the tenant of the delegated call (x-tenant-id of the forwarded context) with the tenant of the session it routes through; another tenant, a missing tenant and a missing call kind are PermissionDenied, and only the node itself (an internal call, system scope) may deliver across tenants. The check applies only on a node that authenticates (the transport is built with the node's frozen auth_disabled): with authentication off every caller is one trust domain and the session's tenant is an unchecked x-tenant-id header (possibly empty) that need not equal the no-auth tenant of a caller, so nothing is refused. Rejected alternative: comparing tenants with authentication off too (every ask to a thin node of a development setup would be refused).

Why unregister_node_impl may mark objects dead by node id in the system scope: admit_tenant_registration stamps every row a tenant registers with the registering node's own id, and register_node/unregister_node require that the context owns the node id (thin:{tenant}: prefix), so a tenant cannot make a session's close mark another tenant's or another node's objects dead.

Liveness. A thin node's liveness is its socket (the WsRegistry); its registry row keeps the heartbeat of its registration (send_heartbeat_impl does not write it, for throughput). The stale-heartbeat scan therefore skips rows whose address is ws://, so a long-lived session is never recorded against and marked DEAD, which would cascade to its objects.

Rejected alternatives: keeping the client id and refusing duplicates only against the session registry (it knows nothing of full nodes); a lookup of the id in the node table at every upgrade to refuse non-thin nodes (a database read per connection at 100,000 connections; the reserved prefix makes the lookup unnecessary); a separate thin-node table (a second membership store).

Auth layer marker

AuthLayerApplied (plexspaces_common) is inserted into the extensions of every request by the layer that stamps the identity headers: node::grpc_auth (gRPC), node::blob_http_router::authenticate (blob REST), node::http_gateway::scoped_grpc_request (the HTTP-to-gRPC bridges) and the dashboard's gRPC bridge; a request without it skipped the layer and is not authenticated, so every reader fails closed. One marker per request tells each service whether the layer authenticated it, so no service needs a frozen switch of its own. Rejected alternatives: an auth_enabled constructor flag per service (each copy could be wired wrongly) and an identity header (a client could forge it unless every layer strips it; an extension cannot cross the wire). AuthLayerApplied::new stays pub because the services of other crates stamp it in tests and a pub(crate) constructor cannot be called across crates; a test-support feature gate was rejected (the layers need the constructor in production, so it would need a second, ungated one). The tenant of MetricsService.RecordMetric and of the typed Record* RPCs is read from the request context built under the marker, like GetMetrics: a request without the marker is Unauthenticated, and under authentication a request without a tenant is refused instead of being counted under the no-auth tenant.

Download header. A downloaded blob's Content-Disposition is built by one function, plexspaces_common::attachment_content_disposition (RFC 6266 / RFC 5987), for the blob REST server and the dashboard download alike: the name (a blob name or, in the dashboard, the request's blob_id) is reduced to its last path segment, cut to 200 characters (MAX_FILE_NAME_CHARS, so the header stays bounded), and sent as a sanitised ASCII filename (every other character becomes _) plus a percent-encoded filename*=UTF-8''..., so a quote, CR/LF or non-ASCII byte can neither break the header nor inject another. Rejected alternative: interpolating the name into a quoted string.

Apps-directory loader health. The wasm-apps-loader readiness checker reports only the COUNT of applications not running (N application(s) not running), or the apps directory loader did not complete when the loader run itself failed; the tenant and application names and the causes are in the node log. The probe may be unauthenticated and is read by every tenant. Rejected alternatives: a capped list of names (still leaks other tenants' names) and the full list with causes.

The main HTTP router declares the same for its routes: the test table of http_routes::policy_gate gives every GatedRoute and OpenRoute an HttpNs (Grpc(rows): the route reaches the service through the in-process gRPC bridge and the named rows are Authorized; RowFilter: the response is filtered by the caller's grant; Exempt(reason)), and the tests compare it with the gRPC table.

scripts/hygiene-check.sh --check new-without-auth lists every context built without the caller's grant (new_without_auth, system_admin(), RequestContext::internal()) in the non-test code of the services, dashboard, blob, node, scheduler, wasm-runtime, application, actor, channel, workflow, keyvalue and cli crates (a conformance.rs harness counts as test code), unless the statement carries a // system-scope: <reason> comment (an operation outside any tenant: sweeps, membership, startup) or a // tenant-source: <where the tenant comes from> comment (a context scoped from a stored row, a verified claim or the application's own deploy scope). The marker counts only inside a // comment and only with a reason. The comment covers the one statement it precedes (it ends where that statement's brackets close, so a call on a later line of a multi-line statement is covered and the next statement is not); a #[cfg(test)] mod, fn or impl is test code. The comment is the allow-list; there is no path allow-list. Rejected alternative: a path allow-list (it silently covers new code in an allowed file).

make hygiene runs scripts/hygiene-check.sh --self-test (the checks against fixture sources: positive, negative, comma continuation, match arm, #[cfg(test)] fn, and the gate itself) and then --blocking: the run fails on new-without-auth, spdx, allow-no-reason, eprintln and doc-links, and only prints the counts of fn-length, test-time, bare-is-err and removed-names. A check moves into BLOCKING_CHECKS when its backlog clears, so the gate tightens one check at a time. The target is part of the all chain (proto build build-examples test test-examples hygiene) and of the ci chain (proto fmt-check lint test check-coverage hygiene). Rejected alternative: blocking on everything at once (the open backlogs would keep the gate red from the first run).

Application service authorization

ApplicationServiceImpl (deploy, undeploy, start, list, status) has no admin check of its own, by design: authorization is enforced at its edges, the gRPC policy table (grpc_auth) and, for HTTP, http_routes::policy_gate. Rejected alternative: a check inside the service, which would need a "system caller" notion because the boot loader and tests call it with system contexts. Two tests keep the edges honest: only_the_gated_edges_use_the_application_service fails when a source file under crates/*/src outside the service starts using the type or the generated gRPC server without being listed (with its gate) in APPLICATION_SERVICE_USERS, and the_application_service_has_no_unlisted_entry_point fails when the service gains a pub or pub(crate) method that is not listed. Limits: tests/, examples/ and sdks/ are not scanned and macro-generated methods are not seen.

Scope headers fail closed. Every HTTP-to-gRPC bridge (actors, deploy, nodes, WebSocket frames) builds x-tenant-id and x-namespace with one function (http_gateway::scoped_grpc_request). A value that is not a valid header (non-ASCII or control characters) is an InvalidArgument error (HTTP 400, or an error frame), never an empty or missing header, because the services read a missing tenant as unscoped. A /ws upgrade with authentication off and an invalid x-tenant-id is refused with 400. The same function forwards the caller's verified identity (user id, admin flag, x-allowed-namespaces) with forward_identity; see Namespace authorization.

Static files. /apps/:app_id/ (index.html) and /apps/:app_id/*file share one file-serving path: canonicalize (async), then prefix-check against the application's directory, so a symlink or .. that resolves outside it never serves the file. Such a path answers the same 404 with the same body as a missing one (the escape is logged at debug, without the requested path). Reason: a distinct status (403 for an existing path outside the directory) would let a caller probe which files exist on the node outside the application directory. Rejected alternative: 403 for an escape.

When authentication is disabled (e.g. PLEXSPACES_DISABLE_AUTH=1), the gateway may use a local default tenant for testing. Client-supplied tenant headers are not part of the public contract.

The middleware sets (from JWT only when auth on):

  • x-tenant-id: From JWT tenant_id claim
  • x-user-id: From JWT sub claim
  • x-user-roles: Comma-separated from JWT roles
  • x-user-groups: Comma-separated from JWT groups
  • x-admin: From JWT is_admin claim. It marks an admin of the caller's own tenant, never a cross-tenant operator: it does not widen user, tenant, token, dashboard or registry queries beyond the tenant claim. Without auth the header is ignored.

Admin scope. There is exactly one cross-tenant scope, RequestContext::system_admin() (is_system_admin(): admin flag with no tenant). It can be built only by node-internal code or granted by a node running without authentication (for example the dashboard and the /api/v1/auth/* routes then act as the system scope because there is no tenant identity to protect). A tenant admin can list, update, mint tokens for and revoke tokens of its own tenant only; naming another tenant is PERMISSION_DENIED (gRPC) / 403 (HTTP), and a record of another tenant is reported as not found. Minted API tokens never carry more privilege than the user they are minted for. There is no platform-operator role for cross-tenant administration with auth enabled. Isolation errors name only the caller's own tenant: they never reveal the id of the tenant that owns the actor or that a spec asked for. The dashboard's node views (GetNodes, GetNodeDashboard) show an authenticated caller only the node's resource use (CPU, memory, uptime): live-actor, message, shard and connected-node counters span every tenant and the cluster name is topology, so they are zeroed outside the system scope.

Token Validation

JWT tokens are validated against:

  1. Algorithm: Pinned to the configured key type (ES256). The token’s alg header is not trusted (algorithm confusion prevention).
  2. Signature: Verified with the EC public key.
  3. Expiration: exp is checked (tokens are rejected when expired).
  4. Issuer and audience: not checked. The validator disables audience validation and does not check iss, so the issuer and allowed_audiences settings have no effect (see the config block above). Isolation rests on the signature (only tokens signed with the node's key are accepted) and on the tenant_id claim.
  5. tenant_id: Must be non-empty in claims for request context creation.

Node-to-node tokens (the node token used over TLS between nodes) are separate: they are validated with a fixed issuer and audience.

Tenant Rules

  • Every request belongs to a non-empty tenant. With authentication enabled it comes from the verified credentials; with authentication disabled it is the constant NO_AUTH_TENANT_ID (default) unless the request names one. Scoped storage (journal, channels, key-value, blobs, task router) and the actor factory reject an empty tenant.
  • The one empty-tenant context is RequestContext::system_admin() (is_system_admin()): an admin context with no tenant or namespace, for node-internal cross-tenant scans (heartbeat sweeps, scheduler, node-death cascades, node and object registry discovery). It is the only context that crosses tenants in the object registry: the x-admin header / is_admin claim marks an admin of the caller's tenant and never widens a registry query beyond it (a tenant-A admin cannot discover tenant B). A request-derived context always carries a tenant (the tenant claim is required with auth, the no-auth tenant is substituted otherwise), so it can never be the system scope. Never build it from client input.
  • A WASM guest cannot choose its tenant or namespace: the host ignores the context argument of its host calls and uses the tenant and namespace the actor was deployed under. A WASM instance without a deploy-time tenant is not created.
  • An application's tenant is the registering caller's (ApplicationManager::register(ctx, app)). A native application whose spec names no tenant is stamped with the tenant of the caller's RequestContext (its actors are spawned in that tenant); a spec that names the SAME tenant is accepted; a spec that names ANOTHER tenant is refused with an ApplicationError::ConfigError that names both tenants, and nothing is registered. It is never silently overridden (that would hide the mismatch) and never silently honoured (a caller could deploy into a tenant it is not). The same rule as registering a virtual actor, and the same refusal as spawning an actor for another tenant above. A WASM application takes the tenant through set_tenant_namespace as before.
  • File-copy deploys take their tenant from tenant_id in app-config.toml (default with authentication disabled); with authentication enabled a file deploy without one is rejected.

Example JWT Token (Claims)

{
  "sub": "user-123",
  "tenant_id": "tenant-456",
  "roles": ["admin", "user"],
  "groups": ["ops"],
  "is_admin": true,
  "namespaces": ["staging", "prod"],
  "exp": 1735689600,
  "iat": 1735603200
}

Auth Error Messages and Local Testing

When a request is rejected due to missing or invalid auth, the response body includes an actionable hint:

  • HTTP 401 (Unauthorized): Missing or invalid JWT. Message includes guidance to provide a valid Authorization: Bearer <token> and, for local testing, to set PLEXSPACES_DISABLE_AUTH=1.
  • HTTP 503 (Service Unavailable): Auth is enabled but no JWT key is configured (or an API token's revocation status cannot be checked). The gateway message reads "Auth enabled but JWT key not configured. Set PLEXSPACES_JWT_PRIVATE_KEY_FILE. For local testing, set PLEXSPACES_DISABLE_AUTH=1."; the other 503 bodies (http_jwt, actor routes) say only "Auth enabled but JWT key not configured".

Local testing without auth:

export PLEXSPACES_DISABLE_AUTH=1
cargo run -p plexspaces-cli -- start --node-id test-node --listen-addr 0.0.0.0:8093

Then call HTTP or gRPC without a token (when the node is configured to allow it).

OAuth/OIDC Authentication

Overview

PlexSpaces supports OAuth 2.0 / OpenID Connect (OIDC) for user authentication. Users authenticate via any OIDC-compliant provider (Google, Okta, Auth0, Keycloak, etc.), and PlexSpaces issues a short-lived session JWT stored in an HttpOnly cookie.

How It Works

Browser                PlexSpaces Node              OIDC Provider
  │                        │                            │
  │  GET /api/v1/auth/oidc/login                        │
  ├───────────────────────►│                            │
  │                        │  302 Redirect to provider  │
  │◄───────────────────────┤                            │
  │                        │                            │
  │  (User authenticates at provider)                   │
  │────────────────────────────────────────────────────►│
  │                        │                            │
  │  GET /api/v1/auth/oidc/callback?code=...&state=...  │
  │◄────────────────────────────────────────────────────│
  ├───────────────────────►│                            │
  │                        │  Exchange code for tokens  │
  │                        ├───────────────────────────►│
  │                        │◄───────────────────────────│
  │                        │                            │
  │                        │  Extract claims, upsert user/tenant
  │                        │  Issue session JWT (15m TTL)
  │  Set-Cookie: psx_session=<JWT>; HttpOnly; Secure    │
  │◄───────────────────────┤                            │
  │  302 → /dashboard      │                            │

Configuration

security:
  disable_auth: false
  oidc:
    enabled: true
    discovery_url: "https://accounts.google.com/.well-known/openid-configuration"
    client_id: "YOUR_CLIENT_ID"          # or PLEXSPACES_OIDC_CLIENT_ID env var
    client_secret: "YOUR_SECRET"         # or PLEXSPACES_OIDC_CLIENT_SECRET env var
    redirect_uri: "http://localhost:8091/api/v1/auth/oidc/callback"
    scopes: ["openid", "email", "profile"]
    tenant_claim: "hd"                   # OIDC claim that maps to tenant slug
    admin_groups: ["platform-admins"]    # OIDC groups that grant admin privilege
    default_tenant_id: "acme"            # Fallback when tenant_claim is absent (not "default": reserved when auth is on)
    group_namespaces:                    # user group (stored on the user, not an IdP claim) -> namespaces (names or "*"); optional
      platform-admins: ["*"]
      dev-team: ["staging", "dev"]
  jwt:
    enable_jwt: true
    private_key_file: "./certs/jwt-es256.pem"  # or PLEXSPACES_JWT_PRIVATE_KEY_FILE / PLEXSPACES_JWT_PRIVATE_KEY
    issuer: "plexspaces"                 # Not enforced on validation

Environment Variables

Variable Description
PLEXSPACES_OIDC_CLIENT_ID OAuth client ID (overrides config)
PLEXSPACES_OIDC_CLIENT_SECRET OAuth client secret (overrides config)
PLEXSPACES_JWT_PRIVATE_KEY Inline ES256 PEM private key (preferred)
PLEXSPACES_JWT_PRIVATE_KEY_FILE Path to ES256 PEM file (generated if missing, only on a node without seed nodes)
PLEXSPACES_DISABLE_AUTH Set to 1 to disable all auth (dev only; on a non-loopback bind it also needs PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=1)

Provider Examples

Google Workspace:

oidc:
  discovery_url: "https://accounts.google.com/.well-known/openid-configuration"
  tenant_claim: "hd"    # Google hosted domain → tenant slug
  scopes: ["openid", "email", "profile"]

Setup: Google Cloud Console → Create OAuth 2.0 Client ID → Add redirect URI.

Okta:

oidc:
  discovery_url: "https://YOUR_ORG.okta.com/.well-known/openid-configuration"
  tenant_claim: "org"
  scopes: ["openid", "email", "profile", "groups"]

Auth0:

oidc:
  discovery_url: "https://YOUR_DOMAIN.auth0.com/.well-known/openid-configuration"
  tenant_claim: "org_id"
  scopes: ["openid", "email", "profile"]

Keycloak (self-hosted):

oidc:
  discovery_url: "http://localhost:8180/realms/plexspaces/.well-known/openid-configuration"
  tenant_claim: "org"
  scopes: ["openid", "email", "profile"]

User → Tenant Mapping

When a user authenticates via OIDC:

  1. The tenant_claim (e.g., hd, org, org_id) is extracted from the ID token claims.
  2. get_or_create_tenant(slug) creates the tenant if it doesn't exist (idempotent upsert).
  3. get_or_create_user(email, tenant_id) creates or updates the user record.
  4. A session JWT is issued with tenant_id, sub, roles, is_admin, and stored as an HttpOnly cookie.

If the tenant_claim is absent from the ID token, the default_tenant_id from config is used.

Admin Detection

Users are granted admin privileges when their stored user groups (set with SetUserGrants, not the IdP's groups claim) include any group listed in admin_groups:

oidc:
  admin_groups: ["platform-admins", "infra-team"]

The is_admin flag is stored on the user record and included in all session JWTs.

Namespace Mapping

The namespaces claim of a session JWT is the user record's allowed_namespaces united with the namespaces that group_namespaces maps from each of the user's stored groups (the groups of the user record, not the IdP's groups claim; provision_user stores none from the IdP), sorted, duplicates removed. It is computed at every login and not stored: changing a mapping or a user's groups takes effect at the next login, and the user record stays the only persisted grant. A user with no allowed_namespaces and no mapped group gets a token that reaches no namespace. Entries are validated and normalized (sorted, deduplicated, "*" subsumes names) when the release file is loaded; an invalid entry stops startup with an error naming the first bad group in sorted order. Syncing the user's groups from an IdP groups claim is not provided.

Changing a user's grants (SetUserGrants)

A user's roles, groups and allowed_namespaces are privileges (they feed admin_groups, group_namespaces and the token cap), so they are changed only by the admin RPC UserService.SetUserGrants, policy TenantAdmin; UpdateUser (member-level, so an account owner can edit itself) does not carry them and edits only display name, avatar and, for admins, the admin flag. GetOrCreateByEmail can set them for a new user but is SystemOnly: account provisioning runs in process during OIDC login and no remote caller reaches it.

SetUserGrants is reachable over gRPC only. The proto http annotation declares PUT /api/v1/auth/users/{user_id}/grants, but the node mounts no such REST route (the auth router serves GET users and tenants, API tokens, JWKS and OIDC login).

  • Replace semantics: each list replaces the stored one; an empty list clears it. Namespaces are validated (validate_namespace_entries) and stored normalized.
  • Who may call: an admin of the user's tenant (the system scope may name any tenant). A user of another tenant is reported as NotFound; a non-admin, the account owner included, gets PermissionDenied.
  • Caps: every namespace must be covered by the caller (cap_requested_namespaces with the user's tenant), so a grant never exceeds the granting side and never crosses tenants; an over-grant is PermissionDenied. A tenant admin holds every namespace of its own tenant, so within a tenant the cap only bites for a granter that is not a full tenant admin or sits in another tenant. Roles and groups carry no privilege in code beyond the OIDC mappings above and are validated for shape only (at most 64 entries, 128 bytes each, no empty entry, no , or control character because the lists travel comma-joined in the x-user-roles / x-user-groups headers).
  • Token revocation: after the update the user's active API tokens whose namespaces are no longer covered by the user's grant (a "*" covers all; an admin user covers all) are revoked through the revocation cache, immediately on this node and within the cache TTL (5 s) on others. This runs on every call, not only when the set shrank, so a retry after a failed revocation finishes the job; tokens without namespaces are left alone. The response reports revoked_api_tokens.
  • Session JWTs are not revoked: a login token already issued keeps its namespaces until it expires (one hour); only API tokens are revocable. An API token minted without namespaces carries none and reaches none, since the layers enforce the claim: mint it (CLI, script or API) with namespaces.

API Tokens (Programmatic Access)

Overview

API tokens provide long-lived programmatic access for CI/CD pipelines, scripts, and SDKs. They follow the GitHub Personal Access Token (PAT) pattern:

  • Format: psx_<64 hex chars> (32 random bytes, hex-encoded)
  • Storage: Only the SHA-256 hash is stored in the database
  • Display: Plaintext is shown exactly once at creation time
  • Validation: Token is hashed → looked up in api_tokens table → checked for revocation/expiry

Creating API Tokens

Via Dashboard: Visit /dashboard/tokens → Click "Create Token" → Set name, scopes, and TTL → Copy the plaintext token immediately.

Via API (with session JWT or existing API token):

curl -X POST http://localhost:8091/api/v1/auth/tokens \
  -H "Authorization: Bearer <session-jwt-or-psx-token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "ci-pipeline", "scopes": ["read", "write"], "namespaces": ["staging"], "ttl_seconds": 7776000}'

namespaces follows the minter cap: omitted means the token reaches no namespace, and an entry the caller (or the target user, when user_id names another user) does not hold is a 403. An invalid entry is a 400.

Response:

{
  "token": {
    "token_id": "01J...",
    "name": "ci-pipeline",
    "prefix": "psx_abc1",
    "scopes": ["read", "write"]
  },
  "plaintext": "psx_a1b2c3d4e5f6..."
}

ttl_seconds defaults to 90 days and must lie between 1 second and 365 days; any other value is a 400 (a larger or negative value would overflow the expiry arithmetic or mint an effectively immortal token).

Using API Tokens

Pass the token as a Bearer token in the Authorization header:

# List users
curl -H "Authorization: Bearer psx_a1b2c3d4e5f6..." \
  http://localhost:8091/api/v1/auth/users

# Send a message to an actor
curl -H "Authorization: Bearer psx_a1b2c3d4e5f6..." \
  -X POST http://localhost:8091/api/v1/actors/my-actor/messages \
  -H "Content-Type: application/json" \
  -d '{"payload": "hello"}'

Token Lifecycle

Operation Endpoint Method
Create /api/v1/auth/tokens POST
List (own tokens) /api/v1/auth/tokens GET
Revoke /api/v1/auth/tokens/:token_id DELETE

Security Properties

  • is_admin snapshotted at creation: The token carries the admin status of the user at creation time. If a user is later demoted, existing tokens retain their original privilege level (revoke and re-create to reflect new permissions).
  • SHA-256 stored, plaintext never persisted: Even if the database is compromised, tokens cannot be recovered.
  • last_used_at tracking: Updated asynchronously on each use for audit purposes.
  • Scoped: Tokens carry explicit scopes (e.g., read, write) checked by handlers.
  • Namespaces snapshotted at creation: the capped namespace list is stored in api_tokens.namespaces_json and signed into the token. User grants live in users.allowed_namespaces_json. Both columns were added to the original 016_users / 018_api_tokens schema files in place (no migration), so an existing database must be recreated.

Unified Auth Design

All three authentication methods produce the same downstream JwtClaims interface:

┌──────────────────────────────────────────────────────────┐
│                    Incoming Request                        │
│                                                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────┐  │
│  │ Session JWT │  │ psx_ Token  │  │ Node token      │  │
│  │ (cookie:HTTP)│ │ (Bearer)    │  │ (gRPC peers)    │  │
│  └──────┬──────┘  └──────┬──────┘  └────────┬────────┘  │
│         │                │                   │           │
│         ▼                ▼                   ▼           │
│  ┌──────────────────────────────────────────────────┐    │
│  │              JwtClaims (uniform)                  │    │
│  │  sub, tenant_id, roles, groups, is_admin         │    │
│  └──────────────────────────────────────────────────┘    │
│         │                                                │
│         ▼                                                │
│  ┌──────────────────────────────────────────────────┐    │
│  │           Downstream Handlers                     │    │
│  │  (auth-method-agnostic, same code path)           │    │
│  └──────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────┘

Testing Authentication

Development Mode (Auth Disabled)

For local development, disable auth entirely:

export PLEXSPACES_DISABLE_AUTH=1
cargo run -p plexspaces-cli -- start --node-id dev-node --listen-addr 127.0.0.1:8091

Disabled authentication is accepted on a loopback address only. To listen on another address (0.0.0.0 in a container or on a development LAN) also set PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=1 (or allow_unauthenticated_public_bind: true under runtime.security); the node then starts with a warning in the log. Without it the node refuses to start with an error naming the setting. See Node-to-Node Authentication.

All endpoints return data as superadmin — no login or token needed:

curl http://localhost:8091/api/v1/auth/users
curl http://localhost:8091/api/v1/auth/tenants
curl -X POST http://localhost:8091/api/v1/auth/tokens \
  -H "Content-Type: application/json" \
  -d '{"name": "test-token"}'

Testing OIDC Login Flow

  1. Configure OIDC provider credentials (see Provider Examples above).
  2. Start the node with auth enabled.
  3. Open browser: http://localhost:8091/api/v1/auth/oidc/login
  4. Authenticate with your provider.
  5. Verify redirect to /dashboard with session cookie set.

Testing API Tokens End-to-End

# 1. Create a token (requires active session or auth disabled)
TOKEN_RESPONSE=$(curl -s -X POST http://localhost:8091/api/v1/auth/tokens \
  -H "Content-Type: application/json" \
  -d '{"name": "e2e-test", "scopes": ["read","write"], "ttl_seconds": 3600}')

PLAINTEXT=$(echo "$TOKEN_RESPONSE" | jq -r '.plaintext')
TOKEN_ID=$(echo "$TOKEN_RESPONSE" | jq -r '.token.token_id')

# 2. Use the token (the token list is open to any authenticated user)
curl -H "Authorization: Bearer $PLAINTEXT" \
  http://localhost:8091/api/v1/auth/tokens

# 3. List tokens
curl -H "Authorization: Bearer $PLAINTEXT" \
  http://localhost:8091/api/v1/auth/tokens

# 4. Revoke the token
curl -H "Authorization: Bearer $PLAINTEXT" \
  -X DELETE http://localhost:8091/api/v1/auth/tokens/$TOKEN_ID

# 5. Verify revoked token is rejected (should return 401)
curl -H "Authorization: Bearer $PLAINTEXT" \
  http://localhost:8091/api/v1/auth/tokens

Testing Access Control

# Non-admin user: the user list is a TenantAdmin method (same as gRPC ListUsers)
curl -H "Authorization: Bearer $USER_TOKEN" \
  http://localhost:8091/api/v1/auth/users
# → 403

# Tenant admin: sees the users of its own tenant only
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  http://localhost:8091/api/v1/auth/users
# → Returns the users of the token holder's tenant (paginated)

# Tenant admin asking for another tenant (refused):
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  "http://localhost:8091/api/v1/auth/users?tenant_id=other-tenant"
# → 403 (only the system scope of a node without authentication may name another tenant)

Dashboard UI Testing

With the node running, visit:

  • http://localhost:8091/dashboard/tokens — Create, view, and revoke API tokens
  • http://localhost:8091/dashboard/users — Browse users and tenants with pagination

Tenant Isolation

RequestContext

All operations require a RequestContext:

use plexspaces_actor::RequestContext;

// Create context from request
let ctx = RequestContext::new("tenant-123".to_string())
    .with_namespace("production".to_string())
    .with_user_id("user-456".to_string());

// Pass to repository/service
let result = repository.get(&ctx, "resource-id").await?;

Repository Pattern

All repository methods require RequestContext:

#[async_trait]
pub trait BlobRepository {
    async fn get(
        &self,
        ctx: &RequestContext,  // Required for tenant isolation
        blob_id: &str,
    ) -> BlobResult<Option<BlobMetadata>>;
}

Service Layer

All service methods require RequestContext:

impl BlobService {
    pub async fn upload_blob(
        &self,
        ctx: &RequestContext,  // Required for tenant isolation
        name: &str,
        data: Vec<u8>,
        // ...
    ) -> BlobResult<BlobMetadata>;
}

SQL Query Patterns

All SQL queries automatically filter by tenant_id and namespace:

-- ✅ Correct: Filters by tenant_id and namespace
SELECT * FROM blob_metadata
WHERE blob_id = $1 AND tenant_id = $2 AND namespace = $3

-- ✅ Correct: Admin/internal context with empty namespace (skips namespace filter)
SELECT * FROM blob_metadata
WHERE blob_id = $1 AND tenant_id = $2
-- Note: Namespace filter skipped when ctx.should_skip_namespace_filter() is true

-- ❌ Wrong: Missing tenant_id filter
SELECT * FROM blob_metadata
WHERE blob_id = $1

Namespace Filtering

Repository lookup methods (read-only operations) support conditional namespace filtering:

  • Normal Contexts: Always filter by both tenant_id and namespace
  • Admin/Internal Contexts with Empty Namespace: Skip namespace filtering to allow cross-namespace queries
// Check if namespace filtering should be skipped
if ctx.should_skip_namespace_filter() {
    // Admin or internal context with empty namespace
    // Query without namespace filter (cross-namespace query)
    sqlx::query("SELECT * FROM kv_store WHERE tenant_id = ? AND key = ?")
        .bind(ctx.tenant_id())
        .bind(key)
        .fetch_optional(&pool)
        .await?
} else {
    // Normal context - filter by namespace
    sqlx::query("SELECT * FROM kv_store WHERE tenant_id = ? AND namespace = ? AND key = ?")
        .bind(ctx.tenant_id())
        .bind(ctx.namespace())
        .bind(key)
        .fetch_optional(&pool)
        .await?
}

Important: Write operations (put, delete, update) always filter by namespace to maintain data isolation.

Tenant Validation

Services validate tenant_id matches:

// Repository validates tenant_id matches context
if metadata.tenant_id != ctx.tenant_id() {
    return Err(BlobError::InvalidInput(
        "Metadata tenant_id does not match context"
    ));
}

WASM Guests: Scope, Sender and Outbound HTTP

A WASM guest does not choose the tenant or the default namespace of a host call. Every host call (key-value, blob, locks, process groups, registry, channels, durability) takes its tenant and default namespace from the application's deployment; the tenant-id/namespace of a WIT context and of registry request messages are ignored, and a registry register naming another namespace is refused. The namespace is the application's and is shared by all of its actors. Messages a guest sends always carry the actor's own id as sender; a sender id in a guest request is overwritten. This is scoping, not authorization: the host builds its context with new_without_auth, which is unrestricted, so a guest that names an actor or a group member by a canonical id of another namespace of the same tenant is not refused by the context (see Namespaces confine API callers, not deployed code).

host-http.http-fetch(link-name, ...) goes through the node-wide runtime.service_links, which carry operator credentials (bearer tokens, API keys). An actor may only use the links its application declares in ApplicationSpec.required_service_links; every other link, and every link when none is declared, is refused before any network call (typed error service link '<name>' is not declared by this application).

A link can also be restricted to tenants: allowed_tenants on the link (runtime.service_links[].allowed_tenants, a list of tenant ids; empty means every tenant) is checked twice. Deploy is rejected (FAILED_PRECONDITION) when the application declares a link whose list excludes its tenant, and http-fetch re-checks on every call with the tenant the application was deployed under (never one named by the guest), so a restriction added later also applies to applications that are already running (typed error service link '<name>' is not available to this tenant).

The outbound client (plexspaces-http-client, see its README) treats the guest's request as untrusted:

  • Headers: the guest cannot set Host, Authorization (denied even when the link sets no credentials of its own), Proxy-Authorization, Connection, Transfer-Encoding, Content-Length, Forwarded, X-Forwarded-*, X-Real-IP, X-Original-URL, X-Rewrite-URL, X-HTTP-Method-Override, Via, Expect, hop-by-hop headers, or any header the link itself sets (API keys). This is a denylist rather than an allowlist so guests keep Content-Type, Accept and their own x-* headers. A link header (default_headers, API key, bearer token) that is not a valid HTTP header makes the node fail at startup instead of silently dropping a credential.
  • Paths: path_and_query is limited to 8 KiB, may not contain control characters, a .. segment (also written percent-encoded or with \), an encoded ., /, \ or % in the path (%2e, %2f, %5c, %25, which a peer or proxy might decode a second time) or a ; path parameter. The query string is not restricted beyond control characters.
  • Redirects are followed only within the link's own origin and base path (a link whose base URL is https://api.example.com/v1 never follows a redirect to /admin); anything else is returned to the guest as the 3xx response. Request and response bodies are capped at 16 MiB.
  • Timeouts: when the link's policy sets none, a request times out after 30 s and a connection attempt after 10 s.
  • Circuit breaker: one breaker per link is shared by every tenant using it, so a guest must not be able to open or wedge it. Requests are validated before the breaker is consulted (a malformed request neither counts as a failure nor uses a half-open probe slot); a 4xx answer other than 408 and 429 does not count as a link failure (the peer answered; 5xx, 408 and 429 do); and every admission ends in a verdict or gives its probe slot back, including when the call is cancelled.
  • The tracing span logs only the first 128 characters of the path, never the query string.

There is no per-actor "capabilities" allow-list: the guest's reach is the set of host interfaces the host links plus the declared links.

forward is a plain send: the receiver sees the forwarding actor as the sender. A guest cannot name another actor as the sender (the receiving side derives tenancy and routing from it), so the original-sender argument is only logged, truncated, at debug level.

Persisted Applications

Applications deployed through the API are stored under the configured runtime.wasm_apps_directory per tenant: <wasm_apps_directory>/<tenant>/<application_id>/ holding app.wasm and application-spec.toml (which records the owning tenant_id). Tenant and application id come from requests, so every path is built from validated names (plexspaces_common::validate_application_id / app_storage_dir: at most 128 bytes, [A-Za-z0-9._-], never only dots) and removal re-checks that the directory's canonical location is exactly <canonical base>/<tenant>/<application_id> before deleting anything (a symlinked directory is refused). The same validator serves HTTP deploy, gRPC deploy, undeploy and the startup loader. Consequences:

  • Undeploy removes only the caller's tenant directory; another tenant's application of the same id is untouched. An id such as ../x, an absolute path or .. is rejected with INVALID_ARGUMENT before anything is purged or deleted.
  • Undeploy falls back to the "stateless" cleanup (purge of the application's namespace state, registrations and files) only when the application manager reports the application as not found for this tenant; any other stop failure leaves the application registered and its files in place. A stateless undeploy of an application the tenant has no state, registration or directory for returns NOT_FOUND.
  • The startup loader reads <tenant>/<application>/ and rejects an entry whose application-spec.toml names a tenant_id different from its tenant directory (a spec without one takes the directory's tenant). Symlinked directories and invalid names are skipped.
  • application-spec.toml is written with a TOML serializer (never by string concatenation), so a hostile version, seed node, argument or facet value cannot inject keys such as tenant_id; a version containing control characters is rejected at deploy time.
  • An upload whose app-config.toml does not parse is a 400: it is not replaced by the default spec, which would silently drop its supervisor tree and required_service_links.
  • Layout: only <wasm_apps_directory>/<tenant>/<application_id>/ is read; a flat <wasm_apps_directory>/<application_id>/ directory is ignored and nothing converts it. Place an application in its tenant directory (mkdir -p apps/<tenant> && mv apps/<application_id> apps/<tenant>/) or redeploy it. Static files uploaded with an application live in <tenant>/<application_id>/static and the /apps/<application_id>/ URL is bound to the first tenant that registers it (another tenant deploying the same id with static files gets 409).

Local Testing

Disabling Auth for Tests

For integration tests or local development, you can disable auth:

export PLEXSPACES_DISABLE_AUTH=1
cargo run -p plexspaces-cli -- start --node-id test-node --listen-addr 127.0.0.1:8093

Or in Docker/Compose (the node listens on 0.0.0.0 inside the container, which needs the explicit opt-in):

# docker-compose.test.yml
services:
  plexspaces-node:
    environment:
      - PLEXSPACES_DISABLE_AUTH=1  # Only for integration tests / local dev
      - PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=1  # Required: non-loopback bind without auth

WARNING: Never disable auth in production!

Integration Test Setup

use plexspaces_actor::RequestContext;

#[tokio::test]
async fn test_blob_operations() {
    // Create test context
    let ctx = RequestContext::new("test-tenant".to_string())
        .with_namespace("test".to_string());
    
    // Test operations
    let metadata = blob_service.upload_blob(
        &ctx,
        "test.txt",
        b"test data".to_vec(),
        None, None, None, HashMap::new(), HashMap::new(), None
    ).await.unwrap();
    
    // Verify tenant isolation
    let data = blob_service.download_blob(&ctx, &metadata.blob_id).await.unwrap();
    assert_eq!(data, b"test data");
}

Security Test Examples

#[tokio::test]
async fn test_tenant_isolation() {
    let ctx1 = RequestContext::new("tenant-1".to_string());
    let ctx2 = RequestContext::new("tenant-2".to_string());
    
    // Upload blob for tenant-1
    let metadata = blob_service.upload_blob(
        &ctx1, "test.txt", b"data".to_vec(),
        None, None, None, HashMap::new(), HashMap::new(), None
    ).await.unwrap();
    
    // Try to access from tenant-2 (should fail)
    let result = blob_service.download_blob(&ctx2, &metadata.blob_id).await;
    assert!(result.is_err()); // Should be NotFound or AccessDenied
}

Best Practices

Secret Management

  1. Never hardcode secrets in config files
  2. Use environment variables for all secrets
  3. Validate secrets are not in config files (automatic)
  4. Rotate secrets regularly
  5. Use secret management (Vault, AWS Secrets Manager, etc.) in production

Certificate Rotation

  1. Set rotation interval in config
  2. Monitor certificate expiration
  3. Update object-registry when rotating
  4. Graceful rotation (old certs valid during transition)

Multi-Tenant Security

  1. Always use RequestContext - never bypass tenant isolation
  2. Validate tenant_id at service boundaries
  3. Filter by tenant_id in all SQL queries
  4. Audit tenant access (log tenant_id in all operations)
  5. Test tenant isolation in integration tests

Audit Logging

All operations should log tenant_id:

tracing::info!(
    tenant_id = %ctx.tenant_id(),
    namespace = %ctx.namespace(),
    blob_id = %blob_id,
    "Blob accessed"
);

Audit log (not provided). The only audit trail is the structured tracing output above. security.proto keeps AuditLogEntry, GetAuditLogsRequest and GetAuditLogsResponse as a draft contract, but no RPC serves them and nothing writes entries. A real audit log would have to be append-only and per-tenant, record who / what / when / outcome, and have a retention policy and a tenant-scoped query API; the draft fields are not a stable contract. The earlier AuthenticateService* / AuthenticateApi* drafts were deleted: tokens are signed JWTs validated by the tower layer (see User API Authentication), so there is no authentication RPC.

Security Headers (Set by Middleware from the verified JWT Only)

When auth is enabled, these headers are set only by the gateway/middleware from the validated JWT; client-sent values are overwritten or removed:

  • x-tenant-id: Tenant identifier (from JWT tenant_id)
  • x-user-id: User identifier (from JWT sub)
  • x-user-roles: Comma-separated roles (from JWT roles)
  • x-user-groups: Comma-separated groups (from JWT groups)
  • x-admin: Admin flag of the caller's own tenant (from JWT is_admin); ignored when auth is disabled
  • x-request-id: Request ID for tracing (generated)

Error Handling

Security errors should not leak information:

// ✅ Good: Generic error message
return Err(BlobError::NotFound(blob_id));

// ❌ Bad: Leaks tenant information
return Err(BlobError::InvalidInput(
    format!("Blob belongs to tenant-{} but you are tenant-{}", ...)
));

Middleware and Observability

HTTP Gateway (JWT)

When auth is enabled the HTTP gateway is default deny: its JWT middleware (http_auth_middleware) authenticates every registered route except the ones listed in PUBLIC_ROUTES (crates/node/src/http_gateway.rs), each with a reason. A route is matched by METHOD and by the pattern it was registered with (axum MatchedPath), exactly and never by path prefix, so a route or a method added later is protected until someone lists it (PublicRoute { method, pattern, reason }; a HEAD is the GET of the same route; the reasons are logged once when the gateway is assembled). On a public route the client-supplied x-tenant-id, x-admin, x-allowed-namespaces and x-user-id headers are removed before the handler runs, since no verification stamps them there. The list: /, /.well-known/jwks.json, POST /api/v1/auth/logout (a state change is not a GET, so a link or image cannot trigger it; the dashboard calls it with fetch), /api/v1/auth/oidc/login, /api/v1/auth/oidc/callback, the dashboard pages (/dashboard, /dashboard/node/:node_id, /dashboard/metrics, /dashboard/metrics/:node_id, /dashboard/application/:name, /dashboard/tenant/:tenant_id, /dashboard/users, /dashboard/tokens), the three dashboard assets under /static/ and the application content routes (/apps/:app_id, /apps/:app_id/, /apps/:app_id/*filepath). /metrics and /ws are not on it. The gateway serves no /health, /ready or /favicon.ico route; a gRPC health probe uses the gRPC health service, not this layer. A URL no HTTP route matches has no pattern. all_http_routes registers no fallback (a test checks it: the assembled routes alone answer 404), but in the real server the HTTP routes are merged with the tonic router, whose fallback answers every other URL through grpc_auth_middleware: the answer is HTTP 200 with grpc-status: 7 (an unclassified method is refused before the credential is read), identical with no token, a bad token and a valid token, so an unknown path never tells a caller whether a credential would have been accepted (test an_unknown_path_is_answered_by_the_grpc_guard_not_by_a_router_404, through seal_grpc_http_server). The no-pattern branch of http_auth_middleware is therefore not reached for an unknown URL in the real server (it stays as a fail-safe). The dashboard pages are static HTML and their data calls go through /api/ with the session cookie; the /apps/ routes stay public until per-application authentication exists, and every response of them carries Content-Security-Policy: sandbox allow-scripts and X-Content-Type-Options: nosniff (see "Deployed app pages" below). With authentication off every route is open. Decision: default deny by registered pattern. Rejected alternatives: a path-prefix allow-list plus an "everything outside /api/ passes" rule (a route outside /api/ was unauthenticated by default) and per-handler checks (one forgotten handler is an open door). A test builds the node's real routes and the dashboard router and requires every registered route to be listed or to answer 401 without a token.

  • Middleware: Validates Authorization: Bearer <token> (or, for browsers, the plexspaces_token session cookie), extracts claims, stamps the identity headers from them and sets the claims in request extensions. Downstream handlers use tenant_id from JWT only (not from path or client headers when auth is on).
  • Observability: On auth failure, the middleware logs without exposing tokens:
    • 503 (no JWT key configured): warn Auth: JWT key not configured (returning 503) with the path field
    • 401: debug Auth: JWT validation failed (401) with the path, has_bearer and has_cookie fields (debug level because an unauthenticated caller controls its frequency)
  • Credential size: the Authorization header (or, for browsers, the plexspaces_token cookie value) may be at most 8 KB on HTTP and gRPC alike (MAX_AUTHORIZATION_BYTES); larger values are rejected unparsed (401).
  • GET /metrics is authenticated like /api/* and is tenant-scoped: with auth on it needs a bearer token (a user JWT or a long-lived API token) and returns only the series labelled with the caller's tenant_id, narrowed for a token whose grant lists namespaces to the series of those namespaces plus the tenant's user series (see "metrics visibility" under Namespace authorization). The node-wide exposition mixes every tenant's series with node-level ones, so it is never served anonymously. There is no platform-operator role, so node-level series (process, topology, aggregate gRPC counters) are not reachable over HTTP with auth on; on a node with auth disabled the endpoint serves the whole registry. See Metrics for the Prometheus scrape configuration.
  • GET /api/v1/nodes lists the cluster's nodes, which is infrastructure rather than tenant data (the registry is not tenant-filtered). The gRPC method is SystemOnly, and the bridge applies the same rule: with auth on it answers 403 to every user token, admin or not; on a node with auth disabled it lists the nodes.

HTTP status of a failed gRPC call

Every HTTP surface that turns a failed gRPC call into a response (the actor gateway actor_call_failure, every dashboard handler, the gRPC-Web rejection of the auth layer) uses ONE function, plexspaces_grpc_middleware::http_status::http_status_of_code, and one table, the grpc-gateway mapping, so a failure has the same status on every surface. The match is exhaustive over tonic::Code: a new code stops compiling instead of becoming a 500. The dashboard handlers that read a store directly (tenant list, actor stop) have no gRPC status to map; they answer 500 with the cause in the log only.

gRPC code HTTP status
Ok 200
InvalidArgument, FailedPrecondition, OutOfRange 400
Unauthenticated 401
PermissionDenied 403
NotFound 404
AlreadyExists, Aborted 409
ResourceExhausted 429
Cancelled 499
Unknown, Internal, DataLoss 500
Unimplemented 501
Unavailable 503
DeadlineExceeded 504

The gRPC-Web rejection of the auth layer is the one narrowing: only Unauthenticated and PermissionDenied change the HTTP status (browsers surface those before the gRPC trailers); every other rejection stays HTTP 200 with the grpc-status header, which is what gRPC-Web clients read. Rejected alternative: a table per handler (the tables would drift).

Error enums carry proto codes (ProcessGroupError::code() -> ProcessGroupErrorCode, BlobError::code() -> BlobErrorCode), and a service derives its gRPC status from the code with one match, never from message text. Process groups reach HTTP through this table. Blob keeps its own code-to-status table because an oversized payload must be a 413, which this table would turn into a 429 (see Error codes).

Browser access (CORS)

CORS applies to the whole single-port server (HTTP routes and gRPC-Web, the path browsers use for gRPC). By default no CORS headers are sent, so browsers allow only same-origin pages (the dashboard served by the node itself). To let another origin call the node, list it in the release config:

runtime:
  grpc:
    cors_allowed_origins:
      - https://console.example.com

Only the listed origins are answered, with a fixed set of methods (GET, POST, PUT, PATCH, DELETE, OPTIONS) and request headers (authorization, content-type, x-grpc-web, x-user-agent, grpc-timeout, x-namespace, x-accept-content-transfer-encoding); grpc-status, grpc-message and grpc-status-details-bin are exposed to scripts. Credentials (cookies) are never allowed cross-origin. Each entry must be exactly what a browser sends in Origin: http:// or https://, a host (name, IPv4 or bracketed IPv6) and an optional port, with no path (not even a trailing /), query, fragment or user info; null is refused. The single entry "*" allows any origin and is for local development only; it may not be combined with other entries. Any other entry stops the node from starting. Native gRPC clients are not subject to CORS.

Listener

The node binds one TCP listener with SO_REUSEADDR (fast restarts) and without SO_REUSEPORT: with that option any other process of the same user could bind the same port and receive a share of the connections, bearer tokens included. Running several nodes on one host needs different ports, as before.

Not rate limited

Failed authentication is not rate limited at the node (a flood of bad tokens costs a signature check each, bounded by the 8 KB credential cap). Rate limit it at the load balancer or proxy.

Deployed app pages (/apps/)

A tenant admin deploys the files under /apps/:app_id/; the node serves them without credentials from the same origin as the dashboard and the API. Script in a deployed HTML or SVG file would therefore run with the node's origin and could call the API with a visitor's session cookie. Every /apps/ response (including 404s and the redirect) carries Content-Security-Policy: sandbox allow-scripts (one map_response layer on static_apps_router) so the page gets an opaque origin: scripts run, but the page has no cookies, no storage of the node origin and its requests to the node are cross-origin, and X-Content-Type-Options: nosniff so a file is not reinterpreted as another type. Test: apps_responses_carry_the_sandbox_headers.

Effect on apps: a page in the sandbox cannot use the session cookie and cannot read a cross-origin API response (the CORS list runtime.grpc.cors_allowed_origins refuses the null origin a sandboxed page sends). The example apps served from /apps/ that call the node from the page are examples/typescript/apps/ws_chat_room/static and examples/typescript/apps/mersenne_prime/static: both open a WebSocket to /ws with a token typed into the page (?token=), which is neither a cookie nor a CORS request, so they keep working; they use no cookie or storage. No other example has a static page (find examples -name '*.html' lists only these two). A page that needs the cookie or to read an API response must wait for a separate origin for apps (tracked as a later item; allow-same-origin would defeat the sandbox and is not set). Rejected alternatives: per-application authentication (not built), serving /apps/ from a separate origin now (needs deployment changes), and sanitizing uploaded HTML (cannot be made complete).

gRPC authentication (node::grpc_auth)

The node's gRPC services (native gRPC and gRPC-Web) are protected by one layer, node::grpc_auth, applied with GrpcHttpServerBuilder::wrap_grpc_routes to all services. The plexspaces-grpc-middleware crate provides the building blocks (JwtClaims::stamp_identity_headers, NodeTokenMinter, the server builder). The crate also contains an interceptor library (InterceptorChain, AuthInterceptor with RBAC, metrics, tracing, rate-limit, retry and compression interceptors, mTLS helpers). It is NOT wired into the node: the node's production gRPC path is grpc_auth only, and inbound mTLS is not enforced. AuthInterceptor shares the node's rules only in its JWT branch (empty-tenant and node tokens rejected, identity headers produced by JwtClaims::stamp_identity_headers), and that branch verifies ES256 tokens only (AuthMiddlewareConfig.jwt_key is the ES256 public key, inline PEM or file path); see the crate README. Its mTLS branch is weaker: any holder of an accepted certificate is let through (all certificates when mtls_trusted_services is empty), and the request's identity headers (tenant, user, roles) are passed on as-is, on the assumption that the calling node already validated the original user. In its mTLS path the peer identity is read only from the verified peer certificate (peer_certificate: DNS subjectAltNames only; the common name is ignored per RFC 6125 and a certificate without a DNS SAN is rejected); the separate peer_service_id field and client-supplied x-peer-service-id/x-peer-certificate/x-forwarded-client-cert/x-client-cert headers are never trusted, and the headers are stripped.

Method policy table

grpc_auth::policy_for maps every gRPC method (exact /package.Service/Method) to who may call it; methods that are not listed are denied, and the node refuses to start if a registered service has no entry (ensure_services_classified over the server builder's service names, exercised by the Node::start integration tests). No test reads generated source: every_classified_method_exists_on_the_real_server starts a node and checks that every classified method of a registered service is routed, and unit tests send every classified path through the real layer as a member, a tenant admin, an internal node call and a delegated node call, checking both denials and the intended ALLOWs against a hand-written access class for EVERY method (kept apart from the policy table and required to cover exactly the classified paths), and again against an independent reading of each table row. What cannot be derived without the proto descriptor is the reverse direction (a new generated method that was never classified); it is denied at runtime by the default rule, so it fails closed rather than open.

User access Meaning
Public no credentials (only grpc.health.v1.Health/Check)
Member any authenticated user; the service scopes data to the token's tenant
TenantAdmin a token with is_admin; acts in its own tenant only
SystemOnly no user token. Reserved for the node-internal system scope; there is no platform-operator role yet, so these operations are not available remotely with auth on
Node access Meaning
Internal a peer acting for itself, in the system scope
Delegated a peer forwarding work for a client, in the client's tenant
InternalOrDelegated / None either kind / peers refused

Highlights: the NodeService control plane (RegisterNodes, UnregisterNode, ConnectNodes, DisconnectNodes), TracingControlService, SystemService (shutdown, backup, config, logs, metrics; Shutdown takes no timeout or graceful flag, so a caller cannot override the operator's shutdown budget), MetricsService exports and ServiceLinkService writes are SystemOnly. The typed Record* RPCs (RecordMessageRouting, RecordActorActivation, RecordChannelMetrics) are SystemOnly with no node access either: they write the framework's own plexspaces_* series, which dashboards and alerts trust, so no remote caller (tenant or peer node) may forge them; the node records those series in process. MetricsService.GetMetrics is SystemOnly for users and InternalOrDelegated for peers: an internal node call sees every series, a delegated call (a peer forwarding a tenant's dashboard query) is answered with that tenant's series only (it carries the wildcard grant, so every namespace of the tenant; the forwarding dashboard filters the rows), and a request without the layer's marker is Unauthenticated. MetricsService.RecordMetric is Member: the handler exports every series under the caller's tenant_id and caps them (128 per tenant, 1024 per node), so tenants cannot overwrite each other or exhaust the registry. UserService.UpdateUser is Member because the handler allows an admin of the user's tenant or the account owner (who cannot grant admin to themselves); making it TenantAdmin would have made the owner path unreachable. UserService.SetUserGrants (roles, groups, allowed namespaces) is TenantAdmin, so an account owner cannot grant itself privileges. UserService.GetOrCreateByEmail stays SystemOnly with no node access: account provisioning is not a network operation (the OIDC login flow calls the user repository in process), so no remote caller reaches it. SchedulingService capacity reads (GetNodeCapacity, ListNodeCapacities: cluster topology, read in the system scope) are SystemOnly for users and Internal for peer nodes (so a peer acting for itself can read them and no tenant, admin or delegated call can), and the handler grants the system scope only on the layer's AuthenticatedNodeCall proof of an internal node call; a missing tenant header proves nothing, so any other request, including every request to a node without authentication (where no layer vouches for anyone), gets PERMISSION_DENIED. NodeService's ListConnectedNodes is SystemOnly as well, and so is its HTTP bridge GET /api/v1/nodes (see above). The dashboard's GetDashboardMetrics and GetMetricsTable are Member but tenant-scoped: a tenant sees only the series labelled with its tenant_id (a tenant_id label filter it supplies is replaced by its own) and, for a token whose grant lists namespaces, only those namespaces' series plus the tenant's user series; only a node without authentication returns the whole recorder. Every method of FirecrackerVmService (CreateVm, BootVm, PauseVm, ResumeVm, StopVm, GetVmState, ListVms, DeployApplication, UndeployApplication) is SystemOnly with no node access, i.e. denied to every caller: the node builds that service but does not register it, and starting host processes and TAP devices needs a platform-operator role that does not exist, so a future registration cannot ship open. A test reads the RPC names from the proto and fails if one has no deny row or a row outlives its method. The host side of Firecracker is hardened independently of this table: VM ids are validated (^[A-Za-z0-9_-]{1,64}$), all files live in a 0700 per-node runtime directory and are only deleted after proving they are inside it, and kernel, rootfs and boot arguments are never taken from a request (see the Firecracker crate). WasmRuntimeService.DeployModule (the service's only RPC), ApplicationService deploy/undeploy, UserService admin RPCs and ObjectRegistry writes are TenantAdmin. Registering an object of type NODE is additionally refused to tenants. Health/Watch needs authentication; the health service caps open watchers at 64 per node and 8 per tenant (MAX_HEALTH_WATCHERS, MAX_HEALTH_WATCHERS_PER_TENANT), so one tenant cannot hold every slot.

Stream lifetime

A token is checked when a call opens, and a server stream can run for much longer. The auth layer therefore ends every streaming response it authenticated at the token's exp: a timer per stream (tokio time, the node's injected Clock for the wall-clock expiry; no polling) drops the handler's stream and the client sees the RPC finish with UNAUTHENTICATED (HTTP trailers on native gRPC, a trailer frame on gRPC-Web; the base64 gRPC-Web text variant just ends). The streaming methods are listed in STREAMING_RESPONSE_METHODS (Health/Watch, ActorService/StreamMessages, NodeService/StreamConnectedNodes, BlobService/DownloadBlob); a method added later that returns a stream must be added there (a test checks every entry is a classified method). An expiry further than 30 years away is capped at 30 years so a huge exp cannot overflow the timer. Unary responses are not wrapped (it would cost a timer and a body adapter per call, and their handler has finished before the response reaches the layer), but every call, unary or streaming, is refused with UNAUTHENTICATED when its credential is already expired on the node's clock. Revocation is not pushed to an open stream: a revoked token's stream ends at its exp at the latest. WebSocket sessions on the HTTP side have their own cutoff at exp plus a periodic revocation recheck (see the WebSocket upgrade section).

The in-flight request counter that the shutdown drain waits for and that feeds the queue depth readiness input is one layer around the whole router, outside the authentication layer (node/in_flight_layer.rs). It reads only the request path (to skip SystemService.BeginShutdown, which drains the node itself; SystemService.Shutdown starts the node shutdown in the background and is SystemOnly) and never an identity, header or body, adds no metric labels, and counts a request for as long as its handler runs, so an unauthenticated call that the auth layer refuses is counted for the instant it takes to refuse it. Streams are not counted (their handler has returned), so an open Watch never holds a shutdown. The readiness threshold (queue_depth_threshold, 10 000 concurrent requests by default) is the bound on how far concurrent callers can push the count; the evaluator publishes a crossing within about 5 s, and the probes read the live count. See Detailed Design: In-flight requests and the drain.

Health/Check and Health/Watch answer by service name: the empty name is the whole node (alive and ready), and the Kubernetes probe names are liveness (alive and not shutting down; it ignores the dependencies, so a database outage fails readiness but never gets the pod restarted), readiness (alive and ready through the cached dependency view; the same rule as the empty name) and startup (startup complete). All of them are NOT_SERVING once graceful shutdown began. Any other name is that service's own status (UNKNOWN when the node does not track it; NOT_FOUND is not used yet), and a service registered under one of the probe names is shadowed by the probe. The shipped manifests and compose files probe with grpc_health_probe -service=startup|liveness|readiness. A Check of liveness or readiness counts one probe of that kind in the probe counters, the empty name counts both, and startup counts none (there is no counter for it). Health/Watch is push-based: there is no polling. The health reporter publishes every status change on a tokio::sync::watch channel (PlexSpacesHealthReporter::subscribe, fed by lifecycle events and by the node's background evaluator, which re-runs the readiness checkers every monitoring_interval), and each Watch stream follows that channel: it sends the current status first, then only changes of the status of its own service name (the empty name is the whole node; a name the node does not track stays UNKNOWN). When the node begins shutting down every stream sends NOT_SERVING and then ends cleanly, so clients re-resolve instead of waiting on a node that is going away; the evaluator keeps running until that final status is published. A watcher that does not read a status within 30 s is dropped and frees its slot. Watch still needs authentication and stays capped (64 per node, 8 per tenant), and it does not count in the health probe counters (the evaluator counts its own runs as source="background", Check as source="probe"). Check is unauthenticated, so it never runs dependency I/O per request: it reads the evaluator's cached results (re-run at most once per twice the evaluation interval, whatever the request rate), and each checker is bounded by a 5 s timeout. See metrics for the probe counters.

Trust model

Caller Proof Identity the handler sees
End user exactly one Authorization: Bearer <JWT> header (no session cookie on gRPC); signature, expiry, non-empty valid tenant_id, API-token revocation (fail closed, see below) x-tenant-id, x-user-id, x-admin, x-user-roles, x-user-groups stamped from the claims; every client-sent value and x-node-caller removed
Peer node, internal call node token + x-plexspaces-call-kind: internal system scope (an x-namespace it carries must be a single valid namespace name, like for every other caller), only on methods whose policy grants Internal; granted by the AuthenticatedNodeCall request extension the layer inserts (x-admin: true and no tenant are also stamped for logs, but authorize nothing)
Peer node, delegated call node token + x-plexspaces-call-kind: delegated exactly one valid x-tenant-id, optional single valid x-namespace and single x-user-id (printable ASCII without whitespace, at most 256 bytes); never admin (x-admin removed); only on methods whose policy grants Delegated
Peer node, wrong or missing kind, or a method outside its kind node token rejected: INVALID_ARGUMENT (kind or tenant), PERMISSION_DENIED (method)
Anyone else none, invalid, expired, foreign key, duplicated or oversized authorization UNAUTHENTICATED (UNAVAILABLE if no JWT key)

gRPC-Web callers additionally get HTTP 401 or 403. System scope is reachable only through an internal node call, and only on positive proof from the layer: plexspaces_actor::AuthenticatedNodeCall (node id and call kind) in the request extensions, which only grpc_auth inserts after validating a node token (extensions never cross the wire, and the layer removes any AuthenticatedNodeCall or JwtClaims it finds before authenticating). request_context_from_grpc_request and request_context_from_grpc_metadata never produce the system scope, whatever headers they see; only request_context_from_grpc_call (used by NodeService, the one service that serves internal node calls) does, when auth is enabled and the proof is present with kind internal. Forged x-node-caller / x-admin / x-plexspaces-call-kind headers without the proof yield no context at all (a tenant is required with auth on). Tenant and namespace names in tokens and delegated headers are validated (plexspaces_common::validate_tenant_name: at most 128 bytes, [A-Za-z0-9._-], reserved names refused, also when tenants are created).

Node token. A JWT signed with the cluster JWT key (NodeTokenMinter): iss=plexspaces-node, aud=plexspaces-node-mesh, sub=node:<node_id>, a node: key id, nbf/iat/exp, lifetime 120 seconds, cached and refreshed after half its life. nbf is backdated by NODE_TOKEN_NBF_SKEW_SECS (5 s) so a receiver whose clock trails the minter's by a fraction of a second does not reject a fresh token; exp is exact. Verification (validate_node_token) requires all of these, with zero clock leeway (clocks of cluster nodes must still be synchronised: a receiver more than 5 s behind rejects the token, and one ahead sees it expire on time), and user-token validation rejects the node issuer and audience, so neither kind of token is accepted in place of the other. The token is never logged or used as a metric label.

Where the token is sent. GrpcConnectionManager channels (NodeChannel) add the token to calls without their own authorization only when the connection is TLS, or the operator opted in to plaintext and the peer is a configured seed node. Otherwise the call fails (UNAUTHENTICATED), it is never sent unauthenticated or to an arbitrary address. The end user's token is never forwarded between nodes. With auth on, the node installs the token source (and refuses unsafe setups) in Node::start, before any outbound call.

Authentication snapshot. Whether authentication is enabled is decided once, in Node::start, after the release's security config has been registered (so the snapshot sees the final config), and frozen in the service locator (ServiceLocatorImpl::freeze_auth_disabled). From then on is_auth_disabled() returns that value for every consumer (the gRPC layer, the HTTP gateway, dashboard and handlers building request contexts); the PLEXSPACES_DISABLE_AUTH environment variable and later register_security_config calls are not consulted, so a handler can never treat a request as authenticated while the layer let it through unauthenticated, or the reverse. Freezing a different value a second time is an error.

API token revocation fails closed. A token with a jti (an API token) is checked against the revocation list on every request, over gRPC and HTTP. If the lookup errors, the request is refused with 503 / UNAVAILABLE (never accepted); if the node started without a revocation list (user database unreachable at startup), every jti token is refused the same way until the node is restarted with the database, while session tokens (no jti) keep working. Lookups are cached for a short TTL (RevocationCachingTokenRepository: 5 s, 10 000 tokens, time from the injected Clock, errors never cached). A cached "revoked" is sticky: a "not revoked" answer from a lookup that started before a revocation and finished after it never replaces it. A revocation made through the node that serves the request is effective immediately; one made on another node or directly in the database is visible after at most the TTL. Counted in plexspaces_auth_revocation_unavailable_total{cause}.

Blast radius. The token proves the caller holds the cluster JWT key; protect that key like a root credential (anyone holding it can mint any user token). A captured node token, replayed within its 120 seconds (there is no per-call nonce), acts as a non-admin member of any tenant, on the delegable methods only; cross-tenant system scope is reachable only on the internal methods (membership pings, node metrics, application status). The call kinds and lists limit the damage of bugs and captured tokens; they are not a boundary against a compromised node. Observability. plexspaces_grpc_auth_rejections_total{reason} (auth_unavailable, unauthenticated, header_too_large, method_unlisted, user_class_denied, node_call_kind_invalid, node_method_denied, node_tenant_invalid, scope_name_invalid, namespace_denied), plexspaces_http_namespace_denials_total (HTTP namespace refusals), plexspaces_grpc_auth_accepted_total{caller="user|node"}, plexspaces_node_token_withheld_total and plexspaces_auth_token_rejections_total. Rejections are logged at debug level (an unauthenticated caller controls their rate).

GetActorStates limits. A request may carry at most MAX_ACTOR_STATES_PER_REQUEST (1000) actor ids (above that: INVALID_ARGUMENT); malformed ids are skipped and counted in plexspaces_actor_states_malformed_ids_total instead of failing the batch, and lookups run with bounded concurrency. The client splits larger questions into batches, bounds each request with ACTOR_STATES_REQUEST_TIMEOUT and every new connection with GRPC_CONNECT_TIMEOUT.

Clients. The CLI sends the token in PLEXSPACES_TOKEN as a bearer token; RemoteActorClient::with_bearer_token does the same for Rust callers. Without a token a node with auth enabled answers UNAUTHENTICATED. The CLI dials host:port and http:// addresses in plaintext and https:// addresses over TLS, trusting only the CA file named by PLEXSPACES_TLS_CA (no system roots; without it an https:// address is refused). It refuses to send PLEXSPACES_TOKEN in plaintext to a non-loopback host (anything but localhost and loopback IPs) unless PLEXSPACES_ALLOW_PLAINTEXT_TOKEN=true is set, and gives up connecting after 10 seconds.

Generated certificates

CertificateGenerator (development and test clusters only) writes ca.key and server.key with mode 0600 set at creation, through a temporary file renamed into place, so a private key is never readable by other users, not even briefly, and a looser pre-existing file is replaced. The four files (CA certificate and key, server certificate and key) are replaced as a set: all are staged first, then the server certificate is removed, the others renamed into place and the server certificate renamed last, so a crash in between leaves a set that fails the "all four exist" check and is regenerated, never a CA that does not match the server certificate. Production certificates should come from cert-manager or Vault PKI.

gRPC tenant-id vs namespace (caller contract)

  • Tenant-id: Callers do not choose tenant by setting metadata themselves when JWT auth is enabled. The JWT proves identity; middleware writes x-tenant-id (and related identity headers) from validated claims. Treat any pre-middleware client value as untrusted; the gRPC auth layer removes client-supplied identity headers and repopulates them from the verified claims.
  • Namespace: x-namespace may be supplied for request scoping (or taken from proto fields / labels where documented). Services build RequestContext via request_context_from_grpc_request so tenant and namespace follow the same validation rules as HTTP.
  • Services: gRPC handlers should use request_context_from_grpc_request (or equivalent) for every RPC that touches tenant-scoped state. Avoid ad hoc RequestContext::new_without_auth("", "") on code paths that perform routing or registry access.
  • WASM host routing: When a WASM actor sends or asks, the host must derive RequestContext from the registered sender actor/runtime metadata instead of manufacturing an empty tenant. That preserves the tenant established by the verified JWT and the namespace already bound to the actor identity.

RequestContext and Auth

  • Auth enabled: request_context_from_grpc_request uses auth_enabled = !service_locator.is_auth_disabled().await. Tenant_id must be present in metadata (set by the auth layer from the verified JWT). If missing, errors include a hint: "Authentication required: provide a valid JWT. For local testing, set PLEXSPACES_DISABLE_AUTH=1."
  • Auth disabled: Tenant_id may come from metadata or defaults for local testing.

Environment Variables

Security Variables

# Outbound node-to-node TLS paths (used when runtime.security.mtls.enable_mtls is true in the
# release config; a path set in the config wins over its variable)
PLEXSPACES_MTLS_CA_CERT=/certs/ca.crt
PLEXSPACES_MTLS_SERVER_CERT=/certs/server.crt
PLEXSPACES_MTLS_SERVER_KEY=/certs/server.key

# JWT (ES256 only; a persistent key is required when auth is enabled)
PLEXSPACES_JWT_PRIVATE_KEY=...          # Inline ES256 PEM (for Docker/K8s secrets)
PLEXSPACES_JWT_PRIVATE_KEY_FILE=...     # Path to ES256 PEM file (generated and saved if missing, only on a node without seed nodes)
# Not read by the node (no external issuer or JWKS validation, no configurable claim names):
#   PLEXSPACES_JWT_ISSUER, PLEXSPACES_JWT_JWKS_URL, PLEXSPACES_JWT_TENANT_ID_CLAIM

# CLI
PLEXSPACES_TOKEN=...                    # User token the CLI presents
PLEXSPACES_TLS_CA=/certs/ca.crt         # CA that signed the node certificate; required to dial https:// nodes
PLEXSPACES_ALLOW_PLAINTEXT_TOKEN=true   # Let the CLI send its token in plaintext to a non-loopback node (unsafe)

# Security
PLEXSPACES_DISABLE_AUTH=false  # Set to true to disable auth for testing (development only)
PLEXSPACES_ALLOW_UNAUTHENTICATED_PUBLIC_BIND=false  # Needed to start with auth disabled on a non-loopback address

Preventing Unauthorized Access

PlexSpaces uses a multi-layered security approach to prevent unauthorized access:

1. Node token over TLS: Preventing Node Impersonation

Problem: An attacker could impersonate a legitimate node in the cluster, or capture the credential nodes use.

Solution: Nodes authenticate to each other with the node token (verified with the cluster JWT key), and the token is sent only over TLS to CA-verified peers, or to configured seed nodes when plaintext is explicitly allowed.

What it does not do: the node does not verify client certificates on inbound connections, so TLS alone does not authenticate a caller; the node token does. Terminate TLS in a proxy or mesh if you also want transport-level client authentication.

Prevents:

  • Calls from processes that do not hold the cluster JWT key
  • Sending the node credential to an address an attacker chose (untrusted destinations get no token)

Does not prevent: a compromised node or a leaked cluster JWT key (both can mint tokens).

2. JWT: Verifying User Identity

Problem: An attacker could send requests without proper authentication.

Solution: JWT tokens verify user identity and extract tenant context.

// JWT middleware validates token and extracts tenant_id
let claims = jwt::decode(&token, &decoding_key, &validation)?;
let tenant_id = claims.claims.get("tenant_id")
    .ok_or(AuthError::MissingTenantId)?;

How it works:

  • User authenticates with identity provider (OAuth2/OIDC)
  • Identity provider issues JWT token with tenant_id claim
  • PlexSpaces validates JWT signature and expiration
  • Tenant_id extracted from claims and added to RequestContext
  • Invalid tokens result in 401 Unauthorized

Prevents:

  • ✅ Unauthenticated requests
  • ✅ Token tampering (signature validation)
  • ✅ Expired token reuse (expiration check)

3. RequestContext: Propagating Security Context

Problem: Tenant context could be lost or tampered with during request processing.

Solution: RequestContext carries tenant_id through the entire call chain.

// RequestContext created from JWT claims
let ctx = RequestContext::new(tenant_id)
    .with_user_id(user_id)
    .with_namespace(namespace);

// Context passed to all service/repository methods
let result = blob_service.upload_blob(&ctx, name, data).await?;

How it works:

  • RequestContext created at API boundary (from JWT)
  • Context is immutable and passed by reference
  • All service methods require RequestContext
  • Context cannot be modified mid-request
  • Missing context results in compilation error (type safety)

Prevents:

  • ✅ Tenant context loss
  • ✅ Context tampering (immutable)
  • ✅ Missing tenant_id (compile-time check)

4. Narrowed Queries: Database-Level Enforcement

Problem: An attacker could bypass application-level checks and access other tenants' data.

Solution: All SQL queries automatically filter by tenant_id and namespace.

// Repository method automatically filters by tenant_id
async fn get(&self, ctx: &RequestContext, blob_id: &str) -> Result<Option<BlobMetadata>> {
    // Query ALWAYS includes tenant_id and namespace filters
    sqlx::query("SELECT * FROM blob_metadata WHERE blob_id = $1 AND tenant_id = $2 AND namespace = $3")
        .bind(blob_id)
        .bind(ctx.tenant_id())  // From RequestContext
        .bind(ctx.namespace())  // From RequestContext
        .fetch_optional(&*self.pool)
        .await
}

How it works:

  • All repository methods require RequestContext
  • SQL queries ALWAYS include WHERE tenant_id = $X AND namespace = $Y
  • Database enforces tenant boundaries (even if application code has bugs)
  • No way to bypass tenant filtering (enforced at type level)

Prevents:

  • ✅ Cross-tenant data access
  • ✅ SQL injection attacks (parameterized queries)
  • ✅ Bypassing application-level checks

5. Method policy and admin classes

Problem: Authenticated users should not reach operator or cross-tenant operations.

Solution: the gRPC policy table (node::grpc_auth) classifies every method. Operator operations are SystemOnly (no user token), code-deploying and tenant-administration operations are TenantAdmin (token with is_admin, acting in its own tenant), everything else is Member and relies on the service's tenant scoping. The HTTP routes of the node's main router follow the same table through the shared gate (http_routes/policy_gate.rs, authorize_caller), and the blob REST routes authenticate in blob_http_router::authenticate (all Member rows). Role names in tokens are carried for handlers but are not an authorization input of the layer.

Prevents:

  • A tenant user changing which nodes the node talks to, reading node-wide configuration or shutting the node down
  • Privilege escalation through a delegated node call (delegated calls are never admin)

6. Complete Security Flow

Here's how all layers work together to prevent unauthorized access:

1. User Request
   ↓
2. Transport (node-to-node)
   - TLS to a CA-verified peer (or a configured seed with the plaintext opt-in)
   - Node token attached by the connection manager
   ↓
3. JWT Validation (if user API)
   - Validates token signature
   - Checks expiration
   - Extracts tenant_id, user_id, roles
   ↓
4. RequestContext Creation
   - tenant_id from JWT (required)
   - user_id from JWT (optional)
   - roles from JWT (optional)
   - request_id generated (ULID)
   ↓
5. Service Layer
   - Validates RequestContext
   - Checks roles (if needed)
   - Passes context to repository
   ↓
6. Repository Layer
   - All queries filter by tenant_id
   - Database enforces boundaries
   - Returns only tenant's data
   ↓
7. Response
   - Data scoped to tenant
   - Audit log includes tenant_id

Security Guarantees

With all layers in place, PlexSpaces guarantees:

  1. No Unauthenticated Access: All requests require a valid JWT (user APIs) or node token (node-to-node) when auth is enabled
  2. No Cross-Tenant Access: Database queries always filter by tenant_id
  3. No Context Tampering: RequestContext is immutable and type-checked
  4. No Privilege Escalation: every gRPC method has an access class; delegated node calls are never admin
  5. No SQL Injection: All queries use parameterized statements
  6. Audit Trail: All operations log tenant_id for compliance

Example: Unauthorized Access Attempt

Let's trace what happens when an attacker tries to access another tenant's data:

// Attacker sends request with tenant-1 JWT
let attacker_ctx = RequestContext::new("tenant-1".to_string());

// Attacker tries to access tenant-2's blob
let blob_id = "blob-123";  // Belongs to tenant-2

// Repository query automatically filters by tenant_id
let result = repository.get(&attacker_ctx, blob_id).await?;
// SQL: SELECT * FROM blob_metadata 
//      WHERE blob_id = 'blob-123' 
//      AND tenant_id = 'tenant-1'  ← From RequestContext
//      AND namespace = 'default'
// Result: None (blob belongs to tenant-2, not tenant-1)

// Attacker gets NotFound error, not tenant-2's data
assert!(result.is_none());

What prevented the attack:

  • ✅ JWT validation ensured attacker can only use tenant-1 context
  • ✅ RequestContext carried tenant-1 (immutable)
  • ✅ SQL query filtered by tenant-1 (database-level enforcement)
  • ✅ No way to bypass tenant filtering (type-safe)

Auth Headers & Security Schemes (OpenAPI-Style)

Overview

PlexSpaces supports propagating authentication credentials and custom headers through the entire call chain via the RequestContext.headers field. This follows the same patterns defined by OpenAPI 3.0 Security Schemes, allowing actors and services to make authenticated outbound calls using credentials propagated from the original request.

Supported Security Schemes

PlexSpaces maps to OpenAPI security scheme types:

OpenAPI Scheme PlexSpaces API Header Key Example Value
type: http, scheme: bearer ctx.with_bearer_token(token) authorization Bearer eyJhbGci...
type: apiKey, in: header ctx.with_api_key_header(name, key) x-api-key (or custom) sk-abc123
type: apiKey, in: query ctx.with_api_key_query(name, key) apikey-query:<name> sk-abc123
type: mutualTLS mTLS at transport layer (TLS, not header) Certificate-based
Custom headers ctx.with_header(name, value) any lowercase name any value

OpenAPI Security Definitions (Proto-Generated)

Security schemes are defined in proto files using protoc-gen-openapiv2 annotations and are automatically included in the generated OpenAPI spec. From common.proto:

option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {
  // ...
  security_definitions: {
    security: {
      key: "BearerAuth";
      value: {
        type: TYPE_API_KEY;
        in: IN_HEADER;
        name: "Authorization";
        description: "JWT Bearer token. Format: Bearer <token>.";
      };
    };
  };
  security: {
    security_requirement: {
      key: "BearerAuth";
      value: {};
    };
  };
};

This generates the standard OpenAPI securityDefinitions / security blocks when protoc-gen-openapiv2 runs. The generated spec (docs/openapi.json) will include:

{
  "securityDefinitions": {
    "BearerAuth": { "type": "apiKey", "name": "Authorization", "in": "header" }
  },
  "security": [{ "BearerAuth": [] }]
}

The node's own API accepts only the signed Bearer JWT. The apiKey rows in the table above describe credentials the node forwards to external HTTP services through service links; the node itself never accepts a static API key (the protos have no API-key messages or OpenAPI schemes).

Rust API

Attaching Credentials

use plexspaces_common::RequestContext;

// Bearer token (most common — JWT)
let ctx = RequestContext::new("tenant-123".to_string(), "prod".to_string(), true)?
    .with_bearer_token("eyJhbGciOiJIUzI1NiIs...".to_string());

// API key in header
let ctx = RequestContext::new("tenant-123".to_string(), "prod".to_string(), false)?
    .with_api_key_header("x-api-key".to_string(), "sk-live-abc123".to_string());

// API key in query parameter
let ctx = RequestContext::new("tenant-123".to_string(), "prod".to_string(), false)?
    .with_api_key_query("api_key".to_string(), "sk-live-abc123".to_string());

// Multiple credentials + custom headers
let ctx = RequestContext::new("tenant-123".to_string(), "prod".to_string(), true)?
    .with_bearer_token("eyJ...".to_string())
    .with_header("x-request-source".to_string(), "mobile-app".to_string())
    .with_header("x-idempotency-key".to_string(), "req-abc-123".to_string());

// Bulk headers (e.g., from inbound HTTP request)
let mut headers = std::collections::HashMap::new();
headers.insert("authorization".to_string(), "Bearer eyJ...".to_string());
headers.insert("x-api-key".to_string(), "sk-abc".to_string());
let ctx = RequestContext::new("tenant-123".to_string(), "prod".to_string(), false)?
    .with_headers(headers);

Reading Credentials

// Bearer token (strips "Bearer " prefix)
if let Some(token) = ctx.bearer_token() {
    // token = "eyJhbGciOiJIUzI1NiIs..."
}

// Any header by name
if let Some(api_key) = ctx.get_header("x-api-key") {
    // api_key = "sk-live-abc123"
}

// API key query parameter
if let Some(key) = ctx.api_key_query("api_key") {
    // key = "sk-live-abc123"
}

// All HTTP headers (excludes internal apikey-query:* entries)
let http_headers = ctx.http_headers();
// Returns: {"authorization": "Bearer ...", "x-api-key": "sk-..."}

// All query-param API keys
let query_params = ctx.api_key_query_params();
// Returns: {"api_key": "sk-..."}

Creating Context from Auth with Headers

// From the validated identity (a proto skeleton) + inbound request headers
let skeleton = RequestContext {
    tenant_id: "tenant-123".to_string(),            // from JWT
    namespace: "prod".to_string(),                  // from request path
    user_id: "user-456".to_string(),                // from JWT sub
    admin: false,                                   // from JWT
    allowed_namespaces: vec!["prod".to_string()],   // from the JWT `namespaces` claim
    headers: inbound_headers,                       // propagate from request
    ..Default::default()
};
let ctx = RequestContext::from_auth(skeleton, /* auth_enabled */ true, /* default_tenant_id */ None)?;

// Work in another namespace of the same caller: authorized, grant kept
let ctx_b = ctx.derive_in_namespace("staging")?;   // NamespaceNotAllowed if "prod" is the only grant

Proto Wire Format

The RequestContext proto message includes a headers map (field 11) and the caller's namespace grant (allowed_namespaces, field 12):

message RequestContext {
  string tenant_id = 1;
  string namespace = 2;
  string user_id = 3;
  string request_id = 4;
  string correlation_id = 5;
  google.protobuf.Timestamp timestamp = 6;
  map<string, string> metadata = 7;
  bool admin = 8;
  bool internal = 9;
  bool auth_enabled = 10;
  map<string, string> headers = 11;  // Auth/credential propagation
  repeated string allowed_namespaces = 12;  // Verified grant; see Namespace authorization
}

Headers survive proto serialization roundtrips:

let original = RequestContext::new_without_auth("t1".into(), "ns".into())
    .with_bearer_token("tok".to_string());

use prost::Message;
let restored = RequestContext::decode(original.encode_to_vec().as_slice()).unwrap();

assert_eq!(restored.bearer_token(), Some("tok"));

WIT (WASM Actors)

The WIT context record includes a headers field for WASM actors:

record context {
    tenant-id: string,
    namespace: string,
    headers: list<tuple<string, string>>,
}

WASM actors can read and propagate auth headers when making outbound calls:

# Python SDK example
from plexspaces import host

def handle_message(ctx, msg):
    # Read bearer token from context
    for name, value in ctx.headers:
        if name == "authorization":
            # Use token for outbound HTTP call
            host.http_request("https://api.example.com/data",
                              headers={"Authorization": value})
// TypeScript SDK example
import { Host } from "plexspaces";

function handleMessage(ctx: Context, msg: Message) {
    const authHeader = ctx.headers.find(([k]) => k === "authorization");
    if (authHeader) {
        // Propagate to downstream service
        Host.tell(targetActor, msgType, payload, {
            headers: [authHeader]
        });
    }
}

Header Propagation Flow

Client Request
    │
    ├── Authorization: Bearer <JWT>
    ├── X-API-Key: sk-abc123
    └── X-Custom: value
         │
         ▼
┌────────────────────────────────┐
│   HTTP Gateway / gRPC Middleware│
│                                │
│  1. Validate JWT / node token  │
│  2. Extract tenant_id from JWT │
│  3. Build RequestContext with   │
│     headers from request       │
└────────────────────────────────┘
         │
         ▼
┌────────────────────────────────┐
│       RequestContext            │
│  tenant_id: "tenant-123"       │
│  namespace: "prod"             │
│  headers:                      │
│    authorization: Bearer <JWT> │
│    x-api-key: sk-abc123        │
│    x-custom: value             │
└────────────────────────────────┘
         │
    ┌────┴────┐
    ▼         ▼
 Actor     Service
  │           │
  │  ctx.bearer_token()
  │  ctx.get_header("x-api-key")
  │           │
  ▼           ▼
 Outbound HTTP/gRPC calls
 with propagated headers

Security Considerations

  1. Bearer tokens from JWT only (when auth enabled): When authentication is enabled, the authorization header in RequestContext.headers is set from the validated JWT token only. Client-supplied Authorization headers are stripped by the auth middleware before being added to the context. This prevents token injection attacks.

  2. Header names are lowercase: All header names are stored in lowercase per HTTP/2 convention. The with_header() and get_header() methods handle case normalization.

  3. Sensitive headers should not be logged: When logging RequestContext, the headers field may contain secrets (tokens, API keys). Use the SecretMasker utility or exclude headers from log output.

  4. Headers propagate across nodes: When an actor message is forwarded to a remote node, RequestContext (including headers) is serialized via proto. Ensure auth headers are appropriate for cross-node propagation (e.g., internal service tokens vs. user tokens).

  5. API keys in query params are less secure: The with_api_key_query() method stores keys internally but they may appear in URLs/logs. Prefer header-based API keys when possible.

  6. WASM actor sandbox: WASM actors can read headers from their context but cannot modify the context of their parent. Headers flow downward (caller → callee), never upward.

Comparison with OpenAPI Security Schemes

Feature OpenAPI 3.0 PlexSpaces RequestContext
Bearer token (HTTP) type: http, scheme: bearer with_bearer_token()
API key in header type: apiKey, in: header with_api_key_header(name, key)
API key in query type: apiKey, in: query with_api_key_query(name, key)
Mutual TLS type: mutualTLS Transport-level mTLS config
OAuth2 / OpenID Connect type: oauth2 / openIdConnect JWT token from OIDC provider → with_bearer_token()
Custom scheme extension (x-*) with_header(name, value)
Multiple schemes (AND) security: [{A, B}] Chain multiple with_* calls
Multiple schemes (OR) security: [{A}, {B}] Different code paths per scheme
Scope-based authorization security: [{A: [read]}] Roles/scopes in JWT claims

Troubleshooting

Common Issues

  1. HTTP 401 Unauthorized when calling /api/v1/actors/...

    • Cause: Auth is enabled but no valid JWT was provided (or token is expired/invalid).
    • Solution: Send Authorization: Bearer <token> with a token created via plexspaces-cli jwt-create (see Creating JWT Tokens (CLI)). Or for local testing only: export PLEXSPACES_DISABLE_AUTH=1 and restart the node.
  2. HTTP 503 when calling /api/v1/actors/...

    • Cause: Auth is enabled but no JWT key is configured.
    • Solution: Set PLEXSPACES_JWT_PRIVATE_KEY_FILE (or PLEXSPACES_JWT_PRIVATE_KEY). Or for local testing: export PLEXSPACES_DISABLE_AUTH=1.
  3. "Missing required tenant_id in RequestContext"

    • Cause: Auth is enabled but tenant_id was not set (e.g. JWT missing tenant_id claim or gRPC metadata not set by middleware).
    • Solution: Ensure JWT includes a non-empty tenant_id claim; ensure gRPC requests go through auth middleware so metadata is set. Error message includes: "For local testing, set PLEXSPACES_DISABLE_AUTH=1."
  4. "HS256 is not supported" / PLEXSPACES_JWT_SECRET is rejected at startup

    • Solution: Unset PLEXSPACES_JWT_SECRET and provide an ES256 key through PLEXSPACES_JWT_PRIVATE_KEY_FILE or PLEXSPACES_JWT_PRIVATE_KEY.
  5. "Metadata tenant_id does not match context tenant_id"

    • Solution: Ensure metadata tenant_id matches RequestContext (tenant_id comes from the verified JWT only when auth is on).
  6. Certificate validation fails

    • Solution: Check CA certificate is correct and not expired.

References

There aren't any published security advisories