diff --git a/docs/examples/multitenancy/operator-values.yaml b/docs/examples/multitenancy/operator-values.yaml new file mode 100644 index 0000000..e71ddea --- /dev/null +++ b/docs/examples/multitenancy/operator-values.yaml @@ -0,0 +1,42 @@ +global: + imagePullSecrets: [{name: nvcr-image-pull-secret}] +agentSandbox: + install: false +llm: + mode: external + external: + baseUrl: https://llm.example.com/v1 + model: served-model + existingSecret: mosaic-inference +mosaicUi: + auth: + existingSecret: mosaic-ui-auth +modules: + ui: {enabled: true} + execution: {enabled: true} + kubernetes: {enabled: true} + observability: {enabled: true} + grafana: {enabled: true} + bcm: {enabled: true} + slurm: {enabled: false} + diagnostics: {enabled: false} + research: {enabled: false} + terminal: {enabled: false} + clusters: {enabled: true} + edit: {enabled: false} +kubernetes: + defaultCluster: local + local: + enabled: true +observability: + prometheusUrl: http://prometheus.monitoring.svc.cluster.local:9090 + alertmanagerUrl: http://alertmanager.monitoring.svc.cluster.local:9093 + grafanaUrl: https://grafana.example.com/grafana + grafanaDatasourceUid: prometheus + grafanaAuth: + existingSecret: mosaic-grafana-auth +bcmMcp: + enabled: true + mode: ssh-adapter + headHost: operator@bcm-head.example.com + hostSshKeySecretName: bcm-host-ssh-key diff --git a/docs/examples/multitenancy/tenant-reader.yaml b/docs/examples/multitenancy/tenant-reader.yaml new file mode 100644 index 0000000..49c77b1 --- /dev/null +++ b/docs/examples/multitenancy/tenant-reader.yaml @@ -0,0 +1,33 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: mosaic-job-reader + namespace: tenant-a +automountServiceAccountToken: false +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: mosaic-job-reader + namespace: tenant-a +rules: + - apiGroups: [""] + resources: [pods, pods/log, events] + verbs: [get, list, watch] + - apiGroups: [batch] + resources: [jobs, cronjobs] + verbs: [get, list, watch] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: mosaic-job-reader + namespace: tenant-a +subjects: + - kind: ServiceAccount + name: mosaic-job-reader + namespace: tenant-a +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: mosaic-job-reader diff --git a/docs/examples/multitenancy/tenant-values.yaml b/docs/examples/multitenancy/tenant-values.yaml new file mode 100644 index 0000000..ce06cf9 --- /dev/null +++ b/docs/examples/multitenancy/tenant-values.yaml @@ -0,0 +1,37 @@ +global: + imagePullSecrets: [{name: nvcr-image-pull-secret}] +agentSandbox: + install: false +llm: + mode: external + external: + baseUrl: https://llm.example.com/v1 + model: served-model + existingSecret: mosaic-inference +mosaicUi: + auth: + existingSecret: mosaic-ui-auth +modules: + ui: {enabled: true} + execution: {enabled: true} + kubernetes: {enabled: true} + observability: {enabled: false} + grafana: {enabled: false} + bcm: {enabled: false} + slurm: {enabled: false} + diagnostics: {enabled: false} + research: {enabled: false} + terminal: {enabled: false} + clusters: {enabled: false} + edit: {enabled: false} +bcmMcp: + enabled: false +kubernetes: + defaultCluster: tenant-a + local: + enabled: false + clusters: + - name: tenant-a + kubeconfigSecretRef: + name: tenant-a-kubeconfig + key: config diff --git a/docs/images/tenant-operator-instances.svg b/docs/images/tenant-operator-instances.svg new file mode 100644 index 0000000..6af90ca --- /dev/null +++ b/docs/images/tenant-operator-instances.svg @@ -0,0 +1,31 @@ + + Separate operator and tenant Mosaic instances + Two independent Helm releases. Each has its own login, state, and credentials. The operator accesses hardware and systems; the tenant accesses only its jobs. Repeat the tenant instance for each tenant. + + + Separate Mosaic instances + + + + + Operator instance + Focus: hardware and systems + Own UI login · agent state · audit storage + Operator credentials · separate Helm release + Tenant instance + Focus: this tenant’s jobs + Own UI login · agent state · audit storage + Scoped credentials · separate Helm release + + + + + + + Cluster infrastructure + Nodes · hardware inventory · system telemetry + Tenant workloads + Jobs · Pods · events · logs + Repeat this instance for each tenant. + + diff --git a/docs/multitenancy.md b/docs/multitenancy.md new file mode 100644 index 0000000..3a4f432 --- /dev/null +++ b/docs/multitenancy.md @@ -0,0 +1,41 @@ +# Tenant and operator deployments + +Deploy separate Mosaic instances for operators and tenants. **Operators focus on hardware and systems. Tenants focus on their jobs.** Each instance is a separate Helm release with its own UI login, agent state, audit storage, and credentials. + +![Separate operator and tenant Mosaic instances, each with its own login, state, and credentials. The operator accesses cluster infrastructure; the tenant accesses only its jobs.](images/tenant-operator-instances.svg) + +| Instance | Focus | Example question | +| --- | --- | --- | +| Operator | Cluster health, hardware inventory, networking, and system telemetry | “Which nodes are unhealthy?” | +| Tenant | Its own jobs, Pods, events, and logs | “Why is my job pending?” | + +Repeat the tenant instance for each tenant. In an enterprise, these audiences can be the infrastructure team and an application team. Within a tenant VM, visibility is limited to resources exposed to that VM or its scoped APIs. + +## Example setup + +These Kubernetes examples are read-only. The provider installs both releases and reuses an existing compatible agent-sandbox controller. + +- **Tenant:** [tenant-values.yaml](examples/multitenancy/tenant-values.yaml) disables infrastructure integrations and the default cluster reader. Apply [tenant-reader.yaml](examples/multitenancy/tenant-reader.yaml) in workload namespace `tenant-a`, provision a kubeconfig for that identity, and store it as Secret `tenant-a-kubeconfig` (key `config`) in `mosaic-tenant-a`. +- **Operator:** [operator-values.yaml](examples/multitenancy/operator-values.yaml) enables cluster monitoring, observability, and [Base Command Manager](bcm.md). Configure the monitoring URLs, `mosaic-grafana-auth` Secret (`username`, `password`), and `bcm-host-ssh-key` Secret (`id_ecdsa`) for its read-only SSH identity in `mosaic-operator`. + +Complete [namespace and registry access setup](installation.md#1-create-the-namespace-and-registry-access) for each release. In each namespace, create a distinct `mosaic-ui-auth` Secret (`username`, `password`, `machineToken`) and `mosaic-inference` Secret (`apiKey`); replace the example inference URL and model with your approved endpoint. + +Install with your approved chart version: + +```bash +helm upgrade --install mosaic-tenant-a \ + oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack \ + --version "$MOSAIC_CHART_VERSION" --namespace mosaic-tenant-a \ + -f tenant-values.yaml --atomic --wait --timeout 12m + +helm upgrade --install mosaic-operator \ + oci://nvcr.io/0948643769302270/afoa-release/mosaic-stack \ + --version "$MOSAIC_CHART_VERSION" --namespace mosaic-operator \ + -f operator-values.yaml --atomic --wait --timeout 12m +``` + +## Access boundaries + +Backend credentials and Kubernetes RBAC enforce access. **The chart's default Kubernetes reader is cluster-wide**, even when installed in a tenant namespace; keep it disabled for tenants. Before handover, verify with the tenant kubeconfig that job and log reads work in `tenant-a`, while other namespaces, nodes, Secrets, and job mutations are denied. + +Give each audience its own UI address and login. Enforce network isolation with authenticated ingress and provider-managed network policies; namespace separation alone is insufficient. Keep storage and credentials separate, including any added MCP or metrics backend. These are separate deployments, not per-user authorization in a shared UI. Cluster administrators and the runtime controller remain trusted; use separately administered clusters or VMs when required for mutually untrusted tenants. diff --git a/index.md b/index.md index 55432b0..58c95da 100644 --- a/index.md +++ b/index.md @@ -41,6 +41,7 @@ Kind installation :hidden: Security model +Tenant and operator deployments Use cases Slurm root cause analysis Headless integration