Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Wombat exposes HTTP endpoints, it aims at the same feature coverage in both UI a
API hubs include:
* `api/impact` is the equivalent of the dashboard but in JSON form, it makes it possible to query active environments and get impact estimations for their services
* `api/simulation` is for getting estimates over entirely manufactured asset descriptions and service activity summary
* `api/modules` is for getting information about registered `WombatModules` and the asset types they provide
* `api/modules` is for listing and managing registered `WombatModules` and their asset types: see [modules](doc/modules.md)
* `api/environments` is for getting information about active environments
* `api/metrics` is for pushing sampled metrics from worker instances to an aggregator, see [distributed setup](doc/distributed.md)
* `api/secrets` provides admin-only management of the optional encrypted database secret store; see [secret management](doc/secret-management.md)
Expand Down Expand Up @@ -219,6 +219,7 @@ More information on each of these modules can be found [in the `wombat-core` doc
* [Secret management](doc/secret-management.md): how to provide necessary credentials to Wombat connectors
* [Metric compaction](doc/compaction.md): how sourced metrics are folded before being served, and the effects of tuning the compaction window
* [Distributed setup](doc/distributed.md): the standalone, aggregator and worker modes, for workers sampling remote environments and pushing their metrics to an aggregator
* [Modules and core deployments](doc/modules.md): enabling modules from Settings > Modules, redeploying the core, and how replicas converge on it

## How to run

Expand Down Expand Up @@ -259,8 +260,10 @@ When running in dev-mode, you'll have access to the Quarkus dev-ui at `/q/dev-ui

Wombat allows user to provide custom modules built with `wombat-core` (see [repository](https://github.com/illuin-tech/wombat-core)) to extend its functionality.

Custom modules are loaded **_at startup_** from every JAR found in `module.extension.path`
(defaults to `extensions`, relative to the working directory). Override it with `-Dmodule.extension.path=...` or `MODULE_EXTENSION_PATH=...`.
Custom modules are loaded **_at startup_** from every JAR found in `wombat.module.extension.path`
(defaults to `extensions`, relative to the working directory). Override it with `-Dwombat.module.extension.path=...` or `WOMBAT_MODULE_EXTENSION_PATH=...`.
A JAR added later is only loaded after a restart. Once loaded, a module is enabled; it is turned off, and the core
redeployed, from **Settings > Modules**: see [modules](doc/modules.md).

When starting up the app with a custom extension, you should see logs like these:

Expand Down
4 changes: 4 additions & 0 deletions doc/access-control.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,17 @@ Endpoints check permissions, and each role grants a fixed set of them:
| `ui.users.read` | ✓ | | |
| `ui.environments.read` | ✓ | ✓ | ✓ |
| `ui.environments.manage` | ✓ | | |
| `ui.modules.read` | ✓ | ✓ | ✓ |
| `ui.modules.manage` | ✓ | | |
| `api.impact.read` | ✓ | ✓ | ✓ |
| `api.simulation.read` | ✓ | ✓ | ✓ |
| `api.environment.read` | ✓ | ✓ | ✓ |
| `api.environment.write`| ✓ | | |
| `api.asset.read` | ✓ | ✓ | ✓ |
| `api.asset.write` | ✓ | | |
| `api.metric.write` | ✓ | | |
| `api.module.read` | ✓ | ✓ | ✓ |
| `api.module.manage` | ✓ | | |
| `api.apikey.manage` | ✓ | ✓ | ✓ |
| `api.apikey.admin` | ✓ | | ✓ |
| `api.user.manage` | ✓ | | |
Expand Down
16 changes: 14 additions & 2 deletions doc/distributed.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ skipped and reported back to the worker in the response status, while the other
malformed request (e.g. an unknown metric family, a missing asset, a missing or reversed window) is still rejected with
`400`, and nothing from it is saved.

A module turned off on the aggregator, from **Settings > Modules**, applies to the next push: no core redeploy is needed
for the metrics of its assets to be skipped. See [modules](modules.md).

### Declared environments

Each asset belongs to the declaration it comes from: the aggregator's own `monitor.environments-file`, or one worker.
Expand Down Expand Up @@ -117,8 +120,17 @@ removes it from the aggregator at the worker's next startup.
No metric is written to the worker's database, so a worker always runs on an in-memory SQLite database, whatever its
`persistence.backend.mode` setting.

Each window is sent in at most one request per metric family. A push that fails (aggregator unreachable, rejected key)
is retried a few times on network errors, then logged and dropped: the worker does not buffer windows it could not send.
A worker enables every module it detects and currently cannot turn them off: it will only declare to the aggregator the assets it samples. The worker deploys its own core at startup, and does not take part in the [core deployments](modules.md#replicas) of the aggregator.

Each window is sent in at most one request per metric family. A push that fails (aggregator unreachable, rejected key) is retried a few times on network errors, then logged and dropped: the worker does not buffer windows it could not send.

## Replicas

Instances sharing one database share their modules' state, and converge on the same [core deployment](modules.md#replicas) within a poll interval of a redeploy.

Sampling orchestration is not shared between instances, though: so each `standalone` or `worker` instance samples every asset it is given.

> ⚠️ In the near future Wombat will gain an orchestration layer for accommodating multi-tenant standalone/aggregator instances ; for now this is only partially supported. If this is something you need please reach out to us.

## Running it locally

Expand Down
123 changes: 123 additions & 0 deletions doc/modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# Modules and core deployments

A **module** provides an asset type: how its assets are described, sampled and evaluated. Wombat ships with core
modules, and loads [custom ones](../README.md#custom-extensions) from the jars of its extension folder. The **core** is
what monitors and evaluates assets: it is built out of the enabled modules, and rebuilt on each **redeploy**.

Modules are managed from **Settings > Modules**, or through `api/modules`.

```mermaid
sequenceDiagram
actor Admin
participant A as Instance A
participant DB as Database
participant B as Instance B
Admin->>DB: toggle a module (modules)
Note over A,B: asset pickers follow at once, the cores do not
Admin->>A: Redeploy core
A->>DB: record the enabled modules as version 8 (core_deployments)
A->>A: build core 8, swap it in
A->>DB: heartbeat: runs 8 (core_instances)
A-->>Admin: core a3f9c21 deployed here
loop every poll interval
B->>DB: latest version?
B->>B: build core 8, swap it in
B->>DB: heartbeat: runs 8
end
```

## The registry

Each instance detects its modules once, at startup: the core modules, and those of the jars found in
`wombat.module.extension.path`. A jar added or removed later is only seen after a restart.

Every detected module is **registered** in the database, enabled, along with the asset type it advertises. Its state is then the registry's, shared by every instance:

| State | Meaning |
|--------------|--------------------------------------------------------------------------------------------------------------------------|
| **Enabled** | Offered in the asset pickers, and held by the next core deployed. |
| **Disabled** | Hidden from the asset pickers, and left out of the next core deployed. Its assets are kept, and still accepted. |
| **Missing** | Registered, but no live instance detects it any more, e.g. its jar was removed. It keeps its state, but is unusable. |

A missing module comes back as it was when its jar does. It can also be deleted, in which case Wombat will forget its state: should its jar come back, it is registered once again, and enabled. A detected module cannot be deleted.

A module detected by another instance, but not by the one serving a page, is not considered missing: that instance only lacks its classes, so it cannot show its schema, nor validate its assets.

## What a module toggle changes, and when

| Where | When |
|--------------------------------------------------------------------|-------------------------------------------------------------------------|
| The asset pickers of the environment pages | At once |
| The metrics workers push to an aggregator | At once: those of a disabled module are skipped |
| Assets declared by a worker, or created through the API | Never: they are accepted whatever the module's state |
| Monitoring (sampling) and evaluation (Impact / Cost estimation) | After a core redeploy, on every instance |

Only a redeploy applies toggles to the core, **even across restarts**: a restarted instance runs the latest
deployment, not the registry. Settings > Modules tells the changes not deployed yet, and which core each instance runs.

## Core deployments

### Redeploying

**Redeploy core** (`POST /api/modules/deployment`) records the modules enabled at that moment under the next version,
then deploys it on the instance which received the request.

Deploying builds a new core and swaps it in: the calls already running finish on the previous core, which is closed a
minute later.

A redeploy resets the evaluation caches, and starts sampling afresh, as a restart does: a sampling run in progress during the swap may overlap the first run of the new core.

A redeploy with no change still rebuilds every core, which resets the caches and retries the instances that failed.

### Replicas

The database is the only channel between instances. Each instance:
* deploys the latest version at startup; the first one to start records version 1 out of the registry;
* every `wombat.module.deployment.poll-interval`, deploys the latest version if it runs an older one, and writes a
**heartbeat**: the version it runs, whether the latest one failed to build there, and the modules it detects.

Replicas thus converge on a redeploy within one poll interval. An instance which missed three heartbeats is considered gone: it is no longer listed, nor counted to tell whether a module is missing. Versions only go forward: an instance never deploys a version older than the one it runs.

Instances detecting different modules, because their extension folders differ, are flagged on the Modules page. A
deployment requesting a module an instance does not detect is still deployed there, without that module, which the
heartbeat reports.

### Failures

A core that fails to build is not swapped in: the instance keeps running its previous core, and its heartbeat reports
the failure and associated error, shown on the Modules page.
The poll does not retry a version that failed: fix the cause, then redeploy. Should the build fail at startup, the instance will not start at all, since there is no previous core to keep running.

Two redeploys requested at the same time on two instances record the same version once: the second gets `409`, and
every instance converges on the first.

### Workers

A [worker](distributed.md#workers) has no UI nor API, and its database is in memory: it enables every module it detects,
deploys its core at startup, and neither polls nor writes a heartbeat. It only declares the assets it samples. To skip
the metrics of a module, disable it on the aggregator.

## Configuration

| Setting | Environment variable | Default | Description |
|------------------------------------------|------------------------------------------|----------------------|---------------------------------------------------------------|
| `wombat.module.extension.path` | `WOMBAT_MODULE_EXTENSION_PATH` | `extensions` | Folder whose jars are loaded as modules, at startup |
| `wombat.module.deployment.poll-interval` | `WOMBAT_MODULE_DEPLOYMENT_POLL_INTERVAL` | `30s` | How often an instance deploys the latest core, and heartbeats |
| `wombat.module.deployment.instance-id` | `WOMBAT_MODULE_DEPLOYMENT_INSTANCE_ID` | `${HOSTNAME:wombat}` | Identifies an instance among the replicas |

The instance id is the pod name in Kubernetes, and `wombat` when `HOSTNAME` is unset: a restarted instance takes its
heartbeat back. Under plain Docker, `HOSTNAME` is the container id, which changes when the container is recreated; set the instance id to keep it.

## API and permissions

| Route | Permission | Does |
|---------------------------------|---------------------|---------------------------------------------------------------------------|
| `GET /api/modules`, `GET /{id}` | `api.module.read` | The modules, their state, whether the core serving the request holds them |
| `GET /api/modules/{id}/schema` | `api.module.read` | The JSON schema of the module's assets |
| `PUT /api/modules/{id}` | `api.module.manage` | Enables or disables a module (`{"enabled": false}`); `409` if missing |
| `DELETE /api/modules/{id}` | `api.module.manage` | Deletes a missing module; `409` if detected |
| `GET /api/modules/deployment` | `api.module.read` | The latest deployment, the live instances, the changes not deployed |
| `POST /api/modules/deployment` | `api.module.manage` | Redeploys the core; `409` if another redeploy was requested at once |

`api.module.manage` is held by admins, and can be granted to API keys. The Modules pages need `ui.modules.read`, held by every role; toggling, redeploying and deleting from them also need `ui.modules.manage`, held by admins. See
[access control](access-control.md).
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ public Response<Integer> push(
private boolean hasActiveModule(AssetIdentity asset)
{
return this.modules.getModule(asset.assetType())
.filter(ModuleMetadata::enabled)
.filter(ModuleMetadata::active)
.isPresent();
}

Expand Down
Loading
Loading