Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions content/validators/changelog/v0.6.0-rc.4.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## v0.6.0-rc.4

Before you upgrade, check the alert rules and readiness probes that read `/health`. A banned validator (`consensus.ban`) and an operator balance below the critical amount (`consensus.operator_funds`) are now `warning` with HTTP 200, not 503: read those conditions to alert on them. A chain RPC or WebSocket endpoint that fails every check for 30 seconds now returns 503 (`chain_rpc_unreachable`, `websocket_disconnected`), where it returned 200. An unreadable block (`sync_block_unreadable`) is `degraded` (200) for its first 5 minutes, and 503 after that.

### New features

- `genlayernode doctor --only config` checks only the config files, merged in `--config` order as `run` merges them: it sends no request and writes nothing, so it runs without network
- with `rollup.genlayerchainrpckeyfile` set, it also reads that key file and fails when the file is missing or empty
- The release signs the node image in both registries (`yeagerai/genlayer-node` and `ghcr.io/genlayerlabs/genlayer-node`) with cosign; `release/README.md` ("Verify the image") gives the `cosign verify` command
- The release LLM config has an Unhardcoded backend, first in the greybox text chain: set `UNHARDCODED_API_KEY` in `.env` to use it; without the key, the chains are as before

### Misc

- `release/docker-compose.yaml` pins the WebDriver, docker-socket-proxy and Alloy images by digest, and updates Alloy to v1.21.3

### Bug fixes

- A node no longer stops its sync and its duties at a block whose bloom filter falsely matches the node's contracts: it reads that block once by its hash and continues; before, it waited until the chain head moved 1000 blocks past the block, or until a restart
- `genlayer_node_bloom_mismatch_settled_by_hash_total` counts the blocks settled this way
- A transaction no longer stays in the commit phase with a vote missing when no validator outside the committee is online: a member that committed now calls `processIdleness` when the commit window ends
- A replacement leader no longer drops its propose duty when its first try reverts with `InvalidValidator` in the same block as the leader change: it tries again while the contract still names it the leader
- On a chain that makes blocks only when transactions arrive, the first block after a quiet period no longer sets the node to `hopelessly_behind` and pauses its sync, and a node at the chain head stays `synced` through a long quiet period

### Security

- `install.sh` and the `release/README.md` steps accept only a checksum signature made by the release workflow on a release line (`refs/heads/vX.Y`), not one made on any branch; use cosign v2.6.2 or later to verify
12 changes: 9 additions & 3 deletions content/validators/docker-compose-monitoring.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,16 @@
# from the new version's directory recreates the old version's containers.
name: genlayer-node

# Images from other projects are pinned by their multi-arch index digest
# (name:tag@sha256:...): Docker pulls the digest, the tag is for people. This
# file is inside the signed release tarball, so its SHA256SUMS signature covers
# these digests. The node image is signed with cosign by the release instead
# (release/README.md, "Verify the image").

services:
webdriver-container:
container_name: genlayer-node-webdriver
image: yeagerai/genlayer-genvm-webdriver:0.0.12
image: yeagerai/genlayer-genvm-webdriver:0.0.12@sha256:2de2cb87c851b9e634535ab94bc3f19202d1b5700f733fb255acd759518f37e8
shm_size: 2gb
security_opt:
- no-new-privileges:true
Expand Down Expand Up @@ -75,7 +81,7 @@ services:
# so keep DOCKER_HOST set to keep those panels populated. Disabling metrics
# permits removing the node's DOCKER_HOST; Alloy still needs this gateway.
docker-socket-proxy:
image: tecnativa/docker-socket-proxy:v0.5.0
image: tecnativa/docker-socket-proxy:v0.5.0@sha256:1f5038b54f06c3e18422902cf00ba21803d1c97805aae032e5e6673d532d3459
container_name: genlayer-node-docker-proxy
environment:
CONTAINERS: 1 # GET /containers/{json,inspect,stats} β€” read-only via POST=0
Expand Down Expand Up @@ -164,7 +170,7 @@ services:
# Set SCRAPE_TARGETS_JSON in .env
# docker compose --profile monitoring up -d
alloy:
image: grafana/alloy:v1.21.2
image: grafana/alloy:v1.21.3@sha256:8ab871206bd2d8f89277f53abffee5f452a293980788d52f85fa183a87a505a4
init: true
container_name: genlayer-node-alloy
labels:
Expand Down
9 changes: 7 additions & 2 deletions content/validators/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,15 @@
# Docker refuses with "container name ... is already in use". With it, `up -d`
# from the new version's directory recreates the old version's containers.
name: genlayer-node
# Images from other projects are pinned by their multi-arch index digest
# (name:tag@sha256:...): Docker pulls the digest, the tag is for people. This
# file is inside the signed release tarball, so its SHA256SUMS signature covers
# these digests. The node image is signed with cosign by the release instead
# (release/README.md, "Verify the image").
services:
webdriver-container:
container_name: genlayer-node-webdriver
image: yeagerai/genlayer-genvm-webdriver:0.0.12
image: yeagerai/genlayer-genvm-webdriver:0.0.12@sha256:2de2cb87c851b9e634535ab94bc3f19202d1b5700f733fb255acd759518f37e8
shm_size: 2gb
security_opt:
- no-new-privileges:true
Expand Down Expand Up @@ -73,7 +78,7 @@ services:
# so keep DOCKER_HOST set to keep those panels populated. Disabling metrics
# permits removing the node's DOCKER_HOST; Alloy still needs this gateway.
docker-socket-proxy:
image: tecnativa/docker-socket-proxy:v0.5.0
image: tecnativa/docker-socket-proxy:v0.5.0@sha256:1f5038b54f06c3e18422902cf00ba21803d1c97805aae032e5e6673d532d3459
container_name: genlayer-node-docker-proxy
environment:
CONTAINERS: 1 # GET /containers/{json,inspect,stats} β€” read-only via POST=0
Expand Down
1 change: 1 addition & 0 deletions content/validators/env.example
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ IOINTELLIGENCEKEY=
LIBERTAI_API_KEY=
MORPHEUS_API_KEY=
OPENAIKEY=
UNHARDCODED_API_KEY=

# Optional logging level override
GENLAYERNODE_LOGGING_LEVEL=INFO
Expand Down
8 changes: 4 additions & 4 deletions content/validators/greybox-setup-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ With **greybox**, the operator picks the models and their order by hand, in two

Greyboxing is deliberately per-validator β€” validators are expected to diverge in provider choice, so there is no single "correct" chain to converge on.

**Default text chain:** deepseek-v3.2 β†’ qwen3-235b β†’ claude-haiku-4.5 β†’ kimi-k2 β†’ glm-5 β†’ llama-3.3 (heurist) β†’ llama-3.3 (ionet)
**Default text chain:** Unhardcoded `route:genvm-validator` (only with `UNHARDCODED_API_KEY`) β†’ deepseek-v3.2 β†’ qwen3-235b β†’ claude-haiku-4.5 β†’ kimi-k2 β†’ glm-5 β†’ llama-3.3 (heurist) β†’ llama-3.3 (ionet)

**Default image chain:** gpt-5.1-mini β†’ gemini-3-flash β†’ claude-haiku-4.5
**Default image chain:** gpt-5.4-mini β†’ gemini-3-flash β†’ claude-haiku-4.5

Chain order is determined by the `meta.greybox` priority numbers on each model in the YAML. Lower number = higher priority. You can change the order by editing these numbers β€” no Lua changes needed.

OpenRouter is the primary aggregator. If it fails, the node falls back to direct provider APIs (heurist, ionet).
With `UNHARDCODED_API_KEY` set, the Unhardcoded route is tried first, and OpenRouter is next. OpenRouter is the primary aggregator otherwise. If it fails, the node falls back to direct provider APIs (heurist, ionet).

## Prerequisites

Expand All @@ -43,7 +43,7 @@ echo "OPENROUTERKEY=sk-or-v1-your-key-here" >> ${ENV_FILE}

### 3. Apply the release LLM config

The tarball ships with a unified config that includes all backends (openrouter, morpheus, heurist, ionet, etc.).
The tarball ships with a unified config that includes all backends (openrouter, unhardcoded, morpheus, heurist, ionet, etc.). A backend whose key is not set in `.env` is disabled at start, and the chains skip it: set `UNHARDCODED_API_KEY` to put the Unhardcoded route first in the text chain.

`NODE_DIR` is the directory of your node release (where you extracted the
tarball). On a host with the `/opt/genlayer-node/bin` symlink layout, it is
Expand Down
30 changes: 23 additions & 7 deletions pages/api-references/genlayer-node.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1900,7 +1900,20 @@ If **no** condition has been observed at all, `status` is `unknown` β€” nothing

A condition can never exceed the liveness cap its catalog entry declares. The GenVM conditions are capped at `degraded`, so a dead LLM provider returns **200**: that node is still booted, synced and voting β€” it votes Timeout on the transactions it cannot execute, which is the designed behaviour rather than an outage. Failing the endpoint would pull a working validator out of a load balancer.

`consensus.ban`, `chain.rpc`, `chain.websocket` and `chain.sync` are capped at `unavailable`, because each of them stops the node doing its job.
503 means the node cannot make progress or cannot act. Only four conditions can return it:

| Condition | Reason | 503 when |
|---|---|---|
| `chain.rpc` | `chain_rpc_unreachable` | The HTTP endpoint has failed every check for 30 seconds. The node cannot submit a duty, read a receipt or read its nonce. |
| `chain.websocket` | `websocket_disconnected` | The WebSocket endpoint has failed every check for 30 seconds. The node receives no new chain heads and no contract events. |
| `chain.sync` | `sync_blocked`, `sync_block_unreadable` | The chain head is above the synced block and the synced block has not moved for 5 minutes. See below. |
| `node.disk` | `disk_space_critical` | 2% or less of the disk is free. The node cannot write new blocks or receipts. |

A shorter transport failure is `suspect`, then `degraded` (HTTP 200). The checks run when a client calls this endpoint, so the 30 seconds are measured in time, not in checks.

`consensus.ban` and `consensus.operator_funds` are capped at `warning`. A banned node, and a node whose operator balance is below the critical amount (reason `operator_balance_critical`), still syncs and serves; leaving rotation or a restart fixes neither.

`consensus.prime` and `node.snapshot_age` are capped at `warning`.

`consensus.stake` is capped at `warning`: below the minimum self-stake the node keeps syncing, and only the consensus contracts do not select the validator for duties. The reason is `self_stake_pending` when a pending self-stake deposit covers the gap (it becomes stake at the validator's prime in a later epoch), and `self_stake_below_minimum` otherwise.

Expand Down Expand Up @@ -1947,14 +1960,14 @@ that owns the manager configuration. Observed `genvm.web` results still apply.
}
```

**Example: the validator is banned β€” HTTP 503**
**Example: the validator is banned β€” HTTP 200**

```json
{
"status": "unavailable",
"status": "warning",
"conditions": {
"consensus.ban": {
"state": "unavailable",
"state": "warning",
"reason": "validator_banned",
"impact": "The consensus contracts do not select this validator for a round. The node earns nothing.",
"action": "Wait for the stated epoch. If the ban is permanent, restake with a new validator.",
Expand All @@ -1971,14 +1984,17 @@ that owns the manager configuration. Observed `genvm.web` results still apply.

**`chain.sync` β€” 503 only when the sync is blocked**

Depth is not a fault. A node syncing from genesis is millions of blocks behind and healthy while its synced block advances. The node reports 503 only when its sync cannot get past a block.
Depth is not a fault. A node syncing from genesis is millions of blocks behind and healthy while its synced block advances. A quiet chain is not a fault either: when the chain head is at the synced block there is nothing to sync, however long no block arrives.

The sync is blocked when the chain head is above the synced block and the synced block has not moved for 5 minutes. The node reads the chain head with a request (`eth_blockNumber`), not only from its head subscription, so a subscription that stopped delivering heads is also seen. The 5 minutes start when there is a block to sync: a failure at a block, an unreadable block, or the first check that saw the head above the synced block.

| Reason | State | HTTP | When |
|---|---|---|---|
| `sync_lagging` | `warning` | 200 | 1–100 blocks behind for 30 s, or the sync cursor cannot be read |
| `sync_catching_up` | `degraded` | 200 | More than 100 blocks behind, at any depth |
| `sync_blocked` | `unavailable` | **503** | The sync has failed at the same block for 5 minutes (`details`: `failing_block`, `failures`, `failing_seconds`) |
| `sync_block_unreadable` | `unavailable` | **503** | The node cannot read the logs of a block (`details`: `unreadable_block`, `unreadable_seconds`) |
| `sync_block_unreadable` | `degraded` | 200 | The node cannot read the logs of a block, for less than 5 minutes (`details`: `unreadable_block`, `unreadable_seconds`) |
| `sync_blocked` | `unavailable` | **503** | The sync is blocked (`details`: `chain_head`, `blocked_seconds`; `failing_block`, `failures`, `failing_seconds` when the sync loop failed at a block) |
| `sync_block_unreadable` | `unavailable` | **503** | The sync is blocked below a block the node cannot read (`details`: as `sync_blocked`, plus `unreadable_block`, `unreadable_seconds`) |

The node never skips a duty because it is behind. It votes Timeout only on a transaction that reads state it has not synced; a new contract deploy still runs.

Expand Down
30 changes: 23 additions & 7 deletions pages/api-references/genlayer-node/ops/health.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,20 @@ If **no** condition has been observed at all, `status` is `unknown` β€” nothing

A condition can never exceed the liveness cap its catalog entry declares. The GenVM conditions are capped at `degraded`, so a dead LLM provider returns **200**: that node is still booted, synced and voting β€” it votes Timeout on the transactions it cannot execute, which is the designed behaviour rather than an outage. Failing the endpoint would pull a working validator out of a load balancer.

`consensus.ban`, `chain.rpc`, `chain.websocket` and `chain.sync` are capped at `unavailable`, because each of them stops the node doing its job.
503 means the node cannot make progress or cannot act. Only four conditions can return it:

| Condition | Reason | 503 when |
|---|---|---|
| `chain.rpc` | `chain_rpc_unreachable` | The HTTP endpoint has failed every check for 30 seconds. The node cannot submit a duty, read a receipt or read its nonce. |
| `chain.websocket` | `websocket_disconnected` | The WebSocket endpoint has failed every check for 30 seconds. The node receives no new chain heads and no contract events. |
| `chain.sync` | `sync_blocked`, `sync_block_unreadable` | The chain head is above the synced block and the synced block has not moved for 5 minutes. See below. |
| `node.disk` | `disk_space_critical` | 2% or less of the disk is free. The node cannot write new blocks or receipts. |

A shorter transport failure is `suspect`, then `degraded` (HTTP 200). The checks run when a client calls this endpoint, so the 30 seconds are measured in time, not in checks.

`consensus.ban` and `consensus.operator_funds` are capped at `warning`. A banned node, and a node whose operator balance is below the critical amount (reason `operator_balance_critical`), still syncs and serves; leaving rotation or a restart fixes neither.

`consensus.prime` and `node.snapshot_age` are capped at `warning`.

`consensus.stake` is capped at `warning`: below the minimum self-stake the node keeps syncing, and only the consensus contracts do not select the validator for duties. The reason is `self_stake_pending` when a pending self-stake deposit covers the gap (it becomes stake at the validator's prime in a later epoch), and `self_stake_below_minimum` otherwise.

Expand Down Expand Up @@ -131,14 +144,14 @@ that owns the manager configuration. Observed `genvm.web` results still apply.
}
```

**Example: the validator is banned β€” HTTP 503**
**Example: the validator is banned β€” HTTP 200**

```json
{
"status": "unavailable",
"status": "warning",
"conditions": {
"consensus.ban": {
"state": "unavailable",
"state": "warning",
"reason": "validator_banned",
"impact": "The consensus contracts do not select this validator for a round. The node earns nothing.",
"action": "Wait for the stated epoch. If the ban is permanent, restake with a new validator.",
Expand All @@ -155,14 +168,17 @@ that owns the manager configuration. Observed `genvm.web` results still apply.

**`chain.sync` β€” 503 only when the sync is blocked**

Depth is not a fault. A node syncing from genesis is millions of blocks behind and healthy while its synced block advances. The node reports 503 only when its sync cannot get past a block.
Depth is not a fault. A node syncing from genesis is millions of blocks behind and healthy while its synced block advances. A quiet chain is not a fault either: when the chain head is at the synced block there is nothing to sync, however long no block arrives.

The sync is blocked when the chain head is above the synced block and the synced block has not moved for 5 minutes. The node reads the chain head with a request (`eth_blockNumber`), not only from its head subscription, so a subscription that stopped delivering heads is also seen. The 5 minutes start when there is a block to sync: a failure at a block, an unreadable block, or the first check that saw the head above the synced block.

| Reason | State | HTTP | When |
|---|---|---|---|
| `sync_lagging` | `warning` | 200 | 1–100 blocks behind for 30 s, or the sync cursor cannot be read |
| `sync_catching_up` | `degraded` | 200 | More than 100 blocks behind, at any depth |
| `sync_blocked` | `unavailable` | **503** | The sync has failed at the same block for 5 minutes (`details`: `failing_block`, `failures`, `failing_seconds`) |
| `sync_block_unreadable` | `unavailable` | **503** | The node cannot read the logs of a block (`details`: `unreadable_block`, `unreadable_seconds`) |
| `sync_block_unreadable` | `degraded` | 200 | The node cannot read the logs of a block, for less than 5 minutes (`details`: `unreadable_block`, `unreadable_seconds`) |
| `sync_blocked` | `unavailable` | **503** | The sync is blocked (`details`: `chain_head`, `blocked_seconds`; `failing_block`, `failures`, `failing_seconds` when the sync loop failed at a block) |
| `sync_block_unreadable` | `unavailable` | **503** | The sync is blocked below a block the node cannot read (`details`: as `sync_blocked`, plus `unreadable_block`, `unreadable_seconds`) |

The node never skips a duty because it is behind. It votes Timeout only on a transaction that reads state it has not synced; a new contract deploy still runs.

Expand Down
Loading