Manual

N2API Manual

This manual contains the detailed deployment, client configuration, provider account, routing, and gateway operations guidance for N2API. The default deployment target is Docker Compose on a small VPS.

Start Locally

From the repository root:

make dev

The default app URL is http://localhost:3000. This zero-configuration path uses predictable development-only credentials. To customize them, copy .env.example to .env and replace every change-me value. The session runner loads the primary checkout's .env and keeps its backup directory when run from a task worktree.

Keep the foreground session open while using the app. Ctrl-C, TERM, or HUP removes development containers, network, and the locally built application image; the named database volume and backups remain. No development service restarts automatically. Use make dev-check for a rebuild/probe/teardown and make dev-stop to recover an abandoned detached stack. The Docker daemon stays available to other projects. See the development lifecycle.

An existing zero-configuration development volume created before startup validation still has the legacy PostgreSQL password stored inside the volume. Migrate that local role once without deleting data, then start the stack again:

docker compose -f deploy/compose.yaml exec postgres psql -U n2api -d n2api -v ON_ERROR_STOP=1 -c "ALTER ROLE n2api PASSWORD 'n2api-local-postgres-password'"
make dev

Do not run this fixed development-password command on a deployment with custom credentials. Update its .env and PostgreSQL role to the same operator-chosen random password instead.

Startup Configuration Safety

N2API validates security-sensitive configuration before opening its listener. N2API_PUBLIC_URL must be an absolute HTTP or HTTPS origin with no credentials, query, fragment, or non-root path. Administrator passwords must contain at least 12 bytes, encryption secrets at least 32 bytes, and the two values must be different. Known template values are rejected. PostgreSQL connection strings are parsed with the same pgx parser used at runtime, and startup errors name variables without echoing credentials or connection strings.

Some valid development and container topologies are intentionally unsafe. They must be acknowledged individually in the comma-separated N2API_ACCEPT_RISKS variable:

• public-http permits a non-loopback N2API_PUBLIC_URL to use HTTP.
• public-bind permits N2API_HOST to listen on a non-loopback address.
• database-plaintext permits a PostgreSQL primary or fallback connection without TLS.
• database-unverified-tls permits a PostgreSQL primary or fallback TLS connection that does not verify both the certificate chain and hostname.

Unknown values, empty list elements, and a blanket all value are rejected. These acknowledgements never bypass malformed URLs or connection strings, placeholder or short secrets, or identical administrator and encryption secrets. The development Compose file supplies public-http, public-bind, and database-plaintext by default and uses separate local-only credentials when no .env exists. Set N2API_ACCEPT_RISKS explicitly in .env to narrow or replace those development defaults.

OPENAI_API_BASE_URL is parsed at startup. The API base URL may use HTTP only when N2API_ALLOW_HTTP_API_UPSTREAMS=true, matching the explicit opt-in used for API-key upstream accounts.

Prometheus Metrics

Prometheus metrics are disabled by default. When enabled, N2API creates a private Prometheus registry and serves /metrics from a separate listener; the gateway, admin API, probes, and static router never expose this handler. The default listener is 127.0.0.1:9090. A non-loopback N2API_METRICS_HOST is rejected unless N2API_METRICS_BEARER_TOKEN is also set. The token protects only the metrics listener and must not reuse an administrator password, API key, provider credential, or encryption secret.

For a scrape from inside the application container, keep the loopback default:

N2API_METRICS_ENABLED=true
N2API_METRICS_HOST=127.0.0.1
N2API_METRICS_PORT=9090
N2API_METRICS_BEARER_TOKEN=

docker compose -f deploy/compose.yaml exec -T n2api \
  wget -q -O - http://127.0.0.1:9090/metrics

For a host-side Prometheus process, set a dedicated random token in .env and apply the metrics override. The override binds the process inside the container to 0.0.0.0, requires bearer authentication, and publishes the port on host loopback by default:

N2API_METRICS_BEARER_TOKEN=replace-with-a-dedicated-random-token
N2API_METRICS_BIND_ADDRESS=127.0.0.1
N2API_METRICS_PORT=9090

N2API_DEV_METRICS=1 make dev

Keep the foreground session open, and scrape from another terminal:

curl --fail --silent --show-error \
  --header "Authorization: Bearer ${N2API_METRICS_BEARER_TOKEN}" \
  http://127.0.0.1:9090/metrics

Ctrl-C closes the metrics development session and removes its containers and network. make dev-stop also recognizes older detached metrics stacks.

Use deploy/compose.metrics.yaml after deploy/compose.release.yaml for an immutable image deployment. Publishing metrics beyond host loopback requires an explicit N2API_METRICS_BIND_ADDRESS plus firewall policy; bearer authentication remains mandatory. Do not send the bearer token over an untrusted network without a TLS reverse proxy or private encrypted transport. Missing or incorrect credentials return HTTP 401.

Metrics are process-local and reset on restart. Labels use fixed allowlists and omit request, key, account, pool, model, IP, token, body, and error details. Request Logs and System Events remain the durable records. Provider inventory is refreshed into memory once per minute, and a scrape reads only that snapshot, in-process counters, and pgxpool.Stat(). A PostgreSQL failure can make the inventory snapshot stale but does not prevent scraping. N2API emits no exemplars, tracing, remote export, Grafana configuration, or outbound telemetry.

n2api_api_key_last_used_total reports fixed updated, skipped, failure, or other outcomes for the conditional API key last-used write. It never uses key identity or database error text as a label.

Reverse Proxy Trust

N2API ignores X-Forwarded-For, X-Real-IP, X-Forwarded-Proto, and X-Forwarded-Host by default. If a reverse proxy connects directly to N2API, set N2API_TRUSTED_PROXY_CIDRS to the proxy's exact address or network. For a same-host proxy using the loopback interface:

N2API_TRUSTED_PROXY_CIDRS=127.0.0.1/32,::1/128

N2API_PUBLIC_URL is the canonical scheme and host for direct requests. A trusted proxy may override them with one valid X-Forwarded-Proto and X-Forwarded-Host value; untrusted requests cannot replace the configured origin with either forwarding headers or a forged HTTP Host value.

For a containerized proxy, use its actual Compose network CIDR after verifying the direct peer address seen by N2API. Do not configure 0.0.0.0/0 or ::/0. An invalid CIDR prevents startup instead of silently weakening the boundary.

In a multi-hop chain, X-Forwarded-For is ordered from the original client on the left to the proxy nearest N2API on the right. N2API walks the list from right to left, skips only configured trusted proxy hops, and uses the first untrusted address as the client. Every proxy hop that should be skipped must be listed in N2API_TRUSTED_PROXY_CIDRS; malformed chains are ignored in favor of the direct peer, including scheme and host metadata. If every visible hop is trusted, the farthest visible address is recorded. The public edge proxy must overwrite or sanitize client-sent forwarding headers before appending its own hop, and the proxy nearest N2API must overwrite X-Forwarded-Proto and X-Forwarded-Host as single values rather than comma-separated lists. The standardized Forwarded header is not currently used.

Administrator Login Protection

Administrator login throttling is enabled by default. N2API tracks failed attempts independently by normalized client IP and lowercase username; either dimension can temporarily deny a request. After five failures, denial starts at one second and doubles up to 60 seconds. Entries expire after 15 minutes of inactivity, the combined map is bounded to 4096 entries, and a successful login clears both identities. Concurrent password checks are reserved atomically, so an initial burst cannot exceed the configured threshold. Rejected requests keep the same invalid_credentials response body for known and unknown usernames. The threshold response includes Retry-After, and requests made during the denial period return HTTP 429 with an integer Retry-After value.

Use N2API_ADMIN_LOGIN_THROTTLE_FAILURES to set the threshold from 1 to 20 and N2API_ADMIN_LOGIN_THROTTLE_MAX_ENTRIES to set the memory bound from 128 to 16384. When the bound is exhausted, new identities fail closed instead of evicting active failure state. Set N2API_ADMIN_LOGIN_THROTTLE_ENABLED=false only as a temporary rollback. State is process-local and resets after a restart, which matches the single-node deployment model. Reverse-proxy deployments must configure N2API_TRUSTED_PROXY_CIDRS correctly so the IP dimension observes the client rather than the nearest proxy.

Repeated login failures are globally aggregated into one System Event per one-minute window while throttling is enabled. The event uses a fixed administrator target; usernames, passwords, and request bodies are never stored in those events.

Administrator sessions expire after seven days by default. Set N2API_ADMIN_SESSION_TTL_HOURS from 1 to 8760 to change the absolute lifetime of newly created sessions. Changing the value does not extend or shorten sessions that already exist, and activity never extends a session's expiry. Open Active sessions from the administrator menu to inspect active login sessions, revoke one session, or revoke every session except the current one. Revoking the current row signs out that browser immediately. Session rows show only bounded client metadata: the creation IP is reduced to an IPv4 /24 or IPv6 /64 network and the User-Agent is cleaned and truncated. Authentication tokens and token hashes are never returned by the session API.

Changing the administrator password keeps the authenticated session that submitted the change and atomically revokes every other active administrator session. The password dialog reports the number revoked. If the password update, session revocation, or security event cannot commit, none of them take effect.

Browser Request Security

N2API rejects unsafe browser requests that carry the administrator session cookie unless Origin matches the trusted external request origin and Fetch Metadata, when present, reports Sec-Fetch-Site: same-origin. Requests from scripts and CLI tools remain compatible when they omit both browser headers. The logout endpoint applies the same browser checks even when no session cookie is present, because its response still changes browser cookie state. Configure N2API_PUBLIC_URL or N2API_TRUSTED_PROXY_CIDRS correctly so N2API can derive the external scheme and host without trusting client-supplied proxy headers.

All responses include MIME sniffing, referrer, permissions, framing, and Content Security Policy headers. HSTS is emitted only when the trusted external scheme is HTTPS. Admin API responses use Cache-Control: no-store. The CSP permits only same-origin resources and data: images. SvelteKit emits build-specific SHA-256 hashes for the static bootstrap script and generated style elements. Inline style attributes remain allowed because the current UI requires them; remote scripts are not allowed.

Container Runtime Identity

The N2API application image runs as the fixed n2api identity with UID and GID 10001. The binary and static admin assets are owned by root:n2api with no write bits, and the runtime user has no home directory or login shell. The image retains the Alpine CA bundle for HTTPS upstream verification but does not need a persistent writable application path.

Inspect a running Compose container without printing its environment:

docker compose -f deploy/compose.yaml exec n2api id
docker compose -f deploy/compose.yaml exec n2api stat -c '%u:%g %a %n' /app/n2api /app/frontend/build/200.html

The first command must report uid=10001(n2api) gid=10001(n2api). The image smoke matrix also verifies readiness, static assets, the CA bundle, application file write denial, and a clean SIGTERM exit within ten seconds on both supported platforms. Future bind mounts must be readable by UID/GID 10001 and must not make application files writable.

Container Runtime Restrictions

The development and release Compose definitions run only the N2API application container with a read-only root filesystem, all Linux capabilities dropped, and no-new-privileges enabled. A writable 16 MiB /tmp tmpfs is available to the unprivileged application identity with noexec, nosuid, and nodev; it is ephemeral and must not be used for persistent data. PostgreSQL is intentionally excluded because its official image requires a persistent writable data path.

N2API uses the unless-stopped restart policy. Compose allows 35 seconds after SIGTERM, longer than N2API's 25-second global shutdown deadline and 20-second request-drain allocation, before Docker sends SIGKILL. Release Compose also sets separate resource defaults:

Service	CPU	Memory	PIDs	nofile soft/hard	/tmp	JSON logs
N2API	1.0	512 MiB	256	4096/8192	16 MiB	10 MiB x 3
PostgreSQL	1.0	768 MiB	256	4096/8192	64 MiB	10 MiB x 3

PostgreSQL additionally receives 128 MiB shared memory. Override the matching N2API_* and POSTGRES_* resource variables only after measuring the host; keep application and database budgets independent. Inspect the effective restrictions without printing container environment variables:

docker inspect "$(docker compose -f deploy/compose.yaml ps -q n2api)" \
  --format '{{.HostConfig.ReadonlyRootfs}} {{.HostConfig.SecurityOpt}} {{.HostConfig.CapDrop}} {{json .HostConfig.Tmpfs}}'

The output must report a read-only root filesystem, no-new-privileges:true, capability drop ALL, and a bounded /tmp tmpfs. Writes outside /tmp must fail. Bind mounts added by operators are separate writable surfaces and should remain read-only unless the application has a documented persistence requirement.

Published Images

The CI Image workflow runs backend/frontend tests, type checks, and ordinary correctness gates for pull requests and main. These runs do not build frontend delivery assets or container images. Security scans remain enabled separately. Release publish calls the same workflow with delivery builds enabled: it runs the mock, gateway, SDK, and PostgreSQL fault checks, then builds and smoke-tests both supported image platforms. The tested multi-platform image receives two source tags before stable promotion:

• main moves to the tested release-build commit when it is still the current default-branch commit; ordinary pushes do not update this tag.
• sha-<12 characters> identifies one tested source commit and is immutable.

Stable promotion adds two more tags without rebuilding that tested image:

• YYYYMMDDNN is an immutable Europe/Berlin CalVer release, for example 2026071401.
• latest moves only when a stable GitHub Release is published.

The Git tag, GitHub Release tag, and container version tag always use the same CalVer value without a v prefix.

Published images support:

• linux/amd64
• linux/arm64

All repository-owned container builds pin readable dependency versions and their multi-platform manifest digests. The current baseline is Bun 1.4.2, Go 1.26.10 on Alpine 3.23, Alpine 3.24.1, and PostgreSQL 18.6 on Alpine 3.23. E2E-only uv and Python images are pinned the same way. A version tag communicates the intended dependency while the digest makes the resolved bytes reproducible on both supported architectures. Update both together; never replace a digest without verifying that it belongs to the adjacent tag.

Dependabot checks Go modules, Bun workspaces, Python contract dependencies, GitHub Actions, Dockerfiles, and Compose definitions every Monday. Patch and minor updates are grouped by ecosystem; major updates remain isolated for explicit review. Every dependency pull request must pass the normal CI Image workflow. Container update pull requests must also pass dev/ci/verify-pinned-dependencies.sh and keep readable version tags next to digests. Both image architectures are smoke-tested during release publication. Static frontend assets are built on the builder's native architecture for both target platforms, avoiding JavaScript tooling under ARM64 emulation.

Every tested image carries the same build identity in the binary and OCI metadata:

• org.opencontainers.image.source identifies the source repository.
• org.opencontainers.image.revision is the complete Git commit SHA.
• org.opencontainers.image.version is the immutable sha-<12 characters> build version.
• org.opencontainers.image.created is the source commit time normalized to UTC RFC 3339.

The commit time, rather than the workflow start time, keeps rebuilds of the same commit traceable to one stable identity. CalVer releases promote the already-tested manifest without rebuilding it, so a running release continues to report its source sha-<12 characters> build version while the CalVer tag identifies the promoted manifest.

For every release source image, CI generates separate SPDX JSON SBOMs and Trivy vulnerability reports for linux/amd64 and linux/arm64. Both evidence jobs read the same immutable parent manifest reference, IMAGE@sha256:..., and select their platform without rebuilding the image. Each platform SBOM is attested to that parent manifest digest in GHCR. CI also retains the SBOM, Trivy JSON, and a non-sensitive metadata file naming the parent digest, platform, and evidence filenames for 14 days.

Unexcepted HIGH and CRITICAL findings with a non-empty Trivy FixedVersion block the platform evidence job. Unfixed findings remain report-only so a base image is not permanently blocked when no remediation exists. Exact CVE or package exceptions may be recorded per platform in security/exceptions.json; they require an owner, reason, and expiry no more than 30 days after creation. Scanner execution, invalid JSON or schema, exception-registry validation, attestation, and artifact upload errors fail closed. Release publish runs verify that the tested parent digest has an SPDX attestation issued by this repository's reusable image-evidence.yml workflow for the selected source commit. Evidence generation first verifies the original CI Image build provenance for that digest and commit; stable tags continue to promote that same digest without rebuilding it.

The read-only Stable Image Security workflow runs weekly and on manual dispatch. It resolves the latest non-draft, non-prerelease GitHub Release, requires its ten-digit CalVer tag and latest image tag to resolve to the same two-platform manifest, then scans that exact IMAGE@digest for both platforms. It does not build, retag, or publish an image and does not create issues. Platform SBOM, Trivy, and relationship metadata artifacts are retained for 14 days; the same fixed HIGH/CRITICAL and expiring-exception policy applies.

Release 2026071401 predates multi-platform publishing and supports only linux/amd64. ARM64 hosts must use a later release. Inspect any tag before deployment with:

N2API_VERSION=YYYYMMDDNN
docker buildx imagetools inspect "ghcr.io/knowsky404/n2api:${N2API_VERSION}"

Release Update Notifications

N2API checks the latest published release from the official KnowSky404/N2API GitHub repository and shows an authenticated administrator a quiet global notice when the running commit is behind that release. Open Release updates from the administrator menu at any time to inspect the current and latest build identities, sanitized release notes, GitHub Release link, and immutable image reference. Dismissing the global notice applies only to that exact release in the current browser; a later release is shown again.

The backend performs the first check asynchronously at startup and refreshes its cache every six hours. The browser reads that cached status every five minutes while an administrator session is active. The modal's refresh control requests an immediate bounded check and has a one-minute cooldown. Failed GitHub requests never affect liveness, readiness, gateway traffic, or database health; after a successful check, the last result remains visible as cached.

Update discovery is read-only. It sends no credentials, deployment configuration, provider data, or administrator identity to GitHub, and the browser never contacts GitHub directly. Set N2API_UPDATE_CHECK_ENABLED=false to disable all release-check traffic; the administrator surface will report that checks are disabled.

The notification does not deploy, restart, back up, restore, or roll back the service. Review the release and immutable image, then use the guarded upgrade plan and reviewed apply workflow. Never grant the web service access to the Docker socket for this feature.

Preview and Publish a Release

Open Actions > Release > Run workflow on the main branch. Keep mode set to preview first. Preview mode calculates the next CalVer, generates release notes from Conventional Commits, and uploads the proposed Release body without requiring an existing source image. The manifest digest remains pending until publication. It does not build images or create Git tags, container tags, or GitHub Releases. A later run may select a newer main commit, so a standalone preview is informational rather than a locked release candidate.

The publish run's generated Release body records that the two platform SBOM attestations and report-only vulnerability evidence are tied to the tested manifest digest. The attestations remain associated with the immutable image in GHCR; the downloadable per-platform evidence artifacts are retained with the originating Release run for 14 days.

Complete the release checklist for the exact candidate before publish approval. In particular, record the last successful real-backup restore drill and every immutable image tag or digest exercised by that drill. Generated CI fixtures do not satisfy this operator recovery gate.

For an approval gate, configure the repository's release environment with a required reviewer. Run the workflow with mode set to publish; it builds and verifies the selected source image before creating a fresh release-preview-* artifact. A readable existing source image is reused without rebuilding it; the original build provenance is verified and both platform SBOM/vulnerability checks run again before the same identity and SPDX attestation checks. If the original build provenance is missing, rerun its failed manifest/provenance job before retrying publication. The publish job then waits for environment approval, consumes the artifact from the same workflow run, and promotes the tested manifest. Candidate sha-* and main tags may already exist while approval is pending; stable tags, Git tags, and GitHub Releases remain unchanged until publication is approved. After approval it creates the Git tag, promotes the tested digest to the CalVer tag, publishes the GitHub Release, and only then moves latest. The workflow never rebuilds the image and refuses to replace a CalVer tag that points to another commit or manifest digest. A rerun can repair latest after a partial failure without replacing an existing Release.

Install Docker on Ubuntu 24.04 ARM64

Remove conflicting distribution packages, then install Docker Engine, Buildx, and Compose from Docker's official apt repository:

mapfile -t conflicting_packages < <(
  dpkg --get-selections \
    docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc \
    2>/dev/null \
    | awk '$2 == "install" { print $1 }'
)
if [ "${#conflicting_packages[@]}" -gt 0 ]; then
  sudo apt remove -y "${conflicting_packages[@]}"
fi
sudo apt update
sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo docker run --rm hello-world

Confirm that the host and Docker installation report ARM64 and Compose v2:

uname -m
dpkg --print-architecture
docker compose version

Expected architecture values are aarch64 and arm64. Docker daemon access is root-equivalent; either keep using sudo for Docker commands or grant access only to a trusted administrator.

Deploy a Published Image

The agent-native operations runbook is the canonical production operations entry point. Use ./ops/n2api describe --format json, then the documented doctor, configuration, and deploy plan/apply sequence. The CLI validates exact image manifests, protects secrets and operation state, rejects stale plans, and records sanitized receipts.

Lower-level Compose fallback

The commands below explain the underlying release inputs for an owner-managed diagnostic or recovery path. They are not the default Agent workflow and must not be used to bypass the canonical CLI's plan, evidence, lock, exact-digest, secret, or receipt contracts.

Check out the release being deployed, copy the example environment file, and pin the same immutable version in .env:

git clone https://github.com/KnowSky404/N2API.git
cd N2API || exit 1
N2API_VERSION=YYYYMMDDNN
git checkout "$N2API_VERSION"
cp .env.example .env
N2API_MANIFEST_DIGEST=sha256:<64-lowercase-hex-characters>
sed -i "s|^N2API_IMAGE=.*|N2API_IMAGE=ghcr.io/knowsky404/n2api:${N2API_VERSION}@${N2API_MANIFEST_DIGEST}|" .env

Replace every change-me value before starting the stack. At minimum, set:

• POSTGRES_PASSWORD to a random value and use the same value in DATABASE_URL. A hex value from openssl rand -hex 32 avoids URL-encoding ambiguity.
• N2API_ADMIN_PASSWORD to a unique strong password.
• N2API_ENCRYPTION_SECRET to a separate long random value. Losing this value makes encrypted provider credentials unrecoverable.
• N2API_PUBLIC_URL to the externally visible origin, including https:// when TLS terminates in front of N2API.
• N2API_ACCEPT_RISKS=public-bind,database-plaintext when using the bundled release topology. The application must listen on the container network, and the bundled PostgreSQL service does not enable TLS. For an external PostgreSQL service, use sslmode=verify-full with a trusted root certificate and omit both database transport risks. sslmode=require and sslmode=verify-ca require the independent database-unverified-tls acknowledgement because they do not verify the server hostname.

N2API_IMAGE must retain both the readable tag and immutable digest. Validate the reference before deployment without pulling or inspecting any secret:

IMAGE_REF="ghcr.io/knowsky404/n2api:${N2API_VERSION}@${N2API_MANIFEST_DIGEST}"
dev/verification/verify-release-image.sh --syntax-only "$IMAGE_REF"

For file-backed core secrets, create owner-readable regular files outside the repository and set N2API_DATABASE_URL_SOURCE_FILE, N2API_ADMIN_PASSWORD_SOURCE_FILE, N2API_ENCRYPTION_SECRET_SOURCE_FILE, and N2API_POSTGRES_PASSWORD_SOURCE_FILE in .env. Then include the core override:

docker compose --env-file .env \
  -f deploy/compose.release.yaml \
  -f deploy/compose.release.secrets.yaml config --quiet
docker compose --env-file .env \
  -f deploy/compose.release.yaml \
  -f deploy/compose.release.secrets.yaml up -d

Metrics file auth additionally requires deploy/compose.metrics.yaml, deploy/compose.metrics.secrets.yaml, and N2API_METRICS_BEARER_TOKEN_SOURCE_FILE. N2API also accepts custom mounts for N2API_ENCRYPTION_PREVIOUS_KEYS_FILE and OPENAI_OAUTH_CLIENT_SECRET_FILE. Never set a direct secret and its _FILE alternative together. Files are limited to 64 KiB, must be regular files rather than symlinks, pipes, or directories, and lose at most one final LF or CRLF.

Encrypted Secret Envelope

New encrypted provider credentials and reusable client-key secrets use the self-describing format n2api:v1:<key-id>:<secret-kind>:<payload>. The version, non-secret key ID, and fixed credential kind are authenticated with the ciphertext. Existing unversioned values remain readable and are not rewritten during an ordinary upgrade.

N2API_ENCRYPTION_KEY_ID identifies the current N2API_ENCRYPTION_SECRET and defaults to default so existing deployments can upgrade without a new required setting. IDs contain only ASCII letters, digits, ., _, or - and are at most 64 characters. Keep N2API_ENCRYPTION_PREVIOUS_KEYS=[] outside an explicit rotation window. During a rotation window it is an ordered JSON array:

N2API_ENCRYPTION_KEY_ID=current-202607
N2API_ENCRYPTION_PREVIOUS_KEYS='[{"id":"default","secret":"<previous-secret>"}]'

The current and previous secrets must each be at least 32 bytes, must be unique, and must differ from the administrator password. At most eight previous keys are accepted. New writes always use the current key. A versioned envelope uses only its named key and must match the credential kind expected by its consuming field; moving an access-token envelope into a refresh-token, proxy, or API-key field is rejected. Only legacy raw-base64 ciphertext tries current and then previous keys in configured order. Invalid versions, missing keys, kind mismatches, or authentication failures stop the credential operation without exposing key material, plaintext, or ciphertext. An unreadable encrypted proxy also stops the affected outbound operation instead of silently bypassing the proxy.

Legacy raw-base64 values predate credential-kind binding, so they retain only the original GCM integrity guarantee until Task 3 re-encrypts them. Version 1 binds the credential kind but not a database row identity; same-kind row substitution remains outside this task's protection boundary.

Before changing encryption keys, take a database backup and complete the isolated restore drill with the exact image and keyring that created it. Then verify every non-empty reversible credential in the live database:

docker compose -f deploy/compose.yaml exec -T n2api \
  /app/n2api admin verify-encryption

verify-encryption is a read-only dry run. It does not run migrations, bootstrap an administrator, rewrite credentials, or start HTTP/background services. Its single JSON document always contains all eight credential types, including zero-count types. Counts cover OAuth code verifiers; provider access, refresh, and ID tokens; provider API keys and proxy URLs; and reusable client API-key secrets and alert action destinations. Every non-empty value has one of these lifecycle states:

• readable_current_key
• readable_previous_key
• readable_legacy
• unreadable_required
• unreadable_expired_or_purgeable
• unreadable_unknown

Only an OAuth code verifier with an explicit expired expires_at or non-null consumed_at can be classified as purgeable. An active verifier is required; provider credentials, reusable client-key secrets, proxy credentials, and alert destinations are always required. Missing lifecycle evidence fails closed as unknown. The report includes an authenticated key ID only after successful decryption. For legacy values, that ID is the key that actually decrypted the value, not an inferred default.

The shortened example below shows one type entry and one failure shape; actual output is never abbreviated and always includes all eight type entries.

{
  "status": "failed",
  "count": 2,
  "lifecycleCounts": [
    {"lifecycleStatus": "readable_previous_key", "count": 1},
    {"lifecycleStatus": "unreadable_required", "count": 1}
  ],
  "types": [
    {
      "table": "provider_account_credentials",
      "credentialKind": "oauth-refresh-token",
      "count": 2,
      "lifecycleCounts": [
        {"lifecycleStatus": "readable_previous_key", "count": 1},
        {"lifecycleStatus": "unreadable_required", "count": 1}
      ]
    }
  ],
  "values": [
    {
      "table": "provider_account_credentials",
      "credentialKind": "oauth-refresh-token",
      "rowId": 41,
      "envelopeFormat": "v1",
      "authenticatedKeyId": "previous-202606",
      "lifecycleStatus": "readable_previous_key",
      "reasonCode": "previous_key_verified"
    },
    {
      "table": "provider_account_credentials",
      "credentialKind": "oauth-refresh-token",
      "rowId": 42,
      "envelopeFormat": "v1",
      "lifecycleStatus": "unreadable_required",
      "reasonCode": "credential_required"
    }
  ]
}

The report never includes plaintext, ciphertext, provider identity, key prefix, state hash, unauthenticated key IDs, or raw crypto/database errors. Exit code 0 means there is no required or unknown unreadable value, including an empty database or a report containing only explicitly purgeable OAuth states. Such a report uses top-level status attention. Exit code 1 means at least one unreadable_required or unreadable_unknown value blocks rotation. Exit code 2 means command usage, configuration, database access, query, or output failed. Do not begin re-encryption while a blocking unreadable value exists or before the backup restore drill succeeds.

Before any future re-encryption implementation can run, perform the read-only rotation preflight. The current key ID must be explicitly configured, the previous keyring must contain at least one key, and the real backup must have been both created and successfully restored in isolation within the previous 24 hours. Use the non-sensitive identifier from the operator restore drill record, never a dump path, public or signed URL, credential, or storage secret:

BACKUP_RECORD_ID=restore-record-YYYYMMDD-NN
BACKUP_CREATED_AT=YYYY-MM-DDTHH:MM:SSZ
BACKUP_RESTORED_AT=YYYY-MM-DDTHH:MM:SSZ
docker compose -f deploy/compose.yaml exec -T n2api \
  /app/n2api admin check-encryption-rotation \
  --backup-id "$BACKUP_RECORD_ID" \
  --backup-created-at "$BACKUP_CREATED_AT" \
  --backup-restored-at "$BACKUP_RESTORED_AT"
unset BACKUP_RECORD_ID BACKUP_CREATED_AT BACKUP_RESTORED_AT

The preflight obtains a dedicated exclusive PostgreSQL session advisory lock and reruns the complete inventory through that same connection. It reports a fixed ordered list of checks, stable reason codes, unreadable counts, the non-sensitive backup record identifier, and the confirmation timestamps. An invalid identifier is not echoed. Status ready and exit code 0 require every gate to pass. Status blocked or contended returns exit code 1; usage, configuration, database, cancellation, unlock, or output failure returns exit code 2 with only a stable error code.

This command is always a dry run. It does not run migrations, rewrite a secret, change a checkpoint, insert a System Event, or otherwise mutate business data. It is the mandatory gate for a future resumable re-encryption implementation, not that implementation and not evidence that key rotation has completed.

Expired OAuth authorization records contain short-lived encrypted PKCE code verifiers. Review them with an explicit UTC cutoff; the command is a dry run unless --execute is present:

OAUTH_STATE_CUTOFF="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
docker compose -f deploy/compose.yaml exec -T n2api \
  /app/n2api admin cleanup-expired-oauth-states \
  --cutoff "$OAUTH_STATE_CUTOFF" --batch-size 1000

The JSON report contains only the cutoff, batch size, eligible count, deleted count, batch count, and dry_run, completed, or contended status. Inspect the eligible count, then repeat the same immutable cutoff with --execute:

docker compose -f deploy/compose.yaml exec -T n2api \
  /app/n2api admin cleanup-expired-oauth-states \
  --cutoff "$OAUTH_STATE_CUTOFF" --batch-size 1000 --execute
unset OAUTH_STATE_CUTOFF

The cutoff cannot be in the future. A row is eligible only when its expires_at is earlier than the cutoff or its non-null consumed_at is earlier than the cutoff. Active authorization states remain untouched. Deletes commit in bounded transactions under a dedicated PostgreSQL session advisory lock; another concurrent worker returns contended without deleting. Repeating a completed command is safe and reports zero deleted rows. A real run records the sanitized oauth.state_cleanup.completed System Event. PostgreSQL, event-write, lock-release, and cancellation failures emit only a structured stable error code; database details and encrypted verifier values are never logged.

Exit code 0 means the dry run or cleanup completed, 1 means another cleanup worker holds the advisory lock, and 2 means invalid arguments, cancellation, database access, cleanup, event recording, lock release, or output failed.

Alert Rules And Delivery

The database stores notification actions and exact-match System Event rules. Action destinations use the dedicated alert-action-destination encryption kind; action and rule reads return only whether a destination is configured, never its plaintext or ciphertext. Supported action records are generic_webhook and ntfy. Rules can filter by category, severity, and action, aggregate within a fixed window, apply a cooldown, deduplicate by rule or event target, and optionally recognize one explicit recovery action.

Each rule is limited to 1024 deduplication states. When the limit is reached, only the oldest idle state may be evicted; active firing state is never silently discarded. Event evaluation and state admission are serialized per rule so concurrent events cannot lose aggregation counts or duplicate a notification decision. Updating a rule atomically clears its prior aggregation and firing state. The schema creates no action or rule by default.

Authenticated owners manage this configuration from the compact /alerting admin page or the /api/admin/alert-actions and /api/admin/alert-rules endpoints. Action responses expose only destinationConfigured; they never return the destination or its ciphertext. Action and rule updates require the current expectedUpdatedAt revision and return 409 stale_update when another edit won the race. An action cannot be deleted while a rule references it. Successful create, update, and delete operations commit their audit System Event in the same PostgreSQL transaction as the configuration change.

The rule editor also exposes a server-owned template catalog at GET /api/admin/alert-rule-templates. Installing a template requires an existing delivery action through POST /api/admin/alert-rule-templates/{key}/install. Installation is explicit, creates the rule disabled, and is idempotent by the persisted template key: retries return the existing rule without changing its delivery action, enabled state, or later owner edits. Deleting a template-derived rule is allowed; it is created again only when an owner explicitly installs the template again. The catalog includes request-log-retention-failed-v1, which matches either complete or partial automatic Request Log retention failures, uses a 24-hour cooldown, and can notify when the next scheduled retention cycle succeeds. Shutdown cancellation updates task status but does not emit the failure signal. Disabling the runner, disabling the saved retention policy, persistent lock contention, or failure to record a System Event can leave a firing rule without a recovery event. The provider-auto-test-failed-v1 template matches both complete and partial Provider account auto-test cycle failures, not individual probe results. It requires two failed cycles within 15 minutes, uses a one-hour cooldown, and can notify when the next scheduled cycle completes successfully. Normal shutdown cancellation updates task status without emitting a failure. The rule starts disabled, and disabling the auto-test runner or failing to record a System Event can leave a firing rule without a recovery event. The provider-account-expired-v1 template matches the first provider_account.expired Runtime warning for each Provider account, uses a 24-hour cooldown, and can notify when that same account emits provider_account.recovered. It starts disabled and is deduplicated by account target. API-upstream configuration edits preserve the current health state, so they do not recover a firing rule. Recovery follows a confirmed 2xx gateway/account-probe response or an explicit Reset local status operator override. The provider-account-circuit-open-v1 template matches the first provider_account.circuit.opened Runtime warning for each Provider account, uses a one-hour cooldown, and can notify when that same account emits provider_account.recovered. It also starts disabled and uses account-target deduplication. Circuit-open state remains active through credential or proxy edits; a confirmed 2xx gateway/account-probe response or explicit Reset local status operator override supplies the recovery event.

API Key budget threshold source events are projected from the durable budget ledger by an always-on monitor that runs once at startup and every five minutes. Each cycle processes at most 100 keys and each key is locked and evaluated in its own PostgreSQL transaction. Strict request-count budgets and observed token and estimated-cost budgets for rolling 24-hour and 30-day windows are independent. Crossing 80 percent emits api_key.budget.threshold_80.crossed; crossing 100 percent emits api_key.budget.threshold_100.crossed. A direct jump to 100 percent emits both. The corresponding .recovered action is emitted only when that exact stream falls below its threshold. Setting a budget to 0, raising its limit, or aging usage out of the rolling window can therefore recover an incident on the next cycle. Revoking a key recovers all its active budget incidents in the revoke transaction.

Budget events use client_api_key_budget targets whose IDs contain only the numeric key ID, budget kind, and window. Metadata is limited to those fields, the threshold, current use, limit, and the key_revoked confirmation when applicable. It never includes the key secret or prefix, Request Log payloads, or raw errors. The persistent crossing table prevents unchanged usage, restarts, or concurrent server instances from duplicating transitions. Source events support custom exact-match rules; no rule is installed or enabled automatically.

The api-key-budget-80-percent-v1 template matches the first 80-percent crossing for each API Key budget stream, uses target-scoped deduplication and a 24-hour cooldown, and can notify when that same stream falls below 80 percent. It starts disabled and must be installed against an existing delivery action. The api-key-budget-100-percent-v1 template independently matches exhaustion for each stream, uses a one-hour cooldown, and recovers only on that stream's 100-percent recovery action. An 80-percent recovery cannot close its firing state. This template also starts disabled and requires explicit installation.

Routing exhaustion source events are projected from Request Logs by a separate always-on monitor that runs at startup and every minute. Migration 45 baselines its persistent checkpoint at the current maximum Request Log ID, so retained history is not replayed after upgrade. Each transaction reads at most 1000 logs in ID order, emits at most 100 transitions, and commits the checkpoint, current per-key firing state, and System Events together. PostgreSQL advisory lock contention or an in-flight Request Log write is a normal skipped cycle rather than a failure. A short SHARE ... NOWAIT table lock prevents a lower sequence ID from committing after the checkpoint has advanced past it. The lock is released immediately after capturing a committed safe maximum ID; event projection never holds it while processing the batch.

A log with routing_pool_error = routing_pool_exhausted emits Runtime error/failure action api_key.routing_pool.exhausted once for that API Key. Only a later log for the same key with a real upstream 2xx, a non-null Provider account, and no routing-pool error emits the Runtime info/success api_key.routing_pool.recovered action. Local responses, rate or budget rejections, other failures, and idle periods do not recover the incident. Events target only the API Key and contain bounded attribution IDs and fallback depth, never request/response content, error text, key material, or pool names. Revoking the key emits a recovery with confirmation: key_revoked in the revoke transaction. Source events support custom exact-match rules; no routing exhaustion rule is installed or enabled automatically.

The routing-pool-exhausted-v1 template matches the first routing-exhaustion event for each API Key, uses target-scoped deduplication and a one-hour cooldown, and can notify on the exact routing-pool recovery action for that key. It starts disabled and must be installed explicitly against an existing delivery action.

The hourly API Key physical-cleanup runner emits scheduler.api_key_purge.failed when a purge transaction fails. The event targets client_api_key_collection, contains only the configured seven-day retention duration, and never includes database errors or API Key identity. Normal shutdown cancellation emits no failure. Every successful cycle, including a zero-row cycle, commits scheduler.api_key_purge.completed with the purge transaction; that completion is the recovery signal for custom rules.

The api-key-purge-failed-v1 template matches the first scheduled API Key purge failure, uses target-scoped deduplication and a 24-hour cooldown, and can notify when the next purge cycle completes successfully. It starts disabled and must be installed explicitly against an existing delivery action. Because failure reporting and completion events use the same PostgreSQL store, an event-storage failure can prevent either the trigger or recovery notification.

The daily System Event retention runner emits scheduler.system_event_retention.failed against the stable system_events/retention target when a delete batch fails. A failure before any committed batch is error/failure; a failure after one or more committed batches is warning/partial and reports the already-deleted row count. The event contains only the UTC cutoff, configured retention days, and deleted count, never SQL, database errors, or deleted-event attributes. Normal shutdown cancellation emits no failure. The next scheduler.system_event_retention.completed cycle is the recovery signal for custom rules. No System Event retention failure rule is installed or enabled automatically.

The system-event-retention-failed-v1 template matches both full and partial scheduled retention failures, fires on the first failed cycle, uses target-scoped deduplication and a 24-hour cooldown, and can notify when the next retention cycle completes successfully. It starts disabled and must be installed explicitly against an existing delivery action. Because retention and its failure events use the same PostgreSQL store, an event-storage failure can prevent either the trigger or recovery notification.

POST /api/admin/alert-actions/{id}/test tests only the saved destination and requires the same action revision. It remains available when the dispatcher or action is disabled, performs one bounded five-second attempt, and returns only a sanitized result: pass/fail status, HTTP status when available, latency, a fixed error code, and whether the failure is retryable. Response bodies, response headers, destination values, ciphertext, and raw network errors are never returned or recorded. Tests are globally serialized in the running process and each action has a persisted 30-second cooldown. The latest sanitized result is stored without changing the action configuration revision and is shown after a refresh or restart. Test audit events are excluded from alert-rule matching to prevent recursive notifications.

Bounded delivery is independently gated by N2API_ALERT_DELIVERY_ENABLED=false. When enabled, PostgreSQL publishes each committed System Event ID to a dedicated listener; events in rolled-back transactions are never sent. The listener uses a dedicated PostgreSQL connection and does not reserve capacity from the business pool. Startup establishes LISTEN synchronously before background source-event monitors run, so their first committed notifications are already queued for the dispatcher.

Size the PostgreSQL server connection limit for the business pool plus control and operator connections:

steady state = pool_max_conns + 1 instance-lock connection
             + 1 LISTEN connection when alert delivery is enabled
startup peak = steady state + 1 migration-lock connection
             + 1 migration-executor connection + operator reserve

pool_max_conns=1 is supported for the application pool, including when alert delivery is enabled. The separate control sessions use the same parsed TLS, authentication, fallback-host, timeout, and runtime settings as DATABASE_URL and publish distinct PostgreSQL application_name values. The listener, one ordered evaluator, and two HTTP workers run outside gateway request processing. Each rule/deduplication stream is stably assigned to one worker so its firing and recovery notifications remain ordered; unrelated streams can deliver concurrently. Queue saturation drops the event without waiting on the gateway and records one aggregate overflow event per reporting window.

Generic Webhook sends a bounded JSON summary. ntfy sends a bounded plain-text summary to its configured topic URL. The dedicated delivery client has a short timeout, does not use environment proxies, never follows redirects, and never stores or logs a destination, query string, response body, or raw network error. Any 2xx response succeeds. Network failures, 408, 425, 429, and 5xx responses receive at most three capped exponential attempts; other responses fail permanently. Exhausted delivery and queue-overflow events are excluded from all rule matching to prevent recursive notification storms.

Rule state and cooldown are committed before the outbound attempt. Delivery failure does not roll them back. The in-process queues are intentionally not a durable outbox: a crash, restart, listener disconnect, or sustained queue saturation can lose a notification. Authenticated GET /api/admin/health reports current counters and sanitized last-result state at tasks.alertDelivery; unauthenticated health responses omit it. Disable the startup gate and all rules to stop outbound delivery.

Changing N2API_ENCRYPTION_SECRET invalidates existing Request Log and System Event cursors and Stateful Responses affinity lookups because those signatures and response-ID HMACs intentionally use only the current secret. Previous encryption keys do not keep old cursors valid. They also do not keep affinity rows valid. After rotation, an existing response ID in a multi-account routing scope fails closed with response_affinity_unknown; only a routing scope whose complete fallback-chain topology contains one account can use the single-account compatibility path. Treat secret rotation as a stateful-session cutover and plan for existing response IDs to stop chaining or loading. This task does not bulk-rewrite stored values. Once the upgraded application creates or refreshes a credential, an older image cannot read that new envelope; rollback then requires the upgraded image with the prior keyring or a database backup taken before new envelope writes.

The release Compose file requires .env and an explicit N2API_IMAGE; it has no latest fallback. Use the immutable CalVer matching the checked-out release or a complete digest reference. Missing required variables are rejected at Compose interpolation time. The stack publishes N2API on 127.0.0.1 by default. Set N2API_BIND_ADDRESS=0.0.0.0 or :: only when an intentionally public host listener is protected by the host firewall or an operator-provided ingress.

Host Binding Modes

Keep N2API_HOST=0.0.0.0 inside the container so other Compose services and the healthcheck can reach N2API. N2API_BIND_ADDRESS controls only the host listener created by Docker:

Deployment path	N2API_BIND_ADDRESS	Required operator action
Reverse proxy on the same host	127.0.0.1 (default), or ::1 for an IPv6-only proxy	Set N2API_PUBLIC_URL to the external HTTPS origin and keep the proxy on the selected loopback family.
Trusted IPv4 LAN	a specific host LAN address, or 0.0.0.0 only when every IPv4 interface is intended	Restrict port 3000 with the host firewall. Add public-http to N2API_ACCEPT_RISKS only when the external origin really uses HTTP.
Direct public IPv4	0.0.0.0	Do not use for secure production: N2API does not terminate TLS, and a firewall does not encrypt credentials or API traffic. Use a TLS ingress on loopback or a Docker network instead. Any temporary public HTTP use requires public-http.
IPv6 listener	a specific host IPv6 address, ::1 for loopback, or :: for all IPv6 interfaces	Confirm the firewall has equivalent IPv6 rules and verify the listener with ss -lntp. A single release mapping publishes only the selected address family.
Docker network only	no published port	Remove the inherited port with a Compose override and connect the ingress container to the same Docker network.

Docker Compose 2.24.4 or later can remove the release port with an override file containing:

services:
  n2api:
    ports: !override []

Apply that file after the release definition. The ingress can then use http://n2api:3000 on the shared Compose network while N2API_PUBLIC_URL remains the browser-visible HTTPS origin:

docker compose -f deploy/compose.release.yaml -f compose.docker-only.yaml \
  --env-file .env config --quiet
docker compose -f deploy/compose.release.yaml -f compose.docker-only.yaml \
  --env-file .env up -d

For simultaneous IPv4 and IPv6 publishing, use a separate override with both long-syntax mappings instead of relying on one N2API_BIND_ADDRESS value:

services:
  n2api:
    ports: !override
      - name: http-ipv4
        target: 3000
        published: "${N2API_PORT:-3000}"
        host_ip: "0.0.0.0"
        protocol: tcp
      - name: http-ipv6
        target: 3000
        published: "${N2API_PORT:-3000}"
        host_ip: "::"
        protocol: tcp

Treat this as public on both address families and apply firewall and TLS ingress rules to both. A protected dual-stack reverse proxy is preferable to publishing N2API's plain HTTP listener directly.

Keep .env readable only by its owner, validate the Compose model without printing the resolved secrets, then pull and start the release:

chmod 600 .env
docker compose -f deploy/compose.release.yaml --env-file .env config --quiet
docker compose -f deploy/compose.release.yaml --env-file .env pull
docker compose -f deploy/compose.release.yaml --env-file .env up -d
docker compose -f deploy/compose.release.yaml --env-file .env ps
curl -fsS http://127.0.0.1:3000/readyz
curl -fsS http://127.0.0.1:3000/livez
curl -fsS http://127.0.0.1:3000/version
curl -fsS http://127.0.0.1:3000/api/admin/health
docker image inspect "ghcr.io/knowsky404/n2api:${N2API_VERSION}" --format '{{.Os}}/{{.Architecture}}'
dev/verification/verify-release-image.sh --container \
  "$(docker compose -f deploy/compose.release.yaml --env-file .env ps -q n2api)" \
  "$IMAGE_REF"

The docker image inspect command must print linux/arm64 on an ARM64 host. The release-image verifier must then confirm that the running container was created from the exact immutable reference. The release Compose file publishes port 3000 on host loopback unless N2API_BIND_ADDRESS changes it.

After the stack is healthy, sign in, connect and test a provider account, enable its supported models, create a client API key, and verify /v1/models and one streaming /v1/responses request with that key.

Back Up and Upgrade

Use the canonical operations runbook for signed backups, isolated restore evidence, upgrade plan/apply, and guarded application rollback. The canonical sequence starts with:

./ops/n2api --env-file .env backup create --evidence-class real_operator --format json
./ops/n2api --env-file .env backup list --format json

The first command can return attention exit 3 after successfully publishing the archive because encrypted off-host custody remains an owner gate.

Lower-level Compose fallback

The raw commands in this section are retained for diagnosis and an owner-controlled recovery when the canonical CLI cannot express the required action. They do not create signed operator evidence or operation receipts and must not replace the CLI's production workflow.

Create a PostgreSQL custom-format backup before every upgrade:

mkdir -p backups
docker compose -f deploy/compose.release.yaml --env-file .env exec -T postgres \
  sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' \
  > "backups/n2api-$(date +%Y%m%d-%H%M%S).dump"

Keep backups outside the Compose volume and periodically verify them with the isolated restore drill below. The repository ignores backups/, but a database dump still belongs in encrypted off-host storage and must never be committed.

Use the exact N2API image tag or digest that should serve the restored data. Provide the administrator credentials and complete encryption keyring from the backup's deployment through the environment; the script does not print them. It creates its own random Compose project, fixed temporary database, internal network, and volume. It never accepts a database URL or Compose project name.

read -rsp 'Restore admin password: ' N2API_RESTORE_ADMIN_PASSWORD; echo
read -rsp 'Restore encryption secret: ' N2API_RESTORE_ENCRYPTION_SECRET; echo
read -rsp 'Restore previous-key JSON (or []): ' N2API_RESTORE_ENCRYPTION_PREVIOUS_KEYS; echo
export N2API_RESTORE_ADMIN_PASSWORD N2API_RESTORE_ENCRYPTION_SECRET N2API_RESTORE_ENCRYPTION_PREVIOUS_KEYS
export N2API_RESTORE_ADMIN_USERNAME='admin'
export N2API_RESTORE_IMAGE='ghcr.io/knowsky404/n2api:YYYYMMDDNN'
export N2API_RESTORE_ENCRYPTION_KEY_ID='default'
dev/verification/restore-backup.sh backups/n2api-YYYYMMDD-HHMMSS.dump
unset N2API_RESTORE_ADMIN_PASSWORD N2API_RESTORE_ENCRYPTION_SECRET N2API_RESTORE_ENCRYPTION_PREVIOUS_KEYS
unset N2API_RESTORE_ENCRYPTION_KEY_ID

Use the exact key ID and previous-key array that were active when the backup was taken. For backups outside a rotation window, enter [] and keep the default key ID unless the deployment had explicitly changed it.

The drill lists and restores the custom archive in one transaction, starts the selected image so pending migrations run, waits for readiness, checks schema version/counts/foreign-key integrity, verifies one restored reusable API key can be decrypted when present, and runs the mock-upstream gateway E2E. Its report contains counts and status labels only. Success and failure both remove the exact temporary containers, network, and volume. A successful generated fixture drill is automated evidence; run the same command on a current real operator backup before claiming the deployment is recoverable.

Restore Drill Schedule And Records

Run the isolated restore drill at least once each calendar month and before every upgrade. Use a fresh pre-upgrade backup. Prove it first with the currently deployed immutable image; when the target release contains migrations, repeat the isolated drill with the proposed immutable image before changing the live stack. A drill is not complete until cleanup has removed its temporary containers, network, and volume.

Complete an operator restore drill record, then reference its non-sensitive identifier from the release checklist. Record UTC backup and drill timestamps, a non-sensitive backup identifier, source and restored schema versions, exact current and candidate image tags and digests, planned window, measured duration, every redacted check outcome, retention condition, and owner sign-off. Keep CI generated-fixture status separate from real operator-backup acceptance. Allocate at least 60 minutes or twice the previous measured duration, whichever is longer; this is a planning window rather than a recovery-time guarantee.

Keep at least the three most recent successful monthly backups. Retain each pre-upgrade backup until the upgraded deployment passes its next monthly restore drill. Backups must be encrypted in off-host storage, with decryption material stored separately from the dump. Keep real operator dumps and storage credentials out of Git, GitHub Actions, logs, and drill records. Do not record public or signed storage URLs, dump contents, complete archive listings, restored row data, or complete object listings. CI may validate only generated non-sensitive fixture dumps and never satisfies the real-backup acceptance gate.

Portable Configuration Export

After signing in, open Gateway and use Export JSON under Portable configuration, or send GET /api/admin/configuration/export with an authenticated administrator session. The response is a no-store JSON attachment named n2api-portable-config-v1-YYYYMMDDTHHMMSSZ.json. The server rejects an export larger than 5 MiB and records the successful security audit event before sending the file.

Format version 1 contains routing pools, memberships and fallbacks; active API non-deleted API key templates (active or disabled) with names, policies, limits, budgets and pool references; provider account names, scheduling fields, models and sanitized base URLs; model, usage-pricing and gateway settings; fingerprint profiles; and error passthrough rules. It also contains every alert action's non-sensitive name, kind, enabled state, and destinationConfigured: true, plus complete alert rule definitions linked through file-local actionRef values. File-local references preserve relationships without making database IDs an import conflict key. API upstream base URLs have user info, query strings and fragments removed. Every custom fingerprint header value is replaced with [redacted], including values that do not look sensitive.

The export never contains administrator or session state, OAuth state or subjects, provider credentials, proxy URLs, client API key hashes, prefixes or encrypted reusable secrets, request logs, system events, provider test history, or runtime failure state. Alert action destinations are never queried or placed in the document, and there is no destination placeholder that a future importer could mistake for an endpoint. Action test results, delivery state, rule matcher state, cooldown timestamps, and configuration timestamps are also omitted. The document reports unsupportedSections: [] and names alertActionDestinations in its top-level redactions list.

Portable configuration is a review and migration aid, not a complete backup and not currently importable. PostgreSQL remains the authoritative recovery source because it retains encrypted credentials and all operational state. Continue taking and verifying database backups even when configuration exports are stored separately.

For a lower-level upgrade, change N2API_IMAGE to the exact target tag@digest, then pull and recreate the stack. This bypasses plan/apply and is therefore an owner-controlled fallback, not an Agent default:

docker compose -f deploy/compose.release.yaml --env-file .env pull
docker compose -f deploy/compose.release.yaml --env-file .env up -d
curl -fsS http://127.0.0.1:3000/readyz

Use immutable CalVer tags or complete digest references for production. The moving latest and main tags are for explicit update experiments and development validation, not unattended release deployment.

Health Probes

N2API exposes separate process and dependency probes:

• GET /livez reports only that the HTTP process can respond. It does not check PostgreSQL or provider accounts.
• GET /readyz reports ready only when PostgreSQL responds, the static admin build contains its application entry document, runtime startup is complete, and a valid Gateway Settings snapshot is available. A stale last-known-good settings snapshot remains ready; an invalid initial snapshot does not. Migrations, administrator bootstrap, and background runner construction finish before the HTTP server starts listening. Provider account availability does not affect readiness.
• GET /healthz remains a compatibility alias for the liveness behavior.
• GET /version is public and returns only the short build version, for example {"version":"sha-0123456789ab"}.
• GET /api/admin/health remains publicly usable for its existing status/database response. With a valid administrator session cookie, it also includes the complete commit SHA and UTC build time under build, plus operational task details such as tasks.requestLogWrite.

Authenticated health also exposes tasks.gatewaySettings, tasks.requestLogRetention, and tasks.alertDelivery without configuration values or destinations. Request Log retention reports whether a persisted policy is configured, keep_all or delete_after_days, oldest/newest times, bounded row estimates, eligible count, relation bytes, and disk risk unknown, ok, watch, or high. Alert delivery reports whether its LISTEN connection is currently connected and the process-local reconnect count in addition to queue, retry, drop, and delivery state.

tasks.requestLogWrite is in-memory status for durable Request Log writes. lastSucceededAt and lastFailedAt are the UTC times of the latest successful and failed writes and are omitted until the corresponding outcome occurs. lastErrorCode is the latest stable write error code, currently request_log_write_failed. consecutiveFailures increments for each failed write and resets to 0 on the next successful write; totalFailures counts all failures since the current process started and does not decrease after recovery. A successful write updates lastSucceededAt but retains the latest failure time, error code, and total count as diagnostic history. All of these values reset when the process restarts, and unauthenticated health responses do not expose them.

After sign-in, the Dashboard shows the short build version in its compact system status. Hover over the value, or focus and activate it, to inspect the complete commit and build time. Signing out removes the authenticated build detail from client state. Local source builds use explicit non-release values: version dev, commit unknown, and build time 1970-01-01T00:00:00Z.

Both development and release Compose configurations use /readyz for the application container healthcheck. A temporary provider outage therefore does not restart or mark the entire gateway unavailable.

Downstream Codex CLI

After adding and testing an API-key upstream account, enable the models that account can serve and create a client key on the API Keys page. Save the following as ~/.codex/n2api.config.toml:

model_provider = "n2api"
model = "gpt-5.4-mini"

[model_providers.n2api]
name = "N2API"
base_url = "http://127.0.0.1:3000/v1"
env_key = "N2API_API_KEY"
wire_api = "responses"

Codex 0.134.0 and later load --profile n2api from that separate file; do not place these settings under a legacy [profiles.n2api] table in ~/.codex/config.toml. Start Codex without putting the client key in shell history, and keep the key scoped to the child shell:

(
  trap 'unset N2API_API_KEY' EXIT INT TERM
  read -rsp 'N2API client key: ' N2API_API_KEY; echo
  export N2API_API_KEY
  codex --profile n2api
)

Replace the base URL when Codex runs on another machine. Verify GET /v1/models with the client key before troubleshooting model requests; the list reflects the key's routing-pool scope and model policy, account model capability, enabled state, and account health. A key without a routing pool returns no models.

Provider Accounts

Use Provider accounts to add API-key upstreams. Set the upstream base URL, API key, models, and optional proxy, then test the account and add it to a routing pool. OAuth connection, reauthorization, token refresh, and fingerprint profiles are no longer supported. Historical OAuth rows and credentials remain in PostgreSQL for backup compatibility and do not participate in routing. Replace their routing-pool memberships with API-key upstreams before use.

• Use the account row to set a display name, priority, and Scheduling preference.
• Select rows on the Provider accounts page to bulk enable or disable provider accounts. Use Enable selected or Disable selected to change scheduling eligibility for the selected exits, and Clear selection to discard the selection without changing accounts.
• Set Bulk priority, Bulk scheduling preference, or Bulk max concurrency, then use Apply scheduling to update selected provider accounts together; bulk priority, bulk scheduling preference, and bulk max concurrency fields use the same validation as each account row.
• Configure supported models on each connected account. These per-account model rows describe account capability for gateway routing.
• Use the Providers table Test models action to diagnose one or more configured models against one exact account. The modal starts with no selection; row checkboxes support one or multiple models, and the tri-state header checkbox selects only the current filtered result set. Tests use that account's stored API-upstream key, never fall back to another account, and persist each model's latest status and latency. They do not enable or disable models or change account scheduling health.
• Selected provider accounts can also receive the same model capability list. Enter one model per line in Bulk models, then use Apply models to replace the selected accounts' manual model lists together; this controls which models the scheduler can route to those accounts.
• Use the Provider accounts page to add or remove selected provider accounts from a routing pool without opening the pool editor. Choose Bulk routing pool, set Pool priority for new pool members, then use Apply pool to add the selected accounts to that pool or Remove pool to remove them while leaving the pool's other members unchanged.
• Use API Keys to control the routing-pool scope and all or selected model policy. Account model configuration remains the source of truth for capability. The gateway defaultModel is used only when a POST request omits model, and the injected model must still be routable in that key's bound pool or explicit fallback chain.
• Client API keys have no model access until they are bound to a routing pool. Within the pool's explicit fallback chain, all keys see every routable configured model and selected keys see only the routable intersection of their selected models.
• Routing pools partition provider accounts for different agents, devices, or risk profiles. A key without a routing pool returns no models and cannot route model requests. A bound key only schedules accounts in its pool and explicit fallback chain, including sticky session bindings scoped to that chain. Pool membership priority is evaluated before the account's global priority inside that pool, so the same provider account can be ranked differently for different client keys. Missing or deleted pools fail closed with routing_pool_unavailable, empty enabled pools fail closed with routing_pool_empty, and Request Logs retain the routing pool name/id for attribution.
• API key and Routing Pool management responses use signed, filter-bound keyset cursors with a default page size of 50 and a maximum of 100. The admin UI loads additional pages explicitly and keeps partial rows visible when a later page fails. The API Keys page supports local search and status filtering by name, prefix, model policy, selected model, active/disabled/deleted status, and limiter state, plus routing-pool filtering, across the rows currently loaded; bulk actions submit only the selected loaded IDs and never imply selection of unloaded pages.
• API key names can be renamed from the API Keys page without rotating the secret, so labels can be kept in sync with devices, agents, or usage purpose.
• New API keys are stored with an encrypted reusable secret. The Prefix column on the API Keys page can copy the full API key again after creation for active or disabled keys; older keys created before encrypted secret storage may need to be rotated if their full value was not saved.
• API keys have three visible states: active, disabled, and deleted. Active and disabled keys can be toggled directly from the API Keys table status column, and disabled keys cannot authenticate gateway requests. Deleting an active or disabled key performs an irreversible logical delete immediately, keeps the row visible during its 7 day retention window, and exposes the scheduled physical deletion time in the deleted status tooltip. Deleted keys can be physically deleted immediately with a second confirmed Delete action. Keys past the retention window are physically removed by startup and hourly cleanup, with API key listing cleanup as a fallback.
• API key budgets are personal operational safeguards, not billing balances. Request budgets are strict rolling 24h and 30d limits backed by atomic PostgreSQL admission. Token and estimated cost limits are explicitly observed-usage budgets: admission rejects once settled observed usage has reached the limit, but one request or concurrent requests can cross it because no trustworthy universal pre-request upper bound exists. 0 disables a budget field. Existing budgeted keys fail closed with budget_initializing while the bounded upgrade backfill initializes their ledger from Request Logs. When a ready key is over budget, clients receive OpenAI-compatible rate_limit_exceeded responses while Request Logs store the precise local reason as api_key_request_budget_exceeded, api_key_token_budget_exceeded, or api_key_cost_budget_exceeded.
• API upstream credentials can be updated from the account row. Rotating the encrypted API key, base URL, or per-account outbound proxy URL preserves current account health; run Test account to confirm the new settings before an expired or circuit-open account becomes schedulable again. Proxy URLs are stored encrypted because they may include credentials, and the admin UI only shows a redacted proxy summary.
• Use Test account before sending client traffic through an account. The action probes one provider account with its current API upstream key. Only a 2xx response confirms recovery and clears local failure status; network errors and every non-2xx response record a failed test without a recovery transition. The account row keeps the last test status, last test time, and last test error so manual checks remain visible after refresh. Each probe also writes provider account test history; use the Providers page History action to expand Recent test history, or fetch the same data from GET /api/admin/provider-accounts/{id}/test-results.
• Gateway account selection records only an attempt timestamp. It does not clear account health. A gateway request confirms account recovery only when its final upstream response is 2xx; redirects, client/server errors, authorization failures, network failures, and configured error passthrough do not recover the account. When fallback succeeds, only the account that produced the final 2xx response recovers.
• The Ops Monitor page shows Recent account tests for the selected monitoring window so manual and automatic probe failures are visible without opening each provider account row. Fetch the same aggregate view from GET /api/admin/ops/account-tests.
• Use Test selected to probe selected provider accounts without probing the whole account pool. It updates the same last-test fields, health fields, and test history as Test account.
• Use Disconnect account when an exit should be removed from the gateway. It deletes the provider account, stops scheduling it for new traffic, and removes its stored credentials and account-scoped model configuration through the database cascade.
• Use Disconnect selected when several exits should be removed together. It deletes the selected provider accounts, stops scheduling them for new traffic, and removes their stored credentials and account-scoped model configuration through the same database cascade.
• Provider account auto tests are disabled by default. N2API_PROVIDER_ACCOUNT_AUTO_TEST_ENABLED and N2API_PROVIDER_ACCOUNT_AUTO_TEST_INTERVAL_SECONDS are startup defaults for Gateway Settings; after sign-in, use the Gateway Settings form to save the runtime auto-test setting. Enable it to run Test all accounts automatically in the backend, and use an interval of 300 seconds or higher for routine checks. Automatic tests update the same last test status, last test time, last test error, test history, and local account health fields shown in Provider accounts and Routing diagnostics.
• Gateway Settings also shows Auto-test status for the in-memory runner. The status row reports whether the runner is active, the last finished time, accounts tested in the last cycle, and the last error when a scheduled probe fails.
• Gateway Settings includes Request log retention for bounded request-log cleanup. A value of 0 disables both manual and automatic deletion. Manual and automatic runs delete logs older than the saved retention window. The status panel reports the current UTC cutoff, exact eligible row count, estimated total row count, oldest/newest timestamps, automatic runner state, and the latest automatic outcome. Clean request logs remains available for an immediate manual run and is disabled while an automatic run owns the cleanup lock.
• Automatic Request Log cleanup has a separate startup gate, N2API_REQUEST_LOG_RETENTION_RUNNER_ENABLED, which defaults to false. A historical positive retention value therefore does not silently become automatic after an upgrade. When enabled, the runner executes once at startup and then every N2API_REQUEST_LOG_RETENTION_INTERVAL_SECONDS seconds, deleting at most N2API_REQUEST_LOG_RETENTION_BATCH_SIZE rows per transaction. Valid intervals are 300-604800 seconds and valid batch sizes are 100-10000 rows.
• Stateful Responses account affinity stores only a domain-separated HMAC of each response ID, scoped to the client key's primary routing pool. N2API_RESPONSE_AFFINITY_TTL_DAYS defaults to 30 days. Expired-row cleanup remains disabled unless N2API_RESPONSE_AFFINITY_RETENTION_RUNNER_ENABLED=true; its interval and single-cycle delete bound are controlled by N2API_RESPONSE_AFFINITY_RETENTION_INTERVAL_SECONDS (300-604800) and N2API_RESPONSE_AFFINITY_RETENTION_BATCH_SIZE (100-10000). Authenticated /api/admin/health includes the runner's last start, success, failure, lock skip, stable error code, and delete count.
• Before enabling automatic Request Log deletion on a long-running instance, restore a current backup successfully in an isolated environment and review the displayed cutoff and eligible count. Keep the runner gate disabled if either check is incomplete. Automatic and manual runs use the same PostgreSQL advisory lock and committed batches are not rolled back if a later batch is canceled or fails.
• Use Pause scheduling when a healthy account should stop receiving traffic for a short window. Set Pause duration seconds on the Provider accounts page before clicking the action; it temporarily opens the account circuit for that window without disabling or deleting the account. Paused and rate-limited rows show the remaining scheduling block in the status column. Use Reset local status to clear the pause early. Reset is an explicit operator override: when it clears unhealthy fields, N2API records both the status-reset audit event and a runtime recovery event even without a provider 2xx.
• Selected provider accounts can be paused and reset together. Use Pause selected to apply the configured Pause duration seconds to every selected account, or Reset selected to apply the same explicit operator override to each selected account. Prefer Test selected when provider-confirmed recovery is required.
• Disabled accounts are kept in PostgreSQL but are not selected for gateway traffic.
• Connected accounts with no configured models are kept in PostgreSQL and can be edited later, but they do not receive model-routed POST traffic.
• Provider-account model configuration is the source of truth for model capability. Accounts with no configured models remain editable but cannot serve model-routed requests.
• Lower account priority numbers are selected before higher account priority numbers. Inside a routing pool, lower pool membership priority is evaluated first, followed by the account's global priority.
• The UI calls the compatibility field Scheduling preference; the JSON/API identifier remains loadFactor. Within the same priority scope, higher values form a strict descending preference tier, not a proportional request weight. A continuously eligible account in a higher scheduling preference tier is considered before every account in a lower tier and can therefore receive every new non-sticky selection. Keep weak or quota-sensitive accounts at scheduling preference 1; raise the value only when that strict preference is intended.
• Within an exactly equal priority and scheduling preference tier, accounts without a recent error are considered before accounts with one, followed by least-recently-used time and account ID. A stored sticky binding is reused while its account remains schedulable. For a new session, sticky FNV hashing only changes order inside the highest tier with exactly equal pool priority (when scoped), global account priority, scheduling preference, and recent-error state; it never promotes a lower scheduling preference tier.
• Provider accounts expose Max concurrency as a per-account concurrency override. 0 inherits the gateway default from Gateway Settings; positive values cap that account independently. Each account row also shows active concurrency as a process-local runtime snapshot beside the effective cap; a cap of 0 is shown as unlimited, and the active count resets when the backend process restarts.
• Rate-limited, circuit-open, expired, and disabled accounts are skipped during gateway account selection.
• Upstream 429 responses mark the account as rate-limited, 401/403 mark it expired, and 5xx responses open a short circuit window before traffic tries another account.
• /v1/models returns 200 with an empty model list for an unbound API key. A model-routed POST made with that key fails closed with 503 and the error code routing_pool_required. For a bound key, all exposes every currently routable configured model in the routing-pool fallback chain, while selected exposes the routable intersection of the key's selected models.
• /v1/models never widens beyond the key's configured routing-pool fallback chain.
• Routing diagnostics can preview scheduler fallback without sending traffic. In Selection preview, choose a Routing pool to inspect its bound routing path. Leaving the pool unset runs an unscoped admin diagnostic across provider accounts; it does not represent access available to an unbound API key. Set Excluded account IDs to a comma-separated list such as 7, 8 to simulate those provider accounts being unavailable; excluded accounts remain visible as blocked candidates with the reason account excluded. Routing preview also shows each candidate's active concurrency and effective account cap; candidates at a positive cap are marked Concurrency full. Each schedulable preview candidate includes a Schedule reason as diagnostic text with its pool/global priority tier, scheduling preference tier, recent-error tier, least-recently-used tie-breaker, account ID, and sticky decision; this explains the current rank and does not change scheduler behavior. Runtime concurrency is not a proportional weight: concurrency-full accounts are excluded and the gateway requests another selection, returning 429 only when no eligible account can accept the request.
• If one enabled account fails before streaming starts, N2API tries another eligible account that supports the same requested model.
• Once upstream streaming has started, N2API preserves that stream and does not retry against another account.
• Provider API keys are encrypted before being stored. Historical OAuth credentials and retired fingerprint records are retained for backup compatibility only.

Gateway Runtime Limits

Gateway management refreshes provider accounts, model routing, and API keys before reporting readiness, so the counts and prerequisite warnings are valid even when /gateway is opened directly.

Gateway management also includes Scheduling health, which summarizes enabled, schedulable, and blocked provider accounts; Blocked reasons groups disabled, expired, rate-limited, and circuit-open exits so account-pool pressure is visible without opening the full Provider accounts table.

The deployment template includes optional in-process gateway guards:

• N2API_GATEWAY_MAX_CONCURRENT_REQUESTS limits total active gateway requests.
• N2API_GATEWAY_MAX_CONCURRENT_REQUESTS_PER_ACCOUNT limits active requests per provider account.
• N2API_GATEWAY_MAX_CONCURRENT_REQUESTS_PER_KEY limits active requests per client API key.
• N2API_GATEWAY_REQUESTS_PER_MINUTE_PER_KEY limits accepted requests per client API key per fixed minute.
• N2API_GATEWAY_TOKENS_PER_MINUTE_PER_KEY limits observed request tokens per client API key per fixed minute.

Set any gateway default value to 0 to disable that guard. These limits are process-local, so N2API holds a PostgreSQL session advisory lock and rejects a second application instance before starting background jobs or the HTTP listener. N2API monitors the dedicated lock session; if it is lost, the service records instance_lock_lost, stops accepting traffic, completes its bounded shutdown, and exits non-zero instead of continuing without the lock. N2API_ALLOW_UNSAFE_MULTI_INSTANCE=true bypasses that protection only for explicit recovery/debugging; authenticated health and startup logs then report unsafe_multi_instance_enabled, and limiter, concurrency, scheduler, and in-memory status correctness are not guaranteed across instances. Keep the override false for normal operation. The API Keys page shows the values loaded by the running service. Per-key values set to 0 inherit the matching gateway default and do not disable that guard for only one key. The API Keys page shows active concurrency for each client key as process-local runtime state; keys at a positive effective cap are marked Concurrency full. It also shows Requests window and Tokens window for each key as process-local fixed one-minute counters with remaining capacity; limited windows at capacity are marked Request limit full or Token limit full, and the counters reset on the next fixed minute or backend restart. Local per-key request/token 429 responses include Retry-After; per-account concurrency skips busy accounts when another eligible account is available and returns 429 only when no eligible account can accept the request.

Gateway request bodies have separate acceptance and in-memory replay limits. N2API_GATEWAY_MAX_ACCEPTED_REQUEST_BODY_BYTES defaults to 4 MiB and is the hard limit for both known-length and chunked requests. N2API_GATEWAY_MAX_IN_MEMORY_REPLAY_BODY_BYTES defaults to 1 MiB and cannot exceed the accepted limit. A body above the accepted limit returns 413 with the stable code request_too_large; when Content-Length proves this before a read, N2API rejects it without consuming the body. A valid body above the replay limit but within the accepted limit is normalized in bounded memory and receives at most one upstream attempt, with no fallback replay. Smaller bodies retain pre-stream retry and fallback behavior. Gateway and client-key concurrency admission occurs before the complete body read, and every body error path releases both slots.

Non-streaming upstream responses are buffered only up to N2API_GATEWAY_MAX_UPSTREAM_RESPONSE_BODY_BYTES plus one byte; the default is 8 MiB and the startup maximum is 64 MiB. N2API verifies this bound before copying the upstream status, headers, or body. An oversized response becomes a 502 response with the stable code upstream_response_too_large; a response read failure uses upstream_response_error. Neither path returns a partial body, upstream headers, or the underlying network error, and truncated data is not parsed as usage or stored in Request Logs. Bounded upstream failure bodies may be inspected in memory for configured passthrough rules, but account health persists only a stable failure classification and never the original body or a truncated prefix. SSE remains streaming and is not subject to this total response-body limit.

Inbound HTTP connections use N2API_HTTP_IDLE_TIMEOUT_SECONDS (60 seconds by default) and N2API_HTTP_MAX_HEADER_BYTES (1 MiB by default). Model-routed POST requests arm a per-request read deadline after gateway and client-key admission and before reading the complete body; N2API_HTTP_REQUEST_BODY_TIMEOUT_SECONDS defaults to 30 seconds. The deadline is cleared after the body read returns while the request context remains active. When the deadline or request cancellation has already fired, it remains in effect until the handler unwinds; every path still releases both admission slots. A slow upload that reaches the deadline returns 408 with request_body_timeout. N2API does not set a global HTTP WriteTimeout, because that would truncate healthy long SSE responses.

Graceful shutdown uses one deadline across the entire process. N2API_SHUTDOWN_TIMEOUT_SECONDS defaults to 25 seconds and is bounded to 6-30 seconds so it always remains below the fixed 35-second Compose grace. N2API_REQUEST_DRAIN_TIMEOUT_SECONDS defaults to 20 seconds, is bounded to 1-25 seconds, and must reserve at least five seconds for background tasks, Alert Delivery, and the Metrics listener. The development, E2E, and release Compose services use a 35-second stop grace period, which must remain greater than the configured application deadline.

Outbound gateway connections use N2API_UPSTREAM_CONNECT_TIMEOUT_SECONDS (10 seconds by default), N2API_UPSTREAM_TLS_HANDSHAKE_TIMEOUT_SECONDS (10 seconds), and N2API_UPSTREAM_RESPONSE_HEADER_TIMEOUT_SECONDS (30 seconds). A timeout before response headers returns 502 with the stable code upstream_timeout; neither the client response nor the provider-account failure summary includes the underlying network address or error. Streaming responses use N2API_UPSTREAM_SSE_IDLE_TIMEOUT_SECONDS (60 seconds by default) as a continuous no-data limit. Each received chunk resets the limit. When it expires, N2API closes the upstream body and records upstream_sse_idle_timeout; because the SSE headers are already committed, the stream closes without appending a JSON error object. Client disconnects also close the upstream body immediately. HTTP request contexts are separate from the signal and background-task contexts. Graceful shutdown first stops accepting new requests and lets active uploads, upstream waits, and SSE streams finish. Only the request-drain deadline cancels remaining request contexts and closes their connections.

Gateway Request Correlation

Gateway clients may send X-Request-ID. N2API trims and accepts an incoming value only when it is 1-100 ASCII characters, begins with an alphanumeric character, and otherwise contains only alphanumeric characters, ., _, :, or -. A missing or invalid value is replaced with a newly generated correlation ID. The accepted or generated value is returned as X-Request-ID and remains the same across every upstream attempt, the Request Log requestId, related System Event correlationId, and request-scoped process-log field correlation_id. An ID assigned or returned by the upstream provider is separate provider data. When the final upstream response supplies a valid X-Request-ID, N2API stores it separately as Request Log upstreamRequestId and includes it in JSON, JSONL, and CSV exports. It is limited to 200 bytes and discarded when it contains control characters or invalid UTF-8. It never replaces N2API's correlation ID or adds a second X-Request-ID value to the client response. Transport failures and responses without a valid upstream ID leave the field empty.

Request Log persistence is best effort. A write failure does not replace or otherwise change the gateway response, including an already successful upstream response. Budget admission and idempotent observed-usage settlement happen independently before the best-effort Request Log write, so a logging failure cannot refund a request or lose already settled budget usage. N2API records the stable process-log error_code request_log_write_failed with the same correlation_id, without logging the database error detail. Use authenticated GET /api/admin/health and inspect tasks.requestLogWrite to distinguish an isolated failure from consecutive failures and a later recovery.

Request Logs keep local gateway rejections diagnosable while client responses stay OpenAI-compatible. Local limit responses still return rate_limit_exceeded to clients, but the stored request-log error identifies the guard as api_key_request_rate_limited, api_key_token_rate_limited, api_key_request_budget_exceeded, api_key_token_budget_exceeded, api_key_cost_budget_exceeded, gateway_concurrency_limited, api_key_concurrency_limited, or provider_account_concurrency_limited.

Request Logs also include gateway fallback diagnostics: attempts count selected provider-account tries, and fallbacks count pre-stream scheduler moves caused by busy accounts or retryable upstream failures.

Routing pool fallback is explicit. The routing pool fallback chain can point to one fallback pool at each step, forming a simple chain such as primary -> secondary. A key tries only its configured pool and that explicit chain; there is no implicit fallback outside the chain. A disabled primary pool fails closed with routing_pool_disabled, an empty primary pool fails closed with routing_pool_empty, cycles fail closed with routing_pool_cycle, and exhausted chains are logged as routing_pool_exhausted.

Request Logs support exact Provider account, Routing pool, API key, Model filter, Usage source, and Session filter fields. On Gateway management and Dashboard, 24h usage rows for Top provider accounts, Top usage sources, Top routing pools, Top routing pool chains, Top client keys, Top models, and Top sessions link to Request Logs with exact provider-account, usage-source, routing-pool, routing-pool-chain, API-key, model, and sticky-session filters when the row identifies a concrete entity.

The authenticated GET /api/admin/request-logs endpoint returns logs, hasMore, and nextCursor. When hasMore is true, pass the opaque nextCursor value back as cursor while keeping every filter unchanged to load older rows. Cursors are signed and filter-bound but not encrypted; clients must not parse or modify them. Restart from the first page when the API returns 400 invalid_input, such as after a cursor is changed, a filter is changed, or the server encryption secret is rotated. Page size may change between requests without invalidating a cursor.

The Request Logs page keeps active filters in the URL and loads 50 rows at a time. Use Load older to append the next cursor page without replacing rows already under review. Applying or clearing filters starts a fresh first page; if a saved cursor is no longer valid, the page also recovers by restarting from the current URL-backed filters.

Request Log CSV and JSONL exports stream the currently applied filters over an explicit half-open UTC range (since <= created_at < before) and do not accumulate the result in application memory. Select a bounded date range before using those formats. Each export writes at most N2API_REQUEST_LOG_EXPORT_MAX_ROWS rows and runs for at most N2API_REQUEST_LOG_EXPORT_TIMEOUT_SECONDS; defaults are 100000 rows and 60 seconds, with valid ranges of 1000-1000000 rows and 5-300 seconds. Reaching the row limit produces a valid, explicitly truncated download and records that outcome in the export event. CSV and JSONL also offer gzip downloads. CSV text cells beginning with a spreadsheet formula marker are escaped before CSV encoding. The compatibility JSON download remains limited to 200 rows.

API upstream accounts and OPENAI_API_BASE_URL require HTTPS by default so upstream API keys are not sent over plaintext HTTP. Set N2API_ALLOW_HTTP_API_UPSTREAMS=true only for trusted local or private HTTP upstreams that you control.

For sticky session routing, clients can send session_id in the POST body. If a client needs a header instead, prefer X-N2API-Session-ID through reverse proxies; session_id remains supported but contains an underscore and may be dropped by default proxy settings. If N2API is behind Nginx and clients send the session_id header, set underscores_in_headers on; in the relevant http or server block. A body session_id overrides either header.

Sticky session bindings are persisted by provider, model, and session_id. A healthy bound account is reused while it remains schedulable; if fallback excludes it before streaming starts, the successful fallback account can rebind that session.

Gateway Compatibility Matrix

The secret-free E2E stack validates the public gateway through raw HTTP and the pinned official OpenAI JavaScript and Python SDKs. All runtime services use an internal Compose network, and the SDK runners reject any gateway hostname other than n2api before creating a client.

Contract	Automated evidence
Models, Chat Completions JSON, and Responses SSE	PostgreSQL-backed Go E2E plus openai JavaScript 6.48.0 and Python 2.46.0 runners
Upstream 401, 403, 429, 500, and 503	Go E2E verifies the client status/error contract and persisted attempt/fallback attribution
Missing or incorrect content type	Go E2E verifies bounded pass-through behavior without inventing upstream metadata
Missing or malformed usage	Go E2E verifies Chat JSON and Responses SSE remain usable while Request Logs retain zero tokens and zero estimated cost instead of fabricating usage
Timeout or disconnect before response headers	Go E2E verifies cancellation and pre-stream fallback without leaking response bodies
Disconnect or clean EOF after the first SSE event	Go E2E verifies N2API never retries after streaming has begun and never fabricates a completion event
Sticky routing, explicit fallback, and local key limits	Go E2E verifies persisted account selection, fallback counters, and OpenAI-compatible 429 responses

Run the SDK contracts locally through the managed lifecycle wrapper:

make test-contracts

The CI correctness matrix also runs the following local entry points. Image build and publication jobs depend on every matrix member:

make test-go-quality
make test-critical-race
make test-control-connections
make test-request-log-profile
make test-management-list-profile
make test-production-deploy

Each runner creates its own account, routing pool, and client key through the admin API, keeps generated secrets in memory only, and performs best-effort API cleanup. Removing the isolated PostgreSQL volume is the final cleanup boundary.

Request Log Query Profile

Run the opt-in Request Log profile against its managed disposable PostgreSQL Compose project after materially changing filters, retention queries, data distribution, or the PostgreSQL major version. The runner does not publish the database port; it connects through the run-owned Docker bridge address, creates a uniquely named schema, loads one million synthetic rows, reports EXPLAIN (ANALYZE, BUFFERS) and index/write metrics, and removes the schema, container, network, and volume during scoped cleanup.

make test-request-log-profile

Do not treat one profile run as a fixed latency target. Compare plan shape, temporary I/O, buffer activity, total index bytes, and the write probe before accepting an index change.

Before upgrading an existing deployment, back up PostgreSQL because the upgrade adds unified provider account tables and client API key model-policy metadata.

Required Services

• n2api: Go application service.
• postgres: PostgreSQL database with a persistent Docker volume.

Redis is intentionally not required for V1. Add it later only if distributed rate limiting, queueing, or multi-instance locking becomes necessary.