Merge pull request #40 from Datasance/fix/sqlite-queue-self-recovery

Fix SQLite write-queue freeze with self-recovery and readiness fast p…

Author: emirhandurmus

CHANGELOG.md

@@ -1,15 +1,17 @@
 # Changelog
 
 
-## [v3.8.2] - 2026-07-25
+## [v3.8.2] - 2026-07-30
 
-Plan 21: liveness/readiness probe split and structured agent auth errors (coordinate with Edgelet v3.8.2). Also embedded OAuth logout/re-login hardening, EdgeOps Console **v1.0.12**, and operator sizing docs.
+Plan 21: liveness/readiness probe split and structured agent auth errors (coordinate with Edgelet v3.8.2). Also embedded OAuth logout/re-login hardening, EdgeOps Console **v1.0.12**, operator sizing docs, and **SQLite write-queue self-recovery** for long-running single-node deployments.
 
 ### Added
 
 - **`GET /api/v3/live`** — public **liveness** probe (process up; always 200 while HTTP server listens).
 - Structured agent fog JWT errors on **`/api/v3/agent/*`**: **`code`** + **`retryable`** on 401 (credential failure) and 503 (Controller dependency failure).
 - **`docs/operations/sizing.md`** — hardware sizing guide by fog count (Kubernetes and Remote ControlPlane).
+- **SQLite transaction recovery settings** — `settings.dbWriteQueueBackpressureDepth` (default **32**), `settings.dbTransactionTimeoutReadinessMs` (**5000**), `settings.dbTransactionTimeoutInteractiveMs` (**15000**), `settings.dbTransactionTimeoutBackgroundMs` (**120000**); env overrides **`DB_WRITE_QUEUE_BACKPRESSURE_DEPTH`**, **`DB_TRANSACTION_TIMEOUT_*_MS`** (see `docs/operations/database-transactions.md`).
+- **OTEL DB metrics** — `db.transaction.timeouts` and `db.write_queue.background_shed` for queue surgery and stuck-transaction visibility.
 
 ### Changed
 
@@ -20,9 +22,15 @@ Plan 21: liveness/readiness probe split and structured agent auth errors (coordi
 - Embedded OAuth BFF authorize sends **`prompt=login`** so each browser sign-in starts a fresh issuer interaction.
 - **`POST /api/v3/user/logout`** (embedded) clears issuer Session/Grant/Interaction state and destroys the BFF express-session when present (refresh-token revocation unchanged).
 - Dependency bumps: OpenTelemetry **0.221.x**, **`body-parser` 1.20.6**, **`js-yaml` 4.3.0**, **`undici` ^7.28.0**; Dockerfile base image digest pins refreshed.
+- **SQLite write queue backpressure** — when total queued depth exceeds **`dbWriteQueueBackpressureDepth`**, new **background** enqueue is rejected (`QueueBackpressureError` → **503** when surfaced through agent auth); **`dbWriteQueueMaxDepth`** (**256**) remains alert-only.
+- **SQLite transaction timeouts** — per-lane timeouts abort stuck work, recycle the sqlite pool, and allow the queue worker to continue (interactive **15s**, background **120s**).
+- **`checkFogToken` transaction scope** — agent handler runs inside the auth transaction via **`runWithTransactionContext`**, so nested **`runInTransaction()`** reuses the parent writer instead of enqueueing a second sqlite transaction.
+- **Readiness database probe (SQLite)** — **`SELECT 1`** for **`GET /api/v3/status`** runs outside the global write queue with a **5s** timeout, so health checks stay responsive under write-queue pressure.
 
 ### Fixed
 
+- **SQLite ControlPlane freeze after multi-day uptime** — a hung sqlite transaction could block the global write queue indefinitely (queue depth > **256**, console/potctl and **`/api/v3/status`** unresponsive until restart). Self-recovery: transaction timeouts, pool recycle, and background queue shedding under sustained backpressure.
+- **Agent auth nested-transaction deadlock (SQLite)** — **`checkFogToken`** authenticated in one transaction then invoked the handler outside ALS, allowing nested writes to deadlock the single-connection pool on hot agent routes (e.g. status/config polling).
 - **Secret PATCH** — omitted **`type`** in the update body now defaults to the existing secret type instead of **400** `Secret type mismatch` (fixes JSON PATCH and YAML secret updates that send only **`data`**).
 - **Embedded OAuth re-login after logout** — stale issuer session could yield **`access_denied`** on callback; logout now tears down issuer/BFF OAuth state and authorize forces fresh login.
 - **OAuth callback errors** — issuer errors (e.g. **`access_denied`**) redirect to **`{consoleUrl}/login?oauthError=...`** instead of **401 JSON** on the API port.

docs/operations/database-transactions.md

@@ -85,6 +85,10 @@ Phased reconcile and grep gates complement ALS: short txs for NATS/platform phas
 |---------|---------|--------------|
 | `settings.sqliteEnterpriseFogWarningThreshold` | 50 | `SQLITE_ENTERPRISE_FOG_WARNING_THRESHOLD` |
 | `settings.dbWriteQueueMaxDepth` | 256 | `DB_WRITE_QUEUE_MAX_DEPTH` |
+| `settings.dbWriteQueueBackpressureDepth` | 32 | `DB_WRITE_QUEUE_BACKPRESSURE_DEPTH` |
+| `settings.dbTransactionTimeoutReadinessMs` | 5000 | `DB_TRANSACTION_TIMEOUT_READINESS_MS` |
+| `settings.dbTransactionTimeoutInteractiveMs` | 15000 | `DB_TRANSACTION_TIMEOUT_INTERACTIVE_MS` |
+| `settings.dbTransactionTimeoutBackgroundMs` | 120000 | `DB_TRANSACTION_TIMEOUT_BACKGROUND_MS` |
 | `settings.dbBusyRetryMaxAttempts` | 8 | `DB_BUSY_RETRY_MAX_ATTEMPTS` |
 | `settings.dbBusyRetryBaseMs` | 25 | `DB_BUSY_RETRY_BASE_MS` |
 | `settings.reconcileOutboxDrainerIntervalSeconds` | 1 | `RECONCILE_OUTBOX_DRAINER_INTERVAL_SECONDS` |
@@ -97,7 +101,19 @@ See `src/config/config.yaml` and [architecture.md](../architecture.md) for pool
 
 ## SQLite write queue backpressure
 
-When total queued work (`interactive` + `background` lanes) exceeds `settings.dbWriteQueueMaxDepth` (default **256**), Controller logs an **error** once per overflow episode. **Interactive requests are not rejected** — the queue continues to drain in priority order. Operators should investigate background job pressure or migrate to mysql/postgres.
+When total queued work (`interactive` + `background` lanes) exceeds `settings.dbWriteQueueBackpressureDepth` (default **32**), **new background enqueue attempts are rejected** with `QueueBackpressureError` (503 to callers when surfaced through agent auth). Interactive work continues to enqueue and drain in priority order.
+
+When depth exceeds `settings.dbWriteQueueMaxDepth` (default **256**), Controller logs an **error** once per overflow episode. Interactive requests are still not rejected at the alert threshold — operators should investigate background job pressure or migrate to mysql/postgres.
+
+### Self-recovery (SQLite)
+
+| Layer | Trigger | Action |
+|-------|---------|--------|
+| **L1 timeout** | Transaction exceeds lane timeout | Reject with `TransactionTimeoutError`; worker continues |
+| **L2 pool recycle** | After interactive/background timeout | Close and re-init sqlite connection pool |
+| **L3 queue surgery** | 3+ interactive timeouts in 60s **or** backpressure sustained 30s | Shed all queued background tasks with `QueueBackpressureError` |
+
+Readiness probes (`SELECT 1` for `/api/v3/status`) run **outside** the global write queue on SQLite so health checks stay responsive when the queue is wedged. Timeout defaults to `dbTransactionTimeoutReadinessMs` (5s); failures return **503** with `Retry-After`.
 
 ---
 
@@ -110,6 +126,8 @@ Instruments are registered at startup in `src/helpers/db-metrics.js` (requires `
 | `db.transaction.duration` | histogram | `label`, `priority`, `provider` | p99 spike correlated with load |
 | `db.write_queue.depth` | gauge | `priority` | **> 100 for 5 min** → investigate background pressure |
 | `db.write_queue.wait_ms` | histogram | `priority` | Sustained high wait → scale DB or reduce background load |
+| `db.transaction.timeouts` | counter | `label`, `priority` | **Any sustained rate** → stuck tx or load spike |
+| `db.write_queue.background_shed` | counter | `reason` | **Any increment** → queue surgery fired; check stuck writers |
 | `db.busy_retries` | counter | `label` | **> 10/min** → lock contention |
 | `db.connection.invalidated` | counter | `provider` | **Any increment** → investigate pool / connection errors |
 | `db.sqlite.fog_count_warning` | counter | — | Fleet exceeded sqlite recommended size |
@@ -202,6 +220,7 @@ npm test -- --grep "grep gates"
 | **Fog platform phased reconcile** | Separate `fogPlatform.*` labels; no monolithic `fogPlatform.natsEnsure` |
 | **OIDC adapter** | All adapter reads/writes through `runInTransaction` with `oidc.adapter.*` labels |
 | **JTI cleanup** | `fog-token-cleanup-job.js` routes through `runInTransaction`, not bare manager call |
+| **Agent auth ALS** | `checkFogToken` runs the handler inside `runWithTransactionContext` so nested `runInTransaction` reuses the auth tx |
 
 When a gate fails, fix the **minimal** violation (pass parent `transaction`, move I/O outside the tx body, or split phases) — do not disable the gate.
 

src/config/config.yaml

@@ -98,11 +98,15 @@ settings:
   jobStartupDelaySeconds: 3                      # Delay before reconcile-heavy background jobs start (default: 3)
   reconcileOutboxDrainerIntervalSeconds: 1       # Outbox drainer poll interval in seconds (default: 1)
   reconcileOutboxDrainerBatchSize: 32            # Max outbox rows per drainer batch (default: 32)
-  wsSessionReconcileIntervalSeconds: 60          # WS exec/log stale DB row reconcile interval (R89)
-  sqliteEnterpriseFogWarningThreshold: 50        # Log + metric when sqlite fog count exceeds (R124)
-  dbWriteQueueMaxDepth: 256                      # SQLite write queue depth before error log (R123)
-  dbBusyRetryMaxAttempts: 8                      # SQLITE_BUSY retry attempts per transaction (Plan 19)
-  dbBusyRetryBaseMs: 25                          # Exponential backoff base for busy retry (Plan 19)
+  wsSessionReconcileIntervalSeconds: 60          # WS exec/log stale DB row reconcile interval
+  sqliteEnterpriseFogWarningThreshold: 50        # Log + metric when sqlite fog count exceeds
+  dbWriteQueueMaxDepth: 256                      # SQLite write queue depth before error log
+  dbWriteQueueBackpressureDepth: 32              # Shed background enqueue when total queue depth exceeds (sqlite)
+  dbTransactionTimeoutReadinessMs: 5000          # Readiness SELECT 1 timeout (sqlite, outside write queue)
+  dbTransactionTimeoutInteractiveMs: 15000       # Interactive sqlite transaction timeout
+  dbTransactionTimeoutBackgroundMs: 120000       # Background sqlite transaction timeout
+  dbBusyRetryMaxAttempts: 8                      # SQLITE_BUSY retry attempts per transaction
+  dbBusyRetryBaseMs: 25                          # Exponential backoff base for busy retry
 
 # Database Configuration
 database:

src/config/env-mapping.js

@@ -80,6 +80,10 @@ module.exports = {
   RECONCILE_OUTBOX_DRAINER_BATCH_SIZE: 'settings.reconcileOutboxDrainerBatchSize',
   SQLITE_ENTERPRISE_FOG_WARNING_THRESHOLD: 'settings.sqliteEnterpriseFogWarningThreshold',
   DB_WRITE_QUEUE_MAX_DEPTH: 'settings.dbWriteQueueMaxDepth',
+  DB_WRITE_QUEUE_BACKPRESSURE_DEPTH: 'settings.dbWriteQueueBackpressureDepth',
+  DB_TRANSACTION_TIMEOUT_READINESS_MS: 'settings.dbTransactionTimeoutReadinessMs',
+  DB_TRANSACTION_TIMEOUT_INTERACTIVE_MS: 'settings.dbTransactionTimeoutInteractiveMs',
+  DB_TRANSACTION_TIMEOUT_BACKGROUND_MS: 'settings.dbTransactionTimeoutBackgroundMs',
   DB_BUSY_RETRY_MAX_ATTEMPTS: 'settings.dbBusyRetryMaxAttempts',
   DB_BUSY_RETRY_BASE_MS: 'settings.dbBusyRetryBaseMs',
 

src/decorators/authorization-decorator.js

@@ -5,7 +5,7 @@ const Errors = require('../helpers/errors')
 const CODES = require('../helpers/agent-auth-error-codes')
 const { classifyCheckFogTokenFailure } = require('../helpers/agent-auth-error-utils')
 const { isTest } = require('../helpers/app-helper')
-const { runInTransaction } = require('../helpers/transaction-runner')
+const { runInTransaction, runWithTransactionContext, PRIORITY_INTERACTIVE } = require('../helpers/transaction-runner')
 
 function checkFogToken (f) {
   return async function (...fArgs) {
@@ -53,28 +53,21 @@ function checkFogToken (f) {
         throw new Errors.AgentAuthenticationError(CODES.AGENT_JWT_MISSING_SUBJECT, 'Agent JWT missing subject claim')
       }
 
-      const fog = await runInTransaction(async (transaction) => {
+      return runInTransaction(async (transaction) => {
         const foundFog = await FogManager.findOne({ uuid: fogUuid }, transaction)
         if (!foundFog) {
-          return null
+          throw new Errors.AgentAuthenticationError(CODES.AGENT_FOG_NOT_FOUND, 'Agent fog not found')
         }
 
         await FogKeyService.verifyJWT(token, fogUuid, transaction)
 
         const timestamp = Date.now()
         await FogManager.updateLastActive(foundFog.uuid, timestamp, transaction)
 
-        return foundFog
-      }, { label: 'checkFogToken' })
-
-      if (!fog) {
-        logger.error(`Fog with UUID ${fogUuid} not found`)
-        throw new Errors.AgentAuthenticationError(CODES.AGENT_FOG_NOT_FOUND, 'Agent fog not found')
-      }
+        fArgs.push(foundFog)
 
-      fArgs.push(fog)
-
-      return f.apply(this, fArgs)
+        return runWithTransactionContext(transaction, PRIORITY_INTERACTIVE, () => f.apply(this, fArgs))
+      }, { label: 'checkFogToken' })
     } catch (error) {
       const classified = classifyCheckFogTokenFailure(error)
       logger.error(`Agent authentication failed: ${classified.message}`, { code: classified.code })

src/helpers/agent-auth-error-utils.js

@@ -2,7 +2,9 @@ const { isSqliteBusyError } = require('./db-busy-retry')
 const CODES = require('./agent-auth-error-codes')
 const {
   AgentAuthenticationError,
-  ServiceUnavailableError
+  ServiceUnavailableError,
+  TransactionTimeoutError,
+  QueueBackpressureError
 } = require('./errors')
 
 function isVaultInfrastructureError (error) {
@@ -19,6 +21,9 @@ function isDatabaseInfrastructureError (error) {
   if (!error) {
     return false
   }
+  if (error instanceof TransactionTimeoutError || error instanceof QueueBackpressureError) {
+    return true
+  }
   if (isSqliteBusyError(error)) {
     return true
   }

src/helpers/db-metrics.js

@@ -10,6 +10,8 @@ let busyRetries = null
 let connectionInvalidated = null
 let sqliteFogCountWarning = null
 let writeQueueDepthInteractive = null
+let transactionTimeouts = null
+let backgroundQueueShed = null
 
 function getMeter () {
   if (!meter) {
@@ -74,6 +76,13 @@ function initDbMetrics (_sequelize, _provider, queueReader) {
       result.observe(depth.background, { priority: 'background' })
     })
   }
+
+  transactionTimeouts = m.createCounter('db.transaction.timeouts', {
+    description: 'SQLite transaction timeouts by label and priority lane'
+  })
+  backgroundQueueShed = m.createCounter('db.write_queue.background_shed', {
+    description: 'Background SQLite write queue tasks shed during recovery surgery'
+  })
 }
 
 function recordTransactionDuration (attributes, durationMs) {
@@ -100,6 +109,19 @@ function recordSqliteFogCountWarning () {
   sqliteFogCountWarning?.add(1)
 }
 
+function recordTransactionTimeout (label, priority) {
+  transactionTimeouts?.add(1, {
+    label: label || 'unknown',
+    priority: priority || 'unknown'
+  })
+}
+
+function recordBackgroundQueueShed (count, reason) {
+  if (count > 0) {
+    backgroundQueueShed?.add(count, { reason: reason || 'unknown' })
+  }
+}
+
 function maybeRecordConnectionInvalidated (error, provider) {
   if (isConnectionInvalidatedError(error)) {
     recordConnectionInvalidated(provider)
@@ -114,5 +136,7 @@ module.exports = {
   recordConnectionInvalidated,
   recordSqliteFogCountWarning,
   recordTransactionDuration,
+  recordTransactionTimeout,
+  recordBackgroundQueueShed,
   recordWriteQueueWaitMs
 }

src/helpers/errors.js

@@ -159,6 +159,26 @@ class ServiceUnavailableError extends Error {
   }
 }
 
+class TransactionTimeoutError extends Error {
+  constructor (label, priority, timeoutMs) {
+    const message = `Database transaction timed out after ${timeoutMs}ms (${label || 'unknown'}, ${priority || 'unknown'})`
+    super(message)
+    this.name = 'TransactionTimeoutError'
+    this.label = label || 'unknown'
+    this.priority = priority || 'unknown'
+    this.timeoutMs = timeoutMs
+    this.retryable = true
+  }
+}
+
+class QueueBackpressureError extends Error {
+  constructor (message = 'SQLite write queue backpressure active') {
+    super(message)
+    this.name = 'QueueBackpressureError'
+    this.retryable = true
+  }
+}
+
 class ReadinessNotReadyError extends Error {
   constructor (code, message, basePayload) {
     super(message)
@@ -184,6 +204,8 @@ module.exports = {
   AuthenticationError,
   AgentAuthenticationError,
   ServiceUnavailableError,
+  TransactionTimeoutError,
+  QueueBackpressureError,
   ReadinessNotReadyError,
   TransactionError,
   ValidationError,