Confine SQL execution to registered snapshots and correct its security contract

README.md

@@ -58,7 +58,7 @@ object storage, derives a typed schema, samples + profiles each
 column, runs a PII pass, proposes joins to your other tables, and
 fits it to a long-lived knowledge base. Then you ask in English. A
 multi-agent pipeline (Grounding → Generation → Critic → Explainer)
-produces governed DuckDB SQL, runs it in a sandboxed connection
+produces governed DuckDB SQL, runs it over registered local Parquet snapshots
 under per-token scopes, returns rows + a chart hint, and keeps a
 conversation thread for drill-down.
 
@@ -96,9 +96,10 @@ supersession.
 
 **flyquery** handles **structured tabular data**. Its model is: file → typed
 schema → Parquet on object storage → DuckDB SQL via multi-agent pipeline. The
-query language is natural-language-to-SQL. The storage engine is Parquet on
-object storage queried in-process by DuckDB. The retrieval primitive is
-schema-element retrieval + table resolution + AST firewall. The user-facing
+query language is natural-language-to-SQL. ObjectStore holds the Parquet
+snapshots, with registered local snapshots queried in-process by DuckDB. The
+retrieval primitive is schema-element retrieval + table resolution + AST
+firewall. The user-facing
 concepts are datasets, tables, snapshots, examples, and metrics.
 
 The query languages, storage engines, retrieval primitives, and user-facing
@@ -211,7 +212,7 @@ curl -s -X POST http://localhost:8520/api/v1/query \
 | **Compression** | .gz, .zip, .bz2 transparently unwrapped |
 | **10-stage pipeline** | Receive → Parse → Reconcile → Sample → Profile → Relations → Describe → PII → Embed → Publish |
 | **4-agent query pipeline** | GroundingAgent → GenerationAgent → CriticAgent → ExplainerAgent |
-| **AST firewall** | sqlglot + DuckDB double-check; single-statement SELECT only on uploaded tables |
+| **AST firewall** | Closed sqlglot admission, registered table checks, and an independent executor check; exact file allowlist with external access disabled ([SQL read boundary](docs/security-model.md#sql-read-boundary)) |
 | **Schema knowledge base** | pgvector HNSW + BM25 + cross-encoder reranker |
 | **Semantic layer** | OSI/MetricFlow YAML metrics, deterministic SQL compilation |
 | **Conversation memory** | Multi-turn drill-down with snapshot pinning |
@@ -280,11 +281,14 @@ with `-m 'not llm'`.
 
 ## Architecture quick-take
 
-flyquery uses a **two-tier storage design**: Parquet on object storage (LocalFs in
-dev; S3/GCS/Azure Blob in production) queried by DuckDB in-process per request.
-Postgres stores all metadata, embeddings, relations, audit, and agent tokens.
-There is no persistent DuckDB database — every query ATTACHes Parquet snapshots
-fresh, eliminating shared mutable state between requests.
+flyquery stores Parquet through ObjectStore adapters and queries registered
+local snapshots with DuckDB in a fresh in-memory connection per request.
+Postgres stores metadata, embeddings, relations, audit, and agent tokens.
+Cloud storage and ingestion adapters are separate from the qualified local
+query workflow. The [SQL read boundary](docs/security-model.md#sql-read-boundary)
+explains the exact file allowlist, copy-on-write, and upstream authorization.
+Statement timeouts are currently unenforced; these in-process controls do not
+constitute an operating-system SQL sandbox.
 
 The **hexagonal adapter ring** means every external boundary (object store, file
 reader, vector store, PII scanner, semantic compiler, rate limiter) is a Python

docs/api-reference.md

@@ -753,12 +753,16 @@ the full pipeline completes.
 {
   "question": "Show top 10 customers by revenue last month",
   "dataset_id": "01906f2a-...",
-  "conversation_id": null,
-  "row_cap": 1000,
-  "statement_timeout_ms": 30000
+  "conversation_id": null
 }
 ```
 
+The current request accepts `question`, `dataset_id`, and optional
+`conversation_id`. Row caps use the executor's global setting; statement
+timeouts are unenforced. There are no per-request `row_cap` or
+`statement_timeout_ms` overrides. See the
+[SQL read boundary](security-model.md#sql-read-boundary).
+
 **Response** `200 OK`
 ```json
 {
@@ -864,8 +868,7 @@ Available only when `workspace.allow_direct_sql = true`.
 ```json
 {
   "sql": "SELECT COUNT(*) FROM orders",
-  "dataset_id": "01906f2a-...",
-  "row_cap": 500
+  "dataset_id": "01906f2a-..."
 }
 ```
 
@@ -1015,7 +1018,7 @@ Source: `queries_controller.py:124` (get_query_result).
 
 Aggregates `flyquery_cost_events` into day / week / month buckets
 scoped to the caller's workspace. Consume
-[`GET /api/v1/cost-events`](#cost-events) for the raw per-call ledger;
+[`GET /api/v1/cost-events`](#get-apiv1cost-events) for the raw per-call ledger;
 this endpoint just rolls it up.
 
 **Query params**: `period` = `day` (default) | `week` | `month`,

docs/architecture.md

@@ -44,8 +44,8 @@ flyquery's specific responsibilities:
   operator approves.
 - **Query execution** — multi-agent pipeline
   (Grounding → Generation → Executor → Critic → Explainer) gated by per-token
-  scopes, an AST firewall, row caps, statement timeouts, PII scanning, and
-  conversation-drill-down memory.
+  scopes, closed AST admission, registered snapshot paths, row caps, PII
+  scanning, and conversation-drill-down memory.
 - **Audit trail** — every upload, snapshot diff, question, candidate SQL,
   executed statement, and cost event, append-only.
 
@@ -219,9 +219,12 @@ flyquery uses a two-tier storage design:
    Backed by `LocalFs` (dev), `S3`, `GCS`, or `Azure Blob` via the `ObjectStore`
    hexagonal port. Selected by `FLYQUERY_OBJECT_STORE`.
 
-2. **DuckDB** — in-process per-request query engine. ATTACHes the Parquet
-   snapshots at query time; no persistent DuckDB database. This eliminates
-   shared mutable state between requests.
+2. **DuckDB** — in-process per-request query engine. Creates views over
+   registered local Parquet snapshots in a fresh in-memory connection.
+   Connections have exact file allowlists and locked external-access settings.
+   See the [SQL read boundary](security-model.md#sql-read-boundary) for admission
+   rules, copy-on-write, and the unenforced statement timeout. ObjectStore
+   provider support does not establish remote query support.
 
 3. **Postgres + pgvector** — metadata, schema knowledge base, relations,
    embeddings, audit, agent tokens. The single persistent transactional store.
@@ -468,17 +471,20 @@ GroundingAgent  →  GroundedContext {
                                                          merge ◄──┘
        │
        ▼
-AST classifier (sqlglot + DuckDB parse) + scope/firewall check
+AST classifier (sqlglot) + ScopeGuard
   single-statement only
-  reject DDL on uploaded tables
+  reject DDL and unknown statements
   reject INSERT/UPDATE/DELETE on uploaded tables
   reject SQL referencing tables outside dataset_allowlist
-  reject DuckDB host-touching functions except httpfs/parquet/json/arrow
+  require registered physical tables and reviewed pure function names
        │
        ▼
 DuckDB executor (in-process, per-request)
-  ATTACH Parquet snapshots as virtual tables
-  SET memory_limit, statement_timeout, LIMIT row_cap+1
+  recheck SQL and supplied registered snapshots
+  create views over registered local Parquet; exact allowed_paths
+  external access / extension autoload off; configuration locked
+  SET memory_limit; LIMIT default_row_cap+1 (SELECT)
+  statement timeout is unenforced
        │
   ┌────┴──────┐
   OK        ERROR
@@ -503,6 +509,10 @@ Recorder → flyquery_queries + audit_events + cost_events
 
 ### Query-pipeline agents
 
+Direct SQL skips the agents and uses the same admission policy with registered
+metadata in both synchronous and SSE handlers. See the
+[SQL read boundary](security-model.md#sql-read-boundary).
+
 | Agent | Output type | Notes |
 |-------|-------------|-------|
 | `GroundingAgent` | `GroundedContext` | Iterative schema expansion up to `EXPAND_ITERS=2` |
@@ -554,16 +564,14 @@ class FileReader(Protocol):
 Implementations: `csv_reader`, `excel_reader`, `json_reader`,
 `parquet_reader`, `avro_reader`, `orc_reader`, `arrow_reader`.
 
-### DialectAdapter port (DuckDB only in v0)
+### Query execution services (DuckDB)
 
-```python
-class DialectAdapter(Protocol):
-    dialect: Literal["duckdb"]
-    async def execute(self, sql, *, attached_tables, row_cap, ...) -> ExecutionResult: ...
-    async def explain(self, sql, *, attached_tables) -> ExplainPlan: ...
-    def ast_classify(self, sql) -> AstClassification: ...
-    def lint(self, sql) -> list[LintFinding]: ...
-```
+`AstClassifier` classifies SQL with sqlglot. `ScopeGuard` checks registered
+metadata and grants before `DuckDBExecutor.execute(sql, attached_tables,
+derived_tables=None)` runs the statement. The executor reads the global
+memory and row-cap settings when constructed. It does not accept a
+per-request timeout or row-cap argument. See the
+[SQL read boundary](security-model.md#sql-read-boundary).
 
 ### Other ports (shared with canon)
 

docs/concurrency.md

@@ -185,46 +185,43 @@ The regression test that covers the drain-with-cancel path is in
 
 ## 5. DuckDB per-request isolation
 
-Every query that hits the DuckDB executor (`POST /query`, `POST /sql:execute`)
-creates a new in-process DuckDB connection:
+Each admitted query execution (`POST /query`, `POST /sql:execute`) creates a
+fresh in-memory DuckDB connection inside the synchronous worker body.
+The asynchronous entry point hands off the request as follows:
 
 ```python
-import duckdb
-
-async def execute(self, sql, *, attached_tables, row_cap, statement_timeout_ms, memory_limit):
-    conn = duckdb.connect(":memory:", read_only=False)  # fresh per call
-    try:
-        conn.execute(f"SET memory_limit='{memory_limit}'")
-        conn.execute(f"SET threads=1")
-        for qname, parquet_key in attached_tables.items():
-            conn.execute(f"ATTACH '{parquet_key}' AS {qname} (READ_ONLY)")
-        # ... execute sql with LIMIT row_cap+1
-    finally:
-        conn.close()
+import asyncio
+
+async def execute(self, sql, attached_tables, derived_tables=None):
+    return await asyncio.to_thread(
+        self._sync, sql, dict(attached_tables), dict(derived_tables or {})
+    )
 ```
 
 Properties of this design:
 
 - **No shared state.** There is no persistent DuckDB database on disk. Each
   connection starts empty.
-- **No shared ATTACH.** Tables are ATTACHed fresh per query. Two concurrent
-  queries for the same dataset attach the same Parquet files independently.
-- **Read-only mode on ingested tables.** `ATTACH ... (READ_ONLY)` prevents
-  writes to Parquet even if the AST firewall were bypassed.
-- **Memory bounded.** `SET memory_limit` caps each connection. With N
-  concurrent queries at 4 GB each, API pods need `N × 4 GB + OS overhead`
-  RAM.
+- **Per-request views.** Registered local Parquet paths become views in each
+  connection. Concurrent queries do not share view or mutable-table state.
+- **Confined file access.** Exact `allowed_paths`, disabled external access
+  and extension autoload/install, and locked configuration restrict the
+  connection. DERIVED DML writes a new scratch Parquet output and leaves the
+  original snapshot unchanged.
+- **Memory budget.** `SET memory_limit` configures each connection's budget.
+  Size API pods for concurrent query memory and process overhead; this
+  setting is not an operating-system process limit. The current executor
+  fixes `threads=2` per connection.
+
+The connection is not read-only. SQL admission and snapshot checks precede
+execution; see the [SQL read boundary](security-model.md#sql-read-boundary)
+for the full policy. The global statement-timeout setting is unenforced,
+so the SELECT row cap does not bound the time required to compute rows.
 
 **Concurrency concern:** DuckDB in-process with asyncio requires care. DuckDB
 operations are synchronous and CPU-bound. They block the asyncio event loop
-thread. flyquery runs DuckDB calls in a thread pool executor:
-
-```python
-import asyncio
-result = await asyncio.get_event_loop().run_in_executor(
-    _duckdb_thread_pool, _execute_sync, conn, sql
-)
-```
+thread. flyquery moves the synchronous execution body to a worker thread
+with `asyncio.to_thread`, as shown above.
 
 This keeps the asyncio event loop responsive for incoming requests while
 DuckDB computes.

docs/deployment-topology.md

@@ -3,8 +3,8 @@
 ## Table of Contents
 
 1. [Overview](#1-overview)
-2. [Single-node (docker-compose)](#2-single-node-docker-compose)
-3. [Multi-node (separate services)](#3-multi-node-separate-services)
+2. [Single-node (docker-compose)](#2-single-node-docker-compose--dev)
+3. [Multi-node (separate services)](#3-multi-node-production--separate-services)
 4. [HA considerations](#4-ha-considerations)
 5. [Service port reference](#5-service-port-reference)
 
@@ -26,6 +26,12 @@ the ingest worker. The retention worker doesn't run pipeline stages
 either — it polls on a fixed interval and only writes recovery /
 cleanup SQL.
 
+The storage-provider choices in these topologies describe ObjectStore
+storage and ingestion. Query execution is qualified for registered local
+Parquet snapshots accessible to the API process. Remote SQL file reads and
+extension-backed query workflows remain outside that qualification; see the
+[SQL read boundary](security-model.md#sql-read-boundary).
+
 For dev / docker-compose convenience, `flyquery worker all` runs both
 workers in a single asyncio event loop — **dev only**, production must
 split them.

docs/deployment.md

@@ -140,10 +140,16 @@ is the canonical reference. Copy to `.env` and edit.
 |----------|---------|-------|
 | `FLYQUERY_DUCKDB_MEMORY_LIMIT` | `4GB` | Per-connection memory cap |
 | `FLYQUERY_DEFAULT_ROW_CAP` | `1000` | Default result row limit |
-| `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` | `30000` | Default query timeout (30 s) |
+| `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` | `30000` | Configuration field; currently unenforced by DuckDBExecutor |
 | `FLYQUERY_RESULT_TTL_HOURS` | `24` | Presigned URL and result object TTL |
 | `FLYQUERY_RESULT_PREVIEW_MAX_BYTES` | `131072` | Preview JSON size cap (128 KiB) |
-| `FLYQUERY_DUCKDB_HTTPFS` | `true` | Enable httpfs for cloud object storage |
+| `FLYQUERY_DUCKDB_HTTPFS` | `true` | Configuration field; does not enable httpfs in DuckDBExecutor |
+
+The qualified query workflow uses registered local Parquet with DuckDB >= 1.3.2.
+External access and extension autoload/install remain disabled; see the
+[SQL read boundary](security-model.md#sql-read-boundary). Query requests have
+no per-request row-cap or statement-timeout override, and the global timeout
+setting does not stop a long-running statement.
 
 ### Ingestion
 
@@ -314,6 +320,11 @@ logging middleware.
 
 ## 10. Object storage configuration
 
+These settings select ObjectStore adapters for stored uploads, ingestion,
+and results. Cloud storage adapter availability does not qualify cloud-backed
+SQL execution. The current query boundary is qualified for registered local
+Parquet; see the [SQL read boundary](security-model.md#sql-read-boundary).
+
 ### Local filesystem (dev)
 
 ```

docs/glossary.md

@@ -178,13 +178,15 @@ lists: `score = 1 / (k + rank)` where `k=FLYQUERY_RRF_K` (default 60).
 Ranks from BM25 and pgvector are fused before the cross-encoder reranker.
 
 **AST classifier**
-The component (sqlglot + DuckDB parser double-check) that classifies
-generated SQL as SELECT, INSERT, UPDATE, DELETE, or DDL, and enforces the
-AST firewall rules. See [security-model.md § 6](security-model.md#6-ast-firewall).
+The sqlglot component that classifies generated, rewritten, or direct SQL
+and admits only reviewed pure function names and supported table sources.
+`ScopeGuard` checks registered metadata and grants; the executor independently
+reclassifies SQL. See the [SQL read boundary](security-model.md#sql-read-boundary).
 
 **ScopeGuard**
-The component that checks the intersection of token scopes, workspace/dataset
-allowlists, AST classification, and table kind before allowing execution.
+The component that checks SQL classification, token scopes, table kind,
+registered table metadata, and dataset allowlists before execution. Actor
+authentication, workspace restrictions, and Postgres RLS remain upstream.
 
 **MetricFlow**
 The OSI/MetricFlow-shape YAML format used for semantic metrics in flyquery.
@@ -212,8 +214,9 @@ changes ship as `Patch+1` within the same month, signalled in CHANGELOG.
 
 **httpfs**
 A DuckDB extension that enables reading remote files via HTTP/S3/GCS/Azure.
-flyquery loads httpfs (along with parquet, json, arrow) and restricts its
-paths to the object-store prefix. No other DuckDB extensions are loaded.
+The current query executor does not load httpfs and disables extension
+autoload/install and external access. Remote query workflows are unqualified;
+see the [SQL read boundary](security-model.md#sql-read-boundary).
 
 **presigned URL**
 A time-limited, single-object URL generated by the ObjectStore for result

docs/operations-runbook.md

@@ -25,7 +25,10 @@ full teardown.
 - Redis (optional; required for `FLYQUERY_EDA_ADAPTER=redis` or
   `FLYQUERY_RATE_LIMIT_BACKEND=redis`).
 - Object storage: local directory, S3 bucket, GCS bucket, or Azure Blob
-  container. IAM/service-account credentials configured.
+  container for storage/ingestion, with IAM/service-account credentials
+  configured. Query execution is qualified for registered local Parquet
+  snapshots accessible to the API process; see the
+  [SQL read boundary](security-model.md#sql-read-boundary).
 - `ANTHROPIC_API_KEY` (or `OPENAI_API_KEY` if using GPT-4o fallback).
 
 ### Step-by-step
@@ -246,23 +249,17 @@ the `TenantContextMiddleware` log output.
 
 ### Agent timeouts
 
-**Symptom:** Query fails with `error_code=AGENT_TIMEOUT` or
-`execution_status=FAILED` after ~30 seconds.
+**Symptom:** The query reports a provider timeout or stalls during execution.
 
-**Cause:** LLM API is slow; `FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` or
-`FLYQUERY_AGENT_MAX_OUTPUT_TOKENS` is too tight.
+**Action:** Identify the stage in the query and agent logs. For a provider
+failure, inspect rate-limit and latency errors from the configured provider.
+For DuckDB, inspect the query plan and reduce expensive scans or joins.
 
-**Fix:**
-```bash
-# Increase statement timeout (ms)
-export FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS=60000
-
-# Check LLM API health
-curl https://api.anthropic.com/v1/messages \
-  -H "x-api-key: $ANTHROPIC_API_KEY" \
-  -H "Content-Type: application/json" \
-  -d '{"model":"claude-haiku-4-5","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}'
-```
+`FLYQUERY_DEFAULT_STATEMENT_TIMEOUT_MS` is currently unenforced by
+DuckDBExecutor, and query requests have no timeout override. Raising it
+does not stop a long-running SQL statement. The SELECT row cap limits
+returned rows, not computation time; see the
+[SQL read boundary](security-model.md#sql-read-boundary).
 
 ---
 
@@ -625,7 +622,7 @@ FOLLOW-UP ACTIONS:
 ### Key runbook pointers
 
 - Worker fleet ops → [§7 Worker fleet](#7-worker-fleet) and [workers.md](workers.md)
-- Stuck worker → [§3 Stuck ingest job](#stuck-ingest-job)
+- Stuck worker → [§3 Stuck ingest job](#ingest-job-stuck-in-running)
 - RLS returning empty → [§3 RLS returning zero rows](#rls-returning-zero-rows)
 - GDPR request → [§4 GDPR purge](#4-gdpr-purge-runbook)
 - Scaling → [§8 Scaling up](#8-scaling-up-more-workers)