fix(deploy): configure a warehouse and a fitting memory limit in the production compose

.env.app.example

@@ -28,6 +28,21 @@ KPUBDATA_BUILDER_ALLOWED_ORIGINS=https://kpubdata-studio.pages.dev
 #   python -c "import os,base64;print(base64.urlsafe_b64encode(os.urandom(32)).decode())"
 KPUBDATA_BUILDER_CREDENTIAL_MASTER_KEY=replace-with-urlsafe-base64-32-bytes
 
+# --- Warehouse 와 리소스 예산 (#993, docs/deploy.md §9) ----------------------
+# Warehouse 루트는 compose 가 /data/warehouse 로 고정한다 (builder-data 볼륨).
+# 아래는 compose 의 기본값이다. 바꿀 때는 합이 컨테이너 한도 안에 들어오게 한다:
+#   DuckDB 연결 수 × KPUBDATA_DUCKDB_MEMORY_LIMIT + KPUBDATA_QUERY_MEMORY_BUDGET_MB + 400 MB
+#   (DuckDB 연결 수 = KPUBDATA_BUILDER_MAX_WORKERS × 4 + 2)
+# 기본값: 10 × 128 MB + 768 MB + 400 MB = 2448 MB ≤ 3 GiB.
+# KPUBDATA_BUILDER_MAX_WORKERS=2
+# KPUBDATA_DUCKDB_THREADS=1
+# KPUBDATA_DUCKDB_MEMORY_LIMIT=128MB     # 64MB 이하에서는 작은 빌드도 실패한다
+# KPUBDATA_DUCKDB_MAX_TEMP_SIZE=1GB
+# KPUBDATA_QUERY_MAX_CONCURRENCY=1
+# KPUBDATA_QUERY_MAX_MEMORY_MB=768
+# KPUBDATA_QUERY_MEMORY_BUDGET_MB=768
+# BUILDER_MEMORY_LIMIT=3G
+
 # --- Caddy 공개 진입 (--profile caddy 사용 시) ------------------------------
 # 공개 도메인. Cloudflare proxied DNS가 이 호스트의 80/443으로 향한다.
 APP_DOMAIN=builder.example.com

CHANGELOG.md

@@ -7,6 +7,7 @@
 - A cross-origin page can read `Content-Disposition`, `X-Request-ID` and `Retry-After`, and the overload 503 (#995). Responses to an allowed origin carried no `Access-Control-Expose-Headers`, so a browser hid every header outside the CORS-safelisted set: Studio could not read a download's file name, the request id, or how long to wait. They are now exposed. The overload 503 is written to the socket before the request is read, so it had no CORS header at all and a browser reported it as a network error; it now carries the deployment's one allowed origin, or `*` when several are allowed (readable by a request made without credentials — a bearer header is not one), with `Retry-After` exposed, and still nothing when no origin is allowed.
 - A run whose table commit failed has one job status (#997). Such a build answers 409 with `warehouse_failures`, so `GET /builds/{run_id}` said `failed` while the job registry held it; once the registry evicted the job or the service restarted, the status was derived from the manifest, whose `status` is `ok`, and the same run read `succeeded`. The manifest path now reads a recorded `warehouse_failures` as `failed` too. The manifest is unchanged, and the contract's `BuildJob.status` says which status this case has.
 - An async build reads its submitter's uploads (#998). `POST /builds` accepted a spec with a `kind: file` source and the job then failed with `file source requires an authenticated, stable principal owner`: the worker passed the submitting owner to the manifest and to credential resolution but not to the file resolver, so only the synchronous `POST /build` could build an uploaded file. The worker now passes the same owner there too. Uploads stay isolated per owner — a job naming another owner's `upload_id` fails as it does synchronously — and a submission with no stable owner still fails with that message. The contract's `submitBuild` description says so; the wire is unchanged.
+- The production compose configures a warehouse and a memory limit its settings fit (#993). `docker-compose.prod.app.yml` set no `KPUBDATA_BUILDER_WAREHOUSE`, so every `/warehouse/*` route answered `warehouse_not_configured`, and capped the container at `512M` — below the default limit of a single DuckDB connection (`1GB`), with ten connections possible. It now sets the warehouse root to `/data/warehouse` and the DuckDB, query and worker limits explicitly (`MAX_WORKERS=2`, `DUCKDB_MEMORY_LIMIT=128MB`, `QUERY_MEMORY_BUDGET_MB=768`, each overridable from `.env`), and raises the container limit to `3G`: 10 connections × 128 MB + 768 MB + 400 MB base = 2448 MB. `tests/unit/test_prod_compose_budget.py` computes that sum from the file and fails when the limit is below it. `docs/deploy.md` §9 records the calculation, and that a DuckDB limit of `64MB` fails even a 20-row build.
 - The image's OS security patches are applied again (#1006). The `apt-get upgrade` layer was served from the build cache with packages from before `libpcre2` 10.42-1+deb12u2, so `docker build and scan` failed every pull request on CVE-2026-103111 although the fix was on the mirror. The layer now takes an `OS_PATCH_DATE` build argument as its cache key; raising the date rebuilds it.
 - A Parquet upload with a Decimal or time column builds (#979). #981 stopped the persisted Bronze file crashing on a Parquet file's native values, but the build still failed one stage later: the dataset card every build writes (#906) passed a `Decimal` or a `time` in its sample rows to `json.dumps` and raised the same `TypeError`, a 502 with no stated cause. The card now gives a cell the text every exporter gives it, alone or inside a list or struct. The persisted `raw_records.jsonl` writes each type as its one standard text instead of `str()`: a datetime is ISO 8601 with a `T` and its numeric offset (it was `2024-01-01 12:30:00+09:00`), `bytes` is hex (it was a Python repr), and a value with no such text fails the persist as before. A Parquet column holding binary or duration values — which no text output could represent and which never built — is refused at ingestion with the column named: `column 'payload' holds binary values, which a parquet upload cannot carry`. `r06_decimal_to_int` pins the Decimal→integer truncation of #918 end to end.
 - A Decimal→integer cast rounds instead of truncating toward zero (#918). `_kind` classified Decimal as "other", reaching a bare `TRY_CAST(... AS BIGINT)` — DuckDB's own cast gives 2.50 → 3 and -0.50 → -1, while Polars (the engine whose semantics Builder promises) truncates toward zero (2 and 0). The value change is silent: null count does not increase, so the cast audit cannot catch it. `_kind` now classifies Decimal as its own family, and the integer cast wraps it in `trunc()` the same way floats already are; boundary tests pin Int64 exact values, one-past-boundary NULL, and null pass-through (#975).

docker-compose.prod.app.yml

@@ -36,6 +36,24 @@ services:
       # 사용자별 Provider credential AES-GCM master key (ADR 0012). 재기동 사이에
       # 반드시 동일 값을 재주입한다 — 교체 시 기존 credential 복호화 불가.
       KPUBDATA_BUILDER_CREDENTIAL_MASTER_KEY: ${KPUBDATA_BUILDER_CREDENTIAL_MASTER_KEY:-}
+      # 테이블 카탈로그 루트 (#993). 없으면 /warehouse/* 가 전부
+      # `warehouse_not_configured` 로 답해 Studio 의 Tables·SQL·Saved Analysis 가
+      # 동작하지 않는다. builder-data 볼륨 안에 둔다.
+      KPUBDATA_BUILDER_WAREHOUSE: /data/warehouse
+      # 리소스 예산 (#993, docs/deploy.md §9). 아래 값들의 합이 deploy.resources 의
+      # memory 한도 안에 들어와야 하고, tests/unit/test_prod_compose_budget.py 가
+      # 그 식을 검사한다. 기본값은 3 GiB 컨테이너 기준이다.
+      #   DuckDB 연결 수 = MAX_WORKERS × 4 (build 당 source) + 2 (preview, composition)
+      #   메모리 = DuckDB 연결 수 × DUCKDB_MEMORY_LIMIT + QUERY_MEMORY_BUDGET_MB + 기본 400 MB
+      KPUBDATA_BUILDER_MAX_WORKERS: ${KPUBDATA_BUILDER_MAX_WORKERS:-2}
+      KPUBDATA_DUCKDB_THREADS: ${KPUBDATA_DUCKDB_THREADS:-1}
+      # 64MB 에서는 20행짜리 빌드도 ResourceLimitError 로 실패한다(실측). 그 아래로
+      # 내리지 않는다.
+      KPUBDATA_DUCKDB_MEMORY_LIMIT: ${KPUBDATA_DUCKDB_MEMORY_LIMIT:-128MB}
+      KPUBDATA_DUCKDB_MAX_TEMP_SIZE: ${KPUBDATA_DUCKDB_MAX_TEMP_SIZE:-1GB}
+      KPUBDATA_QUERY_MAX_CONCURRENCY: ${KPUBDATA_QUERY_MAX_CONCURRENCY:-1}
+      KPUBDATA_QUERY_MAX_MEMORY_MB: ${KPUBDATA_QUERY_MAX_MEMORY_MB:-768}
+      KPUBDATA_QUERY_MEMORY_BUDGET_MB: ${KPUBDATA_QUERY_MEMORY_BUDGET_MB:-768}
     ports:
       # 기본은 루프백이다. 예전 기본값("8000")은 모든 인터페이스에 공개돼서,
       # Caddy 없이 올리면 TLS 없는 8000 으로 API 키가 평문으로 오갔다. 헬스체크는
@@ -66,7 +84,9 @@ services:
     deploy:
       resources:
         limits:
-          memory: 512M
+          # 위 environment 의 리소스 예산과 함께 바꾼다 (#993). 512M 은 DuckDB 연결
+          # 하나의 기본 한도(1GB)보다도 작았다.
+          memory: ${BUILDER_MEMORY_LIMIT:-3G}
 
   # 공개 진입 리버스 프록시. Cloudflare → Caddy → builder.
   # Cloudflare SSL mode: Full(strict). `--profile caddy`로 opt-in.

docs/deploy.md

@@ -196,6 +196,25 @@ query 메모리     = KPUBDATA_QUERY_MEMORY_BUDGET_MB                      (질
 build 가 실패하지 않고 느려진다. thread 8 + query 1 은 1 vCPU 를 넘지만 thread 는 CPU 를 나눠 쓸 뿐
 메모리를 늘리지 않는다. 임시 디스크 8 GB 는 컨테이너의 쓰기 가능한 디스크 안에 있어야 한다.
 
+**운영 compose(`docker-compose.prod.app.yml`)의 기본값** (#993). 컨테이너 한도는 `3G` 이고 아래 합이
+그 안에 들어온다. `tests/unit/test_prod_compose_budget.py` 가 compose 파일에서 같은 식을 계산해, 한도가
+합보다 작아지면 실패한다.
+
+| 항목 | 기본값 | 합 |
+| :--- | :--- | :--- |
+| DuckDB 연결 (build 2 × source 4 + preview 1 + composition 1 = 10) | `KPUBDATA_BUILDER_MAX_WORKERS=2`, `KPUBDATA_DUCKDB_MEMORY_LIMIT=128MB` | 1280 MB |
+| query child | `KPUBDATA_QUERY_MAX_CONCURRENCY=1`, `KPUBDATA_QUERY_MAX_MEMORY_MB=768`, `KPUBDATA_QUERY_MEMORY_BUDGET_MB=768` | 768 MB |
+| 기본 프로세스·HTTP·여유 | 검사가 쓰는 고정값 | 400 MB |
+| **합** | | **2448 MB ≤ 3072 MB** |
+
+`KPUBDATA_DUCKDB_MEMORY_LIMIT` 을 `64MB` 로 두면 20행짜리 테이블의 빌드도
+`the table needs more memory or temporary disk than this deployment allows` 로 실패한다(parity
+시나리오로 실측, `96MB` 부터 통과). 그래서 운영 기본값은 `128MB` 이고, 위 "1 vCPU / 2 GiB 예시"의
+`96MB` 는 여유가 거의 없는 하한이다. 2 GiB 호스트에 맞추려면 `KPUBDATA_DUCKDB_MEMORY_LIMIT` 을 더
+내리지 말고 `KPUBDATA_BUILDER_MAX_WORKERS` 나 query 예산을 줄인다. warehouse 는
+`KPUBDATA_BUILDER_WAREHOUSE=/data/warehouse` 로 켜져 있다 — 없으면 `/warehouse/*` 가
+`warehouse_not_configured` 로 답한다.
+
 query worker 의 종료는 네 경로 모두에서 child process 를 남기지 않는다: 성공, 실패(child 가 스스로
 끝나지 않아도 1초 뒤 terminate), timeout(terminate 를 무시하면 kill), 요청 취소(대기 중인 요청 thread 가
 중단되면 child 도 멈춘다). 클라이언트가 연결을 끊은 것은 서버가 따로 감지하지 않는다 — 그 질의는 timeout

tests/unit/test_prod_compose_budget.py

@@ -0,0 +1,116 @@
+"""The production compose configures a warehouse and a memory limit its own settings fit (#993).
+
+``docker-compose.prod.app.yml`` set no warehouse root, so every ``/warehouse/*`` route
+answered ``warehouse_not_configured``, and capped the container at 512M — less than the
+default limit of one DuckDB connection. This computes the budget ``docs/deploy.md`` §9
+describes from the file's own defaults and fails when the container limit is below it.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+from typing import Any
+
+import pytest
+import yaml
+
+_ROOT = Path(__file__).resolve().parents[2]
+_COMPOSE = _ROOT / "docker-compose.prod.app.yml"
+_ENV_EXAMPLE = _ROOT / ".env.app.example"
+
+#: Sources a build fetches at once (``_MAX_PARALLEL_SOURCES``), each with a connection.
+_SOURCES_PER_BUILD = 4
+#: Connections outside builds: one preview, one composition.
+_EXTRA_CONNECTIONS = 2
+#: The server process, HTTP threads and headroom (docs/deploy.md §9).
+_BASE_MB = 400
+
+_DEFAULT = re.compile(r"^\$\{[A-Z0-9_]+:-(?P<default>[^}]*)\}$")
+_UNITS = {"B": 1, "KB": 10**3, "MB": 10**6, "GB": 10**9, "KIB": 2**10, "MIB": 2**20, "GIB": 2**30}
+_COMPOSE_UNITS = {"K": 2**10, "M": 2**20, "G": 2**30}
+
+
+def _default(value: object) -> str:
+    """What compose substitutes for ``${VAR:-default}`` when VAR is unset."""
+    text = str(value)
+    match = _DEFAULT.match(text)
+    return match.group("default") if match else text
+
+
+def _duckdb_bytes(value: str) -> int:
+    match = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([A-Za-z]+)\s*", value)
+    assert match, value
+    return int(float(match.group(1)) * _UNITS[match.group(2).upper()])
+
+
+def _compose_bytes(value: str) -> int:
+    match = re.fullmatch(r"(\d+)([KMG])", value.upper())
+    assert match, value
+    return int(match.group(1)) * _COMPOSE_UNITS[match.group(2)]
+
+
+def required_bytes(environment: dict[str, Any]) -> int:
+    """The memory the service's own settings can ask for at once."""
+    workers = int(_default(environment["KPUBDATA_BUILDER_MAX_WORKERS"]))
+    connections = workers * _SOURCES_PER_BUILD + _EXTRA_CONNECTIONS
+    duckdb = connections * _duckdb_bytes(_default(environment["KPUBDATA_DUCKDB_MEMORY_LIMIT"]))
+    query = int(_default(environment["KPUBDATA_QUERY_MEMORY_BUDGET_MB"])) * 2**20
+    return duckdb + query + _BASE_MB * 2**20
+
+
+@pytest.fixture(scope="module")
+def builder() -> dict[str, Any]:
+    compose = yaml.safe_load(_COMPOSE.read_text(encoding="utf-8"))
+    service: dict[str, Any] = compose["services"]["builder"]
+    return service
+
+
+def test_the_warehouse_root_is_on_the_data_volume(builder: dict[str, Any]) -> None:
+    root = _default(builder["environment"]["KPUBDATA_BUILDER_WAREHOUSE"])
+
+    assert root.startswith("/data/")
+    assert "builder-data:/data" in builder["volumes"]
+
+
+def test_the_container_limit_covers_the_configured_budget(builder: dict[str, Any]) -> None:
+    limit = _compose_bytes(_default(builder["deploy"]["resources"]["limits"]["memory"]))
+
+    assert limit >= required_bytes(builder["environment"])
+
+
+def test_a_query_reserves_no_more_than_the_query_budget(builder: dict[str, Any]) -> None:
+    """A query that reserves more than the budget can never be admitted."""
+    environment = builder["environment"]
+
+    assert int(_default(environment["KPUBDATA_QUERY_MAX_MEMORY_MB"])) <= int(
+        _default(environment["KPUBDATA_QUERY_MEMORY_BUDGET_MB"])
+    )
+
+
+def test_the_old_512m_limit_fails_the_check(builder: dict[str, Any]) -> None:
+    """The gate itself: the limit this replaces does not cover the budget."""
+    assert _compose_bytes("512M") < required_bytes(builder["environment"])
+
+
+def test_the_default_duckdb_limit_alone_overruns_a_small_container() -> None:
+    """With no DuckDB limit set a connection may take 1GB: ten of them need over 10 GB."""
+    environment = {
+        "KPUBDATA_BUILDER_MAX_WORKERS": "2",
+        "KPUBDATA_DUCKDB_MEMORY_LIMIT": "1GB",
+        "KPUBDATA_QUERY_MEMORY_BUDGET_MB": "768",
+    }
+
+    assert required_bytes(environment) > _compose_bytes("3G")
+
+
+def test_the_env_example_names_every_budget_variable(builder: dict[str, Any]) -> None:
+    text = _ENV_EXAMPLE.read_text(encoding="utf-8")
+    overridable = [
+        name
+        for name, value in builder["environment"].items()
+        if name.startswith(("KPUBDATA_DUCKDB_", "KPUBDATA_QUERY_")) and _DEFAULT.match(str(value))
+    ]
+
+    assert overridable
+    assert [name for name in [*overridable, "BUILDER_MEMORY_LIMIT"] if name not in text] == []