Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@

### Changed

- The kpubdata pin moves to the 0.9 line (`kpubdata>=0.9.0,<0.10`, #990, #1002, kpubdata#780). 0.9.0 is the first release with `Client(env_keys=False)`: on 0.8.0 `serve` refused to start in a multi-user deployment because the operator's keys could not be kept out of a request, and now it starts. Seven imports from kpubdata's private modules — `find_spec`, `discover_specs`, `KPubDataConfig` and `SENSITIVE_PARAM_KEYS` — use the public names, and their allowlist entries are gone. **What changes for a deployment:** kpubdata 0.9.0 declares terms for more datasets, and Builder acts on them — `datago.air_quality` is now `forbidden` (no publish, and no rows through query, preview, exports or downloads), and the 16 specs whose `allowed` was never confirmed are `unknown` (no public publish). kpubdata's own breaking changes apply too: HTTP 401 is `AuthError`, a total count of zero is `0`, NODATA is an empty result.
- The automated deployment to the `app-01` VM is removed. `.github/workflows/deploy.yml` ran on every push to `main` that touched the service, built and pushed an image, then failed at its SSH step — the deployment it was written for is not being used. The workflow and `ops/deploy/app-01-rollout.sh`, which only it ran, are deleted. `docker.yml` still publishes the images to GHCR, and `docker-compose.prod.app.yml` is unchanged for anyone who runs it by hand. `docs/deploy.md`, `docs/CREDENTIAL_SURFACE.md` and ADR 0017 say so.
- A pull request builds the documentation site in strict mode. `mkdocs build --strict` ran only in `docs.yml`, on a push to `main`, so the link #1026 fixed was found after it had merged and the deploy had failed. `ci.yml` has a `Docs build (strict)` job now and the `CI gate` waits for it; `docs.yml` still does the deploy. `tests/unit/test_ci_docs_gate.py` holds that the job exists, that the gate needs it and that it runs the command the deploy runs.
- The contract declares four things the service already sent (#994, API contract 1.80.0, description only). `X-Provider-Key` is a header parameter of the five operations that call a provider (`previewBuild`, `createBuild`, `submitBuild`, `getProviderStatus`, `testProviderConnection`) — it was mentioned only in `info.description`, so a client's drift test had nothing to check it against. The 429 `auth_throttled` and the overload 503 are shared responses any operation can answer (`components.responses.AuthThrottled`, `ServerOverloaded`), and `X-Request-ID` is `components.headers.RequestId`. `tests/unit/test_contract_shared_declarations.py` compares each with what the code sends. Generating the route table from the route adapter, the remaining part of #994, is not done.
Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,11 +102,11 @@ CI(`publish-dataset.yml`)와 패키지 배포 환경에서는 `--no-sources` 플
uv sync --extra dev --extra publish --no-sources
```

`--no-sources`는 `[tool.uv.sources]`를 무시하고, `pyproject.toml`의 `dependencies`에 명시된 **PyPI 릴리스 핀**(`kpubdata>=0.8.0,<0.9`)을 직접 설치합니다. 형제 디렉터리가 없어도 작동합니다.
`--no-sources`는 `[tool.uv.sources]`를 무시하고, `pyproject.toml`의 `dependencies`에 명시된 **PyPI 릴리스 핀**(`kpubdata>=0.9.0,<0.10`)을 직접 설치합니다. 형제 디렉터리가 없어도 작동합니다.

#### 핀 범위(`>=0.8.0,<0.9`)를 이렇게 설정한 이유
#### 핀 범위(`>=0.9.0,<0.10`)를 이렇게 설정한 이유

`kpubdata-builder`는 `kpubdata`의 공개 API만 씁니다(kpubdata ADR 0007, `scripts/check_kpubdata_imports.py`). kpubdata 0.8.0은 코드 열의 앞자리 0을 보존해 `str`로 돌려주는 breaking 릴리스이며(kpubdata#613), #702·#688·#689가 기다리던 라이선스·열 메타데이터 필드를 더합니다. 0.9 이상은 검증 전이라 허용하지 않습니다 (관련 이슈: #213). CI 의 `kpubdata <버전>` 잡이 범위 안의 모든 릴리스로 스위트를 돌립니다(#832).
`kpubdata-builder`는 `kpubdata`의 공개 API만 씁니다(kpubdata ADR 0007, `scripts/check_kpubdata_imports.py`). kpubdata 0.9.0은 `Client(env_keys=False)` 가 들어간 첫 릴리스입니다 — 다중 사용자 모드가 운영자 키를 요청에서 떼어 놓는 데 필요합니다(#990, kpubdata#780). breaking 릴리스이기도 합니다: 모르는 `Client` 인자를 거부하고, HTTP 401 이 `AuthError` 가 되고, 건수 0 을 `None` 이 아니라 `0` 으로 보고하고, NODATA 가 모든 경로에서 빈 결과입니다. 그래서 0.9 라인에 고정합니다.

핀의 정본은 `pyproject.toml`의 `dependencies`입니다. 이 표와 어긋나면 `pyproject.toml`이 맞습니다.

Expand All @@ -115,7 +115,7 @@ uv sync --extra dev --extra publish --no-sources
| 환경 | 명령 | `kpubdata` 소스 |
| :--- | :--- | :--- |
| 로컬 개발 | `uv sync --extra dev` | `../kpubdata` (editable, 형제 디렉터리 필요) |
| CI / 배포 | `uv sync ... --no-sources` | PyPI (`kpubdata>=0.8.0,<0.9`) |
| CI / 배포 | `uv sync ... --no-sources` | PyPI (`kpubdata>=0.9.0,<0.10`) |

## 3. 브랜치 전략과 협업 규칙

Expand Down
11 changes: 6 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,12 @@ classifiers = [
]
dependencies = [
# Builder depends on the kpubdata public API only (ADR 0007 in kpubdata,
# scripts/check_kpubdata_imports.py). 0.8.0 is breaking: code columns keep their
# leading zeros and come back as str (kpubdata#613), and it adds the licence and
# column-metadata fields #702, #688 and #689 wait on. Pin to the 0.8 line; compat
# policy: #213.
"kpubdata>=0.8.0,<0.9",
# scripts/check_kpubdata_imports.py). 0.9.0 is the first release with
# `Client(env_keys=False)`, which the multi-user mode needs to keep the operator's
# keys out of a request (#990, kpubdata#780), and it is breaking: unknown `Client`
# arguments are refused, HTTP 401 is `AuthError`, a total count of zero is `0`, and
# NODATA is an empty result on every path. Pin to the 0.9 line; compat policy: #213.
"kpubdata>=0.9.0,<0.10",
"cryptography>=50,<51",
# The tabular engine the DuckDB migration moves to (ADR 0021, #866). 1.2.0 is the first
# release with every sandbox setting the query profile needs — allowed_paths,
Expand Down
30 changes: 0 additions & 30 deletions scripts/check_kpubdata_imports.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,31 +62,9 @@
"and transport; kpubdata has no public verify API (kpubdata#667). Signatures are "
"pinned by tests/unit/test_kpubdata_internal_surface.py."
)
_SPEC_LOOKUP_REASON = (
"Looks up a SpecDefinition by id; the public Client returns DatasetRef, not the "
"spec model verify needs (kpubdata#667)."
)
_CONFIG_REASON = (
"Reads the operator's provider key the way kpubdata resolves it from the "
"environment; Client does not expose key resolution publicly (kpubdata#667)."
)

#: (path, module, name) -> reason. ``name`` is "" for ``import module``.
ALLOWLIST: dict[tuple[str, str, str], str] = {
(
"src/kpubdata_builder/logging_redaction.py",
"kpubdata.transport._sensitive",
"SENSITIVE_PARAM_KEYS",
): (
"The one canonical list of credential parameter names; Builder keeps no copy "
"so the two cannot drift. Needs a public export (kpubdata#667)."
),
("src/kpubdata_builder/service/providers.py", "kpubdata.config", "KPubDataConfig"): (
_CONFIG_REASON
),
("src/kpubdata_builder/verify/runner.py", "kpubdata.config", "KPubDataConfig"): (
_VERIFY_REASON
),
("src/kpubdata_builder/verify/runner.py", "kpubdata.core.executor", "SpecExecutor"): (
_VERIFY_REASON
),
Expand All @@ -112,14 +90,6 @@
("src/kpubdata_builder/verify/runner.py", "kpubdata.transport.http", "HttpTransport"): (
_VERIFY_REASON
),
("src/kpubdata_builder/agent/monitor.py", "kpubdata.core.spec", "find_spec"): (
_SPEC_LOOKUP_REASON
),
("src/kpubdata_builder/agent/pipeline.py", "kpubdata.core.spec", "find_spec"): (
_SPEC_LOOKUP_REASON
),
("src/kpubdata_builder/cli.py", "kpubdata.core.spec", "find_spec"): _SPEC_LOOKUP_REASON,
("src/kpubdata_builder/cli.py", "kpubdata.core.spec", "discover_specs"): (_SPEC_LOOKUP_REASON),
}


Expand Down
2 changes: 1 addition & 1 deletion src/kpubdata_builder/agent/monitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ def check_approval(dataset_id: str, *, api_key: str | None = None) -> str:
Returns:
Status string: HEALTHY, NEEDS_APPLICATION, INVALID_KEY, etc.
"""
from kpubdata.core.spec import find_spec
from kpubdata import find_spec

from kpubdata_builder.verify import verify_dataset

Expand Down
2 changes: 1 addition & 1 deletion src/kpubdata_builder/agent/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def run_pipeline(
[
sys.executable,
"-c",
"import sys; from kpubdata.core.spec import find_spec; "
"import sys; from kpubdata import find_spec; "
"sys.exit(0 if find_spec(sys.argv[1]) is not None else 1)",
dataset_id,
],
Expand Down
7 changes: 4 additions & 3 deletions src/kpubdata_builder/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -474,7 +474,7 @@ def build_parser() -> argparse.ArgumentParser:
#: Why a client that must not use the operator's keys cannot be built (#990).
ENV_KEYS_UNSUPPORTED = (
"the installed kpubdata cannot keep the environment's provider keys out of a client "
"(Client has no env_keys option; it was added after 0.8.0), so a request without its "
"(Client has no env_keys option; it came with 0.9.0), so a request without its "
"own key would run on the operator's. Install a kpubdata release that has it, or "
"turn KPUBDATA_BUILDER_REQUIRE_OWN_PROVIDER_CREDENTIAL and multi-user mode off."
)
Expand All @@ -485,7 +485,8 @@ def client_keeps_environment_keys_out() -> bool:

kpubdata 0.8.0 has no such option and accepts any keyword without an error, so
passing ``env_keys=False`` to it does nothing and says nothing. Asking the signature
is the only way to know the option is honoured.
is the only way to know the option is honoured. Builder's pin is the 0.9 line, which
has it; the check stays for an environment that installed something older anyway.
"""
import inspect

Expand Down Expand Up @@ -1347,7 +1348,7 @@ def _run_verify(
page_size: Records per test request.
hashes_path: Optional path to previous schema hashes YAML.
"""
from kpubdata.core.spec import discover_specs, find_spec
from kpubdata import discover_specs, find_spec

from .verify import runner as _verify_runner

Expand Down
2 changes: 1 addition & 1 deletion src/kpubdata_builder/logging_redaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
from collections.abc import Iterable
from urllib.parse import quote, quote_plus

from kpubdata.transport._sensitive import SENSITIVE_PARAM_KEYS
from kpubdata import SENSITIVE_PARAM_KEYS

REDACTED = "[REDACTED]"

Expand Down
3 changes: 1 addition & 2 deletions src/kpubdata_builder/service/providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@
from datetime import datetime, timezone
from typing import Literal, Protocol, cast

from kpubdata import Client, Operation
from kpubdata.config import KPubDataConfig
from kpubdata import Client, KPubDataConfig, Operation
from kpubdata.core.models import DatasetRef
from kpubdata.exceptions import (
AuthError,
Expand Down
2 changes: 1 addition & 1 deletion src/kpubdata_builder/verify/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ def verify_dataset(
# ``tests/unit/test_kpubdata_internal_surface.py`` pins this list,
# so upgrade breaks in CI, not at runtime. When starting to use new internal symbols,
# add them to that list too.
from kpubdata.config import KPubDataConfig
from kpubdata import KPubDataConfig
from kpubdata.core.executor import (
SpecExecutor,
check_payload_error,
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/test_verify_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# All tests in this module patch the verify.runner module which is imported
# lazily inside _run_verify via ``from .verify import runner as _verify_runner``.
_RUNNER_MOD = "kpubdata_builder.verify.runner"
_SPEC_MOD = "kpubdata.core.spec"
_SPEC_MOD = "kpubdata"


class TestVerifyParser:
Expand Down
8 changes: 4 additions & 4 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading