diff --git a/CHANGELOG.md b/CHANGELOG.md index fc797643..8b2cf755 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,7 @@ ### Added +- `POST /providers/{provider}/probe` tells what a key can reach, per dataset, without keeping it (#802). The key comes in the `X-Provider-Key` header and is the only one used — a stored credential or one in the server's environment is never tried, in either deployment mode. Each dataset of the provider that kpubdata has a spec for gets one of kpubdata's probe statuses (`available`, `application_required`, `params_invalid` …) and the provider service it belongs to. Nothing is stored. After 45 seconds no further dataset is started; those not reached are named in `not_probed`. Contract 1.87.0. - An upload says when it expires (#1047, API contract 1.86.0, additive). #1045 deletes an upload once it is older than a multi-user deployment's retention period, and its owner had no way to see that coming: the metadata carried `created_at` and the period is a server setting. `POST /uploads` and `GET /uploads/{upload_id}` now carry `expires_at` — `created_at` plus the period in force, the same boundary the clean-up uses — and `null` when nothing will delete the upload (a single-user deployment, or retention turned off). - A build can say which run it retries (#1042, decided in kpubdata#812, API contract 1.85.0, additive). A `run_id` is one attempt and is not reused once it ended, so a retry is a new run: `retry_of` on `POST /build` and `POST /builds` names the earlier one, and it is shown on the job (`BuildJob.retry_of`, also after a restart) and written to the manifest. The named run must be one the caller may read — otherwise the answer is the one reading it would give — and cannot be the run itself. The refusal of a run id that already ended now carries `code: run_id_ended`. - Per-user upload limits in a multi-user deployment (#1045, decided in kpubdata#812, API contract 1.84.0). Nothing bounded what one account could upload, and an upload was kept until its owner deleted it. Each owner may now hold 50 files and 1 GiB in total — an upload past either is refused with 409 `upload_quota_exceeded`, saying which limit and what is in use — and an upload older than 30 days is deleted, when the service starts and when its owner next uploads. `KPUBDATA_BUILDER_UPLOAD_MAX_FILES`, `KPUBDATA_BUILDER_UPLOAD_MAX_TOTAL_BYTES` and `KPUBDATA_BUILDER_UPLOAD_RETENTION_DAYS` override them; `0` turns one off. A single-user deployment applies none. **Retention deletes data:** a saved spec that names an upload past 30 days no longer builds until the file is uploaded again. diff --git a/contract/builder-api.yaml b/contract/builder-api.yaml index be5bbd3d..75bc2955 100644 --- a/contract/builder-api.yaml +++ b/contract/builder-api.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: KPubData Builder Service API - version: "1.86.0" + version: "1.87.0" description: >- KPubData Builder(패키지 `kpubdata-builder`) 서비스 계약. BuildSpec 검증, preview, build 실행, artifact 조회, 계약 버전 조회를 제공한다. CLI와 HTTP service mode(#36)가 동일한 도메인 계약을 @@ -20,6 +20,8 @@ info: 추가한다(#504, additive). v1.8.0은 stable principal별 encrypted Provider credential CRUD와 Provider status/test API를 추가한다(#492, additive). v1.9.0은 `/query` 응답에 child startup과 Polars engine 실행 시간을 추가한다(#523, additive). + v1.87.0 adds `POST /providers/{provider}/probe` (#802, additive): what the key in the request's `X-Provider-Key` header can reach, per source dataset, in kpubdata's probe vocabulary (`available`, `application_required`, `params_invalid` …) with the provider service each dataset belongs to. Only the request's key is used — a stored credential or one in the server's environment is never tried, in either deployment mode — and nothing is stored. Without the header the answer is 400 `provider_key_required`. One call is made per dataset; after 45 seconds no further dataset is started, and those not reached are named in `not_probed` with `complete: false`. + v1.86.0 says when an upload expires (#1047, additive): `UploadMetadata.expires_at`, `created_at` plus the retention period in force, or `null` when nothing will delete the upload. v1.85.0 lets a build point at the run it retries (#1042, additive): `retry_of` on `POST /build` and `POST /builds`, shown on `BuildJob.retry_of` and in the manifest. The refusal of a run id that already ended carries `code: run_id_ended`, and both routes name it as an example. v1.84.0 adds per-user upload limits in a multi-user deployment (#1045, additive): `POST /uploads` answers 409 `upload_quota_exceeded` when the owner's file count or total size would pass the limit, and uploads past the retention period are deleted. @@ -381,6 +383,68 @@ paths: "502": description: 런타임 Provider 카탈로그 조회 실패 + /providers/{provider}/probe: + post: + operationId: probeProviderKey + summary: 요청에 실린 키로 무엇이 되는지 데이터셋별로 확인 (#802) + description: >- + `X-Provider-Key` 헤더의 키만 쓴다. 저장된 credential 이나 서버 환경변수의 키는 + 배포 모드와 무관하게 쓰지 않는다. 키도 결과도 저장하지 않으며, 응답에는 키에서 + 유도한 값이 없다. 데이터셋마다 한 번 호출한다(15초, 재시도·캐시 없음). 45초가 + 지나면 새 호출을 시작하지 않고, 닿지 못한 데이터셋은 `not_probed` 에 적는다. + replay 모드의 Builder(데모·개발용)는 provider 를 호출하지 않고 기록된 응답을 + 재생하므로, 어떤 키에도 `available` 로 답한다 — 키에 대해 아무것도 말하지 않는다. + parameters: + - $ref: "#/components/parameters/ProviderKey" + - $ref: "#/components/parameters/ProviderName" + requestBody: + required: false + content: + application/json: + schema: + $ref: "#/components/schemas/ProviderProbeRequest" + responses: + "200": + description: 데이터셋별 probe 결과 + content: + application/json: + schema: + $ref: "#/components/schemas/ProviderProbeResponse" + examples: + Probed: + summary: 하나는 바로 쓸 수 있고 하나는 활용신청이 필요함 + value: + provider: datago + probed_at: "2026-10-06T09:30:00+00:00" + complete: true + datasets: + - dataset: apt_trade + service_id: RTMSDataSvcAptTradeDev + status: available + detail: "" + http_status: 200 + - dataset: apt_rent + service_id: RTMSDataSvcAptRent + status: application_required + detail: "SERVICE_KEY_IS_NOT_REGISTERED_ERROR" + http_status: 200 + not_probed: [] + "400": + description: >- + `provider_key_required` — 이 provider 의 키가 `X-Provider-Key` 헤더에 없다. + `invalid_request` — body 가 `datasets` 외의 필드를 갖거나, `datasets` 가 비었거나 + probe 할 수 없는 데이터셋을 가리킨다. `invalid_provider_key` — 헤더 형식 오류. + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + "401": + $ref: "#/components/responses/Unauthorized" + "404": + description: 알려지지 않은 Provider + "502": + description: 런타임 Provider 카탈로그 조회 실패, 또는 probe 를 실행할 수 없음 + /providers/{provider}/credential: get: operationId: getProviderCredential @@ -6983,6 +7047,71 @@ components: maximum: 599 description: kpubdata가 구조적으로 신뢰 가능한 status를 제공한 경우에만 포함 + ProviderProbeRequest: + type: object + additionalProperties: false + properties: + datasets: + type: array + minItems: 1 + maxItems: 50 + items: + type: string + description: >- + Dataset names within the provider (as `GET /catalog` lists them). Omitted: + every dataset of the provider that can be probed. + + ProviderProbeResponse: + type: object + required: [provider, probed_at, complete, datasets, not_probed] + properties: + provider: + type: string + probed_at: + type: string + format: date-time + complete: + type: boolean + description: False when `not_probed` names a dataset. + datasets: + type: array + items: + $ref: "#/components/schemas/ProviderProbeDataset" + not_probed: + type: array + items: + type: string + description: >- + Datasets nothing was observed about: the time budget ran out before their + call, or they have no spec to probe with. + + ProviderProbeDataset: + type: object + required: [dataset, service_id, status, detail, http_status] + properties: + dataset: + type: string + description: Dataset name within the provider. + service_id: + type: string + description: >- + The provider service the dataset belongs to. An application at the provider + is granted per service, so datasets of one service share a verdict. + status: + type: string + description: >- + One of kpubdata's `PROBE_STATUSES` (ADR 0005 there): `available`, + `auth_unknown`, `application_required`, `params_invalid`, `rate_limited`, + `temporarily_unavailable`, `network_error`, `insufficient_metadata`, + `retired`. Not an enum here: a kpubdata release may add one, and a client + should show an unknown status as it is. + detail: + type: string + description: What the provider said, with no key in it. May be empty. + http_status: + type: [integer, "null"] + description: The HTTP status of the one call, where it was observed. + StructuredProblem: type: object required: [code, path, message] diff --git a/contract/fixtures/responses.json b/contract/fixtures/responses.json index e7e80928..7710feb1 100644 --- a/contract/fixtures/responses.json +++ b/contract/fixtures/responses.json @@ -1,6 +1,6 @@ { "fixture_format": 1, - "contract_version": "1.86.0", + "contract_version": "1.87.0", "probe_field": "future_optional_field", "rules": "A response may gain optional fields in any minor version; a client ignores fields it does not know. A required field keeps its name and type until the next major version; a client rejects a body whose required field is missing or mistyped.", "fixtures": [ @@ -407,6 +407,88 @@ }, "broken_path": "$.provider" }, + { + "operation_id": "probeProviderKey", + "method": "POST", + "path": "/providers/{provider}/probe", + "status": 200, + "example": "Probed", + "current": { + "provider": "datago", + "probed_at": "2026-10-06T09:30:00+00:00", + "complete": true, + "datasets": [ + { + "dataset": "apt_trade", + "service_id": "RTMSDataSvcAptTradeDev", + "status": "available", + "detail": "", + "http_status": 200 + }, + { + "dataset": "apt_rent", + "service_id": "RTMSDataSvcAptRent", + "status": "application_required", + "detail": "SERVICE_KEY_IS_NOT_REGISTERED_ERROR", + "http_status": 200 + } + ], + "not_probed": [] + }, + "with_additive_fields": { + "provider": "datago", + "probed_at": "2026-10-06T09:30:00+00:00", + "complete": true, + "datasets": [ + { + "dataset": "apt_trade", + "service_id": "RTMSDataSvcAptTradeDev", + "status": "available", + "detail": "", + "http_status": 200, + "future_optional_field": "added by a later minor contract version" + }, + { + "dataset": "apt_rent", + "service_id": "RTMSDataSvcAptRent", + "status": "application_required", + "detail": "SERVICE_KEY_IS_NOT_REGISTERED_ERROR", + "http_status": 200, + "future_optional_field": "added by a later minor contract version" + } + ], + "not_probed": [], + "future_optional_field": "added by a later minor contract version" + }, + "additive_paths": [ + "$.datasets[0]", + "$.datasets[1]", + "$" + ], + "required_type_broken": { + "provider": 12345, + "probed_at": "2026-10-06T09:30:00+00:00", + "complete": true, + "datasets": [ + { + "dataset": "apt_trade", + "service_id": "RTMSDataSvcAptTradeDev", + "status": "available", + "detail": "", + "http_status": 200 + }, + { + "dataset": "apt_rent", + "service_id": "RTMSDataSvcAptRent", + "status": "application_required", + "detail": "SERVICE_KEY_IS_NOT_REGISTERED_ERROR", + "http_status": 200 + } + ], + "not_probed": [] + }, + "broken_path": "$.provider" + }, { "operation_id": "getProviderCredential", "method": "GET", diff --git a/src/kpubdata_builder/service/app.py b/src/kpubdata_builder/service/app.py index 081aaa34..771261ec 100644 --- a/src/kpubdata_builder/service/app.py +++ b/src/kpubdata_builder/service/app.py @@ -51,7 +51,7 @@ from . import datasets as datasets_service from . import monitoring as monitoring_service from . import ownership as ownership_module -from . import pii_reads, publish_credentials, request_credentials +from . import pii_reads, provider_probe, publish_credentials, request_credentials from . import publish as publish_service from .analyses_api import AnalysesApiService, AnalysisStore from .auth import AuthError, Principal, authenticate @@ -419,7 +419,9 @@ def _enforce_ownership() -> bool: # used-run-id refusal carries code run_id_ended (#1042, additive). # 1.85.0 -> 1.86.0: UploadMetadata.expires_at — when the retention period deletes the # upload, or null (#1047, additive). -API_CONTRACT_VERSION = "1.86.0" +# 1.86.0 -> 1.87.0: POST /providers/{provider}/probe — what the key in the request's +# X-Provider-Key header can reach, per dataset; nothing is stored (#802, additive). +API_CONTRACT_VERSION = "1.87.0" #: How long a synchronous ``POST /build`` waits for a build slot before it answers #: ``build_queue_full`` (#1040). Long enough to ride out a short build ahead of it, short @@ -450,6 +452,8 @@ def __init__( upload_repository: UploadRepository | None = None, provider_test_operation: ProviderTestOperation = default_provider_test, provider_test_timeout: float | None = None, + open_probe: provider_probe.OpenProbe = provider_probe.open_kpubdata_probe, + probe_datasets: provider_probe.ListDatasets = provider_probe.spec_dataset_ids, async_max_workers: int = 10, async_max_queue_size: int = 10, max_concurrent_builds: int | None = None, @@ -523,6 +527,8 @@ def __init__( provider_test_operation=self._provider_test_operation, provider_test_timeout=self._provider_test_timeout, test_log=lambda: self._provider_tests(), + open_probe=open_probe, + probe_datasets=probe_datasets, ) # Upload repository initialized only when needed (#498) — pass lambda, # not property value directly (#498) — calling property on every request @@ -813,6 +819,16 @@ def provider_status(self, provider: str, *, principal: Principal) -> ServiceResp """Perform lightweight connection test with current principal's credential.""" return self._providers_service.provider_status(provider, principal=principal) + def probe_provider( + self, + provider: str, + body: Mapping[str, JsonValue] | None, + *, + principal: Principal, + ) -> ServiceResponse: + """Say what the key this request carries can reach, per dataset (#802).""" + return self._providers_service.probe_provider(provider, body, principal=principal) + def provider_credential(self, provider: str, *, principal: Principal) -> ServiceResponse: """Return saved credential metadata for current principal without plaintext.""" return self._providers_service.provider_credential(provider, principal=principal) diff --git a/src/kpubdata_builder/service/provider_probe.py b/src/kpubdata_builder/service/provider_probe.py new file mode 100644 index 00000000..5cb191f0 --- /dev/null +++ b/src/kpubdata_builder/service/provider_probe.py @@ -0,0 +1,159 @@ +"""Probe a provider with the key a request carries, and keep nothing (#802). + +``POST /providers/{provider}/probe`` tells a user what the key they just typed can reach: +per source dataset, one of kpubdata's ``PROBE_STATUSES`` — available, application +required, parameters to check and so on. It differs from the connection test in three +ways: + +- **Only the request's key is used.** It comes in the ``X-Provider-Key`` header (#683). + A stored credential or one in the server's environment is never tried, in any + deployment mode: the client is built with that one key and ``env_keys=False``. +- **Nothing about the key is kept.** The client is closed when the request ends, the + result is not stored, and the response holds nothing derived from the key, so a caller + may keep it under the account. +- **It answers per dataset**, with the provider service each belongs to — an application + at the provider is granted per service, so datasets of one service share a verdict. + +kpubdata makes one call per dataset with a fast-fail transport (15 s, no retry, no cache). +A provider that is down would hold the request for that long per dataset, so the probe +stops starting new calls after ``PROBE_BUDGET_SECONDS`` and names what it did not reach. +""" + +from __future__ import annotations + +import time +from collections.abc import Callable, Iterator, Mapping, Sequence +from contextlib import AbstractContextManager, contextmanager +from dataclasses import dataclass +from datetime import datetime, timezone +from typing import Protocol + +from .. import logging_redaction +from ..spec import JsonValue +from .redaction import redact_secret_text + +#: After this long no further dataset is probed in one request. +PROBE_BUDGET_SECONDS = 45.0 +#: Most datasets one request may name. +MAX_PROBE_DATASETS = 50 + + +class ProbeOutcome(Protocol): + """What the probe reads off kpubdata's ``ProbeResult``.""" + + @property + def service_id(self) -> str: ... + @property + def status(self) -> str: ... + @property + def detail(self) -> str: ... + @property + def http_status(self) -> int | None: ... + + +ProbeOne = Callable[[str], ProbeOutcome | None] +"""``dataset_id -> outcome``, or None when the dataset has no spec to probe.""" + +OpenProbe = Callable[[str, str], AbstractContextManager[ProbeOne]] +"""``(provider, key) -> probe``, open for one request.""" + +ListDatasets = Callable[[str], Sequence[str]] +"""``provider -> the dataset ids (".") that can be probed``.""" + + +def spec_dataset_ids(provider: str) -> list[str]: + """Ids of the spec-defined datasets of ``provider`` — the ones kpubdata can probe.""" + from kpubdata import discover_specs + + return sorted(spec.id for spec in discover_specs() if spec.provider == provider) + + +@contextmanager +def open_kpubdata_probe(provider: str, key: str) -> Iterator[ProbeOne]: + """A kpubdata client holding ``key`` for ``provider`` and nothing else.""" + from kpubdata import Client + + client = Client(provider_keys={provider: key}, cache=False, env_keys=False) + # A provider that puts the key in a path or echoes it would otherwise reach the log. + logging_redaction.register(client, (key,)) + try: + yield client.probe + finally: + try: + client.close() + finally: + logging_redaction.release(client) + + +@dataclass(frozen=True) +class ProbeRequest: + """The datasets a probe request asks for; ``None`` means every one of the provider.""" + + datasets: tuple[str, ...] | None + + +def parse_probe_body(body: Mapping[str, JsonValue] | None) -> ProbeRequest | str: + """The request body, or the reason it is refused.""" + if body is None or not body: + return ProbeRequest(None) + if set(body) != {"datasets"}: + return "the body may hold 'datasets' only" + datasets = body["datasets"] + if not isinstance(datasets, list) or not datasets: + return "datasets must be a non-empty list of dataset names" + if len(datasets) > MAX_PROBE_DATASETS: + return f"datasets may name at most {MAX_PROBE_DATASETS} datasets" + names: list[str] = [] + for item in datasets: + if not isinstance(item, str) or not item.strip(): + return "datasets must be a non-empty list of dataset names" + names.append(item.strip()) + return ProbeRequest(tuple(dict.fromkeys(names))) + + +def run_probe( + provider: str, + key: str, + dataset_ids: Sequence[str], + *, + open_probe: OpenProbe, + budget_seconds: float = PROBE_BUDGET_SECONDS, + clock: Callable[[], float] = time.monotonic, +) -> dict[str, JsonValue]: + """Probe ``dataset_ids`` in order and return the response body. + + A dataset the time budget did not reach is listed in ``not_probed`` and ``complete`` + is false; it gets no status, because nothing was observed about it. + """ + prefix = f"{provider}." + results: list[JsonValue] = [] + not_probed: list[JsonValue] = [] + started = clock() + with open_probe(provider, key) as probe_one: + for dataset_id in dataset_ids: + name = dataset_id.removeprefix(prefix) + if clock() - started >= budget_seconds: + not_probed.append(name) + continue + outcome = probe_one(dataset_id) + if outcome is None: + not_probed.append(name) + continue + results.append( + { + "dataset": name, + "service_id": outcome.service_id, + "status": outcome.status, + # kpubdata keeps a configured key out of ``detail``; this does not + # rely on it. + "detail": redact_secret_text(outcome.detail, (key,)) or "", + "http_status": outcome.http_status, + } + ) + return { + "provider": provider, + "probed_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "complete": not not_probed, + "datasets": results, + "not_probed": not_probed, + } diff --git a/src/kpubdata_builder/service/providers_service.py b/src/kpubdata_builder/service/providers_service.py index 43b28ba1..ce8c8dca 100644 --- a/src/kpubdata_builder/service/providers_service.py +++ b/src/kpubdata_builder/service/providers_service.py @@ -18,6 +18,7 @@ from collections.abc import Callable, Iterable, Mapping from typing import cast +from kpubdata_builder.service import provider_probe, request_credentials from kpubdata_builder.service.auth import Principal from kpubdata_builder.service.ownership import multi_user_mode from kpubdata_builder.service.provider_tests import ProviderTestLog @@ -58,6 +59,8 @@ def __init__( provider_test_operation: ProviderTestOperation, provider_test_timeout: float, test_log: Callable[[], ProviderTestLog | None] = lambda: None, + open_probe: provider_probe.OpenProbe = provider_probe.open_kpubdata_probe, + probe_datasets: provider_probe.ListDatasets = provider_probe.spec_dataset_ids, ) -> None: """Args: test_log: Where each principal's last test per provider is kept (#842); None @@ -69,6 +72,8 @@ def __init__( self._provider_test_operation = provider_test_operation self._provider_test_timeout = provider_test_timeout self._test_log = test_log + self._open_probe = open_probe + self._probe_datasets = probe_datasets # --- Internal queries ------------------------------------------------- @@ -157,6 +162,56 @@ def provider_status(self, provider: str, *, principal: Principal) -> ServiceResp if client is not None: self._close_client(client) + def probe_provider( + self, + provider: str, + body: Mapping[str, JsonValue] | None, + *, + principal: Principal, + ) -> ServiceResponse: + """Say what the key this request carries can reach, per dataset (#802). + + Uses the ``X-Provider-Key`` header's key and no other; stores nothing. + """ + del principal # any authenticated caller; nothing is read or written for them + descriptor = self.known_provider(provider) + if isinstance(descriptor, ServiceResponse): + return descriptor + parsed = provider_probe.parse_probe_body(body) + if isinstance(parsed, str): + return ServiceResponse(400, {"error": parsed, "code": "invalid_request"}) + key = request_credentials.current_key(provider) + if key is None: + return ServiceResponse( + 400, + { + "error": "send the key to probe in the X-Provider-Key header " + f"('{provider}='); a stored key is not used", + "code": "provider_key_required", + }, + ) + available = list(self._probe_datasets(provider)) + if parsed.datasets is None: + dataset_ids = available + else: + dataset_ids = [f"{provider}.{name}" for name in parsed.datasets] + unknown = sorted(set(dataset_ids) - set(available)) + if unknown: + names = ", ".join(item.removeprefix(f"{provider}.") for item in unknown) + return ServiceResponse( + 400, + {"error": f"cannot probe: {names}", "code": "invalid_request"}, + ) + try: + result = provider_probe.run_probe( + provider, key, dataset_ids, open_probe=self._open_probe + ) + except Exception: + # An exception's text may hold the request URL, and with it the key. + _logger_exception("provider probe failed") + return ServiceResponse(502, {"error": "probe unavailable"}) + return ServiceResponse(200, result) + def _remember(self, owner_id: str, result: ProviderTestResult) -> None: """Keep the result as the last test; a store failure never fails the test.""" log = self._test_log() diff --git a/src/kpubdata_builder/service/routes/providers.py b/src/kpubdata_builder/service/routes/providers.py index 7e6e4b21..c2393d0c 100644 --- a/src/kpubdata_builder/service/routes/providers.py +++ b/src/kpubdata_builder/service/routes/providers.py @@ -40,6 +40,8 @@ def route( return service.provider_status(provider, principal=principal) if method == "POST" and operation == "test": return service.provider_status(provider, principal=principal) + if method == "POST" and operation == "probe": + return service.probe_provider(provider, body, principal=principal) if method == "GET" and operation == "credential": return service.provider_credential(provider, principal=principal) if method == "PUT" and operation == "credential": diff --git a/tests/unit/test_contract_shared_declarations.py b/tests/unit/test_contract_shared_declarations.py index edebc564..6baedab4 100644 --- a/tests/unit/test_contract_shared_declarations.py +++ b/tests/unit/test_contract_shared_declarations.py @@ -26,6 +26,7 @@ "submitBuild", "getProviderStatus", "testProviderConnection", + "probeProviderKey", } diff --git a/tests/unit/test_provider_key_probe.py b/tests/unit/test_provider_key_probe.py new file mode 100644 index 00000000..0b3bffd8 --- /dev/null +++ b/tests/unit/test_provider_key_probe.py @@ -0,0 +1,351 @@ +"""``POST /providers/{provider}/probe`` uses the request's key only and keeps nothing (#802). + +The service is driven through ``dispatch`` with the key in the ``X-Provider-Key`` header. +Most tests replace the probe itself, to choose its outcomes; the last ones run the real +kpubdata ``Client`` with its HTTP layer replaced, so the URL, the error text and the log +records are the ones production produces. +""" + +from __future__ import annotations + +import json +import logging +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import httpx +import pytest +import yaml +from kpubdata import PROBE_STATUSES + +from kpubdata_builder import cli +from kpubdata_builder.service import BuilderService, ServiceResponse, dispatch, provider_probe +from kpubdata_builder.spec import JsonValue + +from ._openapi import response_schema, validate + +_CONTRACT = yaml.safe_load( + (Path(__file__).parents[2] / "contract" / "builder-api.yaml").read_text(encoding="utf-8") +) +_PATH = "/providers/{provider}/probe" +_KEY = "canary-value-for-probe-test" +_OPERATOR = "operator-value-that-must-not-be-used" +_IDS = ["datago.apt_rent", "datago.apt_trade", "datago.village_fcst"] + + +@dataclass(frozen=True) +class _Outcome: + service_id: str + status: str + detail: str = "" + http_status: int | None = 200 + + +@dataclass +class _Probe: + """Stands in for kpubdata: records what it was opened with and asked.""" + + outcomes: dict[str, _Outcome | None] = field(default_factory=dict) + opened: list[tuple[str, str]] = field(default_factory=list) + asked: list[str] = field(default_factory=list) + closed: int = 0 + error: Exception | None = None + + @contextmanager + def open(self, provider: str, key: str) -> Iterator[provider_probe.ProbeOne]: + self.opened.append((provider, key)) + try: + yield self._one + finally: + self.closed += 1 + + def _one(self, dataset_id: str) -> _Outcome | None: + self.asked.append(dataset_id) + if self.error is not None: + raise self.error + return self.outcomes.get(dataset_id, _Outcome("svc", "available")) + + +def _service(tmp_path: Path, probe: _Probe, ids: list[str] | None = None) -> BuilderService: + return BuilderService( + output_root=tmp_path, + client_factory=cli._create_client, + open_probe=probe.open, + probe_datasets=lambda provider: [i for i in (ids or _IDS) if i.startswith(provider)], + ) + + +def _post( + service: BuilderService, + body: dict[str, JsonValue] | None = None, + *, + provider: str = "datago", + headers: tuple[str, ...] = (f"datago={_KEY}",), +) -> ServiceResponse: + response = dispatch( + service, "POST", f"/providers/{provider}/probe", body, provider_key_headers=headers + ) + assert isinstance(response, ServiceResponse) + return response + + +def _only_dataset(response: ServiceResponse) -> dict[str, JsonValue]: + datasets = response.body["datasets"] + assert isinstance(datasets, list) and len(datasets) == 1 + (dataset,) = datasets + assert isinstance(dataset, dict) + return dataset + + +def test_each_dataset_gets_its_status_and_service(tmp_path: Path) -> None: + probe = _Probe( + { + "datago.apt_trade": _Outcome("RTMSDataSvcAptTradeDev", "available"), + "datago.apt_rent": _Outcome( + "RTMSDataSvcAptRent", "application_required", "SERVICE_KEY_IS_NOT_REGISTERED_ERROR" + ), + "datago.village_fcst": _Outcome("VilageFcstInfoService_2.0", "params_invalid", "", 400), + } + ) + + response = _post(_service(tmp_path, probe)) + + assert response.status_code == 200 + schema = response_schema(_CONTRACT, _PATH, "post", 200) + assert schema is not None + assert validate(response.body, schema, _CONTRACT) == [] + assert response.body["complete"] is True + assert response.body["not_probed"] == [] + assert response.body["datasets"] == [ + { + "dataset": "apt_rent", + "service_id": "RTMSDataSvcAptRent", + "status": "application_required", + "detail": "SERVICE_KEY_IS_NOT_REGISTERED_ERROR", + "http_status": 200, + }, + { + "dataset": "apt_trade", + "service_id": "RTMSDataSvcAptTradeDev", + "status": "available", + "detail": "", + "http_status": 200, + }, + { + "dataset": "village_fcst", + "service_id": "VilageFcstInfoService_2.0", + "status": "params_invalid", + "detail": "", + "http_status": 400, + }, + ] + # Opened once with the header's key, and closed when the request ended. + assert probe.opened == [("datago", _KEY)] + assert probe.closed == 1 + + +def test_without_the_header_nothing_is_probed_whatever_else_holds_a_key( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Negative: the operator's environment key and a key for another provider are not + used in its place.""" + monkeypatch.setenv("KPUBDATA_DATAGO_API_KEY", _OPERATOR) + probe = _Probe() + service = _service(tmp_path, probe) + + for headers in ((), (f"localdata={_KEY}",)): + response = _post(service, headers=headers) + + assert response.status_code == 400 + assert response.body["code"] == "provider_key_required" + assert probe.opened == [] + + +def test_the_response_holds_nothing_of_the_key(tmp_path: Path) -> None: + probe = _Probe({"datago.apt_trade": _Outcome("svc", "network_error", f"GET /x/{_KEY}/y")}) + + response = _post(_service(tmp_path, probe), {"datasets": ["apt_trade"]}) + + assert response.status_code == 200 + text = json.dumps(response.body, ensure_ascii=False) + assert _KEY not in text + assert "GET /x/" in text + assert probe.asked == ["datago.apt_trade"] + + +def test_a_probe_that_raises_answers_502_without_its_text( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + probe = _Probe(error=RuntimeError(f"https://apis.example/get?serviceKey={_KEY}")) + + with caplog.at_level(logging.DEBUG): + response = _post(_service(tmp_path, probe)) + + assert (response.status_code, response.body) == (502, {"error": "probe unavailable"}) + assert probe.closed == 1 + + +@pytest.mark.parametrize( + "body", + [ + {"datasets": []}, + {"datasets": "apt_trade"}, + {"datasets": ["apt_trade", 3]}, + {"datasets": ["apt_trade"], "key": "x"}, + {"datasets": ["no_such_dataset"]}, + {"datasets": [f"d{index}" for index in range(51)]}, + ], +) +def test_a_body_that_is_not_a_list_of_known_datasets_is_refused( + tmp_path: Path, body: dict[str, JsonValue] +) -> None: + probe = _Probe() + + response = _post(_service(tmp_path, probe), body) + + assert response.status_code == 400 + assert response.body["code"] == "invalid_request" + assert probe.asked == [] + + +def test_an_unknown_provider_is_not_found(tmp_path: Path) -> None: + probe = _Probe() + + response = _post(_service(tmp_path, probe), provider="nosuch", headers=(f"nosuch={_KEY}",)) + + assert response.status_code == 404 + assert probe.opened == [] + + +def test_past_the_budget_no_call_is_started_and_the_rest_is_named() -> None: + probe = _Probe() + ticks = iter([0.0, 0.0, 10.0, 50.0]) + clock: Callable[[], float] = lambda: next(ticks) # noqa: E731 + + body = provider_probe.run_probe( + "datago", _KEY, _IDS, open_probe=probe.open, budget_seconds=45.0, clock=clock + ) + + assert probe.asked == ["datago.apt_rent", "datago.apt_trade"] + assert body["complete"] is False + assert body["not_probed"] == ["village_fcst"] + datasets = body["datasets"] + assert isinstance(datasets, list) and len(datasets) == 2 + + +def test_a_dataset_without_a_spec_is_named_not_given_a_status() -> None: + probe = _Probe({"datago.apt_trade": None}) + + body = provider_probe.run_probe("datago", _KEY, ["datago.apt_trade"], open_probe=probe.open) + + assert (body["datasets"], body["not_probed"], body["complete"]) == ([], ["apt_trade"], False) + + +def test_the_datasets_offered_are_the_ones_kpubdata_has_a_spec_for() -> None: + ids = provider_probe.spec_dataset_ids("datago") + + assert "datago.apt_trade" in ids + assert ids == sorted(ids) + assert all(item.startswith("datago.") for item in ids) + assert provider_probe.spec_dataset_ids("nosuch") == [] + + +# --- The real kpubdata client, with its HTTP layer replaced --- + + +@dataclass +class _Upstream: + """Answers every call and keeps the requests it saw.""" + + status: int = 200 + requests: list[httpx.Request] = field(default_factory=list) + + def handle(self, request: httpx.Request) -> httpx.Response: + self.requests.append(request) + if self.status != 200: + # An upstream that echoes the request, key included. + return httpx.Response(self.status, text=f"rejected: {request.url}") + return httpx.Response( + 200, + json={ + "response": { + "header": {"resultCode": "00", "resultMsg": "OK"}, + "body": {"items": {"item": [{"aptNm": "a"}]}, "totalCount": 1}, + } + }, + ) + + +@pytest.fixture() +def upstream(monkeypatch: pytest.MonkeyPatch) -> _Upstream: + # The cross-repo job replays recorded fixtures instead of calling HTTP; these tests + # need the HTTP path they observe. + monkeypatch.delenv("KPUBDATA_REPLAY_DIR", raising=False) + monkeypatch.delenv("KPUBDATA_MODE", raising=False) + seen = _Upstream() + real_client = httpx.Client + + def mocked_client(**kwargs: Any) -> httpx.Client: + return real_client(transport=httpx.MockTransport(seen.handle), **kwargs) + + monkeypatch.setattr(httpx, "Client", mocked_client) + return seen + + +def _real_service(tmp_path: Path) -> BuilderService: + return BuilderService(output_root=tmp_path, client_factory=cli._create_client) + + +def test_the_real_client_calls_with_the_header_key_and_not_the_operators( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, upstream: _Upstream +) -> None: + monkeypatch.setenv("KPUBDATA_DATAGO_API_KEY", _OPERATOR) + + response = _post(_real_service(tmp_path), {"datasets": ["apt_trade"]}) + + assert response.status_code == 200 + dataset = _only_dataset(response) + assert dataset["status"] in PROBE_STATUSES + assert dataset["status"] == "available" + assert len(upstream.requests) == 1 + sent = str(upstream.requests[0].url) + repr(upstream.requests[0].headers.raw) + assert _KEY in sent + assert _OPERATOR not in sent + + +@pytest.mark.parametrize("status", [401, 403, 500]) +def test_the_key_reaches_no_response_and_no_log_when_the_upstream_echoes_it( + tmp_path: Path, upstream: _Upstream, caplog: pytest.LogCaptureFixture, status: int +) -> None: + upstream.status = status + + with caplog.at_level(logging.DEBUG): + response = _post(_real_service(tmp_path), {"datasets": ["apt_trade"]}) + + assert response.status_code == 200 + dataset = _only_dataset(response) + assert dataset["status"] != "available" + assert len(upstream.requests) == 1 and _KEY in str(upstream.requests[0].url) + assert _KEY not in json.dumps(response.body, ensure_ascii=False) + logged = "\n".join( + logging.Formatter("%(message)s %(exc_text)s").format(record) for record in caplog.records + ) + assert _KEY not in logged + + +def test_the_real_client_does_not_fall_back_to_the_environment( + monkeypatch: pytest.MonkeyPatch, upstream: _Upstream +) -> None: + """Negative: opened for another provider, it has no datago key — and takes none from + the environment, so no call is made.""" + monkeypatch.setenv("KPUBDATA_DATAGO_API_KEY", _OPERATOR) + + with provider_probe.open_kpubdata_probe("localdata", _KEY) as probe_one: + outcome = probe_one("datago.apt_trade") + + assert outcome is not None + assert outcome.status == "auth_unknown" + assert upstream.requests == [] diff --git a/tests/unit/test_service_contract.py b/tests/unit/test_service_contract.py index 53c47e96..49d2f872 100644 --- a/tests/unit/test_service_contract.py +++ b/tests/unit/test_service_contract.py @@ -51,6 +51,7 @@ ("/providers", "GET"): "listProviders", ("/providers/{provider}/status", "GET"): "getProviderStatus", ("/providers/{provider}/test", "POST"): "testProviderConnection", + ("/providers/{provider}/probe", "POST"): "probeProviderKey", ("/providers/{provider}/credential", "GET"): "getProviderCredential", ("/providers/{provider}/credential", "PUT"): "putProviderCredential", ("/providers/{provider}/credential", "DELETE"): "deleteProviderCredential", @@ -119,6 +120,7 @@ ("/providers", "get"), ("/providers/{provider}/status", "get"), ("/providers/{provider}/test", "post"), + ("/providers/{provider}/probe", "post"), ("/providers/{provider}/credential", "get"), ("/providers/{provider}/credential", "put"), ("/providers/{provider}/credential", "delete"), @@ -447,6 +449,7 @@ def test_preview_request_schema_declares_bounded_limit_and_sample_mode() -> None "listProviders", "getProviderStatus", "testProviderConnection", + "probeProviderKey", "getProviderCredential", "putProviderCredential", "deleteProviderCredential", @@ -702,6 +705,7 @@ def test_planned_operations_excluded_from_implementation_check() -> None: "listProviders": {200, 403, 502}, "getProviderStatus": {200, 403, 404, 502}, "testProviderConnection": {200, 403, 404, 502}, + "probeProviderKey": {200, 400, 404, 502}, "getProviderCredential": {200, 403, 404, 502, 503}, "putProviderCredential": {200, 400, 403, 404, 502, 503}, "deleteProviderCredential": {200, 403, 404, 502, 503},