diff --git a/CHANGELOG.md b/CHANGELOG.md index db94c67f..69703325 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,6 +58,7 @@ ### Added +- An upload says when it expires (#1047, API contract 1.86.0, additive). #1045 deletes an upload once it is older than a multi-user deployment's retention period, and its owner had no way to see that coming: the metadata carried `created_at` and the period is a server setting. `POST /uploads` and `GET /uploads/{upload_id}` now carry `expires_at` — `created_at` plus the period in force, the same boundary the clean-up uses — and `null` when nothing will delete the upload (a single-user deployment, or retention turned off). - A build can say which run it retries (#1042, decided in kpubdata#812, API contract 1.85.0, additive). A `run_id` is one attempt and is not reused once it ended, so a retry is a new run: `retry_of` on `POST /build` and `POST /builds` names the earlier one, and it is shown on the job (`BuildJob.retry_of`, also after a restart) and written to the manifest. The named run must be one the caller may read — otherwise the answer is the one reading it would give — and cannot be the run itself. The refusal of a run id that already ended now carries `code: run_id_ended`. - Per-user upload limits in a multi-user deployment (#1045, decided in kpubdata#812, API contract 1.84.0). Nothing bounded what one account could upload, and an upload was kept until its owner deleted it. Each owner may now hold 50 files and 1 GiB in total — an upload past either is refused with 409 `upload_quota_exceeded`, saying which limit and what is in use — and an upload older than 30 days is deleted, when the service starts and when its owner next uploads. `KPUBDATA_BUILDER_UPLOAD_MAX_FILES`, `KPUBDATA_BUILDER_UPLOAD_MAX_TOTAL_BYTES` and `KPUBDATA_BUILDER_UPLOAD_RETENTION_DAYS` override them; `0` turns one off. A single-user deployment applies none. **Retention deletes data:** a saved spec that names an upload past 30 days no longer builds until the file is uploaded again. - The publish recovery routes are in the contract (#994, API contract 1.81.0, additive): `GET` and `DELETE /builds/{run_id}/publish/receipt`, `POST /builds/{run_id}/publish/reconcile` and `GET /builds/{run_id}/publish/audit` have answered since contract 1.19.0 and 1.20.0 but were described only in the contract's prose, so a client had no schema for them and the route tests did not know them. They are now declared with `PublishReceipt`, `PublishReconcileRequest`, `PublishReconcileResponse`, `PublishReceiptReset` and `PublishAuditLog`, and their responses are checked against those schemas. Nothing on the wire changed. diff --git a/contract/builder-api.yaml b/contract/builder-api.yaml index 1475f4ba..be5bbd3d 100644 --- a/contract/builder-api.yaml +++ b/contract/builder-api.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: KPubData Builder Service API - version: "1.85.0" + version: "1.86.0" description: >- KPubData Builder(패키지 `kpubdata-builder`) 서비스 계약. BuildSpec 검증, preview, build 실행, artifact 조회, 계약 버전 조회를 제공한다. CLI와 HTTP service mode(#36)가 동일한 도메인 계약을 @@ -20,6 +20,7 @@ info: 추가한다(#504, additive). v1.8.0은 stable principal별 encrypted Provider credential CRUD와 Provider status/test API를 추가한다(#492, additive). v1.9.0은 `/query` 응답에 child startup과 Polars engine 실행 시간을 추가한다(#523, additive). + v1.86.0 says when an upload expires (#1047, additive): `UploadMetadata.expires_at`, `created_at` plus the retention period in force, or `null` when nothing will delete the upload. v1.85.0 lets a build point at the run it retries (#1042, additive): `retry_of` on `POST /build` and `POST /builds`, shown on `BuildJob.retry_of` and in the manifest. The refusal of a run id that already ended carries `code: run_id_ended`, and both routes name it as an example. v1.84.0 adds per-user upload limits in a multi-user deployment (#1045, additive): `POST /uploads` answers 409 `upload_quota_exceeded` when the owner's file count or total size would pass the limit, and uploads past the retention period are deleted. v1.83.0 makes a `run_id` one attempt (#1042, kpubdata#812): the id of a run that ended without a manifest — one a restart interrupted — is refused to its own submitter too — with 409 on `POST /builds`, as a completed run's id is, and with 400 on `POST /build`, whose 409 is a build response with a different body. @@ -549,6 +550,7 @@ paths: size_bytes: 1024 original_filename: trades.csv created_at: "2026-08-16T09:00:00+00:00" + expires_at: "2026-09-15T09:00:00+00:00" "400": description: 빈 body, 손상된 content, 지원하지 않는 format, 또는 크기 초과 content: @@ -6362,6 +6364,14 @@ components: created_at: type: string format: date-time + expires_at: + type: [string, "null"] + format: date-time + description: >- + When the upload will be deleted (#1047): `created_at` plus the deployment's + retention period (#1045). After it the upload is gone and a spec that names + it no longer builds until the file is uploaded again. `null` when nothing + will delete it — a single-user deployment, or retention turned off. UploadDeleteResponse: type: object diff --git a/contract/fixtures/responses.json b/contract/fixtures/responses.json index 1f7bb1b5..e7e80928 100644 --- a/contract/fixtures/responses.json +++ b/contract/fixtures/responses.json @@ -1,6 +1,6 @@ { "fixture_format": 1, - "contract_version": "1.85.0", + "contract_version": "1.86.0", "probe_field": "future_optional_field", "rules": "A response may gain optional fields in any minor version; a client ignores fields it does not know. A required field keeps its name and type until the next major version; a client rejects a body whose required field is missing or mistyped.", "fixtures": [ @@ -533,7 +533,8 @@ "encoding": "utf-8", "size_bytes": 1024, "original_filename": "trades.csv", - "created_at": "2026-08-16T09:00:00+00:00" + "created_at": "2026-08-16T09:00:00+00:00", + "expires_at": "2026-09-15T09:00:00+00:00" }, "with_additive_fields": { "upload_id": "upl_0123456789abcdef0123456789abcdef", @@ -542,6 +543,7 @@ "size_bytes": 1024, "original_filename": "trades.csv", "created_at": "2026-08-16T09:00:00+00:00", + "expires_at": "2026-09-15T09:00:00+00:00", "future_optional_field": "added by a later minor contract version" }, "additive_paths": [ @@ -553,7 +555,8 @@ "encoding": "utf-8", "size_bytes": 1024, "original_filename": "trades.csv", - "created_at": "2026-08-16T09:00:00+00:00" + "created_at": "2026-08-16T09:00:00+00:00", + "expires_at": "2026-09-15T09:00:00+00:00" }, "broken_path": "$.upload_id" }, diff --git a/src/kpubdata_builder/service/app.py b/src/kpubdata_builder/service/app.py index 83295775..081aaa34 100644 --- a/src/kpubdata_builder/service/app.py +++ b/src/kpubdata_builder/service/app.py @@ -417,7 +417,9 @@ def _enforce_ownership() -> bool: # may answer 409 upload_quota_exceeded (#1045, additive). # 1.84.0 -> 1.85.0: retry_of on both build routes, on BuildJob and in the manifest; the # used-run-id refusal carries code run_id_ended (#1042, additive). -API_CONTRACT_VERSION = "1.85.0" +# 1.85.0 -> 1.86.0: UploadMetadata.expires_at — when the retention period deletes the +# upload, or null (#1047, additive). +API_CONTRACT_VERSION = "1.86.0" #: How long a synchronous ``POST /build`` waits for a build slot before it answers #: ``build_queue_full`` (#1040). Long enough to ride out a short build ahead of it, short diff --git a/src/kpubdata_builder/service/uploads_service.py b/src/kpubdata_builder/service/uploads_service.py index f0b2fb68..567e985e 100644 --- a/src/kpubdata_builder/service/uploads_service.py +++ b/src/kpubdata_builder/service/uploads_service.py @@ -15,6 +15,7 @@ import io import threading from collections.abc import Callable +from datetime import datetime, timedelta, timezone from kpubdata_builder.ingestion import IngestionError from kpubdata_builder.ingestion.tabular_ingest import iter_tabular_batches @@ -28,7 +29,27 @@ RepositoryProvider = Callable[[], UploadRepository] -def upload_metadata_body(metadata: UploadMetadata) -> dict[str, JsonValue]: +def upload_expires_at(metadata: UploadMetadata, limits: UploadLimits | None) -> str | None: + """When the upload will be deleted (#1047), or None when nothing will delete it. + + ``created_at`` plus the retention period in force — the same boundary the purge + uses (``UploadLimits.cutoff``). None in a single-user deployment and when retention + is off. Read at answer time: an operator who changes the period changes the date. + """ + if limits is None or limits.retention_days is None: + return None + try: + created = datetime.fromisoformat(metadata.created_at) + except ValueError: + return None + if created.tzinfo is None: + created = created.replace(tzinfo=timezone.utc) + return (created + timedelta(days=limits.retention_days)).isoformat(timespec="seconds") + + +def upload_metadata_body( + metadata: UploadMetadata, limits: UploadLimits | None = None +) -> dict[str, JsonValue]: """Convert UploadMetadata to wire JSON (#498). Content is never included.""" return { "upload_id": metadata.upload_id, @@ -37,6 +58,9 @@ def upload_metadata_body(metadata: UploadMetadata) -> dict[str, JsonValue]: "size_bytes": metadata.size_bytes, "original_filename": metadata.original_filename, "created_at": metadata.created_at, + # So the owner can see the end coming (#1047): past it the upload is deleted and + # a spec that names it no longer builds. + "expires_at": upload_expires_at(metadata, limits), } @@ -133,7 +157,7 @@ def create_upload( ) except ValueError as exc: return ServiceResponse(400, {"error": str(exc)}) - return ServiceResponse(200, upload_metadata_body(metadata)) + return ServiceResponse(200, upload_metadata_body(metadata, limits)) def purge_expired(self) -> int: """Delete every owner's uploads that are past retention; how many went (#1045).""" @@ -150,7 +174,7 @@ def get_upload(self, upload_id: str, *, principal: Principal) -> ServiceResponse metadata = self._repository().get_metadata(principal.owner_id, upload_id) if metadata is None: return ServiceResponse(404, {"error": f"upload not found: {upload_id}"}) - return ServiceResponse(200, upload_metadata_body(metadata)) + return ServiceResponse(200, upload_metadata_body(metadata, self._limits())) def delete_upload(self, upload_id: str, *, principal: Principal) -> ServiceResponse: """Delete only uploads owned by current principal.""" diff --git a/tests/unit/test_upload_limits.py b/tests/unit/test_upload_limits.py index 35263afb..28b3db04 100644 --- a/tests/unit/test_upload_limits.py +++ b/tests/unit/test_upload_limits.py @@ -290,3 +290,82 @@ def test_a_single_user_deployment_is_unchanged( found = dispatch(service, "GET", f"/uploads/{first}", None) assert isinstance(found, ServiceResponse) assert found.status_code == 200 + + +# --- expires_at: the owner can see the end coming (#1047) --- + + +def _get_upload(service: BuilderService, upload_id: str) -> ServiceResponse: + response = dispatch(service, "GET", f"/uploads/{upload_id}", None) + assert isinstance(response, ServiceResponse) + return response + + +def test_an_upload_says_when_it_expires( + multi_user: None, service: BuilderService, monkeypatch: pytest.MonkeyPatch +) -> None: + _as(monkeypatch, _ALICE) + + created = _upload(service) + + body = created.body + expires = datetime.fromisoformat(str(body["expires_at"])) + assert expires - datetime.fromisoformat(str(body["created_at"])) == timedelta(days=30) + # Reading it back says the same. + assert _get_upload(service, str(body["upload_id"])).body["expires_at"] == body["expires_at"] + + +def test_the_date_follows_the_retention_period_in_force( + multi_user: None, service: BuilderService, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(RETENTION_DAYS_ENV, "7") + _as(monkeypatch, _ALICE) + + body = _upload(service).body + + expires = datetime.fromisoformat(str(body["expires_at"])) + assert expires - datetime.fromisoformat(str(body["created_at"])) == timedelta(days=7) + + +def test_the_date_is_the_moment_the_purge_takes_the_upload( + multi_user: None, service: BuilderService, monkeypatch: pytest.MonkeyPatch +) -> None: + """What the field promises and what the purge does are the same boundary.""" + _as(monkeypatch, _ALICE) + upload_id = str(_upload(service).body["upload_id"]) + + _age(service, upload_id, days=29) + still_there = _get_upload(service, upload_id) + assert still_there.status_code == 200 + assert datetime.fromisoformat(str(still_there.body["expires_at"])) > datetime.now(timezone.utc) + assert service.purge_expired_uploads() == 0 + + _age(service, upload_id, days=31) + past = _get_upload(service, upload_id) + assert datetime.fromisoformat(str(past.body["expires_at"])) < datetime.now(timezone.utc) + assert service.purge_expired_uploads() == 1 + assert _get_upload(service, upload_id).status_code == 404 + + +def test_nothing_expires_when_retention_is_off( + multi_user: None, service: BuilderService, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(RETENTION_DAYS_ENV, "0") + _as(monkeypatch, _ALICE) + + body = _upload(service).body + + assert "expires_at" in body + assert body["expires_at"] is None + + +def test_nothing_expires_in_a_single_user_deployment( + service: BuilderService, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.delenv("ENFORCE_OWNERSHIP", raising=False) + monkeypatch.delenv("OIDC_ISSUER", raising=False) + + body = _upload(service).body + + assert body["expires_at"] is None + assert _get_upload(service, str(body["upload_id"])).body["expires_at"] is None