diff --git a/CHANGELOG.md b/CHANGELOG.md index 596169d..cd4aa05 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## [v0.6.0] - 2026-08-25 + +### Added + +- **Paginated catalog surface**: Introduced `iterate_methods` and `iterate_runs` async generators that follow pagination cursors to fetch entire catalogs without silent truncation, backed by new product models (`MethodPage`, `MethodSummary`, `RunPage`, `RunDetail`, `RunErrorReport`, `MethodFile`). A runaway page backstop raises the new `PagingNotTerminatingError` instead of looping forever on a cursor that cycles across non-empty pages. +- **Run details**: Added `get_run_detail(run_id)` to fetch a single run's execution details, including `mthds_contents` and `inputs`, which are excluded from list views for performance. +- **Validation views & typed reports**: Added a `views` parameter to `validate` and `validate_files` for opt-in structured views (e.g. `VALIDATION_VIEW_INPUT_FORM`), and extended `PipelexValidationReport` with typed `warnings`, `liftable_pipes`, and `input_form` fields. +- **Structured repair proposals**: Added `SuggestedFix` and a fix-operation vocabulary (`FixOpKind`, `FixSafety`, etc.) to `ValidationErrorItem`, along with a new `missing_pipe_code` field. +- **Method file parsers**: Added `parse_method_files` and `serialize_method_files` to convert custom Python source files to and from the platform's at-rest catalog string format. +- **Typed `method_id`**: `execute`, `start`, and `start_and_wait` now accept `method_id` as a first-class, typed keyword parameter. + +### Changed + +- `list_methods` and `list_runs` now return page envelopes (`MethodPage` and `RunPage`) instead of bare arrays, with `q`, `limit`, and `cursor` query parameters forwarded based on presence rather than truthiness. (Breaking) +- `PipelineRun.method_id` and `pipe_code` are now nullable to accurately reflect platform behavior for ad-hoc runs and dynamic pipes. (Breaking) +- `MethodData.python` is now a typed `list[MethodFile]` instead of a raw string, converted automatically at the client boundary. (Breaking) +- `delete_method` now returns a `MethodDeletionAccepted` object instead of `None`, reflecting that deletion is an asynchronous cascade rather than an immediate synchronous action. (Breaking) +- The `extra` parameter on run methods now rejects `method_id`; it must be passed via the dedicated named parameter, and passing a non-string `method_id` raises a `PipelineRequestError` at the client boundary rather than delegating the failure to the server. (Breaking) +- Bumped the `mthds` dependency floor from `>=0.8.1` to `>=0.8.2`. +- **Linting & tooling**: Bumped the `ruff` dev dependency to `0.16.4` to match the version the VS Code extension bundles, converted `pyproject.toml` selector lists to rule names instead of codes, and explicitly ignored `too-many-statements-in-try-clause`. + +### Fixed + +- **Pagination crash**: Fixed a critical bug where `list_methods` and `list_runs` crashed against the deployed platform after the API shifted to `{items, next_cursor}` envelope responses; tests were updated to mock the correct paginated shape. +- **Docs – parity claims**: Updated `docs/architecture.md` to honestly reflect the parity gaps with the TypeScript `@pipelex/sdk` (e.g. deferring `lint`, `format`, `codegen`) instead of claiming a surface-complete client. +- **Docs – import paths**: Fixed a broken import path in the `README.md` quickstart (`PipelexValidationResult` is owned by this package, not `mthds`). +- **Docs – brand attribution**: Corrected docstrings and architecture docs to attribute `TokensUsageRecord` as a Pipelex runtime extension rather than an MTHDS protocol specification. +- **Docs – dead links**: Replaced unopenable internal repository citations in docstrings and comments with explicit, readable rule descriptions. + ## [v0.5.0] - 2026-07-22 ### Added diff --git a/README.md b/README.md index 5759113..c5ef348 100644 --- a/README.md +++ b/README.md @@ -28,12 +28,17 @@ The client is **async-only** (httpx `AsyncClient`) and is an async context manag ```python from pipelex_sdk.client import PipelexAPIClient +from pipelex_sdk.validation_models import VALIDATION_VIEW_INPUT_FORM + async def main() -> None: async with PipelexAPIClient() as client: # 1. Validate an MTHDS bundle. The verdict is always returned (never raised): # a 200 discriminated on `is_valid`, carrying `rendered_markdown`. - report = await client.validate([bundle_text]) + # Structured views are opt-in: asking for `input_form` here is what populates + # `report.input_form` (the per-pipe input-form descriptors); omit `views` and the + # request body carries no `views` key at all. + report = await client.validate([bundle_text], views=[VALIDATION_VIEW_INPUT_FORM]) print(report.rendered_markdown) if not report.is_valid: return @@ -83,9 +88,10 @@ There is no barrel import — package `__init__.py` files stay empty. Import eac - **Client & construction** — `from pipelex_sdk.client import PipelexAPIClient, DEFAULT_API_BASE_URL, MthdsFile` - **Run lifecycle types** — `from pipelex_sdk.runs import RunStatus, RunPublic, RunRead, RunResults, RunResultState, WaitForResultOptions, PollInfo` - **Product wire models** — `from pipelex_sdk.product_models import UserProfile, MethodData, MethodWriteInput, Membership, MembershipsResponse, SubscriptionResponse, PlanView, InvoiceView, OnboardingSubmission, UploadInput, UploadedFile, PipelineRun, ...` -- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError` +- **Validation verdict types** — `from pipelex_sdk.validation_models import PipelexValidationResult, PipelexValidationReport, PipelexInvalidReport, ValidationErrorItem, SuggestedFix, VALIDATION_VIEW_INPUT_FORM, ...` +- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, ...` - **Version** — `from pipelex_sdk.version import __version__` -- **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.runners.api.models import PipelexValidationResult`. +- **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.protocol.models import ValidationResult` (the neutral verdict union that `PipelexValidationResult` narrows). ## Development diff --git a/TODOS.md b/TODOS.md new file mode 100644 index 0000000..f66ca0d --- /dev/null +++ b/TODOS.md @@ -0,0 +1,243 @@ +# TODOS — implementing `wip/updates.md` + +This is the implementation tracker for the design in [`wip/updates.md`](wip/updates.md). The design answers *what* and *why*; this file is the *how*, broken into phases with checkboxes. Tick a box when the item is done and verified, not when it is started. Every design choice that was open has been decided (see `wip/updates.md` §7) and is treated here as settled: `input_form` stays opaque, `MethodData.python` is a typed `list[MethodFile]` with the converter in this repo, the `method_id` type guard lands now, and an unknown `FixOp.kind` raises. + +Ground rules for every phase, from `CLAUDE.md`: + +- Branch: `feature/Typed-method-id-run-option` (already carries the typed `method_id` option and the `delete_method` contract fix). The PR targets `dev`. +- Gate each phase on `make agent-check` **and** `make agent-test`; nothing is "done" before both pass. Run `make check` once at the end as well, since it adds pylint on top of the agent gate. +- Tests use `pytest-mock` only, one `TestClass` per module, no `__init__.py` under `tests/`. Mock at the httpx boundary (`_send`), as the existing modules do. +- Everything accumulates under `## [Unreleased]` in `CHANGELOG.md`. No version bump in this work: the `/release` skill cuts the version, and with the breaking items in Phase 3 (plus those already on the branch) that will be a minor bump. +- Docs move with the code, in the same commit. `docs/architecture.md` is the main one to keep truthful; its "Parity with `@pipelex/sdk`" section currently claims surface-completeness and is wrong. +- No hardcoded counts in code, docs, or commit messages. No volatile state in tracked files (this file records decisions and what landed, never whether the tree is clean or tests are green right now). +- Line numbers quoted below are as of the design date (2026-08-25) and will drift; they are anchors, not contracts. + +Suggested order is the order below. Phase 3 is the most urgent fix (a crash against the deployed platform) but is also the largest breaking change, so the plan puts the purely additive Phase 1 first to keep each commit reviewable; reorder if the crash needs to ship alone. + +## Phase 0 — preflight + +- [x] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.2` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. Confirmed: 0.8.2. +- [x] Run `make agent-check` and `make agent-test` before touching anything, so a later failure is attributable to this work and not to the starting point. Both green at the starting point. +- [x] Re-read `wip/updates.md` §6 and §7 once; if anything there contradicts this file, this file is the stale one. No contradiction found. + +## Phase 1 — the `/v1/validate` surface (`wip/updates.md` §1) + +Purely additive. Every affected response model is `extra="allow"`, so nothing parses differently for a body that lacks the new keys; the work is to type what already arrives and to add the one request knob (`views`) that has no way through today. + +### 1.1 `pipelex_sdk/validation_models.py` — the fix vocabulary and the new fields + +- [x] Add `FixSafety(StrEnum)` with `SAFE = "safe"`, `UNSAFE = "unsafe"`, and an `is_safe` property (house rule: never compare enum values inline; a `match` inside the property). +- [x] Add `FixOpKind(StrEnum)` with the kinds the runtime emits: `SET_KEY = "set_key"`, `ENSURE_TABLE = "ensure_table"`, `DELETE_KEY = "delete_key"`, `DELETE_TABLE = "delete_table"`, `RENAME_TABLE_KEY = "rename_table_key"`, `MOVE_KEY = "move_key"`, `REMAP_VALUE = "remap_value"`. Source of truth: `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact `pipelex-api/docs/openapi/pipelex-api.openapi.yaml`. +- [x] Add `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]`. Deeper nesting is not modelled because the server does not emit it; say so in a comment. +- [x] Add a private `_FixOpBase(BaseModel)` with `model_config = ConfigDict(extra="allow")` and `table_path: list[str]` (empty list means the document root), then one subclass per kind, each with `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key: str, value: TomlValue)`, `EnsureTableOp()`, `DeleteKeyOp(key: str)`, `DeleteTableOp()`, `RenameTableKeyOp(key: str, new_key: str)`, `MoveKeyOp(key: str, new_table_path: list[str], new_key: str)`, `RemapValueOp(key: str, mapping: dict[str, str])`. On `EnsureTableOp` and `DeleteTableOp` declare `table_path: list[str] = Field(min_length=1)`, mirroring the artifact's `minItems: 1`. +- [x] These are **reader** models: no `frozen`, no `extra="forbid"`, none of the runtime's wildcard-refusing validators. Put the two runtime invariants a type cannot carry in the docstrings (`*` is the wildcard segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`). +- [x] Add `FixOp: TypeAlias = Annotated[SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, Field(discriminator="kind")]`. Narrowing is `match op: case SetKeyOp(): …`, exhaustive, no `case _`. +- [x] **Verify** that pydantic accepts the raw wire string (`"set_key"`) against `Literal[FixOpKind.SET_KEY]` both in `validate_python` and `validate_json`, and that the discriminator resolves on it. If it does not on the pinned pydantic, use `Literal["set_key"]` on the models and keep `FixOpKind` as the documented vocabulary (with a test that the two sets agree). +- [x] Add `SuggestedFix(BaseModel, extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]` (typed default factory via `empty_list_factory_of` only if a default is warranted — the runtime always sends `ops`, so leaving it required is fine). +- [x] Add `LiftablePipeEntry(BaseModel, extra="allow")`: `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str] = Field(default_factory=list)`, `absence_source: str`. Mirrors `pipelex/pipelex/pipeline/liftable_pipes.py`. +- [x] Add the view token constant next to the field it gates: `VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form"`. A constant, not a closed enum — the request boundary is deliberately open so a stale token never fails a call. +- [x] On `ValidationErrorItem` add `missing_pipe_code: str | None = None` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. Leave `error_type: str | None` as an open string; do not enum it. +- [x] On `PipelexValidationReport` add `warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem))`, `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry))`, and `input_form: dict[str, Any] | None = None`, each with a docstring: `warnings` never flips `is_valid`; the two lists default empty so a pre-0.52 runner's body still parses; `input_form` is present only when the request named the `input_form` view (a 0.17.0 runner emitted it unconditionally, which `None`-by-default also reads correctly) and is opaque on purpose, keyed like `pipe_io_contracts`. +- [x] `PipelexInvalidReport` gains nothing; add one sentence to its docstring saying why (the invalid arm never carries `warnings` or `input_form` — they derive from a crate that was never assembled). +- [x] Update the module docstring's list of neutrally-named supporting types to include the new ones, and keep the brand rule stated there (the `Pipelex` prefix stays on the two envelopes only). + +### 1.2 `pipelex_sdk/client.py` — `views` on `validate` and `validate_files` + +- [x] `validate(...)`: append `views: list[str] | None = None` after `render`. When `views is not None`, set `extra["views"] = views` **verbatim** — no injection, no de-duplication, an explicit `[]` is sent as `[]`. When `None`, the key is absent from the body. It rides `_post_validate`'s `extra` exactly like `render` and `mthds_sources`; no `mthds-python` change. +- [x] `validate_files(...)`: append `views: list[str] | None = None` after `render` and thread it through to `validate`. +- [x] Docstrings: the sentence "differs from the inherited protocol `validate` in two Pipelex-API ways" becomes three (render injection, `mthds_sources`, `views`); document the `views` semantics (opt-in structured views; `input_form` is the only token today, named by `VALIDATION_VIEW_INPUT_FORM`; unknown tokens are lenient-ignored server-side, never a `422`; the default response stays byte-identical for consumers that discard views). +- [x] The `Returns:` section of `validate` should mention that a valid report now carries `warnings`, `liftable_pipes`, and (when asked) `input_form`. + +### 1.3 Tests + +- [x] `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the `views` key absent from the body when the parameter is omitted; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` injection unchanged when `views` is also passed. +- [x] `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes`, and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None` — this is the regression guard against a future "tighten to required" edit, and it answers the inbox item `../wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` for the Python mirror. +- [x] `tests/unit/test_validation_contract.py`: an invalid body carrying `missing_pipe_code` and a `suggested_fix` with at least two ops of different kinds parses, and `match`-narrowing reaches each op's own members; an `ensure_table` op with an empty `table_path` is rejected; an unknown `kind` raises `pydantic.ValidationError`; `FixSafety` and `FixOpKind` value sets are asserted as the locked vocabularies (same style as `test_category_vocabulary_is_the_locked_set`). +- [x] Keep the canonical bodies where the module already keeps them (module-level constants next to `VALID_BODY`); move to a `tests/unit/test_data.py` only if the module becomes unreadable. + +### 1.4 Docs and changelog + +- [x] `docs/architecture.md` → "`validate` override": add a `views` bullet beside the render bullet; list the typed valid-arm additions (`warnings`, `liftable_pipes`, `input_form`) and the `ValidationErrorItem` additions with the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary; one sentence that `PipeInputContract.optional` became `presence` and that the `fixed` multiplicity carries `item_count` inside the opaque `pipe_io_contracts`, so nobody discovers the new spellings by surprise. +- [x] `docs/architecture.md` → "Brand boundary": the list of neutrally-named supporting types gains the new ones. +- [x] `README.md` quickstart: one line showing `views=[VALIDATION_VIEW_INPUT_FORM]` (or a comment that the input form is opt-in), so the knob is discoverable. +- [x] `CHANGELOG.md` `[Unreleased]` → **Added**: `views` on `validate` / `validate_files`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the fix vocabulary; a note that a body from an older runner still parses (the lists default empty, `input_form` defaults `None`). + +### 1.5 Gate and commit + +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Type the pipelex-api 0.17/0.18 validate contract and add the views opt-in"). + +### Checkpoint 1 + +- [x] Update this file: tick what landed, record the SHA of the Phase 1 commit, note whether pydantic accepted the enum `Literal` tags or the string fallback was needed, and any deviation from §1 of the design with its reason. + +**Landed in `434b2e3`.** Notes: + +- **The enum `Literal` tags work as written; the string fallback was not needed.** Verified against the pinned pydantic (2.13.4) before writing the models: a raw wire `"set_key"` validates against `Literal[FixOpKind.SET_KEY]` in both `validate_python` and `validate_json`, the discriminator resolves on it, an unknown `kind` raises, and `Field(min_length=1)` on `EnsureTableOp.table_path` rejects an empty path. +- **`RemapValueOp.mapping` is left unconstrained**, where the runtime and the OpenAPI artifact both declare `minProperties: 1`. These are reader models: an empty mapping is an advisory no-op, not a parse hazard, and refusing it would fail a whole verdict over a harmless op. The `minItems: 1` on `ensure_table` / `delete_table` was kept because there the empty case is genuinely meaningless (the document root always exists, and cannot be deleted). +- **One Phase-2 item landed early**, because `validation_models.py` was rewritten wholesale here: the `conformance/conformance/validation_contract.py` citation on `ValidationErrorCategory` is already replaced with the rule it was citing. Phase 2 covers the rest. +- No other deviation from §1 of the design. + +## Phase 2 — prose corrections (`wip/updates.md` §2) + +No behaviour change. Two fixes `@pipelex/sdk` 0.14.0 shipped under "Fixed" that apply here for the same reason (`pipelex-sdk` is a public PyPI package). + +- [x] **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py` (module docstring near line 23 and the class docstring near line 128), `docs/run-usage.md` (line 5), `docs/architecture.md` (the `TokensUsageRecord` bullet near line 103): the record is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API pins the wire contract. Reword as `pipelex-sdk-js/src/runs.ts` and its `docs/architecture.md` did. `docs/run-usage.md` line 5 already says the right thing in its second sentence; make the first sentence agree with it. +- [x] **Citations a reader cannot open.** Replace each bare workspace-private path with the rule it was citing: `pipelex_sdk/client.py` near line 138 (`docs/specs/pipelex-platform-api.md` → "the layered extension policy: a hosted client types its own platform's arguments and guards them per layer"); `pipelex_sdk/validation_models.py` near line 44 (`conformance/conformance/validation_contract.py` → "the locked category vocabulary shared with the conformance corpus"); `tests/unit/test_validation_contract.py` module docstring and the docstring of `test_category_vocabulary_is_the_locked_set`; `tests/unit/test_runs.py` near line 12; `tests/unit/test_client_method_id.py` module docstring; `docs/architecture.md` near line 84. Keep the JS mirror references (`pipelex-sdk-js/...`) where they explain a port — those are a sibling public repo, not a private path. +- [x] `CHANGELOG.md` `[Unreleased]` → **Fixed**: two entries mirroring 0.14.0's wording. +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Correct the TokensUsageRecord attribution and drop unopenable citations"). + +## Phase 3 — product paging and nullability (`wip/updates.md` §3) + +Breaking, and the most urgent fix in this plan: `list_methods` and `list_runs` crash against the deployed platform because both routes now answer a `{items, next_cursor}` envelope, and `PipelineRun` requires fields the platform serves as nullable. Wire fields stay snake_case (`next_cursor`), where the JS mirror renamed to `nextCursor` for its own consumers. + +### 3.1 `pipelex_sdk/product_models.py` — models + +- [x] Add `MethodFile(BaseModel, extra="allow")` with `name: str`, `content: str`, defined **before** `MethodData`. Docstring: the at-rest catalog form of one source file (`[{name, content}]`), distinct from `MthdsFile` (`client.py`, validate input) and `MthdsFileItem` (`build_models.py`, build closure) — three shapes for three surfaces, name the difference so nobody merges them. +- [x] Add `parse_method_files(source: str | None) -> list[MethodFile]`: blank source (`None`, `""`, whitespace) and `"[]"` both yield `[]`; a JSON array of `{name, content}` yields those files with blank-content entries dropped; anything else (non-array JSON, a malformed entry, unparseable text) raises `ValueError` with a message naming the expected shape. Implementation: `json.loads` then `TypeAdapter(list[MethodFile])` (built once at module level, TypeAdapter construction is expensive), wrapping `json.JSONDecodeError` / `pydantic.ValidationError` into the `ValueError`. +- [x] Add `serialize_method_files(files: list[MethodFile]) -> str`: drop blank-content entries; an empty result serializes to `""` (the platform's "no source / clear" sentinel), never `"[]"`; otherwise `json.dumps` of `[{name, content}]` only (no extras), stable key order. +- [x] `MethodData`: add `org_id: str`, `created_by_user_id: str`, `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, `python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile))`, plus a `@field_validator("python", mode="before")` that applies `parse_method_files` when the incoming value is a `str` or `None` and passes a list through unchanged (so programmatic construction in tests still works). A `ValueError` from the parser surfaces as `pydantic.ValidationError` from `model_validate`, the same way any malformed response body fails here. +- [x] `MethodWriteInput`: add `python: list[MethodFile] | None = None` with a `@field_serializer("python")` returning `serialize_method_files(value)`. Docstring the three-way contract: `None` → key absent (the write body dumps with `exclude_none=True`) → the stored Python is preserved on `PUT`; `[]` → sent as `""` → clears it; a non-empty list → replaces it. +- [x] Add `MethodSummary(BaseModel, extra="allow")`: `method_id: str`, `name: str`, `description: str | None = None`, `created_at: str`, `deletion_state: MethodDeletionState | None = None`. Docstring: deliberately not a `MethodData` — no `mthds`, no `python`, no `updated_at` — because none is in the index projection and putting `mthds` back is what restored the truncation bug; a method mid-deletion stays in the list so a UI can render "Deleting…" while `get_method` refuses it with a `409`. +- [x] Add `MethodPage(BaseModel, extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. Docstring: opaque cursor, pass it straight back; `None` means last page; no total by design. +- [x] Add `RunErrorReport(BaseModel, extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. +- [x] `PipelineRun`: `method_id: str | None = None` (an ad-hoc run from an inline bundle belongs to no stored method), `pipe_code: str | None = None` (resolved from the bundle's `main_pipe`); add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. Leave `pipe_statuses` as it is. +- [x] Add `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None`. Docstring: `mthds_contents` is what the run actually executed and the only record of it; both fields are absent from the list and the polled status read on purpose (size × page size, size × poll rate). +- [x] Add `RunPage(BaseModel, extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. +- [x] Update the section comments in the module (`# ── Methods catalog`, `# ── Run records`) so the new models sit under the right banner. + +### 3.2 `pipelex_sdk/errors.py` — the runaway-paging error + +- [x] Add one error for `iterate_methods` refusing to keep paging past the ceiling (a name like `PagingNotTerminatingError`), extending whatever base the module's other product errors extend — check the existing hierarchy there first. Message mirrors the JS one: the iterator did not terminate after the ceiling; this is a server-side fault, not a coverage limit. + +### 3.3 `pipelex_sdk/client.py` — list, iterate, detail + +- [x] Add a module helper `_product_query(params: dict[str, str | int | None]) -> str` that keeps entries on **presence** (`is not None`, never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to drop silently) and encodes with `urllib.parse.urlencode`, returning `""` or `?…`. The existing `test_list_runs_encodes_query_value` assertion (`method_id=m%2F1`) must stay green, so keep `/` percent-encoded. +- [x] Add `_MAX_LIST_PAGES: int = 10_000` beside the other module constants (the JS `MAX_PAGES`), with the comment that it is a runaway backstop set far beyond any real catalog, not a coverage cap. +- [x] `list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage` — `GET /v1/methods` with the query built by the helper; parse `MethodPage`. Docstring: `q` is a server-side case-insensitive substring match over name and description across the whole catalog; `limit` defaults to and is capped by the API; ordering is by creation, newest first. +- [x] `iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]` as an `async def` generator: request a page; **before yielding**, stop if `cursor is not None and page.next_cursor == cursor` (the server did not advance; yielding first would double-count); yield every item; stop when `page.next_cursor is None`; otherwise count the page and **raise** the new error once the count reaches `_MAX_LIST_PAGES`; continue **through empty pages** with a live cursor, because `q` is a post-read filter over a bounded index slice per request and `{items: [], next_cursor: "…"}` means "keep going". Docstring says why there is no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. +- [x] `list_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage` — `GET /v1/runs?method_id=…` plus the presence-kept query; parse `RunPage`. Docstring: `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, key conditions rather than filters; a bare date or naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also document the gate the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()`, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — a `403` here means "flag", not "wrong key". +- [x] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). ~~No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing.~~ **Deviation, taken on PR review:** the ceiling applies here too. The plan's reason was too narrow — the empty-page stop only catches a server returning *nothing*, so a cursor cycling across two or more values (`c1 → c2 → c1`) over non-empty pages trips neither it nor the adjacent-cursor check and would loop forever re-yielding the same runs. Greptile and Codex flagged it independently. The fix reuses `_MAX_LIST_PAGES` and `PagingNotTerminatingError` rather than tracking every cursor seen, which would cost unbounded memory for the same protection. +- [x] `get_run_detail(self, run_id: str) -> RunDetail` — `GET /v1/runs/{id}` via `_request_product` with the id path-encoded like the other id routes (`f"{_RUNS}/{quote(run_id, safe='')}"`). Distinct from `get_run_status` (`/status`) and `get_run_result` (`/results`). +- [x] Update the `PipelexAPIClient` class docstring's product-surface bullet and the import block (`MethodPage`, `MethodSummary`, `RunDetail`, `RunPage`, `AsyncIterator` from `collections.abc` under `TYPE_CHECKING` if only used in annotations — it is used at runtime as a return annotation with `from __future__ import annotations`, so `TYPE_CHECKING` is fine). + +### 3.4 Tests + +- [x] `tests/unit/test_client_product.py`: replace the bare-array fixtures at `test_list_methods` and `test_list_runs_encodes_query_value` with envelopes and assert `MethodPage` / `RunPage` come back with `next_cursor`; add query-encoding cases for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped and that an absent parameter leaves no key in the query; a run row with `null` `pipe_code` and `method_id` parses; `get_run_detail` hits `/v1/runs/{id}` with encoding and returns `mthds_contents` and `inputs`; `MethodData` parses the new fields with `python` read from the wire string into `MethodFile` entries and `""` reading as `[]`; `create_method` / `update_method` send `python` three ways (`None` absent, `[]` as `""`, a list as the JSON text). +- [x] New `tests/unit/test_method_files.py` (one `TestClass`): `parse_method_files` on blank / `"[]"` / a valid array / an array with a blank-content entry / a non-array / a malformed entry / unparseable text; `serialize_method_files` on empty / blank-only / mixed; a round-trip is stable. +- [x] New `tests/unit/test_client_paging.py` (one `TestClass`): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding the page; `iterate_runs` stops on an empty page; `iterate_methods` raises the new error past the ceiling (patch `_MAX_LIST_PAGES` down via `mocker.patch` rather than looping ten thousand times); the cursor sent on page N+1 is the `next_cursor` received on page N. Use `mocker.AsyncMock(side_effect=[...])` on `_send` to script the page sequence. +- [x] The `_response` / `_Sent` / `_mock_send` helpers live as private members of `tests/unit/test_client_product.py`. Rather than importing private test helpers across modules, promote a response builder and a `_send` spy to fixtures in a new `tests/unit/conftest.py` for the new modules to use (house rule: fixtures go in `conftest.py`). Migrating `test_client_product.py` onto those fixtures is optional and not part of this change. + +### 3.5 Docs and changelog + +- [x] `docs/architecture.md` → "Pipelex product surface": rewrite the **Methods catalog** bullet for `MethodPage` / `MethodSummary` / `iterate_methods` and the `python` three-way write contract with `MethodFile`; rewrite the **Run records** bullet for `RunPage` / `iterate_runs` / `get_run_detail`, the nullable `PipelineRun` fields and `error`, the instant-only date bounds, and the `ff_api_keys` `403`. State the two iterator stop rules and why they differ. +- [x] `docs/architecture.md` → "Parity with `@pipelex/sdk`": it must stop claiming "surface-complete, with no silent gaps". Rewrite it to list the conscious exclusions honestly: `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — unchanged by the cited releases and deferred. While there, fix the stale "Out of scope for v0.1" bullet that still lists `/v1/build/*` helpers as deferred even though `build_inputs` shipped in 0.5.0. +- [x] `README.md`: if the quickstart gains a listing example, use `iterate_methods` rather than a page loop, so the idiom people copy is the one that cannot truncate. +- [x] `CHANGELOG.md` `[Unreleased]`: **Changed (breaking)** — `list_methods` returns `MethodPage` (items are `MethodSummary`, not `MethodData`), `list_runs` returns `RunPage`, `PipelineRun.method_id` / `pipe_code` are nullable, `MethodData.python` is `list[MethodFile]`; **Added** — `iterate_methods`, `iterate_runs`, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the new `MethodData` fields, `MethodWriteInput.python`, the paging error; **Fixed** — name the crash plainly (iterating the envelope dict yielded its keys, so the first call was `MethodData.model_validate("items")`), and that the unit tests mocked the pre-paging shape. + +### 3.6 Gate and commit + +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Follow the platform's paged method and run lists and stop requiring nullable run fields"). + +### Checkpoint 2 + +- [x] Update this file: tick what landed, record the Phase 2 and Phase 3 commit SHAs, and note any place the Python shapes deliberately diverge from the JS mirror beyond snake_case (there should be none besides the `python` converter living here). + +**Phase 2 landed in `2a8c589`, Phase 3 in `5e01c1b`.** Notes: + +- **Divergences from the JS mirror beyond snake_case:** the `python` converter lives here rather than in `mthds-python` (the decision of `wip/updates.md` §7.2), and `parse_method_files` raises `ValueError` where the JS pair raises `PipelineRequestError` — because the Python parser is reached through a pydantic `field_validator`, where a `ValueError` is the idiomatic signal and surfaces to the caller as a `pydantic.ValidationError` like any other malformed response body. Nothing else diverges. +- **`MethodPage.items` / `RunPage.items` are required**, not defaulted empty. A page body with no `items` is malformed, and failing loudly is the whole point of this phase — the previous shape failed silently in the tests and loudly in production. +- **The shared test fixtures landed in a new `tests/unit/conftest.py`** (`api_client`, `wire_response`, `patch_send`), used by the two new modules. Migrating `test_client_product.py` onto them was left out as the plan allowed; it keeps its own equivalent private helpers. + +## Phase 4 — `method_id` boundary type guard (`wip/updates.md` §4) + +The 2026-08-25 decision in `pipelex-sdk-js/wip/boundary-option-type-validation.md`: a published client validates request-option types at its boundary and raises `PipelineRequestError` rather than dropping or forwarding a wrong-typed value. Its evidence names this repo's bare `if method_id:` in `_merge_hosted_run_extensions`, which drops falsy non-strings and forwards truthy ones to a server `422`. + +- [x] `_merge_hosted_run_extensions` (`client.py` near line 975): replace `if method_id:` with an explicit presence check (`if method_id is not None`) followed by `if not isinstance(method_id, str): raise PipelineRequestError(msg)` naming the received type, then the existing empty-string-is-absent normalization. `None` and `""` still contribute nothing. +- [x] Update the docstring's `Raises:` and the "An absent or empty `method_id`" paragraph. +- [x] `tests/unit/test_client_method_id.py`: one parametrized test over wrong-typed values (`0`, `123`, `[]`, `["mt_1"]`, `{}`) asserting `PipelineRequestError` on `execute` and `start` before any request is sent; keep `test_empty_method_id_is_absent` green. +- [x] `docs/architecture.md` → "Hosted run extensions (`method_id`)": add a bullet that a non-string raises at the boundary, and why (one partition of wrong values across both SDKs). +- [x] `CHANGELOG.md` `[Unreleased]` → **Changed**: the guard, with the JS decision as the reason. +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Reject a non-string method_id at the client boundary"). + +## Phase 5 — wrap-up + +- [x] `make check` (adds pylint to the agent gate) and `make agent-test` pass on the final tree. +- [x] Re-read `docs/architecture.md` end to end for any remaining claim the code no longer supports (parity, `Out of scope`, the validate section, the product section). Three further corrections beyond the ones §3.5 named: the intro line claiming "the full `0.1.0` surface", the parity section's "**Methods** — full coverage" (which contradicted the gap list directly above it), and the "**Models** — full field-for-field match" claim; the Models paragraph now also names the two deliberate divergences (snake_case `next_cursor`, the converter's home). +- [x] Re-read `CHANGELOG.md` `[Unreleased]`: every breaking item is labelled "breaking", no counts, no WIP-doc mentions, the version line untouched (still `0.5.0`). +- [x] `wip/updates.md` stays where it is, with its §7 decisions; this file stays too. Neither is deleted or emptied as part of finishing the work. +- [x] Open the PR against `dev`, then follow `/review-pr-agents` for the Greptile / Codex loop (compare SHAs, not notifications; reply and resolve each thread in one pass). **PR [#14](https://github.com/Pipelex/pipelex-sdk-python/pull/14).** + +### Checkpoint 3 + +- [x] Update this file with the final commit SHAs, the PR number, and anything deferred out of the plan with the reason. + +**PR [#14](https://github.com/Pipelex/pipelex-sdk-python/pull/14), against `dev`.** The five implementation commits, in order: + +| SHA | What | +|---|---| +| `434b2e3` | Phase 1 — the validate surface: the `views` opt-in and the typed 0.17/0.18 contract | +| `2a8c589` | Phase 2 — the `TokensUsageRecord` attribution and the unopenable citations | +| `5e01c1b` | Phase 3 — paged method and run lists, nullable run fields, the `python` converter | +| `cb70fbe` | Phase 4 — the non-string `method_id` boundary guard | +| `a857d02` | Phase 5 — the remaining untrue parity claims in `docs/architecture.md` | + +Deferred out of the plan, with the reason: + +- **`RemapValueOp.mapping` is not constrained to be non-empty**, where the runtime and the OpenAPI artifact both say `minProperties: 1`. Reader models should not fail a whole verdict over an op that is merely a no-op. Recorded at Checkpoint 1. +- **Migrating `test_client_product.py` onto the new `tests/unit/conftest.py` fixtures** was explicitly optional in §3.4 and was not done; the module keeps its own equivalent private helpers. +- **`pipelex_sdk/runs.py`'s "Wire contract mirrors `pipelex-platform`" line** was left alone. It names a service, not a file path, so it is not one of the unopenable citations Phase 2 was about, and widening that phase's scope on my own judgement was not warranted. + +### Corrections found on PR review + +Cubic's pass on `fa8d62a` reported findings against files this plan touched. Three were real and are fixed; the rest were judged not worth acting on, and the reasons are recorded here rather than only in the resolved GitHub threads. + +Fixed: + +- **The `TokensUsageRecord` attribution in `docs/architecture.md` was missed.** Phase 2's first checkbox names that site explicitly, and `CHANGELOG.md` asserts it was corrected, but `2a8c589` only touched the `method_id` citation in that file. `pipelex_sdk/runs.py` and `docs/run-usage.md` were correct. The bullet now carries the same wording as the other two. +- **`CHANGELOG.md` still cited `docs/specs/pipelex-platform-api.md`.** The changelog was outside Phase 2's enumerated citation sites, so the sweep did not reach it — leaving the `[Unreleased]` section claiming "no more citations a reader cannot open" a few entries below a citation a reader cannot open. The sentence now states the rule and points at the sibling public JS SDK instead. +- **The `validate` override intro in `docs/architecture.md` still said "two Pipelex-API extensions"** after Phase 1 added the third. The client docstring was updated at the time; the doc's intro sentence was not. + +Not acted on, with the reason: + +- **Hoisting the `method_id` type guard above `start_and_wait`'s lifecycle handshake.** The claim that a handshake failure can mask the guard is wrong: `_supports_run_lifecycle` swallows its errors and every downstream path still reaches `_merge_hosted_run_extensions`, so no wrong-typed value escapes. The real cost is one probe request, memoized for the client's lifetime. Hoisting would mean either calling the merge helper twice or duplicating the check outside the single documented guard site. +- **A claimed 194-character line in `tests/unit/test_client_validate.py`.** Measured at 127, under the configured 150, and no line in that file is longer; `ruff check` passes. The measurement was simply wrong. +- **Rewriting Checkpoint 2's parenthetical to match the two divergences the note beneath it records.** The parenthetical is the expectation the plan set out with, and the note is the finding; editing the prediction to match the outcome removes the only evidence that the plan's expectation was slightly off. +- **The `../wip/inbox/…` reference in §1.3.** Phase 2's rule is about the *shipped* surface — docstrings, comments and doc pages that travel to PyPI, where a workspace path resolves to nothing and reads as rot. This tracker is not that: it is a working document for the reviewers of this PR, it is written from workspace context throughout (it cites `wip/updates.md` and the JS repo's own `wip/` in the same way), and `../wip/inbox/` is the notation the workspace guide itself prescribes for a sub-repo checkout. The path resolves where the document is read, and the sentence already states its rationale inline before citing the item, so a reader who cannot open it loses only the provenance. +- **Replacing the two `# type: ignore[arg-type]` comments in `tests/unit/test_client_method_id.py` with `cast()`.** A cast would assert to the reader that the value *is* a `str | None`, which is the exact falsehood that test exists to disprove at runtime; the ignore states the truth that this is a deliberate type error, which is the last resort the coding standard permits. + +### Final pre-landing review + +Once the three PR bots reported clean on `301d96e`, a fresh reviewer with no context from this session ran the `gstack` review procedure against this plan, with an adversarial pass folded in. It found no correctness defect in the shipped code, and it verified the wire contracts against `pipelex-server` itself rather than against the docstrings that assert them — including that the two iterators' asymmetric stop rules match the two DynamoDB adapters (the method adapter can mint a cursor for an empty page because `q` filters after the read, the run adapter over-fetches by one and cannot), and that the pydantic three-way write contract for `python` holds for absent / `null` / `""` / `"[]"` alike. + +Fixed: + +- **`README.md` advertised an import that raises `ImportError`.** "Public import paths" sent readers to `from mthds.runners.api.models import PipelexValidationResult`; that name is not in the installed `mthds` at all. The Pipelex narrowing of the verdict union is owned here, which `docs/architecture.md` states in its "Brand boundary" section — so the repo contradicted itself. The section gains a `pipelex_sdk.validation_models` bullet and now cites `mthds.protocol.models.ValidationResult` as the neutral union it narrows. Pre-existing, but this branch edited the quickstart three lines above it. +- **The two page-ceiling tests could not fail.** Their only assertion was `exc_info.value.page_limit`, which just echoes back the constant the test itself patched, so five scripted responses against a limit of two passed whether the raise fired on page 1 or page 5. Each now also asserts `send.call_count`, which was mutation-checked: loosening `pages_seen >= _MAX_LIST_PAGES` to `>` fails both tests where it previously passed. +- **Two hardcoded counts that had already gone stale inside this branch.** `pipelex_sdk/client.py`'s `validate` docstring and `docs/architecture.md` both said "three Pipelex-API ways/extensions" — the same rot the entry above records having corrected once already, which is exactly why the workspace guide forbids writing counts down. Both now say neither two nor three. +- **`execute` and `start` under-documented an exception this branch added.** Their `Raises:` entries named only the `extra`-smuggling trigger; Phase 4's guard raises `PipelineRequestError` for a non-string `method_id` too. The helper's own docstring had been updated at the time, the two public methods' had not. + +Deferred to `wip/pr-14-review-notes.md`, each verified and none blocking: the sdist ships this tracker and the other internal planning documents (a packaging decision that belongs with a release, not inside a feature branch); the client class docstring still dates two surfaces by build-plan phase number; and `start_and_wait`'s `Raises:` omits the `PipelineRequestError` it propagates on both paths. + +One finding reached outside this repo and was filed rather than fixed: `PipelineRun.pipe_statuses` is a field no server has ever filled — it is absent from the platform's `RunPublic` response model and appears nowhere in `pipelex-server` — yet it is declared in this SDK, in `@pipelex/sdk`, and in `pipelex-app`, where run-history progress dots are gated on it and have therefore never rendered. Retiring it here alone would break the parity invariant this package is built on, so the decision belongs to whoever owns the run wire contract: `../wip/inbox/2026-08-25-workspace-pipe-statuses-dead-field-in-three-clients.md`. + +Considered and declined: guarding `iterate_methods` against a `next_cursor` of `""`, which would let a page double-yield. The adversarial pass demonstrated the mechanism and then confirmed the case is unreachable, since the platform's cursor is a base64 `LastEvaluatedKey`. That is the impossible-scenario defensiveness this plan set out to avoid. + +Cubic's pass on `cb391c9` reported four more. Two were real: + +- **`docs/architecture.md` documented an API that does not exist.** The hosted-run-extensions section contrasted `method_id` with "`build_inputs(method_id=…)` and `prepare_inputs`, which are client-side sugar that resolves an id to inline files". Neither helper accepts a `method_id`: `build_inputs` takes a `BuildInputsRequest` of `files` / `pipe_ref` / `format` / `explicit`, `prepare_inputs` takes `files` / `pipe_ref` / `inputs`, and catalog-id resolution for input preparation is recorded as deferred in the 0.5.0 changelog. A reader following that sentence gets a `TypeError`. The contrast it wanted to draw is real, so the bullet now draws it truthfully. This is exactly the class of untrue claim Phase 5 set out to remove from this file, in a section Phase 4 added. +- **This tracker recorded the wrong `mthds` floor.** Phase 0 wrote `>=0.8.1`; `pyproject.toml` has said `>=0.8.2` since `bc17c07`, which is an ancestor of this branch's base — so the floor was already 0.8.2 when the preflight ran, and the number was wrong the day it was written. + +Two were not worth a commit: + +- **Adding this branch's own review-notes file to the sdist inventory in `wip/pr-14-review-notes.md`.** That listing is `tar -tzf` output captured from a build made before the notes file existed. Extending it changes nothing about the finding it supports — that the sdist ships internal planning documents — or about what someone picking the item up would do. +- **The counts in the completion narrative above ("two page-ceiling tests", "two hardcoded counts").** The no-hardcoded-counts rule exists because a live inventory drifts: it "creates diff churn, goes stale silently, and adds no value". None of that applies to a closed record of a finished review round, which gains no members and cannot go stale. The rule targets counting things that are still moving. + +## Known gaps left open on purpose + +- No e2e suite exists in this repo (`tests/` holds only `unit/`), so the live `views` gate and the live paging envelope are pinned only by mocked bodies here; the JS suite pins both live. Adding an e2e suite is separate work. +- The remaining `@pipelex/sdk` surfaces without a Python counterpart (`lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure`) are untouched by the cited releases and stay deferred; Phase 3 makes `docs/architecture.md` say so. +- The protocol-argument type guards (`pipe_code`, `mthds_contents`) belong in `mthds-python` and arrive here with the `mthds` floor bump once that package ships its Phase 1. diff --git a/docs/architecture.md b/docs/architecture.md index 5f8b652..5002ab2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # pipelex-sdk architecture -The design reference for the package. It covers the full `0.1.0` surface and records the parity audit against the TypeScript `@pipelex/sdk` reference (see "Parity with `@pipelex/sdk`" at the end). +The design reference for the package. It covers the shipped surface and records where it stands against the TypeScript `@pipelex/sdk` reference, gaps included (see "Parity with `@pipelex/sdk`" at the end). ## What this is @@ -29,7 +29,7 @@ mthds.runners.api.client.MthdsAPIClient (protocol-only base: transport, body-b MTHDS is the brand of the open standard (the language, the protocol). Pipelex is the brand of the hosted runtime/product. Artifacts that belong to the standard keep neutral, un-prefixed names; Pipelex branding is reserved for genuinely runtime/product-specific surfaces (the durable run lifecycle, the product routes, implementation envelopes). The five protocol routes and their neutral models stay in `mthds`; everything Pipelex-specific lives here. -The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). +The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus` / `LiftablePipeEntry` / `SuggestedFix` / the `FixOp` variants / `FixOpKind` / `FixSafety`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). ## Credentials & configuration @@ -77,6 +77,19 @@ Everything else stays the inherited regime: the protocol's optional 202 async-de The override also **enriches the return type**: it re-validates the base result into a `PipelexExecuteResult` (`pipelex_sdk/execute_result.py`), a `DictRunResultExecute` subtype that adds a resolved `.main_stuff` accessor (dug out of `pipe_output`'s working memory via the response's `main_stuff_name`, raising `MissingMainStuffError` if unlocatable). This gives blocking and durable results the **same output accessor** — `result.main_stuff` — so callers never branch on which path ran. `_map_run_result_to_run_results` reads that accessor too, keeping the resolution single-sourced. +## Hosted run extensions (`method_id`) + +The protocol's run args (`pipe_code`, `mthds_contents`, `inputs`, …) are the base client's named parameters and stay pure. The hosted API's *own* run args are named parameters **here**, on `execute` / `start` / `start_and_wait`. Today that is one, `method_id`, and the reasoning generalizes to every hosted-only argument that follows: + +- **A named parameter, not an `extra` entry.** Typing its own platform's arguments is the one job a layer-3 client exists to do; `extra` stays the escape hatch for an extension this client does not know about (a third-party server, or a newer version of ours). That split, and the reserved-key guard behind it, follow the layered extension policy: a hosted client types its own platform's arguments and guards them per layer, and that guard must never be pushed down into the protocol package. It was previously passed as `extra={"method_id": …}`, which worked but documented nothing and validated nothing. +- **It reaches the wire through the base client's extension mechanism.** `_merge_hosted_run_extensions` folds it into the `extra` mapping handed to `super().execute` / `super().start`, which merges it into the body as a top-level property without knowing what it means. That is the layering working as designed, not a workaround: the protocol client stays catalog-agnostic while the hosted client owns the concept. +- **The guard is per layer.** `method_id` is rejected inside `extra` *here*, and must never become reserved in `mthds`: a protocol client talking to another vendor's server has no business rejecting that vendor's arguments. +- **Pure pass-through — nothing is expanded client-side.** The platform resolves the id against the org's catalog. The input-preparation helpers are the deliberate contrast: `build_inputs` and `prepare_inputs` take the method closure as inline `files` (with an optional `pipe_ref`) and accept no `method_id` at all — resolving a catalog id to its files client-side is deferred, additive work — so the id is a run option only and reaches no other wire body. +- **Alone it is a run source; alongside an inline source it is linkage.** With no inline source the platform resolves and runs the stored method. With one, the inline source is what RUNS (precedence) and the id is recorded as run-history linkage on the Run row — the index key `GET /v1/runs?method_id=` queries. The base client's "something to run" precondition is satisfied because the merged extension mapping is non-empty, so a `method_id`-only run is sent rather than refused client-side. +- **It rides the blocking fallback too.** `start_and_wait` degrades to `POST /v1/execute` against a runner with no run store, and the selector goes with it — so a bare `pipelex-api` answers the `422` that names the key instead of the client silently dropping it and running something else. +- **An empty string is treated as absent.** `method_id=""` selects no method and links nothing, so it is not sent and does not satisfy the precondition. +- **A non-string raises at the boundary.** A published client validates its request-option types where it names them, raising `PipelineRequestError` rather than dropping or forwarding a wrong-typed value — so one wrong value gets one answer. Without the guard the partition is arbitrary: a bare truthiness check drops the falsy wrong types (`0`, `[]`) and forwards the truthy ones (`123`, `["mt_1"]`) to a server `422`, which is a *different* partition than `@pipelex/sdk` makes for the same argument on the same wire. Both SDKs now make the same one. + ## Run lifecycle (hosted extension) The durable run lifecycle (`pipelex_sdk/runs.py` + the client's lifecycle methods) is a **hosted-API extension, not part of the MTHDS Protocol**. Long method runs outlive the hosted gateway's ~30s synchronous cap, so a caller submits a run (`POST /v1/start`), then polls a self-healing endpoint by bare `pipeline_run_id` until it reaches a terminal state. All state lives behind the id (DynamoDB + Temporal on the platform), so a caller can drop the poll loop and resume later with just the id. A bare runner has no run store and `404`s these routes, which the client translates into a clear `RunLifecycleUnavailableError`. @@ -88,7 +101,7 @@ The durable run lifecycle (`pipelex_sdk/runs.py` + the client's lifecycle method - `RunStatus` — the hosted status enum, with `is_terminal` / `is_success` predicates (exhaustive `match`). - `RunRead` — a run record read through the self-healing status path (adds `degraded` + `retry_after_seconds`). - `RunResults` — result artifacts. `main_stuff` (the resolved main output content) is always present for a completed run: on the hosted path it is the `main_stuff.json` S3 artifact; on the bare-runner blocking path the SDK resolves it from the returned working memory via the response's `main_stuff_name`, so both paths deliver the same shape. Consumers read `main_stuff` directly. The full working memory still rides `pipe_output` (blocking path only) for consumers that want it, and `graph_spec` rides the hosted path. A completed run that cannot deliver a main stuff raises `MissingMainStuffError`. Extension-open, so any other server artifact is preserved. -- `TokensUsageRecord` — one client-facing usage record per inference call, carried by `RunResults.tokens_usages` on both paths. A mirror of the wire contract specified in the MTHDS protocol spec, not a shape this SDK owns: every field is optional and the model is extension-open so pre-contract artifacts (relayed verbatim, never migrated) still parse. See [`run-usage.md`](./run-usage.md) for the field reference, the cost/null semantics, and the old-artifact rules. +- `TokensUsageRecord` — one client-facing usage record per inference call, carried by `RunResults.tokens_usages` on both paths. A mirror of the runtime's own record, not a shape this SDK owns — inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins that wire contract: every field is optional and the model is extension-open so pre-contract artifacts (relayed verbatim, never migrated) still parse. See [`run-usage.md`](./run-usage.md) for the field reference, the cost/null semantics, and the old-artifact rules. - `RunResultState` — the single-shot result outcome, a union discriminated on `state` (`running` / `completed` / `failed`). - `WaitForResultOptions` / `PollInfo` — poll-loop tuning and progress info. Async-native cancellation is via `asyncio.CancelledError` (cancel the awaiting task), so there is no `signal` field. @@ -115,16 +128,23 @@ These poll GETs go through `_send_or_unreachable`, so a transport failure surfac ## `validate` override (Pipelex-API presentation + sources) -The protocol `validate` is **overridden** (not inherited) to add the two Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): +The protocol `validate` is **overridden** (not inherited) to add the Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): - **Markdown render is always injected.** `validate(...)` adds `"markdown"` to the `render` list (de-duplicated, caller tokens first) so both a valid `PipelexValidationReport` and a produced `PipelexInvalidReport` carry `rendered_markdown`. Unknown render tokens are server-side lenient-ignored. - **`mthds_sources`** is a named parameter (parallel to `mthds_contents`) threaded onto each diagnostic's `source`; sent only when provided. +- **`views`** is the opt-in for the server's structured views, on both `validate` and `validate_files`. `input_form` — named by the `VALIDATION_VIEW_INPUT_FORM` constant — is the only token today. Unlike `render`, the list travels **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so client-side normalization would only hide what the caller asked for. Left at `None` the key is not sent at all, which is what keeps the default response byte-identical for the consumers that discard views (hook pipelines, CI gates, agent loops). The constant is a constant rather than a closed enum on purpose: the request boundary is deliberately open, so a stale token never fails a call. - **`validate_files(files, …)`** takes `MthdsFile(content, uri?)` records. When any file carries a URI, every content gets a parallel source label — the named file's URI, or a deterministic `inline://file-N.mthds` for an unnamed sibling — so the server never sees a length-mismatched `mthds_sources`. The override reuses the inherited base transport seam `_post_validate` (which builds the body — passing `render` / `mthds_sources` through the protocol's `extra` extension passthrough — sends the request, and raises on a no-verdict non-2xx), then parses the raw 200 body into this SDK's own `PipelexValidationResult` via `PipelexValidationResultAdapter`. Body-building and transport stay shared with the base; only the Pipelex presentation/sources concerns and the branded narrowing live here. The validation models (`PipelexValidationResult` = `PipelexValidationReport | PipelexInvalidReport`, with `rendered_markdown`) are **owned by this SDK** (`pipelex_sdk.validation_models`); they narrow `mthds`'s neutral verdict bases, completing the brand boundary (the resolved follow-up #9). The base `MthdsAPIClient.validate()` returns the neutral `ValidationResult` instead. **Checkpoint-5 decision (validate error regime):** the delegation keeps the inherited `httpx.HTTPStatusError` regime on a *no-verdict* non-2xx, where the JS `validate` raises `ApiResponseError`. Kept as-is (deferred parity), because in both SDKs `validate`'s error regime matches the *other* protocol routes of that SDK — JS routes all raise `ApiResponseError`, Python protocol routes all inherit `httpx.HTTPStatusError` (decision #5). Making Python's `validate` alone raise `ApiResponseError` would make it inconsistent with `execute`/`start`/`models`/`version`, which is worse than the JS divergence. The verdict itself (valid/invalid) is always a 200 either way — only the no-verdict failure *presentation* differs. +**What the report carries (0.17+).** A valid `PipelexValidationReport` adds three typed fields. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: dict[str, Any] | None` carries the per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`; it is present only when the request named the `input_form` view, and it is kept **opaque** for the same reason as `bundle_blueprint` / `pipe_io_contracts` / `graph_spec` — the descriptor vocabulary is owned elsewhere and a second copy here would be free to drift. The two lists default empty and `input_form` defaults `None`, so a body from an older runner still parses; the empty default is also what a clean bundle yields, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them: `warnings` and `input_form` derive from a crate that was never assembled. + +`ValidationErrorItem` gains `missing_pipe_code` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None` — a deterministic repair proposal with a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops`: a list discriminated on `kind` over the closed `FixOpKind` vocabulary (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), narrowed with an exhaustive `match op: case SetKeyOp(): …`. The ops are **reader** models here: `extra="allow"`, no `frozen`, none of the runtime's wildcard-refusing validators, because this SDK only reads fixes where the runtime plans them. A `kind` this SDK does not know fails the whole verdict parse, deliberately and consistently with `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break. + +One movement in the same release reaches no typed field here and is worth knowing anyway, because it changes what a consumer reads out of the opaque dicts: `PipeInputContract.optional` became `presence`, and the `fixed` multiplicity now carries an `item_count`. Both live inside `pipe_io_contracts`, which this SDK carries as `dict[str, Any]` on purpose — nobody should discover the new spellings by surprise. + ## Pipelex product surface (hosted management routes) The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models.py` + the client's product methods). Every route rides the same `{base}/v1/*` surface, `Authorization: Bearer`, org-from-JWT contract as the protocol routes, and goes through `_request_product`, which maps a non-2xx `problem+json` to a typed `ApiResponseError` — **consumers branch on `.code`, never the HTTP status**. @@ -132,22 +152,36 @@ The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models The wire models are snake_case Pydantic v2. Response models are extension-open (`extra="allow"`) so a newly-added server field is preserved, not rejected; input models name exactly what each route accepts. `PipelineRun.status` reuses the run-lifecycle `RunStatus`; `OrgRole`, `PipeStatus`, and the onboarding fields are `StrEnum`s. - **User profile** — `get_me()` → `UserProfile` (`GET /v1/me`). -- **Methods catalog** — `list_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. +- **Methods catalog** — `list_methods()` / `iterate_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. + + **The index is paged.** `list_methods(q=…, limit=…, cursor=…)` returns one `MethodPage` — `items: list[MethodSummary]` plus an opaque `next_cursor` you pass straight back, `None` on the last page, and no total by design. A `MethodSummary` is deliberately **not** a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none of them is in the index projection, and putting `mthds` back is exactly what restored the truncation bug paging removed. A method mid-deletion still appears in the list — so a UI can render "Deleting…" — while `get_method` refuses it with a `409`. Query parameters are kept on **presence**, never truthiness: an explicit empty `q` or `cursor` is bad input the API should reject, not something to drop silently into an unfiltered query that reads as working. + + **`iterate_methods(q=…, limit=…)` is the idiom for the whole catalog**, and there is deliberately no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. It **keeps going through an empty page that carries a live cursor**, because `q` is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` means "keep going". It stops on a `None` cursor, and also when the server hands back the cursor it was just sent — checked *before* yielding, so a stuck cursor never double-counts a page. Past a runaway backstop set far beyond any real catalog it **raises** `PagingNotTerminatingError` rather than returning what it has, because a silently truncated list is the bug being fixed. + + **`MethodData.python` is a typed `list[MethodFile]`, converted at the boundary.** On the wire it is one string: the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python. `parse_method_files` / `serialize_method_files` (in `product_models.py`) carry the rules — a blank source or `"[]"` yields `[]`, blank-content entries are dropped in both directions, and an empty list serializes to `""` rather than `"[]"` because `""` is the platform's clear sentinel. `MethodData` applies the parser through a `field_validator(mode="before")` and `MethodWriteInput` the serializer through a `field_serializer`, which is what makes the platform's three-way write contract fall out of `exclude_none=True`: **`None` → key absent → the stored Python is preserved; `[]` → `""` → cleared; a non-empty list → replaced.** `MethodFile` is distinct from `MthdsFile` (the validate input) and `MthdsFileItem` (the build closure) — three shapes for three surfaces. + + **`delete_method` is asynchronous and its return type says so:** the route answers `202` with a `MethodDeletionAccepted` (`method_id`, `deletion_state`, `deletion_job_id`) the moment the platform has claimed the method and terminated its in-flight workflows — the rest of the cascade (runs, events, S3 objects) is enqueued. A returned value means "accepted", never "gone"; completion is the row disappearing from `list_methods`, not any field of that body. It used to be annotated `-> None` with a docstring promising an empty synchronous delete, which is a misleading contract around a destructive operation. The claim is a conditional write, so a double-clicked delete is a `409 conflict` rather than a second cascade over the same runs. - **Organizations** — `list_memberships()` → `MembershipsResponse` (memberships + active-org feature flags); `create_organization(name)` / `rename_organization(org_id, name)` → `Membership`. Organization *switch* is out of scope (a WorkOS session op, not a `/v1` route). - **Billing** — `get_subscription()`, `list_plans()`, `list_invoices()`, `create_checkout(plan)`. `change_plan(plan)` and `get_billing_portal()` surface a **409 `conflict`** (`ApiResponseError.code`) when there is no subscription yet — start one via `create_checkout` first. - **Pipelex API keys** — `list_pipelex_api_keys()`; `create_pipelex_api_key(label)` and `rotate_pipelex_api_key(id)` return the plaintext `api_key` **once**; `revoke_pipelex_api_key(id)`. Creation surfaces a **409 `pipelex_api_key_limit_reached`** when the per-account limit is hit. Rotation sends no body. - **Gateway (LLM inference) key** — `create_gateway_api_key(promo_code)` **always sends a JSON body** (even with `promo_code=None` → `{"promo_code": null}`); the server 422s an empty body. `get_gateway_api_key()` → status (`gateway_api_key` is `None` until provisioned). - **Onboarding** — `submit_onboarding(OnboardingSubmission)` (`POST /v1/onboarding/submit`, empty 2xx body); absent optional fields are dropped. - **Storage** — `resolve_storage_url(uri)` → presigned URL; `upload(UploadInput)` → the stored file handle. The higher-level `upload_file` / `prepare_inputs` preparation surface built on top of `upload` is now available — see [input-preparation.md](./input-preparation.md). -- **Run records** — `list_runs(method_id)` → `list[PipelineRun]` (the catalog-style list, distinct from the lifecycle status/result routes); `update_run(run_id, UpdateRunInput)` (admin/manual status patch, empty 2xx body). +- **Run records** — `list_runs(method_id, …)` / `iterate_runs(method_id, …)` / `get_run_detail(run_id)` (the catalog-style reads, distinct from the lifecycle status/result routes); `update_run(run_id, UpdateRunInput)` (admin/manual status patch, empty 2xx body). + + **This list is paged too.** `list_runs(method_id, created_from=…, created_to=…, limit=…, cursor=…)` returns a `RunPage` with the same opaque-cursor contract as `MethodPage`. `created_from` / `created_to` are **instants** — ISO-8601 with a UTC offset — and inclusive; they are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Worth knowing and not obvious from the route: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — so a `403` here means "flag", not "wrong key". + + **The two iterators stop on different signals, and the difference is in the server.** `iterate_methods` continues through an empty page with a live cursor; `iterate_runs` treats an **empty page as the end**, because the run date bounds are index key conditions and so a run page is never empty-with-a-cursor. Both share the same runaway page ceiling, and both raise `PagingNotTerminatingError` at it: the empty-page stop only catches a server minting fresh cursors while returning *nothing*, so a cursor that cycles across two or more values (`c1 → c2 → c1`) over non-empty pages trips neither that check nor the adjacent-cursor one. The ceiling is the cheap guard against that whole family — tracking every cursor seen would cost unbounded memory for the same protection. + + **`PipelineRun` fields the platform genuinely serves as null are typed nullable.** `method_id` is `None` for an ad-hoc run from an inline bundle, which belongs to no stored method; `pipe_code` is `None` for a run that let the bundle's `main_pipe` decide. `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None` (`message`, `error_type` — the two fields a consumer may rely on out of the runner's verbose report) join them. `RunDetail`, returned only by `get_run_detail`, adds `mthds_contents` and `inputs`: what the run actually executed, and the only record of it, since a method edited since the run no longer describes what happened. Both are left out of the list and the polled status on purpose — their cost scales with page size and poll rate respectively. ## Health probe `health()` → `GET {origin}/health`. The one route served at the **origin**, NOT under the `/v1` prefix — the origin is derived from the base URL (`_origin_of`, exposed as `self.origin_url`), so a base URL of `https://api.pipelex.com/v1/...` still probes `https://api.pipelex.com/health`. It is **out-of-protocol**: the MTHDS Protocol defines no health route, and `/health` is neither a protocol nor a product surface. It rides `_request_json` (the plainer regime), so a non-2xx raises `PipelineRequestError` rather than the product `ApiResponseError` — liveness needs no `code` taxonomy. Transport failures still map to `ApiUnreachableError`. (Checkpoint-5 decision: kept the plainer regime — see "Error regimes" above.) -## Out of scope for v0.1 +## Out of scope -- `/v1/build/*` helpers (the TS clients carry them; recorded as a conscious deferral). +- The remaining `/v1/build/*` authoring helpers — `build_output`, `build_runner`, `concept`, `pipe_spec`. `build_inputs` shipped in 0.5.0 and is no longer deferred. - Organization *switch* (a WorkOS session operation, not a `/v1` route). - A `~/.pipelex/config` file reader (env-only for now, matching the JS SDK). - A synchronous client facade. @@ -158,13 +192,19 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( ## Parity with `@pipelex/sdk` -This SDK is a faithful port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`). The Checkpoint-5 parity audit walked the JS `src/client.ts`, `src/index.ts` (the public barrel), and `docs/architecture.md`, plus a field-by-field sweep of `runs.ts` / `product-models.ts` / `models.ts` against their Python counterparts. Result: **surface-complete, with no silent gaps** — every JS method, model, and error has a Python equivalent or a consciously-recorded exclusion. +This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tracks it closely, but it is **not surface-complete, and this section is where the gaps are named.** The Checkpoint-5 parity audit walked the JS `src/client.ts`, `src/index.ts` (the public barrel), and `docs/architecture.md`, plus a field-by-field sweep of `runs.ts` / `product-models.ts` / `models.ts`; it concluded surface-completeness, and that conclusion went stale as the JS SDK grew. The honest list of what has no Python counterpart today: + +- **Tooling routes** — `lint`, `format`, `resolve`, `codegen`. +- **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec` (`build_inputs` shipped in 0.5.0 and is **not** a gap). +- **Offline helpers** — `run_codegen_check` (the codegen drift check) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure). + +None of them is moved by the releases this SDK last tracked, and each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. -**Methods** — full coverage: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records), and `health`. +**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), `build_inputs`, the input-preparation surface (`upload_file` / `prepare_inputs`), and `health`. -**Models** — full field-for-field match across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. +**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. -**Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. +**Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`, `PagingNotTerminatingError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. **Resolved parity flags (Checkpoint 5):** @@ -172,6 +212,6 @@ This SDK is a faithful port of the TypeScript `@pipelex/sdk` (`PipelexApiClient` - **`health` error regime** (decision #5) — **kept** the plainer `PipelineRequestError` regime; already matches JS, needs no `code` taxonomy. - **`validate` error regime** (Phase-3 flag) — **deferred**; keeps the inherited `httpx.HTTPStatusError` regime for consistency with the other Python protocol routes (see "`validate` override" above). -**Conscious exclusions:** the `/v1/build/*` authoring helpers (decision #8 — a future follow-up if a consumer needs them) and the organization *switch* (a WorkOS session op, not a `/v1` route). +**Conscious exclusions:** the surfaces listed at the top of this section, and the organization *switch* (a WorkOS session op, not a `/v1` route). **Intentional divergences from the JS SDK** (Python house style / clean inheritance): no barrel (`__init__.py` stays empty; import via full paths); inheritance on `MthdsAPIClient` rather than the JS composition-of-types; async-only; `__version__` derived from installed metadata rather than a hand-synced constant. diff --git a/docs/run-usage.md b/docs/run-usage.md index 2c70606..e9781b0 100644 --- a/docs/run-usage.md +++ b/docs/run-usage.md @@ -2,7 +2,7 @@ A completed run reports what its inference calls consumed as a list of `TokensUsageRecord` objects on `RunResults`, one per inference call, in the order the calls completed. This page covers how to read them, what each field means, and the edge cases the model is deliberately shaped around. -The wire shape is not this SDK's invention: it is specified in Pipelex's protocol spec, under "TokensUsage records on run artifacts", and `pipelex_sdk.runs.TokensUsageRecord` is a client-side mirror of it. `@pipelex/sdk` carries the same mirror in TypeScript. The record is a Pipelex runtime concept rather than part of the MTHDS standard — the MTHDS protocol itself says nothing about usage reporting. +The wire shape is not this SDK's invention. Inference accounting is a Pipelex runtime extension — the MTHDS Protocol does not model it, and says nothing about usage reporting — so the hosted API is what pins the contract, and `pipelex_sdk.runs.TokensUsageRecord` is a client-side mirror of the runtime's own record. `@pipelex/sdk` carries the same mirror in TypeScript. ## Reading the records diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 114b48b..11eafb2 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -23,7 +23,7 @@ import re from time import monotonic from typing import TYPE_CHECKING, Any, NamedTuple, NoReturn, cast -from urllib.parse import quote, urlparse +from urllib.parse import quote, urlencode, urlparse import httpx from mthds.protocol.exceptions import PipelineRequestError @@ -37,6 +37,7 @@ ApiResponseError, ApiUnreachableError, MissingMainStuffError, + PagingNotTerminatingError, PipelineExecuteTimeoutError, RunFailedError, RunLifecycleUnavailableError, @@ -55,11 +56,16 @@ Membership, MembershipsResponse, MethodData, + MethodDeletionAccepted, + MethodPage, + MethodSummary, PipelexApiKeyCreated, PipelexApiKeyList, PipelineRun, PlanView, ResolvedStorageUrl, + RunDetail, + RunPage, SubscriptionResponse, UploadedFile, UserProfile, @@ -79,6 +85,8 @@ from pipelex_sdk.validation_models import PipelexValidationResultAdapter, ValidationErrorItem if TYPE_CHECKING: + from collections.abc import AsyncIterator + from mthds.protocol.models import RunResultStart from mthds.protocol.pipe_output import VariableMultiplicity from mthds.protocol.pipeline_inputs import PipelineInputs @@ -105,6 +113,12 @@ DEFAULT_API_BASE_URL = "https://api.pipelex.com" _POLL_REQUEST_TIMEOUT_SECONDS = 30.0 # single status/result/product GETs; the hosted gateway caps responses at ~30s. + +# A runaway backstop for both paged-list iterators, set far beyond any real catalog — never a +# coverage cap. Reaching it means the server kept minting cursors, which is a fault to raise on, +# not a limit to truncate at. It is what bounds a cursor that CYCLES (`c1 -> c2 -> c1`) over +# non-empty pages: neither iterator's adjacent-cursor check sees a non-adjacent repeat. +_MAX_LIST_PAGES: int = 10_000 _DEFAULT_DEGRADED_RETRY_SECONDS = 5 # matches the platform's `_DEGRADE_RETRY_AFTER_SECONDS`. # The hosted gateway caps synchronous requests at ~30s. A blocking-`execute` failure at/after @@ -125,6 +139,18 @@ # produced validation-error verdict carry `rendered_markdown`; callers may add more tokens. _VALIDATE_MARKDOWN_RENDER_FORMAT = "markdown" +# The HOSTED API's own run args — the layer-3 extensions this client names itself, on top of +# the MTHDS Protocol's basic run args. They are named parameters here and travel to the base +# client through its generic `extra` passthrough, which is exactly the layering: the protocol +# client merges them into the body without knowing what they mean. +# +# Reserved on `extra` for the same reason the protocol args are: one argument must not arrive +# by two paths with different validation. The guard is deliberately PER LAYER — it lives here +# and must never be pushed down into `mthds`, because a protocol client talking to another +# vendor's server has no business rejecting that vendor's arguments. This is the layered +# extension policy: a hosted client types its own platform's arguments and guards them per layer. +_HOSTED_RUN_ARGS: frozenset[str] = frozenset({"method_id"}) + class MthdsFile(BaseModel): """One MTHDS file submitted to `validate_files` — content plus an optional provenance URI. @@ -150,7 +176,9 @@ class PipelexAPIClient(MthdsAPIClient): durable polling extension (added in Phase 2). - **product** (`/v1/me`, `/v1/methods`, `/v1/billing/*`, …) — the hosted product surface (added in Phase 3), reached through `_request_product` so callers branch - on the structured `ApiResponseError.code`, not the HTTP status. + on the structured `ApiResponseError.code`, not the HTTP status. The two list routes + are paged: `list_methods` / `list_runs` answer one `{items, next_cursor}` page, and + `iterate_methods` / `iterate_runs` follow the cursors for the whole catalog. Construction is Pipelex-only — it never reads the `mthds` resolver (`MTHDS_API_KEY` / `MTHDS_BASE_URL`, `~/.mthds/config`), whose values are a credential pair for whatever @@ -323,6 +351,8 @@ async def execute( output_multiplicity: VariableMultiplicity | None = None, dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, + *, + method_id: str | None = None, ) -> PipelexExecuteResult: """Execute a method synchronously and wait for its completion — `POST /v1/execute`. @@ -330,7 +360,8 @@ async def execute( resolved `.main_stuff` accessor, so a blocking result reads its output the same way as a durable one (`result.main_stuff`) instead of digging through `pipe_output`. - Identical to the inherited protocol `execute`, except a failure consistent with the + Identical to the inherited protocol `execute`, except for two things. First, `method_id` + — the hosted platform's own run arg (see below). Second, a failure consistent with the hosted gateway's ~30s synchronous ceiling — a gateway `503`/`504`, or a client-side request timeout, after at least ~28s have elapsed — is translated into a clear `PipelineExecuteTimeoutError` pointing at the durable start+poll path, matching the JS @@ -338,9 +369,31 @@ async def execute( (from the inherited `execute`), and every other non-2xx keeps the inherited `httpx.HTTPStatusError` regime (consistent with the other inherited protocol routes). + Args: + pipe_code: The code identifying the pipe to execute. + mthds_contents: List of MTHDS bundle contents to load. + inputs: Inputs passed to the method. + output_name: Name of the output slot to write to. + output_multiplicity: Output multiplicity setting. + dynamic_output_concept_ref: Override for the dynamic output concept ref. + extra: Server-specific extension args this client does not know about, merged into + the request body as top-level properties. Protocol args and this client's own + hosted args (`method_id`) must be passed as named parameters, not through + `extra` (raises `PipelineRequestError`). + method_id: A stored method's hosted catalog id (`mt_…`) — a pure PASS-THROUGH the + platform resolves against the org's catalog; nothing is expanded client-side, + and it is meaningless off-platform (an open-source runner answers a `422` + naming the key). Alone, the platform resolves and runs the stored method's + source. Alongside `mthds_contents`, the inline source is what RUNS (precedence) + and the id is recorded as run-history linkage on the Run row — the index key + `GET /v1/runs?method_id=` queries, so a run started without it is absent from + its method's history permanently. An empty string is treated as absent. + Raises: PipelineExecuteTimeoutError: The blocking request hit the hosted gateway's ~30s synchronous ceiling — use `start_and_wait` (or `start` + `wait_for_result`). + PipelineRequestError: `extra` carries a protocol arg or a hosted arg, or `method_id` + is present and is not a string. RunStillRunningError: The server answered 202 (the protocol's optional async degrade) — the run continues server-side; resume by `pipeline_run_id`. httpx.HTTPStatusError: Any other non-2xx response (the inherited regime). @@ -354,7 +407,7 @@ async def execute( output_name=output_name, output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, - extra=extra, + extra=_merge_hosted_run_extensions(extra, method_id), ) except (httpx.HTTPStatusError, httpx.TimeoutException) as exc: elapsed_seconds = monotonic() - started_at @@ -377,14 +430,23 @@ async def start( output_multiplicity: VariableMultiplicity | None = None, dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, + *, + method_id: str | None = None, ) -> RunResultStart: """Start a method asynchronously — `POST /v1/start` (202: `pipeline_run_id` only). - Identical to the inherited protocol `start`, except a bare-runner missing-route 404 - (no run store) is translated into a clear `RunLifecycleUnavailableError` instead of a - raw `httpx.HTTPStatusError` — matching the JS SDK and letting `start_and_wait` self-heal - to the blocking-execute fallback. The platform's structured 404s (run not found) keep - their normal `httpx.HTTPStatusError` behavior. + Identical to the inherited protocol `start`, except for `method_id` — the hosted + platform's own run arg, documented on `execute` and carrying the same semantics here — + and that a bare-runner missing-route 404 (no run store) is translated into a clear + `RunLifecycleUnavailableError` instead of a raw `httpx.HTTPStatusError`, matching the JS + SDK and letting `start_and_wait` self-heal to the blocking-execute fallback. The + platform's structured 404s (run not found) keep their normal `httpx.HTTPStatusError` + behavior. + + Raises: + PipelineRequestError: `extra` carries a protocol arg or a hosted arg, or `method_id` + is present and is not a string. + RunLifecycleUnavailableError: The configured server has no run store. """ try: return await super().start( @@ -394,7 +456,7 @@ async def start( output_name=output_name, output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, - extra=extra, + extra=_merge_hosted_run_extensions(extra, method_id), ) except httpx.HTTPStatusError as exc: self._raise_if_lifecycle_unavailable(exc.response, str(exc.request.url)) @@ -407,6 +469,7 @@ async def validate( # type: ignore[override] allow_signatures: bool = False, mthds_sources: list[str] | None = None, render: list[str] | None = None, + views: list[str] | None = None, ) -> PipelexValidationResult: """Parse, validate, and dry-run an MTHDS bundle — `POST /v1/validate`. @@ -416,9 +479,10 @@ async def validate( # type: ignore[override] means no verdict could be produced (request shape, auth, server fault) and surfaces as `httpx.HTTPStatusError` (the inherited protocol error regime). - This override differs from the inherited protocol `validate` in two Pipelex-API ways: + This override differs from the inherited protocol `validate` in these Pipelex-API ways: it always injects `render: ["markdown"]` (so both valid and invalid verdicts carry - `rendered_markdown`), and it accepts `mthds_sources` as a named parameter. + `rendered_markdown`), it accepts `mthds_sources` as a named parameter, and it carries + the `views` opt-in for the server's structured views. Args: mthds_contents: MTHDS contents to load (always a list, even for one file). @@ -428,15 +492,24 @@ async def validate( # type: ignore[override] `source: null`). The server 422s a length mismatch. render: Optional Pipelex-API presentation hints; `"markdown"` is always added. Unknown tokens are server-side lenient-ignored (never a 422). + views: Optional opt-in for the server's structured views. `input_form` — named by + `VALIDATION_VIEW_INPUT_FORM` — is the only token today; unknown tokens are + server-side lenient-ignored (never a 422). Unlike `render`, the list is sent + **verbatim**: nothing is injected and nothing is de-duplicated, and an explicit + `[]` is sent as `[]`. Left at `None` the key is not sent at all, which is what + keeps the default response byte-identical for consumers that discard views. Returns: The 200-diagnostic union: `PipelexValidationReport` (`is_valid: true`) or `PipelexInvalidReport` (`is_valid: false`, with `validation_errors`), each - carrying `rendered_markdown`. + carrying `rendered_markdown`. A valid report also carries `warnings` and + `liftable_pipes`, plus `input_form` when `views` asked for it. """ extra: dict[str, Any] = {"render": _with_validate_markdown_render(render)} if mthds_sources is not None: extra["mthds_sources"] = mthds_sources + if views is not None: + extra["views"] = views # Reuse the inherited transport seam (`_post_validate`) for body-building + the wire call, # then parse the 200-diagnostic body into this SDK's Pipelex-branded narrowing. The base's # own `validate` parses the same body into the neutral `mthds` `ValidationResult`. @@ -448,6 +521,7 @@ async def validate_files( files: list[MthdsFile], allow_signatures: bool = False, render: list[str] | None = None, + views: list[str] | None = None, ) -> PipelexValidationResult: """Validate paired MTHDS files while preserving URI attribution for diagnostics. @@ -455,6 +529,13 @@ async def validate_files( a URI, every content gets a parallel source label (a deterministic `inline://` label for the ones without), so the server never sees a length-mismatched `mthds_sources`. + Args: + files: The MTHDS files to validate, each content plus an optional provenance URI. + allow_signatures: Tolerate unimplemented pipe signatures (strict by default). + render: Optional Pipelex-API presentation hints, threaded to `validate`. + views: Optional structured-view opt-in, threaded to `validate` unchanged — see + `validate` for the semantics. + Raises: PipelineRequestError: If `files` is empty. """ @@ -472,7 +553,7 @@ async def validate_files( else: mthds_sources = None - return await self.validate(mthds_contents, allow_signatures, mthds_sources, render) + return await self.validate(mthds_contents, allow_signatures, mthds_sources, render, views) # ── Hosted extension: durable run lifecycle (NOT part of the protocol) ── # @@ -615,6 +696,8 @@ async def start_and_wait( dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, wait_options: WaitForResultOptions | None = None, + *, + method_id: str | None = None, ) -> RunResults: """Start a run and wait for its result — the whole lifecycle in one call, self-healing across hosted and bare runners. @@ -629,6 +712,10 @@ async def start_and_wait( from `start`, BEFORE any run is created, so the blocking fallback cannot double-run; the negative is cached so later calls skip the durable attempt. + `method_id` — the hosted platform's own run arg, documented on `execute` — is forwarded + on BOTH paths. Dropping it on the blocking fallback would turn a server-side 422 that + names the key into a silently different run. + Raises: RunFailedError: If the run reaches a terminal status other than COMPLETED. RunTimeoutError: If the poll budget elapses (the run keeps executing — resume by id). @@ -643,6 +730,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) except RunLifecycleUnavailableError: self._lifecycle_available = False @@ -654,6 +742,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) return await self.wait_for_result(started.pipeline_run_id, options=wait_options) @@ -665,6 +754,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) async def _execute_blocking( @@ -677,12 +767,15 @@ async def _execute_blocking( output_multiplicity: VariableMultiplicity | None, dynamic_output_concept_ref: str | None, extra: dict[str, Any] | None, + method_id: str | None = None, ) -> RunResults: """Blocking `POST /v1/execute` adapted onto `RunResults` — the bare-runner path. - Forwards every protocol field PLUS the `extra` extension passthrough: an extension-only - call (`{extra}` with no pipe_code/bundle) or a vendor selector riding `extra` must survive - this path, not just the durable one. + Forwards every protocol field PLUS both extension surfaces: the hosted `method_id` and + the generic `extra` passthrough. An extension-only call (`{extra}` with no pipe_code) or + a vendor selector riding `extra` must survive this path, not just the durable one — and + a hosted `method_id` must reach the server here too, so a runner that cannot resolve it + says so instead of the client silently dropping it. """ result = await self.execute( pipe_code=pipe_code, @@ -692,6 +785,7 @@ async def _execute_blocking( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) return _map_run_result_to_run_results(result) @@ -706,10 +800,54 @@ async def get_me(self) -> UserProfile: """The authenticated user's profile — `GET /v1/me`.""" return UserProfile.model_validate(await self._request_product("GET", "me")) - async def list_methods(self) -> list[MethodData]: - """List the caller's saved methods — `GET /v1/methods`.""" - result = await self._request_product("GET", "methods") - return [MethodData.model_validate(item) for item in result] + async def list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage: + """List one page of the caller's saved methods — `GET /v1/methods`. + + Args: + q: Server-side case-insensitive substring match over name and description, + applied across the whole catalog rather than within the returned page. + limit: Page size. The API supplies the default and caps the maximum. + cursor: The `next_cursor` of the previous page, passed back opaquely. + + Returns: + A `MethodPage` of `MethodSummary` rows, ordered by creation, newest first. + `next_cursor` is `None` on the last page. For the whole catalog, prefer + `iterate_methods`, which follows the cursors and cannot truncate. + """ + query = _product_query({"q": q, "limit": limit, "cursor": cursor}) + return MethodPage.model_validate(await self._request_product("GET", f"methods{query}")) + + async def iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]: + """Yield every saved method, following the cursors — `GET /v1/methods`. + + There is deliberately no `list_all_methods()`: an all-at-once helper needs a cap, and + a cap is the silent truncation paging was introduced to remove. + + The loop keeps going **through an empty page that carries a live cursor**, because `q` + is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` + means "keep going", not "done". It stops when `next_cursor` is `None`, and also when the + server hands back the cursor it was just sent — checked *before* yielding, so a stuck + cursor never double-counts a page. + + Raises: + PagingNotTerminatingError: If the server never stops handing out cursors. + """ + cursor: str | None = None + pages_seen = 0 + while True: + page = await self.list_methods(q=q, limit=limit, cursor=cursor) + if cursor is not None and page.next_cursor == cursor: + # The server did not advance. Stop before yielding, or this page is counted twice. + return + for method_summary in page.items: + yield method_summary + if page.next_cursor is None: + return + pages_seen += 1 + if pages_seen >= _MAX_LIST_PAGES: + msg = f"Method paging did not terminate after {_MAX_LIST_PAGES} pages; this is a server-side fault, not a coverage limit." + raise PagingNotTerminatingError(msg, _MAX_LIST_PAGES) + cursor = page.next_cursor async def get_method(self, method_id: str) -> MethodData: """Fetch one method by id — `GET /v1/methods/{id}`.""" @@ -725,9 +863,29 @@ async def update_method(self, method_id: str, write_input: MethodWriteInput) -> body = write_input.model_dump(mode="json", exclude_none=True) return MethodData.model_validate(await self._request_product("PUT", f"methods/{quote(method_id, safe='')}", body=body)) - async def delete_method(self, method_id: str) -> None: - """Delete a method — `DELETE /v1/methods/{id}` (empty body).""" - await self._request_product("DELETE", f"methods/{quote(method_id, safe='')}") + async def delete_method(self, method_id: str) -> MethodDeletionAccepted: + """Erase a method and everything it produced — `DELETE /v1/methods/{id}`. + + **Asynchronous, and the return value says so.** The platform answers `202` the moment it + has claimed the method and terminated its in-flight workflows; the rest of the cascade + (runs, events, S3 objects) is enqueued. So a returned `MethodDeletionAccepted` means + "accepted", never "gone" — completion is the method's row disappearing from + `list_methods`, not any field of the acceptance body. Until then the row stays listed + with a `deletion_state`, which is what lets a UI render it as "Deleting…", while + `get_method` refuses it with a `409`. + + A double-clicked delete is safe: the claim is a conditional write, so the second call is + an `ApiResponseError` (`409 conflict`) rather than a second cascade over the same runs. + An unknown or foreign-org id is a `404`. + + Args: + method_id: The method to erase. + + Returns: + The platform's acceptance — `method_id`, the `deletion_state` the cascade started + in, and the `deletion_job_id` a caller can log or correlate. + """ + return MethodDeletionAccepted.model_validate(await self._request_product("DELETE", f"methods/{quote(method_id, safe='')}")) async def list_memberships(self) -> MembershipsResponse: """The caller's org memberships + active-org feature flags — `GET /v1/organizations/memberships`.""" @@ -865,10 +1023,89 @@ async def prepare_inputs( """ return await _prepare_inputs_impl(self, files=files, pipe_ref=pipe_ref, inputs=inputs) - async def list_runs(self, method_id: str) -> list[PipelineRun]: - """List a method's runs — `GET /v1/runs?method_id={methodId}`.""" - result = await self._request_product("GET", f"{_RUNS}?method_id={quote(method_id, safe='')}") - return [PipelineRun.model_validate(item) for item in result] + async def list_runs( + self, + method_id: str, + *, + created_from: str | None = None, + created_to: str | None = None, + limit: int | None = None, + cursor: str | None = None, + ) -> RunPage: + """List one page of a method's runs — `GET /v1/runs?method_id={methodId}`. + + Args: + method_id: The method whose runs to list. + created_from: Inclusive lower bound on creation, an **instant**: ISO-8601 with a + UTC offset. These are index key conditions rather than filters, so a bare date + or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. + created_to: Inclusive upper bound, same instant-only rule. + limit: Page size. The API supplies the default and caps the maximum. + cursor: The `next_cursor` of the previous page, passed back opaquely. + + Returns: + A `RunPage` of `PipelineRun` rows. For the whole history, prefer `iterate_runs`. + + Raises: + ApiResponseError: On any non-2xx. Note that every `/v1/runs*` product route sits + behind the platform's surface-access gate, which for API-key auth demands the + `ff_api_keys` feature flag and fails closed — so a `403` here means "flag", not + "wrong key". + """ + query = _product_query({"method_id": method_id, "created_from": created_from, "created_to": created_to, "limit": limit, "cursor": cursor}) + return RunPage.model_validate(await self._request_product("GET", f"{_RUNS}{query}")) + + async def iterate_runs( + self, + method_id: str, + *, + created_from: str | None = None, + created_to: str | None = None, + limit: int | None = None, + ) -> AsyncIterator[PipelineRun]: + """Yield every run of a method, following the cursors — `GET /v1/runs`. + + The same loop as `iterate_methods` with one deliberate difference: an **empty page ends + it**. The date bounds are index key conditions rather than a post-read filter, so a run + page is never empty-with-a-cursor. The difference is in the server, not in the client. + + The page ceiling applies here too. The empty-page stop only catches a server minting + fresh cursors while returning *nothing*; a cursor that cycles across two or more values + while every page is non-empty (`c1 → c2 → c1`) trips neither that check nor the + adjacent-cursor one, and would loop forever re-yielding the same runs. The ceiling is + the cheap guard against the whole family — tracking every cursor seen would cost + unbounded memory for the same protection. + + Raises: + PagingNotTerminatingError: If the server never stops handing out cursors. + """ + cursor: str | None = None + pages_seen = 0 + while True: + page = await self.list_runs(method_id, created_from=created_from, created_to=created_to, limit=limit, cursor=cursor) + if cursor is not None and page.next_cursor == cursor: + # The server did not advance. Stop before yielding, or this page is counted twice. + return + if not page.items: + return + for pipeline_run in page.items: + yield pipeline_run + if page.next_cursor is None: + return + pages_seen += 1 + if pages_seen >= _MAX_LIST_PAGES: + msg = f"Run paging did not terminate after {_MAX_LIST_PAGES} pages; this is a server-side fault, not a coverage limit." + raise PagingNotTerminatingError(msg, _MAX_LIST_PAGES) + cursor = page.next_cursor + + async def get_run_detail(self, run_id: str) -> RunDetail: + """Fetch one run record with what it executed — `GET /v1/runs/{id}`. + + Distinct from the two lifecycle reads: `get_run_status` polls `/status`, `get_run_result` + fetches `/results`. This is the catalog-style record, and the only read that carries + `mthds_contents` and `inputs`. + """ + return RunDetail.model_validate(await self._request_product("GET", f"{_RUNS}/{quote(run_id, safe='')}")) async def update_run(self, run_id: str, update_input: UpdateRunInput) -> None: """Patch a run's status (admin/manual) — `PUT /v1/runs/{id}` (empty body).""" @@ -894,6 +1131,52 @@ async def health(self) -> dict[str, Any]: _KNOWN_RUN_STATUS_NAMES: frozenset[str] = frozenset(RunStatus.__members__) +def _merge_hosted_run_extensions(extra: dict[str, Any] | None, method_id: object) -> dict[str, Any] | None: + """Fold the hosted API's own run args into the generic `extra` passthrough handed to the base client. + + This is the seam between layer 3 and layer 2: `method_id` is a named parameter on this + client (it is the hosted platform's argument, so this client must type it), and it reaches + the wire as a top-level body property through the protocol client's extension mechanism — + which merges it without knowing what it means. See `_HOSTED_RUN_ARGS`. + + A **non-string** `method_id` is refused here rather than dropped or forwarded. A published + client validates its request-option types at its own boundary, so that one wrong value gets + one answer: a bare truthiness check would silently drop the falsy wrong types (`0`, `[]`) + and forward the truthy ones (`123`, `["mt_1"]`) to a server `422` — a different partition of + wrong values than the JS client makes for the same argument on the same wire. + + An absent or empty `method_id` still contributes nothing: `method_id=""` selects no method + and links no run, so it is not sent and does not satisfy the base client's "something to + run" precondition, and neither does `None`. `None` is returned for an empty result, leaving + the base's own handling of an absent `extra` untouched. + + Args: + extra: Server-specific extension args from the caller, or None. + method_id: The hosted catalog id, or None. Typed `object` rather than `str | None` + deliberately — this helper *is* the runtime boundary, and the callers it guards + against are the untyped ones a type checker never sees. + + Returns: + The merged extension mapping to hand to the base client, or None if there is nothing. + + Raises: + PipelineRequestError: If `extra` carries a hosted arg this client names itself, or if + `method_id` is present and is not a string. + """ + extensions: dict[str, Any] = dict(extra or {}) + hosted_overlap = extensions.keys() & _HOSTED_RUN_ARGS + if hosted_overlap: + msg = f"extra carries hosted args {sorted(hosted_overlap)} — pass them as named parameters instead." + raise PipelineRequestError(msg) + if method_id is not None: + if not isinstance(method_id, str): + msg = f"method_id must be a string, received {type(method_id).__name__}." + raise PipelineRequestError(msg) + if method_id: + extensions["method_id"] = method_id + return extensions or None + + def _with_validate_markdown_render(render: list[str] | None) -> list[str]: """Ensure `"markdown"` rides the `/validate` render list, preserving order and de-duplicating. @@ -909,6 +1192,19 @@ def _timeout_message(run_id: str, timeout_seconds: float) -> str: return f"Run {run_id} did not reach a terminal state within {timeout_seconds}s; it is still executing server-side and can be resumed by id." +def _product_query(params: dict[str, str | int | None]) -> str: + """Build the query string of a product list route, keeping entries on **presence**. + + Presence (`is not None`), never truthiness: an explicit empty `q` or cursor is bad input + the API should reject, not something to silently drop into an unfiltered query that reads + as working. Returns `""` for no parameters, otherwise a leading `?`. + """ + kept = {key: value for key, value in params.items() if value is not None} + if not kept: + return "" + return "?" + urlencode(kept) + + def _is_gateway_timeout(exc: httpx.HTTPStatusError | httpx.TimeoutException, elapsed_seconds: float) -> bool: """Whether a failed blocking `execute` is the hosted gateway's ~30s synchronous cut-off. diff --git a/pipelex_sdk/errors.py b/pipelex_sdk/errors.py index f7afb63..375f31a 100644 --- a/pipelex_sdk/errors.py +++ b/pipelex_sdk/errors.py @@ -12,6 +12,8 @@ a consumer branches on (decoupled from the HTTP status). - `PipelineExecuteTimeoutError` — a blocking `execute()` killed by the hosted gateway's ~30s synchronous-request ceiling; points the caller at the durable start+poll path. +- `PagingNotTerminatingError` — a paged-list iterator hit its runaway backstop, meaning + the server never stopped handing out cursors. The run-lifecycle errors (`RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`) are owned here (ported from `mthds-python` in @@ -28,7 +30,7 @@ # Explicit re-export (PEP 484 `as` self-alias): the protocol 202-degrade error stays owned by # `mthds`, surfaced here so consumers have a single import home for the run/lifecycle errors. -from mthds.runners.api.exceptions import RunStillRunningError as RunStillRunningError # noqa: PLC0414 +from mthds.runners.api.exceptions import RunStillRunningError as RunStillRunningError # ruff: ignore[useless-import-alias] if TYPE_CHECKING: from pipelex_sdk.runs import RunStatus @@ -163,6 +165,20 @@ def __init__(self, message: str, api_url: str) -> None: self.api_url = api_url +class PagingNotTerminatingError(PipelineRequestError): + """Raised when a paged-list iterator refuses to keep following cursors. + + The ceiling sits far beyond any real catalog, so reaching it is a server-side fault — + an endpoint minting a fresh cursor forever — not a coverage limit the caller can raise. + Raising beats returning, because a silently truncated list is exactly the bug paging + was introduced to remove. + """ + + def __init__(self, message: str, page_limit: int) -> None: + super().__init__(message) + self.page_limit = page_limit + + class InputPreparationError(PipelineRequestError): """Base class for every failure raised by input preparation (`upload_file` / `prepare_inputs`). diff --git a/pipelex_sdk/product_models.py b/pipelex_sdk/product_models.py index 05297f4..287a5ba 100644 --- a/pipelex_sdk/product_models.py +++ b/pipelex_sdk/product_models.py @@ -17,11 +17,13 @@ from __future__ import annotations +import json from enum import StrEnum from typing import Any -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError, field_serializer, field_validator +from pipelex_sdk._pydantic_utils import empty_list_factory_of from pipelex_sdk.runs import RunStatus # ── User profile (`/v1/me`) ───────────────────────────────────────────── @@ -42,6 +44,88 @@ class UserProfile(BaseModel): # ── Methods catalog (`/v1/methods`) ────────────────────────────────────── +class MethodDeletionState(StrEnum): + """Where a method is in the erasure cascade; absent on a normal method.""" + + PENDING = "pending" + IN_PROGRESS = "in_progress" + FAILED = "failed" + + +class MethodFile(BaseModel): + """One named source file of a stored method — the at-rest catalog form. + + This is the shape the hosted platform persists for a method's custom PipeFunc Python: + a JSON `[{name, content}]` array in one wire string. It is deliberately distinct from + two neighbours that look similar and are not: `MthdsFile` (`client.py`) is the *validate* + input, content plus an optional provenance URI; `MthdsFileItem` (`build_models.py`) is + the *build* closure entry. Three shapes for three surfaces — do not merge them. + """ + + model_config = ConfigDict(extra="allow") + + #: Bundle-relative path, e.g. `"funcs/price.py"`. + name: str + #: The file's UTF-8 text content. + content: str + + +_METHOD_FILES_ADAPTER: TypeAdapter[list[MethodFile]] = TypeAdapter(list[MethodFile]) +"""Built once at import — TypeAdapter construction is expensive.""" + +_METHOD_FILES_SHAPE = "a JSON array of {name, content} entries" + + +def _is_blank(content: str) -> bool: + """A file carries no source when its content is empty or whitespace-only.""" + return not content.strip() + + +def parse_method_files(source: str | None) -> list[MethodFile]: + """Parse the catalog wire string into method files. + + A blank source (`None`, `""`, whitespace) and an empty JSON array both yield `[]`. + A JSON `[{name, content}]` array yields those files, with blank-content entries + dropped so the round-trip with `serialize_method_files` is stable. + + Raises: + ValueError: For anything else — a non-array JSON value, an entry that is not a + `{name: str, content: str}` object, or unparseable text. Reached through + `MethodData`'s validator, this surfaces as a `pydantic.ValidationError`, the + same way any other malformed response body fails here. + """ + if source is None or _is_blank(source): + return [] + + try: + parsed = json.loads(source) + except json.JSONDecodeError as exc: + msg = f"Method file source is not valid JSON; expected {_METHOD_FILES_SHAPE}." + raise ValueError(msg) from exc + + try: + files = _METHOD_FILES_ADAPTER.validate_python(parsed) + except ValidationError as exc: + msg = f"Method file source must be {_METHOD_FILES_SHAPE}." + raise ValueError(msg) from exc + + return [file for file in files if not _is_blank(file.content)] + + +def serialize_method_files(files: list[MethodFile]) -> str: + """Serialize method files to the catalog wire string. + + Blank-content entries are dropped (a zero-source file is not persisted), and an empty + result serializes to `""` — the platform's "no source" / "clear the field" sentinel — + never to the literal `"[]"`. Only `name` and `content` cross the wire; anything an + extension-open `MethodFile` picked up on the way in is not written back. + """ + kept = [file for file in files if not _is_blank(file.content)] + if not kept: + return "" + return json.dumps([{"name": file.name, "content": file.content} for file in kept]) + + class MethodData(BaseModel): """One saved method record.""" @@ -51,12 +135,35 @@ class MethodData(BaseModel): name: str #: The `.mthds` bundle source. mthds: str + org_id: str + created_by_user_id: str + description: str | None = None + deletion_state: MethodDeletionState | None = None input_data: dict[str, Any] | None = None #: Legacy persisted output spec; optional. pipe_output: dict[str, Any] | None = None + python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile)) + """The method's custom PipeFunc source files. + + On the wire this is one string — the JSON text of a `[{name, content}]` array, or `""` + for a method with no custom Python. The validator below converts at the boundary so + callers never see that string.""" + created_at: str updated_at: str + @field_validator("python", mode="before") + @classmethod + def _parse_python_files(cls, value: object) -> object: + """Convert the catalog wire string into `MethodFile` entries. + + A `str` or `None` is the wire form and goes through `parse_method_files`; anything + else (a list, from programmatic construction) passes through to normal validation. + """ + if value is None or isinstance(value, str): + return parse_method_files(value) + return value + class MethodWriteInput(BaseModel): """The create/update payload — a rename is a `PUT` with a changed `name`.""" @@ -64,6 +171,67 @@ class MethodWriteInput(BaseModel): name: str mthds: str input_data: dict[str, Any] | None = None + python: list[MethodFile] | None = None + """The custom PipeFunc source files to write, with a deliberate three-way contract. + + The write body is dumped with `exclude_none=True`, so `None` (the default) leaves the key + out entirely and a `PUT` **preserves** the stored Python. An empty list serializes to `""`, + the platform's clear sentinel, which **erases** it. A non-empty list **replaces** it.""" + + @field_serializer("python") + def _serialize_python_files(self, value: list[MethodFile] | None) -> str | None: + """Render the file list as the catalog wire string, leaving `None` for `exclude_none`.""" + if value is None: + return None + return serialize_method_files(value) + + +class MethodSummary(BaseModel): + """One row of the paged method index — `GET /v1/methods`. + + Deliberately **not** a `MethodData`: no `mthds`, no `python`, no `updated_at`, because + none of them is in the index projection. Putting `mthds` back is exactly what restored + the truncation bug paging was introduced to fix. A method mid-deletion still appears + here — so a UI can render "Deleting…" — while `get_method` refuses it with a `409`. + """ + + model_config = ConfigDict(extra="allow") + + method_id: str + name: str + description: str | None = None + created_at: str + deletion_state: MethodDeletionState | None = None + + +class MethodPage(BaseModel): + """One page of the method index — `{items, next_cursor}`. + + The cursor is opaque: pass it straight back as `cursor` to get the next page, and treat + a `None` as the last page. There is no total by design — counting a catalog costs a full + scan, and no caller needs one. + """ + + model_config = ConfigDict(extra="allow") + + items: list[MethodSummary] + next_cursor: str | None = None + + +class MethodDeletionAccepted(BaseModel): + """The `202` acceptance of `DELETE /v1/methods/{id}`. + + Returned the moment the erasure is CLAIMED and handed off, not when it completes. + Nothing in this body means "done": completion is the method's row disappearing from + `list_methods`. What the body buys a caller is a claim it can log and correlate + (`deletion_job_id`) plus the state the cascade started in. + """ + + model_config = ConfigDict(extra="allow") + + method_id: str + deletion_state: MethodDeletionState + deletion_job_id: str # ── Organizations (`/v1/organizations`) ────────────────────────────────── @@ -320,10 +488,11 @@ class UploadedFile(BaseModel): filename: str -# ── Runs list / update (`/v1/runs`) ────────────────────────────────────── +# ── Run records (`/v1/runs`) ───────────────────────────────────────────── # # The run-lifecycle status/results/start routes already live on the client -# (`runs.py`); these are the remaining catalog-style list + admin-update routes. +# (`runs.py`); these are the remaining catalog-style paged list, the single-run +# detail read, and the admin-update route. class PipeStatus(StrEnum): @@ -336,22 +505,69 @@ class PipeStatus(StrEnum): SKIPPED = "skipped" +class RunErrorReport(BaseModel): + """A failed run's error, narrowed to the two fields a consumer may rely on. + + The runner's own report is considerably more verbose; only these two are contractual. + """ + + model_config = ConfigDict(extra="allow") + + message: str | None = None + error_type: str | None = None + + class PipelineRun(BaseModel): """One run record in a method's run list — `GET /v1/runs?method_id=…`.""" model_config = ConfigDict(extra="allow") pipeline_run_id: str - method_id: str - pipe_code: str + method_id: str | None = None + """The stored method this run is linked to, when there is one. An ad-hoc run from an + inline bundle belongs to no stored method, so the platform serves this as null.""" + + pipe_code: str | None = None + """The pipe that ran, when it was named. A run that let the bundle's `main_pipe` decide + has none to report, so the platform serves this as null.""" + + org_id: str | None = None + created_by_user_id: str | None = None workflow_id: str | None = None status: RunStatus result_url: str | None = None + error: RunErrorReport | None = None pipe_statuses: dict[str, PipeStatus] | None = None created_at: str finished_at: str | None = None +class RunDetail(PipelineRun): + """One run read on its own — `GET /v1/runs/{id}`. + + Adds the two heavy fields the list and the polled status deliberately leave out (their + cost scales with page size and poll rate respectively). `mthds_contents` is what the run + actually executed, and the only record of it: a method edited since the run no longer + describes what happened. + """ + + mthds_contents: list[str] | None = None + inputs: dict[str, Any] | None = None + + +class RunPage(BaseModel): + """One page of a method's run list — `{items, next_cursor}`. + + Same opaque-cursor contract as `MethodPage`: pass `next_cursor` straight back, and a + `None` means the last page. + """ + + model_config = ConfigDict(extra="allow") + + items: list[PipelineRun] + next_cursor: str | None = None + + class UpdateRunInput(BaseModel): """The admin/manual run-status patch — `status` is a free string here.""" diff --git a/pipelex_sdk/runs.py b/pipelex_sdk/runs.py index 9aa6cf6..ffba5f3 100644 --- a/pipelex_sdk/runs.py +++ b/pipelex_sdk/runs.py @@ -20,8 +20,9 @@ than redefine. `RunResults.pipe_output` is typed with the protocol's own `DictPipeOutputAbstract` wire model from `mthds` — a shared wire contract the `pipelex` runtime also builds on, not a lifecycle concept. `TokensUsageRecord` -mirrors a runtime wire contract specified in the MTHDS protocol spec; this SDK -follows that shape, it does not define it. +mirrors the runtime's own record: inference accounting is a Pipelex runtime +extension the MTHDS Protocol does not model, so the hosted API is what pins that +wire contract; this SDK follows the shape, it does not define it. Wire contract mirrors `pipelex-platform`: POST /v1/start -> RunResultStart (start, 202) @@ -125,10 +126,11 @@ class RunRead(RunPublic): class TokensUsageRecord(BaseModel): """One inference call's token usage — the client-facing wire record. - Mirrors the runtime's `TokensUsageRecord`, specified in - `docs/specs/pipelex-mthds-protocol.md#tokensusage-records-on-run-artifacts`. The same - shape rides both surfaces: the durable `tokens_usages.json` artifact that the hosted - results route relays, and the blocking execute response's `pipe_output.tokens_usages`. + Mirrors the runtime's `TokensUsageRecord`. Inference accounting is a Pipelex runtime + extension — the MTHDS Protocol does not model it — so the hosted API is what pins this + wire contract. The same shape rides both surfaces: the durable `tokens_usages.json` + artifact that the hosted results route relays, and the blocking execute response's + `pipe_output.tokens_usages`. Every field is optional and the model is extension-open **on purpose**. A record the current runtime emits always carries the full key set (a field with no value is an diff --git a/pipelex_sdk/validation_models.py b/pipelex_sdk/validation_models.py index bfc2da7..63de72b 100644 --- a/pipelex_sdk/validation_models.py +++ b/pipelex_sdk/validation_models.py @@ -10,8 +10,10 @@ `pipelex-sdk`, not in the brand-neutral `mthds` package — mirroring `pipelex-sdk-js/src/models.ts` and the documented brand boundary (`docs/architecture.md` → "Brand boundary"). The report/union types carry the `Pipelex` prefix; the supporting types (`DryRunStatus`, `ValidatedPipeEntry`, -`ValidationErrorCategory`, `ValidationErrorItem`) stay neutrally named — branding the envelope, -not the field names inside it. +`ValidationErrorCategory`, `ValidationErrorItem`, `LiftablePipeEntry`, `SuggestedFix`, the +`FixOp` variants, `FixOpKind`, `FixSafety`) stay neutrally named — branding the envelope, +not the field names inside it. Fixes and lints are language-level concepts, and the runtime +names them brand-neutrally too. `PipelexAPIClient.validate()` returns this `PipelexValidationResult` (parsed via `PipelexValidationResultAdapter`); the protocol base `MthdsAPIClient.validate()` returns the @@ -21,13 +23,21 @@ from __future__ import annotations from enum import StrEnum -from typing import Annotated, Any, TypeAlias +from typing import Annotated, Any, Final, Literal, TypeAlias from mthds.protocol.models import InvalidValidationReport, ValidationDiagnostic, ValidationReport from pydantic import BaseModel, ConfigDict, Field, TypeAdapter from pipelex_sdk._pydantic_utils import empty_list_factory_of +VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form" +"""The one `views` token the server supports today — asks for `PipelexValidationReport.input_form`. + +Deliberately a constant rather than a closed enum: the request boundary is open, the server +resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so +a stale token must never fail a call. +""" + class DryRunStatus(StrEnum): """Per-pipe dry-run sweep outcome on `ValidatedPipeEntry.status`.""" @@ -40,8 +50,8 @@ class DryRunStatus(StrEnum): class ValidationErrorCategory(StrEnum): """The closed `validation_errors[].category` vocabulary (locked). - Mirrors the single source of truth in the conformance suite - (`conformance/conformance/validation_contract.py`); keep in sync with it. + This is the locked category vocabulary shared with the conformance corpus; keep the two + in sync — a category the corpus knows and this set does not fails the whole verdict parse. """ BLUEPRINT_VALIDATION = "blueprint_validation" @@ -50,6 +60,174 @@ class ValidationErrorCategory(StrEnum): DRY_RUN = "dry_run" +class FixSafety(StrEnum): + """Whether a suggested fix is safe to auto-apply (SAFE) or needs an explicit opt-in (UNSAFE).""" + + SAFE = "safe" + UNSAFE = "unsafe" + + @property + def is_safe(self) -> bool: + match self: + case FixSafety.SAFE: + return True + case FixSafety.UNSAFE: + return False + + +class FixOpKind(StrEnum): + """The closed vocabulary of semantic patch operations a `SuggestedFix` is composed of. + + A kind this SDK does not know fails the parse of the whole verdict, which is deliberate and + consistent with `ValidationErrorCategory`: the vocabulary is closed upstream, a new kind is a + runtime release this SDK mirrors, and a loud failure beats a silently unnarrowable op. + """ + + SET_KEY = "set_key" + ENSURE_TABLE = "ensure_table" + DELETE_KEY = "delete_key" + DELETE_TABLE = "delete_table" + RENAME_TABLE_KEY = "rename_table_key" + MOVE_KEY = "move_key" + REMAP_VALUE = "remap_value" + + +TomlScalar: TypeAlias = str | int | float | bool +"""What a `set_key` op can write as a bare TOML value.""" + +TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar] +"""A `set_key` value: a scalar, or a flat scalar mapping written as an inline table. + +Deeper nesting is not modelled because the server does not emit it — the fixes that create a +whole table at once create a flat one. +""" + + +class _FixOpBase(BaseModel): + """Fields every fix op shares: the table it acts in. + + These are **reader** models. The runtime declares its own copies `frozen`, `extra="forbid"`, + with validators that refuse the wildcard segment, because it *plans* fixes; this SDK only + reads them, so the ops follow the response-model convention (`extra="allow"`) and carry none + of those validators — a new server-side member on an op must not break parsing here. + + The two runtime invariants a type cannot carry, recorded so a consumer knows them: `*` is the + wildcard path segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` + and `delete_table` address the table itself rather than its parent, so their `table_path` is + never empty (that one *is* expressed below, mirroring the OpenAPI artifact's `minItems: 1`). + """ + + model_config = ConfigDict(extra="allow") + + table_path: list[str] + """The table the op acts in, e.g. `["pipe", "my_seq"]`. Empty means the document root.""" + + +class SetKeyOp(_FixOpBase): + """Write `key = value` in the addressed table, whatever it currently holds.""" + + kind: Literal[FixOpKind.SET_KEY] + key: str + value: TomlValue + + +class EnsureTableOp(_FixOpBase): + """Create the addressed table when it is absent, leaving an existing one untouched.""" + + kind: Literal[FixOpKind.ENSURE_TABLE] + table_path: list[str] = Field(min_length=1) + + +class DeleteKeyOp(_FixOpBase): + """Drop `key` from the addressed table.""" + + kind: Literal[FixOpKind.DELETE_KEY] + key: str + + +class DeleteTableOp(_FixOpBase): + """Drop the addressed table, including every chunk of one written out of order.""" + + kind: Literal[FixOpKind.DELETE_TABLE] + table_path: list[str] = Field(min_length=1) + + +class RenameTableKeyOp(_FixOpBase): + """Rename `key` to `new_key` in place within the addressed table, keeping its position.""" + + kind: Literal[FixOpKind.RENAME_TABLE_KEY] + key: str + new_key: str + + +class MoveKeyOp(_FixOpBase): + """Relocate `key` from the addressed table into `new_table_path`, under `new_key`.""" + + kind: Literal[FixOpKind.MOVE_KEY] + key: str + new_table_path: list[str] + new_key: str + + +class RemapValueOp(_FixOpBase): + """Rewrite `key`'s value through `mapping`, doing nothing when it is not a mapped value. + + This is the one kind for which `key` may be the wildcard segment `*`, meaning "each key of + the addressed table" — the only shape in which a renamed enumerated value beneath an open + mapping can be repaired at all. + """ + + kind: Literal[FixOpKind.REMAP_VALUE] + key: str + mapping: dict[str, str] + + +FixOp: TypeAlias = Annotated[ + SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, + Field(discriminator="kind"), +] +"""One semantic patch operation, discriminated on `kind`. + +Narrow it with an exhaustive `match op: case SetKeyOp(): ...` — the Python spelling of the +JS mirror's `kind` narrowing. +""" + + +class SuggestedFix(BaseModel): + """A deterministic fix for one validation error, ready for a style-preserving applier. + + `fix_code` is the kebab-case rule id (e.g. `"match-sequence-output"`). `source` is the file + the ops target, when known (multi-file libraries) — an applier must only apply ops to the + file they target. The ops are the machine contract; any rendered diff is presentation. + """ + + model_config = ConfigDict(extra="allow") + + fix_code: str + description: str + safety: FixSafety + source: str | None = None + ops: list[FixOp] + + +class LiftablePipeEntry(BaseModel): + """One pipe the runtime may skip (lift) when an optional slot resolves absent.""" + + model_config = ConfigDict(extra="allow") + + pipe_ref: str + """Namespaced ref of the liftable pipe.""" + + within_pipe_ref: str + """Namespaced ref of the controller in whose flow the lift happens.""" + + skipped_when_absent: list[str] = Field(default_factory=list) + """The slot names whose absence lifts the pipe.""" + + absence_source: str + """Where the possible absence originates (human-readable).""" + + class ValidatedPipeEntry(BaseModel): """One entry of `PipelexValidationReport.validated_pipes[]`.""" @@ -66,10 +244,19 @@ class ValidationErrorItem(ValidationDiagnostic): populated per category and dropped from the wire when unset. Built by pipelex's one shared builder, so the hosted `InvalidReport` and the agent-CLI envelope cannot drift. + + The same item type serves `PipelexValidationReport.warnings[]`, where the locators are + serialized as explicit `null` rather than dropped. Every optional member is therefore + `T | None = None` on purpose — pydantic reads a dropped key and an explicit `null` into the + same `None`, and tightening any of them to required would break the valid arm. """ category: ValidationErrorCategory # pyright: ignore[reportIncompatibleVariableOverride] error_type: str | None = None + """Deliberately an open string, not an enum: the runtime union keeps gaining advisory + members (the `hint_*` lint types), and closing it here would turn every runtime addition + into an SDK break for no consumer benefit.""" + pipe_code: str | None = None concept_code: str | None = None domain_code: str | None = None @@ -77,8 +264,12 @@ class ValidationErrorItem(ValidationDiagnostic): field_path: str | None = None field_name: str | None = None missing_concept_code: str | None = None + missing_pipe_code: str | None = None variable_names: list[str] | None = None declared_concepts: list[str] | None = None + suggested_fix: SuggestedFix | None = None + """The structured repair proposal for this error, when the runtime's fix planner produced + one; absent otherwise.""" class PipelexValidationReport(ValidationReport): @@ -92,13 +283,38 @@ class PipelexValidationReport(ValidationReport): is_runnable: bool = True message: str = "" mthds_contents: list[str] | None = None + warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem)) + """Advisory lints on a bundle that is nonetheless valid — they never flip `is_valid`. + + Same item type as `validation_errors[]`, so one parser serves both channels. Defaults empty + rather than being required, so a body from a runner predating the field still parses; that + default is also what a clean bundle yields, so no caller can tell the two apart.""" + + liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry)) + """The pipes the runtime may skip when an optional slot resolves absent. + + Defaults empty for the same reason as `warnings`: an older runner's body must keep parsing.""" + + input_form: dict[str, Any] | None = None + """Per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`. + + Optional on purpose: it is present only when the request named the `input_form` view + (`VALIDATION_VIEW_INPUT_FORM`), and an older runner emitted it unconditionally — `None` + by default is the one typing that reads a body from either runner correctly. Kept opaque + like `bundle_blueprint`, `pipe_io_contracts` and `graph_spec`, because the descriptor + vocabulary is owned elsewhere and a second copy here would be free to drift.""" + rendered_markdown: str | None = None """Opt-in Pipelex-API presentation extra: the server-rendered Markdown view of the verdict, present only when the request asked for it (`render: ["markdown"]`); absent (None) otherwise.""" class PipelexInvalidReport(InvalidValidationReport[ValidationErrorItem]): - """The invalid arm carrying pipelex's structured `validation_errors[]` (`is_valid: false`).""" + """The invalid arm carrying pipelex's structured `validation_errors[]` (`is_valid: false`). + + It gains none of the valid arm's additions: `warnings` and `input_form` derive from a crate + that was never assembled, so the invalid arm never carries them. + """ rendered_markdown: str | None = None """Opt-in Pipelex-API presentation extra: the server-rendered Markdown view of the invalid diff --git a/pyproject.toml b/pyproject.toml index 9769741..a69d19a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "pipelex-sdk" -version = "0.5.0" +version = "0.6.0" description = "The Python client for the Pipelex hosted API — the MTHDS Protocol surface plus the durable run lifecycle and the Pipelex product surface, built on the `mthds` protocol base." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] @@ -18,7 +18,7 @@ classifiers = [ ] dependencies = [ - "mthds>=0.8.1", + "mthds>=0.8.2", "pydantic>=2.10.6,<3.0.0", "typing-extensions>=4.0.0", "httpx>=0.23.0,<1.0.0", @@ -29,7 +29,7 @@ dev = [ "mypy==1.19.1", "pyright==1.1.411", "pylint==4.0.4", - "ruff==0.14.13", + "ruff==0.16.4", "pytest>=9.0.3", "pytest-mock>=3.12.0,<4.0.0", "pytest-sugar>=1.0.0", @@ -190,121 +190,127 @@ target-version = "py311" preview = true select = ["ALL"] ignore = [ - "ANN201", # Missing return type annotation for public function `my_func` - "ANN202", # Missing return type annotation for private function `my_func` - "ANN204", # Missing return type annotation for special method `my_func` - "ANN206", # Missing return type annotation for classmethod `my_func` - "ANN401", # Dynamically typed expressions (typing.Any) are disallowed in `...` - "ASYNC230", # Async functions should not open files with blocking methods like `open` - "ASYNC240", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path - - "B903", # Class could be dataclass or namedtuple - - "C901", # Is to complex - "COM812", # Checks for the absence of trailing commas. - - "CPY001", # Missing copyright notice at top of file - - "D100", # Missing docstring in public module - "D101", # Missing docstring in public class - "D102", # Missing docstring in public method - "D103", # Missing docstring in public function - "D104", # Missing docstring in public package - "D105", # Missing docstring in magic method - "D107", # Missing docstring in __init__ - "D205", # 1 blank line required between summary line and description - "D400", # First line should end with a period - "D401", # First line of docstring should be in imperative mood: "My docstring...." - "D404", # First word of the docstring should not be "This" - "D415", # First line should end with a period, question mark, or exclamation point - - "DOC201", # `return` is not documented in docstring - "DOC202", # Docstring should not have a returns section because the function doesn't return anything - "DOC402", # `yield` is not documented in docstring - "DOC502", # Raised exception is not explicitly raised: `FileNotFoundError` - "DOC501", # Raised exception `ModuleFileError` missing from docstring - - "DTZ001", # `datetime.datetime()` called without a `tzinfo` argument - "DTZ005", # `datetime.datetime.now()` called without a `tz` argument - - "ERA001", # Found commented-out code - - "FBT001", # Boolean-typed positional argument in function definition - "FBT002", # Boolean default positional argument in function definition - "FBT003", #Boolean positional value in function call - - "FIX002", # Line contains TODO, consider resolving the issue - - "FURB101", # `open` and `read` should be replaced by `Path(file_path.path).read_text(encoding="utf-8")` - "FURB113", # Checks for consecutive calls to append. - "FURB152", # Checks for literals that are similar to constants in math module. - - "LOG004", # `.exception()` call outside exception handlers - - "PLC0105", # `TypeVar` name "SomethingType" does not reflect its covariance; consider renaming it to "SomethingType_co" - "PLC1901", # Checks for comparisons to empty strings. - - "PLR0904", # Too many public methods ( > 20) - "PLR0911", # Too many return statements (/6) - "PLR0912", # Too many branches (/12) - "PLR0913", # Too many arguments in function definition (/5) - "PLR0914", # Too many local variables ( /15) - "PLR0915", # Too many statements (/50) - "PLR0917", # Too many positional arguments ( /5) - "PLR2004", # Magic value used in comparison, consider replacing `2` with a constant variable - "PLR6301", # Too many return statements in `for` loop - "PLR1702", # Too many nested blocks ( > 5) - - "PT013", # Incorrect import of `pytest`; use `import pytest` instead - - "PTH100", # `os.path.abspath()` should be replaced by `Path.resolve()` - "PTH103", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` - "PTH107", # `os.remove()` should be replaced by `Path.unlink()` - "PTH109", # `os.getcwd()` should be replaced by `Path.cwd()` - "PTH118", # `os.path.join()` should be replaced by `Path` with `/` operator - "PTH120", # `os.path.dirname()` should be replaced by `Path.parent` - "PTH110", # `os.path.exists()` should be replaced by `Path.exists()` - "PTH112", # `os.path.isdir()` should be replaced by `Path.is_dir()` - "PTH119", # `os.path.basename()` should be replaced by `Path.name` - "PTH123", # `open()` should be replaced by `Path.open()` - "PTH208", # Use `pathlib.Path.iterdir()` instead. - - "PYI051", # `Literal["auto"]` is redundant in a union with `str` - - "RET505", # superfluous-else-return - - "RUF001", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? - "RUF003", # Comment contains ambiguous `’` (RIGHT SINGLE QUOTATION MARK). Did you mean ``` (GRAVE ACCENT)? - "RUF022", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. - - "SIM105", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` - "SIM108", # Use ternary operator `description = func.__doc__.strip().split("\n")[0] if func.__doc__ else func.__name__` instead of `if`-`else`-block - - "S101", # Use of `assert` detected - "S102", # Use of `exec` detected - "S106", # Possible hardcoded password assigned to argument: "secret" - "S105", # Possible hardcoded password assigned to: "child_secret" - - "S311", # Cryptographically weak pseudo-random number generator - - "TD002", # Missing author in TODO; try: `# TODO(): ...` or `# TODO @: ...` - "TD003", # Missing issue link for this TODO - - "T201", # `print` found + "missing-return-type-undocumented-public-function", # Missing return type annotation for public function `my_func` + "missing-return-type-private-function", # Missing return type annotation for private function `my_func` + "missing-return-type-special-method", # Missing return type annotation for special method `my_func` + "missing-return-type-class-method", # Missing return type annotation for classmethod `my_func` + "any-type", # Dynamically typed expressions (typing.Any) are disallowed in `...` + "blocking-open-call-in-async-function", # Async functions should not open files with blocking methods like `open` + "blocking-path-method-in-async-function", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path + + "class-as-data-structure", # Class could be dataclass or namedtuple + + "complex-structure", # Is to complex + "missing-trailing-comma", # Checks for the absence of trailing commas. + + "missing-copyright-notice", # Missing copyright notice at top of file + + "undocumented-public-module", # Missing docstring in public module + "undocumented-public-class", # Missing docstring in public class + "undocumented-public-method", # Missing docstring in public method + "undocumented-public-function", # Missing docstring in public function + "undocumented-public-package", # Missing docstring in public package + "undocumented-magic-method", # Missing docstring in magic method + "undocumented-public-init", # Missing docstring in __init__ + "missing-blank-line-after-summary", # 1 blank line required between summary line and description + "missing-trailing-period", # First line should end with a period + "non-imperative-mood", # First line of docstring should be in imperative mood: "My docstring...." + "docstring-starts-with-this", # First word of the docstring should not be "This" + "missing-terminal-punctuation", # First line should end with a period, question mark, or exclamation point + + "docstring-missing-returns", # `return` is not documented in docstring + "docstring-extraneous-returns", # Docstring should not have a returns section because the function doesn't return anything + "docstring-missing-yields", # `yield` is not documented in docstring + "docstring-extraneous-exception", # Raised exception is not explicitly raised: `FileNotFoundError` + "docstring-missing-exception", # Raised exception `ModuleFileError` missing from docstring + + "call-datetime-without-tzinfo", # `datetime.datetime()` called without a `tzinfo` argument + "call-datetime-now-without-tzinfo", # `datetime.datetime.now()` called without a `tz` argument + + "commented-out-code", # Found commented-out code + + "boolean-type-hint-positional-argument", # Boolean-typed positional argument in function definition + "boolean-default-value-positional-argument", # Boolean default positional argument in function definition + "boolean-positional-value-in-call", #Boolean positional value in function call + + "line-contains-todo", # Line contains TODO, consider resolving the issue + + "read-whole-file", # `open` and `read` should be replaced by `Path(file_path.path).read_text(encoding="utf-8")` + "repeated-append", # Checks for consecutive calls to append. + "math-constant", # Checks for literals that are similar to constants in math module. + + "log-exception-outside-except-handler", # `.exception()` call outside exception handlers + + "type-name-incorrect-variance", # `TypeVar` name "SomethingType" does not reflect its covariance; consider renaming it to "SomethingType_co" + "compare-to-empty-string", # Checks for comparisons to empty strings. + + "too-many-public-methods", # Too many public methods ( > 20) + "too-many-return-statements", # Too many return statements (/6) + "too-many-branches", # Too many branches (/12) + "too-many-arguments", # Too many arguments in function definition (/5) + "too-many-locals", # Too many local variables ( /15) + "too-many-statements", # Too many statements (/50) + "too-many-positional-arguments", # Too many positional arguments ( /5) + "magic-value-comparison", # Magic value used in comparison, consider replacing `2` with a constant variable + "no-self-use", # Too many return statements in `for` loop + "too-many-nested-blocks", # Too many nested blocks ( > 5) + + "pytest-incorrect-pytest-import", # Incorrect import of `pytest`; use `import pytest` instead + + "os-path-abspath", # `os.path.abspath()` should be replaced by `Path.resolve()` + "os-makedirs", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` + "os-remove", # `os.remove()` should be replaced by `Path.unlink()` + "os-getcwd", # `os.getcwd()` should be replaced by `Path.cwd()` + "os-path-join", # `os.path.join()` should be replaced by `Path` with `/` operator + "os-path-dirname", # `os.path.dirname()` should be replaced by `Path.parent` + "os-path-exists", # `os.path.exists()` should be replaced by `Path.exists()` + "os-path-isdir", # `os.path.isdir()` should be replaced by `Path.is_dir()` + "os-path-basename", # `os.path.basename()` should be replaced by `Path.name` + "builtin-open", # `open()` should be replaced by `Path.open()` + "os-listdir", # Use `pathlib.Path.iterdir()` instead. + + "redundant-literal-union", # `Literal["auto"]` is redundant in a union with `str` + + "superfluous-else-return", # superfluous-else-return + + "ambiguous-unicode-character-string", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? + "ambiguous-unicode-character-comment", # Comment contains ambiguous `’` (RIGHT SINGLE QUOTATION MARK). Did you mean ``` (GRAVE ACCENT)? + "unsorted-dunder-all", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. + + "suppressible-exception", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` + "if-else-block-instead-of-if-exp", # Use ternary operator `description = func.__doc__.strip().split("\n")[0] if func.__doc__ else func.__name__` instead of `if`-`else`-block + + "assert", # Use of `assert` detected + "exec-builtin", # Use of `exec` detected + "hardcoded-password-func-arg", # Possible hardcoded password assigned to argument: "secret" + "hardcoded-password-string", # Possible hardcoded password assigned to: "child_secret" + + "suspicious-non-cryptographic-random-usage", # Cryptographically weak pseudo-random number generator + + "missing-todo-author", # Missing author in TODO; try: `# TODO(): ...` or `# TODO @: ...` + "missing-todo-link", # Missing issue link for this TODO + + "print", # `print` found # TODO: stop ignoring these rules - "BLE001", # Do not catch blind exception: `Exception` - "B027", # Checks for empty methods in abstract base classes without an abstract decorator. - "UP007", # Use `X | Y` for type annotations - "UP036", # Version block is outdated for minimum Python version - "SIM102", # Use a single `if` statement instead of nested `if` statements - "S701", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. - "TRY301", # Abstract `raise` to an inner function - "PERF401", # Use a list comprehension to create a transformed list - "PLW2901", # `for` loop variable `line` overwritten by assignment target - "TRY300", # Consider moving this statement to an `else` block - "UP035", # `typing.List` is deprecated, use `list` instead - "RET503", # Missing explicit `return` at the end of function able to return non-`None` value + "blind-except", # Do not catch blind exception: `Exception` + "empty-method-without-abstract-decorator", # Checks for empty methods in abstract base classes without an abstract decorator. + "non-pep604-annotation-union", # Use `X | Y` for type annotations + "outdated-version-block", # Version block is outdated for minimum Python version + "collapsible-if", # Use a single `if` statement instead of nested `if` statements + "jinja2-autoescape-false", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. + "raise-within-try", # Abstract `raise` to an inner function + "manual-list-comprehension", # Use a list comprehension to create a transformed list + "redefined-loop-name", # `for` loop variable `line` overwritten by assignment target + "try-consider-else", # Consider moving this statement to an `else` block + "deprecated-import", # `typing.List` is deprecated, use `list` instead + "implicit-return", # Missing explicit `return` at the end of function able to return non-`None` value + + # Shrinking a `try` clause changes which statements its handlers cover, so satisfying + # this rule is an error-handling refactor rather than lint cleanup. Ignored across the + # workspace; recorded here even though this repo currently has no findings, so the + # decision does not have to be made again by whoever writes the first long `try`. + "too-many-statements-in-try-clause", # Try clause contains too many statements ] [tool.ruff.lint.flake8-type-checking] @@ -315,14 +321,15 @@ convention = "google" [tool.ruff.lint.per-file-ignores] "tests/**/*.py" = [ - "INP001", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) - "SLF001", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) - "PLC2701", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) - "ARG002", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) + "private-member-access", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) + "import-private-name", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) + "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise ] "examples/**/*.py" = [ - "INP001", # Runnable demo scripts, not an importable package - "T201", # print() is the whole point of a demo script + "implicit-namespace-package", # Runnable demo scripts, not an importable package + "print", # print() is the whole point of a demo script ] [tool.uv] diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py new file mode 100644 index 0000000..84dde2e --- /dev/null +++ b/tests/unit/conftest.py @@ -0,0 +1,55 @@ +"""Shared fixtures for the unit suite — a client, a wire-response builder, and the `_send` spy. + +House rule: fixtures live in `conftest.py`. `test_client_product.py` predates these and keeps +its own equivalent private helpers; migrating it is deliberately not part of this change. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol + +import httpx +import pytest + +from pipelex_sdk.client import PipelexAPIClient + +if TYPE_CHECKING: + from pytest_mock import MockerFixture, MockType + +BASE_URL = "http://localhost:8081" + + +class ResponseBuilder(Protocol): + """Builds one wire response the way the transport would hand it back.""" + + def __call__(self, status_code: int, *, json_body: object | None = None) -> httpx.Response: ... + + +class SendPatcher(Protocol): + """Patches a client's `_send` with a scripted response sequence and returns the spy.""" + + def __call__(self, client: PipelexAPIClient, *responses: httpx.Response) -> MockType: ... + + +@pytest.fixture +def api_client() -> PipelexAPIClient: + return PipelexAPIClient(api_key="test-token", base_url=BASE_URL) + + +@pytest.fixture +def wire_response() -> ResponseBuilder: + def _build(status_code: int, *, json_body: object | None = None) -> httpx.Response: + request = httpx.Request("GET", f"{BASE_URL}/x") + if json_body is None: + return httpx.Response(status_code, request=request) + return httpx.Response(status_code, json=json_body, request=request) + + return _build + + +@pytest.fixture +def patch_send(mocker: MockerFixture) -> SendPatcher: + def _patch(client: PipelexAPIClient, *responses: httpx.Response) -> MockType: + return mocker.patch.object(client, "_send", mocker.AsyncMock(side_effect=list(responses))) + + return _patch diff --git a/tests/unit/test_client_method_id.py b/tests/unit/test_client_method_id.py new file mode 100644 index 0000000..9edf588 --- /dev/null +++ b/tests/unit/test_client_method_id.py @@ -0,0 +1,140 @@ +"""Tests for the hosted `method_id` run option — the layer-3 extension this SDK names itself. + +Mirrors `pipelex-sdk-js/tests/client.test.ts` "hosted method_id option". The doctrine the +assertions pin is the layered extension policy: a hosted client types its own platform's +arguments and guards them per layer, and `extra` stays the escape hatch for the extensions it +does not know about. +""" + +import asyncio + +import httpx +import pytest +from mthds.protocol.exceptions import PipelineRequestError +from pytest_mock import MockerFixture + +from pipelex_sdk.client import PipelexAPIClient + +_BASE_URL = "http://localhost:8081" + +_BARE_VERSION = {"protocol_version": "0.6.0", "implementation": "pipelex-api", "runner_version": "1.2.3"} +_START_BODY = {"pipeline_run_id": "run_1", "state": "RUNNING", "created_at": "2026-08-24T00:00:00Z"} +_EXECUTE_BODY: dict[str, object] = { + "pipeline_run_id": "run-x", + "main_stuff_name": "result", + "pipe_output": { + "working_memory": { + "root": {"result": {"concept": "native.Text", "content": {"text": "hi"}}}, + "aliases": {"main_stuff": "result"}, + }, + "pipeline_run_id": "run-x", + }, +} + + +def _response(status_code: int, *, json: object | None = None) -> httpx.Response: + request = httpx.Request("POST", f"{_BASE_URL}/v1/start") + if json is None: + return httpx.Response(status_code, request=request) + return httpx.Response(status_code, json=json, request=request) + + +class TestHostedMethodIdOption: + def _client(self) -> PipelexAPIClient: + return PipelexAPIClient(api_key="test-token", base_url=_BASE_URL) + + def test_method_id_rides_the_body_as_a_top_level_field(self, mocker: MockerFixture) -> None: + """The typed option reaches the wire exactly where the `extra` passthrough used to put it.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(pipe_code="answer", method_id="mt_1")) + + sent = send.call_args.kwargs["content"].decode("utf-8") + assert '"method_id":"mt_1"' in sent + # The wire body is flat — the option is not nested under an `extra` key. + assert '"extra"' not in sent + + def test_method_id_only_run_is_accepted(self, mocker: MockerFixture) -> None: + """A stored method IS something to run: the platform resolves its source server-side.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(method_id="mt_1")) + + assert '"method_id":"mt_1"' in send.call_args.kwargs["content"].decode("utf-8") + + def test_method_id_alongside_inline_source_is_linkage_not_a_conflict(self, mocker: MockerFixture) -> None: + """Inline source wins as the thing to RUN; the id rides along as run-history linkage. + + Refusing the combination once orphaned every unsaved-buffer run from its method — the Run + row's `method_id` is what writes the index key `GET /v1/runs?method_id=` queries. + """ + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(mthds_contents=['domain = "answer"'], method_id="mt_1")) + + sent = send.call_args.kwargs["content"].decode("utf-8") + assert '"method_id":"mt_1"' in sent + assert '"mthds_contents"' in sent + + def test_start_extra_rejects_a_smuggled_method_id(self) -> None: + """One argument, one path: the layer that NAMES a key must also guard it on `extra`.""" + client = self._client() + with pytest.raises(PipelineRequestError, match="method_id"): + asyncio.run(client.start(pipe_code="p", extra={"method_id": "mt_1"})) + + def test_execute_extra_rejects_a_smuggled_method_id(self) -> None: + """The guard is on the shared merge helper, so both run routes reject identically.""" + client = self._client() + with pytest.raises(PipelineRequestError, match="method_id"): + asyncio.run(client.execute(pipe_code="p", extra={"method_id": "mt_1"})) + + def test_empty_method_id_is_absent(self, mocker: MockerFixture) -> None: + """`method_id=""` selects nothing and links nothing, so it is neither sent nor a run source.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(pipe_code="p", method_id="")) + assert "method_id" not in send.call_args.kwargs["content"].decode("utf-8") + + with pytest.raises(PipelineRequestError): + asyncio.run(client.start(method_id="")) + + @pytest.mark.parametrize("wrong_typed_method_id", [0, 123, [], ["mt_1"], {}, 1.5, True]) + def test_non_string_method_id_raises_before_any_request(self, mocker: MockerFixture, wrong_typed_method_id: object) -> None: + """A wrong-typed selector is refused at the boundary, on both run routes, before the wire. + + Without the guard the partition of wrong values is arbitrary: a bare truthiness check + drops the falsy ones (`0`, `[]`) and forwards the truthy ones (`123`, `["mt_1"]`) to a + server `422`. One wrong value, one answer. + """ + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + asyncio.run(client.execute(pipe_code="p", method_id=wrong_typed_method_id)) # type: ignore[arg-type] + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + asyncio.run(client.start(pipe_code="p", method_id=wrong_typed_method_id)) # type: ignore[arg-type] + + send.assert_not_called() + + def test_blocking_fallback_forwards_method_id(self, mocker: MockerFixture) -> None: + """A bare runner must SEE the selector, so it can answer the 422 that names it. + + Dropping it on the fallback would silently run something else instead of surfacing that + the deployment has no catalog. + """ + client = self._client() + send = mocker.patch.object( + client, + "_send", + mocker.AsyncMock(side_effect=[_response(200, json=_BARE_VERSION), _response(200, json=_EXECUTE_BODY)]), + ) + + asyncio.run(client.start_and_wait(pipe_code="p", method_id="mt_1")) + + execute_call = send.call_args_list[1] + assert execute_call.args[1] == f"{_BASE_URL}/v1/execute" + assert '"method_id":"mt_1"' in execute_call.kwargs["content"].decode("utf-8") diff --git a/tests/unit/test_client_paging.py b/tests/unit/test_client_paging.py new file mode 100644 index 0000000..9955141 --- /dev/null +++ b/tests/unit/test_client_paging.py @@ -0,0 +1,206 @@ +"""Tests for the paged-list iterators — `iterate_methods` and `iterate_runs`. + +Both follow `next_cursor` until the server says stop, but they stop on *different* signals, +and the difference is in the server rather than the client: `q` is a post-read filter over a +bounded index slice, so a method page can be empty with a live cursor; the run date bounds are +index key conditions, so a run page never is. These pin both rules, the stuck-cursor guard, and +the runaway backstop. +""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING, Any, cast +from urllib.parse import parse_qs, urlparse + +import pytest + +from pipelex_sdk.errors import PagingNotTerminatingError + +if TYPE_CHECKING: + from pytest_mock import MockerFixture, MockType + + from pipelex_sdk.client import PipelexAPIClient + from pipelex_sdk.product_models import MethodSummary, PipelineRun + from tests.unit.conftest import ResponseBuilder, SendPatcher + + +def _method(method_id: str) -> dict[str, Any]: + return {"method_id": method_id, "name": f"Method {method_id}", "created_at": "t"} + + +def _run(run_id: str) -> dict[str, Any]: + return {"pipeline_run_id": run_id, "method_id": "m1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"} + + +def _cursors_sent(send: MockType) -> list[str | None]: + """The `cursor` query value of every request the spy recorded, in order.""" + cursors: list[str | None] = [] + for call in send.call_args_list: + url = cast("str", call.args[1]) + query: dict[str, list[str]] = parse_qs(urlparse(url).query) + values = query.get("cursor") + cursors.append(values[0] if values else None) + return cursors + + +async def _drain_methods(client: PipelexAPIClient, **kwargs: Any) -> list[MethodSummary]: + return [summary async for summary in client.iterate_methods(**kwargs)] + + +async def _drain_runs(client: PipelexAPIClient, method_id: str) -> list[PipelineRun]: + return [pipeline_run async for pipeline_run in client.iterate_runs(method_id)] + + +class TestClientPaging: + def test_iterate_methods_follows_the_cursor_and_stops_on_none( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + send = patch_send( + api_client, + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_method("m2")], "next_cursor": None}), + ) + + summaries = asyncio.run(_drain_methods(api_client)) + + assert [summary.method_id for summary in summaries] == ["m1", "m2"] + # The cursor sent on page N+1 is exactly the `next_cursor` received on page N. + assert _cursors_sent(send) == [None, "c1"] + + def test_iterate_methods_continues_through_an_empty_page_with_a_live_cursor( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """`q` filters after the index read, so an empty page means "keep going", not "done".""" + patch_send( + api_client, + wire_response(200, json_body={"items": [], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_method("m9")], "next_cursor": None}), + ) + + summaries = asyncio.run(_drain_methods(api_client, q="needle")) + + assert [summary.method_id for summary in summaries] == ["m9"] + + def test_iterate_methods_stops_on_an_unchanged_cursor_without_re_yielding( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """A server that hands back the cursor it was sent stops the loop *before* yielding.""" + patch_send( + api_client, + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "stuck"}), + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "stuck"}), + ) + + summaries = asyncio.run(_drain_methods(api_client)) + + assert [summary.method_id for summary in summaries] == ["m1"] + + def test_iterate_methods_raises_past_the_page_ceiling( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + mocker: MockerFixture, + ) -> None: + """The backstop raises rather than returning: a truncated list is the bug paging removed.""" + mocker.patch("pipelex_sdk.client._MAX_LIST_PAGES", 2) + pages = [wire_response(200, json_body={"items": [_method(f"m{index}")], "next_cursor": f"c{index}"}) for index in range(5)] + send = patch_send(api_client, *pages) + + with pytest.raises(PagingNotTerminatingError) as exc_info: + asyncio.run(_drain_methods(api_client)) + + assert exc_info.value.page_limit == 2 + # The raise lands ON the ceiling, not one page past it — otherwise a loosened + # comparison would still satisfy `page_limit`, which only echoes the constant. + assert send.call_count == 2 + + def test_iterate_runs_stops_on_an_empty_page( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """Run date bounds are index key conditions, so an empty page really is the end.""" + patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r99")], "next_cursor": None}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1"] + + def test_iterate_runs_follows_the_cursor_and_stops_on_none( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + send = patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": None}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1", "r2"] + assert _cursors_sent(send) == [None, "c1"] + + def test_iterate_runs_raises_on_a_cursor_cycle_over_non_empty_pages( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + mocker: MockerFixture, + ) -> None: + """A cursor cycling across two values trips neither the empty-page nor the adjacent check. + + `c1 → c2 → c1 → …` with every page non-empty would loop forever re-yielding the same + runs; the shared page ceiling is what bounds it. + """ + mocker.patch("pipelex_sdk.client._MAX_LIST_PAGES", 4) + cycle = [ + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + ] + send = patch_send(api_client, *cycle) + + with pytest.raises(PagingNotTerminatingError) as exc_info: + asyncio.run(_drain_runs(api_client, "m1")) + + assert exc_info.value.page_limit == 4 + assert send.call_count == 4 + + def test_iterate_runs_stops_on_an_unchanged_cursor_without_re_yielding( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "stuck"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "stuck"}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1"] diff --git a/tests/unit/test_client_product.py b/tests/unit/test_client_product.py index 75e5ab3..82b5cc9 100644 --- a/tests/unit/test_client_product.py +++ b/tests/unit/test_client_product.py @@ -6,6 +6,7 @@ import asyncio import json +from typing import cast import httpx import pytest @@ -14,6 +15,8 @@ from pipelex_sdk.client import PipelexAPIClient from pipelex_sdk.errors import ApiResponseError from pipelex_sdk.product_models import ( + MethodDeletionState, + MethodFile, MethodWriteInput, OnboardingCurrentTool, OnboardingHeardFrom, @@ -28,6 +31,19 @@ _BASE_URL = "http://localhost:8081" +# The platform's `MethodPublic` shape: `org_id` and `created_by_user_id` are required, and +# `python` crosses the wire as one string holding a JSON `[{name, content}]` array. +_METHOD_BODY: dict[str, object] = { + "method_id": "m1", + "name": "M", + "mthds": "src", + "org_id": "org_1", + "created_by_user_id": "u1", + "python": "", + "created_at": "t", + "updated_at": "t", +} + def _response(status_code: int, *, json_body: object | None = None, content: bytes | None = None) -> httpx.Response: request = httpx.Request("GET", f"{_BASE_URL}/x") @@ -38,6 +54,12 @@ def _response(status_code: int, *, json_body: object | None = None, content: byt return httpx.Response(status_code, request=request) +def _sent_body(send: MockType) -> dict[str, object]: + """The decoded JSON request body of the single recorded `_send` call.""" + content = send.call_args.kwargs["content"] + return cast("dict[str, object]", json.loads(content)) + + class _Sent: """The single `_send` call recorded by the spy, decoded for assertions.""" @@ -79,20 +101,80 @@ def test_get_me(self, mocker: MockerFixture) -> None: # ── Methods catalog ────────────────────────────────────────────── - def test_list_methods(self, mocker: MockerFixture) -> None: + def test_list_methods_returns_the_page_envelope(self, mocker: MockerFixture) -> None: + """The route answers `{items, next_cursor}`; the rows are index projections, not full methods.""" client = self._client() - methods = [{"method_id": "m1", "name": "M", "mthds": "...", "created_at": "t", "updated_at": "t"}] - send = self._mock_send(mocker, client, _response(200, json_body=methods)) + page = { + "items": [{"method_id": "m1", "name": "M", "description": "d", "created_at": "t", "deletion_state": "pending"}], + "next_cursor": "c1", + } + send = self._mock_send(mocker, client, _response(200, json_body=page)) result = asyncio.run(client.list_methods()) assert self._sent(send).url == f"{_BASE_URL}/v1/methods" - assert result[0].method_id == "m1" - assert result[0].name == "M" + assert result.next_cursor == "c1" + assert result.items[0].method_id == "m1" + assert result.items[0].description == "d" + # A method mid-deletion stays listed, so a UI can render "Deleting…". + assert result.items[0].deletion_state is MethodDeletionState.PENDING + + def test_list_methods_keeps_query_params_on_presence(self, mocker: MockerFixture) -> None: + """An explicit empty `q` is forwarded — bad input the API should reject, not something to drop.""" + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_methods(q="", limit=50, cursor="c/1")) + + assert self._sent(send).url == f"{_BASE_URL}/v1/methods?q=&limit=50&cursor=c%2F1" + + def test_list_methods_omits_absent_query_params_entirely(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_methods(limit=5)) + + assert self._sent(send).url == f"{_BASE_URL}/v1/methods?limit=5" + + def test_method_data_parses_the_new_fields_and_the_python_wire_string(self, mocker: MockerFixture) -> None: + """`python` is one wire string; the boundary converts it so callers never see it.""" + client = self._client() + body = {**_METHOD_BODY, "description": "d", "python": '[{"name": "a.py", "content": "x = 1"}]'} + self._mock_send(mocker, client, _response(200, json_body=body)) + + method = asyncio.run(client.get_method("m1")) + + assert method.org_id == "org_1" + assert method.created_by_user_id == "u1" + assert method.description == "d" + assert method.deletion_state is None + assert method.python == [MethodFile(name="a.py", content="x = 1")] + + def test_method_data_reads_the_clear_sentinel_as_no_files(self, mocker: MockerFixture) -> None: + client = self._client() + self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + + assert asyncio.run(client.get_method("m1")).python == [] + + def test_write_input_sends_python_three_ways(self, mocker: MockerFixture) -> None: + """`None` omits the key (preserve), `[]` sends the clear sentinel, a list sends the JSON text.""" + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + + asyncio.run(client.update_method("m1", MethodWriteInput(name="M", mthds="src"))) + assert "python" not in _sent_body(send) + + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + asyncio.run(client.update_method("m1", MethodWriteInput(name="M", mthds="src", python=[]))) + assert _sent_body(send)["python"] == "" + + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + asyncio.run(client.create_method(MethodWriteInput(name="M", mthds="src", python=[MethodFile(name="a.py", content="x = 1")]))) + assert json.loads(cast("str", _sent_body(send)["python"])) == [{"name": "a.py", "content": "x = 1"}] def test_get_method_encodes_id(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "a/b", "name": "M", "mthds": "...", "created_at": "t", "updated_at": "t"} + body = {**_METHOD_BODY, "method_id": "a/b"} send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.get_method("a/b")) @@ -101,7 +183,7 @@ def test_get_method_encodes_id(self, mocker: MockerFixture) -> None: def test_create_method_posts_write_body(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "m1", "name": "M", "mthds": "src", "created_at": "t", "updated_at": "t"} + body = _METHOD_BODY send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.create_method(MethodWriteInput(name="M", mthds="src", input_data={"a": 1}))) @@ -113,7 +195,7 @@ def test_create_method_posts_write_body(self, mocker: MockerFixture) -> None: def test_update_method_puts_and_drops_absent_input_data(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "m1", "name": "Renamed", "mthds": "src", "created_at": "t", "updated_at": "t"} + body = {**_METHOD_BODY, "name": "Renamed"} send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.update_method("m1", MethodWriteInput(name="Renamed", mthds="src"))) @@ -124,16 +206,24 @@ def test_update_method_puts_and_drops_absent_input_data(self, mocker: MockerFixt # input_data is None → dropped from the wire (matches the JS undefined-drop). assert sent.body == {"name": "Renamed", "mthds": "src"} - def test_delete_method_tolerates_empty_204(self, mocker: MockerFixture) -> None: + def test_delete_method_returns_the_202_acceptance(self, mocker: MockerFixture) -> None: + """The erasure is asynchronous: the caller gets the CLAIM, never a "it's gone" signal. + + Completion is the row disappearing from `list_methods`, so the honest return value is the + acceptance body — a `deletion_job_id` to log or correlate, and the state it started in. + """ client = self._client() - send = self._mock_send(mocker, client, _response(204)) + body = {"method_id": "m1", "deletion_state": "pending", "deletion_job_id": "job-1"} + send = self._mock_send(mocker, client, _response(202, json_body=body)) - result = asyncio.run(client.delete_method("m1")) + accepted = asyncio.run(client.delete_method("m/1")) sent = self._sent(send) - assert result is None assert sent.method == "DELETE" - assert sent.url == f"{_BASE_URL}/v1/methods/m1" + assert sent.url == f"{_BASE_URL}/v1/methods/m%2F1" + assert accepted.method_id == "m1" + assert accepted.deletion_state is MethodDeletionState.PENDING + assert accepted.deletion_job_id == "job-1" # ── Organizations ──────────────────────────────────────────────── @@ -367,14 +457,72 @@ def test_upload_base64_payload(self, mocker: MockerFixture) -> None: def test_list_runs_encodes_query_value(self, mocker: MockerFixture) -> None: client = self._client() - runs = [{"pipeline_run_id": "r1", "method_id": "m/1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"}] - send = self._mock_send(mocker, client, _response(200, json_body=runs)) + page = { + "items": [{"pipeline_run_id": "r1", "method_id": "m/1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"}], + "next_cursor": "c1", + } + send = self._mock_send(mocker, client, _response(200, json_body=page)) result = asyncio.run(client.list_runs("m/1")) assert self._sent(send).url == f"{_BASE_URL}/v1/runs?method_id=m%2F1" - assert result[0].pipeline_run_id == "r1" - assert result[0].status is RunStatus.RUNNING + assert result.next_cursor == "c1" + assert result.items[0].pipeline_run_id == "r1" + assert result.items[0].status is RunStatus.RUNNING + + def test_list_runs_keeps_date_bounds_and_paging_params_on_presence(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_runs("m1", created_from="2026-08-01T00:00:00+00:00", created_to="", limit=10, cursor="c1")) + + # Instants are percent-encoded; an explicit empty bound is forwarded, not dropped. + assert self._sent(send).url == ( + f"{_BASE_URL}/v1/runs?method_id=m1&created_from=2026-08-01T00%3A00%3A00%2B00%3A00&created_to=&limit=10&cursor=c1" + ) + + def test_run_row_parses_with_null_method_id_and_pipe_code(self, mocker: MockerFixture) -> None: + """An ad-hoc run belongs to no stored method, and a `main_pipe` run names no pipe.""" + client = self._client() + row = { + "pipeline_run_id": "r1", + "method_id": None, + "pipe_code": None, + "status": "FAILED", + "created_at": "t", + "error": {"message": "boom", "error_type": "PipeExecutionError"}, + } + self._mock_send(mocker, client, _response(200, json_body={"items": [row], "next_cursor": None})) + + result = asyncio.run(client.list_runs("m1")) + + pipeline_run = result.items[0] + assert pipeline_run.method_id is None + assert pipeline_run.pipe_code is None + assert pipeline_run.error is not None + assert pipeline_run.error.message == "boom" + assert pipeline_run.error.error_type == "PipeExecutionError" + + def test_get_run_detail_encodes_id_and_returns_what_ran(self, mocker: MockerFixture) -> None: + """The detail read is the only one carrying `mthds_contents` and `inputs`.""" + client = self._client() + body = { + "pipeline_run_id": "r/1", + "method_id": "m1", + "pipe_code": "p", + "status": "COMPLETED", + "created_at": "t", + "mthds_contents": ["domain = 'x'"], + "inputs": {"topic": "quantum"}, + } + send = self._mock_send(mocker, client, _response(200, json_body=body)) + + detail = asyncio.run(client.get_run_detail("r/1")) + + assert self._sent(send).url == f"{_BASE_URL}/v1/runs/r%2F1" + assert detail.mthds_contents == ["domain = 'x'"] + assert detail.inputs == {"topic": "quantum"} + assert detail.status is RunStatus.COMPLETED def test_update_run_drops_absent_finished_at_and_tolerates_empty_body(self, mocker: MockerFixture) -> None: client = self._client() diff --git a/tests/unit/test_client_validate.py b/tests/unit/test_client_validate.py index 8f8d028..0ceb310 100644 --- a/tests/unit/test_client_validate.py +++ b/tests/unit/test_client_validate.py @@ -1,4 +1,4 @@ -"""Tests for the `validate` override + `validate_files` — render injection, `mthds_sources`, the union round-trip.""" +"""Tests for the `validate` override + `validate_files` — render injection, `mthds_sources`, `views`, the union round-trip.""" import asyncio import json @@ -10,7 +10,7 @@ from pytest_mock import MockerFixture, MockType from pipelex_sdk.client import MthdsFile, PipelexAPIClient -from pipelex_sdk.validation_models import PipelexInvalidReport, PipelexValidationReport +from pipelex_sdk.validation_models import VALIDATION_VIEW_INPUT_FORM, PipelexInvalidReport, PipelexValidationReport _BASE_URL = "http://localhost:8081" @@ -91,6 +91,54 @@ def test_returns_invalid_report_union_arm(self, mocker: MockerFixture) -> None: assert result.rendered_markdown == "## errors" assert result.validation_errors[0].message == "boom" + # ── views ──────────────────────────────────────────────────────── + + def test_views_absent_from_body_by_default(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"])) + + # The opt-in stays opt-in: no `views` key at all, so the response is byte-identical + # for consumers that never asked for a view. + assert "views" not in self._sent_body(send) + + def test_views_sent_verbatim(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], views=[VALIDATION_VIEW_INPUT_FORM, "future_view", VALIDATION_VIEW_INPUT_FORM])) + + # Unlike `render`, nothing is injected and nothing is de-duplicated — the server + # resolves the tokens as a set and lenient-ignores the ones it does not know. + assert self._sent_body(send)["views"] == ["input_form", "future_view", "input_form"] + + def test_explicit_empty_views_is_sent_as_empty_list(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], views=[])) + + assert self._sent_body(send)["views"] == [] + + def test_views_rides_alongside_render_injection(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], render=["html"], views=[VALIDATION_VIEW_INPUT_FORM])) + + body = self._sent_body(send) + assert body["render"] == ["html", "markdown"] + assert body["views"] == ["input_form"] + + def test_validate_files_threads_views_through(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate_files([MthdsFile(content="a")], views=[VALIDATION_VIEW_INPUT_FORM])) + + assert self._sent_body(send)["views"] == ["input_form"] + # ── validate_files ─────────────────────────────────────────────── def test_validate_files_no_uri_omits_mthds_sources(self, mocker: MockerFixture) -> None: diff --git a/tests/unit/test_method_files.py b/tests/unit/test_method_files.py new file mode 100644 index 0000000..eb8528c --- /dev/null +++ b/tests/unit/test_method_files.py @@ -0,0 +1,78 @@ +"""Tests for the method-files catalog converter — `parse_method_files` / `serialize_method_files`. + +Mirrors `mthds-js/src/protocol/method_files.ts`. The at-rest catalog form is one wire string +holding a JSON `[{name, content}]` array, with `""` as the platform's "no source" sentinel; +these pin both sentinels, the blank-content drop, and the round-trip. +""" + +from __future__ import annotations + +import json + +import pytest + +from pipelex_sdk.product_models import MethodFile, parse_method_files, serialize_method_files + + +class TestMethodFiles: + @pytest.mark.parametrize("blank_source", [None, "", " ", "\n\t ", "[]"]) + def test_blank_source_and_empty_array_both_parse_to_no_files(self, blank_source: str | None) -> None: + """A blank source and an explicit empty array both mean "this method has no Python".""" + assert parse_method_files(blank_source) == [] + + def test_parses_a_named_array(self) -> None: + files = parse_method_files('[{"name": "funcs/price.py", "content": "def price(): ...\\n"}]') + assert len(files) == 1 + assert files[0].name == "funcs/price.py" + assert files[0].content == "def price(): ...\n" + + def test_drops_blank_content_entries_on_parse(self) -> None: + """A zero-source file is not a file — dropped on the way in, mirroring serialization.""" + files = parse_method_files('[{"name": "a.py", "content": "x = 1"}, {"name": "empty.py", "content": " "}]') + assert [file.name for file in files] == ["a.py"] + + def test_tolerates_extra_keys_on_an_entry(self) -> None: + """`MethodFile` is extension-open, so a newly-added server key must not fail the read.""" + files = parse_method_files('[{"name": "a.py", "content": "x = 1", "sha": "deadbeef"}]') + assert files[0].name == "a.py" + + @pytest.mark.parametrize( + "bad_source", + [ + "not json at all", + '{"name": "a.py", "content": "x"}', # a JSON object, not an array + '"just a string"', + "42", + '[{"name": "a.py"}]', # entry missing `content` + '[{"content": "x = 1"}]', # entry missing `name` + '[{"name": 1, "content": "x = 1"}]', # `name` is not a string + "[[]]", # entry is not an object + ], + ) + def test_malformed_source_raises_naming_the_expected_shape(self, bad_source: str) -> None: + with pytest.raises(ValueError, match=r"\{name, content\}"): + parse_method_files(bad_source) + + def test_empty_list_serializes_to_the_clear_sentinel(self) -> None: + """`""` is the platform's clear signal; the literal `"[]"` would not clear anything.""" + assert serialize_method_files([]) == "" + + def test_blank_only_list_serializes_to_the_clear_sentinel(self) -> None: + assert serialize_method_files([MethodFile(name="empty.py", content=" \n")]) == "" + + def test_serializes_only_name_and_content(self) -> None: + """Whatever an extension-open `MethodFile` picked up on the way in is not written back.""" + parsed = parse_method_files('[{"name": "a.py", "content": "x = 1", "sha": "deadbeef"}]') + assert json.loads(serialize_method_files(parsed)) == [{"name": "a.py", "content": "x = 1"}] + + def test_serializes_mixed_list_dropping_the_blank_entry(self) -> None: + files = [MethodFile(name="a.py", content="x = 1"), MethodFile(name="empty.py", content=""), MethodFile(name="b.py", content="y = 2")] + assert json.loads(serialize_method_files(files)) == [{"name": "a.py", "content": "x = 1"}, {"name": "b.py", "content": "y = 2"}] + + def test_round_trip_is_stable(self) -> None: + """Serialize → parse → serialize reaches a fixed point, blank entries and all.""" + files = [MethodFile(name="a.py", content="x = 1"), MethodFile(name="empty.py", content=" ")] + once = serialize_method_files(files) + twice = serialize_method_files(parse_method_files(once)) + assert once == twice + assert parse_method_files(twice) == [MethodFile(name="a.py", content="x = 1")] diff --git a/tests/unit/test_runs.py b/tests/unit/test_runs.py index eb98a5a..35536fa 100644 --- a/tests/unit/test_runs.py +++ b/tests/unit/test_runs.py @@ -8,8 +8,8 @@ from pipelex_sdk.runs import RunResults, RunStatus, TokensUsageRecord # A record in the shape the current runtime emits: every contract field present, absent values -# sent as explicit nulls. Mirrors the conformance seed corpus -# (conformance/conformance/usage_records.py), which is what the platform arm asserts on the wire. +# sent as explicit nulls. Mirrors the shared conformance corpus of usage records, which is what +# the platform arm asserts on the wire. _RATED_RECORD: dict[str, Any] = { "model_type": "llm", "inference_model_name": "test-model", diff --git a/tests/unit/test_validation_contract.py b/tests/unit/test_validation_contract.py index 98ef7ba..74789a3 100644 --- a/tests/unit/test_validation_contract.py +++ b/tests/unit/test_validation_contract.py @@ -1,9 +1,9 @@ """Contract round-trip tests for the 200-diagnostic `/validate` union. Pins the Pipelex validation wire models (`pipelex_sdk.validation_models`) against the -canonical example bodies from the protocol spec (`docs/specs/pipelex-mthds-protocol.md`, -`## Validation report union`). Mirrors `mthds-js/tests/unit/protocol/validation-contract.test.ts`: -parse a wire body at the boundary, discriminate on `is_valid`, and assert the narrowed arm. +canonical example bodies of the MTHDS Protocol's validation-report union. Mirrors +`mthds-js/tests/unit/protocol/validation-contract.test.ts`: parse a wire body at the boundary, +discriminate on `is_valid`, and assert the narrowed arm. """ from __future__ import annotations @@ -14,11 +14,20 @@ from pydantic import ValidationError from pipelex_sdk.validation_models import ( + DeleteKeyOp, + DeleteTableOp, DryRunStatus, + EnsureTableOp, + FixOpKind, + FixSafety, + MoveKeyOp, PipelexInvalidReport, PipelexValidationReport, PipelexValidationResult, PipelexValidationResultAdapter, + RemapValueOp, + RenameTableKeyOp, + SetKeyOp, ValidationErrorCategory, ) @@ -86,12 +95,89 @@ "message": "Validation succeeded.", } +# The 0.17+ valid arm: advisory warnings, the liftable inventory, and the opt-in input form. +# The valid arm is dumped WITHOUT `exclude_none`, so an unset locator arrives as an explicit +# `null` where the invalid arm drops the key entirely — same item type, two serializations. +# Mirrors the JS fixture "carries advisory warnings on the VALID arm, with the valid arm's +# explicit nulls" (`pipelex-sdk-js/tests/client.test.ts`). +VALID_BODY_WITH_VIEWS: dict[str, Any] = { + **VALID_BODY, + "warnings": [ + { + "category": "pipe_validation", + "message": "the `!` on `profile` is redundant — the slot is always present", + "error_type": "optional_force_redundant", + "pipe_code": "legal_contracts.summarize", + "concept_code": None, + "domain_code": None, + "source": None, + "field_path": None, + "field_name": None, + "missing_concept_code": None, + "missing_pipe_code": None, + "variable_names": None, + "declared_concepts": None, + "suggested_fix": None, + } + ], + "liftable_pipes": [ + { + "pipe_ref": "legal_contracts.enrich", + "within_pipe_ref": "legal_contracts.summarize", + "skipped_when_absent": ["profile"], + "absence_source": "optional input `profile` of legal_contracts.summarize", + } + ], + "input_form": {"legal_contracts.summarize": {"fields": [{"name": "contract", "kind": "text"}]}}, +} + +# The 0.17+ invalid arm: the new `missing_pipe_code` locator and a structured repair proposal. +INVALID_BODY_WITH_FIX: dict[str, Any] = { + "is_valid": False, + "validation_errors": [ + { + "category": "pipe_validation", + "error_type": "PipeValidationError", + "message": "Sequence step references an unknown pipe.", + "pipe_code": "summarize", + "missing_pipe_code": "enrichh", + "source": "contracts.mthds", + "suggested_fix": { + "fix_code": "match-sequence-output", + "description": "Rename the step and record its output.", + "safety": "safe", + "source": "contracts.mthds", + "ops": [ + {"kind": "rename_table_key", "table_path": ["pipe", "summarize", "steps"], "key": "enrichh", "new_key": "enrich"}, + {"kind": "set_key", "table_path": ["pipe", "summarize"], "key": "output", "value": "legal_contracts.Summary"}, + ], + }, + } + ], + "pending_signatures": [], + "is_runnable": False, + "message": "Validation found errors.", +} + def _parse(body: dict[str, Any]) -> PipelexValidationResult: """Parse a wire body through the real discriminated-union adapter — the exact parse path `PipelexAPIClient.validate()` uses.""" return PipelexValidationResultAdapter.validate_python(body) +def _body_with_single_op(fix_op: dict[str, Any]) -> dict[str, Any]: + """An invalid body whose one error carries a `suggested_fix` holding exactly `fix_op`.""" + return { + **INVALID_BODY_WITH_FIX, + "validation_errors": [ + { + **INVALID_BODY_WITH_FIX["validation_errors"][0], + "suggested_fix": {**INVALID_BODY_WITH_FIX["validation_errors"][0]["suggested_fix"], "ops": [fix_op]}, + } + ], + } + + class TestValidationContract: def test_valid_arm_carries_typed_artifacts(self) -> None: """The valid arm parses to a typed report with structural artifacts.""" @@ -163,8 +249,116 @@ def test_rendered_markdown_is_none_when_absent(self) -> None: assert isinstance(invalid, PipelexInvalidReport) assert invalid.rendered_markdown is None + # ── The 0.17+ valid arm: warnings, liftable pipes, the opt-in input form ── + + def test_valid_arm_carries_warnings_liftable_pipes_and_input_form(self) -> None: + """The 0.17+ valid-arm additions parse into typed fields, `input_form` keyed like `pipe_io_contracts`.""" + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + # Advisory items never flip the verdict. + assert report.is_valid is True + warning = report.warnings[0] + assert warning.category is ValidationErrorCategory.PIPE_VALIDATION + assert warning.error_type == "optional_force_redundant" + assert warning.pipe_code == "legal_contracts.summarize" + liftable = report.liftable_pipes[0] + assert liftable.pipe_ref == "legal_contracts.enrich" + assert liftable.within_pipe_ref == "legal_contracts.summarize" + assert liftable.skipped_when_absent == ["profile"] + assert liftable.absence_source == "optional input `profile` of legal_contracts.summarize" + assert report.input_form is not None + # Keyed exactly like `pipe_io_contracts`, and opaque on purpose. + assert set(report.input_form) == set(report.pipe_io_contracts) + + def test_valid_arm_warning_reads_every_explicit_null_as_none(self) -> None: + """Every explicitly-null locator on a warning reads as `None`. + + The valid arm is dumped without `exclude_none`, so an unset locator arrives as an + explicit `null` where the invalid arm drops the key. This is the regression guard + against a future "tighten to required" edit on `ValidationErrorItem`. + """ + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + warning = report.warnings[0] + assert warning.concept_code is None + assert warning.domain_code is None + assert warning.source is None + assert warning.field_path is None + assert warning.field_name is None + assert warning.missing_concept_code is None + assert warning.missing_pipe_code is None + assert warning.variable_names is None + assert warning.declared_concepts is None + assert warning.suggested_fix is None + + def test_pre_0_52_valid_body_still_parses_with_empty_defaults(self) -> None: + """A body from a runner predating the fields parses: both lists empty, `input_form` None.""" + report = _parse(VALID_BODY) + assert isinstance(report, PipelexValidationReport) + assert report.warnings == [] + assert report.liftable_pipes == [] + assert report.input_form is None + + # ── The 0.17+ invalid arm: `missing_pipe_code` and the fix vocabulary ── + + def test_invalid_arm_carries_missing_pipe_code_and_narrowable_fix_ops(self) -> None: + """A structured `suggested_fix` parses, and `match`-narrowing reaches each op's own members.""" + report = _parse(INVALID_BODY_WITH_FIX) + assert isinstance(report, PipelexInvalidReport) + item = report.validation_errors[0] + assert item.missing_pipe_code == "enrichh" + fix = item.suggested_fix + assert fix is not None + assert fix.fix_code == "match-sequence-output" + assert fix.safety is FixSafety.SAFE + assert fix.safety.is_safe is True + assert fix.source == "contracts.mthds" + + rename_op, set_op = fix.ops + # Narrowing is an exhaustive `match` over the op classes — the Python spelling of the + # JS mirror's `kind` narrowing. + match rename_op: + case RenameTableKeyOp(): + assert rename_op.table_path == ["pipe", "summarize", "steps"] + assert rename_op.key == "enrichh" + assert rename_op.new_key == "enrich" + case SetKeyOp() | EnsureTableOp() | DeleteKeyOp() | DeleteTableOp() | MoveKeyOp() | RemapValueOp(): + pytest.fail("expected a rename_table_key op") + match set_op: + case SetKeyOp(): + assert set_op.table_path == ["pipe", "summarize"] + assert set_op.key == "output" + assert set_op.value == "legal_contracts.Summary" + case EnsureTableOp() | DeleteKeyOp() | DeleteTableOp() | RenameTableKeyOp() | MoveKeyOp() | RemapValueOp(): + pytest.fail("expected a set_key op") + + def test_ensure_table_op_rejects_an_empty_table_path(self) -> None: + """`ensure_table` addresses the table itself, so its path is never empty (artifact `minItems: 1`).""" + bad_body = _body_with_single_op({"kind": "ensure_table", "table_path": []}) + with pytest.raises(ValidationError, match="table_path"): + PipelexValidationResultAdapter.validate_python(bad_body) + + def test_unknown_fix_op_kind_is_rejected(self) -> None: + """An out-of-vocabulary op kind fails the whole verdict parse — the discriminator is closed.""" + bad_body = _body_with_single_op({"kind": "invent_key", "table_path": ["pipe"], "key": "x"}) + with pytest.raises(ValidationError, match="invent_key"): + PipelexValidationResultAdapter.validate_python(bad_body) + + def test_fix_vocabularies_are_the_locked_sets(self) -> None: + """`FixSafety` and `FixOpKind` mirror the runtime's closed vocabularies (drift guard).""" + assert {safety.value for safety in FixSafety} == {"safe", "unsafe"} + assert {kind.value for kind in FixOpKind} == { + "set_key", + "ensure_table", + "delete_key", + "delete_table", + "rename_table_key", + "move_key", + "remap_value", + } + def test_category_vocabulary_is_the_locked_set(self) -> None: - """The closed category set mirrors `conformance/.../validation_contract.py` (drift guard).""" + """The closed category set mirrors the locked conformance vocabulary (drift guard).""" assert {category.value for category in ValidationErrorCategory} == { "blueprint_validation", "pipe_factory", diff --git a/uv.lock b/uv.lock index 9e47f0e..f25455f 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.8.1" +version = "0.8.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "httpx" }, @@ -221,9 +221,9 @@ dependencies = [ { name = "tomlkit" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/24/62/8c94607b2466105906232858dcee74eb8dbc2ed0352fb6b280e1f06f94b9/mthds-0.8.1.tar.gz", hash = "sha256:0c9082c2c5d833605431911635d3290ab1d1ad9d8f6d0eb68a27b89df9d256a2", size = 129886, upload-time = "2026-07-06T21:24:01.313Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ed/87/9fefbf352f2f2794336ae6fa1f72ea7d7a74ce78524e3bce734bea83aa02/mthds-0.8.2.tar.gz", hash = "sha256:1ac24a415f5a4942e93309066ed6b65a553ca379578e2da93f492bc63342b0b6", size = 130890, upload-time = "2026-08-21T10:49:37.05Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/94/29/32de38940a4d1c452285eaa8aec8bf251136a26397ded8ff85edf624b217/mthds-0.8.1-py3-none-any.whl", hash = "sha256:ea421aa00a13009d12168fa84b5fd451d0c513e60febcbfb47431c058b15bbfa", size = 58115, upload-time = "2026-07-06T21:23:59.783Z" }, + { url = "https://files.pythonhosted.org/packages/b7/85/051bf842ead495c9126e04a940cbfd3fada2474599b89be27b6d73e73158/mthds-0.8.2-py3-none-any.whl", hash = "sha256:13d314523afc5f774b7f19f65b94ce32e753a617298aae8d6d11315448bbea9f", size = 58167, upload-time = "2026-08-21T10:49:35.804Z" }, ] [[package]] @@ -303,7 +303,7 @@ wheels = [ [[package]] name = "pipelex-sdk" -version = "0.5.0" +version = "0.6.0" source = { editable = "." } dependencies = [ { name = "httpx" }, @@ -326,7 +326,7 @@ dev = [ [package.metadata] requires-dist = [ { name = "httpx", specifier = ">=0.23.0,<1.0.0" }, - { name = "mthds", specifier = ">=0.8.1" }, + { name = "mthds", specifier = ">=0.8.2" }, { name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" }, { name = "pydantic", specifier = ">=2.10.6,<3.0.0" }, { name = "pylint", marker = "extra == 'dev'", specifier = "==4.0.4" }, @@ -334,7 +334,7 @@ requires-dist = [ { name = "pytest", marker = "extra == 'dev'", specifier = ">=9.0.3" }, { name = "pytest-mock", marker = "extra == 'dev'", specifier = ">=3.12.0,<4.0.0" }, { name = "pytest-sugar", marker = "extra == 'dev'", specifier = ">=1.0.0" }, - { name = "ruff", marker = "extra == 'dev'", specifier = "==0.14.13" }, + { name = "ruff", marker = "extra == 'dev'", specifier = "==0.16.4" }, { name = "typing-extensions", specifier = ">=4.0.0" }, ] provides-extras = ["dev"] @@ -557,28 +557,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.14.13" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/50/0a/1914efb7903174b381ee2ffeebb4253e729de57f114e63595114c8ca451f/ruff-0.14.13.tar.gz", hash = "sha256:83cd6c0763190784b99650a20fec7633c59f6ebe41c5cc9d45ee42749563ad47", size = 6059504, upload-time = "2026-01-15T20:15:16.918Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c3/ae/0deefbc65ca74b0ab1fd3917f94dc3b398233346a74b8bbb0a916a1a6bf6/ruff-0.14.13-py3-none-linux_armv6l.whl", hash = "sha256:76f62c62cd37c276cb03a275b198c7c15bd1d60c989f944db08a8c1c2dbec18b", size = 13062418, upload-time = "2026-01-15T20:14:50.779Z" }, - { url = "https://files.pythonhosted.org/packages/47/df/5916604faa530a97a3c154c62a81cb6b735c0cb05d1e26d5ad0f0c8ac48a/ruff-0.14.13-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:914a8023ece0528d5cc33f5a684f5f38199bbb566a04815c2c211d8f40b5d0ed", size = 13442344, upload-time = "2026-01-15T20:15:07.94Z" }, - { url = "https://files.pythonhosted.org/packages/4c/f3/e0e694dd69163c3a1671e102aa574a50357536f18a33375050334d5cd517/ruff-0.14.13-py3-none-macosx_11_0_arm64.whl", hash = "sha256:d24899478c35ebfa730597a4a775d430ad0d5631b8647a3ab368c29b7e7bd063", size = 12354720, upload-time = "2026-01-15T20:15:09.854Z" }, - { url = "https://files.pythonhosted.org/packages/c3/e8/67f5fcbbaee25e8fc3b56cc33e9892eca7ffe09f773c8e5907757a7e3bdb/ruff-0.14.13-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9aaf3870f14d925bbaf18b8a2347ee0ae7d95a2e490e4d4aea6813ed15ebc80e", size = 12774493, upload-time = "2026-01-15T20:15:20.908Z" }, - { url = "https://files.pythonhosted.org/packages/6b/ce/d2e9cb510870b52a9565d885c0d7668cc050e30fa2c8ac3fb1fda15c083d/ruff-0.14.13-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ac5b7f63dd3b27cc811850f5ffd8fff845b00ad70e60b043aabf8d6ecc304e09", size = 12815174, upload-time = "2026-01-15T20:15:05.74Z" }, - { url = "https://files.pythonhosted.org/packages/88/00/c38e5da58beebcf4fa32d0ddd993b63dfacefd02ab7922614231330845bf/ruff-0.14.13-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:78d2b1097750d90ba82ce4ba676e85230a0ed694178ca5e61aa9b459970b3eb9", size = 13680909, upload-time = "2026-01-15T20:15:14.537Z" }, - { url = "https://files.pythonhosted.org/packages/61/61/cd37c9dd5bd0a3099ba79b2a5899ad417d8f3b04038810b0501a80814fd7/ruff-0.14.13-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:7d0bf87705acbbcb8d4c24b2d77fbb73d40210a95c3903b443cd9e30824a5032", size = 15144215, upload-time = "2026-01-15T20:15:22.886Z" }, - { url = "https://files.pythonhosted.org/packages/56/8a/85502d7edbf98c2df7b8876f316c0157359165e16cdf98507c65c8d07d3d/ruff-0.14.13-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a3eb5da8e2c9e9f13431032fdcbe7681de9ceda5835efee3269417c13f1fed5c", size = 14706067, upload-time = "2026-01-15T20:14:48.271Z" }, - { url = "https://files.pythonhosted.org/packages/7e/2f/de0df127feb2ee8c1e54354dc1179b4a23798f0866019528c938ba439aca/ruff-0.14.13-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:642442b42957093811cd8d2140dfadd19c7417030a7a68cf8d51fcdd5f217427", size = 14133916, upload-time = "2026-01-15T20:14:57.357Z" }, - { url = "https://files.pythonhosted.org/packages/0d/77/9b99686bb9fe07a757c82f6f95e555c7a47801a9305576a9c67e0a31d280/ruff-0.14.13-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4acdf009f32b46f6e8864af19cbf6841eaaed8638e65c8dac845aea0d703c841", size = 13859207, upload-time = "2026-01-15T20:14:55.111Z" }, - { url = "https://files.pythonhosted.org/packages/7d/46/2bdcb34a87a179a4d23022d818c1c236cb40e477faf0d7c9afb6813e5876/ruff-0.14.13-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:591a7f68860ea4e003917d19b5c4f5ac39ff558f162dc753a2c5de897fd5502c", size = 14043686, upload-time = "2026-01-15T20:14:52.841Z" }, - { url = "https://files.pythonhosted.org/packages/1a/a9/5c6a4f56a0512c691cf143371bcf60505ed0f0860f24a85da8bd123b2bf1/ruff-0.14.13-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:774c77e841cc6e046fc3e91623ce0903d1cd07e3a36b1a9fe79b81dab3de506b", size = 12663837, upload-time = "2026-01-15T20:15:18.921Z" }, - { url = "https://files.pythonhosted.org/packages/fe/bb/b920016ece7651fa7fcd335d9d199306665486694d4361547ccb19394c44/ruff-0.14.13-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:61f4e40077a1248436772bb6512db5fc4457fe4c49e7a94ea7c5088655dd21ae", size = 12805867, upload-time = "2026-01-15T20:14:59.272Z" }, - { url = "https://files.pythonhosted.org/packages/7d/b3/0bd909851e5696cd21e32a8fc25727e5f58f1934b3596975503e6e85415c/ruff-0.14.13-py3-none-musllinux_1_2_i686.whl", hash = "sha256:6d02f1428357fae9e98ac7aa94b7e966fd24151088510d32cf6f902d6c09235e", size = 13208528, upload-time = "2026-01-15T20:15:03.732Z" }, - { url = "https://files.pythonhosted.org/packages/3b/3b/e2d94cb613f6bbd5155a75cbe072813756363eba46a3f2177a1fcd0cd670/ruff-0.14.13-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:e399341472ce15237be0c0ae5fbceca4b04cd9bebab1a2b2c979e015455d8f0c", size = 13929242, upload-time = "2026-01-15T20:15:11.918Z" }, - { url = "https://files.pythonhosted.org/packages/6a/c5/abd840d4132fd51a12f594934af5eba1d5d27298a6f5b5d6c3be45301caf/ruff-0.14.13-py3-none-win32.whl", hash = "sha256:ef720f529aec113968b45dfdb838ac8934e519711da53a0456038a0efecbd680", size = 12919024, upload-time = "2026-01-15T20:14:43.647Z" }, - { url = "https://files.pythonhosted.org/packages/c2/55/6384b0b8ce731b6e2ade2b5449bf07c0e4c31e8a2e68ea65b3bafadcecc5/ruff-0.14.13-py3-none-win_amd64.whl", hash = "sha256:6070bd026e409734b9257e03e3ef18c6e1a216f0435c6751d7a8ec69cb59abef", size = 14097887, upload-time = "2026-01-15T20:15:01.48Z" }, - { url = "https://files.pythonhosted.org/packages/4d/e1/7348090988095e4e39560cfc2f7555b1b2a7357deba19167b600fdf5215d/ruff-0.14.13-py3-none-win_arm64.whl", hash = "sha256:7ab819e14f1ad9fe39f246cfcc435880ef7a9390d81a2b6ac7e01039083dd247", size = 13080224, upload-time = "2026-01-15T20:14:45.853Z" }, +version = "0.16.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/8f/d8074b1f25e003164087a8bfe79a0f1a3945135764dbb6aaab04103dcaf9/ruff-0.16.4.tar.gz", hash = "sha256:13171aa9d9af2240ee3504e639de73122c67e74036de5ba2e1d01422cd17e3dc", size = 4899731, upload-time = "2026-08-20T17:43:59.196Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ff/80/779895ef584e089d22f2c6df0d0e99a65ec2df0805f1fffd439415b8c1f0/ruff-0.16.4-py3-none-linux_armv6l.whl", hash = "sha256:df4075f71ddac40b9934af60c3ec8a53047dd5a5fdc43224e6e4e8e9a27cb6f7", size = 10006909, upload-time = "2026-08-20T17:43:16.888Z" }, + { url = "https://files.pythonhosted.org/packages/a9/e6/f553199b5e8927a05cb5c422d921fd0656b29ab976e91c44802107c6b0da/ruff-0.16.4-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:0c95538517af68004306b0fb3214ff2f2af67a65092aee77cd9eb86db6656604", size = 10240201, upload-time = "2026-08-20T17:43:19.337Z" }, + { url = "https://files.pythonhosted.org/packages/1c/70/4a6dc4bb34da4dee35e30f09bbd1bfbdd26f33b62fb9b8df31f08a199cd2/ruff-0.16.4-py3-none-macosx_11_0_arm64.whl", hash = "sha256:963f83df8e69e575b64d67dd447ebbc917db41a14bf38d4593a4183e7aaa8255", size = 9835122, upload-time = "2026-08-20T17:43:21.708Z" }, + { url = "https://files.pythonhosted.org/packages/24/12/c6e22d686372c15bcb7af99831f1a1be96df696491babf4f24e4f942c527/ruff-0.16.4-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:32a5057c7ff3f6e6480a48fccfb3a412a690f48a3d03ac5cf08177d6c2da3ade", size = 9977162, upload-time = "2026-08-20T17:43:24.236Z" }, + { url = "https://files.pythonhosted.org/packages/46/49/72b10ec912f5ab5854992eaf7aa7cd36729b6937d9dc4e0fb41b3bf428ec/ruff-0.16.4-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b3dce8d9b0c57c265b91885a66a567d8ea1372e8eb4e250fa8e5e3f579e99cff", size = 9829789, upload-time = "2026-08-20T17:43:26.966Z" }, + { url = "https://files.pythonhosted.org/packages/fa/80/0f30e32e7f6ee26edc39075502db9d368d788a44a79b55f763eb4ab03796/ruff-0.16.4-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7dc651db49283c69f8e72c834eec4fe5573e4c646856aebece0ce385dceb2a80", size = 10527949, upload-time = "2026-08-20T17:43:29.384Z" }, + { url = "https://files.pythonhosted.org/packages/52/3d/86e8ad3542169e56cac3859a343afdb9df2ad54d35a59ce1e67baee83421/ruff-0.16.4-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3817b87dbcabc92f13b05019257c5b89b5b4d51b5fb20f56fb5235ceb723cd07", size = 11333695, upload-time = "2026-08-20T17:43:31.872Z" }, + { url = "https://files.pythonhosted.org/packages/d0/16/481c29b380c20a0054a8261066665e1b3488e23636c49d0a43e75975b9bb/ruff-0.16.4-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e9fce1499134b2c8c68e5166f95705a5812062bb93aacc5f9873bb1a27084bc7", size = 10727741, upload-time = "2026-08-20T17:43:34.596Z" }, + { url = "https://files.pythonhosted.org/packages/5e/b6/56bc0b8cf45b54b28b3a5e6381c8945d51b5b18adf659454c32295209a31/ruff-0.16.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2d812e482f5a7e02eee26cd73d2a37ebbdf47d795ea63ba1b89110ae93e9fb3", size = 10286522, upload-time = "2026-08-20T17:43:37.288Z" }, + { url = "https://files.pythonhosted.org/packages/e8/8b/b345b4fb110f2fbe2bd31eabd271e5e8b3b7e4ee6c0e02f2dc6be78db000/ruff-0.16.4-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:6baaf984aa7976edf93d3b627fe2d1d22ee94bbca05fa6f90fc76d73924e3454", size = 10584182, upload-time = "2026-08-20T17:43:39.984Z" }, + { url = "https://files.pythonhosted.org/packages/29/e5/827b34041c35f58774a9681a4213994c164fc987800f4dddabcf451da0bf/ruff-0.16.4-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:bdfcf0b28662eb890372d50f92c283bb94e67e7635ed93c7fd533970acff7b2b", size = 10134195, upload-time = "2026-08-20T17:43:42.351Z" }, + { url = "https://files.pythonhosted.org/packages/0f/10/d0bffcdd6729b87afc82ba0ef377173356a7dc8e972f5179968cf2fdf98c/ruff-0.16.4-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:b66b02cb9b04f537643cadf5768e5f98dc461890d530cb67113d71c8c76e605d", size = 9825821, upload-time = "2026-08-20T17:43:44.532Z" }, + { url = "https://files.pythonhosted.org/packages/f5/32/0db2a863b796ca62d83e92a07a3ccf00921b14db02059347576a2fda3d4b/ruff-0.16.4-py3-none-musllinux_1_2_i686.whl", hash = "sha256:8528bf9a4b291a60bf02ea453511e8ce6215bd2b982ee80405b66b008b6c30a0", size = 10267658, upload-time = "2026-08-20T17:43:46.989Z" }, + { url = "https://files.pythonhosted.org/packages/b2/a0/fbdeb59e48c6261f523e56c8f12e9c08fbe693786595cc7e3959207a9232/ruff-0.16.4-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:fbd85d2875fdd67e833213a651f613bbf25303abf6aa822a5121f4531195678d", size = 10697071, upload-time = "2026-08-20T17:43:49.891Z" }, + { url = "https://files.pythonhosted.org/packages/aa/28/0c6dd865859c6d17bc8ccc34cb72b0e02d6c7eb25e8a1e22b5bea681e2c0/ruff-0.16.4-py3-none-win32.whl", hash = "sha256:312769988007aaeb8e189b443ccdd03c0e6374489e053467be6d96518ebff76e", size = 10021687, upload-time = "2026-08-20T17:43:52.281Z" }, + { url = "https://files.pythonhosted.org/packages/a3/03/e724450f621698117f9aa6dd241c94d0274ae96781378dc86745ae29f0e7/ruff-0.16.4-py3-none-win_amd64.whl", hash = "sha256:05d9d27a18c4bcbefada602480ec9e01e0bc949d432e0ced5df77edac195919c", size = 10567657, upload-time = "2026-08-20T17:43:54.78Z" }, + { url = "https://files.pythonhosted.org/packages/0e/fe/da8b9e1347696bb22120b77280ec5ce25d500ca5cb39d5ad6e5c18de19c1/ruff-0.16.4-py3-none-win_arm64.whl", hash = "sha256:a3a61621c9b6f6a89573e938a080e648f1695baa3f58570a3a707bc51ff65a21", size = 10451579, upload-time = "2026-08-20T17:43:57.135Z" }, ] [[package]] diff --git a/wip/pr-14-review-notes.md b/wip/pr-14-review-notes.md new file mode 100644 index 0000000..fbb3c0f --- /dev/null +++ b/wip/pr-14-review-notes.md @@ -0,0 +1,63 @@ +# PR #14 — deferred review findings + +Findings from the pre-landing review of [PR #14](https://github.com/Pipelex/pipelex-sdk-python/pull/14) (`feature/Typed-method-id-run-option`, reviewed at `301d96e`). Each item below was verified against the code and, where the claim was about the built artifact, against a locally built sdist. None of them blocks landing the branch; each is deferred because acting on it reaches outside what this branch set out to change. + +The findings that *were* acted on during the review are recorded in `CHANGELOG.md` under `[Unreleased]`, not here. + +--- + +## 1. The published sdist carries the repo's internal planning documents + +**Status:** Confirmed, pre-existing, widened by this branch. Deferred because the fix is a packaging change to `pyproject.toml`, which this branch does not touch and which has release implications worth deciding on their own. + +`pyproject.toml` declares `build-backend = "hatchling.build"` (`pyproject.toml:40`) and carries no `[tool.hatch.build.targets.sdist]` section, so hatchling falls back to including everything the VCS does not ignore. An sdist built at the review point confirms it. The listing below is that build — version 0.5.0, at `301d96e` — and is kept as it was taken rather than re-run, so it predates this very file and is not the current release's archive: + +``` +$ uv build --sdist +$ tar -tzf dist/pipelex_sdk-0.5.0.tar.gz +pipelex_sdk-0.5.0/CLAUDE.md +pipelex_sdk-0.5.0/TODOS.md +pipelex_sdk-0.5.0/Makefile +pipelex_sdk-0.5.0/uv.lock +pipelex_sdk-0.5.0/docs/HANDOFF.md +pipelex_sdk-0.5.0/wip/pr-11-review-notes.md +pipelex_sdk-0.5.0/wip/updates.md +pipelex_sdk-0.5.0/tests/... +``` + +`CLAUDE.md`, `wip/pr-11-review-notes.md`, `docs/HANDOFF.md`, the `Makefile` and the whole `tests/` tree already shipped this way before the branch, so this is not a regression it introduced. What the branch adds is `TODOS.md` and `wip/updates.md` — the tracker and the design — which together are a substantial share of the archive and are addressed to reviewers of this PR rather than to anyone installing the package. `docs/HANDOFF.md` is a related case already on PyPI: it describes creating this repo from scratch and reads as rot to anyone who finds it in a release. + +Nothing breaks — an sdist is not what `pip install` normally consumes, and none of these files is importable — so this is about what a public package says about itself, not about correctness. + +**If picked up:** declare an explicit sdist include list (or an exclude list covering `wip/`, `TODOS.md`, `CLAUDE.md`, `Makefile` and `docs/HANDOFF.md`), decide deliberately whether `tests/` should stay (some consumers value a testable sdist), and land it with a release rather than inside a feature branch. + +## 2. The client class docstring dates its surfaces by build-plan phase + +**Status:** Confirmed, pre-existing. Deferred because Phase 2 of this branch scoped its citation sweep to bare workspace-private *paths*, and widening that scope mid-branch was a judgement call the tracker declined elsewhere for the same reason. + +`pipelex_sdk/client.py:176` and `pipelex_sdk/client.py:178` describe the run lifecycle as "(added in Phase 2)" and the product surface as "(added in Phase 3)". Those phase numbers refer to the original build plan for this package. They travel to PyPI in the class docstring of the one class every consumer instantiates, where they resolve to nothing — the same failure mode as the repo-relative spec paths Phase 2 replaced, in a different spelling. + +**If picked up:** replace each marker with what the reader actually needs (the release the surface shipped in, or nothing at all), and sweep for the same pattern elsewhere in the shipped modules. + +## 3. `start_and_wait` documents fewer exceptions than it propagates + +**Status:** Confirmed, pre-existing. Deferred as too small to justify widening this branch's diff. + +`pipelex_sdk/client.py` documents `Raises: RunFailedError` and `RunTimeoutError` on `start_and_wait`, but the method reaches `_merge_hosted_run_extensions` on both of its paths — the durable one through `start` and the fallback through `_execute_blocking` → `execute` — so it also propagates `PipelineRequestError` for a reserved key on `extra` and, since this branch, for a non-string `method_id`. The `Raises:` sections of `execute` and `start` were corrected during this review; `start_and_wait` was left alone because its omission predates the branch and is not about anything the branch changed. + +**If picked up:** add the `PipelineRequestError` line to `start_and_wait`, and while there check `wait_for_result` and the product methods for the same drift. + +## 4. `PipelineRun.pipe_statuses` is a contract no server fills, in three repos at once + +**Status:** Confirmed, pre-existing, and the most consequential item here. Deferred because it cannot be resolved inside this repo: removing the field locally would break the parity invariant this package is built on, and the decision belongs to whoever owns the run wire contract. + +`pipelex_sdk/product_models.py` declares `pipe_statuses: dict[str, PipeStatus] | None = None` on `PipelineRun`, with `PipeStatus` as its supporting enum. The platform never sends it. `RunPublic` — the model FastAPI serializes `GET /v1/runs` and `GET /v1/runs/{id}` through — declares no such field (`pipelex-server/shared/src/pipelex_shared/schemas/run.py:180`), and a response model strips whatever it does not declare. A `grep -rn "pipe_statuses"` over the entire `pipelex-server` monorepo returns nothing at all, so no route, worker or Lambda writes it either. The field therefore reads `None` on every run this SDK will ever parse, and a consumer branching on it gets a silently empty answer rather than an error. + +The same dead field exists in the two sibling repos, which is what makes it a workspace question rather than a local cleanup: + +- `pipelex-sdk-js/src/product-models.ts:322` — `pipe_statuses?: Record | null;`, with the enum at `:291`. This SDK is a port of that one, so dropping the field here alone would introduce exactly the parity gap `docs/architecture.md` → "Parity with `@pipelex/sdk`" exists to prevent. +- `pipelex-app/src/types/run.ts:24` — the same declaration, and `pipelex-app/src/components/method/run-history-list.tsx:269` renders a row of per-pipe progress dots gated on `{run.pipe_statuses && ...}`. Because the platform never sends it, that guard is always false and those dots have never appeared. Whether that is a missing feature or an abandoned one is the question to settle. + +So there are two coherent outcomes and this branch is the wrong place to choose between them: either the platform starts projecting per-pipe status onto `RunPublic` (and the webapp's dots light up), or the field is retired from all three clients together. + +**Filed:** `../wip/inbox/2026-08-25-workspace-pipe-statuses-dead-field-in-three-clients.md` (`to: workspace`, naming `pipelex-server/platform`, `pipelex-sdk-js` and `pipelex-app`). Whichever way the decision goes, this SDK follows the JS SDK; it should not move first. diff --git a/wip/updates.md b/wip/updates.md new file mode 100644 index 0000000..dd9109c --- /dev/null +++ b/wip/updates.md @@ -0,0 +1,174 @@ +# Updates warranted by pipelex-api 0.17.0 / 0.18.0, the pipelex-server bump, and `@pipelex/sdk` 0.14.0 + +**Status: implemented.** Every change designed below landed, and shipped in `pipelex-sdk` v0.6.0. [`TODOS.md`](../TODOS.md) is the implementation tracker for this design and carries the per-item status; read it for what was done, and this file for what was intended and why. This document is deliberately left as it was written, so its sections below still speak in the future tense and its §7 decisions stay quotable from the tracker. + +A design for what this repo still owes after the three sources named in the title, written on the `feature/Typed-method-id-run-option` branch, which already carries the typed `method_id` run option and the honest `delete_method` contract. Every claim below was checked against the code it names; line numbers are as of 2026-08-25 and will drift. + +## Verdict + +Yes — three groups of work, in decreasing order of what the cited releases actually ask for: + +1. **The `/v1/validate` surface moved, and this SDK has not followed.** pipelex-api 0.17.0 (via the `pipelex` 0.52.0 pin) added `warnings`, `liftable_pipes`, `input_form`, `missing_pipe_code` and `suggested_fix` to the report; 0.18.0 gated `input_form` behind a new `views` request list. `@pipelex/sdk` 0.14.0 mirrored all of it. Here, nothing crashes — every affected model is `extra="allow"`, so the new fields ride `model_extra` — but nothing is typed either, and there is no way to ask for the `input_form` view at all. This is the direct answer to the question and is purely additive. See §1. +2. **Two documentation corrections `@pipelex/sdk` 0.14.0 made apply verbatim here**: the `TokensUsageRecord` brand attribution, and citations of workspace-private paths from a public package. See §2. +3. **Found while checking, and more urgent than either: `list_methods` and `list_runs` crash against the deployed platform.** The platform reshaped both routes into `{items, next_cursor}` page envelopes (in prod since 2026-08-18 and 2026-08-11 respectively); this SDK still iterates a bare array, and its unit tests mock the old shape, which is why the suite is green. `PipelineRun` also declares `method_id` and `pipe_code` as required strings where the platform serves `str | None`. `@pipelex/sdk` fixed all of this in 0.10.0 / 0.11.0. This is a breaking fix and it is not optional. See §3. + +A fourth item is a decision already taken rather than a release to mirror: the 2026-08-25 boundary-validation decision recorded in `pipelex-sdk-js/wip/boundary-option-type-validation.md` names this repo for its Phase 2. See §4. + +What needs **no** change is listed in §5, so nobody re-derives it. The four design choices that were open in the first draft were decided on 2026-08-25 and are recorded in §7. + +## 1. The validate surface + +### 1.1 Where the Python SDK stands today + +- `PipelexValidationReport` (`pipelex_sdk/validation_models.py:84`) inherits `extra="allow"` from `mthds.protocol.models.ValidationReport`, so a 0.17+ body's `warnings`, `liftable_pipes` and `input_form` land in `model_extra`. Untyped, but parsed. +- `ValidationErrorItem` (`validation_models.py:62`) inherits `extra="allow"` from `ValidationDiagnostic`, so `missing_pipe_code` and `suggested_fix` land in `model_extra` the same way. +- Every optional member of `ValidationErrorItem` is already `T | None = None`. The JS 0.14.0 breaking change (`T` → `T | null`, forced by the valid arm serializing unset locators as explicit `null` inside `warnings[]`) therefore has **no Python counterpart** — pydantic reads both a dropped key and an explicit `null` into the same `None`. That asymmetry still deserves a regression test here, because it is the one thing a future "tighten to required" edit would break; the workspace inbox item `wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` explicitly asks for the Python mirror to be checked on this point, and this section is the answer. +- The hosted `/v1/validate` is the platform proxying to the runner (`pipelex-server/platform/src/pipelex_platform/routers/v1/tooling_proxy.py`), so once `feature/Bump-pipelex` (which moves `api-hosted` to the `pipelex-api` `v0.18.0` tag and the core to `pipelex==0.52.0`) is deployed, `api.pipelex.com` serves exactly the contract a bare 0.18.0 runner serves today, `views` gate included. Nothing in that branch changes any other route this SDK calls; its only non-pin edits are the Temporal dry-validate activity carrying `input_form` through to the route that gates it. + +### 1.2 `views` — the structured-view opt-in on `validate` and `validate_files` + +Add `views: list[str] | None = None` to `PipelexAPIClient.validate` (`pipelex_sdk/client.py:449`) after `render`, and `views: list[str] | None = None` to `validate_files` (`client.py:491`) after `render`. In Python this is not the breaking change it was in TypeScript: the parameter is appended last and callers pass it by keyword, so no existing positional call moves. + +Semantics, mirroring `@pipelex/sdk` exactly: + +- `None` (the default) → the `views` key is **not sent**. This is the invariant that keeps an opt-in view opt-in: the default response stays byte-identical for the consumers that discard it (hook pipelines, CI gates, agent loops). +- A list → sent **verbatim**, including an explicitly empty `[]`. Unlike `render`, nothing is injected and nothing is de-duplicated: the server resolves tokens as a set and lenient-ignores unknown ones (never a `422`), so client-side normalization would only hide what the caller asked for. +- Today `input_form` is the only supported token; a constant for it belongs in `validation_models.py` next to the field it gates, not a closed enum on the parameter — the spec deliberately keeps the request boundary open so a stale token never fails a call. + +It reaches the wire through the base transport seam with no `mthds-python` change: `_post_validate` merges `extra` into the body as top-level keys, and its reserved set `_VALIDATE_REQUEST_ARGS` is only `{mthds_contents, allow_signatures}` (`mthds-python/mthds/runners/api/client.py:383`). `views`, like `render` and `mthds_sources` already, is a Pipelex-API carriage extension, so the protocol client stays unaware of it — the same layering as `method_id` on the run routes. + +### 1.3 The valid arm's new fields + +On `PipelexValidationReport`: + +- `warnings: list[ValidationErrorItem] = Field(default_factory=…)` — advisory lints on a **valid** bundle. Same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid`. This is where the `hint_*` error types ride. +- `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=…)` — pipes the runtime may skip when an optional slot resolves absent. New model `LiftablePipeEntry(extra="allow")` with `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str]` (default empty — the server model defaults it too), `absence_source: str`, mirroring `pipelex/pipelex/pipeline/liftable_pipes.py`. +- `input_form: dict[str, Any] | None = None` — per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`. **Optional on purpose**: it is present only when the request named the `input_form` view (0.18.0), and a 0.17.0 runner emitted it unconditionally, so `None`-by-default is the one typing that reads a body from either runner correctly. Kept **opaque** like `bundle_blueprint`, `pipe_io_contracts` and `graph_spec`, for the same reason `@pipelex/sdk` keeps it opaque: the descriptor vocabulary is owned elsewhere (the runtime's `PipeInputFormDescriptor`, the `@pipelex/mthds-form` kernel, the `docs/specs/mthds-input-form-descriptor.md` contract), and a second copy here would be free to drift. + +Both list fields default to empty rather than being required, and that is a deliberate divergence from the runtime model, where they are always populated. This SDK is pointed at bare runners of whatever version a user runs; a pre-0.52 body with neither key must keep parsing, exactly as `bundle_blueprint` already defaults. The default is also what the runtime emits for a clean bundle, so no caller can tell the two apart — which is the point. + +`PipelexInvalidReport` gains nothing: the invalid arm never carries `warnings` or `input_form` (they derive from a crate that was never assembled), and the e2e evidence on the JS side pins `"warnings" not in report`. + +### 1.4 `ValidationErrorItem` additions and the fix vocabulary + +On `ValidationErrorItem`: `missing_pipe_code: str | None = None` (symmetrical with the existing `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. + +New models in `validation_models.py`, mirroring `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact (`pipelex-api/docs/openapi/pipelex-api.openapi.yaml`, schemas `SuggestedFix`, `SetKeyOp` … `RemapValueOp`, `FixSafety`): + +- `FixSafety(StrEnum)`: `SAFE = "safe"`, `UNSAFE = "unsafe"`, with an `is_safe` property (house style: never compare enum values inline). +- `FixOpKind(StrEnum)`: the seven kinds — `set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`. +- `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]` — what a `set_key` writes; deeper nesting is not modelled because the server does not emit it. +- One model per op, each `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key, value)`, `EnsureTableOp()`, `DeleteKeyOp(key)`, `DeleteTableOp()`, `RenameTableKeyOp(key, new_key)`, `MoveKeyOp(key, new_table_path, new_key)`, `RemapValueOp(key, mapping: dict[str, str])`. All carry `table_path: list[str]` (empty for the document root; the OpenAPI artifact marks it `minItems: 1` on `ensure_table` / `delete_table`, which is worth a `Field(min_length=1)` since it costs nothing and mirrors the artifact). +- `FixOp: TypeAlias = Annotated[SetKeyOp | … | RemapValueOp, Field(discriminator="kind")]` — narrowing is `match op: case SetKeyOp(): …`, which is the Python spelling of the JS `kind` narrowing. +- `SuggestedFix(extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]`. + +Two decisions inside that mirror: + +- **Reader models, not runtime models.** The runtime declares these `frozen`, `extra="forbid"`, with wildcard-refusing validators, because it *plans* fixes. This SDK only reads them, so the ops follow the SDK's response-model convention (`extra="allow"`) and carry none of the validators. A new server-side member on an op must not break parsing here. The two runtime invariants a type cannot carry (`*` is the wildcard segment, refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`) go in docstrings, as the JS mirror did. +- **The `kind` vocabulary is closed, and an unknown kind raises.** A pydantic discriminated union needs `Literal` tags, so a kind this SDK does not know fails the parse of the whole verdict. That is consistent with how `ValidationErrorCategory` already behaves (pinned by `test_unknown_category_is_rejected`): the vocabulary is a closed `StrEnum` upstream, a new kind is a `pipelex` release this SDK mirrors, and a loud failure beats a silently unnarrowable op. The alternative — a catch-all op with `kind: str` through a callable `Discriminator` — is more machinery for a repair proposal that is advisory in the first place; noted in §7 as the one place a reviewer might reasonably disagree. + +`error_type` stays `str | None` — an open string, as in the JS mirror. The 0.17.0 changelog notes the union gained the advisory `HintLintErrorType` members; typing it as a closed enum here would turn every runtime enum addition into an SDK break for no consumer benefit. + +Naming stays neutral (`SuggestedFix`, `FixOp`, `warnings`, `liftable_pipes`, `input_form`): fixes and lints are language-level concepts, and the runtime names them brand-neutrally too. The `Pipelex` prefix stays on the two envelope types only. + +### 1.5 Tests + +- `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the key absent from the body when `None`; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` behaviour unchanged alongside it. +- `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes` and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None`; the existing pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; an invalid body carrying `missing_pipe_code` and a two-op `suggested_fix` parses, with `match`-narrowing reaching each op's own members; an unknown `kind` raises `ValidationError`; `FixSafety` / `FixOpKind` value sets are the locked vocabularies. +- The JS suite also pins the gate **live** (`tests/e2e/tools.e2e.ts`: absent by default, present when asked, unknown token lenient). This repo has no e2e suite at all (`tests/` holds only `unit/`), so that half is not reproducible here today. Not a blocker for this change; recorded in §6 as a known gap rather than silently skipped. + +### 1.6 What stays opaque in the 0.17.0 contract move + +The 0.17.0 changelog lists more `/v1/validate` movements than the ones above, and none of them reaches a typed field here: `PipeInputContract.optional` → `presence`, the `fixed` multiplicity with `item_count`, and the widened `inputs` map all live inside `pipe_io_contracts` / `bundle_blueprint`, which this SDK carries as `dict[str, Any]` on purpose. A consumer that reads those dicts should know the new spellings; the SDK's docs (`docs/architecture.md`, validate section) should name them in one sentence so nobody discovers `presence` by surprise, but no model changes. + +## 2. Already on this branch, and the two documentation corrections still owed + +**Done here, matching `@pipelex/sdk` 0.14.0 field-for-field** (commit `cdd8793`): `method_id` as a typed keyword on `execute` / `start` / `start_and_wait`; the run-source precondition satisfied by a `method_id`-only body; `extra` rejecting `method_id` (`_HOSTED_RUN_ARGS`, `client.py:139`); an empty string treated as absent; the selector forwarded on the blocking fallback; `delete_method` returning `MethodDeletionAccepted`. `tests/unit/test_client_method_id.py` pins every one of the JS cases. Nothing further is owed on those. + +**Still owed** — the two prose fixes 0.14.0 shipped under "Fixed", which apply here for the same reason (`pipelex-sdk` is a public PyPI package): + +- **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py:23` and `:128`, `docs/run-usage.md:5` and `docs/architecture.md:103` say the record is "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API is what pins the wire contract. Reword as the JS mirror did (`pipelex-sdk-js/src/runs.ts`, `docs/architecture.md`). +- **Citations a reader cannot open.** `client.py:138` (`docs/specs/pipelex-platform-api.md`), `validation_models.py:44` (`conformance/conformance/validation_contract.py`), and the test-module docstrings at `tests/unit/test_validation_contract.py:4-5` / `:167`, `tests/unit/test_runs.py:12`, `tests/unit/test_client_method_id.py:4`, plus `docs/architecture.md:84`. Each names a workspace-private path by bare relative reference, which resolves to nothing for anyone who clones this repo and reads as rot. Replace each with the rule it was citing (the layered extension policy; the locked category vocabulary; the shared conformance corpus), as 0.14.0 did. No behaviour change. + +## 3. Found while checking: the product list routes are broken against the deployed platform + +### 3.1 Evidence + +- The platform serves `GET /v1/methods` as `MethodPage` — `{items: MethodSummary[], next_cursor: str | None}` — since `pipelex-server` commit `f4f8764` (2026-08-18, "paginate the method list, which was silently truncating"), and `GET /v1/runs?method_id=` as `RunPage` — `{items: RunPublic[], next_cursor}` — since `2c4e980` (2026-08-11). Both are ancestors of the latest `deploy(prod)` commit (`b9f9555`), so this is what `api.pipelex.com` answers today. Models: `pipelex-server/shared/src/pipelex_shared/schemas/method.py:265-309`, `schemas/run.py:180-236`. +- `list_methods` (`pipelex_sdk/client.py:767`) does `[MethodData.model_validate(item) for item in result]` over the JSON body. Iterating the envelope dict yields its **keys**, so the first call is `MethodData.model_validate("items")` → `pydantic.ValidationError` on every invocation. `list_runs` (`client.py:946`) fails identically. +- The unit tests mock the pre-paging bare arrays (`tests/unit/test_client_product.py:85`, `:379`), which is why nothing is red. +- `PipelineRun` (`product_models.py:369-370`) declares `method_id: str` and `pipe_code: str`; the platform's `RunPublic` declares both `str | None = None`, and both are genuinely null in practice (an ad-hoc run from an inline bundle; a pipe resolved from `main_pipe`). Once the envelope is fixed, the first such row raises. +- `@pipelex/sdk` took all three in 0.10.0 (`listRuns` → `RunPage`, `iterateRuns`, `getRunDetail`, nullable `PipelineRun` fields) and 0.11.0 (`listMethods` → `MethodPage`, `iterateMethods`, `MethodSummary`). `docs/architecture.md` here still claims full parity ("surface-complete, with no silent gaps"), which has been false since 2026-08-11. + +### 3.2 Design + +Breaking, and mirroring the JS shapes — with the wire kept snake_case, so the envelope field is `next_cursor` here where JS renamed it `nextCursor` for its own consumers. + +**Models (`product_models.py`):** + +- `MethodSummary(extra="allow")`: `method_id`, `name`, `description: str | None = None`, `created_at`, `deletion_state: MethodDeletionState | None = None`. Deliberately not a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none is in the index projection and putting `mthds` back is what restored the truncation bug. +- `MethodPage(extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. No total, by design. +- `RunPage(extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. +- `RunErrorReport(extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. +- `PipelineRun`: `method_id: str | None = None`, `pipe_code: str | None = None`; add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. `pipe_statuses` stays as it is (the JS model keeps it optional; the platform's `RunPublic` no longer declares it, and `extra="allow"` covers either way). +- `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None` — the only read that carries what the run actually executed. +- `MethodData`: add `org_id: str`, `created_by_user_id: str` (required on the platform's `MethodPublic` and in the JS model), `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, and `python: list[MethodFile] = Field(default_factory=list)`. See the `python` decision below. +- `MethodWriteInput`: add `python: list[MethodFile] | None = None`. Because the write body is dumped with `exclude_none=True`, the platform's three-way contract falls out naturally: `None` → not sent → the stored Python is preserved; `[]` → serialized as `""` → clears it; a non-empty list → replaces it. Document that on the field. + +**Client (`client.py`):** + +- `list_methods(*, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage`. Query params are added on **presence** (`is not None`), never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to silently drop into an unfiltered query that reads as working. Encode with `urllib.parse.urlencode` rather than string formatting; `q` is free text. +- `iterate_methods(*, q=None, limit=None) -> AsyncIterator[MethodSummary]` — an `async def` generator that follows the cursor. It must keep going **past empty pages** (`q` is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` means "keep going"), stop on `next_cursor is None`, stop when the server hands back the cursor it was sent (checked before yielding, so rows are never double-counted), and **raise** rather than return past a runaway page ceiling set far beyond any real catalog. Deliberately not a `list_all_methods() -> list[…]`: an all-at-once helper needs a cap, and a cap means silently returning a truncated list — the exact bug paging removed. +- `list_runs(method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage`. `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), not days; a naive timestamp is a platform `400`, surfaced as `ApiResponseError`. Same presence semantics. +- `iterate_runs(method_id, *, created_from=None, created_to=None, limit=None) -> AsyncIterator[PipelineRun]` — same loop, except an empty page **does** end it: the date bounds are index key conditions, so a run page is never empty-with-a-cursor. The difference is the server, not the client, and the docstring should say so. +- `get_run_detail(run_id: str) -> RunDetail` — `GET /v1/runs/{id}`, path-encoded like the other id routes. +- One thing to document that the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()` gate (`pipelex-server/platform/src/pipelex_platform/deps.py:345`), which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403`. That arrives here as an `ApiResponseError`, and a reader of `list_runs` should know a `403` means "flag", not "wrong key". + +**The `python` field.** On the wire `MethodPublic.python` is one string: the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python (`pipelex-server/shared/src/pipelex_shared/schemas/method.py:209`, `:251`), and the write side is the same string three ways (omitted → preserve, `""` → clear, text → replace). `@pipelex/sdk` never shows that string to callers: it exposes `MethodFile[]` and converts at the client boundary (`pipelex-sdk-js/src/client.ts:281` on read, `:292` on write) with `parseMethodFiles` / `serializeMethodFiles` from `mthds-js/src/protocol/method_files.ts`. That module is small — parse the JSON, check every entry is `{name: str, content: str}`, drop blank-content entries, and serialize an empty list as `""` rather than `"[]"` because `""` is the platform's clear sentinel. It lives in `mthds/protocol` on the JS side because `pipelex-mcp` consumes the same format and wanted one owner. + +The first draft of this document proposed exposing the raw wire string and asking `mthds-python` for the converter. That was over-engineered: with pydantic the whole converter is a `MethodFile` model plus a `TypeAdapter(list[MethodFile])` and the two sentinel rules, there is no second Python consumer that could drift, and the JS module's own docstring calls this the format "the hosted platform persists" — a Pipelex catalog concern, so this SDK is a proper home for it. **Decision: typed list, converter here.** `product_models.py` gains `MethodFile(name: str, content: str)` and a `parse_method_files(source: str | None) -> list[MethodFile]` / `serialize_method_files(files: list[MethodFile]) -> str` pair carrying the same rules as the JS pair (blank source or `"[]"` → `[]`; blank-content entries dropped on both directions; empty list → `""`). `MethodData` applies the parser through a `field_validator("python", mode="before")`, so `MethodData.model_validate(body)` keeps working unchanged at every call site; `MethodWriteInput` applies the serializer through a `field_serializer("python")`, so the write body still dumps with `exclude_none=True` and the three-way contract holds. Malformed wire text raises `ValueError` inside the validator and therefore surfaces as a `pydantic.ValidationError`, the same way any other malformed response body fails here. If the format ever gains a Python owner in `mthds-python`, this SDK adopts it then; nothing is filed to the inbox for it. + +`get_method_closure` (JS-only client-side sugar that parses the polymorphic `mthds` source into a run-ready closure) stays deferred: it is not moved by any of the cited releases, and the 0.5.0 changelog already recorded it as "deferred and additive" alongside `prepare_inputs`. + +### 3.3 Tests (`tests/unit/test_client_product.py`) + +Replace the two bare-array fixtures with envelopes and add: query encoding for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped; a null `pipe_code` / `method_id` row parsing; `get_run_detail` returning `mthds_contents` and `inputs`; `MethodData` carrying the new fields, with `python` parsed from the wire string into `MethodFile` entries and `""` reading as an empty list; `MethodWriteInput.python` three-way serialization (`None` absent, `[]` sent as `""`, a list sent as the JSON text); the `parse_method_files` / `serialize_method_files` pair round-tripping, dropping blank-content entries, and rejecting a non-array or a malformed entry. For the iterators, in a dedicated module (one `TestClass` per module): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding; `iterate_runs` stops on an empty page; `iterate_methods` raises past the ceiling. + +## 4. Boundary type validation for `method_id` (decision of 2026-08-25) + +`pipelex-sdk-js/wip/boundary-option-type-validation.md` records the decision, taken by Louis, that a published client validates request-option types at its boundary and throws `PipelineRequestError` rather than dropping or forwarding a wrong-typed value. Its evidence section cites this repo directly: `client.py:1003` is a bare `if method_id:`, which drops falsy non-strings (`0`, `[]`) and forwards truthy ones (`123`, `["mt_1"]`) to a server `422` — a *different* partition of wrong values than the JS client makes for the same argument on the same wire. The plan's Phase 2 names the fix: an explicit `is not None` presence check followed by an `isinstance(method_id, str)` check that raises, with `None` and `""` still normalizing to absent. + +The plan sequences the SDKs after the protocol packages so both inherit one behaviour for the protocol-level arguments. That ordering matters for `pipe_code` / `mthds_contents`, whose guards belong in `mthds-python`; it does not constrain `method_id`, which this layer owns outright and whose guard touches only `_merge_hosted_run_extensions` (`client.py:975`). **Recommendation: land the `method_id` guard in this update** — it is a few lines, this branch is already the `method_id` branch, and it closes the repo-specific finding in the JS wip doc — and take the protocol-argument guards later with the `mthds` floor bump once `mthds-python` ships its Phase 1. Decided 2026-08-25: it lands now (§7). + +## 5. Checked, no change needed + +- **pipelex-api 0.17.0's source-less `422` naming unhandled keys** (the `method_id`-at-a-bare-runner diagnosis): this SDK already forwards `method_id` on the blocking fallback precisely so that message reaches the caller; the `execute` docstring already describes it. +- **`storage_scope` / `callback_urls` / `orchestration_mode`** (0.15.0–0.16.0): layer-2 fields the hosted platform sends to the runner; not caller-facing and not an SDK concern. +- **The four OpenAPI schemas that went opaque, the `RunMetadata` split, the `.pipelex/` config schema, and the two authoring changes** (`required = true` + `default_value` rejected; unknown structure-field keys rejected): server-side and inside opaque dicts here; no wire field this SDK types moved. +- **`views` in `mthds-python`**: not needed. It is a Pipelex-API carriage extension exactly like `render`, and the base client's `extra` passthrough already carries it (§1.2). `mthds-js` likewise has no `views`. +- **Explicit-null locators** (JS 0.14.0's `T | null` widening): already `T | None = None` here (§1.1); only a regression test is owed. +- **`ValidationErrorItem.error_type` narrowing** to the new enum members: stays an open `str` by design (§1.4). +- **The 0.14.0 `validate()` positional break**: Python takes `views` by keyword after `render`; no positional call moves. +- **`method_id` typed option and `delete_method`**: already on this branch (§2). + +## 6. Change plan + +Four commits on this branch, each self-contained, each with its docs and changelog lines, each gated on `make agent-check` and `make agent-test`: + +1. **Validate surface** (§1): `validation_models.py` (new fields, `LiftablePipeEntry`, the fix vocabulary), `client.py` (`views` on `validate` / `validate_files`), the two test modules, `docs/architecture.md` (the validate section and the brand-boundary field list gain `warnings` / `liftable_pipes` / `input_form`, and a sentence on the opaque `presence` / `fixed` spellings), `README.md` quickstart mention of `views`. Changelog: **Added** (`views`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary), with a note that an older runner's body still parses. +2. **Prose corrections** (§2): the attribution and citation edits in `runs.py`, `validation_models.py`, `client.py`, the three test docstrings, `docs/run-usage.md`, `docs/architecture.md`. Changelog: **Fixed**, two entries mirroring 0.14.0's wording. +3. **Product paging and nullability** (§3): `product_models.py`, `client.py` (`list_methods`, `iterate_methods`, `list_runs`, `iterate_runs`, `get_run_detail`), `tests/unit/test_client_product.py` plus a new iterator test module, `docs/architecture.md` (product surface section rewritten for pages, the "Parity with `@pipelex/sdk`" section corrected — it must stop claiming surface-completeness and list the conscious exclusions honestly), `README.md` if it gains a listing example. Changelog: **Changed (breaking)** for the two return types and the nullable `PipelineRun` fields, **Added** for the iterators, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the `MethodData` fields and `MethodWriteInput.python`, **Fixed** naming the crash. +4. **`method_id` type guard** (§4): `_merge_hosted_run_extensions`, one wrong-type parametrized test in `test_client_method_id.py`. Changelog: **Changed**. + +The version stays where it is under `## [Unreleased]` until `/release` cuts it; with the breaking items in commit 3 (and the ones already on the branch), that release is a minor bump. + +**Known gaps this plan leaves open, on purpose:** no e2e suite exists in this repo, so the live `views` gate and the live paging envelope are pinned only by mocked bodies here (the JS suite pins both live); the remaining `@pipelex/sdk` surfaces without a Python counterpart — `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — are unchanged by the cited releases and stay the conscious exclusions `docs/architecture.md` already records. + +## 7. Decisions (2026-08-25) + +The four questions the first draft left open, each answered by Louis on 2026-08-25 with the reasoning that settled it. + +1. **`input_form` stays opaque** — `dict[str, Any] | None = None`, matching the JS mirror and the ownership argument in §1.3. A `PipeInputFormDescriptor(fields: list[dict])` shell would type one level and still leave the field vocabulary opaque, which buys little. +2. **`MethodData.python` / `MethodWriteInput.python` are typed `list[MethodFile]`, with the converter in this repo.** The question was first posed as "raw wire string plus an inbox request to `mthds-python` for the parser", and the answer to "why do we need a parser at all?" dissolved that framing: the converter is a dozen lines of pydantic, the format is a Pipelex catalog concern rather than an MTHDS protocol one, and there is no second Python consumer to keep in step. Full design in §3.2; no inbox item is filed. +3. **The `method_id` type guard lands now**, in this update (§4). The protocol-argument guards for `pipe_code` / `mthds_contents` still wait for `mthds-python` to ship its Phase 1 and arrive here with the `mthds` floor bump. +4. **An unknown `FixOp.kind` raises** — closed `Literal` discriminator, `pydantic.ValidationError` on the whole verdict parse, consistent with the closed `ValidationErrorCategory`. The lenient catch-all alternative described in §1.4 was considered and not taken.