From 69568f14d0a35bdc3181bbd0932905dffad94cfe Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 21 Aug 2026 10:59:45 +0200 Subject: [PATCH 01/20] Move the ruff pin to 0.16.2 and take the lint fallout Matches the version the Ruff VS Code extension bundles, so the editor's language server and the CLI run the same binary. The fallout is auto-fix output that make lint would re-apply anyway: selector lists now name rules instead of coding them, and noqa suppressions moved to the new ruff: ignore[...] directive. float-equality-comparison, new under preview, joins the existing tests/ relaxations in per-file-ignores. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 8 ++ README.md | 1 + pipelex_sdk/errors.py | 2 +- pyproject.toml | 241 +++++++++++++++++++++--------------------- uv.lock | 45 ++++---- 5 files changed, 153 insertions(+), 144 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 596169d..77696a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## [Unreleased] + +### Changed + +- **Ruff moves to 0.16.2, matching the version the VS Code extension bundles.** The extension now syncs `pyproject.toml` and `ruff.toml` documents to the language server, because Ruff from 0.16 on lints its own config files. A binary older than that parses those documents as Python source and paints phantom `invalid-syntax` diagnostics across every `pyproject.toml`. Pinning the dev dependency to the exact version the extension ships keeps the editor and the command line on one binary. This is a dev-dependency change: nothing shipped changes, and there is no version bump. + + Two mechanical conversions ride along, both applied by `ruff check --fix` and therefore not deferrable, since `make lint` would re-apply them on first contact: selector lists in `pyproject.toml` now name rules rather than code them, and `# noqa: CODE` suppressions became `# ruff: ignore[rule-name]`. `float-equality-comparison`, newly reported under `preview`, joins the existing `tests/` relaxations — those tests assert float literals that round-trip exactly, so `pytest.approx` would only add noise. + ## [v0.5.0] - 2026-07-22 ### Added diff --git a/README.md b/README.md index 5759113..f32aeea 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ The client is **async-only** (httpx `AsyncClient`) and is an async context manag ```python from pipelex_sdk.client import PipelexAPIClient + async def main() -> None: async with PipelexAPIClient() as client: # 1. Validate an MTHDS bundle. The verdict is always returned (never raised): diff --git a/pipelex_sdk/errors.py b/pipelex_sdk/errors.py index f7afb63..7a8dc2b 100644 --- a/pipelex_sdk/errors.py +++ b/pipelex_sdk/errors.py @@ -28,7 +28,7 @@ # Explicit re-export (PEP 484 `as` self-alias): the protocol 202-degrade error stays owned by # `mthds`, surfaced here so consumers have a single import home for the run/lifecycle errors. -from mthds.runners.api.exceptions import RunStillRunningError as RunStillRunningError # noqa: PLC0414 +from mthds.runners.api.exceptions import RunStillRunningError as RunStillRunningError # ruff: ignore[useless-import-alias] if TYPE_CHECKING: from pipelex_sdk.runs import RunStatus diff --git a/pyproject.toml b/pyproject.toml index 9769741..505489d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -29,7 +29,7 @@ dev = [ "mypy==1.19.1", "pyright==1.1.411", "pylint==4.0.4", - "ruff==0.14.13", + "ruff==0.16.2", "pytest>=9.0.3", "pytest-mock>=3.12.0,<4.0.0", "pytest-sugar>=1.0.0", @@ -190,121 +190,121 @@ target-version = "py311" preview = true select = ["ALL"] ignore = [ - "ANN201", # Missing return type annotation for public function `my_func` - "ANN202", # Missing return type annotation for private function `my_func` - "ANN204", # Missing return type annotation for special method `my_func` - "ANN206", # Missing return type annotation for classmethod `my_func` - "ANN401", # Dynamically typed expressions (typing.Any) are disallowed in `...` - "ASYNC230", # Async functions should not open files with blocking methods like `open` - "ASYNC240", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path - - "B903", # Class could be dataclass or namedtuple - - "C901", # Is to complex - "COM812", # Checks for the absence of trailing commas. - - "CPY001", # Missing copyright notice at top of file - - "D100", # Missing docstring in public module - "D101", # Missing docstring in public class - "D102", # Missing docstring in public method - "D103", # Missing docstring in public function - "D104", # Missing docstring in public package - "D105", # Missing docstring in magic method - "D107", # Missing docstring in __init__ - "D205", # 1 blank line required between summary line and description - "D400", # First line should end with a period - "D401", # First line of docstring should be in imperative mood: "My docstring...." - "D404", # First word of the docstring should not be "This" - "D415", # First line should end with a period, question mark, or exclamation point - - "DOC201", # `return` is not documented in docstring - "DOC202", # Docstring should not have a returns section because the function doesn't return anything - "DOC402", # `yield` is not documented in docstring - "DOC502", # Raised exception is not explicitly raised: `FileNotFoundError` - "DOC501", # Raised exception `ModuleFileError` missing from docstring - - "DTZ001", # `datetime.datetime()` called without a `tzinfo` argument - "DTZ005", # `datetime.datetime.now()` called without a `tz` argument - - "ERA001", # Found commented-out code - - "FBT001", # Boolean-typed positional argument in function definition - "FBT002", # Boolean default positional argument in function definition - "FBT003", #Boolean positional value in function call - - "FIX002", # Line contains TODO, consider resolving the issue - - "FURB101", # `open` and `read` should be replaced by `Path(file_path.path).read_text(encoding="utf-8")` - "FURB113", # Checks for consecutive calls to append. - "FURB152", # Checks for literals that are similar to constants in math module. - - "LOG004", # `.exception()` call outside exception handlers - - "PLC0105", # `TypeVar` name "SomethingType" does not reflect its covariance; consider renaming it to "SomethingType_co" - "PLC1901", # Checks for comparisons to empty strings. - - "PLR0904", # Too many public methods ( > 20) - "PLR0911", # Too many return statements (/6) - "PLR0912", # Too many branches (/12) - "PLR0913", # Too many arguments in function definition (/5) - "PLR0914", # Too many local variables ( /15) - "PLR0915", # Too many statements (/50) - "PLR0917", # Too many positional arguments ( /5) - "PLR2004", # Magic value used in comparison, consider replacing `2` with a constant variable - "PLR6301", # Too many return statements in `for` loop - "PLR1702", # Too many nested blocks ( > 5) - - "PT013", # Incorrect import of `pytest`; use `import pytest` instead - - "PTH100", # `os.path.abspath()` should be replaced by `Path.resolve()` - "PTH103", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` - "PTH107", # `os.remove()` should be replaced by `Path.unlink()` - "PTH109", # `os.getcwd()` should be replaced by `Path.cwd()` - "PTH118", # `os.path.join()` should be replaced by `Path` with `/` operator - "PTH120", # `os.path.dirname()` should be replaced by `Path.parent` - "PTH110", # `os.path.exists()` should be replaced by `Path.exists()` - "PTH112", # `os.path.isdir()` should be replaced by `Path.is_dir()` - "PTH119", # `os.path.basename()` should be replaced by `Path.name` - "PTH123", # `open()` should be replaced by `Path.open()` - "PTH208", # Use `pathlib.Path.iterdir()` instead. - - "PYI051", # `Literal["auto"]` is redundant in a union with `str` - - "RET505", # superfluous-else-return - - "RUF001", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? - "RUF003", # Comment contains ambiguous `’` (RIGHT SINGLE QUOTATION MARK). Did you mean ``` (GRAVE ACCENT)? - "RUF022", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. - - "SIM105", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` - "SIM108", # Use ternary operator `description = func.__doc__.strip().split("\n")[0] if func.__doc__ else func.__name__` instead of `if`-`else`-block - - "S101", # Use of `assert` detected - "S102", # Use of `exec` detected - "S106", # Possible hardcoded password assigned to argument: "secret" - "S105", # Possible hardcoded password assigned to: "child_secret" - - "S311", # Cryptographically weak pseudo-random number generator - - "TD002", # Missing author in TODO; try: `# TODO(): ...` or `# TODO @: ...` - "TD003", # Missing issue link for this TODO - - "T201", # `print` found + "missing-return-type-undocumented-public-function", # Missing return type annotation for public function `my_func` + "missing-return-type-private-function", # Missing return type annotation for private function `my_func` + "missing-return-type-special-method", # Missing return type annotation for special method `my_func` + "missing-return-type-class-method", # Missing return type annotation for classmethod `my_func` + "any-type", # Dynamically typed expressions (typing.Any) are disallowed in `...` + "blocking-open-call-in-async-function", # Async functions should not open files with blocking methods like `open` + "blocking-path-method-in-async-function", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path + + "class-as-data-structure", # Class could be dataclass or namedtuple + + "complex-structure", # Is to complex + "missing-trailing-comma", # Checks for the absence of trailing commas. + + "missing-copyright-notice", # Missing copyright notice at top of file + + "undocumented-public-module", # Missing docstring in public module + "undocumented-public-class", # Missing docstring in public class + "undocumented-public-method", # Missing docstring in public method + "undocumented-public-function", # Missing docstring in public function + "undocumented-public-package", # Missing docstring in public package + "undocumented-magic-method", # Missing docstring in magic method + "undocumented-public-init", # Missing docstring in __init__ + "missing-blank-line-after-summary", # 1 blank line required between summary line and description + "missing-trailing-period", # First line should end with a period + "non-imperative-mood", # First line of docstring should be in imperative mood: "My docstring...." + "docstring-starts-with-this", # First word of the docstring should not be "This" + "missing-terminal-punctuation", # First line should end with a period, question mark, or exclamation point + + "docstring-missing-returns", # `return` is not documented in docstring + "docstring-extraneous-returns", # Docstring should not have a returns section because the function doesn't return anything + "docstring-missing-yields", # `yield` is not documented in docstring + "docstring-extraneous-exception", # Raised exception is not explicitly raised: `FileNotFoundError` + "docstring-missing-exception", # Raised exception `ModuleFileError` missing from docstring + + "call-datetime-without-tzinfo", # `datetime.datetime()` called without a `tzinfo` argument + "call-datetime-now-without-tzinfo", # `datetime.datetime.now()` called without a `tz` argument + + "commented-out-code", # Found commented-out code + + "boolean-type-hint-positional-argument", # Boolean-typed positional argument in function definition + "boolean-default-value-positional-argument", # Boolean default positional argument in function definition + "boolean-positional-value-in-call", #Boolean positional value in function call + + "line-contains-todo", # Line contains TODO, consider resolving the issue + + "read-whole-file", # `open` and `read` should be replaced by `Path(file_path.path).read_text(encoding="utf-8")` + "repeated-append", # Checks for consecutive calls to append. + "math-constant", # Checks for literals that are similar to constants in math module. + + "log-exception-outside-except-handler", # `.exception()` call outside exception handlers + + "type-name-incorrect-variance", # `TypeVar` name "SomethingType" does not reflect its covariance; consider renaming it to "SomethingType_co" + "compare-to-empty-string", # Checks for comparisons to empty strings. + + "too-many-public-methods", # Too many public methods ( > 20) + "too-many-return-statements", # Too many return statements (/6) + "too-many-branches", # Too many branches (/12) + "too-many-arguments", # Too many arguments in function definition (/5) + "too-many-locals", # Too many local variables ( /15) + "too-many-statements", # Too many statements (/50) + "too-many-positional-arguments", # Too many positional arguments ( /5) + "magic-value-comparison", # Magic value used in comparison, consider replacing `2` with a constant variable + "no-self-use", # Too many return statements in `for` loop + "too-many-nested-blocks", # Too many nested blocks ( > 5) + + "pytest-incorrect-pytest-import", # Incorrect import of `pytest`; use `import pytest` instead + + "os-path-abspath", # `os.path.abspath()` should be replaced by `Path.resolve()` + "os-makedirs", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` + "os-remove", # `os.remove()` should be replaced by `Path.unlink()` + "os-getcwd", # `os.getcwd()` should be replaced by `Path.cwd()` + "os-path-join", # `os.path.join()` should be replaced by `Path` with `/` operator + "os-path-dirname", # `os.path.dirname()` should be replaced by `Path.parent` + "os-path-exists", # `os.path.exists()` should be replaced by `Path.exists()` + "os-path-isdir", # `os.path.isdir()` should be replaced by `Path.is_dir()` + "os-path-basename", # `os.path.basename()` should be replaced by `Path.name` + "builtin-open", # `open()` should be replaced by `Path.open()` + "os-listdir", # Use `pathlib.Path.iterdir()` instead. + + "redundant-literal-union", # `Literal["auto"]` is redundant in a union with `str` + + "superfluous-else-return", # superfluous-else-return + + "ambiguous-unicode-character-string", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? + "ambiguous-unicode-character-comment", # Comment contains ambiguous `’` (RIGHT SINGLE QUOTATION MARK). Did you mean ``` (GRAVE ACCENT)? + "unsorted-dunder-all", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. + + "suppressible-exception", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` + "if-else-block-instead-of-if-exp", # Use ternary operator `description = func.__doc__.strip().split("\n")[0] if func.__doc__ else func.__name__` instead of `if`-`else`-block + + "assert", # Use of `assert` detected + "exec-builtin", # Use of `exec` detected + "hardcoded-password-func-arg", # Possible hardcoded password assigned to argument: "secret" + "hardcoded-password-string", # Possible hardcoded password assigned to: "child_secret" + + "suspicious-non-cryptographic-random-usage", # Cryptographically weak pseudo-random number generator + + "missing-todo-author", # Missing author in TODO; try: `# TODO(): ...` or `# TODO @: ...` + "missing-todo-link", # Missing issue link for this TODO + + "print", # `print` found # TODO: stop ignoring these rules - "BLE001", # Do not catch blind exception: `Exception` - "B027", # Checks for empty methods in abstract base classes without an abstract decorator. - "UP007", # Use `X | Y` for type annotations - "UP036", # Version block is outdated for minimum Python version - "SIM102", # Use a single `if` statement instead of nested `if` statements - "S701", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. - "TRY301", # Abstract `raise` to an inner function - "PERF401", # Use a list comprehension to create a transformed list - "PLW2901", # `for` loop variable `line` overwritten by assignment target - "TRY300", # Consider moving this statement to an `else` block - "UP035", # `typing.List` is deprecated, use `list` instead - "RET503", # Missing explicit `return` at the end of function able to return non-`None` value + "blind-except", # Do not catch blind exception: `Exception` + "empty-method-without-abstract-decorator", # Checks for empty methods in abstract base classes without an abstract decorator. + "non-pep604-annotation-union", # Use `X | Y` for type annotations + "outdated-version-block", # Version block is outdated for minimum Python version + "collapsible-if", # Use a single `if` statement instead of nested `if` statements + "jinja2-autoescape-false", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. + "raise-within-try", # Abstract `raise` to an inner function + "manual-list-comprehension", # Use a list comprehension to create a transformed list + "redefined-loop-name", # `for` loop variable `line` overwritten by assignment target + "try-consider-else", # Consider moving this statement to an `else` block + "deprecated-import", # `typing.List` is deprecated, use `list` instead + "implicit-return", # Missing explicit `return` at the end of function able to return non-`None` value ] [tool.ruff.lint.flake8-type-checking] @@ -315,14 +315,15 @@ convention = "google" [tool.ruff.lint.per-file-ignores] "tests/**/*.py" = [ - "INP001", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) - "SLF001", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) - "PLC2701", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) - "ARG002", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) + "private-member-access", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) + "import-private-name", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) + "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise ] "examples/**/*.py" = [ - "INP001", # Runnable demo scripts, not an importable package - "T201", # print() is the whole point of a demo script + "implicit-namespace-package", # Runnable demo scripts, not an importable package + "print", # print() is the whole point of a demo script ] [tool.uv] diff --git a/uv.lock b/uv.lock index 9e47f0e..bb57083 100644 --- a/uv.lock +++ b/uv.lock @@ -334,7 +334,7 @@ requires-dist = [ { name = "pytest", marker = "extra == 'dev'", specifier = ">=9.0.3" }, { name = "pytest-mock", marker = "extra == 'dev'", specifier = ">=3.12.0,<4.0.0" }, { name = "pytest-sugar", marker = "extra == 'dev'", specifier = ">=1.0.0" }, - { name = "ruff", marker = "extra == 'dev'", specifier = "==0.14.13" }, + { name = "ruff", marker = "extra == 'dev'", specifier = "==0.16.2" }, { name = "typing-extensions", specifier = ">=4.0.0" }, ] provides-extras = ["dev"] @@ -557,28 +557,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.14.13" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/50/0a/1914efb7903174b381ee2ffeebb4253e729de57f114e63595114c8ca451f/ruff-0.14.13.tar.gz", hash = "sha256:83cd6c0763190784b99650a20fec7633c59f6ebe41c5cc9d45ee42749563ad47", size = 6059504, upload-time = "2026-01-15T20:15:16.918Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c3/ae/0deefbc65ca74b0ab1fd3917f94dc3b398233346a74b8bbb0a916a1a6bf6/ruff-0.14.13-py3-none-linux_armv6l.whl", hash = "sha256:76f62c62cd37c276cb03a275b198c7c15bd1d60c989f944db08a8c1c2dbec18b", size = 13062418, upload-time = "2026-01-15T20:14:50.779Z" }, - { url = "https://files.pythonhosted.org/packages/47/df/5916604faa530a97a3c154c62a81cb6b735c0cb05d1e26d5ad0f0c8ac48a/ruff-0.14.13-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:914a8023ece0528d5cc33f5a684f5f38199bbb566a04815c2c211d8f40b5d0ed", size = 13442344, upload-time = "2026-01-15T20:15:07.94Z" }, - { url = "https://files.pythonhosted.org/packages/4c/f3/e0e694dd69163c3a1671e102aa574a50357536f18a33375050334d5cd517/ruff-0.14.13-py3-none-macosx_11_0_arm64.whl", hash = "sha256:d24899478c35ebfa730597a4a775d430ad0d5631b8647a3ab368c29b7e7bd063", size = 12354720, upload-time = "2026-01-15T20:15:09.854Z" }, - { url = "https://files.pythonhosted.org/packages/c3/e8/67f5fcbbaee25e8fc3b56cc33e9892eca7ffe09f773c8e5907757a7e3bdb/ruff-0.14.13-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9aaf3870f14d925bbaf18b8a2347ee0ae7d95a2e490e4d4aea6813ed15ebc80e", size = 12774493, upload-time = "2026-01-15T20:15:20.908Z" }, - { url = "https://files.pythonhosted.org/packages/6b/ce/d2e9cb510870b52a9565d885c0d7668cc050e30fa2c8ac3fb1fda15c083d/ruff-0.14.13-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ac5b7f63dd3b27cc811850f5ffd8fff845b00ad70e60b043aabf8d6ecc304e09", size = 12815174, upload-time = "2026-01-15T20:15:05.74Z" }, - { url = "https://files.pythonhosted.org/packages/88/00/c38e5da58beebcf4fa32d0ddd993b63dfacefd02ab7922614231330845bf/ruff-0.14.13-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:78d2b1097750d90ba82ce4ba676e85230a0ed694178ca5e61aa9b459970b3eb9", size = 13680909, upload-time = "2026-01-15T20:15:14.537Z" }, - { url = "https://files.pythonhosted.org/packages/61/61/cd37c9dd5bd0a3099ba79b2a5899ad417d8f3b04038810b0501a80814fd7/ruff-0.14.13-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:7d0bf87705acbbcb8d4c24b2d77fbb73d40210a95c3903b443cd9e30824a5032", size = 15144215, upload-time = "2026-01-15T20:15:22.886Z" }, - { url = "https://files.pythonhosted.org/packages/56/8a/85502d7edbf98c2df7b8876f316c0157359165e16cdf98507c65c8d07d3d/ruff-0.14.13-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a3eb5da8e2c9e9f13431032fdcbe7681de9ceda5835efee3269417c13f1fed5c", size = 14706067, upload-time = "2026-01-15T20:14:48.271Z" }, - { url = "https://files.pythonhosted.org/packages/7e/2f/de0df127feb2ee8c1e54354dc1179b4a23798f0866019528c938ba439aca/ruff-0.14.13-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:642442b42957093811cd8d2140dfadd19c7417030a7a68cf8d51fcdd5f217427", size = 14133916, upload-time = "2026-01-15T20:14:57.357Z" }, - { url = "https://files.pythonhosted.org/packages/0d/77/9b99686bb9fe07a757c82f6f95e555c7a47801a9305576a9c67e0a31d280/ruff-0.14.13-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4acdf009f32b46f6e8864af19cbf6841eaaed8638e65c8dac845aea0d703c841", size = 13859207, upload-time = "2026-01-15T20:14:55.111Z" }, - { url = "https://files.pythonhosted.org/packages/7d/46/2bdcb34a87a179a4d23022d818c1c236cb40e477faf0d7c9afb6813e5876/ruff-0.14.13-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:591a7f68860ea4e003917d19b5c4f5ac39ff558f162dc753a2c5de897fd5502c", size = 14043686, upload-time = "2026-01-15T20:14:52.841Z" }, - { url = "https://files.pythonhosted.org/packages/1a/a9/5c6a4f56a0512c691cf143371bcf60505ed0f0860f24a85da8bd123b2bf1/ruff-0.14.13-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:774c77e841cc6e046fc3e91623ce0903d1cd07e3a36b1a9fe79b81dab3de506b", size = 12663837, upload-time = "2026-01-15T20:15:18.921Z" }, - { url = "https://files.pythonhosted.org/packages/fe/bb/b920016ece7651fa7fcd335d9d199306665486694d4361547ccb19394c44/ruff-0.14.13-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:61f4e40077a1248436772bb6512db5fc4457fe4c49e7a94ea7c5088655dd21ae", size = 12805867, upload-time = "2026-01-15T20:14:59.272Z" }, - { url = "https://files.pythonhosted.org/packages/7d/b3/0bd909851e5696cd21e32a8fc25727e5f58f1934b3596975503e6e85415c/ruff-0.14.13-py3-none-musllinux_1_2_i686.whl", hash = "sha256:6d02f1428357fae9e98ac7aa94b7e966fd24151088510d32cf6f902d6c09235e", size = 13208528, upload-time = "2026-01-15T20:15:03.732Z" }, - { url = "https://files.pythonhosted.org/packages/3b/3b/e2d94cb613f6bbd5155a75cbe072813756363eba46a3f2177a1fcd0cd670/ruff-0.14.13-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:e399341472ce15237be0c0ae5fbceca4b04cd9bebab1a2b2c979e015455d8f0c", size = 13929242, upload-time = "2026-01-15T20:15:11.918Z" }, - { url = "https://files.pythonhosted.org/packages/6a/c5/abd840d4132fd51a12f594934af5eba1d5d27298a6f5b5d6c3be45301caf/ruff-0.14.13-py3-none-win32.whl", hash = "sha256:ef720f529aec113968b45dfdb838ac8934e519711da53a0456038a0efecbd680", size = 12919024, upload-time = "2026-01-15T20:14:43.647Z" }, - { url = "https://files.pythonhosted.org/packages/c2/55/6384b0b8ce731b6e2ade2b5449bf07c0e4c31e8a2e68ea65b3bafadcecc5/ruff-0.14.13-py3-none-win_amd64.whl", hash = "sha256:6070bd026e409734b9257e03e3ef18c6e1a216f0435c6751d7a8ec69cb59abef", size = 14097887, upload-time = "2026-01-15T20:15:01.48Z" }, - { url = "https://files.pythonhosted.org/packages/4d/e1/7348090988095e4e39560cfc2f7555b1b2a7357deba19167b600fdf5215d/ruff-0.14.13-py3-none-win_arm64.whl", hash = "sha256:7ab819e14f1ad9fe39f246cfcc435880ef7a9390d81a2b6ac7e01039083dd247", size = 13080224, upload-time = "2026-01-15T20:14:45.853Z" }, +version = "0.16.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/73/e1/4508a569211b35599016e84ba65c1a992b7a4004b4b6c4bea02a851cba1b/ruff-0.16.2.tar.gz", hash = "sha256:c3d7828d12e8927a6fc65fe38e2c2541b9e762d360a1786d752cb1b8883b3c9c", size = 4885811, upload-time = "2026-08-07T13:31:01.432Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/14/57/db19951540f98859c956b50bdb4d31089b4d91e9f15e2968e7d5193806d5/ruff-0.16.2-py3-none-linux_armv6l.whl", hash = "sha256:3c8de4cf2181f01d57946d87d777aa52916976fc09942aed89938fab5e013318", size = 10847925, upload-time = "2026-08-07T13:30:14.468Z" }, + { url = "https://files.pythonhosted.org/packages/13/5a/995fe85a8470d3e391ac0f7fa8054bb454eaf33ee138196d6172ed1079c0/ruff-0.16.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:9a48cc05c6fbc811ca81b5d7ba95375affea6582d1b8024e455e41afbbf55344", size = 11072662, upload-time = "2026-08-07T13:30:18.143Z" }, + { url = "https://files.pythonhosted.org/packages/32/53/370d767c61c71a971a4ace36703a7ecd8c393956349a7325d7fab2b56827/ruff-0.16.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:a2c0d14fcbb26c91f0f867a6dc9bd71bbc30b1b6151829c884f23faeab2e5700", size = 10566771, upload-time = "2026-08-07T13:30:20.899Z" }, + { url = "https://files.pythonhosted.org/packages/85/d6/9d96948caf5a632be62d62202d5ec914d6856f204fd79eb036e5915e79ea/ruff-0.16.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:335c621622c4650330be50842561c6586ac6971bb8ab5407fe34dcc9efb16bbe", size = 10975825, upload-time = "2026-08-07T13:30:23.517Z" }, + { url = "https://files.pythonhosted.org/packages/3b/92/ea87129b3414acb0b5770563779c51804d37ac67675c7ba35447ddb14773/ruff-0.16.2-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:20e66910f2c37cc753f9ef6580c914a621b80c4fa3549d3e3521e29d0f5bfc3f", size = 10649437, upload-time = "2026-08-07T13:30:26.097Z" }, + { url = "https://files.pythonhosted.org/packages/ac/43/f8f291dcd4af5bb7872b74fdfa41a7cd7c856ca1d4069670971cf1b9f5cb/ruff-0.16.2-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c7e36fbfba65510548156902bcf1350a979a958ce0347ce0f90d73894036b39f", size = 11446761, upload-time = "2026-08-07T13:30:28.752Z" }, + { url = "https://files.pythonhosted.org/packages/71/4a/ef991fb2fcf516ab71f0808adcdd8da5e18c8cde447f4ceaf5f47a5132a5/ruff-0.16.2-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f0eab35f80df8f134aae5d1630e751901321d317cc8e50dc39e36fa3ed34cd12", size = 12336364, upload-time = "2026-08-07T13:30:31.468Z" }, + { url = "https://files.pythonhosted.org/packages/f3/24/f615e74f307e6ca0e56a482872477b856c70d530aa356abfb6dfe5ca8a80/ruff-0.16.2-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:40ea8c0594feb894e89c8c61ab9c103d38b0ea72dfde6c594107147ca31b1140", size = 11630720, upload-time = "2026-08-07T13:30:34.426Z" }, + { url = "https://files.pythonhosted.org/packages/c5/d3/8ef50149e8412a77f7ab409efdef0e2b23803707a3863da4fc64cb23d459/ruff-0.16.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ab3d62dde0b19facdd632008cc4827fc28ada7736c6bd35ab6f1050f0bfed53f", size = 11466130, upload-time = "2026-08-07T13:30:36.958Z" }, + { url = "https://files.pythonhosted.org/packages/dd/a7/a19334985c4dea8c381981fa252cd854c7ee52dc4b1686dc16f4a911c702/ruff-0.16.2-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:e43e1f5b8388da9eca1b9e88328d47a5cec794633ccf6f7484ac2dd15eee92c0", size = 11523634, upload-time = "2026-08-07T13:30:39.822Z" }, + { url = "https://files.pythonhosted.org/packages/6e/6c/96d192b0e742412ceda08c0a50f9669b253dde9fd6a60ea1a10c9fa79a63/ruff-0.16.2-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:c24788a980581e1d7ea3a0cbe4344c4fbeb0a6a9b1f4713aa46bb104f8294690", size = 10949807, upload-time = "2026-08-07T13:30:42.745Z" }, + { url = "https://files.pythonhosted.org/packages/fa/51/e26599ceca11e79ee255c7df515995561edf87e9ca1893284e44d98f5a86/ruff-0.16.2-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:81806b08329130005dd4a8a8394a0c9da8c6f4cafb16ba438d2a2ee6a18bedf1", size = 10646891, upload-time = "2026-08-07T13:30:45.522Z" }, + { url = "https://files.pythonhosted.org/packages/68/01/800c4b1f97bc8d7c6029e06b1f20473a3cf1e13c4933d8f3342add83fc55/ruff-0.16.2-py3-none-musllinux_1_2_i686.whl", hash = "sha256:4ce4e02bad779bef557f541a1b31f20d6abeae1cc05ed1b1ac019d4ffd1044c8", size = 11162063, upload-time = "2026-08-07T13:30:48.131Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d0/1477ea50fc5a0d4b0b71d1d63d50770bdd794d90b43e37a7618e63ec9894/ruff-0.16.2-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:e0422abdf70070255fc4073ce9dfc814cc03db577013761ddd09bc1e4a9a4fbd", size = 11556038, upload-time = "2026-08-07T13:30:50.686Z" }, + { url = "https://files.pythonhosted.org/packages/b8/76/a7776f32048d991e16d4fa8ff91790b877342d3596cc3ed04acdbf1aaedc/ruff-0.16.2-py3-none-win32.whl", hash = "sha256:bf3a63d78fb39f4bf5ac8ae52051c5520505301abe19ba4e204c453b3f09bb0b", size = 10872850, upload-time = "2026-08-07T13:30:53.471Z" }, + { url = "https://files.pythonhosted.org/packages/00/0d/929c800d920e61397d82a01b60bffc68da3052c17d31de59efaad2e4ed75/ruff-0.16.2-py3-none-win_amd64.whl", hash = "sha256:bcabe2f6d0fc7819f1431793005af4e4de7371927d037345bf941252b195b9fa", size = 12023338, upload-time = "2026-08-07T13:30:56.193Z" }, + { url = "https://files.pythonhosted.org/packages/5b/6c/93e26c22c5f78ff87363e07da49c84955affbeb1098bd1936bf3b3f293bf/ruff-0.16.2-py3-none-win_arm64.whl", hash = "sha256:d614e95cedf38a2053fd351c55b103ba30d017d61688fdbfd40ee0412852a99f", size = 11374065, upload-time = "2026-08-07T13:30:58.775Z" }, ] [[package]] From 45f00d233dd51eec8682dfc1780c886e1fbb56fd Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 21 Aug 2026 11:35:56 +0200 Subject: [PATCH 02/20] Take the ruff pin to 0.16.4, the version the extension now bundles The extension has moved on since this sweep was planned: 2026.74.0 bundles 0.16.4, so pinning 0.16.2 would have reopened the editor-versus-CLI gap the bump exists to close. 0.16.4 reports nothing new here and reformats nothing, so this is the pin and the lock only. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 2 +- pyproject.toml | 2 +- uv.lock | 44 ++++++++++++++++++++++---------------------- 3 files changed, 24 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 77696a2..e2bc634 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Changed -- **Ruff moves to 0.16.2, matching the version the VS Code extension bundles.** The extension now syncs `pyproject.toml` and `ruff.toml` documents to the language server, because Ruff from 0.16 on lints its own config files. A binary older than that parses those documents as Python source and paints phantom `invalid-syntax` diagnostics across every `pyproject.toml`. Pinning the dev dependency to the exact version the extension ships keeps the editor and the command line on one binary. This is a dev-dependency change: nothing shipped changes, and there is no version bump. +- **Ruff moves to 0.16.4, matching the version the VS Code extension bundles.** The extension now syncs `pyproject.toml` and `ruff.toml` documents to the language server, because Ruff from 0.16 on lints its own config files. A binary older than that parses those documents as Python source and paints phantom `invalid-syntax` diagnostics across every `pyproject.toml`. Pinning the dev dependency to the exact version the extension ships keeps the editor and the command line on one binary. This is a dev-dependency change: nothing shipped changes, and there is no version bump. Two mechanical conversions ride along, both applied by `ruff check --fix` and therefore not deferrable, since `make lint` would re-apply them on first contact: selector lists in `pyproject.toml` now name rules rather than code them, and `# noqa: CODE` suppressions became `# ruff: ignore[rule-name]`. `float-equality-comparison`, newly reported under `preview`, joins the existing `tests/` relaxations — those tests assert float literals that round-trip exactly, so `pytest.approx` would only add noise. diff --git a/pyproject.toml b/pyproject.toml index 505489d..3ae4f29 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -29,7 +29,7 @@ dev = [ "mypy==1.19.1", "pyright==1.1.411", "pylint==4.0.4", - "ruff==0.16.2", + "ruff==0.16.4", "pytest>=9.0.3", "pytest-mock>=3.12.0,<4.0.0", "pytest-sugar>=1.0.0", diff --git a/uv.lock b/uv.lock index bb57083..28f9207 100644 --- a/uv.lock +++ b/uv.lock @@ -334,7 +334,7 @@ requires-dist = [ { name = "pytest", marker = "extra == 'dev'", specifier = ">=9.0.3" }, { name = "pytest-mock", marker = "extra == 'dev'", specifier = ">=3.12.0,<4.0.0" }, { name = "pytest-sugar", marker = "extra == 'dev'", specifier = ">=1.0.0" }, - { name = "ruff", marker = "extra == 'dev'", specifier = "==0.16.2" }, + { name = "ruff", marker = "extra == 'dev'", specifier = "==0.16.4" }, { name = "typing-extensions", specifier = ">=4.0.0" }, ] provides-extras = ["dev"] @@ -557,27 +557,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.16.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/73/e1/4508a569211b35599016e84ba65c1a992b7a4004b4b6c4bea02a851cba1b/ruff-0.16.2.tar.gz", hash = "sha256:c3d7828d12e8927a6fc65fe38e2c2541b9e762d360a1786d752cb1b8883b3c9c", size = 4885811, upload-time = "2026-08-07T13:31:01.432Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/14/57/db19951540f98859c956b50bdb4d31089b4d91e9f15e2968e7d5193806d5/ruff-0.16.2-py3-none-linux_armv6l.whl", hash = "sha256:3c8de4cf2181f01d57946d87d777aa52916976fc09942aed89938fab5e013318", size = 10847925, upload-time = "2026-08-07T13:30:14.468Z" }, - { url = "https://files.pythonhosted.org/packages/13/5a/995fe85a8470d3e391ac0f7fa8054bb454eaf33ee138196d6172ed1079c0/ruff-0.16.2-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:9a48cc05c6fbc811ca81b5d7ba95375affea6582d1b8024e455e41afbbf55344", size = 11072662, upload-time = "2026-08-07T13:30:18.143Z" }, - { url = "https://files.pythonhosted.org/packages/32/53/370d767c61c71a971a4ace36703a7ecd8c393956349a7325d7fab2b56827/ruff-0.16.2-py3-none-macosx_11_0_arm64.whl", hash = "sha256:a2c0d14fcbb26c91f0f867a6dc9bd71bbc30b1b6151829c884f23faeab2e5700", size = 10566771, upload-time = "2026-08-07T13:30:20.899Z" }, - { url = "https://files.pythonhosted.org/packages/85/d6/9d96948caf5a632be62d62202d5ec914d6856f204fd79eb036e5915e79ea/ruff-0.16.2-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:335c621622c4650330be50842561c6586ac6971bb8ab5407fe34dcc9efb16bbe", size = 10975825, upload-time = "2026-08-07T13:30:23.517Z" }, - { url = "https://files.pythonhosted.org/packages/3b/92/ea87129b3414acb0b5770563779c51804d37ac67675c7ba35447ddb14773/ruff-0.16.2-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:20e66910f2c37cc753f9ef6580c914a621b80c4fa3549d3e3521e29d0f5bfc3f", size = 10649437, upload-time = "2026-08-07T13:30:26.097Z" }, - { url = "https://files.pythonhosted.org/packages/ac/43/f8f291dcd4af5bb7872b74fdfa41a7cd7c856ca1d4069670971cf1b9f5cb/ruff-0.16.2-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c7e36fbfba65510548156902bcf1350a979a958ce0347ce0f90d73894036b39f", size = 11446761, upload-time = "2026-08-07T13:30:28.752Z" }, - { url = "https://files.pythonhosted.org/packages/71/4a/ef991fb2fcf516ab71f0808adcdd8da5e18c8cde447f4ceaf5f47a5132a5/ruff-0.16.2-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f0eab35f80df8f134aae5d1630e751901321d317cc8e50dc39e36fa3ed34cd12", size = 12336364, upload-time = "2026-08-07T13:30:31.468Z" }, - { url = "https://files.pythonhosted.org/packages/f3/24/f615e74f307e6ca0e56a482872477b856c70d530aa356abfb6dfe5ca8a80/ruff-0.16.2-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:40ea8c0594feb894e89c8c61ab9c103d38b0ea72dfde6c594107147ca31b1140", size = 11630720, upload-time = "2026-08-07T13:30:34.426Z" }, - { url = "https://files.pythonhosted.org/packages/c5/d3/8ef50149e8412a77f7ab409efdef0e2b23803707a3863da4fc64cb23d459/ruff-0.16.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ab3d62dde0b19facdd632008cc4827fc28ada7736c6bd35ab6f1050f0bfed53f", size = 11466130, upload-time = "2026-08-07T13:30:36.958Z" }, - { url = "https://files.pythonhosted.org/packages/dd/a7/a19334985c4dea8c381981fa252cd854c7ee52dc4b1686dc16f4a911c702/ruff-0.16.2-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:e43e1f5b8388da9eca1b9e88328d47a5cec794633ccf6f7484ac2dd15eee92c0", size = 11523634, upload-time = "2026-08-07T13:30:39.822Z" }, - { url = "https://files.pythonhosted.org/packages/6e/6c/96d192b0e742412ceda08c0a50f9669b253dde9fd6a60ea1a10c9fa79a63/ruff-0.16.2-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:c24788a980581e1d7ea3a0cbe4344c4fbeb0a6a9b1f4713aa46bb104f8294690", size = 10949807, upload-time = "2026-08-07T13:30:42.745Z" }, - { url = "https://files.pythonhosted.org/packages/fa/51/e26599ceca11e79ee255c7df515995561edf87e9ca1893284e44d98f5a86/ruff-0.16.2-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:81806b08329130005dd4a8a8394a0c9da8c6f4cafb16ba438d2a2ee6a18bedf1", size = 10646891, upload-time = "2026-08-07T13:30:45.522Z" }, - { url = "https://files.pythonhosted.org/packages/68/01/800c4b1f97bc8d7c6029e06b1f20473a3cf1e13c4933d8f3342add83fc55/ruff-0.16.2-py3-none-musllinux_1_2_i686.whl", hash = "sha256:4ce4e02bad779bef557f541a1b31f20d6abeae1cc05ed1b1ac019d4ffd1044c8", size = 11162063, upload-time = "2026-08-07T13:30:48.131Z" }, - { url = "https://files.pythonhosted.org/packages/e4/d0/1477ea50fc5a0d4b0b71d1d63d50770bdd794d90b43e37a7618e63ec9894/ruff-0.16.2-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:e0422abdf70070255fc4073ce9dfc814cc03db577013761ddd09bc1e4a9a4fbd", size = 11556038, upload-time = "2026-08-07T13:30:50.686Z" }, - { url = "https://files.pythonhosted.org/packages/b8/76/a7776f32048d991e16d4fa8ff91790b877342d3596cc3ed04acdbf1aaedc/ruff-0.16.2-py3-none-win32.whl", hash = "sha256:bf3a63d78fb39f4bf5ac8ae52051c5520505301abe19ba4e204c453b3f09bb0b", size = 10872850, upload-time = "2026-08-07T13:30:53.471Z" }, - { url = "https://files.pythonhosted.org/packages/00/0d/929c800d920e61397d82a01b60bffc68da3052c17d31de59efaad2e4ed75/ruff-0.16.2-py3-none-win_amd64.whl", hash = "sha256:bcabe2f6d0fc7819f1431793005af4e4de7371927d037345bf941252b195b9fa", size = 12023338, upload-time = "2026-08-07T13:30:56.193Z" }, - { url = "https://files.pythonhosted.org/packages/5b/6c/93e26c22c5f78ff87363e07da49c84955affbeb1098bd1936bf3b3f293bf/ruff-0.16.2-py3-none-win_arm64.whl", hash = "sha256:d614e95cedf38a2053fd351c55b103ba30d017d61688fdbfd40ee0412852a99f", size = 11374065, upload-time = "2026-08-07T13:30:58.775Z" }, +version = "0.16.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/8f/d8074b1f25e003164087a8bfe79a0f1a3945135764dbb6aaab04103dcaf9/ruff-0.16.4.tar.gz", hash = "sha256:13171aa9d9af2240ee3504e639de73122c67e74036de5ba2e1d01422cd17e3dc", size = 4899731, upload-time = "2026-08-20T17:43:59.196Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ff/80/779895ef584e089d22f2c6df0d0e99a65ec2df0805f1fffd439415b8c1f0/ruff-0.16.4-py3-none-linux_armv6l.whl", hash = "sha256:df4075f71ddac40b9934af60c3ec8a53047dd5a5fdc43224e6e4e8e9a27cb6f7", size = 10006909, upload-time = "2026-08-20T17:43:16.888Z" }, + { url = "https://files.pythonhosted.org/packages/a9/e6/f553199b5e8927a05cb5c422d921fd0656b29ab976e91c44802107c6b0da/ruff-0.16.4-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:0c95538517af68004306b0fb3214ff2f2af67a65092aee77cd9eb86db6656604", size = 10240201, upload-time = "2026-08-20T17:43:19.337Z" }, + { url = "https://files.pythonhosted.org/packages/1c/70/4a6dc4bb34da4dee35e30f09bbd1bfbdd26f33b62fb9b8df31f08a199cd2/ruff-0.16.4-py3-none-macosx_11_0_arm64.whl", hash = "sha256:963f83df8e69e575b64d67dd447ebbc917db41a14bf38d4593a4183e7aaa8255", size = 9835122, upload-time = "2026-08-20T17:43:21.708Z" }, + { url = "https://files.pythonhosted.org/packages/24/12/c6e22d686372c15bcb7af99831f1a1be96df696491babf4f24e4f942c527/ruff-0.16.4-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:32a5057c7ff3f6e6480a48fccfb3a412a690f48a3d03ac5cf08177d6c2da3ade", size = 9977162, upload-time = "2026-08-20T17:43:24.236Z" }, + { url = "https://files.pythonhosted.org/packages/46/49/72b10ec912f5ab5854992eaf7aa7cd36729b6937d9dc4e0fb41b3bf428ec/ruff-0.16.4-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b3dce8d9b0c57c265b91885a66a567d8ea1372e8eb4e250fa8e5e3f579e99cff", size = 9829789, upload-time = "2026-08-20T17:43:26.966Z" }, + { url = "https://files.pythonhosted.org/packages/fa/80/0f30e32e7f6ee26edc39075502db9d368d788a44a79b55f763eb4ab03796/ruff-0.16.4-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7dc651db49283c69f8e72c834eec4fe5573e4c646856aebece0ce385dceb2a80", size = 10527949, upload-time = "2026-08-20T17:43:29.384Z" }, + { url = "https://files.pythonhosted.org/packages/52/3d/86e8ad3542169e56cac3859a343afdb9df2ad54d35a59ce1e67baee83421/ruff-0.16.4-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3817b87dbcabc92f13b05019257c5b89b5b4d51b5fb20f56fb5235ceb723cd07", size = 11333695, upload-time = "2026-08-20T17:43:31.872Z" }, + { url = "https://files.pythonhosted.org/packages/d0/16/481c29b380c20a0054a8261066665e1b3488e23636c49d0a43e75975b9bb/ruff-0.16.4-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e9fce1499134b2c8c68e5166f95705a5812062bb93aacc5f9873bb1a27084bc7", size = 10727741, upload-time = "2026-08-20T17:43:34.596Z" }, + { url = "https://files.pythonhosted.org/packages/5e/b6/56bc0b8cf45b54b28b3a5e6381c8945d51b5b18adf659454c32295209a31/ruff-0.16.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f2d812e482f5a7e02eee26cd73d2a37ebbdf47d795ea63ba1b89110ae93e9fb3", size = 10286522, upload-time = "2026-08-20T17:43:37.288Z" }, + { url = "https://files.pythonhosted.org/packages/e8/8b/b345b4fb110f2fbe2bd31eabd271e5e8b3b7e4ee6c0e02f2dc6be78db000/ruff-0.16.4-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:6baaf984aa7976edf93d3b627fe2d1d22ee94bbca05fa6f90fc76d73924e3454", size = 10584182, upload-time = "2026-08-20T17:43:39.984Z" }, + { url = "https://files.pythonhosted.org/packages/29/e5/827b34041c35f58774a9681a4213994c164fc987800f4dddabcf451da0bf/ruff-0.16.4-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:bdfcf0b28662eb890372d50f92c283bb94e67e7635ed93c7fd533970acff7b2b", size = 10134195, upload-time = "2026-08-20T17:43:42.351Z" }, + { url = "https://files.pythonhosted.org/packages/0f/10/d0bffcdd6729b87afc82ba0ef377173356a7dc8e972f5179968cf2fdf98c/ruff-0.16.4-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:b66b02cb9b04f537643cadf5768e5f98dc461890d530cb67113d71c8c76e605d", size = 9825821, upload-time = "2026-08-20T17:43:44.532Z" }, + { url = "https://files.pythonhosted.org/packages/f5/32/0db2a863b796ca62d83e92a07a3ccf00921b14db02059347576a2fda3d4b/ruff-0.16.4-py3-none-musllinux_1_2_i686.whl", hash = "sha256:8528bf9a4b291a60bf02ea453511e8ce6215bd2b982ee80405b66b008b6c30a0", size = 10267658, upload-time = "2026-08-20T17:43:46.989Z" }, + { url = "https://files.pythonhosted.org/packages/b2/a0/fbdeb59e48c6261f523e56c8f12e9c08fbe693786595cc7e3959207a9232/ruff-0.16.4-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:fbd85d2875fdd67e833213a651f613bbf25303abf6aa822a5121f4531195678d", size = 10697071, upload-time = "2026-08-20T17:43:49.891Z" }, + { url = "https://files.pythonhosted.org/packages/aa/28/0c6dd865859c6d17bc8ccc34cb72b0e02d6c7eb25e8a1e22b5bea681e2c0/ruff-0.16.4-py3-none-win32.whl", hash = "sha256:312769988007aaeb8e189b443ccdd03c0e6374489e053467be6d96518ebff76e", size = 10021687, upload-time = "2026-08-20T17:43:52.281Z" }, + { url = "https://files.pythonhosted.org/packages/a3/03/e724450f621698117f9aa6dd241c94d0274ae96781378dc86745ae29f0e7/ruff-0.16.4-py3-none-win_amd64.whl", hash = "sha256:05d9d27a18c4bcbefada602480ec9e01e0bc949d432e0ced5df77edac195919c", size = 10567657, upload-time = "2026-08-20T17:43:54.78Z" }, + { url = "https://files.pythonhosted.org/packages/0e/fe/da8b9e1347696bb22120b77280ec5ce25d500ca5cb39d5ad6e5c18de19c1/ruff-0.16.4-py3-none-win_arm64.whl", hash = "sha256:a3a61621c9b6f6a89573e938a080e648f1695baa3f58570a3a707bc51ff65a21", size = 10451579, upload-time = "2026-08-20T17:43:57.135Z" }, ] [[package]] From bc17c075e4254a486ce9947e7cc7d78138bc7b55 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 21 Aug 2026 13:05:33 +0200 Subject: [PATCH 03/20] bump mthds to 0.8.2 --- pyproject.toml | 2 +- uv.lock | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 3ae4f29..fbdb9ab 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ classifiers = [ ] dependencies = [ - "mthds>=0.8.1", + "mthds>=0.8.2", "pydantic>=2.10.6,<3.0.0", "typing-extensions>=4.0.0", "httpx>=0.23.0,<1.0.0", diff --git a/uv.lock b/uv.lock index 28f9207..0fa9746 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.8.1" +version = "0.8.2" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "httpx" }, @@ -221,9 +221,9 @@ dependencies = [ { name = "tomlkit" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/24/62/8c94607b2466105906232858dcee74eb8dbc2ed0352fb6b280e1f06f94b9/mthds-0.8.1.tar.gz", hash = "sha256:0c9082c2c5d833605431911635d3290ab1d1ad9d8f6d0eb68a27b89df9d256a2", size = 129886, upload-time = "2026-07-06T21:24:01.313Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ed/87/9fefbf352f2f2794336ae6fa1f72ea7d7a74ce78524e3bce734bea83aa02/mthds-0.8.2.tar.gz", hash = "sha256:1ac24a415f5a4942e93309066ed6b65a553ca379578e2da93f492bc63342b0b6", size = 130890, upload-time = "2026-08-21T10:49:37.05Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/94/29/32de38940a4d1c452285eaa8aec8bf251136a26397ded8ff85edf624b217/mthds-0.8.1-py3-none-any.whl", hash = "sha256:ea421aa00a13009d12168fa84b5fd451d0c513e60febcbfb47431c058b15bbfa", size = 58115, upload-time = "2026-07-06T21:23:59.783Z" }, + { url = "https://files.pythonhosted.org/packages/b7/85/051bf842ead495c9126e04a940cbfd3fada2474599b89be27b6d73e73158/mthds-0.8.2-py3-none-any.whl", hash = "sha256:13d314523afc5f774b7f19f65b94ce32e753a617298aae8d6d11315448bbea9f", size = 58167, upload-time = "2026-08-21T10:49:35.804Z" }, ] [[package]] @@ -326,7 +326,7 @@ dev = [ [package.metadata] requires-dist = [ { name = "httpx", specifier = ">=0.23.0,<1.0.0" }, - { name = "mthds", specifier = ">=0.8.1" }, + { name = "mthds", specifier = ">=0.8.2" }, { name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" }, { name = "pydantic", specifier = ">=2.10.6,<3.0.0" }, { name = "pylint", marker = "extra == 'dev'", specifier = "==4.0.4" }, From 05c309b3d1d21f9f1a6c59716ba63d9cd1c179ce Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 21 Aug 2026 14:15:32 +0200 Subject: [PATCH 04/20] Record the too-many-statements-in-try-clause decision before it is needed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every repo in the workspace running select = ["ALL"] with preview has settled on ignoring too-many-statements-in-try-clause: shrinking a try clause changes which statements its handlers cover, which makes satisfying the rule an error-handling refactor rather than lint cleanup. This repo has no findings for it, so the decision was never written down here. That is fine until someone writes the first long try clause, at which point they hit a rule the workspace has already ruled on and have to reach the same conclusion alone — or reach a different one. Recording the ignore now, with the reasoning inline, costs nothing and removes that. A survey across the five select=ALL repos found their ignore lists share 76 of 99 rules, and that every divergence is inert: each rule a repo lacks fires zero times there. This entry and its counterpart in conformance close the only two gaps that encode a policy rather than local circumstance. No code changes; the rule fires nowhere today. agent-check and agent-test both pass. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 2 ++ pyproject.toml | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e2bc634..8766681 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ Two mechanical conversions ride along, both applied by `ruff check --fix` and therefore not deferrable, since `make lint` would re-apply them on first contact: selector lists in `pyproject.toml` now name rules rather than code them, and `# noqa: CODE` suppressions became `# ruff: ignore[rule-name]`. `float-equality-comparison`, newly reported under `preview`, joins the existing `tests/` relaxations — those tests assert float literals that round-trip exactly, so `pytest.approx` would only add noise. +- **`too-many-statements-in-try-clause` is now ignored explicitly, ahead of ever having a finding.** Every repo in the workspace that runs `select = ["ALL"]` with `preview` has settled on ignoring this rule, because shrinking a `try` clause changes which statements its handlers cover — an error-handling refactor rather than lint cleanup. This repo happened to have no findings, so the decision was never written down here, which meant whoever wrote the first long `try` clause would have had to reach it again on their own. The reasoning now sits inline next to the ignore. No code changes; the rule fires nowhere today. + ## [v0.5.0] - 2026-07-22 ### Added diff --git a/pyproject.toml b/pyproject.toml index fbdb9ab..e466d4a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -305,6 +305,12 @@ ignore = [ "try-consider-else", # Consider moving this statement to an `else` block "deprecated-import", # `typing.List` is deprecated, use `list` instead "implicit-return", # Missing explicit `return` at the end of function able to return non-`None` value + + # Shrinking a `try` clause changes which statements its handlers cover, so satisfying + # this rule is an error-handling refactor rather than lint cleanup. Ignored across the + # workspace; recorded here even though this repo currently has no findings, so the + # decision does not have to be made again by whoever writes the first long `try`. + "too-many-statements-in-try-clause", # Try clause contains too many statements ] [tool.ruff.lint.flake8-type-checking] From cdd87936f4b03a97c5fb9ef5b7544258d3812c59 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 24 Aug 2026 14:51:53 +0200 Subject: [PATCH 05/20] Type method_id as a run option and fix delete_method's contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Promote the hosted platform's `method_id` from the untyped `extra` bag to a first-class keyword parameter on `execute` / `start` / `start_and_wait`. It reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. The argument is now documented, validated, and impossible to smuggle: `method_id` inside `extra` raises, which is breaking for the previously undocumented `extra={"method_id": ...}` path. Because the merged extension mapping is non-empty, a by-id run with no inline source still satisfies the base client's "something to run" precondition; the selector also rides the blocking `execute` fallback, so a bare runner can answer the 422 that names it. The reserved-keys guard stays per layer: `method_id` must never become reserved in `mthds`, which has no business rejecting another vendor's arguments. Doctrine: the workspace spec `docs/specs/pipelex-platform-api.md` -> "Layered extension policy". Also fixes `delete_method`, which was annotated `-> None` and promised an empty synchronous delete. The route answers 202 and the erasure continues asynchronously, so it now returns the new `MethodDeletionAccepted` — a misleading contract around a destructive operation is worth the break. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 7 ++ docs/architecture.md | 14 ++- pipelex_sdk/client.py | 139 +++++++++++++++++++++++++--- pipelex_sdk/product_models.py | 24 +++++ tests/unit/test_client_method_id.py | 122 ++++++++++++++++++++++++ tests/unit/test_client_product.py | 19 +++- 6 files changed, 305 insertions(+), 20 deletions(-) create mode 100644 tests/unit/test_client_method_id.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 8766681..67a3384 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,15 @@ ## [Unreleased] +### Added + +- **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. The doctrine is normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy". + ### Changed +- **Breaking: `delete_method` returns the platform's acceptance instead of `None`.** `DELETE /v1/methods/{id}` answers `202` with `method_id`, `deletion_state`, and `deletion_job_id`, and the erasure continues asynchronously after that; the method was annotated `-> None` with a docstring promising an empty synchronous delete, so a caller reasonably concluded the method was gone when the call returned. It now returns the new `MethodDeletionAccepted` (with `MethodDeletionState`), which means "accepted", never "gone" — completion is the row disappearing from `list_methods`. A misleading contract around a destructive operation is worth a breaking change. +- **Breaking: `extra` now rejects `method_id`.** The client names the key itself, so it must also guard it — one argument must not arrive by two paths with different validation. Callers passing `extra={"method_id": …}` must pass the named parameter instead. The guard is deliberately per layer: `method_id` must never become reserved in `mthds`, which has no business rejecting another vendor's arguments. + - **Ruff moves to 0.16.4, matching the version the VS Code extension bundles.** The extension now syncs `pyproject.toml` and `ruff.toml` documents to the language server, because Ruff from 0.16 on lints its own config files. A binary older than that parses those documents as Python source and paints phantom `invalid-syntax` diagnostics across every `pyproject.toml`. Pinning the dev dependency to the exact version the extension ships keeps the editor and the command line on one binary. This is a dev-dependency change: nothing shipped changes, and there is no version bump. Two mechanical conversions ride along, both applied by `ruff check --fix` and therefore not deferrable, since `make lint` would re-apply them on first contact: selector lists in `pyproject.toml` now name rules rather than code them, and `# noqa: CODE` suppressions became `# ruff: ignore[rule-name]`. `float-equality-comparison`, newly reported under `preview`, joins the existing `tests/` relaxations — those tests assert float literals that round-trip exactly, so `pytest.approx` would only add noise. diff --git a/docs/architecture.md b/docs/architecture.md index 5f8b652..4946324 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -77,6 +77,18 @@ Everything else stays the inherited regime: the protocol's optional 202 async-de The override also **enriches the return type**: it re-validates the base result into a `PipelexExecuteResult` (`pipelex_sdk/execute_result.py`), a `DictRunResultExecute` subtype that adds a resolved `.main_stuff` accessor (dug out of `pipe_output`'s working memory via the response's `main_stuff_name`, raising `MissingMainStuffError` if unlocatable). This gives blocking and durable results the **same output accessor** — `result.main_stuff` — so callers never branch on which path ran. `_map_run_result_to_run_results` reads that accessor too, keeping the resolution single-sourced. +## Hosted run extensions (`method_id`) + +The protocol's run args (`pipe_code`, `mthds_contents`, `inputs`, …) are the base client's named parameters and stay pure. The hosted API's *own* run args are named parameters **here**, on `execute` / `start` / `start_and_wait`. Today that is one, `method_id`, and the reasoning generalizes to every hosted-only argument that follows: + +- **A named parameter, not an `extra` entry.** Typing its own platform's arguments is the one job a layer-3 client exists to do; `extra` stays the escape hatch for an extension this client does not know about (a third-party server, or a newer version of ours). That split, and the reserved-key guard behind it, are normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy" (Rule 5). It was previously passed as `extra={"method_id": …}`, which worked but documented nothing and validated nothing. +- **It reaches the wire through the base client's extension mechanism.** `_merge_hosted_run_extensions` folds it into the `extra` mapping handed to `super().execute` / `super().start`, which merges it into the body as a top-level property without knowing what it means. That is the layering working as designed, not a workaround: the protocol client stays catalog-agnostic while the hosted client owns the concept. +- **The guard is per layer.** `method_id` is rejected inside `extra` *here*, and must never become reserved in `mthds`: a protocol client talking to another vendor's server has no business rejecting that vendor's arguments. +- **Pure pass-through — nothing is expanded client-side.** The platform resolves the id against the org's catalog. Do not confuse this with `build_inputs(method_id=…)` and `prepare_inputs`, which are client-side *sugar* that resolves an id to inline files before the request is built, so no `method_id` ever reaches those wire bodies. +- **Alone it is a run source; alongside an inline source it is linkage.** With no inline source the platform resolves and runs the stored method. With one, the inline source is what RUNS (precedence) and the id is recorded as run-history linkage on the Run row — the index key `GET /v1/runs?method_id=` queries. The base client's "something to run" precondition is satisfied because the merged extension mapping is non-empty, so a `method_id`-only run is sent rather than refused client-side. +- **It rides the blocking fallback too.** `start_and_wait` degrades to `POST /v1/execute` against a runner with no run store, and the selector goes with it — so a bare `pipelex-api` answers the `422` that names the key instead of the client silently dropping it and running something else. +- **An empty string is treated as absent.** `method_id=""` selects no method and links nothing, so it is not sent and does not satisfy the precondition. + ## Run lifecycle (hosted extension) The durable run lifecycle (`pipelex_sdk/runs.py` + the client's lifecycle methods) is a **hosted-API extension, not part of the MTHDS Protocol**. Long method runs outlive the hosted gateway's ~30s synchronous cap, so a caller submits a run (`POST /v1/start`), then polls a self-healing endpoint by bare `pipeline_run_id` until it reaches a terminal state. All state lives behind the id (DynamoDB + Temporal on the platform), so a caller can drop the poll loop and resume later with just the id. A bare runner has no run store and `404`s these routes, which the client translates into a clear `RunLifecycleUnavailableError`. @@ -132,7 +144,7 @@ The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models The wire models are snake_case Pydantic v2. Response models are extension-open (`extra="allow"`) so a newly-added server field is preserved, not rejected; input models name exactly what each route accepts. `PipelineRun.status` reuses the run-lifecycle `RunStatus`; `OrgRole`, `PipeStatus`, and the onboarding fields are `StrEnum`s. - **User profile** — `get_me()` → `UserProfile` (`GET /v1/me`). -- **Methods catalog** — `list_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. +- **Methods catalog** — `list_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. **`delete_method` is asynchronous and its return type says so:** the route answers `202` with a `MethodDeletionAccepted` (`method_id`, `deletion_state`, `deletion_job_id`) the moment the platform has claimed the method and terminated its in-flight workflows — the rest of the cascade (runs, events, S3 objects) is enqueued. A returned value means "accepted", never "gone"; completion is the row disappearing from `list_methods`, not any field of that body. It used to be annotated `-> None` with a docstring promising an empty synchronous delete, which is a misleading contract around a destructive operation. The claim is a conditional write, so a double-clicked delete is a `409 conflict` rather than a second cascade over the same runs. - **Organizations** — `list_memberships()` → `MembershipsResponse` (memberships + active-org feature flags); `create_organization(name)` / `rename_organization(org_id, name)` → `Membership`. Organization *switch* is out of scope (a WorkOS session op, not a `/v1` route). - **Billing** — `get_subscription()`, `list_plans()`, `list_invoices()`, `create_checkout(plan)`. `change_plan(plan)` and `get_billing_portal()` surface a **409 `conflict`** (`ApiResponseError.code`) when there is no subscription yet — start one via `create_checkout` first. - **Pipelex API keys** — `list_pipelex_api_keys()`; `create_pipelex_api_key(label)` and `rotate_pipelex_api_key(id)` return the plaintext `api_key` **once**; `revoke_pipelex_api_key(id)`. Creation surfaces a **409 `pipelex_api_key_limit_reached`** when the per-account limit is hit. Rotation sends no body. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 114b48b..e959c1c 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -55,6 +55,7 @@ Membership, MembershipsResponse, MethodData, + MethodDeletionAccepted, PipelexApiKeyCreated, PipelexApiKeyList, PipelineRun, @@ -125,6 +126,18 @@ # produced validation-error verdict carry `rendered_markdown`; callers may add more tokens. _VALIDATE_MARKDOWN_RENDER_FORMAT = "markdown" +# The HOSTED API's own run args — the layer-3 extensions this client names itself, on top of +# the MTHDS Protocol's basic run args. They are named parameters here and travel to the base +# client through its generic `extra` passthrough, which is exactly the layering: the protocol +# client merges them into the body without knowing what they mean. +# +# Reserved on `extra` for the same reason the protocol args are: one argument must not arrive +# by two paths with different validation. The guard is deliberately PER LAYER — it lives here +# and must never be pushed down into `mthds`, because a protocol client talking to another +# vendor's server has no business rejecting that vendor's arguments. See the workspace spec +# `docs/specs/pipelex-platform-api.md` -> "Layered extension policy" (Rule 5). +_HOSTED_RUN_ARGS: frozenset[str] = frozenset({"method_id"}) + class MthdsFile(BaseModel): """One MTHDS file submitted to `validate_files` — content plus an optional provenance URI. @@ -323,6 +336,8 @@ async def execute( output_multiplicity: VariableMultiplicity | None = None, dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, + *, + method_id: str | None = None, ) -> PipelexExecuteResult: """Execute a method synchronously and wait for its completion — `POST /v1/execute`. @@ -330,7 +345,8 @@ async def execute( resolved `.main_stuff` accessor, so a blocking result reads its output the same way as a durable one (`result.main_stuff`) instead of digging through `pipe_output`. - Identical to the inherited protocol `execute`, except a failure consistent with the + Identical to the inherited protocol `execute`, except for two things. First, `method_id` + — the hosted platform's own run arg (see below). Second, a failure consistent with the hosted gateway's ~30s synchronous ceiling — a gateway `503`/`504`, or a client-side request timeout, after at least ~28s have elapsed — is translated into a clear `PipelineExecuteTimeoutError` pointing at the durable start+poll path, matching the JS @@ -338,9 +354,30 @@ async def execute( (from the inherited `execute`), and every other non-2xx keeps the inherited `httpx.HTTPStatusError` regime (consistent with the other inherited protocol routes). + Args: + pipe_code: The code identifying the pipe to execute. + mthds_contents: List of MTHDS bundle contents to load. + inputs: Inputs passed to the method. + output_name: Name of the output slot to write to. + output_multiplicity: Output multiplicity setting. + dynamic_output_concept_ref: Override for the dynamic output concept ref. + extra: Server-specific extension args this client does not know about, merged into + the request body as top-level properties. Protocol args and this client's own + hosted args (`method_id`) must be passed as named parameters, not through + `extra` (raises `PipelineRequestError`). + method_id: A stored method's hosted catalog id (`mt_…`) — a pure PASS-THROUGH the + platform resolves against the org's catalog; nothing is expanded client-side, + and it is meaningless off-platform (an open-source runner answers a `422` + naming the key). Alone, the platform resolves and runs the stored method's + source. Alongside `mthds_contents`, the inline source is what RUNS (precedence) + and the id is recorded as run-history linkage on the Run row — the index key + `GET /v1/runs?method_id=` queries, so a run started without it is absent from + its method's history permanently. An empty string is treated as absent. + Raises: PipelineExecuteTimeoutError: The blocking request hit the hosted gateway's ~30s synchronous ceiling — use `start_and_wait` (or `start` + `wait_for_result`). + PipelineRequestError: `extra` carries a protocol arg or a hosted arg. RunStillRunningError: The server answered 202 (the protocol's optional async degrade) — the run continues server-side; resume by `pipeline_run_id`. httpx.HTTPStatusError: Any other non-2xx response (the inherited regime). @@ -354,7 +391,7 @@ async def execute( output_name=output_name, output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, - extra=extra, + extra=_merge_hosted_run_extensions(extra, method_id), ) except (httpx.HTTPStatusError, httpx.TimeoutException) as exc: elapsed_seconds = monotonic() - started_at @@ -377,14 +414,22 @@ async def start( output_multiplicity: VariableMultiplicity | None = None, dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, + *, + method_id: str | None = None, ) -> RunResultStart: """Start a method asynchronously — `POST /v1/start` (202: `pipeline_run_id` only). - Identical to the inherited protocol `start`, except a bare-runner missing-route 404 - (no run store) is translated into a clear `RunLifecycleUnavailableError` instead of a - raw `httpx.HTTPStatusError` — matching the JS SDK and letting `start_and_wait` self-heal - to the blocking-execute fallback. The platform's structured 404s (run not found) keep - their normal `httpx.HTTPStatusError` behavior. + Identical to the inherited protocol `start`, except for `method_id` — the hosted + platform's own run arg, documented on `execute` and carrying the same semantics here — + and that a bare-runner missing-route 404 (no run store) is translated into a clear + `RunLifecycleUnavailableError` instead of a raw `httpx.HTTPStatusError`, matching the JS + SDK and letting `start_and_wait` self-heal to the blocking-execute fallback. The + platform's structured 404s (run not found) keep their normal `httpx.HTTPStatusError` + behavior. + + Raises: + PipelineRequestError: `extra` carries a protocol arg or a hosted arg. + RunLifecycleUnavailableError: The configured server has no run store. """ try: return await super().start( @@ -394,7 +439,7 @@ async def start( output_name=output_name, output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, - extra=extra, + extra=_merge_hosted_run_extensions(extra, method_id), ) except httpx.HTTPStatusError as exc: self._raise_if_lifecycle_unavailable(exc.response, str(exc.request.url)) @@ -615,6 +660,8 @@ async def start_and_wait( dynamic_output_concept_ref: str | None = None, extra: dict[str, Any] | None = None, wait_options: WaitForResultOptions | None = None, + *, + method_id: str | None = None, ) -> RunResults: """Start a run and wait for its result — the whole lifecycle in one call, self-healing across hosted and bare runners. @@ -629,6 +676,10 @@ async def start_and_wait( from `start`, BEFORE any run is created, so the blocking fallback cannot double-run; the negative is cached so later calls skip the durable attempt. + `method_id` — the hosted platform's own run arg, documented on `execute` — is forwarded + on BOTH paths. Dropping it on the blocking fallback would turn a server-side 422 that + names the key into a silently different run. + Raises: RunFailedError: If the run reaches a terminal status other than COMPLETED. RunTimeoutError: If the poll budget elapses (the run keeps executing — resume by id). @@ -643,6 +694,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) except RunLifecycleUnavailableError: self._lifecycle_available = False @@ -654,6 +706,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) return await self.wait_for_result(started.pipeline_run_id, options=wait_options) @@ -665,6 +718,7 @@ async def start_and_wait( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) async def _execute_blocking( @@ -677,12 +731,15 @@ async def _execute_blocking( output_multiplicity: VariableMultiplicity | None, dynamic_output_concept_ref: str | None, extra: dict[str, Any] | None, + method_id: str | None = None, ) -> RunResults: """Blocking `POST /v1/execute` adapted onto `RunResults` — the bare-runner path. - Forwards every protocol field PLUS the `extra` extension passthrough: an extension-only - call (`{extra}` with no pipe_code/bundle) or a vendor selector riding `extra` must survive - this path, not just the durable one. + Forwards every protocol field PLUS both extension surfaces: the hosted `method_id` and + the generic `extra` passthrough. An extension-only call (`{extra}` with no pipe_code) or + a vendor selector riding `extra` must survive this path, not just the durable one — and + a hosted `method_id` must reach the server here too, so a runner that cannot resolve it + says so instead of the client silently dropping it. """ result = await self.execute( pipe_code=pipe_code, @@ -692,6 +749,7 @@ async def _execute_blocking( output_multiplicity=output_multiplicity, dynamic_output_concept_ref=dynamic_output_concept_ref, extra=extra, + method_id=method_id, ) return _map_run_result_to_run_results(result) @@ -725,9 +783,29 @@ async def update_method(self, method_id: str, write_input: MethodWriteInput) -> body = write_input.model_dump(mode="json", exclude_none=True) return MethodData.model_validate(await self._request_product("PUT", f"methods/{quote(method_id, safe='')}", body=body)) - async def delete_method(self, method_id: str) -> None: - """Delete a method — `DELETE /v1/methods/{id}` (empty body).""" - await self._request_product("DELETE", f"methods/{quote(method_id, safe='')}") + async def delete_method(self, method_id: str) -> MethodDeletionAccepted: + """Erase a method and everything it produced — `DELETE /v1/methods/{id}`. + + **Asynchronous, and the return value says so.** The platform answers `202` the moment it + has claimed the method and terminated its in-flight workflows; the rest of the cascade + (runs, events, S3 objects) is enqueued. So a returned `MethodDeletionAccepted` means + "accepted", never "gone" — completion is the method's row disappearing from + `list_methods`, not any field of the acceptance body. Until then the row stays listed + with a `deletion_state`, which is what lets a UI render it as "Deleting…", while + `get_method` refuses it with a `409`. + + A double-clicked delete is safe: the claim is a conditional write, so the second call is + an `ApiResponseError` (`409 conflict`) rather than a second cascade over the same runs. + An unknown or foreign-org id is a `404`. + + Args: + method_id: The method to erase. + + Returns: + The platform's acceptance — `method_id`, the `deletion_state` the cascade started + in, and the `deletion_job_id` a caller can log or correlate. + """ + return MethodDeletionAccepted.model_validate(await self._request_product("DELETE", f"methods/{quote(method_id, safe='')}")) async def list_memberships(self) -> MembershipsResponse: """The caller's org memberships + active-org feature flags — `GET /v1/organizations/memberships`.""" @@ -894,6 +972,39 @@ async def health(self) -> dict[str, Any]: _KNOWN_RUN_STATUS_NAMES: frozenset[str] = frozenset(RunStatus.__members__) +def _merge_hosted_run_extensions(extra: dict[str, Any] | None, method_id: str | None) -> dict[str, Any] | None: + """Fold the hosted API's own run args into the generic `extra` passthrough handed to the base client. + + This is the seam between layer 3 and layer 2: `method_id` is a named parameter on this + client (it is the hosted platform's argument, so this client must type it), and it reaches + the wire as a top-level body property through the protocol client's extension mechanism — + which merges it without knowing what it means. See `_HOSTED_RUN_ARGS`. + + An absent or empty `method_id` contributes nothing: `method_id=""` selects no method and + links no run, so it is not sent and does not satisfy the base client's "something to run" + precondition. `None` is returned for an empty result, leaving the base's own handling of an + absent `extra` untouched. + + Args: + extra: Server-specific extension args from the caller, or None. + method_id: The hosted catalog id, or None. + + Returns: + The merged extension mapping to hand to the base client, or None if there is nothing. + + Raises: + PipelineRequestError: If `extra` carries a hosted arg this client names itself. + """ + extensions: dict[str, Any] = dict(extra or {}) + hosted_overlap = extensions.keys() & _HOSTED_RUN_ARGS + if hosted_overlap: + msg = f"extra carries hosted args {sorted(hosted_overlap)} — pass them as named parameters instead." + raise PipelineRequestError(msg) + if method_id: + extensions["method_id"] = method_id + return extensions or None + + def _with_validate_markdown_render(render: list[str] | None) -> list[str]: """Ensure `"markdown"` rides the `/validate` render list, preserving order and de-duplicating. diff --git a/pipelex_sdk/product_models.py b/pipelex_sdk/product_models.py index 05297f4..8f839e1 100644 --- a/pipelex_sdk/product_models.py +++ b/pipelex_sdk/product_models.py @@ -66,6 +66,30 @@ class MethodWriteInput(BaseModel): input_data: dict[str, Any] | None = None +class MethodDeletionState(StrEnum): + """Where a method is in the erasure cascade; absent on a normal method.""" + + PENDING = "pending" + IN_PROGRESS = "in_progress" + FAILED = "failed" + + +class MethodDeletionAccepted(BaseModel): + """The `202` acceptance of `DELETE /v1/methods/{id}`. + + Returned the moment the erasure is CLAIMED and handed off, not when it completes. + Nothing in this body means "done": completion is the method's row disappearing from + `list_methods`. What the body buys a caller is a claim it can log and correlate + (`deletion_job_id`) plus the state the cascade started in. + """ + + model_config = ConfigDict(extra="allow") + + method_id: str + deletion_state: MethodDeletionState + deletion_job_id: str + + # ── Organizations (`/v1/organizations`) ────────────────────────────────── diff --git a/tests/unit/test_client_method_id.py b/tests/unit/test_client_method_id.py new file mode 100644 index 0000000..cc37408 --- /dev/null +++ b/tests/unit/test_client_method_id.py @@ -0,0 +1,122 @@ +"""Tests for the hosted `method_id` run option — the layer-3 extension this SDK names itself. + +Mirrors `pipelex-sdk-js/tests/client.test.ts` "hosted method_id option". The doctrine the +assertions pin is normative in the workspace spec `docs/specs/pipelex-platform-api.md` -> +"Layered extension policy": a hosted client types its own platform's arguments (Rule 5), and +`extra` stays the escape hatch for extensions it does not know about. +""" + +import asyncio + +import httpx +import pytest +from mthds.protocol.exceptions import PipelineRequestError +from pytest_mock import MockerFixture + +from pipelex_sdk.client import PipelexAPIClient + +_BASE_URL = "http://localhost:8081" + +_BARE_VERSION = {"protocol_version": "0.6.0", "implementation": "pipelex-api", "runner_version": "1.2.3"} +_START_BODY = {"pipeline_run_id": "run_1", "state": "RUNNING", "created_at": "2026-08-24T00:00:00Z"} +_EXECUTE_BODY: dict[str, object] = { + "pipeline_run_id": "run-x", + "main_stuff_name": "result", + "pipe_output": { + "working_memory": { + "root": {"result": {"concept": "native.Text", "content": {"text": "hi"}}}, + "aliases": {"main_stuff": "result"}, + }, + "pipeline_run_id": "run-x", + }, +} + + +def _response(status_code: int, *, json: object | None = None) -> httpx.Response: + request = httpx.Request("POST", f"{_BASE_URL}/v1/start") + if json is None: + return httpx.Response(status_code, request=request) + return httpx.Response(status_code, json=json, request=request) + + +class TestHostedMethodIdOption: + def _client(self) -> PipelexAPIClient: + return PipelexAPIClient(api_key="test-token", base_url=_BASE_URL) + + def test_method_id_rides_the_body_as_a_top_level_field(self, mocker: MockerFixture) -> None: + """The typed option reaches the wire exactly where the `extra` passthrough used to put it.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(pipe_code="answer", method_id="mt_1")) + + sent = send.call_args.kwargs["content"].decode("utf-8") + assert '"method_id":"mt_1"' in sent + # The wire body is flat — the option is not nested under an `extra` key. + assert '"extra"' not in sent + + def test_method_id_only_run_is_accepted(self, mocker: MockerFixture) -> None: + """A stored method IS something to run: the platform resolves its source server-side.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(method_id="mt_1")) + + assert '"method_id":"mt_1"' in send.call_args.kwargs["content"].decode("utf-8") + + def test_method_id_alongside_inline_source_is_linkage_not_a_conflict(self, mocker: MockerFixture) -> None: + """Inline source wins as the thing to RUN; the id rides along as run-history linkage. + + Refusing the combination once orphaned every unsaved-buffer run from its method — the Run + row's `method_id` is what writes the index key `GET /v1/runs?method_id=` queries. + """ + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(mthds_contents=['domain = "answer"'], method_id="mt_1")) + + sent = send.call_args.kwargs["content"].decode("utf-8") + assert '"method_id":"mt_1"' in sent + assert '"mthds_contents"' in sent + + def test_start_extra_rejects_a_smuggled_method_id(self) -> None: + """One argument, one path: the layer that NAMES a key must also guard it on `extra`.""" + client = self._client() + with pytest.raises(PipelineRequestError, match="method_id"): + asyncio.run(client.start(pipe_code="p", extra={"method_id": "mt_1"})) + + def test_execute_extra_rejects_a_smuggled_method_id(self) -> None: + """The guard is on the shared merge helper, so both run routes reject identically.""" + client = self._client() + with pytest.raises(PipelineRequestError, match="method_id"): + asyncio.run(client.execute(pipe_code="p", extra={"method_id": "mt_1"})) + + def test_empty_method_id_is_absent(self, mocker: MockerFixture) -> None: + """`method_id=""` selects nothing and links nothing, so it is neither sent nor a run source.""" + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + asyncio.run(client.start(pipe_code="p", method_id="")) + assert "method_id" not in send.call_args.kwargs["content"].decode("utf-8") + + with pytest.raises(PipelineRequestError): + asyncio.run(client.start(method_id="")) + + def test_blocking_fallback_forwards_method_id(self, mocker: MockerFixture) -> None: + """A bare runner must SEE the selector, so it can answer the 422 that names it. + + Dropping it on the fallback would silently run something else instead of surfacing that + the deployment has no catalog. + """ + client = self._client() + send = mocker.patch.object( + client, + "_send", + mocker.AsyncMock(side_effect=[_response(200, json=_BARE_VERSION), _response(200, json=_EXECUTE_BODY)]), + ) + + asyncio.run(client.start_and_wait(pipe_code="p", method_id="mt_1")) + + execute_call = send.call_args_list[1] + assert execute_call.args[1] == f"{_BASE_URL}/v1/execute" + assert '"method_id":"mt_1"' in execute_call.kwargs["content"].decode("utf-8") diff --git a/tests/unit/test_client_product.py b/tests/unit/test_client_product.py index 75e5ab3..0eabde1 100644 --- a/tests/unit/test_client_product.py +++ b/tests/unit/test_client_product.py @@ -14,6 +14,7 @@ from pipelex_sdk.client import PipelexAPIClient from pipelex_sdk.errors import ApiResponseError from pipelex_sdk.product_models import ( + MethodDeletionState, MethodWriteInput, OnboardingCurrentTool, OnboardingHeardFrom, @@ -124,16 +125,24 @@ def test_update_method_puts_and_drops_absent_input_data(self, mocker: MockerFixt # input_data is None → dropped from the wire (matches the JS undefined-drop). assert sent.body == {"name": "Renamed", "mthds": "src"} - def test_delete_method_tolerates_empty_204(self, mocker: MockerFixture) -> None: + def test_delete_method_returns_the_202_acceptance(self, mocker: MockerFixture) -> None: + """The erasure is asynchronous: the caller gets the CLAIM, never a "it's gone" signal. + + Completion is the row disappearing from `list_methods`, so the honest return value is the + acceptance body — a `deletion_job_id` to log or correlate, and the state it started in. + """ client = self._client() - send = self._mock_send(mocker, client, _response(204)) + body = {"method_id": "m1", "deletion_state": "pending", "deletion_job_id": "job-1"} + send = self._mock_send(mocker, client, _response(202, json_body=body)) - result = asyncio.run(client.delete_method("m1")) + accepted = asyncio.run(client.delete_method("m/1")) sent = self._sent(send) - assert result is None assert sent.method == "DELETE" - assert sent.url == f"{_BASE_URL}/v1/methods/m1" + assert sent.url == f"{_BASE_URL}/v1/methods/m%2F1" + assert accepted.method_id == "m1" + assert accepted.deletion_state is MethodDeletionState.PENDING + assert accepted.deletion_job_id == "job-1" # ── Organizations ──────────────────────────────────────────────── From 321e39ae9e41956a57817164d0ec3bceb620b604 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:11:46 +0200 Subject: [PATCH 06/20] updates design --- wip/updates.md | 172 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 wip/updates.md diff --git a/wip/updates.md b/wip/updates.md new file mode 100644 index 0000000..d132265 --- /dev/null +++ b/wip/updates.md @@ -0,0 +1,172 @@ +# Updates warranted by pipelex-api 0.17.0 / 0.18.0, the pipelex-server bump, and `@pipelex/sdk` 0.14.0 + +A design for what this repo still owes after the three sources named in the title, written on the `feature/Typed-method-id-run-option` branch, which already carries the typed `method_id` run option and the honest `delete_method` contract. Every claim below was checked against the code it names; line numbers are as of 2026-08-25 and will drift. + +## Verdict + +Yes — three groups of work, in decreasing order of what the cited releases actually ask for: + +1. **The `/v1/validate` surface moved, and this SDK has not followed.** pipelex-api 0.17.0 (via the `pipelex` 0.52.0 pin) added `warnings`, `liftable_pipes`, `input_form`, `missing_pipe_code` and `suggested_fix` to the report; 0.18.0 gated `input_form` behind a new `views` request list. `@pipelex/sdk` 0.14.0 mirrored all of it. Here, nothing crashes — every affected model is `extra="allow"`, so the new fields ride `model_extra` — but nothing is typed either, and there is no way to ask for the `input_form` view at all. This is the direct answer to the question and is purely additive. See §1. +2. **Two documentation corrections `@pipelex/sdk` 0.14.0 made apply verbatim here**: the `TokensUsageRecord` brand attribution, and citations of workspace-private paths from a public package. See §2. +3. **Found while checking, and more urgent than either: `list_methods` and `list_runs` crash against the deployed platform.** The platform reshaped both routes into `{items, next_cursor}` page envelopes (in prod since 2026-08-18 and 2026-08-11 respectively); this SDK still iterates a bare array, and its unit tests mock the old shape, which is why the suite is green. `PipelineRun` also declares `method_id` and `pipe_code` as required strings where the platform serves `str | None`. `@pipelex/sdk` fixed all of this in 0.10.0 / 0.11.0. This is a breaking fix and it is not optional. See §3. + +A fourth item is a decision already taken rather than a release to mirror: the 2026-08-25 boundary-validation decision recorded in `pipelex-sdk-js/wip/boundary-option-type-validation.md` names this repo for its Phase 2. See §4. + +What needs **no** change is listed in §5, so nobody re-derives it. The four design choices that were open in the first draft were decided on 2026-08-25 and are recorded in §7. + +## 1. The validate surface + +### 1.1 Where the Python SDK stands today + +- `PipelexValidationReport` (`pipelex_sdk/validation_models.py:84`) inherits `extra="allow"` from `mthds.protocol.models.ValidationReport`, so a 0.17+ body's `warnings`, `liftable_pipes` and `input_form` land in `model_extra`. Untyped, but parsed. +- `ValidationErrorItem` (`validation_models.py:62`) inherits `extra="allow"` from `ValidationDiagnostic`, so `missing_pipe_code` and `suggested_fix` land in `model_extra` the same way. +- Every optional member of `ValidationErrorItem` is already `T | None = None`. The JS 0.14.0 breaking change (`T` → `T | null`, forced by the valid arm serializing unset locators as explicit `null` inside `warnings[]`) therefore has **no Python counterpart** — pydantic reads both a dropped key and an explicit `null` into the same `None`. That asymmetry still deserves a regression test here, because it is the one thing a future "tighten to required" edit would break; the workspace inbox item `wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` explicitly asks for the Python mirror to be checked on this point, and this section is the answer. +- The hosted `/v1/validate` is the platform proxying to the runner (`pipelex-server/platform/src/pipelex_platform/routers/v1/tooling_proxy.py`), so once `feature/Bump-pipelex` (which moves `api-hosted` to the `pipelex-api` `v0.18.0` tag and the core to `pipelex==0.52.0`) is deployed, `api.pipelex.com` serves exactly the contract a bare 0.18.0 runner serves today, `views` gate included. Nothing in that branch changes any other route this SDK calls; its only non-pin edits are the Temporal dry-validate activity carrying `input_form` through to the route that gates it. + +### 1.2 `views` — the structured-view opt-in on `validate` and `validate_files` + +Add `views: list[str] | None = None` to `PipelexAPIClient.validate` (`pipelex_sdk/client.py:449`) after `render`, and `views: list[str] | None = None` to `validate_files` (`client.py:491`) after `render`. In Python this is not the breaking change it was in TypeScript: the parameter is appended last and callers pass it by keyword, so no existing positional call moves. + +Semantics, mirroring `@pipelex/sdk` exactly: + +- `None` (the default) → the `views` key is **not sent**. This is the invariant that keeps an opt-in view opt-in: the default response stays byte-identical for the consumers that discard it (hook pipelines, CI gates, agent loops). +- A list → sent **verbatim**, including an explicitly empty `[]`. Unlike `render`, nothing is injected and nothing is de-duplicated: the server resolves tokens as a set and lenient-ignores unknown ones (never a `422`), so client-side normalization would only hide what the caller asked for. +- Today `input_form` is the only supported token; a constant for it belongs in `validation_models.py` next to the field it gates, not a closed enum on the parameter — the spec deliberately keeps the request boundary open so a stale token never fails a call. + +It reaches the wire through the base transport seam with no `mthds-python` change: `_post_validate` merges `extra` into the body as top-level keys, and its reserved set `_VALIDATE_REQUEST_ARGS` is only `{mthds_contents, allow_signatures}` (`mthds-python/mthds/runners/api/client.py:383`). `views`, like `render` and `mthds_sources` already, is a Pipelex-API carriage extension, so the protocol client stays unaware of it — the same layering as `method_id` on the run routes. + +### 1.3 The valid arm's new fields + +On `PipelexValidationReport`: + +- `warnings: list[ValidationErrorItem] = Field(default_factory=…)` — advisory lints on a **valid** bundle. Same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid`. This is where the `hint_*` error types ride. +- `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=…)` — pipes the runtime may skip when an optional slot resolves absent. New model `LiftablePipeEntry(extra="allow")` with `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str]` (default empty — the server model defaults it too), `absence_source: str`, mirroring `pipelex/pipelex/pipeline/liftable_pipes.py`. +- `input_form: dict[str, Any] | None = None` — per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`. **Optional on purpose**: it is present only when the request named the `input_form` view (0.18.0), and a 0.17.0 runner emitted it unconditionally, so `None`-by-default is the one typing that reads a body from either runner correctly. Kept **opaque** like `bundle_blueprint`, `pipe_io_contracts` and `graph_spec`, for the same reason `@pipelex/sdk` keeps it opaque: the descriptor vocabulary is owned elsewhere (the runtime's `PipeInputFormDescriptor`, the `@pipelex/mthds-form` kernel, the `docs/specs/mthds-input-form-descriptor.md` contract), and a second copy here would be free to drift. + +Both list fields default to empty rather than being required, and that is a deliberate divergence from the runtime model, where they are always populated. This SDK is pointed at bare runners of whatever version a user runs; a pre-0.52 body with neither key must keep parsing, exactly as `bundle_blueprint` already defaults. The default is also what the runtime emits for a clean bundle, so no caller can tell the two apart — which is the point. + +`PipelexInvalidReport` gains nothing: the invalid arm never carries `warnings` or `input_form` (they derive from a crate that was never assembled), and the e2e evidence on the JS side pins `"warnings" not in report`. + +### 1.4 `ValidationErrorItem` additions and the fix vocabulary + +On `ValidationErrorItem`: `missing_pipe_code: str | None = None` (symmetrical with the existing `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. + +New models in `validation_models.py`, mirroring `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact (`pipelex-api/docs/openapi/pipelex-api.openapi.yaml`, schemas `SuggestedFix`, `SetKeyOp` … `RemapValueOp`, `FixSafety`): + +- `FixSafety(StrEnum)`: `SAFE = "safe"`, `UNSAFE = "unsafe"`, with an `is_safe` property (house style: never compare enum values inline). +- `FixOpKind(StrEnum)`: the seven kinds — `set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`. +- `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]` — what a `set_key` writes; deeper nesting is not modelled because the server does not emit it. +- One model per op, each `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key, value)`, `EnsureTableOp()`, `DeleteKeyOp(key)`, `DeleteTableOp()`, `RenameTableKeyOp(key, new_key)`, `MoveKeyOp(key, new_table_path, new_key)`, `RemapValueOp(key, mapping: dict[str, str])`. All carry `table_path: list[str]` (empty for the document root; the OpenAPI artifact marks it `minItems: 1` on `ensure_table` / `delete_table`, which is worth a `Field(min_length=1)` since it costs nothing and mirrors the artifact). +- `FixOp: TypeAlias = Annotated[SetKeyOp | … | RemapValueOp, Field(discriminator="kind")]` — narrowing is `match op: case SetKeyOp(): …`, which is the Python spelling of the JS `kind` narrowing. +- `SuggestedFix(extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]`. + +Two decisions inside that mirror: + +- **Reader models, not runtime models.** The runtime declares these `frozen`, `extra="forbid"`, with wildcard-refusing validators, because it *plans* fixes. This SDK only reads them, so the ops follow the SDK's response-model convention (`extra="allow"`) and carry none of the validators. A new server-side member on an op must not break parsing here. The two runtime invariants a type cannot carry (`*` is the wildcard segment, refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`) go in docstrings, as the JS mirror did. +- **The `kind` vocabulary is closed, and an unknown kind raises.** A pydantic discriminated union needs `Literal` tags, so a kind this SDK does not know fails the parse of the whole verdict. That is consistent with how `ValidationErrorCategory` already behaves (pinned by `test_unknown_category_is_rejected`): the vocabulary is a closed `StrEnum` upstream, a new kind is a `pipelex` release this SDK mirrors, and a loud failure beats a silently unnarrowable op. The alternative — a catch-all op with `kind: str` through a callable `Discriminator` — is more machinery for a repair proposal that is advisory in the first place; noted in §7 as the one place a reviewer might reasonably disagree. + +`error_type` stays `str | None` — an open string, as in the JS mirror. The 0.17.0 changelog notes the union gained the advisory `HintLintErrorType` members; typing it as a closed enum here would turn every runtime enum addition into an SDK break for no consumer benefit. + +Naming stays neutral (`SuggestedFix`, `FixOp`, `warnings`, `liftable_pipes`, `input_form`): fixes and lints are language-level concepts, and the runtime names them brand-neutrally too. The `Pipelex` prefix stays on the two envelope types only. + +### 1.5 Tests + +- `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the key absent from the body when `None`; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` behaviour unchanged alongside it. +- `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes` and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None`; the existing pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; an invalid body carrying `missing_pipe_code` and a two-op `suggested_fix` parses, with `match`-narrowing reaching each op's own members; an unknown `kind` raises `ValidationError`; `FixSafety` / `FixOpKind` value sets are the locked vocabularies. +- The JS suite also pins the gate **live** (`tests/e2e/tools.e2e.ts`: absent by default, present when asked, unknown token lenient). This repo has no e2e suite at all (`tests/` holds only `unit/`), so that half is not reproducible here today. Not a blocker for this change; recorded in §6 as a known gap rather than silently skipped. + +### 1.6 What stays opaque in the 0.17.0 contract move + +The 0.17.0 changelog lists more `/v1/validate` movements than the ones above, and none of them reaches a typed field here: `PipeInputContract.optional` → `presence`, the `fixed` multiplicity with `item_count`, and the widened `inputs` map all live inside `pipe_io_contracts` / `bundle_blueprint`, which this SDK carries as `dict[str, Any]` on purpose. A consumer that reads those dicts should know the new spellings; the SDK's docs (`docs/architecture.md`, validate section) should name them in one sentence so nobody discovers `presence` by surprise, but no model changes. + +## 2. Already on this branch, and the two documentation corrections still owed + +**Done here, matching `@pipelex/sdk` 0.14.0 field-for-field** (commit `cdd8793`): `method_id` as a typed keyword on `execute` / `start` / `start_and_wait`; the run-source precondition satisfied by a `method_id`-only body; `extra` rejecting `method_id` (`_HOSTED_RUN_ARGS`, `client.py:139`); an empty string treated as absent; the selector forwarded on the blocking fallback; `delete_method` returning `MethodDeletionAccepted`. `tests/unit/test_client_method_id.py` pins every one of the JS cases. Nothing further is owed on those. + +**Still owed** — the two prose fixes 0.14.0 shipped under "Fixed", which apply here for the same reason (`pipelex-sdk` is a public PyPI package): + +- **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py:23` and `:128`, `docs/run-usage.md:5` and `docs/architecture.md:103` say the record is "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API is what pins the wire contract. Reword as the JS mirror did (`pipelex-sdk-js/src/runs.ts`, `docs/architecture.md`). +- **Citations a reader cannot open.** `client.py:138` (`docs/specs/pipelex-platform-api.md`), `validation_models.py:44` (`conformance/conformance/validation_contract.py`), and the test-module docstrings at `tests/unit/test_validation_contract.py:4-5` / `:167`, `tests/unit/test_runs.py:12`, `tests/unit/test_client_method_id.py:4`, plus `docs/architecture.md:84`. Each names a workspace-private path by bare relative reference, which resolves to nothing for anyone who clones this repo and reads as rot. Replace each with the rule it was citing (the layered extension policy; the locked category vocabulary; the shared conformance corpus), as 0.14.0 did. No behaviour change. + +## 3. Found while checking: the product list routes are broken against the deployed platform + +### 3.1 Evidence + +- The platform serves `GET /v1/methods` as `MethodPage` — `{items: MethodSummary[], next_cursor: str | None}` — since `pipelex-server` commit `f4f8764` (2026-08-18, "paginate the method list, which was silently truncating"), and `GET /v1/runs?method_id=` as `RunPage` — `{items: RunPublic[], next_cursor}` — since `2c4e980` (2026-08-11). Both are ancestors of the latest `deploy(prod)` commit (`b9f9555`), so this is what `api.pipelex.com` answers today. Models: `pipelex-server/shared/src/pipelex_shared/schemas/method.py:265-309`, `schemas/run.py:180-236`. +- `list_methods` (`pipelex_sdk/client.py:767`) does `[MethodData.model_validate(item) for item in result]` over the JSON body. Iterating the envelope dict yields its **keys**, so the first call is `MethodData.model_validate("items")` → `pydantic.ValidationError` on every invocation. `list_runs` (`client.py:946`) fails identically. +- The unit tests mock the pre-paging bare arrays (`tests/unit/test_client_product.py:85`, `:379`), which is why nothing is red. +- `PipelineRun` (`product_models.py:369-370`) declares `method_id: str` and `pipe_code: str`; the platform's `RunPublic` declares both `str | None = None`, and both are genuinely null in practice (an ad-hoc run from an inline bundle; a pipe resolved from `main_pipe`). Once the envelope is fixed, the first such row raises. +- `@pipelex/sdk` took all three in 0.10.0 (`listRuns` → `RunPage`, `iterateRuns`, `getRunDetail`, nullable `PipelineRun` fields) and 0.11.0 (`listMethods` → `MethodPage`, `iterateMethods`, `MethodSummary`). `docs/architecture.md` here still claims full parity ("surface-complete, with no silent gaps"), which has been false since 2026-08-11. + +### 3.2 Design + +Breaking, and mirroring the JS shapes — with the wire kept snake_case, so the envelope field is `next_cursor` here where JS renamed it `nextCursor` for its own consumers. + +**Models (`product_models.py`):** + +- `MethodSummary(extra="allow")`: `method_id`, `name`, `description: str | None = None`, `created_at`, `deletion_state: MethodDeletionState | None = None`. Deliberately not a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none is in the index projection and putting `mthds` back is what restored the truncation bug. +- `MethodPage(extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. No total, by design. +- `RunPage(extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. +- `RunErrorReport(extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. +- `PipelineRun`: `method_id: str | None = None`, `pipe_code: str | None = None`; add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. `pipe_statuses` stays as it is (the JS model keeps it optional; the platform's `RunPublic` no longer declares it, and `extra="allow"` covers either way). +- `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None` — the only read that carries what the run actually executed. +- `MethodData`: add `org_id: str`, `created_by_user_id: str` (required on the platform's `MethodPublic` and in the JS model), `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, and `python: list[MethodFile] = Field(default_factory=list)`. See the `python` decision below. +- `MethodWriteInput`: add `python: list[MethodFile] | None = None`. Because the write body is dumped with `exclude_none=True`, the platform's three-way contract falls out naturally: `None` → not sent → the stored Python is preserved; `[]` → serialized as `""` → clears it; a non-empty list → replaces it. Document that on the field. + +**Client (`client.py`):** + +- `list_methods(*, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage`. Query params are added on **presence** (`is not None`), never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to silently drop into an unfiltered query that reads as working. Encode with `urllib.parse.urlencode` rather than string formatting; `q` is free text. +- `iterate_methods(*, q=None, limit=None) -> AsyncIterator[MethodSummary]` — an `async def` generator that follows the cursor. It must keep going **past empty pages** (`q` is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` means "keep going"), stop on `next_cursor is None`, stop when the server hands back the cursor it was sent (checked before yielding, so rows are never double-counted), and **raise** rather than return past a runaway page ceiling set far beyond any real catalog. Deliberately not a `list_all_methods() -> list[…]`: an all-at-once helper needs a cap, and a cap means silently returning a truncated list — the exact bug paging removed. +- `list_runs(method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage`. `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), not days; a naive timestamp is a platform `400`, surfaced as `ApiResponseError`. Same presence semantics. +- `iterate_runs(method_id, *, created_from=None, created_to=None, limit=None) -> AsyncIterator[PipelineRun]` — same loop, except an empty page **does** end it: the date bounds are index key conditions, so a run page is never empty-with-a-cursor. The difference is the server, not the client, and the docstring should say so. +- `get_run_detail(run_id: str) -> RunDetail` — `GET /v1/runs/{id}`, path-encoded like the other id routes. +- One thing to document that the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()` gate (`pipelex-server/platform/src/pipelex_platform/deps.py:345`), which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403`. That arrives here as an `ApiResponseError`, and a reader of `list_runs` should know a `403` means "flag", not "wrong key". + +**The `python` field.** On the wire `MethodPublic.python` is one string: the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python (`pipelex-server/shared/src/pipelex_shared/schemas/method.py:209`, `:251`), and the write side is the same string three ways (omitted → preserve, `""` → clear, text → replace). `@pipelex/sdk` never shows that string to callers: it exposes `MethodFile[]` and converts at the client boundary (`pipelex-sdk-js/src/client.ts:281` on read, `:292` on write) with `parseMethodFiles` / `serializeMethodFiles` from `mthds-js/src/protocol/method_files.ts`. That module is small — parse the JSON, check every entry is `{name: str, content: str}`, drop blank-content entries, and serialize an empty list as `""` rather than `"[]"` because `""` is the platform's clear sentinel. It lives in `mthds/protocol` on the JS side because `pipelex-mcp` consumes the same format and wanted one owner. + +The first draft of this document proposed exposing the raw wire string and asking `mthds-python` for the converter. That was over-engineered: with pydantic the whole converter is a `MethodFile` model plus a `TypeAdapter(list[MethodFile])` and the two sentinel rules, there is no second Python consumer that could drift, and the JS module's own docstring calls this the format "the hosted platform persists" — a Pipelex catalog concern, so this SDK is a proper home for it. **Decision: typed list, converter here.** `product_models.py` gains `MethodFile(name: str, content: str)` and a `parse_method_files(source: str | None) -> list[MethodFile]` / `serialize_method_files(files: list[MethodFile]) -> str` pair carrying the same rules as the JS pair (blank source or `"[]"` → `[]`; blank-content entries dropped on both directions; empty list → `""`). `MethodData` applies the parser through a `field_validator("python", mode="before")`, so `MethodData.model_validate(body)` keeps working unchanged at every call site; `MethodWriteInput` applies the serializer through a `field_serializer("python")`, so the write body still dumps with `exclude_none=True` and the three-way contract holds. Malformed wire text raises `ValueError` inside the validator and therefore surfaces as a `pydantic.ValidationError`, the same way any other malformed response body fails here. If the format ever gains a Python owner in `mthds-python`, this SDK adopts it then; nothing is filed to the inbox for it. + +`get_method_closure` (JS-only client-side sugar that parses the polymorphic `mthds` source into a run-ready closure) stays deferred: it is not moved by any of the cited releases, and the 0.5.0 changelog already recorded it as "deferred and additive" alongside `prepare_inputs`. + +### 3.3 Tests (`tests/unit/test_client_product.py`) + +Replace the two bare-array fixtures with envelopes and add: query encoding for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped; a null `pipe_code` / `method_id` row parsing; `get_run_detail` returning `mthds_contents` and `inputs`; `MethodData` carrying the new fields, with `python` parsed from the wire string into `MethodFile` entries and `""` reading as an empty list; `MethodWriteInput.python` three-way serialization (`None` absent, `[]` sent as `""`, a list sent as the JSON text); the `parse_method_files` / `serialize_method_files` pair round-tripping, dropping blank-content entries, and rejecting a non-array or a malformed entry. For the iterators, in a dedicated module (one `TestClass` per module): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding; `iterate_runs` stops on an empty page; `iterate_methods` raises past the ceiling. + +## 4. Boundary type validation for `method_id` (decision of 2026-08-25) + +`pipelex-sdk-js/wip/boundary-option-type-validation.md` records the decision, taken by Louis, that a published client validates request-option types at its boundary and throws `PipelineRequestError` rather than dropping or forwarding a wrong-typed value. Its evidence section cites this repo directly: `client.py:1003` is a bare `if method_id:`, which drops falsy non-strings (`0`, `[]`) and forwards truthy ones (`123`, `["mt_1"]`) to a server `422` — a *different* partition of wrong values than the JS client makes for the same argument on the same wire. The plan's Phase 2 names the fix: an explicit `is not None` presence check followed by an `isinstance(method_id, str)` check that raises, with `None` and `""` still normalizing to absent. + +The plan sequences the SDKs after the protocol packages so both inherit one behaviour for the protocol-level arguments. That ordering matters for `pipe_code` / `mthds_contents`, whose guards belong in `mthds-python`; it does not constrain `method_id`, which this layer owns outright and whose guard touches only `_merge_hosted_run_extensions` (`client.py:975`). **Recommendation: land the `method_id` guard in this update** — it is a few lines, this branch is already the `method_id` branch, and it closes the repo-specific finding in the JS wip doc — and take the protocol-argument guards later with the `mthds` floor bump once `mthds-python` ships its Phase 1. Decided 2026-08-25: it lands now (§7). + +## 5. Checked, no change needed + +- **pipelex-api 0.17.0's source-less `422` naming unhandled keys** (the `method_id`-at-a-bare-runner diagnosis): this SDK already forwards `method_id` on the blocking fallback precisely so that message reaches the caller; the `execute` docstring already describes it. +- **`storage_scope` / `callback_urls` / `orchestration_mode`** (0.15.0–0.16.0): layer-2 fields the hosted platform sends to the runner; not caller-facing and not an SDK concern. +- **The four OpenAPI schemas that went opaque, the `RunMetadata` split, the `.pipelex/` config schema, and the two authoring changes** (`required = true` + `default_value` rejected; unknown structure-field keys rejected): server-side and inside opaque dicts here; no wire field this SDK types moved. +- **`views` in `mthds-python`**: not needed. It is a Pipelex-API carriage extension exactly like `render`, and the base client's `extra` passthrough already carries it (§1.2). `mthds-js` likewise has no `views`. +- **Explicit-null locators** (JS 0.14.0's `T | null` widening): already `T | None = None` here (§1.1); only a regression test is owed. +- **`ValidationErrorItem.error_type` narrowing** to the new enum members: stays an open `str` by design (§1.4). +- **The 0.14.0 `validate()` positional break**: Python takes `views` by keyword after `render`; no positional call moves. +- **`method_id` typed option and `delete_method`**: already on this branch (§2). + +## 6. Change plan + +Four commits on this branch, each self-contained, each with its docs and changelog lines, each gated on `make agent-check` and `make agent-test`: + +1. **Validate surface** (§1): `validation_models.py` (new fields, `LiftablePipeEntry`, the fix vocabulary), `client.py` (`views` on `validate` / `validate_files`), the two test modules, `docs/architecture.md` (the validate section and the brand-boundary field list gain `warnings` / `liftable_pipes` / `input_form`, and a sentence on the opaque `presence` / `fixed` spellings), `README.md` quickstart mention of `views`. Changelog: **Added** (`views`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary), with a note that an older runner's body still parses. +2. **Prose corrections** (§2): the attribution and citation edits in `runs.py`, `validation_models.py`, `client.py`, the three test docstrings, `docs/run-usage.md`, `docs/architecture.md`. Changelog: **Fixed**, two entries mirroring 0.14.0's wording. +3. **Product paging and nullability** (§3): `product_models.py`, `client.py` (`list_methods`, `iterate_methods`, `list_runs`, `iterate_runs`, `get_run_detail`), `tests/unit/test_client_product.py` plus a new iterator test module, `docs/architecture.md` (product surface section rewritten for pages, the "Parity with `@pipelex/sdk`" section corrected — it must stop claiming surface-completeness and list the conscious exclusions honestly), `README.md` if it gains a listing example. Changelog: **Changed (breaking)** for the two return types and the nullable `PipelineRun` fields, **Added** for the iterators, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the `MethodData` fields and `MethodWriteInput.python`, **Fixed** naming the crash. +4. **`method_id` type guard** (§4): `_merge_hosted_run_extensions`, one wrong-type parametrized test in `test_client_method_id.py`. Changelog: **Changed**. + +The version stays where it is under `## [Unreleased]` until `/release` cuts it; with the breaking items in commit 3 (and the ones already on the branch), that release is a minor bump. + +**Known gaps this plan leaves open, on purpose:** no e2e suite exists in this repo, so the live `views` gate and the live paging envelope are pinned only by mocked bodies here (the JS suite pins both live); the remaining `@pipelex/sdk` surfaces without a Python counterpart — `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — are unchanged by the cited releases and stay the conscious exclusions `docs/architecture.md` already records. + +## 7. Decisions (2026-08-25) + +The four questions the first draft left open, each answered by Louis on 2026-08-25 with the reasoning that settled it. + +1. **`input_form` stays opaque** — `dict[str, Any] | None = None`, matching the JS mirror and the ownership argument in §1.3. A `PipeInputFormDescriptor(fields: list[dict])` shell would type one level and still leave the field vocabulary opaque, which buys little. +2. **`MethodData.python` / `MethodWriteInput.python` are typed `list[MethodFile]`, with the converter in this repo.** The question was first posed as "raw wire string plus an inbox request to `mthds-python` for the parser", and the answer to "why do we need a parser at all?" dissolved that framing: the converter is a dozen lines of pydantic, the format is a Pipelex catalog concern rather than an MTHDS protocol one, and there is no second Python consumer to keep in step. Full design in §3.2; no inbox item is filed. +3. **The `method_id` type guard lands now**, in this update (§4). The protocol-argument guards for `pipe_code` / `mthds_contents` still wait for `mthds-python` to ship its Phase 1 and arrive here with the `mthds` floor bump. +4. **An unknown `FixOp.kind` raises** — closed `Literal` discriminator, `pydantic.ValidationError` on the whole verdict parse, consistent with the closed `ValidationErrorCategory`. The lenient catch-all alternative described in §1.4 was considered and not taken. From 78b60e0c30452fbe6cd19b97c5afb8f587e167d8 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:17:50 +0200 Subject: [PATCH 07/20] Add the implementation tracker for the wip/updates.md design TODOS.md breaks the design in wip/updates.md into phases with checkboxes, records the decisions that were open as settled, and states the gates each phase has to pass before it counts as done. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- TODOS.md | 169 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 TODOS.md diff --git a/TODOS.md b/TODOS.md new file mode 100644 index 0000000..212c539 --- /dev/null +++ b/TODOS.md @@ -0,0 +1,169 @@ +# TODOS — implementing `wip/updates.md` + +This is the implementation tracker for the design in [`wip/updates.md`](wip/updates.md). The design answers *what* and *why*; this file is the *how*, broken into phases with checkboxes. Tick a box when the item is done and verified, not when it is started. Every design choice that was open has been decided (see `wip/updates.md` §7) and is treated here as settled: `input_form` stays opaque, `MethodData.python` is a typed `list[MethodFile]` with the converter in this repo, the `method_id` type guard lands now, and an unknown `FixOp.kind` raises. + +Ground rules for every phase, from `CLAUDE.md`: + +- Branch: `feature/Typed-method-id-run-option` (already carries the typed `method_id` option and the `delete_method` contract fix). The PR targets `dev`. +- Gate each phase on `make agent-check` **and** `make agent-test`; nothing is "done" before both pass. Run `make check` once at the end as well, since it adds pylint on top of the agent gate. +- Tests use `pytest-mock` only, one `TestClass` per module, no `__init__.py` under `tests/`. Mock at the httpx boundary (`_send`), as the existing modules do. +- Everything accumulates under `## [Unreleased]` in `CHANGELOG.md`. No version bump in this work: the `/release` skill cuts the version, and with the breaking items in Phase 3 (plus those already on the branch) that will be a minor bump. +- Docs move with the code, in the same commit. `docs/architecture.md` is the main one to keep truthful; its "Parity with `@pipelex/sdk`" section currently claims surface-completeness and is wrong. +- No hardcoded counts in code, docs, or commit messages. No volatile state in tracked files (this file records decisions and what landed, never whether the tree is clean or tests are green right now). +- Line numbers quoted below are as of the design date (2026-08-25) and will drift; they are anchors, not contracts. + +Suggested order is the order below. Phase 3 is the most urgent fix (a crash against the deployed platform) but is also the largest breaking change, so the plan puts the purely additive Phase 1 first to keep each commit reviewable; reorder if the crash needs to ship alone. + +## Phase 0 — preflight + +- [ ] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.1` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. +- [ ] Run `make agent-check` and `make agent-test` before touching anything, so a later failure is attributable to this work and not to the starting point. +- [ ] Re-read `wip/updates.md` §6 and §7 once; if anything there contradicts this file, this file is the stale one. + +## Phase 1 — the `/v1/validate` surface (`wip/updates.md` §1) + +Purely additive. Every affected response model is `extra="allow"`, so nothing parses differently for a body that lacks the new keys; the work is to type what already arrives and to add the one request knob (`views`) that has no way through today. + +### 1.1 `pipelex_sdk/validation_models.py` — the fix vocabulary and the new fields + +- [ ] Add `FixSafety(StrEnum)` with `SAFE = "safe"`, `UNSAFE = "unsafe"`, and an `is_safe` property (house rule: never compare enum values inline; a `match` inside the property). +- [ ] Add `FixOpKind(StrEnum)` with the kinds the runtime emits: `SET_KEY = "set_key"`, `ENSURE_TABLE = "ensure_table"`, `DELETE_KEY = "delete_key"`, `DELETE_TABLE = "delete_table"`, `RENAME_TABLE_KEY = "rename_table_key"`, `MOVE_KEY = "move_key"`, `REMAP_VALUE = "remap_value"`. Source of truth: `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact `pipelex-api/docs/openapi/pipelex-api.openapi.yaml`. +- [ ] Add `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]`. Deeper nesting is not modelled because the server does not emit it; say so in a comment. +- [ ] Add a private `_FixOpBase(BaseModel)` with `model_config = ConfigDict(extra="allow")` and `table_path: list[str]` (empty list means the document root), then one subclass per kind, each with `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key: str, value: TomlValue)`, `EnsureTableOp()`, `DeleteKeyOp(key: str)`, `DeleteTableOp()`, `RenameTableKeyOp(key: str, new_key: str)`, `MoveKeyOp(key: str, new_table_path: list[str], new_key: str)`, `RemapValueOp(key: str, mapping: dict[str, str])`. On `EnsureTableOp` and `DeleteTableOp` declare `table_path: list[str] = Field(min_length=1)`, mirroring the artifact's `minItems: 1`. +- [ ] These are **reader** models: no `frozen`, no `extra="forbid"`, none of the runtime's wildcard-refusing validators. Put the two runtime invariants a type cannot carry in the docstrings (`*` is the wildcard segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`). +- [ ] Add `FixOp: TypeAlias = Annotated[SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, Field(discriminator="kind")]`. Narrowing is `match op: case SetKeyOp(): …`, exhaustive, no `case _`. +- [ ] **Verify** that pydantic accepts the raw wire string (`"set_key"`) against `Literal[FixOpKind.SET_KEY]` both in `validate_python` and `validate_json`, and that the discriminator resolves on it. If it does not on the pinned pydantic, use `Literal["set_key"]` on the models and keep `FixOpKind` as the documented vocabulary (with a test that the two sets agree). +- [ ] Add `SuggestedFix(BaseModel, extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]` (typed default factory via `empty_list_factory_of` only if a default is warranted — the runtime always sends `ops`, so leaving it required is fine). +- [ ] Add `LiftablePipeEntry(BaseModel, extra="allow")`: `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str] = Field(default_factory=list)`, `absence_source: str`. Mirrors `pipelex/pipelex/pipeline/liftable_pipes.py`. +- [ ] Add the view token constant next to the field it gates: `VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form"`. A constant, not a closed enum — the request boundary is deliberately open so a stale token never fails a call. +- [ ] On `ValidationErrorItem` add `missing_pipe_code: str | None = None` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. Leave `error_type: str | None` as an open string; do not enum it. +- [ ] On `PipelexValidationReport` add `warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem))`, `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry))`, and `input_form: dict[str, Any] | None = None`, each with a docstring: `warnings` never flips `is_valid`; the two lists default empty so a pre-0.52 runner's body still parses; `input_form` is present only when the request named the `input_form` view (a 0.17.0 runner emitted it unconditionally, which `None`-by-default also reads correctly) and is opaque on purpose, keyed like `pipe_io_contracts`. +- [ ] `PipelexInvalidReport` gains nothing; add one sentence to its docstring saying why (the invalid arm never carries `warnings` or `input_form` — they derive from a crate that was never assembled). +- [ ] Update the module docstring's list of neutrally-named supporting types to include the new ones, and keep the brand rule stated there (the `Pipelex` prefix stays on the two envelopes only). + +### 1.2 `pipelex_sdk/client.py` — `views` on `validate` and `validate_files` + +- [ ] `validate(...)`: append `views: list[str] | None = None` after `render`. When `views is not None`, set `extra["views"] = views` **verbatim** — no injection, no de-duplication, an explicit `[]` is sent as `[]`. When `None`, the key is absent from the body. It rides `_post_validate`'s `extra` exactly like `render` and `mthds_sources`; no `mthds-python` change. +- [ ] `validate_files(...)`: append `views: list[str] | None = None` after `render` and thread it through to `validate`. +- [ ] Docstrings: the sentence "differs from the inherited protocol `validate` in two Pipelex-API ways" becomes three (render injection, `mthds_sources`, `views`); document the `views` semantics (opt-in structured views; `input_form` is the only token today, named by `VALIDATION_VIEW_INPUT_FORM`; unknown tokens are lenient-ignored server-side, never a `422`; the default response stays byte-identical for consumers that discard views). +- [ ] The `Returns:` section of `validate` should mention that a valid report now carries `warnings`, `liftable_pipes`, and (when asked) `input_form`. + +### 1.3 Tests + +- [ ] `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the `views` key absent from the body when the parameter is omitted; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` injection unchanged when `views` is also passed. +- [ ] `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes`, and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None` — this is the regression guard against a future "tighten to required" edit, and it answers the inbox item `../wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` for the Python mirror. +- [ ] `tests/unit/test_validation_contract.py`: an invalid body carrying `missing_pipe_code` and a `suggested_fix` with at least two ops of different kinds parses, and `match`-narrowing reaches each op's own members; an `ensure_table` op with an empty `table_path` is rejected; an unknown `kind` raises `pydantic.ValidationError`; `FixSafety` and `FixOpKind` value sets are asserted as the locked vocabularies (same style as `test_category_vocabulary_is_the_locked_set`). +- [ ] Keep the canonical bodies where the module already keeps them (module-level constants next to `VALID_BODY`); move to a `tests/unit/test_data.py` only if the module becomes unreadable. + +### 1.4 Docs and changelog + +- [ ] `docs/architecture.md` → "`validate` override": add a `views` bullet beside the render bullet; list the typed valid-arm additions (`warnings`, `liftable_pipes`, `input_form`) and the `ValidationErrorItem` additions with the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary; one sentence that `PipeInputContract.optional` became `presence` and that the `fixed` multiplicity carries `item_count` inside the opaque `pipe_io_contracts`, so nobody discovers the new spellings by surprise. +- [ ] `docs/architecture.md` → "Brand boundary": the list of neutrally-named supporting types gains the new ones. +- [ ] `README.md` quickstart: one line showing `views=[VALIDATION_VIEW_INPUT_FORM]` (or a comment that the input form is opt-in), so the knob is discoverable. +- [ ] `CHANGELOG.md` `[Unreleased]` → **Added**: `views` on `validate` / `validate_files`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the fix vocabulary; a note that a body from an older runner still parses (the lists default empty, `input_form` defaults `None`). + +### 1.5 Gate and commit + +- [ ] `make agent-check` and `make agent-test` pass. +- [ ] Commit (suggested message: "Type the pipelex-api 0.17/0.18 validate contract and add the views opt-in"). + +### Checkpoint 1 + +- [ ] Update this file: tick what landed, record the SHA of the Phase 1 commit, note whether pydantic accepted the enum `Literal` tags or the string fallback was needed, and any deviation from §1 of the design with its reason. + +## Phase 2 — prose corrections (`wip/updates.md` §2) + +No behaviour change. Two fixes `@pipelex/sdk` 0.14.0 shipped under "Fixed" that apply here for the same reason (`pipelex-sdk` is a public PyPI package). + +- [ ] **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py` (module docstring near line 23 and the class docstring near line 128), `docs/run-usage.md` (line 5), `docs/architecture.md` (the `TokensUsageRecord` bullet near line 103): the record is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API pins the wire contract. Reword as `pipelex-sdk-js/src/runs.ts` and its `docs/architecture.md` did. `docs/run-usage.md` line 5 already says the right thing in its second sentence; make the first sentence agree with it. +- [ ] **Citations a reader cannot open.** Replace each bare workspace-private path with the rule it was citing: `pipelex_sdk/client.py` near line 138 (`docs/specs/pipelex-platform-api.md` → "the layered extension policy: a hosted client types its own platform's arguments and guards them per layer"); `pipelex_sdk/validation_models.py` near line 44 (`conformance/conformance/validation_contract.py` → "the locked category vocabulary shared with the conformance corpus"); `tests/unit/test_validation_contract.py` module docstring and the docstring of `test_category_vocabulary_is_the_locked_set`; `tests/unit/test_runs.py` near line 12; `tests/unit/test_client_method_id.py` module docstring; `docs/architecture.md` near line 84. Keep the JS mirror references (`pipelex-sdk-js/...`) where they explain a port — those are a sibling public repo, not a private path. +- [ ] `CHANGELOG.md` `[Unreleased]` → **Fixed**: two entries mirroring 0.14.0's wording. +- [ ] `make agent-check` and `make agent-test` pass. +- [ ] Commit (suggested message: "Correct the TokensUsageRecord attribution and drop unopenable citations"). + +## Phase 3 — product paging and nullability (`wip/updates.md` §3) + +Breaking, and the most urgent fix in this plan: `list_methods` and `list_runs` crash against the deployed platform because both routes now answer a `{items, next_cursor}` envelope, and `PipelineRun` requires fields the platform serves as nullable. Wire fields stay snake_case (`next_cursor`), where the JS mirror renamed to `nextCursor` for its own consumers. + +### 3.1 `pipelex_sdk/product_models.py` — models + +- [ ] Add `MethodFile(BaseModel, extra="allow")` with `name: str`, `content: str`, defined **before** `MethodData`. Docstring: the at-rest catalog form of one source file (`[{name, content}]`), distinct from `MthdsFile` (`client.py`, validate input) and `MthdsFileItem` (`build_models.py`, build closure) — three shapes for three surfaces, name the difference so nobody merges them. +- [ ] Add `parse_method_files(source: str | None) -> list[MethodFile]`: blank source (`None`, `""`, whitespace) and `"[]"` both yield `[]`; a JSON array of `{name, content}` yields those files with blank-content entries dropped; anything else (non-array JSON, a malformed entry, unparseable text) raises `ValueError` with a message naming the expected shape. Implementation: `json.loads` then `TypeAdapter(list[MethodFile])` (built once at module level, TypeAdapter construction is expensive), wrapping `json.JSONDecodeError` / `pydantic.ValidationError` into the `ValueError`. +- [ ] Add `serialize_method_files(files: list[MethodFile]) -> str`: drop blank-content entries; an empty result serializes to `""` (the platform's "no source / clear" sentinel), never `"[]"`; otherwise `json.dumps` of `[{name, content}]` only (no extras), stable key order. +- [ ] `MethodData`: add `org_id: str`, `created_by_user_id: str`, `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, `python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile))`, plus a `@field_validator("python", mode="before")` that applies `parse_method_files` when the incoming value is a `str` or `None` and passes a list through unchanged (so programmatic construction in tests still works). A `ValueError` from the parser surfaces as `pydantic.ValidationError` from `model_validate`, the same way any malformed response body fails here. +- [ ] `MethodWriteInput`: add `python: list[MethodFile] | None = None` with a `@field_serializer("python")` returning `serialize_method_files(value)`. Docstring the three-way contract: `None` → key absent (the write body dumps with `exclude_none=True`) → the stored Python is preserved on `PUT`; `[]` → sent as `""` → clears it; a non-empty list → replaces it. +- [ ] Add `MethodSummary(BaseModel, extra="allow")`: `method_id: str`, `name: str`, `description: str | None = None`, `created_at: str`, `deletion_state: MethodDeletionState | None = None`. Docstring: deliberately not a `MethodData` — no `mthds`, no `python`, no `updated_at` — because none is in the index projection and putting `mthds` back is what restored the truncation bug; a method mid-deletion stays in the list so a UI can render "Deleting…" while `get_method` refuses it with a `409`. +- [ ] Add `MethodPage(BaseModel, extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. Docstring: opaque cursor, pass it straight back; `None` means last page; no total by design. +- [ ] Add `RunErrorReport(BaseModel, extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. +- [ ] `PipelineRun`: `method_id: str | None = None` (an ad-hoc run from an inline bundle belongs to no stored method), `pipe_code: str | None = None` (resolved from the bundle's `main_pipe`); add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. Leave `pipe_statuses` as it is. +- [ ] Add `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None`. Docstring: `mthds_contents` is what the run actually executed and the only record of it; both fields are absent from the list and the polled status read on purpose (size × page size, size × poll rate). +- [ ] Add `RunPage(BaseModel, extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. +- [ ] Update the section comments in the module (`# ── Methods catalog`, `# ── Run records`) so the new models sit under the right banner. + +### 3.2 `pipelex_sdk/errors.py` — the runaway-paging error + +- [ ] Add one error for `iterate_methods` refusing to keep paging past the ceiling (a name like `PagingNotTerminatingError`), extending whatever base the module's other product errors extend — check the existing hierarchy there first. Message mirrors the JS one: the iterator did not terminate after the ceiling; this is a server-side fault, not a coverage limit. + +### 3.3 `pipelex_sdk/client.py` — list, iterate, detail + +- [ ] Add a module helper `_product_query(params: dict[str, str | int | None]) -> str` that keeps entries on **presence** (`is not None`, never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to drop silently) and encodes with `urllib.parse.urlencode`, returning `""` or `?…`. The existing `test_list_runs_encodes_query_value` assertion (`method_id=m%2F1`) must stay green, so keep `/` percent-encoded. +- [ ] Add `_MAX_LIST_PAGES: int = 10_000` beside the other module constants (the JS `MAX_PAGES`), with the comment that it is a runaway backstop set far beyond any real catalog, not a coverage cap. +- [ ] `list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage` — `GET /v1/methods` with the query built by the helper; parse `MethodPage`. Docstring: `q` is a server-side case-insensitive substring match over name and description across the whole catalog; `limit` defaults to and is capped by the API; ordering is by creation, newest first. +- [ ] `iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]` as an `async def` generator: request a page; **before yielding**, stop if `cursor is not None and page.next_cursor == cursor` (the server did not advance; yielding first would double-count); yield every item; stop when `page.next_cursor is None`; otherwise count the page and **raise** the new error once the count reaches `_MAX_LIST_PAGES`; continue **through empty pages** with a live cursor, because `q` is a post-read filter over a bounded index slice per request and `{items: [], next_cursor: "…"}` means "keep going". Docstring says why there is no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. +- [ ] `list_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage` — `GET /v1/runs?method_id=…` plus the presence-kept query; parse `RunPage`. Docstring: `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, key conditions rather than filters; a bare date or naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also document the gate the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()`, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — a `403` here means "flag", not "wrong key". +- [ ] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing. +- [ ] `get_run_detail(self, run_id: str) -> RunDetail` — `GET /v1/runs/{id}` via `_request_product` with the id path-encoded like the other id routes (`f"{_RUNS}/{quote(run_id, safe='')}"`). Distinct from `get_run_status` (`/status`) and `get_run_result` (`/results`). +- [ ] Update the `PipelexAPIClient` class docstring's product-surface bullet and the import block (`MethodPage`, `MethodSummary`, `RunDetail`, `RunPage`, `AsyncIterator` from `collections.abc` under `TYPE_CHECKING` if only used in annotations — it is used at runtime as a return annotation with `from __future__ import annotations`, so `TYPE_CHECKING` is fine). + +### 3.4 Tests + +- [ ] `tests/unit/test_client_product.py`: replace the bare-array fixtures at `test_list_methods` and `test_list_runs_encodes_query_value` with envelopes and assert `MethodPage` / `RunPage` come back with `next_cursor`; add query-encoding cases for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped and that an absent parameter leaves no key in the query; a run row with `null` `pipe_code` and `method_id` parses; `get_run_detail` hits `/v1/runs/{id}` with encoding and returns `mthds_contents` and `inputs`; `MethodData` parses the new fields with `python` read from the wire string into `MethodFile` entries and `""` reading as `[]`; `create_method` / `update_method` send `python` three ways (`None` absent, `[]` as `""`, a list as the JSON text). +- [ ] New `tests/unit/test_method_files.py` (one `TestClass`): `parse_method_files` on blank / `"[]"` / a valid array / an array with a blank-content entry / a non-array / a malformed entry / unparseable text; `serialize_method_files` on empty / blank-only / mixed; a round-trip is stable. +- [ ] New `tests/unit/test_client_paging.py` (one `TestClass`): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding the page; `iterate_runs` stops on an empty page; `iterate_methods` raises the new error past the ceiling (patch `_MAX_LIST_PAGES` down via `mocker.patch` rather than looping ten thousand times); the cursor sent on page N+1 is the `next_cursor` received on page N. Use `mocker.AsyncMock(side_effect=[...])` on `_send` to script the page sequence. +- [ ] The `_response` / `_Sent` / `_mock_send` helpers live as private members of `tests/unit/test_client_product.py`. Rather than importing private test helpers across modules, promote a response builder and a `_send` spy to fixtures in a new `tests/unit/conftest.py` for the new modules to use (house rule: fixtures go in `conftest.py`). Migrating `test_client_product.py` onto those fixtures is optional and not part of this change. + +### 3.5 Docs and changelog + +- [ ] `docs/architecture.md` → "Pipelex product surface": rewrite the **Methods catalog** bullet for `MethodPage` / `MethodSummary` / `iterate_methods` and the `python` three-way write contract with `MethodFile`; rewrite the **Run records** bullet for `RunPage` / `iterate_runs` / `get_run_detail`, the nullable `PipelineRun` fields and `error`, the instant-only date bounds, and the `ff_api_keys` `403`. State the two iterator stop rules and why they differ. +- [ ] `docs/architecture.md` → "Parity with `@pipelex/sdk`": it must stop claiming "surface-complete, with no silent gaps". Rewrite it to list the conscious exclusions honestly: `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — unchanged by the cited releases and deferred. While there, fix the stale "Out of scope for v0.1" bullet that still lists `/v1/build/*` helpers as deferred even though `build_inputs` shipped in 0.5.0. +- [ ] `README.md`: if the quickstart gains a listing example, use `iterate_methods` rather than a page loop, so the idiom people copy is the one that cannot truncate. +- [ ] `CHANGELOG.md` `[Unreleased]`: **Changed (breaking)** — `list_methods` returns `MethodPage` (items are `MethodSummary`, not `MethodData`), `list_runs` returns `RunPage`, `PipelineRun.method_id` / `pipe_code` are nullable, `MethodData.python` is `list[MethodFile]`; **Added** — `iterate_methods`, `iterate_runs`, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the new `MethodData` fields, `MethodWriteInput.python`, the paging error; **Fixed** — name the crash plainly (iterating the envelope dict yielded its keys, so the first call was `MethodData.model_validate("items")`), and that the unit tests mocked the pre-paging shape. + +### 3.6 Gate and commit + +- [ ] `make agent-check` and `make agent-test` pass. +- [ ] Commit (suggested message: "Follow the platform's paged method and run lists and stop requiring nullable run fields"). + +### Checkpoint 2 + +- [ ] Update this file: tick what landed, record the Phase 2 and Phase 3 commit SHAs, and note any place the Python shapes deliberately diverge from the JS mirror beyond snake_case (there should be none besides the `python` converter living here). + +## Phase 4 — `method_id` boundary type guard (`wip/updates.md` §4) + +The 2026-08-25 decision in `pipelex-sdk-js/wip/boundary-option-type-validation.md`: a published client validates request-option types at its boundary and raises `PipelineRequestError` rather than dropping or forwarding a wrong-typed value. Its evidence names this repo's bare `if method_id:` in `_merge_hosted_run_extensions`, which drops falsy non-strings and forwards truthy ones to a server `422`. + +- [ ] `_merge_hosted_run_extensions` (`client.py` near line 975): replace `if method_id:` with an explicit presence check (`if method_id is not None`) followed by `if not isinstance(method_id, str): raise PipelineRequestError(msg)` naming the received type, then the existing empty-string-is-absent normalization. `None` and `""` still contribute nothing. +- [ ] Update the docstring's `Raises:` and the "An absent or empty `method_id`" paragraph. +- [ ] `tests/unit/test_client_method_id.py`: one parametrized test over wrong-typed values (`0`, `123`, `[]`, `["mt_1"]`, `{}`) asserting `PipelineRequestError` on `execute` and `start` before any request is sent; keep `test_empty_method_id_is_absent` green. +- [ ] `docs/architecture.md` → "Hosted run extensions (`method_id`)": add a bullet that a non-string raises at the boundary, and why (one partition of wrong values across both SDKs). +- [ ] `CHANGELOG.md` `[Unreleased]` → **Changed**: the guard, with the JS decision as the reason. +- [ ] `make agent-check` and `make agent-test` pass. +- [ ] Commit (suggested message: "Reject a non-string method_id at the client boundary"). + +## Phase 5 — wrap-up + +- [ ] `make check` (adds pylint to the agent gate) and `make agent-test` pass on the final tree. +- [ ] Re-read `docs/architecture.md` end to end for any remaining claim the code no longer supports (parity, `Out of scope`, the validate section, the product section). +- [ ] Re-read `CHANGELOG.md` `[Unreleased]`: every breaking item is labelled "breaking", no counts, no WIP-doc mentions, the version line untouched. +- [ ] `wip/updates.md` stays where it is, with its §7 decisions; this file stays too. Neither is deleted or emptied as part of finishing the work. +- [ ] Open the PR against `dev`, then follow `/review-pr-agents` for the Greptile / Codex loop (compare SHAs, not notifications; reply and resolve each thread in one pass). + +### Checkpoint 3 + +- [ ] Update this file with the final commit SHAs, the PR number, and anything deferred out of the plan with the reason. + +## Known gaps left open on purpose + +- No e2e suite exists in this repo (`tests/` holds only `unit/`), so the live `views` gate and the live paging envelope are pinned only by mocked bodies here; the JS suite pins both live. Adding an e2e suite is separate work. +- The remaining `@pipelex/sdk` surfaces without a Python counterpart (`lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure`) are untouched by the cited releases and stay deferred; Phase 3 makes `docs/architecture.md` say so. +- The protocol-argument type guards (`pipe_code`, `mthds_contents`) belong in `mthds-python` and arrive here with the `mthds` floor bump once that package ships its Phase 1. From 434b2e33d651732c10887b702573860bf1303612 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:23:56 +0200 Subject: [PATCH 08/20] Type the pipelex-api 0.17/0.18 validate contract and add the views opt-in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `/v1/validate` surface moved in pipelex-api 0.17.0 and 0.18.0 and this SDK had not followed. Nothing crashed — every affected model is `extra="allow"`, so the new keys rode `model_extra` — but nothing was typed either, and there was no way to ask for the `input_form` view at all. `validate` and `validate_files` now take a `views` opt-in after `render`. Unlike `render`, the list is sent verbatim: nothing injected, nothing de-duplicated, an explicit `[]` sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know. Left at `None` the key is absent, so the default response stays byte-identical for consumers that discard views. The valid arm now types `warnings`, `liftable_pipes` and `input_form`, and `ValidationErrorItem` types `missing_pipe_code` and `suggested_fix` with the whole fix-op vocabulary as a closed discriminated union. Both lists default empty and `input_form` defaults `None`, so a body from an older runner still parses. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 8 + README.md | 6 +- TODOS.md | 6 +- docs/architecture.md | 9 +- pipelex_sdk/client.py | 27 ++- pipelex_sdk/validation_models.py | 228 ++++++++++++++++++++++++- tests/unit/test_client_validate.py | 52 +++++- tests/unit/test_validation_contract.py | 202 +++++++++++++++++++++- 8 files changed, 517 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67a3384..c29e642 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ ### Added +- **The `/v1/validate` contract of pipelex-api 0.17/0.18 is now typed, and the `input_form` view is askable.** `validate` and `validate_files` take a `views: list[str] | None` opt-in after `render`. `input_form` — named by the new `VALIDATION_VIEW_INPUT_FORM` constant — is the only token the server supports today. Unlike `render`, the list is sent **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so normalizing client-side would only hide what the caller asked for. Left at its `None` default the key is not sent at all, which is what keeps the default response byte-identical for consumers that discard views. The constant is deliberately a constant rather than a closed enum on the parameter: the request boundary is open so a stale token never fails a call. + + A valid `PipelexValidationReport` now carries three typed fields instead of leaving them in `model_extra`: `warnings` (advisory lints on a bundle that is nonetheless valid — the same `ValidationErrorItem` type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid`), `liftable_pipes` (the new `LiftablePipeEntry` inventory of pipes the runtime may skip when an optional slot resolves absent), and `input_form` (the per-pipe descriptors, keyed like `pipe_io_contracts` and kept opaque on purpose — the descriptor vocabulary is owned elsewhere and a second copy here would be free to drift). `ValidationErrorItem` gains `missing_pipe_code`, symmetrical with the existing `missing_concept_code`. + + A body from an older runner still parses unchanged: both lists default empty and `input_form` defaults `None`, which is also exactly what a clean bundle and a view-less request yield, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them — `warnings` and `input_form` derive from a crate that was never assembled. + +- **Structured repair proposals: `SuggestedFix` and the fix-op vocabulary.** `ValidationErrorItem.suggested_fix` is now typed. A `SuggestedFix` carries a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops` — a list discriminated on `kind` over the closed `FixOpKind` set (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), each variant declaring exactly its own members, narrowed with an exhaustive `match op: case SetKeyOp(): …`. These are **reader** models: `extra="allow"`, not `frozen`, and without the runtime's wildcard-refusing validators, because this SDK reads fixes where the runtime plans them — a new server-side member on an op must not break parsing here. An unknown `kind` fails the whole verdict parse, deliberately and consistently with the closed `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break for no consumer benefit. + - **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. The doctrine is normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy". ### Changed diff --git a/README.md b/README.md index f32aeea..4b96381 100644 --- a/README.md +++ b/README.md @@ -28,13 +28,17 @@ The client is **async-only** (httpx `AsyncClient`) and is an async context manag ```python from pipelex_sdk.client import PipelexAPIClient +from pipelex_sdk.validation_models import VALIDATION_VIEW_INPUT_FORM async def main() -> None: async with PipelexAPIClient() as client: # 1. Validate an MTHDS bundle. The verdict is always returned (never raised): # a 200 discriminated on `is_valid`, carrying `rendered_markdown`. - report = await client.validate([bundle_text]) + # Structured views are opt-in: asking for `input_form` here is what populates + # `report.input_form` (the per-pipe input-form descriptors); omit `views` and the + # request body carries no `views` key at all. + report = await client.validate([bundle_text], views=[VALIDATION_VIEW_INPUT_FORM]) print(report.rendered_markdown) if not report.is_valid: return diff --git a/TODOS.md b/TODOS.md index 212c539..4b1e834 100644 --- a/TODOS.md +++ b/TODOS.md @@ -16,9 +16,9 @@ Suggested order is the order below. Phase 3 is the most urgent fix (a crash agai ## Phase 0 — preflight -- [ ] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.1` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. -- [ ] Run `make agent-check` and `make agent-test` before touching anything, so a later failure is attributable to this work and not to the starting point. -- [ ] Re-read `wip/updates.md` §6 and §7 once; if anything there contradicts this file, this file is the stale one. +- [x] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.1` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. Confirmed: 0.8.2. +- [x] Run `make agent-check` and `make agent-test` before touching anything, so a later failure is attributable to this work and not to the starting point. Both green at the starting point. +- [x] Re-read `wip/updates.md` §6 and §7 once; if anything there contradicts this file, this file is the stale one. No contradiction found. ## Phase 1 — the `/v1/validate` surface (`wip/updates.md` §1) diff --git a/docs/architecture.md b/docs/architecture.md index 4946324..8427bef 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -29,7 +29,7 @@ mthds.runners.api.client.MthdsAPIClient (protocol-only base: transport, body-b MTHDS is the brand of the open standard (the language, the protocol). Pipelex is the brand of the hosted runtime/product. Artifacts that belong to the standard keep neutral, un-prefixed names; Pipelex branding is reserved for genuinely runtime/product-specific surfaces (the durable run lifecycle, the product routes, implementation envelopes). The five protocol routes and their neutral models stay in `mthds`; everything Pipelex-specific lives here. -The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). +The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus` / `LiftablePipeEntry` / `SuggestedFix` / the `FixOp` variants / `FixOpKind` / `FixSafety`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). ## Credentials & configuration @@ -131,12 +131,19 @@ The protocol `validate` is **overridden** (not inherited) to add the two Pipelex - **Markdown render is always injected.** `validate(...)` adds `"markdown"` to the `render` list (de-duplicated, caller tokens first) so both a valid `PipelexValidationReport` and a produced `PipelexInvalidReport` carry `rendered_markdown`. Unknown render tokens are server-side lenient-ignored. - **`mthds_sources`** is a named parameter (parallel to `mthds_contents`) threaded onto each diagnostic's `source`; sent only when provided. +- **`views`** is the opt-in for the server's structured views, on both `validate` and `validate_files`. `input_form` — named by the `VALIDATION_VIEW_INPUT_FORM` constant — is the only token today. Unlike `render`, the list travels **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so client-side normalization would only hide what the caller asked for. Left at `None` the key is not sent at all, which is what keeps the default response byte-identical for the consumers that discard views (hook pipelines, CI gates, agent loops). The constant is a constant rather than a closed enum on purpose: the request boundary is deliberately open, so a stale token never fails a call. - **`validate_files(files, …)`** takes `MthdsFile(content, uri?)` records. When any file carries a URI, every content gets a parallel source label — the named file's URI, or a deterministic `inline://file-N.mthds` for an unnamed sibling — so the server never sees a length-mismatched `mthds_sources`. The override reuses the inherited base transport seam `_post_validate` (which builds the body — passing `render` / `mthds_sources` through the protocol's `extra` extension passthrough — sends the request, and raises on a no-verdict non-2xx), then parses the raw 200 body into this SDK's own `PipelexValidationResult` via `PipelexValidationResultAdapter`. Body-building and transport stay shared with the base; only the Pipelex presentation/sources concerns and the branded narrowing live here. The validation models (`PipelexValidationResult` = `PipelexValidationReport | PipelexInvalidReport`, with `rendered_markdown`) are **owned by this SDK** (`pipelex_sdk.validation_models`); they narrow `mthds`'s neutral verdict bases, completing the brand boundary (the resolved follow-up #9). The base `MthdsAPIClient.validate()` returns the neutral `ValidationResult` instead. **Checkpoint-5 decision (validate error regime):** the delegation keeps the inherited `httpx.HTTPStatusError` regime on a *no-verdict* non-2xx, where the JS `validate` raises `ApiResponseError`. Kept as-is (deferred parity), because in both SDKs `validate`'s error regime matches the *other* protocol routes of that SDK — JS routes all raise `ApiResponseError`, Python protocol routes all inherit `httpx.HTTPStatusError` (decision #5). Making Python's `validate` alone raise `ApiResponseError` would make it inconsistent with `execute`/`start`/`models`/`version`, which is worse than the JS divergence. The verdict itself (valid/invalid) is always a 200 either way — only the no-verdict failure *presentation* differs. +**What the report carries (0.17+).** A valid `PipelexValidationReport` adds three typed fields. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: dict[str, Any] | None` carries the per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`; it is present only when the request named the `input_form` view, and it is kept **opaque** for the same reason as `bundle_blueprint` / `pipe_io_contracts` / `graph_spec` — the descriptor vocabulary is owned elsewhere and a second copy here would be free to drift. The two lists default empty and `input_form` defaults `None`, so a body from an older runner still parses; the empty default is also what a clean bundle yields, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them: `warnings` and `input_form` derive from a crate that was never assembled. + +`ValidationErrorItem` gains `missing_pipe_code` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None` — a deterministic repair proposal with a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops`: a list discriminated on `kind` over the closed `FixOpKind` vocabulary (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), narrowed with an exhaustive `match op: case SetKeyOp(): …`. The ops are **reader** models here: `extra="allow"`, no `frozen`, none of the runtime's wildcard-refusing validators, because this SDK only reads fixes where the runtime plans them. A `kind` this SDK does not know fails the whole verdict parse, deliberately and consistently with `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break. + +One movement in the same release reaches no typed field here and is worth knowing anyway, because it changes what a consumer reads out of the opaque dicts: `PipeInputContract.optional` became `presence`, and the `fixed` multiplicity now carries an `item_count`. Both live inside `pipe_io_contracts`, which this SDK carries as `dict[str, Any]` on purpose — nobody should discover the new spellings by surprise. + ## Pipelex product surface (hosted management routes) The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models.py` + the client's product methods). Every route rides the same `{base}/v1/*` surface, `Authorization: Bearer`, org-from-JWT contract as the protocol routes, and goes through `_request_product`, which maps a non-2xx `problem+json` to a typed `ApiResponseError` — **consumers branch on `.code`, never the HTTP status**. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index e959c1c..c76c7e4 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -452,6 +452,7 @@ async def validate( # type: ignore[override] allow_signatures: bool = False, mthds_sources: list[str] | None = None, render: list[str] | None = None, + views: list[str] | None = None, ) -> PipelexValidationResult: """Parse, validate, and dry-run an MTHDS bundle — `POST /v1/validate`. @@ -461,9 +462,10 @@ async def validate( # type: ignore[override] means no verdict could be produced (request shape, auth, server fault) and surfaces as `httpx.HTTPStatusError` (the inherited protocol error regime). - This override differs from the inherited protocol `validate` in two Pipelex-API ways: + This override differs from the inherited protocol `validate` in three Pipelex-API ways: it always injects `render: ["markdown"]` (so both valid and invalid verdicts carry - `rendered_markdown`), and it accepts `mthds_sources` as a named parameter. + `rendered_markdown`), it accepts `mthds_sources` as a named parameter, and it carries + the `views` opt-in for the server's structured views. Args: mthds_contents: MTHDS contents to load (always a list, even for one file). @@ -473,15 +475,24 @@ async def validate( # type: ignore[override] `source: null`). The server 422s a length mismatch. render: Optional Pipelex-API presentation hints; `"markdown"` is always added. Unknown tokens are server-side lenient-ignored (never a 422). + views: Optional opt-in for the server's structured views. `input_form` — named by + `VALIDATION_VIEW_INPUT_FORM` — is the only token today; unknown tokens are + server-side lenient-ignored (never a 422). Unlike `render`, the list is sent + **verbatim**: nothing is injected and nothing is de-duplicated, and an explicit + `[]` is sent as `[]`. Left at `None` the key is not sent at all, which is what + keeps the default response byte-identical for consumers that discard views. Returns: The 200-diagnostic union: `PipelexValidationReport` (`is_valid: true`) or `PipelexInvalidReport` (`is_valid: false`, with `validation_errors`), each - carrying `rendered_markdown`. + carrying `rendered_markdown`. A valid report also carries `warnings` and + `liftable_pipes`, plus `input_form` when `views` asked for it. """ extra: dict[str, Any] = {"render": _with_validate_markdown_render(render)} if mthds_sources is not None: extra["mthds_sources"] = mthds_sources + if views is not None: + extra["views"] = views # Reuse the inherited transport seam (`_post_validate`) for body-building + the wire call, # then parse the 200-diagnostic body into this SDK's Pipelex-branded narrowing. The base's # own `validate` parses the same body into the neutral `mthds` `ValidationResult`. @@ -493,6 +504,7 @@ async def validate_files( files: list[MthdsFile], allow_signatures: bool = False, render: list[str] | None = None, + views: list[str] | None = None, ) -> PipelexValidationResult: """Validate paired MTHDS files while preserving URI attribution for diagnostics. @@ -500,6 +512,13 @@ async def validate_files( a URI, every content gets a parallel source label (a deterministic `inline://` label for the ones without), so the server never sees a length-mismatched `mthds_sources`. + Args: + files: The MTHDS files to validate, each content plus an optional provenance URI. + allow_signatures: Tolerate unimplemented pipe signatures (strict by default). + render: Optional Pipelex-API presentation hints, threaded to `validate`. + views: Optional structured-view opt-in, threaded to `validate` unchanged — see + `validate` for the semantics. + Raises: PipelineRequestError: If `files` is empty. """ @@ -517,7 +536,7 @@ async def validate_files( else: mthds_sources = None - return await self.validate(mthds_contents, allow_signatures, mthds_sources, render) + return await self.validate(mthds_contents, allow_signatures, mthds_sources, render, views) # ── Hosted extension: durable run lifecycle (NOT part of the protocol) ── # diff --git a/pipelex_sdk/validation_models.py b/pipelex_sdk/validation_models.py index bfc2da7..63de72b 100644 --- a/pipelex_sdk/validation_models.py +++ b/pipelex_sdk/validation_models.py @@ -10,8 +10,10 @@ `pipelex-sdk`, not in the brand-neutral `mthds` package — mirroring `pipelex-sdk-js/src/models.ts` and the documented brand boundary (`docs/architecture.md` → "Brand boundary"). The report/union types carry the `Pipelex` prefix; the supporting types (`DryRunStatus`, `ValidatedPipeEntry`, -`ValidationErrorCategory`, `ValidationErrorItem`) stay neutrally named — branding the envelope, -not the field names inside it. +`ValidationErrorCategory`, `ValidationErrorItem`, `LiftablePipeEntry`, `SuggestedFix`, the +`FixOp` variants, `FixOpKind`, `FixSafety`) stay neutrally named — branding the envelope, +not the field names inside it. Fixes and lints are language-level concepts, and the runtime +names them brand-neutrally too. `PipelexAPIClient.validate()` returns this `PipelexValidationResult` (parsed via `PipelexValidationResultAdapter`); the protocol base `MthdsAPIClient.validate()` returns the @@ -21,13 +23,21 @@ from __future__ import annotations from enum import StrEnum -from typing import Annotated, Any, TypeAlias +from typing import Annotated, Any, Final, Literal, TypeAlias from mthds.protocol.models import InvalidValidationReport, ValidationDiagnostic, ValidationReport from pydantic import BaseModel, ConfigDict, Field, TypeAdapter from pipelex_sdk._pydantic_utils import empty_list_factory_of +VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form" +"""The one `views` token the server supports today — asks for `PipelexValidationReport.input_form`. + +Deliberately a constant rather than a closed enum: the request boundary is open, the server +resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so +a stale token must never fail a call. +""" + class DryRunStatus(StrEnum): """Per-pipe dry-run sweep outcome on `ValidatedPipeEntry.status`.""" @@ -40,8 +50,8 @@ class DryRunStatus(StrEnum): class ValidationErrorCategory(StrEnum): """The closed `validation_errors[].category` vocabulary (locked). - Mirrors the single source of truth in the conformance suite - (`conformance/conformance/validation_contract.py`); keep in sync with it. + This is the locked category vocabulary shared with the conformance corpus; keep the two + in sync — a category the corpus knows and this set does not fails the whole verdict parse. """ BLUEPRINT_VALIDATION = "blueprint_validation" @@ -50,6 +60,174 @@ class ValidationErrorCategory(StrEnum): DRY_RUN = "dry_run" +class FixSafety(StrEnum): + """Whether a suggested fix is safe to auto-apply (SAFE) or needs an explicit opt-in (UNSAFE).""" + + SAFE = "safe" + UNSAFE = "unsafe" + + @property + def is_safe(self) -> bool: + match self: + case FixSafety.SAFE: + return True + case FixSafety.UNSAFE: + return False + + +class FixOpKind(StrEnum): + """The closed vocabulary of semantic patch operations a `SuggestedFix` is composed of. + + A kind this SDK does not know fails the parse of the whole verdict, which is deliberate and + consistent with `ValidationErrorCategory`: the vocabulary is closed upstream, a new kind is a + runtime release this SDK mirrors, and a loud failure beats a silently unnarrowable op. + """ + + SET_KEY = "set_key" + ENSURE_TABLE = "ensure_table" + DELETE_KEY = "delete_key" + DELETE_TABLE = "delete_table" + RENAME_TABLE_KEY = "rename_table_key" + MOVE_KEY = "move_key" + REMAP_VALUE = "remap_value" + + +TomlScalar: TypeAlias = str | int | float | bool +"""What a `set_key` op can write as a bare TOML value.""" + +TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar] +"""A `set_key` value: a scalar, or a flat scalar mapping written as an inline table. + +Deeper nesting is not modelled because the server does not emit it — the fixes that create a +whole table at once create a flat one. +""" + + +class _FixOpBase(BaseModel): + """Fields every fix op shares: the table it acts in. + + These are **reader** models. The runtime declares its own copies `frozen`, `extra="forbid"`, + with validators that refuse the wildcard segment, because it *plans* fixes; this SDK only + reads them, so the ops follow the response-model convention (`extra="allow"`) and carry none + of those validators — a new server-side member on an op must not break parsing here. + + The two runtime invariants a type cannot carry, recorded so a consumer knows them: `*` is the + wildcard path segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` + and `delete_table` address the table itself rather than its parent, so their `table_path` is + never empty (that one *is* expressed below, mirroring the OpenAPI artifact's `minItems: 1`). + """ + + model_config = ConfigDict(extra="allow") + + table_path: list[str] + """The table the op acts in, e.g. `["pipe", "my_seq"]`. Empty means the document root.""" + + +class SetKeyOp(_FixOpBase): + """Write `key = value` in the addressed table, whatever it currently holds.""" + + kind: Literal[FixOpKind.SET_KEY] + key: str + value: TomlValue + + +class EnsureTableOp(_FixOpBase): + """Create the addressed table when it is absent, leaving an existing one untouched.""" + + kind: Literal[FixOpKind.ENSURE_TABLE] + table_path: list[str] = Field(min_length=1) + + +class DeleteKeyOp(_FixOpBase): + """Drop `key` from the addressed table.""" + + kind: Literal[FixOpKind.DELETE_KEY] + key: str + + +class DeleteTableOp(_FixOpBase): + """Drop the addressed table, including every chunk of one written out of order.""" + + kind: Literal[FixOpKind.DELETE_TABLE] + table_path: list[str] = Field(min_length=1) + + +class RenameTableKeyOp(_FixOpBase): + """Rename `key` to `new_key` in place within the addressed table, keeping its position.""" + + kind: Literal[FixOpKind.RENAME_TABLE_KEY] + key: str + new_key: str + + +class MoveKeyOp(_FixOpBase): + """Relocate `key` from the addressed table into `new_table_path`, under `new_key`.""" + + kind: Literal[FixOpKind.MOVE_KEY] + key: str + new_table_path: list[str] + new_key: str + + +class RemapValueOp(_FixOpBase): + """Rewrite `key`'s value through `mapping`, doing nothing when it is not a mapped value. + + This is the one kind for which `key` may be the wildcard segment `*`, meaning "each key of + the addressed table" — the only shape in which a renamed enumerated value beneath an open + mapping can be repaired at all. + """ + + kind: Literal[FixOpKind.REMAP_VALUE] + key: str + mapping: dict[str, str] + + +FixOp: TypeAlias = Annotated[ + SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, + Field(discriminator="kind"), +] +"""One semantic patch operation, discriminated on `kind`. + +Narrow it with an exhaustive `match op: case SetKeyOp(): ...` — the Python spelling of the +JS mirror's `kind` narrowing. +""" + + +class SuggestedFix(BaseModel): + """A deterministic fix for one validation error, ready for a style-preserving applier. + + `fix_code` is the kebab-case rule id (e.g. `"match-sequence-output"`). `source` is the file + the ops target, when known (multi-file libraries) — an applier must only apply ops to the + file they target. The ops are the machine contract; any rendered diff is presentation. + """ + + model_config = ConfigDict(extra="allow") + + fix_code: str + description: str + safety: FixSafety + source: str | None = None + ops: list[FixOp] + + +class LiftablePipeEntry(BaseModel): + """One pipe the runtime may skip (lift) when an optional slot resolves absent.""" + + model_config = ConfigDict(extra="allow") + + pipe_ref: str + """Namespaced ref of the liftable pipe.""" + + within_pipe_ref: str + """Namespaced ref of the controller in whose flow the lift happens.""" + + skipped_when_absent: list[str] = Field(default_factory=list) + """The slot names whose absence lifts the pipe.""" + + absence_source: str + """Where the possible absence originates (human-readable).""" + + class ValidatedPipeEntry(BaseModel): """One entry of `PipelexValidationReport.validated_pipes[]`.""" @@ -66,10 +244,19 @@ class ValidationErrorItem(ValidationDiagnostic): populated per category and dropped from the wire when unset. Built by pipelex's one shared builder, so the hosted `InvalidReport` and the agent-CLI envelope cannot drift. + + The same item type serves `PipelexValidationReport.warnings[]`, where the locators are + serialized as explicit `null` rather than dropped. Every optional member is therefore + `T | None = None` on purpose — pydantic reads a dropped key and an explicit `null` into the + same `None`, and tightening any of them to required would break the valid arm. """ category: ValidationErrorCategory # pyright: ignore[reportIncompatibleVariableOverride] error_type: str | None = None + """Deliberately an open string, not an enum: the runtime union keeps gaining advisory + members (the `hint_*` lint types), and closing it here would turn every runtime addition + into an SDK break for no consumer benefit.""" + pipe_code: str | None = None concept_code: str | None = None domain_code: str | None = None @@ -77,8 +264,12 @@ class ValidationErrorItem(ValidationDiagnostic): field_path: str | None = None field_name: str | None = None missing_concept_code: str | None = None + missing_pipe_code: str | None = None variable_names: list[str] | None = None declared_concepts: list[str] | None = None + suggested_fix: SuggestedFix | None = None + """The structured repair proposal for this error, when the runtime's fix planner produced + one; absent otherwise.""" class PipelexValidationReport(ValidationReport): @@ -92,13 +283,38 @@ class PipelexValidationReport(ValidationReport): is_runnable: bool = True message: str = "" mthds_contents: list[str] | None = None + warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem)) + """Advisory lints on a bundle that is nonetheless valid — they never flip `is_valid`. + + Same item type as `validation_errors[]`, so one parser serves both channels. Defaults empty + rather than being required, so a body from a runner predating the field still parses; that + default is also what a clean bundle yields, so no caller can tell the two apart.""" + + liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry)) + """The pipes the runtime may skip when an optional slot resolves absent. + + Defaults empty for the same reason as `warnings`: an older runner's body must keep parsing.""" + + input_form: dict[str, Any] | None = None + """Per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`. + + Optional on purpose: it is present only when the request named the `input_form` view + (`VALIDATION_VIEW_INPUT_FORM`), and an older runner emitted it unconditionally — `None` + by default is the one typing that reads a body from either runner correctly. Kept opaque + like `bundle_blueprint`, `pipe_io_contracts` and `graph_spec`, because the descriptor + vocabulary is owned elsewhere and a second copy here would be free to drift.""" + rendered_markdown: str | None = None """Opt-in Pipelex-API presentation extra: the server-rendered Markdown view of the verdict, present only when the request asked for it (`render: ["markdown"]`); absent (None) otherwise.""" class PipelexInvalidReport(InvalidValidationReport[ValidationErrorItem]): - """The invalid arm carrying pipelex's structured `validation_errors[]` (`is_valid: false`).""" + """The invalid arm carrying pipelex's structured `validation_errors[]` (`is_valid: false`). + + It gains none of the valid arm's additions: `warnings` and `input_form` derive from a crate + that was never assembled, so the invalid arm never carries them. + """ rendered_markdown: str | None = None """Opt-in Pipelex-API presentation extra: the server-rendered Markdown view of the invalid diff --git a/tests/unit/test_client_validate.py b/tests/unit/test_client_validate.py index 8f8d028..0ceb310 100644 --- a/tests/unit/test_client_validate.py +++ b/tests/unit/test_client_validate.py @@ -1,4 +1,4 @@ -"""Tests for the `validate` override + `validate_files` — render injection, `mthds_sources`, the union round-trip.""" +"""Tests for the `validate` override + `validate_files` — render injection, `mthds_sources`, `views`, the union round-trip.""" import asyncio import json @@ -10,7 +10,7 @@ from pytest_mock import MockerFixture, MockType from pipelex_sdk.client import MthdsFile, PipelexAPIClient -from pipelex_sdk.validation_models import PipelexInvalidReport, PipelexValidationReport +from pipelex_sdk.validation_models import VALIDATION_VIEW_INPUT_FORM, PipelexInvalidReport, PipelexValidationReport _BASE_URL = "http://localhost:8081" @@ -91,6 +91,54 @@ def test_returns_invalid_report_union_arm(self, mocker: MockerFixture) -> None: assert result.rendered_markdown == "## errors" assert result.validation_errors[0].message == "boom" + # ── views ──────────────────────────────────────────────────────── + + def test_views_absent_from_body_by_default(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"])) + + # The opt-in stays opt-in: no `views` key at all, so the response is byte-identical + # for consumers that never asked for a view. + assert "views" not in self._sent_body(send) + + def test_views_sent_verbatim(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], views=[VALIDATION_VIEW_INPUT_FORM, "future_view", VALIDATION_VIEW_INPUT_FORM])) + + # Unlike `render`, nothing is injected and nothing is de-duplicated — the server + # resolves the tokens as a set and lenient-ignores the ones it does not know. + assert self._sent_body(send)["views"] == ["input_form", "future_view", "input_form"] + + def test_explicit_empty_views_is_sent_as_empty_list(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], views=[])) + + assert self._sent_body(send)["views"] == [] + + def test_views_rides_alongside_render_injection(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate(["bundle"], render=["html"], views=[VALIDATION_VIEW_INPUT_FORM])) + + body = self._sent_body(send) + assert body["render"] == ["html", "markdown"] + assert body["views"] == ["input_form"] + + def test_validate_files_threads_views_through(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, json_body=_VALID_BODY) + + asyncio.run(client.validate_files([MthdsFile(content="a")], views=[VALIDATION_VIEW_INPUT_FORM])) + + assert self._sent_body(send)["views"] == ["input_form"] + # ── validate_files ─────────────────────────────────────────────── def test_validate_files_no_uri_omits_mthds_sources(self, mocker: MockerFixture) -> None: diff --git a/tests/unit/test_validation_contract.py b/tests/unit/test_validation_contract.py index 98ef7ba..74789a3 100644 --- a/tests/unit/test_validation_contract.py +++ b/tests/unit/test_validation_contract.py @@ -1,9 +1,9 @@ """Contract round-trip tests for the 200-diagnostic `/validate` union. Pins the Pipelex validation wire models (`pipelex_sdk.validation_models`) against the -canonical example bodies from the protocol spec (`docs/specs/pipelex-mthds-protocol.md`, -`## Validation report union`). Mirrors `mthds-js/tests/unit/protocol/validation-contract.test.ts`: -parse a wire body at the boundary, discriminate on `is_valid`, and assert the narrowed arm. +canonical example bodies of the MTHDS Protocol's validation-report union. Mirrors +`mthds-js/tests/unit/protocol/validation-contract.test.ts`: parse a wire body at the boundary, +discriminate on `is_valid`, and assert the narrowed arm. """ from __future__ import annotations @@ -14,11 +14,20 @@ from pydantic import ValidationError from pipelex_sdk.validation_models import ( + DeleteKeyOp, + DeleteTableOp, DryRunStatus, + EnsureTableOp, + FixOpKind, + FixSafety, + MoveKeyOp, PipelexInvalidReport, PipelexValidationReport, PipelexValidationResult, PipelexValidationResultAdapter, + RemapValueOp, + RenameTableKeyOp, + SetKeyOp, ValidationErrorCategory, ) @@ -86,12 +95,89 @@ "message": "Validation succeeded.", } +# The 0.17+ valid arm: advisory warnings, the liftable inventory, and the opt-in input form. +# The valid arm is dumped WITHOUT `exclude_none`, so an unset locator arrives as an explicit +# `null` where the invalid arm drops the key entirely — same item type, two serializations. +# Mirrors the JS fixture "carries advisory warnings on the VALID arm, with the valid arm's +# explicit nulls" (`pipelex-sdk-js/tests/client.test.ts`). +VALID_BODY_WITH_VIEWS: dict[str, Any] = { + **VALID_BODY, + "warnings": [ + { + "category": "pipe_validation", + "message": "the `!` on `profile` is redundant — the slot is always present", + "error_type": "optional_force_redundant", + "pipe_code": "legal_contracts.summarize", + "concept_code": None, + "domain_code": None, + "source": None, + "field_path": None, + "field_name": None, + "missing_concept_code": None, + "missing_pipe_code": None, + "variable_names": None, + "declared_concepts": None, + "suggested_fix": None, + } + ], + "liftable_pipes": [ + { + "pipe_ref": "legal_contracts.enrich", + "within_pipe_ref": "legal_contracts.summarize", + "skipped_when_absent": ["profile"], + "absence_source": "optional input `profile` of legal_contracts.summarize", + } + ], + "input_form": {"legal_contracts.summarize": {"fields": [{"name": "contract", "kind": "text"}]}}, +} + +# The 0.17+ invalid arm: the new `missing_pipe_code` locator and a structured repair proposal. +INVALID_BODY_WITH_FIX: dict[str, Any] = { + "is_valid": False, + "validation_errors": [ + { + "category": "pipe_validation", + "error_type": "PipeValidationError", + "message": "Sequence step references an unknown pipe.", + "pipe_code": "summarize", + "missing_pipe_code": "enrichh", + "source": "contracts.mthds", + "suggested_fix": { + "fix_code": "match-sequence-output", + "description": "Rename the step and record its output.", + "safety": "safe", + "source": "contracts.mthds", + "ops": [ + {"kind": "rename_table_key", "table_path": ["pipe", "summarize", "steps"], "key": "enrichh", "new_key": "enrich"}, + {"kind": "set_key", "table_path": ["pipe", "summarize"], "key": "output", "value": "legal_contracts.Summary"}, + ], + }, + } + ], + "pending_signatures": [], + "is_runnable": False, + "message": "Validation found errors.", +} + def _parse(body: dict[str, Any]) -> PipelexValidationResult: """Parse a wire body through the real discriminated-union adapter — the exact parse path `PipelexAPIClient.validate()` uses.""" return PipelexValidationResultAdapter.validate_python(body) +def _body_with_single_op(fix_op: dict[str, Any]) -> dict[str, Any]: + """An invalid body whose one error carries a `suggested_fix` holding exactly `fix_op`.""" + return { + **INVALID_BODY_WITH_FIX, + "validation_errors": [ + { + **INVALID_BODY_WITH_FIX["validation_errors"][0], + "suggested_fix": {**INVALID_BODY_WITH_FIX["validation_errors"][0]["suggested_fix"], "ops": [fix_op]}, + } + ], + } + + class TestValidationContract: def test_valid_arm_carries_typed_artifacts(self) -> None: """The valid arm parses to a typed report with structural artifacts.""" @@ -163,8 +249,116 @@ def test_rendered_markdown_is_none_when_absent(self) -> None: assert isinstance(invalid, PipelexInvalidReport) assert invalid.rendered_markdown is None + # ── The 0.17+ valid arm: warnings, liftable pipes, the opt-in input form ── + + def test_valid_arm_carries_warnings_liftable_pipes_and_input_form(self) -> None: + """The 0.17+ valid-arm additions parse into typed fields, `input_form` keyed like `pipe_io_contracts`.""" + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + # Advisory items never flip the verdict. + assert report.is_valid is True + warning = report.warnings[0] + assert warning.category is ValidationErrorCategory.PIPE_VALIDATION + assert warning.error_type == "optional_force_redundant" + assert warning.pipe_code == "legal_contracts.summarize" + liftable = report.liftable_pipes[0] + assert liftable.pipe_ref == "legal_contracts.enrich" + assert liftable.within_pipe_ref == "legal_contracts.summarize" + assert liftable.skipped_when_absent == ["profile"] + assert liftable.absence_source == "optional input `profile` of legal_contracts.summarize" + assert report.input_form is not None + # Keyed exactly like `pipe_io_contracts`, and opaque on purpose. + assert set(report.input_form) == set(report.pipe_io_contracts) + + def test_valid_arm_warning_reads_every_explicit_null_as_none(self) -> None: + """Every explicitly-null locator on a warning reads as `None`. + + The valid arm is dumped without `exclude_none`, so an unset locator arrives as an + explicit `null` where the invalid arm drops the key. This is the regression guard + against a future "tighten to required" edit on `ValidationErrorItem`. + """ + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + warning = report.warnings[0] + assert warning.concept_code is None + assert warning.domain_code is None + assert warning.source is None + assert warning.field_path is None + assert warning.field_name is None + assert warning.missing_concept_code is None + assert warning.missing_pipe_code is None + assert warning.variable_names is None + assert warning.declared_concepts is None + assert warning.suggested_fix is None + + def test_pre_0_52_valid_body_still_parses_with_empty_defaults(self) -> None: + """A body from a runner predating the fields parses: both lists empty, `input_form` None.""" + report = _parse(VALID_BODY) + assert isinstance(report, PipelexValidationReport) + assert report.warnings == [] + assert report.liftable_pipes == [] + assert report.input_form is None + + # ── The 0.17+ invalid arm: `missing_pipe_code` and the fix vocabulary ── + + def test_invalid_arm_carries_missing_pipe_code_and_narrowable_fix_ops(self) -> None: + """A structured `suggested_fix` parses, and `match`-narrowing reaches each op's own members.""" + report = _parse(INVALID_BODY_WITH_FIX) + assert isinstance(report, PipelexInvalidReport) + item = report.validation_errors[0] + assert item.missing_pipe_code == "enrichh" + fix = item.suggested_fix + assert fix is not None + assert fix.fix_code == "match-sequence-output" + assert fix.safety is FixSafety.SAFE + assert fix.safety.is_safe is True + assert fix.source == "contracts.mthds" + + rename_op, set_op = fix.ops + # Narrowing is an exhaustive `match` over the op classes — the Python spelling of the + # JS mirror's `kind` narrowing. + match rename_op: + case RenameTableKeyOp(): + assert rename_op.table_path == ["pipe", "summarize", "steps"] + assert rename_op.key == "enrichh" + assert rename_op.new_key == "enrich" + case SetKeyOp() | EnsureTableOp() | DeleteKeyOp() | DeleteTableOp() | MoveKeyOp() | RemapValueOp(): + pytest.fail("expected a rename_table_key op") + match set_op: + case SetKeyOp(): + assert set_op.table_path == ["pipe", "summarize"] + assert set_op.key == "output" + assert set_op.value == "legal_contracts.Summary" + case EnsureTableOp() | DeleteKeyOp() | DeleteTableOp() | RenameTableKeyOp() | MoveKeyOp() | RemapValueOp(): + pytest.fail("expected a set_key op") + + def test_ensure_table_op_rejects_an_empty_table_path(self) -> None: + """`ensure_table` addresses the table itself, so its path is never empty (artifact `minItems: 1`).""" + bad_body = _body_with_single_op({"kind": "ensure_table", "table_path": []}) + with pytest.raises(ValidationError, match="table_path"): + PipelexValidationResultAdapter.validate_python(bad_body) + + def test_unknown_fix_op_kind_is_rejected(self) -> None: + """An out-of-vocabulary op kind fails the whole verdict parse — the discriminator is closed.""" + bad_body = _body_with_single_op({"kind": "invent_key", "table_path": ["pipe"], "key": "x"}) + with pytest.raises(ValidationError, match="invent_key"): + PipelexValidationResultAdapter.validate_python(bad_body) + + def test_fix_vocabularies_are_the_locked_sets(self) -> None: + """`FixSafety` and `FixOpKind` mirror the runtime's closed vocabularies (drift guard).""" + assert {safety.value for safety in FixSafety} == {"safe", "unsafe"} + assert {kind.value for kind in FixOpKind} == { + "set_key", + "ensure_table", + "delete_key", + "delete_table", + "rename_table_key", + "move_key", + "remap_value", + } + def test_category_vocabulary_is_the_locked_set(self) -> None: - """The closed category set mirrors `conformance/.../validation_contract.py` (drift guard).""" + """The closed category set mirrors the locked conformance vocabulary (drift guard).""" assert {category.value for category in ValidationErrorCategory} == { "blueprint_validation", "pipe_factory", From 2a8c58938233f410d44a08f431bf2815435fe1d7 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:25:55 +0200 Subject: [PATCH 09/20] Correct the TokensUsageRecord attribution and drop unopenable citations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two prose fixes `@pipelex/sdk` 0.14.0 shipped, which apply here for the same reason: `pipelex-sdk` is a public PyPI package. `TokensUsageRecord` was described as "specified in the MTHDS protocol spec". It is not — inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins the wire contract and this SDK mirrors the runtime's own record. Getting that wrong misplaces the brand boundary the rest of the architecture is organized around. Several docstrings, comments and doc passages cited internal specs and the conformance suite by bare repo-relative path. Those resolve to nothing for anyone who clones this repo, so they read as rot rather than as a deliberate boundary; each now states the rule it was citing. References to the sibling public pipelex-sdk-js repo are kept where they explain a port. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 6 +++ TODOS.md | 75 ++++++++++++++++------------- docs/architecture.md | 2 +- docs/run-usage.md | 2 +- pipelex_sdk/client.py | 4 +- pipelex_sdk/runs.py | 14 +++--- tests/unit/test_client_method_id.py | 6 +-- tests/unit/test_runs.py | 4 +- 8 files changed, 64 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c29e642..7602a43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,12 @@ - **`too-many-statements-in-try-clause` is now ignored explicitly, ahead of ever having a finding.** Every repo in the workspace that runs `select = ["ALL"]` with `preview` has settled on ignoring this rule, because shrinking a `try` clause changes which statements its handlers cover — an error-handling refactor rather than lint cleanup. This repo happened to have no findings, so the decision was never written down here, which meant whoever wrote the first long `try` clause would have had to reach it again on their own. The reasoning now sits inline next to the ignore. No code changes; the rule fires nowhere today. + +### Fixed + +- **Documentation: `TokensUsageRecord` is attributed to the right brand.** The module and class docstrings in `pipelex_sdk/runs.py`, `docs/run-usage.md`, and `docs/architecture.md` said the record was "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins the wire contract and this SDK mirrors the runtime's own record. Getting the attribution wrong on a public package misplaces the brand boundary the rest of the architecture is organized around. No behaviour changed. +- **Documentation: no more citations a reader cannot open.** Several docstrings, comments and doc passages cited internal specs and the conformance suite by bare repo-relative path — paths that resolve to nothing for anyone who clones this public package, so they read as rot rather than as a deliberate boundary. Each site now states the rule it was citing instead: the layered extension policy behind the hosted run extensions and the reserved-`extra` guard, and the locked category vocabulary shared with the conformance corpus. References to the sibling public `pipelex-sdk-js` repo are kept where they explain a port. No behaviour changed. + ## [v0.5.0] - 2026-07-22 ### Added diff --git a/TODOS.md b/TODOS.md index 4b1e834..40fb925 100644 --- a/TODOS.md +++ b/TODOS.md @@ -26,60 +26,67 @@ Purely additive. Every affected response model is `extra="allow"`, so nothing pa ### 1.1 `pipelex_sdk/validation_models.py` — the fix vocabulary and the new fields -- [ ] Add `FixSafety(StrEnum)` with `SAFE = "safe"`, `UNSAFE = "unsafe"`, and an `is_safe` property (house rule: never compare enum values inline; a `match` inside the property). -- [ ] Add `FixOpKind(StrEnum)` with the kinds the runtime emits: `SET_KEY = "set_key"`, `ENSURE_TABLE = "ensure_table"`, `DELETE_KEY = "delete_key"`, `DELETE_TABLE = "delete_table"`, `RENAME_TABLE_KEY = "rename_table_key"`, `MOVE_KEY = "move_key"`, `REMAP_VALUE = "remap_value"`. Source of truth: `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact `pipelex-api/docs/openapi/pipelex-api.openapi.yaml`. -- [ ] Add `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]`. Deeper nesting is not modelled because the server does not emit it; say so in a comment. -- [ ] Add a private `_FixOpBase(BaseModel)` with `model_config = ConfigDict(extra="allow")` and `table_path: list[str]` (empty list means the document root), then one subclass per kind, each with `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key: str, value: TomlValue)`, `EnsureTableOp()`, `DeleteKeyOp(key: str)`, `DeleteTableOp()`, `RenameTableKeyOp(key: str, new_key: str)`, `MoveKeyOp(key: str, new_table_path: list[str], new_key: str)`, `RemapValueOp(key: str, mapping: dict[str, str])`. On `EnsureTableOp` and `DeleteTableOp` declare `table_path: list[str] = Field(min_length=1)`, mirroring the artifact's `minItems: 1`. -- [ ] These are **reader** models: no `frozen`, no `extra="forbid"`, none of the runtime's wildcard-refusing validators. Put the two runtime invariants a type cannot carry in the docstrings (`*` is the wildcard segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`). -- [ ] Add `FixOp: TypeAlias = Annotated[SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, Field(discriminator="kind")]`. Narrowing is `match op: case SetKeyOp(): …`, exhaustive, no `case _`. -- [ ] **Verify** that pydantic accepts the raw wire string (`"set_key"`) against `Literal[FixOpKind.SET_KEY]` both in `validate_python` and `validate_json`, and that the discriminator resolves on it. If it does not on the pinned pydantic, use `Literal["set_key"]` on the models and keep `FixOpKind` as the documented vocabulary (with a test that the two sets agree). -- [ ] Add `SuggestedFix(BaseModel, extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]` (typed default factory via `empty_list_factory_of` only if a default is warranted — the runtime always sends `ops`, so leaving it required is fine). -- [ ] Add `LiftablePipeEntry(BaseModel, extra="allow")`: `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str] = Field(default_factory=list)`, `absence_source: str`. Mirrors `pipelex/pipelex/pipeline/liftable_pipes.py`. -- [ ] Add the view token constant next to the field it gates: `VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form"`. A constant, not a closed enum — the request boundary is deliberately open so a stale token never fails a call. -- [ ] On `ValidationErrorItem` add `missing_pipe_code: str | None = None` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. Leave `error_type: str | None` as an open string; do not enum it. -- [ ] On `PipelexValidationReport` add `warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem))`, `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry))`, and `input_form: dict[str, Any] | None = None`, each with a docstring: `warnings` never flips `is_valid`; the two lists default empty so a pre-0.52 runner's body still parses; `input_form` is present only when the request named the `input_form` view (a 0.17.0 runner emitted it unconditionally, which `None`-by-default also reads correctly) and is opaque on purpose, keyed like `pipe_io_contracts`. -- [ ] `PipelexInvalidReport` gains nothing; add one sentence to its docstring saying why (the invalid arm never carries `warnings` or `input_form` — they derive from a crate that was never assembled). -- [ ] Update the module docstring's list of neutrally-named supporting types to include the new ones, and keep the brand rule stated there (the `Pipelex` prefix stays on the two envelopes only). +- [x] Add `FixSafety(StrEnum)` with `SAFE = "safe"`, `UNSAFE = "unsafe"`, and an `is_safe` property (house rule: never compare enum values inline; a `match` inside the property). +- [x] Add `FixOpKind(StrEnum)` with the kinds the runtime emits: `SET_KEY = "set_key"`, `ENSURE_TABLE = "ensure_table"`, `DELETE_KEY = "delete_key"`, `DELETE_TABLE = "delete_table"`, `RENAME_TABLE_KEY = "rename_table_key"`, `MOVE_KEY = "move_key"`, `REMAP_VALUE = "remap_value"`. Source of truth: `pipelex/pipelex/suggested_fix.py` and the OpenAPI artifact `pipelex-api/docs/openapi/pipelex-api.openapi.yaml`. +- [x] Add `TomlScalar: TypeAlias = str | int | float | bool` and `TomlValue: TypeAlias = TomlScalar | dict[str, TomlScalar]`. Deeper nesting is not modelled because the server does not emit it; say so in a comment. +- [x] Add a private `_FixOpBase(BaseModel)` with `model_config = ConfigDict(extra="allow")` and `table_path: list[str]` (empty list means the document root), then one subclass per kind, each with `kind: Literal[FixOpKind.X]` and exactly its own members: `SetKeyOp(key: str, value: TomlValue)`, `EnsureTableOp()`, `DeleteKeyOp(key: str)`, `DeleteTableOp()`, `RenameTableKeyOp(key: str, new_key: str)`, `MoveKeyOp(key: str, new_table_path: list[str], new_key: str)`, `RemapValueOp(key: str, mapping: dict[str, str])`. On `EnsureTableOp` and `DeleteTableOp` declare `table_path: list[str] = Field(min_length=1)`, mirroring the artifact's `minItems: 1`. +- [x] These are **reader** models: no `frozen`, no `extra="forbid"`, none of the runtime's wildcard-refusing validators. Put the two runtime invariants a type cannot carry in the docstrings (`*` is the wildcard segment and is refused as a `key` on every kind but `remap_value`; `ensure_table` / `delete_table` need a non-empty `table_path`). +- [x] Add `FixOp: TypeAlias = Annotated[SetKeyOp | EnsureTableOp | DeleteKeyOp | DeleteTableOp | RenameTableKeyOp | MoveKeyOp | RemapValueOp, Field(discriminator="kind")]`. Narrowing is `match op: case SetKeyOp(): …`, exhaustive, no `case _`. +- [x] **Verify** that pydantic accepts the raw wire string (`"set_key"`) against `Literal[FixOpKind.SET_KEY]` both in `validate_python` and `validate_json`, and that the discriminator resolves on it. If it does not on the pinned pydantic, use `Literal["set_key"]` on the models and keep `FixOpKind` as the documented vocabulary (with a test that the two sets agree). +- [x] Add `SuggestedFix(BaseModel, extra="allow")`: `fix_code: str`, `description: str`, `safety: FixSafety`, `source: str | None = None`, `ops: list[FixOp]` (typed default factory via `empty_list_factory_of` only if a default is warranted — the runtime always sends `ops`, so leaving it required is fine). +- [x] Add `LiftablePipeEntry(BaseModel, extra="allow")`: `pipe_ref: str`, `within_pipe_ref: str`, `skipped_when_absent: list[str] = Field(default_factory=list)`, `absence_source: str`. Mirrors `pipelex/pipelex/pipeline/liftable_pipes.py`. +- [x] Add the view token constant next to the field it gates: `VALIDATION_VIEW_INPUT_FORM: Final[str] = "input_form"`. A constant, not a closed enum — the request boundary is deliberately open so a stale token never fails a call. +- [x] On `ValidationErrorItem` add `missing_pipe_code: str | None = None` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None = None`. Leave `error_type: str | None` as an open string; do not enum it. +- [x] On `PipelexValidationReport` add `warnings: list[ValidationErrorItem] = Field(default_factory=empty_list_factory_of(ValidationErrorItem))`, `liftable_pipes: list[LiftablePipeEntry] = Field(default_factory=empty_list_factory_of(LiftablePipeEntry))`, and `input_form: dict[str, Any] | None = None`, each with a docstring: `warnings` never flips `is_valid`; the two lists default empty so a pre-0.52 runner's body still parses; `input_form` is present only when the request named the `input_form` view (a 0.17.0 runner emitted it unconditionally, which `None`-by-default also reads correctly) and is opaque on purpose, keyed like `pipe_io_contracts`. +- [x] `PipelexInvalidReport` gains nothing; add one sentence to its docstring saying why (the invalid arm never carries `warnings` or `input_form` — they derive from a crate that was never assembled). +- [x] Update the module docstring's list of neutrally-named supporting types to include the new ones, and keep the brand rule stated there (the `Pipelex` prefix stays on the two envelopes only). ### 1.2 `pipelex_sdk/client.py` — `views` on `validate` and `validate_files` -- [ ] `validate(...)`: append `views: list[str] | None = None` after `render`. When `views is not None`, set `extra["views"] = views` **verbatim** — no injection, no de-duplication, an explicit `[]` is sent as `[]`. When `None`, the key is absent from the body. It rides `_post_validate`'s `extra` exactly like `render` and `mthds_sources`; no `mthds-python` change. -- [ ] `validate_files(...)`: append `views: list[str] | None = None` after `render` and thread it through to `validate`. -- [ ] Docstrings: the sentence "differs from the inherited protocol `validate` in two Pipelex-API ways" becomes three (render injection, `mthds_sources`, `views`); document the `views` semantics (opt-in structured views; `input_form` is the only token today, named by `VALIDATION_VIEW_INPUT_FORM`; unknown tokens are lenient-ignored server-side, never a `422`; the default response stays byte-identical for consumers that discard views). -- [ ] The `Returns:` section of `validate` should mention that a valid report now carries `warnings`, `liftable_pipes`, and (when asked) `input_form`. +- [x] `validate(...)`: append `views: list[str] | None = None` after `render`. When `views is not None`, set `extra["views"] = views` **verbatim** — no injection, no de-duplication, an explicit `[]` is sent as `[]`. When `None`, the key is absent from the body. It rides `_post_validate`'s `extra` exactly like `render` and `mthds_sources`; no `mthds-python` change. +- [x] `validate_files(...)`: append `views: list[str] | None = None` after `render` and thread it through to `validate`. +- [x] Docstrings: the sentence "differs from the inherited protocol `validate` in two Pipelex-API ways" becomes three (render injection, `mthds_sources`, `views`); document the `views` semantics (opt-in structured views; `input_form` is the only token today, named by `VALIDATION_VIEW_INPUT_FORM`; unknown tokens are lenient-ignored server-side, never a `422`; the default response stays byte-identical for consumers that discard views). +- [x] The `Returns:` section of `validate` should mention that a valid report now carries `warnings`, `liftable_pipes`, and (when asked) `input_form`. ### 1.3 Tests -- [ ] `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the `views` key absent from the body when the parameter is omitted; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` injection unchanged when `views` is also passed. -- [ ] `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes`, and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None` — this is the regression guard against a future "tighten to required" edit, and it answers the inbox item `../wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` for the Python mirror. -- [ ] `tests/unit/test_validation_contract.py`: an invalid body carrying `missing_pipe_code` and a `suggested_fix` with at least two ops of different kinds parses, and `match`-narrowing reaches each op's own members; an `ensure_table` op with an empty `table_path` is rejected; an unknown `kind` raises `pydantic.ValidationError`; `FixSafety` and `FixOpKind` value sets are asserted as the locked vocabularies (same style as `test_category_vocabulary_is_the_locked_set`). -- [ ] Keep the canonical bodies where the module already keeps them (module-level constants next to `VALID_BODY`); move to a `tests/unit/test_data.py` only if the module becomes unreadable. +- [x] `tests/unit/test_client_validate.py`: `views` sent verbatim when given; the `views` key absent from the body when the parameter is omitted; an explicit `[]` sent as `[]`; `validate_files` threads `views` through; `render` injection unchanged when `views` is also passed. +- [x] `tests/unit/test_validation_contract.py`: a valid body carrying `warnings`, `liftable_pipes`, and `input_form` parses into typed fields, with `input_form` keyed like `pipe_io_contracts`; the pre-0.52 `VALID_BODY` still parses with both lists empty and `input_form` `None`; the JS null-bearing warning fixture (`pipelex-sdk-js/tests/client.test.ts`, "carries advisory warnings on the VALID arm, with the valid arm's explicit nulls") parses with every explicit `null` reading as `None` — this is the regression guard against a future "tighten to required" edit, and it answers the inbox item `../wip/inbox/2026-08-25-workspace-validation-error-item-spec-gaps.md` for the Python mirror. +- [x] `tests/unit/test_validation_contract.py`: an invalid body carrying `missing_pipe_code` and a `suggested_fix` with at least two ops of different kinds parses, and `match`-narrowing reaches each op's own members; an `ensure_table` op with an empty `table_path` is rejected; an unknown `kind` raises `pydantic.ValidationError`; `FixSafety` and `FixOpKind` value sets are asserted as the locked vocabularies (same style as `test_category_vocabulary_is_the_locked_set`). +- [x] Keep the canonical bodies where the module already keeps them (module-level constants next to `VALID_BODY`); move to a `tests/unit/test_data.py` only if the module becomes unreadable. ### 1.4 Docs and changelog -- [ ] `docs/architecture.md` → "`validate` override": add a `views` bullet beside the render bullet; list the typed valid-arm additions (`warnings`, `liftable_pipes`, `input_form`) and the `ValidationErrorItem` additions with the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary; one sentence that `PipeInputContract.optional` became `presence` and that the `fixed` multiplicity carries `item_count` inside the opaque `pipe_io_contracts`, so nobody discovers the new spellings by surprise. -- [ ] `docs/architecture.md` → "Brand boundary": the list of neutrally-named supporting types gains the new ones. -- [ ] `README.md` quickstart: one line showing `views=[VALIDATION_VIEW_INPUT_FORM]` (or a comment that the input form is opt-in), so the knob is discoverable. -- [ ] `CHANGELOG.md` `[Unreleased]` → **Added**: `views` on `validate` / `validate_files`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the fix vocabulary; a note that a body from an older runner still parses (the lists default empty, `input_form` defaults `None`). +- [x] `docs/architecture.md` → "`validate` override": add a `views` bullet beside the render bullet; list the typed valid-arm additions (`warnings`, `liftable_pipes`, `input_form`) and the `ValidationErrorItem` additions with the `SuggestedFix` / `FixOp` / `FixSafety` vocabulary; one sentence that `PipeInputContract.optional` became `presence` and that the `fixed` multiplicity carries `item_count` inside the opaque `pipe_io_contracts`, so nobody discovers the new spellings by surprise. +- [x] `docs/architecture.md` → "Brand boundary": the list of neutrally-named supporting types gains the new ones. +- [x] `README.md` quickstart: one line showing `views=[VALIDATION_VIEW_INPUT_FORM]` (or a comment that the input form is opt-in), so the knob is discoverable. +- [x] `CHANGELOG.md` `[Unreleased]` → **Added**: `views` on `validate` / `validate_files`; the typed valid-arm fields; `missing_pipe_code` / `suggested_fix` and the fix vocabulary; a note that a body from an older runner still parses (the lists default empty, `input_form` defaults `None`). ### 1.5 Gate and commit -- [ ] `make agent-check` and `make agent-test` pass. -- [ ] Commit (suggested message: "Type the pipelex-api 0.17/0.18 validate contract and add the views opt-in"). +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Type the pipelex-api 0.17/0.18 validate contract and add the views opt-in"). ### Checkpoint 1 -- [ ] Update this file: tick what landed, record the SHA of the Phase 1 commit, note whether pydantic accepted the enum `Literal` tags or the string fallback was needed, and any deviation from §1 of the design with its reason. +- [x] Update this file: tick what landed, record the SHA of the Phase 1 commit, note whether pydantic accepted the enum `Literal` tags or the string fallback was needed, and any deviation from §1 of the design with its reason. + +**Landed in `434b2e3`.** Notes: + +- **The enum `Literal` tags work as written; the string fallback was not needed.** Verified against the pinned pydantic (2.13.4) before writing the models: a raw wire `"set_key"` validates against `Literal[FixOpKind.SET_KEY]` in both `validate_python` and `validate_json`, the discriminator resolves on it, an unknown `kind` raises, and `Field(min_length=1)` on `EnsureTableOp.table_path` rejects an empty path. +- **`RemapValueOp.mapping` is left unconstrained**, where the runtime and the OpenAPI artifact both declare `minProperties: 1`. These are reader models: an empty mapping is an advisory no-op, not a parse hazard, and refusing it would fail a whole verdict over a harmless op. The `minItems: 1` on `ensure_table` / `delete_table` was kept because there the empty case is genuinely meaningless (the document root always exists, and cannot be deleted). +- **One Phase-2 item landed early**, because `validation_models.py` was rewritten wholesale here: the `conformance/conformance/validation_contract.py` citation on `ValidationErrorCategory` is already replaced with the rule it was citing. Phase 2 covers the rest. +- No other deviation from §1 of the design. ## Phase 2 — prose corrections (`wip/updates.md` §2) No behaviour change. Two fixes `@pipelex/sdk` 0.14.0 shipped under "Fixed" that apply here for the same reason (`pipelex-sdk` is a public PyPI package). -- [ ] **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py` (module docstring near line 23 and the class docstring near line 128), `docs/run-usage.md` (line 5), `docs/architecture.md` (the `TokensUsageRecord` bullet near line 103): the record is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API pins the wire contract. Reword as `pipelex-sdk-js/src/runs.ts` and its `docs/architecture.md` did. `docs/run-usage.md` line 5 already says the right thing in its second sentence; make the first sentence agree with it. -- [ ] **Citations a reader cannot open.** Replace each bare workspace-private path with the rule it was citing: `pipelex_sdk/client.py` near line 138 (`docs/specs/pipelex-platform-api.md` → "the layered extension policy: a hosted client types its own platform's arguments and guards them per layer"); `pipelex_sdk/validation_models.py` near line 44 (`conformance/conformance/validation_contract.py` → "the locked category vocabulary shared with the conformance corpus"); `tests/unit/test_validation_contract.py` module docstring and the docstring of `test_category_vocabulary_is_the_locked_set`; `tests/unit/test_runs.py` near line 12; `tests/unit/test_client_method_id.py` module docstring; `docs/architecture.md` near line 84. Keep the JS mirror references (`pipelex-sdk-js/...`) where they explain a port — those are a sibling public repo, not a private path. -- [ ] `CHANGELOG.md` `[Unreleased]` → **Fixed**: two entries mirroring 0.14.0's wording. -- [ ] `make agent-check` and `make agent-test` pass. -- [ ] Commit (suggested message: "Correct the TokensUsageRecord attribution and drop unopenable citations"). +- [x] **`TokensUsageRecord` attribution.** `pipelex_sdk/runs.py` (module docstring near line 23 and the class docstring near line 128), `docs/run-usage.md` (line 5), `docs/architecture.md` (the `TokensUsageRecord` bullet near line 103): the record is a Pipelex runtime extension the MTHDS Protocol does not model, and the hosted API pins the wire contract. Reword as `pipelex-sdk-js/src/runs.ts` and its `docs/architecture.md` did. `docs/run-usage.md` line 5 already says the right thing in its second sentence; make the first sentence agree with it. +- [x] **Citations a reader cannot open.** Replace each bare workspace-private path with the rule it was citing: `pipelex_sdk/client.py` near line 138 (`docs/specs/pipelex-platform-api.md` → "the layered extension policy: a hosted client types its own platform's arguments and guards them per layer"); `pipelex_sdk/validation_models.py` near line 44 (`conformance/conformance/validation_contract.py` → "the locked category vocabulary shared with the conformance corpus"); `tests/unit/test_validation_contract.py` module docstring and the docstring of `test_category_vocabulary_is_the_locked_set`; `tests/unit/test_runs.py` near line 12; `tests/unit/test_client_method_id.py` module docstring; `docs/architecture.md` near line 84. Keep the JS mirror references (`pipelex-sdk-js/...`) where they explain a port — those are a sibling public repo, not a private path. +- [x] `CHANGELOG.md` `[Unreleased]` → **Fixed**: two entries mirroring 0.14.0's wording. +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Correct the TokensUsageRecord attribution and drop unopenable citations"). ## Phase 3 — product paging and nullability (`wip/updates.md` §3) diff --git a/docs/architecture.md b/docs/architecture.md index 8427bef..d514c8d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -81,7 +81,7 @@ The override also **enriches the return type**: it re-validates the base result The protocol's run args (`pipe_code`, `mthds_contents`, `inputs`, …) are the base client's named parameters and stay pure. The hosted API's *own* run args are named parameters **here**, on `execute` / `start` / `start_and_wait`. Today that is one, `method_id`, and the reasoning generalizes to every hosted-only argument that follows: -- **A named parameter, not an `extra` entry.** Typing its own platform's arguments is the one job a layer-3 client exists to do; `extra` stays the escape hatch for an extension this client does not know about (a third-party server, or a newer version of ours). That split, and the reserved-key guard behind it, are normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy" (Rule 5). It was previously passed as `extra={"method_id": …}`, which worked but documented nothing and validated nothing. +- **A named parameter, not an `extra` entry.** Typing its own platform's arguments is the one job a layer-3 client exists to do; `extra` stays the escape hatch for an extension this client does not know about (a third-party server, or a newer version of ours). That split, and the reserved-key guard behind it, follow the layered extension policy: a hosted client types its own platform's arguments and guards them per layer, and that guard must never be pushed down into the protocol package. It was previously passed as `extra={"method_id": …}`, which worked but documented nothing and validated nothing. - **It reaches the wire through the base client's extension mechanism.** `_merge_hosted_run_extensions` folds it into the `extra` mapping handed to `super().execute` / `super().start`, which merges it into the body as a top-level property without knowing what it means. That is the layering working as designed, not a workaround: the protocol client stays catalog-agnostic while the hosted client owns the concept. - **The guard is per layer.** `method_id` is rejected inside `extra` *here*, and must never become reserved in `mthds`: a protocol client talking to another vendor's server has no business rejecting that vendor's arguments. - **Pure pass-through — nothing is expanded client-side.** The platform resolves the id against the org's catalog. Do not confuse this with `build_inputs(method_id=…)` and `prepare_inputs`, which are client-side *sugar* that resolves an id to inline files before the request is built, so no `method_id` ever reaches those wire bodies. diff --git a/docs/run-usage.md b/docs/run-usage.md index 2c70606..e9781b0 100644 --- a/docs/run-usage.md +++ b/docs/run-usage.md @@ -2,7 +2,7 @@ A completed run reports what its inference calls consumed as a list of `TokensUsageRecord` objects on `RunResults`, one per inference call, in the order the calls completed. This page covers how to read them, what each field means, and the edge cases the model is deliberately shaped around. -The wire shape is not this SDK's invention: it is specified in Pipelex's protocol spec, under "TokensUsage records on run artifacts", and `pipelex_sdk.runs.TokensUsageRecord` is a client-side mirror of it. `@pipelex/sdk` carries the same mirror in TypeScript. The record is a Pipelex runtime concept rather than part of the MTHDS standard — the MTHDS protocol itself says nothing about usage reporting. +The wire shape is not this SDK's invention. Inference accounting is a Pipelex runtime extension — the MTHDS Protocol does not model it, and says nothing about usage reporting — so the hosted API is what pins the contract, and `pipelex_sdk.runs.TokensUsageRecord` is a client-side mirror of the runtime's own record. `@pipelex/sdk` carries the same mirror in TypeScript. ## Reading the records diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index c76c7e4..f744005 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -134,8 +134,8 @@ # Reserved on `extra` for the same reason the protocol args are: one argument must not arrive # by two paths with different validation. The guard is deliberately PER LAYER — it lives here # and must never be pushed down into `mthds`, because a protocol client talking to another -# vendor's server has no business rejecting that vendor's arguments. See the workspace spec -# `docs/specs/pipelex-platform-api.md` -> "Layered extension policy" (Rule 5). +# vendor's server has no business rejecting that vendor's arguments. This is the layered +# extension policy: a hosted client types its own platform's arguments and guards them per layer. _HOSTED_RUN_ARGS: frozenset[str] = frozenset({"method_id"}) diff --git a/pipelex_sdk/runs.py b/pipelex_sdk/runs.py index 9aa6cf6..ffba5f3 100644 --- a/pipelex_sdk/runs.py +++ b/pipelex_sdk/runs.py @@ -20,8 +20,9 @@ than redefine. `RunResults.pipe_output` is typed with the protocol's own `DictPipeOutputAbstract` wire model from `mthds` — a shared wire contract the `pipelex` runtime also builds on, not a lifecycle concept. `TokensUsageRecord` -mirrors a runtime wire contract specified in the MTHDS protocol spec; this SDK -follows that shape, it does not define it. +mirrors the runtime's own record: inference accounting is a Pipelex runtime +extension the MTHDS Protocol does not model, so the hosted API is what pins that +wire contract; this SDK follows the shape, it does not define it. Wire contract mirrors `pipelex-platform`: POST /v1/start -> RunResultStart (start, 202) @@ -125,10 +126,11 @@ class RunRead(RunPublic): class TokensUsageRecord(BaseModel): """One inference call's token usage — the client-facing wire record. - Mirrors the runtime's `TokensUsageRecord`, specified in - `docs/specs/pipelex-mthds-protocol.md#tokensusage-records-on-run-artifacts`. The same - shape rides both surfaces: the durable `tokens_usages.json` artifact that the hosted - results route relays, and the blocking execute response's `pipe_output.tokens_usages`. + Mirrors the runtime's `TokensUsageRecord`. Inference accounting is a Pipelex runtime + extension — the MTHDS Protocol does not model it — so the hosted API is what pins this + wire contract. The same shape rides both surfaces: the durable `tokens_usages.json` + artifact that the hosted results route relays, and the blocking execute response's + `pipe_output.tokens_usages`. Every field is optional and the model is extension-open **on purpose**. A record the current runtime emits always carries the full key set (a field with no value is an diff --git a/tests/unit/test_client_method_id.py b/tests/unit/test_client_method_id.py index cc37408..78bf96d 100644 --- a/tests/unit/test_client_method_id.py +++ b/tests/unit/test_client_method_id.py @@ -1,9 +1,9 @@ """Tests for the hosted `method_id` run option — the layer-3 extension this SDK names itself. Mirrors `pipelex-sdk-js/tests/client.test.ts` "hosted method_id option". The doctrine the -assertions pin is normative in the workspace spec `docs/specs/pipelex-platform-api.md` -> -"Layered extension policy": a hosted client types its own platform's arguments (Rule 5), and -`extra` stays the escape hatch for extensions it does not know about. +assertions pin is the layered extension policy: a hosted client types its own platform's +arguments and guards them per layer, and `extra` stays the escape hatch for the extensions it +does not know about. """ import asyncio diff --git a/tests/unit/test_runs.py b/tests/unit/test_runs.py index eb98a5a..35536fa 100644 --- a/tests/unit/test_runs.py +++ b/tests/unit/test_runs.py @@ -8,8 +8,8 @@ from pipelex_sdk.runs import RunResults, RunStatus, TokensUsageRecord # A record in the shape the current runtime emits: every contract field present, absent values -# sent as explicit nulls. Mirrors the conformance seed corpus -# (conformance/conformance/usage_records.py), which is what the platform arm asserts on the wire. +# sent as explicit nulls. Mirrors the shared conformance corpus of usage records, which is what +# the platform arm asserts on the wire. _RATED_RECORD: dict[str, Any] = { "model_type": "llm", "inference_model_name": "test-model", From 5e01c1b47f363bcc89972dbc7d274ca1f063da20 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:34:14 +0200 Subject: [PATCH 10/20] Follow the platform's paged method and run lists and stop requiring nullable run fields MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `list_methods` and `list_runs` crashed against the deployed platform. Both routes now answer a `{items, next_cursor}` envelope — methods since 2026-08-18, runs since 2026-08-11, both live in production — and this SDK still iterated a bare array. Iterating the envelope dict yields its keys, so the very first call was `MethodData.model_validate("items")` and every invocation raised. Nothing was red because the unit tests mocked the pre-paging shape. `list_methods` now returns a `MethodPage` of `MethodSummary` rows and `list_runs` a `RunPage`, each with the paging and filter parameters the platform serves, kept on presence rather than truthiness. `iterate_methods` and `iterate_runs` follow the cursors; there is deliberately no `list_all_methods()`, because an all-at-once helper needs a cap and a cap is the silent truncation paging removed. The two iterators stop on different signals and the difference is in the server: a method page can be empty with a live cursor because `q` filters after the index read, a run page cannot because its date bounds are key conditions. `PipelineRun.method_id` and `pipe_code` are nullable, which the platform genuinely serves and both cases are ordinary. `MethodData.python` is a typed `list[MethodFile]` converted at the boundary, so the platform's three-way write contract falls out of `exclude_none=True`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 14 ++ TODOS.md | 62 ++++----- docs/architecture.md | 34 ++++- pipelex_sdk/client.py | 157 ++++++++++++++++++++-- pipelex_sdk/errors.py | 16 +++ pipelex_sdk/product_models.py | 212 ++++++++++++++++++++++++++++-- tests/unit/conftest.py | 55 ++++++++ tests/unit/test_client_paging.py | 175 ++++++++++++++++++++++++ tests/unit/test_client_product.py | 163 +++++++++++++++++++++-- tests/unit/test_method_files.py | 78 +++++++++++ 10 files changed, 896 insertions(+), 70 deletions(-) create mode 100644 tests/unit/conftest.py create mode 100644 tests/unit/test_client_paging.py create mode 100644 tests/unit/test_method_files.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7602a43..b6ab78e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,10 +12,20 @@ - **Structured repair proposals: `SuggestedFix` and the fix-op vocabulary.** `ValidationErrorItem.suggested_fix` is now typed. A `SuggestedFix` carries a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops` — a list discriminated on `kind` over the closed `FixOpKind` set (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), each variant declaring exactly its own members, narrowed with an exhaustive `match op: case SetKeyOp(): …`. These are **reader** models: `extra="allow"`, not `frozen`, and without the runtime's wildcard-refusing validators, because this SDK reads fixes where the runtime plans them — a new server-side member on an op must not break parsing here. An unknown `kind` fails the whole verdict parse, deliberately and consistently with the closed `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break for no consumer benefit. +- **Paged listing: `iterate_methods`, `iterate_runs`, and `get_run_detail`.** `iterate_methods(q=…, limit=…)` and `iterate_runs(method_id, created_from=…, created_to=…, limit=…)` are async generators that follow the cursors, and they are the idiom to copy: there is deliberately no `list_all_methods()`, because an all-at-once helper needs a cap and a cap is the silent truncation paging removed. They stop on different signals, and the difference is in the server rather than the client — `iterate_methods` keeps going through an empty page that carries a live cursor, since `q` is a post-read filter over a bounded index slice per request, while `iterate_runs` treats an empty page as the end, since the run date bounds are index key conditions. Both stop when the server hands back the cursor it was sent, checked before yielding so a stuck cursor never double-counts a page. Past a runaway backstop set far beyond any real catalog, `iterate_methods` raises the new `PagingNotTerminatingError` rather than returning what it has. `get_run_detail(run_id)` reads one run through `GET /v1/runs/{id}` — distinct from the lifecycle `get_run_status` (`/status`) and `get_run_result` (`/results`), and the only read carrying `mthds_contents` and `inputs`, which is what the run actually executed and the only record of it. + +- **New product models for the paged surface.** `MethodPage` / `MethodSummary`, `RunPage` / `RunDetail` / `RunErrorReport`, and `MethodFile` with the `parse_method_files` / `serialize_method_files` pair. A `MethodSummary` is deliberately not a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none of them is in the index projection, and putting `mthds` back is what restored the truncation bug. `MethodData` also gains `org_id`, `created_by_user_id`, `description`, `deletion_state`, and a typed `python: list[MethodFile]`; `MethodWriteInput` gains `python: list[MethodFile] | None`. + + On the wire `python` is one string — the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python — and the converter lives here rather than in `mthds`, because the catalog serialization is a Pipelex product concern and there is no second Python consumer to keep in step. `MethodData` applies the parser through a `field_validator(mode="before")`, so `model_validate` keeps working at every call site; `MethodWriteInput` applies the serializer through a `field_serializer`, which makes the platform's three-way write contract fall out of `exclude_none=True`: `None` leaves the key out and preserves the stored Python, `[]` sends the `""` clear sentinel, a non-empty list replaces. Malformed wire text raises inside the validator and surfaces as a `pydantic.ValidationError`, the same way any other malformed response body fails here. + - **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. The doctrine is normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy". ### Changed +- **Breaking: the two product list routes return page envelopes.** `list_methods` returns a `MethodPage` (`items` are `MethodSummary` rows, not `MethodData`) and `list_runs` returns a `RunPage`, each carrying an opaque `next_cursor` that is `None` on the last page. Both take the paging and filter parameters the platform serves — `q` / `limit` / `cursor` on methods, `created_from` / `created_to` / `limit` / `cursor` on runs — kept on **presence** rather than truthiness, so an explicit empty `q` or `cursor` is forwarded for the API to reject rather than dropped silently into an unfiltered query that reads as working. `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, and are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also worth knowing: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed — a `403` there means "flag", not "wrong key". + +- **Breaking: `PipelineRun.method_id` and `pipe_code` are nullable, and `MethodData.python` is a `list[MethodFile]`.** The platform genuinely serves the first two as null and both cases are ordinary: an ad-hoc run from an inline bundle belongs to no stored method, and a run that let the bundle's `main_pipe` decide names no pipe. `PipelineRun` also gains `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None`. `MethodData.python` is converted from its wire string at the boundary, so callers never see that string. + - **Breaking: `delete_method` returns the platform's acceptance instead of `None`.** `DELETE /v1/methods/{id}` answers `202` with `method_id`, `deletion_state`, and `deletion_job_id`, and the erasure continues asynchronously after that; the method was annotated `-> None` with a docstring promising an empty synchronous delete, so a caller reasonably concluded the method was gone when the call returned. It now returns the new `MethodDeletionAccepted` (with `MethodDeletionState`), which means "accepted", never "gone" — completion is the row disappearing from `list_methods`. A misleading contract around a destructive operation is worth a breaking change. - **Breaking: `extra` now rejects `method_id`.** The client names the key itself, so it must also guard it — one argument must not arrive by two paths with different validation. Callers passing `extra={"method_id": …}` must pass the named parameter instead. The guard is deliberately per layer: `method_id` must never become reserved in `mthds`, which has no business rejecting another vendor's arguments. @@ -28,6 +38,10 @@ ### Fixed +- **`list_methods` and `list_runs` crashed against the deployed platform, and the tests hid it.** The platform reshaped both routes into `{items, next_cursor}` envelopes — methods since 2026-08-18, runs since 2026-08-11, both live in production. This SDK still iterated a bare array, and iterating the envelope dict yields its *keys*, so the very first call was `MethodData.model_validate("items")` and every invocation raised `pydantic.ValidationError`. `list_runs` failed identically. Nothing was red because the unit tests mocked the pre-paging shape; they now mock the envelope, and a run row with a null `pipe_code` and `method_id` parses, which is what would have raised next once the envelope was fixed. + +- **Documentation: the parity claim was false and is now an honest list.** `docs/architecture.md` claimed the surface was "surface-complete, with no silent gaps" against `@pipelex/sdk`, which stopped being true as the JS SDK grew; the section now names what has no Python counterpart (`lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure`) and records each as deferred rather than missing. The stale "Out of scope" bullet listing the `/v1/build/*` helpers as deferred wholesale is corrected too — `build_inputs` shipped in 0.5.0. + - **Documentation: `TokensUsageRecord` is attributed to the right brand.** The module and class docstrings in `pipelex_sdk/runs.py`, `docs/run-usage.md`, and `docs/architecture.md` said the record was "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins the wire contract and this SDK mirrors the runtime's own record. Getting the attribution wrong on a public package misplaces the brand boundary the rest of the architecture is organized around. No behaviour changed. - **Documentation: no more citations a reader cannot open.** Several docstrings, comments and doc passages cited internal specs and the conformance suite by bare repo-relative path — paths that resolve to nothing for anyone who clones this public package, so they read as rot rather than as a deliberate boundary. Each site now states the rule it was citing instead: the layered extension policy behind the hosted run extensions and the reserved-`extra` guard, and the locked category vocabulary shared with the conformance corpus. References to the sibling public `pipelex-sdk-js` repo are kept where they explain a port. No behaviour changed. diff --git a/TODOS.md b/TODOS.md index 40fb925..b892550 100644 --- a/TODOS.md +++ b/TODOS.md @@ -94,52 +94,52 @@ Breaking, and the most urgent fix in this plan: `list_methods` and `list_runs` c ### 3.1 `pipelex_sdk/product_models.py` — models -- [ ] Add `MethodFile(BaseModel, extra="allow")` with `name: str`, `content: str`, defined **before** `MethodData`. Docstring: the at-rest catalog form of one source file (`[{name, content}]`), distinct from `MthdsFile` (`client.py`, validate input) and `MthdsFileItem` (`build_models.py`, build closure) — three shapes for three surfaces, name the difference so nobody merges them. -- [ ] Add `parse_method_files(source: str | None) -> list[MethodFile]`: blank source (`None`, `""`, whitespace) and `"[]"` both yield `[]`; a JSON array of `{name, content}` yields those files with blank-content entries dropped; anything else (non-array JSON, a malformed entry, unparseable text) raises `ValueError` with a message naming the expected shape. Implementation: `json.loads` then `TypeAdapter(list[MethodFile])` (built once at module level, TypeAdapter construction is expensive), wrapping `json.JSONDecodeError` / `pydantic.ValidationError` into the `ValueError`. -- [ ] Add `serialize_method_files(files: list[MethodFile]) -> str`: drop blank-content entries; an empty result serializes to `""` (the platform's "no source / clear" sentinel), never `"[]"`; otherwise `json.dumps` of `[{name, content}]` only (no extras), stable key order. -- [ ] `MethodData`: add `org_id: str`, `created_by_user_id: str`, `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, `python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile))`, plus a `@field_validator("python", mode="before")` that applies `parse_method_files` when the incoming value is a `str` or `None` and passes a list through unchanged (so programmatic construction in tests still works). A `ValueError` from the parser surfaces as `pydantic.ValidationError` from `model_validate`, the same way any malformed response body fails here. -- [ ] `MethodWriteInput`: add `python: list[MethodFile] | None = None` with a `@field_serializer("python")` returning `serialize_method_files(value)`. Docstring the three-way contract: `None` → key absent (the write body dumps with `exclude_none=True`) → the stored Python is preserved on `PUT`; `[]` → sent as `""` → clears it; a non-empty list → replaces it. -- [ ] Add `MethodSummary(BaseModel, extra="allow")`: `method_id: str`, `name: str`, `description: str | None = None`, `created_at: str`, `deletion_state: MethodDeletionState | None = None`. Docstring: deliberately not a `MethodData` — no `mthds`, no `python`, no `updated_at` — because none is in the index projection and putting `mthds` back is what restored the truncation bug; a method mid-deletion stays in the list so a UI can render "Deleting…" while `get_method` refuses it with a `409`. -- [ ] Add `MethodPage(BaseModel, extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. Docstring: opaque cursor, pass it straight back; `None` means last page; no total by design. -- [ ] Add `RunErrorReport(BaseModel, extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. -- [ ] `PipelineRun`: `method_id: str | None = None` (an ad-hoc run from an inline bundle belongs to no stored method), `pipe_code: str | None = None` (resolved from the bundle's `main_pipe`); add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. Leave `pipe_statuses` as it is. -- [ ] Add `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None`. Docstring: `mthds_contents` is what the run actually executed and the only record of it; both fields are absent from the list and the polled status read on purpose (size × page size, size × poll rate). -- [ ] Add `RunPage(BaseModel, extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. -- [ ] Update the section comments in the module (`# ── Methods catalog`, `# ── Run records`) so the new models sit under the right banner. +- [x] Add `MethodFile(BaseModel, extra="allow")` with `name: str`, `content: str`, defined **before** `MethodData`. Docstring: the at-rest catalog form of one source file (`[{name, content}]`), distinct from `MthdsFile` (`client.py`, validate input) and `MthdsFileItem` (`build_models.py`, build closure) — three shapes for three surfaces, name the difference so nobody merges them. +- [x] Add `parse_method_files(source: str | None) -> list[MethodFile]`: blank source (`None`, `""`, whitespace) and `"[]"` both yield `[]`; a JSON array of `{name, content}` yields those files with blank-content entries dropped; anything else (non-array JSON, a malformed entry, unparseable text) raises `ValueError` with a message naming the expected shape. Implementation: `json.loads` then `TypeAdapter(list[MethodFile])` (built once at module level, TypeAdapter construction is expensive), wrapping `json.JSONDecodeError` / `pydantic.ValidationError` into the `ValueError`. +- [x] Add `serialize_method_files(files: list[MethodFile]) -> str`: drop blank-content entries; an empty result serializes to `""` (the platform's "no source / clear" sentinel), never `"[]"`; otherwise `json.dumps` of `[{name, content}]` only (no extras), stable key order. +- [x] `MethodData`: add `org_id: str`, `created_by_user_id: str`, `description: str | None = None`, `deletion_state: MethodDeletionState | None = None`, `python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile))`, plus a `@field_validator("python", mode="before")` that applies `parse_method_files` when the incoming value is a `str` or `None` and passes a list through unchanged (so programmatic construction in tests still works). A `ValueError` from the parser surfaces as `pydantic.ValidationError` from `model_validate`, the same way any malformed response body fails here. +- [x] `MethodWriteInput`: add `python: list[MethodFile] | None = None` with a `@field_serializer("python")` returning `serialize_method_files(value)`. Docstring the three-way contract: `None` → key absent (the write body dumps with `exclude_none=True`) → the stored Python is preserved on `PUT`; `[]` → sent as `""` → clears it; a non-empty list → replaces it. +- [x] Add `MethodSummary(BaseModel, extra="allow")`: `method_id: str`, `name: str`, `description: str | None = None`, `created_at: str`, `deletion_state: MethodDeletionState | None = None`. Docstring: deliberately not a `MethodData` — no `mthds`, no `python`, no `updated_at` — because none is in the index projection and putting `mthds` back is what restored the truncation bug; a method mid-deletion stays in the list so a UI can render "Deleting…" while `get_method` refuses it with a `409`. +- [x] Add `MethodPage(BaseModel, extra="allow")`: `items: list[MethodSummary]`, `next_cursor: str | None = None`. Docstring: opaque cursor, pass it straight back; `None` means last page; no total by design. +- [x] Add `RunErrorReport(BaseModel, extra="allow")`: `message: str | None = None`, `error_type: str | None = None` — the two fields a consumer may rely on out of the runner's verbose report. +- [x] `PipelineRun`: `method_id: str | None = None` (an ad-hoc run from an inline bundle belongs to no stored method), `pipe_code: str | None = None` (resolved from the bundle's `main_pipe`); add `org_id: str | None = None`, `created_by_user_id: str | None = None`, `error: RunErrorReport | None = None`. Leave `pipe_statuses` as it is. +- [x] Add `RunDetail(PipelineRun)`: `mthds_contents: list[str] | None = None`, `inputs: dict[str, Any] | None = None`. Docstring: `mthds_contents` is what the run actually executed and the only record of it; both fields are absent from the list and the polled status read on purpose (size × page size, size × poll rate). +- [x] Add `RunPage(BaseModel, extra="allow")`: `items: list[PipelineRun]`, `next_cursor: str | None = None`. +- [x] Update the section comments in the module (`# ── Methods catalog`, `# ── Run records`) so the new models sit under the right banner. ### 3.2 `pipelex_sdk/errors.py` — the runaway-paging error -- [ ] Add one error for `iterate_methods` refusing to keep paging past the ceiling (a name like `PagingNotTerminatingError`), extending whatever base the module's other product errors extend — check the existing hierarchy there first. Message mirrors the JS one: the iterator did not terminate after the ceiling; this is a server-side fault, not a coverage limit. +- [x] Add one error for `iterate_methods` refusing to keep paging past the ceiling (a name like `PagingNotTerminatingError`), extending whatever base the module's other product errors extend — check the existing hierarchy there first. Message mirrors the JS one: the iterator did not terminate after the ceiling; this is a server-side fault, not a coverage limit. ### 3.3 `pipelex_sdk/client.py` — list, iterate, detail -- [ ] Add a module helper `_product_query(params: dict[str, str | int | None]) -> str` that keeps entries on **presence** (`is not None`, never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to drop silently) and encodes with `urllib.parse.urlencode`, returning `""` or `?…`. The existing `test_list_runs_encodes_query_value` assertion (`method_id=m%2F1`) must stay green, so keep `/` percent-encoded. -- [ ] Add `_MAX_LIST_PAGES: int = 10_000` beside the other module constants (the JS `MAX_PAGES`), with the comment that it is a runaway backstop set far beyond any real catalog, not a coverage cap. -- [ ] `list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage` — `GET /v1/methods` with the query built by the helper; parse `MethodPage`. Docstring: `q` is a server-side case-insensitive substring match over name and description across the whole catalog; `limit` defaults to and is capped by the API; ordering is by creation, newest first. -- [ ] `iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]` as an `async def` generator: request a page; **before yielding**, stop if `cursor is not None and page.next_cursor == cursor` (the server did not advance; yielding first would double-count); yield every item; stop when `page.next_cursor is None`; otherwise count the page and **raise** the new error once the count reaches `_MAX_LIST_PAGES`; continue **through empty pages** with a live cursor, because `q` is a post-read filter over a bounded index slice per request and `{items: [], next_cursor: "…"}` means "keep going". Docstring says why there is no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. -- [ ] `list_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage` — `GET /v1/runs?method_id=…` plus the presence-kept query; parse `RunPage`. Docstring: `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, key conditions rather than filters; a bare date or naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also document the gate the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()`, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — a `403` here means "flag", not "wrong key". -- [ ] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing. -- [ ] `get_run_detail(self, run_id: str) -> RunDetail` — `GET /v1/runs/{id}` via `_request_product` with the id path-encoded like the other id routes (`f"{_RUNS}/{quote(run_id, safe='')}"`). Distinct from `get_run_status` (`/status`) and `get_run_result` (`/results`). -- [ ] Update the `PipelexAPIClient` class docstring's product-surface bullet and the import block (`MethodPage`, `MethodSummary`, `RunDetail`, `RunPage`, `AsyncIterator` from `collections.abc` under `TYPE_CHECKING` if only used in annotations — it is used at runtime as a return annotation with `from __future__ import annotations`, so `TYPE_CHECKING` is fine). +- [x] Add a module helper `_product_query(params: dict[str, str | int | None]) -> str` that keeps entries on **presence** (`is not None`, never truthiness — an explicit empty `q` or cursor is bad input the API should reject, not something to drop silently) and encodes with `urllib.parse.urlencode`, returning `""` or `?…`. The existing `test_list_runs_encodes_query_value` assertion (`method_id=m%2F1`) must stay green, so keep `/` percent-encoded. +- [x] Add `_MAX_LIST_PAGES: int = 10_000` beside the other module constants (the JS `MAX_PAGES`), with the comment that it is a runaway backstop set far beyond any real catalog, not a coverage cap. +- [x] `list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage` — `GET /v1/methods` with the query built by the helper; parse `MethodPage`. Docstring: `q` is a server-side case-insensitive substring match over name and description across the whole catalog; `limit` defaults to and is capped by the API; ordering is by creation, newest first. +- [x] `iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]` as an `async def` generator: request a page; **before yielding**, stop if `cursor is not None and page.next_cursor == cursor` (the server did not advance; yielding first would double-count); yield every item; stop when `page.next_cursor is None`; otherwise count the page and **raise** the new error once the count reaches `_MAX_LIST_PAGES`; continue **through empty pages** with a live cursor, because `q` is a post-read filter over a bounded index slice per request and `{items: [], next_cursor: "…"}` means "keep going". Docstring says why there is no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. +- [x] `list_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage` — `GET /v1/runs?method_id=…` plus the presence-kept query; parse `RunPage`. Docstring: `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, key conditions rather than filters; a bare date or naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also document the gate the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()`, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — a `403` here means "flag", not "wrong key". +- [x] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing. +- [x] `get_run_detail(self, run_id: str) -> RunDetail` — `GET /v1/runs/{id}` via `_request_product` with the id path-encoded like the other id routes (`f"{_RUNS}/{quote(run_id, safe='')}"`). Distinct from `get_run_status` (`/status`) and `get_run_result` (`/results`). +- [x] Update the `PipelexAPIClient` class docstring's product-surface bullet and the import block (`MethodPage`, `MethodSummary`, `RunDetail`, `RunPage`, `AsyncIterator` from `collections.abc` under `TYPE_CHECKING` if only used in annotations — it is used at runtime as a return annotation with `from __future__ import annotations`, so `TYPE_CHECKING` is fine). ### 3.4 Tests -- [ ] `tests/unit/test_client_product.py`: replace the bare-array fixtures at `test_list_methods` and `test_list_runs_encodes_query_value` with envelopes and assert `MethodPage` / `RunPage` come back with `next_cursor`; add query-encoding cases for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped and that an absent parameter leaves no key in the query; a run row with `null` `pipe_code` and `method_id` parses; `get_run_detail` hits `/v1/runs/{id}` with encoding and returns `mthds_contents` and `inputs`; `MethodData` parses the new fields with `python` read from the wire string into `MethodFile` entries and `""` reading as `[]`; `create_method` / `update_method` send `python` three ways (`None` absent, `[]` as `""`, a list as the JSON text). -- [ ] New `tests/unit/test_method_files.py` (one `TestClass`): `parse_method_files` on blank / `"[]"` / a valid array / an array with a blank-content entry / a non-array / a malformed entry / unparseable text; `serialize_method_files` on empty / blank-only / mixed; a round-trip is stable. -- [ ] New `tests/unit/test_client_paging.py` (one `TestClass`): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding the page; `iterate_runs` stops on an empty page; `iterate_methods` raises the new error past the ceiling (patch `_MAX_LIST_PAGES` down via `mocker.patch` rather than looping ten thousand times); the cursor sent on page N+1 is the `next_cursor` received on page N. Use `mocker.AsyncMock(side_effect=[...])` on `_send` to script the page sequence. -- [ ] The `_response` / `_Sent` / `_mock_send` helpers live as private members of `tests/unit/test_client_product.py`. Rather than importing private test helpers across modules, promote a response builder and a `_send` spy to fixtures in a new `tests/unit/conftest.py` for the new modules to use (house rule: fixtures go in `conftest.py`). Migrating `test_client_product.py` onto those fixtures is optional and not part of this change. +- [x] `tests/unit/test_client_product.py`: replace the bare-array fixtures at `test_list_methods` and `test_list_runs_encodes_query_value` with envelopes and assert `MethodPage` / `RunPage` come back with `next_cursor`; add query-encoding cases for `q` / `limit` / `cursor` and for `created_from` / `created_to`, including that an explicit empty string is forwarded rather than dropped and that an absent parameter leaves no key in the query; a run row with `null` `pipe_code` and `method_id` parses; `get_run_detail` hits `/v1/runs/{id}` with encoding and returns `mthds_contents` and `inputs`; `MethodData` parses the new fields with `python` read from the wire string into `MethodFile` entries and `""` reading as `[]`; `create_method` / `update_method` send `python` three ways (`None` absent, `[]` as `""`, a list as the JSON text). +- [x] New `tests/unit/test_method_files.py` (one `TestClass`): `parse_method_files` on blank / `"[]"` / a valid array / an array with a blank-content entry / a non-array / a malformed entry / unparseable text; `serialize_method_files` on empty / blank-only / mixed; a round-trip is stable. +- [x] New `tests/unit/test_client_paging.py` (one `TestClass`): `iterate_methods` continues through an empty page with a live cursor and stops on `None`; both iterators stop on an unchanged cursor without re-yielding the page; `iterate_runs` stops on an empty page; `iterate_methods` raises the new error past the ceiling (patch `_MAX_LIST_PAGES` down via `mocker.patch` rather than looping ten thousand times); the cursor sent on page N+1 is the `next_cursor` received on page N. Use `mocker.AsyncMock(side_effect=[...])` on `_send` to script the page sequence. +- [x] The `_response` / `_Sent` / `_mock_send` helpers live as private members of `tests/unit/test_client_product.py`. Rather than importing private test helpers across modules, promote a response builder and a `_send` spy to fixtures in a new `tests/unit/conftest.py` for the new modules to use (house rule: fixtures go in `conftest.py`). Migrating `test_client_product.py` onto those fixtures is optional and not part of this change. ### 3.5 Docs and changelog -- [ ] `docs/architecture.md` → "Pipelex product surface": rewrite the **Methods catalog** bullet for `MethodPage` / `MethodSummary` / `iterate_methods` and the `python` three-way write contract with `MethodFile`; rewrite the **Run records** bullet for `RunPage` / `iterate_runs` / `get_run_detail`, the nullable `PipelineRun` fields and `error`, the instant-only date bounds, and the `ff_api_keys` `403`. State the two iterator stop rules and why they differ. -- [ ] `docs/architecture.md` → "Parity with `@pipelex/sdk`": it must stop claiming "surface-complete, with no silent gaps". Rewrite it to list the conscious exclusions honestly: `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — unchanged by the cited releases and deferred. While there, fix the stale "Out of scope for v0.1" bullet that still lists `/v1/build/*` helpers as deferred even though `build_inputs` shipped in 0.5.0. -- [ ] `README.md`: if the quickstart gains a listing example, use `iterate_methods` rather than a page loop, so the idiom people copy is the one that cannot truncate. -- [ ] `CHANGELOG.md` `[Unreleased]`: **Changed (breaking)** — `list_methods` returns `MethodPage` (items are `MethodSummary`, not `MethodData`), `list_runs` returns `RunPage`, `PipelineRun.method_id` / `pipe_code` are nullable, `MethodData.python` is `list[MethodFile]`; **Added** — `iterate_methods`, `iterate_runs`, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the new `MethodData` fields, `MethodWriteInput.python`, the paging error; **Fixed** — name the crash plainly (iterating the envelope dict yielded its keys, so the first call was `MethodData.model_validate("items")`), and that the unit tests mocked the pre-paging shape. +- [x] `docs/architecture.md` → "Pipelex product surface": rewrite the **Methods catalog** bullet for `MethodPage` / `MethodSummary` / `iterate_methods` and the `python` three-way write contract with `MethodFile`; rewrite the **Run records** bullet for `RunPage` / `iterate_runs` / `get_run_detail`, the nullable `PipelineRun` fields and `error`, the instant-only date bounds, and the `ff_api_keys` `403`. State the two iterator stop rules and why they differ. +- [x] `docs/architecture.md` → "Parity with `@pipelex/sdk`": it must stop claiming "surface-complete, with no silent gaps". Rewrite it to list the conscious exclusions honestly: `lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure` — unchanged by the cited releases and deferred. While there, fix the stale "Out of scope for v0.1" bullet that still lists `/v1/build/*` helpers as deferred even though `build_inputs` shipped in 0.5.0. +- [x] `README.md`: if the quickstart gains a listing example, use `iterate_methods` rather than a page loop, so the idiom people copy is the one that cannot truncate. +- [x] `CHANGELOG.md` `[Unreleased]`: **Changed (breaking)** — `list_methods` returns `MethodPage` (items are `MethodSummary`, not `MethodData`), `list_runs` returns `RunPage`, `PipelineRun.method_id` / `pipe_code` are nullable, `MethodData.python` is `list[MethodFile]`; **Added** — `iterate_methods`, `iterate_runs`, `get_run_detail`, `MethodSummary` / `MethodPage` / `RunPage` / `RunDetail` / `RunErrorReport`, `MethodFile` with `parse_method_files` / `serialize_method_files`, the new `MethodData` fields, `MethodWriteInput.python`, the paging error; **Fixed** — name the crash plainly (iterating the envelope dict yielded its keys, so the first call was `MethodData.model_validate("items")`), and that the unit tests mocked the pre-paging shape. ### 3.6 Gate and commit -- [ ] `make agent-check` and `make agent-test` pass. -- [ ] Commit (suggested message: "Follow the platform's paged method and run lists and stop requiring nullable run fields"). +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Follow the platform's paged method and run lists and stop requiring nullable run fields"). ### Checkpoint 2 diff --git a/docs/architecture.md b/docs/architecture.md index d514c8d..0f5f8c5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,6 +1,6 @@ # pipelex-sdk architecture -The design reference for the package. It covers the full `0.1.0` surface and records the parity audit against the TypeScript `@pipelex/sdk` reference (see "Parity with `@pipelex/sdk`" at the end). +The design reference for the package. It covers the shipped surface and records where it stands against the TypeScript `@pipelex/sdk` reference, gaps included (see "Parity with `@pipelex/sdk`" at the end). ## What this is @@ -151,14 +151,28 @@ The hosted catalog/account routes the webapp drives (`pipelex_sdk/product_models The wire models are snake_case Pydantic v2. Response models are extension-open (`extra="allow"`) so a newly-added server field is preserved, not rejected; input models name exactly what each route accepts. `PipelineRun.status` reuses the run-lifecycle `RunStatus`; `OrgRole`, `PipeStatus`, and the onboarding fields are `StrEnum`s. - **User profile** — `get_me()` → `UserProfile` (`GET /v1/me`). -- **Methods catalog** — `list_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. **`delete_method` is asynchronous and its return type says so:** the route answers `202` with a `MethodDeletionAccepted` (`method_id`, `deletion_state`, `deletion_job_id`) the moment the platform has claimed the method and terminated its in-flight workflows — the rest of the cascade (runs, events, S3 objects) is enqueued. A returned value means "accepted", never "gone"; completion is the row disappearing from `list_methods`, not any field of that body. It used to be annotated `-> None` with a docstring promising an empty synchronous delete, which is a misleading contract around a destructive operation. The claim is a conditional write, so a double-clicked delete is a `409 conflict` rather than a second cascade over the same runs. +- **Methods catalog** — `list_methods()` / `iterate_methods()` / `get_method(id)` / `create_method(MethodWriteInput)` / `update_method(id, MethodWriteInput)` (a rename is a changed `name`) / `delete_method(id)`. The id is path-encoded; an absent `input_data` is dropped from the write body. + + **The index is paged.** `list_methods(q=…, limit=…, cursor=…)` returns one `MethodPage` — `items: list[MethodSummary]` plus an opaque `next_cursor` you pass straight back, `None` on the last page, and no total by design. A `MethodSummary` is deliberately **not** a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none of them is in the index projection, and putting `mthds` back is exactly what restored the truncation bug paging removed. A method mid-deletion still appears in the list — so a UI can render "Deleting…" — while `get_method` refuses it with a `409`. Query parameters are kept on **presence**, never truthiness: an explicit empty `q` or `cursor` is bad input the API should reject, not something to drop silently into an unfiltered query that reads as working. + + **`iterate_methods(q=…, limit=…)` is the idiom for the whole catalog**, and there is deliberately no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. It **keeps going through an empty page that carries a live cursor**, because `q` is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` means "keep going". It stops on a `None` cursor, and also when the server hands back the cursor it was just sent — checked *before* yielding, so a stuck cursor never double-counts a page. Past a runaway backstop set far beyond any real catalog it **raises** `PagingNotTerminatingError` rather than returning what it has, because a silently truncated list is the bug being fixed. + + **`MethodData.python` is a typed `list[MethodFile]`, converted at the boundary.** On the wire it is one string: the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python. `parse_method_files` / `serialize_method_files` (in `product_models.py`) carry the rules — a blank source or `"[]"` yields `[]`, blank-content entries are dropped in both directions, and an empty list serializes to `""` rather than `"[]"` because `""` is the platform's clear sentinel. `MethodData` applies the parser through a `field_validator(mode="before")` and `MethodWriteInput` the serializer through a `field_serializer`, which is what makes the platform's three-way write contract fall out of `exclude_none=True`: **`None` → key absent → the stored Python is preserved; `[]` → `""` → cleared; a non-empty list → replaced.** `MethodFile` is distinct from `MthdsFile` (the validate input) and `MthdsFileItem` (the build closure) — three shapes for three surfaces. + + **`delete_method` is asynchronous and its return type says so:** the route answers `202` with a `MethodDeletionAccepted` (`method_id`, `deletion_state`, `deletion_job_id`) the moment the platform has claimed the method and terminated its in-flight workflows — the rest of the cascade (runs, events, S3 objects) is enqueued. A returned value means "accepted", never "gone"; completion is the row disappearing from `list_methods`, not any field of that body. It used to be annotated `-> None` with a docstring promising an empty synchronous delete, which is a misleading contract around a destructive operation. The claim is a conditional write, so a double-clicked delete is a `409 conflict` rather than a second cascade over the same runs. - **Organizations** — `list_memberships()` → `MembershipsResponse` (memberships + active-org feature flags); `create_organization(name)` / `rename_organization(org_id, name)` → `Membership`. Organization *switch* is out of scope (a WorkOS session op, not a `/v1` route). - **Billing** — `get_subscription()`, `list_plans()`, `list_invoices()`, `create_checkout(plan)`. `change_plan(plan)` and `get_billing_portal()` surface a **409 `conflict`** (`ApiResponseError.code`) when there is no subscription yet — start one via `create_checkout` first. - **Pipelex API keys** — `list_pipelex_api_keys()`; `create_pipelex_api_key(label)` and `rotate_pipelex_api_key(id)` return the plaintext `api_key` **once**; `revoke_pipelex_api_key(id)`. Creation surfaces a **409 `pipelex_api_key_limit_reached`** when the per-account limit is hit. Rotation sends no body. - **Gateway (LLM inference) key** — `create_gateway_api_key(promo_code)` **always sends a JSON body** (even with `promo_code=None` → `{"promo_code": null}`); the server 422s an empty body. `get_gateway_api_key()` → status (`gateway_api_key` is `None` until provisioned). - **Onboarding** — `submit_onboarding(OnboardingSubmission)` (`POST /v1/onboarding/submit`, empty 2xx body); absent optional fields are dropped. - **Storage** — `resolve_storage_url(uri)` → presigned URL; `upload(UploadInput)` → the stored file handle. The higher-level `upload_file` / `prepare_inputs` preparation surface built on top of `upload` is now available — see [input-preparation.md](./input-preparation.md). -- **Run records** — `list_runs(method_id)` → `list[PipelineRun]` (the catalog-style list, distinct from the lifecycle status/result routes); `update_run(run_id, UpdateRunInput)` (admin/manual status patch, empty 2xx body). +- **Run records** — `list_runs(method_id, …)` / `iterate_runs(method_id, …)` / `get_run_detail(run_id)` (the catalog-style reads, distinct from the lifecycle status/result routes); `update_run(run_id, UpdateRunInput)` (admin/manual status patch, empty 2xx body). + + **This list is paged too.** `list_runs(method_id, created_from=…, created_to=…, limit=…, cursor=…)` returns a `RunPage` with the same opaque-cursor contract as `MethodPage`. `created_from` / `created_to` are **instants** — ISO-8601 with a UTC offset — and inclusive; they are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Worth knowing and not obvious from the route: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — so a `403` here means "flag", not "wrong key". + + **The two iterators stop on different signals, and the difference is in the server.** `iterate_methods` continues through an empty page with a live cursor; `iterate_runs` treats an **empty page as the end**, because the run date bounds are index key conditions and so a run page is never empty-with-a-cursor. That stop rule also removes the need for a page ceiling on the run side — a server minting fresh cursors while returning nothing is already caught by it. + + **`PipelineRun` fields the platform genuinely serves as null are typed nullable.** `method_id` is `None` for an ad-hoc run from an inline bundle, which belongs to no stored method; `pipe_code` is `None` for a run that let the bundle's `main_pipe` decide. `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None` (`message`, `error_type` — the two fields a consumer may rely on out of the runner's verbose report) join them. `RunDetail`, returned only by `get_run_detail`, adds `mthds_contents` and `inputs`: what the run actually executed, and the only record of it, since a method edited since the run no longer describes what happened. Both are left out of the list and the polled status on purpose — their cost scales with page size and poll rate respectively. ## Health probe @@ -166,7 +180,7 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( ## Out of scope for v0.1 -- `/v1/build/*` helpers (the TS clients carry them; recorded as a conscious deferral). +- The remaining `/v1/build/*` authoring helpers — `build_output`, `build_runner`, `concept`, `pipe_spec`. `build_inputs` shipped in 0.5.0 and is no longer deferred. - Organization *switch* (a WorkOS session operation, not a `/v1` route). - A `~/.pipelex/config` file reader (env-only for now, matching the JS SDK). - A synchronous client facade. @@ -177,13 +191,19 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( ## Parity with `@pipelex/sdk` -This SDK is a faithful port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`). The Checkpoint-5 parity audit walked the JS `src/client.ts`, `src/index.ts` (the public barrel), and `docs/architecture.md`, plus a field-by-field sweep of `runs.ts` / `product-models.ts` / `models.ts` against their Python counterparts. Result: **surface-complete, with no silent gaps** — every JS method, model, and error has a Python equivalent or a consciously-recorded exclusion. +This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tracks it closely, but it is **not surface-complete, and this section is where the gaps are named.** The Checkpoint-5 parity audit walked the JS `src/client.ts`, `src/index.ts` (the public barrel), and `docs/architecture.md`, plus a field-by-field sweep of `runs.ts` / `product-models.ts` / `models.ts`; it concluded surface-completeness, and that conclusion went stale as the JS SDK grew. The honest list of what has no Python counterpart today: + +- **Tooling routes** — `lint`, `format`, `resolve`, `codegen`. +- **Authoring helpers** — `build_output`, `build_runner`, `concept`, `pipe_spec` (`build_inputs` shipped in 0.5.0 and is **not** a gap). +- **Offline helpers** — `run_codegen_check` (the codegen drift check) and `get_method_closure` (client-side sugar that parses the polymorphic `mthds` source into a run-ready closure). + +None of them is moved by the releases this SDK last tracked, and each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. **Methods** — full coverage: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records), and `health`. **Models** — full field-for-field match across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. -**Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. +**Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`, `PagingNotTerminatingError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. **Resolved parity flags (Checkpoint 5):** @@ -191,6 +211,6 @@ This SDK is a faithful port of the TypeScript `@pipelex/sdk` (`PipelexApiClient` - **`health` error regime** (decision #5) — **kept** the plainer `PipelineRequestError` regime; already matches JS, needs no `code` taxonomy. - **`validate` error regime** (Phase-3 flag) — **deferred**; keeps the inherited `httpx.HTTPStatusError` regime for consistency with the other Python protocol routes (see "`validate` override" above). -**Conscious exclusions:** the `/v1/build/*` authoring helpers (decision #8 — a future follow-up if a consumer needs them) and the organization *switch* (a WorkOS session op, not a `/v1` route). +**Conscious exclusions:** the surfaces listed at the top of this section, and the organization *switch* (a WorkOS session op, not a `/v1` route). **Intentional divergences from the JS SDK** (Python house style / clean inheritance): no barrel (`__init__.py` stays empty; import via full paths); inheritance on `MthdsAPIClient` rather than the JS composition-of-types; async-only; `__version__` derived from installed metadata rather than a hand-synced constant. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index f744005..7993f03 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -23,7 +23,7 @@ import re from time import monotonic from typing import TYPE_CHECKING, Any, NamedTuple, NoReturn, cast -from urllib.parse import quote, urlparse +from urllib.parse import quote, urlencode, urlparse import httpx from mthds.protocol.exceptions import PipelineRequestError @@ -37,6 +37,7 @@ ApiResponseError, ApiUnreachableError, MissingMainStuffError, + PagingNotTerminatingError, PipelineExecuteTimeoutError, RunFailedError, RunLifecycleUnavailableError, @@ -56,11 +57,15 @@ MembershipsResponse, MethodData, MethodDeletionAccepted, + MethodPage, + MethodSummary, PipelexApiKeyCreated, PipelexApiKeyList, PipelineRun, PlanView, ResolvedStorageUrl, + RunDetail, + RunPage, SubscriptionResponse, UploadedFile, UserProfile, @@ -80,6 +85,8 @@ from pipelex_sdk.validation_models import PipelexValidationResultAdapter, ValidationErrorItem if TYPE_CHECKING: + from collections.abc import AsyncIterator + from mthds.protocol.models import RunResultStart from mthds.protocol.pipe_output import VariableMultiplicity from mthds.protocol.pipeline_inputs import PipelineInputs @@ -106,6 +113,11 @@ DEFAULT_API_BASE_URL = "https://api.pipelex.com" _POLL_REQUEST_TIMEOUT_SECONDS = 30.0 # single status/result/product GETs; the hosted gateway caps responses at ~30s. + +# A runaway backstop for the paged-list iterators, set far beyond any real catalog — never a +# coverage cap. Reaching it means the server kept minting cursors, which is a fault to raise on, +# not a limit to truncate at. +_MAX_LIST_PAGES: int = 10_000 _DEFAULT_DEGRADED_RETRY_SECONDS = 5 # matches the platform's `_DEGRADE_RETRY_AFTER_SECONDS`. # The hosted gateway caps synchronous requests at ~30s. A blocking-`execute` failure at/after @@ -163,7 +175,9 @@ class PipelexAPIClient(MthdsAPIClient): durable polling extension (added in Phase 2). - **product** (`/v1/me`, `/v1/methods`, `/v1/billing/*`, …) — the hosted product surface (added in Phase 3), reached through `_request_product` so callers branch - on the structured `ApiResponseError.code`, not the HTTP status. + on the structured `ApiResponseError.code`, not the HTTP status. The two list routes + are paged: `list_methods` / `list_runs` answer one `{items, next_cursor}` page, and + `iterate_methods` / `iterate_runs` follow the cursors for the whole catalog. Construction is Pipelex-only — it never reads the `mthds` resolver (`MTHDS_API_KEY` / `MTHDS_BASE_URL`, `~/.mthds/config`), whose values are a credential pair for whatever @@ -783,10 +797,54 @@ async def get_me(self) -> UserProfile: """The authenticated user's profile — `GET /v1/me`.""" return UserProfile.model_validate(await self._request_product("GET", "me")) - async def list_methods(self) -> list[MethodData]: - """List the caller's saved methods — `GET /v1/methods`.""" - result = await self._request_product("GET", "methods") - return [MethodData.model_validate(item) for item in result] + async def list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage: + """List one page of the caller's saved methods — `GET /v1/methods`. + + Args: + q: Server-side case-insensitive substring match over name and description, + applied across the whole catalog rather than within the returned page. + limit: Page size. The API supplies the default and caps the maximum. + cursor: The `next_cursor` of the previous page, passed back opaquely. + + Returns: + A `MethodPage` of `MethodSummary` rows, ordered by creation, newest first. + `next_cursor` is `None` on the last page. For the whole catalog, prefer + `iterate_methods`, which follows the cursors and cannot truncate. + """ + query = _product_query({"q": q, "limit": limit, "cursor": cursor}) + return MethodPage.model_validate(await self._request_product("GET", f"methods{query}")) + + async def iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]: + """Yield every saved method, following the cursors — `GET /v1/methods`. + + There is deliberately no `list_all_methods()`: an all-at-once helper needs a cap, and + a cap is the silent truncation paging was introduced to remove. + + The loop keeps going **through an empty page that carries a live cursor**, because `q` + is a post-read filter over a bounded index slice per request, so `{items: [], next_cursor: "…"}` + means "keep going", not "done". It stops when `next_cursor` is `None`, and also when the + server hands back the cursor it was just sent — checked *before* yielding, so a stuck + cursor never double-counts a page. + + Raises: + PagingNotTerminatingError: If the server never stops handing out cursors. + """ + cursor: str | None = None + pages_seen = 0 + while True: + page = await self.list_methods(q=q, limit=limit, cursor=cursor) + if cursor is not None and page.next_cursor == cursor: + # The server did not advance. Stop before yielding, or this page is counted twice. + return + for method_summary in page.items: + yield method_summary + if page.next_cursor is None: + return + pages_seen += 1 + if pages_seen >= _MAX_LIST_PAGES: + msg = f"Method paging did not terminate after {_MAX_LIST_PAGES} pages; this is a server-side fault, not a coverage limit." + raise PagingNotTerminatingError(msg, _MAX_LIST_PAGES) + cursor = page.next_cursor async def get_method(self, method_id: str) -> MethodData: """Fetch one method by id — `GET /v1/methods/{id}`.""" @@ -962,10 +1020,76 @@ async def prepare_inputs( """ return await _prepare_inputs_impl(self, files=files, pipe_ref=pipe_ref, inputs=inputs) - async def list_runs(self, method_id: str) -> list[PipelineRun]: - """List a method's runs — `GET /v1/runs?method_id={methodId}`.""" - result = await self._request_product("GET", f"{_RUNS}?method_id={quote(method_id, safe='')}") - return [PipelineRun.model_validate(item) for item in result] + async def list_runs( + self, + method_id: str, + *, + created_from: str | None = None, + created_to: str | None = None, + limit: int | None = None, + cursor: str | None = None, + ) -> RunPage: + """List one page of a method's runs — `GET /v1/runs?method_id={methodId}`. + + Args: + method_id: The method whose runs to list. + created_from: Inclusive lower bound on creation, an **instant**: ISO-8601 with a + UTC offset. These are index key conditions rather than filters, so a bare date + or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. + created_to: Inclusive upper bound, same instant-only rule. + limit: Page size. The API supplies the default and caps the maximum. + cursor: The `next_cursor` of the previous page, passed back opaquely. + + Returns: + A `RunPage` of `PipelineRun` rows. For the whole history, prefer `iterate_runs`. + + Raises: + ApiResponseError: On any non-2xx. Note that every `/v1/runs*` product route sits + behind the platform's surface-access gate, which for API-key auth demands the + `ff_api_keys` feature flag and fails closed — so a `403` here means "flag", not + "wrong key". + """ + query = _product_query({"method_id": method_id, "created_from": created_from, "created_to": created_to, "limit": limit, "cursor": cursor}) + return RunPage.model_validate(await self._request_product("GET", f"{_RUNS}{query}")) + + async def iterate_runs( + self, + method_id: str, + *, + created_from: str | None = None, + created_to: str | None = None, + limit: int | None = None, + ) -> AsyncIterator[PipelineRun]: + """Yield every run of a method, following the cursors — `GET /v1/runs`. + + The same loop as `iterate_methods` with one deliberate difference: an **empty page ends + it**. The date bounds are index key conditions rather than a post-read filter, so a run + page is never empty-with-a-cursor. The difference is in the server, not in the client. + That stop rule also removes the need for a page ceiling here — a server minting fresh + cursors while returning nothing is caught by it. + """ + cursor: str | None = None + while True: + page = await self.list_runs(method_id, created_from=created_from, created_to=created_to, limit=limit, cursor=cursor) + if cursor is not None and page.next_cursor == cursor: + # The server did not advance. Stop before yielding, or this page is counted twice. + return + if not page.items: + return + for pipeline_run in page.items: + yield pipeline_run + if page.next_cursor is None: + return + cursor = page.next_cursor + + async def get_run_detail(self, run_id: str) -> RunDetail: + """Fetch one run record with what it executed — `GET /v1/runs/{id}`. + + Distinct from the two lifecycle reads: `get_run_status` polls `/status`, `get_run_result` + fetches `/results`. This is the catalog-style record, and the only read that carries + `mthds_contents` and `inputs`. + """ + return RunDetail.model_validate(await self._request_product("GET", f"{_RUNS}/{quote(run_id, safe='')}")) async def update_run(self, run_id: str, update_input: UpdateRunInput) -> None: """Patch a run's status (admin/manual) — `PUT /v1/runs/{id}` (empty body).""" @@ -1039,6 +1163,19 @@ def _timeout_message(run_id: str, timeout_seconds: float) -> str: return f"Run {run_id} did not reach a terminal state within {timeout_seconds}s; it is still executing server-side and can be resumed by id." +def _product_query(params: dict[str, str | int | None]) -> str: + """Build the query string of a product list route, keeping entries on **presence**. + + Presence (`is not None`), never truthiness: an explicit empty `q` or cursor is bad input + the API should reject, not something to silently drop into an unfiltered query that reads + as working. Returns `""` for no parameters, otherwise a leading `?`. + """ + kept = {key: value for key, value in params.items() if value is not None} + if not kept: + return "" + return "?" + urlencode(kept) + + def _is_gateway_timeout(exc: httpx.HTTPStatusError | httpx.TimeoutException, elapsed_seconds: float) -> bool: """Whether a failed blocking `execute` is the hosted gateway's ~30s synchronous cut-off. diff --git a/pipelex_sdk/errors.py b/pipelex_sdk/errors.py index 7a8dc2b..375f31a 100644 --- a/pipelex_sdk/errors.py +++ b/pipelex_sdk/errors.py @@ -12,6 +12,8 @@ a consumer branches on (decoupled from the HTTP status). - `PipelineExecuteTimeoutError` — a blocking `execute()` killed by the hosted gateway's ~30s synchronous-request ceiling; points the caller at the durable start+poll path. +- `PagingNotTerminatingError` — a paged-list iterator hit its runaway backstop, meaning + the server never stopped handing out cursors. The run-lifecycle errors (`RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`) are owned here (ported from `mthds-python` in @@ -163,6 +165,20 @@ def __init__(self, message: str, api_url: str) -> None: self.api_url = api_url +class PagingNotTerminatingError(PipelineRequestError): + """Raised when a paged-list iterator refuses to keep following cursors. + + The ceiling sits far beyond any real catalog, so reaching it is a server-side fault — + an endpoint minting a fresh cursor forever — not a coverage limit the caller can raise. + Raising beats returning, because a silently truncated list is exactly the bug paging + was introduced to remove. + """ + + def __init__(self, message: str, page_limit: int) -> None: + super().__init__(message) + self.page_limit = page_limit + + class InputPreparationError(PipelineRequestError): """Base class for every failure raised by input preparation (`upload_file` / `prepare_inputs`). diff --git a/pipelex_sdk/product_models.py b/pipelex_sdk/product_models.py index 8f839e1..287a5ba 100644 --- a/pipelex_sdk/product_models.py +++ b/pipelex_sdk/product_models.py @@ -17,11 +17,13 @@ from __future__ import annotations +import json from enum import StrEnum from typing import Any -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError, field_serializer, field_validator +from pipelex_sdk._pydantic_utils import empty_list_factory_of from pipelex_sdk.runs import RunStatus # ── User profile (`/v1/me`) ───────────────────────────────────────────── @@ -42,6 +44,88 @@ class UserProfile(BaseModel): # ── Methods catalog (`/v1/methods`) ────────────────────────────────────── +class MethodDeletionState(StrEnum): + """Where a method is in the erasure cascade; absent on a normal method.""" + + PENDING = "pending" + IN_PROGRESS = "in_progress" + FAILED = "failed" + + +class MethodFile(BaseModel): + """One named source file of a stored method — the at-rest catalog form. + + This is the shape the hosted platform persists for a method's custom PipeFunc Python: + a JSON `[{name, content}]` array in one wire string. It is deliberately distinct from + two neighbours that look similar and are not: `MthdsFile` (`client.py`) is the *validate* + input, content plus an optional provenance URI; `MthdsFileItem` (`build_models.py`) is + the *build* closure entry. Three shapes for three surfaces — do not merge them. + """ + + model_config = ConfigDict(extra="allow") + + #: Bundle-relative path, e.g. `"funcs/price.py"`. + name: str + #: The file's UTF-8 text content. + content: str + + +_METHOD_FILES_ADAPTER: TypeAdapter[list[MethodFile]] = TypeAdapter(list[MethodFile]) +"""Built once at import — TypeAdapter construction is expensive.""" + +_METHOD_FILES_SHAPE = "a JSON array of {name, content} entries" + + +def _is_blank(content: str) -> bool: + """A file carries no source when its content is empty or whitespace-only.""" + return not content.strip() + + +def parse_method_files(source: str | None) -> list[MethodFile]: + """Parse the catalog wire string into method files. + + A blank source (`None`, `""`, whitespace) and an empty JSON array both yield `[]`. + A JSON `[{name, content}]` array yields those files, with blank-content entries + dropped so the round-trip with `serialize_method_files` is stable. + + Raises: + ValueError: For anything else — a non-array JSON value, an entry that is not a + `{name: str, content: str}` object, or unparseable text. Reached through + `MethodData`'s validator, this surfaces as a `pydantic.ValidationError`, the + same way any other malformed response body fails here. + """ + if source is None or _is_blank(source): + return [] + + try: + parsed = json.loads(source) + except json.JSONDecodeError as exc: + msg = f"Method file source is not valid JSON; expected {_METHOD_FILES_SHAPE}." + raise ValueError(msg) from exc + + try: + files = _METHOD_FILES_ADAPTER.validate_python(parsed) + except ValidationError as exc: + msg = f"Method file source must be {_METHOD_FILES_SHAPE}." + raise ValueError(msg) from exc + + return [file for file in files if not _is_blank(file.content)] + + +def serialize_method_files(files: list[MethodFile]) -> str: + """Serialize method files to the catalog wire string. + + Blank-content entries are dropped (a zero-source file is not persisted), and an empty + result serializes to `""` — the platform's "no source" / "clear the field" sentinel — + never to the literal `"[]"`. Only `name` and `content` cross the wire; anything an + extension-open `MethodFile` picked up on the way in is not written back. + """ + kept = [file for file in files if not _is_blank(file.content)] + if not kept: + return "" + return json.dumps([{"name": file.name, "content": file.content} for file in kept]) + + class MethodData(BaseModel): """One saved method record.""" @@ -51,12 +135,35 @@ class MethodData(BaseModel): name: str #: The `.mthds` bundle source. mthds: str + org_id: str + created_by_user_id: str + description: str | None = None + deletion_state: MethodDeletionState | None = None input_data: dict[str, Any] | None = None #: Legacy persisted output spec; optional. pipe_output: dict[str, Any] | None = None + python: list[MethodFile] = Field(default_factory=empty_list_factory_of(MethodFile)) + """The method's custom PipeFunc source files. + + On the wire this is one string — the JSON text of a `[{name, content}]` array, or `""` + for a method with no custom Python. The validator below converts at the boundary so + callers never see that string.""" + created_at: str updated_at: str + @field_validator("python", mode="before") + @classmethod + def _parse_python_files(cls, value: object) -> object: + """Convert the catalog wire string into `MethodFile` entries. + + A `str` or `None` is the wire form and goes through `parse_method_files`; anything + else (a list, from programmatic construction) passes through to normal validation. + """ + if value is None or isinstance(value, str): + return parse_method_files(value) + return value + class MethodWriteInput(BaseModel): """The create/update payload — a rename is a `PUT` with a changed `name`.""" @@ -64,14 +171,51 @@ class MethodWriteInput(BaseModel): name: str mthds: str input_data: dict[str, Any] | None = None + python: list[MethodFile] | None = None + """The custom PipeFunc source files to write, with a deliberate three-way contract. + The write body is dumped with `exclude_none=True`, so `None` (the default) leaves the key + out entirely and a `PUT` **preserves** the stored Python. An empty list serializes to `""`, + the platform's clear sentinel, which **erases** it. A non-empty list **replaces** it.""" -class MethodDeletionState(StrEnum): - """Where a method is in the erasure cascade; absent on a normal method.""" + @field_serializer("python") + def _serialize_python_files(self, value: list[MethodFile] | None) -> str | None: + """Render the file list as the catalog wire string, leaving `None` for `exclude_none`.""" + if value is None: + return None + return serialize_method_files(value) - PENDING = "pending" - IN_PROGRESS = "in_progress" - FAILED = "failed" + +class MethodSummary(BaseModel): + """One row of the paged method index — `GET /v1/methods`. + + Deliberately **not** a `MethodData`: no `mthds`, no `python`, no `updated_at`, because + none of them is in the index projection. Putting `mthds` back is exactly what restored + the truncation bug paging was introduced to fix. A method mid-deletion still appears + here — so a UI can render "Deleting…" — while `get_method` refuses it with a `409`. + """ + + model_config = ConfigDict(extra="allow") + + method_id: str + name: str + description: str | None = None + created_at: str + deletion_state: MethodDeletionState | None = None + + +class MethodPage(BaseModel): + """One page of the method index — `{items, next_cursor}`. + + The cursor is opaque: pass it straight back as `cursor` to get the next page, and treat + a `None` as the last page. There is no total by design — counting a catalog costs a full + scan, and no caller needs one. + """ + + model_config = ConfigDict(extra="allow") + + items: list[MethodSummary] + next_cursor: str | None = None class MethodDeletionAccepted(BaseModel): @@ -344,10 +488,11 @@ class UploadedFile(BaseModel): filename: str -# ── Runs list / update (`/v1/runs`) ────────────────────────────────────── +# ── Run records (`/v1/runs`) ───────────────────────────────────────────── # # The run-lifecycle status/results/start routes already live on the client -# (`runs.py`); these are the remaining catalog-style list + admin-update routes. +# (`runs.py`); these are the remaining catalog-style paged list, the single-run +# detail read, and the admin-update route. class PipeStatus(StrEnum): @@ -360,22 +505,69 @@ class PipeStatus(StrEnum): SKIPPED = "skipped" +class RunErrorReport(BaseModel): + """A failed run's error, narrowed to the two fields a consumer may rely on. + + The runner's own report is considerably more verbose; only these two are contractual. + """ + + model_config = ConfigDict(extra="allow") + + message: str | None = None + error_type: str | None = None + + class PipelineRun(BaseModel): """One run record in a method's run list — `GET /v1/runs?method_id=…`.""" model_config = ConfigDict(extra="allow") pipeline_run_id: str - method_id: str - pipe_code: str + method_id: str | None = None + """The stored method this run is linked to, when there is one. An ad-hoc run from an + inline bundle belongs to no stored method, so the platform serves this as null.""" + + pipe_code: str | None = None + """The pipe that ran, when it was named. A run that let the bundle's `main_pipe` decide + has none to report, so the platform serves this as null.""" + + org_id: str | None = None + created_by_user_id: str | None = None workflow_id: str | None = None status: RunStatus result_url: str | None = None + error: RunErrorReport | None = None pipe_statuses: dict[str, PipeStatus] | None = None created_at: str finished_at: str | None = None +class RunDetail(PipelineRun): + """One run read on its own — `GET /v1/runs/{id}`. + + Adds the two heavy fields the list and the polled status deliberately leave out (their + cost scales with page size and poll rate respectively). `mthds_contents` is what the run + actually executed, and the only record of it: a method edited since the run no longer + describes what happened. + """ + + mthds_contents: list[str] | None = None + inputs: dict[str, Any] | None = None + + +class RunPage(BaseModel): + """One page of a method's run list — `{items, next_cursor}`. + + Same opaque-cursor contract as `MethodPage`: pass `next_cursor` straight back, and a + `None` means the last page. + """ + + model_config = ConfigDict(extra="allow") + + items: list[PipelineRun] + next_cursor: str | None = None + + class UpdateRunInput(BaseModel): """The admin/manual run-status patch — `status` is a free string here.""" diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py new file mode 100644 index 0000000..84dde2e --- /dev/null +++ b/tests/unit/conftest.py @@ -0,0 +1,55 @@ +"""Shared fixtures for the unit suite — a client, a wire-response builder, and the `_send` spy. + +House rule: fixtures live in `conftest.py`. `test_client_product.py` predates these and keeps +its own equivalent private helpers; migrating it is deliberately not part of this change. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol + +import httpx +import pytest + +from pipelex_sdk.client import PipelexAPIClient + +if TYPE_CHECKING: + from pytest_mock import MockerFixture, MockType + +BASE_URL = "http://localhost:8081" + + +class ResponseBuilder(Protocol): + """Builds one wire response the way the transport would hand it back.""" + + def __call__(self, status_code: int, *, json_body: object | None = None) -> httpx.Response: ... + + +class SendPatcher(Protocol): + """Patches a client's `_send` with a scripted response sequence and returns the spy.""" + + def __call__(self, client: PipelexAPIClient, *responses: httpx.Response) -> MockType: ... + + +@pytest.fixture +def api_client() -> PipelexAPIClient: + return PipelexAPIClient(api_key="test-token", base_url=BASE_URL) + + +@pytest.fixture +def wire_response() -> ResponseBuilder: + def _build(status_code: int, *, json_body: object | None = None) -> httpx.Response: + request = httpx.Request("GET", f"{BASE_URL}/x") + if json_body is None: + return httpx.Response(status_code, request=request) + return httpx.Response(status_code, json=json_body, request=request) + + return _build + + +@pytest.fixture +def patch_send(mocker: MockerFixture) -> SendPatcher: + def _patch(client: PipelexAPIClient, *responses: httpx.Response) -> MockType: + return mocker.patch.object(client, "_send", mocker.AsyncMock(side_effect=list(responses))) + + return _patch diff --git a/tests/unit/test_client_paging.py b/tests/unit/test_client_paging.py new file mode 100644 index 0000000..7ebc3fa --- /dev/null +++ b/tests/unit/test_client_paging.py @@ -0,0 +1,175 @@ +"""Tests for the paged-list iterators — `iterate_methods` and `iterate_runs`. + +Both follow `next_cursor` until the server says stop, but they stop on *different* signals, +and the difference is in the server rather than the client: `q` is a post-read filter over a +bounded index slice, so a method page can be empty with a live cursor; the run date bounds are +index key conditions, so a run page never is. These pin both rules, the stuck-cursor guard, and +the runaway backstop. +""" + +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING, Any, cast +from urllib.parse import parse_qs, urlparse + +import pytest + +from pipelex_sdk.errors import PagingNotTerminatingError + +if TYPE_CHECKING: + from pytest_mock import MockerFixture, MockType + + from pipelex_sdk.client import PipelexAPIClient + from pipelex_sdk.product_models import MethodSummary, PipelineRun + from tests.unit.conftest import ResponseBuilder, SendPatcher + + +def _method(method_id: str) -> dict[str, Any]: + return {"method_id": method_id, "name": f"Method {method_id}", "created_at": "t"} + + +def _run(run_id: str) -> dict[str, Any]: + return {"pipeline_run_id": run_id, "method_id": "m1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"} + + +def _cursors_sent(send: MockType) -> list[str | None]: + """The `cursor` query value of every request the spy recorded, in order.""" + cursors: list[str | None] = [] + for call in send.call_args_list: + url = cast("str", call.args[1]) + query: dict[str, list[str]] = parse_qs(urlparse(url).query) + values = query.get("cursor") + cursors.append(values[0] if values else None) + return cursors + + +async def _drain_methods(client: PipelexAPIClient, **kwargs: Any) -> list[MethodSummary]: + return [summary async for summary in client.iterate_methods(**kwargs)] + + +async def _drain_runs(client: PipelexAPIClient, method_id: str) -> list[PipelineRun]: + return [pipeline_run async for pipeline_run in client.iterate_runs(method_id)] + + +class TestClientPaging: + def test_iterate_methods_follows_the_cursor_and_stops_on_none( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + send = patch_send( + api_client, + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_method("m2")], "next_cursor": None}), + ) + + summaries = asyncio.run(_drain_methods(api_client)) + + assert [summary.method_id for summary in summaries] == ["m1", "m2"] + # The cursor sent on page N+1 is exactly the `next_cursor` received on page N. + assert _cursors_sent(send) == [None, "c1"] + + def test_iterate_methods_continues_through_an_empty_page_with_a_live_cursor( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """`q` filters after the index read, so an empty page means "keep going", not "done".""" + patch_send( + api_client, + wire_response(200, json_body={"items": [], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_method("m9")], "next_cursor": None}), + ) + + summaries = asyncio.run(_drain_methods(api_client, q="needle")) + + assert [summary.method_id for summary in summaries] == ["m9"] + + def test_iterate_methods_stops_on_an_unchanged_cursor_without_re_yielding( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """A server that hands back the cursor it was sent stops the loop *before* yielding.""" + patch_send( + api_client, + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "stuck"}), + wire_response(200, json_body={"items": [_method("m1")], "next_cursor": "stuck"}), + ) + + summaries = asyncio.run(_drain_methods(api_client)) + + assert [summary.method_id for summary in summaries] == ["m1"] + + def test_iterate_methods_raises_past_the_page_ceiling( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + mocker: MockerFixture, + ) -> None: + """The backstop raises rather than returning: a truncated list is the bug paging removed.""" + mocker.patch("pipelex_sdk.client._MAX_LIST_PAGES", 2) + pages = [wire_response(200, json_body={"items": [_method(f"m{index}")], "next_cursor": f"c{index}"}) for index in range(5)] + patch_send(api_client, *pages) + + with pytest.raises(PagingNotTerminatingError) as exc_info: + asyncio.run(_drain_methods(api_client)) + + assert exc_info.value.page_limit == 2 + + def test_iterate_runs_stops_on_an_empty_page( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + """Run date bounds are index key conditions, so an empty page really is the end.""" + patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r99")], "next_cursor": None}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1"] + + def test_iterate_runs_follows_the_cursor_and_stops_on_none( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + send = patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": None}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1", "r2"] + assert _cursors_sent(send) == [None, "c1"] + + def test_iterate_runs_stops_on_an_unchanged_cursor_without_re_yielding( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + ) -> None: + patch_send( + api_client, + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "stuck"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "stuck"}), + ) + + runs = asyncio.run(_drain_runs(api_client, "m1")) + + assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1"] diff --git a/tests/unit/test_client_product.py b/tests/unit/test_client_product.py index 0eabde1..82b5cc9 100644 --- a/tests/unit/test_client_product.py +++ b/tests/unit/test_client_product.py @@ -6,6 +6,7 @@ import asyncio import json +from typing import cast import httpx import pytest @@ -15,6 +16,7 @@ from pipelex_sdk.errors import ApiResponseError from pipelex_sdk.product_models import ( MethodDeletionState, + MethodFile, MethodWriteInput, OnboardingCurrentTool, OnboardingHeardFrom, @@ -29,6 +31,19 @@ _BASE_URL = "http://localhost:8081" +# The platform's `MethodPublic` shape: `org_id` and `created_by_user_id` are required, and +# `python` crosses the wire as one string holding a JSON `[{name, content}]` array. +_METHOD_BODY: dict[str, object] = { + "method_id": "m1", + "name": "M", + "mthds": "src", + "org_id": "org_1", + "created_by_user_id": "u1", + "python": "", + "created_at": "t", + "updated_at": "t", +} + def _response(status_code: int, *, json_body: object | None = None, content: bytes | None = None) -> httpx.Response: request = httpx.Request("GET", f"{_BASE_URL}/x") @@ -39,6 +54,12 @@ def _response(status_code: int, *, json_body: object | None = None, content: byt return httpx.Response(status_code, request=request) +def _sent_body(send: MockType) -> dict[str, object]: + """The decoded JSON request body of the single recorded `_send` call.""" + content = send.call_args.kwargs["content"] + return cast("dict[str, object]", json.loads(content)) + + class _Sent: """The single `_send` call recorded by the spy, decoded for assertions.""" @@ -80,20 +101,80 @@ def test_get_me(self, mocker: MockerFixture) -> None: # ── Methods catalog ────────────────────────────────────────────── - def test_list_methods(self, mocker: MockerFixture) -> None: + def test_list_methods_returns_the_page_envelope(self, mocker: MockerFixture) -> None: + """The route answers `{items, next_cursor}`; the rows are index projections, not full methods.""" client = self._client() - methods = [{"method_id": "m1", "name": "M", "mthds": "...", "created_at": "t", "updated_at": "t"}] - send = self._mock_send(mocker, client, _response(200, json_body=methods)) + page = { + "items": [{"method_id": "m1", "name": "M", "description": "d", "created_at": "t", "deletion_state": "pending"}], + "next_cursor": "c1", + } + send = self._mock_send(mocker, client, _response(200, json_body=page)) result = asyncio.run(client.list_methods()) assert self._sent(send).url == f"{_BASE_URL}/v1/methods" - assert result[0].method_id == "m1" - assert result[0].name == "M" + assert result.next_cursor == "c1" + assert result.items[0].method_id == "m1" + assert result.items[0].description == "d" + # A method mid-deletion stays listed, so a UI can render "Deleting…". + assert result.items[0].deletion_state is MethodDeletionState.PENDING + + def test_list_methods_keeps_query_params_on_presence(self, mocker: MockerFixture) -> None: + """An explicit empty `q` is forwarded — bad input the API should reject, not something to drop.""" + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_methods(q="", limit=50, cursor="c/1")) + + assert self._sent(send).url == f"{_BASE_URL}/v1/methods?q=&limit=50&cursor=c%2F1" + + def test_list_methods_omits_absent_query_params_entirely(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_methods(limit=5)) + + assert self._sent(send).url == f"{_BASE_URL}/v1/methods?limit=5" + + def test_method_data_parses_the_new_fields_and_the_python_wire_string(self, mocker: MockerFixture) -> None: + """`python` is one wire string; the boundary converts it so callers never see it.""" + client = self._client() + body = {**_METHOD_BODY, "description": "d", "python": '[{"name": "a.py", "content": "x = 1"}]'} + self._mock_send(mocker, client, _response(200, json_body=body)) + + method = asyncio.run(client.get_method("m1")) + + assert method.org_id == "org_1" + assert method.created_by_user_id == "u1" + assert method.description == "d" + assert method.deletion_state is None + assert method.python == [MethodFile(name="a.py", content="x = 1")] + + def test_method_data_reads_the_clear_sentinel_as_no_files(self, mocker: MockerFixture) -> None: + client = self._client() + self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + + assert asyncio.run(client.get_method("m1")).python == [] + + def test_write_input_sends_python_three_ways(self, mocker: MockerFixture) -> None: + """`None` omits the key (preserve), `[]` sends the clear sentinel, a list sends the JSON text.""" + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + + asyncio.run(client.update_method("m1", MethodWriteInput(name="M", mthds="src"))) + assert "python" not in _sent_body(send) + + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + asyncio.run(client.update_method("m1", MethodWriteInput(name="M", mthds="src", python=[]))) + assert _sent_body(send)["python"] == "" + + send = self._mock_send(mocker, client, _response(200, json_body=_METHOD_BODY)) + asyncio.run(client.create_method(MethodWriteInput(name="M", mthds="src", python=[MethodFile(name="a.py", content="x = 1")]))) + assert json.loads(cast("str", _sent_body(send)["python"])) == [{"name": "a.py", "content": "x = 1"}] def test_get_method_encodes_id(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "a/b", "name": "M", "mthds": "...", "created_at": "t", "updated_at": "t"} + body = {**_METHOD_BODY, "method_id": "a/b"} send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.get_method("a/b")) @@ -102,7 +183,7 @@ def test_get_method_encodes_id(self, mocker: MockerFixture) -> None: def test_create_method_posts_write_body(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "m1", "name": "M", "mthds": "src", "created_at": "t", "updated_at": "t"} + body = _METHOD_BODY send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.create_method(MethodWriteInput(name="M", mthds="src", input_data={"a": 1}))) @@ -114,7 +195,7 @@ def test_create_method_posts_write_body(self, mocker: MockerFixture) -> None: def test_update_method_puts_and_drops_absent_input_data(self, mocker: MockerFixture) -> None: client = self._client() - body = {"method_id": "m1", "name": "Renamed", "mthds": "src", "created_at": "t", "updated_at": "t"} + body = {**_METHOD_BODY, "name": "Renamed"} send = self._mock_send(mocker, client, _response(200, json_body=body)) asyncio.run(client.update_method("m1", MethodWriteInput(name="Renamed", mthds="src"))) @@ -376,14 +457,72 @@ def test_upload_base64_payload(self, mocker: MockerFixture) -> None: def test_list_runs_encodes_query_value(self, mocker: MockerFixture) -> None: client = self._client() - runs = [{"pipeline_run_id": "r1", "method_id": "m/1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"}] - send = self._mock_send(mocker, client, _response(200, json_body=runs)) + page = { + "items": [{"pipeline_run_id": "r1", "method_id": "m/1", "pipe_code": "p", "status": "RUNNING", "created_at": "t"}], + "next_cursor": "c1", + } + send = self._mock_send(mocker, client, _response(200, json_body=page)) result = asyncio.run(client.list_runs("m/1")) assert self._sent(send).url == f"{_BASE_URL}/v1/runs?method_id=m%2F1" - assert result[0].pipeline_run_id == "r1" - assert result[0].status is RunStatus.RUNNING + assert result.next_cursor == "c1" + assert result.items[0].pipeline_run_id == "r1" + assert result.items[0].status is RunStatus.RUNNING + + def test_list_runs_keeps_date_bounds_and_paging_params_on_presence(self, mocker: MockerFixture) -> None: + client = self._client() + send = self._mock_send(mocker, client, _response(200, json_body={"items": [], "next_cursor": None})) + + asyncio.run(client.list_runs("m1", created_from="2026-08-01T00:00:00+00:00", created_to="", limit=10, cursor="c1")) + + # Instants are percent-encoded; an explicit empty bound is forwarded, not dropped. + assert self._sent(send).url == ( + f"{_BASE_URL}/v1/runs?method_id=m1&created_from=2026-08-01T00%3A00%3A00%2B00%3A00&created_to=&limit=10&cursor=c1" + ) + + def test_run_row_parses_with_null_method_id_and_pipe_code(self, mocker: MockerFixture) -> None: + """An ad-hoc run belongs to no stored method, and a `main_pipe` run names no pipe.""" + client = self._client() + row = { + "pipeline_run_id": "r1", + "method_id": None, + "pipe_code": None, + "status": "FAILED", + "created_at": "t", + "error": {"message": "boom", "error_type": "PipeExecutionError"}, + } + self._mock_send(mocker, client, _response(200, json_body={"items": [row], "next_cursor": None})) + + result = asyncio.run(client.list_runs("m1")) + + pipeline_run = result.items[0] + assert pipeline_run.method_id is None + assert pipeline_run.pipe_code is None + assert pipeline_run.error is not None + assert pipeline_run.error.message == "boom" + assert pipeline_run.error.error_type == "PipeExecutionError" + + def test_get_run_detail_encodes_id_and_returns_what_ran(self, mocker: MockerFixture) -> None: + """The detail read is the only one carrying `mthds_contents` and `inputs`.""" + client = self._client() + body = { + "pipeline_run_id": "r/1", + "method_id": "m1", + "pipe_code": "p", + "status": "COMPLETED", + "created_at": "t", + "mthds_contents": ["domain = 'x'"], + "inputs": {"topic": "quantum"}, + } + send = self._mock_send(mocker, client, _response(200, json_body=body)) + + detail = asyncio.run(client.get_run_detail("r/1")) + + assert self._sent(send).url == f"{_BASE_URL}/v1/runs/r%2F1" + assert detail.mthds_contents == ["domain = 'x'"] + assert detail.inputs == {"topic": "quantum"} + assert detail.status is RunStatus.COMPLETED def test_update_run_drops_absent_finished_at_and_tolerates_empty_body(self, mocker: MockerFixture) -> None: client = self._client() diff --git a/tests/unit/test_method_files.py b/tests/unit/test_method_files.py new file mode 100644 index 0000000..eb8528c --- /dev/null +++ b/tests/unit/test_method_files.py @@ -0,0 +1,78 @@ +"""Tests for the method-files catalog converter — `parse_method_files` / `serialize_method_files`. + +Mirrors `mthds-js/src/protocol/method_files.ts`. The at-rest catalog form is one wire string +holding a JSON `[{name, content}]` array, with `""` as the platform's "no source" sentinel; +these pin both sentinels, the blank-content drop, and the round-trip. +""" + +from __future__ import annotations + +import json + +import pytest + +from pipelex_sdk.product_models import MethodFile, parse_method_files, serialize_method_files + + +class TestMethodFiles: + @pytest.mark.parametrize("blank_source", [None, "", " ", "\n\t ", "[]"]) + def test_blank_source_and_empty_array_both_parse_to_no_files(self, blank_source: str | None) -> None: + """A blank source and an explicit empty array both mean "this method has no Python".""" + assert parse_method_files(blank_source) == [] + + def test_parses_a_named_array(self) -> None: + files = parse_method_files('[{"name": "funcs/price.py", "content": "def price(): ...\\n"}]') + assert len(files) == 1 + assert files[0].name == "funcs/price.py" + assert files[0].content == "def price(): ...\n" + + def test_drops_blank_content_entries_on_parse(self) -> None: + """A zero-source file is not a file — dropped on the way in, mirroring serialization.""" + files = parse_method_files('[{"name": "a.py", "content": "x = 1"}, {"name": "empty.py", "content": " "}]') + assert [file.name for file in files] == ["a.py"] + + def test_tolerates_extra_keys_on_an_entry(self) -> None: + """`MethodFile` is extension-open, so a newly-added server key must not fail the read.""" + files = parse_method_files('[{"name": "a.py", "content": "x = 1", "sha": "deadbeef"}]') + assert files[0].name == "a.py" + + @pytest.mark.parametrize( + "bad_source", + [ + "not json at all", + '{"name": "a.py", "content": "x"}', # a JSON object, not an array + '"just a string"', + "42", + '[{"name": "a.py"}]', # entry missing `content` + '[{"content": "x = 1"}]', # entry missing `name` + '[{"name": 1, "content": "x = 1"}]', # `name` is not a string + "[[]]", # entry is not an object + ], + ) + def test_malformed_source_raises_naming_the_expected_shape(self, bad_source: str) -> None: + with pytest.raises(ValueError, match=r"\{name, content\}"): + parse_method_files(bad_source) + + def test_empty_list_serializes_to_the_clear_sentinel(self) -> None: + """`""` is the platform's clear signal; the literal `"[]"` would not clear anything.""" + assert serialize_method_files([]) == "" + + def test_blank_only_list_serializes_to_the_clear_sentinel(self) -> None: + assert serialize_method_files([MethodFile(name="empty.py", content=" \n")]) == "" + + def test_serializes_only_name_and_content(self) -> None: + """Whatever an extension-open `MethodFile` picked up on the way in is not written back.""" + parsed = parse_method_files('[{"name": "a.py", "content": "x = 1", "sha": "deadbeef"}]') + assert json.loads(serialize_method_files(parsed)) == [{"name": "a.py", "content": "x = 1"}] + + def test_serializes_mixed_list_dropping_the_blank_entry(self) -> None: + files = [MethodFile(name="a.py", content="x = 1"), MethodFile(name="empty.py", content=""), MethodFile(name="b.py", content="y = 2")] + assert json.loads(serialize_method_files(files)) == [{"name": "a.py", "content": "x = 1"}, {"name": "b.py", "content": "y = 2"}] + + def test_round_trip_is_stable(self) -> None: + """Serialize → parse → serialize reaches a fixed point, blank entries and all.""" + files = [MethodFile(name="a.py", content="x = 1"), MethodFile(name="empty.py", content=" ")] + once = serialize_method_files(files) + twice = serialize_method_files(parse_method_files(once)) + assert once == twice + assert parse_method_files(twice) == [MethodFile(name="a.py", content="x = 1")] From cb70fbe5eeee54671b0475f75f2c77963b2aa2d8 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:36:02 +0200 Subject: [PATCH 11/20] Reject a non-string method_id at the client boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A published client validates its request-option types where it names them, raising rather than dropping or forwarding a wrong-typed value. The bare truthiness check this replaces made an arbitrary partition of the wrong values: it silently dropped the falsy ones (`0`, `[]`) and forwarded the truthy ones (`123`, `["mt_1"]`) to a server `422` — a different partition than `@pipelex/sdk` makes for the same argument on the same wire. One wrong value, one answer, and now the same one in both SDKs. `None` and `""` still contribute nothing. The merge helper takes `object` rather than `str | None` because it *is* the runtime boundary, and the callers it guards against are the untyped ones a type checker never sees. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 2 ++ TODOS.md | 22 ++++++++++++-------- docs/architecture.md | 1 + pipelex_sdk/client.py | 31 ++++++++++++++++++++--------- tests/unit/test_client_method_id.py | 18 +++++++++++++++++ 5 files changed, 57 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6ab78e..72c06a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,8 @@ ### Changed +- **A non-string `method_id` is refused at the client boundary.** `execute` / `start` / `start_and_wait` now raise `PipelineRequestError` naming the received type, instead of letting a bare truthiness check decide. That check made an arbitrary partition of the wrong values — it silently dropped the falsy ones (`0`, `[]`) and forwarded the truthy ones (`123`, `["mt_1"]`) to a server `422` — and it was a *different* partition than `@pipelex/sdk` made for the same argument on the same wire. A published client validates its request-option types where it names them, so one wrong value gets one answer. `None` and `""` still contribute nothing, unchanged. The protocol-level arguments (`pipe_code`, `mthds_contents`) are not covered here: their guards belong in `mthds` and arrive with a floor bump. + - **Breaking: the two product list routes return page envelopes.** `list_methods` returns a `MethodPage` (`items` are `MethodSummary` rows, not `MethodData`) and `list_runs` returns a `RunPage`, each carrying an opaque `next_cursor` that is `None` on the last page. Both take the paging and filter parameters the platform serves — `q` / `limit` / `cursor` on methods, `created_from` / `created_to` / `limit` / `cursor` on runs — kept on **presence** rather than truthiness, so an explicit empty `q` or `cursor` is forwarded for the API to reject rather than dropped silently into an unfiltered query that reads as working. `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, and are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also worth knowing: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed — a `403` there means "flag", not "wrong key". - **Breaking: `PipelineRun.method_id` and `pipe_code` are nullable, and `MethodData.python` is a `list[MethodFile]`.** The platform genuinely serves the first two as null and both cases are ordinary: an ad-hoc run from an inline bundle belongs to no stored method, and a run that let the bundle's `main_pipe` decide names no pipe. `PipelineRun` also gains `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None`. `MethodData.python` is converted from its wire string at the boundary, so callers never see that string. diff --git a/TODOS.md b/TODOS.md index b892550..0982b4e 100644 --- a/TODOS.md +++ b/TODOS.md @@ -143,19 +143,25 @@ Breaking, and the most urgent fix in this plan: `list_methods` and `list_runs` c ### Checkpoint 2 -- [ ] Update this file: tick what landed, record the Phase 2 and Phase 3 commit SHAs, and note any place the Python shapes deliberately diverge from the JS mirror beyond snake_case (there should be none besides the `python` converter living here). +- [x] Update this file: tick what landed, record the Phase 2 and Phase 3 commit SHAs, and note any place the Python shapes deliberately diverge from the JS mirror beyond snake_case (there should be none besides the `python` converter living here). + +**Phase 2 landed in `2a8c589`, Phase 3 in `5e01c1b`.** Notes: + +- **Divergences from the JS mirror beyond snake_case:** the `python` converter lives here rather than in `mthds-python` (the decision of `wip/updates.md` §7.2), and `parse_method_files` raises `ValueError` where the JS pair raises `PipelineRequestError` — because the Python parser is reached through a pydantic `field_validator`, where a `ValueError` is the idiomatic signal and surfaces to the caller as a `pydantic.ValidationError` like any other malformed response body. Nothing else diverges. +- **`MethodPage.items` / `RunPage.items` are required**, not defaulted empty. A page body with no `items` is malformed, and failing loudly is the whole point of this phase — the previous shape failed silently in the tests and loudly in production. +- **The shared test fixtures landed in a new `tests/unit/conftest.py`** (`api_client`, `wire_response`, `patch_send`), used by the two new modules. Migrating `test_client_product.py` onto them was left out as the plan allowed; it keeps its own equivalent private helpers. ## Phase 4 — `method_id` boundary type guard (`wip/updates.md` §4) The 2026-08-25 decision in `pipelex-sdk-js/wip/boundary-option-type-validation.md`: a published client validates request-option types at its boundary and raises `PipelineRequestError` rather than dropping or forwarding a wrong-typed value. Its evidence names this repo's bare `if method_id:` in `_merge_hosted_run_extensions`, which drops falsy non-strings and forwards truthy ones to a server `422`. -- [ ] `_merge_hosted_run_extensions` (`client.py` near line 975): replace `if method_id:` with an explicit presence check (`if method_id is not None`) followed by `if not isinstance(method_id, str): raise PipelineRequestError(msg)` naming the received type, then the existing empty-string-is-absent normalization. `None` and `""` still contribute nothing. -- [ ] Update the docstring's `Raises:` and the "An absent or empty `method_id`" paragraph. -- [ ] `tests/unit/test_client_method_id.py`: one parametrized test over wrong-typed values (`0`, `123`, `[]`, `["mt_1"]`, `{}`) asserting `PipelineRequestError` on `execute` and `start` before any request is sent; keep `test_empty_method_id_is_absent` green. -- [ ] `docs/architecture.md` → "Hosted run extensions (`method_id`)": add a bullet that a non-string raises at the boundary, and why (one partition of wrong values across both SDKs). -- [ ] `CHANGELOG.md` `[Unreleased]` → **Changed**: the guard, with the JS decision as the reason. -- [ ] `make agent-check` and `make agent-test` pass. -- [ ] Commit (suggested message: "Reject a non-string method_id at the client boundary"). +- [x] `_merge_hosted_run_extensions` (`client.py` near line 975): replace `if method_id:` with an explicit presence check (`if method_id is not None`) followed by `if not isinstance(method_id, str): raise PipelineRequestError(msg)` naming the received type, then the existing empty-string-is-absent normalization. `None` and `""` still contribute nothing. +- [x] Update the docstring's `Raises:` and the "An absent or empty `method_id`" paragraph. +- [x] `tests/unit/test_client_method_id.py`: one parametrized test over wrong-typed values (`0`, `123`, `[]`, `["mt_1"]`, `{}`) asserting `PipelineRequestError` on `execute` and `start` before any request is sent; keep `test_empty_method_id_is_absent` green. +- [x] `docs/architecture.md` → "Hosted run extensions (`method_id`)": add a bullet that a non-string raises at the boundary, and why (one partition of wrong values across both SDKs). +- [x] `CHANGELOG.md` `[Unreleased]` → **Changed**: the guard, with the JS decision as the reason. +- [x] `make agent-check` and `make agent-test` pass. +- [x] Commit (suggested message: "Reject a non-string method_id at the client boundary"). ## Phase 5 — wrap-up diff --git a/docs/architecture.md b/docs/architecture.md index 0f5f8c5..ddc42c2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -88,6 +88,7 @@ The protocol's run args (`pipe_code`, `mthds_contents`, `inputs`, …) are the b - **Alone it is a run source; alongside an inline source it is linkage.** With no inline source the platform resolves and runs the stored method. With one, the inline source is what RUNS (precedence) and the id is recorded as run-history linkage on the Run row — the index key `GET /v1/runs?method_id=` queries. The base client's "something to run" precondition is satisfied because the merged extension mapping is non-empty, so a `method_id`-only run is sent rather than refused client-side. - **It rides the blocking fallback too.** `start_and_wait` degrades to `POST /v1/execute` against a runner with no run store, and the selector goes with it — so a bare `pipelex-api` answers the `422` that names the key instead of the client silently dropping it and running something else. - **An empty string is treated as absent.** `method_id=""` selects no method and links nothing, so it is not sent and does not satisfy the precondition. +- **A non-string raises at the boundary.** A published client validates its request-option types where it names them, raising `PipelineRequestError` rather than dropping or forwarding a wrong-typed value — so one wrong value gets one answer. Without the guard the partition is arbitrary: a bare truthiness check drops the falsy wrong types (`0`, `[]`) and forwards the truthy ones (`123`, `["mt_1"]`) to a server `422`, which is a *different* partition than `@pipelex/sdk` makes for the same argument on the same wire. Both SDKs now make the same one. ## Run lifecycle (hosted extension) diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 7993f03..2b181cf 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -1115,7 +1115,7 @@ async def health(self) -> dict[str, Any]: _KNOWN_RUN_STATUS_NAMES: frozenset[str] = frozenset(RunStatus.__members__) -def _merge_hosted_run_extensions(extra: dict[str, Any] | None, method_id: str | None) -> dict[str, Any] | None: +def _merge_hosted_run_extensions(extra: dict[str, Any] | None, method_id: object) -> dict[str, Any] | None: """Fold the hosted API's own run args into the generic `extra` passthrough handed to the base client. This is the seam between layer 3 and layer 2: `method_id` is a named parameter on this @@ -1123,28 +1123,41 @@ def _merge_hosted_run_extensions(extra: dict[str, Any] | None, method_id: str | the wire as a top-level body property through the protocol client's extension mechanism — which merges it without knowing what it means. See `_HOSTED_RUN_ARGS`. - An absent or empty `method_id` contributes nothing: `method_id=""` selects no method and - links no run, so it is not sent and does not satisfy the base client's "something to run" - precondition. `None` is returned for an empty result, leaving the base's own handling of an - absent `extra` untouched. + A **non-string** `method_id` is refused here rather than dropped or forwarded. A published + client validates its request-option types at its own boundary, so that one wrong value gets + one answer: a bare truthiness check would silently drop the falsy wrong types (`0`, `[]`) + and forward the truthy ones (`123`, `["mt_1"]`) to a server `422` — a different partition of + wrong values than the JS client makes for the same argument on the same wire. + + An absent or empty `method_id` still contributes nothing: `method_id=""` selects no method + and links no run, so it is not sent and does not satisfy the base client's "something to + run" precondition, and neither does `None`. `None` is returned for an empty result, leaving + the base's own handling of an absent `extra` untouched. Args: extra: Server-specific extension args from the caller, or None. - method_id: The hosted catalog id, or None. + method_id: The hosted catalog id, or None. Typed `object` rather than `str | None` + deliberately — this helper *is* the runtime boundary, and the callers it guards + against are the untyped ones a type checker never sees. Returns: The merged extension mapping to hand to the base client, or None if there is nothing. Raises: - PipelineRequestError: If `extra` carries a hosted arg this client names itself. + PipelineRequestError: If `extra` carries a hosted arg this client names itself, or if + `method_id` is present and is not a string. """ extensions: dict[str, Any] = dict(extra or {}) hosted_overlap = extensions.keys() & _HOSTED_RUN_ARGS if hosted_overlap: msg = f"extra carries hosted args {sorted(hosted_overlap)} — pass them as named parameters instead." raise PipelineRequestError(msg) - if method_id: - extensions["method_id"] = method_id + if method_id is not None: + if not isinstance(method_id, str): + msg = f"method_id must be a string, received {type(method_id).__name__}." + raise PipelineRequestError(msg) + if method_id: + extensions["method_id"] = method_id return extensions or None diff --git a/tests/unit/test_client_method_id.py b/tests/unit/test_client_method_id.py index 78bf96d..9edf588 100644 --- a/tests/unit/test_client_method_id.py +++ b/tests/unit/test_client_method_id.py @@ -102,6 +102,24 @@ def test_empty_method_id_is_absent(self, mocker: MockerFixture) -> None: with pytest.raises(PipelineRequestError): asyncio.run(client.start(method_id="")) + @pytest.mark.parametrize("wrong_typed_method_id", [0, 123, [], ["mt_1"], {}, 1.5, True]) + def test_non_string_method_id_raises_before_any_request(self, mocker: MockerFixture, wrong_typed_method_id: object) -> None: + """A wrong-typed selector is refused at the boundary, on both run routes, before the wire. + + Without the guard the partition of wrong values is arbitrary: a bare truthiness check + drops the falsy ones (`0`, `[]`) and forwards the truthy ones (`123`, `["mt_1"]`) to a + server `422`. One wrong value, one answer. + """ + client = self._client() + send = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(202, json=_START_BODY))) + + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + asyncio.run(client.execute(pipe_code="p", method_id=wrong_typed_method_id)) # type: ignore[arg-type] + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + asyncio.run(client.start(pipe_code="p", method_id=wrong_typed_method_id)) # type: ignore[arg-type] + + send.assert_not_called() + def test_blocking_fallback_forwards_method_id(self, mocker: MockerFixture) -> None: """A bare runner must SEE the selector, so it can answer the 422 that names it. From a857d0214e9093f3fae058458fe07b9d0952d14c Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:37:08 +0200 Subject: [PATCH 12/20] Make the architecture parity section tell the truth end to end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three claims the code no longer supports, found on the Phase-5 read-through: the intro line advertising "the full 0.1.0 surface", the parity section's "Methods — full coverage" (which contradicted the gap list directly above it), and "Models — full field-for-field match". The Models paragraph now also names the two deliberate divergences from the JS mirror: the page envelopes keep the wire's snake_case `next_cursor`, and the method-files converter lives here rather than in the protocol package. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- TODOS.md | 8 ++++---- docs/architecture.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/TODOS.md b/TODOS.md index 0982b4e..694b8d3 100644 --- a/TODOS.md +++ b/TODOS.md @@ -165,10 +165,10 @@ The 2026-08-25 decision in `pipelex-sdk-js/wip/boundary-option-type-validation.m ## Phase 5 — wrap-up -- [ ] `make check` (adds pylint to the agent gate) and `make agent-test` pass on the final tree. -- [ ] Re-read `docs/architecture.md` end to end for any remaining claim the code no longer supports (parity, `Out of scope`, the validate section, the product section). -- [ ] Re-read `CHANGELOG.md` `[Unreleased]`: every breaking item is labelled "breaking", no counts, no WIP-doc mentions, the version line untouched. -- [ ] `wip/updates.md` stays where it is, with its §7 decisions; this file stays too. Neither is deleted or emptied as part of finishing the work. +- [x] `make check` (adds pylint to the agent gate) and `make agent-test` pass on the final tree. +- [x] Re-read `docs/architecture.md` end to end for any remaining claim the code no longer supports (parity, `Out of scope`, the validate section, the product section). Three further corrections beyond the ones §3.5 named: the intro line claiming "the full `0.1.0` surface", the parity section's "**Methods** — full coverage" (which contradicted the gap list directly above it), and the "**Models** — full field-for-field match" claim; the Models paragraph now also names the two deliberate divergences (snake_case `next_cursor`, the converter's home). +- [x] Re-read `CHANGELOG.md` `[Unreleased]`: every breaking item is labelled "breaking", no counts, no WIP-doc mentions, the version line untouched (still `0.5.0`). +- [x] `wip/updates.md` stays where it is, with its §7 decisions; this file stays too. Neither is deleted or emptied as part of finishing the work. - [ ] Open the PR against `dev`, then follow `/review-pr-agents` for the Greptile / Codex loop (compare SHAs, not notifications; reply and resolve each thread in one pass). ### Checkpoint 3 diff --git a/docs/architecture.md b/docs/architecture.md index ddc42c2..b8db7f4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -200,9 +200,9 @@ This SDK is a port of the TypeScript `@pipelex/sdk` (`PipelexApiClient`) and tra None of them is moved by the releases this SDK last tracked, and each stays deferred rather than silently missing. Everything else — the protocol routes, the durable lifecycle, the whole product surface, and the errors — does have a Python equivalent. -**Methods** — full coverage: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records), and `health`. +**Methods** — everything outside the gap list above has a counterpart: protocol (`execute`, `start`, `validate`, `validate_files`, `models`, `version`), durable lifecycle (`get_run_status`, `get_run_result`, `wait_for_result`, `start_and_wait`, the private `_supports_run_lifecycle` / `_execute_blocking`), the whole product surface (profile, methods CRUD with paged listing and the two iterators, organizations, billing, Pipelex API keys, gateway key, onboarding, storage, run records with `get_run_detail`), `build_inputs`, the input-preparation surface (`upload_file` / `prepare_inputs`), and `health`. -**Models** — full field-for-field match across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. +**Models** — field-for-field across the run-lifecycle types and the product wire models. Deliberate idiomatic ports (not gaps): milliseconds → seconds (`interval_seconds` / `timeout_seconds` / `elapsed_seconds`); the JS `AbortSignal` → Python `asyncio` cancellation (no `signal` field); JS inline string-unions promoted to `StrEnum`s (`OrgRole`, `PipeStatus`, the onboarding fields) with identical wire values; response models are `extra="allow"` for forward-compat. The Pipelex validation narrowing is **owned here** (`pipelex_sdk.validation_models`), narrowing `mthds`'s neutral verdict bases (the resolved follow-up #9); the brand-neutral `Dict*` wire concretes (`DictRunResultExecute`) are reused from `mthds` by inheritance — they are a shared wire contract the `pipelex` runtime also builds on — rather than duplicated as `pipelex-sdk-js` does. Two divergences worth naming: the page envelopes keep the wire's snake_case `next_cursor`, where the JS mirror renamed it `nextCursor` for its own consumers; and the method-files catalog converter (`parse_method_files` / `serialize_method_files`) lives in this package, where the JS pair lives in `mthds-js` because `pipelex-mcp` consumes the same format and wanted one owner. There is no second Python consumer, and the catalog serialization is a Pipelex product concern rather than an MTHDS protocol one, so this SDK is a proper home for it. If `mthds-python` ever grows an owner for the format, this SDK adopts it then. **Errors** — `ApiResponseError`, `ApiUnreachableError`, `PipelineExecuteTimeoutError`, `RunFailedError`, `RunTimeoutError`, `RunLifecycleUnavailableError`, `PagingNotTerminatingError` are owned here; `RunStillRunningError` is re-exported from `mthds`. `ClientAuthenticationError` is **not** ported: it is a dormant export in the JS barrel (defined and exported but never raised by the client), and in Python it already lives in `mthds.runners.api.exceptions` — importable directly if ever needed, with no barrel here to re-export it through. From e57e2d62d7aba14c7e24abdd8c8e9652e0054963 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:37:58 +0200 Subject: [PATCH 13/20] Record the PR and the landed SHAs in the implementation tracker Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- TODOS.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/TODOS.md b/TODOS.md index 694b8d3..91cefc5 100644 --- a/TODOS.md +++ b/TODOS.md @@ -169,11 +169,27 @@ The 2026-08-25 decision in `pipelex-sdk-js/wip/boundary-option-type-validation.m - [x] Re-read `docs/architecture.md` end to end for any remaining claim the code no longer supports (parity, `Out of scope`, the validate section, the product section). Three further corrections beyond the ones §3.5 named: the intro line claiming "the full `0.1.0` surface", the parity section's "**Methods** — full coverage" (which contradicted the gap list directly above it), and the "**Models** — full field-for-field match" claim; the Models paragraph now also names the two deliberate divergences (snake_case `next_cursor`, the converter's home). - [x] Re-read `CHANGELOG.md` `[Unreleased]`: every breaking item is labelled "breaking", no counts, no WIP-doc mentions, the version line untouched (still `0.5.0`). - [x] `wip/updates.md` stays where it is, with its §7 decisions; this file stays too. Neither is deleted or emptied as part of finishing the work. -- [ ] Open the PR against `dev`, then follow `/review-pr-agents` for the Greptile / Codex loop (compare SHAs, not notifications; reply and resolve each thread in one pass). +- [x] Open the PR against `dev`, then follow `/review-pr-agents` for the Greptile / Codex loop (compare SHAs, not notifications; reply and resolve each thread in one pass). **PR [#14](https://github.com/Pipelex/pipelex-sdk-python/pull/14).** ### Checkpoint 3 -- [ ] Update this file with the final commit SHAs, the PR number, and anything deferred out of the plan with the reason. +- [x] Update this file with the final commit SHAs, the PR number, and anything deferred out of the plan with the reason. + +**PR [#14](https://github.com/Pipelex/pipelex-sdk-python/pull/14), against `dev`.** The five implementation commits, in order: + +| SHA | What | +|---|---| +| `434b2e3` | Phase 1 — the validate surface: the `views` opt-in and the typed 0.17/0.18 contract | +| `2a8c589` | Phase 2 — the `TokensUsageRecord` attribution and the unopenable citations | +| `5e01c1b` | Phase 3 — paged method and run lists, nullable run fields, the `python` converter | +| `cb70fbe` | Phase 4 — the non-string `method_id` boundary guard | +| `a857d02` | Phase 5 — the remaining untrue parity claims in `docs/architecture.md` | + +Deferred out of the plan, with the reason: + +- **`RemapValueOp.mapping` is not constrained to be non-empty**, where the runtime and the OpenAPI artifact both say `minProperties: 1`. Reader models should not fail a whole verdict over an op that is merely a no-op. Recorded at Checkpoint 1. +- **Migrating `test_client_product.py` onto the new `tests/unit/conftest.py` fixtures** was explicitly optional in §3.4 and was not done; the module keeps its own equivalent private helpers. +- **`pipelex_sdk/runs.py`'s "Wire contract mirrors `pipelex-platform`" line** was left alone. It names a service, not a file path, so it is not one of the unopenable citations Phase 2 was about, and widening that phase's scope on my own judgement was not warranted. ## Known gaps left open on purpose From fa8d62aa49cd3ba85dc87d6a58d995be910141d4 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 09:45:32 +0200 Subject: [PATCH 14/20] Bound iterate_runs against a cursor that cycles over non-empty pages The plan reasoned that the run iterator needed no page ceiling because its empty-page stop already caught a server minting fresh cursors. That reason was too narrow: the empty-page stop only fires when the server returns *nothing*, so a cursor cycling across two or more values (`c1 -> c2 -> c1`) while every page is non-empty trips neither it nor the adjacent-cursor check, and the generator would request forever while re-yielding the same runs. Both iterators now share `_MAX_LIST_PAGES` and raise `PagingNotTerminatingError` at it. Reusing the existing backstop is deliberate over tracking every cursor seen, which would cost unbounded memory for the same protection. Found independently by Greptile and Codex on PR #14. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 2 +- TODOS.md | 2 +- docs/architecture.md | 2 +- pipelex_sdk/client.py | 22 ++++++++++++++++++---- tests/unit/test_client_paging.py | 27 +++++++++++++++++++++++++++ 5 files changed, 48 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 72c06a6..1962ad3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,7 @@ - **Structured repair proposals: `SuggestedFix` and the fix-op vocabulary.** `ValidationErrorItem.suggested_fix` is now typed. A `SuggestedFix` carries a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops` — a list discriminated on `kind` over the closed `FixOpKind` set (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), each variant declaring exactly its own members, narrowed with an exhaustive `match op: case SetKeyOp(): …`. These are **reader** models: `extra="allow"`, not `frozen`, and without the runtime's wildcard-refusing validators, because this SDK reads fixes where the runtime plans them — a new server-side member on an op must not break parsing here. An unknown `kind` fails the whole verdict parse, deliberately and consistently with the closed `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break for no consumer benefit. -- **Paged listing: `iterate_methods`, `iterate_runs`, and `get_run_detail`.** `iterate_methods(q=…, limit=…)` and `iterate_runs(method_id, created_from=…, created_to=…, limit=…)` are async generators that follow the cursors, and they are the idiom to copy: there is deliberately no `list_all_methods()`, because an all-at-once helper needs a cap and a cap is the silent truncation paging removed. They stop on different signals, and the difference is in the server rather than the client — `iterate_methods` keeps going through an empty page that carries a live cursor, since `q` is a post-read filter over a bounded index slice per request, while `iterate_runs` treats an empty page as the end, since the run date bounds are index key conditions. Both stop when the server hands back the cursor it was sent, checked before yielding so a stuck cursor never double-counts a page. Past a runaway backstop set far beyond any real catalog, `iterate_methods` raises the new `PagingNotTerminatingError` rather than returning what it has. `get_run_detail(run_id)` reads one run through `GET /v1/runs/{id}` — distinct from the lifecycle `get_run_status` (`/status`) and `get_run_result` (`/results`), and the only read carrying `mthds_contents` and `inputs`, which is what the run actually executed and the only record of it. +- **Paged listing: `iterate_methods`, `iterate_runs`, and `get_run_detail`.** `iterate_methods(q=…, limit=…)` and `iterate_runs(method_id, created_from=…, created_to=…, limit=…)` are async generators that follow the cursors, and they are the idiom to copy: there is deliberately no `list_all_methods()`, because an all-at-once helper needs a cap and a cap is the silent truncation paging removed. They stop on different signals, and the difference is in the server rather than the client — `iterate_methods` keeps going through an empty page that carries a live cursor, since `q` is a post-read filter over a bounded index slice per request, while `iterate_runs` treats an empty page as the end, since the run date bounds are index key conditions. Both stop when the server hands back the cursor it was sent, checked before yielding so a stuck cursor never double-counts a page. Both also carry a runaway page backstop set far beyond any real catalog, raising the new `PagingNotTerminatingError` rather than returning what they have: the two stop rules above only catch a cursor that repeats *adjacently* or a server returning nothing, so a cursor cycling across several values over non-empty pages would otherwise loop forever re-yielding the same rows. `get_run_detail(run_id)` reads one run through `GET /v1/runs/{id}` — distinct from the lifecycle `get_run_status` (`/status`) and `get_run_result` (`/results`), and the only read carrying `mthds_contents` and `inputs`, which is what the run actually executed and the only record of it. - **New product models for the paged surface.** `MethodPage` / `MethodSummary`, `RunPage` / `RunDetail` / `RunErrorReport`, and `MethodFile` with the `parse_method_files` / `serialize_method_files` pair. A `MethodSummary` is deliberately not a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none of them is in the index projection, and putting `mthds` back is what restored the truncation bug. `MethodData` also gains `org_id`, `created_by_user_id`, `description`, `deletion_state`, and a typed `python: list[MethodFile]`; `MethodWriteInput` gains `python: list[MethodFile] | None`. diff --git a/TODOS.md b/TODOS.md index 91cefc5..98fc113 100644 --- a/TODOS.md +++ b/TODOS.md @@ -118,7 +118,7 @@ Breaking, and the most urgent fix in this plan: `list_methods` and `list_runs` c - [x] `list_methods(self, *, q: str | None = None, limit: int | None = None, cursor: str | None = None) -> MethodPage` — `GET /v1/methods` with the query built by the helper; parse `MethodPage`. Docstring: `q` is a server-side case-insensitive substring match over name and description across the whole catalog; `limit` defaults to and is capped by the API; ordering is by creation, newest first. - [x] `iterate_methods(self, *, q: str | None = None, limit: int | None = None) -> AsyncIterator[MethodSummary]` as an `async def` generator: request a page; **before yielding**, stop if `cursor is not None and page.next_cursor == cursor` (the server did not advance; yielding first would double-count); yield every item; stop when `page.next_cursor is None`; otherwise count the page and **raise** the new error once the count reaches `_MAX_LIST_PAGES`; continue **through empty pages** with a live cursor, because `q` is a post-read filter over a bounded index slice per request and `{items: [], next_cursor: "…"}` means "keep going". Docstring says why there is no `list_all_methods()`: an all-at-once helper needs a cap, and a cap is the silent truncation paging removed. - [x] `list_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None, cursor: str | None = None) -> RunPage` — `GET /v1/runs?method_id=…` plus the presence-kept query; parse `RunPage`. Docstring: `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, key conditions rather than filters; a bare date or naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also document the gate the JS mirror does not spell out: every `/v1/runs*` product route sits behind the platform's `require_surface_access()`, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — a `403` here means "flag", not "wrong key". -- [x] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing. +- [x] `iterate_runs(self, method_id: str, *, created_from: str | None = None, created_to: str | None = None, limit: int | None = None) -> AsyncIterator[PipelineRun]` — same loop, except an **empty page ends it** (date bounds are index key conditions, so a run page is never empty-with-a-cursor; the difference is the server, not the client — say so in the docstring). ~~No page ceiling needed: the empty-page stop already catches a server minting fresh cursors while returning nothing.~~ **Deviation, taken on PR review:** the ceiling applies here too. The plan's reason was too narrow — the empty-page stop only catches a server returning *nothing*, so a cursor cycling across two or more values (`c1 → c2 → c1`) over non-empty pages trips neither it nor the adjacent-cursor check and would loop forever re-yielding the same runs. Greptile and Codex flagged it independently. The fix reuses `_MAX_LIST_PAGES` and `PagingNotTerminatingError` rather than tracking every cursor seen, which would cost unbounded memory for the same protection. - [x] `get_run_detail(self, run_id: str) -> RunDetail` — `GET /v1/runs/{id}` via `_request_product` with the id path-encoded like the other id routes (`f"{_RUNS}/{quote(run_id, safe='')}"`). Distinct from `get_run_status` (`/status`) and `get_run_result` (`/results`). - [x] Update the `PipelexAPIClient` class docstring's product-surface bullet and the import block (`MethodPage`, `MethodSummary`, `RunDetail`, `RunPage`, `AsyncIterator` from `collections.abc` under `TYPE_CHECKING` if only used in annotations — it is used at runtime as a return annotation with `from __future__ import annotations`, so `TYPE_CHECKING` is fine). diff --git a/docs/architecture.md b/docs/architecture.md index b8db7f4..3eaae41 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -171,7 +171,7 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( **This list is paged too.** `list_runs(method_id, created_from=…, created_to=…, limit=…, cursor=…)` returns a `RunPage` with the same opaque-cursor contract as `MethodPage`. `created_from` / `created_to` are **instants** — ISO-8601 with a UTC offset — and inclusive; they are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Worth knowing and not obvious from the route: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed with a `403` — so a `403` here means "flag", not "wrong key". - **The two iterators stop on different signals, and the difference is in the server.** `iterate_methods` continues through an empty page with a live cursor; `iterate_runs` treats an **empty page as the end**, because the run date bounds are index key conditions and so a run page is never empty-with-a-cursor. That stop rule also removes the need for a page ceiling on the run side — a server minting fresh cursors while returning nothing is already caught by it. + **The two iterators stop on different signals, and the difference is in the server.** `iterate_methods` continues through an empty page with a live cursor; `iterate_runs` treats an **empty page as the end**, because the run date bounds are index key conditions and so a run page is never empty-with-a-cursor. Both share the same runaway page ceiling, and both raise `PagingNotTerminatingError` at it: the empty-page stop only catches a server minting fresh cursors while returning *nothing*, so a cursor that cycles across two or more values (`c1 → c2 → c1`) over non-empty pages trips neither that check nor the adjacent-cursor one. The ceiling is the cheap guard against that whole family — tracking every cursor seen would cost unbounded memory for the same protection. **`PipelineRun` fields the platform genuinely serves as null are typed nullable.** `method_id` is `None` for an ad-hoc run from an inline bundle, which belongs to no stored method; `pipe_code` is `None` for a run that let the bundle's `main_pipe` decide. `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None` (`message`, `error_type` — the two fields a consumer may rely on out of the runner's verbose report) join them. `RunDetail`, returned only by `get_run_detail`, adds `mthds_contents` and `inputs`: what the run actually executed, and the only record of it, since a method edited since the run no longer describes what happened. Both are left out of the list and the polled status on purpose — their cost scales with page size and poll rate respectively. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 2b181cf..a14cb87 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -114,9 +114,10 @@ _POLL_REQUEST_TIMEOUT_SECONDS = 30.0 # single status/result/product GETs; the hosted gateway caps responses at ~30s. -# A runaway backstop for the paged-list iterators, set far beyond any real catalog — never a +# A runaway backstop for both paged-list iterators, set far beyond any real catalog — never a # coverage cap. Reaching it means the server kept minting cursors, which is a fault to raise on, -# not a limit to truncate at. +# not a limit to truncate at. It is what bounds a cursor that CYCLES (`c1 -> c2 -> c1`) over +# non-empty pages: neither iterator's adjacent-cursor check sees a non-adjacent repeat. _MAX_LIST_PAGES: int = 10_000 _DEFAULT_DEGRADED_RETRY_SECONDS = 5 # matches the platform's `_DEGRADE_RETRY_AFTER_SECONDS`. @@ -1065,10 +1066,19 @@ async def iterate_runs( The same loop as `iterate_methods` with one deliberate difference: an **empty page ends it**. The date bounds are index key conditions rather than a post-read filter, so a run page is never empty-with-a-cursor. The difference is in the server, not in the client. - That stop rule also removes the need for a page ceiling here — a server minting fresh - cursors while returning nothing is caught by it. + + The page ceiling applies here too. The empty-page stop only catches a server minting + fresh cursors while returning *nothing*; a cursor that cycles across two or more values + while every page is non-empty (`c1 → c2 → c1`) trips neither that check nor the + adjacent-cursor one, and would loop forever re-yielding the same runs. The ceiling is + the cheap guard against the whole family — tracking every cursor seen would cost + unbounded memory for the same protection. + + Raises: + PagingNotTerminatingError: If the server never stops handing out cursors. """ cursor: str | None = None + pages_seen = 0 while True: page = await self.list_runs(method_id, created_from=created_from, created_to=created_to, limit=limit, cursor=cursor) if cursor is not None and page.next_cursor == cursor: @@ -1080,6 +1090,10 @@ async def iterate_runs( yield pipeline_run if page.next_cursor is None: return + pages_seen += 1 + if pages_seen >= _MAX_LIST_PAGES: + msg = f"Run paging did not terminate after {_MAX_LIST_PAGES} pages; this is a server-side fault, not a coverage limit." + raise PagingNotTerminatingError(msg, _MAX_LIST_PAGES) cursor = page.next_cursor async def get_run_detail(self, run_id: str) -> RunDetail: diff --git a/tests/unit/test_client_paging.py b/tests/unit/test_client_paging.py index 7ebc3fa..48b53f5 100644 --- a/tests/unit/test_client_paging.py +++ b/tests/unit/test_client_paging.py @@ -158,6 +158,33 @@ def test_iterate_runs_follows_the_cursor_and_stops_on_none( assert [pipeline_run.pipeline_run_id for pipeline_run in runs] == ["r1", "r2"] assert _cursors_sent(send) == [None, "c1"] + def test_iterate_runs_raises_on_a_cursor_cycle_over_non_empty_pages( + self, + api_client: PipelexAPIClient, + wire_response: ResponseBuilder, + patch_send: SendPatcher, + mocker: MockerFixture, + ) -> None: + """A cursor cycling across two values trips neither the empty-page nor the adjacent check. + + `c1 → c2 → c1 → …` with every page non-empty would loop forever re-yielding the same + runs; the shared page ceiling is what bounds it. + """ + mocker.patch("pipelex_sdk.client._MAX_LIST_PAGES", 4) + cycle = [ + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + wire_response(200, json_body={"items": [_run("r2")], "next_cursor": "c2"}), + wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), + ] + patch_send(api_client, *cycle) + + with pytest.raises(PagingNotTerminatingError) as exc_info: + asyncio.run(_drain_runs(api_client, "m1")) + + assert exc_info.value.page_limit == 4 + def test_iterate_runs_stops_on_an_unchanged_cursor_without_re_yielding( self, api_client: PipelexAPIClient, From 3ac68acb9ce5f78c396edf26f865b016b7b07105 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 10:11:42 +0200 Subject: [PATCH 15/20] Finish the Phase 2 citation sweep and correct two stale doc counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three sites the plan named or implied were left out of sync, all found on PR review and all prose: - `docs/architecture.md`'s `TokensUsageRecord` bullet still attributed the record to the MTHDS protocol spec. Phase 2 named that site and the changelog asserts it was corrected, but the commit only reached the `method_id` citation in that file. It now carries the same wording as `runs.py` and `docs/run-usage.md`: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API pins the wire contract. - `CHANGELOG.md` still cited `docs/specs/pipelex-platform-api.md`, a workspace path that resolves to nothing for anyone who clones this public package — the exact pattern the entry a few lines below claims to have removed. The sentence now states the rule and points at the sibling public JS SDK. - The `validate` override section of `docs/architecture.md` still introduced "two Pipelex-API extensions" after Phase 1 added the third; the client docstring already said three. `TODOS.md` records these alongside the review findings that were judged not worth acting on, with the reason for each. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 2 +- TODOS.md | 17 +++++++++++++++++ docs/architecture.md | 4 ++-- 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1962ad3..1e97cf8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ On the wire `python` is one string — the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python — and the converter lives here rather than in `mthds`, because the catalog serialization is a Pipelex product concern and there is no second Python consumer to keep in step. `MethodData` applies the parser through a `field_validator(mode="before")`, so `model_validate` keeps working at every call site; `MethodWriteInput` applies the serializer through a `field_serializer`, which makes the platform's three-way write contract fall out of `exclude_none=True`: `None` leaves the key out and preserves the stored Python, `[]` sends the `""` clear sentinel, a non-empty list replaces. Malformed wire text raises inside the validator and surfaces as a `pydantic.ValidationError`, the same way any other malformed response body fails here. -- **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. The doctrine is normative in the workspace spec `docs/specs/pipelex-platform-api.md` → "Layered extension policy". +- **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. It is the same layered extension policy the TypeScript `@pipelex/sdk` follows, so both SDKs partition their arguments the same way. ### Changed diff --git a/TODOS.md b/TODOS.md index 98fc113..d2b016f 100644 --- a/TODOS.md +++ b/TODOS.md @@ -191,6 +191,23 @@ Deferred out of the plan, with the reason: - **Migrating `test_client_product.py` onto the new `tests/unit/conftest.py` fixtures** was explicitly optional in §3.4 and was not done; the module keeps its own equivalent private helpers. - **`pipelex_sdk/runs.py`'s "Wire contract mirrors `pipelex-platform`" line** was left alone. It names a service, not a file path, so it is not one of the unopenable citations Phase 2 was about, and widening that phase's scope on my own judgement was not warranted. +### Corrections found on PR review + +Cubic's pass on `fa8d62a` reported findings against files this plan touched. Three were real and are fixed; the rest were judged not worth acting on, and the reasons are recorded here rather than only in the resolved GitHub threads. + +Fixed: + +- **The `TokensUsageRecord` attribution in `docs/architecture.md` was missed.** Phase 2's first checkbox names that site explicitly, and `CHANGELOG.md` asserts it was corrected, but `2a8c589` only touched the `method_id` citation in that file. `pipelex_sdk/runs.py` and `docs/run-usage.md` were correct. The bullet now carries the same wording as the other two. +- **`CHANGELOG.md` still cited `docs/specs/pipelex-platform-api.md`.** The changelog was outside Phase 2's enumerated citation sites, so the sweep did not reach it — leaving the `[Unreleased]` section claiming "no more citations a reader cannot open" a few entries below a citation a reader cannot open. The sentence now states the rule and points at the sibling public JS SDK instead. +- **The `validate` override intro in `docs/architecture.md` still said "two Pipelex-API extensions"** after Phase 1 added the third. The client docstring was updated at the time; the doc's intro sentence was not. + +Not acted on, with the reason: + +- **Hoisting the `method_id` type guard above `start_and_wait`'s lifecycle handshake.** The claim that a handshake failure can mask the guard is wrong: `_supports_run_lifecycle` swallows its errors and every downstream path still reaches `_merge_hosted_run_extensions`, so no wrong-typed value escapes. The real cost is one probe request, memoized for the client's lifetime. Hoisting would mean either calling the merge helper twice or duplicating the check outside the single documented guard site. +- **A claimed 194-character line in `tests/unit/test_client_validate.py`.** Measured at 127, under the configured 150, and no line in that file is longer; `ruff check` passes. The measurement was simply wrong. +- **Rewriting Checkpoint 2's parenthetical to match the two divergences the note beneath it records.** The parenthetical is the expectation the plan set out with, and the note is the finding; editing the prediction to match the outcome removes the only evidence that the plan's expectation was slightly off. +- **Replacing the two `# type: ignore[arg-type]` comments in `tests/unit/test_client_method_id.py` with `cast()`.** A cast would assert to the reader that the value *is* a `str | None`, which is the exact falsehood that test exists to disprove at runtime; the ignore states the truth that this is a deliberate type error, which is the last resort the coding standard permits. + ## Known gaps left open on purpose - No e2e suite exists in this repo (`tests/` holds only `unit/`), so the live `views` gate and the live paging envelope are pinned only by mocked bodies here; the JS suite pins both live. Adding an e2e suite is separate work. diff --git a/docs/architecture.md b/docs/architecture.md index 3eaae41..be7701b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -101,7 +101,7 @@ The durable run lifecycle (`pipelex_sdk/runs.py` + the client's lifecycle method - `RunStatus` — the hosted status enum, with `is_terminal` / `is_success` predicates (exhaustive `match`). - `RunRead` — a run record read through the self-healing status path (adds `degraded` + `retry_after_seconds`). - `RunResults` — result artifacts. `main_stuff` (the resolved main output content) is always present for a completed run: on the hosted path it is the `main_stuff.json` S3 artifact; on the bare-runner blocking path the SDK resolves it from the returned working memory via the response's `main_stuff_name`, so both paths deliver the same shape. Consumers read `main_stuff` directly. The full working memory still rides `pipe_output` (blocking path only) for consumers that want it, and `graph_spec` rides the hosted path. A completed run that cannot deliver a main stuff raises `MissingMainStuffError`. Extension-open, so any other server artifact is preserved. -- `TokensUsageRecord` — one client-facing usage record per inference call, carried by `RunResults.tokens_usages` on both paths. A mirror of the wire contract specified in the MTHDS protocol spec, not a shape this SDK owns: every field is optional and the model is extension-open so pre-contract artifacts (relayed verbatim, never migrated) still parse. See [`run-usage.md`](./run-usage.md) for the field reference, the cost/null semantics, and the old-artifact rules. +- `TokensUsageRecord` — one client-facing usage record per inference call, carried by `RunResults.tokens_usages` on both paths. A mirror of the runtime's own record, not a shape this SDK owns — inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins that wire contract: every field is optional and the model is extension-open so pre-contract artifacts (relayed verbatim, never migrated) still parse. See [`run-usage.md`](./run-usage.md) for the field reference, the cost/null semantics, and the old-artifact rules. - `RunResultState` — the single-shot result outcome, a union discriminated on `state` (`running` / `completed` / `failed`). - `WaitForResultOptions` / `PollInfo` — poll-loop tuning and progress info. Async-native cancellation is via `asyncio.CancelledError` (cancel the awaiting task), so there is no `signal` field. @@ -128,7 +128,7 @@ These poll GETs go through `_send_or_unreachable`, so a transport failure surfac ## `validate` override (Pipelex-API presentation + sources) -The protocol `validate` is **overridden** (not inherited) to add the two Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): +The protocol `validate` is **overridden** (not inherited) to add the three Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): - **Markdown render is always injected.** `validate(...)` adds `"markdown"` to the `render` list (de-duplicated, caller tokens first) so both a valid `PipelexValidationReport` and a produced `PipelexInvalidReport` carry `rendered_markdown`. Unknown render tokens are server-side lenient-ignored. - **`mthds_sources`** is a named parameter (parallel to `mthds_contents`) threaded onto each diagnostic's `source`; sent only when provided. From 301d96ee8f864427f3f08861e20ac5169fb1af1d Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 10:19:28 +0200 Subject: [PATCH 16/20] Record why the tracker keeps its workspace inbox reference Cubic read Phase 2's citation rule as covering this tracker. It does not: the rule is about the shipped surface, where a workspace path resolves to nothing for anyone who installs the package. A tracker read by this PR's reviewers is a different document, it cites the workspace throughout by design, and the sentence in question already states its rationale before citing the item. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- TODOS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TODOS.md b/TODOS.md index d2b016f..db5f9a7 100644 --- a/TODOS.md +++ b/TODOS.md @@ -206,6 +206,7 @@ Not acted on, with the reason: - **Hoisting the `method_id` type guard above `start_and_wait`'s lifecycle handshake.** The claim that a handshake failure can mask the guard is wrong: `_supports_run_lifecycle` swallows its errors and every downstream path still reaches `_merge_hosted_run_extensions`, so no wrong-typed value escapes. The real cost is one probe request, memoized for the client's lifetime. Hoisting would mean either calling the merge helper twice or duplicating the check outside the single documented guard site. - **A claimed 194-character line in `tests/unit/test_client_validate.py`.** Measured at 127, under the configured 150, and no line in that file is longer; `ruff check` passes. The measurement was simply wrong. - **Rewriting Checkpoint 2's parenthetical to match the two divergences the note beneath it records.** The parenthetical is the expectation the plan set out with, and the note is the finding; editing the prediction to match the outcome removes the only evidence that the plan's expectation was slightly off. +- **The `../wip/inbox/…` reference in §1.3.** Phase 2's rule is about the *shipped* surface — docstrings, comments and doc pages that travel to PyPI, where a workspace path resolves to nothing and reads as rot. This tracker is not that: it is a working document for the reviewers of this PR, it is written from workspace context throughout (it cites `wip/updates.md` and the JS repo's own `wip/` in the same way), and `../wip/inbox/` is the notation the workspace guide itself prescribes for a sub-repo checkout. The path resolves where the document is read, and the sentence already states its rationale inline before citing the item, so a reader who cannot open it loses only the provenance. - **Replacing the two `# type: ignore[arg-type]` comments in `tests/unit/test_client_method_id.py` with `cast()`.** A cast would assert to the reader that the value *is* a `str | None`, which is the exact falsehood that test exists to disprove at runtime; the ignore states the truth that this is a deliberate type error, which is the last resort the coding standard permits. ## Known gaps left open on purpose From cb391c96cce03464464e4aadbf6bbb97a076e5e9 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 11:11:21 +0200 Subject: [PATCH 17/20] Fix a broken README import, two toothless tests, and two stale counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-landing review findings, all mechanical and none touching behaviour. The README's "Public import paths" section told readers to reach PipelexValidationResult through mthds.runners.api.models, which raises ImportError — that name is not in the installed mthds at all. The Pipelex narrowing of the verdict union is owned here, exactly as the "Brand boundary" section of docs/architecture.md says, so the repo contradicted itself. The section now carries a pipelex_sdk.validation_models bullet and cites mthds.protocol.models.ValidationResult as the neutral union it narrows. The two page-ceiling tests asserted only PagingNotTerminatingError.page_limit, which echoes back the constant the test itself patched — so five scripted responses against a limit of two passed whether the raise fired on the first page or the last. Each now also asserts the call count, mutation-checked by loosening the comparison in client.py: both tests fail where they previously passed. Two hardcoded counts ("three Pipelex-API ways/extensions") had already gone stale once inside this branch, and the Raises: entries on execute and start named only the extra-smuggling trigger, not the non-string method_id guard this branch added. The deferred findings are recorded in wip/pr-14-review-notes.md, and the one that reaches other repos — pipe_statuses, a field no server fills, declared in this SDK, @pipelex/sdk and pipelex-app — is filed in the workspace inbox. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 2 + README.md | 5 ++- TODOS.md | 17 +++++++++ docs/architecture.md | 2 +- pipelex_sdk/client.py | 8 ++-- tests/unit/test_client_paging.py | 8 +++- wip/pr-14-review-notes.md | 63 ++++++++++++++++++++++++++++++++ 7 files changed, 97 insertions(+), 8 deletions(-) create mode 100644 wip/pr-14-review-notes.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e97cf8..1ffd4d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,8 @@ - **Documentation: `TokensUsageRecord` is attributed to the right brand.** The module and class docstrings in `pipelex_sdk/runs.py`, `docs/run-usage.md`, and `docs/architecture.md` said the record was "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins the wire contract and this SDK mirrors the runtime's own record. Getting the attribution wrong on a public package misplaces the brand boundary the rest of the architecture is organized around. No behaviour changed. - **Documentation: no more citations a reader cannot open.** Several docstrings, comments and doc passages cited internal specs and the conformance suite by bare repo-relative path — paths that resolve to nothing for anyone who clones this public package, so they read as rot rather than as a deliberate boundary. Each site now states the rule it was citing instead: the layered extension policy behind the hosted run extensions and the reserved-`extra` guard, and the locked category vocabulary shared with the conformance corpus. References to the sibling public `pipelex-sdk-js` repo are kept where they explain a port. No behaviour changed. +- **Documentation: the README's import map pointed at a name that does not exist.** "Public import paths" told readers to reach `PipelexValidationResult` through `from mthds.runners.api.models import PipelexValidationResult` — an `ImportError`, since the Pipelex narrowing of the verdict union is owned by this package (`pipelex_sdk.validation_models`), exactly as the "Brand boundary" section of `docs/architecture.md` states. The section now carries a bullet for `pipelex_sdk.validation_models` and cites `mthds.protocol.models.ValidationResult` as the neutral union it narrows. The typed-errors bullet gains `PagingNotTerminatingError` and, like the product-models bullet, is marked non-exhaustive rather than reading as a complete list it never was. + ## [v0.5.0] - 2026-07-22 ### Added diff --git a/README.md b/README.md index 4b96381..c5ef348 100644 --- a/README.md +++ b/README.md @@ -88,9 +88,10 @@ There is no barrel import — package `__init__.py` files stay empty. Import eac - **Client & construction** — `from pipelex_sdk.client import PipelexAPIClient, DEFAULT_API_BASE_URL, MthdsFile` - **Run lifecycle types** — `from pipelex_sdk.runs import RunStatus, RunPublic, RunRead, RunResults, RunResultState, WaitForResultOptions, PollInfo` - **Product wire models** — `from pipelex_sdk.product_models import UserProfile, MethodData, MethodWriteInput, Membership, MembershipsResponse, SubscriptionResponse, PlanView, InvoiceView, OnboardingSubmission, UploadInput, UploadedFile, PipelineRun, ...` -- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError` +- **Validation verdict types** — `from pipelex_sdk.validation_models import PipelexValidationResult, PipelexValidationReport, PipelexInvalidReport, ValidationErrorItem, SuggestedFix, VALIDATION_VIEW_INPUT_FORM, ...` +- **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, ...` - **Version** — `from pipelex_sdk.version import __version__` -- **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.runners.api.models import PipelexValidationResult`. +- **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.protocol.models import ValidationResult` (the neutral verdict union that `PipelexValidationResult` narrows). ## Development diff --git a/TODOS.md b/TODOS.md index db5f9a7..fe196ae 100644 --- a/TODOS.md +++ b/TODOS.md @@ -209,6 +209,23 @@ Not acted on, with the reason: - **The `../wip/inbox/…` reference in §1.3.** Phase 2's rule is about the *shipped* surface — docstrings, comments and doc pages that travel to PyPI, where a workspace path resolves to nothing and reads as rot. This tracker is not that: it is a working document for the reviewers of this PR, it is written from workspace context throughout (it cites `wip/updates.md` and the JS repo's own `wip/` in the same way), and `../wip/inbox/` is the notation the workspace guide itself prescribes for a sub-repo checkout. The path resolves where the document is read, and the sentence already states its rationale inline before citing the item, so a reader who cannot open it loses only the provenance. - **Replacing the two `# type: ignore[arg-type]` comments in `tests/unit/test_client_method_id.py` with `cast()`.** A cast would assert to the reader that the value *is* a `str | None`, which is the exact falsehood that test exists to disprove at runtime; the ignore states the truth that this is a deliberate type error, which is the last resort the coding standard permits. +### Final pre-landing review + +Once the three PR bots reported clean on `301d96e`, a fresh reviewer with no context from this session ran the `gstack` review procedure against this plan, with an adversarial pass folded in. It found no correctness defect in the shipped code, and it verified the wire contracts against `pipelex-server` itself rather than against the docstrings that assert them — including that the two iterators' asymmetric stop rules match the two DynamoDB adapters (the method adapter can mint a cursor for an empty page because `q` filters after the read, the run adapter over-fetches by one and cannot), and that the pydantic three-way write contract for `python` holds for absent / `null` / `""` / `"[]"` alike. + +Fixed: + +- **`README.md` advertised an import that raises `ImportError`.** "Public import paths" sent readers to `from mthds.runners.api.models import PipelexValidationResult`; that name is not in the installed `mthds` at all. The Pipelex narrowing of the verdict union is owned here, which `docs/architecture.md` states in its "Brand boundary" section — so the repo contradicted itself. The section gains a `pipelex_sdk.validation_models` bullet and now cites `mthds.protocol.models.ValidationResult` as the neutral union it narrows. Pre-existing, but this branch edited the quickstart three lines above it. +- **The two page-ceiling tests could not fail.** Their only assertion was `exc_info.value.page_limit`, which just echoes back the constant the test itself patched, so five scripted responses against a limit of two passed whether the raise fired on page 1 or page 5. Each now also asserts `send.call_count`, which was mutation-checked: loosening `pages_seen >= _MAX_LIST_PAGES` to `>` fails both tests where it previously passed. +- **Two hardcoded counts that had already gone stale inside this branch.** `pipelex_sdk/client.py`'s `validate` docstring and `docs/architecture.md` both said "three Pipelex-API ways/extensions" — the same rot the entry above records having corrected once already, which is exactly why the workspace guide forbids writing counts down. Both now say neither two nor three. +- **`execute` and `start` under-documented an exception this branch added.** Their `Raises:` entries named only the `extra`-smuggling trigger; Phase 4's guard raises `PipelineRequestError` for a non-string `method_id` too. The helper's own docstring had been updated at the time, the two public methods' had not. + +Deferred to `wip/pr-14-review-notes.md`, each verified and none blocking: the sdist ships this tracker and the other internal planning documents (a packaging decision that belongs with a release, not inside a feature branch); the client class docstring still dates two surfaces by build-plan phase number; and `start_and_wait`'s `Raises:` omits the `PipelineRequestError` it propagates on both paths. + +One finding reached outside this repo and was filed rather than fixed: `PipelineRun.pipe_statuses` is a field no server has ever filled — it is absent from the platform's `RunPublic` response model and appears nowhere in `pipelex-server` — yet it is declared in this SDK, in `@pipelex/sdk`, and in `pipelex-app`, where run-history progress dots are gated on it and have therefore never rendered. Retiring it here alone would break the parity invariant this package is built on, so the decision belongs to whoever owns the run wire contract: `../wip/inbox/2026-08-25-workspace-pipe-statuses-dead-field-in-three-clients.md`. + +Considered and declined: guarding `iterate_methods` against a `next_cursor` of `""`, which would let a page double-yield. The adversarial pass demonstrated the mechanism and then confirmed the case is unreachable, since the platform's cursor is a base64 `LastEvaluatedKey`. That is the impossible-scenario defensiveness this plan set out to avoid. + ## Known gaps left open on purpose - No e2e suite exists in this repo (`tests/` holds only `unit/`), so the live `views` gate and the live paging envelope are pinned only by mocked bodies here; the JS suite pins both live. Adding an e2e suite is separate work. diff --git a/docs/architecture.md b/docs/architecture.md index be7701b..75bbc68 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -128,7 +128,7 @@ These poll GETs go through `_send_or_unreachable`, so a transport failure surfac ## `validate` override (Pipelex-API presentation + sources) -The protocol `validate` is **overridden** (not inherited) to add the three Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): +The protocol `validate` is **overridden** (not inherited) to add the Pipelex-API extensions the bare protocol route doesn't carry, while keeping the inherited protocol error regime (a no-verdict non-2xx surfaces as `httpx.HTTPStatusError`, not `ApiResponseError` — the verdict itself is always a 200 discriminated on `is_valid`): - **Markdown render is always injected.** `validate(...)` adds `"markdown"` to the `render` list (de-duplicated, caller tokens first) so both a valid `PipelexValidationReport` and a produced `PipelexInvalidReport` carry `rendered_markdown`. Unknown render tokens are server-side lenient-ignored. - **`mthds_sources`** is a named parameter (parallel to `mthds_contents`) threaded onto each diagnostic's `source`; sent only when provided. diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index a14cb87..11eafb2 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -392,7 +392,8 @@ async def execute( Raises: PipelineExecuteTimeoutError: The blocking request hit the hosted gateway's ~30s synchronous ceiling — use `start_and_wait` (or `start` + `wait_for_result`). - PipelineRequestError: `extra` carries a protocol arg or a hosted arg. + PipelineRequestError: `extra` carries a protocol arg or a hosted arg, or `method_id` + is present and is not a string. RunStillRunningError: The server answered 202 (the protocol's optional async degrade) — the run continues server-side; resume by `pipeline_run_id`. httpx.HTTPStatusError: Any other non-2xx response (the inherited regime). @@ -443,7 +444,8 @@ async def start( behavior. Raises: - PipelineRequestError: `extra` carries a protocol arg or a hosted arg. + PipelineRequestError: `extra` carries a protocol arg or a hosted arg, or `method_id` + is present and is not a string. RunLifecycleUnavailableError: The configured server has no run store. """ try: @@ -477,7 +479,7 @@ async def validate( # type: ignore[override] means no verdict could be produced (request shape, auth, server fault) and surfaces as `httpx.HTTPStatusError` (the inherited protocol error regime). - This override differs from the inherited protocol `validate` in three Pipelex-API ways: + This override differs from the inherited protocol `validate` in these Pipelex-API ways: it always injects `render: ["markdown"]` (so both valid and invalid verdicts carry `rendered_markdown`), it accepts `mthds_sources` as a named parameter, and it carries the `views` opt-in for the server's structured views. diff --git a/tests/unit/test_client_paging.py b/tests/unit/test_client_paging.py index 48b53f5..9955141 100644 --- a/tests/unit/test_client_paging.py +++ b/tests/unit/test_client_paging.py @@ -116,12 +116,15 @@ def test_iterate_methods_raises_past_the_page_ceiling( """The backstop raises rather than returning: a truncated list is the bug paging removed.""" mocker.patch("pipelex_sdk.client._MAX_LIST_PAGES", 2) pages = [wire_response(200, json_body={"items": [_method(f"m{index}")], "next_cursor": f"c{index}"}) for index in range(5)] - patch_send(api_client, *pages) + send = patch_send(api_client, *pages) with pytest.raises(PagingNotTerminatingError) as exc_info: asyncio.run(_drain_methods(api_client)) assert exc_info.value.page_limit == 2 + # The raise lands ON the ceiling, not one page past it — otherwise a loosened + # comparison would still satisfy `page_limit`, which only echoes the constant. + assert send.call_count == 2 def test_iterate_runs_stops_on_an_empty_page( self, @@ -178,12 +181,13 @@ def test_iterate_runs_raises_on_a_cursor_cycle_over_non_empty_pages( wire_response(200, json_body={"items": [_run("r2")], "next_cursor": "c2"}), wire_response(200, json_body={"items": [_run("r1")], "next_cursor": "c1"}), ] - patch_send(api_client, *cycle) + send = patch_send(api_client, *cycle) with pytest.raises(PagingNotTerminatingError) as exc_info: asyncio.run(_drain_runs(api_client, "m1")) assert exc_info.value.page_limit == 4 + assert send.call_count == 4 def test_iterate_runs_stops_on_an_unchanged_cursor_without_re_yielding( self, diff --git a/wip/pr-14-review-notes.md b/wip/pr-14-review-notes.md new file mode 100644 index 0000000..99599ba --- /dev/null +++ b/wip/pr-14-review-notes.md @@ -0,0 +1,63 @@ +# PR #14 — deferred review findings + +Findings from the pre-landing review of [PR #14](https://github.com/Pipelex/pipelex-sdk-python/pull/14) (`feature/Typed-method-id-run-option`, reviewed at `301d96e`). Each item below was verified against the code and, where the claim was about the built artifact, against a locally built sdist. None of them blocks landing the branch; each is deferred because acting on it reaches outside what this branch set out to change. + +The findings that *were* acted on during the review are recorded in `CHANGELOG.md` under `[Unreleased]`, not here. + +--- + +## 1. The published sdist carries the repo's internal planning documents + +**Status:** Confirmed, pre-existing, widened by this branch. Deferred because the fix is a packaging change to `pyproject.toml`, which this branch does not touch and which has release implications worth deciding on their own. + +`pyproject.toml` declares `build-backend = "hatchling.build"` (`pyproject.toml:40`) and carries no `[tool.hatch.build.targets.sdist]` section, so hatchling falls back to including everything the VCS does not ignore. Building the current tree confirms it: + +``` +$ uv build --sdist +$ tar -tzf dist/pipelex_sdk-0.5.0.tar.gz +pipelex_sdk-0.5.0/CLAUDE.md +pipelex_sdk-0.5.0/TODOS.md +pipelex_sdk-0.5.0/Makefile +pipelex_sdk-0.5.0/uv.lock +pipelex_sdk-0.5.0/docs/HANDOFF.md +pipelex_sdk-0.5.0/wip/pr-11-review-notes.md +pipelex_sdk-0.5.0/wip/updates.md +pipelex_sdk-0.5.0/tests/... +``` + +`CLAUDE.md`, `wip/pr-11-review-notes.md`, `docs/HANDOFF.md`, the `Makefile` and the whole `tests/` tree already shipped this way before the branch, so this is not a regression it introduced. What the branch adds is `TODOS.md` and `wip/updates.md` — the tracker and the design — which together are a substantial share of the archive and are addressed to reviewers of this PR rather than to anyone installing the package. `docs/HANDOFF.md` is a related case already on PyPI: it describes creating this repo from scratch and reads as rot to anyone who finds it in a release. + +Nothing breaks — an sdist is not what `pip install` normally consumes, and none of these files is importable — so this is about what a public package says about itself, not about correctness. + +**If picked up:** declare an explicit sdist include list (or an exclude list covering `wip/`, `TODOS.md`, `CLAUDE.md`, `Makefile` and `docs/HANDOFF.md`), decide deliberately whether `tests/` should stay (some consumers value a testable sdist), and land it with a release rather than inside a feature branch. + +## 2. The client class docstring dates its surfaces by build-plan phase + +**Status:** Confirmed, pre-existing. Deferred because Phase 2 of this branch scoped its citation sweep to bare workspace-private *paths*, and widening that scope mid-branch was a judgement call the tracker declined elsewhere for the same reason. + +`pipelex_sdk/client.py:176` and `pipelex_sdk/client.py:178` describe the run lifecycle as "(added in Phase 2)" and the product surface as "(added in Phase 3)". Those phase numbers refer to the original build plan for this package. They travel to PyPI in the class docstring of the one class every consumer instantiates, where they resolve to nothing — the same failure mode as the repo-relative spec paths Phase 2 replaced, in a different spelling. + +**If picked up:** replace each marker with what the reader actually needs (the release the surface shipped in, or nothing at all), and sweep for the same pattern elsewhere in the shipped modules. + +## 3. `start_and_wait` documents fewer exceptions than it propagates + +**Status:** Confirmed, pre-existing. Deferred as too small to justify widening this branch's diff. + +`pipelex_sdk/client.py` documents `Raises: RunFailedError` and `RunTimeoutError` on `start_and_wait`, but the method reaches `_merge_hosted_run_extensions` on both of its paths — the durable one through `start` and the fallback through `_execute_blocking` → `execute` — so it also propagates `PipelineRequestError` for a reserved key on `extra` and, since this branch, for a non-string `method_id`. The `Raises:` sections of `execute` and `start` were corrected during this review; `start_and_wait` was left alone because its omission predates the branch and is not about anything the branch changed. + +**If picked up:** add the `PipelineRequestError` line to `start_and_wait`, and while there check `wait_for_result` and the product methods for the same drift. + +## 4. `PipelineRun.pipe_statuses` is a contract no server fills, in three repos at once + +**Status:** Confirmed, pre-existing, and the most consequential item here. Deferred because it cannot be resolved inside this repo: removing the field locally would break the parity invariant this package is built on, and the decision belongs to whoever owns the run wire contract. + +`pipelex_sdk/product_models.py` declares `pipe_statuses: dict[str, PipeStatus] | None = None` on `PipelineRun`, with `PipeStatus` as its supporting enum. The platform never sends it. `RunPublic` — the model FastAPI serializes `GET /v1/runs` and `GET /v1/runs/{id}` through — declares no such field (`pipelex-server/shared/src/pipelex_shared/schemas/run.py:180`), and a response model strips whatever it does not declare. A `grep -rn "pipe_statuses"` over the entire `pipelex-server` monorepo returns nothing at all, so no route, worker or Lambda writes it either. The field therefore reads `None` on every run this SDK will ever parse, and a consumer branching on it gets a silently empty answer rather than an error. + +The same dead field exists in the two sibling repos, which is what makes it a workspace question rather than a local cleanup: + +- `pipelex-sdk-js/src/product-models.ts:322` — `pipe_statuses?: Record | null;`, with the enum at `:291`. This SDK is a port of that one, so dropping the field here alone would introduce exactly the parity gap `docs/architecture.md` → "Parity with `@pipelex/sdk`" exists to prevent. +- `pipelex-app/src/types/run.ts:24` — the same declaration, and `pipelex-app/src/components/method/run-history-list.tsx:269` renders a row of per-pipe progress dots gated on `{run.pipe_statuses && ...}`. Because the platform never sends it, that guard is always false and those dots have never appeared. Whether that is a missing feature or an abandoned one is the question to settle. + +So there are two coherent outcomes and this branch is the wrong place to choose between them: either the platform starts projecting per-pipe status onto `RunPublic` (and the webapp's dots light up), or the field is retired from all three clients together. + +**Filed:** `../wip/inbox/2026-08-25-workspace-pipe-statuses-dead-field-in-three-clients.md` (`to: workspace`, naming `pipelex-server/platform`, `pipelex-sdk-js` and `pipelex-app`). Whichever way the decision goes, this SDK follows the JS SDK; it should not move first. From 1a967c0f968aa9c160bc6ca7d7e4d5a8b5e209e5 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 11:19:10 +0200 Subject: [PATCH 18/20] =?UTF-8?q?Stop=20documenting=20a=20build=5Finputs(m?= =?UTF-8?q?ethod=5Fid=3D=E2=80=A6)=20that=20does=20not=20exist?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hosted-run-extensions section of docs/architecture.md contrasted the method_id run option with "build_inputs(method_id=…) and prepare_inputs, which are client-side sugar that resolves an id to inline files". Neither helper accepts a method_id: build_inputs takes a BuildInputsRequest of files / pipe_ref / format / explicit, prepare_inputs takes files / pipe_ref / inputs, and catalog-id resolution for input preparation is recorded as deferred in the 0.5.0 changelog. Following that sentence gets a TypeError. The contrast the bullet wanted to draw is real — the run routes pass the id through untouched, the input-preparation helpers work on an inline closure — so it now draws it truthfully instead of inventing a signature. Also corrects the mthds floor in the tracker's preflight: it recorded >=0.8.1, but pyproject.toml has said >=0.8.2 since bc17c07, an ancestor of this branch's base, so the number was wrong the day it was written. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- TODOS.md | 12 +++++++++++- docs/architecture.md | 2 +- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/TODOS.md b/TODOS.md index fe196ae..f66ca0d 100644 --- a/TODOS.md +++ b/TODOS.md @@ -16,7 +16,7 @@ Suggested order is the order below. Phase 3 is the most urgent fix (a crash agai ## Phase 0 — preflight -- [x] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.1` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. Confirmed: 0.8.2. +- [x] `make install` and confirm the pinned `mthds` base is the one the code was written against (`uv pip show mthds`; the floor is `>=0.8.2` and the workspace copy is 0.8.2). Nothing in this plan needs a newer `mthds`. Confirmed: 0.8.2. - [x] Run `make agent-check` and `make agent-test` before touching anything, so a later failure is attributable to this work and not to the starting point. Both green at the starting point. - [x] Re-read `wip/updates.md` §6 and §7 once; if anything there contradicts this file, this file is the stale one. No contradiction found. @@ -226,6 +226,16 @@ One finding reached outside this repo and was filed rather than fixed: `Pipeline Considered and declined: guarding `iterate_methods` against a `next_cursor` of `""`, which would let a page double-yield. The adversarial pass demonstrated the mechanism and then confirmed the case is unreachable, since the platform's cursor is a base64 `LastEvaluatedKey`. That is the impossible-scenario defensiveness this plan set out to avoid. +Cubic's pass on `cb391c9` reported four more. Two were real: + +- **`docs/architecture.md` documented an API that does not exist.** The hosted-run-extensions section contrasted `method_id` with "`build_inputs(method_id=…)` and `prepare_inputs`, which are client-side sugar that resolves an id to inline files". Neither helper accepts a `method_id`: `build_inputs` takes a `BuildInputsRequest` of `files` / `pipe_ref` / `format` / `explicit`, `prepare_inputs` takes `files` / `pipe_ref` / `inputs`, and catalog-id resolution for input preparation is recorded as deferred in the 0.5.0 changelog. A reader following that sentence gets a `TypeError`. The contrast it wanted to draw is real, so the bullet now draws it truthfully. This is exactly the class of untrue claim Phase 5 set out to remove from this file, in a section Phase 4 added. +- **This tracker recorded the wrong `mthds` floor.** Phase 0 wrote `>=0.8.1`; `pyproject.toml` has said `>=0.8.2` since `bc17c07`, which is an ancestor of this branch's base — so the floor was already 0.8.2 when the preflight ran, and the number was wrong the day it was written. + +Two were not worth a commit: + +- **Adding this branch's own review-notes file to the sdist inventory in `wip/pr-14-review-notes.md`.** That listing is `tar -tzf` output captured from a build made before the notes file existed. Extending it changes nothing about the finding it supports — that the sdist ships internal planning documents — or about what someone picking the item up would do. +- **The counts in the completion narrative above ("two page-ceiling tests", "two hardcoded counts").** The no-hardcoded-counts rule exists because a live inventory drifts: it "creates diff churn, goes stale silently, and adds no value". None of that applies to a closed record of a finished review round, which gains no members and cannot go stale. The rule targets counting things that are still moving. + ## Known gaps left open on purpose - No e2e suite exists in this repo (`tests/` holds only `unit/`), so the live `views` gate and the live paging envelope are pinned only by mocked bodies here; the JS suite pins both live. Adding an e2e suite is separate work. diff --git a/docs/architecture.md b/docs/architecture.md index 75bbc68..e989345 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -84,7 +84,7 @@ The protocol's run args (`pipe_code`, `mthds_contents`, `inputs`, …) are the b - **A named parameter, not an `extra` entry.** Typing its own platform's arguments is the one job a layer-3 client exists to do; `extra` stays the escape hatch for an extension this client does not know about (a third-party server, or a newer version of ours). That split, and the reserved-key guard behind it, follow the layered extension policy: a hosted client types its own platform's arguments and guards them per layer, and that guard must never be pushed down into the protocol package. It was previously passed as `extra={"method_id": …}`, which worked but documented nothing and validated nothing. - **It reaches the wire through the base client's extension mechanism.** `_merge_hosted_run_extensions` folds it into the `extra` mapping handed to `super().execute` / `super().start`, which merges it into the body as a top-level property without knowing what it means. That is the layering working as designed, not a workaround: the protocol client stays catalog-agnostic while the hosted client owns the concept. - **The guard is per layer.** `method_id` is rejected inside `extra` *here*, and must never become reserved in `mthds`: a protocol client talking to another vendor's server has no business rejecting that vendor's arguments. -- **Pure pass-through — nothing is expanded client-side.** The platform resolves the id against the org's catalog. Do not confuse this with `build_inputs(method_id=…)` and `prepare_inputs`, which are client-side *sugar* that resolves an id to inline files before the request is built, so no `method_id` ever reaches those wire bodies. +- **Pure pass-through — nothing is expanded client-side.** The platform resolves the id against the org's catalog. The input-preparation helpers are the deliberate contrast: `build_inputs` and `prepare_inputs` take the method closure as inline `files` (with an optional `pipe_ref`) and accept no `method_id` at all — resolving a catalog id to its files client-side is deferred, additive work — so the id is a run option only and reaches no other wire body. - **Alone it is a run source; alongside an inline source it is linkage.** With no inline source the platform resolves and runs the stored method. With one, the inline source is what RUNS (precedence) and the id is recorded as run-history linkage on the Run row — the index key `GET /v1/runs?method_id=` queries. The base client's "something to run" precondition is satisfied because the merged extension mapping is non-empty, so a `method_id`-only run is sent rather than refused client-side. - **It rides the blocking fallback too.** `start_and_wait` degrades to `POST /v1/execute` against a runner with no run store, and the selector goes with it — so a bare `pipelex-api` answers the `422` that names the key instead of the client silently dropping it and running something else. - **An empty string is treated as absent.** `method_id=""` selects no method and links nothing, so it is not sent and does not satisfy the precondition. From e9c09e38ff506bc2e85addda446a32f4e896a527 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 11:52:20 +0200 Subject: [PATCH 19/20] Release v0.6.0 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- CHANGELOG.md | 58 +++++++++++++++++--------------------------------- pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 21 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ffd4d3..cd4aa05 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,53 +1,33 @@ # Changelog -## [Unreleased] +## [v0.6.0] - 2026-08-25 ### Added -- **The `/v1/validate` contract of pipelex-api 0.17/0.18 is now typed, and the `input_form` view is askable.** `validate` and `validate_files` take a `views: list[str] | None` opt-in after `render`. `input_form` — named by the new `VALIDATION_VIEW_INPUT_FORM` constant — is the only token the server supports today. Unlike `render`, the list is sent **verbatim**: nothing is injected, nothing is de-duplicated, and an explicit `[]` is sent as `[]`, because the server resolves the tokens as a set and lenient-ignores the ones it does not know (never a `422`), so normalizing client-side would only hide what the caller asked for. Left at its `None` default the key is not sent at all, which is what keeps the default response byte-identical for consumers that discard views. The constant is deliberately a constant rather than a closed enum on the parameter: the request boundary is open so a stale token never fails a call. - - A valid `PipelexValidationReport` now carries three typed fields instead of leaving them in `model_extra`: `warnings` (advisory lints on a bundle that is nonetheless valid — the same `ValidationErrorItem` type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid`), `liftable_pipes` (the new `LiftablePipeEntry` inventory of pipes the runtime may skip when an optional slot resolves absent), and `input_form` (the per-pipe descriptors, keyed like `pipe_io_contracts` and kept opaque on purpose — the descriptor vocabulary is owned elsewhere and a second copy here would be free to drift). `ValidationErrorItem` gains `missing_pipe_code`, symmetrical with the existing `missing_concept_code`. - - A body from an older runner still parses unchanged: both lists default empty and `input_form` defaults `None`, which is also exactly what a clean bundle and a view-less request yield, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them — `warnings` and `input_form` derive from a crate that was never assembled. - -- **Structured repair proposals: `SuggestedFix` and the fix-op vocabulary.** `ValidationErrorItem.suggested_fix` is now typed. A `SuggestedFix` carries a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops` — a list discriminated on `kind` over the closed `FixOpKind` set (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), each variant declaring exactly its own members, narrowed with an exhaustive `match op: case SetKeyOp(): …`. These are **reader** models: `extra="allow"`, not `frozen`, and without the runtime's wildcard-refusing validators, because this SDK reads fixes where the runtime plans them — a new server-side member on an op must not break parsing here. An unknown `kind` fails the whole verdict parse, deliberately and consistently with the closed `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break for no consumer benefit. - -- **Paged listing: `iterate_methods`, `iterate_runs`, and `get_run_detail`.** `iterate_methods(q=…, limit=…)` and `iterate_runs(method_id, created_from=…, created_to=…, limit=…)` are async generators that follow the cursors, and they are the idiom to copy: there is deliberately no `list_all_methods()`, because an all-at-once helper needs a cap and a cap is the silent truncation paging removed. They stop on different signals, and the difference is in the server rather than the client — `iterate_methods` keeps going through an empty page that carries a live cursor, since `q` is a post-read filter over a bounded index slice per request, while `iterate_runs` treats an empty page as the end, since the run date bounds are index key conditions. Both stop when the server hands back the cursor it was sent, checked before yielding so a stuck cursor never double-counts a page. Both also carry a runaway page backstop set far beyond any real catalog, raising the new `PagingNotTerminatingError` rather than returning what they have: the two stop rules above only catch a cursor that repeats *adjacently* or a server returning nothing, so a cursor cycling across several values over non-empty pages would otherwise loop forever re-yielding the same rows. `get_run_detail(run_id)` reads one run through `GET /v1/runs/{id}` — distinct from the lifecycle `get_run_status` (`/status`) and `get_run_result` (`/results`), and the only read carrying `mthds_contents` and `inputs`, which is what the run actually executed and the only record of it. - -- **New product models for the paged surface.** `MethodPage` / `MethodSummary`, `RunPage` / `RunDetail` / `RunErrorReport`, and `MethodFile` with the `parse_method_files` / `serialize_method_files` pair. A `MethodSummary` is deliberately not a `MethodData`: no `mthds`, no `python`, no `updated_at`, because none of them is in the index projection, and putting `mthds` back is what restored the truncation bug. `MethodData` also gains `org_id`, `created_by_user_id`, `description`, `deletion_state`, and a typed `python: list[MethodFile]`; `MethodWriteInput` gains `python: list[MethodFile] | None`. - - On the wire `python` is one string — the JSON text of a `[{name, content}]` array, or `""` for a method with no custom Python — and the converter lives here rather than in `mthds`, because the catalog serialization is a Pipelex product concern and there is no second Python consumer to keep in step. `MethodData` applies the parser through a `field_validator(mode="before")`, so `model_validate` keeps working at every call site; `MethodWriteInput` applies the serializer through a `field_serializer`, which makes the platform's three-way write contract fall out of `exclude_none=True`: `None` leaves the key out and preserves the stored Python, `[]` sends the `""` clear sentinel, a non-empty list replaces. Malformed wire text raises inside the validator and surfaces as a `pydantic.ValidationError`, the same way any other malformed response body fails here. - -- **`method_id` is a typed run option.** `execute`, `start`, and `start_and_wait` take the hosted platform's `method_id` as a first-class keyword parameter. It is a pure pass-through — the platform resolves the id against the org's catalog, and nothing is expanded client-side. Alone it is a run source; alongside `mthds_contents` the inline source is what runs and the id is recorded as run-history linkage on the Run row, which is what makes the run appear under `GET /v1/runs?method_id=`. It rides the blocking `execute` fallback too, so a bare runner answers the `422` that names it rather than the client silently dropping it. An empty string is treated as absent. Mechanically it reaches the wire through the base client's generic extension passthrough, which merges it into the body without knowing what it means — the protocol client stays catalog-agnostic while this client owns the concept. Typing its own platform's arguments is what a hosted client is for; `extra` remains the escape hatch for extensions this client does not know about. It is the same layered extension policy the TypeScript `@pipelex/sdk` follows, so both SDKs partition their arguments the same way. +- **Paginated catalog surface**: Introduced `iterate_methods` and `iterate_runs` async generators that follow pagination cursors to fetch entire catalogs without silent truncation, backed by new product models (`MethodPage`, `MethodSummary`, `RunPage`, `RunDetail`, `RunErrorReport`, `MethodFile`). A runaway page backstop raises the new `PagingNotTerminatingError` instead of looping forever on a cursor that cycles across non-empty pages. +- **Run details**: Added `get_run_detail(run_id)` to fetch a single run's execution details, including `mthds_contents` and `inputs`, which are excluded from list views for performance. +- **Validation views & typed reports**: Added a `views` parameter to `validate` and `validate_files` for opt-in structured views (e.g. `VALIDATION_VIEW_INPUT_FORM`), and extended `PipelexValidationReport` with typed `warnings`, `liftable_pipes`, and `input_form` fields. +- **Structured repair proposals**: Added `SuggestedFix` and a fix-operation vocabulary (`FixOpKind`, `FixSafety`, etc.) to `ValidationErrorItem`, along with a new `missing_pipe_code` field. +- **Method file parsers**: Added `parse_method_files` and `serialize_method_files` to convert custom Python source files to and from the platform's at-rest catalog string format. +- **Typed `method_id`**: `execute`, `start`, and `start_and_wait` now accept `method_id` as a first-class, typed keyword parameter. ### Changed -- **A non-string `method_id` is refused at the client boundary.** `execute` / `start` / `start_and_wait` now raise `PipelineRequestError` naming the received type, instead of letting a bare truthiness check decide. That check made an arbitrary partition of the wrong values — it silently dropped the falsy ones (`0`, `[]`) and forwarded the truthy ones (`123`, `["mt_1"]`) to a server `422` — and it was a *different* partition than `@pipelex/sdk` made for the same argument on the same wire. A published client validates its request-option types where it names them, so one wrong value gets one answer. `None` and `""` still contribute nothing, unchanged. The protocol-level arguments (`pipe_code`, `mthds_contents`) are not covered here: their guards belong in `mthds` and arrive with a floor bump. - -- **Breaking: the two product list routes return page envelopes.** `list_methods` returns a `MethodPage` (`items` are `MethodSummary` rows, not `MethodData`) and `list_runs` returns a `RunPage`, each carrying an opaque `next_cursor` that is `None` on the last page. Both take the paging and filter parameters the platform serves — `q` / `limit` / `cursor` on methods, `created_from` / `created_to` / `limit` / `cursor` on runs — kept on **presence** rather than truthiness, so an explicit empty `q` or `cursor` is forwarded for the API to reject rather than dropped silently into an unfiltered query that reads as working. `created_from` / `created_to` are instants (ISO-8601 with a UTC offset), inclusive, and are index key conditions rather than filters, so a bare date or a naive timestamp is a platform `400` surfaced as `ApiResponseError`. Also worth knowing: every `/v1/runs*` product route sits behind the platform's surface-access gate, which for API-key auth demands the `ff_api_keys` feature flag and fails closed — a `403` there means "flag", not "wrong key". - -- **Breaking: `PipelineRun.method_id` and `pipe_code` are nullable, and `MethodData.python` is a `list[MethodFile]`.** The platform genuinely serves the first two as null and both cases are ordinary: an ad-hoc run from an inline bundle belongs to no stored method, and a run that let the bundle's `main_pipe` decide names no pipe. `PipelineRun` also gains `org_id`, `created_by_user_id`, and a narrowed `error: RunErrorReport | None`. `MethodData.python` is converted from its wire string at the boundary, so callers never see that string. - -- **Breaking: `delete_method` returns the platform's acceptance instead of `None`.** `DELETE /v1/methods/{id}` answers `202` with `method_id`, `deletion_state`, and `deletion_job_id`, and the erasure continues asynchronously after that; the method was annotated `-> None` with a docstring promising an empty synchronous delete, so a caller reasonably concluded the method was gone when the call returned. It now returns the new `MethodDeletionAccepted` (with `MethodDeletionState`), which means "accepted", never "gone" — completion is the row disappearing from `list_methods`. A misleading contract around a destructive operation is worth a breaking change. -- **Breaking: `extra` now rejects `method_id`.** The client names the key itself, so it must also guard it — one argument must not arrive by two paths with different validation. Callers passing `extra={"method_id": …}` must pass the named parameter instead. The guard is deliberately per layer: `method_id` must never become reserved in `mthds`, which has no business rejecting another vendor's arguments. - -- **Ruff moves to 0.16.4, matching the version the VS Code extension bundles.** The extension now syncs `pyproject.toml` and `ruff.toml` documents to the language server, because Ruff from 0.16 on lints its own config files. A binary older than that parses those documents as Python source and paints phantom `invalid-syntax` diagnostics across every `pyproject.toml`. Pinning the dev dependency to the exact version the extension ships keeps the editor and the command line on one binary. This is a dev-dependency change: nothing shipped changes, and there is no version bump. - - Two mechanical conversions ride along, both applied by `ruff check --fix` and therefore not deferrable, since `make lint` would re-apply them on first contact: selector lists in `pyproject.toml` now name rules rather than code them, and `# noqa: CODE` suppressions became `# ruff: ignore[rule-name]`. `float-equality-comparison`, newly reported under `preview`, joins the existing `tests/` relaxations — those tests assert float literals that round-trip exactly, so `pytest.approx` would only add noise. - -- **`too-many-statements-in-try-clause` is now ignored explicitly, ahead of ever having a finding.** Every repo in the workspace that runs `select = ["ALL"]` with `preview` has settled on ignoring this rule, because shrinking a `try` clause changes which statements its handlers cover — an error-handling refactor rather than lint cleanup. This repo happened to have no findings, so the decision was never written down here, which meant whoever wrote the first long `try` clause would have had to reach it again on their own. The reasoning now sits inline next to the ignore. No code changes; the rule fires nowhere today. - +- `list_methods` and `list_runs` now return page envelopes (`MethodPage` and `RunPage`) instead of bare arrays, with `q`, `limit`, and `cursor` query parameters forwarded based on presence rather than truthiness. (Breaking) +- `PipelineRun.method_id` and `pipe_code` are now nullable to accurately reflect platform behavior for ad-hoc runs and dynamic pipes. (Breaking) +- `MethodData.python` is now a typed `list[MethodFile]` instead of a raw string, converted automatically at the client boundary. (Breaking) +- `delete_method` now returns a `MethodDeletionAccepted` object instead of `None`, reflecting that deletion is an asynchronous cascade rather than an immediate synchronous action. (Breaking) +- The `extra` parameter on run methods now rejects `method_id`; it must be passed via the dedicated named parameter, and passing a non-string `method_id` raises a `PipelineRequestError` at the client boundary rather than delegating the failure to the server. (Breaking) +- Bumped the `mthds` dependency floor from `>=0.8.1` to `>=0.8.2`. +- **Linting & tooling**: Bumped the `ruff` dev dependency to `0.16.4` to match the version the VS Code extension bundles, converted `pyproject.toml` selector lists to rule names instead of codes, and explicitly ignored `too-many-statements-in-try-clause`. ### Fixed -- **`list_methods` and `list_runs` crashed against the deployed platform, and the tests hid it.** The platform reshaped both routes into `{items, next_cursor}` envelopes — methods since 2026-08-18, runs since 2026-08-11, both live in production. This SDK still iterated a bare array, and iterating the envelope dict yields its *keys*, so the very first call was `MethodData.model_validate("items")` and every invocation raised `pydantic.ValidationError`. `list_runs` failed identically. Nothing was red because the unit tests mocked the pre-paging shape; they now mock the envelope, and a run row with a null `pipe_code` and `method_id` parses, which is what would have raised next once the envelope was fixed. - -- **Documentation: the parity claim was false and is now an honest list.** `docs/architecture.md` claimed the surface was "surface-complete, with no silent gaps" against `@pipelex/sdk`, which stopped being true as the JS SDK grew; the section now names what has no Python counterpart (`lint`, `format`, `resolve`, `codegen`, `build_output` / `build_runner` / `concept` / `pipe_spec`, `run_codegen_check`, `get_method_closure`) and records each as deferred rather than missing. The stale "Out of scope" bullet listing the `/v1/build/*` helpers as deferred wholesale is corrected too — `build_inputs` shipped in 0.5.0. - -- **Documentation: `TokensUsageRecord` is attributed to the right brand.** The module and class docstrings in `pipelex_sdk/runs.py`, `docs/run-usage.md`, and `docs/architecture.md` said the record was "specified in the MTHDS protocol spec". It is not: inference accounting is a Pipelex runtime extension the MTHDS Protocol does not model, so the hosted API is what pins the wire contract and this SDK mirrors the runtime's own record. Getting the attribution wrong on a public package misplaces the brand boundary the rest of the architecture is organized around. No behaviour changed. -- **Documentation: no more citations a reader cannot open.** Several docstrings, comments and doc passages cited internal specs and the conformance suite by bare repo-relative path — paths that resolve to nothing for anyone who clones this public package, so they read as rot rather than as a deliberate boundary. Each site now states the rule it was citing instead: the layered extension policy behind the hosted run extensions and the reserved-`extra` guard, and the locked category vocabulary shared with the conformance corpus. References to the sibling public `pipelex-sdk-js` repo are kept where they explain a port. No behaviour changed. - -- **Documentation: the README's import map pointed at a name that does not exist.** "Public import paths" told readers to reach `PipelexValidationResult` through `from mthds.runners.api.models import PipelexValidationResult` — an `ImportError`, since the Pipelex narrowing of the verdict union is owned by this package (`pipelex_sdk.validation_models`), exactly as the "Brand boundary" section of `docs/architecture.md` states. The section now carries a bullet for `pipelex_sdk.validation_models` and cites `mthds.protocol.models.ValidationResult` as the neutral union it narrows. The typed-errors bullet gains `PagingNotTerminatingError` and, like the product-models bullet, is marked non-exhaustive rather than reading as a complete list it never was. +- **Pagination crash**: Fixed a critical bug where `list_methods` and `list_runs` crashed against the deployed platform after the API shifted to `{items, next_cursor}` envelope responses; tests were updated to mock the correct paginated shape. +- **Docs – parity claims**: Updated `docs/architecture.md` to honestly reflect the parity gaps with the TypeScript `@pipelex/sdk` (e.g. deferring `lint`, `format`, `codegen`) instead of claiming a surface-complete client. +- **Docs – import paths**: Fixed a broken import path in the `README.md` quickstart (`PipelexValidationResult` is owned by this package, not `mthds`). +- **Docs – brand attribution**: Corrected docstrings and architecture docs to attribute `TokensUsageRecord` as a Pipelex runtime extension rather than an MTHDS protocol specification. +- **Docs – dead links**: Replaced unopenable internal repository citations in docstrings and comments with explicit, readable rule descriptions. ## [v0.5.0] - 2026-07-22 diff --git a/pyproject.toml b/pyproject.toml index e466d4a..a69d19a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "pipelex-sdk" -version = "0.5.0" +version = "0.6.0" description = "The Python client for the Pipelex hosted API — the MTHDS Protocol surface plus the durable run lifecycle and the Pipelex product surface, built on the `mthds` protocol base." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] diff --git a/uv.lock b/uv.lock index 0fa9746..f25455f 100644 --- a/uv.lock +++ b/uv.lock @@ -303,7 +303,7 @@ wheels = [ [[package]] name = "pipelex-sdk" -version = "0.5.0" +version = "0.6.0" source = { editable = "." } dependencies = [ { name = "httpx" }, From eaf4132a5f0c101875b218fc47c36930f9139909 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 25 Aug 2026 12:16:44 +0200 Subject: [PATCH 20/20] Say what is finished, so three docs stop reading as stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of PR #15 raised seven documentation findings, five of them on `wip/updates.md` alone, each asking to rewrite or delete a section of it because the work it designs has since shipped. That is what a design document is for, and this repo already splits the two roles: the design says what and why, and `TODOS.md` is the tracker that carries the per-item status, by the decision recorded in the tracker itself. The sections stay as written; what was missing was a pointer saying so, which is why the file read as stale five separate times. It now opens with one line naming its status, the release it shipped in, and the tracker. The other two findings were real and small. The sdist listing in the PR #14 review notes was taken at that review's own commit and introduced as a build of the current tree, so it now reads as the current release's archive; it says which build it is and why it is kept rather than re-run. And the architecture doc's scope heading still said "for v0.1" over a list maintained through later releases, including a bullet corrected on this branch — the heading dates to the original scaffolding commit and was the half that sweep missed. Documentation only. No Python, no dependency and no changelog change: the wip notes are not release-facing and a heading rename is not either. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011p6KPGP2eiLjni5sErWUre --- docs/architecture.md | 2 +- wip/pr-14-review-notes.md | 2 +- wip/updates.md | 2 ++ 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index e989345..5002ab2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -179,7 +179,7 @@ The wire models are snake_case Pydantic v2. Response models are extension-open ( `health()` → `GET {origin}/health`. The one route served at the **origin**, NOT under the `/v1` prefix — the origin is derived from the base URL (`_origin_of`, exposed as `self.origin_url`), so a base URL of `https://api.pipelex.com/v1/...` still probes `https://api.pipelex.com/health`. It is **out-of-protocol**: the MTHDS Protocol defines no health route, and `/health` is neither a protocol nor a product surface. It rides `_request_json` (the plainer regime), so a non-2xx raises `PipelineRequestError` rather than the product `ApiResponseError` — liveness needs no `code` taxonomy. Transport failures still map to `ApiUnreachableError`. (Checkpoint-5 decision: kept the plainer regime — see "Error regimes" above.) -## Out of scope for v0.1 +## Out of scope - The remaining `/v1/build/*` authoring helpers — `build_output`, `build_runner`, `concept`, `pipe_spec`. `build_inputs` shipped in 0.5.0 and is no longer deferred. - Organization *switch* (a WorkOS session operation, not a `/v1` route). diff --git a/wip/pr-14-review-notes.md b/wip/pr-14-review-notes.md index 99599ba..fbb3c0f 100644 --- a/wip/pr-14-review-notes.md +++ b/wip/pr-14-review-notes.md @@ -10,7 +10,7 @@ The findings that *were* acted on during the review are recorded in `CHANGELOG.m **Status:** Confirmed, pre-existing, widened by this branch. Deferred because the fix is a packaging change to `pyproject.toml`, which this branch does not touch and which has release implications worth deciding on their own. -`pyproject.toml` declares `build-backend = "hatchling.build"` (`pyproject.toml:40`) and carries no `[tool.hatch.build.targets.sdist]` section, so hatchling falls back to including everything the VCS does not ignore. Building the current tree confirms it: +`pyproject.toml` declares `build-backend = "hatchling.build"` (`pyproject.toml:40`) and carries no `[tool.hatch.build.targets.sdist]` section, so hatchling falls back to including everything the VCS does not ignore. An sdist built at the review point confirms it. The listing below is that build — version 0.5.0, at `301d96e` — and is kept as it was taken rather than re-run, so it predates this very file and is not the current release's archive: ``` $ uv build --sdist diff --git a/wip/updates.md b/wip/updates.md index d132265..dd9109c 100644 --- a/wip/updates.md +++ b/wip/updates.md @@ -1,5 +1,7 @@ # Updates warranted by pipelex-api 0.17.0 / 0.18.0, the pipelex-server bump, and `@pipelex/sdk` 0.14.0 +**Status: implemented.** Every change designed below landed, and shipped in `pipelex-sdk` v0.6.0. [`TODOS.md`](../TODOS.md) is the implementation tracker for this design and carries the per-item status; read it for what was done, and this file for what was intended and why. This document is deliberately left as it was written, so its sections below still speak in the future tense and its §7 decisions stay quotable from the tracker. + A design for what this repo still owes after the three sources named in the title, written on the `feature/Typed-method-id-run-option` branch, which already carries the typed `method_id` run option and the honest `delete_method` contract. Every claim below was checked against the code it names; line numbers are as of 2026-08-25 and will drift. ## Verdict