diff --git a/.claude/skills/bump-mthds/SKILL.md b/.claude/skills/bump-mthds/SKILL.md new file mode 100644 index 0000000..97d946b --- /dev/null +++ b/.claude/skills/bump-mthds/SKILL.md @@ -0,0 +1,448 @@ +--- +name: bump-mthds +description: > + Move this repo's exact `mthds` dependency pin to the latest release on PyPI + (or a version you name), re-lock, adapt the client to whatever the new release + broke, run the checks, and write the CHANGELOG entry — stopping before the + commit. Use whenever the user says "bump mthds", "update mthds", "upgrade + mthds", "move to mthds 0.11.0", "get us on the latest mthds", "is our mthds + floor stale", "what's the latest mthds", or asks to build this SDK against a + newly published `mthds` release. Also use when a failure looks like an mthds + version mismatch — a pyright `reportAttributeAccessIssue` on an inherited + `MthdsAPIClient` member (`_send`, `_url`, `_post_validate`), a + `reportIncompatibleMethodOverride` or `reportIncompatibleVariableOverride` on + one of the validate narrowings, an ImportError for a name that used to be + under `mthds.protocol`, a pydantic `Extra inputs are not permitted` from a + protocol model, or a `uv lock` saying mthds is unsatisfiable. This is the + **`mthds` PyPI package** (the MTHDS standard's Python client, from the sibling + `mthds-python` repo) — not the MTHDS spec pages, not the `mthds` npm package, + and not releasing `pipelex-sdk` itself, which is the `release` skill. +--- + +# Bump the `mthds` dependency + +`mthds` is the MTHDS standard's Python client, and this SDK is not merely a +caller of it — it is **built on its inheritance seam**. `PipelexAPIClient` +subclasses `MthdsAPIClient` and reuses its transport (`_send`, `_url`), its +body-builders (`_post_validate`), its constants (`_API_PREFIX`, +`_DEFAULT_REQUEST_TIMEOUT_SECONDS`) and its degrade helper +(`_raise_if_execute_degraded`). `pipelex_sdk/validation_models.py` subclasses +its report and diagnostic models. `pipelex_sdk/errors.py` **re-exports** +`RunStillRunningError` as part of this package's own public surface. + +That is what makes this bump different from bumping an ordinary dependency, in +two directions at once: + +- **Upstream can break you without calling it a break.** Half the surface this + repo depends on is underscore-prefixed. `mthds-python` is free to rename + `_send` in a patch release, because by Python convention that name is private — + but here it is a documented protected extension surface (see `CLAUDE.md`, + "Architecture invariants"). Read every release as if those names were public. +- **You can break your own consumers without writing the line.** A renamed + symbol this package re-exports, or a tightened model this package narrows, is + a breaking change to `pipelex-sdk`'s API even though the diff is one version + string. It has to reach the changelog as one. + +The job is to land the new pin in a state a human can read and commit: pin +moved, lock regenerated, source adapted, checks green, changelog and docs +written, ledger squared. **Stop before committing** — the user stages and commits. + +## What the bump touches + +| File | Why it moves | +|---|---| +| `pyproject.toml` | The pin itself, one line in `[project].dependencies` — `mthds==X.Y.Z`. Possibly also `[tool.ruff.lint.flake8-type-checking].runtime-evaluated-base-classes`, which names `mthds` classes by dotted path (see step 7) | +| `uv.lock` | Regenerated by `make li` | +| `pipelex_sdk/**` | Wherever the new release renamed, split, moved or tightened something this client inherits, imports or re-exports | +| `tests/**` | Same, plus the hand-written httpx fixtures that encode a protocol model's shape | +| `CHANGELOG.md` | An entry under `## [Unreleased]` — always at least the pin line | +| `docs/architecture.md` | Only when the inherited surface or the brand boundary moved — that document names both explicitly | + +Do **not** touch `[project].version`. That is `pipelex-sdk`'s own version and it +moves only at release time, via the `release` skill. + +## The pin is exact, and it always tracks latest + +The requirement is `mthds==X.Y.Z`, not `>=`. That is the opposite of the usual +advice for a published library, so know why before you loosen it: this SDK does +not merely call `mthds`, it inherits its transport and narrows its protocol +models, and those models are `extra="forbid"` shapes the standard's client +tightens release by release. A range would let a resolver hand a consumer a +version this repo never tested against, and the break would surface as a parse +refusal inside `validate` at runtime rather than as a conflict at install time. + +**The standing policy is to move the pin to the latest release, every time**, +even when the new release carries nothing this client uses. Two consequences +worth naming when you report: + +- Every downstream install is forced onto exactly that `mthds`. Say so in the + changelog when the bump is otherwise uneventful, because for a consumer + "pipelex-sdk now requires mthds 0.11.1" *is* the change. +- `pipelex` (the engine) pins `mthds` exactly too, and the two packages + routinely land in one environment. **Two exact pins on different versions do + not resolve at all**, so a skew between the repos is an install-time failure + for anyone holding both, not a silent drift that surfaces later. Read + `pipelex/pyproject.toml` as part of this bump and report the skew if you find + one; keeping the two moving together is why both repos run this same skill. + +## Four numbers, none of which is the others + +This is the single most common way to get confused here, and two of them appear +in the same sentence of an upstream changelog. + +| Number | Where it lives | What it means | +|---|---|---| +| The **`mthds` package version** | `mthds-python/pyproject.toml`, PyPI | The Python client's own release number. **This is what you are bumping.** | +| The **MTHDS standard version** | `MTHDS_STANDARD_VERSION` in `mthds.package.manifest.schema` | The version of the *standard* that client implements. This repo never reads it — it stamps no crates and ships no manifest — so unlike in `pipelex`, there is nothing here to check when it moves. | +| The **spec site's release number** | the `mthds/` repo's own CHANGELOG | The documentation site's release. Coincidentally close to the package number; unrelated to it. | +| **`pipelex-sdk`'s own version** | `[project].version` here | This package's release number. Not yours to move — that is the `release` skill. | + +Read every version reference in the upstream notes against this table before +repeating it in ours. + +## Workflow + +### 1. Orient + +Read the pin **from `pyproject.toml`**, not from the virtualenv: + +```bash +grep -n '"mthds' pyproject.toml +.venv/bin/python -c "import importlib.metadata as m; print(m.version('mthds'))" +``` + +Under an exact pin these two must agree, so any disagreement in either +direction means the same thing: someone edited `pyproject.toml` without +re-running `make li`. (That is a change from when the requirement was a floor, +where a venv resolved above the minimum was normal and unremarkable.) + +Check `git status` and note what was already dirty **before** you start. At the +end you need to separate your changes from theirs and never stage something that +isn't yours. + +Then ask the ledger what it already knows. Both `mthds-python` and `pipelex` +file items here when they land something this SDK will have to absorb, and those +items usually carry the exact diff — file, line, and the shape of the fix — +which is faster and more reliable than rediscovering it from a pyright error: + +```bash +ledger inbound +ledger list --origin mthds-python --status open +``` + +Read past the rows owned by this repo: a row owned by `pipelex` or `pipelex-sdk-js` +is a sibling piece of the same cascade you will be filing into at step 10. Claim +(`ledger claim `) any item that describes the adaptation you are about to do. + +### 2. Resolve the target version + +If the user named a version, use it. Otherwise ask PyPI: + +```bash +curl -s https://pypi.org/pypi/mthds/json \ + | python3 -c "import json,sys; print(json.load(sys.stdin)['info']['version'])" +``` + +`info.version` is the latest non-prerelease, which is what "latest" means for a +pin other people inherit. + +**Ask PyPI, never the sibling checkout.** `mthds-python/pyproject.toml` is a +working tree and is frequently *ahead* of what has been published — a version +sitting there under `## [Unreleased]` is not installable, and a pin naming it +makes `uv lock` fail. If you notice the sibling is ahead, mention it in your +report (it is a preview of the next bump) but do not adopt it. + +If the resolved version equals the current pin, say so and stop: there is +nothing to do, and manufacturing lock churn is worse than reporting "already +current". If the venv disagrees, mention that — `make li` is the real fix there. + +### 3. Digest the upstream changes — *before* editing anything + +For most dependencies you bump first and read the notes later. Not this one. +Because this SDK builds on mthds' types and inheritance seam rather than merely +calling it, the release notes are a work list, and reading them first is what +turns a pyright cascade into a set of expected edits: + +```bash +.venv/bin/python .claude/skills/bump-mthds/scripts/upstream_notes.py 0.9.0 0.11.0 +``` + +The helper reads `../mthds-python/CHANGELOG.md` and prints the released sections +strictly after the old pin up to and including the new one. It skips +`## [Unreleased]` — that section describes work that is *not* in the version you +are adopting, and this repo's changelog is read by people deciding whether an +upgrade will break them. If the checkout predates the target release the script +says so; fall back to: + +```bash +gh release view v0.11.0 --repo mthds-ai/mthds-python +``` + +Read the notes for four things specifically, in this order of how often they +bite here: + +1. **Anything under `mthds.runners.api.client`** — the base class this client + extends. A changed signature on `execute` / `start`, a member moving from + the base into a mixin, a renamed underscore helper. Upstream may not flag + any of these as breaking. +2. **Renames, splits, or module moves under `mthds.protocol`** — imported in + `client.py`, `errors.py`, `validation_models.py`, `execute_result.py`, + `runs.py`, and in the tests. +3. **Parse-time tightening.** mthds' protocol models are `extra="forbid"`, and + this SDK narrows several of them. A model that gains or loses a required + field breaks both the narrowing and the hand-written test fixtures. +4. **Anything that becomes redundant here.** Duplication between this package + and `mthds` is deliberate and temporary in places (see step 6) — an upstream + removal is your cue to delete something local, not to leave it standing. + +### 4. Move the pin + +One line in `pyproject.toml`. Make it a substring edit — replace `==0.11.0` with +`==0.11.1` on that line and leave every other character alone. Use your editor +rather than a shell one-liner: `sed -i` takes a separate empty argument on macOS +and an attached suffix on GNU/Linux, so no single invocation is portable, and +the `release` skill edits this same file the same way. + +Confirm with `grep -n '"mthds' pyproject.toml` before moving on. That grep +returns four hits, not one: the pin, plus the three dotted `mthds.protocol` +paths in the ruff `runtime-evaluated-base-classes` list. Read them now — if the +step-3 notes moved any of those three classes, that list moves in this same edit +(step 7 explains what happens when it doesn't). + +### 5. Re-lock and install + +```bash +make li +``` + +That is `make lock` (`uv lock`) plus `make install` (`uv sync --all-extras`), so +it rewrites `uv.lock` *and* puts the new mthds in `.venv`, which everything +downstream of here depends on. + +**Do not reach for `make update`.** That is `uv lock --upgrade` and it moves +every dependency in the tree at once — including the pinned dev toolchain +(`ruff`, `pyright`, `mypy`, `pylint` are `==` pins in `[project.optional-dependencies].dev` +precisely so lint findings do not appear out of nowhere). Moving the pin is +already enough to make plain `uv lock` re-resolve mthds. + +Check the lock actually moved — a pin edit that didn't take is silent: + +```bash +grep -A1 '^name = "mthds"' uv.lock +``` + +If `uv lock` reports the requirement is unsatisfiable, the version is almost +certainly not on PyPI yet. Re-run the query from step 2. If it genuinely isn't +published, stop and tell the user rather than inventing a git or path source: +adding one is a deliberate decision with a real cost (in uv, a *source* outranks +a version specifier, so the pin you wrote becomes decorative), not a +workaround to apply quietly. + +### 6. Take stock of the seam + +Before running anything, print what the base actually offers now. This is +cheap, and it is the one reading that tells you whether a local workaround has +expired: + +```bash +.venv/bin/python - <<'PY' +import inspect +from mthds.runners.api.client import MthdsAPIClient +from pipelex_sdk.client import PipelexAPIClient +NOISE = {"_abc_impl", "_is_protocol", "_is_runtime_protocol"} # ABC/Protocol machinery, not a seam +base = {name for name, _ in inspect.getmembers(MthdsAPIClient) if not name.startswith("__")} - NOISE +own = set(PipelexAPIClient.__dict__) - NOISE +print("inherited, protected:", sorted(n for n in base - own if n.startswith("_"))) +print("inherited, public: ", sorted(n for n in base - own if not n.startswith("_"))) +print("overridden: ", sorted(base & own)) +PY +``` + +Two things to compare it against: + +- **The protected list is the extension surface `CLAUDE.md` pins.** A name that + vanished from it is the break, and `client.py` is where it lands. +- **The overridden list is where local suppressions live.** `docs/architecture.md` + records that some duplication between this package and `mthds` is transitional: + the lifecycle models are owned here while the base still declared its own + copies, and the narrow `# type: ignore[override]` on `validate` exists for + exactly that divergence. When the base stops declaring a member, the + suppression is no longer buying anything and should go, along with the + paragraph in the docs that explains it. + +### 7. Adapt the source + +Run the type checkers first — they are the fastest and most complete readers of +a protocol break, and they cost seconds where the suite costs minutes. Run both; +they disagree about different things, and this repo gates on both: + +```bash +make pyright +make mypy +``` + +Work from the step-3 notes and any ledger item, not from guesswork. The +adaptations that recur here, in rough order of how often they bite: + +- **A renamed or moved member of the base class.** Fix the call site in + `client.py`. If the *shape* changed rather than the name, adopt the new shape + rather than reconstructing the old one locally. +- **A renamed or split type under `mthds.protocol`.** Fix the import, every + `isinstance` / `match` narrowing, and — critically — the **ruff config**. The + `runtime-evaluated-base-classes` list in `pyproject.toml` names + `mthds.protocol.models.ValidationReport`, `…InvalidValidationReport` and + `…ValidationDiagnostic` **by dotted path**. Ruff does not error on an entry + that no longer resolves; it just silently stops treating those bases as + runtime-evaluated, decides the `mthds.protocol` annotations are type-only, and + moves them into a `TYPE_CHECKING` block — where pydantic cannot resolve them + when it builds the model. The failure is at import time, in the test run, + with a message about an unresolvable annotation and nothing pointing at ruff. + Grep `pyproject.toml` for the old dotted path in the same edit as the import. +- **A tightened model this package narrows or constructs.** mthds' protocol + models are `extra="forbid"`, so a member that used to ride through now raises + at construction. Watch for `model_copy(update={...})` — it does not validate, + so it will happily leave a model in a state its class forbids, and the type + checker is the only thing that notices. +- **A stale suppression.** `reportUnnecessaryTypeIgnoreComment` is `"none"` in + this repo's pyright config, so a `# type: ignore[override]` or + `# pyright: ignore[reportIncompatibleVariableOverride]` that the bump just + made unnecessary will sit there forever without a single warning. Step 6 is + how you find them; check each one by hand against the new base. +- **A new abstract method** on something this repo subclasses. Implement it; + don't `raise NotImplementedError` to get green. + +Failures here are the breaking change announcing itself, not incidental +breakage. Adopt the new API. A bump whose failures were papered over is worse +than no bump. + +### 8. Run the checks + +```bash +make agent-check +make agent-test +``` + +`agent-check` is `fix-unused-imports format lint pyright mypy`. Before you call +the bump done, also run the two gates it leaves out, since CI runs them: + +```bash +make pylint +make check-unused-imports +``` + +**Read test failures carefully — they come in two kinds and only one is a bug.** +Unit tests here mock at the httpx boundary with hand-written JSON, so a +tightened or reshaped upstream model produces a failure that says "your fixture +is stale", not "your code is wrong". Updating the fixture is correct when the +new shape is what the hosted API actually emits; it is a cover-up when the +client should have been the thing that changed. `tests/unit/test_validation_contract.py` +is the one that pins the nesting of the two strictness regimes (closed imported +artifacts inside an extension-open report envelope) — a failure there is about +the contract, not the fixture, and deserves a real answer. + +### 9. Write the changelog and the docs + +Add to `CHANGELOG.md` under `## [Unreleased]`, creating that heading right after +`# Changelog` if it isn't there. Work in progress accumulates there until a +release cuts it into a version heading — do not add a `## [vX.Y.Z]` heading +yourself, and do not bump `[project].version`. + +Every bump gets at least the pin line. Follow the house form, which names the +release *by what it carries*: + +```markdown +- Moved the exact `mthds` pin from `==0.11.0` to `==0.11.1`, . +``` + +A pin move narrows what every downstream environment may install, so it is +breaking for this package's consumers even when the upstream release is not. +Mark it so. + +Anything that reached `pipelex-sdk`'s own surface gets its own entry under +`### Changed`, marked breaking, written for an importer deciding whether this +upgrade will cost them work. Name the old symbol and the new one — a reader +hitting an ImportError searches for the name they had. Write "breaking", not +"pre-1.0 breaking". Three things count as this package's surface even though the +symbol belongs to mthds: the **re-export** in `errors.py`, the **narrowing +subclasses** in `validation_models.py`, and any **imported annotation** that +appears in a public signature. + +If the bump changed the inherited surface, the brand boundary, or retired a +transitional duplication, update `docs/architecture.md` in the same change — it +describes all three by name, and a bump is exactly what makes those paragraphs +stale. If the bump is genuinely uneventful, say that in one sentence and stop. +Padding a quiet bump with upstream detail that doesn't affect this repo makes the +loud ones harder to spot. + +### 10. Square the ledger + +- **Close what you actually landed**, with evidence — the file and line you + changed, and the check that went green. `Closes ` goes in the PR body when + the user opens one. +- **File the parity item.** `pipelex-sdk-js` is this package's twin and consumes + the `mthds` npm package; when a protocol model moves, it usually moves in both + languages. That repo's move is not yours to make from here — file it + (`ledger new --owner pipelex-sdk-js …`) naming the symbol and the version. +- **File the engine item if the pins have diverged.** If `pipelex` names a + different exact `mthds` after this, the two packages no longer co-install at + all — that is not a latent gap but a live break, and it belongs to that repo. +- **File the reverse direction if you found an upstream problem.** A protocol + model that cannot express what the hosted API emits is an item owned by + `mthds-python`, with the payload that broke it. +- `ledger validate`, then `ledger commit`. Nothing else pushes the ledger. + +### 11. Report and stop + +Show the user: + +- The pin move, old → new, and whether `uv.lock` actually followed. +- Whether `pipelex`'s exact `mthds` pin agrees with the one you just wrote. +- What changed on the **inherited seam** (step 6) — that is the part nobody can + see from the diff, and the part most likely to matter next time. +- Every file you changed, separated from what was already dirty when you started. +- The check results, honestly — if `agent-test` failed, say so with the output + rather than reporting a bump as done. If you updated a test fixture, say which + and why it was the fixture that was wrong. +- Whether the bump is **breaking for `pipelex-sdk`'s own importers**, since that + is what decides how the next release is written and who has to move after it. +- Anything left for a human: an upstream change whose adaptation is a judgment + call, or a version the sibling checkout has that PyPI doesn't yet. + +Then stop. Do not commit, branch, or push unless the user asks. + +If they do ask, **stage the files explicitly by path** — never `git add -A`. A +working branch here is `chore/` (spelled out, from the closed prefix set) +and PRs target `dev`. + +## Traps worth remembering + +- **Upstream's private is this repo's contract.** `_send`, `_url`, + `_post_validate`, `_raise_if_execute_degraded` are underscore-prefixed + upstream and load-bearing here. A patch release can move them. +- **The ruff `runtime-evaluated-base-classes` list names mthds classes by dotted + path.** A module move there fails at *runtime*, in pydantic, with nothing + pointing back at the lint config. +- **`reportUnnecessaryTypeIgnoreComment` is off.** Suppressions here never + expire on their own; step 6 is the only thing that finds them. +- **The pin and `pipelex`'s pin are one system.** Both packages name `mthds` + exactly, so a version this repo moves to alone makes the pair uninstallable + together. Check `pipelex/pyproject.toml` in the same pass, and say what you + found even when they agree. +- **The sibling checkout is routinely ahead of PyPI.** Read `mthds-python`'s + changelog for *understanding*, PyPI for *the target version*. A pin naming + an unpublished version fails `uv lock`. +- **`## [Unreleased]` upstream is not in the version you adopted.** Never quote + it as part of the release. The helper script skips it for you. +- **`model_copy(update=…)` does not validate.** It is how a model ends up + holding a value its class forbids, green at runtime, wrong on the type. +- **`make update` is not `make li`.** The former upgrades the whole tree, + including the `==`-pinned linters. +- **A stale test fixture and a real break look identical.** Both are a red test + against a hand-written JSON body. Decide which one you are looking at before + editing either side. +- **`MTHDS_STANDARD_VERSION` is not this repo's problem.** `pipelex` stamps it + onto crates and has to track it; this SDK never reads it. Don't port that step + over from the engine's version of this skill. +- **`[project].version` is `pipelex-sdk`'s own version.** Bumping it is the + `release` skill's job, not this one's. diff --git a/.claude/skills/bump-mthds/scripts/upstream_notes.py b/.claude/skills/bump-mthds/scripts/upstream_notes.py new file mode 100755 index 0000000..a25d77c --- /dev/null +++ b/.claude/skills/bump-mthds/scripts/upstream_notes.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +"""Extract `mthds` release notes for the versions a bump crosses. + +Reads the sibling `mthds-python` checkout's CHANGELOG.md and prints every +released section strictly after ``old_version`` up to and including +``new_version``. + +Three boundaries this exists to get right: + +- ``## [Unreleased]`` is never printed. It describes work that is *not* in the + version being pinned. Quoting it in this repo's changelog is a plain factual + error about what the upgrade contains -- and in this pairing it is a live + hazard rather than a theoretical one, because `mthds-python`'s working tree is + routinely ahead of PyPI. +- The old floor's own section is excluded (it was already in effect) while the + new one's is included. +- A checkout that predates the target release cannot answer, and says so instead + of printing a plausible-looking short range. + +Exits non-zero with an explanation when the checkout cannot answer. Fall back to +``gh release view v --repo mthds-ai/mthds-python`` in that case. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +from packaging.version import InvalidVersion, Version + +# .../pipelex-sdk-python/.claude/skills/bump-mthds/scripts/upstream_notes.py +# parents[4] is this repo's root; its parent is the workspace root. +DEFAULT_CHANGELOG = Path(__file__).resolve().parents[4].parent / "mthds-python" / "CHANGELOG.md" +HEADING = re.compile(r"^## \[v?(?P\d+\.\d+\.\d+[^\]]*)\]") +UNRELEASED = re.compile(r"^## \[Unreleased\]", re.IGNORECASE) +FALLBACK = "gh release view v{version} --repo mthds-ai/mthds-python" + + +def parse_version(raw: str) -> Version: + """Parse a version string into a PEP 440 version. + + Ordering has to hold among prereleases as well as between a prerelease and + the release it leads to, and equality here is normalization-aware, so a + section is matched by the version it denotes rather than by how it was + spelled in the heading. + """ + try: + return Version(raw.strip()) + except InvalidVersion as exc: + msg = f"Not a version this script can compare: {raw!r}" + raise SystemExit(msg) from exc + + +def split_sections(text: str) -> list[tuple[str, str]]: + """Return [(version, body)] for released sections, in file order.""" + sections: list[tuple[str, str]] = [] + current_version: str | None = None + buffer: list[str] = [] + + for line in text.splitlines(): + if UNRELEASED.match(line): + if current_version is not None: + sections.append((current_version, "\n".join(buffer).strip())) + current_version, buffer = None, [] + continue + match = HEADING.match(line) + if match: + if current_version is not None: + sections.append((current_version, "\n".join(buffer).strip())) + current_version, buffer = match.group("version"), [line] + continue + if current_version is not None: + buffer.append(line) + + if current_version is not None: + sections.append((current_version, "\n".join(buffer).strip())) + return sections + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("old_version", help="the floor currently declared, excluded from the output") + parser.add_argument("new_version", help="the version being adopted, included in the output") + parser.add_argument( + "--changelog", + type=Path, + default=DEFAULT_CHANGELOG, + help=f"path to mthds-python's CHANGELOG.md (default: {DEFAULT_CHANGELOG})", + ) + args = parser.parse_args() + + if not args.changelog.is_file(): + print( + f"No mthds-python changelog at {args.changelog}.\nFall back to: {FALLBACK.format(version=args.new_version)}", + file=sys.stderr, + ) + return 2 + + low = parse_version(args.old_version) + high = parse_version(args.new_version) + if low >= high: + print(f"{args.new_version} is not newer than {args.old_version} -- nothing to digest.", file=sys.stderr) + return 2 + + sections = split_sections(args.changelog.read_text(encoding="utf-8")) + known = {parse_version(version) for version, _ in sections} + if high not in known: + print( + f"The checkout at {args.changelog} has no section for {args.new_version} -- it likely predates that release.\n" + f"Fall back to: {FALLBACK.format(version=args.new_version)}", + file=sys.stderr, + ) + return 3 + + wanted = [(version, body) for version, body in sections if low < parse_version(version) <= high] + if not wanted: + print(f"No released sections between {args.old_version} (exclusive) and {args.new_version}.", file=sys.stderr) + return 3 + + print("\n\n".join(body for _, body in wanted)) + if low not in known: + print( + f"\nNote: no section for the old floor {args.old_version} in this checkout, so the range may start earlier than the true gap.", + file=sys.stderr, + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 3cab845..b44662a 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -128,7 +128,7 @@ jobs: name: python-package-distributions path: dist/ - name: Sign the dists with Sigstore - uses: sigstore/gh-action-sigstore-python@v3.0.0 + uses: sigstore/gh-action-sigstore-python@790bc6befb9d733738f18d8f895854b453640ec9 # v3.5.0 with: inputs: >- ./dist/*.tar.gz diff --git a/CHANGELOG.md b/CHANGELOG.md index cd4aa05..b22e916 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +## [v0.7.0] - 2026-08-28 + +### Added + +- **Automation:** New Claude skill (`bump-mthds`) and companion script (`upstream_notes.py`) to automate bumping the `mthds` dependency, regenerating locks, and adapting the codebase to upstream protocol changes. + +### Changed + +- **Dependency:** Pinned `mthds` to an exact version (`mthds==0.11.1`) instead of a floor (`>=0.8.2`), ensuring the SDK and its strict `extra="forbid"` protocol models are always tested against the exact upstream version and preventing runtime parse failures from uncoordinated resolutions. `pipelex` pins the same version, so the two co-install; the two pins must now move in step, because two exact pins on different versions do not resolve at all. (Breaking) +- **Typing:** `PipelexValidationReport.input_form` and `pipe_io_contracts` are now strictly typed via the standard's own client models (`mthds.protocol.input_form.InputForm` and `mthds.protocol.pipe_io_contracts.PipeIOContracts`) rather than opaque dictionaries. As a result, reports with older contracts (e.g. boolean `optional` instead of `presence`, or missing `multiplicity`/`item_count`) no longer parse; the hosted API emits the reshaped contracts and there is intentionally no compatibility shim for older runners. The types are used, never re-exported — `mthds.protocol` stays the one import path for the vocabulary — and `bundle_blueprint` / `graph_spec` stay opaque, since nothing published declares them. (Breaking) +- **Parsing:** List items in input forms now parse into nameless unions (e.g. `DocumentItem` instead of `DocumentField`), so code narrowing a list's item must target the item layer (the named layer silently fails `isinstance` checks). Input-form parsing is also tightened to reject contradictory `required`/`presence` combinations, `gating` on optional slots, and explicit `null`s on wire slots (except `default_value`). (Breaking) +- **Strictness:** The imported artifacts are closed shapes, but the report envelope around them stays extension-open — an unrelated field a future server adds to the report still parses and still rides `model_extra`. The two regimes nest rather than spread, and a test pins both halves. +- **Linting:** Updated Ruff to include `mthds` models (`ValidationReport`, `InvalidValidationReport`, `ValidationDiagnostic`) in `runtime-evaluated-base-classes`, preventing Pydantic resolution errors from annotations mistakenly moved into `TYPE_CHECKING` blocks. +- **Documentation:** Updated `README.md` and `docs/architecture.md` to reflect the move from opaque dictionaries to typed MTHDS imports, detailing strictness boundaries and narrowing strategies, and `docs/ci-cd.md` to record that third-party actions are allowlisted at the enterprise level by exact commit SHA. + +### Fixed + +- **Serialization:** Generating a serialization-mode JSON Schema from `PipelexValidationReport` now outputs the real input-form field shapes instead of an opaque object (resolved via the bump to `mthds` 0.11.1). +- **CI/CD:** Fixed the GitHub Actions publish workflow by pinning `sigstore/gh-action-sigstore-python` to an enterprise-allowlisted SHA for v3.5.0 (`790bc6befb9d733738f18d8f895854b453640ec9`), resolving a deterministic `UnsignedMetadataError` caused by a Sigstore TUF trust-root rotation that broke the previous `v3.0.0` tag. + ## [v0.6.0] - 2026-08-25 ### Added diff --git a/README.md b/README.md index c5ef348..2260e8c 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ There is no barrel import — package `__init__.py` files stay empty. Import eac - **Typed errors** — `from pipelex_sdk.errors import ApiResponseError, ApiUnreachableError, PipelineExecuteTimeoutError, PagingNotTerminatingError, RunFailedError, RunTimeoutError, RunLifecycleUnavailableError, RunStillRunningError, ...` - **Version** — `from pipelex_sdk.version import __version__` - **Protocol surface** (the MTHDS standard's wire types) comes from the `mthds` dependency — e.g. `from mthds.protocol.exceptions import PipelineRequestError`, `from mthds.protocol.models import ValidationResult` (the neutral verdict union that `PipelexValidationResult` narrows). +- **Input-form descriptors and pipe I/O contracts** come from `mthds` too, because they are the standard's artifacts and this SDK only carries them: `from mthds.protocol.input_form import InputForm, InputFormField, ListField, TextField, ...` and `from mthds.protocol.pipe_io_contracts import PipeIOContracts, PipeInputContract, PresenceMarker, IOMultiplicity, ...`. `PipelexValidationReport.input_form` and `.pipe_io_contracts` are typed with them, so a node narrows on its `kind` and a slot's presence and multiplicity read as enums — but `pipelex_sdk` does not re-export the vocabulary, and importing it from here is the one supported path. ## Development diff --git a/TODOS.md b/TODOS.md index f66ca0d..6dea645 100644 --- a/TODOS.md +++ b/TODOS.md @@ -2,6 +2,8 @@ This is the implementation tracker for the design in [`wip/updates.md`](wip/updates.md). The design answers *what* and *why*; this file is the *how*, broken into phases with checkboxes. Tick a box when the item is done and verified, not when it is started. Every design choice that was open has been decided (see `wip/updates.md` §7) and is treated here as settled: `input_form` stays opaque, `MethodData.python` is a typed `list[MethodFile]` with the converter in this repo, the `method_id` type guard lands now, and an unknown `FixOp.kind` raises. +One of those settled choices has since been superseded: `input_form` is no longer opaque, and neither is `pipe_io_contracts` — both are typed by importing the standard's client models now that `mthds` publishes them. The record of that change, and why it honours rather than overrides the ownership argument the opaque ruling rested on, is [`wip/input-form-typed-narrowing.md`](wip/input-form-typed-narrowing.md). Everything else below stands as written. + Ground rules for every phase, from `CLAUDE.md`: - Branch: `feature/Typed-method-id-run-option` (already carries the typed `method_id` option and the `delete_method` contract fix). The PR targets `dev`. diff --git a/docs/architecture.md b/docs/architecture.md index 5002ab2..d000f34 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -31,6 +31,8 @@ MTHDS is the brand of the open standard (the language, the protocol). Pipelex is The Pipelex narrowing of the `/v1/validate` verdict union is one such implementation envelope and lives here (`pipelex_sdk.validation_models`): `PipelexValidationReport` / `PipelexInvalidReport` / the `PipelexValidationResult` union, plus the supporting `ValidationErrorItem` / `ValidationErrorCategory` / `ValidatedPipeEntry` / `DryRunStatus` / `LiftablePipeEntry` / `SuggestedFix` / the `FixOp` variants / `FixOpKind` / `FixSafety`. They narrow the neutral `ValidationReport` / `InvalidValidationReport` / `ValidationResult` bases that `mthds` keeps (in `mthds.protocol.models`). The report/union types carry the `Pipelex` prefix; the supporting types stay neutrally named — branding the envelope, not the fields inside it. The brand-neutral `Dict*` wire concretes (`DictRunResultExecute` and friends) stay in `mthds` — they are a shared wire contract the `pipelex` runtime itself builds on — and this SDK reuses them by inheritance rather than redefining them (a deliberate divergence from `pipelex-sdk-js`, which duplicates both the `Dict*` and the `Pipelex*` types in its own `models.ts`). +The same boundary decides two members *inside* the Pipelex envelope. The input-form descriptor and the pipe I/O contracts are MTHDS artifacts — the standard's own recommended extension fields of the validate report, each with a normative page — so this SDK types them by importing `mthds.protocol.input_form` and `mthds.protocol.pipe_io_contracts` rather than declaring them, and does not re-export them under its own name. A Pipelex-branded envelope may carry a neutral artifact; it may not adopt it. See "Typed by import" below. + ## Credentials & configuration Resolved at construction time, Pipelex-only — the SDK **never** consults the `mthds` resolver (`MTHDS_API_KEY` / `MTHDS_BASE_URL`, `~/.mthds/config`). That config stores a `(base_url, api_key)` credential pair for whatever runner the vendor-neutral `mthds` tooling targets; borrowing the key while ignoring the URL would send a credential configured for another runner to the hosted API (and `mthds`'s base-URL default, a local bare runner on `http://localhost:8081`, would preempt the hosted default). Both chains match the JS SDK exactly: @@ -139,11 +141,26 @@ The override reuses the inherited base transport seam `_post_validate` (which bu **Checkpoint-5 decision (validate error regime):** the delegation keeps the inherited `httpx.HTTPStatusError` regime on a *no-verdict* non-2xx, where the JS `validate` raises `ApiResponseError`. Kept as-is (deferred parity), because in both SDKs `validate`'s error regime matches the *other* protocol routes of that SDK — JS routes all raise `ApiResponseError`, Python protocol routes all inherit `httpx.HTTPStatusError` (decision #5). Making Python's `validate` alone raise `ApiResponseError` would make it inconsistent with `execute`/`start`/`models`/`version`, which is worse than the JS divergence. The verdict itself (valid/invalid) is always a 200 either way — only the no-verdict failure *presentation* differs. -**What the report carries (0.17+).** A valid `PipelexValidationReport` adds three typed fields. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: dict[str, Any] | None` carries the per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`; it is present only when the request named the `input_form` view, and it is kept **opaque** for the same reason as `bundle_blueprint` / `pipe_io_contracts` / `graph_spec` — the descriptor vocabulary is owned elsewhere and a second copy here would be free to drift. The two lists default empty and `input_form` defaults `None`, so a body from an older runner still parses; the empty default is also what a clean bundle yields, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them: `warnings` and `input_form` derive from a crate that was never assembled. +**What the report carries.** A valid `PipelexValidationReport` adds three typed fields beyond the protocol base. `warnings: list[ValidationErrorItem]` are advisory lints on a bundle that is nonetheless valid — the same item type as `validation_errors[]`, so one parser serves both channels, but they never flip `is_valid` (this is where the `hint_*` error types ride). `liftable_pipes: list[LiftablePipeEntry]` inventories the pipes the runtime may skip when an optional slot resolves absent. `input_form: InputForm | None` carries the per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`, and is present only when the request named the `input_form` view. The two lists default empty and `input_form` defaults `None`, so a body from an older runner still parses; the empty default is also what a clean bundle yields, so no caller can tell the two apart. `PipelexInvalidReport` gains none of them: `warnings` and `input_form` derive from a crate that was never assembled. `ValidationErrorItem` gains `missing_pipe_code` (symmetrical with `missing_concept_code`) and `suggested_fix: SuggestedFix | None` — a deterministic repair proposal with a `fix_code`, a `description`, a `FixSafety` (`safe` / `unsafe`, with an `is_safe` property), an optional `source`, and `ops`: a list discriminated on `kind` over the closed `FixOpKind` vocabulary (`set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`, `move_key`, `remap_value`), narrowed with an exhaustive `match op: case SetKeyOp(): …`. The ops are **reader** models here: `extra="allow"`, no `frozen`, none of the runtime's wildcard-refusing validators, because this SDK only reads fixes where the runtime plans them. A `kind` this SDK does not know fails the whole verdict parse, deliberately and consistently with `ValidationErrorCategory`. `error_type` stays an open `str`: the runtime union keeps gaining advisory members, and closing it here would turn every runtime addition into an SDK break. -One movement in the same release reaches no typed field here and is worth knowing anyway, because it changes what a consumer reads out of the opaque dicts: `PipeInputContract.optional` became `presence`, and the `fixed` multiplicity now carries an `item_count`. Both live inside `pipe_io_contracts`, which this SDK carries as `dict[str, Any]` on purpose — nobody should discover the new spellings by surprise. +### Typed by import: the descriptor and the pipe I/O contracts + +Two members of the valid arm are the **standard's** artifacts rather than Pipelex's, and this SDK narrows them by importing the standard's own client models instead of restating their shape: + +- **`pipe_io_contracts: PipeIOContracts`** — `dict[pipe_ref, PipeIOContract]` from `mthds.protocol.pipe_io_contracts`. An input slot reads as typed members: `concept_ref`, a three-valued `presence` (`PresenceMarker`, so an authored `!` is not flattened into a boolean), a `multiplicity` (`IOMultiplicity`), the `item_count` that is non-null exactly on the fixed arm, and the slot's `json_schema`. The output side is deliberately asymmetric — a two-valued `optional` and no schema — because `!` is rejected on an output and the payload a run produces is the run's own result. +- **`input_form: InputForm | None`** — `dict[pipe_ref, PipeInputFormDescriptor]` from `mthds.protocol.input_form`. A descriptor's `fields` are the recursive `InputFormField` union discriminated on `kind`; narrow a node with `match node: case ListField(): …` or an `isinstance` check, importing the per-kind models from `mthds.protocol.input_form`. An `object` node recurses through `fields`, a `list` node through `item` — and the item changes layer. Since `mthds` v0.10.0 the union is split by whether a node names itself: a top-level field is the named union (`TextField`, `DocumentField`, …, each requiring `name: str`), a `ListField.item` is the nameless one (`TextItem`, `DocumentItem`, …), which refuses a `name` at the parse. Narrow a list's item to `DocumentItem`, never `DocumentField`; because each `*Field` derives from its `*Item`, the item layer is the only safe narrowing target in that position. + +**Why import rather than declare.** These artifacts belong to MTHDS: they describe a method's inputs, which is a language-level fact, and any engine derives them from a resolved library with no Pipelex API in the loop. They were carried opaquely until now for a reason that has since expired — when that call was made, no published Python package declared them, so "type it here" could only mean "copy it here", and a copy is free to drift from the runtime that emits it. Since `mthds` 0.9.0 the standard's own client declares both, so typing them means importing them: one declaration per language, nothing to drift from. The principle the opaque ruling was protecting — this SDK is transport and does not own these types — is what an import preserves and a restatement would have broken. + +The types are imported and used, never re-exported from `pipelex_sdk`. Re-exporting would put this package's name on a vocabulary it does not own and hand consumers a second import path to drift against; import them from `mthds.protocol` directly. + +**`bundle_blueprint` and `graph_spec` stay opaque**, for exactly the reason that used to cover all four: no published package declares them, so a type here could only be a copy. When either gets a standard page and a client model, it moves the same way. + +**Strictness composes; it does not spread.** The imported artifacts are **closed** shapes (`extra="forbid"`): a member this `mthds` version does not define is version drift and fails the parse, where catching it is cheap. The report envelope around them stays **extension-open** — `PipelexValidationReport` inherits `extra="allow"` from `mthds`'s `ValidationReport`, and declaring typed fields on a subclass does not touch that config — so an unrelated field a future server adds to the report still parses and still rides `model_extra`, exactly as before. That is the standard's own arrangement: the report is the envelope and grows, the artifact is the view of one version and does not. A test pins both halves, so a later edit cannot quietly close the envelope. + +**The one break.** A valid report whose contracts predate the presence/multiplicity reshape — an input carrying the boolean `optional` instead of `presence`, or missing `multiplicity` / `item_count` — no longer parses, where it used to ride through untyped. The hosted plane emits the reshaped contracts, so this bites only against runners older than that reshape, and there is no compatibility shim by design: an artifact that does not conform to the standard version this package pins is version drift, and saying so at the parse is the point. ## Pipelex product surface (hosted management routes) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index f661c09..f7442bd 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -28,6 +28,7 @@ The lint and test matrices use the repo `Makefile` targets, which honor `PYTHON_ - **PyPI Trusted Publishing**: register `Pipelex/pipelex-sdk-python` as a trusted publisher for the `pipelex-sdk` project, environment `pypi`. No API token secret is needed. - **CLA secrets** (org-level, shared with the other `Pipelex` Python repos): `CLA_GH_APP_ID`, `CLA_GH_APP_PRIVATE_KEY`. The GitHub App must have access to `cla-signatures` and this repo. +- **Actions allowlist**: third-party actions are allowlisted at the *enterprise* level, above both the `Pipelex` and `mthds-ai` organizations, and the allowlist keys on the exact commit SHA. `sigstore/gh-action-sigstore-python` is allowlisted at `790bc6befb9d733738f18d8f895854b453640ec9` (v3.5.0), which is why the publish workflow pins that SHA rather than a tag. Moving it to any other version needs an enterprise admin to add the new SHA first, or the `github-release` job fails before it runs. ## Release flow (summary) diff --git a/pipelex_sdk/client.py b/pipelex_sdk/client.py index 11eafb2..e14c1e6 100644 --- a/pipelex_sdk/client.py +++ b/pipelex_sdk/client.py @@ -503,7 +503,10 @@ async def validate( # type: ignore[override] The 200-diagnostic union: `PipelexValidationReport` (`is_valid: true`) or `PipelexInvalidReport` (`is_valid: false`, with `validation_errors`), each carrying `rendered_markdown`. A valid report also carries `warnings` and - `liftable_pipes`, plus `input_form` when `views` asked for it. + `liftable_pipes`, plus `input_form` when `views` asked for it. `input_form` and + `pipe_io_contracts` are typed by the standard's own models (`mthds.protocol`), + so a field descriptor narrows on its `kind` and a slot's presence and multiplicity + read as enums; import the per-kind types from `mthds.protocol.input_form`. """ extra: dict[str, Any] = {"render": _with_validate_markdown_render(render)} if mthds_sources is not None: diff --git a/pipelex_sdk/validation_models.py b/pipelex_sdk/validation_models.py index 63de72b..96c275f 100644 --- a/pipelex_sdk/validation_models.py +++ b/pipelex_sdk/validation_models.py @@ -15,6 +15,24 @@ not the field names inside it. Fixes and lints are language-level concepts, and the runtime names them brand-neutrally too. +Two members of the valid arm are deliberately **not** Pipelex's to declare. `pipe_io_contracts` +and `input_form` are the standard's own recommended extension fields of the validate report, each +with a normative page and a client model since `mthds` v0.9.0, so they are narrowed here **by +import** — `PipeIOContracts` from `mthds.protocol.pipe_io_contracts`, `InputForm` from +`mthds.protocol.input_form`. That keeps the "this SDK is transport, it does not own these types" +principle intact while the payloads stop being opaque: there is one declaration per language, and +an import cannot drift from it the way a restatement could. The types are used, never re-exported — +a consumer that wants to name a node's type (`ListField`, `PresenceMarker`, …) imports it from +`mthds.protocol` directly, where it belongs. `bundle_blueprint` and `graph_spec` stay opaque for +the reason that used to cover all four: no published package declares them, so a type here could +only be a copy. + +Strictness composes rather than spreads. The imported artifacts are **closed** shapes +(`extra="forbid"`) — a member this `mthds` version does not define is version drift, refused at the +parse — while the report envelope around them stays extension-open (`extra="allow"`, inherited from +`ValidationReport`), so an unrelated field a future server adds to the report still parses and still +rides `model_extra`. + `PipelexAPIClient.validate()` returns this `PipelexValidationResult` (parsed via `PipelexValidationResultAdapter`); the protocol base `MthdsAPIClient.validate()` returns the neutral `mthds` `ValidationResult`. @@ -25,7 +43,9 @@ from enum import StrEnum from typing import Annotated, Any, Final, Literal, TypeAlias +from mthds.protocol.input_form import InputForm from mthds.protocol.models import InvalidValidationReport, ValidationDiagnostic, ValidationReport +from mthds.protocol.pipe_io_contracts import PipeIOContracts from pydantic import BaseModel, ConfigDict, Field, TypeAdapter from pipelex_sdk._pydantic_utils import empty_list_factory_of @@ -276,8 +296,31 @@ class PipelexValidationReport(ValidationReport): """The valid arm narrowed with pipelex's structural artifacts (`is_valid: true`).""" bundle_blueprint: dict[str, Any] = Field(default_factory=dict) - pipe_io_contracts: dict[str, Any] = Field(default_factory=dict) + """The parsed bundle, carried opaquely: no published package declares its shape, so a type + here could only be a copy free to drift from the runtime that emits it.""" + + pipe_io_contracts: PipeIOContracts = Field(default_factory=dict) + """The per-pipe I/O contracts, typed by importing the standard's own client models. + + `PipeIOContracts` is `dict[pipe_ref, PipeIOContract]` (`mthds.protocol.pipe_io_contracts`), so a + declared input slot reads as typed members — `concept_ref`, a three-valued `presence` + (`PresenceMarker`), a `multiplicity` (`IOMultiplicity`), the `item_count` that is non-null exactly + on the fixed arm, and its `json_schema` — and the output side reads its own asymmetric shape + (a two-valued `optional`, because `!` is rejected on an output). The artifact belongs to the + standard, so it is imported rather than restated: one declaration per language is what makes + drift impossible, which is precisely what keeping it opaque used to buy. + + Contracts are **closed** shapes: a member this `mthds` version does not define is version drift + and fails the parse. That closure is scoped to the artifact — the report around it stays + extension-open — and it is the reason a contract from a runner predating the presence/multiplicity + reshape no longer parses. + + Defaults to an empty map rather than `None`: the Pipelex valid arm always states the artifact, so + no caller has to test for its absence.""" + graph_spec: Any = None + """The execution graph, carried opaquely for the same reason as `bundle_blueprint`.""" + validated_pipes: list[ValidatedPipeEntry] = Field(default_factory=empty_list_factory_of(ValidatedPipeEntry)) pending_signatures: list[str] = Field(default_factory=list) is_runnable: bool = True @@ -295,14 +338,26 @@ class PipelexValidationReport(ValidationReport): Defaults empty for the same reason as `warnings`: an older runner's body must keep parsing.""" - input_form: dict[str, Any] | None = None - """Per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`. + input_form: InputForm | None = None + """Per-pipe input-form descriptors, keyed exactly like `pipe_io_contracts`, typed by importing + the standard's own client models. + + `InputForm` is `dict[pipe_ref, PipeInputFormDescriptor]` (`mthds.protocol.input_form`), whose + `fields` are the recursive `InputFormField` union discriminated on `kind`: narrow a node with + `match node: case ListField(): ...` or an `isinstance` check, importing the per-kind models from + `mthds.protocol.input_form`. Imported rather than restated, for the same reason as the contracts, + and closed the same way. + + The recursion changes layer, and a consumer narrowing it has to follow. Since `mthds` v0.10.0 the + union is split in two by whether the node names itself: a top-level field is the **named** union + (`TextField`, `DocumentField`, …, each requiring `name: str`), while a `ListField.item` is the + **nameless** counterpart (`TextItem`, `DocumentItem`, …), which refuses a `name` at the parse. So + a list's item narrows to `DocumentItem`, never `DocumentField` — and since each `*Field` derives + from its `*Item`, only the item layer is a safe narrowing target at that position. Optional on purpose: it is present only when the request named the `input_form` view - (`VALIDATION_VIEW_INPUT_FORM`), and an older runner emitted it unconditionally — `None` - by default is the one typing that reads a body from either runner correctly. Kept opaque - like `bundle_blueprint`, `pipe_io_contracts` and `graph_spec`, because the descriptor - vocabulary is owned elsewhere and a second copy here would be free to drift.""" + (`VALIDATION_VIEW_INPUT_FORM`), and an older runner emitted it unconditionally — `None` by + default is the one typing that reads a body from either runner correctly.""" rendered_markdown: str | None = None """Opt-in Pipelex-API presentation extra: the server-rendered Markdown view of the verdict, diff --git a/pyproject.toml b/pyproject.toml index a69d19a..87bb981 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "pipelex-sdk" -version = "0.6.0" +version = "0.7.0" description = "The Python client for the Pipelex hosted API — the MTHDS Protocol surface plus the durable run lifecycle and the Pipelex product surface, built on the `mthds` protocol base." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] @@ -18,7 +18,7 @@ classifiers = [ ] dependencies = [ - "mthds>=0.8.2", + "mthds==0.11.1", "pydantic>=2.10.6,<3.0.0", "typing-extensions>=4.0.0", "httpx>=0.23.0,<1.0.0", @@ -190,121 +190,121 @@ target-version = "py311" preview = true select = ["ALL"] ignore = [ - "missing-return-type-undocumented-public-function", # Missing return type annotation for public function `my_func` - "missing-return-type-private-function", # Missing return type annotation for private function `my_func` - "missing-return-type-special-method", # Missing return type annotation for special method `my_func` - "missing-return-type-class-method", # Missing return type annotation for classmethod `my_func` - "any-type", # Dynamically typed expressions (typing.Any) are disallowed in `...` - "blocking-open-call-in-async-function", # Async functions should not open files with blocking methods like `open` - "blocking-path-method-in-async-function", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path + "missing-return-type-undocumented-public-function", # Missing return type annotation for public function `my_func` + "missing-return-type-private-function", # Missing return type annotation for private function `my_func` + "missing-return-type-special-method", # Missing return type annotation for special method `my_func` + "missing-return-type-class-method", # Missing return type annotation for classmethod `my_func` + "any-type", # Dynamically typed expressions (typing.Any) are disallowed in `...` + "blocking-open-call-in-async-function", # Async functions should not open files with blocking methods like `open` + "blocking-path-method-in-async-function", # Async functions should not use pathlib.Path methods, use trio.Path or anyio.path "class-as-data-structure", # Class could be dataclass or namedtuple - "complex-structure", # Is to complex + "complex-structure", # Is to complex "missing-trailing-comma", # Checks for the absence of trailing commas. "missing-copyright-notice", # Missing copyright notice at top of file - "undocumented-public-module", # Missing docstring in public module - "undocumented-public-class", # Missing docstring in public class - "undocumented-public-method", # Missing docstring in public method - "undocumented-public-function", # Missing docstring in public function - "undocumented-public-package", # Missing docstring in public package - "undocumented-magic-method", # Missing docstring in magic method - "undocumented-public-init", # Missing docstring in __init__ + "undocumented-public-module", # Missing docstring in public module + "undocumented-public-class", # Missing docstring in public class + "undocumented-public-method", # Missing docstring in public method + "undocumented-public-function", # Missing docstring in public function + "undocumented-public-package", # Missing docstring in public package + "undocumented-magic-method", # Missing docstring in magic method + "undocumented-public-init", # Missing docstring in __init__ "missing-blank-line-after-summary", # 1 blank line required between summary line and description - "missing-trailing-period", # First line should end with a period - "non-imperative-mood", # First line of docstring should be in imperative mood: "My docstring...." - "docstring-starts-with-this", # First word of the docstring should not be "This" - "missing-terminal-punctuation", # First line should end with a period, question mark, or exclamation point - - "docstring-missing-returns", # `return` is not documented in docstring - "docstring-extraneous-returns", # Docstring should not have a returns section because the function doesn't return anything - "docstring-missing-yields", # `yield` is not documented in docstring + "missing-trailing-period", # First line should end with a period + "non-imperative-mood", # First line of docstring should be in imperative mood: "My docstring...." + "docstring-starts-with-this", # First word of the docstring should not be "This" + "missing-terminal-punctuation", # First line should end with a period, question mark, or exclamation point + + "docstring-missing-returns", # `return` is not documented in docstring + "docstring-extraneous-returns", # Docstring should not have a returns section because the function doesn't return anything + "docstring-missing-yields", # `yield` is not documented in docstring "docstring-extraneous-exception", # Raised exception is not explicitly raised: `FileNotFoundError` - "docstring-missing-exception", # Raised exception `ModuleFileError` missing from docstring + "docstring-missing-exception", # Raised exception `ModuleFileError` missing from docstring - "call-datetime-without-tzinfo", # `datetime.datetime()` called without a `tzinfo` argument + "call-datetime-without-tzinfo", # `datetime.datetime()` called without a `tzinfo` argument "call-datetime-now-without-tzinfo", # `datetime.datetime.now()` called without a `tz` argument "commented-out-code", # Found commented-out code - "boolean-type-hint-positional-argument", # Boolean-typed positional argument in function definition + "boolean-type-hint-positional-argument", # Boolean-typed positional argument in function definition "boolean-default-value-positional-argument", # Boolean default positional argument in function definition - "boolean-positional-value-in-call", #Boolean positional value in function call + "boolean-positional-value-in-call", #Boolean positional value in function call "line-contains-todo", # Line contains TODO, consider resolving the issue "read-whole-file", # `open` and `read` should be replaced by `Path(file_path.path).read_text(encoding="utf-8")` "repeated-append", # Checks for consecutive calls to append. - "math-constant", # Checks for literals that are similar to constants in math module. + "math-constant", # Checks for literals that are similar to constants in math module. "log-exception-outside-except-handler", # `.exception()` call outside exception handlers "type-name-incorrect-variance", # `TypeVar` name "SomethingType" does not reflect its covariance; consider renaming it to "SomethingType_co" - "compare-to-empty-string", # Checks for comparisons to empty strings. - - "too-many-public-methods", # Too many public methods ( > 20) - "too-many-return-statements", # Too many return statements (/6) - "too-many-branches", # Too many branches (/12) - "too-many-arguments", # Too many arguments in function definition (/5) - "too-many-locals", # Too many local variables ( /15) - "too-many-statements", # Too many statements (/50) + "compare-to-empty-string", # Checks for comparisons to empty strings. + + "too-many-public-methods", # Too many public methods ( > 20) + "too-many-return-statements", # Too many return statements (/6) + "too-many-branches", # Too many branches (/12) + "too-many-arguments", # Too many arguments in function definition (/5) + "too-many-locals", # Too many local variables ( /15) + "too-many-statements", # Too many statements (/50) "too-many-positional-arguments", # Too many positional arguments ( /5) - "magic-value-comparison", # Magic value used in comparison, consider replacing `2` with a constant variable - "no-self-use", # Too many return statements in `for` loop - "too-many-nested-blocks", # Too many nested blocks ( > 5) + "magic-value-comparison", # Magic value used in comparison, consider replacing `2` with a constant variable + "no-self-use", # Too many return statements in `for` loop + "too-many-nested-blocks", # Too many nested blocks ( > 5) "pytest-incorrect-pytest-import", # Incorrect import of `pytest`; use `import pytest` instead - "os-path-abspath", # `os.path.abspath()` should be replaced by `Path.resolve()` - "os-makedirs", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` - "os-remove", # `os.remove()` should be replaced by `Path.unlink()` - "os-getcwd", # `os.getcwd()` should be replaced by `Path.cwd()` - "os-path-join", # `os.path.join()` should be replaced by `Path` with `/` operator - "os-path-dirname", # `os.path.dirname()` should be replaced by `Path.parent` - "os-path-exists", # `os.path.exists()` should be replaced by `Path.exists()` - "os-path-isdir", # `os.path.isdir()` should be replaced by `Path.is_dir()` + "os-path-abspath", # `os.path.abspath()` should be replaced by `Path.resolve()` + "os-makedirs", # `os.makedirs()` should be replaced by `Path.mkdir(parents=True)` + "os-remove", # `os.remove()` should be replaced by `Path.unlink()` + "os-getcwd", # `os.getcwd()` should be replaced by `Path.cwd()` + "os-path-join", # `os.path.join()` should be replaced by `Path` with `/` operator + "os-path-dirname", # `os.path.dirname()` should be replaced by `Path.parent` + "os-path-exists", # `os.path.exists()` should be replaced by `Path.exists()` + "os-path-isdir", # `os.path.isdir()` should be replaced by `Path.is_dir()` "os-path-basename", # `os.path.basename()` should be replaced by `Path.name` - "builtin-open", # `open()` should be replaced by `Path.open()` - "os-listdir", # Use `pathlib.Path.iterdir()` instead. + "builtin-open", # `open()` should be replaced by `Path.open()` + "os-listdir", # Use `pathlib.Path.iterdir()` instead. "redundant-literal-union", # `Literal["auto"]` is redundant in a union with `str` "superfluous-else-return", # superfluous-else-return - "ambiguous-unicode-character-string", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? + "ambiguous-unicode-character-string", # String contains ambiguous `′` (PRIME). Did you mean ``` (GRAVE ACCENT)? "ambiguous-unicode-character-comment", # Comment contains ambiguous `’` (RIGHT SINGLE QUOTATION MARK). Did you mean ``` (GRAVE ACCENT)? - "unsorted-dunder-all", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. + "unsorted-dunder-all", # Checks for __all__ definitions that are not ordered according to an "isort-style" sort. - "suppressible-exception", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` + "suppressible-exception", # Use `contextlib.suppress(ValueError)` instead of `try`-`except`-`pass` "if-else-block-instead-of-if-exp", # Use ternary operator `description = func.__doc__.strip().split("\n")[0] if func.__doc__ else func.__name__` instead of `if`-`else`-block - "assert", # Use of `assert` detected - "exec-builtin", # Use of `exec` detected + "assert", # Use of `assert` detected + "exec-builtin", # Use of `exec` detected "hardcoded-password-func-arg", # Possible hardcoded password assigned to argument: "secret" - "hardcoded-password-string", # Possible hardcoded password assigned to: "child_secret" + "hardcoded-password-string", # Possible hardcoded password assigned to: "child_secret" "suspicious-non-cryptographic-random-usage", # Cryptographically weak pseudo-random number generator "missing-todo-author", # Missing author in TODO; try: `# TODO(): ...` or `# TODO @: ...` - "missing-todo-link", # Missing issue link for this TODO + "missing-todo-link", # Missing issue link for this TODO "print", # `print` found # TODO: stop ignoring these rules - "blind-except", # Do not catch blind exception: `Exception` - "empty-method-without-abstract-decorator", # Checks for empty methods in abstract base classes without an abstract decorator. - "non-pep604-annotation-union", # Use `X | Y` for type annotations - "outdated-version-block", # Version block is outdated for minimum Python version - "collapsible-if", # Use a single `if` statement instead of nested `if` statements - "jinja2-autoescape-false", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. - "raise-within-try", # Abstract `raise` to an inner function - "manual-list-comprehension", # Use a list comprehension to create a transformed list - "redefined-loop-name", # `for` loop variable `line` overwritten by assignment target - "try-consider-else", # Consider moving this statement to an `else` block - "deprecated-import", # `typing.List` is deprecated, use `list` instead - "implicit-return", # Missing explicit `return` at the end of function able to return non-`None` value + "blind-except", # Do not catch blind exception: `Exception` + "empty-method-without-abstract-decorator", # Checks for empty methods in abstract base classes without an abstract decorator. + "non-pep604-annotation-union", # Use `X | Y` for type annotations + "outdated-version-block", # Version block is outdated for minimum Python version + "collapsible-if", # Use a single `if` statement instead of nested `if` statements + "jinja2-autoescape-false", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function. + "raise-within-try", # Abstract `raise` to an inner function + "manual-list-comprehension", # Use a list comprehension to create a transformed list + "redefined-loop-name", # `for` loop variable `line` overwritten by assignment target + "try-consider-else", # Consider moving this statement to an `else` block + "deprecated-import", # `typing.List` is deprecated, use `list` instead + "implicit-return", # Missing explicit `return` at the end of function able to return non-`None` value # Shrinking a `try` clause changes which statements its handlers cover, so satisfying # this rule is an error-handling refactor rather than lint cleanup. Ignored across the @@ -314,22 +314,33 @@ ignore = [ ] [tool.ruff.lint.flake8-type-checking] -runtime-evaluated-base-classes = ["pydantic.BaseModel"] +# Ruff matches only the base classes named in the class statement, so it cannot see that the +# `/v1/validate` narrowings in `validation_models.py` are pydantic models: they extend `mthds`'s +# report/diagnostic models rather than `BaseModel` directly. Without these entries, an annotation +# imported from `mthds.protocol` (the input-form descriptor, the pipe I/O contracts) is misread as +# type-only and pushed into a `TYPE_CHECKING` block, where pydantic cannot resolve it when it +# builds the model — a runtime failure the linter would have introduced. +runtime-evaluated-base-classes = [ + "pydantic.BaseModel", + "mthds.protocol.models.ValidationReport", + "mthds.protocol.models.InvalidValidationReport", + "mthds.protocol.models.ValidationDiagnostic", +] [tool.ruff.lint.pydocstyle] convention = "google" [tool.ruff.lint.per-file-ignores] "tests/**/*.py" = [ - "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) - "private-member-access", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) - "import-private-name", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) - "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional - "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise + "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) + "private-member-access", # Unit tests legitimately probe private transport/error helpers (e.g. _request_product, _request_json) + "import-private-name", # Unit tests legitimately import private module helpers under test (e.g. _parse_error_body) + "unused-method-argument", # Test-double methods match a Protocol signature; an unused param (e.g. a fake build_inputs ignoring `request`) is intentional + "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise ] "examples/**/*.py" = [ "implicit-namespace-package", # Runnable demo scripts, not an importable package - "print", # print() is the whole point of a demo script + "print", # print() is the whole point of a demo script ] [tool.uv] diff --git a/tests/unit/test_validation_contract.py b/tests/unit/test_validation_contract.py index 74789a3..b009a00 100644 --- a/tests/unit/test_validation_contract.py +++ b/tests/unit/test_validation_contract.py @@ -11,6 +11,8 @@ from typing import Any import pytest +from mthds.protocol.input_form import DocumentField, DocumentItem, ListField, TextField +from mthds.protocol.pipe_io_contracts import IOMultiplicity, PresenceMarker from pydantic import ValidationError from pipelex_sdk.validation_models import ( @@ -38,8 +40,28 @@ "bundle_blueprint": {"source": "contracts.mthds", "domain": "legal_contracts"}, "pipe_io_contracts": { "legal_contracts.summarize": { - "inputs": {"contract": {"concept_ref": "legal_contracts.Contract", "json_schema": {}}}, - "output": {"concept_ref": "legal_contracts.Summary", "multiplicity": "single"}, + "inputs": { + "contract": { + "concept_ref": "legal_contracts.Contract", + "presence": "plain", + "multiplicity": "single", + "item_count": None, + "json_schema": {"type": "object"}, + }, + "attachments": { + "concept_ref": "native.Document", + "presence": "plain", + "multiplicity": "variable", + "item_count": None, + "json_schema": {"type": "array", "items": {"type": "object"}}, + }, + }, + "output": { + "concept_ref": "legal_contracts.Summary", + "multiplicity": "single", + "item_count": None, + "optional": False, + }, } }, "validated_pipes": [{"pipe_ref": "legal_contracts.summarize", "status": "SUCCESS"}], @@ -128,7 +150,35 @@ "absence_source": "optional input `profile` of legal_contracts.summarize", } ], - "input_form": {"legal_contracts.summarize": {"fields": [{"name": "contract", "kind": "text"}]}}, + "input_form": { + "legal_contracts.summarize": { + "fields": [ + { + "kind": "text", + "name": "contract", + "title": "Contract", + "concept_ref": "legal_contracts.Contract", + "required": True, + "presence": "plain", + "gating": True, + "max_length": 20000, + }, + { + "kind": "list", + "name": "attachments", + "concept_ref": "native.Document", + "required": True, + "presence": "plain", + # A variable-length list is required yet never gates: the empty list is a + # legitimate value, which is why the wire states gating instead of deriving it. + "gating": False, + # The item states `required` like any node — only `presence` and `gating` are + # pipe-slot facts a nested node must not carry. + "item": {"kind": "document", "concept_ref": "native.Document", "required": True}, + }, + ] + } + }, } # The 0.17+ invalid arm: the new `missing_pipe_code` locator and a structured repair proposal. @@ -160,6 +210,39 @@ } +# The contracts as a runner predating the presence/multiplicity reshape emitted them: a boolean +# `optional` on the input side, no `presence`, no `multiplicity`, no `item_count`. Typing the field +# by import is what makes this body a parse failure rather than an untyped passenger — the one +# behavioural break of the narrowing, and the shape that documents it. +PRE_RESHAPE_CONTRACTS_BODY: dict[str, Any] = { + **VALID_BODY, + "pipe_io_contracts": { + "legal_contracts.summarize": { + "inputs": {"contract": {"concept_ref": "legal_contracts.Contract", "optional": False, "json_schema": {}}}, + "output": {"concept_ref": "legal_contracts.Summary", "multiplicity": "single"}, + } + }, +} + + +def _body_with_contracts(input_contract: dict[str, Any]) -> dict[str, Any]: + """A valid body whose one pipe declares exactly `input_contract` as its single input slot.""" + return { + **VALID_BODY, + "pipe_io_contracts": { + "legal_contracts.summarize": { + "inputs": {"contract": input_contract}, + "output": {"concept_ref": "legal_contracts.Summary", "multiplicity": "single", "item_count": None, "optional": False}, + } + }, + } + + +def _body_with_descriptor_field(field: dict[str, Any]) -> dict[str, Any]: + """A valid body whose one pipe's input form holds exactly `field`.""" + return {**VALID_BODY, "input_form": {"legal_contracts.summarize": {"fields": [field]}}} + + def _parse(body: dict[str, Any]) -> PipelexValidationResult: """Parse a wire body through the real discriminated-union adapter — the exact parse path `PipelexAPIClient.validate()` uses.""" return PipelexValidationResultAdapter.validate_python(body) @@ -191,6 +274,31 @@ def test_valid_arm_carries_typed_artifacts(self) -> None: assert report.validated_pipes[0].status is DryRunStatus.SUCCESS assert report.mthds_contents == [""] + def test_pipe_io_contracts_read_as_the_standards_models(self) -> None: + """The contracts are typed by import: presence, multiplicity and the output asymmetry read as members.""" + report = _parse(VALID_BODY) + assert isinstance(report, PipelexValidationReport) + contract = report.pipe_io_contracts["legal_contracts.summarize"] + + single = contract.inputs["contract"] + assert single.concept_ref == "legal_contracts.Contract" + assert single.presence is PresenceMarker.PLAIN + assert single.presence.is_optional is False + assert single.multiplicity is IOMultiplicity.SINGLE + assert single.multiplicity.is_plural is False + assert single.item_count is None + assert single.json_schema == {"type": "object"} + + plural = contract.inputs["attachments"] + assert plural.multiplicity is IOMultiplicity.VARIABLE + assert plural.multiplicity.is_plural is True + + # The output side is deliberately asymmetric: a two-valued `optional`, no schema. + assert contract.output.concept_ref == "legal_contracts.Summary" + assert contract.output.multiplicity is IOMultiplicity.SINGLE + assert contract.output.item_count is None + assert contract.output.optional is False + def test_invalid_arm_carries_structured_errors_without_artifacts(self) -> None: """The invalid arm carries typed `validation_errors[]` and no structural artifacts.""" report = _parse(INVALID_BODY) @@ -267,9 +375,114 @@ def test_valid_arm_carries_warnings_liftable_pipes_and_input_form(self) -> None: assert liftable.skipped_when_absent == ["profile"] assert liftable.absence_source == "optional input `profile` of legal_contracts.summarize" assert report.input_form is not None - # Keyed exactly like `pipe_io_contracts`, and opaque on purpose. + # Keyed exactly like `pipe_io_contracts` — the same `pipe_ref` set addresses both artifacts. assert set(report.input_form) == set(report.pipe_io_contracts) + def test_input_form_reads_as_the_standards_models(self) -> None: + """The descriptor is typed by import: nodes narrow on `kind`, and the recursion is typed through.""" + report = _parse(VALID_BODY_WITH_VIEWS) + assert isinstance(report, PipelexValidationReport) + assert report.input_form is not None + descriptor = report.input_form["legal_contracts.summarize"] + + text_node, list_node = descriptor.fields + # A node narrows to its per-kind model, which is what carries that kind's own slots. + assert isinstance(text_node, TextField) + assert text_node.name == "contract" + assert text_node.title == "Contract" + assert text_node.required is True + assert text_node.presence is PresenceMarker.PLAIN + assert text_node.gating is True + assert text_node.max_length == 20000 + + assert isinstance(list_node, ListField) + assert list_node.name == "attachments" + assert list_node.required is True + # Required yet non-gating, stated rather than re-derived from `required`. + assert list_node.gating is False + # No `item_count`: the slot is variable-length, not a fixed `[N]`. + assert list_node.item_count is None + # The recursion is typed through: the item is itself a narrowed node — but on the + # nameless layer. A list's item parses into the `*Item` union, never the `*Field` one. + assert isinstance(list_node.item, DocumentItem) + # `DocumentField` is `DocumentItem` plus `name`, so the negative is what pins the split: + # narrowing to the item layer alone would still admit a named node. + assert not isinstance(list_node.item, DocumentField) + assert list_node.item.concept_ref == "native.Document" + # Pipe-slot facts live on the top-level field only, never on a list's item. + assert list_node.item.presence is None + assert list_node.item.gating is None + + def test_typed_artifacts_do_not_close_the_report_envelope(self) -> None: + """An unrelated future extension field on the report still parses and still rides `model_extra`. + + This is the guard that keeps the two strictness regimes composed the way the standard + intends: the imported artifacts are closed shapes, while the report envelope around them + stays extension-open. A future edit that reached for `extra="forbid"` on the report — or a + narrowing that somehow propagated the artifacts' closure outward — fails here. + """ + report = _parse({**VALID_BODY_WITH_VIEWS, "cost_estimate": {"usd": 0.01}, "some_future_view": ["anything"]}) + assert isinstance(report, PipelexValidationReport) + extra = report.model_extra or {} + assert extra["cost_estimate"] == {"usd": 0.01} + assert extra["some_future_view"] == ["anything"] + # And the typed artifacts parsed all the same. + assert report.input_form is not None + assert "legal_contracts.summarize" in report.pipe_io_contracts + + @pytest.mark.parametrize( + "drifted_body", + [ + pytest.param( + _body_with_contracts( + { + "concept_ref": "legal_contracts.Contract", + "presence": "plain", + "multiplicity": "single", + "item_count": None, + "json_schema": {"type": "object"}, + "tolerance": "lenient", + } + ), + id="undefined-member-inside-an-input-contract", + ), + pytest.param( + _body_with_contracts( + { + "concept_ref": "legal_contracts.Contract", + "presence": "plain", + "multiplicity": "fixed", + "item_count": None, + "json_schema": {"type": "array", "items": {"type": "object"}}, + } + ), + id="fixed-multiplicity-missing-its-item-count", + ), + pytest.param( + _body_with_descriptor_field( + {"kind": "text", "name": "contract", "required": True, "presence": "plain", "gating": True, "widget": "textarea"} + ), + id="undefined-member-inside-a-field-descriptor", + ), + pytest.param( + _body_with_descriptor_field({"kind": "text", "name": "contract", "required": True}), + id="top-level-field-stating-no-pipe-slot-facts", + ), + pytest.param(PRE_RESHAPE_CONTRACTS_BODY, id="pre-reshape-contract-carrying-the-boolean-optional"), + ], + ) + def test_artifact_drift_fails_the_parse(self, drifted_body: dict[str, Any]) -> None: + """Inside an artifact, an undefined member or a violated invariant is version drift and is refused. + + Deliberate, and the standard's own rule (both artifacts are closed shapes) rather than this + SDK's invention: the artifact is a view of one version of the standard and does not grow, + where the report is the envelope and does. The pre-reshape case is the one behavioural break + of typing these fields — a runner older than the presence/multiplicity reshape emits an input + contract this package refuses, where it used to ride through untyped. + """ + with pytest.raises(ValidationError): + _parse(drifted_body) + def test_valid_arm_warning_reads_every_explicit_null_as_none(self) -> None: """Every explicitly-null locator on a warning reads as `None`. @@ -291,8 +504,12 @@ def test_valid_arm_warning_reads_every_explicit_null_as_none(self) -> None: assert warning.declared_concepts is None assert warning.suggested_fix is None - def test_pre_0_52_valid_body_still_parses_with_empty_defaults(self) -> None: - """A body from a runner predating the fields parses: both lists empty, `input_form` None.""" + def test_valid_body_without_the_view_fields_parses_with_empty_defaults(self) -> None: + """A verdict that carries none of the opt-in members parses: both lists empty, `input_form` None. + + That is what a caller who never asked for a view reads, and also what a runner predating + those members emits — the defaults make the two indistinguishable, on purpose. + """ report = _parse(VALID_BODY) assert isinstance(report, PipelexValidationReport) assert report.warnings == [] diff --git a/uv.lock b/uv.lock index f25455f..f0ce831 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.8.2" +version = "0.11.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "httpx" }, @@ -221,9 +221,9 @@ dependencies = [ { name = "tomlkit" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/ed/87/9fefbf352f2f2794336ae6fa1f72ea7d7a74ce78524e3bce734bea83aa02/mthds-0.8.2.tar.gz", hash = "sha256:1ac24a415f5a4942e93309066ed6b65a553ca379578e2da93f492bc63342b0b6", size = 130890, upload-time = "2026-08-21T10:49:37.05Z" } +sdist = { url = "https://files.pythonhosted.org/packages/44/9a/93d688320010ad14a31c863c3882fabc2f8637622f69fd8e4aed64a3f2bb/mthds-0.11.1.tar.gz", hash = "sha256:5f52b703835abe9e40a000d76023e2bd29124655c049b1ef9eeab4a8761b6e2c", size = 170790, upload-time = "2026-08-28T11:17:02.146Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/b7/85/051bf842ead495c9126e04a940cbfd3fada2474599b89be27b6d73e73158/mthds-0.8.2-py3-none-any.whl", hash = "sha256:13d314523afc5f774b7f19f65b94ce32e753a617298aae8d6d11315448bbea9f", size = 58167, upload-time = "2026-08-21T10:49:35.804Z" }, + { url = "https://files.pythonhosted.org/packages/e5/1c/74abc1b8b63813c0de69218b003e777cc92a7327d21b90911c10502b2e69/mthds-0.11.1-py3-none-any.whl", hash = "sha256:760cd7d3f7e86b8f962e1d3a4e87e6146b94968b9422ddfc22e006a3fc305ba3", size = 71954, upload-time = "2026-08-28T11:17:00.67Z" }, ] [[package]] @@ -303,7 +303,7 @@ wheels = [ [[package]] name = "pipelex-sdk" -version = "0.6.0" +version = "0.7.0" source = { editable = "." } dependencies = [ { name = "httpx" }, @@ -326,7 +326,7 @@ dev = [ [package.metadata] requires-dist = [ { name = "httpx", specifier = ">=0.23.0,<1.0.0" }, - { name = "mthds", specifier = ">=0.8.2" }, + { name = "mthds", specifier = "==0.11.1" }, { name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" }, { name = "pydantic", specifier = ">=2.10.6,<3.0.0" }, { name = "pylint", marker = "extra == 'dev'", specifier = "==4.0.4" }, diff --git a/wip/input-form-typed-narrowing.md b/wip/input-form-typed-narrowing.md new file mode 100644 index 0000000..1d4477a --- /dev/null +++ b/wip/input-form-typed-narrowing.md @@ -0,0 +1,44 @@ +# Typing the descriptor and the contracts by import (`mthds.protocol`) + +This is this repo's tracker for **Stage 3.4** of the workspace input-form program (`../../wip/input-form/plan.md`), carried by ledger item `L-260826-c9b76b`. The program plan holds the *why* and the sequence; this file holds what the change is here, the decisions taken while making it, and what a reader needs to know afterwards. + +## The instruction + +Stage 3.4 applies decision **D-1**: the wire types of the input-form descriptor and of the pipe I/O contracts belong to the standard's clients — `mthds/protocol` in TypeScript, `mthds.protocol` in Python. Every SDK therefore narrows its opaque field **by import** rather than by restating the shape. Here that means two fields of `PipelexValidationReport` stop being bare mappings and start being the standard's own models, and the `mthds` floor moves to the version that publishes them. + +## What this retires, and why it is not a reversal + +The first program ruled (its D4) that `input_form` stays opaque, and the reason it gave was ownership plus drift: the descriptor vocabulary is owned elsewhere, so a second copy inside this SDK would be free to drift from it. That reasoning was sound and its conclusion is now obsolete, because the premise changed. When D4 was taken, no published Python package declared the descriptor, so "type it here" could only mean "restate it here" — a copy, and therefore drift. Since `mthds` 0.9.0 the standard's own client declares both artifacts, so typing them here means importing them: one declaration per language, nothing to drift from. D-1 supersedes D4 on that basis, and the principle D4 was protecting — this SDK is transport and does not own these types — is exactly what an import preserves and a restatement would have broken. + +## Where the boundary now sits + +Two fields of the valid arm are typed by import: `pipe_io_contracts: PipeIOContracts` (from `mthds.protocol.pipe_io_contracts`) and `input_form: InputForm | None` (from `mthds.protocol.input_form`). Two remain opaque, and for the reason that used to cover all four: `bundle_blueprint` and `graph_spec` have no published declaration to import, so a type here could only be a copy. When one of them gets a standard page and a client model, it moves the same way. + +The types are imported and used, never re-exported from `pipelex_sdk`. Re-exporting them would put this package's name on a vocabulary it does not own and would give consumers a second import path to drift against; a consumer that wants to name a node's type imports it from `mthds.protocol.input_form` directly, which is also how it reaches the per-kind models for narrowing. + +## Strictness: closed artifacts inside an open envelope + +This is the one thing worth getting exactly right, because getting it wrong turns a strictness improvement into a regression. + +The standard's models are **closed** shapes (`extra="forbid"`, decision D-5): a member this version of `mthds` does not define is version drift and fails the parse. The validate report itself is **extension-open** per the protocol's extension policy, and stays that way — `PipelexValidationReport` inherits `model_config = ConfigDict(extra="allow")` from `mthds`'s `ValidationReport`, and declaring two typed fields on a subclass does not touch that config. So the two closures compose the way the standard intends and nest rather than spread: + +- An unrelated field the server adds to the **report** — a new artifact, a cost estimate, another opt-in view — still parses and still rides `model_extra`, exactly as before this change. The `input-form-does-not-close-the-report` test pins that, and it is the regression guard against a future edit that reaches for `extra="forbid"` on the envelope. +- An undefined member **inside** a contract or a field descriptor now fails the parse. That is deliberate, it is the standard's own rule rather than this SDK's invention, and it is scoped to the artifact. + +## The break + +A valid report whose `pipe_io_contracts` predates the reshape — an input contract carrying the boolean `optional` instead of `presence`, or missing `multiplicity` / `item_count` — no longer parses, where before it rode through untyped. The hosted plane emits the reshaped contracts, so this is a break against runners older than that reshape and not against the API this SDK targets. No compatibility shim: an artifact that does not conform to the version of the standard this package pins is version drift, and reporting it at the parse is the whole point of D-5. + +## Checklist + +- [x] `mthds` floor moved to `>=0.9.0` in `pyproject.toml`, lockfile refreshed. +- [x] `pipe_io_contracts` and `input_form` typed by import in `pipelex_sdk/validation_models.py`, with the module docstring stating which members are typed by import, which stay opaque, and why. +- [x] Wire fixtures in `tests/unit/test_validation_contract.py` updated to conformant payloads — they were written for the opaque era and state neither the reshaped contract members nor the descriptor's pipe-slot facts. +- [x] Tests: the artifacts read as typed members; the report stays extension-open around them; drift inside an artifact and a violated cross-field invariant both fail the parse; a pre-reshape contract no longer parses. +- [x] `docs/architecture.md` — the validate section says what is typed and what stays opaque, and the paragraph that told a reader to go read the spellings out of an opaque mapping is retired. +- [x] `README.md` — the import map names where the descriptor and contract types come from. +- [x] `CHANGELOG.md` under `## [Unreleased]`. + +## Release + +None from this item. Decision **D-7** of the program plan: every Stage 3 repo lands on `dev` and records its warrant under `## [Unreleased]`; the versions are cut together at the plan's release cascade (item 4.0), when Stage 4 needs published artifacts. Where the plan or the ledger item says "minor bump" for this item, that means the changelog warrant, not a version cut. diff --git a/wip/updates.md b/wip/updates.md index dd9109c..b0013f1 100644 --- a/wip/updates.md +++ b/wip/updates.md @@ -169,6 +169,8 @@ The version stays where it is under `## [Unreleased]` until `/release` cuts it; The four questions the first draft left open, each answered by Louis on 2026-08-25 with the reasoning that settled it. 1. **`input_form` stays opaque** — `dict[str, Any] | None = None`, matching the JS mirror and the ownership argument in §1.3. A `PipeInputFormDescriptor(fields: list[dict])` shell would type one level and still leave the field vocabulary opaque, which buys little. + + **Superseded, and by its own reasoning.** This answer assumed that typing the descriptor here meant declaring it here, which was true while no published Python package declared it — and a declaration here would have been a copy free to drift, exactly as the ownership argument said. `mthds` 0.9.0 changed the premise by publishing the artifact as a normative page with client models, so the field is now typed **by import** rather than restated, which keeps the ownership argument satisfied instead of overriding it. The same move applies to `pipe_io_contracts`. See [`input-form-typed-narrowing.md`](input-form-typed-narrowing.md). 2. **`MethodData.python` / `MethodWriteInput.python` are typed `list[MethodFile]`, with the converter in this repo.** The question was first posed as "raw wire string plus an inbox request to `mthds-python` for the parser", and the answer to "why do we need a parser at all?" dissolved that framing: the converter is a dozen lines of pydantic, the format is a Pipelex catalog concern rather than an MTHDS protocol one, and there is no second Python consumer to keep in step. Full design in §3.2; no inbox item is filed. 3. **The `method_id` type guard lands now**, in this update (§4). The protocol-argument guards for `pipe_code` / `mthds_contents` still wait for `mthds-python` to ship its Phase 1 and arrive here with the `mthds` floor bump. 4. **An unknown `FixOp.kind` raises** — closed `Literal` discriminator, `pydantic.ValidationError` on the whole verdict parse, consistent with the closed `ValidationErrorCategory`. The lenient catch-all alternative described in §1.4 was considered and not taken.