From 0e2e07b07c2901fa555f10df9a55b6381aa70e81 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 6 Sep 2026 22:45:25 +0200 Subject: [PATCH 01/21] codegen --- CHANGELOG.md | 1 + docs/decisions.md | 6 +- pipelex-codex/skills/pipelex-design/SKILL.md | 2 +- pipelex-codex/skills/pipelex-edit/SKILL.md | 4 +- pipelex-vibe/skills/pipelex-design/SKILL.md | 3 +- pipelex-vibe/skills/pipelex-edit/SKILL.md | 4 +- pipelex/skills/pipelex-design/SKILL.md | 3 +- pipelex/skills/pipelex-edit/SKILL.md | 4 +- templates/skills/pipelex-design/SKILL.md.j2 | 5 +- templates/skills/pipelex-edit/SKILL.md.j2 | 4 +- tests/unit/test_gen_skill_docs.py | 28 ++ wip/pipelex-integrate/brief.md | 70 +++++ wip/pipelex-integrate/design.md | 271 ++++++++++++++++++ wip/pipelex-integrate/plan.md | 158 ++++++++++ .../upstream-dependencies.md | 26 ++ 15 files changed, 569 insertions(+), 20 deletions(-) create mode 100644 wip/pipelex-integrate/brief.md create mode 100644 wip/pipelex-integrate/design.md create mode 100644 wip/pipelex-integrate/plan.md create mode 100644 wip/pipelex-integrate/upstream-dependencies.md diff --git a/CHANGELOG.md b/CHANGELOG.md index b6fb750..4b38d6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Changed +- **`pipelex-design` is model-invocable.** The skill shipped `disable-model-invocation: true` on Claude and Vibe, so it could only be reached by typing `/pipelex-design`. That made `pipelex-edit`'s routing a dead end — it classified a structural change, then had to hand the user a slash command to type and throw away the baseline verdict it had just produced. The flag is gone on every target, the skill's description now carries natural-language triggers ("design a method", "create a pipeline", "add a step", "rewire this pipeline") so the model can actually reach it, and `pipelex-edit` names the affected pipes and invokes `/pipelex-design` directly instead of stopping. The consent gate stays where it belongs: the design run still announces its captured contract in one line before writing anything. - **Tooling:** Pinned `ruff` to an exact `0.16.4`, replacing the `>=0.6.8` floor. The exact pin matches what the Ruff VS Code extension now bundles, which matters because Ruff 0.16 lints `pyproject.toml` itself: the extension syncs the config file to the language server, and a pre-0.16 binary parses it as Python source and paints phantom `invalid-syntax` diagnostics on lines like `requires-python`. A floor let the editor and the CLI resolve to different binaries; an exact pin cannot. This is a dev dependency, so nothing shipped changes, and the upgrade produced no lint findings and no reformatting. ## [0.5.0] - 2026-08-01 diff --git a/docs/decisions.md b/docs/decisions.md index e91e82d..081d530 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -123,9 +123,9 @@ Skill adoption follows the target, not the tool surface: **only `pipelex-inputs` The CLI-era plugin had `mthds-edit` next to `mthds-build`. The port splits that ground along the **contract line** instead of recreating a monolithic edit skill: -- **`pipelex-edit`** is the modification entry point and the model-invocable half — it owns the natural-language triggers ("change this pipe", "rename this concept"). It handles contract-preserving edits itself (prompts, descriptions, model refs, operator settings, mechanical renames) under a baseline-verdict discipline: whole-bundle `mthds_validate` before and after, never edit on a broken baseline, inputs-refresh check when a rename touches the client-facing template. -- **`pipelex-design`** owns structural and contract changes via its "Editing an existing method" re-entry section. Re-entry is complexity-adaptive: validate the baseline first, then edit the smallest coherent region directly when its complete shallow graph and propagated contracts can be understood together; use same-contract signatures and re-refinement for nested, uncertain, cross-module, or staged changes. Contract changes still propagate through parent wiring, concept reshapes still include every field-reading consumer, and organization runs only when re-entry leaves a construction-shaped layout. The skill stays `disable-model-invocation: true` — a design run is a commitment the user opts into explicitly — so `pipelex-edit` routes by *telling the user* to run `/pipelex-design`, never by invoking it. -- **Why the split:** the routing surface and the methodology have different homes. Edit intent must auto-trigger from natural phrases, which design deliberately cannot (explicit-invoke only); the propagating-change discipline (contract identity, concept shapes, backlog draining) must live in exactly one skill or it drifts. The hook is not a substitute for either: its semantic-validation stage is fail-open (skipped without `PIPELEX_API_KEY`), so `pipelex-edit` always takes the whole-bundle MCP verdict as the authoritative check. +- **`pipelex-edit`** is the modification entry point for contract-preserving work — it owns the natural-language triggers ("change this pipe", "rename this concept"). It handles contract-preserving edits itself (prompts, descriptions, model refs, operator settings, mechanical renames) under a baseline-verdict discipline: whole-bundle `mthds_validate` before and after, never edit on a broken baseline, inputs-refresh check when a rename touches the client-facing template. +- **`pipelex-design`** owns structural and contract changes via its "Editing an existing method" re-entry section. Re-entry is complexity-adaptive: validate the baseline first, then edit the smallest coherent region directly when its complete shallow graph and propagated contracts can be understood together; use same-contract signatures and re-refinement for nested, uncertain, cross-module, or staged changes. Contract changes still propagate through parent wiring, concept reshapes still include every field-reading consumer, and organization runs only when re-entry leaves a construction-shaped layout. **The skill is model-invocable (changed 2026-08-29; it shipped `disable-model-invocation: true` through 0.5.0).** The original reasoning — a design run is a commitment the user opts into explicitly — turned the routing into a dead end: `pipelex-edit` classified a structural change correctly and could then only *tell* the user to type `/pipelex-design`, discarding the baseline verdict and classification it had just produced and costing a turn for a handoff the user had already asked for. The consent gate that matters is inside the skill, not on its invocation — it announces the captured contract in one line before writing anything, and infers the construction mode rather than asking. So `pipelex-edit` now names the affected pipes and invokes `/pipelex-design` directly, and the design skill's description carries natural-language triggers ("design a method", "create a pipeline", "add a step", "rewire this pipeline") so the model can reach it without a slash command. Removing the flag alone would have been inert: with a purely descriptive description, nothing would ever have triggered it. +- **Why the split:** the routing surface and the methodology have different homes. Both halves auto-trigger from natural phrases, so the split is no longer about invocability — it is that the cheap contract-preserving path must not drag the whole design methodology behind it, and the propagating-change discipline (contract identity, concept shapes, backlog draining) must live in exactly one skill or it drifts. The hook is not a substitute for either: its semantic-validation stage is fail-open (skipped without `PIPELEX_API_KEY`), so `pipelex-edit` always takes the whole-bundle MCP verdict as the authoritative check. ## MCP tool vs skill naming convention (2026-07-16) diff --git a/pipelex-codex/skills/pipelex-design/SKILL.md b/pipelex-codex/skills/pipelex-design/SKILL.md index 574f682..70daa01 100644 --- a/pipelex-codex/skills/pipelex-design/SKILL.md +++ b/pipelex-codex/skills/pipelex-design/SKILL.md @@ -1,6 +1,6 @@ --- name: pipelex-design -description: Design a MTHDS method bundle top-down with a construction workflow matched to its complexity — build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven stepwise refinement for deep, uncertain, staged, or resumable work. Re-enters existing methods with the same adaptive choice for structural and contract changes. +description: Design a MTHDS method bundle (.mthds files) top-down, contract-first. Use when the user says "design a method", "create a pipeline", "build a .mthds", "write a method that does X", "turn this workflow into MTHDS", "scaffold a method", or asks for a structural or contract change to an existing bundle — "add a step", "rewire this pipeline", "change what this pipe takes or produces", "reshape this concept", "refactor the flow". Construction is complexity-adaptive: a fully understood shallow graph is written directly as a coherent runnable bundle, while deep, uncertain, staged, or resumable work goes through validated signature-driven stepwise refinement. Re-enters existing methods with the same adaptive choice. --- diff --git a/pipelex-codex/skills/pipelex-edit/SKILL.md b/pipelex-codex/skills/pipelex-edit/SKILL.md index a805689..0873f06 100644 --- a/pipelex-codex/skills/pipelex-edit/SKILL.md +++ b/pipelex-codex/skills/pipelex-edit/SKILL.md @@ -9,7 +9,7 @@ description: Edit an existing MTHDS method bundle (.mthds files). Use when the u Modify an existing MTHDS method bundle. There are two classes of change; this skill applies the first and routes the second: - **Contract-preserving edits** (this skill): prompt and instruction text, `description` and `system_prompt` wording, model references, operator settings, and mechanical renames of pipes, concepts, or input variables. The method's structure — which pipes exist, how they wire, what each one takes and produces — stays the same. -- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: tell the user which pipe(s) the change touches and that `/pipelex-design` re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work), and stop. Never attempt a partial structural edit here. +- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: say in one line which pipe(s) the change touches, then invoke `/pipelex-design` — it re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work). Never attempt a partial structural edit here. ## Requirements — the Pipelex MCP tools @@ -52,7 +52,7 @@ Validate the whole bundle **before editing**: call `mthds_validate` with `files` ### Step 3: Classify the change -Check the requested change against the scope split at the top. Structural or contract-changing → route to `/pipelex-design` now, before any files change. Everything else proceeds. +Check the requested change against the scope split at the top. Structural or contract-changing → hand off to `/pipelex-design` now, before any files change. Everything else proceeds. ### Step 4: Apply the edits diff --git a/pipelex-vibe/skills/pipelex-design/SKILL.md b/pipelex-vibe/skills/pipelex-design/SKILL.md index f68133a..68b2a4e 100644 --- a/pipelex-vibe/skills/pipelex-design/SKILL.md +++ b/pipelex-vibe/skills/pipelex-design/SKILL.md @@ -1,7 +1,6 @@ --- name: pipelex-design -description: Design a MTHDS method bundle top-down with a construction workflow matched to its complexity — build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven stepwise refinement for deep, uncertain, staged, or resumable work. Re-enters existing methods with the same adaptive choice for structural and contract changes. -disable-model-invocation: true +description: Design a MTHDS method bundle (.mthds files) top-down, contract-first. Use when the user says "design a method", "create a pipeline", "build a .mthds", "write a method that does X", "turn this workflow into MTHDS", "scaffold a method", or asks for a structural or contract change to an existing bundle — "add a step", "rewire this pipeline", "change what this pipe takes or produces", "reshape this concept", "refactor the flow". Construction is complexity-adaptive: a fully understood shallow graph is written directly as a coherent runnable bundle, while deep, uncertain, staged, or resumable work goes through validated signature-driven stepwise refinement. Re-enters existing methods with the same adaptive choice. --- diff --git a/pipelex-vibe/skills/pipelex-edit/SKILL.md b/pipelex-vibe/skills/pipelex-edit/SKILL.md index 2e03681..e3454b2 100644 --- a/pipelex-vibe/skills/pipelex-edit/SKILL.md +++ b/pipelex-vibe/skills/pipelex-edit/SKILL.md @@ -9,7 +9,7 @@ description: Edit an existing MTHDS method bundle (.mthds files). Use when the u Modify an existing MTHDS method bundle. There are two classes of change; this skill applies the first and routes the second: - **Contract-preserving edits** (this skill): prompt and instruction text, `description` and `system_prompt` wording, model references, operator settings, and mechanical renames of pipes, concepts, or input variables. The method's structure — which pipes exist, how they wire, what each one takes and produces — stays the same. -- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: tell the user which pipe(s) the change touches and that `/pipelex-design` re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work), and stop. Never attempt a partial structural edit here. +- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: say in one line which pipe(s) the change touches, then invoke `/pipelex-design` — it re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work). Never attempt a partial structural edit here. ## Requirements — the Pipelex MCP tools @@ -52,7 +52,7 @@ Validate the whole bundle **before editing**: call `mthds_validate` with `files` ### Step 3: Classify the change -Check the requested change against the scope split at the top. Structural or contract-changing → route to `/pipelex-design` now, before any files change. Everything else proceeds. +Check the requested change against the scope split at the top. Structural or contract-changing → hand off to `/pipelex-design` now, before any files change. Everything else proceeds. ### Step 4: Apply the edits diff --git a/pipelex/skills/pipelex-design/SKILL.md b/pipelex/skills/pipelex-design/SKILL.md index 499b23d..e2a2c35 100644 --- a/pipelex/skills/pipelex-design/SKILL.md +++ b/pipelex/skills/pipelex-design/SKILL.md @@ -1,7 +1,6 @@ --- name: pipelex-design -description: Design a MTHDS method bundle top-down with a construction workflow matched to its complexity — build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven stepwise refinement for deep, uncertain, staged, or resumable work. Re-enters existing methods with the same adaptive choice for structural and contract changes. -disable-model-invocation: true +description: Design a MTHDS method bundle (.mthds files) top-down, contract-first. Use when the user says "design a method", "create a pipeline", "build a .mthds", "write a method that does X", "turn this workflow into MTHDS", "scaffold a method", or asks for a structural or contract change to an existing bundle — "add a step", "rewire this pipeline", "change what this pipe takes or produces", "reshape this concept", "refactor the flow". Construction is complexity-adaptive: a fully understood shallow graph is written directly as a coherent runnable bundle, while deep, uncertain, staged, or resumable work goes through validated signature-driven stepwise refinement. Re-enters existing methods with the same adaptive choice. allowed-tools: - Bash - Read diff --git a/pipelex/skills/pipelex-edit/SKILL.md b/pipelex/skills/pipelex-edit/SKILL.md index 3dc75ea..1ef13e1 100644 --- a/pipelex/skills/pipelex-edit/SKILL.md +++ b/pipelex/skills/pipelex-edit/SKILL.md @@ -18,7 +18,7 @@ allowed-tools: Modify an existing MTHDS method bundle. There are two classes of change; this skill applies the first and routes the second: - **Contract-preserving edits** (this skill): prompt and instruction text, `description` and `system_prompt` wording, model references, operator settings, and mechanical renames of pipes, concepts, or input variables. The method's structure — which pipes exist, how they wire, what each one takes and produces — stays the same. -- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: tell the user which pipe(s) the change touches and that `/pipelex-design` re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work), and stop. Never attempt a partial structural edit here. +- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: say in one line which pipe(s) the change touches, then invoke `/pipelex-design` — it re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work). Never attempt a partial structural edit here. ## Requirements — the Pipelex MCP tools @@ -61,7 +61,7 @@ Validate the whole bundle **before editing**: call `mthds_validate` with `files` ### Step 3: Classify the change -Check the requested change against the scope split at the top. Structural or contract-changing → route to `/pipelex-design` now, before any files change. Everything else proceeds. +Check the requested change against the scope split at the top. Structural or contract-changing → hand off to `/pipelex-design` now, before any files change. Everything else proceeds. ### Step 4: Apply the edits diff --git a/templates/skills/pipelex-design/SKILL.md.j2 b/templates/skills/pipelex-design/SKILL.md.j2 index 4839b19..a51a331 100644 --- a/templates/skills/pipelex-design/SKILL.md.j2 +++ b/templates/skills/pipelex-design/SKILL.md.j2 @@ -1,9 +1,6 @@ --- name: pipelex-design -description: Design a MTHDS method bundle top-down with a construction workflow matched to its complexity — build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven stepwise refinement for deep, uncertain, staged, or resumable work. Re-enters existing methods with the same adaptive choice for structural and contract changes. -{% if platform != "codex" -%} -disable-model-invocation: true -{% endif -%} +description: Design a MTHDS method bundle (.mthds files) top-down, contract-first. Use when the user says "design a method", "create a pipeline", "build a .mthds", "write a method that does X", "turn this workflow into MTHDS", "scaffold a method", or asks for a structural or contract change to an existing bundle — "add a step", "rewire this pipeline", "change what this pipe takes or produces", "reshape this concept", "refactor the flow". Construction is complexity-adaptive: a fully understood shallow graph is written directly as a coherent runnable bundle, while deep, uncertain, staged, or resumable work goes through validated signature-driven stepwise refinement. Re-enters existing methods with the same adaptive choice. {% include "skills/shared/frontmatter.md.j2" %} {%- if platform == "claude" %} - mcp__plugin_pipelex_pipelex__mthds_validate diff --git a/templates/skills/pipelex-edit/SKILL.md.j2 b/templates/skills/pipelex-edit/SKILL.md.j2 index 5615bb5..07225a1 100644 --- a/templates/skills/pipelex-edit/SKILL.md.j2 +++ b/templates/skills/pipelex-edit/SKILL.md.j2 @@ -13,7 +13,7 @@ description: Edit an existing MTHDS method bundle (.mthds files). Use when the u Modify an existing MTHDS method bundle. There are two classes of change; this skill applies the first and routes the second: - **Contract-preserving edits** (this skill): prompt and instruction text, `description` and `system_prompt` wording, model references, operator settings, and mechanical renames of pipes, concepts, or input variables. The method's structure — which pipes exist, how they wire, what each one takes and produces — stays the same. -- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: tell the user which pipe(s) the change touches and that `/pipelex-design` re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work), and stop. Never attempt a partial structural edit here. +- **Structural or contract changes** (route to `/pipelex-design`): adding, removing, or rewiring steps; changing any pipe's `inputs`/`output` beyond a pure rename; reshaping a concept's structure; refactoring the flow. These propagate — the parent wiring, concept shapes, and contracts all move together — so they are design work: say in one line which pipe(s) the change touches, then invoke `/pipelex-design` — it re-enters existing methods adaptively (a direct coherent edit for a fully understood shallow region, or signature-driven reopening for nested, uncertain, cross-module, or staged work). Never attempt a partial structural edit here. ## Requirements — the Pipelex MCP tools @@ -56,7 +56,7 @@ Validate the whole bundle **before editing**: call `mthds_validate` with `files` ### Step 3: Classify the change -Check the requested change against the scope split at the top. Structural or contract-changing → route to `/pipelex-design` now, before any files change. Everything else proceeds. +Check the requested change against the scope split at the top. Structural or contract-changing → hand off to `/pipelex-design` now, before any files change. Everything else proceeds. ### Step 4: Apply the edits diff --git a/tests/unit/test_gen_skill_docs.py b/tests/unit/test_gen_skill_docs.py index d891f25..68be201 100644 --- a/tests/unit/test_gen_skill_docs.py +++ b/tests/unit/test_gen_skill_docs.py @@ -774,6 +774,34 @@ def test_every_platform_renders_the_adaptive_workflow(self, target_name: str) -> assert "{%" not in body assert "{{" not in body + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_design_is_model_invocable_on_every_platform(self, target_name: str) -> None: + """The design skill must stay reachable without a slash command. + + It shipped ``disable-model-invocation: true`` through 0.5.0, which made + ``pipelex-edit``'s structural-change routing a dead end. Both halves of + the fix are pinned: the flag is gone, and the description carries the + natural-language triggers without which removing the flag is inert. + """ + config = load_target_config(self.REPO_ROOT / "targets", target_name) + rendered = render_templates( + self.REPO_ROOT / "templates", + self.REPO_ROOT, + config.template_vars, + include_skills=["pipelex-design"], + target_name=config.name, + ) + body = next(content for path, content in rendered.items() if path.match("skills/pipelex-design/SKILL.md")) + assert "disable-model-invocation" not in body + assert 'Use when the user says "design a method"' in body + assert '"add a step", "rewire this pipeline"' in body + + def test_edit_hands_structural_changes_off_by_invoking_design(self) -> None: + edit = (self.SKILLS / "pipelex-edit" / "SKILL.md.j2").read_text(encoding="utf-8") + assert "then invoke `/pipelex-design`" in edit + assert "hand off to `/pipelex-design` now, before any files change" in edit + assert "and stop. Never attempt a partial structural edit here." not in edit + def test_adjacent_skills_describe_organization_as_conditional(self) -> None: organize = (self.SKILLS / "pipelex-organize" / "SKILL.md.j2").read_text(encoding="utf-8") edit = (self.SKILLS / "pipelex-edit" / "SKILL.md.j2").read_text(encoding="utf-8") diff --git a/wip/pipelex-integrate/brief.md b/wip/pipelex-integrate/brief.md new file mode 100644 index 0000000..d7b28ab --- /dev/null +++ b/wip/pipelex-integrate/brief.md @@ -0,0 +1,70 @@ +--- +status: draft +item: L-260830-344594 +--- + +# Brief: a skill that integrates a method into a Python or TypeScript codebase + +This is a brief, not a design and not a plan. It says what the skill is for, what already exists that the design must not reinvent, and which questions the design session has to settle. Everything below that reads like a decision is a *finding* — a fact about the surrounding system that constrains the design — except where it is explicitly marked as an open question. + +Written from `pipelex-mcp` on 2026-08-30, the day `mthds_codegen` finished review on that repo's `feature/CodegenTool`. The tool is unreleased at the time of writing; the plugin's baked launcher is `npx -y @pipelex/mcp@latest`, so nothing here waits on a version pin moving. + +## The gap + +Every skill this plugin ships today acts on `.mthds` files: explain one, design one, reorganize one, edit one, fill its inputs. None of them touches the codebase that will *call* the method. So the moment a user is happy with a method, the plugin stops helping, and the last mile — turning a validated bundle into typed code an application actually runs — is left to the model's improvisation. + +That last mile is not improvisation-shaped. It is deterministic, it has a known-good shape, and getting it wrong is quiet: hand-written types drift from the bundle with nothing to detect it, a formatter run breaks a trust chain nobody knew was there, two methods generated into one directory report as permanently stale. The work is exactly what a skill is for. + +`mthds_codegen` is what makes it possible now. Before it, a consumer had to build its own harness: `pipelex-starter-js` wrote `scripts/codegen.mts` over `@pipelex/sdk`'s `client.codegen()`, and `pipelex-starter-python` shells out to a `pipelex` CLI that it deliberately does not depend on (`PIPELEX ?= pipelex` in its Makefile, with a comment explaining that the starter talks to the hosted API and the runtime is not its dependency). Both are per-project scaffolding that no third project can reuse. The MCP tool does the same projection from the workshop with nothing but an API key. + +## What the skill does, in one paragraph + +Given a method — local `.mthds` files, a catalog id, or a published address — and a codebase, the skill picks the codegen target that matches the project, writes the generated tree into the project at a location that fits its layout, makes the surrounding tooling leave that tree alone, wires the offline drift check into whatever gate the project already runs, and then writes the call site: a typed function that runs the method through `@pipelex/sdk` or `pipelex-sdk` and parses its output with the generated binder. Re-running it on a project that already has a generated tree is the common case, not the rare one. + +## Read these first + +Ground truth, in the order the design session should read it: + +- `../pipelex-mcp/SPEC.md` → **"Codegen Scope (`mthds_codegen`)"** and its subsection **"The write arm (`output_dir`) — local workshop only"**. The full input and output shapes, the verdict discipline, the error taxonomy, the overwrite rule, the orphan rule, and the size-bounding rule. This is the contract; do not infer it from the tool description. +- `../pipelex-starter-js/docs/codegen.md`. **The whole document is the reference design for a TypeScript integration** — the tree layout, why the generated files are excluded from Prettier and ESLint but not from `tsc`, the split between the keyed regeneration action and the keyless CI check, and the two artifacts the codegen route does not produce. If the design session reads one thing beyond the SPEC, read this. +- `../pipelex-starter-python/Makefile` (the `codegen` and `codegen-check` targets) and its `CLAUDE.md` bullet "The typed models are generated, never hand-written". The Python shape, and the CLI dependency this skill is meant to remove. +- `../pipelex-starter-js/src/actions/runSummarizePdfPipeline.ts` and `src/types/generateImagePipeline.ts`. What a finished call site looks like, and what the hand-written layer *above* a generated binder is for. +- `../docs/specs/pipelex-codegen.md` → "Two axes: what to project and for whom". Why the target enum is what it is, and why there is no `language` alias. + +## Findings the design has to build on + +**The write arm is the whole efficiency argument.** Passing `output_dir` makes the workshop write the tree to disk and withhold every artifact byte from every stream: the structured result carries `path`, `bytes` and `written_to` per file and the summary carries no fenced blocks. Without it, a method with a handful of concepts puts tens of kilobytes through the model's context *twice*, once as `structuredContent.artifacts[].content` and again as fenced Markdown, and the model then re-emits all of it through file writes. The skill should always pass `output_dir`. The plugin only ever declares the local workshop launcher, so the arm is always available to it. + +**The bytes are load-bearing and must not be touched.** Each artifact carries a stamp holding its own content hash, and `codegen.lock` holds the hash of every artifact. A reformat, a re-serialized lock, a trimmed trailing newline — any of them breaks the chain and turns the offline check red. This is why both starters exclude their generated tree from their formatters and linters while keeping it inside the type checker, and why the skill has to make that exclusion itself rather than mentioning it. The concrete edit differs per project (`.prettierignore`, an ESLint flat-config `ignores`, `[tool.ruff] exclude`, Biome, Black), which makes it a detection problem rather than a fixed patch. + +**One directory per method, and it is not a style preference.** After writing, the tool walks the directory and reports any stamped file the new lock does not list as an *orphan*, and it never deletes one. Two methods generated into the same directory therefore report as permanently non-current, by design. The skill must place each method in its own directory and must not offer "clean up the orphans" as advice, because the moment a user has two methods in one place that advice deletes real files. + +**Regeneration overwrites its own output and refuses everything else.** A destination that does not exist is written; a regular file carrying a codegen stamp is overwritten whether or not somebody hand-edited it; anything else — an unstamped file, a symlink whatever it points at, a directory — refuses the entire write with an `input_domain` error naming the file. So pointing `output_dir` at a directory that holds hand-written code fails loudly and leaves the tree byte-identical, which is the behaviour the skill should rely on rather than pre-checking around. + +**The target is required, has no default, and its rule is about audience rather than language.** `ts-zod` emits `types.ts` (zod schemas plus inferred types, depending only on zod) and `binder.ts` (a parse/serialize pair per concept); keep both. `python-pydantic` emits `models.py`, plain `BaseModel`s, for a Python consumer with no Pipelex runtime. `python-structures` emits `structures.py`, runtime `StructuredContent` classes, and is wanted only by a Pipelex host or a `@pipe_func` implementation. The two Python targets differ by audience, not by language, so "this is a Python project" does not pick one. Field keys are wire-native snake_case in every target, TypeScript included. + +**TypeScript gets a keyless drift gate for free; Python does not.** `@pipelex/sdk` exports `runCodegenCheck`, pure hashing with no engine and no network, so a TypeScript project that already depends on the SDK can add a CI check with no new dependency. `pipelex-sdk` (Python) has the `/v1/codegen` wire models and the `codegen()` call but no check at all, so a Python project's only offline gate today is the `pipelex` CLI — the runtime dependency a hosted-API consumer took the SDK to avoid. Filed as [L-260830-4e43cd](http://localhost:4747/i/L-260830-4e43cd) against `pipelex-sdk-python`. **The skill ships with the asymmetry documented rather than waiting on it**, but the design should decide what it tells a Python user in the meantime. + +**The lock signs the artifacts, not their sources.** It answers "has this generated tree been tampered with", not "has the bundle changed since I generated". `pipelex-starter-js` invented a `sources.json` sidecar to close that gap — a SHA-256 per `.mthds` source in the closure, plus a second map for the artifacts it emits that the lock cannot sign. The MCP write arm produces no such sidecar. Whether the skill recreates it, relies on the user regenerating after every bundle edit, or leaves source drift undetected is an open question, and it is the one with the most direct effect on whether the integration stays honest over time. + +## Open questions the design must settle + +1. **Where the skill's responsibility ends.** Generating and placing the types is deterministic. Writing the call site is not: it means reading an unfamiliar codebase and matching its conventions. Does the skill write a complete typed function, a single annotated example the user adapts, or only the types plus instructions? "Properly integrate" argues for the first; the plugin's existing skills all stop at the `.mthds` boundary, so this is the largest new commitment in the proposal and it should be decided deliberately rather than by momentum. + +2. **Which run source to recommend, and whether to guard its drift.** The generated types pin a `crate_fingerprint` from the closure that produced them. A run by `method_id` executes whatever the catalog holds *at run time*, so somebody editing the stored method silently invalidates the committed types with nothing to detect it. A committed bundle regenerated in lockstep, or a `method_ref` pinned at an immutable tag, keeps the two together. This is a real correctness question about the shape the skill produces, not a preference. + +3. **Whether the skill can produce a contracts artifact at all.** `pipelex-starter-js` needs the pipe IO contracts and the wire input-form descriptor to gate its run inputs, and gets them from `/v1/validate` with `views: ["input_form"]`. Through MCP, both ride `_meta`, which is the view-only channel and **never reaches the model's context** — and the workshop has no views. So a workshop agent cannot see them today. Decide whether the skill needs them; if it does, that is a `pipelex-mcp` follow-up to file, not something to work around in a skill. + +4. **Detecting the project, and how far to go on a guess.** Language, package layout, where generated code conventionally lives, which formatter and linter are in play, which CI gate to extend. Some of this is cheap and reliable (a `package.json`, a `pyproject.toml`); some is not (is this a Pipelex host, which decides between the two Python targets). Name the checks the skill runs and what it does when they are inconclusive — the plugin's other skills have a stop-and-ask posture worth matching. + +5. **The second invocation.** Re-running against an existing generated tree is the common case: the user edited the bundle and wants the types refreshed. What the skill re-derives, what it takes from what is already on disk, and what it leaves alone are the difference between a skill that is used once and one that is used weekly. + +6. **The skill's name and its place in the family.** The existing set is `pipelex-explain`, `pipelex-design`, `pipelex-organize`, `pipelex-edit`, `pipelex-inputs` — all verbs on a bundle. This one is a verb on a codebase. Whatever it is called, its description has to make a model reach for it on "use this method in my app", "generate types for this method", "call this from my code", and not on the bundle-authoring phrasings the other skills already claim. + +## Constraints and scope + +- Templates are the source of truth. Edit `templates/skills//SKILL.md.j2`, run `make build`, never touch the generated `pipelex*/` outputs. `templates/skills/pipelex-inputs/SKILL.md.j2` is the closest sibling to model: MCP-backed, multi-step, with Claude-only `allowed-tools` frontmatter guarded by `{% if platform == "claude" %}`. +- The skill renders to all three targets. Tool names differ per host (`mcp__plugin_pipelex_pipelex__mthds_codegen` on Claude, `mcp__pipelex__mthds_codegen` on Codex), which the existing templates already handle by referring to tools generically in prose. +- Add `mthds_codegen` to the tool lists in `CLAUDE.md` ("Key dependency") and `docs/decisions.md` when the skill lands, exactly as `mthds_prepare_inputs` was added. +- Out of scope: anything under `../pipelex-mcp/`. If the skill needs a tool change, file it against that repo rather than working around it here. +- Out of scope: changing either starter. They are the reference shape to learn from, not a deliverable. diff --git a/wip/pipelex-integrate/design.md b/wip/pipelex-integrate/design.md new file mode 100644 index 0000000..be1e09d --- /dev/null +++ b/wip/pipelex-integrate/design.md @@ -0,0 +1,271 @@ +--- +status: active +item: L-260830-344594 +--- + +# Design — `pipelex-integrate`: wire an MTHDS method into a Python or TypeScript codebase + +**Written 2026-08-30**, from the brief beside this file ([`brief.md`](brief.md)), against the sources it names: `pipelex-mcp/SPEC.md` → "Codegen Scope" and "The write arm", `pipelex-starter-js/docs/codegen.md`, `pipelex-starter-python/Makefile` and `docs/codegen.md`, the two starter call sites, and `docs/specs/pipelex-codegen.md`. **Status: draft — awaiting ratification.** The decision boxes at the end are what a ratification answers; the implementation tracker is [`plan.md`](plan.md) and starts after they are answered. Ledger item `L-260830-344594`. File and line references were accurate on the writing date; verify them against the code before implementing. + +Everything the brief lists as a *finding* is taken as a constraint and not re-argued here. This document settles the brief's six open questions and the smaller decisions the implementation needs, in the order a reader of the skill would meet them. + +## 1. What the skill is + +`pipelex-integrate` takes a method — a local `.mthds` bundle, a published address (`method_ref`), or a catalog id (`method_id`) — and a Python or TypeScript codebase, and leaves the codebase able to call the method with types that cannot silently drift from it. Concretely, it: + +1. picks the codegen target that matches the project's language and audience; +2. has the workshop write the generated tree into a dedicated directory per method, through `mthds_codegen`'s write arm, so no artifact byte ever passes through the model; +3. makes the project's formatters and linters leave that tree alone while its type checker keeps covering it; +4. records how the tree was generated in a small sidecar beside the lock, so the next run knows what to refresh and a bundle edit is detectable; +5. wires the offline drift check into the gate the project already runs, where one exists for the language; +6. writes one typed call-site module per method, running the method through `@pipelex/sdk` or `pipelex-sdk` and narrowing its output with the generated binder; +7. verifies the result with the project's own type checker and the drift gate it just installed. + +Re-running it on a project that already carries a generated tree is **refresh mode**, the common case: the bundle changed, the types are regenerated in place, the call site is touched only if the types no longer fit it. + +**What it is not.** It is not a build tool (no watch mode, no build-time hook, no per-project harness — the workshop is the harness), not a runner (it never executes the method; `/pipelex-inputs` prepares inputs and offers a run), and not a design skill (a method that does not validate or is not runnable is sent back to `/pipelex-design`). It does not write tests, wire UI routes or CLI commands, or edit existing business code; it stops at one callable module per method, and a user who wants more says so in the conversation. + +## 2. The shape it produces + +The target state is the reference design both starters converged on, minus the per-project scaffolding the workshop replaces. + +**TypeScript** (`ts-zod`): + +``` +methods//main.mthds # the source of truth, committed (files source only) +src/generated// + types.ts # zod schemas + inferred types — written verbatim by the workshop + binder.ts # parse / serialize — written verbatim by the workshop + codegen.lock # the trust-chain lock — written verbatim by the workshop + sources.json # the skill's sidecar: how this tree was generated (§4.6) +src/pipelex/.ts # the call site: one typed function per method (§4.1) +src/pipelex/client.ts # shared: the PipelexApiClient factory, created once per project +src/pipelex/wireOutput.ts # shared, temporary: the wire-null normalizer (§4.8) +scripts/codegen-check.mjs # the offline gate, copied from the skill's references (§4.5) +``` + +**Python** (`python-pydantic`, or `python-structures` for a Pipelex host): + +``` +methods//main.mthds +/generated/__init__.py # the skill's: makes the trees importable (unstamped, never an artifact) +/generated// + __init__.py # the skill's + models.py # or structures.py — written verbatim by the workshop + codegen.lock # written verbatim by the workshop + sources.json # the sidecar +/pipelex/.py # the call site +``` + +Paths are the defaults; §5 says how the skill adapts them to a project that already has a convention. The directory names are the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python), derived from the bundle's root domain, the address's package name, or the catalog method's name. + +Two artifacts the starter's tree carries are **deliberately absent**: `contracts.ts` (the pipe IO contracts and the input-form descriptor — §4.3) and the starter's `derived` map in `sources.json` (the skill emits no derived artifact the lock cannot sign, so there is nothing to record). + +## 3. The procedure + +The skill is automatic by default, with the same mode rules as `pipelex-inputs` (an explicit user signal wins; a genuinely ambiguous decision pauses for one question). Every MCP call branches on the structured verdict, never on transport, and the `config`-class stop discipline is the plugin's usual one. + +1. **Identify the method and the project.** The method comes from the conversation (a bundle directory, an address, an `mt_…` id, or a name resolved through `mthds_list_methods`). The project root is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the user's working area; a workspace holding several is a question, not a guess. Detection rules are in §5. +2. **Prove the method is integrable.** `mthds_validate` on the selector: `is_valid: true` **and** `is_runnable: true` with no pending signatures. A scaffold with pending signatures has a concept set but cannot run, so integrating it produces a call site that cannot succeed — route to `/pipelex-design` instead. An invalid method carries its `validation_errors[]` to `/pipelex-design` / `/pipelex-edit` the way `pipelex-inputs` does. +3. **Read the pipe's signature.** `mthds_inputs_template` with the same selector and **`explicit: true`** — the one skill call in the plugin that wants the ceremonial envelope, because it needs each input's declared concept ref to type the call site (§4.3, §4.9). Record the resolved `pipe_ref`. The main pipe's output concept and multiplicity come from the bundle for a files source, and from the heuristic in §4.3 otherwise. +4. **Choose the target** (§4.2) and **the destination** (§5). State both in one line before writing anything. +5. **Make the tooling leave the tree alone — before the tree exists.** Add the generated directory to the formatter's and linter's ignore lists per §5, confirm the type checker's include still covers it, and confirm it is not gitignored. This ordering is load-bearing: the first project-wide `format` run after generation would otherwise rewrite the stamps. +6. **Generate.** `mthds_codegen` with the selector, `target`, and `output_dir` expressed **relative to the workshop's working directory** (§4.4). Branch: `is_valid: false` → back to step 2's repair route; `status: "error"` located at `output_dir` → a foreign file or a containment escape, handled per §6; `runtime` mid-write → call again once with the same `output_dir`, as the tool's own hint says. On success, confirm `is_current: true` and an empty `orphans[]`; a non-empty `orphans[]` is reported by name and never cleaned (§4.7). +7. **Write the sidecar** `sources.json` beside the lock (§4.6). +8. **Add the dependencies the generated code needs**, with the project's own package manager: `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen — §4.2). State what is added; interactive mode confirms first. +9. **Write the call site** and its shared helpers (§4.1, §4.8). +10. **Wire the offline gate** into the project's existing aggregate check (§4.5). TypeScript gets the lock check and the source-hash check; Python gets the sidecar only, with the asymmetry stated in the report. +11. **Verify.** Run the project's formatter on the files the skill wrote (never on the generated tree — the exclusion from step 5 is what makes a project-wide run safe), then its type checker, then the drift gate. A failure here is the skill's to fix before it reports. +12. **Report.** What was generated and where, the target and why, the call site's signature, what changed in the tooling config, how to refresh (`/pipelex-integrate` again after a bundle edit), and — for Python — that no offline drift check exists yet and refresh is the guard. + +**Refresh mode** (§4.10) enters at step 1 when a sidecar for the method already exists, skips steps 5, 8 and 10 except to verify they still hold, and touches the call site only if the regenerated types no longer type-check against it. + +## 4. Decisions + +### 4.1 Where the skill's responsibility ends — one callable module per method (brief Q1) + +**Decision: the skill writes a complete, typed, callable function per method — and stops there.** Not an annotated example, not "types plus instructions": a function the application can import and call. The brief's "properly integrate" reads that way, and the alternative leaves the last mile to improvisation, which is what the skill exists to remove. + +The commitment is bounded precisely, so it does not grow into "rewrite my app": + +- **One new module per method**, placed by the project's convention (§5): it exports one async function named after the method, whose parameters are the pipe's inputs typed from the explicit template (§4.9) and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned address / id (§4.2), runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait`, which takes the durable path on the hosted API and the blocking path on a bare runner), and narrows `main_stuff` through the generated binder (`parse` in TypeScript, `Model.model_validate` in Python). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. +- **At most two shared helpers, created once per project and reused by every later method**: a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction pattern that matches how the project builds its other clients), and — TypeScript only, until `L-260820-ee327d` lands — the wire-null normalizer of §4.8. If the project already has a Pipelex client module, the skill uses it. +- **File-bearing inputs** (`Document`, `Image`) are typed as the canonical content dict `{ url }` — an `http(s)` URL or a `pipelex-storage://` reference. The module does not upload; its docstring points at the SDK's `prepareInputs` for callers holding local files or bytes, which is the SDK's own signature-driven upload and not something to hand-roll per method. +- **Not written:** tests, routes, UI, CLI commands, error-model integration beyond letting the SDK's typed errors propagate, retries, caching. In interactive mode the user can ask for any of these and the skill does them as ordinary coding work in the conversation; they are not part of the skill's contract. + +The call site follows the project's conventions where it can see them (module style, quoting, error handling, sync versus async) and the SDK's defaults where it cannot. Python projects that are synchronous throughout get a thin sync wrapper around the async function; async-native projects (FastAPI, an existing async codebase) get the async function alone. + +### 4.2 Which run source the call site uses, and the guard on its drift (brief Q2) + +**Decision: the call site runs the method from the same source the types were generated from, the sidecar records that source, and the skill never mixes sources.** The shape depends on the selector: + +| Types generated from | The call site runs | Drift guard | +| --- | --- | --- | +| **Local files** (the recommended shape) | the committed bundle, read at call time as `mthds_contents` | the sidecar's source hashes: a bundle edit without a regeneration is detectable offline (TypeScript gate) and by the next skill run (both languages), and `pipelex-edit` / `pipelex-design` announce it at edit time (§7) | +| **`method_ref@tag`** | `method_ref` at the **same pinned tag** | the tag is the pin: a tag is immutable in practice, and the run ack carries the fetched `commit_sha` as provenance; a `method_ref` with no tag is refused for a committed integration (the address would float) | +| **`method_id`** | `method_id` | **none offline** — the catalog is unversioned, so an edit to the stored method invalidates the committed types with nothing to detect it; the skill says so in one line, recommends committing the source or publishing an address, and proceeds only on the user's say-so, recording the id so refresh mode is at least the guard | + +The rule that matters is the negative one: **never generate from one source and run from another.** Types from a local bundle over a call site that runs by `method_id` is the shape that drifts silently, because the two halves have no relation the tooling can see. If the user wants the call site to run a catalog method, the types are generated from that id too, with the warning above. + +A bundle that is outside the project root (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first — the call site loads it at runtime and the sources must be versioned with the code — and the user is told. A bundle already inside the project stays where it is; the sidecar records its paths. + +### 4.3 The contracts artifact, and where the pipe signature comes from (brief Q3) + +**Decision: the skill does not produce a contracts artifact.** `pipelex-starter-js` needs `PIPE_IO_CONTRACTS` and `INPUT_FORM` to drive a form kernel and gate run inputs in a Server Action; a typed function has neither concern — its parameter types *are* the input gate, and the SDK validates the run request. Nothing in the skill's scope consumes the descriptor. + +What the skill does need is the **pipe's signature**: input names with their concept refs, and the main pipe's output concept with its multiplicity. Today this reaches the model through two channels and one gap: + +- **Inputs** — `mthds_inputs_template` with `explicit: true` returns each input as `{concept, content}` in `structuredContent`; the concept ref is exactly what types the parameter. This is why the skill departs from the plugin's `explicit: false` pin (§4.9). +- **Output, files source** — read from the bundle: the root's `main_pipe` and that pipe's `output` declaration, multiplicity included. The model can read the files; no tool call is needed. +- **Output, `method_ref` / `method_id` source — the gap.** `mthds_validate` carries `main_pipe_ref` and `pipe_io_contracts` on the view-only `_meta` channel, which never reaches the model, and the workshop has no views. So for a method whose source is not on disk, no in-context channel names the output concept. **v1 heuristic:** after generation, the candidate output concepts are the generated types minus the input concepts minus natives; exactly one candidate is taken and stated as an assumption; several is one question to the user, listing them; multiplicity is assumed single unless the user says otherwise. For a public `method_ref` the model may additionally read the package's `.mthds` at the tag from the repository to answer exactly. +- **The follow-up.** The heuristic is a stopgap. The correct fix is in `pipelex-mcp`: a compact main-pipe signature — `main_pipe_ref`, input names → concept refs, output concept ref and multiplicity — promoted from `_meta` to `structuredContent` on a valid `mthds_validate` verdict (or on `mthds_codegen`'s valid arm). It is small where the full descriptor is not, and it is what any integrating agent needs. Filed against `pipelex-mcp` as `L-260830-e8b2e0`, discovered from this item; when it lands, the heuristic and its question are deleted. + +### 4.4 The write arm is mandatory, and the model never writes an artifact (brief finding) + +**Decision: every `mthds_codegen` call passes `output_dir`; a refused or failed write is handled as a refusal, never by re-calling without `output_dir` and writing the bytes from the conversation.** The brief's efficiency argument decides the first half. The second half is a correctness rule: an artifact re-emitted through the model is one trailing newline away from a broken stamp, and the whole point of the trust chain is that the tree on disk is byte-identical to what the engine emitted. The tool's own posture ("a refused or failed write is a no-verdict, never a fallback to riding the content") is the skill's posture too. + +Three consequences the skill has to carry: + +- **`output_dir` is relative to the workshop's working directory, which is the directory the harness was launched in** — the host spawns the server there (`process.cwd()` in `pipelex-mcp/src/local/server.ts`), the launcher wrapper does not `cd`, and containment is real-path-checked against it (`workspace-boundary.ts`, `resolveSaveDir`). The skill computes the generated directory's path relative to the session's initial working directory and passes that. **A project root outside that directory cannot be written to**, and the skill stops with the instruction to relaunch the harness from the project (or, on Vibe, to register the workshop with that working directory) — it does not fall back to riding content. +- **The generated files are never opened for editing and never formatted.** Step 5 of §3 precedes step 6 for this reason, and step 11 formats only the files the skill authored. +- **A destination that refuses is a destination that was wrong.** The writer refuses any unstamped file, symlink, or directory at an artifact path and leaves the tree byte-identical; the skill treats that as "this is not a dedicated generated directory", picks or asks for one that is, and never pre-clears anything. + +### 4.5 The offline drift gate — one line for TypeScript, an honest gap for Python (brief finding) + +**Decision, TypeScript:** the skill installs a small **plain-ESM script** (`scripts/codegen-check.mjs`, copied verbatim from the skill's `references/`) that walks each generated directory the caller names, runs `@pipelex/sdk`'s `runCodegenCheck` over it with the lock read from disk, and compares the sidecar's source hashes against the committed `.mthds` files; it prints drifts by category and exits `0` current / `1` drift or stale source / `2` no verdict, the exit contract the starter established. It is `.mjs` rather than `.ts` so it runs under plain `node` in any project regardless of its TypeScript build setup; it imports nothing but `@pipelex/sdk` and Node builtins; and it writes through `process.stdout` / `process.stderr` so a `no-console` lint rule does not fire on it. The skill registers it as an npm script (`codegen:check`) and **extends the project's existing aggregate gate** — a `check` / `ci` / `validate` script, a Makefile `check` target, or the obvious lint/test step in an existing GitHub workflow — rather than inventing a new one; a project with no aggregate gate gets the script and a sentence in the report saying where to call it. + +The script exists because `@pipelex/sdk` ships the check as a pure function and no CLI; `L-260820-2ba0f4` (upstream the walk/write/sidecar workflow into the SDK as a CLI or exported functions) is the item that retires it, at which point the gate becomes a one-line invocation and the skill stops copying a script into projects. The skill's reference script is written so that swap is mechanical. + +**Decision, Python:** no gate is installed. `pipelex-sdk` has no offline check (`L-260830-4e43cd`), and the only one that exists — `pipelex codegen check` — needs the `pipelex` runtime, which a `python-pydantic` consumer deliberately does not have. The skill does **not** add `pipelex` as a dependency to get a gate; that would reverse the decision the target expresses. It writes the sidecar anyway (refresh mode and the editing skills' staleness notice both read it — §7), and the report states the asymmetry plainly: the generated tree is protected by its stamps and lock, but nothing in CI proves it current; refresh with `/pipelex-integrate` after every bundle edit. The one exception is the `python-structures` audience, whose project already depends on `pipelex`: there, `pipelex codegen check ` is wired into the existing gate exactly as `pipelex-starter-python` does. When `L-260830-4e43cd` lands, the Python branch gains its one-line gate and the asymmetry paragraph is deleted. + +### 4.6 The sidecar — `sources.json`, the skill's own memory (brief finding, Q5) + +**Decision: the skill writes an unstamped `sources.json` beside every lock it causes to exist**, and it is the only state the skill keeps. The lock signs the artifacts, not their sources, and the write arm produces no sidecar; without one, the second invocation has to re-derive the selector, the target, and the destination from the conversation every time, and a bundle edit is undetectable except by regenerating and diffing. + +The file keeps the starter's name and the starter's `sources` map, deliberately: `@pipelex/sdk` documents `sources.json` as the sidecar its orphan rule is designed to tolerate (an unstamped `.json` is never an artifact and never an orphan), and `L-260820-2ba0f4` plans to upstream a sidecar under that name, so converging now costs nothing and a later engine-owned sidecar can subsume this one. The shape: + +```json +{ + "comment": "Written by /pipelex-integrate. `method` and `target` are how this tree was generated — re-run the skill to refresh it. `sources` is the SHA-256 of each local .mthds source, so a bundle edit that was never regenerated is detectable. Not part of the codegen lock; do not hand-edit.", + "generator": "pipelex-integrate", + "method": { "files": ["methods/summarize-pdf/main.mthds"] }, + "target": "ts-zod", + "pipe": { + "pipe_ref": "summarize.summarize_pdf", + "inputs": { "document": "native.Document", "context": "native.Text" }, + "output": "summarize.DocumentSummary" + }, + "sources": { "methods/summarize-pdf/main.mthds": "" } +} +``` + +`method` is exactly one of `{files}`, `{method_ref}`, `{method_id}` — the selector as it was passed. Paths are relative to the project root, not to the workshop's working directory, so the file survives a relaunch from elsewhere. `pipe` records what the call site was typed against, so refresh mode can tell a signature change from a body change without re-reading the call site. `sources` is empty for a `method_ref` / `method_id` source; the `output` string uses the language's multiplicity notation (`Concept[]`) when the pipe produces a list. Hashes are over raw bytes; a CRLF checkout therefore reads as stale, and the remedy is a regeneration that changes nothing — the starter normalizes line endings and the upstreamed sidecar should too, but a false stale whose fix is a no-op is acceptable in the skill for the sake of a hash the model can compute with `shasum` / `sha256sum` / `hashlib`. + +### 4.7 One directory per method, orphans reported and never deleted (brief finding) + +**Decision: the skill always writes each method to its own directory, and when `orphans[]` is non-empty it names them, says what they are, and does nothing else.** The tool never deletes an orphan and neither does the skill: the moment two methods share a directory, "clean up the orphans" deletes real files. A non-empty `orphans[]` on a fresh integration means the chosen directory was not fresh — an earlier generation, a different target, or an engine rename — and the report says a dedicated directory per generation is the fix, in the tool's own words. On a refresh into the method's own directory, an orphan can only be an artifact the engine stopped emitting; the report names it and leaves the decision to the user. + +A directory that already holds a `codegen.lock` is refresh mode only if its sidecar names the same method; otherwise the skill chooses a different directory name and says why. + +### 4.8 The wire-`null` mismatch — a shared helper with an expiry (finding surfaced in design) + +The ts-zod emitter projects a non-required field as `.optional()` (`pipelex/pipelex/codegen/emitters/ts_zod.py`, `_field` modifier), which in zod means `| undefined` and rejects `null` — but the runtime serializes an unset optional field as an explicit `null` (`WorkingMemory.dump_for_transport` is a `model_dump(serialize_as_any=True)` with no `exclude_none`). `pipelex-starter-js` confirmed this against a live hosted run and carries `dropWireNulls` as a schema-guided workaround; the defect is tracked as `L-260820-ee327d` (P1, `pipelex`). Python is unaffected: the pydantic projection emits `str | None = None`. + +**Decision: until `L-260820-ee327d` lands, the TypeScript call site parses through a shared `wireOutput(results, schema)` helper, copied once per project from the skill's `references/`, that drops a `null` only where the concept's own zod schema says the field is optional with no default, descends declared objects and arrays, and passes anything opaque (`z.unknown()`, `z.record()` keys, unions) through untouched.** A blind deep null-strip is rejected for the reason the starter's design note gives: inside a `z.record()` a `null` is data. The helper's header names the item it waits on and says the file is deleted, not maintained, when the emitter projects `.nullish()`; the plan carries a checkpoint to re-check the item before shipping, since it may land first — in which case the helper is never written and the call site parses `main_stuff` directly. + +### 4.9 The template shape — `explicit: true`, the plugin's one exception + +The plugin's standing rule pins every `mthds_inputs_template` call to `explicit: false` (`docs/decisions.md`, "the light template stays pinned"), because the three existing call sites only show or key-compare the template. This skill is different in kind: it needs **each input's declared concept ref** to write a typed parameter, and the light shape carries values without concepts. **Decision: `pipelex-integrate` passes `explicit: true`, and the decisions record gains a sentence naming it as the exception and why.** The rule's rationale (the envelope is an authoring aid the other skills do not need) is unchanged; a fourth call site with a need for concept identity is exactly the case the rule said would justify it. + +### 4.10 The second invocation — refresh mode (brief Q5) + +**Decision: refresh mode re-derives nothing the sidecar already records, regenerates in place, and leaves alone everything the regeneration did not invalidate.** + +Entered when: the user asks to refresh, regenerate, or update the types; `pipelex-edit` or `pipelex-design` hand off after editing a bundle a sidecar names (§7); or the skill finds a sidecar for the method it was asked to integrate. + +| Taken from disk | Re-derived | Left alone | +| --- | --- | --- | +| the selector, target, destination, and pipe record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared before regenerating so the report can say whether the bundle actually changed; the pipe signature, through `mthds_inputs_template` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client and wire-output helpers, the gate wiring, tests, other methods' trees | + +The regeneration is one `mthds_codegen` call with the recorded arguments. Its `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — reported as such, with the diff left to the user's commit); changed means the concept set moved. Then the project's type checker runs. **The call site is edited only if it no longer type-checks or the sidecar's `pipe` record no longer matches the template** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last, with the new hashes and pipe record. + +### 4.11 The name and its place in the family (brief Q6) + +**Decision: `pipelex-integrate`.** `pipelex-codegen` — the name the `pipelex-mcp` codegen design floated (`wip/mcp-codegen/design.md`, "not filed, recorded so it is not rediscovered") — is rejected because it is the tool's name in skill clothing, which the plugin's naming convention forbids: tools are the contract and skills are the manual, named after user tasks. The task is integrating a method into an application; codegen is one step of it. + +The description is written to trigger on the codebase phrasings — "use this method in my app", "call this from my code", "generate types for this method", "wire the method into", "typed client", "refresh the generated types", "regenerate the types" — and to stay silent on the authoring phrasings the other skills claim ("design", "edit", "explain", "prepare inputs"). It is model-invocable, like every skill in the plugin since `pipelex-design` lost its flag; the consent gate is inside (the one-line statement of target and destination before writing). + +Its place: it is the skill after the method is done. `pipelex-design` and `pipelex-inputs` close by pointing at it when there is a codebase in the workspace (§7). + +## 5. Detecting the project + +The rule: **a cheap, reliable signal decides; an inconclusive one asks one question; the Python audience is never guessed when `pipelex` is a dependency.** The language-specific detail lives in the skill's `references/typescript.md` and `references/python.md`; this is the contract those files implement. + +| Question | Signals, in order | When inconclusive | +| --- | --- | --- | +| **Which project?** | the user named it; else the nearest `package.json` / `pyproject.toml` (or `setup.py`, `requirements.txt`) above the working area; a workspace with several (a monorepo, a full-stack repo) | ask which app; never pick one | +| **Language → target** | `package.json` → `ts-zod` (a JavaScript project with no TypeScript build — no `tsconfig.json`, no bundler or runtime that strips types — is asked, because `types.ts` needs one); `pyproject.toml` → Python | both present at one root: ask | +| **Python audience → target** | `pipelex` **not** among the project's dependencies → `python-pydantic`, no question (`python-structures` imports the runtime and would not even load); `pipelex` present → `python-structures` if `@pipe_func` or `StructuredContent` appear in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | +| **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock*` → bun; `uv.lock` → uv, `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none: npm / `pip install` into the active environment, stated | +| **Generated-tree root** | an existing directory already holding generated code (`generated/`, `gen/`, `__generated__/`) → beside it; else `src/generated/` if `src/` exists, else `generated/`; Python: `/generated/` where the package is the one `[project].name` or the setuptools `packages` list names, or the top-level directory holding `__init__.py` | ask | +| **Formatter and linter exclusions** | Prettier (`.prettierrc*` or a `prettier` dev dependency) → `.prettierignore` entry; ESLint flat config (`eslint.config.*`) → `globalIgnores` / `ignores` entry, legacy `.eslintrc*` → `.eslintignore`; Biome (`biome.json*`) → the files-ignore key its version uses; Ruff (`[tool.ruff]`) → `exclude` / `extend-exclude`; Black → `extend-exclude`; isort → `skip` / `extend_skip_glob` | a tool the table does not name: read its config, add the equivalent, say so | +| **Type checker still covers the tree** | `tsconfig.json` `include` / `exclude`; `[tool.pyright]` `include`; `[tool.mypy]` `packages` / `files` | an exclusion that would drop the tree is *not* added; the report says which check covers it | +| **Aggregate gate to extend** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows` job with a lint or test step; `.pre-commit-config.yaml` | none: the script alone, and a sentence in the report | +| **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, a `services/` or `clients/` package) → beside it; else `src/pipelex/` / `/pipelex/` | — | +| **Not gitignored** | the generated root and the sidecar must be committable; a `.gitignore` pattern that swallows them is reported, and the path is un-ignored on confirmation | — | + +## 6. Failure posture + +| Condition | The skill | +| --- | --- | +| `mthds_codegen` or `mthds_inputs_template` or `mthds_validate` absent | STOP with the plugin's one-line MCP-connection message (the platform-specific wording every MCP-backed skill renders) | +| `status: "error"`, class `config` — including the hosted `FF_PLAYGROUND` **403**, which is a feature gate and not a key problem | STOP, surface `hint` verbatim; never say "check your key" for a 403 | +| `status: "error"`, class `config`, `kind: "paywall"` | STOP, surface the plan-limit message | +| `is_valid: false` | route the `validation_errors[]` to `/pipelex-design` or `/pipelex-edit`; a by-id method's stored content is broken where it is edited, not here | +| validate says not runnable / pending signatures | STOP: finish the method with `/pipelex-design`; nothing generated | +| `input_domain` at `output_dir`: containment escape | STOP: the project is outside the workshop's working directory — relaunch the harness from the project root; never ride content | +| `input_domain` at `output_dir`: foreign file named | the directory is not a dedicated generated directory — choose another or ask; never delete or move the named file | +| `input_domain` at `method_ref` / `method_id` | report the selector failure as the tool words it (unknown or foreign-org id, unfetchable address, the structures refusal, a registry-form ref) | +| `runtime` after a partial write | call again once with the same `output_dir` (the writer overwrites its own stamped files); then report what landed, in the tool's words | +| success with `orphans[]` non-empty | report by name, never delete (§4.7) | +| success with `is_current: false` | a write the check disowns — report the `drifts[]` verbatim and stop; do not commit a tree the check rejects | +| `orphans_truncated: true` | say orphan detection was partial rather than reporting a clean tree | +| the project's type check fails after the call site is written | the skill's to fix (its own code), then re-run; a failure inside the generated tree is reported, not patched | +| `mthds_list_methods` absent | integrate by id or files only; never stop for it (soft dependency) | + +## 7. Family wiring + +Three small edits to the existing skills, each one sentence or one step: + +- **`pipelex-edit` Step 7 (Report) and `pipelex-design`'s re-entry delivery:** after a successful edit to a bundle, look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file that changed; for each, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. This is the plugin-native forgetting-guard, and it is language-agnostic — it is what gives the Python side a guard at all until `L-260830-4e43cd`. +- **`pipelex-design`'s delivery step 4 and `pipelex-inputs`' closing report:** one line — when the workspace holds a `package.json` or `pyproject.toml`, `/pipelex-integrate` wires the method into that code. +- **The `MCP_SKILLS` tuple in the tests, the README's skill list and MCP tool list, `CLAUDE.md`'s "Key dependency", and `docs/decisions.md`** gain the skill and `mthds_codegen`, exactly as `mthds_prepare_inputs` was added. + +## 8. Follow-ups, filed or linked + +| Item | Repo | Relation | Why | +| --- | --- | --- | --- | +| `L-260820-ee327d` | `pipelex` | related — the §4.8 helper waits on it | ts-zod `.optional()` rejects the runtime's explicit `null`; when it lands, the wire-output helper is deleted from the skill's references and the call site parses directly | +| `L-260820-2ba0f4` | `pipelex-sdk-js` | related — retires the §4.5 script | upstream the walk / write / sidecar workflow into `@pipelex/sdk` as a CLI or exported functions; the skill's gate becomes one line | +| `L-260830-4e43cd` | `pipelex-sdk-python` | discovered-from (already) | the Python offline check; when it lands, the Python branch gains its gate and the asymmetry paragraph goes | +| `L-260830-e8b2e0` | `pipelex-mcp` | discovered-from this item (filed 2026-08-30) | promote a compact main-pipe signature (`main_pipe_ref`, inputs → concept refs, output concept + multiplicity) from `_meta` into `structuredContent`, so a by-ref / by-id integration is typed exactly instead of by the §4.3 heuristic | +| `L-260829-563e9e` | workspace | related, informational | the pipe-selector campaign adds `pipe_ref` to the run request; the sidecar already records the qualified ref, and the call site moves from `pipe_code` to `pipe_ref` when the SDKs take it | + +## 9. Out of scope, stated so it is not rediscovered + +No change to `pipelex-mcp` from this repo (the follow-up above is filed, not worked around); no change to either starter (they are the reference, not a deliverable); no JSON Schema target until the engine serves one (`L-260829-7b7917` → `L-260829-263b9e` → `L-260829-68c7cf`); no watch mode or build-time regeneration; no per-project reimplementation of write-if-changed or orphan cleanup (the writer overwrites and reports; the SDK upstreaming owns the rest); no `contracts.ts`; no tests, routes, or UI; no hosted-console branching (the plugin only ever declares the workshop, so the write arm is always available to it). + +## Decision boxes for ratification + +| Box | Ruling | Ratified? | +| --- | --- | --- | +| **1 — Scope of the call site** | One complete typed callable module per method plus at most two shared helpers; no tests, routes, or UI (§4.1) | Yes, as written — 2026-08-30 | +| **2 — Run source** | The call site runs from the source the types came from; `method_id` allowed with a one-line warning; never mixed (§4.2) | Yes, as written — 2026-08-30 | +| **3 — No contracts artifact** | Signature from `explicit: true` template + bundle read; heuristic for by-ref / by-id until the `pipelex-mcp` follow-up lands (§4.3) | Yes, as written — 2026-08-30 | +| **4 — Write arm only** | Always `output_dir`, relative to the harness's launch directory; never ride content; stop on containment escape (§4.4) | Yes, as written — 2026-08-30 | +| **5 — Gate asymmetry** | TypeScript: reference `codegen-check.mjs` into the existing gate; Python: no gate, sidecar plus refresh, stated in the report (§4.5) | Yes, as written — 2026-08-30 | +| **6 — Sidecar** | `sources.json`, starter-compatible `sources` map plus `generator` / `method` / `target` / `pipe` (§4.6) | Yes, as written — 2026-08-30 | +| **7 — Wire-null helper** | Shared schema-guided `wireOutput` per TypeScript project until `L-260820-ee327d`, with a pre-ship re-check (§4.8) | Yes, as written — 2026-08-30 | +| **8 — `explicit: true`** | The plugin's one exception to the light-template pin, recorded in `docs/decisions.md` (§4.9) | Yes, as written — 2026-08-30 | +| **9 — Name** | `pipelex-integrate`, model-invocable, codebase-phrasing triggers (§4.11) | Yes, as written — 2026-08-30 | +| **10 — Family wiring** | Staleness notice in `pipelex-edit` / `pipelex-design`; one-line hand-off in `pipelex-design` / `pipelex-inputs` (§7) | Yes, as written — 2026-08-30 | diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md new file mode 100644 index 0000000..e193332 --- /dev/null +++ b/wip/pipelex-integrate/plan.md @@ -0,0 +1,158 @@ +--- +status: active +item: L-260830-344594 +--- + +# Plan — `pipelex-integrate`: the implementation tracker + +**Written 2026-08-30** as the execution tracker for [`design.md`](design.md). It schedules; it does not re-argue — when this file and the design disagree, the design wins unless the disagreement is logged under "Deviations" below. Section references (`§N`) are to the design. Ledger item `L-260830-344594`; the phases name the follow-up items they wait on or file. + +**Status: active** since 2026-08-30, when the ten decision boxes of `design.md` were ratified as written (Phase 0). Work proceeds from Phase 1. + +## How to work a phase + +- `ledger claim L-260830-344594` before touching code; renew the claim once you are on the working branch. +- The working branch is `feature/Codegen` (already created for this item); the PR targets `dev` and its body carries `Closes L-260830-344594`. A merged PR is landed with `/ledger-land`. +- **This checkout is shared with other sessions.** Stage the files you touched explicitly (`git add `), never `git add -A`; the branch already carries uncommitted work on `pipelex-design` / `pipelex-edit` from another piece of work, and a phase here must not sweep it into its commit. Never run a formatter over files you did not author. +- Templates are the source of truth: edit `templates/skills/…/*.j2` and `skills/pipelex-integrate/references/*`, then `make build`; never edit `pipelex*/` outputs. Before pushing: `make agent-check` and `make agent-test`. +- `mthds_codegen` is **unreleased in `@pipelex/mcp`** at writing. Development and dogfood run against the local `../pipelex-mcp` checkout through the repo skill `/pipelex-mcp-source`; **switch back to `@latest` before any commit** and let that skill confirm no dev switch leaked into `targets/defaults.toml`. +- Version discipline: everything accumulates under `[Unreleased]` in `CHANGELOG.md`; the release phase cuts the heading and bumps the version through `/release`. +- At each checkpoint: tick the boxes, record the SHAs and versions outcomes landed in (never live git state), reconcile deviations into the later phases, and leave this file cold-start ready. + +## Standing context for every phase + +- **The write arm is the only arm the skill uses** (§4.4). If a dogfood run ever tempts a "just write the bytes from the response" fallback, that is a bug in the run, not a feature to add. +- **The generated tree is never opened for editing, formatted, or linted** — by the skill, and by the session working this plan. A dogfood run that reformats a generated file has invalidated its own verdict; regenerate and start the scenario again. +- **Two upstream items can land during this work and each deletes a piece of it.** `L-260820-ee327d` (ts-zod `.nullish()`) deletes the wire-output helper (§4.8); `L-260830-4e43cd` (Python offline check) deletes the Python asymmetry paragraph (§4.5). Phases 1 and 4 each carry a box to re-check both before proceeding, and a landed item is recorded under "Decisions taken along the way" with what was removed. +- **Ledger ids never appear in user-facing skill text or reports.** They belong in this tracker, the design, and `docs/decisions.md`; the skill's report to a user says "the Python SDK has no offline drift check yet", never the item that tracks it. + +## Phase 0 — ratify, file, link + +No code. Owner: the session that reads the design with Louis. + +- [x] `ledger claim L-260830-344594` (2026-08-30). +- [x] Walk the ten decision boxes at the end of `design.md` with Louis; record each ruling (ratified as drafted, or amended, with the amendment written into the design's section) in the box's "Ratified?" column with the date — done 2026-08-30: all ten ratified as written, no amendments. +- [x] Flip `design.md` and this file to `status: active` in the same change that records the ratification — done 2026-08-30. +- [x] File the `pipelex-mcp` follow-up from §4.3 — filed 2026-08-30 as `L-260830-e8b2e0`: promote a compact main-pipe signature — `main_pipe_ref`, input names → concept refs, output concept ref and multiplicity — from `_meta` into `structuredContent` on a valid `mthds_validate` verdict (or on `mthds_codegen`'s valid arm), with the evidence from `pipelex-mcp/SPEC.md` → "Validation Scope" (the `_meta`-only rule and its token rationale) and the §4.3 heuristic as the consumer that needs it. Its id is recorded in §8 of the design and under "Where everything is" below. +- [x] `ledger link L-260830-344594 --related L-260820-ee327d`, `--related L-260820-2ba0f4`, `--related L-260829-563e9e` (`L-260830-4e43cd` is already discovered-from) — done 2026-08-30. +- [x] `ledger ref L-260830-344594` attached `plan:pipelex-plugins/wip/pipelex-integrate/design.md` and `plan:pipelex-plugins/wip/pipelex-integrate/plan.md` beside the existing brief ref — done 2026-08-30. +- [x] `ledger validate`, then `ledger commit` — done 2026-08-30 for the filing above; re-validated after the ratification edits (the ratification changed only these two documents, not the ledger). + +## Phase 1 — the skill template and its references + +Owner: `pipelex-plugins`. Everything in this phase renders into all three targets; nothing in it is platform-specific except the `allowed-tools` frontmatter and the MCP-absent message, which the shared patterns already handle. + +**Pre-flight** + +- [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` (`ledger show`). If either has closed and shipped in the hosted engine / the Python SDK, strike the corresponding piece below before writing it and log the deviation. +- [ ] Confirm the static-asset mechanism works end to end before relying on it: `scripts/gen_skill_docs.py` → `setup_static_assets` copies `skills//references/` into every target; `scripts/check.py` → `check_stale_references` resolves `references/…` links from a rendered `SKILL.md`; `check_no_templates_in_output` tolerates `.mjs` / `.ts` files under `skills/`. This is the **first skill in the repo to ship references**, so a root `skills/` directory does not exist yet; create it and note in `docs/build-targets.md` (Phase 2) that the mechanism is now in use. + +**The template — `templates/skills/pipelex-integrate/SKILL.md.j2`** + +- [ ] Frontmatter: `name`, the description from §4.11 (codebase-phrasing triggers, silent on authoring phrasings), the shared `frontmatter.md.j2` include, and on Claude the `allowed-tools` entries for `mthds_codegen`, `mthds_inputs_template`, `mthds_validate`, `mthds_list_methods`. No `disable-model-invocation`. +- [ ] "Requirements — the Pipelex MCP tools": `mthds_codegen`, `mthds_inputs_template` and `mthds_validate` required with the plugin's standard MCP-absent STOP message (copy the exact conditional block from `pipelex-inputs`, so `TestSkillFailureDiscipline.test_absent_tools_stop_message_matches_platform` passes on every target); `mthds_list_methods` soft. The `config`-class stop, with the **403 feature-gate wording** from §6 (a 403 is not a key problem). +- [ ] Mode selection: automatic default; the interactive signals; the one-question rule from §5. +- [ ] The procedure of §3, as numbered steps, each naming its MCP call, its arguments (`explicit: true` on the template call — with the sentence saying this is the deliberate exception to the plugin's light-template pin), and its verdict branches. +- [ ] The `output_dir` rule of §4.4 spelled out for the model: compute it relative to the session's initial working directory; never absolute; never ride content; the relaunch instruction on a containment escape. +- [ ] The exclusions-before-generation ordering (§3 step 5), stated as a rule with its reason. +- [ ] The orphan rule of §4.7 in the tool's own wording; never delete. +- [ ] The sidecar section: the exact `sources.json` shape of §4.6, how hashes are computed (`shasum -a 256` / `sha256sum` / `hashlib`, raw bytes), paths relative to the project root. +- [ ] The call-site section (§4.1): what one module contains, the two shared helpers, the input-type mapping table from concept ref to language type (Text → `string`/`str`, Number → `number`/`float | int`, YesNo → `boolean`/`bool`, Date → ISO string, Image/Document → `{ url }`, structured → the generated type, `[]` → arrays, `?` → optional), the `main_stuff` narrowing, the `prepareInputs` pointer for file-bearing callers, the sync-wrapper rule for synchronous Python projects. Language detail is delegated to the two reference files. +- [ ] Refresh mode as its own section (§4.10): the taken / re-derived / left-alone table, the fingerprint comparison and the restamp-only case, the "call site edited only if it no longer type-checks or the `pipe` record moved" rule. +- [ ] The verification step (§3 step 11) and the report (§3 step 12), including the Python asymmetry sentence. +- [ ] A failure table condensed from §6. +- [ ] `## Reference`: links to `references/typescript.md`, `references/python.md`, and the two shared language references. + +**The references — `skills/pipelex-integrate/references/`** + +- [ ] `typescript.md`: the detection signals of §5 for a TypeScript project (project root, TS-capable build, package manager, generated root, Prettier / ESLint flat and legacy / Biome exclusion edits with the exact config keys, `tsconfig` coverage, aggregate gate, call-site location, `.gitignore`); the call-site module template with the `getPipelexClient` helper and the `wireOutput` import; the `codegen:check` npm script and how to append it to `check` / a Makefile / a workflow step. +- [ ] `python.md`: the same for Python (import package discovery, `python-pydantic` vs `python-structures` per §5, uv / poetry / pipenv / pip, `[tool.ruff]` `exclude` and `extend-exclude`, Black, isort, pyright / mypy coverage, `__init__.py` creation for the generated package and each method subpackage, setuptools `packages` / `package-data` when the project is packaged, the async call-site template plus the sync wrapper, the `pipelex codegen check` wiring for the `python-structures` audience only, the asymmetry sentence for everyone else). +- [ ] `codegen-check.mjs` (§4.5): plain ESM, Node builtins + `@pipelex/sdk` only; takes generated directories as arguments; per directory reads `codegen.lock` from disk, walks recursively (pruning `node_modules`, `.git`, `dist`, `build`, `.next`), filters with `isStampableArtifactPath`, decodes strictly, runs `runCodegenCheck`, prints drifts by category; then reads `sources.json` and compares each `sources` hash against the file on disk, reporting `stale-source` with the "run `/pipelex-integrate` to refresh" remedy; exit `0` / `1` / `2` with the precedence no-verdict > drift > current; output through `process.stdout` / `process.stderr`. Header comment names what it is and that `@pipelex/sdk` upstreaming retires it. +- [ ] `wire-output.ts` (§4.8, **skip if `L-260820-ee327d` has landed**): `wireOutput(results, schema)` and the schema-guided `dropWireNulls`, trimmed from `pipelex-starter-js/src/lib/wireOutput.ts` — objects, arrays, `z.lazy`, optional-without-default only; opaque schemas passed through; a depth cap; no `server-only` import, no Next-specific error types. Header comment states it is a workaround with an expiry and what deletes it. + +**Build and tests** + +- [ ] `make build`; confirm `pipelex/`, `pipelex-codex/`, `pipelex-vibe/` each carry `skills/pipelex-integrate/SKILL.md` and the `references/` directory verbatim. +- [ ] `tests/unit/test_gen_skill_docs.py`: add `"pipelex-integrate"` to `TestSkillFailureDiscipline.MCP_SKILLS`. +- [ ] New `TestPipelexIntegrateDiscipline` pinning the load-bearing sentences in the real template, rendered on all three targets: `output_dir` is always passed; content is never ridden; orphans are never deleted; one directory per method; generated files are never edited or formatted; exclusions precede generation; `explicit: true` on the template call; the `method_id` warning; refresh mode leaves the call site alone unless the types moved; the 403 wording. Plus one test that the references land in every target's output. +- [ ] `make agent-check`, `make agent-test`. + +**CHECKPOINT 1** — the template and references render on every target and the tests pin their rules. Record here: the commit SHA, what was struck because an upstream item landed, and anything the template could not express without a reference file. + +## Phase 2 — family wiring and documentation + +Owner: `pipelex-plugins`. Small, deliberate edits; each one is one sentence or one step (§7). + +- [ ] `templates/skills/pipelex-edit/SKILL.md.j2` Step 7: the `sources.json` staleness notice and the `/pipelex-integrate` offer. +- [ ] `templates/skills/pipelex-design/SKILL.md.j2`: the same notice in the re-entry delivery; the one-line hand-off in "Common runnable gate and delivery" step 4 (when a `package.json` / `pyproject.toml` is in the workspace). +- [ ] `templates/skills/pipelex-inputs/SKILL.md.j2`: the one-line hand-off in the closing report. +- [ ] `docs/decisions.md`: a dated entry — the skill's scope line (§4.1), the name ruling over `pipelex-codegen` (§4.11), the `explicit: true` exception appended to the light-template decision (§4.9), the write-arm-only and never-ride-content rule (§4.4), the sidecar (§4.6), the wire-null helper with its expiry (§4.8), the gate asymmetry (§4.5), and that this is the first skill to ship `references/`. +- [ ] `CLAUDE.md` "Key dependency": add `mthds_codegen` (the write arm, `output_dir`) beside the other tools; the structure block gains the `pipelex-integrate` template and the root `skills/` directory. +- [ ] `README.md`: the skill in "What's inside" and `mthds_codegen` in the MCP server bullet; the Claude and Codex sections' skill lists. +- [ ] `docs/build-targets.md`: the `skills//references/` mechanism is now in use, with `pipelex-integrate` as the example. +- [ ] `CHANGELOG.md` `[Unreleased]` → "Added": the skill, in the changelog's existing voice (what it does, the write arm, the sidecar, the gate asymmetry, the wire-null helper and its expiry, `explicit: true`); "Changed": the three family one-liners. +- [ ] `make build`, `make check`, `make agent-test`. + +## Phase 3 — dogfood against the local workshop + +Owner: `pipelex-plugins`, with `../pipelex-mcp` on a build that carries `mthds_codegen` (`feature/CodegenTool` or later). Scratch projects live in the session scratchpad, **never** in either starter (§9). Switch the launcher with `/pipelex-mcp-source` to the local checkout for the phase, and back before committing. + +Two scratch projects, each created from scratch by the session so the skill meets a cold codebase: a minimal TypeScript project (`package.json`, `tsconfig.json`, Prettier and ESLint flat config, a `check` script, `src/`), and a minimal Python project (`pyproject.toml` with `[tool.ruff]` and `[tool.pyright]`, `uv.lock`, one import package). Each holds a committed `methods//main.mthds` copied from the cookbook or written with `/pipelex-design`. + +- [ ] **TS-1 fresh integration, files source.** Run `/pipelex-integrate`. Verify: the tree landed under `src/generated//` with `is_current: true` and no orphans; `.prettierignore` and the ESLint `ignores` entry were added **before** the write; `zod` and `@pipelex/sdk` were added with npm; the call site, client helper, and wire-output helper exist where §5 says; `npm run check` (extended with `codegen:check`) passes; `tsc --noEmit` passes; `sources.json` matches §4.6. +- [ ] **TS-2 the bytes are untouched.** `git diff --stat` shows no change under `src/generated/` after the skill's own format run; `npm run codegen:check` exits 0. +- [ ] **TS-3 refresh after a bundle edit.** Change a concept field in `main.mthds` (through `/pipelex-edit`, which should announce the staleness — Phase 2 wiring), run the skill again: the sidecar comparison reports the changed source, the fingerprint moved, the call site is edited only if the type check demanded it, `sources.json` carries the new hash, `codegen:check` is green again. Then edit only a prompt (no concept change): the fingerprint is unchanged and the report says restamp-or-nothing. +- [ ] **TS-4 stale-source gate.** Edit the bundle and do *not* refresh: `npm run codegen:check` exits 1 with `stale-source` and the refresh remedy. +- [ ] **TS-5 `method_ref` source.** Integrate a published address at a tag (a `github.com/Pipelex/…@vX.Y.Z` package): the call site runs by `method_ref`, the sidecar's `sources` is empty, the output-concept heuristic of §4.3 either finds one candidate or asks — record which. +- [ ] **TS-6 `method_id` warning.** Integrate a catalog method by id: the one-line warning appears, the recommendation is stated, and the skill proceeds only on confirmation. +- [ ] **TS-7 orphan.** Generate a second method into the first method's directory on purpose (by naming the dir explicitly): `orphans[]` is reported by name, nothing is deleted, the fix sentence is the tool's. +- [ ] **TS-8 foreign file.** Point at a directory holding a hand-written `types.ts`: the refusal is surfaced, the file is untouched, the skill chooses or asks for another directory. +- [ ] **TS-9 containment escape.** Launch the harness from a sibling directory so the project is outside the workshop's working directory: the skill stops with the relaunch instruction and does not ride content. +- [ ] **TS-10 no key.** Unset `PIPELEX_API_KEY` in the workshop's environment: the `config` stop with the hint verbatim, nothing written. +- [ ] **PY-1 fresh integration, pydantic audience.** No `pipelex` dependency → `python-pydantic` chosen without a question; `/generated/__init__.py` and the subpackage `__init__.py` created; `[tool.ruff] exclude` gains the tree; pyright still covers it; `pydantic` and `pipelex-sdk` added with uv; the async call site plus a sync wrapper if the project is synchronous; the report states the gate asymmetry; `uv run pyright` passes. +- [ ] **PY-2 structures audience.** Add `pipelex` as a dependency and a `@pipe_func` file: `python-structures` is chosen (or offered first when only the dependency is present); `pipelex codegen check ` is wired into the existing gate. +- [ ] **PY-3 refresh after a bundle edit**, as TS-3, including the `/pipelex-edit` staleness notice. +- [ ] **Vibe render sanity.** Read `pipelex-vibe/skills/pipelex-integrate/SKILL.md` once for the manual-registration wording of the MCP-absent message and the absence of Claude-only frontmatter. +- [ ] Reconcile every finding into the template and references; re-run the affected scenarios; `make build`, `make agent-check`, `make agent-test`. +- [ ] `/pipelex-mcp-source` back to `@latest`; confirm the diff carries no launcher change. + +**CHECKPOINT 2** — every scenario above has been run at least once against the local workshop and its finding reconciled. Record here: the `pipelex-mcp` SHA the dogfood ran against, the scenarios that exposed a template change (and the change), any scenario that could not be run and why, and the exact wording the §4.3 heuristic produced in TS-5. + +## Phase 4 — release + +Owner: `pipelex-plugins`. **Gate, hard:** a published `@pipelex/mcp` version that carries `mthds_codegen` with the write arm — name the version here before starting; an open `pipelex-mcp` release item is not a gate. The plugin's launcher is `@latest`, so nothing in this repo moves for it, but a plugin released before the tool is a skill that stops at "tool absent" for every user. + +- [ ] Published `@pipelex/mcp` version carrying `mthds_codegen`: `__________` (fill in). +- [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` one last time; strike or keep the helper and the asymmetry paragraph accordingly, and log it. +- [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. +- [ ] Open the PR against `dev` with `Closes L-260830-344594` in the body; work the review rounds per the workspace's tightening-bar rule; land with `/ledger-land`. +- [ ] `/release` → the next minor (`0.6.0`), which cuts the changelog heading, bumps every target TOML and the Claude marketplace, and opens the release PR against `main`. +- [ ] After the release merges: `/ledger-land` on the release PR, and this file's `status` flips to `landed` when the tooling performs it. + +## Deferred and out of scope + +Carried in the design's §9 and §8; repeated here only where a phase might be tempted: + +- No `contracts.ts`, no watch mode, no tests / routes / UI written by the skill, no per-project write-if-changed or orphan cleanup, no hosted-console branching. +- The by-ref / by-id output-concept heuristic (§4.3) ships as designed; the exact answer arrives with the `pipelex-mcp` follow-up and is not worked around here. +- The Python offline gate waits on `L-260830-4e43cd`; the TypeScript script is retired by `L-260820-2ba0f4`; the wire-null helper by `L-260820-ee327d`. None of the three is worked from this repo. +- The call site uses `pipe_code` (bare) until the SDKs take `pipe_ref` (`L-260829-563e9e`); the sidecar already records the qualified ref so the switch is a one-line edit per call site when it comes. + +## Decisions taken along the way + +- **2026-08-30 — Phase 0 ratification.** Louis walked the ten decision boxes of `design.md` one at a time, each presented against its alternatives (a narrower or wider call-site scope; refusing `method_id` or accepting a floating `method_ref`; blocking by-ref / by-id on the `pipelex-mcp` follow-up or keeping a contracts artifact; a content fallback or consented pre-clearing on the write arm; a hash-only Python gate or no gates at all; dropping the sidecar's `pipe` record or LF-normalizing its hashes; fixing the emitter first or a blind null-strip; lifting the light-template pin plugin-wide or reading concepts from the bundle; `pipelex-codegen` or a user-invocable-only skill; a staleness notice alone or an auto-refresh from `pipelex-edit`). Every box was ratified as written; the design's sections stand unamended and both documents flipped to `active` in this change. +- **2026-08-30 — upstream dependencies reviewed (Louis).** None of the items the design leans on (`L-260820-ee327d`, `L-260820-2ba0f4`, `L-260830-4e43cd`, `L-260830-e8b2e0`, `L-260829-563e9e`) is a member of the build-retirement epic `L-260829-848001` or appears in its plan; they sit on the codegen trust-chain axis, not the descriptor-route axis. Two decisions: `L-260830-e8b2e0` is linked *related* to that epic (not a member), to be sequenced after `L-260829-dfaed4` once the workshop holds the input-form descriptor; and `L-260820-ee327d` is prioritized ahead of Phase 3, so the Phase 1 pre-flight re-check may strike `wire-output.ts` before it is written. The summary is `upstream-dependencies.md` beside this file. + +## Deviations from the design + +*(empty at writing — a deviation is logged here with its reason before the code that embodies it is committed.)* + +## Where everything is + +- Brief: `wip/pipelex-integrate/brief.md`. Design: `wip/pipelex-integrate/design.md`. This tracker: `wip/pipelex-integrate/plan.md`. +- The tool contract: `../pipelex-mcp/SPEC.md` → "Codegen Scope (`mthds_codegen`)" and "The write arm (`output_dir`) — local workshop only". The writer: `../pipelex-mcp/src/capabilities/codegen-writer.ts`; containment: `workspace-boundary.ts`. +- Reference integrations (read, never changed): `../pipelex-starter-js/docs/codegen.md`, `src/generated//`, `src/lib/wireOutput.ts`, `scripts/codegen-check.mts`; `../pipelex-starter-python/Makefile` (`codegen`, `codegen-check`), `docs/codegen.md`, `piper/generated/`. +- The offline check the TypeScript gate wraps: `../pipelex-sdk-js/src/codegen-check.ts` (`runCodegenCheck`, `isStampableArtifactPath`), documented in `docs/crate-routes.md` → "The offline check". +- The sibling skill to model: `templates/skills/pipelex-inputs/SKILL.md.j2`; the tests to extend: `tests/unit/test_gen_skill_docs.py` (`TestSkillFailureDiscipline`, `TestPipelexInputsSizeLimitDiscipline` as the pattern). +- The static-asset mechanism: `scripts/gen_skill_docs.py` → `setup_static_assets`; documented in `docs/build-targets.md` → "Template vs output directories". +- Ledger: this item `L-260830-344594`; discovered `L-260830-4e43cd`; related `L-260820-ee327d`, `L-260820-2ba0f4`, `L-260829-563e9e`; the `pipelex-mcp` follow-up filed in Phase 0: `L-260830-e8b2e0`. diff --git a/wip/pipelex-integrate/upstream-dependencies.md b/wip/pipelex-integrate/upstream-dependencies.md new file mode 100644 index 0000000..1eec4de --- /dev/null +++ b/wip/pipelex-integrate/upstream-dependencies.md @@ -0,0 +1,26 @@ +--- +status: active +item: L-260830-344594 +--- + +# Upstream dependencies of `pipelex-integrate` — what matters + +A reading companion to `design.md` §8. The design ships the skill on today's surfaces and carries a stopgap for each of the items below; this file says what each item is, why it matters to the skill, what the stopgap costs, and what was decided about it on 2026-08-30. None of these items is a member of the build-retirement epic (`L-260829-848001`) or appears in its plan: that program is about the input-form descriptor route and retiring `/v1/build/*`, while these all sit on the codegen trust-chain axis. The two campaigns touch at exactly one point, noted under the `pipelex-mcp` item. + +## The items + +- **The ts-zod wire-`null` defect — `L-260820-ee327d`, owner `pipelex`, P1.** The only real bug in the set. The TypeScript emitter projects an optional field as `.optional()`, which accepts an *absent* value but rejects `null`; the runtime serializes an unset optional as an explicit `null` (`model_dump` with no `exclude_none`). So a generated TypeScript binder fails on the runtime's own output the moment a method leaves an optional field empty. Python is unaffected (`str | None = None`). The fix is a one-line emitter change to `.nullish()`. **Stopgap:** the schema-guided `wireOutput` helper the skill copies once into each TypeScript project (design §4.8) — a file the plugin owns living in user repos, deleted when the fix ships. **Decision 2026-08-30:** Louis prioritizes this fix ahead of the skill's Phase 3 dogfood; the plan's Phase 1 pre-flight re-check strikes `wire-output.ts` before it is written if the fix has shipped in the hosted engine. + +- **No offline check CLI in `@pipelex/sdk` — `L-260820-2ba0f4`, owner `pipelex-sdk-js`, P3.** The SDK exports the drift check as a pure function (`runCodegenCheck`) and ships no command, so the skill copies a small `codegen-check.mjs` into each TypeScript project and wires it into the project's existing aggregate gate. It works; it is ergonomics. When the SDK gains the CLI, the gate becomes one line and the reference script stops being copied (design §4.5). Low urgency. + +- **No offline check at all in `pipelex-sdk` — `L-260830-4e43cd`, owner `pipelex-sdk-python`.** The Python SDK has no equivalent of even the pure function. The only check that exists, `pipelex codegen check`, lives in the `pipelex` runtime, which a `python-pydantic` consumer deliberately does not install — and the skill will not add the runtime as a dependency to get a gate (design §4.5). So Python integrations get **no CI gate**: nothing in CI proves the generated tree matches the bundle. **Stopgap:** the `sources.json` hashes plus the staleness notice in `pipelex-edit` / `pipelex-design` (design §7), and the report says the asymmetry plainly. This is the largest user-visible gap and it is a porting job, not a design question. The `python-structures` audience is the exception: its project already depends on `pipelex` and gets `pipelex codegen check` wired in. + +- **The main-pipe signature is invisible to the model — `L-260830-e8b2e0`, owner `pipelex-mcp`.** When the method comes from a `method_ref` or `method_id` rather than local files, nothing the workshop returns names the output concept: `mthds_validate` carries `main_pipe_ref` and `pipe_io_contracts` on the view-only `_meta` channel, which never reaches the model. **Stopgap:** the heuristic of design §4.3 (candidate outputs are the generated types minus the input concepts minus natives; one candidate is taken as a stated assumption, several become one question). Files-based integrations, the recommended shape, are unaffected. **The one touch point with build-retirement:** that program moves the workshop onto `POST /v1/input-form` (member `L-260829-dfaed4`), and the descriptor it fetches already carries `main_pipe_ref` and the inputs' concept refs — but **not** the output concept and its multiplicity, which is the half the heuristic guesses. **Decision 2026-08-30:** linked *related* to the epic, not as a member, and best sequenced right after `L-260829-dfaed4`, when exposing the signature in `structuredContent` is nearly free. + +- **The pipe selector — `L-260829-563e9e`, owner `workspace`.** Not a gap in the skill, only a dependency to know about: the run routes do not yet accept `pipe_ref`, so the call site sends `pipe_code` today, and its one selector line changes when the campaign lands. Already ratified (`L-260830-a67fff`) and gating build-retirement's Wave A2; nothing to do from here. + +## What this means for the skill + +- The skill does not wait on any of these. Each stopgap names the item that deletes it, so the design's shape holds whether an item lands before or after the skill ships. +- The order that matters is the one Louis set: `L-260820-ee327d` first (it removes a file from every TypeScript user's codebase), then `L-260830-e8b2e0` once the workshop holds the descriptor, with the two SDK check items behind them. +- The Python gate gap (`L-260830-4e43cd`) is the one users will notice. Until it lands, the editing skills' staleness notice is the only forgetting-guard Python has, which is why that notice is part of the skill's family wiring rather than an optional nicety. From 54cf04dcd0eff36737f3b4b56bdc94aba297fd91 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 00:11:58 +0200 Subject: [PATCH 02/21] Add pipelex-integrate and pipelex-scaffold, the campaign's two new skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pipelex-integrate` wires an MTHDS method into an existing TypeScript or Python codebase: it reads the method's signature from the validate verdict's `main_pipe`, adds the formatter and linter exclusions before anything is generated, drives `mthds_codegen`'s write arm so no artifact byte passes through the model, writes a `sources.json` sidecar, an offline drift gate and a typed call site, and defers wholesale to a project that already owns a codegen harness. `pipelex-scaffold` is the front door to a new project: a starter through its own bootstrap skill, or the ecosystem's own initializer, and it is MCP-free. Both ship `references/` — the second and third skills to do so — including `codegen-check.mjs`, the offline gate copied into TypeScript projects. `wire-output.ts` was struck before it was written: the ts-zod `.nullish()` emitter fix shipped in pipelex v0.56.0, which makes the helper lossy rather than protective. The rest of the family gained one-line wiring: `pipelex-edit` and `pipelex-design` announce a stale sidecar and offer the refresh, `pipelex-design`'s delivery forks between the two new skills on whether the workspace holds a project manifest, and `pipelex-inputs` hands off at its closing report. Advances L-260830-344594 Advances L-260906-8ac105 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DYrbgEm55V4QP3uJmgTqRS --- CHANGELOG.md | 3 + CLAUDE.md | 10 +- README.md | 8 +- docs/build-targets.md | 2 +- docs/decisions.md | 17 +- pipelex-codex/skills/pipelex-design/SKILL.md | 4 +- pipelex-codex/skills/pipelex-edit/SKILL.md | 2 + pipelex-codex/skills/pipelex-inputs/SKILL.md | 2 + .../skills/pipelex-integrate/SKILL.md | 209 +++++++++++++++++ .../references/codegen-check.mjs | 166 +++++++++++++ .../pipelex-integrate/references/python.md | 102 ++++++++ .../references/typescript.md | 118 ++++++++++ .../skills/pipelex-scaffold/SKILL.md | 158 +++++++++++++ .../references/initializers.md | 38 +++ .../pipelex-scaffold/references/starters.md | 59 +++++ pipelex-vibe/skills/pipelex-design/SKILL.md | 4 +- pipelex-vibe/skills/pipelex-edit/SKILL.md | 2 + pipelex-vibe/skills/pipelex-inputs/SKILL.md | 2 + .../skills/pipelex-integrate/SKILL.md | 209 +++++++++++++++++ .../references/codegen-check.mjs | 166 +++++++++++++ .../pipelex-integrate/references/python.md | 102 ++++++++ .../references/typescript.md | 118 ++++++++++ pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 158 +++++++++++++ .../references/initializers.md | 38 +++ .../pipelex-scaffold/references/starters.md | 59 +++++ pipelex/skills/pipelex-design/SKILL.md | 4 +- pipelex/skills/pipelex-edit/SKILL.md | 2 + pipelex/skills/pipelex-inputs/SKILL.md | 2 + pipelex/skills/pipelex-integrate/SKILL.md | 220 ++++++++++++++++++ .../references/codegen-check.mjs | 166 +++++++++++++ .../pipelex-integrate/references/python.md | 102 ++++++++ .../references/typescript.md | 118 ++++++++++ pipelex/skills/pipelex-scaffold/SKILL.md | 165 +++++++++++++ .../references/initializers.md | 38 +++ .../pipelex-scaffold/references/starters.md | 59 +++++ .../references/codegen-check.mjs | 166 +++++++++++++ skills/pipelex-integrate/references/python.md | 102 ++++++++ .../references/typescript.md | 118 ++++++++++ .../references/initializers.md | 38 +++ .../pipelex-scaffold/references/starters.md | 59 +++++ templates/skills/pipelex-design/SKILL.md.j2 | 4 +- templates/skills/pipelex-edit/SKILL.md.j2 | 2 + templates/skills/pipelex-inputs/SKILL.md.j2 | 2 + .../skills/pipelex-integrate/SKILL.md.j2 | 215 +++++++++++++++++ templates/skills/pipelex-scaffold/SKILL.md.j2 | 158 +++++++++++++ tests/unit/test_gen_skill_docs.py | 2 +- tests/unit/test_pipelex_integrate_skill.py | 132 +++++++++++ tests/unit/test_pipelex_scaffold_skill.py | 111 +++++++++ wip/pipelex-integrate/design.md | 46 ++-- wip/pipelex-integrate/plan.md | 163 +++++++++---- wip/pipelex-integrate/scaffold-design.md | 107 +++++++++ 51 files changed, 3980 insertions(+), 77 deletions(-) create mode 100644 pipelex-codex/skills/pipelex-integrate/SKILL.md create mode 100644 pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs create mode 100644 pipelex-codex/skills/pipelex-integrate/references/python.md create mode 100644 pipelex-codex/skills/pipelex-integrate/references/typescript.md create mode 100644 pipelex-codex/skills/pipelex-scaffold/SKILL.md create mode 100644 pipelex-codex/skills/pipelex-scaffold/references/initializers.md create mode 100644 pipelex-codex/skills/pipelex-scaffold/references/starters.md create mode 100644 pipelex-vibe/skills/pipelex-integrate/SKILL.md create mode 100644 pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs create mode 100644 pipelex-vibe/skills/pipelex-integrate/references/python.md create mode 100644 pipelex-vibe/skills/pipelex-integrate/references/typescript.md create mode 100644 pipelex-vibe/skills/pipelex-scaffold/SKILL.md create mode 100644 pipelex-vibe/skills/pipelex-scaffold/references/initializers.md create mode 100644 pipelex-vibe/skills/pipelex-scaffold/references/starters.md create mode 100644 pipelex/skills/pipelex-integrate/SKILL.md create mode 100644 pipelex/skills/pipelex-integrate/references/codegen-check.mjs create mode 100644 pipelex/skills/pipelex-integrate/references/python.md create mode 100644 pipelex/skills/pipelex-integrate/references/typescript.md create mode 100644 pipelex/skills/pipelex-scaffold/SKILL.md create mode 100644 pipelex/skills/pipelex-scaffold/references/initializers.md create mode 100644 pipelex/skills/pipelex-scaffold/references/starters.md create mode 100644 skills/pipelex-integrate/references/codegen-check.mjs create mode 100644 skills/pipelex-integrate/references/python.md create mode 100644 skills/pipelex-integrate/references/typescript.md create mode 100644 skills/pipelex-scaffold/references/initializers.md create mode 100644 skills/pipelex-scaffold/references/starters.md create mode 100644 templates/skills/pipelex-integrate/SKILL.md.j2 create mode 100644 templates/skills/pipelex-scaffold/SKILL.md.j2 create mode 100644 tests/unit/test_pipelex_integrate_skill.py create mode 100644 tests/unit/test_pipelex_scaffold_skill.py create mode 100644 wip/pipelex-integrate/scaffold-design.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 2687553..a914430 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,15 @@ ### Added +- **`pipelex-integrate` — wire an MTHDS method into a Python or TypeScript codebase.** Given a local bundle, a published `method_ref` at a tag, or a catalog `method_id`, and a project, the skill picks the codegen target by audience (`ts-zod`; `python-pydantic` for a hosted-API consumer, `python-structures` for a Pipelex host), has the workshop's `mthds_codegen` write the generated tree into one dedicated directory per method through its `output_dir` arm — no artifact byte ever passes through the model, and a refused write is never worked around by writing bytes from the conversation — excludes that tree from the project's formatters and linters *before* it exists while keeping it under the type checker, records a `sources.json` sidecar (selector, target, pipe signature, source hashes) so a second run is a refresh and a bundle edit is detectable, wires an offline drift gate (`scripts/codegen-check.mjs` over `@pipelex/sdk`'s `runCodegenCheck`) into a TypeScript project's existing check, and writes one typed call-site module per method over `startAndWaitForResult` / `start_and_wait` and the generated binder or model. The call site is typed from the main pipe's signature on the validate verdict. A Python consumer gets no offline gate yet — the Python SDK has no check and the skill will not add the `pipelex` runtime to get one — and the report says so. A project made from a Pipelex starter keeps its own codegen harness: the skill runs the project's `codegen` script or `make add-method` and never writes a second layout beside the first. Ships `references/typescript.md`, `references/python.md` and `references/codegen-check.mjs`. +- **`pipelex-scaffold` — the front door to a project that does not exist yet.** Two branches and no templates of its own: one of the Pipelex starters (`pipelex-starter-js` for a web app with contract-rendered forms, `pipelex-starter-python` for a CLI or service), acquired as a fresh-history local clone by default or through `gh repo create --template` after confirmation, committed once as it came, then renamed by the clone's **own** `bootstrap` skill, read from its `SKILL.md` and never reimplemented; or the ecosystem's initializer (`uv init --package`, `npm create next-app@latest`, …) when the user wants their framework. Both branches end with the env-file convention, the key filled only from the shell environment and never asked for in the conversation, and a hand-off to `/pipelex-integrate`. It is the plugin's third MCP-free skill. Ships `references/starters.md` and `references/initializers.md`. - **`pipelex-synthetic-inputs` — a skill that renders the files a method needs, from code.** PDFs through `reportlab` (canvas letters, multi-page Platypus reports, tables, and a composed line-item document whose totals come from its items) and PNGs through `Pillow` and `matplotlib` in four categories: `chart` (bar, line, pie, scatter), `diagram` (a node/edge list laid out on a grid with clipped arrows), `document_scan` (an A4-at-150-dpi page put through a seeded skew/tint/grain/vignette post-process, for OCR and document-understanding methods) and `screenshot` (window chrome, sidebar, stat tiles, and a status-badged table or card grid). Word and Excel come along from `pipelex-inputs`. No AI is involved anywhere, and only packages whose licences are compatible with MIT are used. **Photographs and handwriting are deliberately out of scope** — code cannot render either to a standard a vision model would accept, so the skill says so and asks for a real file instead of handing a method an imitation. It is MCP-free, the second such skill after `pipelex-explain`, and it installs what it needs itself: `uv` with ephemeral packages, or a venv it creates under the user's cache directory when `uv` is absent. Nothing is installed into the project, and installing a *tool* always asks first. - **The recipes refuse to render a file that would be silently wrong.** A recipe is copied and adapted, so the content block is where things go wrong: each one now bounds its content against the page it is drawing on and stops with the cure rather than exiting 0 on a file that lies. A `document_scan` whose items overrun the page, a `screenshot` whose rows run off the canvas, a `diagram` with two nodes in one grid cell, an unsubstituted `` still in the path — every one of these used to print success and hand a method an input missing exactly the field it was meant to read. Line-item descriptions wrap instead of overprinting the quantity beside them, the PDF table recipe sizes its columns instead of drawing them off the paper, and every recipe renders beside its target and renames on success, so a crash can no longer truncate a file the user already had. - **The shipped recipes are executed by tests.** `tests/recipes` extracts every runnable block from the skill's reference files and runs it, asserting the declared file exists with the right magic bytes, a PDF's page objects, and a PNG's declared pixel size — on both rungs of the environment ladder. The predecessor's image recipe pointed at a bundle that no longer existed and nothing noticed, which is the failure this closes. Opt in with `make test-recipes`; the default run deselects the `recipes` marker because a cold `uv` cache downloads packages. ### Changed +- **The family hands the method to the code.** `pipelex-design`'s delivery and `pipelex-inputs`' closing report point at `/pipelex-integrate` when the workspace holds a codebase and, for the design skill, at `/pipelex-scaffold` when it holds none; `pipelex-edit` and `pipelex-design`'s re-entry announce stale generated types after a bundle edit by reading the `sources.json` sidecars `pipelex-integrate` writes — the only drift guard a Python consumer has until its SDK gains an offline check. The plugin's light-template pin (`explicit: false` on every `mthds_inputs_template` call) gains one exception: `pipelex-integrate` passes `explicit: true` on its fallback path, when the verdict carries no main-pipe signature, because it needs each input's concept ref to type a parameter. - **`pipelex-design` is model-invocable.** The skill shipped `disable-model-invocation: true` on Claude and Vibe, so it could only be reached by typing `/pipelex-design`. That made `pipelex-edit`'s routing a dead end — it classified a structural change, then had to hand the user a slash command to type and throw away the baseline verdict it had just produced. The flag is gone on every target, the skill's description now carries natural-language triggers ("design a method", "create a pipeline", "add a step", "rewire this pipeline") so the model can actually reach it, and `pipelex-edit` names the affected pipes and invokes `/pipelex-design` directly instead of stopping. The consent gate stays where it belongs: the design run still announces its captured contract in one line before writing anything. - **`pipelex-inputs` delegates file generation instead of doing it inline.** (Breaking) Its Document Generation section and Fallback Strategy block are gone, and the `native.Image` / `native.Document` strategy rows now delegate to `pipelex-synthetic-inputs`. "Generate File Inputs" is the delegation contract — the request fields to pass, the bare path that comes back and goes into `inputs.json`, and what happens when the factory cannot deliver: that one input is left unfilled with the reason in the report, and the rest of the flow continues. Anyone relying on the inline recipes should read them in the new skill's `references/`, where they are deeper and now executed by tests. - **Tooling:** Pinned `ruff` to an exact `0.16.4`, replacing the `>=0.6.8` floor. The exact pin matches what the Ruff VS Code extension now bundles, which matters because Ruff 0.16 lints `pyproject.toml` itself: the extension syncs the config file to the language server, and a pre-0.16 binary parses it as Python source and paints phantom `invalid-syntax` diagnostics on lines like `requires-python`. A floor let the editor and the CLI resolve to different binaries; an exact pin cannot. This is a dev dependency, so nothing shipped changes, and the upgrade produced no lint findings and no reformatting. diff --git a/CLAUDE.md b/CLAUDE.md index aefb0e8..cc146c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -35,6 +35,8 @@ templates/ # SOURCE OF TRUTH — all .j2 templates live here │ ├── pipelex-edit/SKILL.md.j2 # Contract-preserving edits to an existing bundle; routes structural changes to pipelex-design (MCP-backed) │ ├── pipelex-inputs/SKILL.md.j2 # inputs.json preparation (MCP-backed) │ ├── pipelex-synthetic-inputs/SKILL.md.j2 # File factory: render PDFs/PNGs/Office files from code, no AI (no MCP dependency) +│ ├── pipelex-integrate/SKILL.md.j2 # Wire a method into a TS/Python codebase: codegen write arm, exclusions, sidecar, gate, typed call site (MCP-backed) +│ ├── pipelex-scaffold/SKILL.md.j2 # Front door to a new project: a starter through its own bootstrap, or the ecosystem's initializer (no MCP dependency) │ └── shared/ │ ├── frontmatter.md.j2 # Common YAML frontmatter (included by templates) │ ├── mthds-reference.md.j2 # MTHDS language reference (rendered per target) @@ -49,7 +51,9 @@ templates/ # SOURCE OF TRUTH — all .j2 templates live here └── assets/check.mjs # Vendored wasm+API validation bundle (static asset, built in pipelex-sdk-js) skills/ # SOURCE OF TRUTH for static (non-templated) skill assets, copied verbatim into every target ├── pipelex-design/references/writing-mthds.md # MTHDS authoring reference -└── pipelex-synthetic-inputs/references/ # pdf.md, png.md, office.md — runnable recipes, executed by tests/recipes +├── pipelex-synthetic-inputs/references/ # pdf.md, png.md, office.md — runnable recipes, executed by tests/recipes +├── pipelex-integrate/references/ # typescript.md, python.md, codegen-check.mjs — detection tables, call-site templates, the offline gate copied into TS projects +└── pipelex-scaffold/references/ # starters.md, initializers.md — the two starters side by side, the ecosystem initializers pipelex/ # Claude prod plugin (generated, checked in) pipelex-codex/ # Codex plugin (generated, checked in) pipelex-vibe/ # Mistral Vibe target (generated, checked in; loaded via skill_paths) @@ -137,6 +141,6 @@ So there is nothing to enable — the bundled hook loads on its own (hooks are S ## Key dependency -The plugin imports nothing and requires no install. Validation rides on the vendored `check.mjs` bundle (wasm engine + `@pipelex/sdk` → hosted API) and, for the MCP-backed skills (`pipelex-design`, `pipelex-organize`, `pipelex-edit`, `pipelex-inputs`), on the plugin-declared `pipelex-mcp` server (tools `mthds_validate` / `mthds_inputs_template`; `mthds_prepare_inputs`, which uploads `pipelex-inputs`' file-bearing values to Pipelex storage and rewrites them to `pipelex-storage://` references so a run can reach them; plus the `mthds_run` family powering `pipelex-inputs`' closing offer to run; declared in the Claude and Codex manifests, manual registration on Vibe). The baked declaration is the **local workshop launcher** — `npx -y @pipelex/mcp@latest` over stdio, from the `[vars.mcp_server]` block in `targets/defaults.toml` — never a hosted URL: the hosted console is a connector users add in their host's own UI (see the README's "One install, one server" section and `docs/decisions.md`). Credential delivery: on Claude the manifest's `userConfig` prompts for the API key / base URL at enable time (keychain-stored) and the MCP entry spawns the `launch-pipelex-mcp.sh` wrapper, which receives them as `PIPELEX_PLUGIN_*` via `${user_config.*}` substitution and promotes each to `PIPELEX_API_KEY`/`PIPELEX_BASE_URL` **only when non-empty** — the canonical credential channel (required for Claude Desktop, which carries no shell env), with the non-empty guard keeping an unfilled option from shadowing a shell-exported key; injecting `PIPELEX_*` directly instead makes an empty option surface as a config-class `Unauthorized` that hard-stops every MCP-backed skill; on Codex the manifest forwards `PIPELEX_API_KEY`/`PIPELEX_BASE_URL` by name via `env_vars` because Codex whitelist-filters MCP spawn env. Dev override: point `command`/`args` at a local checkout in `targets/defaults.toml` + `make build` on Claude; a same-named `[mcp_servers.pipelex]` entry in `~/.codex/config.toml` on Codex. +The plugin imports nothing and requires no install. Validation rides on the vendored `check.mjs` bundle (wasm engine + `@pipelex/sdk` → hosted API) and, for the MCP-backed skills (`pipelex-design`, `pipelex-organize`, `pipelex-edit`, `pipelex-inputs`, `pipelex-integrate`), on the plugin-declared `pipelex-mcp` server (tools `mthds_validate` / `mthds_inputs_template`; `mthds_codegen`, whose write arm — `output_dir`, relative to the workshop's working directory — writes `pipelex-integrate`'s generated trees to disk so no artifact byte passes through the model; `mthds_prepare_inputs`, which uploads `pipelex-inputs`' file-bearing values to Pipelex storage and rewrites them to `pipelex-storage://` references so a run can reach them; plus the `mthds_run` family powering `pipelex-inputs`' closing offer to run; declared in the Claude and Codex manifests, manual registration on Vibe). The baked declaration is the **local workshop launcher** — `npx -y @pipelex/mcp@latest` over stdio, from the `[vars.mcp_server]` block in `targets/defaults.toml` — never a hosted URL: the hosted console is a connector users add in their host's own UI (see the README's "One install, one server" section and `docs/decisions.md`). Credential delivery: on Claude the manifest's `userConfig` prompts for the API key / base URL at enable time (keychain-stored) and the MCP entry spawns the `launch-pipelex-mcp.sh` wrapper, which receives them as `PIPELEX_PLUGIN_*` via `${user_config.*}` substitution and promotes each to `PIPELEX_API_KEY`/`PIPELEX_BASE_URL` **only when non-empty** — the canonical credential channel (required for Claude Desktop, which carries no shell env), with the non-empty guard keeping an unfilled option from shadowing a shell-exported key; injecting `PIPELEX_*` directly instead makes an empty option surface as a config-class `Unauthorized` that hard-stops every MCP-backed skill; on Codex the manifest forwards `PIPELEX_API_KEY`/`PIPELEX_BASE_URL` by name via `env_vars` because Codex whitelist-filters MCP spawn env. Dev override: point `command`/`args` at a local checkout in `targets/defaults.toml` + `make build` on Claude; a same-named `[mcp_servers.pipelex]` entry in `~/.codex/config.toml` on Codex. -**`pipelex-synthetic-inputs` depends on none of that.** It is the second MCP-free skill after `pipelex-explain` — no tool, no key, no Pipelex service — and its only dependency is a Python it can reach: `uv` with ephemeral `--with` packages on the normal rung, and a venv it creates itself under `${XDG_CACHE_HOME:-$HOME/.cache}/pipelex-plugins/synth-venv` when `uv` is absent. Swapping the runner line is the *only* difference between the two rungs, and `tests/recipes` proves it by running real recipes through both. When neither rung is reachable the skill stops with the exact missing piece and, called from `pipelex-inputs`, returns no path so that one input is left unfilled rather than aborting the flow. Keep it out of the `MCP_SKILLS` tuple in `tests/unit/test_gen_skill_docs.py`. +**`pipelex-synthetic-inputs` depends on none of that.** It is the second MCP-free skill after `pipelex-explain` — no tool, no key, no Pipelex service — (`pipelex-scaffold` is the third: git, the starters' own bootstrap scripts and the ecosystem's initializers are all it uses, and it hands every project it creates to `pipelex-integrate`) and its only dependency is a Python it can reach: `uv` with ephemeral `--with` packages on the normal rung, and a venv it creates itself under `${XDG_CACHE_HOME:-$HOME/.cache}/pipelex-plugins/synth-venv` when `uv` is absent. Swapping the runner line is the *only* difference between the two rungs, and `tests/recipes` proves it by running real recipes through both. When neither rung is reachable the skill stops with the exact missing piece and, called from `pipelex-inputs`, returns no path so that one input is left unfilled rather than aborting the flow. Keep it and `pipelex-scaffold` out of the `MCP_SKILLS` tuple in `tests/unit/test_gen_skill_docs.py`. diff --git a/README.md b/README.md index a051a01..ac395de 100644 --- a/README.md +++ b/README.md @@ -8,11 +8,11 @@ This is the plugin generation that pairs with the hosted Pipelex API and the (cl ## What's inside -- **Skills** — Pipelex skills for working with MTHDS bundles: `pipelex-explain` (read and explain a bundle), `pipelex-design` (design a method contract-first with a complexity-adaptive workflow: build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven refinement for deep, uncertain, staged, or resumable work), `pipelex-organize` (regroup a construction-shaped or otherwise awkward layout into coherent module files — or a single file when the method is simple — proven equivalent through the MCP validation verdict; automatically follows `pipelex-design` only when the converged layout needs it), `pipelex-edit` (contract-preserving edits to an existing bundle — prompts, model references, mechanical renames — proven with a before/after MCP validation verdict; structural or contract changes route to `/pipelex-design`, which re-enters the affected region directly or through signatures according to its complexity), `pipelex-inputs` (prepare an `inputs.json` for a method — placeholder template, synthetic data, user files, or a mix — from the input template the MCP server projects; targets a local bundle or a registered catalog method by its `mt_…` id, uploads any local files to Pipelex storage so the run can reach them, and closes by offering to start the run through the MCP run tools), and `pipelex-synthetic-inputs` (render the files a method needs when the user has none — PDFs through reportlab, PNGs through Pillow and matplotlib as charts, diagrams, scanned-looking pages or app screenshots, plus Word and Excel; `pipelex-inputs` delegates to it for every file-typed input). - `pipelex-synthetic-inputs` is the plugin's second MCP-free skill after `pipelex-explain`: it needs no API key and no Pipelex service. It renders files from code — no image-generation model, no hosted method — using only packages whose licences are compatible with MIT, and it installs them itself, through `uv`'s ephemeral environments or an isolated venv under your cache directory when `uv` is absent. Nothing is installed into your project, and installing a *tool* always asks first. Photographs and handwriting are deliberately out of scope: code cannot render either convincingly, and the skill asks for your own file rather than handing a method an imitation. +- **Skills** — Pipelex skills for working with MTHDS bundles: `pipelex-explain` (read and explain a bundle), `pipelex-design` (design a method contract-first with a complexity-adaptive workflow: build a fully understood shallow graph directly as a coherent runnable bundle, or use validated signature-driven refinement for deep, uncertain, staged, or resumable work), `pipelex-organize` (regroup a construction-shaped or otherwise awkward layout into coherent module files — or a single file when the method is simple — proven equivalent through the MCP validation verdict; automatically follows `pipelex-design` only when the converged layout needs it), `pipelex-edit` (contract-preserving edits to an existing bundle — prompts, model references, mechanical renames — proven with a before/after MCP validation verdict; structural or contract changes route to `/pipelex-design`, which re-enters the affected region directly or through signatures according to its complexity), `pipelex-inputs` (prepare an `inputs.json` for a method — placeholder template, synthetic data, user files, or a mix — from the input template the MCP server projects; targets a local bundle or a registered catalog method by its `mt_…` id, uploads any local files to Pipelex storage so the run can reach them, and closes by offering to start the run through the MCP run tools), `pipelex-synthetic-inputs` (render the files a method needs when the user has none — PDFs through reportlab, PNGs through Pillow and matplotlib as charts, diagrams, scanned-looking pages or app screenshots, plus Word and Excel; `pipelex-inputs` delegates to it for every file-typed input), `pipelex-integrate` (wire a method into an existing TypeScript or Python codebase: the workshop's `mthds_codegen` writes drift-proof generated types into one directory per method, the skill excludes that tree from formatters, records a `sources.json` sidecar, wires the offline drift check into the project's gate on TypeScript, and writes one typed call site that runs the method through `@pipelex/sdk` or `pipelex-sdk`; a project made from a Pipelex starter keeps its own codegen harness), and `pipelex-scaffold` (start a project where none exists — one of the Pipelex starter templates run through its own `bootstrap` skill, or the ecosystem's initializer — and hand it to `pipelex-integrate`). + `pipelex-synthetic-inputs` is the plugin's second MCP-free skill after `pipelex-explain` (`pipelex-scaffold` is the third — it needs only git, the starters' own scripts and the ecosystem's initializers): it needs no API key and no Pipelex service. It renders files from code — no image-generation model, no hosted method — using only packages whose licences are compatible with MIT, and it installs them itself, through `uv`'s ephemeral environments or an isolated venv under your cache directory when `uv` is absent. Nothing is installed into your project, and installing a *tool* always asks first. Photographs and handwriting are deliberately out of scope: code cannot render either convincingly, and the skill asks for your own file rather than handing a method an imitation. - **Hooks** — a CLI-free validation hook that checks `.mthds` files on edit (Claude/Codex `PostToolUse`, Mistral Vibe `post_tool`). On every target, lint and format run locally through a bundled WASM engine (offline, no credentials — the file is also auto-formatted in place), and full semantic validation calls the hosted Pipelex API when `PIPELEX_API_KEY` is set. Everything fails open: no Node → the hook no-ops; no key / API unreachable → only the validate stage is skipped. See [docs/hooks.md](docs/hooks.md). -- **MCP server declaration** — on Claude Code and Codex the plugin declares the `pipelex-mcp` server as the **local workshop launcher** (`npx -y @pipelex/mcp@latest`, stdio; tools `mthds_validate` for bundle validation, `mthds_inputs_template` for input templates, `mthds_prepare_inputs` to upload a filled template's file values to Pipelex storage so a run can reach them, and the `mthds_run` family for durable runs — each takes submitted file contents or a registered method's catalog id (`mt_…`) as `method_id`, operating on the method's current stored content; by-id calls require an API key since the catalog is org-scoped), which the MCP-backed skills require. The harness spawns it automatically at session start, and it authenticates to the Pipelex API with the key from the plugin configuration (prompted at enable time on Claude Code) or, as a fallback, `PIPELEX_API_KEY` from your session environment — the same credential the hook uses. Unlike the fail-open hook, the MCP-backed skills stop with a setup instruction when the tools are absent. The hosted console is never baked into the plugin — see [One install, one server](#one-install-one-server--workshop-vs-console) and [docs/decisions.md](docs/decisions.md). +- **MCP server declaration** — on Claude Code and Codex the plugin declares the `pipelex-mcp` server as the **local workshop launcher** (`npx -y @pipelex/mcp@latest`, stdio; tools `mthds_validate` for bundle validation (its verdict carries the main pipe's signature, which `pipelex-integrate` types call sites from), `mthds_inputs_template` for input templates, `mthds_codegen` to project a method's concepts into typed code (`ts-zod`, `python-pydantic`, `python-structures`) and, with `output_dir`, write the tree to disk, `mthds_prepare_inputs` to upload a filled template's file values to Pipelex storage so a run can reach them, and the `mthds_run` family for durable runs — each takes submitted file contents or a registered method's catalog id (`mt_…`) as `method_id`, operating on the method's current stored content; by-id calls require an API key since the catalog is org-scoped), which the MCP-backed skills require. The harness spawns it automatically at session start, and it authenticates to the Pipelex API with the key from the plugin configuration (prompted at enable time on Claude Code) or, as a fallback, `PIPELEX_API_KEY` from your session environment — the same credential the hook uses. Unlike the fail-open hook, the MCP-backed skills stop with a setup instruction when the tools are absent. The hosted console is never baked into the plugin — see [One install, one server](#one-install-one-server--workshop-vs-console) and [docs/decisions.md](docs/decisions.md). - **Language reference** — the shared MTHDS language reference docs that ground the skills (the *language* stays MTHDS — that's the standard; Pipelex is the tooling, product, and service). ## Install @@ -39,7 +39,7 @@ A value set in the plugin configuration wins over the environment; an empty plug Everything fails open: with no key (or the API unreachable) the local lint/format verdicts still apply and only the validate stage is skipped — no blocked edits, no nagging. **Privacy note:** with a key set, the `.mthds` files around the edited one are sent to the API on each validate call; leave the key unset (both channels) to keep validation fully local (lint/format only). -The plugin also declares the **`pipelex-mcp` server** as the **local workshop launcher** (`npx -y @pipelex/mcp@latest`, stdio), which the MCP-backed skills (`pipelex-design`, `pipelex-organize`, `pipelex-edit`, `pipelex-inputs`) use for validation, input templates, and runs. Claude Code spawns it automatically at session start — no extra install: Node.js is already required by the harness, and the spawned server authenticates with the same plugin-config values (falling back to `PIPELEX_API_KEY` / `PIPELEX_BASE_URL` from the session environment). Without a key the tools still connect, but validation calls return a `config` no-verdict explaining how to set one. To use a local `pipelex-mcp` checkout instead, edit the `[vars.mcp_server]` block in `targets/defaults.toml` (e.g. `command = "node"`, `args = ["/path/to/pipelex-mcp/dist/local/main.js"]`) and rebuild via the dogfood loop below. +The plugin also declares the **`pipelex-mcp` server** as the **local workshop launcher** (`npx -y @pipelex/mcp@latest`, stdio), which the MCP-backed skills (`pipelex-design`, `pipelex-organize`, `pipelex-edit`, `pipelex-inputs`, `pipelex-integrate`) use for validation, input templates, codegen, and runs. Claude Code spawns it automatically at session start — no extra install: Node.js is already required by the harness, and the spawned server authenticates with the same plugin-config values (falling back to `PIPELEX_API_KEY` / `PIPELEX_BASE_URL` from the session environment). Without a key the tools still connect, but validation calls return a `config` no-verdict explaining how to set one. To use a local `pipelex-mcp` checkout instead, edit the `[vars.mcp_server]` block in `targets/defaults.toml` (e.g. `command = "node"`, `args = ["/path/to/pipelex-mcp/dist/local/main.js"]`) and rebuild via the dogfood loop below. ### Codex diff --git a/docs/build-targets.md b/docs/build-targets.md index fe085b6..e2377c6 100644 --- a/docs/build-targets.md +++ b/docs/build-targets.md @@ -37,7 +37,7 @@ scripts/gen_skill_docs.py renders .j2 templates with merged variables **`templates/`** holds all `.j2` source files. Never edit files in `pipelex/`, `pipelex-codex/`, or `pipelex-vibe/` directly — they are generated output. -**`skills/`** at the repo root (if present) holds only static per-skill assets (`references/` subdirectories) that are copied into every target. **`pipelex/`**, **`pipelex-codex/`**, and **`pipelex-vibe/`** are generated output directories (build artifacts checked into git). +**`skills/`** at the repo root (if present) holds only static per-skill assets (`references/` subdirectories) that are copied into every target. Several skills use it — `pipelex-design`, `pipelex-synthetic-inputs`, `pipelex-integrate`, `pipelex-scaffold` — and a reference need not be Markdown: `pipelex-integrate` ships `codegen-check.mjs`, a script the skill copies verbatim into TypeScript projects, so a reference edit is followed by `make build` and, for a script, by running it (`node --check` at the least). **`pipelex/`**, **`pipelex-codex/`**, and **`pipelex-vibe/`** are generated output directories (build artifacts checked into git). ## Target configuration diff --git a/docs/decisions.md b/docs/decisions.md index d7fd57f..d111798 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -117,7 +117,7 @@ Skill adoption follows the target, not the tool surface: **only `pipelex-inputs` **A storage size rejection is terminal; the skill never works around it.** When an `inputs`-located `mthds_prepare_inputs` error reports that any file-bearing input exceeds the service limit, `pipelex-inputs` surfaces the exact message, hint, affected asset, and provided size/limit, then stops the preparation attempt. It must not compress, optimize, re-encode, resize, downsample, split, truncate, extract, convert, derive, or substitute the asset before or after the call, and a preflight size check cannot be used to do so proactively. The user's original, the copied bundle file, and the local-path `inputs.json` remain unchanged; there is no retry with altered content and no run call or offer. Work resumes only if the user supplies a different acceptable input/reference or the service limit changes. This boundary applies to every file type. It is deliberately separate from an unreadable-path failure, where retrying the exact same bytes with an absolute request path remains valid and does not rewrite `inputs.json`. -**The light shape stays pinned: every skill call passes `explicit: false`.** The flip's rationale (concept identity and canonical content shape visible by default) is real, but it is an *authoring-aid* argument, and the three call sites here don't need the aid: `pipelex-design` and `pipelex-edit` only ever *show* or *key-compare* the template, and `pipelex-inputs` fills it against the strategy tables it already carries. Adopting the envelope would mean rewriting Step 2's example, the value-shapes table, and all four fill strategies for no behavior change — `mthds_prepare_inputs` and `mthds_run` accept both shapes identically. So the flag is pinned, not the prose rewritten. Revisit as its own pass if concept identity turns out to improve synthetic-data quality or structured-input filling; the switch is one argument in three places. +**The light shape stays pinned: every skill call passes `explicit: false`.** The flip's rationale (concept identity and canonical content shape visible by default) is real, but it is an *authoring-aid* argument, and the three call sites here don't need the aid: `pipelex-design` and `pipelex-edit` only ever *show* or *key-compare* the template, and `pipelex-inputs` fills it against the strategy tables it already carries. Adopting the envelope would mean rewriting Step 2's example, the value-shapes table, and all four fill strategies for no behavior change — `mthds_prepare_inputs` and `mthds_run` accept both shapes identically. So the flag is pinned, not the prose rewritten. Revisit as its own pass if concept identity turns out to improve synthetic-data quality or structured-input filling; the switch is one argument in three places. **One exception, added 2026-09-06: `pipelex-integrate` passes `explicit: true` on its fallback path** — when the validate verdict carries no `main_pipe` and the user has named the pipe to integrate — because it needs each input's declared concept ref to type a call-site parameter, and the light shape carries values without concepts. On its ordinary path it reads the signature from `structuredContent.main_pipe` and makes no template call at all; the pin stands everywhere else. ## Edit vs design — who owns method modification (2026-07-16) @@ -146,6 +146,21 @@ So file synthesis became **`pipelex-synthetic-inputs`**, a skill of its own. `pi - **Prose recipes, guarded by execution.** The reference files carry complete runnable blocks the agent copies and adapts, rather than shipped generator scripts — the content of a synthetic file differs every time, and a parameterized script would become a JSON DSL for documents. What was missing from the predecessor is the guard, and it is the reason the old image recipe rotted unnoticed: `tests/recipes` extracts every block, runs it, and checks the file's magic bytes, a PDF's page objects and a PNG's declared pixel size. Opt-in (`make test-recipes`), since a cold `uv` cache downloads packages. A router `SKILL.md` plus one reference per format keeps every invocation from paying for every format. - **The skill owns its environment, and asks before installing a tool.** A ladder run once per invocation: `uv` with ephemeral `--with` packages first; failing that, a venv the skill creates under `${XDG_CACHE_HOME:-$HOME/.cache}/pipelex-plugins/synth-venv` and fills itself. Swapping the runner line is the only difference between the rungs, which `tests/recipes` proves by running real recipes through both. Nothing is installed into the user's project; installing a **tool** (`uv` itself, a system package) always asks first, in every mode, because a package in a cache and a binary on the machine are different commitments. When no rung is reachable the skill stops with the exact missing piece and the command that supplies it — and, called from `pipelex-inputs`, returns no path so that one input is left unfilled with its reason in the report, rather than aborting the whole inputs flow. +## The method reaches the code — `pipelex-integrate` and `pipelex-scaffold` (2026-09-06) + +Every skill before these acts on `.mthds` files; none touched the codebase that calls the method, so the plugin stopped helping at the moment a user was happy with a method. Two skills close that gap, decided with Louis on 2026-08-30 (the integrate design's ten boxes) and 2026-09-06 (the widening to greenfield projects, four rulings). The designs are `wip/pipelex-integrate/design.md` and `wip/pipelex-integrate/scaffold-design.md`; the tracker is `plan.md` beside them. + +- **`pipelex-integrate` writes a complete typed call site, and stops there.** One module per method — an async function typed from the pipe's signature, running through the SDK's self-healing lifecycle call and narrowing `main_stuff` through the generated binder or model — plus at most two shared helpers; no tests, routes or UI. The name `pipelex-codegen` was rejected: tools are the contract and skills are the manual, named after user tasks, and codegen is one step of integrating. +- **The write arm is the only arm.** Every `mthds_codegen` call passes `output_dir`; a refused or failed write is a refusal, never a fallback to writing the returned bytes from the conversation — a re-emitted artifact is one trailing newline from a broken stamp. Generated files are never edited, formatted or linted; the tooling exclusions go in before the tree exists; one directory per method; orphans are reported and never deleted. +- **The signature comes from the verdict.** `mthds_validate`'s `structuredContent.main_pipe` types the call site for every selector. The by-elimination heuristic the first design carried for by-ref and by-id sources was made obsolete by `pipelex-mcp` before it shipped and was never written; an older workshop takes a fallback (the template for inputs, the bundle for the output), and a by-ref or by-id source without a signature stops rather than guessing. +- **The sidecar `sources.json` is the skill's only state** — selector, target, pipe record, source hashes — so refresh mode re-derives nothing and a bundle edit is detectable; `pipelex-edit` and `pipelex-design` announce staleness from it. +- **The drift gate is asymmetric, and says so.** TypeScript gets `references/codegen-check.mjs` over `@pipelex/sdk`'s `runCodegenCheck`, wired into the project's existing gate. A `python-pydantic` consumer gets no gate, because the Python SDK has no offline check yet and adding the `pipelex` runtime for one would reverse what the target means; the report says so, and the editing skills' notice is the guard meanwhile. +- **No wire-null helper.** The design carried a schema-guided `wireOutput` helper against the ts-zod `.optional()` defect; the emitter fix (`.nullish()`) shipped in pipelex v0.56.0 before the skill did, and the helper — lossy once the emitter is fixed — is never written. +- **A project that owns a codegen harness keeps it.** A starter-derived project regenerates, checks and scaffolds through its own scripts; the skill defers to them and never writes a second layout beside the first. This is the personas split: the mechanism may move into the SDK, the policy stays with the consumer. +- **`pipelex-scaffold` is the front door, with two branches and no templates of its own.** A project comes from one of the two GitHub-template starters, run through the clone's own `bootstrap` skill (read from its `SKILL.md`, never reimplemented here), or from the ecosystem's initializer when the user wants their framework. Cookiecutter and copier were rejected: the starters are living apps with CI and end-to-end tests that a Jinja-ified template could not keep, and the rename is already a deterministic script with a dry run. The skill acquires with a fresh-history local clone by default (what the template button produces) or `gh repo create --template` after confirmation, makes exactly one commit — the pristine template, so the Python bootstrap's `git mv` works and the rename is a reviewable diff — fills the key only from the shell environment, adds no SDK dependency, and is the plugin's third MCP-free skill. Whether a CLI front door for non-agent users is still wanted is a separate open decision in the workspace ledger. +- **Two skills, not one.** The integrate design is large and the trigger families differ ("use this method in my app" against "start a new project"); the greenfield tail is integrate anyway, so the hand-off chain is the plugin's existing pattern. +- **References carry the language detail, and one of them is a script.** `pipelex-integrate` ships `typescript.md`, `python.md` and `codegen-check.mjs` — the first non-Markdown reference, copied byte for byte into every target and then into users' TypeScript projects, so a change to it is verified by running it, not by rendering it; `pipelex-scaffold` ships `starters.md` and `initializers.md`. + ## License & distribution **Apache 2.0**; repo made public when ready (required for easy marketplace install). Versions start at **0.1.0** (plugin and marketplace). GitHub home assumed `Pipelex/pipelex-plugins` — confirm at first push. diff --git a/pipelex-codex/skills/pipelex-design/SKILL.md b/pipelex-codex/skills/pipelex-design/SKILL.md index 70daa01..4cfb7fc 100644 --- a/pipelex-codex/skills/pipelex-design/SKILL.md +++ b/pipelex-codex/skills/pipelex-design/SKILL.md @@ -182,7 +182,7 @@ After the gate: 1. **Organize only when the layout needs it.** A direct result that is already coherent skips `/pipelex-organize`. A converged stepwise result normally invokes it automatically because one-definition-per-file construction history and satisfied headers need regrouping. A naturally coherent result in either mode does not take an organization round trip solely for process compliance. 2. **Project the input schema.** Call `mthds_inputs_template` with the final whole-bundle `files` submission plus `explicit: false`. Show the returned compact template, but **do not save it as `inputs.json`** — input preparation belongs exclusively to `/pipelex-inputs`. 3. **Present the flow.** Point to the interactive method graph where the host rendered the valid verdict's view; in terminal hosts, present a concise text flow of the final structure. -4. **Hand off inputs.** Suggest preparing real inputs with `/pipelex-inputs`. +4. **Hand off inputs — and the code.** Suggest preparing real inputs with `/pipelex-inputs`. Then, when the workspace holds a codebase (a `package.json` or a `pyproject.toml`), say that `/pipelex-integrate` wires the method into it with generated types and a typed call site; when it holds none and the user wants an application around the method, `/pipelex-scaffold` creates one and hands it to `/pipelex-integrate`. > **NEVER write `inputs.json` manually.** If the user provides files, paths, or wants to run with real data, invoke `/pipelex-inputs` — it handles the template, path resolution, placeholder formatting, and file copying. @@ -207,7 +207,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex-codex/skills/pipelex-edit/SKILL.md b/pipelex-codex/skills/pipelex-edit/SKILL.md index 0873f06..856d473 100644 --- a/pipelex-codex/skills/pipelex-edit/SKILL.md +++ b/pipelex-codex/skills/pipelex-edit/SKILL.md @@ -76,6 +76,8 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. +**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. + ## Reference - [MTHDS Language Reference](../shared/mthds-reference.md) — read for concept definitions and syntax before editing constructs you haven't touched recently diff --git a/pipelex-codex/skills/pipelex-inputs/SKILL.md b/pipelex-codex/skills/pipelex-inputs/SKILL.md index 5736336..d1f76c5 100644 --- a/pipelex-codex/skills/pipelex-inputs/SKILL.md +++ b/pipelex-codex/skills/pipelex-inputs/SKILL.md @@ -353,6 +353,8 @@ After assembling the inputs, confirm readiness: (Or, for the Template strategy: point out which placeholders the user still needs to fill.) +When the workspace holds a codebase (a `package.json` or a `pyproject.toml`) and the method is not yet wired into it, add one line: `/pipelex-integrate` generates the method's types into the project and writes a typed call site that runs it. + ### Offer to run When the inputs are complete, close by offering to run the method. Offer — never start unprompted: a run executes on the hosted Pipelex API and **spends inference credit**. diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md new file mode 100644 index 0000000..482f7fe --- /dev/null +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -0,0 +1,209 @@ +--- +name: pipelex-integrate +description: Wire an MTHDS method into a Python or TypeScript codebase with generated, drift-proof types and one typed call site that runs it through @pipelex/sdk or pipelex-sdk. Use when the user says "use this method in my app", "call this from my code", "generate types for this method", "wire the method into my project", "add this pipeline to my service", "typed client for this method", "integrate the method", "refresh the generated types", "regenerate the types", "the types are stale", or wants application code that runs a .mthds method — from a local bundle, a catalog id (mt_…) or a published method_ref address. Also the refresh path after a bundle edit. Not for authoring or editing the method itself (/pipelex-design, /pipelex-edit), and not for a project that does not exist yet (/pipelex-scaffold). + +--- + +# Integrate an MTHDS method into a codebase + +Take a method — a local `.mthds` bundle, a published address (`method_ref`), or a catalog id (`method_id`) — and a Python or TypeScript project, and leave the project able to call the method with types that cannot silently drift from it. Concretely: + +1. pick the codegen target that matches the project's language **and audience**; +2. have the Pipelex workshop write the generated tree into a dedicated directory per method, through `mthds_codegen`'s write arm, so no generated byte ever passes through you; +3. make the project's formatters and linters leave that tree alone while its type checker keeps covering it; +4. record how the tree was generated in a small sidecar beside the lock, so the next run knows what to refresh and a bundle edit is detectable; +5. wire the offline drift check into the gate the project already runs, where one exists for the language; +6. write one typed call-site module per method, running it through `@pipelex/sdk` or `pipelex-sdk` and narrowing its output with the generated binder or model; +7. verify with the project's own type checker and the gate you just installed. + +Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. + +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. + +## Requirements — the Pipelex MCP tools + +This skill generates through **`mthds_codegen`**, proves the method through **`mthds_validate`**, and reads a pipe's inputs through **`mthds_inputs_template`** on its fallback path — all served by the plugin's `pipelex` MCP server. They are required: never hand-write a generated file, and never derive a signature from the `.mthds` source when the verdict carries it. + +- **If a tool is absent from this session** (the MCP server isn't connected), STOP and tell the user in one line: *"The Pipelex MCP server isn't connected — the plugin manifest spawns the local workshop (`npx -y @pipelex/mcp@latest`), so its absence usually means `node`/`npx` is unavailable or the spawn failed. Check the plugin's MCP connection."* +- **If a call returns `status: "error"` with an error of class `config`**, STOP the same way and surface the error's `hint` verbatim. Two `config` errors deserve a precise reading: a **403** on `mthds_codegen` is a feature gate, not a key problem — its hint says code generation is not enabled for the organization on the hosted API; never answer it with "check your key" — and `kind: "paywall"` is the plan limit, whose hint points at billing. +- The server authenticates with **`PIPELEX_API_KEY`** from its environment — the same variable the plugin's validation hook documents. +- **`mthds_list_methods`** is optional: it resolves a catalog method the user names without its `mt_…` id. When it is absent, integrate by id, address or files; never stop for it. + +## Mode + +Automatic by default: state the target, the destination and the generator in one line before writing anything, decide the routine calls yourself, and pause only for a genuinely ambiguous decision (which app in a monorepo; which of two Python audiences; a `method_id` source). Explicit user signals win — "just do it" is automatic, "walk me through" is interactive, and in interactive mode the dependency additions and the tooling edits are confirmed before they happen. Every MCP call branches on the structured verdict, never on transport. + +## The rules that never bend + +- **The write arm, always.** Every `mthds_codegen` call passes `output_dir`. A refused or failed write is handled as a refusal — never by calling again without `output_dir` and writing the returned bytes yourself. A generated file re-emitted through the conversation is one trailing newline away from a broken stamp, and the whole point of the trust chain is that the tree on disk is byte-identical to what the engine emitted. +- **Generated files are never opened for editing, never formatted, never linted.** Each artifact carries a stamp with its own content hash and the lock hashes every artifact; a reformat, a trimmed newline or a re-serialized lock turns the offline check red. This is why the tooling exclusions are made **before** the tree exists. +- **One directory per method.** After writing, the workshop reports any stamped file the new lock does not list as an orphan and never deletes it; two methods in one directory therefore read as permanently non-current, by design. You never delete an orphan either, and you never offer "clean up the orphans" — the moment two methods share a directory, that advice deletes real files. +- **Never generate from one source and run from another.** Types from a local bundle over a call site that runs by `method_id` is the shape that drifts silently. The sidecar records the selector; the call site uses the same one. +- **A project that owns a codegen harness keeps it.** Never write a second generated layout beside the one the project already has. +- **No `dropWireNulls` / `wireOutput` helper.** The ts-zod emitter projects optional fields as `.nullish()`, so a generated schema parses the runtime's explicit `null`s directly; a null-stripping helper is lossy (it removes legitimate nulls inside opaque fields) and must not be written into a project. + +## Process + +### Step 1: Identify the method and the project + +**The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: + +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. +- **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. + +**The project** is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the working area. A workspace holding several (a monorepo, a full-stack repo) is a question — which app? — never a guess. **No project at all** → this is not an integration yet: offer the `pipelex-scaffold` skill (open `../pipelex-scaffold/SKILL.md`), which creates one and hands it back here. + +Then look for a **codegen harness**: a `codegen` script in `package.json` or a `codegen` Makefile target, a `sources.json` carrying a `derived` map, `docs/codegen.md` or `docs/add-method.md`, a `methods/` directory beside `src/generated/` or `/generated/`. Either of the first two decides; the rest only corroborate. If the project has one, follow [A project that owns a codegen harness](#a-project-that-owns-a-codegen-harness) from here. + +If a `sources.json` with `"generator": "pipelex-integrate"` already names this method, this is [refresh mode](#refresh-mode). + +### Step 2: Prove the method is integrable + +Call **`mthds_validate`** with the selector. Branch: + +- `status: "ok"`, `is_valid: true`, `is_runnable: true`, `pending_signatures: []` → integrable; keep the verdict, step 3 reads from it. +- `is_valid: true` but **not runnable** or `pending_signatures` non-empty → a scaffold with a concept set but no runnable pipes; integrating it produces a call site that cannot succeed. STOP: finish the method with `/pipelex-design` first. Nothing is generated. +- `is_valid: false` → route the `validation_errors[]` to `/pipelex-design` or `/pipelex-edit`; a by-id method's stored content is fixed where it is edited, not here. +- `status: "error"` → class `config` stops per the Requirements; class `input_domain` at `method_ref` / `method_id` is reported in the tool's own words (an unknown or foreign-organization id — the catalog is org-scoped, so another org's method reads exactly like a miss — an unfetchable address, a registry-form ref); class `runtime` → retry once, then report. + +### Step 3: Read the pipe's signature — from the verdict + +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. + +**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. + +### Step 4: Choose the target, the destination and the generator + +State the three in one line before writing. The rule for the target is about **audience**, not language: + +| Project | Target | Emits | +|---|---|---| +| `package.json` with a TypeScript build (a `tsconfig.json`, or a runtime/bundler that strips types) | `ts-zod` | `types.ts` (zod schemas + inferred types, depends only on `zod`) and `binder.ts` (`parse` / `serialize`); keep both | +| `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | +| `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | + +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). + +### Step 5: Make the tooling leave the tree alone — before the tree exists + +Add the generated directory to the formatter's and linter's ignore lists per the language reference (`.prettierignore`, an ESLint flat-config `ignores`, Biome; `[tool.ruff] exclude`, Black, isort), confirm the type checker's include **still covers it** (an exclusion that would drop it is not added), and confirm it is not gitignored. Do this **before** step 6: the first project-wide `format` run after generation would otherwise rewrite the stamps and turn the check red. + +### Step 6: Generate + +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root — do not ride content instead. + +Branch on the structured result: + +- `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. +- `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. +- `status: "error"`, class `input_domain` located at `output_dir`: + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. +- `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. +- Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. +- Success with **`is_current: false`** → a write the check disowns. Report the `drifts[]` verbatim (`path`, `category`, `detail`) and stop; never commit a tree the check rejects. + +### Step 7: Write the sidecar + +Write an **unstamped `sources.json`** beside the lock — the only state this skill keeps. The lock signs the artifacts, not their sources; without the sidecar the next run re-derives everything and a bundle edit is undetectable. Shape: + +```json +{ + "comment": "Written by /pipelex-integrate. `method` and `target` are how this tree was generated — re-run the skill to refresh it. `sources` is the SHA-256 of each local .mthds source, so a bundle edit that was never regenerated is detectable. Not part of the codegen lock; do not hand-edit.", + "generator": "pipelex-integrate", + "method": { "files": ["methods/summarize-pdf/main.mthds"] }, + "target": "ts-zod", + "pipe": { + "pipe_ref": "summarize.summarize_pdf", + "inputs": { "document": "native.Document", "context": "native.Text?" }, + "output": "summarize.DocumentSummary" + }, + "sources": { "methods/summarize-pdf/main.mthds": "" } +} +``` + +`method` is exactly one of `{files}`, `{method_ref}`, `{method_id}`, as passed. Paths are relative to the **project root**, not to the workshop's working directory. `pipe` records what the call site was typed against — `Concept` single, `Concept[]` a list, `Concept?` optional — so refresh mode can tell a signature change from a body change. `sources` is empty for a `method_ref` / `method_id` source. Hashes are over the raw bytes: `shasum -a 256 ` / `sha256sum ` / `hashlib.sha256(path.read_bytes())`. + +### Step 8: Add the dependencies the generated code needs + +With the project's own package manager (read the lockfile): `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen). State what is added; interactive mode confirms first. + +### Step 9: Write the call site + +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. + +| Declared concept | TypeScript parameter | Python parameter | +|---|---|---| +| `native.Text` (or a refinement) | `string` | `str` | +| `native.Number` | `number` | `float` | +| `native.YesNo` | `boolean` | `bool` | +| `native.Date` | `string` (ISO 8601) | `str` (ISO 8601) | +| `native.Image`, `native.Document` | `{ url: string }` — an `http(s)` URL or a `pipelex-storage://` reference | `dict[str, Any]` with a `url` key | +| a structured concept, or a composite native (`Page`, `TextAndImages`, `JSON`) | the generated type from `types.ts` | the generated model from `models.py` | +| `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | +| not required | optional parameter (`?`) | `T \| None = None` | + +**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. + +### Step 10: Wire the offline drift gate + +- **TypeScript**: copy [references/codegen-check.mjs](references/codegen-check.mjs) **verbatim** to `scripts/codegen-check.mjs`, register `"codegen:check": "node scripts/codegen-check.mjs …"` in `package.json`, and **extend the project's existing aggregate gate** — a `check` / `ci` / `validate` / `verify` script, a Makefile `check` target, the lint or test step of an existing workflow — rather than inventing a new one. The script runs `@pipelex/sdk`'s `runCodegenCheck` over each directory, compares the sidecar's source hashes against the committed `.mthds` files, and exits `0` current / `1` drift or stale source / `2` no verdict. A project with no aggregate gate gets the script and one sentence in the report saying where to call it. +- **Python, `python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists (`pipelex codegen check`) needs the `pipelex` runtime a consumer deliberately does not have — do not add `pipelex` as a dependency to get a gate. The sidecar is still written (refresh mode and the editing skills' staleness notice read it), and the report says plainly: the tree is protected by its stamps and lock, but nothing in CI proves it current; refresh with this skill after every bundle edit. +- **Python, `python-structures`**: the project already depends on `pipelex`, so `pipelex codegen check ` (exit `0` / `1` / `2`) is wired into its existing gate. + +### Step 11: Verify + +Run the project's formatter **on the files you wrote only** — never on the generated tree; the step-5 exclusions are what make a later project-wide run safe — then its type checker, then the gate you installed. A failure in your own code is yours to fix before reporting; a failure inside the generated tree is reported, never patched. + +### Step 12: Report + +What was generated and where; the target and why; the call site's signature; what changed in the tooling config; how to refresh (this skill again after a bundle edit); for a `python-pydantic` project, that no offline drift check exists yet and refresh is the guard; for a `method_id` source, that the catalog is unversioned. Then the hand-off: `/pipelex-inputs` prepares inputs and offers a run. + +## Refresh mode + +Entered when the user asks to refresh, regenerate or update the types; when `/pipelex-edit` or `/pipelex-design` hand off after editing a bundle a sidecar names; or when step 1 finds a sidecar for the method. **Re-derive nothing the sidecar already records, regenerate in place, and leave alone everything the regeneration did not invalidate.** + +| Taken from disk | Re-derived | Left alone | +|---|---|---| +| the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | + +One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one that adopted their pattern, regenerates every method in one place, checks them in one place, and keeps its own sidecar. Writing this skill's tree beside that would leave two regeneration paths, two sidecar dialects and files the workshop never emits. So on such a project: + +- **place the method where the project keeps them** — `methods//main.mthds`, or the project's manifest form for a catalog or published method; +- **run the project's generator** — `make add-method METHOD=…` when the project has it and the method is remote, its `codegen` script or Makefile target otherwise; +- **write the call site the way the project's docs and existing methods do** (`docs/codegen.md`, `docs/add-method.md`, the existing actions or CLI commands), not the shape of step 9; +- **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); +- **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. + +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| a required tool is absent | STOP with the one-line MCP-connection message above | +| `status: "error"`, class `config` — including the codegen **403** feature gate | STOP, surface `hint` verbatim; never say "check your key" for a 403 | +| `status: "error"`, class `config`, `kind: "paywall"` | STOP, surface the plan-limit hint | +| `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | +| not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | +| `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | +| `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | +| success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | +| success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | +| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `mthds_list_methods` absent | integrate by id, address or files; never stop for it | + +## Reference + +- [references/typescript.md](references/typescript.md) — detecting a TypeScript project (build, package manager, generated root, Prettier / ESLint / Biome exclusions, `tsconfig` coverage, the aggregate gate, the call-site location), the call-site module and client helper templates, the `codegen:check` wiring, the harness a starter-derived project owns. +- [references/python.md](references/python.md) — the same for Python (import package, `python-pydantic` vs `python-structures`, uv / poetry / pipenv / pip, Ruff / Black / isort exclusions, pyright / mypy coverage, `__init__.py` and package data, the async module plus its sync wrapper, `pipelex codegen check` for the structures audience, the asymmetry sentence for everyone else). +- [references/codegen-check.mjs](references/codegen-check.mjs) — the offline gate copied verbatim into TypeScript projects. +- [MTHDS Language Reference](../shared/mthds-reference.md) — for reading a bundle's `main_pipe` and `output` declarations on the fallback path. diff --git a/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs new file mode 100644 index 0000000..ade01da --- /dev/null +++ b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs @@ -0,0 +1,166 @@ +#!/usr/bin/env node +// codegen-check.mjs — the offline drift gate for Pipelex-generated trees. +// +// Copied verbatim into a project by /pipelex-integrate. Run it from the project root +// with one generated directory per argument: +// +// node scripts/codegen-check.mjs src/generated/summarize-pdf src/generated/extract-entities +// +// For each directory it (1) runs @pipelex/sdk's runCodegenCheck over the stamped files +// against codegen.lock — pure hashing, no engine, no network, no API key — and (2) compares +// the SHA-256 recorded for each .mthds source in sources.json against the file on disk, so +// a bundle edited without a regeneration is caught as `stale-source`. +// +// Exit codes: 0 current · 1 drift or stale source · 2 no verdict (no lock, an unreadable +// file, a symlink in the tree). Precedence across directories: 2 > 1 > 0. +// +// It imports only Node builtins and @pipelex/sdk, and writes through process.stdout / +// process.stderr so a no-console lint rule stays quiet. When @pipelex/sdk ships this check +// as a command, replace this file with that one line. + +import { createHash } from "node:crypto"; +import { lstat, readdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import process from "node:process"; +import { CodegenLockError, isStampableArtifactPath, runCodegenCheck } from "@pipelex/sdk"; + +const EXIT_CURRENT = 0; +const EXIT_DRIFT = 1; +const EXIT_NO_VERDICT = 2; + +const LOCK_FILENAME = "codegen.lock"; +const SIDECAR_FILENAME = "sources.json"; +const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); + +const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); + +const out = (line) => process.stdout.write(`${line}\n`); +const err = (line) => process.stderr.write(`${line}\n`); + +/** Every regular file under `root`, as sorted forward-slash paths relative to it. Refuses symlinks. */ +async function walk(root, relative = "") { + const absolute = relative ? path.join(root, relative) : root; + const entries = await readdir(absolute, { withFileTypes: true }); + const paths = []; + for (const entry of entries) { + const rel = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isSymbolicLink()) { + throw new Error(`refusing to read through the symlink ${rel}`); + } + if (entry.isDirectory()) { + if (PRUNED_DIRECTORIES.has(entry.name)) continue; + paths.push(...(await walk(root, rel))); + } else if (entry.isFile()) { + paths.push(rel); + } + } + return paths.sort(); +} + +async function readStrict(filePath) { + return strictUtf8.decode(await readFile(filePath)); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** The lock check: { code, lines } — never throws. */ +async function checkTree(dir) { + let lockContent; + try { + lockContent = await readStrict(path.join(dir, LOCK_FILENAME)); + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.code === "ENOENT" ? "not found" : error.message}`] }; + } + + let files; + try { + const rootStat = await lstat(dir); + if (rootStat.isSymbolicLink()) throw new Error("the generated directory itself is a symlink"); + const stampable = (await walk(dir)).filter((rel) => isStampableArtifactPath(rel)); + files = []; + for (const rel of stampable) { + files.push({ path: rel, content: await readStrict(path.join(dir, rel)) }); + } + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${error.message}`] }; + } + + try { + const report = await runCodegenCheck({ lockContent, files }); + if (report.isCurrent) { + return { + code: EXIT_CURRENT, + lines: [` ${files.length} artifact(s) current (crate ${report.crateFingerprint.slice(0, 12)}, engine ${report.engineVersion})`], + }; + } + return { code: EXIT_DRIFT, lines: report.drifts.map((drift) => ` ${drift.category}: ${drift.path} — ${drift.detail}`) }; + } catch (error) { + if (error instanceof CodegenLockError) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.message}`] }; + } + throw error; + } +} + +/** The sidecar check against the .mthds sources, relative to the project root: { code, lines }. */ +async function checkSources(dir) { + let sidecar; + try { + sidecar = JSON.parse(await readStrict(path.join(dir, SIDECAR_FILENAME))); + } catch (error) { + if (error.code === "ENOENT") { + return { code: EXIT_CURRENT, lines: [` no ${SIDECAR_FILENAME} — source staleness not checked`] }; + } + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; + } + + const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + const lines = []; + for (const [source, recorded] of Object.entries(sources).sort()) { + let onDisk; + try { + onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); + } catch (error) { + lines.push(` stale-source: ${source} — recorded as a source but ${error.code === "ENOENT" ? "no longer on disk" : `unreadable (${error.message})`}`); + continue; + } + if (onDisk !== recorded) { + lines.push(` stale-source: ${source} — edited since the types were generated`); + } + } + return { code: lines.length ? EXIT_DRIFT : EXIT_CURRENT, lines }; +} + +function worse(a, b) { + // Precedence: no verdict > drift > current. + if (a === EXIT_NO_VERDICT || b === EXIT_NO_VERDICT) return EXIT_NO_VERDICT; + if (a === EXIT_DRIFT || b === EXIT_DRIFT) return EXIT_DRIFT; + return EXIT_CURRENT; +} + +async function main(argv) { + const dirs = argv.slice(2); + if (dirs.length === 0) { + err("usage: node scripts/codegen-check.mjs [ ...]"); + return EXIT_NO_VERDICT; + } + + let exitCode = EXIT_CURRENT; + for (const dir of dirs) { + out(`${dir}`); + const tree = await checkTree(dir); + const sources = tree.code === EXIT_NO_VERDICT ? { code: EXIT_CURRENT, lines: [] } : await checkSources(dir); + const code = worse(tree.code, sources.code); + const write = code === EXIT_CURRENT ? out : err; + for (const line of [...tree.lines, ...sources.lines]) write(line); + if (code === EXIT_DRIFT) err(" Run /pipelex-integrate to refresh the generated types."); + exitCode = worse(exitCode, code); + } + + out(`\ncodegen-check: ${exitCode === EXIT_CURRENT ? "current" : exitCode === EXIT_DRIFT ? "drift" : "no verdict"}`); + return exitCode; +} + +process.exit(await main(process.argv)); diff --git a/pipelex-codex/skills/pipelex-integrate/references/python.md b/pipelex-codex/skills/pipelex-integrate/references/python.md new file mode 100644 index 0000000..cc64358 --- /dev/null +++ b/pipelex-codex/skills/pipelex-integrate/references/python.md @@ -0,0 +1,102 @@ +# Integrating into a Python project + +Companion to `/pipelex-integrate` for a project that has a `pyproject.toml` (or `setup.py` / `requirements.txt`). The SDK facts were checked against `pipelex-sdk` as of 2026-09 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Audience → target** | `pipelex` **not** among the dependencies (`[project].dependencies`, `requirements*.txt`) → `python-pydantic`, no question: `python-structures` imports the runtime and would not even load; `pipelex` present → `python-structures` if `@pipe_func` or `from pipelex.core.stuffs.structured_content import StructuredContent` appears in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | +| **Import package** | the package `[project].name` names (dashes → underscores), or the setuptools `packages` list, or the top-level directory holding `__init__.py` (`src//` in a src layout) | ask | +| **Package manager** | the lockfile: `uv.lock` → uv (`uv add`), `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none → `pip install` into the active environment, stated | +| **Generated root** | an existing directory already holding generated code → beside it; else `/generated/` | ask | +| **Formatter and linter** | `[tool.ruff]` → add the generated directory to `exclude` (or `extend-exclude`); `[tool.black]` → `extend-exclude`; `[tool.isort]` → `skip` / `extend_skip_glob` | a tool this table does not name: read its config, add the equivalent, say so | +| **Type checker still covers the tree** | `[tool.pyright]` `include` / `exclude`; `[tool.mypy]` `packages` / `files` / `exclude` | an exclusion that would drop it is **not** added; generated code stays type-checked | +| **Packaged for distribution** | a `[build-system]` table and an import package | the bundle goes **inside** the package (`/methods//main.mthds`) and `*.mthds` plus `codegen.lock` are registered as package data, as the Python starter does — a wheel that ships the call site must ship the bundle it loads at call time; an unpackaged app keeps `methods//` at the project root | +| **Aggregate gate** | a Makefile `check` target; `.github/workflows/*.yml`; `.pre-commit-config.yaml`; a `nox`/`tox` session | none: nothing to wire for `python-pydantic` anyway (below) | +| **Call-site location** | the project's existing service / client layer (`services/`, `clients/`, `api/`) → beside it | `/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | reported and un-ignored on confirmation | +| **Owns a codegen harness** | a `codegen` Makefile target; `docs/codegen.md`; `/generated/` beside `/methods/` | the harness section below | + +Why the Ruff exclusion is not optional: `ruff format` rewrites the generated bytes, breaks every stamp, and makes the drift check report the tree as hand-edited. The exclusion goes in before generation, so the first project-wide format after it is already safe. + +## The generated tree + +`mthds_codegen` with `target: "python-pydantic"` and `output_dir: "/generated/"` writes: + +- `models.py` — stamped; `from __future__ import annotations`, `from pydantic import BaseModel, Field`; one plain `BaseModel` subclass per concept, natives included, with `Field(..., description=…)`; no Pipelex import. Non-required fields are `T | None = None`. +- `codegen.lock` — TOML with the crate fingerprint, the engine version and one `[[artifacts]]` entry per stamped file. + +With `target: "python-structures"` the file is `structures.py` — `StructuredContent` subclasses from `pipelex.core.stuffs.structured_content`, natives not re-emitted — for a Pipelex host whose `@pipe_func` functions return them. + +**The skill creates the `__init__.py` files** the tree needs to be importable: `/generated/__init__.py` (once per project, a one-line docstring) and `/generated//__init__.py` (empty). Codegen never emits them and they are never artifacts: they carry no stamp, so the writer does not touch them and the check does not count them as orphans. When the project is packaged, list the generated subpackages where the project lists its packages and add `codegen.lock` as package data, as the Python starter's `pyproject.toml` does. Beside the lock the skill writes `sources.json`, unstamped. + +## The call site + +One module per method. `summarize_pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle inside the package: + +```python +# /pipelex/summarize_pdf.py +"""Run the summarize_pdf method through the hosted Pipelex API. + +`document` is an http(s) URL or a pipelex-storage:// reference, as ``{"url": ...}``. For a +local file or bytes, call ``client.prepare_inputs(files=[...], inputs=...)`` first: it uploads +and rewrites the value. It treats any string it does not recognise as data:, http(s):// or +pipelex-storage:// as a LOCAL FILE PATH it reads and uploads, so gate schemes before handing +values from an untrusted caller to it. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +from pipelex_sdk.client import PipelexAPIClient + +from .generated.summarize_pdf.models import DocumentSummary + +PIPE_CODE = "summarize_pdf" +BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" + + +async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + inputs: dict[str, Any] = {"document": document} + if context is not None: + inputs["context"] = context + async with PipelexAPIClient() as client: + results = await client.start_and_wait( + pipe_code=PIPE_CODE, + mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + inputs=inputs, + ) + return DocumentSummary.model_validate(results.main_stuff) + + +def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + """The blocking wrapper for a synchronous caller.""" + return asyncio.run(summarize_pdf(document=document, context=context)) +``` + +Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: + +- **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. +- **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. + +Facts the module leans on: `PipelexAPIClient()` reads `PIPELEX_API_KEY` and `PIPELEX_BASE_URL` (default `https://api.pipelex.com`) itself; it is async-only and used as an async context manager; there is **no barrel** in `pipelex_sdk` by design, so every import is a full module path (`pipelex_sdk.client`, `pipelex_sdk.runs`, `pipelex_sdk.errors`); `start_and_wait(...)` takes the durable path on the hosted API (default 2 s poll, 20 min budget, `wait_options=WaitForResultOptions(...)` to change them) and falls back to blocking on a bare runner, returning `RunResults` whose `main_stuff` is always present for a completed run. Let the typed errors from `pipelex_sdk.errors` propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own handling only where it already wraps its other clients. + +Parameters are keyword-only, typed from the signature: `str` for Text and Date (ISO 8601), `float` for Number, `bool` for YesNo, `dict[str, Any]` with a `url` key for Image and Document, the generated model for a structured concept or a composite native, `list[T]` for a list, `T | None = None` for a non-required input. Names are the pipe's input names as declared. A project that already constructs a `PipelexAPIClient` somewhere gets that construction reused instead of a fresh `async with` per call; an async-native project (FastAPI, an existing async codebase) gets the async function alone, without the `_sync` wrapper. + +## The drift gate — an honest asymmetry + +- **`python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists — `pipelex codegen check ` — lives in the `pipelex` runtime, which a hosted-API consumer deliberately does not install. Do not add `pipelex` as a dependency to get a gate; that reverses the decision the target expresses. Write the sidecar anyway (refresh mode and `/pipelex-edit`'s staleness notice read it) and put this sentence in the report: *the generated tree is protected by its stamps and lock, but nothing in CI proves it current; run `/pipelex-integrate` again after every bundle edit.* When the Python SDK gains the check, this section becomes one line. +- **`python-structures`**: the project already depends on `pipelex`, so wire `pipelex codegen check /generated/` into its existing gate (a Makefile `check` target, a workflow step). Exit codes: `0` current, `1` drift, `2` no lock or an unreadable lock. Offline: no engine boot, no network, no key. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-python` has `make codegen` and `make codegen-check`, both shelling out to a `pipelex` CLI the starter deliberately does not depend on (`PIPELEX=` in its Makefile), methods under `/methods//`, trees under `/generated//` with hand-committed `__init__.py` files, and typed CLI commands under its mode sub-packages. On such a project: + +- place the bundle under `/methods//main.mthds` and add the two `codegen` / `codegen-check` lines the Makefile pattern uses for its other methods; +- run `make codegen` when a `pipelex` CLI is reachable; when it is not, call `mthds_codegen` with `output_dir` set to `/generated/` — the harness's own layout, byte-identical (same engine, same stamps, same lock, no sidecar in this starter) — and say `make codegen` is the refresh once `PIPELEX=` points at an install; +- write the CLI command the way `docs/cli-architecture.md` and the existing commands do, importing the generated model from `.generated..models`; +- verify with `make agent-check` and `make agent-test`; no `sources.json` of this skill's shape and no second generated directory. diff --git a/pipelex-codex/skills/pipelex-integrate/references/typescript.md b/pipelex-codex/skills/pipelex-integrate/references/typescript.md new file mode 100644 index 0000000..bce5bfa --- /dev/null +++ b/pipelex-codex/skills/pipelex-integrate/references/typescript.md @@ -0,0 +1,118 @@ +# Integrating into a TypeScript project + +Companion to `/pipelex-integrate` for a project that has a `package.json`. Everything here follows the shape the Pipelex JS starter converged on; the SDK facts were checked against `@pipelex/sdk` 0.17 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **TypeScript build** | a `tsconfig.json`; a `typescript` dev dependency; a bundler or runtime that strips types (Next.js, Vite, tsx, Bun, Deno) | a plain JavaScript project is asked — `types.ts` needs a TypeScript build; the alternative is `python-pydantic`'s sibling in this language, which does not exist yet, so the honest answer is "add TypeScript or skip codegen" | +| **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock` / `bun.lockb` → bun | none → npm, stated | +| **Generated root** | an existing directory already holding generated code (`generated/`, `gen/`, `__generated__/`) → beside it; else `src/generated/` when `src/` exists; else `generated/` | ask | +| **Formatter** | `.prettierrc*` or a `prettier` dev dependency → add `src/generated/` to `.prettierignore` (create the file if absent); `biome.json*` → the files-ignore key its version uses (`files.ignore` before Biome 2, `files.includes` with a `!` negation from Biome 2 on) | a formatter this table does not name: read its config, add the equivalent, say so | +| **Linter** | `eslint.config.*` (flat config) → add `"src/generated/**"` to `globalIgnores([...])` or an `{ ignores: [...] }` entry; legacy `.eslintrc*` → `.eslintignore`; Biome as above | same | +| **Type checker still covers the tree** | `tsconfig.json` `include` / `exclude` — the generated directory must stay inside `include` and outside `exclude` | an exclusion that would drop it is **not** added; the report says the typecheck is the check that covers generated code | +| **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | +| **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | + +Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. + +## The generated tree + +`mthds_codegen` with `target: "ts-zod"` and `output_dir: "src/generated/"` writes: + +- `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. +- `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. +- `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. + +Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. + +## The call site + +One module per method. `summarize-pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle: + +```ts +// src/pipelex/summarizePdf.ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import type { RunResults } from "@pipelex/sdk"; +import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; +import type { DocumentSummary } from "../generated/summarize-pdf/types"; +import { getPipelexClient } from "./client"; + +const PIPE_CODE = "summarize_pdf"; +const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); + +export interface SummarizePdfInputs { + /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, + * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * it uploads and rewrites the value. Note: prepareInputs treats any string it does not + * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and + * uploads, so a public endpoint must gate schemes before handing values to it. */ + document: { url: string }; + context?: string; +} + +export async function summarizePdf(inputs: SummarizePdfInputs): Promise { + const bundle = await readFile(BUNDLE_PATH, "utf8"); + const results: RunResults = await getPipelexClient().startAndWaitForResult({ + pipe_code: PIPE_CODE, + mthds_contents: [bundle], + inputs, + }); + return parseDocumentSummary(results.main_stuff); +} +``` + +Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: + +- **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. +- **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. + +The shared client helper, created once per project and reused by every method (if the project already constructs a `PipelexApiClient` somewhere, import that instead): + +```ts +// src/pipelex/client.ts +import { PipelexApiClient } from "@pipelex/sdk"; + +let client: PipelexApiClient | undefined; + +/** Reads PIPELEX_API_KEY and PIPELEX_BASE_URL (default https://api.pipelex.com) from the environment. */ +export function getPipelexClient(): PipelexApiClient { + client ??= new PipelexApiClient(); + return client; +} +``` + +`startAndWaitForResult(options, pollOptions?)` takes the durable path (`start` + poll, default 2 s interval, 20 min budget) on the hosted API and falls back to a blocking `execute` on a bare runner; it returns `RunResults` whose `main_stuff` is always present for a completed run. Let the SDK's typed errors propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own error handling only where it already wraps its other clients. A server-side framework seam (a Next.js Server Action, an Express handler) is the caller's; this module is framework-free. + +Parameter types from the signature: `string` for Text and Date (ISO 8601), `number` for Number, `boolean` for YesNo, `{ url: string }` for Image and Document, the generated type for a structured concept or a composite native, `T[]` for a list, `?` for a non-required input. Key names are the pipe's input names as declared, snake_case included. + +## The offline gate + +Copy `references/codegen-check.mjs` verbatim to `scripts/codegen-check.mjs`. It imports only Node builtins and `@pipelex/sdk`, runs under plain `node` whatever the project's TypeScript build, and prints through `process.stdout` / `process.stderr` so a `no-console` rule does not fire. Register it and extend the existing gate: + +```json +{ + "scripts": { + "codegen:check": "node scripts/codegen-check.mjs src/generated/summarize-pdf", + "check": "npm run lint && npm run typecheck && npm run codegen:check" + } +} +``` + +Add every method's directory to the `codegen:check` line as it is integrated. Exit codes: `0` current, `1` drift or stale source, `2` no verdict (no lock, an unreadable file, a symlink in the tree). It runs from the project root because `sources.json` records source paths relative to it. When `@pipelex/sdk` ships this as a command of its own, the script is replaced by that one line. + +## The Node-only boundary + +`readFile`, `node:path` and `process.cwd()` in the call site are server-side facts. In a framework with a client/server split (Next.js, Remix, SvelteKit), the module belongs on the server side — a Server Action, a route handler, a loader — and the JS starter marks such modules with `import "server-only"`. Never import it from a component that renders in the browser: the API key would leave the server. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js`, or one that copied its codegen kit, has: `npm run codegen` (regenerates every `methods/*` tree through the hosted `/v1/codegen`, writes `contracts.ts` and `sources.json` with a `derived` map), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (semantic, keyed), and `make add-method METHOD=` (manifest, tree, action trio, narrower, form, tab — one shot, never overwrites). On such a project: + +- a local bundle goes under `methods//main.mthds`, then `npm run codegen`; the fan-out follows `docs/codegen.md` and the existing actions under `src/actions/` and narrowers under `src/types/`; +- a catalog or published method goes through `make add-method`; +- the verification is `make check`; the refresh is `npm run codegen`; no `sources.json` of this skill's shape, no `scripts/codegen-check.mjs`, no second generated directory. diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md new file mode 100644 index 0000000..6f50fbe --- /dev/null +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -0,0 +1,158 @@ +--- +name: pipelex-scaffold +description: Start a new project that will call MTHDS methods through Pipelex, in TypeScript or Python — from one of the Pipelex starter templates or from the ecosystem's own initializer — and hand it to /pipelex-integrate. Use when the user says "start a new project with Pipelex", "I have a method and need an app around it", "create a Next.js app that runs my method", "set up a Pipelex project from scratch", "new Python CLI for this method", "which starter should I use", "bootstrap a Pipelex project", or wants a codebase where none exists yet. Also use when the user is standing in a freshly cloned pipelex-starter-js or pipelex-starter-python that has not been renamed yet — this skill runs the template's own bootstrap for them. Not for adding Pipelex to code that already exists: that is /pipelex-integrate. + +--- + +# Scaffold a project for Pipelex methods + +Give a user who has no project yet a project that is ready for `/pipelex-integrate`. This skill has exactly two branches and carries no templates of its own: + +- **One of the Pipelex starters** when the user wants the opinionated shape: `pipelex-starter-js` for a web app whose forms are rendered from the methods' own contracts, `pipelex-starter-python` for a CLI or service that runs methods in the three execution modes. You acquire the template, commit it once as it came, then run the clone's **own** `bootstrap` skill — the rename logic lives in the starters and is never reimplemented here. +- **The ecosystem's own initializer** when the user wants their framework or a minimal project: `uv init --package`, `npm create next-app@latest`, `django-admin startproject`, whatever the framework documents. You run it; you never assemble a project by hand. + +Both branches end the same way: an env file that follows the starters' convention, one pristine commit that makes everything after it reviewable, and the hand-off — to `/pipelex-integrate` when a method exists, to `/pipelex-design` first when none does. + +**What this skill is not.** Not a template engine (no cookiecutter, no copier, no framework matrix of its own), not a bootstrap (the starters own theirs), not a runner or a dev-server launcher, not a deployer. It needs no MCP tool and no API key: git, the starters' scripts and the ecosystem's initializers are all it uses. + +## Choosing the branch + +A cheap, reliable signal decides; an inconclusive one asks one question; nothing is guessed twice. + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | +| **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | + +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. + +[references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. + +## Mode + +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. + +## Branch A — one of the starters + +### Step 1: Prerequisites + +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. + +- **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. +- **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). +- **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. + +### Step 2: Acquire the template + +**Local, the default.** Clone shallow, read the template's identity, then detach from it: + +```bash +git clone --depth 1 https://github.com/Pipelex/.git +git -C rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf /.git && git -C init -b main +``` + +The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. + +**GitHub, on request.** When the user asked for a repository on GitHub: + +```bash +gh repo create / --template Pipelex/ --private --clone +``` + +Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. + +Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. + +### Step 3: Commit the pristine template — exactly once + +```bash +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. + +### Step 4: Run the clone's own bootstrap + +Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written. The project's skills are not loaded in this session — it began elsewhere — so read the file; do not look for a `/bootstrap` command. Run every command it gives from inside the project directory (`cd && …`, or `-C `), because that skill assumes it is standing in the repo root. + +Feed it what the conversation already holds — the project name, title, description, author, repository URL, license — so that it asks once, consolidated, for whatever is left, exactly as its own Step 2 says. It dry-runs, previews, runs, re-syncs the lock file, runs the project's own checks (`make all` on JS; `make agent-check` and `make agent-test` on Python), and removes itself. Its rules stand unchanged: it never commits, its edits stay uncommitted for the user's review (the Python renames are staged by `git mv`, which its skill explains), and a red check is fixed, never skipped. **Add nothing to that procedure and reimplement none of it.** If the clone carries no bootstrap skill — a future template dropped it — follow the README's "manual equivalent" list and say that the template changed. + +### Step 5: The env file + +```bash +cp /.env.example /.env.local # JS: Next.js reads .env.local +cp /.env.example /.env # Python: python-dotenv reads .env +``` + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. + +### Step 6: Verify and hand off + +The bootstrap's own checks are the verification; do not start `make dev`. Write the report (below), then hand the user's method to `/pipelex-integrate`, which recognizes the starter's codegen harness (`npm run codegen`, `make codegen`, `make add-method`) and defers to it rather than writing a second one. + +## Branch B — the ecosystem's initializer + +### Step 1: Prerequisites + +As in branch A, for the language chosen. + +### Step 2: Run the initializer — never assemble by hand + +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. + +Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. + +### Step 3: Version control and the pristine commit + +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: + +```bash +git -C add -A && git -C commit -m "Scaffold project" +``` + +### Step 4: The env file + +Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: + +``` +PIPELEX_BASE_URL=https://api.pipelex.com +PIPELEX_API_KEY= +``` + +### Step 5: Hand off + +Add **no** SDK dependency and create **no** empty `methods/` directory: `/pipelex-integrate` adds `@pipelex/sdk` or `pipelex-sdk` when it writes the first call site, and creates `methods//` when it places the first bundle. A project with nothing to integrate yet has nothing Pipelex-shaped in it beyond the env convention, and that is correct. + +## The report + +Say, in this order: what was created and where; which template or initializer it came from, at which version and SHA; that this skill made exactly one commit and what it holds; what the bootstrap changed and that those changes are uncommitted for review, in the bootstrap's own words (branch A); which env file was written and whether the key was filled from the environment or left for the user; the demos the starter still carries and where the README's removal checklist is (branch A); and the hand-off. + +Two lines are easy to forget and matter: + +- **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd `, then starting Codex there, is how they arrive. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | +| The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | +| The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | +| An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | +| `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | + +## Reference + +- [references/starters.md](references/starters.md) — the two starters side by side: what each brings, its prerequisite floors, the acquisition commands, its env file, its bootstrap, its demos and their removal checklist, and the codegen harness `/pipelex-integrate` will find. +- [references/initializers.md](references/initializers.md) — per language, the minimal default and the common frameworks' non-interactive initializers, whether each runs `git init`, and where the import package or `src/` root lands. +- `/pipelex-integrate` — the skill this one hands every project to. diff --git a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md new file mode 100644 index 0000000..98b683f --- /dev/null +++ b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md @@ -0,0 +1,38 @@ +# Ecosystem initializers + +Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. + +After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +## Python + +| Want | Command | `git init`? | Where the import package lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | +| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | + +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. + +## TypeScript / JavaScript + +| Want | Command | `git init`? | Where `src/` lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | +| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Node library | the minimal recipe above | no | `src/` | +| pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | + +`npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. + +## What every branch-B project shares afterwards + +- One commit, the pristine scaffold, so the user's first real change is a clean diff. +- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/pipelex-codex/skills/pipelex-scaffold/references/starters.md b/pipelex-codex/skills/pipelex-scaffold/references/starters.md new file mode 100644 index 0000000..52ccdc3 --- /dev/null +++ b/pipelex-codex/skills/pipelex-scaffold/references/starters.md @@ -0,0 +1,59 @@ +# The two Pipelex starters + +Both are GitHub **template repositories** under the `Pipelex` organization. Each is a real, CI-tested application against the hosted Pipelex API, not a parameterized template: the identity you see in a fresh clone (`pipelex-starter-js` / `Pipelex Starter`, or `piper` / `Piper`) is a placeholder that the starter's own `bootstrap` skill rewrites. Read the clone's `README.md` after acquiring it — the sections named below are where the details live, and they move as the starters evolve. + +## Side by side + +| | `pipelex-starter-js` | `pipelex-starter-python` | +|---|---|---| +| **Shape** | Next.js (App Router), React, TypeScript strict, Tailwind; a web app with one tab per method whose input form is rendered from the method's own contract by `@pipelex/mthds-form` | A Typer CLI with one command per method, printing JSON on stdout and a cost report on stderr; three execution modes (`blocking`, `attended`, `detached`) as separate sub-packages | +| **Pick it when** | people will use the methods in a browser: forms, uploads, live run status | the methods run from a terminal, a script, a batch job or a service, and the user wants Python | +| **SDK** | `@pipelex/sdk` | `pipelex-sdk` (import package `pipelex_sdk`) — the `pipelex` runtime is **not** a dependency | +| **Methods live in** | `methods//main.mthds`, or `methods//method.json` for a method that lives elsewhere (a catalog id or a published address) | `/methods//main.mthds` | +| **Generated types** | `src/generated//` — `types.ts`, `binder.ts`, `contracts.ts`, `codegen.lock`, `sources.json` | `/generated//` — `models.py`, `codegen.lock` | +| **Codegen harness** | `npm run codegen` (keyed, dev), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (keyed, semantic); `make add-method METHOD=` scaffolds a remote method end to end | `make codegen` / `make codegen-check` — **both shell out to a `pipelex` CLI the starter does not depend on** (`PIPELEX=` in the Makefile); `/pipelex-integrate` knows this and writes into the same layout when that CLI is absent | +| **Toolchain floor** | Node ≥ the `engines.node` field of `package.json` (22.12 at writing); `npm` | `uv`; a Python inside `requires-python` of `pyproject.toml` (3.11–3.14 at writing) | +| **Env file** | `.env.local`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY`, `NEXT_PUBLIC_EXECUTION_MODE` | `.env`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY` | +| **Checks the bootstrap runs** | `npm install --package-lock-only`, then `make all` (lint, format check, typecheck, unit tests, build) | `make li` (lock + sync), then `make agent-check` and `make agent-test` | +| **Agent-facing files** | `CLAUDE.md`, `AGENTS.md`; skills `bootstrap`, `release`, `bump-sdk`, `bump-mthds-form` | `CLAUDE.md`; skills `bootstrap`, `release` | +| **Docs worth reading after bootstrap** | `docs/codegen.md`, `docs/add-method.md`, `docs/input-form.md`, `docs/adopt-in-an-existing-project.md`; README → "Swap in your own pipeline" and "Remove an example" | `docs/codegen.md`, `docs/cli-architecture.md`; README → the per-command sections | +| **Demos it carries** | several demo methods, one tab each; keep them as references or strip them with the README's "Remove an example" checklist | several demo methods, one CLI command each; keep them as references or remove the command and its method directory together | + +## Acquisition + +Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): + +```bash +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git -C rev-parse HEAD +rm -rf /.git && git -C init -b main +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. + +GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): + +```bash +gh auth status +gh repo create / --template Pipelex/pipelex-starter-js --private --clone +gh repo create / --template Pipelex/pipelex-starter-python --private --clone +``` + +## The bootstrap you delegate to + +Both starters carry `.claude/skills/bootstrap/SKILL.md` with a bundled script (`scripts/bootstrap.mjs` / `scripts/bootstrap.py`). Read the file in the clone and follow it; the shape is the same on both: + +1. **Preflight** — confirms the identity is still the template's (`package.json` name `pipelex-starter-js`; `pyproject.toml` `name = "piper"`), notes a dirty tree, and on JS makes sure `node_modules/` exists (`make install`). +2. **Collect** — the package name (kebab on JS, underscores on Python, everything else derives from it), a display title, a one-line description; optionally author name **and** email (never one without the other), the repository URL, and the license (MIT kept, proprietary, or another SPDX id; the copyright holder and year). Pass what the conversation already holds so it asks once for the rest. +3. **Dry run** — the script with `--dry-run` prints the plan; the user confirms. +4. **Run** — the same command without `--dry-run`; on Python the package directory is renamed with `git mv`, which is why the pristine commit must exist first. `--clean` strips the template-only prose; keep it unless the user wants the template charter kept. +5. **Verify** — the lock file is re-synced and the project's own checks run; red is fixed, not skipped. +6. **Self-removal** — `rm -rf .claude/skills/bootstrap`, unstaged like everything else; the user reviews with `git status` and `git diff` and commits when ready. + +The starter's rules are yours while you run it: never commit on the user's behalf, always dry-run first, never touch `.github/` or the `release` skill's logic. + +## What `/pipelex-integrate` finds afterwards + +A bootstrapped starter is a project that **owns a codegen harness**, and `/pipelex-integrate` defers to it: on JS it drops a bundle under `methods//` and runs `npm run codegen`, or runs `make add-method METHOD=…` for a catalog or published method, then follows `docs/codegen.md` and the existing actions for the fan-out; on Python it places the bundle under `/methods//` and runs `make codegen` when a `pipelex` CLI is available, writing into `/generated//` through the Pipelex workshop when it is not. It never writes a second generated layout beside the starter's own. diff --git a/pipelex-vibe/skills/pipelex-design/SKILL.md b/pipelex-vibe/skills/pipelex-design/SKILL.md index 68b2a4e..903fbba 100644 --- a/pipelex-vibe/skills/pipelex-design/SKILL.md +++ b/pipelex-vibe/skills/pipelex-design/SKILL.md @@ -182,7 +182,7 @@ After the gate: 1. **Organize only when the layout needs it.** A direct result that is already coherent skips `/pipelex-organize`. A converged stepwise result normally invokes it automatically because one-definition-per-file construction history and satisfied headers need regrouping. A naturally coherent result in either mode does not take an organization round trip solely for process compliance. 2. **Project the input schema.** Call `mthds_inputs_template` with the final whole-bundle `files` submission plus `explicit: false`. Show the returned compact template, but **do not save it as `inputs.json`** — input preparation belongs exclusively to `/pipelex-inputs`. 3. **Present the flow.** Point to the interactive method graph where the host rendered the valid verdict's view; in terminal hosts, present a concise text flow of the final structure. -4. **Hand off inputs.** Suggest preparing real inputs with `/pipelex-inputs`. +4. **Hand off inputs — and the code.** Suggest preparing real inputs with `/pipelex-inputs`. Then, when the workspace holds a codebase (a `package.json` or a `pyproject.toml`), say that `/pipelex-integrate` wires the method into it with generated types and a typed call site; when it holds none and the user wants an application around the method, `/pipelex-scaffold` creates one and hands it to `/pipelex-integrate`. > **NEVER write `inputs.json` manually.** If the user provides files, paths, or wants to run with real data, invoke `/pipelex-inputs` — it handles the template, path resolution, placeholder formatting, and file copying. @@ -207,7 +207,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex-vibe/skills/pipelex-edit/SKILL.md b/pipelex-vibe/skills/pipelex-edit/SKILL.md index e3454b2..42d6f50 100644 --- a/pipelex-vibe/skills/pipelex-edit/SKILL.md +++ b/pipelex-vibe/skills/pipelex-edit/SKILL.md @@ -76,6 +76,8 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. +**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. + ## Reference - [MTHDS Language Reference](../shared/mthds-reference.md) — read for concept definitions and syntax before editing constructs you haven't touched recently diff --git a/pipelex-vibe/skills/pipelex-inputs/SKILL.md b/pipelex-vibe/skills/pipelex-inputs/SKILL.md index 12d0b7a..a32c91d 100644 --- a/pipelex-vibe/skills/pipelex-inputs/SKILL.md +++ b/pipelex-vibe/skills/pipelex-inputs/SKILL.md @@ -353,6 +353,8 @@ After assembling the inputs, confirm readiness: (Or, for the Template strategy: point out which placeholders the user still needs to fill.) +When the workspace holds a codebase (a `package.json` or a `pyproject.toml`) and the method is not yet wired into it, add one line: `/pipelex-integrate` generates the method's types into the project and writes a typed call site that runs it. + ### Offer to run When the inputs are complete, close by offering to run the method. Offer — never start unprompted: a run executes on the hosted Pipelex API and **spends inference credit**. diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md new file mode 100644 index 0000000..c3d2cd3 --- /dev/null +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -0,0 +1,209 @@ +--- +name: pipelex-integrate +description: Wire an MTHDS method into a Python or TypeScript codebase with generated, drift-proof types and one typed call site that runs it through @pipelex/sdk or pipelex-sdk. Use when the user says "use this method in my app", "call this from my code", "generate types for this method", "wire the method into my project", "add this pipeline to my service", "typed client for this method", "integrate the method", "refresh the generated types", "regenerate the types", "the types are stale", or wants application code that runs a .mthds method — from a local bundle, a catalog id (mt_…) or a published method_ref address. Also the refresh path after a bundle edit. Not for authoring or editing the method itself (/pipelex-design, /pipelex-edit), and not for a project that does not exist yet (/pipelex-scaffold). + +--- + +# Integrate an MTHDS method into a codebase + +Take a method — a local `.mthds` bundle, a published address (`method_ref`), or a catalog id (`method_id`) — and a Python or TypeScript project, and leave the project able to call the method with types that cannot silently drift from it. Concretely: + +1. pick the codegen target that matches the project's language **and audience**; +2. have the Pipelex workshop write the generated tree into a dedicated directory per method, through `mthds_codegen`'s write arm, so no generated byte ever passes through you; +3. make the project's formatters and linters leave that tree alone while its type checker keeps covering it; +4. record how the tree was generated in a small sidecar beside the lock, so the next run knows what to refresh and a bundle edit is detectable; +5. wire the offline drift check into the gate the project already runs, where one exists for the language; +6. write one typed call-site module per method, running it through `@pipelex/sdk` or `pipelex-sdk` and narrowing its output with the generated binder or model; +7. verify with the project's own type checker and the gate you just installed. + +Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. + +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. + +## Requirements — the Pipelex MCP tools + +This skill generates through **`mthds_codegen`**, proves the method through **`mthds_validate`**, and reads a pipe's inputs through **`mthds_inputs_template`** on its fallback path — all served by the plugin's `pipelex` MCP server. They are required: never hand-write a generated file, and never derive a signature from the `.mthds` source when the verdict carries it. + +- **If a tool is absent from this session** (the MCP server isn't connected), STOP and tell the user in one line: *"The Pipelex MCP server isn't connected — on Mistral Vibe the local workshop (`npx -y @pipelex/mcp@latest`) is not auto-spawned: register it in Vibe's MCP configuration with `PIPELEX_API_KEY` in its environment, then retry."* +- **If a call returns `status: "error"` with an error of class `config`**, STOP the same way and surface the error's `hint` verbatim. Two `config` errors deserve a precise reading: a **403** on `mthds_codegen` is a feature gate, not a key problem — its hint says code generation is not enabled for the organization on the hosted API; never answer it with "check your key" — and `kind: "paywall"` is the plan limit, whose hint points at billing. +- The server authenticates with **`PIPELEX_API_KEY`** from its environment — the same variable the plugin's validation hook documents. +- **`mthds_list_methods`** is optional: it resolves a catalog method the user names without its `mt_…` id. When it is absent, integrate by id, address or files; never stop for it. + +## Mode + +Automatic by default: state the target, the destination and the generator in one line before writing anything, decide the routine calls yourself, and pause only for a genuinely ambiguous decision (which app in a monorepo; which of two Python audiences; a `method_id` source). Explicit user signals win — "just do it" is automatic, "walk me through" is interactive, and in interactive mode the dependency additions and the tooling edits are confirmed before they happen. Every MCP call branches on the structured verdict, never on transport. + +## The rules that never bend + +- **The write arm, always.** Every `mthds_codegen` call passes `output_dir`. A refused or failed write is handled as a refusal — never by calling again without `output_dir` and writing the returned bytes yourself. A generated file re-emitted through the conversation is one trailing newline away from a broken stamp, and the whole point of the trust chain is that the tree on disk is byte-identical to what the engine emitted. +- **Generated files are never opened for editing, never formatted, never linted.** Each artifact carries a stamp with its own content hash and the lock hashes every artifact; a reformat, a trimmed newline or a re-serialized lock turns the offline check red. This is why the tooling exclusions are made **before** the tree exists. +- **One directory per method.** After writing, the workshop reports any stamped file the new lock does not list as an orphan and never deletes it; two methods in one directory therefore read as permanently non-current, by design. You never delete an orphan either, and you never offer "clean up the orphans" — the moment two methods share a directory, that advice deletes real files. +- **Never generate from one source and run from another.** Types from a local bundle over a call site that runs by `method_id` is the shape that drifts silently. The sidecar records the selector; the call site uses the same one. +- **A project that owns a codegen harness keeps it.** Never write a second generated layout beside the one the project already has. +- **No `dropWireNulls` / `wireOutput` helper.** The ts-zod emitter projects optional fields as `.nullish()`, so a generated schema parses the runtime's explicit `null`s directly; a null-stripping helper is lossy (it removes legitimate nulls inside opaque fields) and must not be written into a project. + +## Process + +### Step 1: Identify the method and the project + +**The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: + +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. +- **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. + +**The project** is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the working area. A workspace holding several (a monorepo, a full-stack repo) is a question — which app? — never a guess. **No project at all** → this is not an integration yet: offer the `pipelex-scaffold` skill (open `../pipelex-scaffold/SKILL.md`), which creates one and hands it back here. + +Then look for a **codegen harness**: a `codegen` script in `package.json` or a `codegen` Makefile target, a `sources.json` carrying a `derived` map, `docs/codegen.md` or `docs/add-method.md`, a `methods/` directory beside `src/generated/` or `/generated/`. Either of the first two decides; the rest only corroborate. If the project has one, follow [A project that owns a codegen harness](#a-project-that-owns-a-codegen-harness) from here. + +If a `sources.json` with `"generator": "pipelex-integrate"` already names this method, this is [refresh mode](#refresh-mode). + +### Step 2: Prove the method is integrable + +Call **`mthds_validate`** with the selector. Branch: + +- `status: "ok"`, `is_valid: true`, `is_runnable: true`, `pending_signatures: []` → integrable; keep the verdict, step 3 reads from it. +- `is_valid: true` but **not runnable** or `pending_signatures` non-empty → a scaffold with a concept set but no runnable pipes; integrating it produces a call site that cannot succeed. STOP: finish the method with `/pipelex-design` first. Nothing is generated. +- `is_valid: false` → route the `validation_errors[]` to `/pipelex-design` or `/pipelex-edit`; a by-id method's stored content is fixed where it is edited, not here. +- `status: "error"` → class `config` stops per the Requirements; class `input_domain` at `method_ref` / `method_id` is reported in the tool's own words (an unknown or foreign-organization id — the catalog is org-scoped, so another org's method reads exactly like a miss — an unfetchable address, a registry-form ref); class `runtime` → retry once, then report. + +### Step 3: Read the pipe's signature — from the verdict + +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. + +**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. + +### Step 4: Choose the target, the destination and the generator + +State the three in one line before writing. The rule for the target is about **audience**, not language: + +| Project | Target | Emits | +|---|---|---| +| `package.json` with a TypeScript build (a `tsconfig.json`, or a runtime/bundler that strips types) | `ts-zod` | `types.ts` (zod schemas + inferred types, depends only on `zod`) and `binder.ts` (`parse` / `serialize`); keep both | +| `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | +| `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | + +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). + +### Step 5: Make the tooling leave the tree alone — before the tree exists + +Add the generated directory to the formatter's and linter's ignore lists per the language reference (`.prettierignore`, an ESLint flat-config `ignores`, Biome; `[tool.ruff] exclude`, Black, isort), confirm the type checker's include **still covers it** (an exclusion that would drop it is not added), and confirm it is not gitignored. Do this **before** step 6: the first project-wide `format` run after generation would otherwise rewrite the stamps and turn the check red. + +### Step 6: Generate + +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root (or register the workshop with that working directory) — do not ride content instead. + +Branch on the structured result: + +- `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. +- `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. +- `status: "error"`, class `input_domain` located at `output_dir`: + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. +- `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. +- Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. +- Success with **`is_current: false`** → a write the check disowns. Report the `drifts[]` verbatim (`path`, `category`, `detail`) and stop; never commit a tree the check rejects. + +### Step 7: Write the sidecar + +Write an **unstamped `sources.json`** beside the lock — the only state this skill keeps. The lock signs the artifacts, not their sources; without the sidecar the next run re-derives everything and a bundle edit is undetectable. Shape: + +```json +{ + "comment": "Written by /pipelex-integrate. `method` and `target` are how this tree was generated — re-run the skill to refresh it. `sources` is the SHA-256 of each local .mthds source, so a bundle edit that was never regenerated is detectable. Not part of the codegen lock; do not hand-edit.", + "generator": "pipelex-integrate", + "method": { "files": ["methods/summarize-pdf/main.mthds"] }, + "target": "ts-zod", + "pipe": { + "pipe_ref": "summarize.summarize_pdf", + "inputs": { "document": "native.Document", "context": "native.Text?" }, + "output": "summarize.DocumentSummary" + }, + "sources": { "methods/summarize-pdf/main.mthds": "" } +} +``` + +`method` is exactly one of `{files}`, `{method_ref}`, `{method_id}`, as passed. Paths are relative to the **project root**, not to the workshop's working directory. `pipe` records what the call site was typed against — `Concept` single, `Concept[]` a list, `Concept?` optional — so refresh mode can tell a signature change from a body change. `sources` is empty for a `method_ref` / `method_id` source. Hashes are over the raw bytes: `shasum -a 256 ` / `sha256sum ` / `hashlib.sha256(path.read_bytes())`. + +### Step 8: Add the dependencies the generated code needs + +With the project's own package manager (read the lockfile): `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen). State what is added; interactive mode confirms first. + +### Step 9: Write the call site + +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. + +| Declared concept | TypeScript parameter | Python parameter | +|---|---|---| +| `native.Text` (or a refinement) | `string` | `str` | +| `native.Number` | `number` | `float` | +| `native.YesNo` | `boolean` | `bool` | +| `native.Date` | `string` (ISO 8601) | `str` (ISO 8601) | +| `native.Image`, `native.Document` | `{ url: string }` — an `http(s)` URL or a `pipelex-storage://` reference | `dict[str, Any]` with a `url` key | +| a structured concept, or a composite native (`Page`, `TextAndImages`, `JSON`) | the generated type from `types.ts` | the generated model from `models.py` | +| `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | +| not required | optional parameter (`?`) | `T \| None = None` | + +**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. + +### Step 10: Wire the offline drift gate + +- **TypeScript**: copy [references/codegen-check.mjs](references/codegen-check.mjs) **verbatim** to `scripts/codegen-check.mjs`, register `"codegen:check": "node scripts/codegen-check.mjs …"` in `package.json`, and **extend the project's existing aggregate gate** — a `check` / `ci` / `validate` / `verify` script, a Makefile `check` target, the lint or test step of an existing workflow — rather than inventing a new one. The script runs `@pipelex/sdk`'s `runCodegenCheck` over each directory, compares the sidecar's source hashes against the committed `.mthds` files, and exits `0` current / `1` drift or stale source / `2` no verdict. A project with no aggregate gate gets the script and one sentence in the report saying where to call it. +- **Python, `python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists (`pipelex codegen check`) needs the `pipelex` runtime a consumer deliberately does not have — do not add `pipelex` as a dependency to get a gate. The sidecar is still written (refresh mode and the editing skills' staleness notice read it), and the report says plainly: the tree is protected by its stamps and lock, but nothing in CI proves it current; refresh with this skill after every bundle edit. +- **Python, `python-structures`**: the project already depends on `pipelex`, so `pipelex codegen check ` (exit `0` / `1` / `2`) is wired into its existing gate. + +### Step 11: Verify + +Run the project's formatter **on the files you wrote only** — never on the generated tree; the step-5 exclusions are what make a later project-wide run safe — then its type checker, then the gate you installed. A failure in your own code is yours to fix before reporting; a failure inside the generated tree is reported, never patched. + +### Step 12: Report + +What was generated and where; the target and why; the call site's signature; what changed in the tooling config; how to refresh (this skill again after a bundle edit); for a `python-pydantic` project, that no offline drift check exists yet and refresh is the guard; for a `method_id` source, that the catalog is unversioned. Then the hand-off: `/pipelex-inputs` prepares inputs and offers a run. + +## Refresh mode + +Entered when the user asks to refresh, regenerate or update the types; when `/pipelex-edit` or `/pipelex-design` hand off after editing a bundle a sidecar names; or when step 1 finds a sidecar for the method. **Re-derive nothing the sidecar already records, regenerate in place, and leave alone everything the regeneration did not invalidate.** + +| Taken from disk | Re-derived | Left alone | +|---|---|---| +| the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | + +One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one that adopted their pattern, regenerates every method in one place, checks them in one place, and keeps its own sidecar. Writing this skill's tree beside that would leave two regeneration paths, two sidecar dialects and files the workshop never emits. So on such a project: + +- **place the method where the project keeps them** — `methods//main.mthds`, or the project's manifest form for a catalog or published method; +- **run the project's generator** — `make add-method METHOD=…` when the project has it and the method is remote, its `codegen` script or Makefile target otherwise; +- **write the call site the way the project's docs and existing methods do** (`docs/codegen.md`, `docs/add-method.md`, the existing actions or CLI commands), not the shape of step 9; +- **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); +- **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. + +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| a required tool is absent | STOP with the one-line MCP-connection message above | +| `status: "error"`, class `config` — including the codegen **403** feature gate | STOP, surface `hint` verbatim; never say "check your key" for a 403 | +| `status: "error"`, class `config`, `kind: "paywall"` | STOP, surface the plan-limit hint | +| `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | +| not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | +| `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | +| `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | +| success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | +| success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | +| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `mthds_list_methods` absent | integrate by id, address or files; never stop for it | + +## Reference + +- [references/typescript.md](references/typescript.md) — detecting a TypeScript project (build, package manager, generated root, Prettier / ESLint / Biome exclusions, `tsconfig` coverage, the aggregate gate, the call-site location), the call-site module and client helper templates, the `codegen:check` wiring, the harness a starter-derived project owns. +- [references/python.md](references/python.md) — the same for Python (import package, `python-pydantic` vs `python-structures`, uv / poetry / pipenv / pip, Ruff / Black / isort exclusions, pyright / mypy coverage, `__init__.py` and package data, the async module plus its sync wrapper, `pipelex codegen check` for the structures audience, the asymmetry sentence for everyone else). +- [references/codegen-check.mjs](references/codegen-check.mjs) — the offline gate copied verbatim into TypeScript projects. +- [MTHDS Language Reference](../shared/mthds-reference.md) — for reading a bundle's `main_pipe` and `output` declarations on the fallback path. diff --git a/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs new file mode 100644 index 0000000..ade01da --- /dev/null +++ b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs @@ -0,0 +1,166 @@ +#!/usr/bin/env node +// codegen-check.mjs — the offline drift gate for Pipelex-generated trees. +// +// Copied verbatim into a project by /pipelex-integrate. Run it from the project root +// with one generated directory per argument: +// +// node scripts/codegen-check.mjs src/generated/summarize-pdf src/generated/extract-entities +// +// For each directory it (1) runs @pipelex/sdk's runCodegenCheck over the stamped files +// against codegen.lock — pure hashing, no engine, no network, no API key — and (2) compares +// the SHA-256 recorded for each .mthds source in sources.json against the file on disk, so +// a bundle edited without a regeneration is caught as `stale-source`. +// +// Exit codes: 0 current · 1 drift or stale source · 2 no verdict (no lock, an unreadable +// file, a symlink in the tree). Precedence across directories: 2 > 1 > 0. +// +// It imports only Node builtins and @pipelex/sdk, and writes through process.stdout / +// process.stderr so a no-console lint rule stays quiet. When @pipelex/sdk ships this check +// as a command, replace this file with that one line. + +import { createHash } from "node:crypto"; +import { lstat, readdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import process from "node:process"; +import { CodegenLockError, isStampableArtifactPath, runCodegenCheck } from "@pipelex/sdk"; + +const EXIT_CURRENT = 0; +const EXIT_DRIFT = 1; +const EXIT_NO_VERDICT = 2; + +const LOCK_FILENAME = "codegen.lock"; +const SIDECAR_FILENAME = "sources.json"; +const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); + +const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); + +const out = (line) => process.stdout.write(`${line}\n`); +const err = (line) => process.stderr.write(`${line}\n`); + +/** Every regular file under `root`, as sorted forward-slash paths relative to it. Refuses symlinks. */ +async function walk(root, relative = "") { + const absolute = relative ? path.join(root, relative) : root; + const entries = await readdir(absolute, { withFileTypes: true }); + const paths = []; + for (const entry of entries) { + const rel = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isSymbolicLink()) { + throw new Error(`refusing to read through the symlink ${rel}`); + } + if (entry.isDirectory()) { + if (PRUNED_DIRECTORIES.has(entry.name)) continue; + paths.push(...(await walk(root, rel))); + } else if (entry.isFile()) { + paths.push(rel); + } + } + return paths.sort(); +} + +async function readStrict(filePath) { + return strictUtf8.decode(await readFile(filePath)); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** The lock check: { code, lines } — never throws. */ +async function checkTree(dir) { + let lockContent; + try { + lockContent = await readStrict(path.join(dir, LOCK_FILENAME)); + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.code === "ENOENT" ? "not found" : error.message}`] }; + } + + let files; + try { + const rootStat = await lstat(dir); + if (rootStat.isSymbolicLink()) throw new Error("the generated directory itself is a symlink"); + const stampable = (await walk(dir)).filter((rel) => isStampableArtifactPath(rel)); + files = []; + for (const rel of stampable) { + files.push({ path: rel, content: await readStrict(path.join(dir, rel)) }); + } + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${error.message}`] }; + } + + try { + const report = await runCodegenCheck({ lockContent, files }); + if (report.isCurrent) { + return { + code: EXIT_CURRENT, + lines: [` ${files.length} artifact(s) current (crate ${report.crateFingerprint.slice(0, 12)}, engine ${report.engineVersion})`], + }; + } + return { code: EXIT_DRIFT, lines: report.drifts.map((drift) => ` ${drift.category}: ${drift.path} — ${drift.detail}`) }; + } catch (error) { + if (error instanceof CodegenLockError) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.message}`] }; + } + throw error; + } +} + +/** The sidecar check against the .mthds sources, relative to the project root: { code, lines }. */ +async function checkSources(dir) { + let sidecar; + try { + sidecar = JSON.parse(await readStrict(path.join(dir, SIDECAR_FILENAME))); + } catch (error) { + if (error.code === "ENOENT") { + return { code: EXIT_CURRENT, lines: [` no ${SIDECAR_FILENAME} — source staleness not checked`] }; + } + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; + } + + const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + const lines = []; + for (const [source, recorded] of Object.entries(sources).sort()) { + let onDisk; + try { + onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); + } catch (error) { + lines.push(` stale-source: ${source} — recorded as a source but ${error.code === "ENOENT" ? "no longer on disk" : `unreadable (${error.message})`}`); + continue; + } + if (onDisk !== recorded) { + lines.push(` stale-source: ${source} — edited since the types were generated`); + } + } + return { code: lines.length ? EXIT_DRIFT : EXIT_CURRENT, lines }; +} + +function worse(a, b) { + // Precedence: no verdict > drift > current. + if (a === EXIT_NO_VERDICT || b === EXIT_NO_VERDICT) return EXIT_NO_VERDICT; + if (a === EXIT_DRIFT || b === EXIT_DRIFT) return EXIT_DRIFT; + return EXIT_CURRENT; +} + +async function main(argv) { + const dirs = argv.slice(2); + if (dirs.length === 0) { + err("usage: node scripts/codegen-check.mjs [ ...]"); + return EXIT_NO_VERDICT; + } + + let exitCode = EXIT_CURRENT; + for (const dir of dirs) { + out(`${dir}`); + const tree = await checkTree(dir); + const sources = tree.code === EXIT_NO_VERDICT ? { code: EXIT_CURRENT, lines: [] } : await checkSources(dir); + const code = worse(tree.code, sources.code); + const write = code === EXIT_CURRENT ? out : err; + for (const line of [...tree.lines, ...sources.lines]) write(line); + if (code === EXIT_DRIFT) err(" Run /pipelex-integrate to refresh the generated types."); + exitCode = worse(exitCode, code); + } + + out(`\ncodegen-check: ${exitCode === EXIT_CURRENT ? "current" : exitCode === EXIT_DRIFT ? "drift" : "no verdict"}`); + return exitCode; +} + +process.exit(await main(process.argv)); diff --git a/pipelex-vibe/skills/pipelex-integrate/references/python.md b/pipelex-vibe/skills/pipelex-integrate/references/python.md new file mode 100644 index 0000000..cc64358 --- /dev/null +++ b/pipelex-vibe/skills/pipelex-integrate/references/python.md @@ -0,0 +1,102 @@ +# Integrating into a Python project + +Companion to `/pipelex-integrate` for a project that has a `pyproject.toml` (or `setup.py` / `requirements.txt`). The SDK facts were checked against `pipelex-sdk` as of 2026-09 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Audience → target** | `pipelex` **not** among the dependencies (`[project].dependencies`, `requirements*.txt`) → `python-pydantic`, no question: `python-structures` imports the runtime and would not even load; `pipelex` present → `python-structures` if `@pipe_func` or `from pipelex.core.stuffs.structured_content import StructuredContent` appears in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | +| **Import package** | the package `[project].name` names (dashes → underscores), or the setuptools `packages` list, or the top-level directory holding `__init__.py` (`src//` in a src layout) | ask | +| **Package manager** | the lockfile: `uv.lock` → uv (`uv add`), `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none → `pip install` into the active environment, stated | +| **Generated root** | an existing directory already holding generated code → beside it; else `/generated/` | ask | +| **Formatter and linter** | `[tool.ruff]` → add the generated directory to `exclude` (or `extend-exclude`); `[tool.black]` → `extend-exclude`; `[tool.isort]` → `skip` / `extend_skip_glob` | a tool this table does not name: read its config, add the equivalent, say so | +| **Type checker still covers the tree** | `[tool.pyright]` `include` / `exclude`; `[tool.mypy]` `packages` / `files` / `exclude` | an exclusion that would drop it is **not** added; generated code stays type-checked | +| **Packaged for distribution** | a `[build-system]` table and an import package | the bundle goes **inside** the package (`/methods//main.mthds`) and `*.mthds` plus `codegen.lock` are registered as package data, as the Python starter does — a wheel that ships the call site must ship the bundle it loads at call time; an unpackaged app keeps `methods//` at the project root | +| **Aggregate gate** | a Makefile `check` target; `.github/workflows/*.yml`; `.pre-commit-config.yaml`; a `nox`/`tox` session | none: nothing to wire for `python-pydantic` anyway (below) | +| **Call-site location** | the project's existing service / client layer (`services/`, `clients/`, `api/`) → beside it | `/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | reported and un-ignored on confirmation | +| **Owns a codegen harness** | a `codegen` Makefile target; `docs/codegen.md`; `/generated/` beside `/methods/` | the harness section below | + +Why the Ruff exclusion is not optional: `ruff format` rewrites the generated bytes, breaks every stamp, and makes the drift check report the tree as hand-edited. The exclusion goes in before generation, so the first project-wide format after it is already safe. + +## The generated tree + +`mthds_codegen` with `target: "python-pydantic"` and `output_dir: "/generated/"` writes: + +- `models.py` — stamped; `from __future__ import annotations`, `from pydantic import BaseModel, Field`; one plain `BaseModel` subclass per concept, natives included, with `Field(..., description=…)`; no Pipelex import. Non-required fields are `T | None = None`. +- `codegen.lock` — TOML with the crate fingerprint, the engine version and one `[[artifacts]]` entry per stamped file. + +With `target: "python-structures"` the file is `structures.py` — `StructuredContent` subclasses from `pipelex.core.stuffs.structured_content`, natives not re-emitted — for a Pipelex host whose `@pipe_func` functions return them. + +**The skill creates the `__init__.py` files** the tree needs to be importable: `/generated/__init__.py` (once per project, a one-line docstring) and `/generated//__init__.py` (empty). Codegen never emits them and they are never artifacts: they carry no stamp, so the writer does not touch them and the check does not count them as orphans. When the project is packaged, list the generated subpackages where the project lists its packages and add `codegen.lock` as package data, as the Python starter's `pyproject.toml` does. Beside the lock the skill writes `sources.json`, unstamped. + +## The call site + +One module per method. `summarize_pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle inside the package: + +```python +# /pipelex/summarize_pdf.py +"""Run the summarize_pdf method through the hosted Pipelex API. + +`document` is an http(s) URL or a pipelex-storage:// reference, as ``{"url": ...}``. For a +local file or bytes, call ``client.prepare_inputs(files=[...], inputs=...)`` first: it uploads +and rewrites the value. It treats any string it does not recognise as data:, http(s):// or +pipelex-storage:// as a LOCAL FILE PATH it reads and uploads, so gate schemes before handing +values from an untrusted caller to it. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +from pipelex_sdk.client import PipelexAPIClient + +from .generated.summarize_pdf.models import DocumentSummary + +PIPE_CODE = "summarize_pdf" +BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" + + +async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + inputs: dict[str, Any] = {"document": document} + if context is not None: + inputs["context"] = context + async with PipelexAPIClient() as client: + results = await client.start_and_wait( + pipe_code=PIPE_CODE, + mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + inputs=inputs, + ) + return DocumentSummary.model_validate(results.main_stuff) + + +def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + """The blocking wrapper for a synchronous caller.""" + return asyncio.run(summarize_pdf(document=document, context=context)) +``` + +Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: + +- **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. +- **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. + +Facts the module leans on: `PipelexAPIClient()` reads `PIPELEX_API_KEY` and `PIPELEX_BASE_URL` (default `https://api.pipelex.com`) itself; it is async-only and used as an async context manager; there is **no barrel** in `pipelex_sdk` by design, so every import is a full module path (`pipelex_sdk.client`, `pipelex_sdk.runs`, `pipelex_sdk.errors`); `start_and_wait(...)` takes the durable path on the hosted API (default 2 s poll, 20 min budget, `wait_options=WaitForResultOptions(...)` to change them) and falls back to blocking on a bare runner, returning `RunResults` whose `main_stuff` is always present for a completed run. Let the typed errors from `pipelex_sdk.errors` propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own handling only where it already wraps its other clients. + +Parameters are keyword-only, typed from the signature: `str` for Text and Date (ISO 8601), `float` for Number, `bool` for YesNo, `dict[str, Any]` with a `url` key for Image and Document, the generated model for a structured concept or a composite native, `list[T]` for a list, `T | None = None` for a non-required input. Names are the pipe's input names as declared. A project that already constructs a `PipelexAPIClient` somewhere gets that construction reused instead of a fresh `async with` per call; an async-native project (FastAPI, an existing async codebase) gets the async function alone, without the `_sync` wrapper. + +## The drift gate — an honest asymmetry + +- **`python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists — `pipelex codegen check ` — lives in the `pipelex` runtime, which a hosted-API consumer deliberately does not install. Do not add `pipelex` as a dependency to get a gate; that reverses the decision the target expresses. Write the sidecar anyway (refresh mode and `/pipelex-edit`'s staleness notice read it) and put this sentence in the report: *the generated tree is protected by its stamps and lock, but nothing in CI proves it current; run `/pipelex-integrate` again after every bundle edit.* When the Python SDK gains the check, this section becomes one line. +- **`python-structures`**: the project already depends on `pipelex`, so wire `pipelex codegen check /generated/` into its existing gate (a Makefile `check` target, a workflow step). Exit codes: `0` current, `1` drift, `2` no lock or an unreadable lock. Offline: no engine boot, no network, no key. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-python` has `make codegen` and `make codegen-check`, both shelling out to a `pipelex` CLI the starter deliberately does not depend on (`PIPELEX=` in its Makefile), methods under `/methods//`, trees under `/generated//` with hand-committed `__init__.py` files, and typed CLI commands under its mode sub-packages. On such a project: + +- place the bundle under `/methods//main.mthds` and add the two `codegen` / `codegen-check` lines the Makefile pattern uses for its other methods; +- run `make codegen` when a `pipelex` CLI is reachable; when it is not, call `mthds_codegen` with `output_dir` set to `/generated/` — the harness's own layout, byte-identical (same engine, same stamps, same lock, no sidecar in this starter) — and say `make codegen` is the refresh once `PIPELEX=` points at an install; +- write the CLI command the way `docs/cli-architecture.md` and the existing commands do, importing the generated model from `.generated..models`; +- verify with `make agent-check` and `make agent-test`; no `sources.json` of this skill's shape and no second generated directory. diff --git a/pipelex-vibe/skills/pipelex-integrate/references/typescript.md b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md new file mode 100644 index 0000000..bce5bfa --- /dev/null +++ b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md @@ -0,0 +1,118 @@ +# Integrating into a TypeScript project + +Companion to `/pipelex-integrate` for a project that has a `package.json`. Everything here follows the shape the Pipelex JS starter converged on; the SDK facts were checked against `@pipelex/sdk` 0.17 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **TypeScript build** | a `tsconfig.json`; a `typescript` dev dependency; a bundler or runtime that strips types (Next.js, Vite, tsx, Bun, Deno) | a plain JavaScript project is asked — `types.ts` needs a TypeScript build; the alternative is `python-pydantic`'s sibling in this language, which does not exist yet, so the honest answer is "add TypeScript or skip codegen" | +| **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock` / `bun.lockb` → bun | none → npm, stated | +| **Generated root** | an existing directory already holding generated code (`generated/`, `gen/`, `__generated__/`) → beside it; else `src/generated/` when `src/` exists; else `generated/` | ask | +| **Formatter** | `.prettierrc*` or a `prettier` dev dependency → add `src/generated/` to `.prettierignore` (create the file if absent); `biome.json*` → the files-ignore key its version uses (`files.ignore` before Biome 2, `files.includes` with a `!` negation from Biome 2 on) | a formatter this table does not name: read its config, add the equivalent, say so | +| **Linter** | `eslint.config.*` (flat config) → add `"src/generated/**"` to `globalIgnores([...])` or an `{ ignores: [...] }` entry; legacy `.eslintrc*` → `.eslintignore`; Biome as above | same | +| **Type checker still covers the tree** | `tsconfig.json` `include` / `exclude` — the generated directory must stay inside `include` and outside `exclude` | an exclusion that would drop it is **not** added; the report says the typecheck is the check that covers generated code | +| **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | +| **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | + +Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. + +## The generated tree + +`mthds_codegen` with `target: "ts-zod"` and `output_dir: "src/generated/"` writes: + +- `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. +- `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. +- `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. + +Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. + +## The call site + +One module per method. `summarize-pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle: + +```ts +// src/pipelex/summarizePdf.ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import type { RunResults } from "@pipelex/sdk"; +import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; +import type { DocumentSummary } from "../generated/summarize-pdf/types"; +import { getPipelexClient } from "./client"; + +const PIPE_CODE = "summarize_pdf"; +const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); + +export interface SummarizePdfInputs { + /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, + * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * it uploads and rewrites the value. Note: prepareInputs treats any string it does not + * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and + * uploads, so a public endpoint must gate schemes before handing values to it. */ + document: { url: string }; + context?: string; +} + +export async function summarizePdf(inputs: SummarizePdfInputs): Promise { + const bundle = await readFile(BUNDLE_PATH, "utf8"); + const results: RunResults = await getPipelexClient().startAndWaitForResult({ + pipe_code: PIPE_CODE, + mthds_contents: [bundle], + inputs, + }); + return parseDocumentSummary(results.main_stuff); +} +``` + +Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: + +- **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. +- **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. + +The shared client helper, created once per project and reused by every method (if the project already constructs a `PipelexApiClient` somewhere, import that instead): + +```ts +// src/pipelex/client.ts +import { PipelexApiClient } from "@pipelex/sdk"; + +let client: PipelexApiClient | undefined; + +/** Reads PIPELEX_API_KEY and PIPELEX_BASE_URL (default https://api.pipelex.com) from the environment. */ +export function getPipelexClient(): PipelexApiClient { + client ??= new PipelexApiClient(); + return client; +} +``` + +`startAndWaitForResult(options, pollOptions?)` takes the durable path (`start` + poll, default 2 s interval, 20 min budget) on the hosted API and falls back to a blocking `execute` on a bare runner; it returns `RunResults` whose `main_stuff` is always present for a completed run. Let the SDK's typed errors propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own error handling only where it already wraps its other clients. A server-side framework seam (a Next.js Server Action, an Express handler) is the caller's; this module is framework-free. + +Parameter types from the signature: `string` for Text and Date (ISO 8601), `number` for Number, `boolean` for YesNo, `{ url: string }` for Image and Document, the generated type for a structured concept or a composite native, `T[]` for a list, `?` for a non-required input. Key names are the pipe's input names as declared, snake_case included. + +## The offline gate + +Copy `references/codegen-check.mjs` verbatim to `scripts/codegen-check.mjs`. It imports only Node builtins and `@pipelex/sdk`, runs under plain `node` whatever the project's TypeScript build, and prints through `process.stdout` / `process.stderr` so a `no-console` rule does not fire. Register it and extend the existing gate: + +```json +{ + "scripts": { + "codegen:check": "node scripts/codegen-check.mjs src/generated/summarize-pdf", + "check": "npm run lint && npm run typecheck && npm run codegen:check" + } +} +``` + +Add every method's directory to the `codegen:check` line as it is integrated. Exit codes: `0` current, `1` drift or stale source, `2` no verdict (no lock, an unreadable file, a symlink in the tree). It runs from the project root because `sources.json` records source paths relative to it. When `@pipelex/sdk` ships this as a command of its own, the script is replaced by that one line. + +## The Node-only boundary + +`readFile`, `node:path` and `process.cwd()` in the call site are server-side facts. In a framework with a client/server split (Next.js, Remix, SvelteKit), the module belongs on the server side — a Server Action, a route handler, a loader — and the JS starter marks such modules with `import "server-only"`. Never import it from a component that renders in the browser: the API key would leave the server. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js`, or one that copied its codegen kit, has: `npm run codegen` (regenerates every `methods/*` tree through the hosted `/v1/codegen`, writes `contracts.ts` and `sources.json` with a `derived` map), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (semantic, keyed), and `make add-method METHOD=` (manifest, tree, action trio, narrower, form, tab — one shot, never overwrites). On such a project: + +- a local bundle goes under `methods//main.mthds`, then `npm run codegen`; the fan-out follows `docs/codegen.md` and the existing actions under `src/actions/` and narrowers under `src/types/`; +- a catalog or published method goes through `make add-method`; +- the verification is `make check`; the refresh is `npm run codegen`; no `sources.json` of this skill's shape, no `scripts/codegen-check.mjs`, no second generated directory. diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md new file mode 100644 index 0000000..90a4401 --- /dev/null +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -0,0 +1,158 @@ +--- +name: pipelex-scaffold +description: Start a new project that will call MTHDS methods through Pipelex, in TypeScript or Python — from one of the Pipelex starter templates or from the ecosystem's own initializer — and hand it to /pipelex-integrate. Use when the user says "start a new project with Pipelex", "I have a method and need an app around it", "create a Next.js app that runs my method", "set up a Pipelex project from scratch", "new Python CLI for this method", "which starter should I use", "bootstrap a Pipelex project", or wants a codebase where none exists yet. Also use when the user is standing in a freshly cloned pipelex-starter-js or pipelex-starter-python that has not been renamed yet — this skill runs the template's own bootstrap for them. Not for adding Pipelex to code that already exists: that is /pipelex-integrate. + +--- + +# Scaffold a project for Pipelex methods + +Give a user who has no project yet a project that is ready for `/pipelex-integrate`. This skill has exactly two branches and carries no templates of its own: + +- **One of the Pipelex starters** when the user wants the opinionated shape: `pipelex-starter-js` for a web app whose forms are rendered from the methods' own contracts, `pipelex-starter-python` for a CLI or service that runs methods in the three execution modes. You acquire the template, commit it once as it came, then run the clone's **own** `bootstrap` skill — the rename logic lives in the starters and is never reimplemented here. +- **The ecosystem's own initializer** when the user wants their framework or a minimal project: `uv init --package`, `npm create next-app@latest`, `django-admin startproject`, whatever the framework documents. You run it; you never assemble a project by hand. + +Both branches end the same way: an env file that follows the starters' convention, one pristine commit that makes everything after it reviewable, and the hand-off — to `/pipelex-integrate` when a method exists, to `/pipelex-design` first when none does. + +**What this skill is not.** Not a template engine (no cookiecutter, no copier, no framework matrix of its own), not a bootstrap (the starters own theirs), not a runner or a dev-server launcher, not a deployer. It needs no MCP tool and no API key: git, the starters' scripts and the ecosystem's initializers are all it uses. + +## Choosing the branch + +A cheap, reliable signal decides; an inconclusive one asks one question; nothing is guessed twice. + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | +| **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | + +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. + +[references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. + +## Mode + +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. + +## Branch A — one of the starters + +### Step 1: Prerequisites + +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. + +- **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. +- **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). +- **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. + +### Step 2: Acquire the template + +**Local, the default.** Clone shallow, read the template's identity, then detach from it: + +```bash +git clone --depth 1 https://github.com/Pipelex/.git +git -C rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf /.git && git -C init -b main +``` + +The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. + +**GitHub, on request.** When the user asked for a repository on GitHub: + +```bash +gh repo create / --template Pipelex/ --private --clone +``` + +Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. + +Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. + +### Step 3: Commit the pristine template — exactly once + +```bash +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. + +### Step 4: Run the clone's own bootstrap + +Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written. The project's skills are not loaded in this session — it began elsewhere — so read the file; do not look for a `/bootstrap` command. Run every command it gives from inside the project directory (`cd && …`, or `-C `), because that skill assumes it is standing in the repo root. + +Feed it what the conversation already holds — the project name, title, description, author, repository URL, license — so that it asks once, consolidated, for whatever is left, exactly as its own Step 2 says. It dry-runs, previews, runs, re-syncs the lock file, runs the project's own checks (`make all` on JS; `make agent-check` and `make agent-test` on Python), and removes itself. Its rules stand unchanged: it never commits, its edits stay uncommitted for the user's review (the Python renames are staged by `git mv`, which its skill explains), and a red check is fixed, never skipped. **Add nothing to that procedure and reimplement none of it.** If the clone carries no bootstrap skill — a future template dropped it — follow the README's "manual equivalent" list and say that the template changed. + +### Step 5: The env file + +```bash +cp /.env.example /.env.local # JS: Next.js reads .env.local +cp /.env.example /.env # Python: python-dotenv reads .env +``` + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. + +### Step 6: Verify and hand off + +The bootstrap's own checks are the verification; do not start `make dev`. Write the report (below), then hand the user's method to `/pipelex-integrate`, which recognizes the starter's codegen harness (`npm run codegen`, `make codegen`, `make add-method`) and defers to it rather than writing a second one. + +## Branch B — the ecosystem's initializer + +### Step 1: Prerequisites + +As in branch A, for the language chosen. + +### Step 2: Run the initializer — never assemble by hand + +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. + +Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. + +### Step 3: Version control and the pristine commit + +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: + +```bash +git -C add -A && git -C commit -m "Scaffold project" +``` + +### Step 4: The env file + +Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: + +``` +PIPELEX_BASE_URL=https://api.pipelex.com +PIPELEX_API_KEY= +``` + +### Step 5: Hand off + +Add **no** SDK dependency and create **no** empty `methods/` directory: `/pipelex-integrate` adds `@pipelex/sdk` or `pipelex-sdk` when it writes the first call site, and creates `methods//` when it places the first bundle. A project with nothing to integrate yet has nothing Pipelex-shaped in it beyond the env convention, and that is correct. + +## The report + +Say, in this order: what was created and where; which template or initializer it came from, at which version and SHA; that this skill made exactly one commit and what it holds; what the bootstrap changed and that those changes are uncommitted for review, in the bootstrap's own words (branch A); which env file was written and whether the key was filled from the environment or left for the user; the demos the starter still carries and where the README's removal checklist is (branch A); and the hand-off. + +Two lines are easy to forget and matter: + +- **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd `, then starting Mistral Vibe there, is how they arrive. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | +| The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | +| The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | +| An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | +| `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | + +## Reference + +- [references/starters.md](references/starters.md) — the two starters side by side: what each brings, its prerequisite floors, the acquisition commands, its env file, its bootstrap, its demos and their removal checklist, and the codegen harness `/pipelex-integrate` will find. +- [references/initializers.md](references/initializers.md) — per language, the minimal default and the common frameworks' non-interactive initializers, whether each runs `git init`, and where the import package or `src/` root lands. +- `/pipelex-integrate` — the skill this one hands every project to. diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md new file mode 100644 index 0000000..98b683f --- /dev/null +++ b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md @@ -0,0 +1,38 @@ +# Ecosystem initializers + +Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. + +After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +## Python + +| Want | Command | `git init`? | Where the import package lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | +| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | + +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. + +## TypeScript / JavaScript + +| Want | Command | `git init`? | Where `src/` lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | +| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Node library | the minimal recipe above | no | `src/` | +| pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | + +`npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. + +## What every branch-B project shares afterwards + +- One commit, the pristine scaffold, so the user's first real change is a clean diff. +- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md new file mode 100644 index 0000000..52ccdc3 --- /dev/null +++ b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md @@ -0,0 +1,59 @@ +# The two Pipelex starters + +Both are GitHub **template repositories** under the `Pipelex` organization. Each is a real, CI-tested application against the hosted Pipelex API, not a parameterized template: the identity you see in a fresh clone (`pipelex-starter-js` / `Pipelex Starter`, or `piper` / `Piper`) is a placeholder that the starter's own `bootstrap` skill rewrites. Read the clone's `README.md` after acquiring it — the sections named below are where the details live, and they move as the starters evolve. + +## Side by side + +| | `pipelex-starter-js` | `pipelex-starter-python` | +|---|---|---| +| **Shape** | Next.js (App Router), React, TypeScript strict, Tailwind; a web app with one tab per method whose input form is rendered from the method's own contract by `@pipelex/mthds-form` | A Typer CLI with one command per method, printing JSON on stdout and a cost report on stderr; three execution modes (`blocking`, `attended`, `detached`) as separate sub-packages | +| **Pick it when** | people will use the methods in a browser: forms, uploads, live run status | the methods run from a terminal, a script, a batch job or a service, and the user wants Python | +| **SDK** | `@pipelex/sdk` | `pipelex-sdk` (import package `pipelex_sdk`) — the `pipelex` runtime is **not** a dependency | +| **Methods live in** | `methods//main.mthds`, or `methods//method.json` for a method that lives elsewhere (a catalog id or a published address) | `/methods//main.mthds` | +| **Generated types** | `src/generated//` — `types.ts`, `binder.ts`, `contracts.ts`, `codegen.lock`, `sources.json` | `/generated//` — `models.py`, `codegen.lock` | +| **Codegen harness** | `npm run codegen` (keyed, dev), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (keyed, semantic); `make add-method METHOD=` scaffolds a remote method end to end | `make codegen` / `make codegen-check` — **both shell out to a `pipelex` CLI the starter does not depend on** (`PIPELEX=` in the Makefile); `/pipelex-integrate` knows this and writes into the same layout when that CLI is absent | +| **Toolchain floor** | Node ≥ the `engines.node` field of `package.json` (22.12 at writing); `npm` | `uv`; a Python inside `requires-python` of `pyproject.toml` (3.11–3.14 at writing) | +| **Env file** | `.env.local`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY`, `NEXT_PUBLIC_EXECUTION_MODE` | `.env`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY` | +| **Checks the bootstrap runs** | `npm install --package-lock-only`, then `make all` (lint, format check, typecheck, unit tests, build) | `make li` (lock + sync), then `make agent-check` and `make agent-test` | +| **Agent-facing files** | `CLAUDE.md`, `AGENTS.md`; skills `bootstrap`, `release`, `bump-sdk`, `bump-mthds-form` | `CLAUDE.md`; skills `bootstrap`, `release` | +| **Docs worth reading after bootstrap** | `docs/codegen.md`, `docs/add-method.md`, `docs/input-form.md`, `docs/adopt-in-an-existing-project.md`; README → "Swap in your own pipeline" and "Remove an example" | `docs/codegen.md`, `docs/cli-architecture.md`; README → the per-command sections | +| **Demos it carries** | several demo methods, one tab each; keep them as references or strip them with the README's "Remove an example" checklist | several demo methods, one CLI command each; keep them as references or remove the command and its method directory together | + +## Acquisition + +Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): + +```bash +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git -C rev-parse HEAD +rm -rf /.git && git -C init -b main +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. + +GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): + +```bash +gh auth status +gh repo create / --template Pipelex/pipelex-starter-js --private --clone +gh repo create / --template Pipelex/pipelex-starter-python --private --clone +``` + +## The bootstrap you delegate to + +Both starters carry `.claude/skills/bootstrap/SKILL.md` with a bundled script (`scripts/bootstrap.mjs` / `scripts/bootstrap.py`). Read the file in the clone and follow it; the shape is the same on both: + +1. **Preflight** — confirms the identity is still the template's (`package.json` name `pipelex-starter-js`; `pyproject.toml` `name = "piper"`), notes a dirty tree, and on JS makes sure `node_modules/` exists (`make install`). +2. **Collect** — the package name (kebab on JS, underscores on Python, everything else derives from it), a display title, a one-line description; optionally author name **and** email (never one without the other), the repository URL, and the license (MIT kept, proprietary, or another SPDX id; the copyright holder and year). Pass what the conversation already holds so it asks once for the rest. +3. **Dry run** — the script with `--dry-run` prints the plan; the user confirms. +4. **Run** — the same command without `--dry-run`; on Python the package directory is renamed with `git mv`, which is why the pristine commit must exist first. `--clean` strips the template-only prose; keep it unless the user wants the template charter kept. +5. **Verify** — the lock file is re-synced and the project's own checks run; red is fixed, not skipped. +6. **Self-removal** — `rm -rf .claude/skills/bootstrap`, unstaged like everything else; the user reviews with `git status` and `git diff` and commits when ready. + +The starter's rules are yours while you run it: never commit on the user's behalf, always dry-run first, never touch `.github/` or the `release` skill's logic. + +## What `/pipelex-integrate` finds afterwards + +A bootstrapped starter is a project that **owns a codegen harness**, and `/pipelex-integrate` defers to it: on JS it drops a bundle under `methods//` and runs `npm run codegen`, or runs `make add-method METHOD=…` for a catalog or published method, then follows `docs/codegen.md` and the existing actions for the fan-out; on Python it places the bundle under `/methods//` and runs `make codegen` when a `pipelex` CLI is available, writing into `/generated//` through the Pipelex workshop when it is not. It never writes a second generated layout beside the starter's own. diff --git a/pipelex/skills/pipelex-design/SKILL.md b/pipelex/skills/pipelex-design/SKILL.md index e2a2c35..3cb8f63 100644 --- a/pipelex/skills/pipelex-design/SKILL.md +++ b/pipelex/skills/pipelex-design/SKILL.md @@ -191,7 +191,7 @@ After the gate: 1. **Organize only when the layout needs it.** A direct result that is already coherent skips `/pipelex-organize`. A converged stepwise result normally invokes it automatically because one-definition-per-file construction history and satisfied headers need regrouping. A naturally coherent result in either mode does not take an organization round trip solely for process compliance. 2. **Project the input schema.** Call `mthds_inputs_template` with the final whole-bundle `files` submission plus `explicit: false`. Show the returned compact template, but **do not save it as `inputs.json`** — input preparation belongs exclusively to `/pipelex-inputs`. 3. **Present the flow.** Point to the interactive method graph where the host rendered the valid verdict's view; in terminal hosts, present a concise text flow of the final structure. -4. **Hand off inputs.** Suggest preparing real inputs with `/pipelex-inputs`. +4. **Hand off inputs — and the code.** Suggest preparing real inputs with `/pipelex-inputs`. Then, when the workspace holds a codebase (a `package.json` or a `pyproject.toml`), say that `/pipelex-integrate` wires the method into it with generated types and a typed call site; when it holds none and the user wants an application around the method, `/pipelex-scaffold` creates one and hands it to `/pipelex-integrate`. > **NEVER write `inputs.json` manually.** If the user provides files, paths, or wants to run with real data, invoke `/pipelex-inputs` — it handles the template, path resolution, placeholder formatting, and file copying. @@ -216,7 +216,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex/skills/pipelex-edit/SKILL.md b/pipelex/skills/pipelex-edit/SKILL.md index 1ef13e1..3fa7367 100644 --- a/pipelex/skills/pipelex-edit/SKILL.md +++ b/pipelex/skills/pipelex-edit/SKILL.md @@ -85,6 +85,8 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. +**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. + ## Reference - [MTHDS Language Reference](../shared/mthds-reference.md) — read for concept definitions and syntax before editing constructs you haven't touched recently diff --git a/pipelex/skills/pipelex-inputs/SKILL.md b/pipelex/skills/pipelex-inputs/SKILL.md index 13a02dd..167224c 100644 --- a/pipelex/skills/pipelex-inputs/SKILL.md +++ b/pipelex/skills/pipelex-inputs/SKILL.md @@ -365,6 +365,8 @@ After assembling the inputs, confirm readiness: (Or, for the Template strategy: point out which placeholders the user still needs to fill.) +When the workspace holds a codebase (a `package.json` or a `pyproject.toml`) and the method is not yet wired into it, add one line: `/pipelex-integrate` generates the method's types into the project and writes a typed call site that runs it. + ### Offer to run When the inputs are complete, close by offering to run the method. Offer — never start unprompted: a run executes on the hosted Pipelex API and **spends inference credit**. diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md new file mode 100644 index 0000000..bb8999d --- /dev/null +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -0,0 +1,220 @@ +--- +name: pipelex-integrate +description: Wire an MTHDS method into a Python or TypeScript codebase with generated, drift-proof types and one typed call site that runs it through @pipelex/sdk or pipelex-sdk. Use when the user says "use this method in my app", "call this from my code", "generate types for this method", "wire the method into my project", "add this pipeline to my service", "typed client for this method", "integrate the method", "refresh the generated types", "regenerate the types", "the types are stale", or wants application code that runs a .mthds method — from a local bundle, a catalog id (mt_…) or a published method_ref address. Also the refresh path after a bundle edit. Not for authoring or editing the method itself (/pipelex-design, /pipelex-edit), and not for a project that does not exist yet (/pipelex-scaffold). +allowed-tools: + - Bash + - Read + - Write + - Edit + - Grep + - Glob + + - mcp__plugin_pipelex_pipelex__mthds_validate + - mcp__plugin_pipelex_pipelex__mthds_codegen + - mcp__plugin_pipelex_pipelex__mthds_inputs_template + - mcp__plugin_pipelex_pipelex__mthds_list_methods +--- + +# Integrate an MTHDS method into a codebase + +Take a method — a local `.mthds` bundle, a published address (`method_ref`), or a catalog id (`method_id`) — and a Python or TypeScript project, and leave the project able to call the method with types that cannot silently drift from it. Concretely: + +1. pick the codegen target that matches the project's language **and audience**; +2. have the Pipelex workshop write the generated tree into a dedicated directory per method, through `mthds_codegen`'s write arm, so no generated byte ever passes through you; +3. make the project's formatters and linters leave that tree alone while its type checker keeps covering it; +4. record how the tree was generated in a small sidecar beside the lock, so the next run knows what to refresh and a bundle edit is detectable; +5. wire the offline drift check into the gate the project already runs, where one exists for the language; +6. write one typed call-site module per method, running it through `@pipelex/sdk` or `pipelex-sdk` and narrowing its output with the generated binder or model; +7. verify with the project's own type checker and the gate you just installed. + +Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. + +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. + +## Requirements — the Pipelex MCP tools + +This skill generates through **`mthds_codegen`**, proves the method through **`mthds_validate`**, and reads a pipe's inputs through **`mthds_inputs_template`** on its fallback path — all served by the plugin's `pipelex` MCP server. They are required: never hand-write a generated file, and never derive a signature from the `.mthds` source when the verdict carries it. + +- **If a tool is absent from this session** (the MCP server isn't connected), STOP and tell the user in one line: *"The Pipelex MCP server isn't connected — the plugin manifest spawns the local workshop (`npx -y @pipelex/mcp@latest`), so its absence usually means `node`/`npx` is unavailable or the spawn failed. Check the plugin's MCP connection (`/mcp`)."* +- **If a call returns `status: "error"` with an error of class `config`**, STOP the same way and surface the error's `hint` verbatim. Two `config` errors deserve a precise reading: a **403** on `mthds_codegen` is a feature gate, not a key problem — its hint says code generation is not enabled for the organization on the hosted API; never answer it with "check your key" — and `kind: "paywall"` is the plan limit, whose hint points at billing. +- The server authenticates with **`PIPELEX_API_KEY`** from its environment — the same variable the plugin's validation hook documents. +- **`mthds_list_methods`** is optional: it resolves a catalog method the user names without its `mt_…` id. When it is absent, integrate by id, address or files; never stop for it. + +## Mode + +Automatic by default: state the target, the destination and the generator in one line before writing anything, decide the routine calls yourself, and pause only for a genuinely ambiguous decision (which app in a monorepo; which of two Python audiences; a `method_id` source). Explicit user signals win — "just do it" is automatic, "walk me through" is interactive, and in interactive mode the dependency additions and the tooling edits are confirmed before they happen. Every MCP call branches on the structured verdict, never on transport. + +## The rules that never bend + +- **The write arm, always.** Every `mthds_codegen` call passes `output_dir`. A refused or failed write is handled as a refusal — never by calling again without `output_dir` and writing the returned bytes yourself. A generated file re-emitted through the conversation is one trailing newline away from a broken stamp, and the whole point of the trust chain is that the tree on disk is byte-identical to what the engine emitted. +- **Generated files are never opened for editing, never formatted, never linted.** Each artifact carries a stamp with its own content hash and the lock hashes every artifact; a reformat, a trimmed newline or a re-serialized lock turns the offline check red. This is why the tooling exclusions are made **before** the tree exists. +- **One directory per method.** After writing, the workshop reports any stamped file the new lock does not list as an orphan and never deletes it; two methods in one directory therefore read as permanently non-current, by design. You never delete an orphan either, and you never offer "clean up the orphans" — the moment two methods share a directory, that advice deletes real files. +- **Never generate from one source and run from another.** Types from a local bundle over a call site that runs by `method_id` is the shape that drifts silently. The sidecar records the selector; the call site uses the same one. +- **A project that owns a codegen harness keeps it.** Never write a second generated layout beside the one the project already has. +- **No `dropWireNulls` / `wireOutput` helper.** The ts-zod emitter projects optional fields as `.nullish()`, so a generated schema parses the runtime's explicit `null`s directly; a null-stripping helper is lossy (it removes legitimate nulls inside opaque fields) and must not be written into a project. + +## Process + +### Step 1: Identify the method and the project + +**The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: + +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. +- **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. + +**The project** is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the working area. A workspace holding several (a monorepo, a full-stack repo) is a question — which app? — never a guess. **No project at all** → this is not an integration yet: offer `/pipelex-scaffold`, which creates one and hands it back here. + +Then look for a **codegen harness**: a `codegen` script in `package.json` or a `codegen` Makefile target, a `sources.json` carrying a `derived` map, `docs/codegen.md` or `docs/add-method.md`, a `methods/` directory beside `src/generated/` or `/generated/`. Either of the first two decides; the rest only corroborate. If the project has one, follow [A project that owns a codegen harness](#a-project-that-owns-a-codegen-harness) from here. + +If a `sources.json` with `"generator": "pipelex-integrate"` already names this method, this is [refresh mode](#refresh-mode). + +### Step 2: Prove the method is integrable + +Call **`mthds_validate`** with the selector. Branch: + +- `status: "ok"`, `is_valid: true`, `is_runnable: true`, `pending_signatures: []` → integrable; keep the verdict, step 3 reads from it. +- `is_valid: true` but **not runnable** or `pending_signatures` non-empty → a scaffold with a concept set but no runnable pipes; integrating it produces a call site that cannot succeed. STOP: finish the method with `/pipelex-design` first. Nothing is generated. +- `is_valid: false` → route the `validation_errors[]` to `/pipelex-design` or `/pipelex-edit`; a by-id method's stored content is fixed where it is edited, not here. +- `status: "error"` → class `config` stops per the Requirements; class `input_domain` at `method_ref` / `method_id` is reported in the tool's own words (an unknown or foreign-organization id — the catalog is org-scoped, so another org's method reads exactly like a miss — an unfetchable address, a registry-form ref); class `runtime` → retry once, then report. + +### Step 3: Read the pipe's signature — from the verdict + +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. + +**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. + +### Step 4: Choose the target, the destination and the generator + +State the three in one line before writing. The rule for the target is about **audience**, not language: + +| Project | Target | Emits | +|---|---|---| +| `package.json` with a TypeScript build (a `tsconfig.json`, or a runtime/bundler that strips types) | `ts-zod` | `types.ts` (zod schemas + inferred types, depends only on `zod`) and `binder.ts` (`parse` / `serialize`); keep both | +| `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | +| `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | + +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). + +### Step 5: Make the tooling leave the tree alone — before the tree exists + +Add the generated directory to the formatter's and linter's ignore lists per the language reference (`.prettierignore`, an ESLint flat-config `ignores`, Biome; `[tool.ruff] exclude`, Black, isort), confirm the type checker's include **still covers it** (an exclusion that would drop it is not added), and confirm it is not gitignored. Do this **before** step 6: the first project-wide `format` run after generation would otherwise rewrite the stamps and turn the check red. + +### Step 6: Generate + +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root — do not ride content instead. + +Branch on the structured result: + +- `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. +- `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. +- `status: "error"`, class `input_domain` located at `output_dir`: + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. +- `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. +- Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. +- Success with **`is_current: false`** → a write the check disowns. Report the `drifts[]` verbatim (`path`, `category`, `detail`) and stop; never commit a tree the check rejects. + +### Step 7: Write the sidecar + +Write an **unstamped `sources.json`** beside the lock — the only state this skill keeps. The lock signs the artifacts, not their sources; without the sidecar the next run re-derives everything and a bundle edit is undetectable. Shape: + +```json +{ + "comment": "Written by /pipelex-integrate. `method` and `target` are how this tree was generated — re-run the skill to refresh it. `sources` is the SHA-256 of each local .mthds source, so a bundle edit that was never regenerated is detectable. Not part of the codegen lock; do not hand-edit.", + "generator": "pipelex-integrate", + "method": { "files": ["methods/summarize-pdf/main.mthds"] }, + "target": "ts-zod", + "pipe": { + "pipe_ref": "summarize.summarize_pdf", + "inputs": { "document": "native.Document", "context": "native.Text?" }, + "output": "summarize.DocumentSummary" + }, + "sources": { "methods/summarize-pdf/main.mthds": "" } +} +``` + +`method` is exactly one of `{files}`, `{method_ref}`, `{method_id}`, as passed. Paths are relative to the **project root**, not to the workshop's working directory. `pipe` records what the call site was typed against — `Concept` single, `Concept[]` a list, `Concept?` optional — so refresh mode can tell a signature change from a body change. `sources` is empty for a `method_ref` / `method_id` source. Hashes are over the raw bytes: `shasum -a 256 ` / `sha256sum ` / `hashlib.sha256(path.read_bytes())`. + +### Step 8: Add the dependencies the generated code needs + +With the project's own package manager (read the lockfile): `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen). State what is added; interactive mode confirms first. + +### Step 9: Write the call site + +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. + +| Declared concept | TypeScript parameter | Python parameter | +|---|---|---| +| `native.Text` (or a refinement) | `string` | `str` | +| `native.Number` | `number` | `float` | +| `native.YesNo` | `boolean` | `bool` | +| `native.Date` | `string` (ISO 8601) | `str` (ISO 8601) | +| `native.Image`, `native.Document` | `{ url: string }` — an `http(s)` URL or a `pipelex-storage://` reference | `dict[str, Any]` with a `url` key | +| a structured concept, or a composite native (`Page`, `TextAndImages`, `JSON`) | the generated type from `types.ts` | the generated model from `models.py` | +| `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | +| not required | optional parameter (`?`) | `T \| None = None` | + +**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. + +### Step 10: Wire the offline drift gate + +- **TypeScript**: copy [references/codegen-check.mjs](references/codegen-check.mjs) **verbatim** to `scripts/codegen-check.mjs`, register `"codegen:check": "node scripts/codegen-check.mjs …"` in `package.json`, and **extend the project's existing aggregate gate** — a `check` / `ci` / `validate` / `verify` script, a Makefile `check` target, the lint or test step of an existing workflow — rather than inventing a new one. The script runs `@pipelex/sdk`'s `runCodegenCheck` over each directory, compares the sidecar's source hashes against the committed `.mthds` files, and exits `0` current / `1` drift or stale source / `2` no verdict. A project with no aggregate gate gets the script and one sentence in the report saying where to call it. +- **Python, `python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists (`pipelex codegen check`) needs the `pipelex` runtime a consumer deliberately does not have — do not add `pipelex` as a dependency to get a gate. The sidecar is still written (refresh mode and the editing skills' staleness notice read it), and the report says plainly: the tree is protected by its stamps and lock, but nothing in CI proves it current; refresh with this skill after every bundle edit. +- **Python, `python-structures`**: the project already depends on `pipelex`, so `pipelex codegen check ` (exit `0` / `1` / `2`) is wired into its existing gate. + +### Step 11: Verify + +Run the project's formatter **on the files you wrote only** — never on the generated tree; the step-5 exclusions are what make a later project-wide run safe — then its type checker, then the gate you installed. A failure in your own code is yours to fix before reporting; a failure inside the generated tree is reported, never patched. + +### Step 12: Report + +What was generated and where; the target and why; the call site's signature; what changed in the tooling config; how to refresh (this skill again after a bundle edit); for a `python-pydantic` project, that no offline drift check exists yet and refresh is the guard; for a `method_id` source, that the catalog is unversioned. Then the hand-off: `/pipelex-inputs` prepares inputs and offers a run. + +## Refresh mode + +Entered when the user asks to refresh, regenerate or update the types; when `/pipelex-edit` or `/pipelex-design` hand off after editing a bundle a sidecar names; or when step 1 finds a sidecar for the method. **Re-derive nothing the sidecar already records, regenerate in place, and leave alone everything the regeneration did not invalidate.** + +| Taken from disk | Re-derived | Left alone | +|---|---|---| +| the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | + +One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one that adopted their pattern, regenerates every method in one place, checks them in one place, and keeps its own sidecar. Writing this skill's tree beside that would leave two regeneration paths, two sidecar dialects and files the workshop never emits. So on such a project: + +- **place the method where the project keeps them** — `methods//main.mthds`, or the project's manifest form for a catalog or published method; +- **run the project's generator** — `make add-method METHOD=…` when the project has it and the method is remote, its `codegen` script or Makefile target otherwise; +- **write the call site the way the project's docs and existing methods do** (`docs/codegen.md`, `docs/add-method.md`, the existing actions or CLI commands), not the shape of step 9; +- **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); +- **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. + +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| a required tool is absent | STOP with the one-line MCP-connection message above | +| `status: "error"`, class `config` — including the codegen **403** feature gate | STOP, surface `hint` verbatim; never say "check your key" for a 403 | +| `status: "error"`, class `config`, `kind: "paywall"` | STOP, surface the plan-limit hint | +| `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | +| not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | +| `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | +| `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | +| success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | +| success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | +| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `mthds_list_methods` absent | integrate by id, address or files; never stop for it | + +## Reference + +- [references/typescript.md](references/typescript.md) — detecting a TypeScript project (build, package manager, generated root, Prettier / ESLint / Biome exclusions, `tsconfig` coverage, the aggregate gate, the call-site location), the call-site module and client helper templates, the `codegen:check` wiring, the harness a starter-derived project owns. +- [references/python.md](references/python.md) — the same for Python (import package, `python-pydantic` vs `python-structures`, uv / poetry / pipenv / pip, Ruff / Black / isort exclusions, pyright / mypy coverage, `__init__.py` and package data, the async module plus its sync wrapper, `pipelex codegen check` for the structures audience, the asymmetry sentence for everyone else). +- [references/codegen-check.mjs](references/codegen-check.mjs) — the offline gate copied verbatim into TypeScript projects. +- [MTHDS Language Reference](../shared/mthds-reference.md) — for reading a bundle's `main_pipe` and `output` declarations on the fallback path. diff --git a/pipelex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs new file mode 100644 index 0000000..ade01da --- /dev/null +++ b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs @@ -0,0 +1,166 @@ +#!/usr/bin/env node +// codegen-check.mjs — the offline drift gate for Pipelex-generated trees. +// +// Copied verbatim into a project by /pipelex-integrate. Run it from the project root +// with one generated directory per argument: +// +// node scripts/codegen-check.mjs src/generated/summarize-pdf src/generated/extract-entities +// +// For each directory it (1) runs @pipelex/sdk's runCodegenCheck over the stamped files +// against codegen.lock — pure hashing, no engine, no network, no API key — and (2) compares +// the SHA-256 recorded for each .mthds source in sources.json against the file on disk, so +// a bundle edited without a regeneration is caught as `stale-source`. +// +// Exit codes: 0 current · 1 drift or stale source · 2 no verdict (no lock, an unreadable +// file, a symlink in the tree). Precedence across directories: 2 > 1 > 0. +// +// It imports only Node builtins and @pipelex/sdk, and writes through process.stdout / +// process.stderr so a no-console lint rule stays quiet. When @pipelex/sdk ships this check +// as a command, replace this file with that one line. + +import { createHash } from "node:crypto"; +import { lstat, readdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import process from "node:process"; +import { CodegenLockError, isStampableArtifactPath, runCodegenCheck } from "@pipelex/sdk"; + +const EXIT_CURRENT = 0; +const EXIT_DRIFT = 1; +const EXIT_NO_VERDICT = 2; + +const LOCK_FILENAME = "codegen.lock"; +const SIDECAR_FILENAME = "sources.json"; +const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); + +const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); + +const out = (line) => process.stdout.write(`${line}\n`); +const err = (line) => process.stderr.write(`${line}\n`); + +/** Every regular file under `root`, as sorted forward-slash paths relative to it. Refuses symlinks. */ +async function walk(root, relative = "") { + const absolute = relative ? path.join(root, relative) : root; + const entries = await readdir(absolute, { withFileTypes: true }); + const paths = []; + for (const entry of entries) { + const rel = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isSymbolicLink()) { + throw new Error(`refusing to read through the symlink ${rel}`); + } + if (entry.isDirectory()) { + if (PRUNED_DIRECTORIES.has(entry.name)) continue; + paths.push(...(await walk(root, rel))); + } else if (entry.isFile()) { + paths.push(rel); + } + } + return paths.sort(); +} + +async function readStrict(filePath) { + return strictUtf8.decode(await readFile(filePath)); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** The lock check: { code, lines } — never throws. */ +async function checkTree(dir) { + let lockContent; + try { + lockContent = await readStrict(path.join(dir, LOCK_FILENAME)); + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.code === "ENOENT" ? "not found" : error.message}`] }; + } + + let files; + try { + const rootStat = await lstat(dir); + if (rootStat.isSymbolicLink()) throw new Error("the generated directory itself is a symlink"); + const stampable = (await walk(dir)).filter((rel) => isStampableArtifactPath(rel)); + files = []; + for (const rel of stampable) { + files.push({ path: rel, content: await readStrict(path.join(dir, rel)) }); + } + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${error.message}`] }; + } + + try { + const report = await runCodegenCheck({ lockContent, files }); + if (report.isCurrent) { + return { + code: EXIT_CURRENT, + lines: [` ${files.length} artifact(s) current (crate ${report.crateFingerprint.slice(0, 12)}, engine ${report.engineVersion})`], + }; + } + return { code: EXIT_DRIFT, lines: report.drifts.map((drift) => ` ${drift.category}: ${drift.path} — ${drift.detail}`) }; + } catch (error) { + if (error instanceof CodegenLockError) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.message}`] }; + } + throw error; + } +} + +/** The sidecar check against the .mthds sources, relative to the project root: { code, lines }. */ +async function checkSources(dir) { + let sidecar; + try { + sidecar = JSON.parse(await readStrict(path.join(dir, SIDECAR_FILENAME))); + } catch (error) { + if (error.code === "ENOENT") { + return { code: EXIT_CURRENT, lines: [` no ${SIDECAR_FILENAME} — source staleness not checked`] }; + } + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; + } + + const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + const lines = []; + for (const [source, recorded] of Object.entries(sources).sort()) { + let onDisk; + try { + onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); + } catch (error) { + lines.push(` stale-source: ${source} — recorded as a source but ${error.code === "ENOENT" ? "no longer on disk" : `unreadable (${error.message})`}`); + continue; + } + if (onDisk !== recorded) { + lines.push(` stale-source: ${source} — edited since the types were generated`); + } + } + return { code: lines.length ? EXIT_DRIFT : EXIT_CURRENT, lines }; +} + +function worse(a, b) { + // Precedence: no verdict > drift > current. + if (a === EXIT_NO_VERDICT || b === EXIT_NO_VERDICT) return EXIT_NO_VERDICT; + if (a === EXIT_DRIFT || b === EXIT_DRIFT) return EXIT_DRIFT; + return EXIT_CURRENT; +} + +async function main(argv) { + const dirs = argv.slice(2); + if (dirs.length === 0) { + err("usage: node scripts/codegen-check.mjs [ ...]"); + return EXIT_NO_VERDICT; + } + + let exitCode = EXIT_CURRENT; + for (const dir of dirs) { + out(`${dir}`); + const tree = await checkTree(dir); + const sources = tree.code === EXIT_NO_VERDICT ? { code: EXIT_CURRENT, lines: [] } : await checkSources(dir); + const code = worse(tree.code, sources.code); + const write = code === EXIT_CURRENT ? out : err; + for (const line of [...tree.lines, ...sources.lines]) write(line); + if (code === EXIT_DRIFT) err(" Run /pipelex-integrate to refresh the generated types."); + exitCode = worse(exitCode, code); + } + + out(`\ncodegen-check: ${exitCode === EXIT_CURRENT ? "current" : exitCode === EXIT_DRIFT ? "drift" : "no verdict"}`); + return exitCode; +} + +process.exit(await main(process.argv)); diff --git a/pipelex/skills/pipelex-integrate/references/python.md b/pipelex/skills/pipelex-integrate/references/python.md new file mode 100644 index 0000000..cc64358 --- /dev/null +++ b/pipelex/skills/pipelex-integrate/references/python.md @@ -0,0 +1,102 @@ +# Integrating into a Python project + +Companion to `/pipelex-integrate` for a project that has a `pyproject.toml` (or `setup.py` / `requirements.txt`). The SDK facts were checked against `pipelex-sdk` as of 2026-09 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Audience → target** | `pipelex` **not** among the dependencies (`[project].dependencies`, `requirements*.txt`) → `python-pydantic`, no question: `python-structures` imports the runtime and would not even load; `pipelex` present → `python-structures` if `@pipe_func` or `from pipelex.core.stuffs.structured_content import StructuredContent` appears in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | +| **Import package** | the package `[project].name` names (dashes → underscores), or the setuptools `packages` list, or the top-level directory holding `__init__.py` (`src//` in a src layout) | ask | +| **Package manager** | the lockfile: `uv.lock` → uv (`uv add`), `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none → `pip install` into the active environment, stated | +| **Generated root** | an existing directory already holding generated code → beside it; else `/generated/` | ask | +| **Formatter and linter** | `[tool.ruff]` → add the generated directory to `exclude` (or `extend-exclude`); `[tool.black]` → `extend-exclude`; `[tool.isort]` → `skip` / `extend_skip_glob` | a tool this table does not name: read its config, add the equivalent, say so | +| **Type checker still covers the tree** | `[tool.pyright]` `include` / `exclude`; `[tool.mypy]` `packages` / `files` / `exclude` | an exclusion that would drop it is **not** added; generated code stays type-checked | +| **Packaged for distribution** | a `[build-system]` table and an import package | the bundle goes **inside** the package (`/methods//main.mthds`) and `*.mthds` plus `codegen.lock` are registered as package data, as the Python starter does — a wheel that ships the call site must ship the bundle it loads at call time; an unpackaged app keeps `methods//` at the project root | +| **Aggregate gate** | a Makefile `check` target; `.github/workflows/*.yml`; `.pre-commit-config.yaml`; a `nox`/`tox` session | none: nothing to wire for `python-pydantic` anyway (below) | +| **Call-site location** | the project's existing service / client layer (`services/`, `clients/`, `api/`) → beside it | `/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | reported and un-ignored on confirmation | +| **Owns a codegen harness** | a `codegen` Makefile target; `docs/codegen.md`; `/generated/` beside `/methods/` | the harness section below | + +Why the Ruff exclusion is not optional: `ruff format` rewrites the generated bytes, breaks every stamp, and makes the drift check report the tree as hand-edited. The exclusion goes in before generation, so the first project-wide format after it is already safe. + +## The generated tree + +`mthds_codegen` with `target: "python-pydantic"` and `output_dir: "/generated/"` writes: + +- `models.py` — stamped; `from __future__ import annotations`, `from pydantic import BaseModel, Field`; one plain `BaseModel` subclass per concept, natives included, with `Field(..., description=…)`; no Pipelex import. Non-required fields are `T | None = None`. +- `codegen.lock` — TOML with the crate fingerprint, the engine version and one `[[artifacts]]` entry per stamped file. + +With `target: "python-structures"` the file is `structures.py` — `StructuredContent` subclasses from `pipelex.core.stuffs.structured_content`, natives not re-emitted — for a Pipelex host whose `@pipe_func` functions return them. + +**The skill creates the `__init__.py` files** the tree needs to be importable: `/generated/__init__.py` (once per project, a one-line docstring) and `/generated//__init__.py` (empty). Codegen never emits them and they are never artifacts: they carry no stamp, so the writer does not touch them and the check does not count them as orphans. When the project is packaged, list the generated subpackages where the project lists its packages and add `codegen.lock` as package data, as the Python starter's `pyproject.toml` does. Beside the lock the skill writes `sources.json`, unstamped. + +## The call site + +One module per method. `summarize_pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle inside the package: + +```python +# /pipelex/summarize_pdf.py +"""Run the summarize_pdf method through the hosted Pipelex API. + +`document` is an http(s) URL or a pipelex-storage:// reference, as ``{"url": ...}``. For a +local file or bytes, call ``client.prepare_inputs(files=[...], inputs=...)`` first: it uploads +and rewrites the value. It treats any string it does not recognise as data:, http(s):// or +pipelex-storage:// as a LOCAL FILE PATH it reads and uploads, so gate schemes before handing +values from an untrusted caller to it. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +from pipelex_sdk.client import PipelexAPIClient + +from .generated.summarize_pdf.models import DocumentSummary + +PIPE_CODE = "summarize_pdf" +BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" + + +async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + inputs: dict[str, Any] = {"document": document} + if context is not None: + inputs["context"] = context + async with PipelexAPIClient() as client: + results = await client.start_and_wait( + pipe_code=PIPE_CODE, + mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + inputs=inputs, + ) + return DocumentSummary.model_validate(results.main_stuff) + + +def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + """The blocking wrapper for a synchronous caller.""" + return asyncio.run(summarize_pdf(document=document, context=context)) +``` + +Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: + +- **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. +- **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. + +Facts the module leans on: `PipelexAPIClient()` reads `PIPELEX_API_KEY` and `PIPELEX_BASE_URL` (default `https://api.pipelex.com`) itself; it is async-only and used as an async context manager; there is **no barrel** in `pipelex_sdk` by design, so every import is a full module path (`pipelex_sdk.client`, `pipelex_sdk.runs`, `pipelex_sdk.errors`); `start_and_wait(...)` takes the durable path on the hosted API (default 2 s poll, 20 min budget, `wait_options=WaitForResultOptions(...)` to change them) and falls back to blocking on a bare runner, returning `RunResults` whose `main_stuff` is always present for a completed run. Let the typed errors from `pipelex_sdk.errors` propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own handling only where it already wraps its other clients. + +Parameters are keyword-only, typed from the signature: `str` for Text and Date (ISO 8601), `float` for Number, `bool` for YesNo, `dict[str, Any]` with a `url` key for Image and Document, the generated model for a structured concept or a composite native, `list[T]` for a list, `T | None = None` for a non-required input. Names are the pipe's input names as declared. A project that already constructs a `PipelexAPIClient` somewhere gets that construction reused instead of a fresh `async with` per call; an async-native project (FastAPI, an existing async codebase) gets the async function alone, without the `_sync` wrapper. + +## The drift gate — an honest asymmetry + +- **`python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists — `pipelex codegen check ` — lives in the `pipelex` runtime, which a hosted-API consumer deliberately does not install. Do not add `pipelex` as a dependency to get a gate; that reverses the decision the target expresses. Write the sidecar anyway (refresh mode and `/pipelex-edit`'s staleness notice read it) and put this sentence in the report: *the generated tree is protected by its stamps and lock, but nothing in CI proves it current; run `/pipelex-integrate` again after every bundle edit.* When the Python SDK gains the check, this section becomes one line. +- **`python-structures`**: the project already depends on `pipelex`, so wire `pipelex codegen check /generated/` into its existing gate (a Makefile `check` target, a workflow step). Exit codes: `0` current, `1` drift, `2` no lock or an unreadable lock. Offline: no engine boot, no network, no key. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-python` has `make codegen` and `make codegen-check`, both shelling out to a `pipelex` CLI the starter deliberately does not depend on (`PIPELEX=` in its Makefile), methods under `/methods//`, trees under `/generated//` with hand-committed `__init__.py` files, and typed CLI commands under its mode sub-packages. On such a project: + +- place the bundle under `/methods//main.mthds` and add the two `codegen` / `codegen-check` lines the Makefile pattern uses for its other methods; +- run `make codegen` when a `pipelex` CLI is reachable; when it is not, call `mthds_codegen` with `output_dir` set to `/generated/` — the harness's own layout, byte-identical (same engine, same stamps, same lock, no sidecar in this starter) — and say `make codegen` is the refresh once `PIPELEX=` points at an install; +- write the CLI command the way `docs/cli-architecture.md` and the existing commands do, importing the generated model from `.generated..models`; +- verify with `make agent-check` and `make agent-test`; no `sources.json` of this skill's shape and no second generated directory. diff --git a/pipelex/skills/pipelex-integrate/references/typescript.md b/pipelex/skills/pipelex-integrate/references/typescript.md new file mode 100644 index 0000000..bce5bfa --- /dev/null +++ b/pipelex/skills/pipelex-integrate/references/typescript.md @@ -0,0 +1,118 @@ +# Integrating into a TypeScript project + +Companion to `/pipelex-integrate` for a project that has a `package.json`. Everything here follows the shape the Pipelex JS starter converged on; the SDK facts were checked against `@pipelex/sdk` 0.17 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **TypeScript build** | a `tsconfig.json`; a `typescript` dev dependency; a bundler or runtime that strips types (Next.js, Vite, tsx, Bun, Deno) | a plain JavaScript project is asked — `types.ts` needs a TypeScript build; the alternative is `python-pydantic`'s sibling in this language, which does not exist yet, so the honest answer is "add TypeScript or skip codegen" | +| **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock` / `bun.lockb` → bun | none → npm, stated | +| **Generated root** | an existing directory already holding generated code (`generated/`, `gen/`, `__generated__/`) → beside it; else `src/generated/` when `src/` exists; else `generated/` | ask | +| **Formatter** | `.prettierrc*` or a `prettier` dev dependency → add `src/generated/` to `.prettierignore` (create the file if absent); `biome.json*` → the files-ignore key its version uses (`files.ignore` before Biome 2, `files.includes` with a `!` negation from Biome 2 on) | a formatter this table does not name: read its config, add the equivalent, say so | +| **Linter** | `eslint.config.*` (flat config) → add `"src/generated/**"` to `globalIgnores([...])` or an `{ ignores: [...] }` entry; legacy `.eslintrc*` → `.eslintignore`; Biome as above | same | +| **Type checker still covers the tree** | `tsconfig.json` `include` / `exclude` — the generated directory must stay inside `include` and outside `exclude` | an exclusion that would drop it is **not** added; the report says the typecheck is the check that covers generated code | +| **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | +| **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | + +Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. + +## The generated tree + +`mthds_codegen` with `target: "ts-zod"` and `output_dir: "src/generated/"` writes: + +- `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. +- `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. +- `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. + +Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. + +## The call site + +One module per method. `summarize-pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle: + +```ts +// src/pipelex/summarizePdf.ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import type { RunResults } from "@pipelex/sdk"; +import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; +import type { DocumentSummary } from "../generated/summarize-pdf/types"; +import { getPipelexClient } from "./client"; + +const PIPE_CODE = "summarize_pdf"; +const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); + +export interface SummarizePdfInputs { + /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, + * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * it uploads and rewrites the value. Note: prepareInputs treats any string it does not + * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and + * uploads, so a public endpoint must gate schemes before handing values to it. */ + document: { url: string }; + context?: string; +} + +export async function summarizePdf(inputs: SummarizePdfInputs): Promise { + const bundle = await readFile(BUNDLE_PATH, "utf8"); + const results: RunResults = await getPipelexClient().startAndWaitForResult({ + pipe_code: PIPE_CODE, + mthds_contents: [bundle], + inputs, + }); + return parseDocumentSummary(results.main_stuff); +} +``` + +Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: + +- **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. +- **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. + +The shared client helper, created once per project and reused by every method (if the project already constructs a `PipelexApiClient` somewhere, import that instead): + +```ts +// src/pipelex/client.ts +import { PipelexApiClient } from "@pipelex/sdk"; + +let client: PipelexApiClient | undefined; + +/** Reads PIPELEX_API_KEY and PIPELEX_BASE_URL (default https://api.pipelex.com) from the environment. */ +export function getPipelexClient(): PipelexApiClient { + client ??= new PipelexApiClient(); + return client; +} +``` + +`startAndWaitForResult(options, pollOptions?)` takes the durable path (`start` + poll, default 2 s interval, 20 min budget) on the hosted API and falls back to a blocking `execute` on a bare runner; it returns `RunResults` whose `main_stuff` is always present for a completed run. Let the SDK's typed errors propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own error handling only where it already wraps its other clients. A server-side framework seam (a Next.js Server Action, an Express handler) is the caller's; this module is framework-free. + +Parameter types from the signature: `string` for Text and Date (ISO 8601), `number` for Number, `boolean` for YesNo, `{ url: string }` for Image and Document, the generated type for a structured concept or a composite native, `T[]` for a list, `?` for a non-required input. Key names are the pipe's input names as declared, snake_case included. + +## The offline gate + +Copy `references/codegen-check.mjs` verbatim to `scripts/codegen-check.mjs`. It imports only Node builtins and `@pipelex/sdk`, runs under plain `node` whatever the project's TypeScript build, and prints through `process.stdout` / `process.stderr` so a `no-console` rule does not fire. Register it and extend the existing gate: + +```json +{ + "scripts": { + "codegen:check": "node scripts/codegen-check.mjs src/generated/summarize-pdf", + "check": "npm run lint && npm run typecheck && npm run codegen:check" + } +} +``` + +Add every method's directory to the `codegen:check` line as it is integrated. Exit codes: `0` current, `1` drift or stale source, `2` no verdict (no lock, an unreadable file, a symlink in the tree). It runs from the project root because `sources.json` records source paths relative to it. When `@pipelex/sdk` ships this as a command of its own, the script is replaced by that one line. + +## The Node-only boundary + +`readFile`, `node:path` and `process.cwd()` in the call site are server-side facts. In a framework with a client/server split (Next.js, Remix, SvelteKit), the module belongs on the server side — a Server Action, a route handler, a loader — and the JS starter marks such modules with `import "server-only"`. Never import it from a component that renders in the browser: the API key would leave the server. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js`, or one that copied its codegen kit, has: `npm run codegen` (regenerates every `methods/*` tree through the hosted `/v1/codegen`, writes `contracts.ts` and `sources.json` with a `derived` map), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (semantic, keyed), and `make add-method METHOD=` (manifest, tree, action trio, narrower, form, tab — one shot, never overwrites). On such a project: + +- a local bundle goes under `methods//main.mthds`, then `npm run codegen`; the fan-out follows `docs/codegen.md` and the existing actions under `src/actions/` and narrowers under `src/types/`; +- a catalog or published method goes through `make add-method`; +- the verification is `make check`; the refresh is `npm run codegen`; no `sources.json` of this skill's shape, no `scripts/codegen-check.mjs`, no second generated directory. diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md new file mode 100644 index 0000000..f9a1991 --- /dev/null +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -0,0 +1,165 @@ +--- +name: pipelex-scaffold +description: Start a new project that will call MTHDS methods through Pipelex, in TypeScript or Python — from one of the Pipelex starter templates or from the ecosystem's own initializer — and hand it to /pipelex-integrate. Use when the user says "start a new project with Pipelex", "I have a method and need an app around it", "create a Next.js app that runs my method", "set up a Pipelex project from scratch", "new Python CLI for this method", "which starter should I use", "bootstrap a Pipelex project", or wants a codebase where none exists yet. Also use when the user is standing in a freshly cloned pipelex-starter-js or pipelex-starter-python that has not been renamed yet — this skill runs the template's own bootstrap for them. Not for adding Pipelex to code that already exists: that is /pipelex-integrate. +allowed-tools: + - Bash + - Read + - Write + - Edit + - Grep + - Glob + +--- + +# Scaffold a project for Pipelex methods + +Give a user who has no project yet a project that is ready for `/pipelex-integrate`. This skill has exactly two branches and carries no templates of its own: + +- **One of the Pipelex starters** when the user wants the opinionated shape: `pipelex-starter-js` for a web app whose forms are rendered from the methods' own contracts, `pipelex-starter-python` for a CLI or service that runs methods in the three execution modes. You acquire the template, commit it once as it came, then run the clone's **own** `bootstrap` skill — the rename logic lives in the starters and is never reimplemented here. +- **The ecosystem's own initializer** when the user wants their framework or a minimal project: `uv init --package`, `npm create next-app@latest`, `django-admin startproject`, whatever the framework documents. You run it; you never assemble a project by hand. + +Both branches end the same way: an env file that follows the starters' convention, one pristine commit that makes everything after it reviewable, and the hand-off — to `/pipelex-integrate` when a method exists, to `/pipelex-design` first when none does. + +**What this skill is not.** Not a template engine (no cookiecutter, no copier, no framework matrix of its own), not a bootstrap (the starters own theirs), not a runner or a dev-server launcher, not a deployer. It needs no MCP tool and no API key: git, the starters' scripts and the ecosystem's initializers are all it uses. + +## Choosing the branch + +A cheap, reliable signal decides; an inconclusive one asks one question; nothing is guessed twice. + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | +| **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | + +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. + +[references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. + +## Mode + +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. + +## Branch A — one of the starters + +### Step 1: Prerequisites + +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. + +- **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. +- **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). +- **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. + +### Step 2: Acquire the template + +**Local, the default.** Clone shallow, read the template's identity, then detach from it: + +```bash +git clone --depth 1 https://github.com/Pipelex/.git +git -C rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf /.git && git -C init -b main +``` + +The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. + +**GitHub, on request.** When the user asked for a repository on GitHub: + +```bash +gh repo create / --template Pipelex/ --private --clone +``` + +Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. + +Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. + +### Step 3: Commit the pristine template — exactly once + +```bash +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. + +### Step 4: Run the clone's own bootstrap + +Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written. The project's skills are not loaded in this session — it began elsewhere — so read the file; do not look for a `/bootstrap` command. Run every command it gives from inside the project directory (`cd && …`, or `-C `), because that skill assumes it is standing in the repo root. + +Feed it what the conversation already holds — the project name, title, description, author, repository URL, license — so that it asks once, consolidated, for whatever is left, exactly as its own Step 2 says. It dry-runs, previews, runs, re-syncs the lock file, runs the project's own checks (`make all` on JS; `make agent-check` and `make agent-test` on Python), and removes itself. Its rules stand unchanged: it never commits, its edits stay uncommitted for the user's review (the Python renames are staged by `git mv`, which its skill explains), and a red check is fixed, never skipped. **Add nothing to that procedure and reimplement none of it.** If the clone carries no bootstrap skill — a future template dropped it — follow the README's "manual equivalent" list and say that the template changed. + +### Step 5: The env file + +```bash +cp /.env.example /.env.local # JS: Next.js reads .env.local +cp /.env.example /.env # Python: python-dotenv reads .env +``` + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. + +### Step 6: Verify and hand off + +The bootstrap's own checks are the verification; do not start `make dev`. Write the report (below), then hand the user's method to `/pipelex-integrate`, which recognizes the starter's codegen harness (`npm run codegen`, `make codegen`, `make add-method`) and defers to it rather than writing a second one. + +## Branch B — the ecosystem's initializer + +### Step 1: Prerequisites + +As in branch A, for the language chosen. + +### Step 2: Run the initializer — never assemble by hand + +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. + +Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. + +### Step 3: Version control and the pristine commit + +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: + +```bash +git -C add -A && git -C commit -m "Scaffold project" +``` + +### Step 4: The env file + +Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: + +``` +PIPELEX_BASE_URL=https://api.pipelex.com +PIPELEX_API_KEY= +``` + +### Step 5: Hand off + +Add **no** SDK dependency and create **no** empty `methods/` directory: `/pipelex-integrate` adds `@pipelex/sdk` or `pipelex-sdk` when it writes the first call site, and creates `methods//` when it places the first bundle. A project with nothing to integrate yet has nothing Pipelex-shaped in it beyond the env convention, and that is correct. + +## The report + +Say, in this order: what was created and where; which template or initializer it came from, at which version and SHA; that this skill made exactly one commit and what it holds; what the bootstrap changed and that those changes are uncommitted for review, in the bootstrap's own words (branch A); which env file was written and whether the key was filled from the environment or left for the user; the demos the starter still carries and where the README's removal checklist is (branch A); and the hand-off. + +Two lines are easy to forget and matter: + +- **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd && claude` is how they arrive. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it with `/pipelex-integrate`; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | +| The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | +| The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | +| An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | +| `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | + +## Reference + +- [references/starters.md](references/starters.md) — the two starters side by side: what each brings, its prerequisite floors, the acquisition commands, its env file, its bootstrap, its demos and their removal checklist, and the codegen harness `/pipelex-integrate` will find. +- [references/initializers.md](references/initializers.md) — per language, the minimal default and the common frameworks' non-interactive initializers, whether each runs `git init`, and where the import package or `src/` root lands. +- `/pipelex-integrate` — the skill this one hands every project to. diff --git a/pipelex/skills/pipelex-scaffold/references/initializers.md b/pipelex/skills/pipelex-scaffold/references/initializers.md new file mode 100644 index 0000000..98b683f --- /dev/null +++ b/pipelex/skills/pipelex-scaffold/references/initializers.md @@ -0,0 +1,38 @@ +# Ecosystem initializers + +Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. + +After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +## Python + +| Want | Command | `git init`? | Where the import package lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | +| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | + +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. + +## TypeScript / JavaScript + +| Want | Command | `git init`? | Where `src/` lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | +| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Node library | the minimal recipe above | no | `src/` | +| pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | + +`npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. + +## What every branch-B project shares afterwards + +- One commit, the pristine scaffold, so the user's first real change is a clean diff. +- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/pipelex/skills/pipelex-scaffold/references/starters.md b/pipelex/skills/pipelex-scaffold/references/starters.md new file mode 100644 index 0000000..52ccdc3 --- /dev/null +++ b/pipelex/skills/pipelex-scaffold/references/starters.md @@ -0,0 +1,59 @@ +# The two Pipelex starters + +Both are GitHub **template repositories** under the `Pipelex` organization. Each is a real, CI-tested application against the hosted Pipelex API, not a parameterized template: the identity you see in a fresh clone (`pipelex-starter-js` / `Pipelex Starter`, or `piper` / `Piper`) is a placeholder that the starter's own `bootstrap` skill rewrites. Read the clone's `README.md` after acquiring it — the sections named below are where the details live, and they move as the starters evolve. + +## Side by side + +| | `pipelex-starter-js` | `pipelex-starter-python` | +|---|---|---| +| **Shape** | Next.js (App Router), React, TypeScript strict, Tailwind; a web app with one tab per method whose input form is rendered from the method's own contract by `@pipelex/mthds-form` | A Typer CLI with one command per method, printing JSON on stdout and a cost report on stderr; three execution modes (`blocking`, `attended`, `detached`) as separate sub-packages | +| **Pick it when** | people will use the methods in a browser: forms, uploads, live run status | the methods run from a terminal, a script, a batch job or a service, and the user wants Python | +| **SDK** | `@pipelex/sdk` | `pipelex-sdk` (import package `pipelex_sdk`) — the `pipelex` runtime is **not** a dependency | +| **Methods live in** | `methods//main.mthds`, or `methods//method.json` for a method that lives elsewhere (a catalog id or a published address) | `/methods//main.mthds` | +| **Generated types** | `src/generated//` — `types.ts`, `binder.ts`, `contracts.ts`, `codegen.lock`, `sources.json` | `/generated//` — `models.py`, `codegen.lock` | +| **Codegen harness** | `npm run codegen` (keyed, dev), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (keyed, semantic); `make add-method METHOD=` scaffolds a remote method end to end | `make codegen` / `make codegen-check` — **both shell out to a `pipelex` CLI the starter does not depend on** (`PIPELEX=` in the Makefile); `/pipelex-integrate` knows this and writes into the same layout when that CLI is absent | +| **Toolchain floor** | Node ≥ the `engines.node` field of `package.json` (22.12 at writing); `npm` | `uv`; a Python inside `requires-python` of `pyproject.toml` (3.11–3.14 at writing) | +| **Env file** | `.env.local`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY`, `NEXT_PUBLIC_EXECUTION_MODE` | `.env`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY` | +| **Checks the bootstrap runs** | `npm install --package-lock-only`, then `make all` (lint, format check, typecheck, unit tests, build) | `make li` (lock + sync), then `make agent-check` and `make agent-test` | +| **Agent-facing files** | `CLAUDE.md`, `AGENTS.md`; skills `bootstrap`, `release`, `bump-sdk`, `bump-mthds-form` | `CLAUDE.md`; skills `bootstrap`, `release` | +| **Docs worth reading after bootstrap** | `docs/codegen.md`, `docs/add-method.md`, `docs/input-form.md`, `docs/adopt-in-an-existing-project.md`; README → "Swap in your own pipeline" and "Remove an example" | `docs/codegen.md`, `docs/cli-architecture.md`; README → the per-command sections | +| **Demos it carries** | several demo methods, one tab each; keep them as references or strip them with the README's "Remove an example" checklist | several demo methods, one CLI command each; keep them as references or remove the command and its method directory together | + +## Acquisition + +Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): + +```bash +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git -C rev-parse HEAD +rm -rf /.git && git -C init -b main +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. + +GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): + +```bash +gh auth status +gh repo create / --template Pipelex/pipelex-starter-js --private --clone +gh repo create / --template Pipelex/pipelex-starter-python --private --clone +``` + +## The bootstrap you delegate to + +Both starters carry `.claude/skills/bootstrap/SKILL.md` with a bundled script (`scripts/bootstrap.mjs` / `scripts/bootstrap.py`). Read the file in the clone and follow it; the shape is the same on both: + +1. **Preflight** — confirms the identity is still the template's (`package.json` name `pipelex-starter-js`; `pyproject.toml` `name = "piper"`), notes a dirty tree, and on JS makes sure `node_modules/` exists (`make install`). +2. **Collect** — the package name (kebab on JS, underscores on Python, everything else derives from it), a display title, a one-line description; optionally author name **and** email (never one without the other), the repository URL, and the license (MIT kept, proprietary, or another SPDX id; the copyright holder and year). Pass what the conversation already holds so it asks once for the rest. +3. **Dry run** — the script with `--dry-run` prints the plan; the user confirms. +4. **Run** — the same command without `--dry-run`; on Python the package directory is renamed with `git mv`, which is why the pristine commit must exist first. `--clean` strips the template-only prose; keep it unless the user wants the template charter kept. +5. **Verify** — the lock file is re-synced and the project's own checks run; red is fixed, not skipped. +6. **Self-removal** — `rm -rf .claude/skills/bootstrap`, unstaged like everything else; the user reviews with `git status` and `git diff` and commits when ready. + +The starter's rules are yours while you run it: never commit on the user's behalf, always dry-run first, never touch `.github/` or the `release` skill's logic. + +## What `/pipelex-integrate` finds afterwards + +A bootstrapped starter is a project that **owns a codegen harness**, and `/pipelex-integrate` defers to it: on JS it drops a bundle under `methods//` and runs `npm run codegen`, or runs `make add-method METHOD=…` for a catalog or published method, then follows `docs/codegen.md` and the existing actions for the fan-out; on Python it places the bundle under `/methods//` and runs `make codegen` when a `pipelex` CLI is available, writing into `/generated//` through the Pipelex workshop when it is not. It never writes a second generated layout beside the starter's own. diff --git a/skills/pipelex-integrate/references/codegen-check.mjs b/skills/pipelex-integrate/references/codegen-check.mjs new file mode 100644 index 0000000..ade01da --- /dev/null +++ b/skills/pipelex-integrate/references/codegen-check.mjs @@ -0,0 +1,166 @@ +#!/usr/bin/env node +// codegen-check.mjs — the offline drift gate for Pipelex-generated trees. +// +// Copied verbatim into a project by /pipelex-integrate. Run it from the project root +// with one generated directory per argument: +// +// node scripts/codegen-check.mjs src/generated/summarize-pdf src/generated/extract-entities +// +// For each directory it (1) runs @pipelex/sdk's runCodegenCheck over the stamped files +// against codegen.lock — pure hashing, no engine, no network, no API key — and (2) compares +// the SHA-256 recorded for each .mthds source in sources.json against the file on disk, so +// a bundle edited without a regeneration is caught as `stale-source`. +// +// Exit codes: 0 current · 1 drift or stale source · 2 no verdict (no lock, an unreadable +// file, a symlink in the tree). Precedence across directories: 2 > 1 > 0. +// +// It imports only Node builtins and @pipelex/sdk, and writes through process.stdout / +// process.stderr so a no-console lint rule stays quiet. When @pipelex/sdk ships this check +// as a command, replace this file with that one line. + +import { createHash } from "node:crypto"; +import { lstat, readdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import process from "node:process"; +import { CodegenLockError, isStampableArtifactPath, runCodegenCheck } from "@pipelex/sdk"; + +const EXIT_CURRENT = 0; +const EXIT_DRIFT = 1; +const EXIT_NO_VERDICT = 2; + +const LOCK_FILENAME = "codegen.lock"; +const SIDECAR_FILENAME = "sources.json"; +const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); + +const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); + +const out = (line) => process.stdout.write(`${line}\n`); +const err = (line) => process.stderr.write(`${line}\n`); + +/** Every regular file under `root`, as sorted forward-slash paths relative to it. Refuses symlinks. */ +async function walk(root, relative = "") { + const absolute = relative ? path.join(root, relative) : root; + const entries = await readdir(absolute, { withFileTypes: true }); + const paths = []; + for (const entry of entries) { + const rel = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isSymbolicLink()) { + throw new Error(`refusing to read through the symlink ${rel}`); + } + if (entry.isDirectory()) { + if (PRUNED_DIRECTORIES.has(entry.name)) continue; + paths.push(...(await walk(root, rel))); + } else if (entry.isFile()) { + paths.push(rel); + } + } + return paths.sort(); +} + +async function readStrict(filePath) { + return strictUtf8.decode(await readFile(filePath)); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** The lock check: { code, lines } — never throws. */ +async function checkTree(dir) { + let lockContent; + try { + lockContent = await readStrict(path.join(dir, LOCK_FILENAME)); + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.code === "ENOENT" ? "not found" : error.message}`] }; + } + + let files; + try { + const rootStat = await lstat(dir); + if (rootStat.isSymbolicLink()) throw new Error("the generated directory itself is a symlink"); + const stampable = (await walk(dir)).filter((rel) => isStampableArtifactPath(rel)); + files = []; + for (const rel of stampable) { + files.push({ path: rel, content: await readStrict(path.join(dir, rel)) }); + } + } catch (error) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${error.message}`] }; + } + + try { + const report = await runCodegenCheck({ lockContent, files }); + if (report.isCurrent) { + return { + code: EXIT_CURRENT, + lines: [` ${files.length} artifact(s) current (crate ${report.crateFingerprint.slice(0, 12)}, engine ${report.engineVersion})`], + }; + } + return { code: EXIT_DRIFT, lines: report.drifts.map((drift) => ` ${drift.category}: ${drift.path} — ${drift.detail}`) }; + } catch (error) { + if (error instanceof CodegenLockError) { + return { code: EXIT_NO_VERDICT, lines: [` no verdict: ${LOCK_FILENAME} — ${error.message}`] }; + } + throw error; + } +} + +/** The sidecar check against the .mthds sources, relative to the project root: { code, lines }. */ +async function checkSources(dir) { + let sidecar; + try { + sidecar = JSON.parse(await readStrict(path.join(dir, SIDECAR_FILENAME))); + } catch (error) { + if (error.code === "ENOENT") { + return { code: EXIT_CURRENT, lines: [` no ${SIDECAR_FILENAME} — source staleness not checked`] }; + } + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; + } + + const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + const lines = []; + for (const [source, recorded] of Object.entries(sources).sort()) { + let onDisk; + try { + onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); + } catch (error) { + lines.push(` stale-source: ${source} — recorded as a source but ${error.code === "ENOENT" ? "no longer on disk" : `unreadable (${error.message})`}`); + continue; + } + if (onDisk !== recorded) { + lines.push(` stale-source: ${source} — edited since the types were generated`); + } + } + return { code: lines.length ? EXIT_DRIFT : EXIT_CURRENT, lines }; +} + +function worse(a, b) { + // Precedence: no verdict > drift > current. + if (a === EXIT_NO_VERDICT || b === EXIT_NO_VERDICT) return EXIT_NO_VERDICT; + if (a === EXIT_DRIFT || b === EXIT_DRIFT) return EXIT_DRIFT; + return EXIT_CURRENT; +} + +async function main(argv) { + const dirs = argv.slice(2); + if (dirs.length === 0) { + err("usage: node scripts/codegen-check.mjs [ ...]"); + return EXIT_NO_VERDICT; + } + + let exitCode = EXIT_CURRENT; + for (const dir of dirs) { + out(`${dir}`); + const tree = await checkTree(dir); + const sources = tree.code === EXIT_NO_VERDICT ? { code: EXIT_CURRENT, lines: [] } : await checkSources(dir); + const code = worse(tree.code, sources.code); + const write = code === EXIT_CURRENT ? out : err; + for (const line of [...tree.lines, ...sources.lines]) write(line); + if (code === EXIT_DRIFT) err(" Run /pipelex-integrate to refresh the generated types."); + exitCode = worse(exitCode, code); + } + + out(`\ncodegen-check: ${exitCode === EXIT_CURRENT ? "current" : exitCode === EXIT_DRIFT ? "drift" : "no verdict"}`); + return exitCode; +} + +process.exit(await main(process.argv)); diff --git a/skills/pipelex-integrate/references/python.md b/skills/pipelex-integrate/references/python.md new file mode 100644 index 0000000..cc64358 --- /dev/null +++ b/skills/pipelex-integrate/references/python.md @@ -0,0 +1,102 @@ +# Integrating into a Python project + +Companion to `/pipelex-integrate` for a project that has a `pyproject.toml` (or `setup.py` / `requirements.txt`). The SDK facts were checked against `pipelex-sdk` as of 2026-09 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Audience → target** | `pipelex` **not** among the dependencies (`[project].dependencies`, `requirements*.txt`) → `python-pydantic`, no question: `python-structures` imports the runtime and would not even load; `pipelex` present → `python-structures` if `@pipe_func` or `from pipelex.core.stuffs.structured_content import StructuredContent` appears in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | +| **Import package** | the package `[project].name` names (dashes → underscores), or the setuptools `packages` list, or the top-level directory holding `__init__.py` (`src//` in a src layout) | ask | +| **Package manager** | the lockfile: `uv.lock` → uv (`uv add`), `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none → `pip install` into the active environment, stated | +| **Generated root** | an existing directory already holding generated code → beside it; else `/generated/` | ask | +| **Formatter and linter** | `[tool.ruff]` → add the generated directory to `exclude` (or `extend-exclude`); `[tool.black]` → `extend-exclude`; `[tool.isort]` → `skip` / `extend_skip_glob` | a tool this table does not name: read its config, add the equivalent, say so | +| **Type checker still covers the tree** | `[tool.pyright]` `include` / `exclude`; `[tool.mypy]` `packages` / `files` / `exclude` | an exclusion that would drop it is **not** added; generated code stays type-checked | +| **Packaged for distribution** | a `[build-system]` table and an import package | the bundle goes **inside** the package (`/methods//main.mthds`) and `*.mthds` plus `codegen.lock` are registered as package data, as the Python starter does — a wheel that ships the call site must ship the bundle it loads at call time; an unpackaged app keeps `methods//` at the project root | +| **Aggregate gate** | a Makefile `check` target; `.github/workflows/*.yml`; `.pre-commit-config.yaml`; a `nox`/`tox` session | none: nothing to wire for `python-pydantic` anyway (below) | +| **Call-site location** | the project's existing service / client layer (`services/`, `clients/`, `api/`) → beside it | `/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | reported and un-ignored on confirmation | +| **Owns a codegen harness** | a `codegen` Makefile target; `docs/codegen.md`; `/generated/` beside `/methods/` | the harness section below | + +Why the Ruff exclusion is not optional: `ruff format` rewrites the generated bytes, breaks every stamp, and makes the drift check report the tree as hand-edited. The exclusion goes in before generation, so the first project-wide format after it is already safe. + +## The generated tree + +`mthds_codegen` with `target: "python-pydantic"` and `output_dir: "/generated/"` writes: + +- `models.py` — stamped; `from __future__ import annotations`, `from pydantic import BaseModel, Field`; one plain `BaseModel` subclass per concept, natives included, with `Field(..., description=…)`; no Pipelex import. Non-required fields are `T | None = None`. +- `codegen.lock` — TOML with the crate fingerprint, the engine version and one `[[artifacts]]` entry per stamped file. + +With `target: "python-structures"` the file is `structures.py` — `StructuredContent` subclasses from `pipelex.core.stuffs.structured_content`, natives not re-emitted — for a Pipelex host whose `@pipe_func` functions return them. + +**The skill creates the `__init__.py` files** the tree needs to be importable: `/generated/__init__.py` (once per project, a one-line docstring) and `/generated//__init__.py` (empty). Codegen never emits them and they are never artifacts: they carry no stamp, so the writer does not touch them and the check does not count them as orphans. When the project is packaged, list the generated subpackages where the project lists its packages and add `codegen.lock` as package data, as the Python starter's `pyproject.toml` does. Beside the lock the skill writes `sources.json`, unstamped. + +## The call site + +One module per method. `summarize_pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle inside the package: + +```python +# /pipelex/summarize_pdf.py +"""Run the summarize_pdf method through the hosted Pipelex API. + +`document` is an http(s) URL or a pipelex-storage:// reference, as ``{"url": ...}``. For a +local file or bytes, call ``client.prepare_inputs(files=[...], inputs=...)`` first: it uploads +and rewrites the value. It treats any string it does not recognise as data:, http(s):// or +pipelex-storage:// as a LOCAL FILE PATH it reads and uploads, so gate schemes before handing +values from an untrusted caller to it. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +from pipelex_sdk.client import PipelexAPIClient + +from .generated.summarize_pdf.models import DocumentSummary + +PIPE_CODE = "summarize_pdf" +BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" + + +async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + inputs: dict[str, Any] = {"document": document} + if context is not None: + inputs["context"] = context + async with PipelexAPIClient() as client: + results = await client.start_and_wait( + pipe_code=PIPE_CODE, + mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + inputs=inputs, + ) + return DocumentSummary.model_validate(results.main_stuff) + + +def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: + """The blocking wrapper for a synchronous caller.""" + return asyncio.run(summarize_pdf(document=document, context=context)) +``` + +Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: + +- **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. +- **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. + +Facts the module leans on: `PipelexAPIClient()` reads `PIPELEX_API_KEY` and `PIPELEX_BASE_URL` (default `https://api.pipelex.com`) itself; it is async-only and used as an async context manager; there is **no barrel** in `pipelex_sdk` by design, so every import is a full module path (`pipelex_sdk.client`, `pipelex_sdk.runs`, `pipelex_sdk.errors`); `start_and_wait(...)` takes the durable path on the hosted API (default 2 s poll, 20 min budget, `wait_options=WaitForResultOptions(...)` to change them) and falls back to blocking on a bare runner, returning `RunResults` whose `main_stuff` is always present for a completed run. Let the typed errors from `pipelex_sdk.errors` propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own handling only where it already wraps its other clients. + +Parameters are keyword-only, typed from the signature: `str` for Text and Date (ISO 8601), `float` for Number, `bool` for YesNo, `dict[str, Any]` with a `url` key for Image and Document, the generated model for a structured concept or a composite native, `list[T]` for a list, `T | None = None` for a non-required input. Names are the pipe's input names as declared. A project that already constructs a `PipelexAPIClient` somewhere gets that construction reused instead of a fresh `async with` per call; an async-native project (FastAPI, an existing async codebase) gets the async function alone, without the `_sync` wrapper. + +## The drift gate — an honest asymmetry + +- **`python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists — `pipelex codegen check ` — lives in the `pipelex` runtime, which a hosted-API consumer deliberately does not install. Do not add `pipelex` as a dependency to get a gate; that reverses the decision the target expresses. Write the sidecar anyway (refresh mode and `/pipelex-edit`'s staleness notice read it) and put this sentence in the report: *the generated tree is protected by its stamps and lock, but nothing in CI proves it current; run `/pipelex-integrate` again after every bundle edit.* When the Python SDK gains the check, this section becomes one line. +- **`python-structures`**: the project already depends on `pipelex`, so wire `pipelex codegen check /generated/` into its existing gate (a Makefile `check` target, a workflow step). Exit codes: `0` current, `1` drift, `2` no lock or an unreadable lock. Offline: no engine boot, no network, no key. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-python` has `make codegen` and `make codegen-check`, both shelling out to a `pipelex` CLI the starter deliberately does not depend on (`PIPELEX=` in its Makefile), methods under `/methods//`, trees under `/generated//` with hand-committed `__init__.py` files, and typed CLI commands under its mode sub-packages. On such a project: + +- place the bundle under `/methods//main.mthds` and add the two `codegen` / `codegen-check` lines the Makefile pattern uses for its other methods; +- run `make codegen` when a `pipelex` CLI is reachable; when it is not, call `mthds_codegen` with `output_dir` set to `/generated/` — the harness's own layout, byte-identical (same engine, same stamps, same lock, no sidecar in this starter) — and say `make codegen` is the refresh once `PIPELEX=` points at an install; +- write the CLI command the way `docs/cli-architecture.md` and the existing commands do, importing the generated model from `.generated..models`; +- verify with `make agent-check` and `make agent-test`; no `sources.json` of this skill's shape and no second generated directory. diff --git a/skills/pipelex-integrate/references/typescript.md b/skills/pipelex-integrate/references/typescript.md new file mode 100644 index 0000000..bce5bfa --- /dev/null +++ b/skills/pipelex-integrate/references/typescript.md @@ -0,0 +1,118 @@ +# Integrating into a TypeScript project + +Companion to `/pipelex-integrate` for a project that has a `package.json`. Everything here follows the shape the Pipelex JS starter converged on; the SDK facts were checked against `@pipelex/sdk` 0.17 and move only when that package does. + +## Detecting the project + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **TypeScript build** | a `tsconfig.json`; a `typescript` dev dependency; a bundler or runtime that strips types (Next.js, Vite, tsx, Bun, Deno) | a plain JavaScript project is asked — `types.ts` needs a TypeScript build; the alternative is `python-pydantic`'s sibling in this language, which does not exist yet, so the honest answer is "add TypeScript or skip codegen" | +| **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock` / `bun.lockb` → bun | none → npm, stated | +| **Generated root** | an existing directory already holding generated code (`generated/`, `gen/`, `__generated__/`) → beside it; else `src/generated/` when `src/` exists; else `generated/` | ask | +| **Formatter** | `.prettierrc*` or a `prettier` dev dependency → add `src/generated/` to `.prettierignore` (create the file if absent); `biome.json*` → the files-ignore key its version uses (`files.ignore` before Biome 2, `files.includes` with a `!` negation from Biome 2 on) | a formatter this table does not name: read its config, add the equivalent, say so | +| **Linter** | `eslint.config.*` (flat config) → add `"src/generated/**"` to `globalIgnores([...])` or an `{ ignores: [...] }` entry; legacy `.eslintrc*` → `.eslintignore`; Biome as above | same | +| **Type checker still covers the tree** | `tsconfig.json` `include` / `exclude` — the generated directory must stay inside `include` and outside `exclude` | an exclusion that would drop it is **not** added; the report says the typecheck is the check that covers generated code | +| **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | +| **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | +| **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | + +Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. + +## The generated tree + +`mthds_codegen` with `target: "ts-zod"` and `output_dir: "src/generated/"` writes: + +- `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. +- `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. +- `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. + +Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. + +## The call site + +One module per method. `summarize-pdf` with a `document: native.Document` input, an optional `context: native.Text`, and a `summarize.DocumentSummary` output, from a committed bundle: + +```ts +// src/pipelex/summarizePdf.ts +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import type { RunResults } from "@pipelex/sdk"; +import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; +import type { DocumentSummary } from "../generated/summarize-pdf/types"; +import { getPipelexClient } from "./client"; + +const PIPE_CODE = "summarize_pdf"; +const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); + +export interface SummarizePdfInputs { + /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, + * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * it uploads and rewrites the value. Note: prepareInputs treats any string it does not + * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and + * uploads, so a public endpoint must gate schemes before handing values to it. */ + document: { url: string }; + context?: string; +} + +export async function summarizePdf(inputs: SummarizePdfInputs): Promise { + const bundle = await readFile(BUNDLE_PATH, "utf8"); + const results: RunResults = await getPipelexClient().startAndWaitForResult({ + pipe_code: PIPE_CODE, + mthds_contents: [bundle], + inputs, + }); + return parseDocumentSummary(results.main_stuff); +} +``` + +Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: + +- **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. +- **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. + +The shared client helper, created once per project and reused by every method (if the project already constructs a `PipelexApiClient` somewhere, import that instead): + +```ts +// src/pipelex/client.ts +import { PipelexApiClient } from "@pipelex/sdk"; + +let client: PipelexApiClient | undefined; + +/** Reads PIPELEX_API_KEY and PIPELEX_BASE_URL (default https://api.pipelex.com) from the environment. */ +export function getPipelexClient(): PipelexApiClient { + client ??= new PipelexApiClient(); + return client; +} +``` + +`startAndWaitForResult(options, pollOptions?)` takes the durable path (`start` + poll, default 2 s interval, 20 min budget) on the hosted API and falls back to a blocking `execute` on a bare runner; it returns `RunResults` whose `main_stuff` is always present for a completed run. Let the SDK's typed errors propagate — `RunFailedError`, `RunTimeoutError` (the run keeps going; resume by `pipeline_run_id`), `ApiResponseError` (branch on `.code`), `ApiUnreachableError`, `MissingMainStuffError` — and add the project's own error handling only where it already wraps its other clients. A server-side framework seam (a Next.js Server Action, an Express handler) is the caller's; this module is framework-free. + +Parameter types from the signature: `string` for Text and Date (ISO 8601), `number` for Number, `boolean` for YesNo, `{ url: string }` for Image and Document, the generated type for a structured concept or a composite native, `T[]` for a list, `?` for a non-required input. Key names are the pipe's input names as declared, snake_case included. + +## The offline gate + +Copy `references/codegen-check.mjs` verbatim to `scripts/codegen-check.mjs`. It imports only Node builtins and `@pipelex/sdk`, runs under plain `node` whatever the project's TypeScript build, and prints through `process.stdout` / `process.stderr` so a `no-console` rule does not fire. Register it and extend the existing gate: + +```json +{ + "scripts": { + "codegen:check": "node scripts/codegen-check.mjs src/generated/summarize-pdf", + "check": "npm run lint && npm run typecheck && npm run codegen:check" + } +} +``` + +Add every method's directory to the `codegen:check` line as it is integrated. Exit codes: `0` current, `1` drift or stale source, `2` no verdict (no lock, an unreadable file, a symlink in the tree). It runs from the project root because `sources.json` records source paths relative to it. When `@pipelex/sdk` ships this as a command of its own, the script is replaced by that one line. + +## The Node-only boundary + +`readFile`, `node:path` and `process.cwd()` in the call site are server-side facts. In a framework with a client/server split (Next.js, Remix, SvelteKit), the module belongs on the server side — a Server Action, a route handler, a loader — and the JS starter marks such modules with `import "server-only"`. Never import it from a component that renders in the browser: the API key would leave the server. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js`, or one that copied its codegen kit, has: `npm run codegen` (regenerates every `methods/*` tree through the hosted `/v1/codegen`, writes `contracts.ts` and `sources.json` with a `derived` map), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (semantic, keyed), and `make add-method METHOD=` (manifest, tree, action trio, narrower, form, tab — one shot, never overwrites). On such a project: + +- a local bundle goes under `methods//main.mthds`, then `npm run codegen`; the fan-out follows `docs/codegen.md` and the existing actions under `src/actions/` and narrowers under `src/types/`; +- a catalog or published method goes through `make add-method`; +- the verification is `make check`; the refresh is `npm run codegen`; no `sources.json` of this skill's shape, no `scripts/codegen-check.mjs`, no second generated directory. diff --git a/skills/pipelex-scaffold/references/initializers.md b/skills/pipelex-scaffold/references/initializers.md new file mode 100644 index 0000000..98b683f --- /dev/null +++ b/skills/pipelex-scaffold/references/initializers.md @@ -0,0 +1,38 @@ +# Ecosystem initializers + +Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. + +After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +## Python + +| Want | Command | `git init`? | Where the import package lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | +| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | + +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. + +## TypeScript / JavaScript + +| Want | Command | `git init`? | Where `src/` lands | +|---|---|---|---| +| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | +| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Node library | the minimal recipe above | no | `src/` | +| pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | + +`npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. + +## What every branch-B project shares afterwards + +- One commit, the pristine scaffold, so the user's first real change is a clean diff. +- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/skills/pipelex-scaffold/references/starters.md b/skills/pipelex-scaffold/references/starters.md new file mode 100644 index 0000000..52ccdc3 --- /dev/null +++ b/skills/pipelex-scaffold/references/starters.md @@ -0,0 +1,59 @@ +# The two Pipelex starters + +Both are GitHub **template repositories** under the `Pipelex` organization. Each is a real, CI-tested application against the hosted Pipelex API, not a parameterized template: the identity you see in a fresh clone (`pipelex-starter-js` / `Pipelex Starter`, or `piper` / `Piper`) is a placeholder that the starter's own `bootstrap` skill rewrites. Read the clone's `README.md` after acquiring it — the sections named below are where the details live, and they move as the starters evolve. + +## Side by side + +| | `pipelex-starter-js` | `pipelex-starter-python` | +|---|---|---| +| **Shape** | Next.js (App Router), React, TypeScript strict, Tailwind; a web app with one tab per method whose input form is rendered from the method's own contract by `@pipelex/mthds-form` | A Typer CLI with one command per method, printing JSON on stdout and a cost report on stderr; three execution modes (`blocking`, `attended`, `detached`) as separate sub-packages | +| **Pick it when** | people will use the methods in a browser: forms, uploads, live run status | the methods run from a terminal, a script, a batch job or a service, and the user wants Python | +| **SDK** | `@pipelex/sdk` | `pipelex-sdk` (import package `pipelex_sdk`) — the `pipelex` runtime is **not** a dependency | +| **Methods live in** | `methods//main.mthds`, or `methods//method.json` for a method that lives elsewhere (a catalog id or a published address) | `/methods//main.mthds` | +| **Generated types** | `src/generated//` — `types.ts`, `binder.ts`, `contracts.ts`, `codegen.lock`, `sources.json` | `/generated//` — `models.py`, `codegen.lock` | +| **Codegen harness** | `npm run codegen` (keyed, dev), `npm run codegen:check` (offline, in `make check`), `npm run codegen:verify` (keyed, semantic); `make add-method METHOD=` scaffolds a remote method end to end | `make codegen` / `make codegen-check` — **both shell out to a `pipelex` CLI the starter does not depend on** (`PIPELEX=` in the Makefile); `/pipelex-integrate` knows this and writes into the same layout when that CLI is absent | +| **Toolchain floor** | Node ≥ the `engines.node` field of `package.json` (22.12 at writing); `npm` | `uv`; a Python inside `requires-python` of `pyproject.toml` (3.11–3.14 at writing) | +| **Env file** | `.env.local`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY`, `NEXT_PUBLIC_EXECUTION_MODE` | `.env`, from `.env.example`: `PIPELEX_BASE_URL`, `PIPELEX_API_KEY` | +| **Checks the bootstrap runs** | `npm install --package-lock-only`, then `make all` (lint, format check, typecheck, unit tests, build) | `make li` (lock + sync), then `make agent-check` and `make agent-test` | +| **Agent-facing files** | `CLAUDE.md`, `AGENTS.md`; skills `bootstrap`, `release`, `bump-sdk`, `bump-mthds-form` | `CLAUDE.md`; skills `bootstrap`, `release` | +| **Docs worth reading after bootstrap** | `docs/codegen.md`, `docs/add-method.md`, `docs/input-form.md`, `docs/adopt-in-an-existing-project.md`; README → "Swap in your own pipeline" and "Remove an example" | `docs/codegen.md`, `docs/cli-architecture.md`; README → the per-command sections | +| **Demos it carries** | several demo methods, one tab each; keep them as references or strip them with the README's "Remove an example" checklist | several demo methods, one CLI command each; keep them as references or remove the command and its method directory together | + +## Acquisition + +Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): + +```bash +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git -C rev-parse HEAD +rm -rf /.git && git -C init -b main +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. + +GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): + +```bash +gh auth status +gh repo create / --template Pipelex/pipelex-starter-js --private --clone +gh repo create / --template Pipelex/pipelex-starter-python --private --clone +``` + +## The bootstrap you delegate to + +Both starters carry `.claude/skills/bootstrap/SKILL.md` with a bundled script (`scripts/bootstrap.mjs` / `scripts/bootstrap.py`). Read the file in the clone and follow it; the shape is the same on both: + +1. **Preflight** — confirms the identity is still the template's (`package.json` name `pipelex-starter-js`; `pyproject.toml` `name = "piper"`), notes a dirty tree, and on JS makes sure `node_modules/` exists (`make install`). +2. **Collect** — the package name (kebab on JS, underscores on Python, everything else derives from it), a display title, a one-line description; optionally author name **and** email (never one without the other), the repository URL, and the license (MIT kept, proprietary, or another SPDX id; the copyright holder and year). Pass what the conversation already holds so it asks once for the rest. +3. **Dry run** — the script with `--dry-run` prints the plan; the user confirms. +4. **Run** — the same command without `--dry-run`; on Python the package directory is renamed with `git mv`, which is why the pristine commit must exist first. `--clean` strips the template-only prose; keep it unless the user wants the template charter kept. +5. **Verify** — the lock file is re-synced and the project's own checks run; red is fixed, not skipped. +6. **Self-removal** — `rm -rf .claude/skills/bootstrap`, unstaged like everything else; the user reviews with `git status` and `git diff` and commits when ready. + +The starter's rules are yours while you run it: never commit on the user's behalf, always dry-run first, never touch `.github/` or the `release` skill's logic. + +## What `/pipelex-integrate` finds afterwards + +A bootstrapped starter is a project that **owns a codegen harness**, and `/pipelex-integrate` defers to it: on JS it drops a bundle under `methods//` and runs `npm run codegen`, or runs `make add-method METHOD=…` for a catalog or published method, then follows `docs/codegen.md` and the existing actions for the fan-out; on Python it places the bundle under `/methods//` and runs `make codegen` when a `pipelex` CLI is available, writing into `/generated//` through the Pipelex workshop when it is not. It never writes a second generated layout beside the starter's own. diff --git a/templates/skills/pipelex-design/SKILL.md.j2 b/templates/skills/pipelex-design/SKILL.md.j2 index a51a331..3f8ddbe 100644 --- a/templates/skills/pipelex-design/SKILL.md.j2 +++ b/templates/skills/pipelex-design/SKILL.md.j2 @@ -186,7 +186,7 @@ After the gate: 1. **Organize only when the layout needs it.** A direct result that is already coherent skips `/pipelex-organize`. A converged stepwise result normally invokes it automatically because one-definition-per-file construction history and satisfied headers need regrouping. A naturally coherent result in either mode does not take an organization round trip solely for process compliance. 2. **Project the input schema.** Call `mthds_inputs_template` with the final whole-bundle `files` submission plus `explicit: false`. Show the returned compact template, but **do not save it as `inputs.json`** — input preparation belongs exclusively to `/pipelex-inputs`. 3. **Present the flow.** Point to the interactive method graph where the host rendered the valid verdict's view; in terminal hosts, present a concise text flow of the final structure. -4. **Hand off inputs.** Suggest preparing real inputs with `/pipelex-inputs`. +4. **Hand off inputs — and the code.** Suggest preparing real inputs with `/pipelex-inputs`. Then, when the workspace holds a codebase (a `package.json` or a `pyproject.toml`), say that `/pipelex-integrate` wires the method into it with generated types and a typed call site; when it holds none and the user wants an application around the method, `/pipelex-scaffold` creates one and hands it to `/pipelex-integrate`. > **NEVER write `inputs.json` manually.** If the user provides files, paths, or wants to run with real data, invoke `/pipelex-inputs` — it handles the template, path resolution, placeholder formatting, and file copying. @@ -211,7 +211,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/templates/skills/pipelex-edit/SKILL.md.j2 b/templates/skills/pipelex-edit/SKILL.md.j2 index 07225a1..3c7b607 100644 --- a/templates/skills/pipelex-edit/SKILL.md.j2 +++ b/templates/skills/pipelex-edit/SKILL.md.j2 @@ -80,6 +80,8 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. +**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. + ## Reference - [MTHDS Language Reference](../shared/mthds-reference.md) — read for concept definitions and syntax before editing constructs you haven't touched recently diff --git a/templates/skills/pipelex-inputs/SKILL.md.j2 b/templates/skills/pipelex-inputs/SKILL.md.j2 index b85723e..6cd0fa7 100644 --- a/templates/skills/pipelex-inputs/SKILL.md.j2 +++ b/templates/skills/pipelex-inputs/SKILL.md.j2 @@ -360,6 +360,8 @@ After assembling the inputs, confirm readiness: (Or, for the Template strategy: point out which placeholders the user still needs to fill.) +When the workspace holds a codebase (a `package.json` or a `pyproject.toml`) and the method is not yet wired into it, add one line: `/pipelex-integrate` generates the method's types into the project and writes a typed call site that runs it. + ### Offer to run When the inputs are complete, close by offering to run the method. Offer — never start unprompted: a run executes on the hosted Pipelex API and **spends inference credit**. diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 new file mode 100644 index 0000000..743bb41 --- /dev/null +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -0,0 +1,215 @@ +--- +name: pipelex-integrate +description: Wire an MTHDS method into a Python or TypeScript codebase with generated, drift-proof types and one typed call site that runs it through @pipelex/sdk or pipelex-sdk. Use when the user says "use this method in my app", "call this from my code", "generate types for this method", "wire the method into my project", "add this pipeline to my service", "typed client for this method", "integrate the method", "refresh the generated types", "regenerate the types", "the types are stale", or wants application code that runs a .mthds method — from a local bundle, a catalog id (mt_…) or a published method_ref address. Also the refresh path after a bundle edit. Not for authoring or editing the method itself (/pipelex-design, /pipelex-edit), and not for a project that does not exist yet (/pipelex-scaffold). +{% include "skills/shared/frontmatter.md.j2" %} +{%- if platform == "claude" %} + - mcp__plugin_pipelex_pipelex__mthds_validate + - mcp__plugin_pipelex_pipelex__mthds_codegen + - mcp__plugin_pipelex_pipelex__mthds_inputs_template + - mcp__plugin_pipelex_pipelex__mthds_list_methods +{%- endif %} +--- + +# Integrate an MTHDS method into a codebase + +Take a method — a local `.mthds` bundle, a published address (`method_ref`), or a catalog id (`method_id`) — and a Python or TypeScript project, and leave the project able to call the method with types that cannot silently drift from it. Concretely: + +1. pick the codegen target that matches the project's language **and audience**; +2. have the Pipelex workshop write the generated tree into a dedicated directory per method, through `mthds_codegen`'s write arm, so no generated byte ever passes through you; +3. make the project's formatters and linters leave that tree alone while its type checker keeps covering it; +4. record how the tree was generated in a small sidecar beside the lock, so the next run knows what to refresh and a bundle edit is detectable; +5. wire the offline drift check into the gate the project already runs, where one exists for the language; +6. write one typed call-site module per method, running it through `@pipelex/sdk` or `pipelex-sdk` and narrowing its output with the generated binder or model; +7. verify with the project's own type checker and the gate you just installed. + +Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. + +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. + +## Requirements — the Pipelex MCP tools + +This skill generates through **`mthds_codegen`**, proves the method through **`mthds_validate`**, and reads a pipe's inputs through **`mthds_inputs_template`** on its fallback path — all served by the plugin's `pipelex` MCP server. They are required: never hand-write a generated file, and never derive a signature from the `.mthds` source when the verdict carries it. + +- **If a tool is absent from this session** (the MCP server isn't connected), STOP and tell the user in one line: *"The Pipelex MCP server isn't connected —{% if platform == "mistral-vibe" %} on {{ harness_name }} the local workshop (`{{ mcp_server.command }} {{ mcp_server.args | join(" ") }}`) is not auto-spawned: register it in Vibe's MCP configuration with `PIPELEX_API_KEY` in its environment, then retry.{% else %} the plugin manifest spawns the local workshop (`{{ mcp_server.command }} {{ mcp_server.args | join(" ") }}`), so its absence usually means `node`/`npx` is unavailable or the spawn failed. Check the plugin's MCP connection{% if platform == "claude" %} (`/mcp`){% endif %}.{% endif %}"* +- **If a call returns `status: "error"` with an error of class `config`**, STOP the same way and surface the error's `hint` verbatim. Two `config` errors deserve a precise reading: a **403** on `mthds_codegen` is a feature gate, not a key problem — its hint says code generation is not enabled for the organization on the hosted API; never answer it with "check your key" — and `kind: "paywall"` is the plan limit, whose hint points at billing. +- The server authenticates with **`PIPELEX_API_KEY`** from its environment — the same variable the plugin's validation hook documents. +- **`mthds_list_methods`** is optional: it resolves a catalog method the user names without its `mt_…` id. When it is absent, integrate by id, address or files; never stop for it. + +## Mode + +Automatic by default: state the target, the destination and the generator in one line before writing anything, decide the routine calls yourself, and pause only for a genuinely ambiguous decision (which app in a monorepo; which of two Python audiences; a `method_id` source). Explicit user signals win — "just do it" is automatic, "walk me through" is interactive, and in interactive mode the dependency additions and the tooling edits are confirmed before they happen. Every MCP call branches on the structured verdict, never on transport. + +## The rules that never bend + +- **The write arm, always.** Every `mthds_codegen` call passes `output_dir`. A refused or failed write is handled as a refusal — never by calling again without `output_dir` and writing the returned bytes yourself. A generated file re-emitted through the conversation is one trailing newline away from a broken stamp, and the whole point of the trust chain is that the tree on disk is byte-identical to what the engine emitted. +- **Generated files are never opened for editing, never formatted, never linted.** Each artifact carries a stamp with its own content hash and the lock hashes every artifact; a reformat, a trimmed newline or a re-serialized lock turns the offline check red. This is why the tooling exclusions are made **before** the tree exists. +- **One directory per method.** After writing, the workshop reports any stamped file the new lock does not list as an orphan and never deletes it; two methods in one directory therefore read as permanently non-current, by design. You never delete an orphan either, and you never offer "clean up the orphans" — the moment two methods share a directory, that advice deletes real files. +- **Never generate from one source and run from another.** Types from a local bundle over a call site that runs by `method_id` is the shape that drifts silently. The sidecar records the selector; the call site uses the same one. +- **A project that owns a codegen harness keeps it.** Never write a second generated layout beside the one the project already has. +- **No `dropWireNulls` / `wireOutput` helper.** The ts-zod emitter projects optional fields as `.nullish()`, so a generated schema parses the runtime's explicit `null`s directly; a null-stripping helper is lossy (it removes legitimate nulls inside opaque fields) and must not be written into a project. + +## Process + +### Step 1: Identify the method and the project + +**The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: + +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. +- **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. + +**The project** is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the working area. A workspace holding several (a monorepo, a full-stack repo) is a question — which app? — never a guess. **No project at all** → this is not an integration yet: offer {% if platform == "claude" %}`/pipelex-scaffold`{% else %}the `pipelex-scaffold` skill (open `../pipelex-scaffold/SKILL.md`){% endif %}, which creates one and hands it back here. + +Then look for a **codegen harness**: a `codegen` script in `package.json` or a `codegen` Makefile target, a `sources.json` carrying a `derived` map, `docs/codegen.md` or `docs/add-method.md`, a `methods/` directory beside `src/generated/` or `/generated/`. Either of the first two decides; the rest only corroborate. If the project has one, follow [A project that owns a codegen harness](#a-project-that-owns-a-codegen-harness) from here. + +If a `sources.json` with `"generator": "pipelex-integrate"` already names this method, this is [refresh mode](#refresh-mode). + +### Step 2: Prove the method is integrable + +Call **`mthds_validate`** with the selector. Branch: + +- `status: "ok"`, `is_valid: true`, `is_runnable: true`, `pending_signatures: []` → integrable; keep the verdict, step 3 reads from it. +- `is_valid: true` but **not runnable** or `pending_signatures` non-empty → a scaffold with a concept set but no runnable pipes; integrating it produces a call site that cannot succeed. STOP: finish the method with `/pipelex-design` first. Nothing is generated. +- `is_valid: false` → route the `validation_errors[]` to `/pipelex-design` or `/pipelex-edit`; a by-id method's stored content is fixed where it is edited, not here. +- `status: "error"` → class `config` stops per the Requirements; class `input_domain` at `method_ref` / `method_id` is reported in the tool's own words (an unknown or foreign-organization id — the catalog is org-scoped, so another org's method reads exactly like a miss — an unfetchable address, a registry-form ref); class `runtime` → retry once, then report. + +### Step 3: Read the pipe's signature — from the verdict + +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. + +**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. + +### Step 4: Choose the target, the destination and the generator + +State the three in one line before writing. The rule for the target is about **audience**, not language: + +| Project | Target | Emits | +|---|---|---| +| `package.json` with a TypeScript build (a `tsconfig.json`, or a runtime/bundler that strips types) | `ts-zod` | `types.ts` (zod schemas + inferred types, depends only on `zod`) and `binder.ts` (`parse` / `serialize`); keep both | +| `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | +| `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | + +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). + +### Step 5: Make the tooling leave the tree alone — before the tree exists + +Add the generated directory to the formatter's and linter's ignore lists per the language reference (`.prettierignore`, an ESLint flat-config `ignores`, Biome; `[tool.ruff] exclude`, Black, isort), confirm the type checker's include **still covers it** (an exclusion that would drop it is not added), and confirm it is not gitignored. Do this **before** step 6: the first project-wide `format` run after generation would otherwise rewrite the stamps and turn the check red. + +### Step 6: Generate + +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root{% if platform == "mistral-vibe" %} (or register the workshop with that working directory){% endif %} — do not ride content instead. + +Branch on the structured result: + +- `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. +- `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. +- `status: "error"`, class `input_domain` located at `output_dir`: + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. +- `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. +- Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. +- Success with **`is_current: false`** → a write the check disowns. Report the `drifts[]` verbatim (`path`, `category`, `detail`) and stop; never commit a tree the check rejects. + +### Step 7: Write the sidecar + +Write an **unstamped `sources.json`** beside the lock — the only state this skill keeps. The lock signs the artifacts, not their sources; without the sidecar the next run re-derives everything and a bundle edit is undetectable. Shape: + +```json +{ + "comment": "Written by /pipelex-integrate. `method` and `target` are how this tree was generated — re-run the skill to refresh it. `sources` is the SHA-256 of each local .mthds source, so a bundle edit that was never regenerated is detectable. Not part of the codegen lock; do not hand-edit.", + "generator": "pipelex-integrate", + "method": { "files": ["methods/summarize-pdf/main.mthds"] }, + "target": "ts-zod", + "pipe": { + "pipe_ref": "summarize.summarize_pdf", + "inputs": { "document": "native.Document", "context": "native.Text?" }, + "output": "summarize.DocumentSummary" + }, + "sources": { "methods/summarize-pdf/main.mthds": "" } +} +``` + +`method` is exactly one of `{files}`, `{method_ref}`, `{method_id}`, as passed. Paths are relative to the **project root**, not to the workshop's working directory. `pipe` records what the call site was typed against — `Concept` single, `Concept[]` a list, `Concept?` optional — so refresh mode can tell a signature change from a body change. `sources` is empty for a `method_ref` / `method_id` source. Hashes are over the raw bytes: `shasum -a 256 ` / `sha256sum ` / `hashlib.sha256(path.read_bytes())`. + +### Step 8: Add the dependencies the generated code needs + +With the project's own package manager (read the lockfile): `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen). State what is added; interactive mode confirms first. + +### Step 9: Write the call site + +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. + +| Declared concept | TypeScript parameter | Python parameter | +|---|---|---| +| `native.Text` (or a refinement) | `string` | `str` | +| `native.Number` | `number` | `float` | +| `native.YesNo` | `boolean` | `bool` | +| `native.Date` | `string` (ISO 8601) | `str` (ISO 8601) | +| `native.Image`, `native.Document` | `{ url: string }` — an `http(s)` URL or a `pipelex-storage://` reference | `dict[str, Any]` with a `url` key | +| a structured concept, or a composite native (`Page`, `TextAndImages`, `JSON`) | the generated type from `types.ts` | the generated model from `models.py` | +| `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | +| not required | optional parameter (`?`) | `T \| None = None` | + +**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. + +### Step 10: Wire the offline drift gate + +- **TypeScript**: copy [references/codegen-check.mjs](references/codegen-check.mjs) **verbatim** to `scripts/codegen-check.mjs`, register `"codegen:check": "node scripts/codegen-check.mjs …"` in `package.json`, and **extend the project's existing aggregate gate** — a `check` / `ci` / `validate` / `verify` script, a Makefile `check` target, the lint or test step of an existing workflow — rather than inventing a new one. The script runs `@pipelex/sdk`'s `runCodegenCheck` over each directory, compares the sidecar's source hashes against the committed `.mthds` files, and exits `0` current / `1` drift or stale source / `2` no verdict. A project with no aggregate gate gets the script and one sentence in the report saying where to call it. +- **Python, `python-pydantic`**: **no gate is installed.** `pipelex-sdk` has no offline check yet, and the only one that exists (`pipelex codegen check`) needs the `pipelex` runtime a consumer deliberately does not have — do not add `pipelex` as a dependency to get a gate. The sidecar is still written (refresh mode and the editing skills' staleness notice read it), and the report says plainly: the tree is protected by its stamps and lock, but nothing in CI proves it current; refresh with this skill after every bundle edit. +- **Python, `python-structures`**: the project already depends on `pipelex`, so `pipelex codegen check ` (exit `0` / `1` / `2`) is wired into its existing gate. + +### Step 11: Verify + +Run the project's formatter **on the files you wrote only** — never on the generated tree; the step-5 exclusions are what make a later project-wide run safe — then its type checker, then the gate you installed. A failure in your own code is yours to fix before reporting; a failure inside the generated tree is reported, never patched. + +### Step 12: Report + +What was generated and where; the target and why; the call site's signature; what changed in the tooling config; how to refresh (this skill again after a bundle edit); for a `python-pydantic` project, that no offline drift check exists yet and refresh is the guard; for a `method_id` source, that the catalog is unversioned. Then the hand-off: `/pipelex-inputs` prepares inputs and offers a run. + +## Refresh mode + +Entered when the user asks to refresh, regenerate or update the types; when `/pipelex-edit` or `/pipelex-design` hand off after editing a bundle a sidecar names; or when step 1 finds a sidecar for the method. **Re-derive nothing the sidecar already records, regenerate in place, and leave alone everything the regeneration did not invalidate.** + +| Taken from disk | Re-derived | Left alone | +|---|---|---| +| the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | + +One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. + +## A project that owns a codegen harness + +A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one that adopted their pattern, regenerates every method in one place, checks them in one place, and keeps its own sidecar. Writing this skill's tree beside that would leave two regeneration paths, two sidecar dialects and files the workshop never emits. So on such a project: + +- **place the method where the project keeps them** — `methods//main.mthds`, or the project's manifest form for a catalog or published method; +- **run the project's generator** — `make add-method METHOD=…` when the project has it and the method is remote, its `codegen` script or Makefile target otherwise; +- **write the call site the way the project's docs and existing methods do** (`docs/codegen.md`, `docs/add-method.md`, the existing actions or CLI commands), not the shape of step 9; +- **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); +- **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. + +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| a required tool is absent | STOP with the one-line MCP-connection message above | +| `status: "error"`, class `config` — including the codegen **403** feature gate | STOP, surface `hint` verbatim; never say "check your key" for a 403 | +| `status: "error"`, class `config`, `kind: "paywall"` | STOP, surface the plan-limit hint | +| `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | +| not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | +| `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | +| `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | +| success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | +| success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | +| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `mthds_list_methods` absent | integrate by id, address or files; never stop for it | + +## Reference + +- [references/typescript.md](references/typescript.md) — detecting a TypeScript project (build, package manager, generated root, Prettier / ESLint / Biome exclusions, `tsconfig` coverage, the aggregate gate, the call-site location), the call-site module and client helper templates, the `codegen:check` wiring, the harness a starter-derived project owns. +- [references/python.md](references/python.md) — the same for Python (import package, `python-pydantic` vs `python-structures`, uv / poetry / pipenv / pip, Ruff / Black / isort exclusions, pyright / mypy coverage, `__init__.py` and package data, the async module plus its sync wrapper, `pipelex codegen check` for the structures audience, the asymmetry sentence for everyone else). +- [references/codegen-check.mjs](references/codegen-check.mjs) — the offline gate copied verbatim into TypeScript projects. +- [MTHDS Language Reference](../shared/mthds-reference.md) — for reading a bundle's `main_pipe` and `output` declarations on the fallback path. diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 new file mode 100644 index 0000000..0297634 --- /dev/null +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -0,0 +1,158 @@ +--- +name: pipelex-scaffold +description: Start a new project that will call MTHDS methods through Pipelex, in TypeScript or Python — from one of the Pipelex starter templates or from the ecosystem's own initializer — and hand it to /pipelex-integrate. Use when the user says "start a new project with Pipelex", "I have a method and need an app around it", "create a Next.js app that runs my method", "set up a Pipelex project from scratch", "new Python CLI for this method", "which starter should I use", "bootstrap a Pipelex project", or wants a codebase where none exists yet. Also use when the user is standing in a freshly cloned pipelex-starter-js or pipelex-starter-python that has not been renamed yet — this skill runs the template's own bootstrap for them. Not for adding Pipelex to code that already exists: that is /pipelex-integrate. +{% include "skills/shared/frontmatter.md.j2" %} +--- + +# Scaffold a project for Pipelex methods + +Give a user who has no project yet a project that is ready for `/pipelex-integrate`. This skill has exactly two branches and carries no templates of its own: + +- **One of the Pipelex starters** when the user wants the opinionated shape: `pipelex-starter-js` for a web app whose forms are rendered from the methods' own contracts, `pipelex-starter-python` for a CLI or service that runs methods in the three execution modes. You acquire the template, commit it once as it came, then run the clone's **own** `bootstrap` skill — the rename logic lives in the starters and is never reimplemented here. +- **The ecosystem's own initializer** when the user wants their framework or a minimal project: `uv init --package`, `npm create next-app@latest`, `django-admin startproject`, whatever the framework documents. You run it; you never assemble a project by hand. + +Both branches end the same way: an env file that follows the starters' convention, one pristine commit that makes everything after it reviewable, and the hand-off — to `/pipelex-integrate` when a method exists, to `/pipelex-design` first when none does. + +**What this skill is not.** Not a template engine (no cookiecutter, no copier, no framework matrix of its own), not a bootstrap (the starters own theirs), not a runner or a dev-server launcher, not a deployer. It needs no MCP tool and no API key: git, the starters' scripts and the ecosystem's initializers are all it uses. + +## Choosing the branch + +A cheap, reliable signal decides; an inconclusive one asks one question; nothing is guessed twice. + +| Question | Signals, in order | When inconclusive | +|---|---|---| +| **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | +| **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | + +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. + +[references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. + +## Mode + +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. + +## Branch A — one of the starters + +### Step 1: Prerequisites + +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. + +- **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. +- **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). +- **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. + +### Step 2: Acquire the template + +**Local, the default.** Clone shallow, read the template's identity, then detach from it: + +```bash +git clone --depth 1 https://github.com/Pipelex/.git +git -C rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf /.git && git -C init -b main +``` + +The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. + +**GitHub, on request.** When the user asked for a repository on GitHub: + +```bash +gh repo create / --template Pipelex/ --private --clone +``` + +Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. + +Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. + +### Step 3: Commit the pristine template — exactly once + +```bash +git -C add -A && git -C commit -m "Start from Pipelex/ ()" +``` + +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. + +### Step 4: Run the clone's own bootstrap + +Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written. The project's skills are not loaded in this session — it began elsewhere — so read the file; do not look for a `/bootstrap` command. Run every command it gives from inside the project directory (`cd && …`, or `-C `), because that skill assumes it is standing in the repo root. + +Feed it what the conversation already holds — the project name, title, description, author, repository URL, license — so that it asks once, consolidated, for whatever is left, exactly as its own Step 2 says. It dry-runs, previews, runs, re-syncs the lock file, runs the project's own checks (`make all` on JS; `make agent-check` and `make agent-test` on Python), and removes itself. Its rules stand unchanged: it never commits, its edits stay uncommitted for the user's review (the Python renames are staged by `git mv`, which its skill explains), and a red check is fixed, never skipped. **Add nothing to that procedure and reimplement none of it.** If the clone carries no bootstrap skill — a future template dropped it — follow the README's "manual equivalent" list and say that the template changed. + +### Step 5: The env file + +```bash +cp /.env.example /.env.local # JS: Next.js reads .env.local +cp /.env.example /.env # Python: python-dotenv reads .env +``` + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. + +### Step 6: Verify and hand off + +The bootstrap's own checks are the verification; do not start `make dev`. Write the report (below), then hand the user's method to `/pipelex-integrate`, which recognizes the starter's codegen harness (`npm run codegen`, `make codegen`, `make add-method`) and defers to it rather than writing a second one. + +## Branch B — the ecosystem's initializer + +### Step 1: Prerequisites + +As in branch A, for the language chosen. + +### Step 2: Run the initializer — never assemble by hand + +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. + +Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. + +### Step 3: Version control and the pristine commit + +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: + +```bash +git -C add -A && git -C commit -m "Scaffold project" +``` + +### Step 4: The env file + +Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: + +``` +PIPELEX_BASE_URL=https://api.pipelex.com +PIPELEX_API_KEY= +``` + +### Step 5: Hand off + +Add **no** SDK dependency and create **no** empty `methods/` directory: `/pipelex-integrate` adds `@pipelex/sdk` or `pipelex-sdk` when it writes the first call site, and creates `methods//` when it places the first bundle. A project with nothing to integrate yet has nothing Pipelex-shaped in it beyond the env convention, and that is correct. + +## The report + +Say, in this order: what was created and where; which template or initializer it came from, at which version and SHA; that this skill made exactly one commit and what it holds; what the bootstrap changed and that those changes are uncommitted for review, in the bootstrap's own words (branch A); which env file was written and whether the key was filled from the environment or left for the user; the demos the starter still carries and where the README's removal checklist is (branch A); and the hand-off. + +Two lines are easy to forget and matter: + +- **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; {% if platform == "claude" %}`cd && claude`{% else %}`cd `, then starting {{ harness_name }} there,{% endif %} is how they arrive. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it{% if platform == "claude" %} with `/pipelex-integrate`{% else %} by opening `../pipelex-integrate/SKILL.md` and following it{% endif %}; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. + +## When something goes wrong + +| Condition | Do this | +|---|---| +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | +| The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | +| The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | +| An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | +| `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | + +## Reference + +- [references/starters.md](references/starters.md) — the two starters side by side: what each brings, its prerequisite floors, the acquisition commands, its env file, its bootstrap, its demos and their removal checklist, and the codegen harness `/pipelex-integrate` will find. +- [references/initializers.md](references/initializers.md) — per language, the minimal default and the common frameworks' non-interactive initializers, whether each runs `git init`, and where the import package or `src/` root lands. +- `/pipelex-integrate` — the skill this one hands every project to. diff --git a/tests/unit/test_gen_skill_docs.py b/tests/unit/test_gen_skill_docs.py index ef5566a..46f01dc 100644 --- a/tests/unit/test_gen_skill_docs.py +++ b/tests/unit/test_gen_skill_docs.py @@ -602,7 +602,7 @@ def test_edit_offers_restore_on_no_verdict(self) -> None: body = (self.REPO_TEMPLATES / "pipelex-edit" / "SKILL.md.j2").read_text(encoding="utf-8") assert "applied but **unproven**" in body - MCP_SKILLS = ("pipelex-design", "pipelex-organize", "pipelex-edit", "pipelex-inputs") + MCP_SKILLS = ("pipelex-design", "pipelex-organize", "pipelex-edit", "pipelex-inputs", "pipelex-integrate") @pytest.mark.parametrize( "target_name, manifest_spawns", diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py new file mode 100644 index 0000000..c88f714 --- /dev/null +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -0,0 +1,132 @@ +"""Pin the load-bearing rules of the pipelex-integrate skill template and its references.""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir, setup_static_assets + + +class TestPipelexIntegrateSkill: + """The skill is executable guidance, so these tests guard the rules a user's + codebase depends on: the write arm is the only arm, generated bytes are never + touched, one directory per method, orphans are never deleted, sources are never + mixed, a project-owned harness is kept, and the signature comes from the verdict. + """ + + REPO_ROOT = Path(__file__).parents[2] + TEMPLATE = REPO_ROOT / "templates" / "skills" / "pipelex-integrate" / "SKILL.md.j2" + REFERENCES_DIR = REPO_ROOT / "skills" / "pipelex-integrate" / "references" + REFERENCES = ("typescript.md", "python.md", "codegen-check.mjs") + RULES = ( + "**The write arm, always.** Every `mthds_codegen` call passes `output_dir`.", + "never by calling again without `output_dir` and writing the returned bytes yourself", + "**Generated files are never opened for editing, never formatted, never linted.**", + "**One directory per method.**", + "You never delete an orphan either", + "**Never generate from one source and run from another.**", + "**A project that owns a codegen harness keeps it.** Never write a second generated layout", + "**No `dropWireNulls` / `wireOutput` helper.**", + "Do this **before** step 6", + ) + + @property + def integrate(self) -> str: + return self.TEMPLATE.read_text(encoding="utf-8") + + def test_the_rules_that_never_bend_are_stated(self) -> None: + body = self.integrate + for rule in self.RULES: + assert rule in body, f"missing rule: {rule}" + + def test_signature_comes_from_the_verdict_and_the_heuristic_is_absent(self) -> None: + body = self.integrate + assert "A valid verdict carries **`main_pipe`**" in body + assert "**`explicit: true`** — the one call in this plugin that wants the ceremonial" in body + # The by-elimination output-concept heuristic was designed, then made obsolete before it shipped. + assert "candidate output concepts" not in body + assert "minus the input concepts minus natives" not in body + # A by-ref / by-id source with no signature stops instead of guessing. + assert "STOP and say the workshop is too old to type this integration exactly, rather than guessing" in body + + def test_the_wire_null_helper_is_never_installed(self) -> None: + body = self.integrate + assert "wire-output.ts" not in body + assert "wireOutput(results" not in body + for reference in self.REFERENCES: + assert "dropWireNulls" not in (self.REFERENCES_DIR / reference).read_text(encoding="utf-8") + + def test_failure_posture_pins_the_403_and_the_orphans(self) -> None: + body = self.integrate + assert "a **403** on `mthds_codegen` is a feature gate, not a key problem" in body + assert 'never say "check your key" for a 403' in body + assert '`kind: "paywall"`' in body + assert "report by name, never delete" in body + assert "never delete, move or clear the named file" in body + assert "report `drifts[]` verbatim and stop" in body + + def test_method_id_warns_and_refresh_leaves_the_call_site_alone(self) -> None: + body = self.integrate + assert "The catalog is unversioned" in body + assert "proceed only on the user's say-so" in body + assert "**The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature**" in body + assert '"generator": "pipelex-integrate"' in body + + def test_python_gate_asymmetry_is_honest(self) -> None: + body = self.integrate + assert "**Python, `python-pydantic`**: **no gate is installed.**" in body + assert "do not add `pipelex` as a dependency to get a gate" in body + assert "pipelex codegen check " in body + python = (self.REFERENCES_DIR / "python.md").read_text(encoding="utf-8") + assert "**no gate is installed.**" in python + assert "there is **no barrel** in `pipelex_sdk` by design" in python + + def test_the_check_script_imports_only_node_builtins_and_the_sdk(self) -> None: + script = (self.REFERENCES_DIR / "codegen-check.mjs").read_text(encoding="utf-8") + imports = re.findall(r'^import .* from "([^"]+)";$', script, flags=re.MULTILINE) + assert imports, "the script should import through static ESM imports" + for module in imports: + assert module.startswith("node:") or module == "@pipelex/sdk", f"unexpected import: {module}" + assert "runCodegenCheck" in script and "isStampableArtifactPath" in script + assert "process.stdout.write" in script and "process.stderr.write" in script + assert "console." not in script + + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_every_platform_renders_the_skill_and_its_references(self, target_name: str) -> None: + config = load_target_config(self.REPO_ROOT / "targets", target_name) + rendered = render_templates( + self.REPO_ROOT / "templates", + self.REPO_ROOT, + config.template_vars, + include_skills=["pipelex-integrate"], + target_name=config.name, + ) + body = next(content for path, content in rendered.items() if path.match("skills/pipelex-integrate/SKILL.md")) + assert "# Integrate an MTHDS method into a codebase" in body + assert "{%" not in body + assert "{{" not in body + for rule in self.RULES: + assert rule in body, f"{target_name}: missing rule: {rule}" + if target_name == "prod": + assert "mcp__plugin_pipelex_pipelex__mthds_codegen" in body + assert "offer `/pipelex-scaffold`" in body + else: + assert "mcp__" not in body + assert "open `../pipelex-scaffold/SKILL.md`" in body + + references_dir = resolve_output_dir(self.REPO_ROOT, config.source) / "skills" / "pipelex-integrate" / "references" + for reference in self.REFERENCES: + assert (references_dir / reference).is_file(), f"{target_name}: missing references/{reference}" + + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_the_build_copies_the_references_byte_for_byte(self, target_name: str, tmp_path: Path) -> None: + """The check script is executable know-how: a stale or re-encoded copy is a broken gate.""" + config = load_target_config(self.REPO_ROOT / "targets", target_name) + setup_static_assets(self.REPO_ROOT, tmp_path, self.REPO_ROOT / "templates", config.include_skills) + produced = tmp_path / "skills" / "pipelex-integrate" / "references" + for reference in self.REFERENCES: + assert (produced / reference).is_file(), f"{target_name}: the build did not copy references/{reference}" + assert (produced / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes() diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py new file mode 100644 index 0000000..c056c97 --- /dev/null +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -0,0 +1,111 @@ +"""Pin the pipelex-scaffold skill: two branches, no templates of its own, one commit, delegated bootstrap.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir, setup_static_assets + + +class TestPipelexScaffoldSkill: + """The skill is executable guidance, so these tests guard what a user's new + project depends on: nothing is written into a non-empty directory, the skill + makes exactly one commit, the starters' bootstrap is delegated and never + reimplemented, the key never crosses the conversation, and no MCP tool is needed. + """ + + REPO_ROOT = Path(__file__).parents[2] + SKILLS = REPO_ROOT / "templates" / "skills" + TEMPLATE = SKILLS / "pipelex-scaffold" / "SKILL.md.j2" + REFERENCES_DIR = REPO_ROOT / "skills" / "pipelex-scaffold" / "references" + REFERENCES = ("starters.md", "initializers.md") + RULES = ( + "exactly two branches and carries no templates of its own", + "no cookiecutter, no copier, no framework matrix of its own", + "never write into a directory that exists and is not empty", + "This is the **one commit this skill makes**", + "Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written", + "**Add nothing to that procedure and reimplement none of it.**", + "**Never print a key, and never ask for one in the conversation.**", + "**state the exact command and confirm before running it**", + "do not start `make dev`", + "Add **no** SDK dependency and create **no** empty `methods/` directory", + "Nothing beyond what the initializer writes is authored by this skill", + ) + + @property + def scaffold(self) -> str: + return self.TEMPLATE.read_text(encoding="utf-8") + + def test_the_rules_are_stated(self) -> None: + body = self.scaffold + for rule in self.RULES: + assert rule in body, f"missing rule: {rule}" + + def test_fresh_clone_shortcut_and_template_checkout_stop(self) -> None: + body = self.scaffold + assert "**The fresh-clone shortcut.**" in body + assert "Do not clone again." in body + assert "this is the template, not a copy of it" in body + + def test_declares_no_mcp_tool(self) -> None: + """The scaffold skill is MCP-free: no allowed-tools entry, no MCP-absent STOP message.""" + body = self.scaffold + assert "mcp__" not in body + assert "plugin manifest spawns" not in body + assert "It needs no MCP tool and no API key" in body + + def test_integrate_hands_a_missing_project_to_scaffold(self) -> None: + integrate = (self.SKILLS / "pipelex-integrate" / "SKILL.md.j2").read_text(encoding="utf-8") + assert "**No project at all** → this is not an integration yet: offer" in integrate + assert "pipelex-scaffold" in integrate + + def test_references_describe_both_starters_and_the_initializers(self) -> None: + starters = (self.REFERENCES_DIR / "starters.md").read_text(encoding="utf-8") + assert "pipelex-starter-js" in starters and "pipelex-starter-python" in starters + assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git" in starters + assert "gh repo create / --template Pipelex/pipelex-starter-python" in starters + assert "shell out to a `pipelex` CLI the starter does not depend on" in starters + initializers = (self.REFERENCES_DIR / "initializers.md").read_text(encoding="utf-8") + assert "uv init --package " in initializers + assert "npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes" in initializers + assert "No SDK dependency" in initializers + + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_every_platform_renders_the_skill_and_its_references(self, target_name: str) -> None: + config = load_target_config(self.REPO_ROOT / "targets", target_name) + rendered = render_templates( + self.REPO_ROOT / "templates", + self.REPO_ROOT, + config.template_vars, + include_skills=["pipelex-scaffold"], + target_name=config.name, + ) + body = next(content for path, content in rendered.items() if path.match("skills/pipelex-scaffold/SKILL.md")) + assert "# Scaffold a project for Pipelex methods" in body + assert "{%" not in body + assert "{{" not in body + assert "mcp__" not in body + for rule in self.RULES: + assert rule in body, f"{target_name}: missing rule: {rule}" + if target_name == "prod": + assert "`cd && claude`" in body + assert "with `/pipelex-integrate`" in body + else: + assert "`cd && claude`" not in body + assert "opening `../pipelex-integrate/SKILL.md`" in body + + references_dir = resolve_output_dir(self.REPO_ROOT, config.source) / "skills" / "pipelex-scaffold" / "references" + for reference in self.REFERENCES: + assert (references_dir / reference).is_file(), f"{target_name}: missing references/{reference}" + + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_the_build_copies_the_references_byte_for_byte(self, target_name: str, tmp_path: Path) -> None: + config = load_target_config(self.REPO_ROOT / "targets", target_name) + setup_static_assets(self.REPO_ROOT, tmp_path, self.REPO_ROOT / "templates", config.include_skills) + produced = tmp_path / "skills" / "pipelex-scaffold" / "references" + for reference in self.REFERENCES: + assert (produced / reference).is_file(), f"{target_name}: the build did not copy references/{reference}" + assert (produced / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes() diff --git a/wip/pipelex-integrate/design.md b/wip/pipelex-integrate/design.md index be1e09d..54271e7 100644 --- a/wip/pipelex-integrate/design.md +++ b/wip/pipelex-integrate/design.md @@ -5,7 +5,7 @@ item: L-260830-344594 # Design — `pipelex-integrate`: wire an MTHDS method into a Python or TypeScript codebase -**Written 2026-08-30**, from the brief beside this file ([`brief.md`](brief.md)), against the sources it names: `pipelex-mcp/SPEC.md` → "Codegen Scope" and "The write arm", `pipelex-starter-js/docs/codegen.md`, `pipelex-starter-python/Makefile` and `docs/codegen.md`, the two starter call sites, and `docs/specs/pipelex-codegen.md`. **Status: draft — awaiting ratification.** The decision boxes at the end are what a ratification answers; the implementation tracker is [`plan.md`](plan.md) and starts after they are answered. Ledger item `L-260830-344594`. File and line references were accurate on the writing date; verify them against the code before implementing. +**Written 2026-08-30**, from the brief beside this file ([`brief.md`](brief.md)), against the sources it names: `pipelex-mcp/SPEC.md` → "Codegen Scope" and "The write arm", `pipelex-starter-js/docs/codegen.md`, `pipelex-starter-python/Makefile` and `docs/codegen.md`, the two starter call sites, and `docs/specs/pipelex-codegen.md`. **Status: active** — the ten decision boxes at the end were ratified on 2026-08-30, and the document was **amended on 2026-09-06** (boxes 3 and 8 reworded, box 11 added) when a second design session widened the campaign to greenfield projects; that session's sibling document is [`scaffold-design.md`](scaffold-design.md) and the implementation tracker for both skills is [`plan.md`](plan.md). Ledger item `L-260830-344594`. File and line references were accurate on the writing date; verify them against the code before implementing. Everything the brief lists as a *finding* is taken as a constraint and not re-argued here. This document settles the brief's six open questions and the smaller decisions the implementation needs, in the order a reader of the skill would meet them. @@ -27,7 +27,7 @@ Re-running it on a project that already carries a generated tree is **refresh mo ## 2. The shape it produces -The target state is the reference design both starters converged on, minus the per-project scaffolding the workshop replaces. +The target state is the reference design both starters converged on, minus the per-project scaffolding the workshop replaces. On a project that already owns that scaffolding — a codegen harness of its own — the shape is the project's, not this one (§4.12). **TypeScript** (`ts-zod`): @@ -67,10 +67,10 @@ The skill is automatic by default, with the same mode rules as `pipelex-inputs` 1. **Identify the method and the project.** The method comes from the conversation (a bundle directory, an address, an `mt_…` id, or a name resolved through `mthds_list_methods`). The project root is the nearest directory holding a `package.json` or a `pyproject.toml` (or `setup.py` / `requirements.txt`) above the user's working area; a workspace holding several is a question, not a guess. Detection rules are in §5. 2. **Prove the method is integrable.** `mthds_validate` on the selector: `is_valid: true` **and** `is_runnable: true` with no pending signatures. A scaffold with pending signatures has a concept set but cannot run, so integrating it produces a call site that cannot succeed — route to `/pipelex-design` instead. An invalid method carries its `validation_errors[]` to `/pipelex-design` / `/pipelex-edit` the way `pipelex-inputs` does. -3. **Read the pipe's signature.** `mthds_inputs_template` with the same selector and **`explicit: true`** — the one skill call in the plugin that wants the ceremonial envelope, because it needs each input's declared concept ref to type the call site (§4.3, §4.9). Record the resolved `pipe_ref`. The main pipe's output concept and multiplicity come from the bundle for a files source, and from the heuristic in §4.3 otherwise. -4. **Choose the target** (§4.2) and **the destination** (§5). State both in one line before writing anything. +3. **Read the pipe's signature from the verdict.** A valid `mthds_validate` verdict carries `structuredContent.main_pipe` — the main pipe's namespaced ref, each declared input with its fully-qualified concept ref, multiplicity and `required` flag, and the produced concept with its multiplicity and `optional` flag — which is everything the call site is typed against (§4.3). Record it. Only when `main_pipe` is absent (the bundle declares no main pipe) does the skill ask which pipe to integrate and read that pipe's inputs from `mthds_inputs_template` with `explicit: true` (§4.9), its output from the bundle for a files source. +4. **Choose the target** (§4.2), **the destination** (§5), and **the generator** — the project's own codegen harness when it has one, the workshop's write arm otherwise (§4.12). State all of it in one line before writing anything. 5. **Make the tooling leave the tree alone — before the tree exists.** Add the generated directory to the formatter's and linter's ignore lists per §5, confirm the type checker's include still covers it, and confirm it is not gitignored. This ordering is load-bearing: the first project-wide `format` run after generation would otherwise rewrite the stamps. -6. **Generate.** `mthds_codegen` with the selector, `target`, and `output_dir` expressed **relative to the workshop's working directory** (§4.4). Branch: `is_valid: false` → back to step 2's repair route; `status: "error"` located at `output_dir` → a foreign file or a containment escape, handled per §6; `runtime` mid-write → call again once with the same `output_dir`, as the tool's own hint says. On success, confirm `is_current: true` and an empty `orphans[]`; a non-empty `orphans[]` is reported by name and never cleaned (§4.7). +6. **Generate.** On a project that owns a codegen harness, run that harness instead: it owns the tree, the lock and the check, steps 7, 8 and 10 are skipped because the harness already provides them, and step 9 follows the project's own documentation (§4.12). Otherwise `mthds_codegen` with the selector, `target`, and `output_dir` expressed **relative to the workshop's working directory** (§4.4). Branch: `is_valid: false` → back to step 2's repair route; `status: "error"` located at `output_dir` → a foreign file or a containment escape, handled per §6; `runtime` mid-write → call again once with the same `output_dir`, as the tool's own hint says. On success, confirm `is_current: true` and an empty `orphans[]`; a non-empty `orphans[]` is reported by name and never cleaned (§4.7). 7. **Write the sidecar** `sources.json` beside the lock (§4.6). 8. **Add the dependencies the generated code needs**, with the project's own package manager: `zod` and `@pipelex/sdk` for TypeScript; `pydantic` and `pipelex-sdk` for Python (`python-structures` needs `pipelex`, which is already present by the time that target is chosen — §4.2). State what is added; interactive mode confirms first. 9. **Write the call site** and its shared helpers (§4.1, §4.8). @@ -113,12 +113,12 @@ A bundle that is outside the project root (a `pipelex-wip/` directory elsewhere **Decision: the skill does not produce a contracts artifact.** `pipelex-starter-js` needs `PIPE_IO_CONTRACTS` and `INPUT_FORM` to drive a form kernel and gate run inputs in a Server Action; a typed function has neither concern — its parameter types *are* the input gate, and the SDK validates the run request. Nothing in the skill's scope consumes the descriptor. -What the skill does need is the **pipe's signature**: input names with their concept refs, and the main pipe's output concept with its multiplicity. Today this reaches the model through two channels and one gap: +What the skill does need is the **pipe's signature**: input names with their concept refs and multiplicity, and the main pipe's output concept with its multiplicity. -- **Inputs** — `mthds_inputs_template` with `explicit: true` returns each input as `{concept, content}` in `structuredContent`; the concept ref is exactly what types the parameter. This is why the skill departs from the plugin's `explicit: false` pin (§4.9). -- **Output, files source** — read from the bundle: the root's `main_pipe` and that pipe's `output` declaration, multiplicity included. The model can read the files; no tool call is needed. -- **Output, `method_ref` / `method_id` source — the gap.** `mthds_validate` carries `main_pipe_ref` and `pipe_io_contracts` on the view-only `_meta` channel, which never reaches the model, and the workshop has no views. So for a method whose source is not on disk, no in-context channel names the output concept. **v1 heuristic:** after generation, the candidate output concepts are the generated types minus the input concepts minus natives; exactly one candidate is taken and stated as an assumption; several is one question to the user, listing them; multiplicity is assumed single unless the user says otherwise. For a public `method_ref` the model may additionally read the package's `.mthds` at the tag from the repository to answer exactly. -- **The follow-up.** The heuristic is a stopgap. The correct fix is in `pipelex-mcp`: a compact main-pipe signature — `main_pipe_ref`, input names → concept refs, output concept ref and multiplicity — promoted from `_meta` to `structuredContent` on a valid `mthds_validate` verdict (or on `mthds_codegen`'s valid arm). It is small where the full descriptor is not, and it is what any integrating agent needs. Filed against `pipelex-mcp` as `L-260830-e8b2e0`, discovered from this item; when it lands, the heuristic and its question are deleted. +**Amended 2026-09-06 — the gap this section originally worked around is closed.** As written on 2026-08-30, the signature reached the model through the `explicit: true` inputs template and a bundle read for a files source, and not at all for a `method_ref` or `method_id` source, because `mthds_validate` carried `main_pipe_ref` and `pipe_io_contracts` only on the view-only `_meta` channel; the section carried a by-elimination heuristic for the output concept and one question to confirm it, and filed `L-260830-e8b2e0` against `pipelex-mcp` for the real fix. That follow-up landed: a valid `mthds_validate` verdict now carries `structuredContent.main_pipe` — the main pipe's namespaced ref, each declared input with its fully-qualified `concept_ref`, multiplicity and `required` flag, and the produced concept with its multiplicity and `optional` flag — on the local workshop as much as on the hosted console, surviving a pending-signature verdict and `include_graph: false` (`pipelex-mcp/src/capabilities/validate.ts`, `mainPipeSignatureOf`; `pipelex-mcp/SPEC.md` → "Validation Scope"). So: + +- **Every selector reads the signature from the step-2 verdict** — files, `method_ref` and `method_id` alike — and the heuristic and its question are **never written**. `L-260831-b67e18`, filed to delete them, closes with the skill's first release. +- **`main_pipe` is absent only when the bundle declares no main pipe**, and it is omitted whole rather than partially. That path asks which pipe to integrate, reads its inputs from `mthds_inputs_template` with `explicit: true` (§4.9), and its output from the bundle for a files source; a by-ref or by-id method with no main pipe and no way to read its output declaration is reported as not integrable as it stands. ### 4.4 The write arm is mandatory, and the model never writes an artifact (brief finding) @@ -173,9 +173,11 @@ The ts-zod emitter projects a non-required field as `.optional()` (`pipelex/pipe **Decision: until `L-260820-ee327d` lands, the TypeScript call site parses through a shared `wireOutput(results, schema)` helper, copied once per project from the skill's `references/`, that drops a `null` only where the concept's own zod schema says the field is optional with no default, descends declared objects and arrays, and passes anything opaque (`z.unknown()`, `z.record()` keys, unions) through untouched.** A blind deep null-strip is rejected for the reason the starter's design note gives: inside a `z.record()` a `null` is data. The helper's header names the item it waits on and says the file is deleted, not maintained, when the emitter projects `.nullish()`; the plan carries a checkpoint to re-check the item before shipping, since it may land first — in which case the helper is never written and the call site parses `main_stuff` directly. +**Status 2026-09-06:** the emitter fix merged to `pipelex` `dev` as `pipelex#1177` — a non-required field now emits `.nullish()`, a defaulted one `.nullable().default(…)`. It reaches the skill's users only through a `pipelex` release deployed to the hosted codegen route, which is what the plan's pre-flight re-check waits for. Once that deploy is live the helper is not merely unnecessary, it is **lossy** (it strips legitimate nulls inside opaque fields such as `native.JSON`'s map), so from then on the skill must never write it into a project. + ### 4.9 The template shape — `explicit: true`, the plugin's one exception -The plugin's standing rule pins every `mthds_inputs_template` call to `explicit: false` (`docs/decisions.md`, "the light template stays pinned"), because the three existing call sites only show or key-compare the template. This skill is different in kind: it needs **each input's declared concept ref** to write a typed parameter, and the light shape carries values without concepts. **Decision: `pipelex-integrate` passes `explicit: true`, and the decisions record gains a sentence naming it as the exception and why.** The rule's rationale (the envelope is an authoring aid the other skills do not need) is unchanged; a fourth call site with a need for concept identity is exactly the case the rule said would justify it. +The plugin's standing rule pins every `mthds_inputs_template` call to `explicit: false` (`docs/decisions.md`, "the light template stays pinned"), because the three existing call sites only show or key-compare the template. This skill is different in kind: it needs **each input's declared concept ref** to write a typed parameter, and the light shape carries values without concepts. **Decision, as amended 2026-09-06: `pipelex-integrate` passes `explicit: true` only on its fallback path — when the validate verdict carries no `main_pipe` and the user has named the pipe to integrate — and the decisions record names that as the exception and why.** On the ordinary path the signature comes from `structuredContent.main_pipe` (§4.3) and no template call is made at all. The rule's rationale (the envelope is an authoring aid the other skills do not need) is unchanged; a call site that needs concept identity and has no other channel to it is exactly the case the rule said would justify it. ### 4.10 The second invocation — refresh mode (brief Q5) @@ -197,13 +199,22 @@ The description is written to trigger on the codebase phrasings — "use this me Its place: it is the skill after the method is done. `pipelex-design` and `pipelex-inputs` close by pointing at it when there is a codebase in the workspace (§7). +### 4.12 A project that owns a codegen harness — defer to it (amendment, 2026-09-06) + +**Decision: when the project already owns a codegen harness, the skill uses that harness for generation and follows the project's own documentation for the fan-out; the workshop's write arm is for projects that have none.** A project made from `pipelex-starter-js` or `pipelex-starter-python` — or one that adopted their pattern — regenerates every method in one place (`npm run codegen` / `make codegen`), checks them in one place (`codegen:check` / `make codegen-check`, already inside its aggregate gate), records the sidecar it invented (`sources.json` with its `derived` map, on the JS side), and on the JS side scaffolds a catalog or published method end to end with `make add-method` (manifest, tree, action trio, narrower, form, tab). Writing the plugin's generic tree beside that would leave the project with two regeneration paths, two sidecar dialects and one `contracts.ts` the workshop never emits. The personas decision (`wip/devx/codegen-personas-and-trust-chain.md`) draws the same line: the *mechanism* may move into the SDK, but the *policy* — layout, env, CI wiring — belongs to the consumer, and a starter-derived project is a consumer with a policy. + +The signals are cheap: a `codegen` script in `package.json` or a `codegen` Makefile target; a `sources.json` carrying a `derived` map; a `docs/codegen.md` or `docs/add-method.md`; a `methods/` directory beside a `src/generated/` or `/generated/` tree. Either of the first two decides; the rest corroborate. On such a project the skill: places the method where the project keeps them (`methods//main.mthds`, or the project's manifest form for a catalog or published method); runs the project's generator — `make add-method METHOD=…` when the project has it and the method is remote, its `codegen` script otherwise; writes the call site the way the project's docs and existing methods do rather than the shape of §4.1; runs the project's own aggregate gate as the verification; and writes no sidecar of its own, because the harness has one. Refresh mode on such a project is the project's `codegen` script, and the skill says so instead of calling `mthds_codegen`. + +**The harness owns the layout and the check; its generator is preferred, not mandatory.** When the harness's generator cannot run — today the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on (`L-260906-a2cd5b`) — the workshop writes into the harness's own layout instead, which is byte-identical there (same engine, same stamps, same lock, and the Python starter keeps no sidecar), and the report says the project's `make codegen` is the refresh once its prerequisite is met. What the skill never does is write a second layout beside the first. The Python starter's missing `add-method` is `L-260906-aa5083`; until it lands, a remote method on a Python starter-derived project is placed and generated by the skill following the project's docs, which is the same work `add-method` would do. + ## 5. Detecting the project The rule: **a cheap, reliable signal decides; an inconclusive one asks one question; the Python audience is never guessed when `pipelex` is a dependency.** The language-specific detail lives in the skill's `references/typescript.md` and `references/python.md`; this is the contract those files implement. | Question | Signals, in order | When inconclusive | | --- | --- | --- | -| **Which project?** | the user named it; else the nearest `package.json` / `pyproject.toml` (or `setup.py`, `requirements.txt`) above the working area; a workspace with several (a monorepo, a full-stack repo) | ask which app; never pick one | +| **Which project?** | the user named it; else the nearest `package.json` / `pyproject.toml` (or `setup.py`, `requirements.txt`) above the working area; a workspace with several (a monorepo, a full-stack repo) | ask which app; never pick one — and when there is **no project at all**, offer `/pipelex-scaffold` (§7) rather than asking where to integrate | +| **Owns a codegen harness?** | a `codegen` script in `package.json` or a `codegen` Makefile target; a `sources.json` with a `derived` map; `docs/codegen.md` / `docs/add-method.md`; `methods/` beside a generated tree | either of the first two decides (§4.12); the rest only corroborate | | **Language → target** | `package.json` → `ts-zod` (a JavaScript project with no TypeScript build — no `tsconfig.json`, no bundler or runtime that strips types — is asked, because `types.ts` needs one); `pyproject.toml` → Python | both present at one root: ask | | **Python audience → target** | `pipelex` **not** among the project's dependencies → `python-pydantic`, no question (`python-structures` imports the runtime and would not even load); `pipelex` present → `python-structures` if `@pipe_func` or `StructuredContent` appear in the code, else one question with `python-structures` offered first | the user's explicit request wins over all of this | | **Package manager** | the lockfile: `package-lock.json` → npm, `pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lock*` → bun; `uv.lock` → uv, `poetry.lock` → poetry, `Pipfile.lock` → pipenv | none: npm / `pip install` into the active environment, stated | @@ -240,6 +251,7 @@ Three small edits to the existing skills, each one sentence or one step: - **`pipelex-edit` Step 7 (Report) and `pipelex-design`'s re-entry delivery:** after a successful edit to a bundle, look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file that changed; for each, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. This is the plugin-native forgetting-guard, and it is language-agnostic — it is what gives the Python side a guard at all until `L-260830-4e43cd`. - **`pipelex-design`'s delivery step 4 and `pipelex-inputs`' closing report:** one line — when the workspace holds a `package.json` or `pyproject.toml`, `/pipelex-integrate` wires the method into that code. - **The `MCP_SKILLS` tuple in the tests, the README's skill list and MCP tool list, `CLAUDE.md`'s "Key dependency", and `docs/decisions.md`** gain the skill and `mthds_codegen`, exactly as `mthds_prepare_inputs` was added. +- **`pipelex-scaffold`** (the greenfield sibling, [`scaffold-design.md`](scaffold-design.md)) hands every project it creates to this skill; in return, step 1 of §3 offers `/pipelex-scaffold` when it finds no project at all — no `package.json`, no `pyproject.toml` above the working area — instead of asking where to integrate. `pipelex-design`'s delivery step gains the same fork: a codebase in the workspace → `/pipelex-integrate`; none → `/pipelex-scaffold`. ## 8. Follow-ups, filed or linked @@ -250,6 +262,9 @@ Three small edits to the existing skills, each one sentence or one step: | `L-260830-4e43cd` | `pipelex-sdk-python` | discovered-from (already) | the Python offline check; when it lands, the Python branch gains its gate and the asymmetry paragraph goes | | `L-260830-e8b2e0` | `pipelex-mcp` | discovered-from this item (filed 2026-08-30) | promote a compact main-pipe signature (`main_pipe_ref`, inputs → concept refs, output concept + multiplicity) from `_meta` into `structuredContent`, so a by-ref / by-id integration is typed exactly instead of by the §4.3 heuristic | | `L-260829-563e9e` | workspace | related, informational | the pipe-selector campaign adds `pipe_ref` to the run request; the sidecar already records the qualified ref, and the call site moves from `pipe_code` to `pipe_ref` when the SDKs take it | +| `L-260831-b67e18` | `pipelex-plugins` | closes with this skill's first release | asks for the §4.3 heuristic to be deleted; it is never written, so the release that ships the skill is its evidence | +| `L-260906-a2cd5b`, `L-260906-aa5083` | `pipelex-starter-python` | discovered-from this item (filed 2026-09-06) | the Python starter's harness shells out to a runtime CLI it does not depend on, and lacks the JS starter's `add-method`; §4.12 defers to the harness as it stands until they land | +| `L-260906-84bb41` | workspace | related, decision (filed 2026-09-06) | whether a `create-pipelex-app` CLI front door is still wanted beside `pipelex-scaffold` for users who do not work through an agent | ## 9. Out of scope, stated so it is not rediscovered @@ -261,11 +276,12 @@ No change to `pipelex-mcp` from this repo (the follow-up above is filed, not wor | --- | --- | --- | | **1 — Scope of the call site** | One complete typed callable module per method plus at most two shared helpers; no tests, routes, or UI (§4.1) | Yes, as written — 2026-08-30 | | **2 — Run source** | The call site runs from the source the types came from; `method_id` allowed with a one-line warning; never mixed (§4.2) | Yes, as written — 2026-08-30 | -| **3 — No contracts artifact** | Signature from `explicit: true` template + bundle read; heuristic for by-ref / by-id until the `pipelex-mcp` follow-up lands (§4.3) | Yes, as written — 2026-08-30 | +| **3 — No contracts artifact** | Signature from the validate verdict's `main_pipe` for every selector; the `explicit: true` template only when no main pipe is declared; the by-ref / by-id heuristic is never written (§4.3) | Yes, as written — 2026-08-30; amended 2026-09-06 after the `pipelex-mcp` follow-up landed | | **4 — Write arm only** | Always `output_dir`, relative to the harness's launch directory; never ride content; stop on containment escape (§4.4) | Yes, as written — 2026-08-30 | | **5 — Gate asymmetry** | TypeScript: reference `codegen-check.mjs` into the existing gate; Python: no gate, sidecar plus refresh, stated in the report (§4.5) | Yes, as written — 2026-08-30 | | **6 — Sidecar** | `sources.json`, starter-compatible `sources` map plus `generator` / `method` / `target` / `pipe` (§4.6) | Yes, as written — 2026-08-30 | | **7 — Wire-null helper** | Shared schema-guided `wireOutput` per TypeScript project until `L-260820-ee327d`, with a pre-ship re-check (§4.8) | Yes, as written — 2026-08-30 | -| **8 — `explicit: true`** | The plugin's one exception to the light-template pin, recorded in `docs/decisions.md` (§4.9) | Yes, as written — 2026-08-30 | +| **8 — `explicit: true`** | The plugin's one exception to the light-template pin, now confined to the no-main-pipe fallback path, recorded in `docs/decisions.md` (§4.9) | Yes, as written — 2026-08-30; narrowed 2026-09-06 | | **9 — Name** | `pipelex-integrate`, model-invocable, codebase-phrasing triggers (§4.11) | Yes, as written — 2026-08-30 | -| **10 — Family wiring** | Staleness notice in `pipelex-edit` / `pipelex-design`; one-line hand-off in `pipelex-design` / `pipelex-inputs` (§7) | Yes, as written — 2026-08-30 | +| **10 — Family wiring** | Staleness notice in `pipelex-edit` / `pipelex-design`; one-line hand-off in `pipelex-design` / `pipelex-inputs`; the `pipelex-scaffold` fork on "no project" (§7) | Yes, as written — 2026-08-30; scaffold fork added 2026-09-06 | +| **11 — Harness deference** | A project that owns a codegen harness keeps it: its generator, layout, check and docs drive the integration; the write arm is for projects with none, and never writes a second layout beside the first (§4.12) | Yes — 2026-09-06 | diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index e193332..1055e95 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -5,17 +5,17 @@ item: L-260830-344594 # Plan — `pipelex-integrate`: the implementation tracker -**Written 2026-08-30** as the execution tracker for [`design.md`](design.md). It schedules; it does not re-argue — when this file and the design disagree, the design wins unless the disagreement is logged under "Deviations" below. Section references (`§N`) are to the design. Ledger item `L-260830-344594`; the phases name the follow-up items they wait on or file. +**Written 2026-08-30** as the execution tracker for [`design.md`](design.md); **amended 2026-09-06** to carry the campaign's second skill, `pipelex-scaffold`, whose design is [`scaffold-design.md`](scaffold-design.md). It schedules; it does not re-argue — when this file and a design disagree, the design wins unless the disagreement is logged under "Deviations" below. Section references (`§N`) are to `design.md`; `S§N` is `scaffold-design.md`. Ledger items `L-260830-344594` (integrate) and `L-260906-8ac105` (scaffold); the phases name the follow-up items they wait on or file. -**Status: active** since 2026-08-30, when the ten decision boxes of `design.md` were ratified as written (Phase 0). Work proceeds from Phase 1. +**Status: active** since 2026-08-30, when the ten decision boxes of `design.md` were ratified as written (Phase 0). The 2026-09-06 amendments to `design.md` (boxes 3, 8 and 10 reworded, box 11 added) and `scaffold-design.md`'s boxes A–I were ratified in that session (Phase 0b); Phases 1 and 1b are both open. ## How to work a phase - `ledger claim L-260830-344594` before touching code; renew the claim once you are on the working branch. -- The working branch is `feature/Codegen` (already created for this item); the PR targets `dev` and its body carries `Closes L-260830-344594`. A merged PR is landed with `/ledger-land`. -- **This checkout is shared with other sessions.** Stage the files you touched explicitly (`git add `), never `git add -A`; the branch already carries uncommitted work on `pipelex-design` / `pipelex-edit` from another piece of work, and a phase here must not sweep it into its commit. Never run a formatter over files you did not author. -- Templates are the source of truth: edit `templates/skills/…/*.j2` and `skills/pipelex-integrate/references/*`, then `make build`; never edit `pipelex*/` outputs. Before pushing: `make agent-check` and `make agent-test`. -- `mthds_codegen` is **unreleased in `@pipelex/mcp`** at writing. Development and dogfood run against the local `../pipelex-mcp` checkout through the repo skill `/pipelex-mcp-source`; **switch back to `@latest` before any commit** and let that skill confirm no dev switch leaked into `targets/defaults.toml`. +- The working branch is `feature/Codegen`, in the worktree `_pipelex-plugins--codegen`; both skills ship on it. The PR targets `dev` and its body carries `Closes L-260830-344594` and `Closes L-260906-8ac105`. A merged PR is landed with `/ledger-land`. +- **This checkout may be shared with other sessions.** Stage the files you touched explicitly (`git add `), never `git add -A`, so a phase never sweeps another session's work into its commit. Never run a formatter over files you did not author. +- Templates are the source of truth: edit `templates/skills/…/*.j2`, `skills/pipelex-integrate/references/*` and `skills/pipelex-scaffold/references/*`, then `make build`; never edit `pipelex*/` outputs. Before pushing: `make agent-check` and `make agent-test`. +- `mthds_codegen` shipped in `@pipelex/mcp` 0.13.0, but the validate verdict's `main_pipe` signature is still unreleased there at the 2026-09-06 pause. Development and dogfood run against the local `../pipelex-mcp` checkout through the repo skill `/pipelex-mcp-source`; **switch back to `@latest` before any commit** and let that skill confirm no dev switch leaked into `targets/defaults.toml`. - Version discipline: everything accumulates under `[Unreleased]` in `CHANGELOG.md`; the release phase cuts the heading and bumps the version through `/release`. - At each checkpoint: tick the boxes, record the SHAs and versions outcomes landed in (never live git state), reconcile deviations into the later phases, and leave this file cold-start ready. @@ -23,7 +23,7 @@ item: L-260830-344594 - **The write arm is the only arm the skill uses** (§4.4). If a dogfood run ever tempts a "just write the bytes from the response" fallback, that is a bug in the run, not a feature to add. - **The generated tree is never opened for editing, formatted, or linted** — by the skill, and by the session working this plan. A dogfood run that reformats a generated file has invalidated its own verdict; regenerate and start the scenario again. -- **Two upstream items can land during this work and each deletes a piece of it.** `L-260820-ee327d` (ts-zod `.nullish()`) deletes the wire-output helper (§4.8); `L-260830-4e43cd` (Python offline check) deletes the Python asymmetry paragraph (§4.5). Phases 1 and 4 each carry a box to re-check both before proceeding, and a landed item is recorded under "Decisions taken along the way" with what was removed. +- **Two upstream items can land during this work and each deletes a piece of it.** `L-260820-ee327d` (ts-zod `.nullish()`) deletes the wire-output helper (§4.8) — its fix merged to `pipelex` `dev` as `pipelex#1177` on 2026-09-01, so the question is now whether a `pipelex` release carrying it is deployed to the hosted codegen route; `L-260830-4e43cd` (Python offline check) deletes the Python asymmetry paragraph (§4.5). Phases 1 and 4 each carry a box to re-check both before proceeding, and a landed item is recorded under "Decisions taken along the way" with what was removed. A third, `L-260830-e8b2e0` (the main-pipe signature in the validate verdict), already landed before Phase 1 began: the §4.3 heuristic is never written, and `L-260831-b67e18` closes with the first release. - **Ledger ids never appear in user-facing skill text or reports.** They belong in this tracker, the design, and `docs/decisions.md`; the skill's report to a user says "the Python SDK has no offline drift check yet", never the item that tracks it. ## Phase 0 — ratify, file, link @@ -38,60 +38,104 @@ No code. Owner: the session that reads the design with Louis. - [x] `ledger ref L-260830-344594` attached `plan:pipelex-plugins/wip/pipelex-integrate/design.md` and `plan:pipelex-plugins/wip/pipelex-integrate/plan.md` beside the existing brief ref — done 2026-08-30. - [x] `ledger validate`, then `ledger commit` — done 2026-08-30 for the filing above; re-validated after the ratification edits (the ratification changed only these two documents, not the ledger). +## Phase 0b — the second design session (2026-09-06) + +No code. Owner: the session that widened the campaign with Louis. + +- [x] Research the SDKs, the starters and the codegen contract as they stand, and the workspace's scaffolding and personas documents — done 2026-09-06; the findings are in the decisions log below. +- [x] Decide the shape with Louis: two skills rather than one; a from-scratch project comes from a starter or the ecosystem's initializer, never from a cookiecutter or copier template; `pipelex-integrate` defers to a project-owned codegen harness; the greenfield skill is `pipelex-scaffold` — done 2026-09-06, all four as recommended. +- [x] Amend `design.md`: §3 steps 3, 4 and 6; §4.3 (signature from `main_pipe`, heuristic never written); §4.8 (status of the emitter fix); §4.9 (`explicit: true` confined to the fallback); new §4.12 (harness deference); §5 (two rows); §7 (scaffold fork); §8 (new follow-ups); boxes 3, 8, 10 reworded, box 11 added — done 2026-09-06. +- [x] File the items: `L-260906-8ac105` (the scaffold skill, this repo), `L-260906-a2cd5b` and `L-260906-aa5083` (Python starter parity), `L-260906-84bb41` (the `create-pipelex-app` decision) — done 2026-09-06. +- [x] Write `scaffold-design.md` as a draft with its own decision boxes — done 2026-09-06. +- [x] Walk `scaffold-design.md`'s boxes A–I with Louis; record each ruling in its "Ratified?" column with the date; flip the document to `status: active` in the same change — done 2026-09-06: all nine ratified as written, no amendments. + ## Phase 1 — the skill template and its references Owner: `pipelex-plugins`. Everything in this phase renders into all three targets; nothing in it is platform-specific except the `allowed-tools` frontmatter and the MCP-absent message, which the shared patterns already handle. **Pre-flight** -- [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` (`ledger show`). If either has closed and shipped in the hosted engine / the Python SDK, strike the corresponding piece below before writing it and log the deviation. -- [ ] Confirm the static-asset mechanism works end to end before relying on it: `scripts/gen_skill_docs.py` → `setup_static_assets` copies `skills//references/` into every target; `scripts/check.py` → `check_stale_references` resolves `references/…` links from a rendered `SKILL.md`; `check_no_templates_in_output` tolerates `.mjs` / `.ts` files under `skills/`. This is the **first skill in the repo to ship references**, so a root `skills/` directory does not exist yet; create it and note in `docs/build-targets.md` (Phase 2) that the mechanism is now in use. +- [x] Re-check `L-260820-ee327d` and `L-260830-4e43cd` (`ledger show`). For the first, the fix is merged to `pipelex` `dev` (`pipelex#1177`): the check is whether a release carrying it is deployed to the hosted codegen route — if so, strike `wire-output.ts` below and never write the helper (it is lossy once the emitter is fixed, §4.8). For the second, if the Python SDK ships the offline check, strike the asymmetry sentence. Log either as a deviation. +- [x] Confirm the static-asset mechanism works end to end before relying on it: `scripts/gen_skill_docs.py` → `setup_static_assets` copies `skills//references/` into every target; `scripts/check.py` → `check_stale_references` resolves `references/…` links from a rendered `SKILL.md`; `check_no_templates_in_output` tolerates `.mjs` / `.ts` files under `skills/`. This is the **first skill in the repo to ship references**, so a root `skills/` directory does not exist yet; create it and note in `docs/build-targets.md` (Phase 2) that the mechanism is now in use. **The template — `templates/skills/pipelex-integrate/SKILL.md.j2`** -- [ ] Frontmatter: `name`, the description from §4.11 (codebase-phrasing triggers, silent on authoring phrasings), the shared `frontmatter.md.j2` include, and on Claude the `allowed-tools` entries for `mthds_codegen`, `mthds_inputs_template`, `mthds_validate`, `mthds_list_methods`. No `disable-model-invocation`. -- [ ] "Requirements — the Pipelex MCP tools": `mthds_codegen`, `mthds_inputs_template` and `mthds_validate` required with the plugin's standard MCP-absent STOP message (copy the exact conditional block from `pipelex-inputs`, so `TestSkillFailureDiscipline.test_absent_tools_stop_message_matches_platform` passes on every target); `mthds_list_methods` soft. The `config`-class stop, with the **403 feature-gate wording** from §6 (a 403 is not a key problem). -- [ ] Mode selection: automatic default; the interactive signals; the one-question rule from §5. -- [ ] The procedure of §3, as numbered steps, each naming its MCP call, its arguments (`explicit: true` on the template call — with the sentence saying this is the deliberate exception to the plugin's light-template pin), and its verdict branches. -- [ ] The `output_dir` rule of §4.4 spelled out for the model: compute it relative to the session's initial working directory; never absolute; never ride content; the relaunch instruction on a containment escape. -- [ ] The exclusions-before-generation ordering (§3 step 5), stated as a rule with its reason. -- [ ] The orphan rule of §4.7 in the tool's own wording; never delete. -- [ ] The sidecar section: the exact `sources.json` shape of §4.6, how hashes are computed (`shasum -a 256` / `sha256sum` / `hashlib`, raw bytes), paths relative to the project root. -- [ ] The call-site section (§4.1): what one module contains, the two shared helpers, the input-type mapping table from concept ref to language type (Text → `string`/`str`, Number → `number`/`float | int`, YesNo → `boolean`/`bool`, Date → ISO string, Image/Document → `{ url }`, structured → the generated type, `[]` → arrays, `?` → optional), the `main_stuff` narrowing, the `prepareInputs` pointer for file-bearing callers, the sync-wrapper rule for synchronous Python projects. Language detail is delegated to the two reference files. -- [ ] Refresh mode as its own section (§4.10): the taken / re-derived / left-alone table, the fingerprint comparison and the restamp-only case, the "call site edited only if it no longer type-checks or the `pipe` record moved" rule. -- [ ] The verification step (§3 step 11) and the report (§3 step 12), including the Python asymmetry sentence. -- [ ] A failure table condensed from §6. -- [ ] `## Reference`: links to `references/typescript.md`, `references/python.md`, and the two shared language references. +- [x] Frontmatter: `name`, the description from §4.11 (codebase-phrasing triggers, silent on authoring phrasings), the shared `frontmatter.md.j2` include, and on Claude the `allowed-tools` entries for `mthds_codegen`, `mthds_inputs_template`, `mthds_validate`, `mthds_list_methods`. No `disable-model-invocation`. +- [x] "Requirements — the Pipelex MCP tools": `mthds_codegen`, `mthds_inputs_template` and `mthds_validate` required with the plugin's standard MCP-absent STOP message (copy the exact conditional block from `pipelex-inputs`, so `TestSkillFailureDiscipline.test_absent_tools_stop_message_matches_platform` passes on every target); `mthds_list_methods` soft. The `config`-class stop, with the **403 feature-gate wording** from §6 (a 403 is not a key problem). +- [x] Mode selection: automatic default; the interactive signals; the one-question rule from §5. +- [x] The procedure of §3, as numbered steps, each naming its MCP call, its arguments and its verdict branches: the signature is read from the validate verdict's `structuredContent.main_pipe` (§4.3); `mthds_inputs_template` with `explicit: true` appears only on the no-main-pipe fallback, with the sentence naming it as the deliberate exception to the plugin's light-template pin (§4.9); the generator choice — the project's harness or the write arm — is stated in the one-line announcement (§3 step 4). +- [x] The harness-deference section (§4.12): the detection signals and which of them decide; what a harness project skips (exclusions, sidecar, dependencies, gate) and what it follows instead (the project's `codegen` script, `add-method`, its docs, its aggregate gate); the write-arm-into-the-harness's-layout fallback when the harness's generator cannot run; refresh mode on a harness project; the rule that no second layout is ever written beside the first. +- [x] The `output_dir` rule of §4.4 spelled out for the model: compute it relative to the session's initial working directory; never absolute; never ride content; the relaunch instruction on a containment escape. +- [x] The exclusions-before-generation ordering (§3 step 5), stated as a rule with its reason. +- [x] The orphan rule of §4.7 in the tool's own wording; never delete. +- [x] The sidecar section: the exact `sources.json` shape of §4.6, how hashes are computed (`shasum -a 256` / `sha256sum` / `hashlib`, raw bytes), paths relative to the project root. +- [x] The call-site section (§4.1): what one module contains, the two shared helpers, the input-type mapping table from concept ref to language type (Text → `string`/`str`, Number → `number`/`float | int`, YesNo → `boolean`/`bool`, Date → ISO string, Image/Document → `{ url }`, structured → the generated type, `[]` → arrays, `?` → optional), the `main_stuff` narrowing, the `prepareInputs` pointer for file-bearing callers, the sync-wrapper rule for synchronous Python projects. Language detail is delegated to the two reference files. +- [x] Refresh mode as its own section (§4.10): the taken / re-derived / left-alone table, the fingerprint comparison and the restamp-only case, the "call site edited only if it no longer type-checks or the `pipe` record moved" rule. +- [x] The verification step (§3 step 11) and the report (§3 step 12), including the Python asymmetry sentence. +- [x] A failure table condensed from §6. +- [x] `## Reference`: links to `references/typescript.md`, `references/python.md`, and the two shared language references. **The references — `skills/pipelex-integrate/references/`** -- [ ] `typescript.md`: the detection signals of §5 for a TypeScript project (project root, TS-capable build, package manager, generated root, Prettier / ESLint flat and legacy / Biome exclusion edits with the exact config keys, `tsconfig` coverage, aggregate gate, call-site location, `.gitignore`); the call-site module template with the `getPipelexClient` helper and the `wireOutput` import; the `codegen:check` npm script and how to append it to `check` / a Makefile / a workflow step. -- [ ] `python.md`: the same for Python (import package discovery, `python-pydantic` vs `python-structures` per §5, uv / poetry / pipenv / pip, `[tool.ruff]` `exclude` and `extend-exclude`, Black, isort, pyright / mypy coverage, `__init__.py` creation for the generated package and each method subpackage, setuptools `packages` / `package-data` when the project is packaged, the async call-site template plus the sync wrapper, the `pipelex codegen check` wiring for the `python-structures` audience only, the asymmetry sentence for everyone else). -- [ ] `codegen-check.mjs` (§4.5): plain ESM, Node builtins + `@pipelex/sdk` only; takes generated directories as arguments; per directory reads `codegen.lock` from disk, walks recursively (pruning `node_modules`, `.git`, `dist`, `build`, `.next`), filters with `isStampableArtifactPath`, decodes strictly, runs `runCodegenCheck`, prints drifts by category; then reads `sources.json` and compares each `sources` hash against the file on disk, reporting `stale-source` with the "run `/pipelex-integrate` to refresh" remedy; exit `0` / `1` / `2` with the precedence no-verdict > drift > current; output through `process.stdout` / `process.stderr`. Header comment names what it is and that `@pipelex/sdk` upstreaming retires it. -- [ ] `wire-output.ts` (§4.8, **skip if `L-260820-ee327d` has landed**): `wireOutput(results, schema)` and the schema-guided `dropWireNulls`, trimmed from `pipelex-starter-js/src/lib/wireOutput.ts` — objects, arrays, `z.lazy`, optional-without-default only; opaque schemas passed through; a depth cap; no `server-only` import, no Next-specific error types. Header comment states it is a workaround with an expiry and what deletes it. +- [x] `typescript.md`: the detection signals of §5 for a TypeScript project (project root, TS-capable build, package manager, generated root, Prettier / ESLint flat and legacy / Biome exclusion edits with the exact config keys, `tsconfig` coverage, aggregate gate, call-site location, `.gitignore`); the call-site module template with the `getPipelexClient` helper and the `wireOutput` import; the `codegen:check` npm script and how to append it to `check` / a Makefile / a workflow step. +- [x] `python.md`: the same for Python (import package discovery, `python-pydantic` vs `python-structures` per §5, uv / poetry / pipenv / pip, `[tool.ruff]` `exclude` and `extend-exclude`, Black, isort, pyright / mypy coverage, `__init__.py` creation for the generated package and each method subpackage, setuptools `packages` / `package-data` when the project is packaged, the async call-site template plus the sync wrapper, the `pipelex codegen check` wiring for the `python-structures` audience only, the asymmetry sentence for everyone else). +- [x] Both language references gain a "the project owns a codegen harness" section (§4.12): the starter's scripts and Makefile targets by name, the docs to read before writing the fan-out (`docs/codegen.md`, `docs/add-method.md`, the README's "swap in your own pipeline"), where a starter keeps its methods and its manifest form for a remote method, and — Python — the honest sentence about the `pipelex` CLI prerequisite of `make codegen` with the write-arm fallback into `/generated//`. +- [x] `codegen-check.mjs` (§4.5): plain ESM, Node builtins + `@pipelex/sdk` only; takes generated directories as arguments; per directory reads `codegen.lock` from disk, walks recursively (pruning `node_modules`, `.git`, `dist`, `build`, `.next`), filters with `isStampableArtifactPath`, decodes strictly, runs `runCodegenCheck`, prints drifts by category; then reads `sources.json` and compares each `sources` hash against the file on disk, reporting `stale-source` with the "run `/pipelex-integrate` to refresh" remedy; exit `0` / `1` / `2` with the precedence no-verdict > drift > current; output through `process.stdout` / `process.stderr`. Header comment names what it is and that `@pipelex/sdk` upstreaming retires it. +- [x] `wire-output.ts` (§4.8, **skip if `L-260820-ee327d` has landed**): `wireOutput(results, schema)` and the schema-guided `dropWireNulls`, trimmed from `pipelex-starter-js/src/lib/wireOutput.ts` — objects, arrays, `z.lazy`, optional-without-default only; opaque schemas passed through; a depth cap; no `server-only` import, no Next-specific error types. Header comment states it is a workaround with an expiry and what deletes it. **Build and tests** -- [ ] `make build`; confirm `pipelex/`, `pipelex-codex/`, `pipelex-vibe/` each carry `skills/pipelex-integrate/SKILL.md` and the `references/` directory verbatim. -- [ ] `tests/unit/test_gen_skill_docs.py`: add `"pipelex-integrate"` to `TestSkillFailureDiscipline.MCP_SKILLS`. -- [ ] New `TestPipelexIntegrateDiscipline` pinning the load-bearing sentences in the real template, rendered on all three targets: `output_dir` is always passed; content is never ridden; orphans are never deleted; one directory per method; generated files are never edited or formatted; exclusions precede generation; `explicit: true` on the template call; the `method_id` warning; refresh mode leaves the call site alone unless the types moved; the 403 wording. Plus one test that the references land in every target's output. -- [ ] `make agent-check`, `make agent-test`. +- [x] `make build`; confirm `pipelex/`, `pipelex-codex/`, `pipelex-vibe/` each carry `skills/pipelex-integrate/SKILL.md` and the `references/` directory verbatim. +- [x] `tests/unit/test_gen_skill_docs.py`: add `"pipelex-integrate"` to `TestSkillFailureDiscipline.MCP_SKILLS`. +- [x] New `TestPipelexIntegrateDiscipline` pinning the load-bearing sentences in the real template, rendered on all three targets: `output_dir` is always passed; content is never ridden; orphans are never deleted; one directory per method; generated files are never edited or formatted; exclusions precede generation; the signature is read from `main_pipe` and `explicit: true` appears only on the fallback; a harness project keeps its harness and no second layout is written; the `method_id` warning; refresh mode leaves the call site alone unless the types moved; the 403 wording. Plus one test that the references land in every target's output. +- [x] `make agent-check`, `make agent-test`. **CHECKPOINT 1** — the template and references render on every target and the tests pin their rules. Record here: the commit SHA, what was struck because an upstream item landed, and anything the template could not express without a reference file. +**Reached 2026-09-06 (uncommitted at the session pause — the SHA is recorded when the work is committed).** Struck: `wire-output.ts` (the emitter fix shipped in pipelex v0.56.0, see the pre-flight entry below). The pre-flight note above calling this "the first skill in the repo to ship references" was stale when ticked — `pipelex-design` and `pipelex-synthetic-inputs` already did; the mechanism (`setup_static_assets`, `static_asset_mismatches`) was confirmed to copy non-Markdown files too, which is what `codegen-check.mjs` needs. The discipline tests live in their own modules, `tests/unit/test_pipelex_integrate_skill.py` and `tests/unit/test_pipelex_scaffold_skill.py` (one class each), rather than in `test_gen_skill_docs.py`, which only gained `pipelex-integrate` in `MCP_SKILLS`. + +## Phase 1b — the `pipelex-scaffold` template and its references + +Owner: `pipelex-plugins`. **Gate:** `scaffold-design.md`'s boxes ratified (Phase 0b). MCP-free: no `allowed-tools` MCP entries, no MCP-absent message, and the skill stays out of `MCP_SKILLS`; the template to model is `templates/skills/pipelex-synthetic-inputs/SKILL.md.j2` (MCP-free, references-bearing, a stop posture on a missing toolchain). + +**The template — `templates/skills/pipelex-scaffold/SKILL.md.j2`** + +- [x] Frontmatter: `name`, the description from S§8 (greenfield phrasings, silent on the integrate phrasings, the fresh-clone shortcut named), the shared `frontmatter.md.j2` include. No `disable-model-invocation`. +- [x] The branch table of S§2, including "here" for an empty working directory and the un-bootstrapped-clone shortcut. +- [x] Branch A as numbered steps (S§3): the prerequisites and the stop posture; the two acquisition forms with the exact commands, the fresh-history rationale and the `gh` confirmation; the one pristine commit and why (`git mv`, reviewable diff); the delegation to the clone's own `bootstrap` SKILL.md with the inputs passed through and every command run from inside the project; the env file and the key rule; the hand-off to `/pipelex-integrate` with the harness note. +- [x] Branch B as numbered steps (S§4): initializer first, non-interactive flags, the hand-the-command-to-the-user fallback, the language defaults, git init when the initializer did not, the pristine commit, the two-line `.env.example`, nothing else Pipelex-shaped. +- [x] The report (S§5) with the session note as its own line. +- [x] Mode (S§6) and a failure table condensed from S§7, including the "this is the template's own checkout" stop. +- [x] `## Reference`: links to `references/starters.md` and `references/initializers.md`. + +**The references — `skills/pipelex-scaffold/references/`** + +- [x] `starters.md`: the two starters side by side — what each brings and what it costs; the prerequisite floors and where each starter states them (`package.json` `engines`, `pyproject.toml` `requires-python`); the clone and `gh` commands; the env file name per starter; where the bootstrap skill lives and what it will ask; the demos and where each README's removal checklist is; the JS starter's `make add-method` and the Python starter's `make codegen` prerequisite (`L-260906-a2cd5b`), stated as the project's own, for the integrate hand-off. +- [x] `initializers.md`: per language, the minimal default and the common frameworks with their non-interactive invocations, whether each runs `git init` itself, and where the import package or `src/` root lands — the facts `pipelex-integrate`'s detection will read next. + +**Build and tests** + +- [x] `make build`; confirm every target carries `skills/pipelex-scaffold/SKILL.md` and the references. +- [x] New `TestPipelexScaffoldDiscipline` pinning, on all three targets: never writes into a non-empty directory; exactly one commit and it is the pristine template; the bootstrap is delegated (the template names `.claude/skills/bootstrap/SKILL.md` and reimplements no rename); the key is never printed or asked for; `gh repo create` is confirmed; no dev server is started; no SDK dependency is added. Plus the assertion that `pipelex-scaffold` is **not** in `MCP_SKILLS` and renders no MCP-absent message. +- [x] `make agent-check`, `make agent-test`. + +**CHECKPOINT 1b** — the scaffold template and references render on every target and the tests pin their rules. Record here: the commit SHA and anything the design had to give up because a starter's bootstrap behaves differently from what S§3 assumed. + +**Reached 2026-09-06 (uncommitted at the session pause).** Nothing given up; the design was written against the two bootstrap skills as they stand. Phase 2 (family wiring and docs) was completed in the same session; `make build`, `make agent-check` and `make agent-test` are green on all three targets. + ## Phase 2 — family wiring and documentation Owner: `pipelex-plugins`. Small, deliberate edits; each one is one sentence or one step (§7). -- [ ] `templates/skills/pipelex-edit/SKILL.md.j2` Step 7: the `sources.json` staleness notice and the `/pipelex-integrate` offer. -- [ ] `templates/skills/pipelex-design/SKILL.md.j2`: the same notice in the re-entry delivery; the one-line hand-off in "Common runnable gate and delivery" step 4 (when a `package.json` / `pyproject.toml` is in the workspace). -- [ ] `templates/skills/pipelex-inputs/SKILL.md.j2`: the one-line hand-off in the closing report. -- [ ] `docs/decisions.md`: a dated entry — the skill's scope line (§4.1), the name ruling over `pipelex-codegen` (§4.11), the `explicit: true` exception appended to the light-template decision (§4.9), the write-arm-only and never-ride-content rule (§4.4), the sidecar (§4.6), the wire-null helper with its expiry (§4.8), the gate asymmetry (§4.5), and that this is the first skill to ship `references/`. -- [ ] `CLAUDE.md` "Key dependency": add `mthds_codegen` (the write arm, `output_dir`) beside the other tools; the structure block gains the `pipelex-integrate` template and the root `skills/` directory. -- [ ] `README.md`: the skill in "What's inside" and `mthds_codegen` in the MCP server bullet; the Claude and Codex sections' skill lists. -- [ ] `docs/build-targets.md`: the `skills//references/` mechanism is now in use, with `pipelex-integrate` as the example. -- [ ] `CHANGELOG.md` `[Unreleased]` → "Added": the skill, in the changelog's existing voice (what it does, the write arm, the sidecar, the gate asymmetry, the wire-null helper and its expiry, `explicit: true`); "Changed": the three family one-liners. -- [ ] `make build`, `make check`, `make agent-test`. +- [x] `templates/skills/pipelex-edit/SKILL.md.j2` Step 7: the `sources.json` staleness notice and the `/pipelex-integrate` offer. +- [x] `templates/skills/pipelex-design/SKILL.md.j2`: the same notice in the re-entry delivery; the fork in "Common runnable gate and delivery" step 4 — a `package.json` / `pyproject.toml` in the workspace → `/pipelex-integrate`, none → `/pipelex-scaffold` (§7, S§8). +- [x] `templates/skills/pipelex-inputs/SKILL.md.j2`: the one-line hand-off in the closing report. +- [x] `docs/decisions.md`: a dated entry for each skill — integrate: the scope line (§4.1), the name ruling over `pipelex-codegen` (§4.11), the `explicit: true` exception confined to the fallback, appended to the light-template decision (§4.9), the write-arm-only and never-ride-content rule (§4.4), the sidecar (§4.6), the wire-null helper with its expiry (§4.8), the gate asymmetry (§4.5), harness deference (§4.12), and that this is the first skill to ship `references/`; scaffold: two skills rather than one, starter-or-initializer and why not cookiecutter, the one pristine commit, bootstrap delegated to the starters, MCP-free (S§1, S§3, S§4). +- [x] `CLAUDE.md` "Key dependency": add `mthds_codegen` (the write arm, `output_dir`) beside the other tools; the structure block gains both templates and the root `skills/` directory; the MCP-free paragraph names `pipelex-scaffold` as the third MCP-free skill. +- [x] `README.md`: both skills in "What's inside" and `mthds_codegen` in the MCP server bullet; the Claude and Codex sections' skill lists. +- [x] `docs/build-targets.md`: the `skills//references/` mechanism is now in use, with `pipelex-integrate` as the example. +- [x] `CHANGELOG.md` `[Unreleased]` → "Added": both skills, in the changelog's existing voice (integrate: what it does, the write arm, the sidecar, the gate asymmetry, harness deference, the wire-null helper and its expiry if still shipped; scaffold: the two branches, the delegation to the starters' bootstrap, MCP-free); "Changed": the family one-liners. +- [x] `make build`, `make check`, `make agent-test`. ## Phase 3 — dogfood against the local workshop @@ -112,20 +156,41 @@ Two scratch projects, each created from scratch by the session so the skill meet - [ ] **PY-1 fresh integration, pydantic audience.** No `pipelex` dependency → `python-pydantic` chosen without a question; `/generated/__init__.py` and the subpackage `__init__.py` created; `[tool.ruff] exclude` gains the tree; pyright still covers it; `pydantic` and `pipelex-sdk` added with uv; the async call site plus a sync wrapper if the project is synchronous; the report states the gate asymmetry; `uv run pyright` passes. - [ ] **PY-2 structures audience.** Add `pipelex` as a dependency and a `@pipe_func` file: `python-structures` is chosen (or offered first when only the dependency is present); `pipelex codegen check ` is wired into the existing gate. - [ ] **PY-3 refresh after a bundle edit**, as TS-3, including the `/pipelex-edit` staleness notice. + +Harness scenarios (§4.12) — each on a fresh scratch copy of a starter, never on the starter checkout itself: + +- [ ] **TS-11 harness project, local method.** A scratch copy of `pipelex-starter-js` plus a new `methods//main.mthds`: the skill detects the harness, runs `npm run codegen`, writes the call site the way `docs/codegen.md` and the existing actions do, writes no sidecar of its own and no `src/generated/` tree of the plugin's shape, and `make check` is green. Refresh is reported as `npm run codegen`. +- [ ] **TS-12 harness project, remote method.** The same copy with a `method_ref` at a tag: `make add-method METHOD=…` is what runs; nothing else is written by the skill. +- [ ] **PY-4 harness project without a `pipelex` CLI.** A scratch copy of `pipelex-starter-python` plus a new method, with no `pipelex` on the PATH: the write arm writes into `/generated//` — the harness's layout — the report names `make codegen`'s prerequisite as the project's own and points at `L-260906-a2cd5b`'s subject in plain words, and no second layout exists. + +Scaffold scenarios (S§3–S§7) — scratch directories in the session scratchpad; the network is needed for the clones and installs: + +- [ ] **SC-1 JS starter, local clone, named directory, inputs up front.** Prerequisites checked and reported; the clone lands with fresh history; exactly one commit, its message carrying the template version and SHA; the clone's own `bootstrap` SKILL.md is followed from inside the project without re-asking what was given; `.env.local` is written with the key from the environment; `make all` is green; the bootstrap has removed itself; the report carries the session note and the hand-off. +- [ ] **SC-2 Python starter, local clone.** As SC-1; `git mv` of the package directory succeeds because of the pristine commit; `make agent-check` and `make agent-test` are green. +- [ ] **SC-3 "here".** An empty working directory: the clone lands in place and everything else is as SC-1. +- [ ] **SC-4 non-empty target.** The refusal, the question, nothing written. +- [ ] **SC-5 un-bootstrapped clone in the working directory.** Branch A enters at the bootstrap step and nothing is cloned. +- [ ] **SC-6 ecosystem, Python.** `uv init --package` in a fresh directory, the pristine commit, `.env.example` + `.env` (ignored), no SDK dependency, no `methods/`; then `/pipelex-integrate` from the same session lands the PY-1 shape. +- [ ] **SC-7 ecosystem, TypeScript, named framework.** `npm create next-app@latest … --yes`, the initializer's own `git init` respected, the pristine commit on top, the env pair; then `/pipelex-integrate` lands the TS-1 shape. +- [ ] **SC-8 missing toolchain.** A PATH without `node`: the stop message names Node and the starter README's floor; nothing is cloned. +- [ ] **SC-9 `gh repo create --template`.** Run only on Louis's explicit say-so, against a throwaway private repository he deletes afterwards: the confirmation appears before the command, visibility is asked, the GitHub-made initial commit is respected and no second pristine commit is made. +- [ ] **SC-10 no key in the environment.** The env file is written with an empty key, the report says where a key comes from, and nothing asks for it in the conversation. - [ ] **Vibe render sanity.** Read `pipelex-vibe/skills/pipelex-integrate/SKILL.md` once for the manual-registration wording of the MCP-absent message and the absence of Claude-only frontmatter. - [ ] Reconcile every finding into the template and references; re-run the affected scenarios; `make build`, `make agent-check`, `make agent-test`. - [ ] `/pipelex-mcp-source` back to `@latest`; confirm the diff carries no launcher change. -**CHECKPOINT 2** — every scenario above has been run at least once against the local workshop and its finding reconciled. Record here: the `pipelex-mcp` SHA the dogfood ran against, the scenarios that exposed a template change (and the change), any scenario that could not be run and why, and the exact wording the §4.3 heuristic produced in TS-5. +**CHECKPOINT 2** — every scenario above has been run at least once against the local workshop and its finding reconciled. Record here: the `pipelex-mcp` SHA the dogfood ran against, the starter SHAs the harness and scaffold scenarios cloned, the scenarios that exposed a template change (and the change), and any scenario that could not be run and why. ## Phase 4 — release Owner: `pipelex-plugins`. **Gate, hard:** a published `@pipelex/mcp` version that carries `mthds_codegen` with the write arm — name the version here before starting; an open `pipelex-mcp` release item is not a gate. The plugin's launcher is `@latest`, so nothing in this repo moves for it, but a plugin released before the tool is a skill that stops at "tool absent" for every user. -- [ ] Published `@pipelex/mcp` version carrying `mthds_codegen`: `__________` (fill in). +- [ ] Published `@pipelex/mcp` version carrying `mthds_codegen`: **0.13.0** (published 2026-08-30; the plugin's `@latest` already resolves to it). +- [ ] Published `@pipelex/mcp` version carrying **`main_pipe` on the validate verdict**: `__________` (fill in — it sits under `[Unreleased]` in `pipelex-mcp/CHANGELOG.md` at writing). The skill's ordinary path reads the signature from there; on an older workshop it takes the fallback of §4.3 (template for the inputs, the bundle for the output) and says the workshop predates the signature. Ship after this release so users never meet the fallback by default. - [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` one last time; strike or keep the helper and the asymmetry paragraph accordingly, and log it. - [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. -- [ ] Open the PR against `dev` with `Closes L-260830-344594` in the body; work the review rounds per the workspace's tightening-bar rule; land with `/ledger-land`. +- [ ] Open the PR against `dev` with `Closes L-260830-344594` and `Closes L-260906-8ac105` in the body; work the review rounds per the workspace's tightening-bar rule; land with `/ledger-land`. `L-260831-b67e18` closes in the same landing, with the release as evidence that the heuristic never shipped. +- [ ] File the starter-README pointer items (S§9, "to file at release"): both starters' "Use this template" sections name `/pipelex-scaffold` beside the button and `/bootstrap`. - [ ] `/release` → the next minor (`0.6.0`), which cuts the changelog heading, bumps every target TOML and the Claude marketplace, and opens the release PR against `main`. - [ ] After the release merges: `/ledger-land` on the release PR, and this file's `status` flips to `landed` when the tooling performs it. @@ -137,22 +202,32 @@ Carried in the design's §9 and §8; repeated here only where a phase might be t - The by-ref / by-id output-concept heuristic (§4.3) ships as designed; the exact answer arrives with the `pipelex-mcp` follow-up and is not worked around here. - The Python offline gate waits on `L-260830-4e43cd`; the TypeScript script is retired by `L-260820-2ba0f4`; the wire-null helper by `L-260820-ee327d`. None of the three is worked from this repo. - The call site uses `pipe_code` (bare) until the SDKs take `pipe_ref` (`L-260829-563e9e`); the sidecar already records the qualified ref so the switch is a one-line edit per call site when it comes. +- No cookiecutter or copier conversion of the starters, and no `create-pipelex-app` from this repo: the former was rejected on 2026-09-06 (the starters are living apps with CI and end-to-end tests a Jinja-ified template could not keep, and the rename is already a deterministic script with a dry run); the latter is the decision `L-260906-84bb41`. +- No change to either starter from here, including the Python starter's harness asymmetry (`L-260906-a2cd5b`, `L-260906-aa5083`): `pipelex-integrate` defers to the harness as it stands and the report says so. ## Decisions taken along the way - **2026-08-30 — Phase 0 ratification.** Louis walked the ten decision boxes of `design.md` one at a time, each presented against its alternatives (a narrower or wider call-site scope; refusing `method_id` or accepting a floating `method_ref`; blocking by-ref / by-id on the `pipelex-mcp` follow-up or keeping a contracts artifact; a content fallback or consented pre-clearing on the write arm; a hash-only Python gate or no gates at all; dropping the sidecar's `pipe` record or LF-normalizing its hashes; fixing the emitter first or a blind null-strip; lifting the light-template pin plugin-wide or reading concepts from the bundle; `pipelex-codegen` or a user-invocable-only skill; a staleness notice alone or an auto-refresh from `pipelex-edit`). Every box was ratified as written; the design's sections stand unamended and both documents flipped to `active` in this change. - **2026-08-30 — upstream dependencies reviewed (Louis).** None of the items the design leans on (`L-260820-ee327d`, `L-260820-2ba0f4`, `L-260830-4e43cd`, `L-260830-e8b2e0`, `L-260829-563e9e`) is a member of the build-retirement epic `L-260829-848001` or appears in its plan; they sit on the codegen trust-chain axis, not the descriptor-route axis. Two decisions: `L-260830-e8b2e0` is linked *related* to that epic (not a member), to be sequenced after `L-260829-dfaed4` once the workshop holds the input-form descriptor; and `L-260820-ee327d` is prioritized ahead of Phase 3, so the Phase 1 pre-flight re-check may strike `wire-output.ts` before it is written. The summary is `upstream-dependencies.md` beside this file. +- **2026-09-06 — the second design session (Louis).** The question was how to organize the skills that plug Pipelex methods into a development project, existing or new, across Python and TypeScript, and whether the starters should become cookiecutter-style templates. Research first: the SDKs (`@pipelex/sdk` has `codegen()`, `runCodegenCheck`, `prepareInputs`, `startAndWaitForResult`, a barrel; `pipelex-sdk` has `codegen()` and `prepare_inputs` but no offline check, no inputs template since `build_inputs` was removed, and deliberately no barrel), the starters (both are GitHub templates with script-driven `bootstrap` and `release` skills; the JS one also has `bump-sdk`, `bump-mthds-form`, `make add-method`, `AGENTS.md` and `docs/adopt-in-an-existing-project.md`; the Python one regenerates through a `pipelex` CLI it does not depend on), the codegen contract (`docs/specs/pipelex-codegen.md`; two axes, three targets; `pipelex-mcp`'s write arm; the lock and stamps), and the workspace documents (`wip/devx/scaffolding.md`'s unbuilt Proposals 2 and 4; `codegen-personas-and-trust-chain.md`'s mechanism-versus-policy split). Two upstream changes since 2026-08-30 were found: the validate verdict now carries `main_pipe` (`L-260830-e8b2e0` landed, `L-260831-b67e18` filed to delete the heuristic), and the ts-zod null fix merged to `pipelex` `dev` (`pipelex#1177`). Four rulings, each as recommended: **two skills** (`pipelex-integrate` as ratified plus a thin greenfield skill) over one skill with a preamble or integrate alone; **a starter or the ecosystem's initializer, no cookiecutter or copier**; **harness deference** in `pipelex-integrate` over always using the write arm; **`pipelex-scaffold`** over `pipelex-new-project` and `pipelex-bootstrap`. `design.md` was amended accordingly (boxes 3, 8, 10 reworded, 11 added), `scaffold-design.md` was written as a draft with boxes A–I, and the items `L-260906-8ac105`, `L-260906-a2cd5b`, `L-260906-aa5083` and `L-260906-84bb41` were filed. + +- **2026-09-06 — Phase 1 pre-flight.** `L-260820-ee327d` is **closed**: the ts-zod `.nullish()` fix (`pipelex#1177`) is in `pipelex` **v0.56.0**, and `pipelex-api` pins `pipelex==0.56.0`, so the hosted codegen route carries it as soon as that api is deployed. Under §4.8 the helper is now lossy rather than protective: **`wire-output.ts` is struck** and is never written into a project; the TypeScript call site parses `main_stuff` through the generated binder directly. `L-260830-4e43cd` is still open: the Python asymmetry sentence stays. The published `@pipelex/mcp` is 0.13.0, which carries `mthds_codegen` with the write arm but **not yet `main_pipe`** on the validate verdict (that entry is under `[Unreleased]`); the skill therefore keeps an honest fallback for an older workshop — inputs from `mthds_inputs_template` with `explicit: true`, output from the bundle for a files source, a stop for a by-ref or by-id source with the advice to refresh the workshop — and Phase 4 gains the release that carries the signature as a gate. + +- **2026-09-06 — session paused after Phases 1, 1b and 2, before Phase 3.** Everything is in the worktree `_pipelex-plugins--codegen` on `feature/Codegen`, **uncommitted**: the two templates, their references, the two test modules, the family-wiring edits to `pipelex-design` / `pipelex-edit` / `pipelex-inputs`, `docs/decisions.md`, `CLAUDE.md`, `README.md`, `docs/build-targets.md`, `CHANGELOG.md`, the regenerated `pipelex*/` outputs, and the amended `design.md`, `plan.md` and new `scaffold-design.md`. **Cold start:** `ledger claim L-260830-344594 --renew` from the worktree, read this file's Phase 3, review the diff (`git status`, `git diff`), commit with explicit `git add ` per the rules above, then run Phase 3 against the local `../pipelex-mcp` checkout (`/pipelex-mcp-source`) — the published `@pipelex/mcp` 0.13.0 lacks `main_pipe`, so the ordinary signature path only exercises on the local build. Two small follow-ups noticed and not done: the Phase 4 `create-pipelex-app` decision is `L-260906-84bb41`; the synthetic-inputs campaign's `plan.md` still says `active` after its item closed (doctor `doc-active-after-close`, unrelated to this campaign). + ## Deviations from the design *(empty at writing — a deviation is logged here with its reason before the code that embodies it is committed.)* ## Where everything is -- Brief: `wip/pipelex-integrate/brief.md`. Design: `wip/pipelex-integrate/design.md`. This tracker: `wip/pipelex-integrate/plan.md`. +- Brief: `wip/pipelex-integrate/brief.md`. Designs: `wip/pipelex-integrate/design.md` (integrate) and `wip/pipelex-integrate/scaffold-design.md` (scaffold). Upstream reading companion: `upstream-dependencies.md`. This tracker: `wip/pipelex-integrate/plan.md`. +- The starters the scaffold skill acquires and the harness rule defers to: `../pipelex-starter-js/.claude/skills/bootstrap/SKILL.md` (+ `scripts/bootstrap.mjs`), `../pipelex-starter-python/.claude/skills/bootstrap/SKILL.md` (+ `scripts/bootstrap.py`); their harnesses: `../pipelex-starter-js/package.json` scripts `codegen` / `codegen:check` / `codegen:verify` / `add-method` and `Makefile`, `../pipelex-starter-python/Makefile` (`codegen`, `codegen-check`). +- The MCP-free template to model for scaffold: `templates/skills/pipelex-synthetic-inputs/SKILL.md.j2`. - The tool contract: `../pipelex-mcp/SPEC.md` → "Codegen Scope (`mthds_codegen`)" and "The write arm (`output_dir`) — local workshop only". The writer: `../pipelex-mcp/src/capabilities/codegen-writer.ts`; containment: `workspace-boundary.ts`. - Reference integrations (read, never changed): `../pipelex-starter-js/docs/codegen.md`, `src/generated//`, `src/lib/wireOutput.ts`, `scripts/codegen-check.mts`; `../pipelex-starter-python/Makefile` (`codegen`, `codegen-check`), `docs/codegen.md`, `piper/generated/`. - The offline check the TypeScript gate wraps: `../pipelex-sdk-js/src/codegen-check.ts` (`runCodegenCheck`, `isStampableArtifactPath`), documented in `docs/crate-routes.md` → "The offline check". - The sibling skill to model: `templates/skills/pipelex-inputs/SKILL.md.j2`; the tests to extend: `tests/unit/test_gen_skill_docs.py` (`TestSkillFailureDiscipline`, `TestPipelexInputsSizeLimitDiscipline` as the pattern). - The static-asset mechanism: `scripts/gen_skill_docs.py` → `setup_static_assets`; documented in `docs/build-targets.md` → "Template vs output directories". -- Ledger: this item `L-260830-344594`; discovered `L-260830-4e43cd`; related `L-260820-ee327d`, `L-260820-2ba0f4`, `L-260829-563e9e`; the `pipelex-mcp` follow-up filed in Phase 0: `L-260830-e8b2e0`. +- Ledger: the integrate item `L-260830-344594` and the scaffold item `L-260906-8ac105`; discovered `L-260830-4e43cd`, `L-260906-a2cd5b`, `L-260906-aa5083`; related `L-260820-ee327d`, `L-260820-2ba0f4`, `L-260829-563e9e`, `L-260906-84bb41`; the `pipelex-mcp` follow-up filed in Phase 0, `L-260830-e8b2e0`, and its consequence for this repo, `L-260831-b67e18`. diff --git a/wip/pipelex-integrate/scaffold-design.md b/wip/pipelex-integrate/scaffold-design.md new file mode 100644 index 0000000..3e75d4b --- /dev/null +++ b/wip/pipelex-integrate/scaffold-design.md @@ -0,0 +1,107 @@ +--- +status: active +item: L-260906-8ac105 +--- + +# Design — `pipelex-scaffold`: the front door to a project that does not exist yet + +**Written 2026-09-06**, in the design session that widened this campaign from one skill to two. The shape was decided with Louis in that session and is not re-argued here: two skills rather than one (`pipelex-integrate` for an existing codebase, this one for a project that does not exist yet); a from-scratch project comes from one of our two GitHub-template starters or from the ecosystem's own initializer, never from a cookiecutter or copier template the plugin would have to carry; `pipelex-integrate` defers to a project that owns a codegen harness (`design.md` §4.12); and the name is `pipelex-scaffold`. **Status: active** — the decision boxes at the end, which settle the procedure and the smaller calls the implementation needs, were ratified as written on 2026-09-06, the day this document was drafted. The sibling document is [`design.md`](design.md); the tracker for both skills is [`plan.md`](plan.md). Ledger item `L-260906-8ac105`. File and line references were accurate on the writing date; verify them against the code before implementing. + +## 1. What the skill is + +`pipelex-scaffold` gives a user who has no project yet a project that is ready for `pipelex-integrate`. It has exactly two branches and carries no templates of its own: + +- **One of our starters** when the user wants the opinionated shape: `pipelex-starter-js` for a web app whose forms are rendered from the methods' own contracts, `pipelex-starter-python` for a CLI or service that runs methods in the three execution modes. The skill acquires the template, commits it once as it came, then runs the clone's own `bootstrap` skill — the rename logic stays in the starters and is never duplicated here. +- **The ecosystem's own initializer** when the user wants their framework or a minimal project: `uv init --package`, `npm create next-app@latest`, `django-admin startproject`, whatever the framework documents. The skill runs it; it never assembles a project by hand. + +Both branches share a tail: the env file and the key convention, the pristine commit that makes everything after it reviewable, and the hand-off — to `/pipelex-integrate` when a method exists, to `/pipelex-design` first when none does. + +**What it is not.** It is not a template engine (no cookiecutter, no copier, no framework matrix of its own), not a bootstrap (the starters own theirs), not a runner or a dev-server launcher, not a deployer, and not `create-pipelex-app` — whether a CLI front door for users who do not work through an agent is still wanted is `L-260906-84bb41`, a decision, not this skill's scope. It is **MCP-free**: git, the starters' scripts and the ecosystem's initializers are all it needs, which puts it beside `pipelex-explain` and `pipelex-synthetic-inputs` and out of the `MCP_SKILLS` tuple in the tests. + +## 2. Choosing the branch + +The rule from `pipelex-integrate` applies: a cheap, reliable signal decides; an inconclusive one asks one question; nothing is guessed twice. + +| Question | Signals, in order | When inconclusive | +| --- | --- | --- | +| **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | +| **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, Remix, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or strip) | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | + +A starter clone that is already in the working directory and has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `piper` — is **branch A entered at step 3**: acquisition already happened, and the skill goes straight to running the clone's bootstrap. + +## 3. Branch A — one of our starters + +1. **Prerequisites.** JavaScript: Node at or above the floor the starter's `package.json` `engines` names (22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and npm. Python: `uv` (the starter's Makefile installs and locks with it) and a Python inside the starter's `requires-python` range (3.11 to 3.14 at writing) that `uv python find` can see. Both: git. The GitHub branch also needs `gh` authenticated (`gh auth status`). A missing piece **stops** the skill with the exact thing missing and the starter README's own line about it; the skill never installs a toolchain. +2. **Acquire.** + - **Local, the default.** `git clone --depth 1 https://github.com/Pipelex/.git `; read the template's version from its `package.json` / `pyproject.toml` and its head SHA; then detach from the template — remove the clone's `.git`, `git init -b main` — so that `git status`, `git remote` and a future push belong to the user's project and not to the template. This is what GitHub's "Use this template" button produces: a copy with no history and no remote. The starters' READMEs say "don't clone it directly" to humans for exactly that reason, and the fresh history is how the skill honours it. + - **GitHub, on request.** `gh repo create / --template Pipelex/ --private --clone` (visibility is the user's call, asked, default private). Creating a repository on GitHub is an outward-facing action: the skill states the exact command and confirms before running it. GitHub writes the initial commit itself; the skill continues at step 3. + - Both take the template's **default-branch head**, and the pristine commit below records the version and SHA it came from. Pinning a release tag is not offered unless the user asks; the starters cut releases, and a user who wants one names it. +3. **Commit the pristine template — exactly once.** `git add -A && git commit -m "Start from Pipelex/ ()"` from inside the directory. This is the one commit the skill makes, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in this commit — it is the template as it came. +4. **Run the clone's own bootstrap.** Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written, with every command run from inside the project directory (the skill's cwd is wherever the harness was launched, so commands carry `-C ` or a `cd &&`). Feed it what the conversation already holds — the project name, title, description, author, repository URL, license — so that it asks once, consolidated, for whatever is left, exactly as its Step 2 says. It dry-runs, previews, runs, re-syncs the lock file, runs the project's own checks (`make all` on JS; `make agent-check` and `make agent-test` on Python), and removes itself. Its rules stand unchanged: it never commits, its edits stay unstaged for the user's review (the Python renames are staged by `git mv`, which its own skill explains), and a red check is fixed, not skipped. The plugin skill adds nothing to that procedure and reimplements none of it. If the clone carries no bootstrap skill — a future template dropped it — the skill follows the README's "manual equivalent" list and says the template changed. +5. **The env file.** `cp .env.example .env.local` on the JS starter (Next.js reads `.env.local`), `cp .env.example .env` on the Python one (`python-dotenv`). Fill `PIPELEX_API_KEY` from the shell environment when it is set there, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that the file is where it goes. The skill never prints a key and never asks for one in the conversation. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored (both starters ignore it) before writing a key into it. +6. **Verify and hand off.** The bootstrap's own checks are the verification; the skill does not start `make dev`. The report (§5) names the demos the starter still carries and where the README's removal checklist is, and hands the user's method to `/pipelex-integrate`, which recognizes the starter's codegen harness and defers to it (`design.md` §4.12). + +## 4. Branch B — the ecosystem's initializer + +1. **Prerequisites** as in branch A, for the language chosen. +2. **Run the initializer, never assemble by hand.** + - **A named framework** uses its documented initializer with its non-interactive flags where it has them — `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`, `uv init --package ` then `uv add fastapi`, `django-admin startproject `, and so on; the skill's `references/initializers.md` carries the invocations for the common ones. An initializer that only runs interactively is handed to the user to run (`! ` in the prompt runs it in the session), and the skill resumes when it is done. + - **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. Nothing beyond what those initializers write is authored by the skill. +3. **Version control and the pristine commit.** If the initializer did not `git init` (some do), `git init -b main`; then the one commit, `"Scaffold project"`, for the same reason as branch A: everything the user does next is a reviewable diff against it. +4. **The env file.** Write `.env.example` with the two lines the starters share (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), add `.env` to `.gitignore` if it is not already ignored, and copy the example to `.env` with the same key rule as branch A. +5. **Hand off.** No SDK dependency is added and no empty `methods/` directory is created: `pipelex-integrate` adds `@pipelex/sdk` / `pipelex-sdk` when it writes the first call site, and creates `methods//` when it places the first bundle. A project with nothing to integrate yet has nothing Pipelex-shaped in it beyond the env convention, which is correct. + +## 5. The shared tail — the report + +The report says: what was created and where; which template or initializer it came from, at which version and SHA; that the skill made exactly one commit and what it contains; what the bootstrap changed and that those changes are uncommitted for review, in the bootstrap's own words; which env file was written and whether the key was filled from the environment or left for the user; the demos the starter still carries and where the README's removal checklist is (branch A); and the hand-off — `/pipelex-integrate` for a method that exists (a bundle elsewhere on disk is copied into the project by that skill, `design.md` §4.2), `/pipelex-design` first when none does. + +One line in the report is easy to forget and matters: **the project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session, which began elsewhere; `cd && claude` (or the host's equivalent) is how they arrive. Until then `/pipelex-integrate` still works from here, because the workshop writes anywhere under the directory the harness was launched in and the new project sits there. + +## 6. Mode and questions + +Automatic by default, with the plugin's usual rules: an explicit user signal wins; a genuinely ambiguous branch is one question (§2), asked once; a project with every input given up front proceeds without re-asking. Two things always confirm: `gh repo create`, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not — it is on a directory the skill just created, holding the template as it came, and no user content is at stake. + +## 7. Failure posture + +| Condition | The skill | +| --- | --- | +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| `gh` is absent or not authenticated | fall back to the local clone and say the GitHub repository can be created later with `gh repo create --source .` | +| The clone carries no `bootstrap` skill | follow the README's manual list, say the template changed | +| The bootstrap's checks are red | the bootstrap's own rule: fix the cause and re-run; never hand off on red | +| An initializer is interactive with no non-interactive form | hand the command to the user to run in the session, resume after | +| `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file, say where a key comes from and where it goes; never ask for it in the conversation | +| The working directory is the template's own checkout (a `Pipelex/` remote) | STOP: this is the template, not a copy of it — acquire a copy | + +## 8. Name, triggers and family wiring + +**`pipelex-scaffold`**, model-invocable. Its description triggers on the greenfield phrasings — "start a new project with Pipelex", "I have a method and need an app around it", "create a Next.js app that runs my method", "set up a Pipelex project from scratch", "which starter should I use", "bootstrap a Pipelex project" — and stays silent on the existing-codebase phrasings `pipelex-integrate` claims ("use this method in my app", "call this from my code", "generate types"). It says in so many words that a user already standing in a fresh starter clone is served by that clone's own `/bootstrap`, which this skill runs for them (§2, the shortcut). + +Family wiring, each one sentence: `pipelex-integrate`'s step 1 offers `/pipelex-scaffold` when it finds no project at all (`design.md` §5, §7); `pipelex-design`'s delivery step forks the same way — a codebase in the workspace → `/pipelex-integrate`, none → `/pipelex-scaffold`; the README's skill list, `CLAUDE.md`'s structure block and `docs/decisions.md` gain the skill, with the note that it is the plugin's third MCP-free skill. + +## 9. Follow-ups, filed or to file + +| Item | Repo | Relation | Why | +| --- | --- | --- | --- | +| `L-260906-84bb41` | workspace | related, decision | whether a `create-pipelex-app` CLI front door is still wanted for users who do not work through an agent | +| `L-260906-a2cd5b` | `pipelex-starter-python` | informational | the Python starter's `make codegen` needs a `pipelex` CLI the starter does not depend on; the report on a Python starter says so honestly and points at `/pipelex-integrate`, which writes into the harness's layout meanwhile (`design.md` §4.12) | +| `L-260906-aa5083` | `pipelex-starter-python` | informational | the Python starter lacks `AGENTS.md` and `add-method`; nothing in this skill waits on it | +| *to file at release* | both starters | docs | the READMEs' "Use this template" sections should name `/pipelex-scaffold` as the agent front door beside the button and `/bootstrap` — filed when the skill ships, so the pointer never precedes the thing it points at | + +## Decision boxes for ratification + +| Box | Ruling | Ratified? | +| --- | --- | --- | +| **A — Two branches, no templates of its own** | Our starter or the ecosystem's initializer; MCP-free; the language defaults for branch B are `uv init --package` and `npm init` + `tsc --init` (§1, §4) | Yes, as written — 2026-09-06 | +| **B — Acquisition** | Local clone with fresh history by default, matching what the template button produces; `gh repo create --template` only on request, confirmed, visibility asked; default-branch head with version and SHA recorded (§3 step 2) | Yes, as written — 2026-09-06 | +| **C — Exactly one commit** | The pristine template or scaffold, before the bootstrap, so `git mv` works and the bootstrap's edits are a reviewable diff; everything after stays uncommitted under the bootstrap's own rules (§3 step 3, §4 step 3) | Yes, as written — 2026-09-06 | +| **D — Bootstrap is delegated** | The clone's own `bootstrap` skill, read from its `SKILL.md` and run from inside the project with the inputs already known; the README's manual list as the fallback; an un-bootstrapped clone in the working directory enters here (§2, §3 step 4) | Yes, as written — 2026-09-06 | +| **E — The env file and the key** | The starter's own env file; the key filled only from the shell environment, never printed, never asked in the conversation; the base URL untouched (§3 step 5, §4 step 4) | Yes, as written — 2026-09-06 | +| **F — Nothing Pipelex-shaped beyond the env** | No SDK dependency and no empty `methods/` from this skill; `pipelex-integrate` adds both when there is something to integrate (§4 step 5) | Yes, as written — 2026-09-06 | +| **G — The session note** | The report says the project's own instructions and skills load in a session started inside it, and that `/pipelex-integrate` works from here meanwhile (§5) | Yes, as written — 2026-09-06 | +| **H — Verification** | The bootstrap's own checks on branch A, the initializer's result and a clean `git status` after the pristine commit on branch B; the skill never starts a dev server (§3 step 6) | Yes, as written — 2026-09-06 | +| **I — Name and triggers** | `pipelex-scaffold`, model-invocable, greenfield phrasings only, with the fresh-clone shortcut named in the description (§8) | Yes, as written — 2026-09-06 | From 4572156f5d24411dc3dfb53f1c8d732e670c117f Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 00:39:35 +0200 Subject: [PATCH 03/21] Reconcile three defects the integrate dogfood exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven Phase 3 scenarios ran as cold headless sessions against the local workshop — a fresh agent per scenario, given a user's own phrasing and no knowledge of the campaign. Three findings survived. The refresh-mode rule read the wrong value. `crate_fingerprint` covers the whole bundle, not its concept set, so a prompt-only edit moves it while the projected code is byte-identical — every artifact's single stamp line moves with it and nothing else does. The honest signal is the lock's `artifacts[].content_hash`, which is computed beneath the stamp; both the skill and the design's §4.10 now read that instead. The skill and its Python reference disagreed about where a bundle lives: Step 1 said an in-project bundle never moves, while the reference's packaged row put it inside the import package. The reference is right, and a wheel build proved it rather than assuming it — a root-level `methods/` does not ship. Step 1 now names the exception and asks for `git mv`. The family's staleness notice searched beside the bundle and reported a clean bill, while the sidecar sat beside the generated tree where the skill puts it. The location had been written as background prose rather than as a search instruction; `pipelex-edit` and `pipelex-design` now say to search the whole project and give the grep. Re-run, the notice fired correctly. Advances L-260830-344594 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DYrbgEm55V4QP3uJmgTqRS --- pipelex-codex/skills/pipelex-design/SKILL.md | 2 +- pipelex-codex/skills/pipelex-edit/SKILL.md | 2 +- .../skills/pipelex-integrate/SKILL.md | 4 +-- pipelex-vibe/skills/pipelex-design/SKILL.md | 2 +- pipelex-vibe/skills/pipelex-edit/SKILL.md | 2 +- .../skills/pipelex-integrate/SKILL.md | 4 +-- pipelex/skills/pipelex-design/SKILL.md | 2 +- pipelex/skills/pipelex-edit/SKILL.md | 2 +- pipelex/skills/pipelex-integrate/SKILL.md | 4 +-- templates/skills/pipelex-design/SKILL.md.j2 | 2 +- templates/skills/pipelex-edit/SKILL.md.j2 | 2 +- .../skills/pipelex-integrate/SKILL.md.j2 | 4 +-- wip/pipelex-integrate/design.md | 2 +- wip/pipelex-integrate/plan.md | 30 ++++++++++++------- 14 files changed, 37 insertions(+), 27 deletions(-) diff --git a/pipelex-codex/skills/pipelex-design/SKILL.md b/pipelex-codex/skills/pipelex-design/SKILL.md index 4cfb7fc..c83a35c 100644 --- a/pipelex-codex/skills/pipelex-design/SKILL.md +++ b/pipelex-codex/skills/pipelex-design/SKILL.md @@ -207,7 +207,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. Search the whole project for `sources.json` files carrying `"generator": "pipelex-integrate"` — they sit beside each generated tree (`src/generated//`, `/generated//`), never beside the bundle — and for each one whose `sources` name a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex-codex/skills/pipelex-edit/SKILL.md b/pipelex-codex/skills/pipelex-edit/SKILL.md index 856d473..13e705d 100644 --- a/pipelex-codex/skills/pipelex-edit/SKILL.md +++ b/pipelex-codex/skills/pipelex-edit/SKILL.md @@ -76,7 +76,7 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. -**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. +**Generated types may now be stale.** Search the **whole project** for `sources.json` files carrying `"generator": "pipelex-integrate"` — `grep -rl '"pipelex-integrate"' --include=sources.json .` — and keep the ones whose `sources` name a file this edit changed. The sidecar sits beside the **generated tree**, never beside the bundle: `src/generated//` or `/generated//`. Looking only next to the `.mthds` file finds nothing and reports a clean bill that is wrong. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. ## Reference diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index 482f7fe..a006684 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -48,7 +48,7 @@ Automatic by default: state the target, the destination and the generator in one **The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: -- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is, with one exception the Python reference names: a **packaged** project (a `[build-system]` table and an import package) needs the bundle *inside* that package for the wheel to ship it beside the call site that loads it — propose the move, say why, and make it with `git mv` so it reads as one rename. - **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. - **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. ## A project that owns a codegen harness diff --git a/pipelex-vibe/skills/pipelex-design/SKILL.md b/pipelex-vibe/skills/pipelex-design/SKILL.md index 903fbba..2e4861b 100644 --- a/pipelex-vibe/skills/pipelex-design/SKILL.md +++ b/pipelex-vibe/skills/pipelex-design/SKILL.md @@ -207,7 +207,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. Search the whole project for `sources.json` files carrying `"generator": "pipelex-integrate"` — they sit beside each generated tree (`src/generated//`, `/generated//`), never beside the bundle — and for each one whose `sources` name a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex-vibe/skills/pipelex-edit/SKILL.md b/pipelex-vibe/skills/pipelex-edit/SKILL.md index 42d6f50..81b9d09 100644 --- a/pipelex-vibe/skills/pipelex-edit/SKILL.md +++ b/pipelex-vibe/skills/pipelex-edit/SKILL.md @@ -76,7 +76,7 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. -**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. +**Generated types may now be stale.** Search the **whole project** for `sources.json` files carrying `"generator": "pipelex-integrate"` — `grep -rl '"pipelex-integrate"' --include=sources.json .` — and keep the ones whose `sources` name a file this edit changed. The sidecar sits beside the **generated tree**, never beside the bundle: `src/generated//` or `/generated//`. Looking only next to the `.mthds` file finds nothing and reports a clean bill that is wrong. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. ## Reference diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index c3d2cd3..3817c36 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -48,7 +48,7 @@ Automatic by default: state the target, the destination and the generator in one **The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: -- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is, with one exception the Python reference names: a **packaged** project (a `[build-system]` table and an import package) needs the bundle *inside* that package for the wheel to ship it beside the call site that loads it — propose the move, say why, and make it with `git mv` so it reads as one rename. - **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. - **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. ## A project that owns a codegen harness diff --git a/pipelex/skills/pipelex-design/SKILL.md b/pipelex/skills/pipelex-design/SKILL.md index 3cb8f63..5f5d3d0 100644 --- a/pipelex/skills/pipelex-design/SKILL.md +++ b/pipelex/skills/pipelex-design/SKILL.md @@ -216,7 +216,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. Search the whole project for `sources.json` files carrying `"generator": "pipelex-integrate"` — they sit beside each generated tree (`src/generated//`, `/generated//`), never beside the bundle — and for each one whose `sources` name a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/pipelex/skills/pipelex-edit/SKILL.md b/pipelex/skills/pipelex-edit/SKILL.md index 3fa7367..a4d5dd6 100644 --- a/pipelex/skills/pipelex-edit/SKILL.md +++ b/pipelex/skills/pipelex-edit/SKILL.md @@ -85,7 +85,7 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. -**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. +**Generated types may now be stale.** Search the **whole project** for `sources.json` files carrying `"generator": "pipelex-integrate"` — `grep -rl '"pipelex-integrate"' --include=sources.json .` — and keep the ones whose `sources` name a file this edit changed. The sidecar sits beside the **generated tree**, never beside the bundle: `src/generated//` or `/generated//`. Looking only next to the `.mthds` file finds nothing and reports a clean bill that is wrong. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. ## Reference diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index bb8999d..6c98ec7 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -59,7 +59,7 @@ Automatic by default: state the target, the destination and the generator in one **The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: -- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is, with one exception the Python reference names: a **packaged** project (a `[build-system]` table and an import package) needs the bundle *inside* that package for the wheel to ship it beside the call site that loads it — propose the move, say why, and make it with `git mv` so it reads as one rename. - **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. - **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. @@ -179,7 +179,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. ## A project that owns a codegen harness diff --git a/templates/skills/pipelex-design/SKILL.md.j2 b/templates/skills/pipelex-design/SKILL.md.j2 index 3f8ddbe..dd08454 100644 --- a/templates/skills/pipelex-design/SKILL.md.j2 +++ b/templates/skills/pipelex-design/SKILL.md.j2 @@ -211,7 +211,7 @@ Structural changes to an existing method — adding, removing, or rewiring steps - Do not predeclare deeper descendants while their parent is only a signature. Introduce those child signatures when that parent receives its concrete controller definition, keeping every pending signature reachable. - Validate the atomic scaffold, then drain its structured backlog with Step S2. 5. **Recover rather than leave an unproven edit.** If a post-edit call returns no verdict, or the edited region cannot be made valid after two focused fixes, restore the retained baseline contents and report the failure. -6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. If a `sources.json` carrying `"generator": "pipelex-integrate"` names a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. +6. **Converge and deliver.** Restore at least the baseline verdict. Run `/pipelex-organize` only if signature-driven re-entry produced a construction-shaped layout that needs regrouping. Re-project the input template; if `inputs.json` exists and the client surface changed, flag the drift and hand the refresh to `/pipelex-inputs`. Search the whole project for `sources.json` files carrying `"generator": "pipelex-integrate"` — they sit beside each generated tree (`src/generated//`, `/generated//`), never beside the bundle — and for each one whose `sources` name a bundle file this re-entry changed, say the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them. --- diff --git a/templates/skills/pipelex-edit/SKILL.md.j2 b/templates/skills/pipelex-edit/SKILL.md.j2 index 3c7b607..d719cb8 100644 --- a/templates/skills/pipelex-edit/SKILL.md.j2 +++ b/templates/skills/pipelex-edit/SKILL.md.j2 @@ -80,7 +80,7 @@ When the edit could have changed the input template — a renamed main-pipe inpu State what changed (files and constructs), give the verdict line from the summary, and where the host renders MCP views, point to the method graph that accompanied the valid verdict. If inputs were refreshed or invalidated, say so. Suggest `/pipelex-inputs` when the user wants to prepare inputs or run the method. -**Generated types may now be stale.** Look for `sources.json` files carrying `"generator": "pipelex-integrate"` whose `sources` name a file this edit changed — a project keeps one beside each generated tree, typically under `src/generated//` or `/generated//`. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. +**Generated types may now be stale.** Search the **whole project** for `sources.json` files carrying `"generator": "pipelex-integrate"` — `grep -rl '"pipelex-integrate"' --include=sources.json .` — and keep the ones whose `sources` name a file this edit changed. The sidecar sits beside the **generated tree**, never beside the bundle: `src/generated//` or `/generated//`. Looking only next to the `.mthds` file finds nothing and reports a clean bill that is wrong. For each, say that the generated types in that directory are now stale and offer `/pipelex-integrate` to refresh them: it regenerates in place and touches the call site only if the types no longer fit it. This notice is the only drift guard a Python consumer has, so do not skip it. ## Reference diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index 743bb41..fdef5ed 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -54,7 +54,7 @@ Automatic by default: state the target, the destination and the generator in one **The method** comes from the conversation, in one of three selector forms — pass exactly one to every tool call, never two: -- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is. +- **Local files**, the recommended shape: every `.mthds` file of the bundle, as `files` items. Prefer `{path: }` per file (the workshop resolves a path against **its own** working directory, wherever the harness launched it, so pass absolute paths); `{content, uri}` is the inline fallback. A bundle that lives **outside the project** (a `pipelex-wip/` directory elsewhere on disk) is copied into the project under `methods//` first, because the call site loads it at runtime and the sources must be versioned with the code; tell the user. A bundle already inside the project stays where it is, with one exception the Python reference names: a **packaged** project (a `[build-system]` table and an import package) needs the bundle *inside* that package for the wheel to ship it beside the call site that loads it — propose the move, say why, and make it with `git mv` so it reads as one rename. - **A published address** — `method_ref: "github.com//[/]@"`. The tag is the pin: a `method_ref` **without a tag** floats and is refused for a committed integration; ask for the tag. - **A catalog id** — `method_id: "mt_…"` (resolve a name through `mthds_list_methods` when present). The catalog is unversioned: an edit to the stored method silently invalidates committed types with nothing offline to detect it. Say so in one line, recommend committing the source or publishing an address, and proceed only on the user's say-so. @@ -174,7 +174,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. The new `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — say so and leave the diff to the user's commit); changed means the concept set moved. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. ## A project that owns a codegen harness diff --git a/wip/pipelex-integrate/design.md b/wip/pipelex-integrate/design.md index 54271e7..ad5ded7 100644 --- a/wip/pipelex-integrate/design.md +++ b/wip/pipelex-integrate/design.md @@ -189,7 +189,7 @@ Entered when: the user asks to refresh, regenerate, or update the types; `pipele | --- | --- | --- | | the selector, target, destination, and pipe record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared before regenerating so the report can say whether the bundle actually changed; the pipe signature, through `mthds_inputs_template` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client and wire-output helpers, the gate wiring, tests, other methods' trees | -The regeneration is one `mthds_codegen` call with the recorded arguments. Its `crate_fingerprint` against the old lock's says what happened: unchanged means a restamp at most (an engine bump rewrites every stamp with no semantic change — reported as such, with the diff left to the user's commit); changed means the concept set moved. Then the project's type checker runs. **The call site is edited only if it no longer type-checks or the sidecar's `pipe` record no longer matches the template** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last, with the new hashes and pipe record. +The regeneration is one `mthds_codegen` call with the recorded arguments. What happened is read from the lock's `artifacts[].content_hash`, **not** from the fingerprint — a Phase 3 correction to this section, logged under the plan's deviations: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it, and with it the single stamp line at the head of every artifact, while the projected code stays byte-identical. Content hashes unchanged is therefore the restamp case (reported as such, with that one-line-per-file diff left to the user's commit); a content hash that moved is the concept set moving. An engine bump is the same reading. Then the project's type checker runs. **The call site is edited only if it no longer type-checks or the sidecar's `pipe` record no longer matches the template** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last, with the new hashes and pipe record. ### 4.11 The name and its place in the family (brief Q6) diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index 1055e95..4b73156 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -92,7 +92,7 @@ Owner: `pipelex-plugins`. Everything in this phase renders into all three target **CHECKPOINT 1** — the template and references render on every target and the tests pin their rules. Record here: the commit SHA, what was struck because an upstream item landed, and anything the template could not express without a reference file. -**Reached 2026-09-06 (uncommitted at the session pause — the SHA is recorded when the work is committed).** Struck: `wire-output.ts` (the emitter fix shipped in pipelex v0.56.0, see the pre-flight entry below). The pre-flight note above calling this "the first skill in the repo to ship references" was stale when ticked — `pipelex-design` and `pipelex-synthetic-inputs` already did; the mechanism (`setup_static_assets`, `static_asset_mismatches`) was confirmed to copy non-Markdown files too, which is what `codegen-check.mjs` needs. The discipline tests live in their own modules, `tests/unit/test_pipelex_integrate_skill.py` and `tests/unit/test_pipelex_scaffold_skill.py` (one class each), rather than in `test_gen_skill_docs.py`, which only gained `pipelex-integrate` in `MCP_SKILLS`. +**Reached 2026-09-06; committed 2026-09-07 as `54cf04d`, together with Phases 1b and 2.** Struck: `wire-output.ts` (the emitter fix shipped in pipelex v0.56.0, see the pre-flight entry below). The pre-flight note above calling this "the first skill in the repo to ship references" was stale when ticked — `pipelex-design` and `pipelex-synthetic-inputs` already did; the mechanism (`setup_static_assets`, `static_asset_mismatches`) was confirmed to copy non-Markdown files too, which is what `codegen-check.mjs` needs. The discipline tests live in their own modules, `tests/unit/test_pipelex_integrate_skill.py` and `tests/unit/test_pipelex_scaffold_skill.py` (one class each), rather than in `test_gen_skill_docs.py`, which only gained `pipelex-integrate` in `MCP_SKILLS`. ## Phase 1b — the `pipelex-scaffold` template and its references @@ -121,7 +121,7 @@ Owner: `pipelex-plugins`. **Gate:** `scaffold-design.md`'s boxes ratified (Phase **CHECKPOINT 1b** — the scaffold template and references render on every target and the tests pin their rules. Record here: the commit SHA and anything the design had to give up because a starter's bootstrap behaves differently from what S§3 assumed. -**Reached 2026-09-06 (uncommitted at the session pause).** Nothing given up; the design was written against the two bootstrap skills as they stand. Phase 2 (family wiring and docs) was completed in the same session; `make build`, `make agent-check` and `make agent-test` are green on all three targets. +**Reached 2026-09-06; committed 2026-09-07 as `54cf04d`.** Nothing given up; the design was written against the two bootstrap skills as they stand. Phase 2 (family wiring and docs) was completed in the same session; `make build`, `make agent-check` and `make agent-test` are green on all three targets. ## Phase 2 — family wiring and documentation @@ -143,19 +143,23 @@ Owner: `pipelex-plugins`, with `../pipelex-mcp` on a build that carries `mthds_c Two scratch projects, each created from scratch by the session so the skill meets a cold codebase: a minimal TypeScript project (`package.json`, `tsconfig.json`, Prettier and ESLint flat config, a `check` script, `src/`), and a minimal Python project (`pyproject.toml` with `[tool.ruff]` and `[tool.pyright]`, `uv.lock`, one import package). Each holds a committed `methods//main.mthds` copied from the cookbook or written with `/pipelex-design`. -- [ ] **TS-1 fresh integration, files source.** Run `/pipelex-integrate`. Verify: the tree landed under `src/generated//` with `is_current: true` and no orphans; `.prettierignore` and the ESLint `ignores` entry were added **before** the write; `zod` and `@pipelex/sdk` were added with npm; the call site, client helper, and wire-output helper exist where §5 says; `npm run check` (extended with `codegen:check`) passes; `tsc --noEmit` passes; `sources.json` matches §4.6. -- [ ] **TS-2 the bytes are untouched.** `git diff --stat` shows no change under `src/generated/` after the skill's own format run; `npm run codegen:check` exits 0. -- [ ] **TS-3 refresh after a bundle edit.** Change a concept field in `main.mthds` (through `/pipelex-edit`, which should announce the staleness — Phase 2 wiring), run the skill again: the sidecar comparison reports the changed source, the fingerprint moved, the call site is edited only if the type check demanded it, `sources.json` carries the new hash, `codegen:check` is green again. Then edit only a prompt (no concept change): the fingerprint is unchanged and the report says restamp-or-nothing. -- [ ] **TS-4 stale-source gate.** Edit the bundle and do *not* refresh: `npm run codegen:check` exits 1 with `stale-source` and the refresh remedy. +- [x] **TS-1 fresh integration, files source.** Run `/pipelex-integrate`. Verify: the tree landed under `src/generated//` with `is_current: true` and no orphans; `.prettierignore` and the ESLint `ignores` entry were added **before** the write; `zod` and `@pipelex/sdk` were added with npm; the call site, client helper, and wire-output helper exist where §5 says; `npm run check` (extended with `codegen:check`) passes; `tsc --noEmit` passes; `sources.json` matches §4.6. +- [x] **TS-2 the bytes are untouched.** `git diff --stat` shows no change under `src/generated/` after the skill's own format run; `npm run codegen:check` exits 0. +- [x] **TS-3 refresh after a bundle edit.** Change a concept field in `main.mthds` (through `/pipelex-edit`, which should announce the staleness — Phase 2 wiring), run the skill again: the sidecar comparison reports the changed source, the fingerprint moved, the call site is edited only if the type check demanded it, `sources.json` carries the new hash, `codegen:check` is green again. Then edit only a prompt (no concept change): the fingerprint is unchanged and the report says restamp-or-nothing. +- [x] **TS-4 stale-source gate.** Edit the bundle and do *not* refresh: `npm run codegen:check` exits 1 with `stale-source` and the refresh remedy. - [ ] **TS-5 `method_ref` source.** Integrate a published address at a tag (a `github.com/Pipelex/…@vX.Y.Z` package): the call site runs by `method_ref`, the sidecar's `sources` is empty, the output-concept heuristic of §4.3 either finds one candidate or asks — record which. - [ ] **TS-6 `method_id` warning.** Integrate a catalog method by id: the one-line warning appears, the recommendation is stated, and the skill proceeds only on confirmation. - [ ] **TS-7 orphan.** Generate a second method into the first method's directory on purpose (by naming the dir explicitly): `orphans[]` is reported by name, nothing is deleted, the fix sentence is the tool's. - [ ] **TS-8 foreign file.** Point at a directory holding a hand-written `types.ts`: the refusal is surfaced, the file is untouched, the skill chooses or asks for another directory. - [ ] **TS-9 containment escape.** Launch the harness from a sibling directory so the project is outside the workshop's working directory: the skill stops with the relaunch instruction and does not ride content. - [ ] **TS-10 no key.** Unset `PIPELEX_API_KEY` in the workshop's environment: the `config` stop with the hint verbatim, nothing written. -- [ ] **PY-1 fresh integration, pydantic audience.** No `pipelex` dependency → `python-pydantic` chosen without a question; `/generated/__init__.py` and the subpackage `__init__.py` created; `[tool.ruff] exclude` gains the tree; pyright still covers it; `pydantic` and `pipelex-sdk` added with uv; the async call site plus a sync wrapper if the project is synchronous; the report states the gate asymmetry; `uv run pyright` passes. -- [ ] **PY-2 structures audience.** Add `pipelex` as a dependency and a `@pipe_func` file: `python-structures` is chosen (or offered first when only the dependency is present); `pipelex codegen check ` is wired into the existing gate. -- [ ] **PY-3 refresh after a bundle edit**, as TS-3, including the `/pipelex-edit` staleness notice. +- [x] **PY-1 fresh integration, pydantic audience.** No `pipelex` dependency → `python-pydantic` chosen without a question; `/generated/__init__.py` and the subpackage `__init__.py` created; `[tool.ruff] exclude` gains the tree; pyright still covers it; `pydantic` and `pipelex-sdk` added with uv; the async call site plus a sync wrapper if the project is synchronous; the report states the gate asymmetry; `uv run pyright` passes. +- [x] **PY-2 structures audience.** Add `pipelex` as a dependency and a `@pipe_func` file: `python-structures` is chosen (or offered first when only the dependency is present); `pipelex codegen check ` is wired into the existing gate. +- [x] **PY-3 refresh after a bundle edit**, as TS-3, including the `/pipelex-edit` staleness notice. + +**Integrate scenarios run 2026-09-07.** The seven boxes above are green; TS-5 through TS-10 and the harness and scaffold scenarios below are not yet run. How they were driven, because it is not obvious from the boxes: each scenario is a **fresh headless session** — `claude -p "" --plugin-dir /pipelex --model sonnet` — launched from inside the scratch project, so the skill met a cold agent with no knowledge of this campaign, and the plugin came from the worktree rather than the installed marketplace copy. Sonnet rather than the default model, deliberately: a skill whose text carries the work should carry it on the ordinary workhorse. The workshop was the local `../pipelex-mcp` checkout at `efddebe` (`make build-local`), which is what makes the `main_pipe` path exercisable at all — published `@pipelex/mcp` 0.13.0 still lacks it. Transcripts are in the session scratchpad under `runs/`. + +**One environment fact that cost a run and is worth writing down.** The first TS-1 attempt failed at `mthds_validate` with a `config`-class `Forbidden`. The key in the shell environment is **dev-scoped**: `GET /v1/me` answers 200 on `https://api-dev.pipelex.com` and 403 on `https://api.pipelex.com`, which the workshop defaults to. Every run since carries `PIPELEX_BASE_URL=https://api-dev.pipelex.com`. The skill's own behaviour on that failure was exactly the design: it stopped, surfaced the tool's `hint` verbatim, wrote nothing, and did not guess a signature — `git status` was clean afterwards. Phase 4's live run against the **published** workshop needs a key that works on whichever API that run targets. Harness scenarios (§4.12) — each on a fresh scratch copy of a starter, never on the starter checkout itself: @@ -216,9 +220,15 @@ Carried in the design's §9 and §8; repeated here only where a phase might be t - **2026-09-06 — session paused after Phases 1, 1b and 2, before Phase 3.** Everything is in the worktree `_pipelex-plugins--codegen` on `feature/Codegen`, **uncommitted**: the two templates, their references, the two test modules, the family-wiring edits to `pipelex-design` / `pipelex-edit` / `pipelex-inputs`, `docs/decisions.md`, `CLAUDE.md`, `README.md`, `docs/build-targets.md`, `CHANGELOG.md`, the regenerated `pipelex*/` outputs, and the amended `design.md`, `plan.md` and new `scaffold-design.md`. **Cold start:** `ledger claim L-260830-344594 --renew` from the worktree, read this file's Phase 3, review the diff (`git status`, `git diff`), commit with explicit `git add ` per the rules above, then run Phase 3 against the local `../pipelex-mcp` checkout (`/pipelex-mcp-source`) — the published `@pipelex/mcp` 0.13.0 lacks `main_pipe`, so the ordinary signature path only exercises on the local build. Two small follow-ups noticed and not done: the Phase 4 `create-pipelex-app` decision is `L-260906-84bb41`; the synthetic-inputs campaign's `plan.md` still says `active` after its item closed (doctor `doc-active-after-close`, unrelated to this campaign). +- **2026-09-07 — Phases 1, 1b and 2 committed, and the integrate half of Phase 3 run.** The session opened by verifying the paused work (`make check`, `make agent-test` green on all three targets, no dev launcher switch in the tree) and committing it as `54cf04d`. Phase 3 then ran the seven integrate scenarios the two languages share, each as a cold headless session against the local workshop. **What the skill got right without being watched:** it validated before generating; it wrote the formatter and linter exclusions *before* the tree existed, unprompted, in both languages; every `mthds_codegen` call passed `output_dir` and no artifact byte came back through the conversation; the sidecar matched §4.6 to the letter, including a `pipe` record read from the verdict's `main_pipe` rather than from the bundle; refresh mode re-derived only what the table allows and left the call site alone both times the signature held; and the target choice was made silently and correctly on the audience signal in both Python projects — `python-pydantic` for the consumer, `python-structures` for the host carrying a `@pipe_func`. The TypeScript gate went red on a stale source with exit 1 and the refresh remedy, and green again after the refresh. **What it got wrong** is the three deviations logged below; all three are reconciled into the templates and the design, and the one that could be re-tested in the same session was, and passed. **One thing struck through by evidence rather than by reading a ledger row:** the generated `owner: z.string().nullish()` confirms the ts-zod emitter fix is live on the API the dogfood used, so `wire-output.ts` stays unwritten. + ## Deviations from the design -*(empty at writing — a deviation is logged here with its reason before the code that embodies it is committed.)* +**2026-09-07 — §4.10's fingerprint rule was wrong, and the dogfood proved it.** The design said a `crate_fingerprint` unchanged against the old lock's means a restamp at most, and a changed one means the concept set moved. Scenario TS-3's second half — a **system-prompt-only** edit, no concept touched — moved the fingerprint (`fc27bd…` → `088d66…`), because the fingerprint covers the whole bundle rather than its concept set. What did not move was the lock's `artifacts[].content_hash`: the only changed line in `types.ts` and in `binder.ts` was the one stamp line carrying the fingerprint, and the artifact hashes are computed over the content beneath the stamp, so they are the honest signal. Both §4.10 and the skill's refresh-mode section now read the outcome from `artifacts[].content_hash`, with the fingerprint named as the whole-bundle value it is. The cold agent recovered on its own — it diffed the tree and reported the restamp correctly — but only because it went and looked; the rule as written would have had it announce a concept change that had not happened. + +**2026-09-07 — the skill and its Python reference contradicted each other about where a bundle lives.** Step 1 said flatly that a bundle already inside the project stays where it is; `references/python.md`'s "Packaged for distribution" row said a packaged project's bundle goes *inside* the import package. Scenario PY-1 hit the seam and followed the reference, moving `methods/gantt/` to `reportkit/methods/gantt/` with `git mv` and saying why. The reference is right, and the dogfood proved it rather than assuming it: `uv build --wheel` on the result ships `reportkit/methods/gantt/main.mthds` and `reportkit/generated/gantt/codegen.lock`, which a root-level `methods/` would not have. Step 1 now names the packaged-project exception, points at the reference for it, and asks for `git mv` so the move reads as one rename. PY-2 later made the same move on its own and verified its own wheel. + +**2026-09-07 — the family's staleness notice looked in the wrong place.** Phase 2 wired `/pipelex-edit` and `/pipelex-design` to warn when an edit invalidates a generated tree. Run against the real thing, `/pipelex-edit` searched for `sources.json` **beside the bundle**, found none, and reported that no downstream refresh was needed — while the sidecar sat in `src/generated/gantt/`, exactly where the skill puts it. The sentence had named the location as background prose ("a project keeps one beside each generated tree, typically under…") instead of as an instruction about where to search. Both wirings now say to search the whole project, give the grep, and state that the sidecar never sits beside the bundle. Re-run afterwards on PY-1, the notice fired correctly — via the `/pipelex-design` route, since a concept-structure change is structural, so both edited wirings were exercised. ## Where everything is From 40e789066ce94f0e002a2e636807a7759fe49e8a Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Mon, 7 Sep 2026 08:22:15 +0200 Subject: [PATCH 04/21] Record the api-dev ruling and park the Python target question upstream MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two rulings from Louis, both recorded in the tracker rather than acted on in the templates. This work stays on api-dev.pipelex.com until he says otherwise: the key on this machine is dev-scoped and updating the prod environment is off his list for a while. Phase 4's live-run step said "live" in a way that read as production; it now says the published workshop against the dev API, and notes that a prod 403 is the expected state rather than a defect to chase. The Python two-audience rule stands as ratified. Asked why a host carrying a @pipe_func gets python-structures at all, the answer is that the runtime enforces it — func_registry.py:372 refuses a pipe func whose return type is not a StuffContent subclass — so the target is what makes a pipe func writable. Rather than re-cut the rule, Louis asked for the requirement to be relaxed at its source. That is L-260907-3ea0c0 against pipelex, and the target question is parked on it. Advances L-260830-344594 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DYrbgEm55V4QP3uJmgTqRS --- wip/pipelex-integrate/plan.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index 4b73156..cc0da1d 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -192,7 +192,7 @@ Owner: `pipelex-plugins`. **Gate, hard:** a published `@pipelex/mcp` version tha - [ ] Published `@pipelex/mcp` version carrying `mthds_codegen`: **0.13.0** (published 2026-08-30; the plugin's `@latest` already resolves to it). - [ ] Published `@pipelex/mcp` version carrying **`main_pipe` on the validate verdict**: `__________` (fill in — it sits under `[Unreleased]` in `pipelex-mcp/CHANGELOG.md` at writing). The skill's ordinary path reads the signature from there; on an older workshop it takes the fallback of §4.3 (template for the inputs, the bundle for the output) and says the workshop predates the signature. Ship after this release so users never meet the fallback by default. - [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` one last time; strike or keep the helper and the asymmetry paragraph accordingly, and log it. -- [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. +- [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. "Live" here means the published workshop, **not** the production API: Louis ruled on 2026-09-07 that this work stays on `https://api-dev.pipelex.com` until he says otherwise, because the key on his machine is dev-scoped and updating the prod environment is off his list for a while. So the run carries `PIPELEX_BASE_URL=https://api-dev.pipelex.com`, and a prod 403 is the expected state rather than a defect to chase. - [ ] Open the PR against `dev` with `Closes L-260830-344594` and `Closes L-260906-8ac105` in the body; work the review rounds per the workspace's tightening-bar rule; land with `/ledger-land`. `L-260831-b67e18` closes in the same landing, with the release as evidence that the heuristic never shipped. - [ ] File the starter-README pointer items (S§9, "to file at release"): both starters' "Use this template" sections name `/pipelex-scaffold` beside the button and `/bootstrap`. - [ ] `/release` → the next minor (`0.6.0`), which cuts the changelog heading, bumps every target TOML and the Claude marketplace, and opens the release PR against `main`. @@ -222,6 +222,8 @@ Carried in the design's §9 and §8; repeated here only where a phase might be t - **2026-09-07 — Phases 1, 1b and 2 committed, and the integrate half of Phase 3 run.** The session opened by verifying the paused work (`make check`, `make agent-test` green on all three targets, no dev launcher switch in the tree) and committing it as `54cf04d`. Phase 3 then ran the seven integrate scenarios the two languages share, each as a cold headless session against the local workshop. **What the skill got right without being watched:** it validated before generating; it wrote the formatter and linter exclusions *before* the tree existed, unprompted, in both languages; every `mthds_codegen` call passed `output_dir` and no artifact byte came back through the conversation; the sidecar matched §4.6 to the letter, including a `pipe` record read from the verdict's `main_pipe` rather than from the bundle; refresh mode re-derived only what the table allows and left the call site alone both times the signature held; and the target choice was made silently and correctly on the audience signal in both Python projects — `python-pydantic` for the consumer, `python-structures` for the host carrying a `@pipe_func`. The TypeScript gate went red on a stale source with exit 1 and the refresh remedy, and green again after the refresh. **What it got wrong** is the three deviations logged below; all three are reconciled into the templates and the design, and the one that could be re-tested in the same session was, and passed. **One thing struck through by evidence rather than by reading a ledger row:** the generated `owner: z.string().nullish()` confirms the ts-zod emitter fix is live on the API the dogfood used, so `wire-output.ts` stays unwritten. +- **2026-09-07 — the Python two-audience rule is left as ratified, and the real fix was sent upstream (Louis).** Reviewing the PY-2 result, Louis questioned why a host carrying a `@pipe_func` gets `python-structures` at all, since that target puts Pipelex classes into the project's own generated types. The rule was checked rather than defended: the runtime enforces it hard — `pipelex/system/registries/func_registry.py:372` refuses a pipe func with *"return type must be a subclass of StuffContent, but is '…'"* — so `python-structures` is what makes a pipe func writable, and the skill never adds `pipelex` to reach that target. The imprecision the dogfood did expose is that §5 keys the choice on the **project** (a `@pipe_func` appears somewhere) when the decisive question is **per-method** (does a pipe func return *this* method's concepts?); in PY-2 nothing consumed the structures but the call site, which a plain `BaseModel` would have served. Three rules were put to Louis — a per-method test with `python-pydantic` offered first, the ratified project-level rule unchanged, or never choosing structures unaided. **He chose none of them and asked instead that the requirement be relaxed at its source**, which is `L-260907-3ea0c0` against `pipelex`: let a `@pipe_func` return a plain pydantic model and have the runtime adapt it. `StuffContent` is already pydantic (`stuff_content.py:15`) and what it adds over `BaseModel` is a generic rendering and dump surface, so the ask is plausible rather than speculative. **§5 therefore stands unchanged and this is not a deviation** — the target question is parked on that item, and the rule is revisited when it lands, not before. + ## Deviations from the design **2026-09-07 — §4.10's fingerprint rule was wrong, and the dogfood proved it.** The design said a `crate_fingerprint` unchanged against the old lock's means a restamp at most, and a changed one means the concept set moved. Scenario TS-3's second half — a **system-prompt-only** edit, no concept touched — moved the fingerprint (`fc27bd…` → `088d66…`), because the fingerprint covers the whole bundle rather than its concept set. What did not move was the lock's `artifacts[].content_hash`: the only changed line in `types.ts` and in `binder.ts` was the one stamp line carrying the fingerprint, and the artifact hashes are computed over the content beneath the stamp, so they are the honest signal. Both §4.10 and the skill's refresh-mode section now read the outcome from `artifacts[].content_hash`, with the fingerprint named as the whole-bundle value it is. The cold agent recovered on its own — it diffed the tree and reported the restamp correctly — but only because it went and looked; the rule as written would have had it announce a concept change that had not happened. From 245c978e5f0b8b2e03d8782570d3cca79fcf919c Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sat, 12 Sep 2026 23:56:40 +0200 Subject: [PATCH 05/21] Name all three reasons the validate verdict carries no main_pipe pipelex-mcp dev settles the entry pipe from the report's default_pipe_ref, which follows a package's METHODS.toml, and omits main_pipe whole in three cases: the method settles no entry pipe, the entry pipe's contract did not come back whole, or the workshop predates the signature. The skill named only the first and the third, and told a by-ref or by-id source to blame an old workshop. Step 3 now names all three, says the call site is typed and run against main_pipe.pipe_ref (which can differ from the bundle's own main_pipe for a published package), and the stop for a by-ref or by-id source no longer asserts which cause applies. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-integrate/SKILL.md | 6 +++--- pipelex-vibe/skills/pipelex-integrate/SKILL.md | 6 +++--- pipelex/skills/pipelex-integrate/SKILL.md | 6 +++--- templates/skills/pipelex-integrate/SKILL.md.j2 | 6 +++--- tests/unit/test_pipelex_integrate_skill.py | 6 +++++- 5 files changed, 17 insertions(+), 13 deletions(-) diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index 3f620a3..2bd83af 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -69,9 +69,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -197,7 +197,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index aaa4b41..12e1194 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -69,9 +69,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -197,7 +197,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index 65b045f..10050a7 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -80,9 +80,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -208,7 +208,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index d2a070d..8b918cf 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -75,9 +75,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** It is omitted whole when the bundle declares no main pipe — or when the workshop predates the signature (older `@pipelex/mcp` releases do not carry it; `npx -y @pipelex/mcp@latest` refreshes it). Then: ask which pipe to integrate if the bundle declares no main pipe; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle for a files source. For a `method_ref` or `method_id` source with no `main_pipe`, the output concept has no in-context channel: STOP and say the workshop is too old to type this integration exactly, rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -203,7 +203,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the workshop predates the signature — refresh `@pipelex/mcp` and retry; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index c88f714..030ec01 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -50,7 +50,11 @@ def test_signature_comes_from_the_verdict_and_the_heuristic_is_absent(self) -> N assert "candidate output concepts" not in body assert "minus the input concepts minus natives" not in body # A by-ref / by-id source with no signature stops instead of guessing. - assert "STOP and say the workshop is too old to type this integration exactly, rather than guessing" in body + assert "STOP and say the verdict carries no signature to type this integration exactly" in body + # The entry pipe is the verdict's, which follows a package manifest; its absence has three causes, not one. + assert "`main_pipe.pipe_ref` is the pipe a run with no pipe selector executes" in body + assert "the method settles **no entry pipe**" in body + assert "the workshop **predates the signature**" in body def test_the_wire_null_helper_is_never_installed(self) -> None: body = self.integrate From 161c2a15a3ce0cfd6a41d490efc435bb50b349eb Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sat, 12 Sep 2026 23:58:35 +0200 Subject: [PATCH 06/21] Refuse an occupied generated directory without a sidecar naming the method Two dogfood findings on the integrate skill. First, every method of a target emits the same file names, so generating a second method into a directory that already holds another method's tree overwrites it outright and reports no orphan. The template only guarded a lock whose sidecar names a different method, while design section 4.7 says any lock not named by this method's sidecar; step 4 now carries that rule, including a lock with no sidecar at all and a directory the user named. Second, faced with a hand-written file at an artifact path, the skill offered to delete it. It now says never to offer that, and applies the rule when the file is noticed before the tool is called. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-integrate/SKILL.md | 8 ++++---- pipelex-vibe/skills/pipelex-integrate/SKILL.md | 8 ++++---- pipelex/skills/pipelex-integrate/SKILL.md | 8 ++++---- templates/skills/pipelex-integrate/SKILL.md.j2 | 8 ++++---- tests/unit/test_pipelex_integrate_skill.py | 4 ++++ 5 files changed, 20 insertions(+), 16 deletions(-) diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index 2bd83af..e9b9d94 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -83,7 +83,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -98,7 +98,7 @@ Branch on the structured result: - `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. - `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. - `status: "error"`, class `input_domain` located at `output_dir`: - - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file, and never offer to** — the file is the user's and not in the way; the directory choice was wrong. The same holds when you notice such a file before calling the tool: pick or ask for another directory. - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. - `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. - Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. ## A project that owns a codegen harness @@ -192,7 +192,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | -| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index 12e1194..5629e52 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -83,7 +83,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -98,7 +98,7 @@ Branch on the structured result: - `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. - `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. - `status: "error"`, class `input_domain` located at `output_dir`: - - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file, and never offer to** — the file is the user's and not in the way; the directory choice was wrong. The same holds when you notice such a file before calling the tool: pick or ask for another directory. - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. - `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. - Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. ## A project that owns a codegen harness @@ -192,7 +192,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | -| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index 10050a7..3403252 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -94,7 +94,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -109,7 +109,7 @@ Branch on the structured result: - `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. - `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. - `status: "error"`, class `input_domain` located at `output_dir`: - - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file, and never offer to** — the file is the user's and not in the way; the directory choice was wrong. The same holds when you notice such a file before calling the tool: pick or ask for another directory. - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. - `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. - Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. @@ -179,7 +179,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. ## A project that owns a codegen harness @@ -203,7 +203,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | -| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index 8b918cf..d7899a4 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -89,7 +89,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -104,7 +104,7 @@ Branch on the structured result: - `status: "ok"`, `is_valid: true`, and `output_dir` present in the result → written. Confirm **`is_current: true`** and an empty **`orphans[]`**. Every `artifacts[]` entry and the `lock` carry `written_to`; nothing carries `content`. - `is_valid: false` → the method regressed since step 2 — back to step 2's repair route. - `status: "error"`, class `input_domain` located at `output_dir`: - - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file.** + - the message says `output_dir already holds …, which this tool does not own` → a foreign file, a symlink or a directory sits at an artifact path. The whole write was refused and the tree is byte-identical; this is not a dedicated generated directory. Choose or ask for one that is. **Never delete, move or "clear" the named file, and never offer to** — the file is the user's and not in the way; the directory choice was wrong. The same holds when you notice such a file before calling the tool: pick or ask for another directory. - the hint says files stay inside the directory the host started the server in → the containment escape above: STOP with the relaunch instruction. - `status: "error"`, class `runtime`, `retryable: true` (a partial write) → call again **once** with the same `output_dir`, as the hint says: regeneration overwrites its own stamped files. Then report what landed, in the tool's words. - Success with **`orphans[]` non-empty** → the chosen directory was not fresh (an earlier generation, another target, an engine rename). Report the paths by name, say what they are, do nothing else; a dedicated directory per generation is the fix, in the tool's own words. `orphans_truncated: true` → say orphan detection was partial rather than reporting a clean tree. @@ -174,7 +174,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` whose sidecar names a **different** method is not this method's: choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. ## A project that owns a codegen harness @@ -198,7 +198,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | -| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file | +| `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index 030ec01..69017ef 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -70,6 +70,10 @@ def test_failure_posture_pins_the_403_and_the_orphans(self) -> None: assert '`kind: "paywall"`' in body assert "report by name, never delete" in body assert "never delete, move or clear the named file" in body + assert "and never offer to" in body + # A lock is not this method's without a sidecar naming it: same file names mean a silent overwrite, not an orphan. + assert "is this method's only when a `sources.json` beside it names this method" in body + assert "would silently overwrite the other method's stamped files rather than report an orphan" in body assert "report `drifts[]` verbatim and stop" in body def test_method_id_warns_and_refresh_leaves_the_call_site_alone(self) -> None: From 907698e161d6fae4ffe8e8daac0d1b406b16a6c1 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:00:21 +0200 Subject: [PATCH 07/21] Keep the scaffold skill from offering to clear a non-empty directory Asked to scaffold into a directory holding a file of the user's, the skill refused to write into it and then offered to move the file aside and merge it back afterwards. The refusal is the whole rule (scaffold-design.md section 7): the answer to an occupied directory is another directory, never making room in this one. The branch table and the failure table now say so, and a test pins it. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-scaffold/SKILL.md | 4 ++-- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 4 ++-- pipelex/skills/pipelex-scaffold/SKILL.md | 4 ++-- templates/skills/pipelex-scaffold/SKILL.md.j2 | 4 ++-- tests/unit/test_pipelex_scaffold_skill.py | 1 + 5 files changed, 9 insertions(+), 8 deletions(-) diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 6f50fbe..9bfca98 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -142,7 +142,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index 90a4401..d10386d 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -142,7 +142,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index f9a1991..cd77f10 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -30,7 +30,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -149,7 +149,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index 0297634..a09599a 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -142,7 +142,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index c056c97..395129c 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -25,6 +25,7 @@ class TestPipelexScaffoldSkill: "exactly two branches and carries no templates of its own", "no cookiecutter, no copier, no framework matrix of its own", "never write into a directory that exists and is not empty", + "never offer to move, delete or merge what it holds to make room", "This is the **one commit this skill makes**", "Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written", "**Add nothing to that procedure and reimplement none of it.**", From be82a3b5043eedec738b4eccbe3da4ace899762a Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:02:14 +0200 Subject: [PATCH 08/21] Let the scaffold skill reach a runtime a version manager already holds Dogfooded on a PATH with no node, the skill found the machine's nvm, activated it and carried on cloning and installing. That is the useful answer and it installs nothing, but the skill's prerequisites read as if PATH were the whole test, so the behaviour was outside what it says. It now distinguishes a runtime the machine lacks from one only the PATH is missing: check nvm, fnm, volta, asdf or mise, use what they already hold, say which one was used and that the user's own shell may not have it, and stop only when no runtime can be reached that way. Installing a toolchain is still never done. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-scaffold/SKILL.md | 4 ++-- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 4 ++-- pipelex/skills/pipelex-scaffold/SKILL.md | 4 ++-- templates/skills/pipelex-scaffold/SKILL.md.j2 | 4 ++-- tests/unit/test_pipelex_scaffold_skill.py | 1 + 5 files changed, 9 insertions(+), 8 deletions(-) diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 9bfca98..1ace51f 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -38,7 +38,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win ### Step 1: Prerequisites -Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). @@ -141,7 +141,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| -| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | | The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index d10386d..fed5303 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -38,7 +38,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win ### Step 1: Prerequisites -Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). @@ -141,7 +141,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| -| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | | The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index cd77f10..9be9760 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -45,7 +45,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win ### Step 1: Prerequisites -Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). @@ -148,7 +148,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| -| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | | The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index a09599a..e14aabc 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -38,7 +38,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win ### Step 1: Prerequisites -Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain. +Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). @@ -141,7 +141,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| -| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | +| A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | | The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 395129c..51baa58 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -34,6 +34,7 @@ class TestPipelexScaffoldSkill: "do not start `make dev`", "Add **no** SDK dependency and create **no** empty `methods/` directory", "Nothing beyond what the initializer writes is authored by this skill", + "a runtime the machine already has and only the `PATH` is missing is not a missing piece", ) @property From 82c2c15ef705ddc3f8613d342e13b807ff364570 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:04:48 +0200 Subject: [PATCH 09/21] Name the emitter's extensionless import as a known defect Two dogfood runs on cold TypeScript projects failed their own type check inside the generated tree: the ts-zod emitter writes binder.ts's import as `from "./types"`, which a plain Node ESM project (type module with moduleResolution nodenext or node16) rejects, and so does Node at runtime. Both runs diagnosed it from scratch and refused to patch the stamped file, which is the right posture but not something each user should have to rediscover. The failure table and the TypeScript reference now name the defect, say the fix is upstream in the emitter, and say that moving the project to a bundler resolution is the user's decision, not the skill's. Filed against pipelex as L-260912-857a5a. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-integrate/SKILL.md | 1 + .../skills/pipelex-integrate/references/typescript.md | 2 ++ pipelex-vibe/skills/pipelex-integrate/SKILL.md | 1 + pipelex-vibe/skills/pipelex-integrate/references/typescript.md | 2 ++ pipelex/skills/pipelex-integrate/SKILL.md | 1 + pipelex/skills/pipelex-integrate/references/typescript.md | 2 ++ skills/pipelex-integrate/references/typescript.md | 2 ++ templates/skills/pipelex-integrate/SKILL.md.j2 | 1 + tests/unit/test_pipelex_integrate_skill.py | 3 +++ 9 files changed, 15 insertions(+) diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index e9b9d94..cd977d8 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -199,6 +199,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | | `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/pipelex-codex/skills/pipelex-integrate/references/typescript.md b/pipelex-codex/skills/pipelex-integrate/references/typescript.md index bce5bfa..84dc2d9 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/typescript.md +++ b/pipelex-codex/skills/pipelex-integrate/references/typescript.md @@ -25,6 +25,8 @@ Why the exclusions are not optional: the ts-zod emitter prints at Prettier's def - `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. - `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. + + **Known defect:** `binder.ts` imports its sibling as `from "./types"`, with no extension. On a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) that fails the type check with `TS2835` and the compiled code with `ERR_MODULE_NOT_FOUND`; a bundler resolution (`bundler`, `node10`) is unaffected, which is why the JS starter never meets it. The fix belongs to the emitter. Say so plainly and leave the file alone: it is stamped and hashed, so a patch breaks the trust chain and the next regeneration drops it. Whether to move the project to a bundler resolution meanwhile is the user's decision. - `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index 5629e52..80d18e8 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -199,6 +199,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | | `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/pipelex-vibe/skills/pipelex-integrate/references/typescript.md b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md index bce5bfa..84dc2d9 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/typescript.md +++ b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md @@ -25,6 +25,8 @@ Why the exclusions are not optional: the ts-zod emitter prints at Prettier's def - `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. - `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. + + **Known defect:** `binder.ts` imports its sibling as `from "./types"`, with no extension. On a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) that fails the type check with `TS2835` and the compiled code with `ERR_MODULE_NOT_FOUND`; a bundler resolution (`bundler`, `node10`) is unaffected, which is why the JS starter never meets it. The fix belongs to the emitter. Say so plainly and leave the file alone: it is stamped and hashed, so a patch breaks the trust chain and the next regeneration drops it. Whether to move the project to a bundler resolution meanwhile is the user's decision. - `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index 3403252..3369dfa 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -210,6 +210,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | | `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/pipelex/skills/pipelex-integrate/references/typescript.md b/pipelex/skills/pipelex-integrate/references/typescript.md index bce5bfa..84dc2d9 100644 --- a/pipelex/skills/pipelex-integrate/references/typescript.md +++ b/pipelex/skills/pipelex-integrate/references/typescript.md @@ -25,6 +25,8 @@ Why the exclusions are not optional: the ts-zod emitter prints at Prettier's def - `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. - `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. + + **Known defect:** `binder.ts` imports its sibling as `from "./types"`, with no extension. On a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) that fails the type check with `TS2835` and the compiled code with `ERR_MODULE_NOT_FOUND`; a bundler resolution (`bundler`, `node10`) is unaffected, which is why the JS starter never meets it. The fix belongs to the emitter. Say so plainly and leave the file alone: it is stamped and hashed, so a patch breaks the trust chain and the next regeneration drops it. Whether to move the project to a bundler resolution meanwhile is the user's decision. - `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. diff --git a/skills/pipelex-integrate/references/typescript.md b/skills/pipelex-integrate/references/typescript.md index bce5bfa..84dc2d9 100644 --- a/skills/pipelex-integrate/references/typescript.md +++ b/skills/pipelex-integrate/references/typescript.md @@ -25,6 +25,8 @@ Why the exclusions are not optional: the ts-zod emitter prints at Prettier's def - `types.ts` — stamped; `import { z } from "zod"`; one `export const XSchema = z.object({...})` and `export type X = z.infer` per concept, natives included; non-required fields are `.nullish()`, so the schema parses the runtime's explicit `null`s directly. - `binder.ts` — stamped; `export function parseX(wire: unknown): X` and `export function serializeX(value: X): X` per concept, over the pure schemas. Field keys are wire-native snake_case. + + **Known defect:** `binder.ts` imports its sibling as `from "./types"`, with no extension. On a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) that fails the type check with `TS2835` and the compiled code with `ERR_MODULE_NOT_FOUND`; a bundler resolution (`bundler`, `node10`) is unaffected, which is why the JS starter never meets it. The fix belongs to the emitter. Say so plainly and leave the file alone: it is stamped and hashed, so a patch breaks the trust chain and the next regeneration drops it. Whether to move the project to a bundler resolution meanwhile is the user's decision. - `codegen.lock` — TOML: `lock_version`, `crate_fingerprint`, `engine_version`, one `[[artifacts]]` entry per stamped file with its `content_hash`. Beside them the skill writes `sources.json`, unstamped; it is never an artifact and never an orphan. Everything stamped is read-only and formatter-free. A consumer imports the type from `types.ts` and the parser from `binder.ts`; anything it wants to add goes in its own module, never in the generated one. diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index d7899a4..a362261 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -205,6 +205,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | | `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index 69017ef..90c0a72 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -75,6 +75,9 @@ def test_failure_posture_pins_the_403_and_the_orphans(self) -> None: assert "is this method's only when a `sources.json` beside it names this method" in body assert "would silently overwrite the other method's stamped files rather than report an orphan" in body assert "report `drifts[]` verbatim and stop" in body + # The emitter's extensionless import is a known defect: named, never patched in the stamped tree. + assert 'a known defect of the ts-zod emitter, which writes `from "./types"`' in body + assert "Changing the project's `moduleResolution` is the user's call to make, not yours" in body def test_method_id_warns_and_refresh_leaves_the_call_site_alone(self) -> None: body = self.integrate From 917ee3bc12c768ef4a27fbadfcc7547fe83db4f9 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:04:53 +0200 Subject: [PATCH 10/21] Catch a project the workshop cannot reach before generating, not after Dogfooded with the harness launched in a sibling directory, the skill never met the containment error the rule is written around: it passed an output_dir relative to the workshop's working directory, the write landed legally beside the wrong project, and the run then moved the tree into place and carried on. The bytes survived that, but the project was left with a refresh that hits the same mismatch every time and a sidecar whose project-relative paths name a project the workshop cannot see. Step 6 now says to read the path from the workshop's working directory to the generated directory before calling, treat a climbing path or a project root elsewhere on disk as the stop, and never write beside the wrong project to move the tree over afterwards. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-integrate/SKILL.md | 3 ++- pipelex-vibe/skills/pipelex-integrate/SKILL.md | 3 ++- pipelex/skills/pipelex-integrate/SKILL.md | 3 ++- templates/skills/pipelex-integrate/SKILL.md.j2 | 3 ++- tests/unit/test_pipelex_integrate_skill.py | 2 ++ 5 files changed, 10 insertions(+), 4 deletions(-) diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index cd977d8..35988da 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -91,7 +91,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root — do not ride content instead. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -192,6 +192,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| the project root is not the workshop's working directory or below it (the harness was launched elsewhere) | STOP **before** generating, with the same relaunch instruction — the tool cannot catch this, because a path inside the workshop is legal wherever it points; never write beside the wrong project and move the tree over | | `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index 80d18e8..f89ce7d 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -91,7 +91,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root (or register the workshop with that working directory) — do not ride content instead. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root (or register the workshop with that working directory) — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -192,6 +192,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| the project root is not the workshop's working directory or below it (the harness was launched elsewhere) | STOP **before** generating, with the same relaunch instruction — the tool cannot catch this, because a path inside the workshop is legal wherever it points; never write beside the wrong project and move the tree over | | `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index 3369dfa..eeadb0b 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -102,7 +102,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root — do not ride content instead. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -203,6 +203,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| the project root is not the workshop's working directory or below it (the harness was launched elsewhere) | STOP **before** generating, with the same relaunch instruction — the tool cannot catch this, because a path inside the workshop is legal wherever it points; never write beside the wrong project and move the tree over | | `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index a362261..077c274 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -97,7 +97,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. A project outside that directory cannot be written to: STOP with the instruction to relaunch the harness from the project root{% if platform == "mistral-vibe" %} (or register the workshop with that working directory){% endif %} — do not ride content instead. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root{% if platform == "mistral-vibe" %} (or register the workshop with that working directory){% endif %} — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -198,6 +198,7 @@ The harness owns the layout and the check; its generator is preferred, not manda | `is_valid: false` | route `validation_errors[]` to `/pipelex-design` or `/pipelex-edit` | | not runnable, or `pending_signatures` non-empty | STOP: finish the method with `/pipelex-design`; nothing generated | | `input_domain` at `output_dir`, containment escape | STOP: relaunch the harness from the project root; never ride content | +| the project root is not the workshop's working directory or below it (the harness was launched elsewhere) | STOP **before** generating, with the same relaunch instruction — the tool cannot catch this, because a path inside the workshop is legal wherever it points; never write beside the wrong project and move the tree over | | `input_domain` at `output_dir`, a file this tool does not own | not a dedicated generated directory — choose another or ask; never delete, move or clear the named file, and never offer to | | `input_domain` at `method_ref` / `method_id` | report the selector failure in the tool's words | | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index 90c0a72..3d6fa04 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -31,6 +31,8 @@ class TestPipelexIntegrateSkill: "**A project that owns a codegen harness keeps it.** Never write a second generated layout", "**No `dropWireNulls` / `wireOutput` helper.**", "Do this **before** step 6", + "**Check containment before the call rather than waiting for an error**", + "never write the tree into the workshop's own directory and move it across afterwards", ) @property From 3fe3327245a863eb52bb081b97f044d7801114bb Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:07:08 +0200 Subject: [PATCH 11/21] Record what the dogfood changed, in the designs and the decisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two designs carry the amendments the Phase 3 runs forced — the three reasons a verdict has no main_pipe and the entry pipe following a package manifest, containment read before the first write, an occupied generated directory refused however it was named, a non-empty target never cleared, and a runtime behind a version manager — and docs/decisions.md gains a dated entry for the same set so a reader of the repo finds them without opening the campaign. The changelog's integrate entry now names the two silent failures the skill refuses and the emitter defect it reports rather than patches. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- docs/decisions.md | 11 +++++++++++ wip/pipelex-integrate/design.md | 8 +++++--- wip/pipelex-integrate/scaffold-design.md | 4 ++-- 4 files changed, 19 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fe439e1..3d1f687 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- **`pipelex-integrate` — wire an MTHDS method into a Python or TypeScript codebase.** Given a local bundle, a published `method_ref` at a tag, or a catalog `method_id`, and a project, the skill picks the codegen target by audience (`ts-zod`; `python-pydantic` for a hosted-API consumer, `python-structures` for a Pipelex host), has the workshop's `mthds_codegen` write the generated tree into one dedicated directory per method through its `output_dir` arm — no artifact byte ever passes through the model, and a refused write is never worked around by writing bytes from the conversation — excludes that tree from the project's formatters and linters *before* it exists while keeping it under the type checker, records a `sources.json` sidecar (selector, target, pipe signature, source hashes) so a second run is a refresh and a bundle edit is detectable, wires an offline drift gate (`scripts/codegen-check.mjs` over `@pipelex/sdk`'s `runCodegenCheck`) into a TypeScript project's existing check, and writes one typed call-site module per method over `startAndWaitForResult` / `start_and_wait` and the generated binder or model. The call site is typed from the main pipe's signature on the validate verdict. A Python consumer gets no offline gate yet — the Python SDK has no check and the skill will not add the `pipelex` runtime to get one — and the report says so. A project made from a Pipelex starter keeps its own codegen harness: the skill runs the project's `codegen` script or `make add-method` and never writes a second layout beside the first. Ships `references/typescript.md`, `references/python.md` and `references/codegen-check.mjs`. +- **`pipelex-integrate` — wire an MTHDS method into a Python or TypeScript codebase.** Given a local bundle, a published `method_ref` at a tag, or a catalog `method_id`, and a project, the skill picks the codegen target by audience (`ts-zod`; `python-pydantic` for a hosted-API consumer, `python-structures` for a Pipelex host), has the workshop's `mthds_codegen` write the generated tree into one dedicated directory per method through its `output_dir` arm — no artifact byte ever passes through the model, and a refused write is never worked around by writing bytes from the conversation — excludes that tree from the project's formatters and linters *before* it exists while keeping it under the type checker, records a `sources.json` sidecar (selector, target, pipe signature, source hashes) so a second run is a refresh and a bundle edit is detectable, wires an offline drift gate (`scripts/codegen-check.mjs` over `@pipelex/sdk`'s `runCodegenCheck`) into a TypeScript project's existing check, and writes one typed call-site module per method over `startAndWaitForResult` / `start_and_wait` and the generated binder or model. The call site is typed from the main pipe's signature on the validate verdict. A Python consumer gets no offline gate yet — the Python SDK has no check and the skill will not add the `pipelex` runtime to get one — and the report says so. A project made from a Pipelex starter keeps its own codegen harness: the skill runs the project's `codegen` script or `make add-method` and never writes a second layout beside the first. Two things it refuses outright, because both are silent: generating a second method into a directory that already holds another method's tree (every method of a target emits the same file names, so that overwrites the first and reports no orphan), and reaching a project the workshop was not launched in (a path inside the workshop is legal wherever it points, so containment is read before the first write and a tree that landed beside the wrong project is never moved across). One upstream defect it names rather than patches: the `ts-zod` emitter writes `binder.ts`'s sibling import without a file extension, which a plain Node ESM project rejects at type-check and at runtime while a bundler resolution accepts — the generated tree is stamped and hashed, so the fix is the emitter's and the skill says so. Ships `references/typescript.md`, `references/python.md` and `references/codegen-check.mjs`. - **`pipelex-scaffold` — the front door to a project that does not exist yet.** Two branches and no templates of its own: one of the Pipelex starters (`pipelex-starter-js` for a web app with contract-rendered forms, `pipelex-starter-python` for a CLI or service), acquired as a fresh-history local clone by default or through `gh repo create --template` after confirmation, committed once as it came, then renamed by the clone's **own** `bootstrap` skill, read from its `SKILL.md` and never reimplemented; or the ecosystem's initializer (`uv init --package`, `npm create next-app@latest`, …) when the user wants their framework. Both branches end with the env-file convention, the key filled only from the shell environment and never asked for in the conversation, and a hand-off to `/pipelex-integrate`. It is the plugin's third MCP-free skill. Ships `references/starters.md` and `references/initializers.md`. - **`pipelex-vibe/mcp/vibe-mcp.toml` — the Mistral Vibe target bakes the workshop launcher**: Vibe has no plugin manifest, so the target now ships the `npx -y @pipelex/mcp@latest` launcher as a `[[mcp_servers]]` stdio entry to append to the end of `~/.vibe/config.toml`, rendered from the same `[vars.mcp_server]` block as the Claude and Codex manifests and enforced by `make check`. Write your API key into the entry's `env` table, because Vibe spawns stdio servers with a minimal environment and never sees an exported `PIPELEX_API_KEY`; the MCP-backed skills' Vibe stop message now points at the fragment and says so, and their Vibe auth line no longer claims the server reads the session environment. Before appending, delete the `mcp_servers = []` line a new Vibe config carries and any `pipelex` server registered by hand: either leftover stops Vibe from starting. Vibe records its configuration, that key included, in every session log under `~/.vibe/logs/session/`, so redact the key before sharing one. - **`pipelex-synthetic-inputs` — a skill that renders the files a method needs, from code.** PDFs through `reportlab` (canvas letters, multi-page Platypus reports, tables, and a composed line-item document whose totals come from its items) and PNGs through `Pillow` and `matplotlib` in four categories: `chart` (bar, line, pie, scatter), `diagram` (a node/edge list laid out on a grid with clipped arrows), `document_scan` (an A4-at-150-dpi page put through a seeded skew/tint/grain/vignette post-process, for OCR and document-understanding methods) and `screenshot` (window chrome, sidebar, stat tiles, and a status-badged table or card grid). Word and Excel come along from `pipelex-inputs`. No AI is involved anywhere, and only packages whose licences are compatible with MIT are used. **Photographs and handwriting are deliberately out of scope** — code cannot render either to a standard a vision model would accept, so the skill says so and asks for a real file instead of handing a method an imitation. It is MCP-free, the second such skill after `pipelex-explain`, and it installs what it needs itself: `uv` with ephemeral packages, or a venv it creates under the user's cache directory when `uv` is absent. Nothing is installed into the project, and installing a *tool* always asks first. diff --git a/docs/decisions.md b/docs/decisions.md index 849b2a9..4e55e90 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -174,6 +174,17 @@ Every skill before these acts on `.mthds` files; none touched the codebase that - **Two skills, not one.** The integrate design is large and the trigger families differ ("use this method in my app" against "start a new project"); the greenfield tail is integrate anyway, so the hand-off chain is the plugin's existing pattern. - **References carry the language detail, and one of them is a script.** `pipelex-integrate` ships `typescript.md`, `python.md` and `codegen-check.mjs` — the first non-Markdown reference, copied byte for byte into every target and then into users' TypeScript projects, so a change to it is verified by running it, not by rendering it; `pipelex-scaffold` ships `starters.md` and `initializers.md`. +## What the integrate and scaffold dogfood changed (2026-09-12) + +Both skills were run as cold headless sessions against a local `pipelex-mcp` build of `dev`, in scratch projects and in scratch copies of both starters. The rules below are what those runs changed; the scenarios and their verdicts are in `wip/pipelex-integrate/plan.md`, and the designs carry the same amendments. + +- **The validate verdict's `main_pipe` is absent for three reasons, and the skill asserts none of them.** The entry pipe is the API's `default_pipe_ref` — for a published package, its `METHODS.toml` entry rather than the bundle's own declaration, which is why the call site is typed and run against `main_pipe.pipe_ref`. The signature is omitted whole when nothing settles an entry pipe, when the pipe's contract does not narrow, or when the workshop predates the signature. The skill named only the first and the third, and told a by-ref or by-id source to blame an old workshop. +- **An occupied generated directory is refused even when the user names it.** Every method of a target emits the same file names, so generating a second method into another method's directory overwrites it and reports an empty `orphans[]` — a clean-looking generation. The guard is the sidecar: a directory is this method's only when a `sources.json` there names it, a lock with no sidecar included. +- **Containment is read before the first write, not after an error.** A path inside the workshop's working directory is legal wherever it points, so a harness launched beside the project accepts a write into the wrong tree; a run that hit this moved the tree across afterwards and left the project with a refresh that fails the same way every time. The skill now reads the path from the workshop's working directory first, and never moves a tree into place. +- **A file the skill does not own is never cleared, and never offered for clearing.** Integrate met a hand-written file at an artifact path and offered to delete it; scaffold met a non-empty target directory and offered to move its contents aside and merge them back. The answer in both cases is another directory. +- **A runtime behind a version manager is not a missing toolchain.** Scaffold, on a `PATH` without `node`, found the machine's `nvm` and carried on — which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, says which it used, and stops only when no runtime can be reached. +- **One upstream defect is named rather than worked around.** The `ts-zod` emitter writes `binder.ts`'s sibling import with no file extension, which a plain Node ESM project rejects at type-check and at runtime while a bundler resolution accepts — which is why the JS starter never met it. Filed as `L-260912-857a5a` against `pipelex`. The tree is stamped and hashed, so the skill reports it, never patches the file, never drops the tree from the type checker, and leaves a change of `moduleResolution` to the user. + ## License & distribution **Apache 2.0**; repo made public when ready (required for easy marketplace install). Versions start at **0.1.0** (plugin and marketplace). GitHub home assumed `Pipelex/pipelex-plugins` — confirm at first push. diff --git a/wip/pipelex-integrate/design.md b/wip/pipelex-integrate/design.md index ad5ded7..87b5850 100644 --- a/wip/pipelex-integrate/design.md +++ b/wip/pipelex-integrate/design.md @@ -118,7 +118,7 @@ What the skill does need is the **pipe's signature**: input names with their con **Amended 2026-09-06 — the gap this section originally worked around is closed.** As written on 2026-08-30, the signature reached the model through the `explicit: true` inputs template and a bundle read for a files source, and not at all for a `method_ref` or `method_id` source, because `mthds_validate` carried `main_pipe_ref` and `pipe_io_contracts` only on the view-only `_meta` channel; the section carried a by-elimination heuristic for the output concept and one question to confirm it, and filed `L-260830-e8b2e0` against `pipelex-mcp` for the real fix. That follow-up landed: a valid `mthds_validate` verdict now carries `structuredContent.main_pipe` — the main pipe's namespaced ref, each declared input with its fully-qualified `concept_ref`, multiplicity and `required` flag, and the produced concept with its multiplicity and `optional` flag — on the local workshop as much as on the hosted console, surviving a pending-signature verdict and `include_graph: false` (`pipelex-mcp/src/capabilities/validate.ts`, `mainPipeSignatureOf`; `pipelex-mcp/SPEC.md` → "Validation Scope"). So: - **Every selector reads the signature from the step-2 verdict** — files, `method_ref` and `method_id` alike — and the heuristic and its question are **never written**. `L-260831-b67e18`, filed to delete them, closes with the skill's first release. -- **`main_pipe` is absent only when the bundle declares no main pipe**, and it is omitted whole rather than partially. That path asks which pipe to integrate, reads its inputs from `mthds_inputs_template` with `explicit: true` (§4.9), and its output from the bundle for a files source; a by-ref or by-id method with no main pipe and no way to read its output declaration is reported as not integrable as it stands. +- **`main_pipe` is omitted whole rather than partially, for three reasons** (amended 2026-09-12, Phase 3, against `pipelex-mcp` `dev` at `791b9ce`): the verdict settles **no entry pipe** — the report's `default_pipe_ref` is a stated `null`, which happens when nothing declares a main pipe and when a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, exactly the cases where a selector-less run would fail to resolve one too; the entry pipe's **contract did not narrow** (a malformed member, or no contract entry at all, omits the signature and leaves the verdict untouched); or the **workshop predates the signature** (`@pipelex/mcp` 0.13.0 and earlier). The entry pipe is the API's `default_pipe_ref`, so for a published package it is the manifest's pipe and not necessarily the bundle's own declaration — which is why the call site is typed and run against `main_pipe.pipe_ref`. The original wording named only the first of the three and told a by-ref or by-id source to blame an old workshop; the skill now states all three and asserts none of them (`pipelex-mcp/SPEC.md` → "The main pipe's signature rides `structuredContent`"). That path asks which pipe to integrate, reads its inputs from `mthds_inputs_template` with `explicit: true` (§4.9), and its output from the bundle for a files source; a by-ref or by-id method with no main pipe and no way to read its output declaration is reported as not integrable as it stands. ### 4.4 The write arm is mandatory, and the model never writes an artifact (brief finding) @@ -126,7 +126,7 @@ What the skill does need is the **pipe's signature**: input names with their con Three consequences the skill has to carry: -- **`output_dir` is relative to the workshop's working directory, which is the directory the harness was launched in** — the host spawns the server there (`process.cwd()` in `pipelex-mcp/src/local/server.ts`), the launcher wrapper does not `cd`, and containment is real-path-checked against it (`workspace-boundary.ts`, `resolveSaveDir`). The skill computes the generated directory's path relative to the session's initial working directory and passes that. **A project root outside that directory cannot be written to**, and the skill stops with the instruction to relaunch the harness from the project (or, on Vibe, to register the workshop with that working directory) — it does not fall back to riding content. +- **`output_dir` is relative to the workshop's working directory, which is the directory the harness was launched in** — the host spawns the server there (`process.cwd()` in `pipelex-mcp/src/local/server.ts`), the launcher wrapper does not `cd`, and containment is real-path-checked against it (`workspace-boundary.ts`, `resolveSaveDir`). The skill computes the generated directory's path relative to the session's initial working directory and passes that. **A project root outside that directory cannot be written to**, and the skill stops with the instruction to relaunch the harness from the project (or, on Vibe, to register the workshop with that working directory) — it does not fall back to riding content. **Amended 2026-09-12 (Phase 3):** the tool cannot be relied on to raise that error, because a path inside the workshop's working directory is legal wherever it points — a harness launched in a sibling of the project accepts a write that lands beside the wrong project. So containment is checked *before* the first call, by reading the path from the workshop's working directory to the generated directory, and a tree that has landed outside the project is **not** repaired by moving it across: the bytes survive a move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. - **The generated files are never opened for editing and never formatted.** Step 5 of §3 precedes step 6 for this reason, and step 11 formats only the files the skill authored. - **A destination that refuses is a destination that was wrong.** The writer refuses any unstamped file, symlink, or directory at an artifact path and leaves the tree byte-identical; the skill treats that as "this is not a dedicated generated directory", picks or asks for one that is, and never pre-clears anything. @@ -165,7 +165,7 @@ The file keeps the starter's name and the starter's `sources` map, deliberately: **Decision: the skill always writes each method to its own directory, and when `orphans[]` is non-empty it names them, says what they are, and does nothing else.** The tool never deletes an orphan and neither does the skill: the moment two methods share a directory, "clean up the orphans" deletes real files. A non-empty `orphans[]` on a fresh integration means the chosen directory was not fresh — an earlier generation, a different target, or an engine rename — and the report says a dedicated directory per generation is the fix, in the tool's own words. On a refresh into the method's own directory, an orphan can only be an artifact the engine stopped emitting; the report names it and leaves the decision to the user. -A directory that already holds a `codegen.lock` is refresh mode only if its sidecar names the same method; otherwise the skill chooses a different directory name and says why. +A directory that already holds a `codegen.lock` is refresh mode only if its sidecar names the same method; otherwise the skill chooses a different directory name and says why — **including when the lock has no sidecar at all, and including when the user named that directory** (amended 2026-09-12, Phase 3). The orphan report is not the guard here: every method of a target emits the same file names, so a second method generated into an occupied directory overwrites the first one's stamped files and reports an empty `orphans[]`, which reads as a clean generation. ### 4.8 The wire-`null` mismatch — a shared helper with an expiry (finding surfaced in design) @@ -270,6 +270,8 @@ Three small edits to the existing skills, each one sentence or one step: No change to `pipelex-mcp` from this repo (the follow-up above is filed, not worked around); no change to either starter (they are the reference, not a deliverable); no JSON Schema target until the engine serves one (`L-260829-7b7917` → `L-260829-263b9e` → `L-260829-68c7cf`); no watch mode or build-time regeneration; no per-project reimplementation of write-if-changed or orphan cleanup (the writer overwrites and reports; the SDK upstreaming owns the rest); no `contracts.ts`; no tests, routes, or UI; no hosted-console branching (the plugin only ever declares the workshop, so the write arm is always available to it). +**One upstream defect the skill names rather than works around** (found 2026-09-12, Phase 3): the ts-zod emitter writes `binder.ts`'s sibling import as `from "./types"`, with no extension, which a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) rejects at type-check and at runtime, while a bundler resolution — the JS starter's, which is why the starter never met it — accepts. It is `L-260912-857a5a` against `pipelex`. The skill reports it, never patches the stamped file, never drops the tree from the type checker, and leaves a change of the project's `moduleResolution` to the user. + ## Decision boxes for ratification | Box | Ruling | Ratified? | diff --git a/wip/pipelex-integrate/scaffold-design.md b/wip/pipelex-integrate/scaffold-design.md index 3e75d4b..f551398 100644 --- a/wip/pipelex-integrate/scaffold-design.md +++ b/wip/pipelex-integrate/scaffold-design.md @@ -33,7 +33,7 @@ A starter clone that is already in the working directory and has not been bootst ## 3. Branch A — one of our starters -1. **Prerequisites.** JavaScript: Node at or above the floor the starter's `package.json` `engines` names (22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and npm. Python: `uv` (the starter's Makefile installs and locks with it) and a Python inside the starter's `requires-python` range (3.11 to 3.14 at writing) that `uv python find` can see. Both: git. The GitHub branch also needs `gh` authenticated (`gh auth status`). A missing piece **stops** the skill with the exact thing missing and the starter README's own line about it; the skill never installs a toolchain. +1. **Prerequisites.** JavaScript: Node at or above the floor the starter's `package.json` `engines` names (22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and npm. Python: `uv` (the starter's Makefile installs and locks with it) and a Python inside the starter's `requires-python` range (3.11 to 3.14 at writing) that `uv python find` can see. Both: git. The GitHub branch also needs `gh` authenticated (`gh auth status`). A missing piece **stops** the skill with the exact thing missing and the starter README's own line about it; the skill never installs a toolchain. **Amended 2026-09-12 (Phase 3):** a runtime the machine already has and only the `PATH` is missing is not a missing piece — dogfooded on a `PATH` without `node`, the skill found the machine's `nvm`, activated it and carried on, which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, uses what they already hold, says which one it used and that the user's own shell may not have it, and stops only when no runtime can be reached that way. 2. **Acquire.** - **Local, the default.** `git clone --depth 1 https://github.com/Pipelex/.git `; read the template's version from its `package.json` / `pyproject.toml` and its head SHA; then detach from the template — remove the clone's `.git`, `git init -b main` — so that `git status`, `git remote` and a future push belong to the user's project and not to the template. This is what GitHub's "Use this template" button produces: a copy with no history and no remote. The starters' READMEs say "don't clone it directly" to humans for exactly that reason, and the fresh history is how the skill honours it. - **GitHub, on request.** `gh repo create / --template Pipelex/ --private --clone` (visibility is the user's call, asked, default private). Creating a repository on GitHub is an outward-facing action: the skill states the exact command and confirms before running it. GitHub writes the initial commit itself; the skill continues at step 3. @@ -68,7 +68,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win | Condition | The skill | | --- | --- | | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete or write into it | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, **and never offer to make room** — amended 2026-09-12 (Phase 3), after a run refused the directory and then offered to move the user's file aside and merge it back afterwards | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone and say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list, say the template changed | From 8356f2ca2e00d800b83586bc5fd60c122bb783af Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:14:30 +0200 Subject: [PATCH 12/21] Say how to test for the key without putting it in the transcript MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A dogfood run checking whether PIPELEX_API_KEY was set printed the raw value into a tool call's output, and then told the user to consider rotating it. The skill said never to print a key but never said how to look for one, which leaves the obvious `env | grep PIPELEX` as the path of least resistance. It now gives the presence test that reveals nothing, and says that echoing the value — in a diagnostic, a message or a command substitution — makes the key one to rotate. Co-Authored-By: Claude Opus 5 (1M context) --- pipelex-codex/skills/pipelex-scaffold/SKILL.md | 3 ++- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 3 ++- pipelex/skills/pipelex-scaffold/SKILL.md | 3 ++- templates/skills/pipelex-scaffold/SKILL.md.j2 | 3 ++- tests/unit/test_pipelex_scaffold_skill.py | 1 + 5 files changed, 9 insertions(+), 4 deletions(-) diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 1ace51f..1f8243d 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -88,7 +88,7 @@ cp /.env.example /.env.local # JS: Next.js reads .env.local cp /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -149,6 +149,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index fed5303..f025c32 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -88,7 +88,7 @@ cp /.env.example /.env.local # JS: Next.js reads .env.local cp /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -149,6 +149,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index 9be9760..4341d83 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -95,7 +95,7 @@ cp /.env.example /.env.local # JS: Next.js reads .env.local cp /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -156,6 +156,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index e14aabc..ba639aa 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -88,7 +88,7 @@ cp /.env.example /.env.local # JS: Next.js reads .env.local cp /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -149,6 +149,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 51baa58..8cf6b63 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -30,6 +30,7 @@ class TestPipelexScaffoldSkill: "Read `/.claude/skills/bootstrap/SKILL.md` and follow it as written", "**Add nothing to that procedure and reimplement none of it.**", "**Never print a key, and never ask for one in the conversation.**", + "a key in the transcript is a key to rotate", "**state the exact command and confirm before running it**", "do not start `make dev`", "Add **no** SDK dependency and create **no** empty `methods/` directory", From 95828e26d5f9f9c3654d0723f19924524d2be588 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:15:36 +0200 Subject: [PATCH 13/21] Close Phase 3 in the tracker, with the two scenarios that stay open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every remaining scenario is recorded with its verdict and its evidence: the six that changed a template and the commit each fix landed in, the two that only surfaced the upstream emitter defect, and the two that cannot be run here — SC-9 needs the founder's say-so for a GitHub repository, and SC-10's empty-key branch is unreachable on a machine whose shell profile exports a key into every tool shell. Checkpoint 2 records the workshop and starter SHAs the runs used. Phase 4 gains the upstream re-check, the TS-1 half of its live run against the published workshop, and the fact that the cut carrying main_pipe is still unpublished, which is what this pull request's merge waits on. Co-Authored-By: Claude Opus 5 (1M context) --- wip/pipelex-integrate/plan.md | 93 ++++++++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 24 deletions(-) diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index cc0da1d..b5392ba 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -147,12 +147,12 @@ Two scratch projects, each created from scratch by the session so the skill meet - [x] **TS-2 the bytes are untouched.** `git diff --stat` shows no change under `src/generated/` after the skill's own format run; `npm run codegen:check` exits 0. - [x] **TS-3 refresh after a bundle edit.** Change a concept field in `main.mthds` (through `/pipelex-edit`, which should announce the staleness — Phase 2 wiring), run the skill again: the sidecar comparison reports the changed source, the fingerprint moved, the call site is edited only if the type check demanded it, `sources.json` carries the new hash, `codegen:check` is green again. Then edit only a prompt (no concept change): the fingerprint is unchanged and the report says restamp-or-nothing. - [x] **TS-4 stale-source gate.** Edit the bundle and do *not* refresh: `npm run codegen:check` exits 1 with `stale-source` and the refresh remedy. -- [ ] **TS-5 `method_ref` source.** Integrate a published address at a tag (a `github.com/Pipelex/…@vX.Y.Z` package): the call site runs by `method_ref`, the sidecar's `sources` is empty, the output-concept heuristic of §4.3 either finds one candidate or asks — record which. -- [ ] **TS-6 `method_id` warning.** Integrate a catalog method by id: the one-line warning appears, the recommendation is stated, and the skill proceeds only on confirmation. -- [ ] **TS-7 orphan.** Generate a second method into the first method's directory on purpose (by naming the dir explicitly): `orphans[]` is reported by name, nothing is deleted, the fix sentence is the tool's. -- [ ] **TS-8 foreign file.** Point at a directory holding a hand-written `types.ts`: the refusal is surfaced, the file is untouched, the skill chooses or asks for another directory. -- [ ] **TS-9 containment escape.** Launch the harness from a sibling directory so the project is outside the workshop's working directory: the skill stops with the relaunch instruction and does not ride content. -- [ ] **TS-10 no key.** Unset `PIPELEX_API_KEY` in the workshop's environment: the `config` stop with the hint verbatim, nothing written. +- [x] **TS-5 `method_ref` source.** Integrate a published address at a tag (a `github.com/Pipelex/…@vX.Y.Z` package): the call site runs by `method_ref`, the sidecar's `sources` is empty, the output-concept heuristic of §4.3 either finds one candidate or asks — record which. +- [x] **TS-6 `method_id` warning.** Integrate a catalog method by id: the one-line warning appears, the recommendation is stated, and the skill proceeds only on confirmation. +- [x] **TS-7 orphan.** Generate a second method into the first method's directory on purpose (by naming the dir explicitly): `orphans[]` is reported by name, nothing is deleted, the fix sentence is the tool's. +- [x] **TS-8 foreign file.** Point at a directory holding a hand-written `types.ts`: the refusal is surfaced, the file is untouched, the skill chooses or asks for another directory. +- [x] **TS-9 containment escape.** Launch the harness from a sibling directory so the project is outside the workshop's working directory: the skill stops with the relaunch instruction and does not ride content. +- [x] **TS-10 no key.** Unset `PIPELEX_API_KEY` in the workshop's environment: the `config` stop with the hint verbatim, nothing written. - [x] **PY-1 fresh integration, pydantic audience.** No `pipelex` dependency → `python-pydantic` chosen without a question; `/generated/__init__.py` and the subpackage `__init__.py` created; `[tool.ruff] exclude` gains the tree; pyright still covers it; `pydantic` and `pipelex-sdk` added with uv; the async call site plus a sync wrapper if the project is synchronous; the report states the gate asymmetry; `uv run pyright` passes. - [x] **PY-2 structures audience.** Add `pipelex` as a dependency and a `@pipe_func` file: `python-structures` is chosen (or offered first when only the dependency is present); `pipelex codegen check ` is wired into the existing gate. - [x] **PY-3 refresh after a bundle edit**, as TS-3, including the `/pipelex-edit` staleness notice. @@ -163,36 +163,67 @@ Two scratch projects, each created from scratch by the session so the skill meet Harness scenarios (§4.12) — each on a fresh scratch copy of a starter, never on the starter checkout itself: -- [ ] **TS-11 harness project, local method.** A scratch copy of `pipelex-starter-js` plus a new `methods//main.mthds`: the skill detects the harness, runs `npm run codegen`, writes the call site the way `docs/codegen.md` and the existing actions do, writes no sidecar of its own and no `src/generated/` tree of the plugin's shape, and `make check` is green. Refresh is reported as `npm run codegen`. -- [ ] **TS-12 harness project, remote method.** The same copy with a `method_ref` at a tag: `make add-method METHOD=…` is what runs; nothing else is written by the skill. -- [ ] **PY-4 harness project without a `pipelex` CLI.** A scratch copy of `pipelex-starter-python` plus a new method, with no `pipelex` on the PATH: the write arm writes into `/generated//` — the harness's layout — the report names `make codegen`'s prerequisite as the project's own and points at `L-260906-a2cd5b`'s subject in plain words, and no second layout exists. +- [x] **TS-11 harness project, local method.** A scratch copy of `pipelex-starter-js` plus a new `methods//main.mthds`: the skill detects the harness, runs `npm run codegen`, writes the call site the way `docs/codegen.md` and the existing actions do, writes no sidecar of its own and no `src/generated/` tree of the plugin's shape, and `make check` is green. Refresh is reported as `npm run codegen`. +- [x] **TS-12 harness project, remote method.** The same copy with a `method_ref` at a tag: `make add-method METHOD=…` is what runs; nothing else is written by the skill. +- [x] **PY-4 harness project without a `pipelex` CLI.** A scratch copy of `pipelex-starter-python` plus a new method, with no `pipelex` on the PATH: the write arm writes into `/generated//` — the harness's layout — the report names `make codegen`'s prerequisite as the project's own and points at `L-260906-a2cd5b`'s subject in plain words, and no second layout exists. Scaffold scenarios (S§3–S§7) — scratch directories in the session scratchpad; the network is needed for the clones and installs: -- [ ] **SC-1 JS starter, local clone, named directory, inputs up front.** Prerequisites checked and reported; the clone lands with fresh history; exactly one commit, its message carrying the template version and SHA; the clone's own `bootstrap` SKILL.md is followed from inside the project without re-asking what was given; `.env.local` is written with the key from the environment; `make all` is green; the bootstrap has removed itself; the report carries the session note and the hand-off. -- [ ] **SC-2 Python starter, local clone.** As SC-1; `git mv` of the package directory succeeds because of the pristine commit; `make agent-check` and `make agent-test` are green. -- [ ] **SC-3 "here".** An empty working directory: the clone lands in place and everything else is as SC-1. -- [ ] **SC-4 non-empty target.** The refusal, the question, nothing written. -- [ ] **SC-5 un-bootstrapped clone in the working directory.** Branch A enters at the bootstrap step and nothing is cloned. -- [ ] **SC-6 ecosystem, Python.** `uv init --package` in a fresh directory, the pristine commit, `.env.example` + `.env` (ignored), no SDK dependency, no `methods/`; then `/pipelex-integrate` from the same session lands the PY-1 shape. -- [ ] **SC-7 ecosystem, TypeScript, named framework.** `npm create next-app@latest … --yes`, the initializer's own `git init` respected, the pristine commit on top, the env pair; then `/pipelex-integrate` lands the TS-1 shape. -- [ ] **SC-8 missing toolchain.** A PATH without `node`: the stop message names Node and the starter README's floor; nothing is cloned. +- [x] **SC-1 JS starter, local clone, named directory, inputs up front.** Prerequisites checked and reported; the clone lands with fresh history; exactly one commit, its message carrying the template version and SHA; the clone's own `bootstrap` SKILL.md is followed from inside the project without re-asking what was given; `.env.local` is written with the key from the environment; `make all` is green; the bootstrap has removed itself; the report carries the session note and the hand-off. +- [x] **SC-2 Python starter, local clone.** As SC-1; `git mv` of the package directory succeeds because of the pristine commit; `make agent-check` and `make agent-test` are green. +- [x] **SC-3 "here".** An empty working directory: the clone lands in place and everything else is as SC-1. +- [x] **SC-4 non-empty target.** The refusal, the question, nothing written. +- [x] **SC-5 un-bootstrapped clone in the working directory.** Branch A enters at the bootstrap step and nothing is cloned. +- [x] **SC-6 ecosystem, Python.** `uv init --package` in a fresh directory, the pristine commit, `.env.example` + `.env` (ignored), no SDK dependency, no `methods/`; then `/pipelex-integrate` from the same session lands the PY-1 shape. +- [x] **SC-7 ecosystem, TypeScript, named framework.** `npm create next-app@latest … --yes`, the initializer's own `git init` respected, the pristine commit on top, the env pair; then `/pipelex-integrate` lands the TS-1 shape. +- [x] **SC-8 missing toolchain.** A PATH without `node`: the stop message names Node and the starter README's floor; nothing is cloned. - [ ] **SC-9 `gh repo create --template`.** Run only on Louis's explicit say-so, against a throwaway private repository he deletes afterwards: the confirmation appears before the command, visibility is asked, the GitHub-made initial commit is respected and no second pristine commit is made. - [ ] **SC-10 no key in the environment.** The env file is written with an empty key, the report says where a key comes from, and nothing asks for it in the conversation. -- [ ] **Vibe render sanity.** Read `pipelex-vibe/skills/pipelex-integrate/SKILL.md` once for the manual-registration wording of the MCP-absent message and the absence of Claude-only frontmatter. -- [ ] Reconcile every finding into the template and references; re-run the affected scenarios; `make build`, `make agent-check`, `make agent-test`. -- [ ] `/pipelex-mcp-source` back to `@latest`; confirm the diff carries no launcher change. +- [x] **Vibe render sanity.** Read `pipelex-vibe/skills/pipelex-integrate/SKILL.md` once for the manual-registration wording of the MCP-absent message and the absence of Claude-only frontmatter. +- [x] Reconcile every finding into the template and references; re-run the affected scenarios; `make build`, `make agent-check`, `make agent-test`. +- [x] `/pipelex-mcp-source` back to `@latest`; confirm the diff carries no launcher change. + +**The rest of Phase 3, run 2026-09-12 against `pipelex-mcp` `dev` at `791b9ce`.** One cold headless session per scenario; "the skill" means `pipelex-integrate` unless a line says otherwise. Transcripts are in the session scratchpad under `runs/`. + +- **TS-5** — the signature came from the verdict for a by-ref source (`documents.extract_document_markdown(document: native.Document) -> native.Text`), `sources` was empty in the sidecar, the call site ran by `method_ref` at the tag, the exclusions went in before the write, the gate was wired. It then found an upstream defect rather than a skill one: the generated `binder.ts` imports `./types` with no extension, which this project's `nodenext` resolution and Node itself both reject. The run refused to patch the stamped file and refused to drop the tree from the type checker, which is the designed posture. Filed as `L-260912-857a5a` against `pipelex`; the skill now names it (`82c2c15`). The §4.3 heuristic this box still mentions was never written, so there was nothing to ask. +- **TS-6** — the id was resolved through `mthds_list_methods`, the verdict carried `main_pipe` (`mcp_e2e_fixture.name_one_word`), the unversioned-catalog warning and its recommendation appeared, and the run stopped for confirmation with nothing written. +- **TS-7** — exposed the overwrite hazard and produced a fix (`161c2a1`); re-run, the skill refused the directory the prompt named and generated into `src/generated/summarize-pdf/`, saying why. This box's premise is wrong, and the deviation says why: two methods of one target produce a silent overwrite, not orphans. +- **TS-8** — the hand-written file was noticed before any tool call, left untouched, and another directory offered. The first pass also offered to delete it, now forbidden (`161c2a1`); the re-run offers the other directory or asks the user to move the file themselves. +- **TS-9** — failed as designed-for and produced a fix (`917ee3b`). First pass: no tool error, a legal write beside the wrong project, then a move into place. Re-run: stopped before generating, named the relaunch, and left both directories exactly as they were. +- **TS-10** — the `config` stop with the tool's hint verbatim, nothing written, clean tree. +- **TS-11** — harness detected, `npm run codegen` run, the repo's own pattern followed (bundle loader, narrower, action trio with its test, form, tab), no sidecar and no second layout of the plugin's shape, `make all` green including `codegen:check`. The run went beyond the ask and drove the dev server with Playwright to watch the new tab render. +- **TS-12** — `make add-method METHOD=github.com/Pipelex/methods/documents@v0.1.0` is what ran, everything written came from the harness, `make all` green. Reached only on explicit invocation: left to itself, the project's own `AGENTS.md` drove the same command without the skill. +- **PY-4** — with no `pipelex` on the `PATH` the workshop wrote into the harness's own layout (`piper/generated/keyword_extract/`), the Makefile's `codegen` / `codegen-check` lines and `pyproject.toml`'s packages and package data gained the method, the report named `make codegen`'s missing prerequisite as the project's own, no second layout exists, `make agent-check` and `make agent-test` green. +- **SC-1** — prerequisites checked, clone with fresh history, one pristine commit (`Start from Pipelex/pipelex-starter-js 0.4.0 (e104e9a3…)`), the clone's own bootstrap followed from inside the project without re-asking, `.env.local` filled from the shell key without printing it, `make all` green, bootstrap self-removed, the report carried the session note and the hand-off. +- **SC-2** — as SC-1 on the Python starter, pristine commit `Start from Pipelex/pipelex-starter-python 0.15.0 (72bb8e06…)`, the `git mv` renames staged because that commit existed, `make agent-check` and `make agent-test` green. +- **SC-3** — the clone landed in the empty working directory and the rest was SC-1. Run on the Python starter rather than the JS one, to spend one npm install instead of two; what "here" exercises is the placement, not the starter. +- **SC-4** — refused the non-empty directory, asked, wrote nothing. It also offered to move the user's file aside, now forbidden (`907698e`). +- **SC-5** — entered at the bootstrap and cloned nothing. The route was the clone's **own** `/bootstrap` rather than `pipelex-scaffold`: a session started inside the clone loads the project's skills, which is what `scaffold-design.md` §2 says that user should get. +- **SC-6** — `uv init --package`, one pristine commit, the env pair with `.env` ignored, no SDK dependency and no `methods/` from the scaffold; then the integrate tail landed the PY-1 shape in the same session — `python-pydantic` without a question, the bundle moved inside the package because the project is packaged, `models.py` + lock + sidecar, the async call site with its sync wrapper, the gate asymmetry stated. No formatter or type checker was configured, so there was nothing to exclude and nothing to wire, and the report said so. +- **SC-7** — `create-next-app` ran, its own `git init` and initial commit respected, and the integrate tail landed the TS-1 shape (`server-only` on the call site, the ESLint ignore, the drift script standalone because the project has no aggregate gate). Branch B itself was never entered: the run executed the initializer directly and invoked only `pipelex-integrate`, so the env pair was not written — see the observation in the decisions log. +- **SC-8** — did not stop, and produced a fix (`be82a3b`): with no `node` on the `PATH` the run found the machine's `nvm`, activated it and carried on. The MCP server failed to spawn for the same reason and the scaffold skill ran regardless, which is the MCP-free claim holding under the one condition that tests it. +- **SC-9** — **not run.** It needs a throwaway GitHub repository created under Louis's explicit say-so, which no agent may give itself. The box stays open. +- **SC-10** — **not testable on this machine, and the box stays open.** The shell profile exports `PIPELEX_API_KEY`, and Claude Code initializes every tool shell from that profile, so the key comes back even when the session is launched with it unset and when session settings set it empty — both were tried. What the attempts did show: the positive branch of the same rule three times over (SC-1, SC-2, SC-3 filled the env file from the shell without printing the value) and `.env.example` always written with an empty key. They also showed the hazard the rule exists for: one run printed the raw key into a tool call's own output while testing whether it was set, and said so. The skill now carries the presence test that reveals nothing (`8356f2c`). Testing the empty-key branch honestly needs a machine whose profile does not export a key. +- **Vibe render sanity** — `pipelex-vibe/skills/pipelex-integrate/SKILL.md` and its scaffold sibling carry no `mcp__` tool names and no `allowed-tools` frontmatter, the MCP-absent line points at `mcp/vibe-mcp.toml` and its `env` table (dev's new wording, which this branch's merge adopted), and both reference directories land in the target. **CHECKPOINT 2** — every scenario above has been run at least once against the local workshop and its finding reconciled. Record here: the `pipelex-mcp` SHA the dogfood ran against, the starter SHAs the harness and scaffold scenarios cloned, the scenarios that exposed a template change (and the change), and any scenario that could not be run and why. +**Reached 2026-09-12, with two scenarios open and named.** The workshop was `pipelex-mcp` `dev` at `791b9ce`, built from a `git archive` of that commit inside the session scratchpad; the starters were `pipelex-starter-js` at `3bf44f0` (version 0.4.0) for the harness copies and `pipelex-starter-python` at `abbe7c2` (version 0.15.0), while the scaffold scenarios cloned `main` from GitHub (`e104e9a3` / `72bb8e06`). Every run carried `PIPELEX_BASE_URL=https://api-dev.pipelex.com`. + +Six scenarios changed the templates, each fix committed on its own: TS-9 the containment pre-check (`917ee3b`), TS-7 the refusal of an occupied generated directory (`161c2a1`), TS-8 and SC-4 the rule that neither skill offers to clear what is in the way (`161c2a1`, `907698e`), SC-8 the runtime a version manager already holds (`be82a3b`), the SC-10 attempt the non-printing key test (`8356f2c`). TS-5 and the published-workshop run changed nothing but surfaced `L-260912-857a5a`, whose honest statement the skill now carries (`82c2c15`). One more fix preceded the scenarios, from reading the cut the skill will ship against rather than from a run: the three reasons a verdict carries no `main_pipe` (`245c978`). + +Not run: **SC-9**, which needs the founder's say-so to create a GitHub repository, and **SC-10**, which this machine cannot express because its shell profile exports a key into every tool shell. Both are recorded above with what stands in for them. + +Two findings went to other repositories rather than being worked here: `L-260912-857a5a` (`pipelex`, the ts-zod extensionless import) and `L-260912-248c68` (`pipelex-starter-python`, whose bootstrap writes the distribution spelling into Python imports, reproduced three times in this batch). + ## Phase 4 — release Owner: `pipelex-plugins`. **Gate, hard:** a published `@pipelex/mcp` version that carries `mthds_codegen` with the write arm — name the version here before starting; an open `pipelex-mcp` release item is not a gate. The plugin's launcher is `@latest`, so nothing in this repo moves for it, but a plugin released before the tool is a skill that stops at "tool absent" for every user. -- [ ] Published `@pipelex/mcp` version carrying `mthds_codegen`: **0.13.0** (published 2026-08-30; the plugin's `@latest` already resolves to it). -- [ ] Published `@pipelex/mcp` version carrying **`main_pipe` on the validate verdict**: `__________` (fill in — it sits under `[Unreleased]` in `pipelex-mcp/CHANGELOG.md` at writing). The skill's ordinary path reads the signature from there; on an older workshop it takes the fallback of §4.3 (template for the inputs, the bundle for the output) and says the workshop predates the signature. Ship after this release so users never meet the fallback by default. -- [ ] Re-check `L-260820-ee327d` and `L-260830-4e43cd` one last time; strike or keep the helper and the asymmetry paragraph accordingly, and log it. -- [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. "Live" here means the published workshop, **not** the production API: Louis ruled on 2026-09-07 that this work stays on `https://api-dev.pipelex.com` until he says otherwise, because the key on his machine is dev-scoped and updating the prod environment is off his list for a while. So the run carries `PIPELEX_BASE_URL=https://api-dev.pipelex.com`, and a prod 403 is the expected state rather than a defect to chase. +- [x] Published `@pipelex/mcp` version carrying `mthds_codegen`: **0.13.0** (published 2026-08-30; the plugin's `@latest` already resolves to it). +- [ ] Published `@pipelex/mcp` version carrying **`main_pipe` on the validate verdict**: **none yet — unreleased as of 2026-09-12.** It sits under `[Unreleased]` in `pipelex-mcp/CHANGELOG.md` on `dev` at `791b9ce`; the cut that publishes it is `L-260911-ca8256`, and that publish is the founder's gate. This pull request must not merge before it, because a plugin released first gives every user the fallback path by default. (fill in — it sits under `[Unreleased]` in `pipelex-mcp/CHANGELOG.md` at writing). The skill's ordinary path reads the signature from there; on an older workshop it takes the fallback of §4.3 (template for the inputs, the bundle for the output) and says the workshop predates the signature. Ship after this release so users never meet the fallback by default. +- [x] Re-check `L-260820-ee327d` and `L-260830-4e43cd` one last time; strike or keep the helper and the asymmetry paragraph accordingly, and log it. **2026-09-12:** `L-260820-ee327d` stays closed — the helper was struck in Phase 1 and the dogfood confirmed the live emitter projects `.nullish()`, so it is never written. `L-260830-4e43cd` is still open, so the Python asymmetry paragraph stays as written; SC-6's report stated it unprompted. +- [ ] One live run of TS-1 and PY-1 against the **published** `@pipelex/mcp@latest` (not the local checkout), on the prod plugin output. **The TS-1 half is done, 2026-09-12**, on this branch's built `pipelex/` output against `npx -y @pipelex/mcp@latest` (0.13.0) and api-dev: because that release carries no `main_pipe`, the run took the §4.3 fallback exactly as designed — `mthds_inputs_template` with `explicit: true` for the inputs — then generated through the write arm, wrote the sidecar, wired the gate, and reported `L-260912-857a5a` in the skill's new words. So the branch is correct against the published workshop and against the next cut. The PY-1 half is still to run. "Live" here means the published workshop, **not** the production API: Louis ruled on 2026-09-07 that this work stays on `https://api-dev.pipelex.com` until he says otherwise, because the key on his machine is dev-scoped and updating the prod environment is off his list for a while. So the run carries `PIPELEX_BASE_URL=https://api-dev.pipelex.com`, and a prod 403 is the expected state rather than a defect to chase. - [ ] Open the PR against `dev` with `Closes L-260830-344594` and `Closes L-260906-8ac105` in the body; work the review rounds per the workspace's tightening-bar rule; land with `/ledger-land`. `L-260831-b67e18` closes in the same landing, with the release as evidence that the heuristic never shipped. - [ ] File the starter-README pointer items (S§9, "to file at release"): both starters' "Use this template" sections name `/pipelex-scaffold` beside the button and `/bootstrap`. - [ ] `/release` → the next minor (`0.6.0`), which cuts the changelog heading, bumps every target TOML and the Claude marketplace, and opens the release PR against `main`. @@ -224,6 +255,10 @@ Carried in the design's §9 and §8; repeated here only where a phase might be t - **2026-09-07 — the Python two-audience rule is left as ratified, and the real fix was sent upstream (Louis).** Reviewing the PY-2 result, Louis questioned why a host carrying a `@pipe_func` gets `python-structures` at all, since that target puts Pipelex classes into the project's own generated types. The rule was checked rather than defended: the runtime enforces it hard — `pipelex/system/registries/func_registry.py:372` refuses a pipe func with *"return type must be a subclass of StuffContent, but is '…'"* — so `python-structures` is what makes a pipe func writable, and the skill never adds `pipelex` to reach that target. The imprecision the dogfood did expose is that §5 keys the choice on the **project** (a `@pipe_func` appears somewhere) when the decisive question is **per-method** (does a pipe func return *this* method's concepts?); in PY-2 nothing consumed the structures but the call site, which a plain `BaseModel` would have served. Three rules were put to Louis — a per-method test with `python-pydantic` offered first, the ratified project-level rule unchanged, or never choosing structures unaided. **He chose none of them and asked instead that the requirement be relaxed at its source**, which is `L-260907-3ea0c0` against `pipelex`: let a `@pipe_func` return a plain pydantic model and have the runtime adapt it. `StuffContent` is already pydantic (`stuff_content.py:15`) and what it adds over `BaseModel` is a generic rendering and dump surface, so the ask is plausible rather than speculative. **§5 therefore stands unchanged and this is not a deviation** — the target question is parked on that item, and the rule is revisited when it lands, not before. +- **2026-09-12 — the rest of Phase 3, and how the dev override was taken.** The remaining scenarios were run the way 2026-09-07's were: one cold headless session each (`claude -p "" --plugin-dir --model sonnet`), launched inside a scratch project, with `PIPELEX_BASE_URL=https://api-dev.pipelex.com` per the 2026-09-07 ruling. Two things were done differently and are worth keeping. **The dev override was a copy, not a switch**: instead of pointing `targets/defaults.toml` at a local checkout through `/pipelex-mcp-source`, the built `pipelex/` output was copied into the session scratchpad and only that copy's `hooks/launch-pipelex-mcp.sh` was repointed at a local `pipelex-mcp` build. The committed tree therefore never carried a launcher switch at any moment, which is the failure mode that skill exists to catch. **The `pipelex-mcp` build was `dev` at `791b9ce`, built from a `git archive` of that commit inside the scratchpad**, so the read-only neighbour repository was never touched — and it is the build that matters, because `main_pipe` is still unreleased (`npx @pipelex/mcp@latest` resolves to 0.13.0, which has no signature). The installed marketplace `pipelex` plugin was disabled per session (`--settings '{"enabledPlugins":{"pipelex@pipelex-plugins":false}}'`) so a stale 0.5.0 could not answer instead of the branch. + +- **2026-09-12 — what a skill's description can and cannot win.** Three runs are worth recording because they say something about where these skills sit rather than about their text. Asked to "generate the TypeScript types for methods/summarize-pdf … then give me a typed function", a cold session did the work itself and never invoked `pipelex-integrate`, though the phrasing is in its description; invoked explicitly, the same prompt produced the right refusal. In a starter-derived project the project's own `CLAUDE.md` and `AGENTS.md` load with the session and win: a run asked to add a published method went straight to `make add-method` without the skill, which is the same outcome the skill's harness deference prescribes, and a run standing in an un-bootstrapped clone reached for the clone's own `bootstrap` skill rather than `pipelex-scaffold`, which is what `scaffold-design.md` §2 says that user should get. A fourth, the same shape as the first: asked to create a Next.js app and then wire a method into it, a run executed `create-next-app` itself and invoked only `pipelex-integrate`, so branch B was never entered and the one thing the skill adds over a bare initializer — the `.env.example` / `.env` pair and the key rule — did not happen, while the initializer's own commit stood as the baseline the design asks for anyway. None of the four was treated as a text defect: two are a model's judgment on a prompt it can execute directly, and two are the project's own instructions doing their job. No description can compel invocation, so the lesson is about what a skill may assume rather than about its triggers — `pipelex-integrate` must keep working on a project that never met `pipelex-scaffold`, which is exactly what these runs show it doing. Worth re-reading if the skills ever appear unused in practice. + ## Deviations from the design **2026-09-07 — §4.10's fingerprint rule was wrong, and the dogfood proved it.** The design said a `crate_fingerprint` unchanged against the old lock's means a restamp at most, and a changed one means the concept set moved. Scenario TS-3's second half — a **system-prompt-only** edit, no concept touched — moved the fingerprint (`fc27bd…` → `088d66…`), because the fingerprint covers the whole bundle rather than its concept set. What did not move was the lock's `artifacts[].content_hash`: the only changed line in `types.ts` and in `binder.ts` was the one stamp line carrying the fingerprint, and the artifact hashes are computed over the content beneath the stamp, so they are the honest signal. Both §4.10 and the skill's refresh-mode section now read the outcome from `artifacts[].content_hash`, with the fingerprint named as the whole-bundle value it is. The cold agent recovered on its own — it diffed the tree and reported the restamp correctly — but only because it went and looked; the rule as written would have had it announce a concept change that had not happened. @@ -232,6 +267,16 @@ Carried in the design's §9 and §8; repeated here only where a phase might be t **2026-09-07 — the family's staleness notice looked in the wrong place.** Phase 2 wired `/pipelex-edit` and `/pipelex-design` to warn when an edit invalidates a generated tree. Run against the real thing, `/pipelex-edit` searched for `sources.json` **beside the bundle**, found none, and reported that no downstream refresh was needed — while the sidecar sat in `src/generated/gantt/`, exactly where the skill puts it. The sentence had named the location as background prose ("a project keeps one beside each generated tree, typically under…") instead of as an instruction about where to search. Both wirings now say to search the whole project, give the grep, and state that the sidecar never sits beside the bundle. Re-run afterwards on PY-1, the notice fired correctly — via the `/pipelex-design` route, since a concept-structure change is structural, so both edited wirings were exercised. +**2026-09-12 — the verdict's `main_pipe` is absent for three reasons, not one.** §4.3 as amended on 2026-09-06 said the signature is omitted only when the bundle declares no main pipe. Read against `pipelex-mcp` `dev` at `791b9ce` (`src/capabilities/validate.ts`, `mainPipeSignatureOf`; `SPEC.md` → "The main pipe's signature rides `structuredContent`"), it is also omitted when the server settles **no entry pipe** — a stated `default_pipe_ref: null`, which covers a published package whose `METHODS.toml` names a pipe the closure does not declare or declares in several domains — and when the entry pipe's **contract does not narrow**, since a partial signature is never emitted. The entry pipe is `default_pipe_ref` rather than the bundle's declaration, so for a published package the call site must be typed and run against `main_pipe.pipe_ref`. The skill's stop for a by-ref or by-id source no longer asserts that the workshop is too old, because it cannot know that. Landed in `245c978`. + +**2026-09-12 — an occupied generated directory is refused however it was named, and the orphan report is not the guard.** §4.7's "otherwise the skill chooses a different directory name" had narrowed, in the skill's text, to a lock *whose sidecar names a different method*. Scenario TS-7 showed why that is not enough: every method of a target emits the same file names, so generating a second method into a directory already holding another method's tree overwrites it and reports an empty `orphans[]` — a clean-looking generation that has destroyed the other method's types. The rule now refuses any directory holding a `codegen.lock` that no sidecar names for this method, a lock with no sidecar at all included, and says why instead of obeying the user's directory. Landed in `161c2a1`. + +**2026-09-12 — the containment stop has to be read before the call, because the tool cannot raise it.** §4.4 wrote the escape as a tool error to branch on. Scenario TS-9 — the harness launched in a sibling of the project — never produced one: `output_dir` was computed relative to the workshop's working directory as instructed, the write landed legally *beside* the wrong project, and the run then moved the tree into the project byte for byte and carried on. No content rode through the model, but the result is a project whose every later refresh meets the same mismatch and whose sidecar paths name a project the workshop cannot see. Step 6 now reads the path from the workshop's working directory to the generated directory before calling, treats a climbing path or a project root elsewhere on disk as the stop, and forbids writing beside the wrong project to move the tree over. Landed in `917ee3b`. + +**2026-09-12 — neither skill may offer to clear something that is in the way.** §4.4 says the skill never pre-clears a destination and S§7 says a non-empty target directory is refused; both runs obeyed the refusal and then offered to do it with consent — integrate offered to delete a hand-written `types.ts`, scaffold offered to move a `README.md` aside and merge it back. The answer to an occupied destination is another destination, so both skills now say never to offer. Landed in `161c2a1` and `907698e`. + +**2026-09-12 — a runtime a version manager already holds is not a missing toolchain.** S§3 step 1 stops on a missing prerequisite and never installs one. On a `PATH` without `node`, scaffold found the machine's `nvm`, activated it and carried on — installing nothing, and the only useful answer available. The prerequisites now distinguish a runtime the machine lacks from one only the `PATH` is missing, name the managers to check (`nvm`, `fnm`, `volta`, `asdf`, `mise`), and require the report to say which was used and that the user's own shell may not have it. Landed in `be82a3b`. + ## Where everything is - Brief: `wip/pipelex-integrate/brief.md`. Designs: `wip/pipelex-integrate/design.md` (integrate) and `wip/pipelex-integrate/scaffold-design.md` (scaffold). Upstream reading companion: `upstream-dependencies.md`. This tracker: `wip/pipelex-integrate/plan.md`. From 8cfd835ab377d40bd73a9c81603c7be376efa9ee Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:43:32 +0200 Subject: [PATCH 14/21] Compile and run the code these skills emit, and fix what that found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dogfood matrix exercised the skills in real sessions; nothing ever compiled or ran the code they write into a user's project, and that is where almost every defect was. The TypeScript call site declared its inputs as an `interface`. TypeScript gives a type alias of an object type an implicit index signature and an interface none, so it was not assignable to the SDK's `inputs: Record` and failed TS2322 on every resolution, bundler included — the first thing `tsc --noEmit` would have said. Both call sites read only `main.mthds` while generation takes every file of the bundle, so a multi-file method got correct types over a run that could not load what they were projected from; a bundle is one closure, and both now submit all of it. The module's own relative imports are extensionless, which is right only on a bundler resolution, so they are named as the agent's to extend with `.js` — not as the emitter defect, which is about the stamped tree and was the row they used to land on. The scaffold's `uv add` recipes ran wherever the agent stood rather than in the project `uv init` had just made: from a parent that is the user's own pyproject.toml and lockfile, and from nowhere it simply fails. The Django row already used the subshell the other two needed. The minimal TypeScript recipe reaches its commit with a populated `node_modules/` and no `.gitignore` from either `npm init -y` or `tsc --init`, and its `--module nodenext` with `"type": "module"` is exactly the shape the emitter defect breaks, which the reference now says where the recipe is chosen. The offline gate said `current` on two inputs it should not have. A sidecar whose `sources` was present but not an object was coerced to `{}`: nothing checked, nothing printed, exit 0 — the one input both silent and green, with an array the shape to expect since `method.files` beside it really is one. `null` is why the fix does not use `??`. And the decoder stripped a leading BOM before hashing, so a BOM'd artifact matched the lock while the bytes on disk were hand-edited. Both fail closed now, the absent and empty cases announce themselves, and `process.exitCode` replaces `process.exit` so the drift lines survive a pipe. Co-Authored-By: Claude Opus 5 (1M context) --- .../references/codegen-check.mjs | 30 ++++++++++++++++--- .../pipelex-integrate/references/python.md | 13 ++++++-- .../references/typescript.md | 30 +++++++++++++------ .../references/initializers.md | 10 +++++-- .../references/codegen-check.mjs | 30 ++++++++++++++++--- .../pipelex-integrate/references/python.md | 13 ++++++-- .../references/typescript.md | 30 +++++++++++++------ .../references/initializers.md | 10 +++++-- .../references/codegen-check.mjs | 30 ++++++++++++++++--- .../pipelex-integrate/references/python.md | 13 ++++++-- .../references/typescript.md | 30 +++++++++++++------ .../references/initializers.md | 10 +++++-- .../references/codegen-check.mjs | 30 ++++++++++++++++--- skills/pipelex-integrate/references/python.md | 13 ++++++-- .../references/typescript.md | 30 +++++++++++++------ .../references/initializers.md | 10 +++++-- 16 files changed, 256 insertions(+), 76 deletions(-) diff --git a/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs index ade01da..0e9fe5e 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs @@ -32,7 +32,10 @@ const LOCK_FILENAME = "codegen.lock"; const SIDECAR_FILENAME = "sources.json"; const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); -const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); +// `ignoreBOM: true` keeps a leading BOM in the decoded string. The default strips it, so an +// artifact given a BOM would hash as its un-BOM'd self, match the lock, and report current while +// the bytes on disk are hand-edited — the gate's one job is to not say that. +const strictUtf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); const out = (line) => process.stdout.write(`${line}\n`); const err = (line) => process.stderr.write(`${line}\n`); @@ -116,9 +119,25 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } - const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to + // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and + // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it + // in the sidecar really is an array), and `null` is why this does not use `??`, which would + // quietly turn an explicit null into the legitimate absent case. + const sources = sidecar?.sources; + if (sources === undefined) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + if (typeof sources !== "object" || sources === null || Array.isArray(sources)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — \`sources\` is not an object, so staleness cannot be ruled out`] }; + } + const recordedSources = Object.entries(sources).sort(); + if (recordedSources.length === 0) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + const lines = []; - for (const [source, recorded] of Object.entries(sources).sort()) { + for (const [source, recorded] of recordedSources) { let onDisk; try { onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); @@ -163,4 +182,7 @@ async function main(argv) { return exitCode; } -process.exit(await main(process.argv)); +// `process.exitCode` rather than `process.exit`: stdout is a pipe under `npm run`, `make` and every +// CI runner, where writes are asynchronous and `process.exit` drops the ones still pending. The code +// would survive either way; the drift lines explaining it are what gets truncated. +process.exitCode = await main(process.argv); diff --git a/pipelex-codex/skills/pipelex-integrate/references/python.md b/pipelex-codex/skills/pipelex-integrate/references/python.md index cc64358..8acdbde 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/python.md +++ b/pipelex-codex/skills/pipelex-integrate/references/python.md @@ -57,7 +57,14 @@ from pipelex_sdk.client import PipelexAPIClient from .generated.summarize_pdf.models import DocumentSummary PIPE_CODE = "summarize_pdf" -BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" +BUNDLE_DIR = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" + + +def _read_bundle() -> list[str]: + """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that + imports a sibling needs that sibling submitted with it, or the run fails to load what the + generated models were projected from.""" + return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: @@ -67,7 +74,7 @@ async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) async with PipelexAPIClient() as client: results = await client.start_and_wait( pipe_code=PIPE_CODE, - mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + mthds_contents=_read_bundle(), inputs=inputs, ) return DocumentSummary.model_validate(results.main_stuff) @@ -78,7 +85,7 @@ def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) return asyncio.run(summarize_pdf(document=document, context=context)) ``` -Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: +Variants by selector, replacing the `mthds_contents=` argument and dropping `BUNDLE_DIR` and `_read_bundle`: - **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. - **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. diff --git a/pipelex-codex/skills/pipelex-integrate/references/typescript.md b/pipelex-codex/skills/pipelex-integrate/references/typescript.md index 84dc2d9..9ed2da6 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/typescript.md +++ b/pipelex-codex/skills/pipelex-integrate/references/typescript.md @@ -15,7 +15,7 @@ Companion to `/pipelex-integrate` for a project that has a `package.json`. Every | **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | | **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | | **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | -| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below, but **read the script before believing it**: `"codegen": "graphql-codegen"` or a protobuf generator satisfies the name and generates no MTHDS types, and deferring to it would skip the dependencies, the exclusions, the sidecar and the gate while generating nothing. Pipelex-specific evidence — it calls `mthds_codegen`, a `pipelex` CLI, or reads `methods/` — is what makes it this method's harness; without that, integrate normally and leave the unrelated harness alone | Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. @@ -37,7 +37,7 @@ One module per method. `summarize-pdf` with a `document: native.Document` input, ```ts // src/pipelex/summarizePdf.ts -import { readFile } from "node:fs/promises"; +import { readdir, readFile } from "node:fs/promises"; import path from "node:path"; import type { RunResults } from "@pipelex/sdk"; import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; @@ -45,30 +45,42 @@ import type { DocumentSummary } from "../generated/summarize-pdf/types"; import { getPipelexClient } from "./client"; const PIPE_CODE = "summarize_pdf"; -const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); +const BUNDLE_DIR = path.join(process.cwd(), "methods", "summarize-pdf"); + +/** Every `.mthds` file of the bundle, sorted, as the run's `mthds_contents`. A bundle is one + * closure: a main file that imports a sibling needs that sibling submitted with it, or the + * run fails to load what the generated types were projected from. `recursive` needs Node + * >= 20.1 (or >= 18.17); below that, walk the directory yourself. */ +async function readBundle(): Promise { + const names = (await readdir(BUNDLE_DIR, { recursive: true })).filter((name) => name.endsWith(".mthds")).sort(); + return Promise.all(names.map((name) => readFile(path.join(BUNDLE_DIR, name), "utf8"))); +} -export interface SummarizePdfInputs { +export type SummarizePdfInputs = { /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, - * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * run `getPipelexClient().prepareInputs({ files: (await readBundle()).map((c) => ({ content: c })), inputs })` first — * it uploads and rewrites the value. Note: prepareInputs treats any string it does not * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and * uploads, so a public endpoint must gate schemes before handing values to it. */ document: { url: string }; context?: string; -} +}; export async function summarizePdf(inputs: SummarizePdfInputs): Promise { - const bundle = await readFile(BUNDLE_PATH, "utf8"); const results: RunResults = await getPipelexClient().startAndWaitForResult({ pipe_code: PIPE_CODE, - mthds_contents: [bundle], + mthds_contents: await readBundle(), inputs, }); return parseDocumentSummary(results.main_stuff); } ``` -Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: +**`SummarizePdfInputs` is a `type`, not an `interface`, and that is load-bearing.** The SDK takes `inputs: Record`, and TypeScript gives a type alias of an object type an implicit index signature while an `interface` gets none — so an interface here fails with `TS2322: Index signature for type 'string' is missing`. It fails on every resolution, bundler included, and it is the first thing a `tsc --noEmit` would have caught. Keep it a `type`. + +**The three relative imports above are extensionless, which is correct only on a bundler resolution.** On the plain Node ESM shape that meets the emitter's `TS2835` defect (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) this module needs `.js` on each of them — `"../generated/summarize-pdf/binder.js"`, `"../generated/summarize-pdf/types.js"`, `"./client.js"`. That is your own module and so your own fix, unlike the stamped `binder.ts`: write the extensions when the project's resolution demands them, and do not report your own module's `TS2835` as the emitter's defect. + +Variants by selector, replacing the `mthds_contents` line and dropping `BUNDLE_DIR` and `readBundle`: - **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. - **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. diff --git a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md index 98b683f..7602a15 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md @@ -10,13 +10,15 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | | A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | -| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | `uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +**Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. + ## TypeScript / JavaScript | Want | Command | `git init`? | Where `src/` lands | @@ -31,6 +33,8 @@ After the initializer: `git init -b main` only if it did not initialize a reposi `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs index ade01da..0e9fe5e 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs @@ -32,7 +32,10 @@ const LOCK_FILENAME = "codegen.lock"; const SIDECAR_FILENAME = "sources.json"; const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); -const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); +// `ignoreBOM: true` keeps a leading BOM in the decoded string. The default strips it, so an +// artifact given a BOM would hash as its un-BOM'd self, match the lock, and report current while +// the bytes on disk are hand-edited — the gate's one job is to not say that. +const strictUtf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); const out = (line) => process.stdout.write(`${line}\n`); const err = (line) => process.stderr.write(`${line}\n`); @@ -116,9 +119,25 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } - const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to + // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and + // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it + // in the sidecar really is an array), and `null` is why this does not use `??`, which would + // quietly turn an explicit null into the legitimate absent case. + const sources = sidecar?.sources; + if (sources === undefined) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + if (typeof sources !== "object" || sources === null || Array.isArray(sources)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — \`sources\` is not an object, so staleness cannot be ruled out`] }; + } + const recordedSources = Object.entries(sources).sort(); + if (recordedSources.length === 0) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + const lines = []; - for (const [source, recorded] of Object.entries(sources).sort()) { + for (const [source, recorded] of recordedSources) { let onDisk; try { onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); @@ -163,4 +182,7 @@ async function main(argv) { return exitCode; } -process.exit(await main(process.argv)); +// `process.exitCode` rather than `process.exit`: stdout is a pipe under `npm run`, `make` and every +// CI runner, where writes are asynchronous and `process.exit` drops the ones still pending. The code +// would survive either way; the drift lines explaining it are what gets truncated. +process.exitCode = await main(process.argv); diff --git a/pipelex-vibe/skills/pipelex-integrate/references/python.md b/pipelex-vibe/skills/pipelex-integrate/references/python.md index cc64358..8acdbde 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/python.md +++ b/pipelex-vibe/skills/pipelex-integrate/references/python.md @@ -57,7 +57,14 @@ from pipelex_sdk.client import PipelexAPIClient from .generated.summarize_pdf.models import DocumentSummary PIPE_CODE = "summarize_pdf" -BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" +BUNDLE_DIR = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" + + +def _read_bundle() -> list[str]: + """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that + imports a sibling needs that sibling submitted with it, or the run fails to load what the + generated models were projected from.""" + return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: @@ -67,7 +74,7 @@ async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) async with PipelexAPIClient() as client: results = await client.start_and_wait( pipe_code=PIPE_CODE, - mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + mthds_contents=_read_bundle(), inputs=inputs, ) return DocumentSummary.model_validate(results.main_stuff) @@ -78,7 +85,7 @@ def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) return asyncio.run(summarize_pdf(document=document, context=context)) ``` -Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: +Variants by selector, replacing the `mthds_contents=` argument and dropping `BUNDLE_DIR` and `_read_bundle`: - **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. - **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. diff --git a/pipelex-vibe/skills/pipelex-integrate/references/typescript.md b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md index 84dc2d9..9ed2da6 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/typescript.md +++ b/pipelex-vibe/skills/pipelex-integrate/references/typescript.md @@ -15,7 +15,7 @@ Companion to `/pipelex-integrate` for a project that has a `package.json`. Every | **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | | **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | | **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | -| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below, but **read the script before believing it**: `"codegen": "graphql-codegen"` or a protobuf generator satisfies the name and generates no MTHDS types, and deferring to it would skip the dependencies, the exclusions, the sidecar and the gate while generating nothing. Pipelex-specific evidence — it calls `mthds_codegen`, a `pipelex` CLI, or reads `methods/` — is what makes it this method's harness; without that, integrate normally and leave the unrelated harness alone | Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. @@ -37,7 +37,7 @@ One module per method. `summarize-pdf` with a `document: native.Document` input, ```ts // src/pipelex/summarizePdf.ts -import { readFile } from "node:fs/promises"; +import { readdir, readFile } from "node:fs/promises"; import path from "node:path"; import type { RunResults } from "@pipelex/sdk"; import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; @@ -45,30 +45,42 @@ import type { DocumentSummary } from "../generated/summarize-pdf/types"; import { getPipelexClient } from "./client"; const PIPE_CODE = "summarize_pdf"; -const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); +const BUNDLE_DIR = path.join(process.cwd(), "methods", "summarize-pdf"); + +/** Every `.mthds` file of the bundle, sorted, as the run's `mthds_contents`. A bundle is one + * closure: a main file that imports a sibling needs that sibling submitted with it, or the + * run fails to load what the generated types were projected from. `recursive` needs Node + * >= 20.1 (or >= 18.17); below that, walk the directory yourself. */ +async function readBundle(): Promise { + const names = (await readdir(BUNDLE_DIR, { recursive: true })).filter((name) => name.endsWith(".mthds")).sort(); + return Promise.all(names.map((name) => readFile(path.join(BUNDLE_DIR, name), "utf8"))); +} -export interface SummarizePdfInputs { +export type SummarizePdfInputs = { /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, - * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * run `getPipelexClient().prepareInputs({ files: (await readBundle()).map((c) => ({ content: c })), inputs })` first — * it uploads and rewrites the value. Note: prepareInputs treats any string it does not * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and * uploads, so a public endpoint must gate schemes before handing values to it. */ document: { url: string }; context?: string; -} +}; export async function summarizePdf(inputs: SummarizePdfInputs): Promise { - const bundle = await readFile(BUNDLE_PATH, "utf8"); const results: RunResults = await getPipelexClient().startAndWaitForResult({ pipe_code: PIPE_CODE, - mthds_contents: [bundle], + mthds_contents: await readBundle(), inputs, }); return parseDocumentSummary(results.main_stuff); } ``` -Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: +**`SummarizePdfInputs` is a `type`, not an `interface`, and that is load-bearing.** The SDK takes `inputs: Record`, and TypeScript gives a type alias of an object type an implicit index signature while an `interface` gets none — so an interface here fails with `TS2322: Index signature for type 'string' is missing`. It fails on every resolution, bundler included, and it is the first thing a `tsc --noEmit` would have caught. Keep it a `type`. + +**The three relative imports above are extensionless, which is correct only on a bundler resolution.** On the plain Node ESM shape that meets the emitter's `TS2835` defect (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) this module needs `.js` on each of them — `"../generated/summarize-pdf/binder.js"`, `"../generated/summarize-pdf/types.js"`, `"./client.js"`. That is your own module and so your own fix, unlike the stamped `binder.ts`: write the extensions when the project's resolution demands them, and do not report your own module's `TS2835` as the emitter's defect. + +Variants by selector, replacing the `mthds_contents` line and dropping `BUNDLE_DIR` and `readBundle`: - **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. - **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md index 98b683f..7602a15 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md @@ -10,13 +10,15 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | | A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | -| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | `uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +**Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. + ## TypeScript / JavaScript | Want | Command | `git init`? | Where `src/` lands | @@ -31,6 +33,8 @@ After the initializer: `git init -b main` only if it did not initialize a reposi `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/pipelex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs index ade01da..0e9fe5e 100644 --- a/pipelex/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs @@ -32,7 +32,10 @@ const LOCK_FILENAME = "codegen.lock"; const SIDECAR_FILENAME = "sources.json"; const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); -const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); +// `ignoreBOM: true` keeps a leading BOM in the decoded string. The default strips it, so an +// artifact given a BOM would hash as its un-BOM'd self, match the lock, and report current while +// the bytes on disk are hand-edited — the gate's one job is to not say that. +const strictUtf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); const out = (line) => process.stdout.write(`${line}\n`); const err = (line) => process.stderr.write(`${line}\n`); @@ -116,9 +119,25 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } - const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to + // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and + // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it + // in the sidecar really is an array), and `null` is why this does not use `??`, which would + // quietly turn an explicit null into the legitimate absent case. + const sources = sidecar?.sources; + if (sources === undefined) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + if (typeof sources !== "object" || sources === null || Array.isArray(sources)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — \`sources\` is not an object, so staleness cannot be ruled out`] }; + } + const recordedSources = Object.entries(sources).sort(); + if (recordedSources.length === 0) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + const lines = []; - for (const [source, recorded] of Object.entries(sources).sort()) { + for (const [source, recorded] of recordedSources) { let onDisk; try { onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); @@ -163,4 +182,7 @@ async function main(argv) { return exitCode; } -process.exit(await main(process.argv)); +// `process.exitCode` rather than `process.exit`: stdout is a pipe under `npm run`, `make` and every +// CI runner, where writes are asynchronous and `process.exit` drops the ones still pending. The code +// would survive either way; the drift lines explaining it are what gets truncated. +process.exitCode = await main(process.argv); diff --git a/pipelex/skills/pipelex-integrate/references/python.md b/pipelex/skills/pipelex-integrate/references/python.md index cc64358..8acdbde 100644 --- a/pipelex/skills/pipelex-integrate/references/python.md +++ b/pipelex/skills/pipelex-integrate/references/python.md @@ -57,7 +57,14 @@ from pipelex_sdk.client import PipelexAPIClient from .generated.summarize_pdf.models import DocumentSummary PIPE_CODE = "summarize_pdf" -BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" +BUNDLE_DIR = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" + + +def _read_bundle() -> list[str]: + """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that + imports a sibling needs that sibling submitted with it, or the run fails to load what the + generated models were projected from.""" + return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: @@ -67,7 +74,7 @@ async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) async with PipelexAPIClient() as client: results = await client.start_and_wait( pipe_code=PIPE_CODE, - mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + mthds_contents=_read_bundle(), inputs=inputs, ) return DocumentSummary.model_validate(results.main_stuff) @@ -78,7 +85,7 @@ def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) return asyncio.run(summarize_pdf(document=document, context=context)) ``` -Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: +Variants by selector, replacing the `mthds_contents=` argument and dropping `BUNDLE_DIR` and `_read_bundle`: - **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. - **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. diff --git a/pipelex/skills/pipelex-integrate/references/typescript.md b/pipelex/skills/pipelex-integrate/references/typescript.md index 84dc2d9..9ed2da6 100644 --- a/pipelex/skills/pipelex-integrate/references/typescript.md +++ b/pipelex/skills/pipelex-integrate/references/typescript.md @@ -15,7 +15,7 @@ Companion to `/pipelex-integrate` for a project that has a `package.json`. Every | **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | | **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | | **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | -| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below, but **read the script before believing it**: `"codegen": "graphql-codegen"` or a protobuf generator satisfies the name and generates no MTHDS types, and deferring to it would skip the dependencies, the exclusions, the sidecar and the gate while generating nothing. Pipelex-specific evidence — it calls `mthds_codegen`, a `pipelex` CLI, or reads `methods/` — is what makes it this method's harness; without that, integrate normally and leave the unrelated harness alone | Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. @@ -37,7 +37,7 @@ One module per method. `summarize-pdf` with a `document: native.Document` input, ```ts // src/pipelex/summarizePdf.ts -import { readFile } from "node:fs/promises"; +import { readdir, readFile } from "node:fs/promises"; import path from "node:path"; import type { RunResults } from "@pipelex/sdk"; import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; @@ -45,30 +45,42 @@ import type { DocumentSummary } from "../generated/summarize-pdf/types"; import { getPipelexClient } from "./client"; const PIPE_CODE = "summarize_pdf"; -const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); +const BUNDLE_DIR = path.join(process.cwd(), "methods", "summarize-pdf"); + +/** Every `.mthds` file of the bundle, sorted, as the run's `mthds_contents`. A bundle is one + * closure: a main file that imports a sibling needs that sibling submitted with it, or the + * run fails to load what the generated types were projected from. `recursive` needs Node + * >= 20.1 (or >= 18.17); below that, walk the directory yourself. */ +async function readBundle(): Promise { + const names = (await readdir(BUNDLE_DIR, { recursive: true })).filter((name) => name.endsWith(".mthds")).sort(); + return Promise.all(names.map((name) => readFile(path.join(BUNDLE_DIR, name), "utf8"))); +} -export interface SummarizePdfInputs { +export type SummarizePdfInputs = { /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, - * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * run `getPipelexClient().prepareInputs({ files: (await readBundle()).map((c) => ({ content: c })), inputs })` first — * it uploads and rewrites the value. Note: prepareInputs treats any string it does not * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and * uploads, so a public endpoint must gate schemes before handing values to it. */ document: { url: string }; context?: string; -} +}; export async function summarizePdf(inputs: SummarizePdfInputs): Promise { - const bundle = await readFile(BUNDLE_PATH, "utf8"); const results: RunResults = await getPipelexClient().startAndWaitForResult({ pipe_code: PIPE_CODE, - mthds_contents: [bundle], + mthds_contents: await readBundle(), inputs, }); return parseDocumentSummary(results.main_stuff); } ``` -Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: +**`SummarizePdfInputs` is a `type`, not an `interface`, and that is load-bearing.** The SDK takes `inputs: Record`, and TypeScript gives a type alias of an object type an implicit index signature while an `interface` gets none — so an interface here fails with `TS2322: Index signature for type 'string' is missing`. It fails on every resolution, bundler included, and it is the first thing a `tsc --noEmit` would have caught. Keep it a `type`. + +**The three relative imports above are extensionless, which is correct only on a bundler resolution.** On the plain Node ESM shape that meets the emitter's `TS2835` defect (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) this module needs `.js` on each of them — `"../generated/summarize-pdf/binder.js"`, `"../generated/summarize-pdf/types.js"`, `"./client.js"`. That is your own module and so your own fix, unlike the stamped `binder.ts`: write the extensions when the project's resolution demands them, and do not report your own module's `TS2835` as the emitter's defect. + +Variants by selector, replacing the `mthds_contents` line and dropping `BUNDLE_DIR` and `readBundle`: - **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. - **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. diff --git a/pipelex/skills/pipelex-scaffold/references/initializers.md b/pipelex/skills/pipelex-scaffold/references/initializers.md index 98b683f..7602a15 100644 --- a/pipelex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex/skills/pipelex-scaffold/references/initializers.md @@ -10,13 +10,15 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | | A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | -| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | `uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +**Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. + ## TypeScript / JavaScript | Want | Command | `git init`? | Where `src/` lands | @@ -31,6 +33,8 @@ After the initializer: `git init -b main` only if it did not initialize a reposi `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/skills/pipelex-integrate/references/codegen-check.mjs b/skills/pipelex-integrate/references/codegen-check.mjs index ade01da..0e9fe5e 100644 --- a/skills/pipelex-integrate/references/codegen-check.mjs +++ b/skills/pipelex-integrate/references/codegen-check.mjs @@ -32,7 +32,10 @@ const LOCK_FILENAME = "codegen.lock"; const SIDECAR_FILENAME = "sources.json"; const PRUNED_DIRECTORIES = new Set(["node_modules", ".git", "dist", "build", ".next"]); -const strictUtf8 = new TextDecoder("utf-8", { fatal: true }); +// `ignoreBOM: true` keeps a leading BOM in the decoded string. The default strips it, so an +// artifact given a BOM would hash as its un-BOM'd self, match the lock, and report current while +// the bytes on disk are hand-edited — the gate's one job is to not say that. +const strictUtf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); const out = (line) => process.stdout.write(`${line}\n`); const err = (line) => process.stderr.write(`${line}\n`); @@ -116,9 +119,25 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } - const sources = sidecar && typeof sidecar.sources === "object" && sidecar.sources !== null ? sidecar.sources : {}; + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to + // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and + // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it + // in the sidecar really is an array), and `null` is why this does not use `??`, which would + // quietly turn an explicit null into the legitimate absent case. + const sources = sidecar?.sources; + if (sources === undefined) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + if (typeof sources !== "object" || sources === null || Array.isArray(sources)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — \`sources\` is not an object, so staleness cannot be ruled out`] }; + } + const recordedSources = Object.entries(sources).sort(); + if (recordedSources.length === 0) { + return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; + } + const lines = []; - for (const [source, recorded] of Object.entries(sources).sort()) { + for (const [source, recorded] of recordedSources) { let onDisk; try { onDisk = sha256(await readFile(path.resolve(process.cwd(), source))); @@ -163,4 +182,7 @@ async function main(argv) { return exitCode; } -process.exit(await main(process.argv)); +// `process.exitCode` rather than `process.exit`: stdout is a pipe under `npm run`, `make` and every +// CI runner, where writes are asynchronous and `process.exit` drops the ones still pending. The code +// would survive either way; the drift lines explaining it are what gets truncated. +process.exitCode = await main(process.argv); diff --git a/skills/pipelex-integrate/references/python.md b/skills/pipelex-integrate/references/python.md index cc64358..8acdbde 100644 --- a/skills/pipelex-integrate/references/python.md +++ b/skills/pipelex-integrate/references/python.md @@ -57,7 +57,14 @@ from pipelex_sdk.client import PipelexAPIClient from .generated.summarize_pdf.models import DocumentSummary PIPE_CODE = "summarize_pdf" -BUNDLE_PATH = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" / "main.mthds" +BUNDLE_DIR = Path(__file__).resolve().parent.parent / "methods" / "summarize_pdf" + + +def _read_bundle() -> list[str]: + """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that + imports a sibling needs that sibling submitted with it, or the run fails to load what the + generated models were projected from.""" + return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: @@ -67,7 +74,7 @@ async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) async with PipelexAPIClient() as client: results = await client.start_and_wait( pipe_code=PIPE_CODE, - mthds_contents=[BUNDLE_PATH.read_text(encoding="utf-8")], + mthds_contents=_read_bundle(), inputs=inputs, ) return DocumentSummary.model_validate(results.main_stuff) @@ -78,7 +85,7 @@ def summarize_pdf_sync(*, document: dict[str, Any], context: str | None = None) return asyncio.run(summarize_pdf(document=document, context=context)) ``` -Variants by selector, replacing the `mthds_contents=` argument and dropping the bundle path: +Variants by selector, replacing the `mthds_contents=` argument and dropping `BUNDLE_DIR` and `_read_bundle`: - **`method_ref`** at a tag: `method_ref="github.com//[/]@"`, with `pipe_code=PIPE_CODE` or omitted to run the package's declared pipe. - **`method_id`**: `method_id="mt_…"`; the module docstring says the catalog is unversioned. diff --git a/skills/pipelex-integrate/references/typescript.md b/skills/pipelex-integrate/references/typescript.md index 84dc2d9..9ed2da6 100644 --- a/skills/pipelex-integrate/references/typescript.md +++ b/skills/pipelex-integrate/references/typescript.md @@ -15,7 +15,7 @@ Companion to `/pipelex-integrate` for a project that has a `package.json`. Every | **Aggregate gate** | `package.json` `scripts.check` / `ci` / `validate` / `verify`; a Makefile `check` target; a `.github/workflows/*.yml` job with a lint or test step; `.pre-commit-config.yaml` | none: the `codegen:check` script alone, and a sentence in the report saying where to call it | | **Call-site location** | the project's existing service / action / client layer (`src/actions/`, `src/services/`, `src/lib/`, `src/server/`) → beside it | `src/pipelex/` | | **Not gitignored** | the generated root and `sources.json` must be committable | a `.gitignore` pattern that swallows them is reported and un-ignored on confirmation | -| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below | +| **Owns a codegen harness** | `scripts.codegen` in `package.json`; `sources.json` with a `derived` map; `docs/codegen.md`; `make add-method` | either of the first two → the harness section below, but **read the script before believing it**: `"codegen": "graphql-codegen"` or a protobuf generator satisfies the name and generates no MTHDS types, and deferring to it would skip the dependencies, the exclusions, the sidecar and the gate while generating nothing. Pipelex-specific evidence — it calls `mthds_codegen`, a `pipelex` CLI, or reads `methods/` — is what makes it this method's harness; without that, integrate normally and leave the unrelated harness alone | Why the exclusions are not optional: the ts-zod emitter prints at Prettier's defaults (80 columns). A project that prints at another width, or a linter with an autofix, rewrites the bytes, breaks every stamp, and makes the offline check report the whole tree as hand-edited. The type checker, by contrast, must keep covering the tree — that is the check that catches a call site drifting from its types. @@ -37,7 +37,7 @@ One module per method. `summarize-pdf` with a `document: native.Document` input, ```ts // src/pipelex/summarizePdf.ts -import { readFile } from "node:fs/promises"; +import { readdir, readFile } from "node:fs/promises"; import path from "node:path"; import type { RunResults } from "@pipelex/sdk"; import { parseDocumentSummary } from "../generated/summarize-pdf/binder"; @@ -45,30 +45,42 @@ import type { DocumentSummary } from "../generated/summarize-pdf/types"; import { getPipelexClient } from "./client"; const PIPE_CODE = "summarize_pdf"; -const BUNDLE_PATH = path.join(process.cwd(), "methods", "summarize-pdf", "main.mthds"); +const BUNDLE_DIR = path.join(process.cwd(), "methods", "summarize-pdf"); + +/** Every `.mthds` file of the bundle, sorted, as the run's `mthds_contents`. A bundle is one + * closure: a main file that imports a sibling needs that sibling submitted with it, or the + * run fails to load what the generated types were projected from. `recursive` needs Node + * >= 20.1 (or >= 18.17); below that, walk the directory yourself. */ +async function readBundle(): Promise { + const names = (await readdir(BUNDLE_DIR, { recursive: true })).filter((name) => name.endsWith(".mthds")).sort(); + return Promise.all(names.map((name) => readFile(path.join(BUNDLE_DIR, name), "utf8"))); +} -export interface SummarizePdfInputs { +export type SummarizePdfInputs = { /** An http(s) URL or a pipelex-storage:// reference. For a local file or bytes, - * run `getPipelexClient().prepareInputs({ files: [{ content: bundle }], inputs })` first — + * run `getPipelexClient().prepareInputs({ files: (await readBundle()).map((c) => ({ content: c })), inputs })` first — * it uploads and rewrites the value. Note: prepareInputs treats any string it does not * recognise as data:, http(s):// or pipelex-storage:// as a LOCAL FILE PATH it reads and * uploads, so a public endpoint must gate schemes before handing values to it. */ document: { url: string }; context?: string; -} +}; export async function summarizePdf(inputs: SummarizePdfInputs): Promise { - const bundle = await readFile(BUNDLE_PATH, "utf8"); const results: RunResults = await getPipelexClient().startAndWaitForResult({ pipe_code: PIPE_CODE, - mthds_contents: [bundle], + mthds_contents: await readBundle(), inputs, }); return parseDocumentSummary(results.main_stuff); } ``` -Variants by selector, replacing the `mthds_contents` line and dropping the bundle read: +**`SummarizePdfInputs` is a `type`, not an `interface`, and that is load-bearing.** The SDK takes `inputs: Record`, and TypeScript gives a type alias of an object type an implicit index signature while an `interface` gets none — so an interface here fails with `TS2322: Index signature for type 'string' is missing`. It fails on every resolution, bundler included, and it is the first thing a `tsc --noEmit` would have caught. Keep it a `type`. + +**The three relative imports above are extensionless, which is correct only on a bundler resolution.** On the plain Node ESM shape that meets the emitter's `TS2835` defect (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) this module needs `.js` on each of them — `"../generated/summarize-pdf/binder.js"`, `"../generated/summarize-pdf/types.js"`, `"./client.js"`. That is your own module and so your own fix, unlike the stamped `binder.ts`: write the extensions when the project's resolution demands them, and do not report your own module's `TS2835` as the emitter's defect. + +Variants by selector, replacing the `mthds_contents` line and dropping `BUNDLE_DIR` and `readBundle`: - **`method_ref`** at a tag: `{ method_ref: "github.com//[/]@", pipe_code: PIPE_CODE, inputs }` — omit `pipe_code` to run the package's declared pipe. - **`method_id`**: `{ method_id: "mt_…", inputs }` — the catalog resolves the stored method; the module's header says the catalog is unversioned. diff --git a/skills/pipelex-scaffold/references/initializers.md b/skills/pipelex-scaffold/references/initializers.md index 98b683f..7602a15 100644 --- a/skills/pipelex-scaffold/references/initializers.md +++ b/skills/pipelex-scaffold/references/initializers.md @@ -10,13 +10,15 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | | A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && uv add "fastapi[standard]"` | yes | as minimal | -| Django project | `uv init --package && uv add django && (cd && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && uv add typer` | yes | as minimal | +| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | `uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +**Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. + ## TypeScript / JavaScript | Want | Command | `git init`? | Where `src/` lands | @@ -31,6 +33,8 @@ After the initializer: `git init -b main` only if it did not initialize a reposi `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. From 6a8625fcb53ef38dce7c253bd01492d9273e0428 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:43:48 +0200 Subject: [PATCH 15/21] Keep the dogfood's own guards from misfiring on the states they create MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each of the three integrate guards the dogfood added refuses a case it is itself responsible for. The occupied-directory rule reads a `codegen.lock` with no sidecar as another generation's. But the lock is written in step 6 and the sidecar in step 7, so every stop between them — a reported `is_current: false`, a non-empty `orphans[]`, a partial write whose retry also failed — leaves exactly that state for the method being integrated right now, and the rule then sends the agent to a different directory: a stranded half-tree plus a second tree for one method, which is the fragmentation the rule exists to prevent. A harness-owned layout is the same shape for a different reason, keeping no sidecar by design, so the rule forbade the write the harness section prescribes two screens later. Both are named exceptions now, with regeneration in place as the answer. The containment pre-check was a lexical reading, which is wrong in both directions: a symlink inside the workshop pointing at a project outside it reads as contained, and on macOS a project under `/tmp` reads as outside because the workshop's own `process.cwd()` is the `/private` form of the same place. Both sides are resolved before comparing. The reading is also taken as soon as step 1 names the project, because both inputs are known there and by the old placement a bundle copy, possibly a staged `git mv`, and the step-5 exclusions are already on disk — so a stop that happens anyway now says what is already written, none of it the agent's to revert. Two more from the same family. `pipe_ref` is namespaced and the run route takes the bare code, and nothing said to strip the domain — a confusion that type-checks, passes the offline gate, and fails only on a real run, which step 11 never does. A `variable` or `fixed` output arrives as an array that a single-concept parser rejects, so narrowing follows the multiplicity step 3 recorded. And the three causes of an absent `main_pipe` are all three in the failure table, where they matter most because the remedy differs by cause: a contract that did not come back whole comes from the runner, and no workshop refresh touches it. The same signature rides the verdict's text summary, so a host that does not surface structured content is not a method without a signature. Co-Authored-By: Claude Opus 5 (1M context) --- .../skills/pipelex-integrate/SKILL.md | 19 ++++++++++--------- .../skills/pipelex-integrate/SKILL.md | 19 ++++++++++--------- pipelex/skills/pipelex-integrate/SKILL.md | 19 ++++++++++--------- .../skills/pipelex-integrate/SKILL.md.j2 | 19 ++++++++++--------- 4 files changed, 40 insertions(+), 36 deletions(-) diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index 35988da..accf990 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -69,9 +69,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. **`pipe_ref` and the run's `pipe_code` are not the same string**: `pipe_ref` is namespaced (`summarize.summarize_pdf`) and the run route takes the code alone (`summarize_pdf`), so strip the domain when the signature becomes the call site's `PIPE_CODE`. The sidecar records the namespaced `pipe_ref`, the call site passes the bare code, and nothing catches a confusion between them — a namespaced `pipe_code` type-checks, passes the offline gate, and fails only when the method is actually run, which neither gate in step 11 does. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases — so it is the **same fact on a second channel, not a fourth cause**, and that is what makes it useful: when the structured field did not reach you but a `## Main pipe` line did, read the signature from there rather than treating the method as unsignatured. A host that does not surface structured content, or a cached tool schema, is the usual reason, and it is not a property of the method. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -83,7 +83,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. **One lock-without-sidecar is this method's own, and relocating is the wrong answer for it**: the lock is written in step 6 and the sidecar in step 7, so every stop between them — a reported `is_current: false`, a non-empty `orphans[]`, a partial write whose retry also failed — leaves exactly that state for the method you are integrating now. When the destination is the one this method would have chosen and the tree's artifacts are the target's, treat it as an interrupted run of your own: regenerate in place, which overwrites its own stamped files, and write the sidecar that was missing. Two trees for one method is the fragmentation this rule exists to prevent, not a way out of it. Where you cannot tell whose the tree is, ask — never relocate silently, and never clear it. A harness-owned layout is a further exception, named in its own section. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -91,7 +91,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. **Resolve both sides before comparing them** (`pwd -P`, `realpath`), because a lexical reading is wrong in both directions: a symlink inside the workshop pointing at a project outside it reads as contained, and on macOS a project under `/tmp` or `/var` reads as outside when the workshop's own `process.cwd()` is the `/private/...` form of the same place. **Both inputs are known as soon as step 1 names the project, so do this reading there** and only restate it here: by the time you reach this step you have already copied a bundle into the project, possibly staged a `git mv`, and written the step-5 exclusions, so a stop now leaves all of that on disk — if you arrive here anyway, say in the stop exactly what is already written, because none of it is yours to revert. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -131,7 +131,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ ### Step 9: Write the call site -**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads **every `.mthds` file of the committed bundle** at call time (files source) — a bundle is one closure, so a main file that imports a sibling needs that sibling in the same `mthds_contents`, and a call site that submits `main.mthds` alone fails at load time on exactly the methods codegen handled correctly — or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). **Narrow according to the output's `multiplicity`, which step 3 recorded** — the single-concept form above is right only for `single`. A `variable` or `fixed` output arrives as an array, and a generated single-object parser rejects an array, so a run that succeeded fails in the narrowing: map the parser over the items and return `T[]` / `list[T]` (`z.array(Schema)`, `TypeAdapter(list[Model])`), and for `fixed` say in the report that the declared `item_count` is not checked unless you check it. `main_stuff` is `unknown` / `Any`, so the type checker does not catch this one either — it surfaces only on a real run, which step 11 never does. Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. | Declared concept | TypeScript parameter | Python parameter | |---|---|---| @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why, or let the user overrule by naming the method that directory is for, as step 4 allows. The exception is a harness-owned layout, which keeps no sidecar by design — see that section. ## A project that owns a codegen harness @@ -180,7 +180,7 @@ A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one tha - **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); - **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. -The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. **This is the one destination step 4's sidecar rule does not govern**: the harness owns the directory and deliberately keeps no `sources.json`, so on every run after the first it holds a lock with no sidecar — which would otherwise read as another generation's. The harness branch is entered at step 1, before that rule applies; write into the layout again rather than inventing a second directory name, which is the thing this whole section exists to prevent. ## When something goes wrong @@ -198,9 +198,10 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | first look for the one-line signature in the verdict's **text summary** — it carries the same thing and reaches a host that does not surface structured content. Genuinely absent → STOP: all three causes of step 3, and the remedy differs by cause — the workshop predates it (refresh `@pipelex/mcp` and retry), the method settles no entry pipe, or the entry pipe's contract did not come back whole, which comes from the runner behind `/v1/validate` and no refresh fixes. Never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | -| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | +| `TS2835` on a relative import, in **your own** call-site module | yours to fix, per step 11: on a project whose resolution demands extensions, the module's own imports of the generated tree and its client helper take `.js` (`from "../generated//binder.js"`). Do not route this to the row below — that one is about the stamped tree only | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. On this project shape the integration does not merely fail a check — the compiled code cannot load the module at all (`ERR_MODULE_NOT_FOUND`), so the report says the integration cannot execute until the emitter is fixed or the user changes `moduleResolution`, not that a check is red. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index f89ce7d..1f3e250 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -69,9 +69,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. **`pipe_ref` and the run's `pipe_code` are not the same string**: `pipe_ref` is namespaced (`summarize.summarize_pdf`) and the run route takes the code alone (`summarize_pdf`), so strip the domain when the signature becomes the call site's `PIPE_CODE`. The sidecar records the namespaced `pipe_ref`, the call site passes the bare code, and nothing catches a confusion between them — a namespaced `pipe_code` type-checks, passes the offline gate, and fails only when the method is actually run, which neither gate in step 11 does. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases — so it is the **same fact on a second channel, not a fourth cause**, and that is what makes it useful: when the structured field did not reach you but a `## Main pipe` line did, read the signature from there rather than treating the method as unsignatured. A host that does not surface structured content, or a cached tool schema, is the usual reason, and it is not a property of the method. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -83,7 +83,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. **One lock-without-sidecar is this method's own, and relocating is the wrong answer for it**: the lock is written in step 6 and the sidecar in step 7, so every stop between them — a reported `is_current: false`, a non-empty `orphans[]`, a partial write whose retry also failed — leaves exactly that state for the method you are integrating now. When the destination is the one this method would have chosen and the tree's artifacts are the target's, treat it as an interrupted run of your own: regenerate in place, which overwrites its own stamped files, and write the sidecar that was missing. Two trees for one method is the fragmentation this rule exists to prevent, not a way out of it. Where you cannot tell whose the tree is, ask — never relocate silently, and never clear it. A harness-owned layout is a further exception, named in its own section. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -91,7 +91,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root (or register the workshop with that working directory) — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. **Resolve both sides before comparing them** (`pwd -P`, `realpath`), because a lexical reading is wrong in both directions: a symlink inside the workshop pointing at a project outside it reads as contained, and on macOS a project under `/tmp` or `/var` reads as outside when the workshop's own `process.cwd()` is the `/private/...` form of the same place. **Both inputs are known as soon as step 1 names the project, so do this reading there** and only restate it here: by the time you reach this step you have already copied a bundle into the project, possibly staged a `git mv`, and written the step-5 exclusions, so a stop now leaves all of that on disk — if you arrive here anyway, say in the stop exactly what is already written, because none of it is yours to revert. Then STOP with the instruction to relaunch the harness from the project root (or register the workshop with that working directory) — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -131,7 +131,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ ### Step 9: Write the call site -**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads **every `.mthds` file of the committed bundle** at call time (files source) — a bundle is one closure, so a main file that imports a sibling needs that sibling in the same `mthds_contents`, and a call site that submits `main.mthds` alone fails at load time on exactly the methods codegen handled correctly — or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). **Narrow according to the output's `multiplicity`, which step 3 recorded** — the single-concept form above is right only for `single`. A `variable` or `fixed` output arrives as an array, and a generated single-object parser rejects an array, so a run that succeeded fails in the narrowing: map the parser over the items and return `T[]` / `list[T]` (`z.array(Schema)`, `TypeAdapter(list[Model])`), and for `fixed` say in the report that the declared `item_count` is not checked unless you check it. `main_stuff` is `unknown` / `Any`, so the type checker does not catch this one either — it surfaces only on a real run, which step 11 never does. Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. | Declared concept | TypeScript parameter | Python parameter | |---|---|---| @@ -168,7 +168,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why, or let the user overrule by naming the method that directory is for, as step 4 allows. The exception is a harness-owned layout, which keeps no sidecar by design — see that section. ## A project that owns a codegen harness @@ -180,7 +180,7 @@ A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one tha - **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); - **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. -The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. **This is the one destination step 4's sidecar rule does not govern**: the harness owns the directory and deliberately keeps no `sources.json`, so on every run after the first it holds a lock with no sidecar — which would otherwise read as another generation's. The harness branch is entered at step 1, before that rule applies; write into the layout again rather than inventing a second directory name, which is the thing this whole section exists to prevent. ## When something goes wrong @@ -198,9 +198,10 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | first look for the one-line signature in the verdict's **text summary** — it carries the same thing and reaches a host that does not surface structured content. Genuinely absent → STOP: all three causes of step 3, and the remedy differs by cause — the workshop predates it (refresh `@pipelex/mcp` and retry), the method settles no entry pipe, or the entry pipe's contract did not come back whole, which comes from the runner behind `/v1/validate` and no refresh fixes. Never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | -| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | +| `TS2835` on a relative import, in **your own** call-site module | yours to fix, per step 11: on a project whose resolution demands extensions, the module's own imports of the generated tree and its client helper take `.js` (`from "../generated//binder.js"`). Do not route this to the row below — that one is about the stamped tree only | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. On this project shape the integration does not merely fail a check — the compiled code cannot load the module at all (`ERR_MODULE_NOT_FOUND`), so the report says the integration cannot execute until the emitter is fixed or the user changes `moduleResolution`, not that a check is red. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index eeadb0b..ceba242 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -80,9 +80,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. **`pipe_ref` and the run's `pipe_code` are not the same string**: `pipe_ref` is namespaced (`summarize.summarize_pdf`) and the run route takes the code alone (`summarize_pdf`), so strip the domain when the signature becomes the call site's `PIPE_CODE`. The sidecar records the namespaced `pipe_ref`, the call site passes the bare code, and nothing catches a confusion between them — a namespaced `pipe_code` type-checks, passes the offline gate, and fails only when the method is actually run, which neither gate in step 11 does. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases — so it is the **same fact on a second channel, not a fourth cause**, and that is what makes it useful: when the structured field did not reach you but a `## Main pipe` line did, read the signature from there rather than treating the method as unsignatured. A host that does not surface structured content, or a cached tool schema, is the usual reason, and it is not a property of the method. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -94,7 +94,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. **One lock-without-sidecar is this method's own, and relocating is the wrong answer for it**: the lock is written in step 6 and the sidecar in step 7, so every stop between them — a reported `is_current: false`, a non-empty `orphans[]`, a partial write whose retry also failed — leaves exactly that state for the method you are integrating now. When the destination is the one this method would have chosen and the tree's artifacts are the target's, treat it as an interrupted run of your own: regenerate in place, which overwrites its own stamped files, and write the sidecar that was missing. Two trees for one method is the fragmentation this rule exists to prevent, not a way out of it. Where you cannot tell whose the tree is, ask — never relocate silently, and never clear it. A harness-owned layout is a further exception, named in its own section. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -102,7 +102,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. **Resolve both sides before comparing them** (`pwd -P`, `realpath`), because a lexical reading is wrong in both directions: a symlink inside the workshop pointing at a project outside it reads as contained, and on macOS a project under `/tmp` or `/var` reads as outside when the workshop's own `process.cwd()` is the `/private/...` form of the same place. **Both inputs are known as soon as step 1 names the project, so do this reading there** and only restate it here: by the time you reach this step you have already copied a bundle into the project, possibly staged a `git mv`, and written the step-5 exclusions, so a stop now leaves all of that on disk — if you arrive here anyway, say in the stop exactly what is already written, because none of it is yours to revert. Then STOP with the instruction to relaunch the harness from the project root — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -142,7 +142,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ ### Step 9: Write the call site -**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads **every `.mthds` file of the committed bundle** at call time (files source) — a bundle is one closure, so a main file that imports a sibling needs that sibling in the same `mthds_contents`, and a call site that submits `main.mthds` alone fails at load time on exactly the methods codegen handled correctly — or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). **Narrow according to the output's `multiplicity`, which step 3 recorded** — the single-concept form above is right only for `single`. A `variable` or `fixed` output arrives as an array, and a generated single-object parser rejects an array, so a run that succeeded fails in the narrowing: map the parser over the items and return `T[]` / `list[T]` (`z.array(Schema)`, `TypeAdapter(list[Model])`), and for `fixed` say in the report that the declared `item_count` is not checked unless you check it. `main_stuff` is `unknown` / `Any`, so the type checker does not catch this one either — it surfaces only on a real run, which step 11 never does. Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. | Declared concept | TypeScript parameter | Python parameter | |---|---|---| @@ -179,7 +179,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why, or let the user overrule by naming the method that directory is for, as step 4 allows. The exception is a harness-owned layout, which keeps no sidecar by design — see that section. ## A project that owns a codegen harness @@ -191,7 +191,7 @@ A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one tha - **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); - **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. -The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. **This is the one destination step 4's sidecar rule does not govern**: the harness owns the directory and deliberately keeps no `sources.json`, so on every run after the first it holds a lock with no sidecar — which would otherwise read as another generation's. The harness branch is entered at step 1, before that rule applies; write into the layout again rather than inventing a second directory name, which is the thing this whole section exists to prevent. ## When something goes wrong @@ -209,9 +209,10 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | first look for the one-line signature in the verdict's **text summary** — it carries the same thing and reaches a host that does not surface structured content. Genuinely absent → STOP: all three causes of step 3, and the remedy differs by cause — the workshop predates it (refresh `@pipelex/mcp` and retry), the method settles no entry pipe, or the entry pipe's contract did not come back whole, which comes from the runner behind `/v1/validate` and no refresh fixes. Never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | -| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | +| `TS2835` on a relative import, in **your own** call-site module | yours to fix, per step 11: on a project whose resolution demands extensions, the module's own imports of the generated tree and its client helper take `.js` (`from "../generated//binder.js"`). Do not route this to the row below — that one is about the stamped tree only | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. On this project shape the integration does not merely fail a check — the compiled code cannot load the module at all (`ERR_MODULE_NOT_FOUND`), so the report says the integration cannot execute until the emitter is fixed or the user changes `moduleResolution`, not that a check is red. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index 077c274..f50176b 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -75,9 +75,9 @@ Call **`mthds_validate`** with the selector. Branch: ### Step 3: Read the pipe's signature — from the verdict -A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. +A valid verdict carries **`main_pipe`** in its structured content: `pipe_ref` (the namespaced `domain.pipe_code`), `inputs[]` — each with `name`, `concept_ref`, `multiplicity` (`single` / `variable` / `fixed` with `item_count`) and `required` — and `output` with `concept_ref`, `multiplicity` and `optional`. That is everything the call site is typed against, for every selector alike; record it. **`pipe_ref` and the run's `pipe_code` are not the same string**: `pipe_ref` is namespaced (`summarize.summarize_pdf`) and the run route takes the code alone (`summarize_pdf`), so strip the domain when the signature becomes the call site's `PIPE_CODE`. The sidecar records the namespaced `pipe_ref`, the call site passes the bare code, and nothing catches a confusion between them — a namespaced `pipe_code` type-checks, passes the offline gate, and fails only when the method is actually run, which neither gate in step 11 does. Type and run the call site against **that** pipe: `main_pipe.pipe_ref` is the pipe a run with no pipe selector executes — for a published package, the entry its `METHODS.toml` names, which can differ from the `main_pipe` its bundle declares. -**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. +**The fallback, when `main_pipe` is absent.** The workshop omits it whole — never a partial signature, and the verdict is unaffected — in three cases: the method settles **no entry pipe** (the bundle declares no `main_pipe`, or a published package's `METHODS.toml` names a pipe the closure does not declare or declares in several domains, which is exactly when a run with no pipe selector would fail too); the entry pipe's contract did not come back whole; or the workshop **predates the signature** (`@pipelex/mcp` 0.13.0 and earlier; `npx -y @pipelex/mcp@latest` refreshes it). The one-line signature in the verdict's text summary is missing in the same cases — so it is the **same fact on a second channel, not a fourth cause**, and that is what makes it useful: when the structured field did not reach you but a `## Main pipe` line did, read the signature from there rather than treating the method as unsignatured. A host that does not surface structured content, or a cached tool schema, is the usual reason, and it is not a property of the method. Then, for a **files source**: take the pipe the bundle declares as `main_pipe`, or ask which pipe to integrate when it declares none; call **`mthds_inputs_template`** with the selector, that `pipe_ref`, and **`explicit: true`** — the one call in this plugin that wants the ceremonial `{concept, content}` envelope, because it is the concept ref per input you need — and read the pipe's `output` declaration from the bundle. For a **`method_ref` or `method_id` source** the output concept has no in-context channel: STOP and say the verdict carries no signature to type this integration exactly — the workshop may predate it, or the method settles no entry pipe — rather than guessing. ### Step 4: Choose the target, the destination and the generator @@ -89,7 +89,7 @@ State the three in one line before writing. The rule for the target is about **a | `pyproject.toml`, `pipelex` **not** among the dependencies | `python-pydantic` | `models.py`, plain `BaseModel`s, no Pipelex import — for a consumer of the hosted API | | `pyproject.toml`, `pipelex` **is** a dependency and the code uses `@pipe_func` or `StructuredContent` | `python-structures` | `structures.py`, runtime `StructuredContent` classes — only for a Pipelex host; it imports the runtime and would not even load elsewhere | -`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). +`pipelex` present but neither signal in the code → one question, `python-structures` offered first. A JavaScript project with no TypeScript build is asked, because `types.ts` needs one. Field keys are wire-native snake_case in every target, TypeScript included. The destination is one dedicated directory per method — `src/generated//` or `/generated//` by default, beside any generated code the project already keeps; the method's directory name is the method's, in the language's casing (`summarize-pdf` in TypeScript, `summarize_pdf` in Python). **A directory that already holds a `codegen.lock` is this method's only when a `sources.json` beside it names this method.** A lock with no sidecar, or a sidecar naming another method, means the directory belongs to another generation — even when the user names it — and every method of a target emits the same file names, so generating into it would silently overwrite the other method's stamped files rather than report an orphan: choose another directory, say why, and let the user overrule only by naming the method that directory is for. **One lock-without-sidecar is this method's own, and relocating is the wrong answer for it**: the lock is written in step 6 and the sidecar in step 7, so every stop between them — a reported `is_current: false`, a non-empty `orphans[]`, a partial write whose retry also failed — leaves exactly that state for the method you are integrating now. When the destination is the one this method would have chosen and the tree's artifacts are the target's, treat it as an interrupted run of your own: regenerate in place, which overwrites its own stamped files, and write the sidecar that was missing. Two trees for one method is the fragmentation this rule exists to prevent, not a way out of it. Where you cannot tell whose the tree is, ask — never relocate silently, and never clear it. A harness-owned layout is a further exception, named in its own section. The generator is the workshop's write arm — or the project's harness, per its section. Detection detail: [references/typescript.md](references/typescript.md), [references/python.md](references/python.md). ### Step 5: Make the tooling leave the tree alone — before the tree exists @@ -97,7 +97,7 @@ Add the generated directory to the formatter's and linter's ignore lists per the ### Step 6: Generate -Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. Then STOP with the instruction to relaunch the harness from the project root{% if platform == "mistral-vibe" %} (or register the workshop with that working directory){% endif %} — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. +Call **`mthds_codegen`** with the selector, `target`, and **`output_dir`** — the generated directory's path **relative to the workshop's working directory**, which is the directory the harness was launched in (the launcher does not `cd`); never absolute. **Check containment before the call rather than waiting for an error**, because a wrong `output_dir` can be perfectly legal: the workshop's working directory is the directory this session started in, so the project is reachable only when its root is that directory or below it. Take the path from the workshop's working directory to the generated directory and read it — one that has to climb out (`../`) means the project is not under the workshop, and a project root elsewhere on disk means the same thing even when some path inside the workshop would be accepted. **Resolve both sides before comparing them** (`pwd -P`, `realpath`), because a lexical reading is wrong in both directions: a symlink inside the workshop pointing at a project outside it reads as contained, and on macOS a project under `/tmp` or `/var` reads as outside when the workshop's own `process.cwd()` is the `/private/...` form of the same place. **Both inputs are known as soon as step 1 names the project, so do this reading there** and only restate it here: by the time you reach this step you have already copied a bundle into the project, possibly staged a `git mv`, and written the step-5 exclusions, so a stop now leaves all of that on disk — if you arrive here anyway, say in the stop exactly what is already written, because none of it is yours to revert. Then STOP with the instruction to relaunch the harness from the project root{% if platform == "mistral-vibe" %} (or register the workshop with that working directory){% endif %} — do not ride content instead, do not pass a climbing path, and **never write the tree into the workshop's own directory and move it across afterwards**: the bytes would survive the move, but every later refresh meets the same mismatch and the sidecar's project-relative paths describe a project the workshop cannot see. Branch on the structured result: @@ -137,7 +137,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ ### Step 9: Write the call site -**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads the committed bundle at call time (files source) or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. +**One new module per method**, placed by the project's convention (its service / action / client layer when it has one; else `src/pipelex/.ts` or `/pipelex/.py`). It exports one async function named after the method whose parameters are the pipe's inputs, typed from the signature, and whose return type is the generated output type. It loads **every `.mthds` file of the committed bundle** at call time (files source) — a bundle is one closure, so a main file that imports a sibling needs that sibling in the same `mthds_contents`, and a call site that submits `main.mthds` alone fails at load time on exactly the methods codegen handled correctly — or names the pinned `method_ref` / the `method_id`, runs it through the SDK's self-healing lifecycle call (`startAndWaitForResult` / `start_and_wait` — the durable path on the hosted API, blocking on a bare runner), and narrows `main_stuff` through the generated binder (`parse(results.main_stuff)`) or model (`Model.model_validate(results.main_stuff)`). **Narrow according to the output's `multiplicity`, which step 3 recorded** — the single-concept form above is right only for `single`. A `variable` or `fixed` output arrives as an array, and a generated single-object parser rejects an array, so a run that succeeded fails in the narrowing: map the parser over the items and return `T[]` / `list[T]` (`z.array(Schema)`, `TypeAdapter(list[Model])`), and for `fixed` say in the report that the declared `item_count` is not checked unless you check it. `main_stuff` is `unknown` / `Any`, so the type checker does not catch this one either — it surfaces only on a real run, which step 11 never does. Credentials come from the environment through the SDK's own defaults (`PIPELEX_API_KEY`, `PIPELEX_BASE_URL`); the module never reads them itself. | Declared concept | TypeScript parameter | Python parameter | |---|---|---| @@ -174,7 +174,7 @@ Entered when the user asks to refresh, regenerate or update the types; when `/pi |---|---|---| | the selector, target, destination and `pipe` record — from the sidecar; the previous `crate_fingerprint` — from the lock | the source hashes (files source), compared **before** regenerating so the report can say whether the bundle actually changed; the signature, through `mthds_validate` again, compared to the sidecar's `pipe` record | the tooling exclusions (verified, re-added only if missing), the dependencies, the client helper, the gate wiring, tests, other methods' trees | -One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why. +One `mthds_codegen` call with the recorded arguments. Read what happened from the lock's **`artifacts[].content_hash`**, not from the fingerprint: `crate_fingerprint` covers the whole bundle, so a prompt-only edit moves it — and with it the one stamp line at the head of every artifact — while the projected code is byte-identical. Content hashes unchanged therefore means a pure restamp: say so, and leave that one-line-per-file diff to the user's commit. A content hash that moved is the concept set actually moving. Then the type checker. **The call site is edited only if it no longer type-checks or the `pipe` record no longer matches the signature** — a renamed input, a reshaped output — and the edit is the minimal one, stated in the report. The sidecar is rewritten last. A directory holding a `codegen.lock` with no sidecar naming this method is not this method's (step 4): choose another directory name and say why, or let the user overrule by naming the method that directory is for, as step 4 allows. The exception is a harness-owned layout, which keeps no sidecar by design — see that section. ## A project that owns a codegen harness @@ -186,7 +186,7 @@ A project made from `pipelex-starter-js` or `pipelex-starter-python`, or one tha - **skip steps 5, 7, 8 and 10** — the exclusions, dependencies, sidecar and gate already exist — and verify with the project's own aggregate gate (`make check` / `make all`); - **refresh is the project's `codegen` script**, and you say so instead of calling `mthds_codegen`. -The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. +The harness owns the layout and the check; its generator is preferred, not mandatory. When it cannot run — the Python starter's `make codegen` shells out to a `pipelex` CLI the starter does not depend on — call `mthds_codegen` with `output_dir` set to **the harness's own layout** (`/generated//`), which is byte-identical there (same engine, same stamps, same lock, and that starter keeps no sidecar), and say the project's `make codegen` is the refresh once its prerequisite is met. Never a second layout beside the first. **This is the one destination step 4's sidecar rule does not govern**: the harness owns the directory and deliberately keeps no `sources.json`, so on every run after the first it holds a lock with no sidecar — which would otherwise read as another generation's. The harness branch is entered at step 1, before that rule applies; write into the layout again rather than inventing a second directory name, which is the thing this whole section exists to prevent. ## When something goes wrong @@ -204,9 +204,10 @@ The harness owns the layout and the check; its generator is preferred, not manda | `runtime`, `retryable: true` after a partial write | call again once with the same `output_dir`; then report what landed | | success with `orphans[]` non-empty | report by name, never delete; `orphans_truncated: true` → say detection was partial | | success with `is_current: false` | report `drifts[]` verbatim and stop; do not commit a tree the check rejects | -| `main_pipe` absent on a by-ref / by-id source | STOP: the verdict carries no signature — the workshop predates it (refresh `@pipelex/mcp` and retry) or the method settles no entry pipe; never guess the output concept | +| `main_pipe` absent on a by-ref / by-id source | first look for the one-line signature in the verdict's **text summary** — it carries the same thing and reaches a host that does not surface structured content. Genuinely absent → STOP: all three causes of step 3, and the remedy differs by cause — the workshop predates it (refresh `@pipelex/mcp` and retry), the method settles no entry pipe, or the entry pipe's contract did not come back whole, which comes from the runner behind `/v1/validate` and no refresh fixes. Never guess the output concept | | the project's type check fails after the call site is written | your code — fix and re-run; a failure inside the generated tree is reported, not patched | -| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | +| `TS2835` on a relative import, in **your own** call-site module | yours to fix, per step 11: on a project whose resolution demands extensions, the module's own imports of the generated tree and its client helper take `.js` (`from "../generated//binder.js"`). Do not route this to the row below — that one is about the stamped tree only | +| `TS2835` in the generated `binder.ts`: a relative import needs a file extension | a known defect of the ts-zod emitter, which writes `from "./types"`. On this project shape the integration does not merely fail a check — the compiled code cannot load the module at all (`ERR_MODULE_NOT_FOUND`), so the report says the integration cannot execute until the emitter is fixed or the user changes `moduleResolution`, not that a check is red. It bites a plain Node ESM project (`"type": "module"` with `moduleResolution` `nodenext` or `node16`) and not a bundler one. Report it — the fix is upstream; never patch the stamped file (a regeneration loses the patch and the stamp is hashed) and never drop the tree from the type checker. Changing the project's `moduleResolution` is the user's call to make, not yours | | `mthds_list_methods` absent | integrate by id, address or files; never stop for it | ## Reference From a9c819d4f4a794d17ada27df53157575b4620d96 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 00:44:06 +0200 Subject: [PATCH 16/21] Close the write half of the key rule, and the scaffold's destructive edges MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `8356f2c` fixed looking a key up and left putting one down. It sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter — and a tool call's parameters are the transcript, so that is the one form of write that cannot be made safe. The value now moves only through a shell that expands the variable itself. Reading the env file back afterwards is refused by name, because that is the reflex after writing and the first move when a later step fails, and the rule had enumerated shell constructions only. Confirming the write uses a file-side presence test that reveals nothing, and the report says the value was taken from the environment and not validated — a placeholder passes `-n` and fails the first run, and this skill never calls the API, so it cannot tell. The row carrying this guidance had an unescaped `|` inside a code span and rendered as four broken cells. Reaching a runtime behind a version manager means resolving it to a path and carrying that into every later command. Shell state does not survive a command here, and the profile that would have had the runtime is the one that did not — so sourcing `nvm.sh` bought nothing, the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap failed on `node: command not found`, which is worse than the honest stop. Three things the clause does not license: a shim that resolves and then answers nothing is not a runtime, the starter's floor still applies to a version a manager happens to hold, and `volta` and `mise` install a version they lack, which is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch — the report used to promise the opposite unconditionally. Three destructive edges. The `rm -rf /.git` was not chained to the clone that creates ``, so on a path that already held a repository it destroyed the user's history. The `.env` copy had no `-n`, and the fresh-clone shortcut enters that step in a directory the user was already working in, where it would overwrite a key they had filled. And the non-empty refusal read as absolute while the fresh-clone shortcut six lines below enters branch A in a directory non-empty by definition; it is scoped to a directory you are creating a project in, which removes the contradiction without changing what it refuses — whether `.git`-only and `.DS_Store`-only should still be refused is a judgment call, filed rather than taken. Also: `npm create next-app` without the `--` separator, so npm ate the flags and the advertised non-interactive command prompted; a `.gitignore` read before staging, since `npm init -y` writes none; an initializer that commits as well as `git init`s has already made the pristine commit, as `create-next-app` does; and the decisions log dated the model-invocable change to a day with no session behind it, against the commit that removed the flag. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 3 + docs/decisions.md | 2 +- .../skills/pipelex-scaffold/SKILL.md | 32 +++++++--- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 32 +++++++--- pipelex/skills/pipelex-scaffold/SKILL.md | 32 +++++++--- templates/skills/pipelex-scaffold/SKILL.md.j2 | 32 +++++++--- tests/unit/test_pipelex_integrate_skill.py | 62 +++++++++++++++++++ tests/unit/test_pipelex_scaffold_skill.py | 29 +++++++++ 8 files changed, 187 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d1f687..d4352e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,9 @@ ### Fixed +- **The code the two new skills emit now compiles and runs the cases it claimed.** A review pass over the emitted artifacts — as opposed to the skill prose, which the dogfood matrix had exercised — found that none of them had ever been compiled or executed. The TypeScript call site declared its inputs as an `interface`, which TypeScript gives no implicit index signature and so cannot assign to the SDK's `inputs: Record`: it failed `TS2322` on every resolution, bundler included. Both call sites read only `main.mthds`, so any multi-file bundle produced types that were correct and a run that could not load what they were projected from; they now submit every `.mthds` file of the bundle, because a bundle is one closure. The verdict's `pipe_ref` is namespaced and the run route takes the bare code, and nothing said to strip the domain — a confusion that type-checks, passes the offline gate, and surfaces only on a real run. A list-valued output is narrowed according to its `multiplicity` instead of being parsed as a single concept. The call site's own relative imports are named as the agent's to extend with `.js` where the project's resolution demands it, rather than being misrouted to the emitter defect that never applies to them. The scaffold's `uv add` recipes ran in whatever directory the agent stood in, which from a parent is the user's own project and lockfile rather than the one just created; the minimal TypeScript recipe would have committed `node_modules/` into the commit meant to be a readable baseline; `npm create next-app` was written without the `--` separator, so npm ate the flags and the advertised non-interactive command prompted; the `.env` copy had no `-n` and would overwrite a key the user had already filled on the fresh-clone path; and the `rm -rf /.git` was not chained to the clone that creates ``. The offline drift gate reported `current` for a sidecar whose `sources` was present but not an object — checking nothing, printing nothing, exiting 0 — and for an artifact given a byte-order mark, which the default decoder strips before hashing; both now fail closed, no branch is both silent and green, and the verdict's explanation survives a pipe. +- **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. +- **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. - **The file factory no longer touches the user's own project.** Every `uv run` line in the skill and its references now passes `--no-project`. Without it `uv run` walks up from the working directory, finds the nearest project, and *syncs* it — so rendering a test PDF inside a checkout created a `.venv/` and wrote a `uv.lock` the user never asked for, in a repository the skill has no business modifying. `--no-project` resolves the ephemeral `--with` packages against nothing at all, which is what the recipes always meant; a test asserts every shipped runner line carries the flag. - **The PDF reference no longer offers a public URL that has stopped serving a PDF.** The documented last resort answered with an HTML error page rather than a document, so an agent following the recipe would have handed a method a file that is not a PDF at the exact moment it had already failed to render one. There is now no last resort for any format: the skill asks the user for their own file, and says plainly that it will neither fabricate one nor substitute a document the brief did not ask for. diff --git a/docs/decisions.md b/docs/decisions.md index 4e55e90..2b3f1dc 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -137,7 +137,7 @@ Skill adoption follows the target, not the tool surface: **only `pipelex-inputs` The CLI-era plugin had `mthds-edit` next to `mthds-build`. The port splits that ground along the **contract line** instead of recreating a monolithic edit skill: - **`pipelex-edit`** is the modification entry point for contract-preserving work — it owns the natural-language triggers ("change this pipe", "rename this concept"). It handles contract-preserving edits itself (prompts, descriptions, model refs, operator settings, mechanical renames) under a baseline-verdict discipline: whole-bundle `mthds_validate` before and after, never edit on a broken baseline, inputs-refresh check when a rename touches the client-facing template. -- **`pipelex-design`** owns structural and contract changes via its "Editing an existing method" re-entry section. Re-entry is complexity-adaptive: validate the baseline first, then edit the smallest coherent region directly when its complete shallow graph and propagated contracts can be understood together; use same-contract signatures and re-refinement for nested, uncertain, cross-module, or staged changes. Contract changes still propagate through parent wiring, concept reshapes still include every field-reading consumer, and organization runs only when re-entry leaves a construction-shaped layout. **The skill is model-invocable (changed 2026-08-29; it shipped `disable-model-invocation: true` through 0.5.0).** The original reasoning — a design run is a commitment the user opts into explicitly — turned the routing into a dead end: `pipelex-edit` classified a structural change correctly and could then only *tell* the user to type `/pipelex-design`, discarding the baseline verdict and classification it had just produced and costing a turn for a handoff the user had already asked for. The consent gate that matters is inside the skill, not on its invocation — it announces the captured contract in one line before writing anything, and infers the construction mode rather than asking. So `pipelex-edit` now names the affected pipes and invokes `/pipelex-design` directly, and the design skill's description carries natural-language triggers ("design a method", "create a pipeline", "add a step", "rewire this pipeline") so the model can reach it without a slash command. Removing the flag alone would have been inert: with a purely descriptive description, nothing would ever have triggered it. +- **`pipelex-design`** owns structural and contract changes via its "Editing an existing method" re-entry section. Re-entry is complexity-adaptive: validate the baseline first, then edit the smallest coherent region directly when its complete shallow graph and propagated contracts can be understood together; use same-contract signatures and re-refinement for nested, uncertain, cross-module, or staged changes. Contract changes still propagate through parent wiring, concept reshapes still include every field-reading consumer, and organization runs only when re-entry leaves a construction-shaped layout. **The skill is model-invocable (changed 2026-09-06; it shipped `disable-model-invocation: true` through 0.5.0).** The original reasoning — a design run is a commitment the user opts into explicitly — turned the routing into a dead end: `pipelex-edit` classified a structural change correctly and could then only *tell* the user to type `/pipelex-design`, discarding the baseline verdict and classification it had just produced and costing a turn for a handoff the user had already asked for. The consent gate that matters is inside the skill, not on its invocation — it announces the captured contract in one line before writing anything, and infers the construction mode rather than asking. So `pipelex-edit` now names the affected pipes and invokes `/pipelex-design` directly, and the design skill's description carries natural-language triggers ("design a method", "create a pipeline", "add a step", "rewire this pipeline") so the model can reach it without a slash command. Removing the flag alone would have been inert: with a purely descriptive description, nothing would ever have triggered it. - **Why the split:** the routing surface and the methodology have different homes. Both halves auto-trigger from natural phrases, so the split is no longer about invocability — it is that the cheap contract-preserving path must not drag the whole design methodology behind it, and the propagating-change discipline (contract identity, concept shapes, backlog draining) must live in exactly one skill or it drifts. The hook is not a substitute for either: its semantic-validation stage is fail-open (skipped without `PIPELEX_API_KEY`), so `pipelex-edit` always takes the whole-bundle MCP verdict as the authoritative check. ## MCP tool vs skill naming convention (2026-07-16) diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 1f8243d..a37ab77 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -40,6 +40,10 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. +**Activating it means resolving it to an absolute path, not sourcing a shell.** Your shell state does not survive from one command to the next — each one starts again from the user's profile, which is the profile that did not have the runtime — so `. nvm.sh` or `eval "$(fnm env)"` in one call buys nothing in the next. Resolve the binary once (`ls "$NVM_DIR"/versions/node/*/bin/node`, `volta which node`, `mise which node`, `asdf which node`, `fnm exec --using= -- which node`), keep that directory, and prefix **every** later command with it — `PATH=":$PATH" …` — the clone's bootstrap and its `make all` / `make agent-check` included, because those are separate commands too. Verify the runtime answers under that prefix **before** step 2, so a machine you cannot actually reach stops while nothing has been created; discovering it at step 4 has already spent the pristine commit. + +Three things this clause does not license. **A shim is not a runtime**: `asdf` and `mise` put a `node` on the `PATH` that exists and then fails with "no version set", so the test is that `node --version` *answers*, not that the binary resolves — and that case is a stop, not a manager to activate. **The floor still applies**: a manager holding Node 18 does not satisfy the starter's `engines` floor, and "a runtime the machine already has" never means a version below it. And **`volta` and `mise` install on first use** — `volta run`, `mise x` and `mise use` will fetch a version they do not have — which is the toolchain install this step forbids: use only a version the manager already holds, and stop rather than let it download one. Note too that `nvm`, `fnm` and `volta` manage Node alone and can never supply `uv`. + - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). - **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. @@ -49,12 +53,14 @@ Check before touching anything, and **stop** on a missing piece with the exact t **Local, the default.** Clone shallow, read the template's identity, then detach from it: ```bash -git clone --depth 1 https://github.com/Pipelex/.git +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf /.git && git -C init -b main ``` +The `|| exit` on the clone is not decoration: the line below it deletes a `.git` directory, and if the clone never ran — a network failure, or `` already existing — that `rm -rf` finds whatever `.git` is actually at that path. On a directory the skill just created it destroys nothing; on a repository of the user's it destroys their history irrecoverably. Run the destructive line only behind a clone that succeeded, and never type it on a path you have not just created. + The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. **GitHub, on request.** When the user asked for a repository on GitHub: @@ -84,11 +90,17 @@ Feed it what the conversation already holds — the project name, title, descrip ### Step 5: The env file ```bash -cp /.env.example /.env.local # JS: Next.js reads .env.local -cp /.env.example /.env # Python: python-dotenv reads .env +cp -n /.env.example /.env.local # JS: Next.js reads .env.local +cp -n /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +`-n` because this is the one step that can destroy something of the user's. On the fresh-clone shortcut the directory is one they were already working in, and a plain `cp` would overwrite an `.env.local` they had filled with their own key — the skill's whole posture is that nothing of the user's is ever cleared, and an env file is the most expensive thing in the tree to lose. An existing env file is left exactly as it is; read whether it already carries a key with the file-side test below, and say in the report that you kept theirs. + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. + +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. + +A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -102,19 +114,21 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: ```bash git -C add -A && git -C commit -m "Scaffold project" ``` +**An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. + ### Step 4: The env file Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: @@ -135,7 +149,7 @@ Say, in this order: what was created and where; which template or initializer it Two lines are easy to forget and matter: - **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd `, then starting Codex there, is how they arrive. -- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. One exception, and it is the version-manager machine of step 1: the workshop is spawned with `npx` on the **harness's** own `PATH`, which no activation of yours reaches, so a `node` only reachable through `nvm` or `fnm` means no workshop at all. Say so there instead — the hand-off needs the harness restarted from a shell where the runtime is active. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. ## When something goes wrong @@ -149,7 +163,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | -| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index f025c32..46b0be2 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -40,6 +40,10 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. +**Activating it means resolving it to an absolute path, not sourcing a shell.** Your shell state does not survive from one command to the next — each one starts again from the user's profile, which is the profile that did not have the runtime — so `. nvm.sh` or `eval "$(fnm env)"` in one call buys nothing in the next. Resolve the binary once (`ls "$NVM_DIR"/versions/node/*/bin/node`, `volta which node`, `mise which node`, `asdf which node`, `fnm exec --using= -- which node`), keep that directory, and prefix **every** later command with it — `PATH=":$PATH" …` — the clone's bootstrap and its `make all` / `make agent-check` included, because those are separate commands too. Verify the runtime answers under that prefix **before** step 2, so a machine you cannot actually reach stops while nothing has been created; discovering it at step 4 has already spent the pristine commit. + +Three things this clause does not license. **A shim is not a runtime**: `asdf` and `mise` put a `node` on the `PATH` that exists and then fails with "no version set", so the test is that `node --version` *answers*, not that the binary resolves — and that case is a stop, not a manager to activate. **The floor still applies**: a manager holding Node 18 does not satisfy the starter's `engines` floor, and "a runtime the machine already has" never means a version below it. And **`volta` and `mise` install on first use** — `volta run`, `mise x` and `mise use` will fetch a version they do not have — which is the toolchain install this step forbids: use only a version the manager already holds, and stop rather than let it download one. Note too that `nvm`, `fnm` and `volta` manage Node alone and can never supply `uv`. + - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). - **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. @@ -49,12 +53,14 @@ Check before touching anything, and **stop** on a missing piece with the exact t **Local, the default.** Clone shallow, read the template's identity, then detach from it: ```bash -git clone --depth 1 https://github.com/Pipelex/.git +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf /.git && git -C init -b main ``` +The `|| exit` on the clone is not decoration: the line below it deletes a `.git` directory, and if the clone never ran — a network failure, or `` already existing — that `rm -rf` finds whatever `.git` is actually at that path. On a directory the skill just created it destroys nothing; on a repository of the user's it destroys their history irrecoverably. Run the destructive line only behind a clone that succeeded, and never type it on a path you have not just created. + The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. **GitHub, on request.** When the user asked for a repository on GitHub: @@ -84,11 +90,17 @@ Feed it what the conversation already holds — the project name, title, descrip ### Step 5: The env file ```bash -cp /.env.example /.env.local # JS: Next.js reads .env.local -cp /.env.example /.env # Python: python-dotenv reads .env +cp -n /.env.example /.env.local # JS: Next.js reads .env.local +cp -n /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +`-n` because this is the one step that can destroy something of the user's. On the fresh-clone shortcut the directory is one they were already working in, and a plain `cp` would overwrite an `.env.local` they had filled with their own key — the skill's whole posture is that nothing of the user's is ever cleared, and an env file is the most expensive thing in the tree to lose. An existing env file is left exactly as it is; read whether it already carries a key with the file-side test below, and say in the report that you kept theirs. + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. + +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. + +A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -102,19 +114,21 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: ```bash git -C add -A && git -C commit -m "Scaffold project" ``` +**An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. + ### Step 4: The env file Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: @@ -135,7 +149,7 @@ Say, in this order: what was created and where; which template or initializer it Two lines are easy to forget and matter: - **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd `, then starting Mistral Vibe there, is how they arrive. -- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. One exception, and it is the version-manager machine of step 1: the workshop is spawned with `npx` on the **harness's** own `PATH`, which no activation of yours reaches, so a `node` only reachable through `nvm` or `fnm` means no workshop at all. Say so there instead — the hand-off needs the harness restarted from a shell where the runtime is active. Hand the method to it by opening `../pipelex-integrate/SKILL.md` and following it; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. ## When something goes wrong @@ -149,7 +163,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | -| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index 4341d83..97373c9 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -30,7 +30,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -47,6 +47,10 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. +**Activating it means resolving it to an absolute path, not sourcing a shell.** Your shell state does not survive from one command to the next — each one starts again from the user's profile, which is the profile that did not have the runtime — so `. nvm.sh` or `eval "$(fnm env)"` in one call buys nothing in the next. Resolve the binary once (`ls "$NVM_DIR"/versions/node/*/bin/node`, `volta which node`, `mise which node`, `asdf which node`, `fnm exec --using= -- which node`), keep that directory, and prefix **every** later command with it — `PATH=":$PATH" …` — the clone's bootstrap and its `make all` / `make agent-check` included, because those are separate commands too. Verify the runtime answers under that prefix **before** step 2, so a machine you cannot actually reach stops while nothing has been created; discovering it at step 4 has already spent the pristine commit. + +Three things this clause does not license. **A shim is not a runtime**: `asdf` and `mise` put a `node` on the `PATH` that exists and then fails with "no version set", so the test is that `node --version` *answers*, not that the binary resolves — and that case is a stop, not a manager to activate. **The floor still applies**: a manager holding Node 18 does not satisfy the starter's `engines` floor, and "a runtime the machine already has" never means a version below it. And **`volta` and `mise` install on first use** — `volta run`, `mise x` and `mise use` will fetch a version they do not have — which is the toolchain install this step forbids: use only a version the manager already holds, and stop rather than let it download one. Note too that `nvm`, `fnm` and `volta` manage Node alone and can never supply `uv`. + - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). - **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. @@ -56,12 +60,14 @@ Check before touching anything, and **stop** on a missing piece with the exact t **Local, the default.** Clone shallow, read the template's identity, then detach from it: ```bash -git clone --depth 1 https://github.com/Pipelex/.git +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf /.git && git -C init -b main ``` +The `|| exit` on the clone is not decoration: the line below it deletes a `.git` directory, and if the clone never ran — a network failure, or `` already existing — that `rm -rf` finds whatever `.git` is actually at that path. On a directory the skill just created it destroys nothing; on a repository of the user's it destroys their history irrecoverably. Run the destructive line only behind a clone that succeeded, and never type it on a path you have not just created. + The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. **GitHub, on request.** When the user asked for a repository on GitHub: @@ -91,11 +97,17 @@ Feed it what the conversation already holds — the project name, title, descrip ### Step 5: The env file ```bash -cp /.env.example /.env.local # JS: Next.js reads .env.local -cp /.env.example /.env # Python: python-dotenv reads .env +cp -n /.env.example /.env.local # JS: Next.js reads .env.local +cp -n /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +`-n` because this is the one step that can destroy something of the user's. On the fresh-clone shortcut the directory is one they were already working in, and a plain `cp` would overwrite an `.env.local` they had filled with their own key — the skill's whole posture is that nothing of the user's is ever cleared, and an env file is the most expensive thing in the tree to lose. An existing env file is left exactly as it is; read whether it already carries a key with the file-side test below, and say in the report that you kept theirs. + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. + +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. + +A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -109,19 +121,21 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: ```bash git -C add -A && git -C commit -m "Scaffold project" ``` +**An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. + ### Step 4: The env file Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: @@ -142,7 +156,7 @@ Say, in this order: what was created and where; which template or initializer it Two lines are easy to forget and matter: - **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; `cd && claude` is how they arrive. -- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it with `/pipelex-integrate`; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. One exception, and it is the version-manager machine of step 1: the workshop is spawned with `npx` on the **harness's** own `PATH`, which no activation of yours reaches, so a `node` only reachable through `nvm` or `fnm` means no workshop at all. Say so there instead — the hand-off needs the harness restarted from a shell where the runtime is active. Hand the method to it with `/pipelex-integrate`; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. ## When something goes wrong @@ -156,7 +170,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | -| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index ba639aa..e6c77a2 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -40,6 +40,10 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win Check before touching anything, and **stop** on a missing piece with the exact thing missing and the starter README's own line about it. Never install a toolchain — but a runtime the machine already has and only the `PATH` is missing is not a missing piece: when `node` or `uv` is absent while a version manager on the machine carries one (`nvm`, `fnm`, `volta`, `asdf`, `mise`), activate it for this work and say in the report which one you used and that the user's own shell may not have it. Stop only when no usable runtime can be reached that way. +**Activating it means resolving it to an absolute path, not sourcing a shell.** Your shell state does not survive from one command to the next — each one starts again from the user's profile, which is the profile that did not have the runtime — so `. nvm.sh` or `eval "$(fnm env)"` in one call buys nothing in the next. Resolve the binary once (`ls "$NVM_DIR"/versions/node/*/bin/node`, `volta which node`, `mise which node`, `asdf which node`, `fnm exec --using= -- which node`), keep that directory, and prefix **every** later command with it — `PATH=":$PATH" …` — the clone's bootstrap and its `make all` / `make agent-check` included, because those are separate commands too. Verify the runtime answers under that prefix **before** step 2, so a machine you cannot actually reach stops while nothing has been created; discovering it at step 4 has already spent the pristine commit. + +Three things this clause does not license. **A shim is not a runtime**: `asdf` and `mise` put a `node` on the `PATH` that exists and then fails with "no version set", so the test is that `node --version` *answers*, not that the binary resolves — and that case is a stop, not a manager to activate. **The floor still applies**: a manager holding Node 18 does not satisfy the starter's `engines` floor, and "a runtime the machine already has" never means a version below it. And **`volta` and `mise` install on first use** — `volta run`, `mise x` and `mise use` will fetch a version they do not have — which is the toolchain install this step forbids: use only a version the manager already holds, and stop rather than let it download one. Note too that `nvm`, `fnm` and `volta` manage Node alone and can never supply `uv`. + - **JavaScript**: Node at or above the floor the starter's `package.json` `engines` field names (`node --version`; 22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and `npm`. - **Python**: `uv` on the PATH (the starter installs and locks with it) and a Python inside the starter's `requires-python` range that `uv python find` can see (3.11 to 3.14 at writing). - **Both**: `git`. The GitHub form also needs `gh` authenticated — `gh auth status`. @@ -49,12 +53,14 @@ Check before touching anything, and **stop** on a missing piece with the exact t **Local, the default.** Clone shallow, read the template's identity, then detach from it: ```bash -git clone --depth 1 https://github.com/Pipelex/.git +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf /.git && git -C init -b main ``` +The `|| exit` on the clone is not decoration: the line below it deletes a `.git` directory, and if the clone never ran — a network failure, or `` already existing — that `rm -rf` finds whatever `.git` is actually at that path. On a directory the skill just created it destroys nothing; on a repository of the user's it destroys their history irrecoverably. Run the destructive line only behind a clone that succeeded, and never type it on a path you have not just created. + The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. **GitHub, on request.** When the user asked for a repository on GitHub: @@ -84,11 +90,17 @@ Feed it what the conversation already holds — the project name, title, descrip ### Step 5: The env file ```bash -cp /.env.example /.env.local # JS: Next.js reads .env.local -cp /.env.example /.env # Python: python-dotenv reads .env +cp -n /.env.example /.env.local # JS: Next.js reads .env.local +cp -n /.env.example /.env # Python: python-dotenv reads .env ``` -Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset` — and write it with a redirection or an in-place edit that never echoes the value; `env | grep PIPELEX`, `echo $PIPELEX_API_KEY` and a command substitution in a message all put the key in the transcript, which is not yours to spend. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. +`-n` because this is the one step that can destroy something of the user's. On the fresh-clone shortcut the directory is one they were already working in, and a plain `cp` would overwrite an `.env.local` they had filled with their own key — the skill's whole posture is that nothing of the user's is ever cleared, and an env file is the most expensive thing in the tree to lose. An existing env file is left exactly as it is; read whether it already carries a key with the file-side test below, and say in the report that you kept theirs. + +Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. + +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. + +A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. ### Step 6: Verify and hand off @@ -102,19 +114,21 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest --ts --app --src-dir --eslint --use-npm --yes`; `uv init --package ` then `uv add "fastapi[standard]"`; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. Then the one commit, for the same reason as branch A: +If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: ```bash git -C add -A && git -C commit -m "Scaffold project" ``` +**An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. + ### Step 4: The env file Write `.env.example` with the two lines the starters share, make sure `.env` is gitignored, and copy the example to `.env` under the same key rule as branch A: @@ -135,7 +149,7 @@ Say, in this order: what was created and where; which template or initializer it Two lines are easy to forget and matter: - **The project's own instructions and skills load in a session started inside it.** Its `CLAUDE.md` / `AGENTS.md` and its `release` and `bump-*` skills are not in the current session; {% if platform == "claude" %}`cd && claude`{% else %}`cd `, then starting {{ harness_name }} there,{% endif %} is how they arrive. -- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. Hand the method to it{% if platform == "claude" %} with `/pipelex-integrate`{% else %} by opening `../pipelex-integrate/SKILL.md` and following it{% endif %}; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. +- **`/pipelex-integrate` still works from here meanwhile**, because the Pipelex workshop writes anywhere under the directory the harness was launched in, and the new project sits there. One exception, and it is the version-manager machine of step 1: the workshop is spawned with `npx` on the **harness's** own `PATH`, which no activation of yours reaches, so a `node` only reachable through `nvm` or `fnm` means no workshop at all. Say so there instead — the hand-off needs the harness restarted from a shell where the runtime is active. Hand the method to it{% if platform == "claude" %} with `/pipelex-integrate`{% else %} by opening `../pipelex-integrate/SKILL.md` and following it{% endif %}; a bundle that lives elsewhere on disk is copied into the project by that skill. No method yet → `/pipelex-design` first. ## When something goes wrong @@ -149,7 +163,7 @@ Two lines are easy to forget and matter: | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | -| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env | grep PIPELEX`, never echo the value — a key in the transcript is a key to rotate | +| you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | | The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | ## Reference diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index 3d6fa04..f32c25a 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -33,6 +33,19 @@ class TestPipelexIntegrateSkill: "Do this **before** step 6", "**Check containment before the call rather than waiting for an error**", "never write the tree into the workshop's own directory and move it across afterwards", + # The run route takes the bare pipe code; the verdict and the sidecar carry the namespaced ref. + "**`pipe_ref` and the run's `pipe_code` are not the same string**", + # A bundle is one closure: the call site submits every file, not just main.mthds. + "It loads **every `.mthds` file of the committed bundle** at call time", + # The containment reading is lexically wrong in both directions without resolving. + "**Resolve both sides before comparing them**", + "**Both inputs are known as soon as step 1 names the project, so do this reading there**", + # A lock with no sidecar is also what this method's own interrupted run leaves. + "**One lock-without-sidecar is this method's own, and relocating is the wrong answer for it**", + # The harness keeps no sidecar by design, so step 4's rule cannot govern its layout. + "**This is the one destination step 4's sidecar rule does not govern**", + # A list output arrives as an array and a single-object parser rejects it. + "**Narrow according to the output's `multiplicity`, which step 3 recorded**", ) @property @@ -57,6 +70,10 @@ def test_signature_comes_from_the_verdict_and_the_heuristic_is_absent(self) -> N assert "`main_pipe.pipe_ref` is the pipe a run with no pipe selector executes" in body assert "the method settles **no entry pipe**" in body assert "the workshop **predates the signature**" in body + # The third cause was stated in step 3 but dropped from the failure table, whose remedy differs by cause. + assert "the entry pipe's contract did not come back whole, which comes from the runner" in body + # The same signature rides the text summary, so a host that drops structured content is not a dead end. + assert "the **same fact on a second channel, not a fourth cause**" in body def test_the_wire_null_helper_is_never_installed(self) -> None: body = self.integrate @@ -81,6 +98,51 @@ def test_failure_posture_pins_the_403_and_the_orphans(self) -> None: assert 'a known defect of the ts-zod emitter, which writes `from "./types"`' in body assert "Changing the project's `moduleResolution` is the user's call to make, not yours" in body + def test_the_call_sites_submit_the_whole_bundle_not_just_main(self) -> None: + """Generation takes every `.mthds` file of the bundle, so the run must too. + + A call site that submits `main.mthds` alone type-checks, passes the offline gate, and then + fails at load time on exactly the multi-file methods codegen handled correctly. + """ + typescript = (self.REFERENCES_DIR / "typescript.md").read_text(encoding="utf-8") + assert "mthds_contents: await readBundle()," in typescript + # An interface has no implicit index signature, so it is not assignable to Record. + assert "export type SummarizePdfInputs = {" in typescript + assert "export interface SummarizePdfInputs" not in typescript + assert "**`SummarizePdfInputs` is a `type`, not an `interface`, and that is load-bearing.**" in typescript + # A script merely named `codegen` is not evidence of a Pipelex harness. + assert "read the script before believing it" in typescript + assert 'name.endsWith(".mthds")' in typescript + assert "mthds_contents: [bundle]" not in typescript + # Its own relative imports need extensions on the resolution that meets the emitter defect. + assert "**The three relative imports above are extensionless, which is correct only on a bundler resolution.**" in typescript + + python = (self.REFERENCES_DIR / "python.md").read_text(encoding="utf-8") + assert "mthds_contents=_read_bundle()," in python + assert 'sorted(BUNDLE_DIR.rglob("*.mthds"))' in python + assert "BUNDLE_PATH" not in python + + def test_the_gate_fails_closed_on_a_malformed_sources_and_does_not_truncate(self) -> None: + """The offline gate is the one executable artifact here, so two properties are pinned. + + A sidecar whose `sources` is present but not an object must fail the way an unreadable one + does: coerced to `{}` it would check nothing, print nothing and exit 0 — the single input + that is both silent and green. And the verdict's explanation must survive a pipe, which + `process.exit` does not guarantee. + """ + gate = (self.REFERENCES_DIR / "codegen-check.mjs").read_text(encoding="utf-8") + # No `??` here: it would turn an explicit null into the legitimate absent case. + assert "const sources = sidecar?.sources;" in gate + assert "?? {}" not in gate + assert 'if (typeof sources !== "object" || sources === null || Array.isArray(sources)) {' in gate + assert "is not an object, so staleness cannot be ruled out" in gate + # No branch is both silent and green: absent and empty both announce themselves. + assert "records no sources — a by-ref or by-id integration; source staleness does not apply" in gate + assert "process.exitCode = await main(process.argv);" in gate + # The default decoder strips a BOM, which would hash a BOM'd artifact as current. + assert 'new TextDecoder("utf-8", { fatal: true, ignoreBOM: true })' in gate + assert "process.exit(await main" not in gate + def test_method_id_warns_and_refresh_leaves_the_call_site_alone(self) -> None: body = self.integrate assert "The catalog is unversioned" in body diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 8cf6b63..a37274c 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -36,6 +36,25 @@ class TestPipelexScaffoldSkill: "Add **no** SDK dependency and create **no** empty `methods/` directory", "Nothing beyond what the initializer writes is authored by this skill", "a runtime the machine already has and only the `PATH` is missing is not a missing piece", + # A sourced shell does not survive the next command, so the runtime is resolved to a path. + "**Activating it means resolving it to an absolute path, not sourcing a shell.**", + "**A shim is not a runtime**", + # A file-editing tool needs the literal value in its parameters, which is the transcript. + "**The value moves only through a shell that expands the variable itself, and never through you.**", + "**no reading the env file back**", + "and not validated", + # The destructive line runs only behind a clone that succeeded. + "git clone --depth 1 https://github.com/Pipelex/.git || exit", + # cp -n: an env file the user already filled is the most expensive thing in the tree to lose. + "cp -n /.env.example /.env.local", + # npm eats the flags without the separator and create-next-app then prompts. + "npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes", + # uv add resolves the project from its working directory, which from the parent is the user's. + "**from inside ``**", + # An initializer that commits has already made the pristine commit. + "**An initializer that commits as well as `git init`s has already made this commit.**", + # npm init -y and tsc --init write no .gitignore, so git add -A would commit node_modules. + "**Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**", ) @property @@ -75,6 +94,16 @@ def test_references_describe_both_starters_and_the_initializers(self) -> None: assert "uv init --package " in initializers assert "npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes" in initializers assert "No SDK dependency" in initializers + # Every `uv add` runs inside the new project: from the parent it writes to the user's own. + assert "**Every `uv add` above runs inside ``, and the parentheses are why.**" in initializers + for recipe in ( + '(cd && uv add "fastapi[standard]")', + "(cd && uv add typer)", + "(cd && uv add django && uv run django-admin startproject config .)", + ): + assert recipe in initializers, f"uv add not scoped to the project: {recipe}" + # The minimal TS default is the very resolution the emitter defect breaks. + assert "is exactly the shape that meets the ts-zod emitter's extensionless-import defect" in initializers @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) def test_every_platform_renders_the_skill_and_its_references(self, target_name: str) -> None: From 41c0b7ac8f348377eae7e57d5b50bc86e7bf31ef Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 01:13:05 +0200 Subject: [PATCH 17/21] Run the scaffold's recipes into the integrate call site, and fix what met there MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 1 compiled the TypeScript snippet; this round ran the initializers and then integrated into the project they actually produce, which is where the two skills meet and where every defect below lived. `tsc --init` writes `"types": []` as an active key, switching off the `@types/node` the recipe installs the line before, so the call site failed TS2591 on `node:fs/promises`, `node:path` and `process`. Every `uv init` now passes `--no-workspace`: inside a directory that already holds a `pyproject.toml`, a bare one appends `[tool.uv.workspace]` to the user's own file and leaves the lock at the parent root. The Vite and Express follow-on `npm install` lines are parenthesised for the same reason the `uv add` lines were, and npm is worse — it finds the parent and exits 0. Branch B's pristine commit could reach an enclosing repository, because `git -C ` scopes nothing and `uv init` only `git init`s a standalone project; the step now tests `rev-parse --show-toplevel`, both commits carry `-- .`, and the read-back names paths. The clone guard was missing from `references/starters.md`, the file the skill names as carrying every command. The offline gate still reported `current` for a sidecar that was valid JSON and not an object, because `sidecar?.sources` does to the sidecar what `??` would have done to `sources`. Also: the helper budget said two and listed one, licensing the wire-output helper the campaign struck; and the reference-copy test compared a fresh `copytree` against its own source, so it could not see a stale committed copy. Each fix is pinned by a test that failed before it, the gate's by one that runs the script rather than reading it. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 3 + docs/decisions.md | 2 +- .../skills/pipelex-integrate/SKILL.md | 4 +- .../references/codegen-check.mjs | 11 +- .../pipelex-integrate/references/python.md | 9 +- .../skills/pipelex-scaffold/SKILL.md | 12 +- .../references/initializers.md | 22 ++-- .../pipelex-scaffold/references/starters.md | 8 +- .../skills/pipelex-integrate/SKILL.md | 4 +- .../references/codegen-check.mjs | 11 +- .../pipelex-integrate/references/python.md | 9 +- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 12 +- .../references/initializers.md | 22 ++-- .../pipelex-scaffold/references/starters.md | 8 +- pipelex/skills/pipelex-integrate/SKILL.md | 4 +- .../references/codegen-check.mjs | 11 +- .../pipelex-integrate/references/python.md | 9 +- pipelex/skills/pipelex-scaffold/SKILL.md | 12 +- .../references/initializers.md | 22 ++-- .../pipelex-scaffold/references/starters.md | 8 +- .../references/codegen-check.mjs | 11 +- skills/pipelex-integrate/references/python.md | 9 +- .../references/initializers.md | 22 ++-- .../pipelex-scaffold/references/starters.md | 8 +- .../skills/pipelex-integrate/SKILL.md.j2 | 4 +- templates/skills/pipelex-scaffold/SKILL.md.j2 | 12 +- tests/unit/test_pipelex_integrate_skill.py | 119 ++++++++++++++++-- tests/unit/test_pipelex_scaffold_skill.py | 70 +++++++++-- 28 files changed, 362 insertions(+), 96 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d4352e6..2c3b9b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,9 @@ ### Fixed - **The code the two new skills emit now compiles and runs the cases it claimed.** A review pass over the emitted artifacts — as opposed to the skill prose, which the dogfood matrix had exercised — found that none of them had ever been compiled or executed. The TypeScript call site declared its inputs as an `interface`, which TypeScript gives no implicit index signature and so cannot assign to the SDK's `inputs: Record`: it failed `TS2322` on every resolution, bundler included. Both call sites read only `main.mthds`, so any multi-file bundle produced types that were correct and a run that could not load what they were projected from; they now submit every `.mthds` file of the bundle, because a bundle is one closure. The verdict's `pipe_ref` is namespaced and the run route takes the bare code, and nothing said to strip the domain — a confusion that type-checks, passes the offline gate, and surfaces only on a real run. A list-valued output is narrowed according to its `multiplicity` instead of being parsed as a single concept. The call site's own relative imports are named as the agent's to extend with `.js` where the project's resolution demands it, rather than being misrouted to the emitter defect that never applies to them. The scaffold's `uv add` recipes ran in whatever directory the agent stood in, which from a parent is the user's own project and lockfile rather than the one just created; the minimal TypeScript recipe would have committed `node_modules/` into the commit meant to be a readable baseline; `npm create next-app` was written without the `--` separator, so npm ate the flags and the advertised non-interactive command prompted; the `.env` copy had no `-n` and would overwrite a key the user had already filled on the fresh-clone path; and the `rm -rf /.git` was not chained to the clone that creates ``. The offline drift gate reported `current` for a sidecar whose `sources` was present but not an object — checking nothing, printing nothing, exiting 0 — and for an artifact given a byte-order mark, which the default decoder strips before hashing; both now fail closed, no branch is both silent and green, and the verdict's explanation survives a pipe. +- **Running the emitted artifacts a second time found the defects the first pass's own fixes left behind.** Where the previous round compiled the TypeScript call site, this one ran the scaffold's initializer recipes and then integrated into the project they actually produce, which is where the two halves meet. `tsc --init` writes `"types": []` as an active key — switching off the `@types/node` the recipe installed on the line before — so the call site failed `TS2591` on `node:fs/promises`, on `node:path` and on `process`, with cascading implicit-any on the bundle reader; nothing in the errors points at the tsconfig, so the recipe now sets `"types": ["node"]` itself and says which TypeScript major it got, because the `tsc --init` template moved with the major. Every `uv init` now carries `--no-workspace`: run inside a directory that already holds a `pyproject.toml`, a bare one does not create a standalone project at all but appends a `[tool.uv.workspace]` table to **the user's own file** and leaves the lock at the parent root, so the new project had no lockfile and did not resolve alone — the same hazard the `uv add` parentheses were added for, one command earlier, and the parentheses could not see it. The follow-on `npm install` lines in the Vite and Express recipes are parenthesised for that reason too, and npm is the worse of the two: where `uv add` fails outright when it finds no project nearby, npm finds the parent, writes the dependency into the user's `package.json`, puts `node_modules/` in the user's tree, and exits 0. Branch B's pristine commit could reach an enclosing repository: `git -C ` sets git's working directory and scopes nothing, so with no `.git` of its own `` is governed by the user's repo and a pathspec-less `add -A` staged that whole worktree — the user's unrelated files committed under this skill's message. The step now tests `git -C rev-parse --show-toplevel` instead of trusting a list of initializers that `git init` (`uv init` is on that list only when it creates a standalone project), both commits carry a `-- .` pathspec, and the read-back names paths rather than a count that cannot tell a correct scaffold from a swept-up worktree. The clone guard the last round added to the skill was missing from `references/starters.md`, the file the skill names as carrying every command — so the one irrecoverable line shipped unguarded in the copy an agent reads it from. And the offline gate still reported `current`, exit 0, for a sidecar whose whole content was valid JSON but not an object (`null`, `[]`, a string, a number, a boolean): the previous round guarded `sources` against exactly this and reached it through `sidecar?.sources`, where optional chaining does to the sidecar precisely what `??` would have done to `sources`. All of these are now pinned by tests, the gate's by one that runs the script rather than reading it. +- **The helper budget said two and listed one.** `pipelex-integrate` announced "at most two shared helpers", named the client factory, and closed with "and nothing else" — residue of the wire-output helper that was struck for being lossy and is forbidden by name a few lines below. A stale count would be cosmetic; this one was written permission to create the single thing the campaign removed. It is one helper, in the skill, in both places it was stated, and in `docs/decisions.md`. +- **The reference-copy test asserted that `shutil` copies bytes.** It built a fresh tree into a temp directory and compared it against the source it had just been built from, so it could not observe the only failure that matters — a stale committed copy under `pipelex/`, `pipelex-codex/` or `pipelex-vibe/`, which is what a user installs. It now compares the committed copies themselves, and its three-target parametrization exercises three different trees instead of three identical calls. - **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. - **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. diff --git a/docs/decisions.md b/docs/decisions.md index 2b3f1dc..f48e01b 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -163,7 +163,7 @@ So file synthesis became **`pipelex-synthetic-inputs`**, a skill of its own. `pi Every skill before these acts on `.mthds` files; none touched the codebase that calls the method, so the plugin stopped helping at the moment a user was happy with a method. Two skills close that gap, decided with Louis on 2026-08-30 (the integrate design's ten boxes) and 2026-09-06 (the widening to greenfield projects, four rulings). The designs are `wip/pipelex-integrate/design.md` and `wip/pipelex-integrate/scaffold-design.md`; the tracker is `plan.md` beside them. -- **`pipelex-integrate` writes a complete typed call site, and stops there.** One module per method — an async function typed from the pipe's signature, running through the SDK's self-healing lifecycle call and narrowing `main_stuff` through the generated binder or model — plus at most two shared helpers; no tests, routes or UI. The name `pipelex-codegen` was rejected: tools are the contract and skills are the manual, named after user tasks, and codegen is one step of integrating. +- **`pipelex-integrate` writes a complete typed call site, and stops there.** One module per method — an async function typed from the pipe's signature, running through the SDK's self-healing lifecycle call and narrowing `main_stuff` through the generated binder or model — plus exactly one shared helper, the client factory; no tests, routes or UI. The name `pipelex-codegen` was rejected: tools are the contract and skills are the manual, named after user tasks, and codegen is one step of integrating. - **The write arm is the only arm.** Every `mthds_codegen` call passes `output_dir`; a refused or failed write is a refusal, never a fallback to writing the returned bytes from the conversation — a re-emitted artifact is one trailing newline from a broken stamp. Generated files are never edited, formatted or linted; the tooling exclusions go in before the tree exists; one directory per method; orphans are reported and never deleted. - **The signature comes from the verdict.** `mthds_validate`'s `structuredContent.main_pipe` types the call site for every selector. The by-elimination heuristic the first design carried for by-ref and by-id sources was made obsolete by `pipelex-mcp` before it shipped and was never written; an older workshop takes a fallback (the template for inputs, the bundle for the output), and a by-ref or by-id source without a signature stops rather than guessing. - **The sidecar `sources.json` is the skill's only state** — selector, target, pipe record, source hashes — so refresh mode re-derives nothing and a bundle edit is detectable; `pipelex-edit` and `pipelex-design` announce staleness from it. diff --git a/pipelex-codex/skills/pipelex-integrate/SKILL.md b/pipelex-codex/skills/pipelex-integrate/SKILL.md index accf990..b188db1 100644 --- a/pipelex-codex/skills/pipelex-integrate/SKILL.md +++ b/pipelex-codex/skills/pipelex-integrate/SKILL.md @@ -18,7 +18,7 @@ Take a method — a local `.mthds` bundle, a published address (`method_ref`), o Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. -**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus one shared helper, and a user who wants more says so. ## Requirements — the Pipelex MCP tools @@ -144,7 +144,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ | `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | | not required | optional parameter (`?`) | `T \| None = None` | -**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. +**Exactly one shared helper, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. There is no second one — the wire-output helper that would have been it was struck for being lossy, and writing one is forbidden below, so read "one" as the whole budget and not as a floor. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. ### Step 10: Wire the offline drift gate diff --git a/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs index 0e9fe5e..74e9156 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex-codex/skills/pipelex-integrate/references/codegen-check.mjs @@ -119,12 +119,21 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } + // The same hazard one level up, and it is why the line below is not `sidecar?.sources`. A file + // whose whole content is `null`, `[]`, `"x"`, `42` or `true` is valid JSON and is not an object, + // and optional chaining turns every one of them into `undefined` — the legitimate absent case — + // so the gate would print "a by-ref or by-id integration" and exit 0 over a sidecar that says + // nothing of the kind. On the sidecar, `?.` does exactly what `??` would do on `sources`. + if (typeof sidecar !== "object" || sidecar === null || Array.isArray(sidecar)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — not a JSON object, so staleness cannot be ruled out`] }; + } + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it // in the sidecar really is an array), and `null` is why this does not use `??`, which would // quietly turn an explicit null into the legitimate absent case. - const sources = sidecar?.sources; + const sources = sidecar.sources; if (sources === undefined) { return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; } diff --git a/pipelex-codex/skills/pipelex-integrate/references/python.md b/pipelex-codex/skills/pipelex-integrate/references/python.md index 8acdbde..f099f30 100644 --- a/pipelex-codex/skills/pipelex-integrate/references/python.md +++ b/pipelex-codex/skills/pipelex-integrate/references/python.md @@ -64,7 +64,14 @@ def _read_bundle() -> list[str]: """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that imports a sibling needs that sibling submitted with it, or the run fails to load what the generated models were projected from.""" - return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + contents = [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + if not contents: + # `rglob` on a missing or empty directory returns nothing and raises nothing, so without + # this the run goes out with `mthds_contents=[]` and fails server-side, naming the pipe + # rather than the path that is actually wrong. The TypeScript twin gets this for free: + # `readdir` throws ENOENT. + raise FileNotFoundError(f"no .mthds files under {BUNDLE_DIR}") + return contents async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index a37ab77..fc0edd1 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -76,7 +76,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -115,18 +115,22 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand - **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. -- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: +**Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. + +**Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" ``` +The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. + **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. ### Step 4: The env file diff --git a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md index 7602a15..e3631ee 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md @@ -8,14 +8,16 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where the import package lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | -| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | -| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | +| **Minimal (the default when no framework is named)** | `uv init --package --no-workspace ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app --no-workspace ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package --no-workspace && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package --no-workspace && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package --no-workspace && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | -`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package --no-workspace .` — it names the package after the directory. + +**`--no-workspace` is on every `uv init` above, and it is the same hazard the `uv add` parentheses below address, one command earlier.** Run inside a directory that already holds a `pyproject.toml` — the scaffold being made inside an existing project, which is the common case — a bare `uv init --package ` does not create a standalone project at all. It prints `Adding as member of workspace …`, appends a `[tool.uv.workspace]` table naming `` to **the user's own `pyproject.toml`**, and then the first `uv add` writes the lockfile at the *parent* root, so the new project has no `uv.lock` of its own and does not resolve standalone. Editing a file of the user's is exactly what this skill does not do, and a project that needs its parent to resolve is not the project the report says was handed over. `--no-workspace` leaves the parent untouched and gives `` its own lock. With no parent project it changes nothing, so it is safe to pass always, which is why it is not conditional. **Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. @@ -25,16 +27,20 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | -| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | -| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Express server | the minimal recipe above, then `(cd && npm install express && npm install --save-dev @types/express)` | no | `src/` | | Node library | the minimal recipe above | no | `src/` | | pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**Every follow-on `npm install` above is parenthesised for the same reason every `uv add` is, and npm is the worse of the two.** `npm install` resolves the project it writes to from its *working* directory upwards, so run from the parent it adds the dependency to the user's own `package.json` and puts `node_modules/` in the user's tree — and where `uv add` at least fails outright when it finds no project nearby, npm finds the parent and silently succeeds, leaving the new project with nothing installed and no error to read. The `cd ` inside the minimal recipe's own `&&` chain does not reach a follow-on issued as a separate command, because shell state does not survive from one command to the next. Use the subshell, or `npm install --prefix `. + **The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. +**And `tsc --init` writes `"types": []` as an active key, which switches `@types/node` off on the line after the recipe installed it.** That empty array is the current `tsc --init` template's own default (TypeScript 7 writes it, with `// "types": ["node"],` commented out three lines below), and it means no `@types` package is loaded at all — so the call site `/pipelex-integrate` writes next fails with `TS2591` on `node:fs/promises`, on `node:path` and on `process`, and then cascading `TS7006` implicit-any on the `readdir` callback. Nothing about the errors points at the tsconfig, so set `"types": ["node"]` as part of the recipe rather than leaving it for the integration to discover. `npm install --save-dev typescript` also resolves to TypeScript 7 now; say which major the project got, because `tsc --init`'s defaults moved with it. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/pipelex-codex/skills/pipelex-scaffold/references/starters.md b/pipelex-codex/skills/pipelex-scaffold/references/starters.md index 52ccdc3..c305e13 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/starters.md @@ -24,15 +24,17 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +**The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/pipelex-vibe/skills/pipelex-integrate/SKILL.md b/pipelex-vibe/skills/pipelex-integrate/SKILL.md index 1f3e250..3b711fd 100644 --- a/pipelex-vibe/skills/pipelex-integrate/SKILL.md +++ b/pipelex-vibe/skills/pipelex-integrate/SKILL.md @@ -18,7 +18,7 @@ Take a method — a local `.mthds` bundle, a published address (`method_ref`), o Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. -**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus one shared helper, and a user who wants more says so. ## Requirements — the Pipelex MCP tools @@ -144,7 +144,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ | `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | | not required | optional parameter (`?`) | `T \| None = None` | -**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. +**Exactly one shared helper, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. There is no second one — the wire-output helper that would have been it was struck for being lossy, and writing one is forbidden below, so read "one" as the whole budget and not as a floor. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. ### Step 10: Wire the offline drift gate diff --git a/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs index 0e9fe5e..74e9156 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex-vibe/skills/pipelex-integrate/references/codegen-check.mjs @@ -119,12 +119,21 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } + // The same hazard one level up, and it is why the line below is not `sidecar?.sources`. A file + // whose whole content is `null`, `[]`, `"x"`, `42` or `true` is valid JSON and is not an object, + // and optional chaining turns every one of them into `undefined` — the legitimate absent case — + // so the gate would print "a by-ref or by-id integration" and exit 0 over a sidecar that says + // nothing of the kind. On the sidecar, `?.` does exactly what `??` would do on `sources`. + if (typeof sidecar !== "object" || sidecar === null || Array.isArray(sidecar)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — not a JSON object, so staleness cannot be ruled out`] }; + } + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it // in the sidecar really is an array), and `null` is why this does not use `??`, which would // quietly turn an explicit null into the legitimate absent case. - const sources = sidecar?.sources; + const sources = sidecar.sources; if (sources === undefined) { return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; } diff --git a/pipelex-vibe/skills/pipelex-integrate/references/python.md b/pipelex-vibe/skills/pipelex-integrate/references/python.md index 8acdbde..f099f30 100644 --- a/pipelex-vibe/skills/pipelex-integrate/references/python.md +++ b/pipelex-vibe/skills/pipelex-integrate/references/python.md @@ -64,7 +64,14 @@ def _read_bundle() -> list[str]: """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that imports a sibling needs that sibling submitted with it, or the run fails to load what the generated models were projected from.""" - return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + contents = [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + if not contents: + # `rglob` on a missing or empty directory returns nothing and raises nothing, so without + # this the run goes out with `mthds_contents=[]` and fails server-side, naming the pipe + # rather than the path that is actually wrong. The TypeScript twin gets this for free: + # `readdir` throws ENOENT. + raise FileNotFoundError(f"no .mthds files under {BUNDLE_DIR}") + return contents async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index 46b0be2..909b451 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -76,7 +76,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -115,18 +115,22 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand - **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. -- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: +**Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. + +**Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" ``` +The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. + **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. ### Step 4: The env file diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md index 7602a15..e3631ee 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md @@ -8,14 +8,16 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where the import package lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | -| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | -| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | +| **Minimal (the default when no framework is named)** | `uv init --package --no-workspace ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app --no-workspace ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package --no-workspace && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package --no-workspace && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package --no-workspace && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | -`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package --no-workspace .` — it names the package after the directory. + +**`--no-workspace` is on every `uv init` above, and it is the same hazard the `uv add` parentheses below address, one command earlier.** Run inside a directory that already holds a `pyproject.toml` — the scaffold being made inside an existing project, which is the common case — a bare `uv init --package ` does not create a standalone project at all. It prints `Adding as member of workspace …`, appends a `[tool.uv.workspace]` table naming `` to **the user's own `pyproject.toml`**, and then the first `uv add` writes the lockfile at the *parent* root, so the new project has no `uv.lock` of its own and does not resolve standalone. Editing a file of the user's is exactly what this skill does not do, and a project that needs its parent to resolve is not the project the report says was handed over. `--no-workspace` leaves the parent untouched and gives `` its own lock. With no parent project it changes nothing, so it is safe to pass always, which is why it is not conditional. **Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. @@ -25,16 +27,20 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | -| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | -| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Express server | the minimal recipe above, then `(cd && npm install express && npm install --save-dev @types/express)` | no | `src/` | | Node library | the minimal recipe above | no | `src/` | | pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**Every follow-on `npm install` above is parenthesised for the same reason every `uv add` is, and npm is the worse of the two.** `npm install` resolves the project it writes to from its *working* directory upwards, so run from the parent it adds the dependency to the user's own `package.json` and puts `node_modules/` in the user's tree — and where `uv add` at least fails outright when it finds no project nearby, npm finds the parent and silently succeeds, leaving the new project with nothing installed and no error to read. The `cd ` inside the minimal recipe's own `&&` chain does not reach a follow-on issued as a separate command, because shell state does not survive from one command to the next. Use the subshell, or `npm install --prefix `. + **The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. +**And `tsc --init` writes `"types": []` as an active key, which switches `@types/node` off on the line after the recipe installed it.** That empty array is the current `tsc --init` template's own default (TypeScript 7 writes it, with `// "types": ["node"],` commented out three lines below), and it means no `@types` package is loaded at all — so the call site `/pipelex-integrate` writes next fails with `TS2591` on `node:fs/promises`, on `node:path` and on `process`, and then cascading `TS7006` implicit-any on the `readdir` callback. Nothing about the errors points at the tsconfig, so set `"types": ["node"]` as part of the recipe rather than leaving it for the integration to discover. `npm install --save-dev typescript` also resolves to TypeScript 7 now; say which major the project got, because `tsc --init`'s defaults moved with it. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md index 52ccdc3..c305e13 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md @@ -24,15 +24,17 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +**The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/pipelex/skills/pipelex-integrate/SKILL.md b/pipelex/skills/pipelex-integrate/SKILL.md index ceba242..eece2ce 100644 --- a/pipelex/skills/pipelex-integrate/SKILL.md +++ b/pipelex/skills/pipelex-integrate/SKILL.md @@ -29,7 +29,7 @@ Take a method — a local `.mthds` bundle, a published address (`method_ref`), o Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. -**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus one shared helper, and a user who wants more says so. ## Requirements — the Pipelex MCP tools @@ -155,7 +155,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ | `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | | not required | optional parameter (`?`) | `T \| None = None` | -**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. +**Exactly one shared helper, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. There is no second one — the wire-output helper that would have been it was struck for being lossy, and writing one is forbidden below, so read "one" as the whole budget and not as a floor. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. ### Step 10: Wire the offline drift gate diff --git a/pipelex/skills/pipelex-integrate/references/codegen-check.mjs b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs index 0e9fe5e..74e9156 100644 --- a/pipelex/skills/pipelex-integrate/references/codegen-check.mjs +++ b/pipelex/skills/pipelex-integrate/references/codegen-check.mjs @@ -119,12 +119,21 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } + // The same hazard one level up, and it is why the line below is not `sidecar?.sources`. A file + // whose whole content is `null`, `[]`, `"x"`, `42` or `true` is valid JSON and is not an object, + // and optional chaining turns every one of them into `undefined` — the legitimate absent case — + // so the gate would print "a by-ref or by-id integration" and exit 0 over a sidecar that says + // nothing of the kind. On the sidecar, `?.` does exactly what `??` would do on `sources`. + if (typeof sidecar !== "object" || sidecar === null || Array.isArray(sidecar)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — not a JSON object, so staleness cannot be ruled out`] }; + } + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it // in the sidecar really is an array), and `null` is why this does not use `??`, which would // quietly turn an explicit null into the legitimate absent case. - const sources = sidecar?.sources; + const sources = sidecar.sources; if (sources === undefined) { return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; } diff --git a/pipelex/skills/pipelex-integrate/references/python.md b/pipelex/skills/pipelex-integrate/references/python.md index 8acdbde..f099f30 100644 --- a/pipelex/skills/pipelex-integrate/references/python.md +++ b/pipelex/skills/pipelex-integrate/references/python.md @@ -64,7 +64,14 @@ def _read_bundle() -> list[str]: """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that imports a sibling needs that sibling submitted with it, or the run fails to load what the generated models were projected from.""" - return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + contents = [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + if not contents: + # `rglob` on a missing or empty directory returns nothing and raises nothing, so without + # this the run goes out with `mthds_contents=[]` and fails server-side, naming the pipe + # rather than the path that is actually wrong. The TypeScript twin gets this for free: + # `readdir` throws ENOENT. + raise FileNotFoundError(f"no .mthds files under {BUNDLE_DIR}") + return contents async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index 97373c9..2ea73e9 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -83,7 +83,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -122,18 +122,22 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand - **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. -- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: +**Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. + +**Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" ``` +The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. + **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. ### Step 4: The env file diff --git a/pipelex/skills/pipelex-scaffold/references/initializers.md b/pipelex/skills/pipelex-scaffold/references/initializers.md index 7602a15..e3631ee 100644 --- a/pipelex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex/skills/pipelex-scaffold/references/initializers.md @@ -8,14 +8,16 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where the import package lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | -| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | -| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | +| **Minimal (the default when no framework is named)** | `uv init --package --no-workspace ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app --no-workspace ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package --no-workspace && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package --no-workspace && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package --no-workspace && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | -`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package --no-workspace .` — it names the package after the directory. + +**`--no-workspace` is on every `uv init` above, and it is the same hazard the `uv add` parentheses below address, one command earlier.** Run inside a directory that already holds a `pyproject.toml` — the scaffold being made inside an existing project, which is the common case — a bare `uv init --package ` does not create a standalone project at all. It prints `Adding as member of workspace …`, appends a `[tool.uv.workspace]` table naming `` to **the user's own `pyproject.toml`**, and then the first `uv add` writes the lockfile at the *parent* root, so the new project has no `uv.lock` of its own and does not resolve standalone. Editing a file of the user's is exactly what this skill does not do, and a project that needs its parent to resolve is not the project the report says was handed over. `--no-workspace` leaves the parent untouched and gives `` its own lock. With no parent project it changes nothing, so it is safe to pass always, which is why it is not conditional. **Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. @@ -25,16 +27,20 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | -| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | -| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Express server | the minimal recipe above, then `(cd && npm install express && npm install --save-dev @types/express)` | no | `src/` | | Node library | the minimal recipe above | no | `src/` | | pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**Every follow-on `npm install` above is parenthesised for the same reason every `uv add` is, and npm is the worse of the two.** `npm install` resolves the project it writes to from its *working* directory upwards, so run from the parent it adds the dependency to the user's own `package.json` and puts `node_modules/` in the user's tree — and where `uv add` at least fails outright when it finds no project nearby, npm finds the parent and silently succeeds, leaving the new project with nothing installed and no error to read. The `cd ` inside the minimal recipe's own `&&` chain does not reach a follow-on issued as a separate command, because shell state does not survive from one command to the next. Use the subshell, or `npm install --prefix `. + **The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. +**And `tsc --init` writes `"types": []` as an active key, which switches `@types/node` off on the line after the recipe installed it.** That empty array is the current `tsc --init` template's own default (TypeScript 7 writes it, with `// "types": ["node"],` commented out three lines below), and it means no `@types` package is loaded at all — so the call site `/pipelex-integrate` writes next fails with `TS2591` on `node:fs/promises`, on `node:path` and on `process`, and then cascading `TS7006` implicit-any on the `readdir` callback. Nothing about the errors points at the tsconfig, so set `"types": ["node"]` as part of the recipe rather than leaving it for the integration to discover. `npm install --save-dev typescript` also resolves to TypeScript 7 now; say which major the project got, because `tsc --init`'s defaults moved with it. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/pipelex/skills/pipelex-scaffold/references/starters.md b/pipelex/skills/pipelex-scaffold/references/starters.md index 52ccdc3..c305e13 100644 --- a/pipelex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex/skills/pipelex-scaffold/references/starters.md @@ -24,15 +24,17 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +**The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/skills/pipelex-integrate/references/codegen-check.mjs b/skills/pipelex-integrate/references/codegen-check.mjs index 0e9fe5e..74e9156 100644 --- a/skills/pipelex-integrate/references/codegen-check.mjs +++ b/skills/pipelex-integrate/references/codegen-check.mjs @@ -119,12 +119,21 @@ async function checkSources(dir) { return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — unreadable (${error.message}), so staleness cannot be ruled out`] }; } + // The same hazard one level up, and it is why the line below is not `sidecar?.sources`. A file + // whose whole content is `null`, `[]`, `"x"`, `42` or `true` is valid JSON and is not an object, + // and optional chaining turns every one of them into `undefined` — the legitimate absent case — + // so the gate would print "a by-ref or by-id integration" and exit 0 over a sidecar that says + // nothing of the kind. On the sidecar, `?.` does exactly what `??` would do on `sources`. + if (typeof sidecar !== "object" || sidecar === null || Array.isArray(sidecar)) { + return { code: EXIT_DRIFT, lines: [` stale-source: ${SIDECAR_FILENAME} — not a JSON object, so staleness cannot be ruled out`] }; + } + // A present-but-wrong-shaped `sources` must fail the way an unreadable sidecar does. Coerced to // {} it would check nothing, print nothing and exit 0 — the one input that is both silent and // green. An array is the shape to beware (`typeof [] === "object"`, and `method.files` beside it // in the sidecar really is an array), and `null` is why this does not use `??`, which would // quietly turn an explicit null into the legitimate absent case. - const sources = sidecar?.sources; + const sources = sidecar.sources; if (sources === undefined) { return { code: EXIT_CURRENT, lines: [` ${SIDECAR_FILENAME} records no sources — a by-ref or by-id integration; source staleness does not apply`] }; } diff --git a/skills/pipelex-integrate/references/python.md b/skills/pipelex-integrate/references/python.md index 8acdbde..f099f30 100644 --- a/skills/pipelex-integrate/references/python.md +++ b/skills/pipelex-integrate/references/python.md @@ -64,7 +64,14 @@ def _read_bundle() -> list[str]: """Every `.mthds` file of the bundle, sorted. A bundle is one closure: a main file that imports a sibling needs that sibling submitted with it, or the run fails to load what the generated models were projected from.""" - return [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + contents = [path.read_text(encoding="utf-8") for path in sorted(BUNDLE_DIR.rglob("*.mthds"))] + if not contents: + # `rglob` on a missing or empty directory returns nothing and raises nothing, so without + # this the run goes out with `mthds_contents=[]` and fails server-side, naming the pipe + # rather than the path that is actually wrong. The TypeScript twin gets this for free: + # `readdir` throws ENOENT. + raise FileNotFoundError(f"no .mthds files under {BUNDLE_DIR}") + return contents async def summarize_pdf(*, document: dict[str, Any], context: str | None = None) -> DocumentSummary: diff --git a/skills/pipelex-scaffold/references/initializers.md b/skills/pipelex-scaffold/references/initializers.md index 7602a15..e3631ee 100644 --- a/skills/pipelex-scaffold/references/initializers.md +++ b/skills/pipelex-scaffold/references/initializers.md @@ -8,14 +8,16 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where the import package lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `uv init --package ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | -| A script-style app rather than a package | `uv init --app ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | -| FastAPI service | `uv init --package && (cd && uv add "fastapi[standard]")` | yes | as minimal | -| Django project | `uv init --package && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | -| Typer CLI (the Python starter's shape, without the starter) | `uv init --package && (cd && uv add typer)` | yes | as minimal | +| **Minimal (the default when no framework is named)** | `uv init --package --no-workspace ` | yes (`--vcs none` to skip) | `src//` with `__init__.py` and a console-script entry in `pyproject.toml` | +| A script-style app rather than a package | `uv init --app --no-workspace ` | yes | `main.py` at the root — no import package; `/pipelex-integrate` will put the generated tree under `generated/` at the root | +| FastAPI service | `uv init --package --no-workspace && (cd && uv add "fastapi[standard]")` | yes | as minimal | +| Django project | `uv init --package --no-workspace && (cd && uv add django && uv run django-admin startproject config .)` | yes (from `uv init`) | the Django project package `config/` plus `src//`; ask which one owns the Pipelex call sites | +| Typer CLI (the Python starter's shape, without the starter) | `uv init --package --no-workspace && (cd && uv add typer)` | yes | as minimal | | An existing `pyproject.toml` layout the user prefers (poetry, pdm, hatch) | the tool the user names: `poetry new `, `pdm init --non-interactive`, `hatch new ` | poetry: no; pdm: no; hatch: no | per tool — `poetry new` and `hatch new` make `/` or `src//` | -`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package .` — it names the package after the directory. +`uv init` refuses a directory that already holds a project; on an empty "here" directory use `uv init --package --no-workspace .` — it names the package after the directory. + +**`--no-workspace` is on every `uv init` above, and it is the same hazard the `uv add` parentheses below address, one command earlier.** Run inside a directory that already holds a `pyproject.toml` — the scaffold being made inside an existing project, which is the common case — a bare `uv init --package ` does not create a standalone project at all. It prints `Adding as member of workspace …`, appends a `[tool.uv.workspace]` table naming `` to **the user's own `pyproject.toml`**, and then the first `uv add` writes the lockfile at the *parent* root, so the new project has no `uv.lock` of its own and does not resolve standalone. Editing a file of the user's is exactly what this skill does not do, and a project that needs its parent to resolve is not the project the report says was handed over. `--no-workspace` leaves the parent untouched and gives `` its own lock. With no parent project it changes nothing, so it is safe to pass always, which is why it is not conditional. **Every `uv add` above runs inside ``, and the parentheses are why.** `uv add` resolves the project from its *working* directory upwards, so run from the parent it writes the dependency into whatever project it finds there — the user's own `pyproject.toml` and lockfile, when the scaffold is being made inside an existing workspace — or fails outright when it finds none. Neither is the new project. `uv add --directory ` is equivalent if you prefer a flag to a subshell. @@ -25,16 +27,20 @@ After the initializer: `git init -b main` only if it did not initialize a reposi |---|---|---|---| | **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | -| Vite + React | `npm create vite@latest -- --template react-ts` then `npm install` | no | `src/` | +| Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | -| Express server | the minimal recipe above, then `npm install express && npm install --save-dev @types/express` | no | `src/` | +| Express server | the minimal recipe above, then `(cd && npm install express && npm install --save-dev @types/express)` | no | `src/` | | Node library | the minimal recipe above | no | `src/` | | pnpm / yarn / bun instead of npm | replace `npm create` with `pnpm create` / `yarn create` / `bun create`, and the install command accordingly; `/pipelex-integrate` reads the lockfile to pick the package manager for what it adds | — | — | `npm create @latest -- `: the `--` is what passes the flags to the initializer rather than to npm. The Next.js `--yes` accepts the initializer's defaults for every prompt not covered by a flag. +**Every follow-on `npm install` above is parenthesised for the same reason every `uv add` is, and npm is the worse of the two.** `npm install` resolves the project it writes to from its *working* directory upwards, so run from the parent it adds the dependency to the user's own `package.json` and puts `node_modules/` in the user's tree — and where `uv add` at least fails outright when it finds no project nearby, npm finds the parent and silently succeeds, leaving the new project with nothing installed and no error to read. The `cd ` inside the minimal recipe's own `&&` chain does not reach a follow-on issued as a separate command, because shell state does not survive from one command to the next. Use the subshell, or `npm install --prefix `. + **The minimal recipe's `--module nodenext` plus `"type": "module"` is exactly the shape that meets the ts-zod emitter's extensionless-import defect** (`pipelex-integrate`'s `references/typescript.md`, "Known defect"): the generated `binder.ts` fails the type check with `TS2835` and will not load at runtime. So say so when you hand a minimal TypeScript project to `/pipelex-integrate`, and when the user has no reason to prefer Node's own resolution, prefer a bundler-backed setup (Vite, Next.js) or `--module esnext --moduleResolution bundler`, which the defect does not touch. The two recipes also need `node_modules/` and `dist/` in a `.gitignore` before the pristine commit — neither `npm init -y` nor `tsc --init` writes one. +**And `tsc --init` writes `"types": []` as an active key, which switches `@types/node` off on the line after the recipe installed it.** That empty array is the current `tsc --init` template's own default (TypeScript 7 writes it, with `// "types": ["node"],` commented out three lines below), and it means no `@types` package is loaded at all — so the call site `/pipelex-integrate` writes next fails with `TS2591` on `node:fs/promises`, on `node:path` and on `process`, and then cascading `TS7006` implicit-any on the `readdir` callback. Nothing about the errors points at the tsconfig, so set `"types": ["node"]` as part of the recipe rather than leaving it for the integration to discover. `npm install --save-dev typescript` also resolves to TypeScript 7 now; say which major the project got, because `tsc --init`'s defaults moved with it. + ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. diff --git a/skills/pipelex-scaffold/references/starters.md b/skills/pipelex-scaffold/references/starters.md index 52ccdc3..c305e13 100644 --- a/skills/pipelex-scaffold/references/starters.md +++ b/skills/pipelex-scaffold/references/starters.md @@ -24,15 +24,17 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +**The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/templates/skills/pipelex-integrate/SKILL.md.j2 b/templates/skills/pipelex-integrate/SKILL.md.j2 index f50176b..d2130ea 100644 --- a/templates/skills/pipelex-integrate/SKILL.md.j2 +++ b/templates/skills/pipelex-integrate/SKILL.md.j2 @@ -24,7 +24,7 @@ Take a method — a local `.mthds` bundle, a published address (`method_ref`), o Re-running on a project that already carries a generated tree is **[refresh mode](#refresh-mode)**, the common case. A project that already **owns a codegen harness** — one made from a Pipelex starter — keeps it: see [that section](#a-project-that-owns-a-codegen-harness) before generating anything. -**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus at most two shared helpers, and a user who wants more says so. +**What this skill is not.** Not a build tool (no watch mode, no per-project harness — the workshop is the harness), not a runner (`/pipelex-inputs` prepares inputs and offers a run), not a design skill (a method that does not validate or cannot run goes back to `/pipelex-design`). It writes no tests, routes, UI or CLI commands and edits no existing business code: it stops at one callable module per method plus one shared helper, and a user who wants more says so. ## Requirements — the Pipelex MCP tools @@ -150,7 +150,7 @@ With the project's own package manager (read the lockfile): `zod` and `@pipelex/ | `Concept[]` / `Concept[N]` | `T[]` | `list[T]` | | not required | optional parameter (`?`) | `T \| None = None` | -**At most two shared helpers, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. +**Exactly one shared helper, created once per project and reused by every later method:** a client factory (`getPipelexClient()` / a `PipelexAPIClient` construction that matches how the project builds its other clients), and nothing else. There is no second one — the wire-output helper that would have been it was struck for being lossy, and writing one is forbidden below, so read "one" as the whole budget and not as a floor. If the project already has a Pipelex client module, use it. The module **does not upload**: its docstring points callers holding local files or bytes at the SDK's `prepareInputs` / `prepare_inputs`, which is the SDK's own signature-driven upload — and carries the SDK's warning that `prepareInputs` treats any string it does not recognise as `data:`, `http(s)://` or `pipelex-storage://` as a **local filesystem path** it reads and uploads, so a public endpoint must gate schemes before handing values to it. Python projects that are synchronous throughout get a thin `asyncio.run` wrapper beside the async function; async-native projects get the async function alone. Follow the project's conventions where you can see them (module style, quoting, error handling) and the SDK's defaults where you cannot; let the SDK's typed errors propagate. Module templates: the language references. ### Step 10: Wire the offline drift gate diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index e6c77a2..a68d09a 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -76,7 +76,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -115,18 +115,22 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand - **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. -- **No framework named** takes the language's own minimal initializer: Python → `uv init --package `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. +- **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. ### Step 3: Version control and the pristine commit -If the initializer did not `git init` on its own (some do — `uv init` and `create-next-app` among them), run `git init -b main` in the directory. **Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and `git add -A` would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged (`git -C diff --cached --stat | tail -1`) before committing. Then the one commit, for the same reason as branch A: +**Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. + +**Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" ``` +The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. + **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. ### Step 4: The env file diff --git a/tests/unit/test_pipelex_integrate_skill.py b/tests/unit/test_pipelex_integrate_skill.py index f32c25a..17092f2 100644 --- a/tests/unit/test_pipelex_integrate_skill.py +++ b/tests/unit/test_pipelex_integrate_skill.py @@ -3,11 +3,13 @@ from __future__ import annotations import re +import shutil +import subprocess from pathlib import Path import pytest -from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir, setup_static_assets +from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir class TestPipelexIntegrateSkill: @@ -82,6 +84,20 @@ def test_the_wire_null_helper_is_never_installed(self) -> None: for reference in self.REFERENCES: assert "dropWireNulls" not in (self.REFERENCES_DIR / reference).read_text(encoding="utf-8") + def test_the_helper_budget_is_one_and_does_not_license_a_second(self) -> None: + """The budget was two while the wire-output helper was the second. That helper was struck for + being lossy and is forbidden outright below, so a surviving "at most two" is not merely a + stale count — it is written permission to create the one thing the campaign removed, in a + sentence that then says "and nothing else" about a list of one. + """ + body = self.integrate + assert "**Exactly one shared helper, created once per project" in body + assert "one callable module per method plus one shared helper" in body + assert "two shared helpers" not in body + assert "at most two" not in body + decisions = (self.REPO_ROOT / "docs" / "decisions.md").read_text(encoding="utf-8") + assert "two shared helpers" not in decisions + def test_failure_posture_pins_the_403_and_the_orphans(self) -> None: body = self.integrate assert "a **403** on `mthds_codegen` is a feature gate, not a key problem" in body @@ -131,9 +147,16 @@ def test_the_gate_fails_closed_on_a_malformed_sources_and_does_not_truncate(self `process.exit` does not guarantee. """ gate = (self.REFERENCES_DIR / "codegen-check.mjs").read_text(encoding="utf-8") - # No `??` here: it would turn an explicit null into the legitimate absent case. - assert "const sources = sidecar?.sources;" in gate + # Neither `??` nor `?.` on the way to `sources`: both turn a hostile value into the + # legitimate absent case. `?.` is the subtler of the two — a sidecar whose whole content is + # `null`, `[]`, `"x"` or `42` is valid JSON and not an object, and `sidecar?.sources` makes + # every one of them `undefined`, so the gate claims "a by-ref or by-id integration" and + # exits 0 over a file that says nothing of the kind. Guard the sidecar, then its `sources`. + assert "const sources = sidecar.sources;" in gate + assert "const sources = sidecar?.sources" not in gate assert "?? {}" not in gate + assert 'if (typeof sidecar !== "object" || sidecar === null || Array.isArray(sidecar)) {' in gate + assert "not a JSON object, so staleness cannot be ruled out" in gate assert 'if (typeof sources !== "object" || sources === null || Array.isArray(sources)) {' in gate assert "is not an object, so staleness cannot be ruled out" in gate # No branch is both silent and green: absent and empty both announce themselves. @@ -143,6 +166,79 @@ def test_the_gate_fails_closed_on_a_malformed_sources_and_does_not_truncate(self assert 'new TextDecoder("utf-8", { fatal: true, ignoreBOM: true })' in gate assert "process.exit(await main" not in gate + @pytest.mark.parametrize( + ("sidecar", "expected_exit"), + [ + ("null", 1), + ("[]", 1), + ('"a string"', 1), + ("42", 1), + ("true", 1), + ('{"method": {"id": "mt_x"}}', 0), + ('{"sources": {}}', 0), + ('{"sources": null}', 1), + ('{"sources": []}', 1), + ], + ) + def test_the_gate_runs_and_refuses_a_non_object_sidecar(self, sidecar: str, expected_exit: int, tmp_path: Path) -> None: + """Run the gate, do not read it. Every case here was found by executing it, and the + non-object-sidecar ones reported `current` with exit 0 before the guard went in — the + failure the file's own header calls its one job to avoid. + + The lock check is made to yield no verdict (no `codegen.lock`), which the script reports on + its own and which suppresses the sidecar check — so the cases are driven through a tree that + reaches `checkSources`, i.e. one with a lock. Skipped when no `node` is on the PATH. + """ + node = shutil.which("node") + if node is None: + pytest.skip("no node on the PATH") + gate = tmp_path / "codegen-check.mjs" + gate.write_bytes((self.REFERENCES_DIR / "codegen-check.mjs").read_bytes()) + # A lock the SDK cannot even be asked about would short-circuit the sidecar check, so stub + # the one import and let the tree report itself current. + stub = tmp_path / "node_modules" / "@pipelex" / "sdk" + stub.mkdir(parents=True) + (stub / "package.json").write_text( + '{"name":"@pipelex/sdk","version":"0.0.0-stub","type":"module","main":"index.js","exports":{".":"./index.js"}}', + encoding="utf-8", + ) + (stub / "index.js").write_text( + "export class CodegenLockError extends Error {}\n" + "export const isStampableArtifactPath = (p) => p.endsWith('.ts') || p.endsWith('.py');\n" + "export const runCodegenCheck = async () => ({ isCurrent: true, drifts: [], " + "crateFingerprint: 'stubfingerprint', engineVersion: '0.0.0' });\n", + encoding="utf-8", + ) + tree = tmp_path / "generated" / "m" + tree.mkdir(parents=True) + (tree / "codegen.lock").write_text("lock_version = 1\n", encoding="utf-8") + (tree / "types.ts").write_text("export const x = 1;\n", encoding="utf-8") + (tree / "sources.json").write_text(sidecar, encoding="utf-8") + result = subprocess.run( + [node, str(gate), "generated/m"], + cwd=tmp_path, + capture_output=True, + text=True, + check=False, + ) + assert result.returncode == expected_exit, ( + f"sidecar {sidecar!r}: expected exit {expected_exit}, got {result.returncode}\nstdout: {result.stdout}\nstderr: {result.stderr}" + ) + if expected_exit == 0: + assert "codegen-check: current" in result.stdout + else: + assert "codegen-check: drift" in result.stdout + + def test_the_python_call_site_refuses_an_empty_bundle(self) -> None: + """`Path.rglob` on a missing or empty directory returns nothing and raises nothing, so a + wrong `BUNDLE_DIR` would submit `mthds_contents=[]` and fail server-side against the pipe + instead of locally against the path. The TypeScript twin gets this free from `readdir`'s + ENOENT; the Python one has to ask. + """ + python = (self.REFERENCES_DIR / "python.md").read_text(encoding="utf-8") + assert "if not contents:" in python + assert 'raise FileNotFoundError(f"no .mthds files under {BUNDLE_DIR}")' in python + def test_method_id_warns_and_refresh_leaves_the_call_site_alone(self) -> None: body = self.integrate assert "The catalog is unversioned" in body @@ -197,11 +293,16 @@ def test_every_platform_renders_the_skill_and_its_references(self, target_name: assert (references_dir / reference).is_file(), f"{target_name}: missing references/{reference}" @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) - def test_the_build_copies_the_references_byte_for_byte(self, target_name: str, tmp_path: Path) -> None: - """The check script is executable know-how: a stale or re-encoded copy is a broken gate.""" + def test_the_committed_references_match_the_source_byte_for_byte(self, target_name: str) -> None: + """The references are executable know-how, and the committed target copies are the ones a + user installs — so compare against those, not against a fresh `copytree` into a tmp dir, + which only ever asserts that `shutil` copies bytes. A stale committed copy is the whole + failure mode, and it is invisible to any assertion that rebuilds its own expected side. + """ config = load_target_config(self.REPO_ROOT / "targets", target_name) - setup_static_assets(self.REPO_ROOT, tmp_path, self.REPO_ROOT / "templates", config.include_skills) - produced = tmp_path / "skills" / "pipelex-integrate" / "references" + installed = resolve_output_dir(self.REPO_ROOT, config.source) / "skills" / "pipelex-integrate" / "references" for reference in self.REFERENCES: - assert (produced / reference).is_file(), f"{target_name}: the build did not copy references/{reference}" - assert (produced / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes() + assert (installed / reference).is_file(), f"{target_name}: missing references/{reference}" + assert (installed / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes(), ( + f"{target_name}: references/{reference} is stale — run `make build`" + ) diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index a37274c..f5acd3e 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -6,7 +6,7 @@ import pytest -from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir, setup_static_assets +from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir class TestPipelexScaffoldSkill: @@ -53,8 +53,12 @@ class TestPipelexScaffoldSkill: "**from inside ``**", # An initializer that commits has already made the pristine commit. "**An initializer that commits as well as `git init`s has already made this commit.**", - # npm init -y and tsc --init write no .gitignore, so git add -A would commit node_modules. - "**Confirm there is a `.gitignore` covering the dependency tree and the build output before you stage anything**", + # The default TS branch is prescribed in the skill; both its costs live in the reference. + "**Read [references/initializers.md](references/initializers.md) before running either**", + # npm init -y and tsc --init write no .gitignore, so the staging would commit node_modules. + "**Then confirm there is a `.gitignore` covering the dependency tree and the build output**", + # with no .git of its own is governed by the repo enclosing it — the user's. + "**Test whether `` is its own repository; never infer it from which initializer ran.**", ) @property @@ -87,11 +91,16 @@ def test_integrate_hands_a_missing_project_to_scaffold(self) -> None: def test_references_describe_both_starters_and_the_initializers(self) -> None: starters = (self.REFERENCES_DIR / "starters.md").read_text(encoding="utf-8") assert "pipelex-starter-js" in starters and "pipelex-starter-python" in starters - assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git" in starters + # The `|| exit` is the guard on the `rm -rf /.git` below it: a clone that never ran + # leaves that line to delete whatever `.git` is at that path — a user's history, if + # was theirs. The SKILL.md carries it; so must the reference the skill names as the source + # of every command, or the guard exists only in the copy nobody executes from. + assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit" in starters + assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit" in starters + assert "The `|| exit` on the clone is load-bearing" in starters assert "gh repo create / --template Pipelex/pipelex-starter-python" in starters assert "shell out to a `pipelex` CLI the starter does not depend on" in starters initializers = (self.REFERENCES_DIR / "initializers.md").read_text(encoding="utf-8") - assert "uv init --package " in initializers assert "npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes" in initializers assert "No SDK dependency" in initializers # Every `uv add` runs inside the new project: from the parent it writes to the user's own. @@ -102,8 +111,43 @@ def test_references_describe_both_starters_and_the_initializers(self) -> None: "(cd && uv add django && uv run django-admin startproject config .)", ): assert recipe in initializers, f"uv add not scoped to the project: {recipe}" + # Every `uv init` carries --no-workspace. Without it, run inside a directory that already + # holds a pyproject.toml, uv appends a [tool.uv.workspace] table to the USER'S file and puts + # the lock at the parent root, so the new project does not resolve standalone. Same hazard + # class as the `uv add` parentheses above, one command earlier. + assert "uv init --package --no-workspace " in initializers + assert "uv init --app --no-workspace " in initializers + assert "`--no-workspace` is on every `uv init` above" in initializers + assert "| `uv init --package `" not in initializers, "a bare uv init absorbs into the parent workspace" + assert "uv init --package &&" not in initializers, "a bare uv init absorbs into the parent workspace" + # npm resolves the project it writes to upward exactly as uv does, and unlike uv it finds the + # parent and silently succeeds, so every follow-on install is scoped too. + for recipe in ("(cd && npm install)", "(cd && npm install express && npm install --save-dev @types/express)"): + assert recipe in initializers, f"npm install not scoped to the project: {recipe}" + assert "**Every follow-on `npm install` above is parenthesised" in initializers # The minimal TS default is the very resolution the emitter defect breaks. assert "is exactly the shape that meets the ts-zod emitter's extensionless-import defect" in initializers + # `tsc --init` writes an active `"types": []`, which switches off the @types/node the line + # before it installed — the integrate call site then fails TS2591 on node:path and process. + assert '`tsc --init` writes `"types": []` as an active key' in initializers + assert 'set `"types": ["node"]` as part of the recipe' in initializers + + def test_the_pristine_commit_cannot_reach_an_enclosing_repository(self) -> None: + """`git -C ` sets git's working directory and scopes nothing. With no `.git` of its own, + is governed by whatever repo encloses it — the user's — and a pathspec-less `add -A` + stages that whole worktree, committing the user's unrelated files under this skill's message. + `uv init` is on the list of initializers that `git init`, but only when it creates a + standalone project; inside an existing one it makes a workspace member and no repo. + """ + body = self.TEMPLATE.read_text(encoding="utf-8") + assert "git -C rev-parse --show-toplevel` must print `` itself" in body + assert "never infer it from which initializer ran" in body + # Both pristine commits carry the pathspec, so the staging cannot escape . + assert body.count("git -C add -A -- .") == 2 + assert "git -C add -A &&" not in body + # The read-back must name paths; a --stat count cannot tell a scaffold from a swept worktree. + assert "git -C diff --cached --name-only" in body + assert "git -C diff --cached --stat | tail -1" not in body @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) def test_every_platform_renders_the_skill_and_its_references(self, target_name: str) -> None: @@ -134,10 +178,16 @@ def test_every_platform_renders_the_skill_and_its_references(self, target_name: assert (references_dir / reference).is_file(), f"{target_name}: missing references/{reference}" @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) - def test_the_build_copies_the_references_byte_for_byte(self, target_name: str, tmp_path: Path) -> None: + def test_the_committed_references_match_the_source_byte_for_byte(self, target_name: str) -> None: + """The references are executable know-how, and the committed target copies are the ones a + user installs — so compare against those, not against a fresh `copytree` into a tmp dir, + which only ever asserts that `shutil` copies bytes. A stale committed copy is the whole + failure mode, and it is invisible to any assertion that rebuilds its own expected side. + """ config = load_target_config(self.REPO_ROOT / "targets", target_name) - setup_static_assets(self.REPO_ROOT, tmp_path, self.REPO_ROOT / "templates", config.include_skills) - produced = tmp_path / "skills" / "pipelex-scaffold" / "references" + installed = resolve_output_dir(self.REPO_ROOT, config.source) / "skills" / "pipelex-scaffold" / "references" for reference in self.REFERENCES: - assert (produced / reference).is_file(), f"{target_name}: the build did not copy references/{reference}" - assert (produced / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes() + assert (installed / reference).is_file(), f"{target_name}: missing references/{reference}" + assert (installed / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes(), ( + f"{target_name}: references/{reference} is stale — run `make build`" + ) From d53f3cf4962407edbb305d9ee894206e5a7ca477 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 14:39:05 +0200 Subject: [PATCH 18/21] Read a directory holding nothing but .git as empty L-260912-724b71, ruled 2026-09-13. The non-empty refusal treated any existing entry as occupancy, so the skill refused a state it produces itself: `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, and branch B runs `git init -b main` in the directory it is working in one step later. The cruft list was declined in the same ruling, so the exception is written as one directory entry named `.git` and never as a predicate over ignorable files that a later reader could extend without a decision. Every other entry still refuses, and the test pins the declined names as refusing so that admitting one means rewording an asserted sentence. Never offering to clear anything is untouched: narrowing what counts as occupied is not permission to empty what is. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + docs/decisions.md | 1 + .../skills/pipelex-scaffold/SKILL.md | 4 +- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 4 +- pipelex/skills/pipelex-scaffold/SKILL.md | 4 +- templates/skills/pipelex-scaffold/SKILL.md.j2 | 4 +- tests/unit/test_pipelex_scaffold_skill.py | 52 +++++++++++++++++++ wip/pipelex-integrate/plan.md | 7 +++ wip/pipelex-integrate/scaffold-design.md | 6 ++- 9 files changed, 73 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 52341ac..b5858cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,7 @@ - **The helper budget said two and listed one.** `pipelex-integrate` announced "at most two shared helpers", named the client factory, and closed with "and nothing else" — residue of the wire-output helper that was struck for being lossy and is forbidden by name a few lines below. A stale count would be cosmetic; this one was written permission to create the single thing the campaign removed. It is one helper, in the skill, in both places it was stated, and in `docs/decisions.md`. - **The reference-copy test asserted that `shutil` copies bytes.** It built a fresh tree into a temp directory and compared it against the source it had just been built from, so it could not observe the only failure that matters — a stale committed copy under `pipelex/`, `pipelex-codex/` or `pipelex-vibe/`, which is what a user installs. It now compares the committed copies themselves, and its three-target parametrization exercises three different trees instead of three identical calls. - **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. +- **A directory holding nothing but `.git` is somewhere `pipelex-scaffold` will build.** The skill refused any target directory that was not empty, which made it refuse the state it produces itself one step later: `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive, and the skill's own initializer branch runs `git init -b main` in the directory it is working in. A lone `.git` now reads as empty. Everything else goes on refusing — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included — because the exception is one directory entry named in full and not a class of files the agent may decide to overlook, which is the judgement the refusal exists to prevent. Nothing about clearing changed: a directory that is not empty is still never emptied, and making room is still never offered. - **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. - **The file factory no longer touches the user's own project.** Every `uv run` line in the skill and its references now passes `--no-project`. Without it `uv run` walks up from the working directory, finds the nearest project, and *syncs* it — so rendering a test PDF inside a checkout created a `.venv/` and wrote a `uv.lock` the user never asked for, in a repository the skill has no business modifying. `--no-project` resolves the ephemeral `--with` packages against nothing at all, which is what the recipes always meant; a test asserts every shipped runner line carries the flag. diff --git a/docs/decisions.md b/docs/decisions.md index decb2ab..ba156b7 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -182,6 +182,7 @@ Both skills were run as cold headless sessions against a local `pipelex-mcp` bui - **An occupied generated directory is refused even when the user names it.** Every method of a target emits the same file names, so generating a second method into another method's directory overwrites it and reports an empty `orphans[]` — a clean-looking generation. The guard is the sidecar: a directory is this method's only when a `sources.json` there names it, a lock with no sidecar included. - **Containment is read before the first write, not after an error.** A path inside the workshop's working directory is legal wherever it points, so a harness launched beside the project accepts a write into the wrong tree; a run that hit this moved the tree across afterwards and left the project with a refresh that fails the same way every time. The skill now reads the path from the workshop's working directory first, and never moves a tree into place. - **A file the skill does not own is never cleared, and never offered for clearing.** Integrate met a hand-written file at an artifact path and offered to delete it; scaffold met a non-empty target directory and offered to move its contents aside and merge them back. The answer in both cases is another directory. +- **A directory holding nothing but `.git` reads as empty; everything else still refuses.** (Ruled 2026-09-13, `L-260912-724b71`.) The non-empty refusal treated any existing entry as occupancy, which made the skill refuse a state it produces itself: `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, branch B runs `git init -b main` in the directory it is working in one step later, and `references/initializers.md` already documents `uv init --package --no-workspace .` for the empty-"here" case. A lone `.git` is now read as empty. **The cruft list was declined in the same ruling** — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` go on refusing until a real report names one — because a list that grows by guesswork is how this rule drifts back into the agent judging which of a user's files matter, which is the thing the refusal exists to forbid; so the exception is written as one directory entry by name and never as a predicate over ignorable files. The separate rule that the skill never *offers* to clear anything is untouched: a directory read as empty is never cleared either way. - **A runtime behind a version manager is not a missing toolchain.** Scaffold, on a `PATH` without `node`, found the machine's `nvm` and carried on — which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, says which it used, and stops only when no runtime can be reached. - **One upstream defect is named rather than worked around.** The `ts-zod` emitter writes `binder.ts`'s sibling import with no file extension, which a plain Node ESM project rejects at type-check and at runtime while a bundler resolution accepts — which is why the JS starter never met it. Filed as `L-260912-857a5a` against `pipelex`. The tree is stamped and hashed, so the skill reports it, never patches the file, never drops the tree from the type checker, and leaves a change of `moduleResolution` to the user. diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index fc0edd1..421f729 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -160,7 +160,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index 909b451..b44ee8e 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -160,7 +160,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index 2ea73e9..e5dc3c3 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -30,7 +30,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -167,7 +167,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index a68d09a..3a63079 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -160,7 +160,7 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, and never offer to | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index f5acd3e..5e16999 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -65,6 +65,23 @@ class TestPipelexScaffoldSkill: def scaffold(self) -> str: return self.TEMPLATE.read_text(encoding="utf-8") + def render(self, target_name: str) -> str: + """The skill as one target's users read it, rendered from `templates/` in memory. + + A rule asserted on the template alone is a rule that may never reach a user: the platform + conditionals are resolved here, and the committed trees under `pipelex*/` are built from + exactly this call. + """ + config = load_target_config(self.REPO_ROOT / "targets", target_name) + rendered = render_templates( + self.REPO_ROOT / "templates", + self.REPO_ROOT, + config.template_vars, + include_skills=["pipelex-scaffold"], + target_name=config.name, + ) + return next(content for path, content in rendered.items() if path.match("skills/pipelex-scaffold/SKILL.md")) + def test_the_rules_are_stated(self) -> None: body = self.scaffold for rule in self.RULES: @@ -76,6 +93,41 @@ def test_fresh_clone_shortcut_and_template_checkout_stop(self) -> None: assert "Do not clone again." in body assert "this is the template, not a copy of it" in body + def test_only_a_lone_git_reads_as_empty_and_no_cruft_list_joins_it(self) -> None: + """`L-260912-724b71`, ruled 2026-09-13: a directory holding nothing but `.git` is empty. + + The refusal it narrows is the right one — the agent has no business deciding which of a + user's files matter — and the exception exists because `mkdir my-app && cd my-app && git + init` is an ordinary opening move and branch B runs `git init -b main` in the directory it + is working in one step later, so without it the skill refuses a state it produces itself. + + The cruft list was deliberately declined in the same ruling: `.DS_Store`, `.idea/`, + `.vscode/` and `Thumbs.db` keep refusing until a real report names one, because a list that + grows by guesswork is how this rule drifts back into the judgement it forbids. So this test + pins the exception as an entry named `.git` rather than as a predicate over ignorable + files, and pins the four declined names as still-refusing — adding any of them to the + exception means rewording a sentence asserted here, which is the point. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + # The exception, at both sites, each stated as one named entry and not as a category. + assert "**and a directory whose only entry is `.git` is empty for this rule**" in body + assert "**A directory holding nothing but `.git` is empty here and is written into**" in body + assert "branch B runs `git init -b main` in the directory it is working in one step later" in body + # And the refusal everything else still meets, with the declined names spelled out. + assert ( + "**A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else**" + in body + ) + assert "not a class of files you may decide to overlook" in body + assert "Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included" in body + assert "that exception is the one directory entry by name and not a class" in body + assert "`.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse" in body + # Untouched by the ruling: a directory read as empty is never cleared, so the skill + # still never offers to make room. Narrowing what counts as occupied is not permission + # to empty what is. + assert "never offer to move, delete or merge what it holds to make room" in body + assert "never delete, move or write into it, and never offer to" in body + def test_declares_no_mcp_tool(self) -> None: """The scaffold skill is MCP-free: no allowed-tools entry, no MCP-absent STOP message.""" body = self.scaffold diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index b339e95..387e793 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -109,6 +109,7 @@ Owner: `pipelex-plugins`. **Gate:** `scaffold-design.md`'s boxes ratified (Phase - [x] The report (S§5) with the session note as its own line. - [x] Mode (S§6) and a failure table condensed from S§7, including the "this is the template's own checkout" stop. - [x] `## Reference`: links to `references/starters.md` and `references/initializers.md`. +- [x] The non-empty refusal as `L-260912-724b71` ruled it on 2026-09-13: a lone `.git` reads as empty, at both sites (the "Where" row and the failure-table stop row), written as that one directory entry by name — the cruft list was declined, so `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` still refuse and the test pins them refusing. Never *offering* to clear is untouched. **The references — `skills/pipelex-scaffold/references/`** @@ -309,6 +310,12 @@ Nothing met the bar, so nothing was fixed, and four findings were deferred with Coverage was short in the same place as round 1 and will stay short: Codex refused within seconds on "You've hit your usage limit … try again at Sep 19th, 2026 10:23 AM", so this branch has had no Codex pass at any round and cannot get one before the quota resets. cubic ran and returned an empty issue list, and said itself that a clean pass over a diff that is overwhelmingly prose is weak evidence. The official `code-review` lens produced the round's only findings, and its own fork reviewed the wrong repository again despite a brief carrying the worktree's absolute path — noted with the evidence on `L-260912-b16eb4`, the canonical item, of which round 1's `L-260913-b0caa5` is a duplicate. +**2026-09-13 — the scaffold resumed on its own branch, and the founder's ruling on the non-empty refusal went in.** `feature/Scaffold-skill` took `dev` as a **merge** and never a rebase, because a rebase rewrites every commit and the merge gate checks that a recorded review pass's SHA is still an ancestor — twice on this campaign that has been the difference between a landable branch and one whose review record evaporated. Every conflict sat in a file the removal commit `e449b91` had narrowed to what shipped without the scaffold (`CHANGELOG.md`, `CLAUDE.md`, `README.md`, `docs/build-targets.md`, `docs/decisions.md`), plus the files `#18` and `#20` advanced after the cut (the integrate template and its three rendered trees, its test, this tracker and `design.md`); the resolution takes `dev`'s later text as the base everywhere and restores the scaffold halves into it, `pipelex-integrate`'s two forward references included — the "no project at all" branch offers `/pipelex-scaffold` again rather than stopping, since the skill it names now ships. + +`L-260912-724b71` was ruled on 2026-09-13 and is implemented: **a directory holding nothing but `.git` reads as empty and is written into; everything else goes on refusing.** The cruft list was declined in the same ruling, so the exception is written as one directory entry named `.git` and never as a predicate over ignorable files — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` still refuse, and the test pins them as refusing so that admitting one means rewording an asserted sentence. The rule that the skill never *offers* to clear anything is untouched: narrowing what counts as occupied is not permission to empty what is. The sites are the template's "Where" row and its failure-table stop row, with `scaffold-design.md` §2 and §7 amended and `docs/decisions.md` carrying the ruling. + +**What this branch deliberately did not do**, because the cut was for a decomposition problem and not a review problem: no second broad dogfood campaign. The earlier one produced fixes that became the next round's defect surface — round 1 returned thirty-four findings with fourteen inside the implementer's own fixes, and round 2 made the ratio worse. The only execution here was a targeted mutation battery on the new ruling test, restored and proven by sha256 rather than by `git status`. SC-9 and SC-10 stay unattempted and stay open for the reasons already recorded: SC-9 needs the founder's say-so to create a throwaway GitHub repository, and SC-10's empty-key branch is unreachable on a machine whose shell profile exports a key into every tool shell. + ## Where everything is - Brief: `wip/pipelex-integrate/brief.md`. Designs: `wip/pipelex-integrate/design.md` (integrate) and `wip/pipelex-integrate/scaffold-design.md` (scaffold). Upstream reading companion: `upstream-dependencies.md`. This tracker: `wip/pipelex-integrate/plan.md`. diff --git a/wip/pipelex-integrate/scaffold-design.md b/wip/pipelex-integrate/scaffold-design.md index f551398..0fb37d2 100644 --- a/wip/pipelex-integrate/scaffold-design.md +++ b/wip/pipelex-integrate/scaffold-design.md @@ -26,7 +26,7 @@ The rule from `pipelex-integrate` applies: a cheap, reliable signal decides; an | --- | --- | --- | | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, Remix, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty | +| **Where** | the directory the user named; **"here"** when the working directory is empty; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty — **except one whose only entry is `.git`, which reads as empty** (amended 2026-09-13, `L-260912-724b71`; see §7) | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | A starter clone that is already in the working directory and has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `piper` — is **branch A entered at step 3**: acquisition already happened, and the skill goes straight to running the clone's bootstrap. @@ -68,7 +68,7 @@ Automatic by default, with the plugin's usual rules: an explicit user signal win | Condition | The skill | | --- | --- | | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain | -| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, **and never offer to make room** — amended 2026-09-12 (Phase 3), after a run refused the directory and then offered to move the user's file aside and merge it back afterwards | +| The target directory exists and is not empty | STOP, ask for another; never delete, move or write into it, **and never offer to make room** — amended 2026-09-12 (Phase 3), after a run refused the directory and then offered to move the user's file aside and merge it back afterwards. **A directory holding nothing but `.git` reads as empty and is written into** — amended 2026-09-13 on Louis's ruling of `L-260912-724b71`; the exception is that one entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` go on refusing | | `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | | `gh` is absent or not authenticated | fall back to the local clone and say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list, say the template changed | @@ -92,6 +92,8 @@ Family wiring, each one sentence: `pipelex-integrate`'s step 1 offers `/pipelex- | `L-260906-aa5083` | `pipelex-starter-python` | informational | the Python starter lacks `AGENTS.md` and `add-method`; nothing in this skill waits on it | | *to file at release* | both starters | docs | the READMEs' "Use this template" sections should name `/pipelex-scaffold` as the agent front door beside the button and `/bootstrap` — filed when the skill ships, so the pointer never precedes the thing it points at | +**Amended 2026-09-13 — a directory holding nothing but `.git` reads as empty.** The non-empty refusal (§2's "Where" row, §7's second row) treated any existing entry as occupancy. Louis ruled `L-260912-724b71` on 2026-09-13: a lone `.git` is not occupancy and the directory is written into. Three things carried it. `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive at this skill; branch B itself runs `git init -b main` in the directory it is working in (§4 step 3), so the skill was refusing a state it produces one step later; and `references/initializers.md` already documents `uv init --package --no-workspace .` for exactly the empty-"here" case. **The cruft list was deliberately declined in the same ruling** — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` keep refusing until a real report names one — because a list that grows by guesswork is how this rule drifts back into the agent judging which of a user's files matter, which is what the original refusal was right to forbid. So the exception is written as one directory entry named `.git` and never as a predicate over ignorable files. **What the ruling does not touch:** a directory read as empty is never *cleared*, so §7's "never offer to make room" stands unchanged — narrowing what counts as occupied is not permission to empty what is. + ## Decision boxes for ratification | Box | Ruling | Ratified? | From 51395ef3ea7db5f678b119d3b6ecbd269abbfe6a Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 15:11:03 +0200 Subject: [PATCH 19/21] Make the scaffold's recipes keep the guarantees its references state Review round 1 read the skill as a document to execute rather than to proofread, and found the places where the one line an agent runs disagreed with the reference it is told to read. The named-framework recipe still prescribed a bare `uv init --package `, so it appended a `[tool.uv.workspace]` table to the user's own pyproject.toml -- the exact hazard the reference documents in bold as fixed everywhere. The key write was an unconditional append, which on the fresh-clone shortcut landed a second PIPELEX_API_KEY after one the user had filled; dotenv resolves a repeated name to the later line, so a stale exported value replaced a working key while the report claimed theirs was kept. The pristine commit carried its `-- .` pathspec on the staging but not on the commit, and a bare `git commit` commits the whole index, so a user's work staged elsewhere in an enclosing repository rode along under this skill's message. `gh repo create --clone` takes no destination and clones into ./, which every later step addressed as . The minimal TypeScript recipe opened with `mkdir `, which aborts its own chain in the "here" directory the skill permits, and its default resolution is the one the same file calls broken. The fresh-clone shortcut and the template-checkout stop shared a detection signal and prescribed opposite actions. The discipline test pinned the --no-workspace rule in the reference and never in the skill body, which is how the half-application shipped; it now runs on the template and all three renders, and was shown to fail when either guard is struck. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + .../skills/pipelex-scaffold/SKILL.md | 16 ++++++------ .../references/initializers.md | 4 +-- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 16 ++++++------ .../references/initializers.md | 4 +-- pipelex/skills/pipelex-scaffold/SKILL.md | 16 ++++++------ .../references/initializers.md | 4 +-- .../references/initializers.md | 4 +-- templates/skills/pipelex-scaffold/SKILL.md.j2 | 16 ++++++------ tests/unit/test_pipelex_scaffold_skill.py | 25 +++++++++++++++++++ .../scaffold-review-deferrals.md | 24 ++++++++++++++++++ 11 files changed, 90 insertions(+), 40 deletions(-) create mode 100644 wip/pipelex-integrate/scaffold-review-deferrals.md diff --git a/CHANGELOG.md b/CHANGELOG.md index b5858cf..79c3194 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ - **The reference-copy test asserted that `shutil` copies bytes.** It built a fresh tree into a temp directory and compared it against the source it had just been built from, so it could not observe the only failure that matters — a stale committed copy under `pipelex/`, `pipelex-codex/` or `pipelex-vibe/`, which is what a user installs. It now compares the committed copies themselves, and its three-target parametrization exercises three different trees instead of three identical calls. - **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. - **A directory holding nothing but `.git` is somewhere `pipelex-scaffold` will build.** The skill refused any target directory that was not empty, which made it refuse the state it produces itself one step later: `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive, and the skill's own initializer branch runs `git init -b main` in the directory it is working in. A lone `.git` now reads as empty. Everything else goes on refusing — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included — because the exception is one directory entry named in full and not a class of files the agent may decide to overlook, which is the judgement the refusal exists to prevent. Nothing about clearing changed: a directory that is not empty is still never emptied, and making room is still never offered. +- **The scaffold's recipes now match the guarantees its own references state.** A review round over the skill as an executable document, rather than over its prose, found the places where the two disagreed. The named-framework recipe still prescribed a bare `uv init --package ` while `references/initializers.md` said in bold that `--no-workspace` is on every `uv init` above — so the one line an agent executes appended a `[tool.uv.workspace]` table to the user's own `pyproject.toml`, the exact hazard the reference documents, and the discipline test pinned the reference alone and never the skill body. The key write was an unconditional append, which on the fresh-clone shortcut put a second `PIPELEX_API_KEY` after one the user had already filled; every dotenv reader resolves a repeated name to the later line, so a stale exported value silently replaced a working key while the report said theirs was kept — it is gated on a file-side presence test now. `gh repo create --clone` takes no destination and clones into `./`, which every later step addressed as ``. The pristine commit carried its `-- .` pathspec on the staging but not on the commit, and a bare `git commit` commits the whole index, so anything the user had staged in an enclosing repository rode along under this skill's message — both commits carry it now, and the guard's description says what it actually guarantees. The minimal TypeScript recipe opened with `mkdir `, which aborts its own `&&` chain when `` is the "here" directory the skill explicitly permits, and its default resolution is the one the same file calls broken, now flagged where the command is chosen rather than only below it. The fresh-clone shortcut and the template-checkout stop shared a detection signal and prescribed opposite actions; the shortcut now names the `origin` that separates a copy from the template itself. - **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. - **The file factory no longer touches the user's own project.** Every `uv run` line in the skill and its references now passes `--no-project`. Without it `uv run` walks up from the working directory, finds the nearest project, and *syncs* it — so rendering a test PDF inside a checkout created a `.venv/` and wrote a `uv.lock` the user never asked for, in a repository the skill has no business modifying. `--no-project` resolves the ephemeral `--with` packages against nothing at all, which is what the recipes always meant; a test asserts every shipped runner line carries the flag. diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 421f729..9959b89 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -26,7 +26,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing | **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | -**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. [references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. @@ -69,14 +69,14 @@ The clone's `.git` is removed on purpose: it is the template's history and remot gh repo create / --template Pipelex/ --private --clone ``` -Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. +`--clone` takes no destination argument: it clones into `./` under the current working directory, so for this form `` **is** `` — either choose the repository name to match the directory the "Where" question settled, or rebind `` to `./` before step 4, because every step after this one addresses `` literally. Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -98,7 +98,7 @@ cp -n /.env.example /.env # Python: python-dotenv reads .env Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. -**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed. **Append only when the file does not already carry a key**, which is the other half of the `-n` above: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local || printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. On the fresh-clone shortcut the file is one the user may have filled themselves, and an unconditional append puts a second assignment *after* theirs — every dotenv reader resolves a repeated name to the later line, so their working key is silently replaced by whatever the shell happened to export, while the report tells them you kept theirs. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. @@ -114,7 +114,7 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package --no-workspace ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. @@ -126,10 +126,10 @@ Nothing beyond what the initializer writes is authored by this skill: no example **Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A -- . && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" -- . ``` -The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. +The `-- .` pathspec is the second half of the guard, and it is on **both** commands for a reason: `add -A -- .` bounds what this command stages, but a bare `git commit` then commits the *whole index*, so anything the user had staged elsewhere in an enclosing repository before the session goes into the commit under this skill's message. With the pathspec on the commit too, the staging and the commit are both held to `` and below even when the repository turns out to be an enclosing one, and the user's own staged work is left staged and uncommitted where they put it. **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. @@ -168,7 +168,7 @@ Two lines are easy to forget and matter: | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | | you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | -| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory. This is what separates it from the fresh-clone shortcut, which is a copy of the template and carries somebody else's `origin` or none | ## Reference diff --git a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md index e3631ee..9610cb4 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md @@ -25,7 +25,7 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where `src/` lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| **Minimal (the default when no framework is named)** | `mkdir -p && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` — **but read the resolution note below first: prefer `--module esnext --moduleResolution bundler` unless the user actually wants Node's own resolution**, because `nodenext` is the shape that meets the emitter defect | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | | Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | @@ -44,5 +44,5 @@ After the initializer: `git init -b main` only if it did not initialize a reposi ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. -- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- `.env.example` and `.env` written, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. Both land *after* the pristine commit, which holds the initializer's output as it came, so they stay untracked for the user to review and commit — the same posture branch A leaves the bootstrap's edits in. - Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index b44ee8e..38d7b29 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -26,7 +26,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing | **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | -**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. [references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. @@ -69,14 +69,14 @@ The clone's `.git` is removed on purpose: it is the template's history and remot gh repo create / --template Pipelex/ --private --clone ``` -Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. +`--clone` takes no destination argument: it clones into `./` under the current working directory, so for this form `` **is** `` — either choose the repository name to match the directory the "Where" question settled, or rebind `` to `./` before step 4, because every step after this one addresses `` literally. Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -98,7 +98,7 @@ cp -n /.env.example /.env # Python: python-dotenv reads .env Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. -**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed. **Append only when the file does not already carry a key**, which is the other half of the `-n` above: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local || printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. On the fresh-clone shortcut the file is one the user may have filled themselves, and an unconditional append puts a second assignment *after* theirs — every dotenv reader resolves a repeated name to the later line, so their working key is silently replaced by whatever the shell happened to export, while the report tells them you kept theirs. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. @@ -114,7 +114,7 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package --no-workspace ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. @@ -126,10 +126,10 @@ Nothing beyond what the initializer writes is authored by this skill: no example **Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A -- . && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" -- . ``` -The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. +The `-- .` pathspec is the second half of the guard, and it is on **both** commands for a reason: `add -A -- .` bounds what this command stages, but a bare `git commit` then commits the *whole index*, so anything the user had staged elsewhere in an enclosing repository before the session goes into the commit under this skill's message. With the pathspec on the commit too, the staging and the commit are both held to `` and below even when the repository turns out to be an enclosing one, and the user's own staged work is left staged and uncommitted where they put it. **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. @@ -168,7 +168,7 @@ Two lines are easy to forget and matter: | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | | you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | -| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory. This is what separates it from the fresh-clone shortcut, which is a copy of the template and carries somebody else's `origin` or none | ## Reference diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md index e3631ee..9610cb4 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md @@ -25,7 +25,7 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where `src/` lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| **Minimal (the default when no framework is named)** | `mkdir -p && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` — **but read the resolution note below first: prefer `--module esnext --moduleResolution bundler` unless the user actually wants Node's own resolution**, because `nodenext` is the shape that meets the emitter defect | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | | Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | @@ -44,5 +44,5 @@ After the initializer: `git init -b main` only if it did not initialize a reposi ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. -- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- `.env.example` and `.env` written, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. Both land *after* the pristine commit, which holds the initializer's output as it came, so they stay untracked for the user to review and commit — the same posture branch A leaves the bootstrap's edits in. - Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index e5dc3c3..ba8cb19 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -33,7 +33,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing | **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | -**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. [references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. @@ -76,14 +76,14 @@ The clone's `.git` is removed on purpose: it is the template's history and remot gh repo create / --template Pipelex/ --private --clone ``` -Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. +`--clone` takes no destination argument: it clones into `./` under the current working directory, so for this form `` **is** `` — either choose the repository name to match the directory the "Where" question settled, or rebind `` to `./` before step 4, because every step after this one addresses `` literally. Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -105,7 +105,7 @@ cp -n /.env.example /.env # Python: python-dotenv reads .env Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. -**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed. **Append only when the file does not already carry a key**, which is the other half of the `-n` above: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local || printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. On the fresh-clone shortcut the file is one the user may have filled themselves, and an unconditional append puts a second assignment *after* theirs — every dotenv reader resolves a repeated name to the later line, so their working key is silently replaced by whatever the shell happened to export, while the report tells them you kept theirs. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. @@ -121,7 +121,7 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package --no-workspace ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run (typing `! ` in the prompt runs it inside this session), and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. @@ -133,10 +133,10 @@ Nothing beyond what the initializer writes is authored by this skill: no example **Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A -- . && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" -- . ``` -The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. +The `-- .` pathspec is the second half of the guard, and it is on **both** commands for a reason: `add -A -- .` bounds what this command stages, but a bare `git commit` then commits the *whole index*, so anything the user had staged elsewhere in an enclosing repository before the session goes into the commit under this skill's message. With the pathspec on the commit too, the staging and the commit are both held to `` and below even when the repository turns out to be an enclosing one, and the user's own staged work is left staged and uncommitted where they put it. **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. @@ -175,7 +175,7 @@ Two lines are easy to forget and matter: | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | | you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | -| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory. This is what separates it from the fresh-clone shortcut, which is a copy of the template and carries somebody else's `origin` or none | ## Reference diff --git a/pipelex/skills/pipelex-scaffold/references/initializers.md b/pipelex/skills/pipelex-scaffold/references/initializers.md index e3631ee..9610cb4 100644 --- a/pipelex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex/skills/pipelex-scaffold/references/initializers.md @@ -25,7 +25,7 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where `src/` lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| **Minimal (the default when no framework is named)** | `mkdir -p && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` — **but read the resolution note below first: prefer `--module esnext --moduleResolution bundler` unless the user actually wants Node's own resolution**, because `nodenext` is the shape that meets the emitter defect | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | | Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | @@ -44,5 +44,5 @@ After the initializer: `git init -b main` only if it did not initialize a reposi ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. -- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- `.env.example` and `.env` written, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. Both land *after* the pristine commit, which holds the initializer's output as it came, so they stay untracked for the user to review and commit — the same posture branch A leaves the bootstrap's edits in. - Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/skills/pipelex-scaffold/references/initializers.md b/skills/pipelex-scaffold/references/initializers.md index e3631ee..9610cb4 100644 --- a/skills/pipelex-scaffold/references/initializers.md +++ b/skills/pipelex-scaffold/references/initializers.md @@ -25,7 +25,7 @@ After the initializer: `git init -b main` only if it did not initialize a reposi | Want | Command | `git init`? | Where `src/` lands | |---|---|---|---| -| **Minimal (the default when no framework is named)** | `mkdir && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | +| **Minimal (the default when no framework is named)** | `mkdir -p && cd && npm init -y && npm install --save-dev typescript @types/node && npx tsc --init --strict --module nodenext --target es2022 --rootDir src --outDir dist` then set `"type": "module"` in `package.json` — **but read the resolution note below first: prefer `--module esnext --moduleResolution bundler` unless the user actually wants Node's own resolution**, because `nodenext` is the shape that meets the emitter defect | no | `src/` (create it); `/pipelex-integrate` puts the generated tree under `src/generated/` | | Next.js app (the JS starter's shape, without the starter) | `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes` | yes (`--disable-git` to skip) | `src/app/`; generated tree under `src/generated/` | | Vite + React | `npm create vite@latest -- --template react-ts` then `(cd && npm install)` | no | `src/` | | Hono server | `npm create hono@latest -- --template nodejs --pm npm --install` | no | `src/` | @@ -44,5 +44,5 @@ After the initializer: `git init -b main` only if it did not initialize a reposi ## What every branch-B project shares afterwards - One commit, the pristine scaffold, so the user's first real change is a clean diff. -- `.env.example` committed, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. +- `.env.example` and `.env` written, `.env` ignored, `PIPELEX_API_KEY` filled only from the shell environment. Both land *after* the pristine commit, which holds the initializer's output as it came, so they stay untracked for the user to review and commit — the same posture branch A leaves the bootstrap's edits in. - Nothing else Pipelex-shaped: the SDK dependency, the `methods/` directory and the generated tree arrive with the first `/pipelex-integrate`. diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index 3a63079..062ce8e 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -26,7 +26,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing | **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | -**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. +**The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. [references/starters.md](references/starters.md) compares the two starters and carries every command below; [references/initializers.md](references/initializers.md) carries the initializers. @@ -69,14 +69,14 @@ The clone's `.git` is removed on purpose: it is the template's history and remot gh repo create / --template Pipelex/ --private --clone ``` -Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. +`--clone` takes no destination argument: it clones into `./` under the current working directory, so for this form `` **is** `` — either choose the repository name to match the directory the "Where" question settled, or rebind `` to `./` before step 4, because every step after this one addresses `` literally. Visibility is the user's call: ask, default `--private`. Creating a repository on GitHub is outward-facing, so **state the exact command and confirm before running it**, in every mode. GitHub writes the initial commit itself — skip step 3 and continue at step 4. If `gh` is absent or not authenticated, fall back to the local clone and say the repository can be created later with `gh repo create --source .`. Both forms take the template's default-branch head. Do not offer a release tag unless the user asks for one. ### Step 3: Commit the pristine template — exactly once ```bash -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. @@ -98,7 +98,7 @@ cp -n /.env.example /.env # Python: python-dotenv reads .env Fill `PIPELEX_API_KEY` **from the shell environment when it is set there**, and leave it empty otherwise, telling the user where a key comes from (`app.pipelex.com`) and that this file is where it goes. **Never print a key, and never ask for one in the conversation.** Test for it without printing it — `[ -n "${PIPELEX_API_KEY:-}" ] && echo set || echo unset`. -**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed: `printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. +**The value moves only through a shell that expands the variable itself, and never through you.** A redirection is safe precisely because the shell does the expanding and only the variable's *name* is transcribed. **Append only when the file does not already carry a key**, which is the other half of the `-n` above: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local || printf 'PIPELEX_API_KEY=%s\n' "$PIPELEX_API_KEY" >> /.env.local`. On the fresh-clone shortcut the file is one the user may have filled themselves, and an unconditional append puts a second assignment *after* theirs — every dotenv reader resolves a repeated name to the later line, so their working key is silently replaced by whatever the shell happened to export, while the report tells them you kept theirs. A file-editing tool is the one form that cannot be made safe, whatever it is called — it takes a **literal** string, so you would have to know the value to pass it, and a tool call's parameters are the transcript. So: no file-editing tool on a line carrying the key, no `env | grep PIPELEX`, no `echo $PIPELEX_API_KEY`, no command substitution in a message, and **no reading the env file back** once written — `cat .env.local`, a `grep` over it, or opening it to check your work is the reflex after writing and the first move when a later step fails, and it puts the key in the transcript just as surely. To confirm the write landed, test the file the same way you tested the environment: `grep -q '^PIPELEX_API_KEY=.\+' /.env.local && echo filled || echo empty`. A key in the transcript is a key to rotate, and it is not yours to spend. A value that passed the test is copied verbatim and never inspected, so say in the report that it was taken from the environment **and not validated** — a placeholder someone exported once passes a presence test and fails the first run, and this skill never calls the API, so it cannot tell the difference. `PIPELEX_BASE_URL` stays as the example ships it. Confirm the file is gitignored before writing a key into it — both starters ignore it, but check. @@ -114,7 +114,7 @@ As in branch A, for the language chosen. ### Step 2: Run the initializer — never assemble by hand -- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. +- **A named framework** uses its documented initializer with its non-interactive flags: `npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes`, where the `--` is what passes the flags to the initializer instead of to npm and without it `create-next-app` prompts; `uv init --package --no-workspace ` then `uv add "fastapi[standard]"` **from inside ``**, because `uv add` writes to whatever project its working directory resolves to and from the parent that is the user's, not the new one; and so on — [references/initializers.md](references/initializers.md) carries the common ones. An initializer that only runs interactively is handed to the user to run{% if platform == "claude" %} (typing `! ` in the prompt runs it inside this session){% endif %}, and you resume when it is done. - **No framework named** takes the language's own minimal initializer: Python → `uv init --package --no-workspace `, which gives the import package `/pipelex-integrate` wants and a console-script entry; TypeScript → `npm init -y`, then `npm install --save-dev typescript @types/node` and `npx tsc --init` with strict mode, ES modules and a `src/` root. **Read [references/initializers.md](references/initializers.md) before running either**, and not only for the flags: it is where the two costs of the TypeScript default are written down — it is the resolution that meets the emitter's extensionless-import defect, and `tsc --init` switches off the `@types/node` the line before it installed — and both are the kind of thing the integration, not the scaffold, gets blamed for. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. @@ -126,10 +126,10 @@ Nothing beyond what the initializer writes is authored by this skill: no example **Then confirm there is a `.gitignore` covering the dependency tree and the build output**: `npm init -y` and `tsc --init` write none, so the minimal TypeScript recipe — and the Express and library recipes built on it — reach this step with a populated `node_modules/` and nothing excluding it, and the staging below would commit the whole dependency tree into the one commit that is supposed to be a readable baseline. Write `node_modules/`, `dist/` and `.env` into a `.gitignore` first where the initializer left none, and read back what is staged — `git -C diff --cached --name-only` over the paths themselves, not a `--stat | tail -1` whose count cannot tell a correct scaffold from a swept-up worktree — before committing. Then the one commit, for the same reason as branch A: ```bash -git -C add -A -- . && git -C commit -m "Scaffold project" +git -C add -A -- . && git -C commit -m "Scaffold project" -- . ``` -The `-- .` pathspec is the second half of the guard: it holds the staging to `` and below even when the repository turns out to be an enclosing one, so the failure mode degrades from committing the user's work to committing into the wrong repository. +The `-- .` pathspec is the second half of the guard, and it is on **both** commands for a reason: `add -A -- .` bounds what this command stages, but a bare `git commit` then commits the *whole index*, so anything the user had staged elsewhere in an enclosing repository before the session goes into the commit under this skill's message. With the pathspec on the commit too, the staging and the commit are both held to `` and below even when the repository turns out to be an enclosing one, and the user's own staged work is left staged and uncommitted where they put it. **An initializer that commits as well as `git init`s has already made this commit.** `create-next-app` is one: it runs `git init`, stages everything and commits, so the tree is clean and the command above stops with `nothing to commit` — which is the initializer having done the job, not a failure of it. Take that commit as the pristine one, exactly as branch A takes GitHub's, and name it and its message in the report. Never force a second empty commit on top of it. @@ -168,7 +168,7 @@ Two lines are easy to forget and matter: | An initializer is interactive with no non-interactive form | hand the command to the user to run in the session; resume after | | `PIPELEX_API_KEY` is not in the shell environment | leave the value empty in the env file; say where a key comes from and where it goes; never ask for it in the conversation | | you need to know whether a key is set | test it without printing it (`[ -n "${PIPELEX_API_KEY:-}" ] && echo set`); never `env \| grep PIPELEX`, never echo the value, never move it with a file-editing tool, never read the env file back — a key in the transcript is a key to rotate | -| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory | +| The working directory is the template's own checkout (its `origin` remote points at `Pipelex/pipelex-starter-…`) | STOP: this is the template, not a copy of it — acquire a copy in another directory. This is what separates it from the fresh-clone shortcut, which is a copy of the template and carries somebody else's `origin` or none | ## Reference diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 5e16999..0da8bb2 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -87,6 +87,26 @@ def test_the_rules_are_stated(self) -> None: for rule in self.RULES: assert rule in body, f"missing rule: {rule}" + def test_the_skill_body_carries_the_guards_the_references_state(self) -> None: + """Two guards were stated in `references/initializers.md` and pinned only there. + + The skill body is the document an agent actually executes; a reference it is told to read + is a second hop. Round 2 added `--no-workspace` to every `uv init` in the reference and + asserted it there, while the named-framework recipe *in the skill* kept the bare form — so + the reference said "on every `uv init` above" and the executable line one hop away + contradicted it. These assertions run on the template and on all three renders, because a + rule that holds only in `templates/` is a rule no user reads. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + # A bare `uv init --package ` appends a [tool.uv.workspace] table to the USER'S + # own pyproject.toml and leaves the new project without its own lock. + assert "uv init --package --no-workspace " in body + assert "uv init --package " not in body, "a bare uv init absorbs into the parent workspace" + # The append is gated: on the fresh-clone shortcut the env file may be one the user + # filled, and a second assignment after theirs is the one dotenv resolves to. + assert "grep -q '^PIPELEX_API_KEY=.\\+' /.env.local || printf" in body + assert ">> /.env.local`.\n" not in body, "an ungated append shadows a key the user already filled" + def test_fresh_clone_shortcut_and_template_checkout_stop(self) -> None: body = self.scaffold assert "**The fresh-clone shortcut.**" in body @@ -196,6 +216,11 @@ def test_the_pristine_commit_cannot_reach_an_enclosing_repository(self) -> None: assert "never infer it from which initializer ran" in body # Both pristine commits carry the pathspec, so the staging cannot escape . assert body.count("git -C add -A -- .") == 2 + # `add -A -- .` bounds only the staging; a bare `git commit` commits the whole index, + # so anything the user had staged in an enclosing repo would ride along. Both commits + # carry the pathspec, which leaves their staged work staged. + assert 'commit -m "Start from Pipelex/ ()" -- .' in body + assert 'commit -m "Scaffold project" -- .' in body assert "git -C add -A &&" not in body # The read-back must name paths; a --stat count cannot tell a scaffold from a swept worktree. assert "git -C diff --cached --name-only" in body diff --git a/wip/pipelex-integrate/scaffold-review-deferrals.md b/wip/pipelex-integrate/scaffold-review-deferrals.md new file mode 100644 index 0000000..4026297 --- /dev/null +++ b/wip/pipelex-integrate/scaffold-review-deferrals.md @@ -0,0 +1,24 @@ +--- +status: active +item: L-260906-8ac105 +--- + +# `pipelex-scaffold` — findings review round 1 confirmed and did not fix + +Round 1 of `/rev` on `pipelex-plugins#21` (branch `feature/Scaffold-skill`, 2026-09-13) ran cubic, the Codex review and adversarial passes, and the official `code-review`. What it fixed is in the pull request and the changelog. This file is the trace for what it confirmed and deliberately left, so none of it is a finding that merely evaporated. + +Everything below was read and verified in the tree — none of it rests on a reviewer's word alone. Each entry says why it was not fixed, which is always either "real but not important" or "the fix is a decision, not an edit". + +## Carried elsewhere + +- **A directory holding only `.git` is one branch A cannot clone into** — the sharpest finding of the round, raised by three reviewers. It needs a ruling because every fix changes behaviour the founder ruled on, so it is its own decision item: `L-260913-f28d9d`. +- **A branch-B project writes `.env` that nothing loads into the process** — rediscovered by the Codex review, already open as `L-260912-059765`. No new trace needed. + +## Deferred here + +- **The report says "this skill made exactly one commit" on paths where it made none.** The GitHub form (`gh repo create --template`) and a self-committing initializer (`create-next-app`) both produce the pristine commit themselves, and the skill correctly says to adopt it rather than force a second one. The unconditional report line at the template's "The report" section then misstates provenance. Real, and cosmetic in effect: the same step already tells the agent to name the commit and its message, so the user sees the truth beside the wrong sentence. Worth one clause next time this file is opened. +- **Step 1 checks a version floor only step 2 can read.** Branch A's prerequisites check Node against "the floor the starter's `package.json` `engines` field names", but step 1 runs before the clone exists, so the authoritative value is unreadable and the check necessarily runs against the hardcoded `22.12` the text itself hedges as "at writing". Harmless while the floor is stable; it becomes wrong silently when a starter raises it. +- **"It is the plugin's third MCP-free skill" is a hardcoded count**, in `CHANGELOG.md` and `docs/decisions.md`. The workspace guide forbids counts in docs because they go stale silently. It is accurate today and the phrasing has precedent already on `dev`, so changing it here would be a lone deviation; it wants doing across the repo at once or not at all. +- **Branch B's ownership sentence is broader than the steps beneath it.** "Nothing beyond what the initializer writes is authored by this skill" is glossed immediately with "no example code, no folder layout of its own, no opinion the framework did not ship", which scopes it to project shape — but steps 3 and 4 then author a `.gitignore` and the env pair. An agent following the explicit imperatives is not actually misled, which is why this is a wording imprecision rather than a defect. +- **The minimal TypeScript recipe still defaults to `--module nodenext`.** Round 1 annotated the default row so the caveat is visible where the command is chosen, rather than twelve lines below it, but it did not change the command. Making the bundler form the default is a recipe change that should be executed before it ships — this campaign's own history is that recipes composed inside a review round became the next round's defects. +- **`.env.example` ships an empty `PIPELEX_API_KEY=`, so the gated append can still leave two assignments.** Round 1 closed the defect that mattered — an append landing *after* a key the user had already filled, which every dotenv reader resolves to the later line. What remains is only the placeholder case, where the later line is the one the skill intends and the value is correct. A user editing the first line and seeing nothing change is the cost; rewriting in place instead of appending would need a `sed -i` whose BSD/GNU spelling differs, which is not worth trading a portability trap for a tidiness gain. From 3a57f7d73364348caab741f6510416ce05c87ac6 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 15:32:39 +0200 Subject: [PATCH 20/21] Let branch A acquire into the directory the lone-.git rule admits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A directory whose only entry is `.git` reads as empty, which the initializer branch can honour and the starter branch could not: its first command on the chosen directory is `git clone`, and git refuses a destination that already holds a `.git`. The skill accepted a directory it then could not populate. Branch A now clones into a mktemp path beside the target, discards the template's history there before anything moves, re-reads the directory immediately before the copy, and carries the contents in with `cp -R "$tmp"/. /` so that the entries beginning with a dot come too — reaching the end state the default recipe reaches, with the user's repository, branch, history and remote left standing. The recipe is executed rather than read. A new test extracts it from the skill and runs it against a directory that does not exist, an empty one, one holding a real repository with commits on a named branch, one holding `.git` beside a file of the user's, and one holding only the cruft the earlier ruling declined to exempt — reading back the surviving commits, branch and reflog, and logging every path any `rm` is pointed at to prove none lies under the chosen directory. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- docs/decisions.md | 1 + .../skills/pipelex-scaffold/SKILL.md | 31 +- .../pipelex-scaffold/references/starters.md | 15 + pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 31 +- .../pipelex-scaffold/references/starters.md | 15 + pipelex/skills/pipelex-scaffold/SKILL.md | 31 +- .../pipelex-scaffold/references/starters.md | 15 + .../pipelex-scaffold/references/starters.md | 15 + templates/skills/pipelex-scaffold/SKILL.md.j2 | 31 +- tests/unit/test_pipelex_scaffold_skill.py | 370 ++++++++++++++++++ wip/pipelex-integrate/plan.md | 1 + wip/pipelex-integrate/scaffold-design.md | 3 + .../scaffold-review-deferrals.md | 2 +- 14 files changed, 545 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 79c3194..55a100f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,7 @@ - **The helper budget said two and listed one.** `pipelex-integrate` announced "at most two shared helpers", named the client factory, and closed with "and nothing else" — residue of the wire-output helper that was struck for being lossy and is forbidden by name a few lines below. A stale count would be cosmetic; this one was written permission to create the single thing the campaign removed. It is one helper, in the skill, in both places it was stated, and in `docs/decisions.md`. - **The reference-copy test asserted that `shutil` copies bytes.** It built a fresh tree into a temp directory and compared it against the source it had just been built from, so it could not observe the only failure that matters — a stale committed copy under `pipelex/`, `pipelex-codex/` or `pipelex-vibe/`, which is what a user installs. It now compares the committed copies themselves, and its three-target parametrization exercises three different trees instead of three identical calls. - **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. -- **A directory holding nothing but `.git` is somewhere `pipelex-scaffold` will build.** The skill refused any target directory that was not empty, which made it refuse the state it produces itself one step later: `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive, and the skill's own initializer branch runs `git init -b main` in the directory it is working in. A lone `.git` now reads as empty. Everything else goes on refusing — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included — because the exception is one directory entry named in full and not a class of files the agent may decide to overlook, which is the judgement the refusal exists to prevent. Nothing about clearing changed: a directory that is not empty is still never emptied, and making room is still never offered. +- **A directory holding nothing but `.git` is somewhere `pipelex-scaffold` will build.** The skill refused any target directory that was not empty, which made it refuse the state it produces itself one step later: `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive, and the skill's own initializer branch runs `git init -b main` in the directory it is working in. A lone `.git` now reads as empty, and **both branches serve it**: the initializer runs in the directory as it stands, while the starter branch — which `git clone` cannot aim at a directory already holding a `.git` — acquires into a temporary path beside it, discards the template's history while the clone is still its own, and copies the template in, dotfiles included, so the repository the user made goes on standing with their branch, their history and their remote. Everything else goes on refusing — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included — because the exception is one directory entry named in full and not a class of files the agent may decide to overlook, which is the judgement the refusal exists to prevent. Nothing about clearing changed: a directory that is not empty is still never emptied, and making room is still never offered. - **The scaffold's recipes now match the guarantees its own references state.** A review round over the skill as an executable document, rather than over its prose, found the places where the two disagreed. The named-framework recipe still prescribed a bare `uv init --package ` while `references/initializers.md` said in bold that `--no-workspace` is on every `uv init` above — so the one line an agent executes appended a `[tool.uv.workspace]` table to the user's own `pyproject.toml`, the exact hazard the reference documents, and the discipline test pinned the reference alone and never the skill body. The key write was an unconditional append, which on the fresh-clone shortcut put a second `PIPELEX_API_KEY` after one the user had already filled; every dotenv reader resolves a repeated name to the later line, so a stale exported value silently replaced a working key while the report said theirs was kept — it is gated on a file-side presence test now. `gh repo create --clone` takes no destination and clones into `./`, which every later step addressed as ``. The pristine commit carried its `-- .` pathspec on the staging but not on the commit, and a bare `git commit` commits the whole index, so anything the user had staged in an enclosing repository rode along under this skill's message — both commits carry it now, and the guard's description says what it actually guarantees. The minimal TypeScript recipe opened with `mkdir `, which aborts its own `&&` chain when `` is the "here" directory the skill explicitly permits, and its default resolution is the one the same file calls broken, now flagged where the command is chosen rather than only below it. The fresh-clone shortcut and the template-checkout stop shared a detection signal and prescribed opposite actions; the shortcut now names the `origin` that separates a copy from the template itself. - **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. diff --git a/docs/decisions.md b/docs/decisions.md index ba156b7..5312032 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -183,6 +183,7 @@ Both skills were run as cold headless sessions against a local `pipelex-mcp` bui - **Containment is read before the first write, not after an error.** A path inside the workshop's working directory is legal wherever it points, so a harness launched beside the project accepts a write into the wrong tree; a run that hit this moved the tree across afterwards and left the project with a refresh that fails the same way every time. The skill now reads the path from the workshop's working directory first, and never moves a tree into place. - **A file the skill does not own is never cleared, and never offered for clearing.** Integrate met a hand-written file at an artifact path and offered to delete it; scaffold met a non-empty target directory and offered to move its contents aside and merge them back. The answer in both cases is another directory. - **A directory holding nothing but `.git` reads as empty; everything else still refuses.** (Ruled 2026-09-13, `L-260912-724b71`.) The non-empty refusal treated any existing entry as occupancy, which made the skill refuse a state it produces itself: `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, branch B runs `git init -b main` in the directory it is working in one step later, and `references/initializers.md` already documents `uv init --package --no-workspace .` for the empty-"here" case. A lone `.git` is now read as empty. **The cruft list was declined in the same ruling** — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` go on refusing until a real report names one — because a list that grows by guesswork is how this rule drifts back into the agent judging which of a user's files matter, which is the thing the refusal exists to forbid; so the exception is written as one directory entry by name and never as a predicate over ignorable files. The separate rule that the skill never *offers* to clear anything is untouched: a directory read as empty is never cleared either way. +- **Branch A acquires beside a directory that already holds a repository, so the lone-`.git` exception means one thing on both branches.** (Ruled 2026-09-13, `L-260913-f28d9d`.) The ruling above is implementable on the initializer branch, where `uv init` accepts such a directory, and unimplementable on the starter branch, whose first command on the chosen directory is `git clone` — which refuses any destination already holding a `.git`. The skill therefore admitted a directory it could not populate, and the founder's own motivating case dead-ended for anyone who wanted the opinionated starter rather than the initializer. **Scoping the allowance to the initializer branch was rejected**: the exception exists because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, which is not specific to initializers, so that user has exactly the same claim on the starter and would have met the refusal the exception was written to remove. Branch A now clones into a `mktemp` path beside the directory, discards the template's history there, re-reads the directory and copies the template in. Three properties make it safe and each is executed rather than asserted, in `tests/unit/test_pipelex_scaffold_skill.py`: **no `rm -rf` ever addresses a path under the chosen directory**, because the discard is spent on the temporary path before anything moves; **`cp -R "$tmp"/. /` carries the entries beginning with a dot**, which `mv "$tmp"/*` drops while exiting `0`; and **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so a collision is impossible rather than unlikely and anything else stops the run with nothing copied. The end state is the one the default recipe reaches — the template in place, no template history, no Pipelex remote — reached by a different route, and nothing is initialised because the repository is already the user's. - **A runtime behind a version manager is not a missing toolchain.** Scaffold, on a `PATH` without `node`, found the machine's `nvm` and carried on — which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, says which it used, and stops only when no runtime can be reached. - **One upstream defect is named rather than worked around.** The `ts-zod` emitter writes `binder.ts`'s sibling import with no file extension, which a plain Node ESM project rejects at type-check and at runtime while a bundler resolution accepts — which is why the JS starter never met it. Filed as `L-260912-857a5a` against `pipelex`. The tree is stamped and hashed, so the skill reports it, never patches the file, never drops the tree from the type checker, and leaves a change of `moduleResolution` to the user. diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 9959b89..1c6f2f3 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -63,6 +63,29 @@ The `|| exit` on the clone is not decoration: the line below it deletes a `.git` The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. +**Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. + +**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. + +**The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. + +**The chain goes out as one command.** The guards hold only inside one shell — the same reason the `|| exit` above is load-bearing, stated in full in [references/starters.md](references/starters.md) — and split across separate calls this one loses its cleanup too, leaving the temporary directory beside the user's project with no line left to remove it. + +**Nothing is initialized here.** The default recipe ends `git init -b main` because it has just deleted the only repository at that path. This one ends on the user's repository, their branch and their remote, which is the whole point of taking the long way round. + **GitHub, on request.** When the user asked for a repository on GitHub: ```bash @@ -79,7 +102,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. ### Step 4: Run the clone's own bootstrap @@ -160,8 +183,8 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | -| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse. On branch A that directory is served by the acquisition beside it (Step 2) and never by a `git clone` into it, which git refuses outright | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory, and on the acquisition beside an existing repository the chain removes its own temporary path and copies nothing | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | diff --git a/pipelex-codex/skills/pipelex-scaffold/references/starters.md b/pipelex-codex/skills/pipelex-scaffold/references/starters.md index c305e13..7f3c08a 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/starters.md @@ -35,6 +35,21 @@ The version comes from `package.json` (`"version"`) on JS and from `pyproject.to **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. +Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index 38d7b29..6821ac7 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -63,6 +63,29 @@ The `|| exit` on the clone is not decoration: the line below it deletes a `.git` The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. +**Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. + +**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. + +**The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. + +**The chain goes out as one command.** The guards hold only inside one shell — the same reason the `|| exit` above is load-bearing, stated in full in [references/starters.md](references/starters.md) — and split across separate calls this one loses its cleanup too, leaving the temporary directory beside the user's project with no line left to remove it. + +**Nothing is initialized here.** The default recipe ends `git init -b main` because it has just deleted the only repository at that path. This one ends on the user's repository, their branch and their remote, which is the whole point of taking the long way round. + **GitHub, on request.** When the user asked for a repository on GitHub: ```bash @@ -79,7 +102,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. ### Step 4: Run the clone's own bootstrap @@ -160,8 +183,8 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | -| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse. On branch A that directory is served by the acquisition beside it (Step 2) and never by a `git clone` into it, which git refuses outright | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory, and on the acquisition beside an existing repository the chain removes its own temporary path and copies nothing | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md index c305e13..7f3c08a 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md @@ -35,6 +35,21 @@ The version comes from `package.json` (`"version"`) on JS and from `pyproject.to **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. +Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index ba8cb19..0db7692 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -30,7 +30,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -70,6 +70,29 @@ The `|| exit` on the clone is not decoration: the line below it deletes a `.git` The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. +**Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. + +**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. + +**The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. + +**The chain goes out as one command.** The guards hold only inside one shell — the same reason the `|| exit` above is load-bearing, stated in full in [references/starters.md](references/starters.md) — and split across separate calls this one loses its cleanup too, leaving the temporary directory beside the user's project with no line left to remove it. + +**Nothing is initialized here.** The default recipe ends `git init -b main` because it has just deleted the only repository at that path. This one ends on the user's repository, their branch and their remote, which is the whole point of taking the long way round. + **GitHub, on request.** When the user asked for a repository on GitHub: ```bash @@ -86,7 +109,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. ### Step 4: Run the clone's own bootstrap @@ -167,8 +190,8 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | -| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse. On branch A that directory is served by the acquisition beside it (Step 2) and never by a `git clone` into it, which git refuses outright | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory, and on the acquisition beside an existing repository the chain removes its own temporary path and copies nothing | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | diff --git a/pipelex/skills/pipelex-scaffold/references/starters.md b/pipelex/skills/pipelex-scaffold/references/starters.md index c305e13..7f3c08a 100644 --- a/pipelex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex/skills/pipelex-scaffold/references/starters.md @@ -35,6 +35,21 @@ The version comes from `package.json` (`"version"`) on JS and from `pyproject.to **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. +Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/skills/pipelex-scaffold/references/starters.md b/skills/pipelex-scaffold/references/starters.md index c305e13..7f3c08a 100644 --- a/skills/pipelex-scaffold/references/starters.md +++ b/skills/pipelex-scaffold/references/starters.md @@ -35,6 +35,21 @@ The version comes from `package.json` (`"version"`) on JS and from `pyproject.to **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. +Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. + GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index 062ce8e..b908dde 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later; else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -63,6 +63,29 @@ The `|| exit` on the clone is not decoration: the line below it deletes a `.git` The clone's `.git` is removed on purpose: it is the template's history and remote, and leaving it would make `git status` and a future `git push` belong to Pipelex's template rather than to the user's project. This is exactly what GitHub's "Use this template" button produces — a copy with no history and no remote — and it is why the starters' READMEs tell humans not to clone directly. Fresh history is how you honour that. +**Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: + +```bash +tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } +git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message +# the template version: package.json "version" (JS) or pyproject.toml version (Python) +rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +rm -rf "$tmp" +``` + +**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. + +**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. + +**The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. + +**The chain goes out as one command.** The guards hold only inside one shell — the same reason the `|| exit` above is load-bearing, stated in full in [references/starters.md](references/starters.md) — and split across separate calls this one loses its cleanup too, leaving the temporary directory beside the user's project with no line left to remove it. + +**Nothing is initialized here.** The default recipe ends `git init -b main` because it has just deleted the only repository at that path. This one ends on the user's repository, their branch and their remote, which is the whole point of taking the long way round. + **GitHub, on request.** When the user asked for a repository on GitHub: ```bash @@ -79,7 +102,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. ### Step 4: Run the clone's own bootstrap @@ -160,8 +183,8 @@ Two lines are easy to forget and matter: | Condition | Do this | |---|---| | A toolchain piece is missing (Node below the floor, no `uv`, no git) | STOP, name the exact missing piece and the starter README's line about it; never install a toolchain — first check a version manager the machine already has (`nvm`, `fnm`, `volta`, `asdf`, `mise`) and use its runtime, saying so | -| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse | -| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory | +| The target directory exists and is not empty — anything at all beyond a lone `.git` | STOP, ask for another; never delete, move or write into it, and never offer to. **A directory holding nothing but `.git` is empty here and is written into**; that exception is the one directory entry by name and not a class, so `.DS_Store`, `.idea/`, `.vscode/`, `Thumbs.db` and anything else still refuse. On branch A that directory is served by the acquisition beside it (Step 2) and never by a `git clone` into it, which git refuses outright | +| `git clone` fails (network, permissions) | report git's error verbatim; nothing to clean up beyond an empty directory, and on the acquisition beside an existing repository the chain removes its own temporary path and copies nothing | | `gh` is absent or not authenticated | fall back to the local clone; say the GitHub repository can be created later with `gh repo create --source .` | | The clone carries no `bootstrap` skill | follow the README's manual list; say the template changed | | The bootstrap's checks are red | its own rule: fix the cause and re-run; never hand off on red | diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 0da8bb2..6b0fffc 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -2,12 +2,42 @@ from __future__ import annotations +import os +import re +import shutil +import subprocess from pathlib import Path import pytest from scripts.gen_skill_docs import load_target_config, render_templates, resolve_output_dir +REPO_ROOT = Path(__file__).parents[2] +SKILL_TEMPLATE = REPO_ROOT / "templates" / "skills" / "pipelex-scaffold" / "SKILL.md.j2" +STARTERS_REFERENCE = REPO_ROOT / "skills" / "pipelex-scaffold" / "references" / "starters.md" + +BASH_BLOCK = re.compile(r"```bash\n(.*?)```", re.DOTALL) +# The one string both acquisition recipes point at a real remote, swapped for a +# local repository so the recipes run as shipped without touching the network. +STARTER_URL = "https://github.com/Pipelex/.git" + +NEEDS_GIT = pytest.mark.skipif(shutil.which("git") is None, reason="the acquisition recipes are git") + + +def _bash_blocks(text: str) -> list[str]: + return [match.group(1) for match in BASH_BLOCK.finditer(text)] + + +def _recipe(text: str, marker: str) -> str: + """The one shipped bash block containing `marker`, verbatim. + + Pinned to exactly one so that a recipe split in two, or a second one written + beside it, fails here instead of letting this suite execute an arbitrary half. + """ + blocks = [block for block in _bash_blocks(text) if marker in block] + assert len(blocks) == 1, f"expected exactly one bash block containing {marker!r}, found {len(blocks)}" + return blocks[0] + class TestPipelexScaffoldSkill: """The skill is executable guidance, so these tests guard what a user's new @@ -148,6 +178,56 @@ def test_only_a_lone_git_reads_as_empty_and_no_cruft_list_joins_it(self) -> None assert "never offer to move, delete or merge what it holds to make room" in body assert "never delete, move or write into it, and never offer to" in body + def test_branch_a_acquires_into_the_directory_the_lone_git_rule_admits(self) -> None: + """`L-260913-f28d9d`, ruled 2026-09-13: branch A serves the lone-`.git` directory too. + + The earlier ruling made a directory whose only entry is `.git` read as empty, which branch + B can honour because `uv init` accepts such a directory. Branch A could not: its first + command on the chosen directory is `git clone`, and git refuses a destination already + holding a `.git`. So the skill admitted a directory it then could not populate, and the + founder's own motivating case — `mkdir my-app && cd my-app && git init` — dead-ended for + anyone who wanted the starter rather than the initializer. Scoping the allowance to branch + B was rejected: it would answer that user with the refusal the exception was written to + remove, and the ruling would mean two different things depending on which branch they + landed in. + + Branch A now acquires beside the directory and moves in. These are the claims the recipe + one file over is executed against in `TestScaffoldAcquisitionRecipes`; asserted on the + template, on all three renders and on the reference, because round 1 found a guard that + had been pinned against the reference alone and was missing from the document an agent + actually executes. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + assert "**Local, into a directory that already holds a repository.**" in body + # Both halves of the ruling: the same end state, and the user's repository left alone. + assert "The end state is the one the default recipe reaches" in body + assert "the repository the user made goes on standing instead of being replaced" in body + # Ordering: the destructive line is spent on the temporary path before anything moves. + assert "**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.**" in body + assert 'the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`' in body + # Dotfiles: the naive glob drops them and still exits 0. + assert '**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.**' in body + assert "The glob matches no entry beginning with a dot" in body + # Collision: one admitted entry, so the template can only add. + assert '**The `ls -A` line is the "Where" rule read again, against the copy.**' in body + assert "a collision is therefore impossible rather than merely unlikely" in body + assert "Nothing of the user's is overwritten, moved or deleted to make room" in body + # The guards and the cleanup hold only inside one shell. + assert "**The chain goes out as one command.**" in body + # The user's repository is not re-initialised; the pristine commit lands on their branch. + assert "**Nothing is initialized here.**" in body + assert "the commit lands on the user's branch, on top of their history" in body + # And the failure table sends branch A down this route rather than at a clone. + assert "On branch A that directory is served by the acquisition beside it (Step 2) and never by a `git clone` into it" in body + + # The reference is the file the skill names as carrying every command, so the recipe and + # the three properties that make it safe are stated there as well as in the skill body. + reference = STARTERS_REFERENCE.read_text(encoding="utf-8") + assert 'tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1' in reference + assert "**No `rm -rf` in it addresses a path under ``**" in reference + assert '**`cp -R "$tmp"/. /` carries the entries beginning with a dot**' in reference + assert "**the `ls -A` line admits exactly one entry**" in reference + def test_declares_no_mcp_tool(self) -> None: """The scaffold skill is MCP-free: no allowed-tools entry, no MCP-absent STOP message.""" body = self.scaffold @@ -268,3 +348,293 @@ def test_the_committed_references_match_the_source_byte_for_byte(self, target_na assert (installed / reference).read_bytes() == (self.REFERENCES_DIR / reference).read_bytes(), ( f"{target_name}: references/{reference} is stale — run `make build`" ) + + +@NEEDS_GIT +class TestScaffoldAcquisitionRecipes: + """Branch A's two acquisition recipes, extracted from the skill and executed. + + `L-260913-f28d9d` was ruled with a condition attached: the recipe is proven by being + run, not by being read. Three reasons, each specific. It touches the one area of this + skill that deletes; this campaign's history is that recipes composed during review + rounds became the next round's defects; and `references/starters.md` already warns that + the `|| exit` guard holds only inside one shell, which is the single thing standing + between a `rm -rf` and a user's repository. + + So these tests read the bash blocks out of the skill template, swap the starter's URL + for a local repository standing in for it, and run the bytes as shipped. Nothing is + mocked and nothing is paraphrased: a recipe reworded in the skill is the recipe that + runs here. No network, so this is part of the default suite rather than opt-in. + """ + + DEFAULT_MARKER = "rm -rf /.git && git -C init -b main" + PRESERVING_MARKER = "mktemp -d" + + @staticmethod + def _commit(repository: Path, message: str) -> None: + subprocess.run( + ["git", "-C", str(repository), "-c", "user.email=t@example.com", "-c", "user.name=Test", "commit", "-q", "-m", message], + check=True, + ) + + @pytest.fixture(scope="class") + def starter(self, tmp_path_factory: pytest.TempPathFactory) -> Path: + """A local repository standing in for a starter template. + + It carries what makes the move hard rather than what makes it look real: entries + beginning with a dot at the top level and nested inside one, which is the failure + the shipped `cp -R "$tmp"/.` form exists to avoid. + """ + template = tmp_path_factory.mktemp("starter-template") + (template / "src").mkdir() + (template / ".github" / "workflows").mkdir(parents=True) + (template / ".claude" / "skills" / "bootstrap").mkdir(parents=True) + (template / "package.json").write_text('{"name": "pipelex-starter-js", "version": "0.4.2"}\n', encoding="utf-8") + (template / "README.md").write_text("# Starter\n", encoding="utf-8") + (template / "src" / "index.ts").write_text("export const x = 1\n", encoding="utf-8") + (template / ".gitignore").write_text("node_modules/\n.env.local\n", encoding="utf-8") + (template / ".env.example").write_text("PIPELEX_BASE_URL=https://api.pipelex.com\nPIPELEX_API_KEY=\n", encoding="utf-8") + (template / ".github" / "workflows" / "ci.yml").write_text("name: ci\n", encoding="utf-8") + (template / ".claude" / "skills" / "bootstrap" / "SKILL.md").write_text("# bootstrap\n", encoding="utf-8") + subprocess.run(["git", "-C", str(template), "init", "-q", "-b", "main"], check=True) + subprocess.run(["git", "-C", str(template), "add", "-A"], check=True) + self._commit(template, "the template as it came") + return template + + # Every entry the stand-in starter ships, so a dropped one is named rather than counted. + TEMPLATE_ENTRIES = frozenset({"package.json", "README.md", "src", ".gitignore", ".env.example", ".github", ".claude"}) + DOTTED_ENTRIES = frozenset({".gitignore", ".env.example", ".github", ".claude"}) + + def _run( + self, + recipe: str, + *, + starter: Path, + target: Path, + path_prefix: Path | None = None, + ) -> subprocess.CompletedProcess[str]: + """The recipe as shipped, with only the remote and `` bound.""" + script = recipe.replace(STARTER_URL, f"file://{starter}").replace("", str(target)) + assert "" not in script and "github.com" not in script, "a placeholder survived the binding" + environment = dict(os.environ) + if path_prefix is not None: + environment["PATH"] = f"{path_prefix}{os.pathsep}{environment['PATH']}" + return subprocess.run(["bash", "-c", script], capture_output=True, text=True, check=False, env=environment) + + @property + def default_recipe(self) -> str: + return _recipe(SKILL_TEMPLATE.read_text(encoding="utf-8"), self.DEFAULT_MARKER) + + @property + def preserving_recipe(self) -> str: + return _recipe(SKILL_TEMPLATE.read_text(encoding="utf-8"), self.PRESERVING_MARKER) + + @staticmethod + def _entries(directory: Path) -> set[str]: + return {entry.name for entry in directory.iterdir()} + + @staticmethod + def _make_repository(directory: Path, branch: str) -> None: + directory.mkdir(parents=True, exist_ok=True) + subprocess.run(["git", "-C", str(directory), "init", "-q", "-b", branch], check=True) + + @staticmethod + def _temporaries_beside(target: Path) -> list[str]: + return [entry.name for entry in target.parent.iterdir() if entry.name.startswith(".pipelex-starter-")] + + def test_the_default_recipe_populates_a_directory_that_does_not_exist(self, starter: Path, tmp_path: Path) -> None: + target = tmp_path / "my-app" + result = self._run(self.default_recipe, starter=starter, target=target) + assert result.returncode == 0, result.stderr + assert self._entries(target) == self.TEMPLATE_ENTRIES | {".git"} + # Fresh history and no remote: what GitHub's "Use this template" button produces. + assert subprocess.run(["git", "-C", str(target), "remote"], capture_output=True, text=True, check=True).stdout == "" + assert subprocess.run(["git", "-C", str(target), "log", "-1"], capture_output=True, text=True, check=False).returncode != 0 + + def test_the_default_recipe_populates_an_empty_directory(self, starter: Path, tmp_path: Path) -> None: + target = tmp_path / "my-app" + target.mkdir() + result = self._run(self.default_recipe, starter=starter, target=target) + assert result.returncode == 0, result.stderr + assert self._entries(target) == self.TEMPLATE_ENTRIES | {".git"} + + def test_the_default_recipe_cannot_serve_a_lone_git_directory(self, starter: Path, tmp_path: Path) -> None: + """The reproduction the ruling was made on, kept executable. + + This is why the preserving recipe exists, and the assertion that would go green if + someone decided one recipe was enough after all. The `|| exit` holds, so the user's + repository is untouched — the cost is a dead end, not damage. + """ + target = tmp_path / "my-app" + self._make_repository(target, "main") + result = self._run(self.default_recipe, starter=starter, target=target) + assert result.returncode != 0 + assert "already exists and is not an empty directory" in result.stderr + assert self._entries(target) == {".git"} + + def test_the_preserving_recipe_leaves_the_users_repository_standing(self, starter: Path, tmp_path: Path) -> None: + """The ruling itself: their commit, their branch, their reflog, their remote. + + An assertion that the run succeeded is not the claim being made — the claim is that + the repository the user made survived it, so every part of it is read back. + """ + target = tmp_path / "my-app" + self._make_repository(target, "trunk") + (target / "NOTES.md").write_text("my notes\n", encoding="utf-8") + subprocess.run(["git", "-C", str(target), "add", "NOTES.md"], check=True) + self._commit(target, "my own first commit") + subprocess.run(["git", "-C", str(target), "rm", "-q", "NOTES.md"], check=True) + self._commit(target, "and then I emptied the worktree") + subprocess.run(["git", "-C", str(target), "remote", "add", "origin", "https://github.com/someone/theirs.git"], check=True) + + def read(*arguments: str) -> str: + return subprocess.run(["git", "-C", str(target), *arguments], capture_output=True, text=True, check=True).stdout + + commits_before, branch_before, reflog_before = read("log", "--format=%H"), read("rev-parse", "--abbrev-ref", "HEAD"), read("reflog") + assert self._entries(target) == {".git"}, "the fixture is not the lone-.git shape the ruling is about" + + result = self._run(self.preserving_recipe, starter=starter, target=target) + assert result.returncode == 0, result.stderr + + assert read("log", "--format=%H") == commits_before, "a commit of the user's did not survive" + assert read("rev-parse", "--abbrev-ref", "HEAD") == branch_before == "trunk\n" + assert read("reflog") == reflog_before, "the reflog was rewritten" + assert "https://github.com/someone/theirs.git" in read("remote", "-v"), "the user's remote is gone" + assert "pipelex-starter" not in read("remote", "-v"), "the template's remote came with it" + # Their first commit still holds the file they put in it, so nothing was rewritten quietly. + assert "NOTES.md" in read("show", "--stat", "--format=", f"{commits_before.split()[-1]}") + assert read("fsck", "--no-progress") == "" + # And the template arrived, so this is an acquisition and not a no-op that preserved + # the repository by doing nothing at all. + assert self._entries(target) == self.TEMPLATE_ENTRIES | {".git"} + + def test_the_preserving_recipe_carries_every_entry_beginning_with_a_dot(self, starter: Path, tmp_path: Path) -> None: + """`mv "$tmp"/*` drops these and exits 0, so the failure looks exactly like success. + + Read off the destination rather than reasoned about from the glob, which is the + whole point: a starter that arrives without its `.gitignore` commits `node_modules/` + into the baseline, and nothing in the run says so. + """ + target = tmp_path / "my-app" + self._make_repository(target, "main") + result = self._run(self.preserving_recipe, starter=starter, target=target) + assert result.returncode == 0, result.stderr + assert self.DOTTED_ENTRIES <= self._entries(target) + # Nested inside a dot-directory too, not just at the top level. + assert (target / ".github" / "workflows" / "ci.yml").is_file() + assert (target / ".claude" / "skills" / "bootstrap" / "SKILL.md").is_file() + # And the recipe never reaches for the glob that would have dropped them. + assert 'mv "$tmp"/*' not in self.preserving_recipe + + def test_the_preserving_recipe_refuses_a_directory_holding_git_and_anything_else(self, starter: Path, tmp_path: Path) -> None: + """Not the ruled case. The exception is one entry named `.git`, never `.git` and friends.""" + target = tmp_path / "my-app" + self._make_repository(target, "main") + (target / "my-file.txt").write_text("mine\n", encoding="utf-8") + result = self._run(self.preserving_recipe, starter=starter, target=target) + assert result.returncode != 0 + assert self._entries(target) == {".git", "my-file.txt"} + assert (target / "my-file.txt").read_text(encoding="utf-8") == "mine\n" + assert self._temporaries_beside(target) == [] + + def test_the_preserving_recipe_refuses_a_directory_holding_only_ignorable_cruft(self, starter: Path, tmp_path: Path) -> None: + """The names the earlier ruling deliberately declined go on refusing. + + `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` are not an exception waiting to be + granted: a list that grows by guesswork is how this rule drifts back into the agent + judging which of a user's files matter, which is what the refusal exists to forbid. + The recipe's own `ls -A` line is that rule in executable form, so it is read here. + """ + target = tmp_path / "my-app" + target.mkdir() + (target / ".idea").mkdir() + (target / ".vscode").mkdir() + (target / ".DS_Store").write_text("", encoding="utf-8") + (target / "Thumbs.db").write_text("", encoding="utf-8") + result = self._run(self.preserving_recipe, starter=starter, target=target) + assert result.returncode != 0 + assert self._entries(target) == {".idea", ".vscode", ".DS_Store", "Thumbs.db"} + assert self._temporaries_beside(target) == [] + + def test_no_delete_in_the_preserving_recipe_addresses_a_path_under_the_target(self, starter: Path, tmp_path: Path) -> None: + """The ordering claim, recorded rather than argued. + + Every argument every `rm` is given is logged by a shim on the `PATH`, and the run is + read back: the only paths a delete may be pointed at are the temporary clone's `.git` + and the temporary clone itself. This is what makes the recipe safe to aim at a + directory holding somebody's repository, and it is the assertion that would fail if + the discard were ever reordered to after the copy. + """ + real_rm = shutil.which("rm") + assert real_rm is not None, "these tests already require a POSIX userland" + log = tmp_path / "rm-targets.log" + shim_bin = tmp_path / "shim-bin" + shim_bin.mkdir() + shim = shim_bin / "rm" + shim.write_text( + f'#!/bin/sh\nfor a in "$@"; do case "$a" in -*) ;; *) echo "$a" >> "{log}";; esac; done\nexec {real_rm} "$@"\n', + encoding="utf-8", + ) + shim.chmod(0o755) + + target = tmp_path / "my-app" + self._make_repository(target, "main") + result = self._run(self.preserving_recipe, starter=starter, target=target, path_prefix=shim_bin) + assert result.returncode == 0, result.stderr + + targets = [line for line in log.read_text(encoding="utf-8").splitlines() if line] + assert targets, "the shim recorded nothing — the recipe no longer deletes, or the shim was bypassed" + under_the_users_directory = [line for line in targets if Path(line) == target or target in Path(line).parents] + assert under_the_users_directory == [], f"a delete was pointed inside the user's directory: {under_the_users_directory}" + assert all(".pipelex-starter-" in line for line in targets), f"a delete left the temporary path: {targets}" + + def test_the_preserving_recipe_removes_its_temporary_path_on_success_and_on_refusal(self, starter: Path, tmp_path: Path) -> None: + """A temporary directory left beside the user's project is litter they did not make, + and on the refusal paths it is litter with a whole starter inside it.""" + succeeding = tmp_path / "ok" / "my-app" + self._make_repository(succeeding, "main") + assert self._run(self.preserving_recipe, starter=starter, target=succeeding).returncode == 0 + assert self._temporaries_beside(succeeding) == [] + + refusing = tmp_path / "no" / "my-app" + self._make_repository(refusing, "main") + (refusing / "theirs.txt").write_text("mine\n", encoding="utf-8") + assert self._run(self.preserving_recipe, starter=starter, target=refusing).returncode != 0 + assert self._temporaries_beside(refusing) == [] + + # A clone that cannot run at all: the failure the `|| exit` chain was written for. + unreachable = tmp_path / "gone" / "my-app" + self._make_repository(unreachable, "main") + missing_remote = self.preserving_recipe.replace(STARTER_URL, f"file://{tmp_path / 'no-such-repository'}") + assert self._run(missing_remote, starter=starter, target=unreachable).returncode != 0 + assert self._temporaries_beside(unreachable) == [] + assert self._entries(unreachable) == {".git"} + + def test_the_temporary_path_is_beside_the_target_and_collision_proof(self, starter: Path, tmp_path: Path) -> None: + """Beside, so the acquisition never crosses a filesystem or a small `/tmp`; named by + `mktemp`, so two runs in the same parent cannot land on each other.""" + recipe = self.preserving_recipe + assert 'tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX")' in recipe + target = tmp_path / "my-app" + self._make_repository(target, "main") + # The name is generated, so the same recipe run twice in one parent must not collide. + assert self._run(recipe, starter=starter, target=target).returncode == 0 + second = tmp_path / "other-app" + self._make_repository(second, "main") + assert self._run(recipe, starter=starter, target=second).returncode == 0 + assert self._temporaries_beside(target) == [] + + @pytest.mark.parametrize("target_name", ["prod", "codex", "mistral-vibe"]) + def test_the_recipes_executed_here_are_the_bytes_every_target_ships(self, target_name: str) -> None: + """This suite executes the template, so the renders must carry the same block. + + The acquisition recipes carry no Jinja, which is what makes reading the template + safe — but that is a claim about the renders, so it is read off them rather than + argued. `make agent-check` proves the committed trees are fresh; this proves the + freshness is of these lines, which are the ones a user installs and runs. + """ + config = load_target_config(REPO_ROOT / "targets", target_name) + installed = (resolve_output_dir(REPO_ROOT, config.source) / "skills" / "pipelex-scaffold" / "SKILL.md").read_text(encoding="utf-8") + template = SKILL_TEMPLATE.read_text(encoding="utf-8") + for marker in (self.DEFAULT_MARKER, self.PRESERVING_MARKER): + assert _recipe(installed, marker) == _recipe(template, marker), f"{target_name}: the shipped recipe is not the one executed here" diff --git a/wip/pipelex-integrate/plan.md b/wip/pipelex-integrate/plan.md index 387e793..a1923d0 100644 --- a/wip/pipelex-integrate/plan.md +++ b/wip/pipelex-integrate/plan.md @@ -109,6 +109,7 @@ Owner: `pipelex-plugins`. **Gate:** `scaffold-design.md`'s boxes ratified (Phase - [x] The report (S§5) with the session note as its own line. - [x] Mode (S§6) and a failure table condensed from S§7, including the "this is the template's own checkout" stop. - [x] `## Reference`: links to `references/starters.md` and `references/initializers.md`. +- [x] Branch A's acquisition into a directory whose only entry is `.git`, as `L-260913-f28d9d` ruled it on 2026-09-13: a `mktemp` path beside the target, the template's history discarded there before anything moves, the directory re-read immediately before the copy, `cp -R "$tmp"/. /` so the dotfiles come too, the temporary path removed on every exit, and no `git init` because the repository is the user's. Stated in the skill, in `references/starters.md` and in the failure table; executed against every directory shape by `TestScaffoldAcquisitionRecipes`. - [x] The non-empty refusal as `L-260912-724b71` ruled it on 2026-09-13: a lone `.git` reads as empty, at both sites (the "Where" row and the failure-table stop row), written as that one directory entry by name — the cruft list was declined, so `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` still refuse and the test pins them refusing. Never *offering* to clear is untouched. **The references — `skills/pipelex-scaffold/references/`** diff --git a/wip/pipelex-integrate/scaffold-design.md b/wip/pipelex-integrate/scaffold-design.md index 0fb37d2..3bb3c3f 100644 --- a/wip/pipelex-integrate/scaffold-design.md +++ b/wip/pipelex-integrate/scaffold-design.md @@ -36,6 +36,7 @@ A starter clone that is already in the working directory and has not been bootst 1. **Prerequisites.** JavaScript: Node at or above the floor the starter's `package.json` `engines` names (22.12 at writing — the SDK is ESM-only and the starter's e2e specs `require()` it), and npm. Python: `uv` (the starter's Makefile installs and locks with it) and a Python inside the starter's `requires-python` range (3.11 to 3.14 at writing) that `uv python find` can see. Both: git. The GitHub branch also needs `gh` authenticated (`gh auth status`). A missing piece **stops** the skill with the exact thing missing and the starter README's own line about it; the skill never installs a toolchain. **Amended 2026-09-12 (Phase 3):** a runtime the machine already has and only the `PATH` is missing is not a missing piece — dogfooded on a `PATH` without `node`, the skill found the machine's `nvm`, activated it and carried on, which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, uses what they already hold, says which one it used and that the user's own shell may not have it, and stops only when no runtime can be reached that way. 2. **Acquire.** - **Local, the default.** `git clone --depth 1 https://github.com/Pipelex/.git `; read the template's version from its `package.json` / `pyproject.toml` and its head SHA; then detach from the template — remove the clone's `.git`, `git init -b main` — so that `git status`, `git remote` and a future push belong to the user's project and not to the template. This is what GitHub's "Use this template" button produces: a copy with no history and no remote. The starters' READMEs say "don't clone it directly" to humans for exactly that reason, and the fresh history is how the skill honours it. + - **Local, into a directory that already holds a repository** (added 2026-09-13 on Louis's ruling of `L-260913-f28d9d`; see §11). `git clone` refuses a destination already holding a `.git`, so the one directory shape §2's "Where" row reads as empty is the one this default cannot serve. The skill clones into a `mktemp` path beside the directory, discards the template's `.git` there — before anything moves, so no deletion ever addresses a path under the user's directory — re-reads the directory (`ls -A` must be exactly `.git`, which makes a collision impossible and refuses anything else with nothing copied), copies the contents in with `cp -R "$tmp"/. /` so that the entries beginning with a dot come too, and removes the temporary path on every exit. Nothing is initialised: the repository is the user's and it stands, and the pristine commit lands on their branch. - **GitHub, on request.** `gh repo create / --template Pipelex/ --private --clone` (visibility is the user's call, asked, default private). Creating a repository on GitHub is an outward-facing action: the skill states the exact command and confirms before running it. GitHub writes the initial commit itself; the skill continues at step 3. - Both take the template's **default-branch head**, and the pristine commit below records the version and SHA it came from. Pinning a release tag is not offered unless the user asks; the starters cut releases, and a user who wants one names it. 3. **Commit the pristine template — exactly once.** `git add -A && git commit -m "Start from Pipelex/ ()"` from inside the directory. This is the one commit the skill makes, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in this commit — it is the template as it came. @@ -94,6 +95,8 @@ Family wiring, each one sentence: `pipelex-integrate`'s step 1 offers `/pipelex- **Amended 2026-09-13 — a directory holding nothing but `.git` reads as empty.** The non-empty refusal (§2's "Where" row, §7's second row) treated any existing entry as occupancy. Louis ruled `L-260912-724b71` on 2026-09-13: a lone `.git` is not occupancy and the directory is written into. Three things carried it. `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive at this skill; branch B itself runs `git init -b main` in the directory it is working in (§4 step 3), so the skill was refusing a state it produces one step later; and `references/initializers.md` already documents `uv init --package --no-workspace .` for exactly the empty-"here" case. **The cruft list was deliberately declined in the same ruling** — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` keep refusing until a real report names one — because a list that grows by guesswork is how this rule drifts back into the agent judging which of a user's files matter, which is what the original refusal was right to forbid. So the exception is written as one directory entry named `.git` and never as a predicate over ignorable files. **What the ruling does not touch:** a directory read as empty is never *cleared*, so §7's "never offer to make room" stands unchanged — narrowing what counts as occupied is not permission to empty what is. +**Amended 2026-09-13 — branch A acquires beside such a directory.** The amendment above is honoured by branch B, where `uv init` accepts a directory holding a `.git`, and cannot be honoured by branch A, whose first command on the chosen directory is `git clone` — which refuses any destination already holding a `.git`. The skill therefore accepted a directory it could not populate, and the motivating case the ruling itself cites dead-ended for a user who wanted the starter. Louis ruled `L-260913-f28d9d` on 2026-09-13: branch A clones to a temporary path beside the target, moves the template's files in, discards the temporary clone's own `.git`, and lets the user's repository stand (§3 step 2). **Scoping the allowance to branch B was rejected** — the exception exists because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, which is not specific to initializers, so that user has the same claim on the opinionated starter and would have met the refusal the exception was written to remove. This is the only option under which the ruling means one thing on both branches. **The end state is the one branch A already reaches by its other path** — the template in place, no template history, no Pipelex remote — so it is a different route to the same place, except that a repository the user already had survives instead of being replaced. The ruling carried a condition: the recipe is proven by being executed, not by being read, because it touches the one area of this skill that deletes and this campaign's history is that recipes composed inside review rounds became the next round's defects. It is executed against every directory shape by `TestScaffoldAcquisitionRecipes`. + ## Decision boxes for ratification | Box | Ruling | Ratified? | diff --git a/wip/pipelex-integrate/scaffold-review-deferrals.md b/wip/pipelex-integrate/scaffold-review-deferrals.md index 4026297..ec868a0 100644 --- a/wip/pipelex-integrate/scaffold-review-deferrals.md +++ b/wip/pipelex-integrate/scaffold-review-deferrals.md @@ -11,7 +11,7 @@ Everything below was read and verified in the tree — none of it rests on a rev ## Carried elsewhere -- **A directory holding only `.git` is one branch A cannot clone into** — the sharpest finding of the round, raised by three reviewers. It needs a ruling because every fix changes behaviour the founder ruled on, so it is its own decision item: `L-260913-f28d9d`. +- **A directory holding only `.git` is one branch A cannot clone into** — the sharpest finding of the round, raised by three reviewers. It needed a ruling because every fix changes behaviour the founder ruled on, so it became its own decision item, `L-260913-f28d9d`. Ruled on 2026-09-13 — branch A acquires beside such a directory and leaves the user's repository standing — and implemented on this branch, so it is no longer carried. - **A branch-B project writes `.env` that nothing loads into the process** — rediscovered by the Codex review, already open as `L-260912-059765`. No new trace needed. ## Deferred here From 96bb083f37ab1736fce071f6a7ba513dd5c17a07 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 13 Sep 2026 15:56:31 +0200 Subject: [PATCH 21/21] Serve the destination the acquisition was written for, and the rules stated in one copy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 2 read round 1's own fixes and the founder's ruling together. The preserving acquisition derived its temporary path from the destination's spelling. `dirname .` is `.`, so a destination given as `.` — the ordinary one, since the user is standing in the directory they just `git init`-ed — put the temporary clone inside the destination, where the emptiness re-check found it beside `.git` and refused. The acquisition served every shape except the one the ruling exists for, and every recipe test bound the destination to an absolute path, which is why it took an execution test spelled `.` to show it. The destination is resolved first now, which also gives the chain a variable to quote, so a directory whose name holds a space reaches `cp` whole. The references carried both starters' clone URLs on consecutive lines of a block whose own prose calls it one chain: run as written, the second clone fails on the destination the first just filled and its handler deletes the successful clone before anything is copied. The blocks name one starter now. Three rules held in one copy and were executed from another. Round 1 put the `-- .` pathspec on the pristine commit as well as its staging and wrote that it is on both commands for a reason, while the reference kept the bare commit — the half-application that round was convened to fix, mirrored. The initializers reference prescribed the inference the skill body forbids by name in favour of testing `rev-parse --show-toplevel`. And the lone-`.git` rule justified itself with a `git init` that must not run in precisely that case. The confirmation rule was reconciled with the ruling it predates: it exempted the pristine commit as landing on a directory this skill just created, which the preserving acquisition makes false on all three grounds, so that commit now says what it will stage and asks. The initializer branch locks its Python project before the hand-off, since `uv init` writes neither a lock file nor an environment and `pipelex-integrate` reads an absent lock file as pip. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 + docs/decisions.md | 2 +- .../skills/pipelex-scaffold/SKILL.md | 19 ++- .../references/initializers.md | 4 +- .../pipelex-scaffold/references/starters.md | 24 +-- pipelex-vibe/skills/pipelex-scaffold/SKILL.md | 19 ++- .../references/initializers.md | 4 +- .../pipelex-scaffold/references/starters.md | 24 +-- pipelex/skills/pipelex-scaffold/SKILL.md | 19 ++- .../references/initializers.md | 4 +- .../pipelex-scaffold/references/starters.md | 24 +-- .../references/initializers.md | 4 +- .../pipelex-scaffold/references/starters.md | 24 +-- templates/skills/pipelex-scaffold/SKILL.md.j2 | 19 ++- tests/unit/test_pipelex_scaffold_skill.py | 155 ++++++++++++++++-- .../scaffold-review-deferrals.md | 12 +- 16 files changed, 265 insertions(+), 94 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 55a100f..69cd0b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,8 @@ - **The guards the dogfood pass added no longer misfire on the cases they created.** The occupied-directory rule refused a `codegen.lock` with no sidecar as another generation's, but the lock is written a step before the sidecar, so every stop between them leaves exactly that state for the method being integrated — and a harness-owned layout keeps no sidecar by design, so the rule forbade the write its own harness section prescribes. Both are now named exceptions, with regeneration in place as the answer rather than a second tree for one method. The containment pre-check is read on resolved paths, because a lexical reading passes a symlink pointing outside the workshop and fails a project under `/tmp` that the tool would have accepted; it is also taken as soon as the project is identified, since by the time the old placement ran, a bundle copy and the tooling exclusions were already on disk. Reaching a runtime behind `nvm`, `fnm`, `volta`, `asdf` or `mise` means resolving it to a path and carrying that into every later command, not sourcing a shell that the next command will not inherit — otherwise the prerequisite read as met, the pristine commit was spent, and the delegated bootstrap then failed; a shim that answers nothing is not a runtime, the starter's version floor still applies, and a manager that would install a version it lacks is the toolchain install this step forbids. The workshop is spawned on the harness's own `PATH`, so on that machine the hand-off needs a relaunch rather than the unconditional promise the report used to make. The three stated causes of an absent `main_pipe` are all three in the failure table, where the remedy differs by cause, and the same signature is read from the verdict's text summary when the structured field did not arrive. - **A directory holding nothing but `.git` is somewhere `pipelex-scaffold` will build.** The skill refused any target directory that was not empty, which made it refuse the state it produces itself one step later: `mkdir my-app && cd my-app && git init` is an ordinary way for a user to arrive, and the skill's own initializer branch runs `git init -b main` in the directory it is working in. A lone `.git` now reads as empty, and **both branches serve it**: the initializer runs in the directory as it stands, while the starter branch — which `git clone` cannot aim at a directory already holding a `.git` — acquires into a temporary path beside it, discards the template's history while the clone is still its own, and copies the template in, dotfiles included, so the repository the user made goes on standing with their branch, their history and their remote. Everything else goes on refusing — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included — because the exception is one directory entry named in full and not a class of files the agent may decide to overlook, which is the judgement the refusal exists to prevent. Nothing about clearing changed: a directory that is not empty is still never emptied, and making room is still never offered. - **The scaffold's recipes now match the guarantees its own references state.** A review round over the skill as an executable document, rather than over its prose, found the places where the two disagreed. The named-framework recipe still prescribed a bare `uv init --package ` while `references/initializers.md` said in bold that `--no-workspace` is on every `uv init` above — so the one line an agent executes appended a `[tool.uv.workspace]` table to the user's own `pyproject.toml`, the exact hazard the reference documents, and the discipline test pinned the reference alone and never the skill body. The key write was an unconditional append, which on the fresh-clone shortcut put a second `PIPELEX_API_KEY` after one the user had already filled; every dotenv reader resolves a repeated name to the later line, so a stale exported value silently replaced a working key while the report said theirs was kept — it is gated on a file-side presence test now. `gh repo create --clone` takes no destination and clones into `./`, which every later step addressed as ``. The pristine commit carried its `-- .` pathspec on the staging but not on the commit, and a bare `git commit` commits the whole index, so anything the user had staged in an enclosing repository rode along under this skill's message — both commits carry it now, and the guard's description says what it actually guarantees. The minimal TypeScript recipe opened with `mkdir `, which aborts its own `&&` chain when `` is the "here" directory the skill explicitly permits, and its default resolution is the one the same file calls broken, now flagged where the command is chosen rather than only below it. The fresh-clone shortcut and the template-checkout stop shared a detection signal and prescribed opposite actions; the shortcut now names the `origin` that separates a copy from the template itself. +- **The acquisition that serves a lone `.git` now serves it when the user says "here".** A second review round, over the recipe as something executed rather than read, found the temporary path derived from the destination's *spelling*: `dirname .` is `.`, so a destination given as `.` — the ordinary one, since the user is standing in the directory they just `git init`-ed — put the temporary clone **inside** the destination, where the emptiness re-check found it beside `.git` and refused. The acquisition worked on every shape except the one it was written for, and it was invisible because every recipe test bound the destination to an absolute path; the destination is resolved first now, which also lets every later mention be quoted, so a directory whose name holds a space reaches `cp` whole. The same round found the references' acquisition blocks carrying both starters' clone URLs on consecutive lines of a block whose own prose calls it one chain — run as written, the second clone fails on the destination the first just filled and its handler deletes the successful clone before anything is copied — while the skill body had always used a single placeholder; the blocks name one starter now. The confirmation rule was reconciled with the ruling it predates: it exempted the pristine commit as landing "on a directory this skill just created", which the preserving acquisition made false on all three of its grounds, so that commit now says what it will stage and asks, in every mode. And the initializer branch locks its Python project before handing it over — `uv init` writes a `pyproject.toml` and neither a lock file nor an environment, and `pipelex-integrate` reads an absent lock file as `pip install` into an active environment that `uv init` never made. +- **Two rules that were stated in one copy and executed from another.** The round before this one put the `-- .` pathspec on the pristine commit as well as on its staging and wrote that it is on both commands for a reason; `references/starters.md` kept the bare commit, so the rule held only where an agent does not read the command from — the same half-application that round was convened to fix, in the mirror direction, and its changelog line said both commits carried it when one did not. The initializers reference told the reader to run `git init -b main` "only if it did not initialize a repository itself", which is the inference the skill body forbids by name in favour of testing `git -C rev-parse --show-toplevel`, and the difference is the case that motivates the test: `uv init` initializes nothing when the parent already holds a project, and staging in a directory governed by an enclosing repository sweeps the user's whole worktree into this skill's commit. Separately, the lone-`.git` rule justified itself with "branch B runs `git init -b main` in the directory it is working in one step later" — which in that very case it must not, the user having just run it themselves. - **A key cannot reach the transcript through the write half of the step that handles it.** The rule covered looking a key up and not putting one down: it sanctioned "an in-place edit", which for a file-editing tool means passing the literal value as a parameter, and a tool call's parameters are the transcript. The value now moves only through a shell that expands the variable itself, reading the env file back afterwards is refused by name, and confirming the write uses a file-side presence test that reveals nothing. The report says the value was taken from the environment **and not validated**, because a placeholder passes a presence test and fails the first run. The failure-table row that carried this guidance had an unescaped `|` inside a code span and rendered as four broken cells. - **Hosted-console connector instructions in the README**: the passage told readers to put an API key in the connector URL (`?api_key=plx_sk_...`) or an `Authorization: Bearer` header, a channel removed from the console in `@pipelex/mcp` 0.12.0 that no longer connects at all. It now says to add the connector by its plain URL and sign in with your Pipelex account, and to remove and re-add any connector registered the old way. - **The file factory no longer touches the user's own project.** Every `uv run` line in the skill and its references now passes `--no-project`. Without it `uv run` walks up from the working directory, finds the nearest project, and *syncs* it — so rendering a test PDF inside a checkout created a `.venv/` and wrote a `uv.lock` the user never asked for, in a repository the skill has no business modifying. `--no-project` resolves the ephemeral `--with` packages against nothing at all, which is what the recipes always meant; a test asserts every shipped runner line carries the flag. diff --git a/docs/decisions.md b/docs/decisions.md index 5312032..aed6725 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -183,7 +183,7 @@ Both skills were run as cold headless sessions against a local `pipelex-mcp` bui - **Containment is read before the first write, not after an error.** A path inside the workshop's working directory is legal wherever it points, so a harness launched beside the project accepts a write into the wrong tree; a run that hit this moved the tree across afterwards and left the project with a refresh that fails the same way every time. The skill now reads the path from the workshop's working directory first, and never moves a tree into place. - **A file the skill does not own is never cleared, and never offered for clearing.** Integrate met a hand-written file at an artifact path and offered to delete it; scaffold met a non-empty target directory and offered to move its contents aside and merge them back. The answer in both cases is another directory. - **A directory holding nothing but `.git` reads as empty; everything else still refuses.** (Ruled 2026-09-13, `L-260912-724b71`.) The non-empty refusal treated any existing entry as occupancy, which made the skill refuse a state it produces itself: `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, branch B runs `git init -b main` in the directory it is working in one step later, and `references/initializers.md` already documents `uv init --package --no-workspace .` for the empty-"here" case. A lone `.git` is now read as empty. **The cruft list was declined in the same ruling** — `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` go on refusing until a real report names one — because a list that grows by guesswork is how this rule drifts back into the agent judging which of a user's files matter, which is the thing the refusal exists to forbid; so the exception is written as one directory entry by name and never as a predicate over ignorable files. The separate rule that the skill never *offers* to clear anything is untouched: a directory read as empty is never cleared either way. -- **Branch A acquires beside a directory that already holds a repository, so the lone-`.git` exception means one thing on both branches.** (Ruled 2026-09-13, `L-260913-f28d9d`.) The ruling above is implementable on the initializer branch, where `uv init` accepts such a directory, and unimplementable on the starter branch, whose first command on the chosen directory is `git clone` — which refuses any destination already holding a `.git`. The skill therefore admitted a directory it could not populate, and the founder's own motivating case dead-ended for anyone who wanted the opinionated starter rather than the initializer. **Scoping the allowance to the initializer branch was rejected**: the exception exists because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, which is not specific to initializers, so that user has exactly the same claim on the starter and would have met the refusal the exception was written to remove. Branch A now clones into a `mktemp` path beside the directory, discards the template's history there, re-reads the directory and copies the template in. Three properties make it safe and each is executed rather than asserted, in `tests/unit/test_pipelex_scaffold_skill.py`: **no `rm -rf` ever addresses a path under the chosen directory**, because the discard is spent on the temporary path before anything moves; **`cp -R "$tmp"/. /` carries the entries beginning with a dot**, which `mv "$tmp"/*` drops while exiting `0`; and **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so a collision is impossible rather than unlikely and anything else stops the run with nothing copied. The end state is the one the default recipe reaches — the template in place, no template history, no Pipelex remote — reached by a different route, and nothing is initialised because the repository is already the user's. +- **Branch A acquires beside a directory that already holds a repository, so the lone-`.git` exception means one thing on both branches.** (Ruled 2026-09-13, `L-260913-f28d9d`.) The ruling above is implementable on the initializer branch, where `uv init` accepts such a directory, and unimplementable on the starter branch, whose first command on the chosen directory is `git clone` — which refuses any destination already holding a `.git`. The skill therefore admitted a directory it could not populate, and the founder's own motivating case dead-ended for anyone who wanted the opinionated starter rather than the initializer. **Scoping the allowance to the initializer branch was rejected**: the exception exists because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive, which is not specific to initializers, so that user has exactly the same claim on the starter and would have met the refusal the exception was written to remove. Branch A now clones into a `mktemp` path beside the directory, discards the template's history there, re-reads the directory and copies the template in. The properties that make it safe are executed rather than asserted, in `tests/unit/test_pipelex_scaffold_skill.py`: **no `rm -rf` ever addresses a path under the chosen directory**, because the discard is spent on the temporary path before anything moves; **`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot**, which `mv "$tmp"/*` drops while exiting `0`; and **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so a collision is impossible rather than unlikely and anything else stops the run with nothing copied. The end state is the one the default recipe reaches — the template in place, no template history, no Pipelex remote — reached by a different route, and nothing is initialised because the repository is already the user's. **The destination is resolved to an absolute path before the temporary path is derived from it** (added in review round 2): the chosen directory is usually spelled `.`, since the user is standing in the directory they just `git init`-ed, and `dirname .` is `.` — so deriving the sibling from the spelling made the temporary directory a *child*, which the `ls -A` line then saw beside `.git` and refused. The recipe served every shape except the one the ruling was written for, and every recipe test bound `` to an absolute path, which is why it took an execution test spelled `.` to show it. - **A runtime behind a version manager is not a missing toolchain.** Scaffold, on a `PATH` without `node`, found the machine's `nvm` and carried on — which installs nothing and is the useful answer. It now checks `nvm`, `fnm`, `volta`, `asdf` and `mise`, says which it used, and stops only when no runtime can be reached. - **One upstream defect is named rather than worked around.** The `ts-zod` emitter writes `binder.ts`'s sibling import with no file extension, which a plain Node ESM project rejects at type-check and at runtime while a bundler resolution accepts — which is why the JS starter never met it. Filed as `L-260912-857a5a` against `pipelex`. The tree is stamped and hashed, so the skill reports it, never patches the file, never drops the tree from the type checker, and leaves a change of `moduleResolution` to the user. diff --git a/pipelex-codex/skills/pipelex-scaffold/SKILL.md b/pipelex-codex/skills/pipelex-scaffold/SKILL.md index 1c6f2f3..6906aa2 100644 --- a/pipelex-codex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-codex/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and a repository the user made is not work of theirs to write over: branch B runs its initializer in the directory as it stands and leaves that repository alone — it does **not** re-run `git init` there, since Step 3's test finds `` is already its own repository — while branch A acquires beside it and moves in, so the repository already there goes on standing either way (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -32,7 +32,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing ## Mode -Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation **on a directory this skill created** — it holds the template as it came, and no user content is at stake. **The acquisition into a directory that already held a repository is the exception**, and it is a third thing that always confirms: there the commit lands on the user's branch, on top of their history, and `add -A -- .` records whatever their worktree was already showing along with the template (Step 3). None of the three grounds above holds, so state what will be staged and what it will land on, and ask — in every mode. ## Branch A — one of the starters @@ -66,19 +66,22 @@ The clone's `.git` is removed on purpose: it is the template's history and remot **Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` +**The first line resolves the destination, and that is what keeps the temporary path a sibling rather than a child.** `` is very often `.` here: `mkdir my-app && cd my-app && git init` is the "Where" rule's own account of how a user arrives at a directory holding nothing but `.git`, and they then ask for the project *here*. `dirname .` is `.`, so deriving the parent from the spelling would put the temporary directory **inside** the destination, where the `ls -A` line below finds it sitting beside `.git` and refuses — every time, on exactly the case this section exists to serve. Resolving to an absolute path first also pins the destination for the rest of the chain, so no later line can be re-read against a working directory that has moved, and it is what lets every mention below be quoted: a name with a space reaches `cp` whole instead of arriving as two arguments. + **No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. -**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. +**`cp -R "$tmp"/. "$dir"/`, and never `mv "$tmp"/* "$dir"/`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A "$dir"` after the copy rather than trusting the form. **The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. @@ -102,7 +105,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. That is the commit the Mode section sends back for confirmation, and this is what to put in front of the user: `git -C status --short` before staging says what will ride along, and it is the difference between a baseline commit and a line in their history that says "Start from Pipelex/…" over a change they made. ### Step 4: Run the clone's own bootstrap @@ -142,6 +145,8 @@ As in branch A, for the language chosen. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. +**On Python, finish with `uv sync` from inside ``.** `uv init` writes a `pyproject.toml` and stops: no lock file, no environment. `/pipelex-integrate` picks the package manager off the lock file and reads no lock file as `pip install` into the active environment, so a project handed over without one is a uv project installed into with pip — and `uv init` left no environment for pip to find either. The recipes that end in a `uv add` are locked by that command; the minimal and script forms, which are exactly what "no framework named" selects, are locked only by this line. [references/initializers.md](references/initializers.md) carries it with the rest of the post-initializer sequence. + ### Step 3: Version control and the pristine commit **Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. diff --git a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md index 9610cb4..bd2e5c0 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/initializers.md @@ -2,7 +2,9 @@ Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. -After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. +After the initializer: `git init -b main` only if `git -C rev-parse --show-toplevel` does not print `` itself — **the test, never the inference from which initializer ran**, because the `git init?` columns below are conditional and `/pipelex-scaffold`'s Step 3 gives the case that makes them so (`uv init` initializes nothing when the parent directory already holds a project, and staging in a `` governed by an enclosing repository sweeps the user's whole worktree into this skill's commit). Then one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +**On Python, run `uv sync` from inside `` before the pristine commit, and commit the `uv.lock` it writes.** `uv init` writes a `pyproject.toml` and nothing else: no lock file and no environment. That matters at the hand-off rather than here, because `/pipelex-integrate` reads the lock file to decide how to install — `uv.lock` selects uv, and **nothing** selects `pip install` into whatever environment happens to be active. A project handed over without its lock is therefore a uv project that the next skill installs into with pip, or cannot install into at all, because `uv init` left no environment for pip to find either. The recipes below that end in a `uv add` get a lock from that command; the minimal and script forms, which are the defaults when no framework is named, get one only from this step. ## Python diff --git a/pipelex-codex/skills/pipelex-scaffold/references/starters.md b/pipelex-codex/skills/pipelex-scaffold/references/starters.md index 7f3c08a..08550a3 100644 --- a/pipelex-codex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-codex/skills/pipelex-scaffold/references/starters.md @@ -21,41 +21,41 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each ## Acquisition +`` below is the one the choice above settled — `pipelex-starter-js` or `pipelex-starter-python` — and each block is a single chain for that one starter, never a menu to run top to bottom. + Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. **The `-- .` pathspec is on the commit as well as on the staging**, for the reason `/pipelex-scaffold`'s Step 3 gives in full: `add -A -- .` bounds what is staged, but a bare `git commit` then commits the whole index, so anything the user had staged elsewhere in an enclosing repository rides along under this skill's message. **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` -Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. +Every part of that chain is load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **The first line resolves `` to an absolute path before the parent is computed from it**, without which a destination spelled `.` — the ordinary spelling, since the user is usually standing in the directory they just `git init`-ed — puts the temporary directory inside the destination and the `ls -A` line then refuses every time. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* "$dir"/` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash gh auth status -gh repo create / --template Pipelex/pipelex-starter-js --private --clone -gh repo create / --template Pipelex/pipelex-starter-python --private --clone +gh repo create / --template Pipelex/ --private --clone ``` ## The bootstrap you delegate to diff --git a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md index 6821ac7..067bdde 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/SKILL.md +++ b/pipelex-vibe/skills/pipelex-scaffold/SKILL.md @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and a repository the user made is not work of theirs to write over: branch B runs its initializer in the directory as it stands and leaves that repository alone — it does **not** re-run `git init` there, since Step 3's test finds `` is already its own repository — while branch A acquires beside it and moves in, so the repository already there goes on standing either way (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -32,7 +32,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing ## Mode -Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation **on a directory this skill created** — it holds the template as it came, and no user content is at stake. **The acquisition into a directory that already held a repository is the exception**, and it is a third thing that always confirms: there the commit lands on the user's branch, on top of their history, and `add -A -- .` records whatever their worktree was already showing along with the template (Step 3). None of the three grounds above holds, so state what will be staged and what it will land on, and ask — in every mode. ## Branch A — one of the starters @@ -66,19 +66,22 @@ The clone's `.git` is removed on purpose: it is the template's history and remot **Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` +**The first line resolves the destination, and that is what keeps the temporary path a sibling rather than a child.** `` is very often `.` here: `mkdir my-app && cd my-app && git init` is the "Where" rule's own account of how a user arrives at a directory holding nothing but `.git`, and they then ask for the project *here*. `dirname .` is `.`, so deriving the parent from the spelling would put the temporary directory **inside** the destination, where the `ls -A` line below finds it sitting beside `.git` and refuses — every time, on exactly the case this section exists to serve. Resolving to an absolute path first also pins the destination for the rest of the chain, so no later line can be re-read against a working directory that has moved, and it is what lets every mention below be quoted: a name with a space reaches `cp` whole instead of arriving as two arguments. + **No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. -**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. +**`cp -R "$tmp"/. "$dir"/`, and never `mv "$tmp"/* "$dir"/`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A "$dir"` after the copy rather than trusting the form. **The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. @@ -102,7 +105,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. That is the commit the Mode section sends back for confirmation, and this is what to put in front of the user: `git -C status --short` before staging says what will ride along, and it is the difference between a baseline commit and a line in their history that says "Start from Pipelex/…" over a change they made. ### Step 4: Run the clone's own bootstrap @@ -142,6 +145,8 @@ As in branch A, for the language chosen. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. +**On Python, finish with `uv sync` from inside ``.** `uv init` writes a `pyproject.toml` and stops: no lock file, no environment. `/pipelex-integrate` picks the package manager off the lock file and reads no lock file as `pip install` into the active environment, so a project handed over without one is a uv project installed into with pip — and `uv init` left no environment for pip to find either. The recipes that end in a `uv add` are locked by that command; the minimal and script forms, which are exactly what "no framework named" selects, are locked only by this line. [references/initializers.md](references/initializers.md) carries it with the rest of the post-initializer sequence. + ### Step 3: Version control and the pristine commit **Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md index 9610cb4..bd2e5c0 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/initializers.md @@ -2,7 +2,9 @@ Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. -After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. +After the initializer: `git init -b main` only if `git -C rev-parse --show-toplevel` does not print `` itself — **the test, never the inference from which initializer ran**, because the `git init?` columns below are conditional and `/pipelex-scaffold`'s Step 3 gives the case that makes them so (`uv init` initializes nothing when the parent directory already holds a project, and staging in a `` governed by an enclosing repository sweeps the user's whole worktree into this skill's commit). Then one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +**On Python, run `uv sync` from inside `` before the pristine commit, and commit the `uv.lock` it writes.** `uv init` writes a `pyproject.toml` and nothing else: no lock file and no environment. That matters at the hand-off rather than here, because `/pipelex-integrate` reads the lock file to decide how to install — `uv.lock` selects uv, and **nothing** selects `pip install` into whatever environment happens to be active. A project handed over without its lock is therefore a uv project that the next skill installs into with pip, or cannot install into at all, because `uv init` left no environment for pip to find either. The recipes below that end in a `uv add` get a lock from that command; the minimal and script forms, which are the defaults when no framework is named, get one only from this step. ## Python diff --git a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md index 7f3c08a..08550a3 100644 --- a/pipelex-vibe/skills/pipelex-scaffold/references/starters.md +++ b/pipelex-vibe/skills/pipelex-scaffold/references/starters.md @@ -21,41 +21,41 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each ## Acquisition +`` below is the one the choice above settled — `pipelex-starter-js` or `pipelex-starter-python` — and each block is a single chain for that one starter, never a menu to run top to bottom. + Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. **The `-- .` pathspec is on the commit as well as on the staging**, for the reason `/pipelex-scaffold`'s Step 3 gives in full: `add -A -- .` bounds what is staged, but a bare `git commit` then commits the whole index, so anything the user had staged elsewhere in an enclosing repository rides along under this skill's message. **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` -Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. +Every part of that chain is load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **The first line resolves `` to an absolute path before the parent is computed from it**, without which a destination spelled `.` — the ordinary spelling, since the user is usually standing in the directory they just `git init`-ed — puts the temporary directory inside the destination and the `ls -A` line then refuses every time. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* "$dir"/` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash gh auth status -gh repo create / --template Pipelex/pipelex-starter-js --private --clone -gh repo create / --template Pipelex/pipelex-starter-python --private --clone +gh repo create / --template Pipelex/ --private --clone ``` ## The bootstrap you delegate to diff --git a/pipelex/skills/pipelex-scaffold/SKILL.md b/pipelex/skills/pipelex-scaffold/SKILL.md index 0db7692..daab34a 100644 --- a/pipelex/skills/pipelex-scaffold/SKILL.md +++ b/pipelex/skills/pipelex-scaffold/SKILL.md @@ -30,7 +30,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and a repository the user made is not work of theirs to write over: branch B runs its initializer in the directory as it stands and leaves that repository alone — it does **not** re-run `git init` there, since Step 3's test finds `` is already its own repository — while branch A acquires beside it and moves in, so the repository already there goes on standing either way (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -39,7 +39,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing ## Mode -Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation **on a directory this skill created** — it holds the template as it came, and no user content is at stake. **The acquisition into a directory that already held a repository is the exception**, and it is a third thing that always confirms: there the commit lands on the user's branch, on top of their history, and `add -A -- .` records whatever their worktree was already showing along with the template (Step 3). None of the three grounds above holds, so state what will be staged and what it will land on, and ask — in every mode. ## Branch A — one of the starters @@ -73,19 +73,22 @@ The clone's `.git` is removed on purpose: it is the template's history and remot **Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` +**The first line resolves the destination, and that is what keeps the temporary path a sibling rather than a child.** `` is very often `.` here: `mkdir my-app && cd my-app && git init` is the "Where" rule's own account of how a user arrives at a directory holding nothing but `.git`, and they then ask for the project *here*. `dirname .` is `.`, so deriving the parent from the spelling would put the temporary directory **inside** the destination, where the `ls -A` line below finds it sitting beside `.git` and refuses — every time, on exactly the case this section exists to serve. Resolving to an absolute path first also pins the destination for the rest of the chain, so no later line can be re-read against a working directory that has moved, and it is what lets every mention below be quoted: a name with a space reaches `cp` whole instead of arriving as two arguments. + **No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. -**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. +**`cp -R "$tmp"/. "$dir"/`, and never `mv "$tmp"/* "$dir"/`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A "$dir"` after the copy rather than trusting the form. **The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. @@ -109,7 +112,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. That is the commit the Mode section sends back for confirmation, and this is what to put in front of the user: `git -C status --short` before staging says what will ride along, and it is the difference between a baseline commit and a line in their history that says "Start from Pipelex/…" over a change they made. ### Step 4: Run the clone's own bootstrap @@ -149,6 +152,8 @@ As in branch A, for the language chosen. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. +**On Python, finish with `uv sync` from inside ``.** `uv init` writes a `pyproject.toml` and stops: no lock file, no environment. `/pipelex-integrate` picks the package manager off the lock file and reads no lock file as `pip install` into the active environment, so a project handed over without one is a uv project installed into with pip — and `uv init` left no environment for pip to find either. The recipes that end in a `uv add` are locked by that command; the minimal and script forms, which are exactly what "no framework named" selects, are locked only by this line. [references/initializers.md](references/initializers.md) carries it with the rest of the post-initializer sequence. + ### Step 3: Version control and the pristine commit **Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. diff --git a/pipelex/skills/pipelex-scaffold/references/initializers.md b/pipelex/skills/pipelex-scaffold/references/initializers.md index 9610cb4..bd2e5c0 100644 --- a/pipelex/skills/pipelex-scaffold/references/initializers.md +++ b/pipelex/skills/pipelex-scaffold/references/initializers.md @@ -2,7 +2,9 @@ Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. -After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. +After the initializer: `git init -b main` only if `git -C rev-parse --show-toplevel` does not print `` itself — **the test, never the inference from which initializer ran**, because the `git init?` columns below are conditional and `/pipelex-scaffold`'s Step 3 gives the case that makes them so (`uv init` initializes nothing when the parent directory already holds a project, and staging in a `` governed by an enclosing repository sweeps the user's whole worktree into this skill's commit). Then one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +**On Python, run `uv sync` from inside `` before the pristine commit, and commit the `uv.lock` it writes.** `uv init` writes a `pyproject.toml` and nothing else: no lock file and no environment. That matters at the hand-off rather than here, because `/pipelex-integrate` reads the lock file to decide how to install — `uv.lock` selects uv, and **nothing** selects `pip install` into whatever environment happens to be active. A project handed over without its lock is therefore a uv project that the next skill installs into with pip, or cannot install into at all, because `uv init` left no environment for pip to find either. The recipes below that end in a `uv add` get a lock from that command; the minimal and script forms, which are the defaults when no framework is named, get one only from this step. ## Python diff --git a/pipelex/skills/pipelex-scaffold/references/starters.md b/pipelex/skills/pipelex-scaffold/references/starters.md index 7f3c08a..08550a3 100644 --- a/pipelex/skills/pipelex-scaffold/references/starters.md +++ b/pipelex/skills/pipelex-scaffold/references/starters.md @@ -21,41 +21,41 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each ## Acquisition +`` below is the one the choice above settled — `pipelex-starter-js` or `pipelex-starter-python` — and each block is a single chain for that one starter, never a menu to run top to bottom. + Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. **The `-- .` pathspec is on the commit as well as on the staging**, for the reason `/pipelex-scaffold`'s Step 3 gives in full: `add -A -- .` bounds what is staged, but a bare `git commit` then commits the whole index, so anything the user had staged elsewhere in an enclosing repository rides along under this skill's message. **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` -Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. +Every part of that chain is load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **The first line resolves `` to an absolute path before the parent is computed from it**, without which a destination spelled `.` — the ordinary spelling, since the user is usually standing in the directory they just `git init`-ed — puts the temporary directory inside the destination and the `ls -A` line then refuses every time. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* "$dir"/` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash gh auth status -gh repo create / --template Pipelex/pipelex-starter-js --private --clone -gh repo create / --template Pipelex/pipelex-starter-python --private --clone +gh repo create / --template Pipelex/ --private --clone ``` ## The bootstrap you delegate to diff --git a/skills/pipelex-scaffold/references/initializers.md b/skills/pipelex-scaffold/references/initializers.md index 9610cb4..bd2e5c0 100644 --- a/skills/pipelex-scaffold/references/initializers.md +++ b/skills/pipelex-scaffold/references/initializers.md @@ -2,7 +2,9 @@ Branch B of `/pipelex-scaffold` runs an initializer that already exists and authors nothing of its own beyond what that initializer writes. This file lists the common ones with their non-interactive forms. Flags change between versions: when a command below prompts anyway or rejects a flag, read its `--help` and prefer its own current non-interactive form over improvising a layout by hand. An initializer with no non-interactive form is handed to the user to run in the session. -After the initializer: `git init -b main` only if it did not initialize a repository itself, one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. +After the initializer: `git init -b main` only if `git -C rev-parse --show-toplevel` does not print `` itself — **the test, never the inference from which initializer ran**, because the `git init?` columns below are conditional and `/pipelex-scaffold`'s Step 3 gives the case that makes them so (`uv init` initializes nothing when the parent directory already holds a project, and staging in a `` governed by an enclosing repository sweeps the user's whole worktree into this skill's commit). Then one pristine commit (`Scaffold project`), the two-line `.env.example` (`PIPELEX_BASE_URL=https://api.pipelex.com`, `PIPELEX_API_KEY=`), `.env` gitignored and copied from it. No SDK dependency — `/pipelex-integrate` adds it. + +**On Python, run `uv sync` from inside `` before the pristine commit, and commit the `uv.lock` it writes.** `uv init` writes a `pyproject.toml` and nothing else: no lock file and no environment. That matters at the hand-off rather than here, because `/pipelex-integrate` reads the lock file to decide how to install — `uv.lock` selects uv, and **nothing** selects `pip install` into whatever environment happens to be active. A project handed over without its lock is therefore a uv project that the next skill installs into with pip, or cannot install into at all, because `uv init` left no environment for pip to find either. The recipes below that end in a `uv add` get a lock from that command; the minimal and script forms, which are the defaults when no framework is named, get one only from this step. ## Python diff --git a/skills/pipelex-scaffold/references/starters.md b/skills/pipelex-scaffold/references/starters.md index 7f3c08a..08550a3 100644 --- a/skills/pipelex-scaffold/references/starters.md +++ b/skills/pipelex-scaffold/references/starters.md @@ -21,41 +21,41 @@ Both are GitHub **template repositories** under the `Pipelex` organization. Each ## Acquisition +`` below is the one the choice above settled — `pipelex-starter-js` or `pipelex-starter-python` — and each block is a single chain for that one starter, never a menu to run top to bottom. + Local clone with fresh history (the default — it produces what GitHub's "Use this template" button produces, a copy with no history and no remote): ```bash -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit +git clone --depth 1 https://github.com/Pipelex/.git || exit git -C rev-parse HEAD rm -rf /.git && git -C init -b main -git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" +git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. +The version comes from `package.json` (`"version"`) on JS and from `pyproject.toml` (`version =`) on Python, read before the commit. **The `-- .` pathspec is on the commit as well as on the staging**, for the reason `/pipelex-scaffold`'s Step 3 gives in full: `add -A -- .` bounds what is staged, but a bare `git commit` then commits the whole index, so anything the user had staged elsewhere in an enclosing repository rides along under this skill's message. **The `|| exit` on the clone is load-bearing and is not decoration**, for the reason `/pipelex-scaffold`'s Step 2 gives in full: the line below it deletes a `.git` directory, and a clone that never ran — a network failure, or `` already existing — leaves that `rm -rf` to find whatever `.git` is actually at that path, destroying a repository of the user's irrecoverably. The guard only holds inside one shell, so when the two lines go out as separate commands, check the clone's exit status yourself before typing the `rm -rf`, and never type it on a path you have not just created. Into a directory whose only entry is `.git` — the one shape the "Where" rule reads as empty and `git clone` still refuses — acquire beside it and move in, so the user's own repository stands and the end state is the same as above: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git "$tmp" || { rm -rf "$tmp"; exit 1; } -git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git "$tmp" || { rm -rf "$tmp"; exit 1; } +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 +git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` -Three things in that chain are load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. /` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* /` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. +Every part of that chain is load-bearing, and `/pipelex-scaffold`'s Step 2 gives each in full. **The first line resolves `` to an absolute path before the parent is computed from it**, without which a destination spelled `.` — the ordinary spelling, since the user is usually standing in the directory they just `git init`-ed — puts the temporary directory inside the destination and the `ls -A` line then refuses every time. **No `rm -rf` in it addresses a path under ``**: the template's history is discarded while the clone is still at a path `mktemp` made for this command, before anything moves, so the guarded-deletion problem above does not arise here at all. **`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot** — `.gitignore`, `.env.example`, `.github/`, `.claude/` — every one of which `mv "$tmp"/* "$dir"/` leaves behind while exiting `0`. And **the `ls -A` line admits exactly one entry**, `.git`, which the clone no longer has, so the template can only add to the directory and anything else stops the run with nothing copied and the temporary path removed. It is one chain and goes out as one command, for the reason the paragraph above gives. GitHub repository, on request and after confirmation (visibility asked, default private; GitHub makes the initial commit, so no pristine commit of your own): ```bash gh auth status -gh repo create / --template Pipelex/pipelex-starter-js --private --clone -gh repo create / --template Pipelex/pipelex-starter-python --private --clone +gh repo create / --template Pipelex/ --private --clone ``` ## The bootstrap you delegate to diff --git a/templates/skills/pipelex-scaffold/SKILL.md.j2 b/templates/skills/pipelex-scaffold/SKILL.md.j2 index b908dde..37be71f 100644 --- a/templates/skills/pipelex-scaffold/SKILL.md.j2 +++ b/templates/skills/pipelex-scaffold/SKILL.md.j2 @@ -23,7 +23,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing |---|---|---| | **Language** | the user's word; the language the method's consumer is written in; a framework the user named | ask | | **Which branch** | a **named framework** the starters do not carry (FastAPI, Django, Express, Hono, a plain library, a Lambda) → the initializer; **"minimal"**, **"no demo code"**, **"just a project"** → the initializer; a **web app people use in a browser**, forms, an upload flow → the JS starter; a **CLI, script, batch job, worker or service** in Python → the Python starter | one question offering the matching starter first, saying what it brings (durable runs, forms or CLI modes, codegen wiring, CI, its own `release` skill) and what it costs (demos to keep as references or to strip) | -| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and branch B runs `git init -b main` in the directory it is working in one step later, while branch A acquires beside it and moves in so that the repository already there goes on standing (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | +| **Where** | the directory the user named; **"here"** when the working directory is empty — **and a directory whose only entry is `.git` is empty for this rule**, because `mkdir my-app && cd my-app && git init` is an ordinary way to arrive here and a repository the user made is not work of theirs to write over: branch B runs its initializer in the directory as it stands and leaves that repository alone — it does **not** re-run `git init` there, since Step 3's test finds `` is already its own repository — while branch A acquires beside it and moves in, so the repository already there goes on standing either way (Step 2); else a kebab-case directory named after the project | ask; never write into a directory that exists and is not empty, and never offer to move, delete or merge what it holds to make room — the answer is another directory. **A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else** — not a class of files you may decide to overlook. Every other entry still refuses, `.DS_Store`, `.idea/`, `.vscode/` and `Thumbs.db` included: judging which of a user's files matter is the thing this rule exists to forbid, and a list that grows by guesswork is how it would come back. The rule is about **a directory you are creating a project in**, which is why the fresh-clone shortcut below is not an exception to it: there the project is already there and you are finishing it, not writing over someone's work | | **GitHub or local** | the user asked for a GitHub repository → `gh repo create --template`, after confirmation; otherwise a local clone with fresh history | local | **The fresh-clone shortcut.** A starter clone already in the working directory that has not been bootstrapped — `package.json` still says `pipelex-starter-js`, or `pyproject.toml` still says `name = "piper"` — **and whose `origin` does not point at `Pipelex/pipelex-starter-…`** is branch A entered at step 4: acquisition already happened, so go straight to running the clone's bootstrap. Do not clone again. @@ -32,7 +32,7 @@ A cheap, reliable signal decides; an inconclusive one asks one question; nothing ## Mode -Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation — it is on a directory this skill just created, holding the template as it came, and no user content is at stake. +Automatic by default, with the plugin's usual rules: an explicit user signal wins ("just do it" → automatic; "walk me through" → interactive); a genuinely ambiguous branch is one question, asked once; a request that gave every input up front proceeds without re-asking. Two things always confirm, in every mode: **`gh repo create`**, because it creates a repository on GitHub, and whatever the clone's bootstrap skill confirms on its own account. The pristine commit does not need confirmation **on a directory this skill created** — it holds the template as it came, and no user content is at stake. **The acquisition into a directory that already held a repository is the exception**, and it is a third thing that always confirms: there the commit lands on the user's branch, on top of their history, and `add -A -- .` records whatever their worktree was already showing along with the template (Step 3). None of the three grounds above holds, so state what will be staged and what it will land on, and ask — in every mode. ## Branch A — one of the starters @@ -66,19 +66,22 @@ The clone's `.git` is removed on purpose: it is the template's history and remot **Local, into a directory that already holds a repository.** The "Where" rule reads a directory whose only entry is `.git` as empty, and `git clone` cannot serve it: git refuses any destination already holding a `.git` and stops with `destination path '' already exists and is not an empty directory`. So acquire **beside** the directory and move in. The end state is the one the default recipe reaches — the template in ``, no template history, no Pipelex remote — and the repository the user made goes on standing instead of being replaced: ```bash -tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1 +dir=$(cd && pwd) || exit 1 +tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1 git clone --depth 1 https://github.com/Pipelex/.git "$tmp" || { rm -rf "$tmp"; exit 1; } git -C "$tmp" rev-parse HEAD # the template SHA, for the commit message # the template version: package.json "version" (JS) or pyproject.toml version (Python) rm -rf "$tmp/.git" || { rm -rf "$tmp"; exit 1; } -[ "$(ls -A )" = ".git" ] || { rm -rf "$tmp"; exit 1; } -cp -R "$tmp"/. / || { rm -rf "$tmp"; exit 1; } +[ "$(ls -A "$dir")" = ".git" ] || { rm -rf "$tmp"; exit 1; } +cp -R "$tmp"/. "$dir"/ || { rm -rf "$tmp"; exit 1; } rm -rf "$tmp" ``` +**The first line resolves the destination, and that is what keeps the temporary path a sibling rather than a child.** `` is very often `.` here: `mkdir my-app && cd my-app && git init` is the "Where" rule's own account of how a user arrives at a directory holding nothing but `.git`, and they then ask for the project *here*. `dirname .` is `.`, so deriving the parent from the spelling would put the temporary directory **inside** the destination, where the `ls -A` line below finds it sitting beside `.git` and refuses — every time, on exactly the case this section exists to serve. Resolving to an absolute path first also pins the destination for the rest of the chain, so no later line can be re-read against a working directory that has moved, and it is what lets every mention below be quoted: a name with a space reaches `cp` whole instead of arriving as two arguments. + **No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.** The template's history is discarded while the clone is still at a path `mktemp` made for this one command, so the destructive line is spent before anything moves: the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`. Compare the default recipe, where the same line runs on `` itself and `|| exit` is the whole thing standing between it and a user's history. Here there is nothing for a guard to hold, which is what makes this the form you may aim at a directory holding somebody's repository. -**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A ` after the copy rather than trusting the form. +**`cp -R "$tmp"/. "$dir"/`, and never `mv "$tmp"/* "$dir"/`.** The glob matches no entry beginning with a dot, so the naive move leaves `.gitignore`, `.env.example`, `.github/` and `.claude/` behind, exits `0`, and the line after it deletes the temporary directory they are still sitting in — a starter arriving without its `.gitignore`, whose pristine commit then swallows `node_modules/`, reported as a success. The trailing `/.` copies the directory's *contents*, dotfiles included, with no shell globbing involved at all. Confirm it with `ls -A "$dir"` after the copy rather than trusting the form. **The `ls -A` line is the "Where" rule read again, against the copy.** It is not the decision — the "Where" question settled that — it is the last look before anything lands, put next to the copy so nothing can change between the two. It admits exactly one entry, `.git`, which the clone has not had since the line above: a collision is therefore impossible rather than merely unlikely, and the template can only add to the directory. Anything else — `.git` beside a file of the user's, a `.DS_Store`, a `README.md` they wrote — stops here with nothing copied, the temporary path removed and the directory as it was. A discarded shallow clone is the cheap half of that trade. Nothing of the user's is overwritten, moved or deleted to make room, here or anywhere. @@ -102,7 +105,7 @@ Both forms take the template's default-branch head. Do not offer a release tag u git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- . ``` -This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. +This is the **one commit this skill makes**, and it is load-bearing twice over: the Python starter's bootstrap renames the package directory with `git mv`, which refuses a path git does not track, and a committed baseline is what turns the bootstrap's edits into a diff the user can read before committing them. Nothing of the user's is in it — it is the template as it came. One qualification on the acquisition into a directory that already held a repository: the commit lands on the user's branch, on top of their history rather than opening a new one, and `add -A -- .` also records whatever deletion their worktree was already showing — their own pending change and not one this skill made, so name it in the report instead of undoing it. That is the commit the Mode section sends back for confirmation, and this is what to put in front of the user: `git -C status --short` before staging says what will ride along, and it is the difference between a baseline commit and a line in their history that says "Start from Pipelex/…" over a change they made. ### Step 4: Run the clone's own bootstrap @@ -142,6 +145,8 @@ As in branch A, for the language chosen. Nothing beyond what the initializer writes is authored by this skill: no example code, no folder layout of its own, no opinion the framework did not ship. +**On Python, finish with `uv sync` from inside ``.** `uv init` writes a `pyproject.toml` and stops: no lock file, no environment. `/pipelex-integrate` picks the package manager off the lock file and reads no lock file as `pip install` into the active environment, so a project handed over without one is a uv project installed into with pip — and `uv init` left no environment for pip to find either. The recipes that end in a `uv add` are locked by that command; the minimal and script forms, which are exactly what "no framework named" selects, are locked only by this line. [references/initializers.md](references/initializers.md) carries it with the rest of the post-initializer sequence. + ### Step 3: Version control and the pristine commit **Test whether `` is its own repository; never infer it from which initializer ran.** `git -C rev-parse --show-toplevel` must print `` itself, and when it does not, run `git init -b main` in the directory before staging anything. The list of initializers that `git init` on their own is not a substitute for that test, because membership in it is conditional: `uv init` initializes a repository when it creates a standalone project and **does not** when the parent directory already holds one, where it makes `` a workspace member of the enclosing project instead. A `` with no `.git` of its own is governed by whatever repository encloses it — the user's — and `git -C ` sets git's working directory without scoping anything, so the staging below would sweep that whole worktree: the user's unrelated untracked files, wherever they sit, committed into their repository under this skill's message. That is the one outcome this step exists to prevent, and the read-back catches it only if you read the paths and not just the count. diff --git a/tests/unit/test_pipelex_scaffold_skill.py b/tests/unit/test_pipelex_scaffold_skill.py index 6b0fffc..b6451b0 100644 --- a/tests/unit/test_pipelex_scaffold_skill.py +++ b/tests/unit/test_pipelex_scaffold_skill.py @@ -15,6 +15,7 @@ REPO_ROOT = Path(__file__).parents[2] SKILL_TEMPLATE = REPO_ROOT / "templates" / "skills" / "pipelex-scaffold" / "SKILL.md.j2" STARTERS_REFERENCE = REPO_ROOT / "skills" / "pipelex-scaffold" / "references" / "starters.md" +INITIALIZERS_REFERENCE = REPO_ROOT / "skills" / "pipelex-scaffold" / "references" / "initializers.md" BASH_BLOCK = re.compile(r"```bash\n(.*?)```", re.DOTALL) # The one string both acquisition recipes point at a real remote, swapped for a @@ -162,7 +163,7 @@ def test_only_a_lone_git_reads_as_empty_and_no_cruft_list_joins_it(self) -> None # The exception, at both sites, each stated as one named entry and not as a category. assert "**and a directory whose only entry is `.git` is empty for this rule**" in body assert "**A directory holding nothing but `.git` is empty here and is written into**" in body - assert "branch B runs `git init -b main` in the directory it is working in one step later" in body + assert "a repository the user made is not work of theirs to write over" in body # And the refusal everything else still meets, with the declined names spelled out. assert ( "**A lone `.git` is the only entry that does not make a directory non-empty, and that is a ruling about `.git` and nothing else**" @@ -206,8 +207,13 @@ def test_branch_a_acquires_into_the_directory_the_lone_git_rule_admits(self) -> assert "**No `rm -rf` here ever addresses a path under ``, and that is the ordering rather than a coincidence.**" in body assert 'the only two paths any delete is pointed at are `"$tmp/.git"` and `"$tmp"`' in body # Dotfiles: the naive glob drops them and still exits 0. - assert '**`cp -R "$tmp"/. /`, and never `mv "$tmp"/* /`.**' in body + assert '**`cp -R "$tmp"/. "$dir"/`, and never `mv "$tmp"/* "$dir"/`.**' in body assert "The glob matches no entry beginning with a dot" in body + # The destination is resolved before its parent is computed, so `.` cannot make the + # temporary path a child of the target. Round 2 found the unresolved form refusing + # every "scaffold here", which is the ruling's own motivating case. + assert "**The first line resolves the destination, and that is what keeps the temporary path a sibling rather than a child.**" in body + assert "would put the temporary directory **inside** the destination" in body # Collision: one admitted entry, so the template can only add. assert '**The `ls -A` line is the "Where" rule read again, against the copy.**' in body assert "a collision is therefore impossible rather than merely unlikely" in body @@ -223,10 +229,77 @@ def test_branch_a_acquires_into_the_directory_the_lone_git_rule_admits(self) -> # The reference is the file the skill names as carrying every command, so the recipe and # the three properties that make it safe are stated there as well as in the skill body. reference = STARTERS_REFERENCE.read_text(encoding="utf-8") - assert 'tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX") || exit 1' in reference + assert "dir=$(cd && pwd) || exit 1" in reference + assert 'tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX") || exit 1' in reference assert "**No `rm -rf` in it addresses a path under ``**" in reference - assert '**`cp -R "$tmp"/. /` carries the entries beginning with a dot**' in reference + assert '**`cp -R "$tmp"/. "$dir"/` carries the entries beginning with a dot**' in reference assert "**the `ls -A` line admits exactly one entry**" in reference + assert "**The first line resolves `` to an absolute path before the parent is computed from it**" in reference + + def test_every_acquisition_block_names_one_starter(self) -> None: + """Round 2: the reference's blocks each carried both starters' URLs on consecutive lines. + + Read as the chain the prose calls them — "It is one chain and goes out as one command" — + the second clone lands on a destination the first has just filled, fails, and its handler + deletes the successful clone before anything is copied. The skill body had always used a + single `` placeholder, so this was the body-and-reference divergence round 1 was + told to watch for, reappearing on the other side. + """ + reference = STARTERS_REFERENCE.read_text(encoding="utf-8") + assert "pipelex-starter-js.git" not in reference + assert "pipelex-starter-python.git" not in reference + assert "--template Pipelex/pipelex-starter-js" not in reference + assert "--template Pipelex/pipelex-starter-python" not in reference + # The placeholder is bound where the blocks begin, so `` is not left dangling. + assert "`` below is the one the choice above settled" in reference + assert "never a menu to run top to bottom" in reference + + def test_the_lone_git_rule_does_not_promise_a_git_init_that_must_not_run(self) -> None: + """Round 2: the "Where" row justified the exception with behaviour that cannot occur there. + + It read "branch B runs `git init -b main` in the directory it is working in one step later" + — but in the lone-`.git` case the user has already run `git init`, so Step 3's own test finds + `` is its own repository and branch B must not re-run it. The rationale invited an agent + to expect the one command the step exists to gate. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + assert "it does **not** re-run `git init` there" in body + assert "since Step 3's test finds `` is already its own repository" in body + assert "branch B runs `git init -b main` in the directory it is working in one step later" not in body + + def test_the_pristine_commit_confirms_when_it_lands_on_the_users_repository(self) -> None: + """Round 2: the Mode section's carve-out was made false by the preserving acquisition. + + It read "the pristine commit does not need confirmation — it is on a directory this skill + just created, holding the template as it came, and no user content is at stake". On the + preserving path none of those three grounds holds: the directory is the user's, the commit + lands on their branch, and `add -A -- .` sweeps in whatever their worktree was already + showing. Step 3 was qualified when the ruling landed and the Mode section was not, which is + the half-application this suite exists to catch. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + assert "The pristine commit does not need confirmation **on a directory this skill created**" in body + assert "**The acquisition into a directory that already held a repository is the exception**" in body + assert "None of the three grounds above holds" in body + # And Step 3 says what to put in front of the user rather than only what to report. + assert "That is the commit the Mode section sends back for confirmation" in body + assert "`git -C status --short` before staging says what will ride along" in body + + def test_branch_b_locks_the_python_project_before_the_hand_off(self) -> None: + """Round 2: `uv init` writes a `pyproject.toml` and neither a lock file nor an environment. + + `/pipelex-integrate` picks the package manager off the lock file and reads its absence as + `pip install` into the active environment, so the default Python scaffold — the minimal and + script forms, which are what "no framework named" selects — handed over a uv project for the + next skill to install into with pip, and into no environment at all. Asserted on the body and + on the reference, because either alone is the half-application. + """ + for body in [self.scaffold] + [self.render(target) for target in ("prod", "codex", "mistral-vibe")]: + assert "**On Python, finish with `uv sync` from inside ``.**" in body + assert "reads no lock file as `pip install` into the active environment" in body + reference = INITIALIZERS_REFERENCE.read_text(encoding="utf-8") + assert "**On Python, run `uv sync` from inside `` before the pristine commit, and commit the `uv.lock` it writes.**" in reference + assert "`uv init` writes a `pyproject.toml` and nothing else: no lock file and no environment." in reference def test_declares_no_mcp_tool(self) -> None: """The scaffold skill is MCP-free: no allowed-tools entry, no MCP-absent STOP message.""" @@ -247,12 +320,21 @@ def test_references_describe_both_starters_and_the_initializers(self) -> None: # leaves that line to delete whatever `.git` is at that path — a user's history, if # was theirs. The SKILL.md carries it; so must the reference the skill names as the source # of every command, or the guard exists only in the copy nobody executes from. - assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-js.git || exit" in starters - assert "git clone --depth 1 https://github.com/Pipelex/pipelex-starter-python.git || exit" in starters + assert "git clone --depth 1 https://github.com/Pipelex/.git || exit" in starters assert "The `|| exit` on the clone is load-bearing" in starters - assert "gh repo create / --template Pipelex/pipelex-starter-python" in starters - assert "shell out to a `pipelex` CLI the starter does not depend on" in starters + # Round 1 put the `-- .` pathspec on the commit as well as on the staging and wrote that it + # is on "both commands for a reason"; the reference kept the old bare commit, so the rule + # held only in the copy an agent does not read the commands out of. The mirror of the + # half-application round 1 was itself convened to fix. + assert 'git -C add -A -- . && git -C commit -m "Start from Pipelex/ ()" -- .' in starters + assert "**The `-- .` pathspec is on the commit as well as on the staging**" in starters + # And the initializers reference must send the reader to the repository test rather than to + # the inference the skill body forbids by name. initializers = (self.REFERENCES_DIR / "initializers.md").read_text(encoding="utf-8") + assert "**the test, never the inference from which initializer ran**" in initializers + assert "`git init -b main` only if `git -C rev-parse --show-toplevel` does not print `` itself" in initializers + assert "gh repo create / --template Pipelex/ --private --clone" in starters + assert "shell out to a `pipelex` CLI the starter does not depend on" in starters assert "npm create next-app@latest -- --ts --app --src-dir --eslint --use-npm --yes" in initializers assert "No SDK dependency" in initializers # Every `uv add` runs inside the new project: from the parent it writes to the user's own. @@ -412,14 +494,28 @@ def _run( starter: Path, target: Path, path_prefix: Path | None = None, + dir_literal: str | None = None, + cwd: Path | None = None, ) -> subprocess.CompletedProcess[str]: - """The recipe as shipped, with only the remote and `` bound.""" - script = recipe.replace(STARTER_URL, f"file://{starter}").replace("", str(target)) + """The recipe as shipped, with only the remote and `` bound. + + `dir_literal` binds `` to a spelling other than the target's absolute path — `.`, say — + and `cwd` is the directory the shell starts in, which is what makes such a spelling mean the + target at all. + """ + script = recipe.replace(STARTER_URL, f"file://{starter}").replace("", dir_literal or str(target)) assert "" not in script and "github.com" not in script, "a placeholder survived the binding" environment = dict(os.environ) if path_prefix is not None: environment["PATH"] = f"{path_prefix}{os.pathsep}{environment['PATH']}" - return subprocess.run(["bash", "-c", script], capture_output=True, text=True, check=False, env=environment) + return subprocess.run( + ["bash", "-c", script], + capture_output=True, + text=True, + check=False, + env=environment, + cwd=None if cwd is None else str(cwd), + ) @property def default_recipe(self) -> str: @@ -610,11 +706,46 @@ def test_the_preserving_recipe_removes_its_temporary_path_on_success_and_on_refu assert self._temporaries_beside(unreachable) == [] assert self._entries(unreachable) == {".git"} + @pytest.mark.parametrize("spelling", [".", "./"]) + def test_the_preserving_recipe_serves_the_destination_spelled_here(self, starter: Path, tmp_path: Path, spelling: str) -> None: + """The destination is usually `.`, and the recipe has to survive being told so. + + `mkdir my-app && cd my-app && git init` is the "Where" rule's own account of how a user + reaches a directory holding nothing but `.git`, and they then ask for the project *here* — + so `` binds to `.`, not to a path with a parent to speak of. Computing the parent from + that spelling gives `.` again, which puts the temporary directory inside the destination; + the `ls -A` line then finds it beside `.git` and refuses, every time, on the one case the + ruling was written to serve. Every other recipe test binds `` to an absolute path, + which is exactly why this went unnoticed until round 2. + """ + target = tmp_path / "my-app" + self._make_repository(target, "main") + (target / "NOTES.md").write_text("theirs\n", encoding="utf-8") + subprocess.run(["git", "-C", str(target), "add", "NOTES.md"], check=True) + self._commit(target, "the user's own commit") + (target / "NOTES.md").unlink() + head = subprocess.run(["git", "-C", str(target), "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout + + result = self._run(self.preserving_recipe, starter=starter, target=target, dir_literal=spelling, cwd=target) + + assert result.returncode == 0, result.stderr + assert self.TEMPLATE_ENTRIES <= self._entries(target) + assert self.DOTTED_ENTRIES <= self._entries(target) + # The temporary path was a sibling, and it was cleaned up. + assert self._temporaries_beside(target) == [] + assert [entry for entry in self._entries(target) if entry.startswith(".pipelex-starter-")] == [] + # And the user's repository is untouched: same commit, same branch. + assert subprocess.run(["git", "-C", str(target), "rev-parse", "HEAD"], capture_output=True, text=True, check=True).stdout == head + assert ( + subprocess.run(["git", "-C", str(target), "rev-parse", "--abbrev-ref", "HEAD"], capture_output=True, text=True, check=True).stdout.strip() + == "main" + ) + def test_the_temporary_path_is_beside_the_target_and_collision_proof(self, starter: Path, tmp_path: Path) -> None: """Beside, so the acquisition never crosses a filesystem or a small `/tmp`; named by `mktemp`, so two runs in the same parent cannot land on each other.""" recipe = self.preserving_recipe - assert 'tmp=$(mktemp -d "$(dirname )/.pipelex-starter-XXXXXX")' in recipe + assert 'tmp=$(mktemp -d "$(dirname "$dir")/.pipelex-starter-XXXXXX")' in recipe target = tmp_path / "my-app" self._make_repository(target, "main") # The name is generated, so the same recipe run twice in one parent must not collide. diff --git a/wip/pipelex-integrate/scaffold-review-deferrals.md b/wip/pipelex-integrate/scaffold-review-deferrals.md index ec868a0..6785def 100644 --- a/wip/pipelex-integrate/scaffold-review-deferrals.md +++ b/wip/pipelex-integrate/scaffold-review-deferrals.md @@ -3,7 +3,7 @@ status: active item: L-260906-8ac105 --- -# `pipelex-scaffold` — findings review round 1 confirmed and did not fix +# `pipelex-scaffold` — findings the review rounds confirmed and did not fix Round 1 of `/rev` on `pipelex-plugins#21` (branch `feature/Scaffold-skill`, 2026-09-13) ran cubic, the Codex review and adversarial passes, and the official `code-review`. What it fixed is in the pull request and the changelog. This file is the trace for what it confirmed and deliberately left, so none of it is a finding that merely evaporated. @@ -22,3 +22,13 @@ Everything below was read and verified in the tree — none of it rests on a rev - **Branch B's ownership sentence is broader than the steps beneath it.** "Nothing beyond what the initializer writes is authored by this skill" is glossed immediately with "no example code, no folder layout of its own, no opinion the framework did not ship", which scopes it to project shape — but steps 3 and 4 then author a `.gitignore` and the env pair. An agent following the explicit imperatives is not actually misled, which is why this is a wording imprecision rather than a defect. - **The minimal TypeScript recipe still defaults to `--module nodenext`.** Round 1 annotated the default row so the caveat is visible where the command is chosen, rather than twelve lines below it, but it did not change the command. Making the bundler form the default is a recipe change that should be executed before it ships — this campaign's own history is that recipes composed inside a review round became the next round's defects. - **`.env.example` ships an empty `PIPELEX_API_KEY=`, so the gated append can still leave two assignments.** Round 1 closed the defect that mattered — an append landing *after* a key the user had already filled, which every dotenv reader resolves to the later line. What remains is only the placeholder case, where the later line is the one the skill intends and the value is correct. A user editing the first line and seeing nothing change is the cost; rewriting in place instead of appending would need a `sed -i` whose BSD/GNU spelling differs, which is not worth trading a portability trap for a tidiness gain. + +## Deferred in round 2 + +Round 2 (2026-09-13, profile 4) read round 1's own fixes and the founder's ruling together. What it fixed is in the pull request and the changelog; these it confirmed and left, at a bar that admits confirmed defects and drops improvements. + +- **`` is a placeholder and the document never quotes it.** Raised by cubic as its highest finding, and the sharpest edge is real: `rm -rf /.git` in the default recipe would, under a `` holding a space, expand to `rm -rf my app/.git` and delete a sibling named `my`. Every shape was executed on this machine's BSD userland before deferring, and all of them fail closed — a space makes `$(dirname …)` emit two lines so `mktemp` fails; `ls -A my app` errors and the guard's string can never equal `.git`; a leading dash is eaten as `ls` options; and the destructive line is unreachable because `git clone my app` rejects the extra argument and `|| exit` fires first. The preserving recipe is quoted throughout as of this round, since resolving the destination gave it a variable to quote. What is left is a documentation-wide convention — quoting the placeholder in every recipe, or one sentence telling the agent to quote whatever it substitutes — which should be done across the file in one pass rather than at the one site a round happened to touch. +- **The recipes have more than one canonical copy.** cubic's observation that the skill body and `references/starters.md` both carry the acquisition and initializer commands, and that these have already drifted twice — round 1 found a guard pinned against the reference alone, round 2 found the pathspec and the double-clone on the other side. The diagnosis is right and the remedy is a structural change (one source the body, the reference and the tests all read), which is a decision rather than an edit and should not be made inside a review round. +- **The suite asserts prose sentences as well as behaviour.** cubic reads the exact-sentence assertions across the template and three renders as coupling that makes restructuring expensive. Declined on the merits rather than only on the bar: the skill *is* prose that instructs an agent, so a sentence going missing is the failure mode, and every half-application this branch has suffered was caught by exactly such an assertion. The executable recipe tests already carry the behavioural half. +- **The report is not required to say the commit landed on the user's existing branch.** Raised by `code-review`. The Mode section now confirms that commit before it is made, which puts the fact in front of the user at the moment it matters; saying it again afterwards is an improvement, and it applies identically to both branches, so it wants doing once for both. +- **A `` whose `.git` is a file — a linked worktree or a submodule working directory — is written into.** Weighed deliberately this round rather than merely noted, and left. Executed end to end: `ls -A` returns exactly `.git`, the guard passes, the copy lands, and the pristine commit goes onto that worktree's branch in the shared repository. It is left because it is still the user's own repository standing, because `git rev-parse --show-toplevel` prints `` itself for such a directory so branch B accepts the identical case already — and making branch A alone refuse would re-create the A/B disagreement `L-260913-f28d9d` was ruled to remove — and because the shape is barely reachable: a worktree or submodule directory normally holds its branch's files, so `ls -A` returns more than `.git` and the chain refuses before copying. Nothing is destroyed on the path that does reach the copy, and the commit is an ordinary one the user can reset.