Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5e7f519
✨ add dotnet-nuget-update skill
aicia-bot Sep 7, 2026
9861cc1
💬 update readme with dotnet-nuget-update entry
aicia-bot Sep 7, 2026
2b2dd03
✨ document dotnet-nuget-update in changelog and add version links
Sep 7, 2026
4be6ae5
♻️ improve dotnet-nuget-update script robustness
aicia-bot Sep 7, 2026
77f40fd
🐛 fix conflict detection and property resolution
aicia-bot Sep 7, 2026
0827577
🐛 fix xml entity decoding in condition value extraction
Sep 7, 2026
8f577f1
📝 clarify target-framework scanner documentation limits
aicia-bot Sep 8, 2026
1fbb072
♻️ refactor property resolution in get-targetframeworks
aicia-bot Sep 8, 2026
3fbb983
✅ add framework-scoping eval case with fixtures
aicia-bot Sep 8, 2026
6cafe93
✅ add test automation for target framework resolution
aicia-bot Sep 8, 2026
8e0ca2f
♻️ extract grouping examples to dedicated reference
aicia-bot Sep 10, 2026
93efc53
✅ add edge-case tests for untracked files and scope narrowing
aicia-bot Sep 10, 2026
287be79
💬 update changelog for 0.10.0 release
Sep 10, 2026
5e8f6dc
📝 clarify validation procedures and concurrent execution guidance
Sep 10, 2026
29f8fe1
✅ update validation assertions for git-visual-commits skill
Sep 10, 2026
5099139
💬 clarify optional eval preparation policy
gimlichael Sep 10, 2026
d6a9f55
✨ add section validation to git-keep-a-changelog
gimlichael Sep 10, 2026
035686e
✅ update validation assertions for git-visual-commits skill
gimlichael Sep 10, 2026
c0353a6
♻️ refactor dotnet-nuget-update for bounded parallel package audits
aicia-bot Sep 11, 2026
6d8e281
✅ add skill template validation tests for grouping references
aicia-bot Sep 11, 2026
e0cad5f
👷 eval-runners: add codex runner path validation tests
aicia-bot Sep 11, 2026
740e330
💬 update repository readme and release guidance
aicia-bot Sep 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 11 additions & 17 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Where `skill-creator` says to spawn with-skill and baseline subagents in the sam

### Asking for an eval

`eval <skill>`, `evaluate <skill>`, `eval this skill`, `prepare evals for <skill>`, and `evaluate <skill> using the existing evals` are all requests for this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too.
`eval <skill>`, `evaluate <skill>`, `eval this skill`, `prepare evals for <skill>`, `evaluate <skill> using the existing evals`, `please do an eval`, and similar explicit evaluation requests all invoke this workflow. Treat them as instructions to prepare the package, never to run it, and never as a request to write new eval cases unless the user asks for that too.

Resolve the execution configuration before running the package preparation script. Normalize explicit user intent immediately and do not ask again for a harness the user already supplied: `Codex` -> `codex`; `GitHub Copilot`, `GitHub Copilot CLI`, or `Copilot` -> `github-copilot`; `OpenCode` -> `opencode`; matching is case-insensitive. In an interactive agent session, offer Codebelt Reference first (`GitHub Copilot CLI` + `claude-haiku-4.5`) and let `scripts/prepare-skill-evals.ps1` validate that model internally; if it is unavailable, show the current discovered Copilot models and ask for a replacement. If the user selects Codex without a model, pass `-Runner codex` and let preparation resolve repository defaults (`gpt-5.6-luna` with low reasoning) and validate the model. For manual selection, ask for Harness only when the user did not already name one, then discover current models for that harness with `scripts/Get-HarnessModels.ps1`. OpenCode discovery mirrors every model exposed by all configured OpenCode providers; it exposes exact `provider/model` selectors and retains display and availability metadata when available, but availability is presentation metadata only and never filters the selectable catalog. If OpenCode is selected and no model was explicitly supplied, present every discovered selector to the user, ask the user to choose one, and stop until that choice is made. Do not choose the first, free, recommended, previous-iteration, previous-successful, or previous-failed model on the user's behalf. If the user explicitly supplies an OpenCode selector, preserve it verbatim in `execution-profile.json`; discovery may verify or annotate it, but discovery failure or incomplete metadata must never substitute another model. GitHub Copilot and Codex discovery lists all currently available models. Never guess stale model ids, silently switch harnesses, or generate an execution-ready package with a null runner or model.

Expand Down Expand Up @@ -91,21 +91,13 @@ Useful switches: `-Eval <id...>` to prepare a subset, `-Iteration <n>` plus `-Fo

The expected output and the assertions are the grading key. They belong in `eval-metadata.json`, outside every run directory, and must never appear in either prompt — a baseline handed the answer key is not a baseline.

### Eval preparation is a completion gate
### Optional eval preparation

Adding or modifying any repo-managed skill triggers this workflow. It is not something the user asks for separately, and "the change is small" or "the evals did not change" does not exempt it. Touching `SKILL.md`, `FORMS.md`, `references/`, `scripts/`, `assets/`, or `evals/` under `skills/<name>/` is a skill change.
Eval package preparation is optional. A skill change does not require package preparation, and it does not require choosing a harness or model. Prepare a package only when the user explicitly asks for an evaluation, such as `eval <skill>`, `evaluate <skill>`, `please do an eval`, or a similar request. A request to create, modify, validate, or release a skill does not implicitly ask for eval preparation.

After the final skill edit is in place, run:
When the user explicitly asks for an eval, follow [Asking for an eval](#asking-for-an-eval), resolve the Harness + Model choice, and run the preparation command described there. Preparation remains model-free and never executes the generated prompts. If a package is prepared, name its prompt path in the completion message and hand it over as described below. Do not ask for a harness, prepare a package, or report an eval handoff solely because a skill changed.

```
pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>
```

It resolves every repo-managed skill this branch changed, uncommitted work included, and prepares a package for each. With no skill changed it says so and exits clean, which satisfies the gate.

Then name the prepared prompt paths in the completion message so the user knows what is waiting for them. Preparing and reporting satisfies this gate. Executing a prompt never does, and an agent that runs one has broken the Priority 1 rule rather than completed the gate.

Run it before `scripts/sync-skill-install.ps1`, which stays the last gate because it must observe the final state of every file. See [Blocking Completion Gates](#blocking-completion-gates).
`scripts/sync-skill-install.ps1` remains the last gate when a repo-managed skill was edited. An explicitly requested eval adds package preparation before handoff, but package preparation is not a completion gate. See [Blocking Completion Gates](#blocking-completion-gates).

### Manual execution boundary

Expand Down Expand Up @@ -181,13 +173,15 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills
- When `files` is present, keep the paths relative to `skills/<name>/` and validate that every fixture exists
- Treat eval prompts, expected outcomes, and assertions as versioned review specifications; their presence never authorizes automated model execution
- Start with `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -MetadataOnly` for a sub-second repository-wide metadata and fixture check
- Run only the changed skill's deterministic validator and focused regression scripts during iteration; independent read-only checks may use bounded local parallelism, while shared-file mutations stay sequential
- Run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` once before completion for the repository gate
- Run only the changed skill's deterministic validator and focused regression scripts during iteration; run independent checks concurrently with bounded local parallelism, while shared-file mutations stay sequential.
- For the repository completion gate, run the independent suites from `.github/workflows/validate-skill-templates.yml` concurrently. Use that matrix as the source of truth for script paths and `-Suite` values, including the transport-specific conformance suites and phase-specific integrity suites. Use separate PowerShell 7 processes, bounded concurrency with at least two workers, isolated temporary fixtures, and per-suite exit codes. Include `-Full` for DocFX when required by the task. Every required suite must pass.
- Do not invoke the sequential aggregate `scripts/validate-skill-templates.ps1` with no `-Suite` or with `-Suite All` for agent-driven validation. Do not run the CI suite matrix in a sequential loop. If concurrent execution is unavailable, report the limitation instead of silently falling back to sequential execution.
- An explicit user instruction to skip or stop testing overrides these validation gates for that session. Stop any tests you started, run no further validation, and report the unvalidated work.
- Follow the top-level **AI/LLM Evaluation Automation Prohibition** for every eval. No per-skill or third-party requirement overrides it.
- To compare a skill against a baseline, prepare a package with **Portable Eval Handoff** and hand `RUN-THIS.prompt.md` to the user; the repository agent never runs the prompts, while the user-directed external executor runs, grades, and reports the paired comparison
- Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them

If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` and run `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>` before considering the work complete. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts.
If you add a new skill or modify an existing repo-managed skill, keep that skill's `evals/evals.json` present and update it when the eval definitions or fixtures change. Package preparation is optional and is triggered only by an explicit eval request; when requested, use `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>`. Use `-CodebeltReference` instead only after its dynamic Copilot model check passes. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts.

## Git Identity

Expand Down Expand Up @@ -317,7 +311,7 @@ Before any completion message, reread the skill instructions and the current con

For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required.

Whenever a repo-managed skill was edited, two gates apply in a fixed order. `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>` (or `-CodebeltReference` after dynamic availability verification) runs first and prepares the eval packages for the changed skills, reporting the prompt paths. `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report the actual output of both; an earlier run in the same session satisfies neither. See [Eval preparation is a completion gate](#eval-preparation-is-a-completion-gate) and [Local Install Sync](#local-install-sync).
When a repo-managed skill was edited, `scripts/sync-skill-install.ps1` runs last, because every other step can still change a file. Report its actual output; an earlier run in the same session satisfies neither. If the user explicitly requested an eval, prepare the package before handing it over as described in [Optional eval preparation](#optional-eval-preparation), but do not treat that preparation as a completion gate. See [Local Install Sync](#local-install-sync).

## User Input UX

Expand Down
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,32 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.10.0] - 2026-09-11

This is a minor release that introduces `dotnet-nuget-update`, a deterministic NuGet dependency audit and update workflow for .NET repositories. The skill owns the complete-audit invariant, ensuring every declared package version is accounted for before any update is applied. It handles both central package management (`Directory.Packages.props`) and project-level `PackageReference` versioning, tracks stable and prerelease intent, preserves TFM-band pins (keeping `net9` or `net10` packages within their matching major when that major is the compatibility signal), and supports both normal mode (auto-applies patch/minor/revision, batches majors for approval) and yolo mode (auto-applies safe classes only, reports held majors). All scripts are deterministic and offline-testable via bundled fixtures. Alongside that, `git-visual-commits` gained refactoring and three new eval cases for edge-case coverage, `git-keep-a-changelog` gained improved entity-classification infrastructure and a dedicated reference document for section validation, and repository-level validation enhancements ensure all skills conform to updated testing requirements.

### Added

- `dotnet-nuget-update` skill for auditing and updating NuGet dependencies with complete declaration accounting, supporting both central package management and project-level versioning, two interactive modes (normal with approval batching, yolo for safe updates only), stable/prerelease intent inference, and TFM-band awareness so conditional `net9`/`net10` package declarations stay within their matching major when that major is the compatibility signal rather than jumping to the newest overall release,
- bundled deterministic scripts for `dotnet-nuget-update`: `Get-DependencyAudit.ps1` for complete graph enumeration before any edit, `Get-PackageGraph.ps1` for central-package condition resolution, `Get-TargetFrameworks.ps1` for TFM matrix discovery, `Resolve-NuGetVersion.ps1` and `Compare-Version.ps1` for version investigation, `Apply-PackageUpdates.ps1` for minimal structural XML edits preserving comments and line endings, and `Get-NuGetSources.ps1` for feed configuration visibility,
- comprehensive test coverage for `dotnet-nuget-update`: regression harnesses for dependency audit, package graph, TFM-band logic, project-level package references, version comparison, and update application, together with offline-testable fixtures covering central-package scenarios, mixed stable/prerelease intent, multi-TFM bands, plain project references, and XML comment pinning,
- per-process memoization for live or offline flat-container NuGet version feeds in `dotnet-nuget-update`, with filesystem flat-container fixtures supporting deterministic offline testing,
- complete-audit invariant validation in `dotnet-nuget-update` ensuring `current + auto + approval + unresolved == declared` before reporting the repository as updated,
- history-aware update tracking in `dotnet-nuget-update` with adjacent XML comments surfacing as `note` fields so pin rationale remains visible during audits and auto-updates marked with `READ THE NOTE before applying` receive explicit attention,
- README catalog entry, installation snippet, and "Why dotnet-nuget-update?" community health section explaining the need for complete dependency graph auditing, TFM-band awareness, and preservation of intentional pins and compatibility markers,
- `references/section-validation.md` in `git-keep-a-changelog` documenting the section-classification contract for resolved release entities,
- test harness `scripts/test-resolve-release-entity.ps1` in `git-keep-a-changelog` validating entity-classification behavior across multiple release scenarios.

### Changed

- `git-visual-commits` SKILL.md refactored for improved clarity, with single-category quality-gate guidance hardened and commit-language best practices restructured to emphasize inventory refresh and per-path explanation before classification,
- `git-visual-commits` evals expanded with three new eval cases (26, 27, 28) covering untracked file discovery, path handling with spaces and untracked directories, and explicit scope narrowing,
- `git-keep-a-changelog` SKILL.md improved with clarifications on entity classification workflow, pending-change handling, and section-validation procedures,
- `git-keep-a-changelog` evals updated to reflect improved entity-classification guidance and concurrent validation requirements,
- Repository validation infrastructure in `scripts/validate-skill-templates.ps1` hardened with concurrent suite dispatch, bounded concurrency of at least two workers, per-suite exit codes, and explicit prohibition on sequential aggregate runs,
- Repository agent guidelines in `AGENTS.md` clarified with explicit concurrent execution guidance, per-suite validation, and user-instruction overrides for testing suspension,
- README.md documentation updated with concurrent validation procedures and explicit messaging about running independent validation suites in parallel with bounded concurrency.

## [0.9.1] - 2026-09-07

This patch release adds harness-agnostic Eval Runner execution boundary infrastructure without changing the paired evaluation methodology or existing report schemas, while optimizing skill descriptions and refactoring repository-level authoring guidance. Prepared packages now carry `execution-profile.json`, package-local runner protocol tools, and normalized `execution-result.json` evidence. The deterministic fake runner is the conformance reference, with Codex, GitHub Copilot CLI, and OpenCode as supported real adapters. Repository automation remains model-free; only a human-directed external Eval Orchestrator may invoke the selected runner, and unsupported isolation fails closed.
Expand Down Expand Up @@ -638,6 +664,7 @@ This is a minor release that introduces two complementary git workflow skills, e

- Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output.

[0.10.0]: https://github.com/codebeltnet/agentic/compare/v0.9.1...v0.10.0
[0.9.1]: https://github.com/codebeltnet/agentic/compare/v0.9.0...v0.9.1
[0.9.0]: https://github.com/codebeltnet/agentic/compare/v0.8.2...v0.9.0
[0.8.2]: https://github.com/codebeltnet/agentic/compare/v0.8.1...v0.8.2
Expand Down
8 changes: 5 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,15 @@ Evals let you verify the skill works and measure improvement over a baseline. Ev

Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something.

Evals are prepared, not executed, from this repository. Adding or modifying a repo-managed skill requires preparing the packages for every skill the branch touched, which is a completion gate rather than an optional extra:
Evals are prepared, not executed, from this repository. Package preparation is optional and happens only after an explicit eval request, such as `eval <skill>`, `evaluate <skill>`, or `please do an eval`. Adding or modifying a repo-managed skill does not require a package, a harness choice, or a model choice.

When an eval is requested, prepare the changed packages with:

```console
pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner github-copilot
```

Run it after the last skill edit and before `scripts/sync-skill-install.ps1`, which stays last. For a single skill on demand, use:
For a single skill on demand, use:

```console
pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Skill <skill-name> -Runner <runner-id> -Model <runner-native-model>
Expand Down Expand Up @@ -149,7 +151,7 @@ pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD
- [ ] At least one eval in `evals/evals.json`
- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name
- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder
- [ ] `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>` or `-CodebeltReference` was run after the last skill edit, and the prepared prompt paths were reported
- [ ] If an eval was explicitly requested, `pwsh -NoProfile -NonInteractive -File ./scripts/prepare-skill-evals.ps1 -Changed -Runner <runner-id> -Model <runner-native-model>` or `-CodebeltReference` was run, and the prepared prompt paths were reported
- [ ] If an external evaluation was run, each result includes the producing model and the package contains the first-party `report.html`, exact upstream `skill-creator-report.html`, `benchmark.json`, and `benchmark.md`; use `-CollectResults` only for explicitly authorized forensic recovery of an existing package
- [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior
- [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too
Expand Down
Loading
Loading