Repository navigation
chore: close the v0.3.5.1 open ends - #109
Merged
Merged
Conversation
The notes in 80 of the files named the old engine path brownfield-recipes/deterministic-audit/, and every file named the generator as tools/eval/. The generator and its test now name dev/eval/ and the real output folder, and all files are regenerated. Roadmap-Step: 1.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make eval-expected-check runs generate_fixture_expected.py --check, which compares the committed eval-expected files with a fresh engine run (a few seconds). It joins make check and the CI static job, so a stale file can no longer reach dev unseen. The docs that list the parts of check say so. Roadmap-Step: 1.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REQ-SH-030 said the security verb MUST run the dependency scan, while TOOL-04 passes a scan under any verb. The owner chose the check's reading: a verb MUST run the scan and it SHOULD be security. Spec decision 0003 records why; no verdict moves. Roadmap-Step: 3.1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The audit bundle rewrote only the link to the shared prompt. It now uses the same rewrite as the implement bundle. A link to a carried file points at its place in the bundle. Any other link becomes plain text naming the framework path. The bundle also carries the Tools CLI sections for its five tools, so the SKILL.md tool links resolve. Both copies of the schema docs are rewritten for their own folder. A new test holds every link in the bundle resolvable. Roadmap-Step: 1.3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hook that read the branch and only printed a warning passed TOOL-07, though the commit went ahead. A guard that reads the branch must now also exit non-zero (exit 1 to 255, process.exit(1), sys.exit(1) or a bare false) in its lefthook command block or hook script; comments are skipped. no-commit-to-branch always stops the commit. No wired guard that exits is PARTIAL, and branch coverage is read only on the guards that exit. Ruleset 2026-11-10.3; no catalogue bump. One-defect fixture tool-07-guard-no-exit (a lefthook guard that only warns, PARTIAL) with its eval file, six in-process tests, rubric and changelog. Self-audit: TOOL-07 still PASS (the lefthook guard exits 1). Roadmap-Step: 4.4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aseline The eval file of the new TOOL-07 fixture holds the fixture's fixed repo_sha, which detect-secrets reads as a high-entropy string. The gates of the 4.4 commit ran before the new file was staged, so make security did not scan it. The baseline now lists it, like the other eval files. Roadmap-Step: 4.4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One pinned nested module passed EXEC-01 even when a sibling module had no pin. The check now reads every module manifest (package.json, pyproject.toml, go.mod) of a pinned ecosystem. A module is reached by a pin in its own manifest, or by a pin file or (node, python) a pinning manifest in its folder or a parent folder; a parent go.mod does not reach a nested Go module. A module no pin reaches is PARTIAL and is named in the evidence. Two tests that asserted PASS for exactly this gap now assert PARTIAL; their single-module cases are kept. Ruleset 2026-11-10.4; no catalogue bump. One-defect fixture exec-01-sibling-module-unpinned with its eval file and baseline entry, ten in-process tests, rubric, audit master list and changelog. Self-audit: EXEC-01 still PASS. Roadmap-Step: 4.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s docs Step 8 of add-repo-hygiene told the agent to confirm a guessed Renovate form. Renovate's docs give a preset for it: security:only-security-updates turns off other updates with matchPackageNames ["*"] and turns on vulnerabilityAlerts and osvVulnerabilityAlerts. The step now writes that preset with minimumReleaseAge "7 days", says vulnerabilityAlerts works on GitHub only, and names the osv.dev limits on other platforms. Roadmap-Step: 2.2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The playbook called the GitLab variable names assumed. GitLab's predefined variables list CI_PIPELINE_SOURCE, CI_MERGE_REQUEST_SOURCE_BRANCH_NAME and CI_MERGE_REQUEST_TARGET_BRANCH_NAME with the meaning the job uses. The step now cites the page, says the branch names exist only in merge request pipelines, and warns that a pipeline with only .pre jobs does not run. Roadmap-Step: 2.3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's list of supported ecosystems has pre-commit: Dependabot updates the rev: lines of .pre-commit-config.yaml, with version updates only and no security updates. Step 8 of add-repo-hygiene now adds a pre-commit entry when the file exists and the policy is not security-only. The coverage bullet is rewrapped, and its 45-word sentence is split. Roadmap-Step: 2.4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
progress.py wrote the framework commits and the source reports' git_sha as 12 digits, but copied --commit and --merged-sha as given, so a full 40-digit id could land in the state, where secret scanners read it as a token. Both are now cut to 12 digits; a shorter id (7 to 11) is kept. The schema still accepts 7 to 40, so its version does not change. state.md, the Tools CLI reference and two new tests say so. Roadmap-Step: 1.2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the small open ends left after v0.3.3 (plan v0.3.5.1). Two engine verdict changes (rulesets
2026-11-10.3and.4), one spec decision, one new gate step, and playbook facts checked against the vendors' docs.What changed
0ccd527,d25eae4make eval-expected-checkjoinsmake checkand the CIstaticjob (about 6 s).ddf8b05progress.pystores--commitand--merged-shacut to 12 digits, like the other commit fields. Schema unchanged (7 to 40).6a98a5daf0b6a1security:only-security-updates).18ec3b3.pre-only pipelines do not run.9db8c0bpre-commitecosystem (version updates only) added to the hygiene playbook.5182077REQ-SH-030: a verb MUST run the dependency scan, and it SHOULD besecurity(spec decision 0003). No verdict moves.71b4e61,54c10e0TOOL-07: a guard that reads the branch must exit non-zero, orPARTIAL. Ruleset2026-11-10.3, fixturetool-07-guard-no-exit.86cab41EXEC-01: a module manifest that no pin reaches givesPARTIAL. Ruleset2026-11-10.4, fixtureexec-01-sibling-module-unpinned.Verdicts that move: a repository whose only branch guard prints a warning drops from
PASStoPARTIALonTOOL-07. A monorepo where one pinned module covered an unpinned sibling drops fromPASStoPARTIALonEXEC-01. In the 4.5 commit, twoEXEC-01tests that assertedPASSfor exactly that gap now assertPARTIAL; their single-module cases are kept.How it was checked
make check,make security,make docs-lint,make self-auditandmake coverage: passed before each commit.6a98a5dwas gated on its own base; the next run covered the combined tree.make validate-skills: both bundles valid.make commit-lint: 11 subjects pass.TOOL-07andEXEC-01stayPASSon this repository.Not in this PR
docs/reference/eval-results.mdandtools-cli.mdstill nametools/eval/score.py(found, not fixed).🤖 Generated with Claude Code