self-audit.md

Self-Audit

Audience: Everyone Source of truth: schema/examples/self-audit/2026-11-09.json (the engine's report for this repository) and the root .audit-baseline.json (the same report, kept as the CI ratchet). This page restates the report; the report wins. Status: Live. Regenerated by hand when the baseline changes (see Keep it true).

---

The framework runs its own deterministic engine on itself, read-only, and publishes what it finds. The result is not a pass: it is the Phase 1 score of this repository, the findings that remain, and a CI ratchet that stops the score from falling.

The result

Field	Value
Phase-1 score	0.9264
Date	2026-11-10
Engine	python-agentic-audit 0.3.2
Ruleset	2026-11-10.2 (hash 9b2a2e2562a10c7a)
Applicable checks	74 (60 scoreable)
Findings	1: DEGRADER
Scope	the whole repository except artefacts/ and dist/; no user-level configuration read

Since the 2026-11-06 report (0.8402): HYG-02 closed with .github/CODEOWNERS and HYG-06 with the security policy, SECURITY.md. On 2026-11-09, TOOL-04 closed with the dependency audit in make security (ADR 0007), and SEC-01 and SEC-03 closed when the .env rules in .gitignore began to match at any depth, and IDX-01 closed when .mcp.json began to register codebase-memory-mcp. EXEC-01 and EXEC-07 opened as owner exceptions. EXEC-01 closed with ruleset 2026-11-09.8, which reads the go directive of a nested go.mod (dev/tools/actionlint/go.mod). EXEC-07 closed with ruleset 2026-11-09.11, which reads pyrightconfig.json as a language-server mirror of the mypy gate. The checks added since, CI-05 (workflow files are linted), EXEC-08 (tool versions visible to dependency automation), TOOL-07 (local branch guard), AGT-12 (communication rules stated) and DOC-05 (docs are readable), all pass.

The score was 0.6762 on 2026-11-05, before the repository adopted the Phase 1 requirements it asks of others (see the changelog and the engine's ruleset history).

Verdict	Checks
PASS	67
PARTIAL	3
FAIL	0
UNKNOWN	4
ATTEST	0
Total	74

UNKNOWN and ATTEST earn no credit. Four checks are UNKNOWN because their evidence needs a Tier-C probe (--run-gates, not used here) or the user-level configuration that --no-user-config leaves unread, so this score understates the repository rather than certifying it.

What is not a pass

Every check that is not PASS, with the reason it is open.

Check	Verdict	Severity	Why it is open
BASE-02	PARTIAL	DEGRADER	The T5 playbook was applied (D32): .quality-baseline.json holds the counts (ruff 0, mypy 0, Markdown lint outside the ratchet 735) and make format-check, make typecheck and make docs-lint fail when one goes up. The check still reads PARTIAL by design: it credits only a numeric budget flag (--max-warnings N), and none of these tools has one. Engine issue #45 tracks it; the coverage floor (91) is unchanged and BASE-04 passes
EXEC-02	PARTIAL	BLOCKER	No finding: the lockfiles of the pinned dev tools (dev/tools/uv.lock, dev/tools/package-lock.json and the go.sum of each Go tool module, dev/tools/osv/ and dev/tools/actionlint/) cannot be checked against their manifests without running the package managers. dev/tests/test_tool_pins.py checks that they agree; the engine does not read that test
NAV-05	PARTIAL	COSMETIC	No finding, informational and unscored (D34): 40% of public symbols documented (476 of 1188). Reviewed on 2026-11-09 and left as it is: a ruff D1 ratchet would baseline about 700 violations for a cosmetic check, and mypy strict and the tests carry the meaning
IDX-02, IDX-03, TST-02, CON-04	UNKNOWN	mixed	Need a probe, the user-level index store, or --emit-baseline; not run here

No owner exception is open. The IDX-01 exception (D21), the TOOL-04 exception (D33) and the EXEC-01 and EXEC-07 exceptions ended on 2026-11-09: .mcp.json now registers codebase-memory-mcp, make security now runs a dependency audit, the engine reads a nested go.mod, and it reads pyrightconfig.json as a mirror of the mypy gate. AGT-08 stays as it is for v0.2, because a stated harness list closes the gap even when a named harness does not read AGENTS.md natively, and is revisited in v0.3 (D31); NAV-05 is informational and unscored (D34; reviewed again 2026-11-09). The engine has no way to record an owner exception, so this page is the record. Everything else is either a deliberate design choice stated in the repository, a value waiting for the owner, planned work or open debt; none is hidden in the score.

The ratchet

make self-audit runs the engine against .audit-baseline.json. The comparison is by finding id (ADR-0001): the run exits 2 only when a finding appears that the baseline did not have, and 0 otherwise. Fixing a finding never fails the run. CI runs it as the self-audit job on Linux, and CI Gate needs it.

To check that the ratchet bites, a scratch copy of the tracked files with .python-version removed makes the run exit 2 and name the regression EXEC-01:python.

Reproduce

From the repository root, with Python 3.11+ and uv (see make setup):

make self-audit             # reads the repository, writes dist/self-audit/audit-report.{json,md}; exit 2 on a regression
cat dist/self-audit/audit-report.md

Nothing is written inside the tracked tree: dist/ is git-ignored. The report differs from the published one only in provenance.scanned_at, the target and the commit (see Neutralised path).

Keep it true

When a finding is fixed, accept the smaller baseline on purpose:

make update-audit-baseline  # rewrites .audit-baseline.json from a fresh run; review the diff

Never run it to get a failing make self-audit green: a new finding is fixed or recorded with an owner reason here first. After updating it, copy the report to schema/examples/self-audit/<date>.json, update the tables above, and run make update-secrets-baseline (the report holds a ruleset hash and a git SHA that read as high-entropy strings), reviewing the diff.

Neutralised path

The framework repository is public, so the committed baseline and the published example are the engine's output with two hand edits: provenance.target is . (the engine writes the absolute local path), and the IDX-02 summary quotes <repository>/.codebase-memory/graph.db.zst instead of the absolute index path. Nothing else is changed. make update-audit-baseline copies the raw report, so follow it with the same two edits before committing:

sed -i 's#"target": "[^"]*"#"target": "."#; s#[^ ,"]*/\.codebase-memory#<repository>/.codebase-memory#' .audit-baseline.json
grep -c '/home/' .audit-baseline.json   # must print 0 (a report from `make self-audit` also has a `baseline` path: use the update target's)

The ratchet compares by finding id, so the edit does not affect it. The engine should write a neutral value itself; that request is issue #49, and the edit goes when it is fixed.