diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..7e27ac6 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,52 @@ + + +## What + + + +## Ledger + + + +- Task: +- Bug: +- Decision: +- Jira: + + + + + +## Evidence + + + +- [ ] `make test-unit` passes +- [ ] `make test` passes +- [ ] New logic has tests in this PR (or: N/A because …) +- [ ] Acceptance Criteria in the ledger file are checked off, with evidence recorded **in the file** + +## Risk + + diff --git a/.github/workflows/ledger.yml b/.github/workflows/ledger.yml new file mode 100644 index 0000000..e9893d0 --- /dev/null +++ b/.github/workflows/ledger.yml @@ -0,0 +1,71 @@ +# The first CI this repo has had. See ADR-0034, and +# docs/bugs/open/bug-repo-does-not-meet-own-standards.md — a repo whose purpose +# is enforcing lint and tests on other repos had neither for itself. +name: tests and ledger + +on: + push: + branches: [main] + pull_request: + +permissions: + contents: read + +jobs: + tests: + name: unit tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v5 + with: + enable-cache: true + + - name: Set up Python + run: uv python install 3.11 + + - name: Install dependencies + run: uv sync --all-extras --dev + + - name: Run unit tests + run: make test-unit + + ledger: + name: ledger consistency + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # Need the base commit to diff against for the companion check. + fetch-depth: 0 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install PyYAML + run: pip install --quiet pyyaml + + # Blocking: the ledger must be internally consistent. A broken link or a + # status that disagrees with its directory is a real defect. + - name: Validate the ledger + run: python3 scripts/check_ledger.py --all + + # Advisory for now: warns when a code change has no task/bug/ADR + # companion. Flipping this to blocking is its own task — + # docs/tasks/pending/task-make-ledger-check-blocking.md + - name: PR companion check (advisory) + if: github.event_name == 'pull_request' + continue-on-error: true + env: + PR_BODY: ${{ github.event.pull_request.body }} + BASE_SHA: ${{ github.event.pull_request.base.sha }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: | + printf '%s' "$PR_BODY" > /tmp/pr-body.txt + python3 scripts/check_ledger.py \ + --diff "${BASE_SHA}..${HEAD_SHA}" \ + --body /tmp/pr-body.txt diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..e6584fd --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,311 @@ +# AGENTS.md — Operational Handbook + +The handbook for anyone working in this repo, human or agent. `CLAUDE.md` holds the +never-violate runtime rules and the script reference; this file holds the process. + +Read this before your first change. Read [`PLAN.md`](PLAN.md) to find out what to work on. + +--- + +## 1. What this system is + +Given an approved RHAISTRAT strategy and its epic decomposition, generate implementation code +against target repos, review it, and open a PR — autonomously. One parameterized epic per run. + +``` +RFE (rfe-creator) + → Strategy (strat-creator) + → Epic decomposition (epic-creator) + → Code generation (epic-code-gen) ← this system + → PR on target repo → CI → human review → merge +``` + +Tracked in Jira under [RHAIFIRST-168](https://redhat.atlassian.net/browse/RHAIFIRST-168) +("Epic Code Generation — Hardening & Production Rollout"). + +### The four repos + +| Repo | Host | Role | +|---|---|---| +| **epic-code-gen** | GitHub `ederign/epic-code-gen` | The brains. Skill, agents, all orchestration logic. **The ledger lives here.** | +| **epic-code-gen-pipeline** | GitLab `redhat/rhel-ai/agentic-ci/` | Thin GitLab CI shell. Clones this repo, runs it, pushes results. | +| **epic-code-gen-pipeline-data** | GitLab `redhat/rhel-ai/agentic-ci/` | Git-as-database. Durable state + per-epic artifacts. Machine-written. | +| **epic-code-gen-dashboard** | GitLab `redhat/rhel-ai/agentic-ci/` | Reads the data repo, publishes to GitLab Pages. | + +Design rationale for the split is [ADR-0005](docs/decisions/). Full topology is +[`docs/architecture/01-system-overview.md`](docs/architecture/01-system-overview.md). + +--- + +## 2. The ledger + +This repo uses a filesystem-native work ledger, adapted from +[jctanner's Agent Work Ledger](https://gist.github.com/jctanner/7f1d5f132cf3f9b7fc67fbb3e3c8ff4c). +It exists so that **project state can be understood without chat history**, and so that an agent +which has lost all context can resume. + +``` +PLAN.md Navigation index. Not the plan. +docs/ + architecture/ How the system is built. Explanatory, not chronological. + decisions/ ADR-NNNN-*.md — why it is built that way. + plans/ Phase plans, the arc of the work. + milestones/ Groupings that map to Jira epics. + tasks/{pending,current,blocked,done}/ + bugs/{open,fixed,wontfix}/ + notes/session-log.md Dated activity log. +``` + +### Five principles + +1. **`PLAN.md` is an index, not the plan.** It links out. It never grows content of its own. +2. **Tasks are files.** One file per meaningful unit of work. +3. **State is location.** A task's status is the directory it sits in. Change status with + `git mv`, so the transition appears in the diff and in `git log --follow`. +4. **Decisions are recorded.** If you chose between real alternatives, write an ADR. Chat history + and commit messages are not durable enough — that gap is why this ledger exists. +5. **Bugs are first-class.** Record a defect **when you discover it**, even if you are not fixing + it, even if you caused it. An unrecorded bug is indistinguishable from a bug nobody knows about. + +### Frontmatter + +Every ledger file starts with: + +```yaml +--- +id: task-deterministic-scoring # kebab-case; must equal the filename minus .md +title: Compute review scores from findings, not reviewer judgment +type: task # task | bug | adr | milestone | plan +status: done # must agree with the directory it is in +repos: [epic-code-gen] # which repos this touches +jira: RHAIFIRST-374 # omit if unticketed +commits: [a7326fe, 788f16f] # REQUIRED for done/fixed. Evidence. +decisions: [ADR-0022] # optional cross-links +--- +``` + +This is deliberately **not** registered in `scripts/artifact_utils.py` `SCHEMAS`. That module +governs pipeline runtime artifacts; coupling docs to it would let a doc typo fail a codegen run. +`scripts/check_ledger.py` validates ledger frontmatter instead. + +Link related ledger files inline with `[[id]]`. A `[[id]]` that doesn't resolve yet is fine — it +marks something worth writing. + +### Templates + +**Task** — `docs/tasks//.md` + +```markdown +# Task: + +## Goal +## Context +## Acceptance Criteria +- [ ] ... +## Files Likely Involved +## Status +## Notes +``` + +**Bug** — `docs/bugs/<state>/<kebab-title>.md` + +```markdown +# Bug: <title> + +## Summary +## Reproduction +## Expected +## Actual +## Impact <!-- Critical | High | Medium | Low --> +## Related Tasks + +<!-- Optional, and strongly preferred when they exist: --> +## Observed incident <!-- dates, job URLs, trace excerpts --> +## Evidence <!-- artifact paths, file:line, reproduction on a clean checkout --> +``` + +`## Observed incident` and `## Evidence` are local additions to the upstream template. The best +bug reports in this project's history (RHAIFIRST-374, 391, 392) had them, and dropping them would +have thrown away the forensics that made those bugs fixable. + +**ADR** — `docs/decisions/ADR-NNNN-<kebab-title>.md` + +```markdown +# ADR-NNNN: <title> + +## Status +<!-- Proposed | Accepted | Accepted, under review | Superseded by ADR-NNNN | Rejected --> +## Context +## Decision +## Consequences +### Positive +### Negative +``` + +Number ADRs sequentially, never reuse a number, never renumber. Superseding an ADR means writing a +new one and editing the old one's Status — not editing the old one's Decision. + +--- + +## 3. The PR companion rule + +> **Every PR must reference at least one ledger file in its `## Ledger` section.** +> +> - New or changed behavior → a file in `docs/tasks/`, moved to `done/` in the same PR that lands +> the work. +> - A defect → a file in `docs/bugs/`, created **when discovered** (even if not fixed), moved to +> `fixed/` by the PR that fixes it. +> - An architectural decision → an `ADR-NNNN` in `docs/decisions/`, added in the same PR as the +> change it justifies. +> - Docs-only, typo, or dependency-bump PRs: write `Ledger: none — <reason>`. +> +> **A task or bug is not done until its Acceptance Criteria are checked and the evidence — commit +> SHAs, test names, job URLs — is recorded in the file. Do not declare success in the PR body and +> leave the ledger file empty.** + +That last paragraph is the whole point. The failure it prevents is real and recent: in +RHAIFIRST-391 the orchestrator wrote its own review files, dismissed a reviewer's Critical +finding, estimated scores in prose rather than computing them, and opened a PR — and the job +exited 0 reporting success. Recording evidence is what makes "done" falsifiable. + +`scripts/check_ledger.py` enforces a weak form of this in CI. It is **advisory** today +(warns, doesn't block) while the backlog is still being seeded; flipping it to blocking is its own +pending task. The rule is not advisory. + +### Companion Jira — on demand + +Every feature and bug should have a companion Jira issue under +[RHAIFIRST-168](https://redhat.atlassian.net/browse/RHAIFIRST-168), **opened when it is needed, not +eagerly.** The ledger file comes first and is always required; the Jira issue is the outward-facing +half and gets created when someone outside this repo needs to see it. + +Open one when any of these is true: + +- The work is being planned, scheduled, or reported on outside this repo. +- Someone else needs to be assigned to it, or it needs to block/relate to other Jira work. +- It is a defect with real consequences that a stakeholder should know about. +- You are about to start work on it. + +Don't open one for: internal hygiene the team already agreed on, a bug you are fixing in the same PR +that found it, or anything that would exist only to satisfy a rule. + +When you do open one: + +1. Add `jira: RHAIFIRST-NNN` to the ledger file's frontmatter. +2. Put the ledger path in the Jira issue, so the link is bidirectional. +3. Keep Jira the **summary** and the ledger file the **detail** — do not maintain the same prose twice. + The Jira description should be enough to triage; the ledger file is where the evidence lives. + +`check_ledger.py` deliberately does **not** require `jira:`. A `done` or `fixed` file needs evidence — +commit SHAs *or* a Jira key — so unticketed work can still close honestly on commits alone. That is why +several backfilled tasks carry commits and no Jira: the work shipped before anyone thought to file it, +and inventing tickets after the fact would be theatre. + +Cross-links go stale silently. If a Jira issue's status and its ledger file's directory disagree, the +ledger is what you are looking at — fix whichever is wrong, and prefer moving the file to editing the +status field. + +### When to write an ADR + +Write one if any of these is true: + +- You chose between approaches that a competent engineer might reasonably have decided differently. +- You are accepting a known cost (duplication, a manual step, a fat image) to buy something. +- You are doing something that will look like a mistake to someone who doesn't know why. +- You are reversing or narrowing an earlier decision. + +The third case matters most here. Several of this system's sharpest choices look wrong on sight: +reviewers are dispatched *without* `agentType` so they can inherit `Write` ([ADR-0027]); merge +logic is *deliberately* duplicated across a repo boundary ([ADR-0014]); `stream-claude.py` kills +its parent with `SIGTERM` and exits 42 on purpose. Each needed an ADR and didn't have one. + +--- + +## 4. Working agreements + +### Testing + +- After any change under `scripts/`: `make test-unit`. +- Before pushing: `make test-unit` and `python3 scripts/check_ledger.py --all`. +- **A change is not done until tests pass.** Not "tests pass locally except one" — pass. +- New logic gets tests in the same PR. `scripts/check_ledger.py` is not exempt. + +Current suite: **703 tests** across 19 files in `tests/`, ~2.5 min. + +> **Do not use `make test`** — it always fails. It depends on `test-integration`, which runs +> `pytest -m integration`; nothing carries that marker, so pytest exits 5 and make reports a failure +> regardless of results. This has been true on `main` since the marker was added. See +> [[bug-make-test-fails-on-empty-integration-target]] and [[task-fix-make-test-target]]; once fixed, +> `make test` becomes the right command again. (`README.md` also claims 186 tests; it is stale by 517.) + +### Evidence standard + +This system's core failure mode is **confident, plausible, wrong**. It reviews its own generated +code, so an unverified claim becomes a score becomes a merged PR. Hold yourself to the standard the +reviewers are held to: + +- Read the source before asserting a mechanism. Cite `file:line`. +- "Works because X" needs the lines that prove X, or it doesn't go in. +- Don't trust a commit subject as a description of a commit's contents. Read the diff. +- Don't trust a subagent's report as fact. It is a lead. Verify before recording. +- Reporting that something failed is always better than reporting a success that isn't real. + +### Hard rules + +These have all been violated at least once and cost a day each. See `CLAUDE.md` for the full list +with commands. + +1. **Never write `run-metadata.yaml` whole — always merge.** Two producers share the file. A + whole-file write deletes the other's fields and silently deadlocks the epic (RHAIFIRST-374). +2. **A check that couldn't run is `unrunnable`, not `failed`.** Scoring an environment fault as + bad code once produced `lint=5.0` from a missing `uv`. +3. **Never hand-write `validation.json`.** Use `validate_target.py --out`. The authenticity gate + exists because a hand-written file once scored `lint=8.0` while Prettier was failing. +4. **Run all scripts from the project root**, never from inside `.target-repo/`. +5. **Never run `run_pipeline.py` locally.** It runs in CI only. `--dry-run` is fine. + +### Commits and PRs + +- Branch from `main`; don't commit to `main` directly. +- Subject line says what changed and, where it fits, why: `Stop the review-response path hiding + why the fix agent failed` beats `fix review response`. +- Reference the Jira key when one exists. +- Keep a PR to one phase or one concern. Five reviewable PRs beat one 115-file PR. + +--- + +## 5. Workflow + +1. Read `PLAN.md`. +2. Pick a task from `docs/tasks/pending/` (or write one — unplanned work still gets a file). +3. `git mv` it to `docs/tasks/current/`, set `status: current`. Commit that alone, so the claim is + visible before the work lands. +4. Do the work. Append discoveries to the task's `## Notes` as you go, not at the end. +5. File a bug the moment you find one — separate file, `docs/bugs/open/`. Do not fold an + unrelated fix into your task. +6. Write an ADR if you made a decision (§3). +7. Check the Acceptance Criteria boxes and record evidence: commit SHAs, test names, job URLs. +8. `git mv` to `docs/tasks/done/`, set `status: done`, fill `commits:`. +9. Update `PLAN.md` if the active set changed. +10. Append an entry to `docs/notes/session-log.md`. + +If you end up blocked: `git mv` to `docs/tasks/blocked/`, and record in `## Notes` what +specifically unblocks it. "Blocked" with no exit condition is abandonment with better branding. + +--- + +## 6. Conventions + +- **Python**: stdlib only where practical. The only runtime dependency is `pyyaml`. Modules are + flat in `scripts/` with `sys.path.insert` bootstrapping rather than an installed package — a + deliberate choice under review ([ADR-0003]). +- **No new YAML parser.** There are already six. Use `artifact_utils.read_frontmatter`. +- **No new HTTP client.** Use `jira_utils` or `github_utils`. +- **Agent definitions** live in `.claude/agents/`, one file per agent. A reviewer's `tools:` line + is documentation, not enforcement ([ADR-0027]). +- **Reviewers classify severity; Python computes scores.** Never let a model choose a number + ([ADR-0022]). +- **Artifacts** are written under `artifacts/` (gitignored) and persisted to the data repo by CI. + +Full reference: `CLAUDE.md`. Contracts for every artifact file: +[`docs/architecture/03-artifact-contracts.md`](docs/architecture/03-artifact-contracts.md). diff --git a/Dockerfile.ci b/Dockerfile.ci index 5bbcd1b..3672aa5 100644 --- a/Dockerfile.ci +++ b/Dockerfile.ci @@ -33,12 +33,28 @@ RUN curl -fsSL "https://go.dev/dl/go1.24.4.linux-${TARGETARCH}.tar.gz" | tar -C ENV PATH="/usr/local/go/bin:${PATH}" ENV GOPATH="/home/claude-ci/go" -# Node.js 22 (required by odh-dashboard: engines.node >= 22) -RUN curl -fsSL https://rpm.nodesource.com/setup_22.x | bash - \ +# Node.js 26 — the highest floor any target sets, so one runtime serves all. +# odh-dashboard needs >= 22; rh-forge-ui needs ^24.15.0 || >= 26 AND sets +# engine-strict=true in .npmrc, which turns a too-old Node from a warning into +# a failed install. Was 22, which satisfied the first and hard-failed the +# second. +RUN curl -fsSL https://rpm.nodesource.com/setup_26.x | bash - \ && dnf install -y --nodocs nodejs \ && dnf clean all \ && npm install -g yarn markdownlint-cli +# pnpm, via the corepack shim Node ships, so a repo pinning +# `packageManager: pnpm@x.y.z` gets that exact version rather than whatever +# was current at image build time. A pnpm-workspace repo installed with npm +# resolves a different dependency tree than its CI has. +# `corepack enable pnpm`, not a bare `corepack enable`: the latter also +# installs a yarn shim that shadows the yarn npm just put on PATH, which +# odh-dashboard depends on. +ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 +RUN corepack enable pnpm \ + && corepack prepare pnpm@latest --activate \ + && pnpm --version + # JS/TS check tooling. `npm run lint` prepends the repo's node_modules/.bin, # so a repo that pins its own eslint/tsc/vitest still gets that one; these are # the fallback for when it does not resolve, which otherwise exits 127 and @@ -96,6 +112,8 @@ RUN python3 --version \ && go version \ && node --version \ && npm --version \ + && pnpm --version \ + && yarn --version \ && eslint --version \ && tsc --version \ && vitest --version \ diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..286ae60 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,90 @@ +# Project Plan + +Navigation index for the [work ledger](AGENTS.md#2-the-ledger). This file links out; it does not +hold content of its own. Process rules live in [`AGENTS.md`](AGENTS.md). + +**Initiative:** [RHAIFIRST-168](https://redhat.atlassian.net/browse/RHAIFIRST-168) — Epic Code +Generation — Hardening & Production Rollout. + +--- + +## Where we are + +Out of POC. The system has processed 24 epics across 7 target repos and landed 5 merged PRs, with +score progressions like 2.4 → 4.9 → 7.2 → 9.4 over four review iterations. What it lacks is not +capability but **engineering discipline**: until this ledger there were no ADRs, no architecture +docs, no CI on this repo, and no link between a code change and a recorded reason. + +Current focus: repo hygiene and process, then the open review-gate defects +([M6](docs/milestones/)) that make a passing score less trustworthy than it looks. + +--- + +## Milestones + +| Milestone | Jira | Status | +|---|---|---| +| [M1 — POC validation](docs/milestones/) | RHAIFIRST-136 | Closed | +| [M2 — CI pipeline & dashboard](docs/milestones/) | RHAIFIRST-200 | Closed | +| [M3 — Jira automation](docs/milestones/) | RHAIFIRST-208 | Closed | +| [M4 — Review-response pipeline](docs/milestones/) | RHAIFIRST-212 | Closed | +| [M5 — State integrity](docs/milestones/) | RHAIFIRST-374/375/376 | Closed | +| [M6 — Review-gate hardening](docs/milestones/) | RHAIFIRST-391/392/393 | **Open** | +| [M7 — Engineering process](docs/milestones/) | — | **Open** | + +--- + +## Active tasks + +- [`docs/tasks/current/`](docs/tasks/current/) — nothing claimed right now. + +Pick up work from [`docs/tasks/pending/`](docs/tasks/pending/). See +[AGENTS.md §5](AGENTS.md#5-workflow) for how to claim it. + +--- + +## Open bugs + +Highest impact first. Full list in [`docs/bugs/open/`](docs/bugs/open/). + +| Bug | Impact | Jira | +|---|---|---| +| Review gate is advisory — PRs open from unreviewed versions | Critical | RHAIFIRST-391 | +| Baseline target-repo check failures scored as bad code (both repos tried) | High | RHAIFIRST-392 | +| Failed review-response cycle marks comments processed, dropping feedback | High | RHAIFIRST-393 | +| Multi-strategy runs lose the run record for all but the last strategy | High | *needs one* | +| `shell=True` command injection in `validate_target.py` | High | RHAIFIRST-194 | +| `make test` always fails (test-integration collects nothing) | Medium | *needs one* | +| `max_iterations` defaults to 3 in one path and 10 in five others | Medium | *needs one* | + +Jira issues are opened [on demand](AGENTS.md#companion-jira--on-demand), so *needs one* means "not yet +visible outside this repo" — not that it is untracked. + +--- + +## Decisions + +All ADRs: [`docs/decisions/`](docs/decisions/). + +The load-bearing ones, if you read only five: + +- **ADR-0013** — nine CI states; `status` vs `codegen_outcome`, one owner per field. +- **ADR-0014** — merge, never write, `run-metadata.yaml`. +- **ADR-0022** — reviewers classify severity; Python computes the score. +- **ADR-0025** — `unrunnable` ≠ `failed`. +- **ADR-0027** — reviewers dispatched without `agentType`, so `tools:` is documentation. + +--- + +## Architecture + +[`docs/architecture/`](docs/architecture/) — start with the +[system overview](docs/architecture/01-system-overview.md), then the +[state machine](docs/architecture/02-pipeline-state-machine.md) and +[artifact contracts](docs/architecture/03-artifact-contracts.md). + +--- + +## Log + +[`docs/notes/session-log.md`](docs/notes/session-log.md) — dated activity, 2026-06-22 onward. diff --git a/config/repo_mapping.json b/config/repo_mapping.json index 996e537..fadb901 100644 --- a/config/repo_mapping.json +++ b/config/repo_mapping.json @@ -17,7 +17,9 @@ "ederign/kale": { "keywords": ["kale", "kale extension", "kale jupyterlab", "sample notebook catalog", "kale pipeline"] }, - "ederign/openc-ui-by-agentic-sdlc": { - "keywords": ["openc-ui", "openc-ui-by-agentic-sdlc", "openclaw", "openshell", "conversation-ui", "conversation surface", "gateway client", "gateway-client"] + "rh-forge/rh-forge-ui": { + "keywords": ["rh-forge", "rh-forge-ui", "forge ui", "openc-ui", "openclaw", "openshell", "conversation-ui", "conversation surface", "gateway client", "gateway-client", "draft proposals", "drafts list", "home-page drawer", "review drawer", "draft service", "outbox poll"], + "fork_owner": "ederign", + "gh_token_var": "RH_FORGE_GITHUB_TOKEN" } } diff --git a/docs/architecture/01-system-overview.md b/docs/architecture/01-system-overview.md new file mode 100644 index 0000000..f3aebf0 --- /dev/null +++ b/docs/architecture/01-system-overview.md @@ -0,0 +1,137 @@ +--- +id: 01-system-overview +title: System overview — how a Jira epic becomes a merged PR +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +decisions: [ADR-0005, ADR-0006, ADR-0007, ADR-0008, ADR-0009] +--- + +# System overview + +## Position in the wider chain + +``` +RFE (rfe-creator) + → Strategy (strat-creator) + → Epic decomposition (epic-creator) + → Code generation (epic-code-gen) ← this system + → PR on target repo → CI → human review → merge +``` + +## The four repos + +| Repo | Role | Written by | +|---|---|---| +| `epic-code-gen` | The brains: skill, 13 agents, 20 scripts, 703 tests | humans + agents | +| `epic-code-gen-pipeline` | Thin GitLab CI shell, 16 files | humans | +| `epic-code-gen-pipeline-data` | Git-as-database: state + artifacts | CI bot | +| `epic-code-gen-dashboard` | Reads the data repo → GitLab Pages | humans | + +Rationale: [ADR-0005]. The pipeline repo clones the brains repo at run time rather than vendoring it, so +logic ships without touching CI. + +## One run, end to end + +``` + operator sets STRATEGY_KEYS, presses "play" on codegen-run (manual trigger) + │ + ▼ + ┌─────────────────────── GitLab job (6h timeout) ────────────────────────┐ + │ before_script: setup-env.sh → clone-data-repo.sh │ + │ │ + │ run-codegen.sh: │ + │ clone epic-code-gen → /tmp/claude-workdir │ + │ start otel-collector.py (127.0.0.1:4318) │ + │ start progress heartbeat (every 300s) │ + │ python3 scripts/run_pipeline.py $KEYS --ci --data-repo /tmp/data-repo│ + │ │ │ + │ ├─ fetch strategy children from Jira, build dependency DAG │ + │ ├─ classify eligibility per epic │ + │ └─ for each eligible epic: ONE state transition │ + │ └─ if generating: claude -p → /epic-codegen skill │ + │ │ + │ after_script: pipeline-post.sh → push-results.py → data repo commit │ + └────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + trigger-dashboard → epic-code-gen-dashboard pipeline → GitLab Pages +``` + +`after_script` rather than `script` is deliberate (`aff11df`): results persist even when codegen times +out or crashes. Note it is bounded by `RUNNER_AFTER_SCRIPT_TIMEOUT` (5 min default), independently of the +6-hour job timeout. + +## The convergence loop + +The system does **not** drive one epic to completion in one job. Each run advances every actionable epic +by exactly one step ([ADR-0009]): + +``` +run 1: Pending → Ready clone, readiness, toolchain preflight +run 2: Ready → ReviewPending generate + review (the expensive one) +run 3: ReviewPending → PRCreated open the PR +run 4: PRCreated → PRCreated no new comments — a valid no-op +run 5: PRChangesRequested → PRCreated address review feedback +run 6: PRCreated → Done PR merged upstream +``` + +Full transition graph: [02-pipeline-state-machine.md](02-pipeline-state-machine.md). + +This is why the pipeline never blocks on a human: it *observes* that a human acted, on the next run. +It also means latency is measured in runs, and the trigger is manual — so wall-clock latency is really +"how often does someone press the button." + +## Inside one epic's generation + +The `/epic-codegen` skill, four phases ([04-codegen-skill-phases.md](04-codegen-skill-phases.md)): + +``` +Phase 1 Spec & Plan + pattern discovery (explicit refs, concept search, 5–10 siblings, conventions) + → brainstorming subagent → spec → spec review gate → writing-plans → plan +Phase 2 Implementation + Superpowers SDD, orchestrator acts as the human partner +Phase 3 Review — 6 agents in parallel + architecture 30% · tests 30% · lint 20% · intent 20% (scored) + wiring · interactions (advisory) + → score_reviews.py computes the number from finding counts +Phase 4 Iterate or complete + pass ≥8.0 → final diff · near-miss ≥7.0 on exhaustion → PR anyway + fail → triage → fix agent → new version (budget 10) +``` + +## Two authorities, and the seam between them + +| Question | Authority | +|---|---| +| Which epics exist, and is this one eligible? | **Jira** ([ADR-0007]) | +| Where is this epic in the pipeline, and what scored what? | **the data repo** ([ADR-0008]) | + +That seam is where the worst bug lived: RHAIFIRST-374, an epic whose state file said `completed` — a word +the state machine did not know — was skipped on every run while CI reported success, deadlocking three +dependents. The fix ([ADR-0013], [ADR-0014], [ADR-0015]) is one owner per field, merge-never-write, and +loud failure on the unrecognized. + +## Where things are + +| Concern | Location | +|---|---| +| Orchestration | `scripts/run_pipeline.py` (1,973 lines) | +| Review loop | `scripts/review_cycle.py` | +| Scoring | `scripts/score_reviews.py` | +| Schemas & state vocabularies | `scripts/artifact_utils.py` | +| Review response | `scripts/review_response.py` | +| Target repo assessment | `validate_target.py`, `repo_readiness.py` | +| Agents | `.claude/agents/` (13 files) | +| The skill | `.claude/skills/epic-codegen/SKILL.md` (829 lines) | +| CI image | `Dockerfile.ci` | + +## Scale to date + +24 epics across 7 target repos, 39 recorded pipeline passes, 9 PRs (5 merged), ~$80 logged Claude spend, +20.8% reported completion. Score progressions: RHAI-74 2.4 → 4.9 → 7.2 → 9.4; RHAI-64 2.6 → 2.65 → 6.1 → +6.7 → 8.2. Passing scores cluster 7.9–9.4. + +Honest counterweight: **39 of the data repo's 104 commits are humans hand-editing state to unwedge the +pipeline.** See [10-known-limitations.md](10-known-limitations.md). diff --git a/docs/architecture/02-pipeline-state-machine.md b/docs/architecture/02-pipeline-state-machine.md new file mode 100644 index 0000000..668515e --- /dev/null +++ b/docs/architecture/02-pipeline-state-machine.md @@ -0,0 +1,187 @@ +--- +id: 02-pipeline-state-machine +title: The CI state machine — nine states and every transition +type: plan +status: current +repos: [epic-code-gen] +decisions: [ADR-0009, ADR-0013, ADR-0015] +--- + +# The CI state machine + +Until this document, the transition graph existed **only** as `if/elif` in +`scripts/run_pipeline.py:1127-1974`. Everything below is read from that code; line numbers are cited so +it can be re-verified when the code moves. + +Each pipeline run takes **exactly one action per epic** ([ADR-0009]). Progress happens across runs. + +## States + +`CI_STATES` (`artifact_utils.py:31`) — the vocabulary of the `status` field in `run-metadata.yaml`, +owned solely by `run_pipeline.py` ([ADR-0013]): + +| State | Meaning | +|---|---| +| `Pending` | Never processed. Freshly discovered from Jira. | +| `Ready` | Eligible to generate. Dependencies resolved. | +| `Generating` | Codegen in flight. **Transient** — only seen if a run died mid-generation. | +| `ReviewPending` | Code generated and scored; awaiting the PR decision. | +| `PRCreated` | PR is open upstream. Waiting on human/bot review. | +| `PRChangesRequested` | Review left actionable feedback. | +| `Done` | **Terminal.** PR merged. | +| `Blocked` | Waiting on a dependency epic. | +| `Failed` | **Terminal.** Unrecoverable without intervention. | + +`CI_TERMINAL_STATES = {"Done", "Failed"}` (`run_pipeline.py:87`). + +## Dispatch + +`ci_process_epic()` (`:1127`) runs these guards **before** any state handler: + +| Order | Guard | Result | +|---|---|---| +| 1 | `state is None` | `_init_epic_state()`, save, continue as `Pending` | +| 2 | `normalize_epic_state()` | maps foreign/legacy values onto `CI_STATES` ([ADR-0015]) | +| 3 | `not is_codegen_project(epic)` | `SKIPPED` — "Project not in codegen scope" | +| 4 | `has_skip_label(epic)` | `SKIPPED` — "Skipped (epic-code-gen-skip)" | +| 5 | `current in CI_TERMINAL_STATES` | `SKIPPED` — "Terminal state: …" | +| 6 | dispatch to `_ci_handle_<state>` | — | +| 7 | **unrecognised state** | `FAILED`, and the state file is **left untouched** | + +Guard 7 is the RHAIFIRST-374 fix. The comment at `:1175-1181` is explicit about why the state is not +overwritten: *"we do not understand this document, and overwriting it would destroy the evidence a human +needs, so the run keeps failing until someone fixes it."* Previously this branch was a silent `SKIPPED` +that still exited 0, so an epic was skipped forever while CI reported success. + +`Generating` deliberately routes to `_ci_handle_ready` (`:1163-1164`) — a run that died mid-generation +retries generation. + +## Transitions + +Every `return` in every handler. Action is one of `PROCESSED` / `SKIPPED` / `BLOCKED` / `FAILED`; +`FAILED` makes `main()` exit 1. + +### `Pending` — `_ci_handle_pending` (`:1206`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `Blocked` | BLOCKED | unresolved dependencies | 1223 | +| `Ready` | PROCESSED | classified as ready | 1228 | + +### `Ready` / `Generating` — `_ci_handle_ready` (`:1231`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `Ready` | PROCESSED | `--dry-run` | 1236 | +| `Failed` | FAILED | target repo setup failed | 1250 | +| **`Ready`** | **FAILED** | **toolchain preflight gap — no code generated** | 1266 | +| `ReviewPending` | PROCESSED | codegen completed | 1305 | +| `Failed` | FAILED | codegen failed | 1312 | + +Line 1266 is the deliberate action/state disagreement: a missing tool is an environment fault, so state +stays `Ready` to retry once the image is fixed, but the action is `FAILED` so the run exits non-zero and +is visibly broken ([ADR-0025]). Consequence: it exits 1 on **every** run until someone rebuilds the +image — no backoff, no alert hook. + +### `ReviewPending` — `_ci_handle_review_pending` (`:1315`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `ReviewPending` | SKIPPED | no scores yet | 1331 | +| `PRCreated` | PROCESSED | passed → PR opened | 1366 | +| `Failed` | FAILED | PR creation failed | 1373 | +| `PRCreated` | PROCESSED | **near-miss (≥7.0) → PR opened anyway** | 1391 | +| `Failed` | FAILED | iterations exhausted below near-miss | 1399 | +| **`Ready`** | PROCESSED | **failed but budget remains → iterate again** | 1404 | + +The `→ Ready` edge at 1404 is the review loop's outer cycle, and the only backward edge in the graph. + +> **Known defect:** this handler re-implements the pass rule (`avg >= 8.0 and dims_ok`, hard-coded 6.0 +> floor, `:1348`) instead of reading the `verdict` `score_reviews.py` already computed. Two copies of the +> rule, and only the `score_reviews` copy fails on a foreign `validation.json` — so they can disagree. +> See `docs/bugs/open/`. + +### `PRCreated` — `_ci_handle_pr_created` (`:1408`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `PRCreated` | SKIPPED | no PR URL / no GitHub token / no status change | 1413, 1424, 1457 | +| *derived* | PROCESSED | PR state changed | 1460 | +| `Done` | PROCESSED | **PR merged** | 1470 | +| `Ready` | PROCESSED | PR closed unmerged → regenerate | 1475 | +| *derived* | SKIPPED | other derived state | 1476 | +| `Done` | PROCESSED | PR merged, detected via `gh` fallback | 1486 | +| `PRCreated` | SKIPPED | PR still open | 1487 | + +### `PRChangesRequested` — `_ci_handle_pr_changes` (`:1490`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `PRChangesRequested` | SKIPPED | dry-run / nothing actionable | 1495, 1515 | +| `PRChangesRequested` | PROCESSED | — | 1499 | +| `Failed` | FAILED | setup / review-response failure | 1510, 1531, 1633 | +| `PRCreated` | SKIPPED | rebase-only cycle, **no iteration consumed** | 1612 | +| `PRCreated` | PROCESSED | fixes applied and pushed | 1626 | + +Line 1612 implements the [ADR-0031] rule: a cycle that rebases nothing and finds nothing actionable must +not consume an iteration, or an unaddressable review loops until the budget is gone. + +### `Blocked` — `_ci_handle_blocked` (`:1687`) + +| → | Action | Condition | Line | +|---|---|---|---| +| `Blocked` | BLOCKED | still blocked | 1703 | +| → `_ci_handle_ready` | *(inherited)* | deps resolved — **falls through in the same run** | 1706+ | + +When dependencies resolve, the handler sets `Ready`, deletes `blocked_by`, and immediately delegates to +`_ci_handle_ready`, reporting `from: "Blocked"`. So an unblocked epic generates in the *same* pass rather +than waiting for the next one (`1045c53`). + +> **Subtlety worth knowing:** the dependency check reads each dependency's **data-repo state** and +> requires `status == "Done"` (`:1694-1700`) — not its Jira status. Initial eligibility classification in +> `_ci_handle_pending` uses **Jira**. Two different authorities for "is this dependency finished", +> depending on which state you are in. + +## Graph + +``` + ┌─────────┐ + │ Pending │ + └────┬────┘ + deps unmet │ │ ready + ┌──────┘ └──────┐ + ▼ ▼ + ┌─────────┐ ┌──────────┐ + │ Blocked │────►│ Ready │◄──────────┐ + └─────────┘ deps└────┬─────┘ │ + (falls through) │ │ + │ codegen │ fail, budget left + ▼ │ (1404) + ┌────────────────┐ │ + │ ReviewPending │─────────┘ + └───────┬────────┘ + pass / near-miss │ + ▼ + ┌────────────────┐ changes requested + │ PRCreated │◄─────────────────┐ + └───┬────────┬───┘ │ + merged │ │ review feedback │ + ▼ ▼ │ + ┌──────┐ ┌─────────────────────┐ │ + │ Done │ │ PRChangesRequested │────┘ + └──────┘ └─────────────────────┘ + (terminal) fixes pushed + + Any state ──► Failed (terminal) on unrecoverable error + PRCreated ──► Ready if PR closed unmerged (1475) +``` + +## Reading a run + +`pipeline-runs/<run_id>.json` records `action`, `result`, `reason`, and `transitions` per epic; +`actions.json` records `{epic, from, to, version}` and feeds the dashboard's state-log view. In the data +repo, `<strategy>/run-log.jsonl` is the durable append-only history — 39 recorded passes to date. + +Observed `to`-state distribution across those 39 passes: `Ready` 20, `PRCreated` 16, `Blocked` 10, +`Done` 4, `Failed` 3, `ReviewPending` 3, `Generating` 1. `PRChangesRequested` appears only as a `from`, +which is expected — it is entered by observing GitHub, not by a transition the pipeline records. diff --git a/docs/architecture/03-artifact-contracts.md b/docs/architecture/03-artifact-contracts.md new file mode 100644 index 0000000..6d78cdc --- /dev/null +++ b/docs/architecture/03-artifact-contracts.md @@ -0,0 +1,192 @@ +--- +id: 03-artifact-contracts +title: Artifact contracts — the exact shape of every file the pipeline passes +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +decisions: [ADR-0001, ADR-0002, ADR-0013, ADR-0014, ADR-0024] +--- + +# Artifact contracts + +The pipeline's components communicate through files ([ADR-0001]). These are the contracts. Only the +first three are schema-validated; the rest are conventions enforced by whoever reads them. + +## Schema-validated: `artifact_utils.SCHEMAS` + +`scripts/artifact_utils.py:164`. Validated on read via `frontmatter.py read`. + +### `epic-task` — `artifacts/epic-tasks/<EPIC_ID>.md` + +| Field | Type | Req | Notes | +|---|---|---|---| +| `epic_id` | str | ✔ | `^[A-Z][A-Z0-9]+-\d+(-E\d+)?$` | +| `title` | str | ✔ | | +| `strategy_key` | str | ✔ | `^[A-Z][A-Z0-9]+-\d+$` | +| `target_repo` | str | ✔ | | +| `status` | str | ✔ | enum: `Pending, Ready, InProgress, Generated, Validated, Failed` | +| `target_branch` | str | | default `""` | +| `components`, `jira_labels`, `dependencies`, `blocks` | list | | `dependencies` = blocked-by | +| `effort_size` | str | | enum `S, M, L, XL` | +| `readiness_score`, `codegen_branch`, `pr_url`, `jira_status` | | | | + +Note this `status` enum is a **fourth** vocabulary, distinct from `CI_STATES` and `CODEGEN_OUTCOMES`. It +describes the epic-task artifact, not the run. + +### `codegen-run` + +`epic_id` ✔, `status` (enum = `CI_STATES`), `codegen_outcome` (enum = `CODEGEN_OUTCOMES`), `iterations` ✔ +(default 0), `max_iterations` ✔ (default 10), `started_at`, `completed_at`, `target_repo` ✔, +`target_branch`, `codegen_branch` ✔, `validation` (dict: `lint_pass`, `typecheck_pass`, `tests_pass`). + +### `codegen-review` — **dead** + +`epic_id`, `recommendation` (`approve|revise|reject`), `total_score`, `scores{lint, typecheck, tests, +intent_coverage, architecture}`. + +Nothing in the pipeline has ever written one. Its `scores` keys (`typecheck`, `intent_coverage`) predate +the live dimensions. `find_codegen_review` and `rebuild_index` are dead with it. See `docs/bugs/open/`. + +## `run-metadata.yaml` — the state file + +**Not schema-validated.** Two writers, one file; always merge, never write ([ADR-0014]). + +| Field | Owner | +|---|---| +| `status` | `run_pipeline.py` — `CI_STATES` only | +| `codegen_outcome` | the `/epic-codegen` skill — `CODEGEN_OUTCOMES` only | + +`PIPELINE_OWNED_KEYS` (which the skill may never overwrite): `status`, `status_normalized_from`, +`current_version`, `max_iterations`, `pr_state`, `timestamps`, `scores`, `blocked_by`, `failure_reason`, +`tooling_missing`. + +A real terminal file (`RHAISTRAT-2162/RHAI-74`): + +```yaml +epic_id: RHAI-74 +strategy_key: RHAISTRAT-2162 +target_repo: ederign/kale +branch: epic/RHAI-74 +base_sha: c2264752da1a7f4add3bf138b08fbbc982d901ea +status: Done +current_version: 7 # pipeline-owned; counts review-response cycles +max_iterations: 10 +pr_state: merged +versions: 4 # skill-owned; counts scored codegen iterations +final_version: 4 +final_score: 9.4 +verdict: pass +pr_url: https://github.com/ederign/kale/pull/11 +fork_owner: dora-the-ai-coder +score_progression: {v1: 2.4, v2: 4.9, v3: 7.2, v4: 9.4} +dimension_scores: {architecture: 9.0, tests: 9.0, lint: 10.0, intent: 10.0} +files_changed: 8 +lines_added: 1081 +tests_count: 53 +timestamps: {last_run: '2026-07-29T14:57:23.963082+00:00'} +``` + +### Known drift — four generations coexist in the data repo + +| Generation | Distinctive shape | +|---|---| +| `RHAISTRAT-1749/RHOAIENG-72528` | nested `scores` with an **absolute** `reviews_dir`; no findings counts | +| `RHAISTRAT-1699/RHOAIENG-72103` | **flat** `scores{architecture, tests, lint, intent, weighted_average, verdict}` + a `features` block | +| `RHAISTRAT-1508/RHAI-64` | `iterations`, `final_version: v5` (**string**), `final_verdict`, `dimensions:` not `scores:` | +| `RHAISTRAT-1961/RHAI-68` | current: `readiness`, `codegen_outcome`, `scores_by_dimension` **and** nested `scores` | + +Two names for one concept: `dimension_scores` (what production writes) vs `scores_by_dimension` (what +`SKILL.md`, `run_index.py`, and `frontmatter.py` reference). Three counters with no documented +relationship: `current_version`, `versions`, `final_version`. `RHAISTRAT-2352/RHAI-264` still carries the +pre-fix `status: completed`. + +## `validation.json` — written only by `validate_target.py --out` + +``` +repo_path, language, marker, +commands: {lint|typecheck|test: <command string>}, +checks: [ {name, command, passed, exit_code, output (≤5000 chars), + unrunnable, missing_tool}, … ], +all_passed, unrunnable: [names], missing_tools: [executables], has_unrunnable +``` + +`all_passed = all checks passed AND len(checks) > 0 AND no unrunnable`. **Consumers must read +`all_passed`, never per-check keys.** + +**Authenticity gate** ([ADR-0024]): `VALIDATION_DOCUMENT_KEYS = ("all_passed", "checks")`. +`validation_document_status()` → `ok` | `missing` | `foreign` | `unreadable`; `foreign`/`unreadable` +force `verdict: fail`. + +`--preflight` returns a **different** shape: `{language, required, found: {tool: path|null}, missing, +ok}`. Exit 2 = missing tool; exit 1 = failing check. + +## `scores.json` — written by `review_cycle.py score` + +``` +{ + "reviews_dir": str, + "dimensions": { "<dim>": { "score": float, "weight": float, "weighted": float, + "file": str, + "findings": {"critical": int, "important": int, "minor": int} } }, + "weighted_average": float, + "verdict": "pass" | "near-miss" | "fail" | "incomplete", + "missing": [dim], "errors": [str], + "validation": {"status": "ok"|"missing"|"foreign"|"unreadable", "detail": str|null} +} +``` + +Constants (`score_reviews.py:34-49`): weights architecture .30 / tests .30 / lint .20 / intent .20; +`CRITICAL_WEIGHT 5.0`, `IMPORTANT_WEIGHT 1.5`, `MINOR_WEIGHT 0.5`, `CRITICAL_CAP 5.0`; +`PASS_THRESHOLD 8.0`, `NEAR_MISS_THRESHOLD 7.0`, `MIN_DIMENSION_SCORE 6.0`, `HARD_FLOOR 5.0`. + +**Findings are parsed by regex over markdown**: a heading matching +`^#{1,4}\s+(critical|important|minor)$` opens a section; a finding is a line matching `^\d+\.\s+\*\*`. +Dimension name comes from the filename via `^review-(\w+)\.md$`. This **fails open** — an unrecognized +heading yields zero findings and therefore 10.0. + +## Review files — `review-<dimension>.md` + +Written by reviewer agents ([ADR-0021]). Required: findings grouped under `#### Critical` / +`#### Important` / `#### Minor`, each numbered `N. **Title**`. **No score in the output.** + +Per-dimension extras: architecture → `### Convention Compliance`, `### Integration Assessment`; +tests → `### AC Coverage` table, `### Edge Cases`; lint → `### Validation Results` table; +intent → `### AC-to-Diff Mapping`, `### Pass Criteria Verification`, `### Scope Fidelity`, +`### UX Acceptance Criteria Verification`, `### Scope Creep Check`. + +Unscored ([ADR-0028]): `review-wiring.md` (`### Wiring Traces` table), `review-interactions.md`. + +## Smaller contracts + +| File | Shape | +|---|---| +| `pre-setup.json` | `{validation: <full validate dict>, readiness_output: <markdown **string**>, language, deps_installed}` | +| `pr-replies.json` | processed review-comment IDs per version | +| `tmp/epic-codegen-<EPIC>.json` | despite `.json`, `state.py`'s `key: value` **line format** | +| `tmp/accepted-findings-<EPIC>.json` | real JSON: `[{finding, dimension, accepted_in, reason}]` | +| `index.json` | `{runs: [<full run-metadata dict>], total, summary: {<outcome>: count}}` | +| `pipeline-runs/<run_id>.json` | `{run_id, start_time, end_time, strategies: {<KEY>: {total_epics, summary{processed,skipped,blocked,failed}, epics: {<EPIC>: {title, jira_status, action, result, reason, dependencies, blocks, transitions, pr_url, timestamp}}}}}` | +| `pipeline-runs/actions.json` | `{epic, from, to, version}` per transition — input to `push-results.py` | +| `config/review_config.json` | `{bot_reviewers[], our_user, max_review_iterations: 5, validation_retry_limit: 3}` | +| `config/repo_mapping.json` | `{"<owner/repo>": {"keywords": [...]}}` | + +## Data-repo aggregates + +Regenerated by `push-results.py` on every push; consumed by the dashboard. + +- `<strategy>/strategy-summary.json` — `{strategy_key, generated_at, stats{total, done, in_progress, + blocked, failed}, epics[{epic_id, status, current_version, pr_url, pr_state, scores, target_repo}]}`. + `in_progress` is derived as `total − done − failed − blocked`. +- `summary.json` (top level) — `{generated_at, stats{strategies, total_epics, total_done, + completion_rate}, strategies[…]}`. +- `<strategy>/run-log.jsonl` — append-only, one line per pass: `{timestamp, strategy, epics_processed, + epics_skipped, epics_blocked, actions[], otel_cost_usd?}`. + +> **Two live data-quality defects in these aggregates**, both tracked in `docs/bugs/open/`: +> `summary.json` reports 7 strategies but double-counts `RHAISTRAT-1699`, because the manual archive +> directory `RHAISTRAT-1699-before-ux-ac/` declares the same `strategy_key` inside its summary. And +> `scores` is `null` for every epic whose metadata uses `dimension_scores`/`dimensions`/ +> `scores_by_dimension` instead of a nested `scores` block — including RHAI-74, which shows +> `scores: null` beside `final_score: 9.4`. + +There is **no `index.json` in the data repo**; `run_index.py` writes it into `artifacts/codegen-runs/`. diff --git a/docs/architecture/04-codegen-skill-phases.md b/docs/architecture/04-codegen-skill-phases.md new file mode 100644 index 0000000..64ed905 --- /dev/null +++ b/docs/architecture/04-codegen-skill-phases.md @@ -0,0 +1,92 @@ +--- +id: 04-codegen-skill-phases +title: The /epic-codegen skill — four phases, fourteen steps +type: plan +status: current +repos: [epic-code-gen] +decisions: [ADR-0016, ADR-0017, ADR-0018, ADR-0019, ADR-0020] +--- + +# The `/epic-codegen` skill + +`.claude/skills/epic-codegen/SKILL.md`, 829 lines. Handles **one epic per invocation**. + +``` +/epic-codegen EPIC_ID [--max-iterations N] [--dry-run] [--fork-owner USER] + [--gh-token-var VARNAME] [--checks lint,test,typecheck] +``` + +Defaults: `--max-iterations 10`, `--fork-owner dora-the-ai-coder`, +`--gh-token-var EPIC_CODEGEN_GITHUB_TOKEN`. + +Two framing rules from the preamble: **the epic strategy IS the product owner**, and **every script runs +from the project root**, never from inside `.target-repo/`. + +## Autonomous operation + +SDD has 12 human checkpoints; an autonomous pipeline cannot stop at any of them ([ADR-0017]). `SKILL.md` +maps each to a resolution derived from the epic's acceptance criteria — pre-flight conflicts, implementer +questions, `BLOCKED`, `NEEDS_CONTEXT`, plan-mandated findings, finishing — plus a clarifications table +covering continuous execution, `DONE_WITH_CONCERNS`, reviewer ⚠️ marks, fix-report validation, and the +progress ledger. SDD's own final review and finishing steps are skipped, because this pipeline has its +own review phase. + +## Phase 1 — Spec & Plan (steps 1–9) + +| Step | What | +|---|---| +| 1 | Parse the epic-task file | +| 2 | Init state (`tmp/epic-codegen-<EPIC_ID>.json`) — [ADR-0004] | +| 3 | Clone the target repo, create `epic/<EPIC_ID>` | +| 4 | Validate + readiness, short-circuiting on `pre-setup.json` if the orchestrator pre-staged it | +| 5 | Read the strategy, including the **authoritative** "Staff Engineer Input" section | +| 6 | Read repo context — scan every agent-readiness file: `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `GEMINI.md`, `COPILOT.md`, `CONVENTIONS.md`, `CONSTITUTION.md` | +| **7a–7d** | **Pattern discovery** — explicit refs · concept search · target file + 5–10 siblings + sibling dirs + callers · conventions docs ([ADR-0018]) | +| 7.5 | Parse the UX prototype via `node scripts/parse_prototype.js` ([ADR-0020]) | +| 7.6 | Extract UX ACs → `ux-acceptance-criteria.md` | +| 8 | Write `context-brief.md`, dispatch `design-spec-generator` → **brainstorming** ([ADR-0016]). Retry-then-**fail**, never fallback | +| 8.5 | **Spec review gate** — `spec-reviewer` validates the spec against real repo patterns | +| 9 | Dispatch `plan-generator` → **writing-plans**, then 4-point plan validation | + +Step 7 must complete **before** step 8 (`9b65e8c`), or the design gets invented and then justified. + +## Phase 2 — Implementation (steps 10–13.5) + +| Step | What | +|---|---| +| 10 | Record `BASE_SHA` | +| 11 | Init `.target-repo/.superpowers/sdd/` | +| 12 | `Skill("superpowers:subagent-driven-development")` with the autonomy overrides | +| 13 | Save version artifacts. **`validation.json` via `validate_target.py --out` — NEVER hand-written** ([ADR-0024]) | +| 13.5 | Wiring verification ([ADR-0028]) | + +## Phase 3 + 4 — Review, iterate, complete (step 14) + +The 9-step **review dispatch loop**, delegated to `review_cycle.py` ([ADR-0026]) and documented in +[06-review-and-scoring.md](06-review-and-scoring.md). Two hard rules live here: + +- **Never write review files yourself.** They are the reviewers' output; authoring them is fabricating + evidence. This is exactly what RHAIFIRST-391 did. +- **No `agentType` for reviewers** — with it they cannot write their review files ([ADR-0027]). + +Progress is logged to `tmp/progress.log`, which `run-codegen.sh`'s heartbeat tails every 300s so a 6-hour +run is observable. + +Post-loop: `pass` → save final diff · `near-miss` (≥7.0) → PR anyway on exhaustion · `fail` with budget +left → triage → fix → new version. + +## Reference sections + +`SKILL.md` also carries `## Run Metadata` (the two-writer ownership rule — [ADR-0014]), +`## Model Selection`, `## Review Dimensions`, `## State Recovery`, `## Error Handling`, `## Rules` (11 +hard rules), and — the most useful artifact in the file — a **21-row `## File Handoffs` table** mapping +each artifact to its writer and reader. That table is the direct ancestor of +[03-artifact-contracts.md](03-artifact-contracts.md). + +## Caveats + +- **829 lines of orchestration with no tests.** Every rule in it is a prompt instruction, so every rule is + advisory in the way prompts are advisory — which is the root of RHAIFIRST-391. +- Step 4's `pre-setup.json` short-circuit reads a `validation` object the orchestrator captured **before** + installing dependencies. Currently harmless (only `language` is used) but a loaded gun; open bug. +- `--dry-run` produces a diff and no PR. diff --git a/docs/architecture/05-agent-roster.md b/docs/architecture/05-agent-roster.md new file mode 100644 index 0000000..e86c548 --- /dev/null +++ b/docs/architecture/05-agent-roster.md @@ -0,0 +1,87 @@ +--- +id: 05-agent-roster +title: Agent roster — all 13 agents and their contracts +type: plan +status: current +repos: [epic-code-gen] +decisions: [ADR-0021, ADR-0027, ADR-0028, ADR-0029] +--- + +# Agent roster + +Thirteen agent definitions in `.claude/agents/`, one file each ([ADR-0021]). All inherit the session +model — no per-agent overrides ([ADR-0029]). + +> **Read [ADR-0027] before trusting a `tools:` line.** Reviewers and verifiers are dispatched *without* +> `agentType`, so they inherit the parent's full tool set — including `Write`, which they need to produce +> their output. For those agents `tools:` documents intent, it does not constrain. Generators *are* +> dispatched with `agentType` (SKILL.md:287, 378, 416, 440) and their tool lists are enforced. + +## Scored reviewers + +Findings → deterministic score ([ADR-0022]). Each writes `REVIEW_FILE`. **None outputs a score.** + +| Agent | Weight | Declared tools | Required sections beyond findings | +|---|---|---|---| +| `architecture-reviewer` | 30% | Read, Glob, Grep | `### Convention Compliance`, `### Integration Assessment` | +| `tests-reviewer` | 30% | Read, Glob, Grep | `### AC Coverage` table (`AC \| Test \| file:line \| Covered?`), `### Edge Cases` | +| `lint-reviewer` | 20% | Read, Glob, Grep | `### Validation Results` table; reads `VALIDATION_FILE` | +| `intent-reviewer` | 20% | Read, Glob, Grep | `### AC-to-Diff Mapping`, `### Pass Criteria Verification`, `### Scope Fidelity`, `### UX Acceptance Criteria Verification`, `### Scope Creep Check` | + +Shared findings contract: `### Findings` → `#### Critical` / `#### Important` / `#### Minor`, each finding +numbered `N. **Title**`. This markdown *is* the machine interface — `score_reviews.py` parses it by regex +and **fails open** (unrecognized heading → zero findings → 10.0). + +`tests-reviewer` explicitly scopes integration/e2e tests out ("NOT a finding"). `intent-reviewer` is the +richest at 174 lines and reads the epic file directly. + +## Unscored verifiers + +Findings inform triage only ([ADR-0028]). + +| Agent | Traces | Notes | +|---|---|---| +| `wiring-verifier` | trigger → chain → outcome per AC | `### Wiring Traces` table. "Minor: none expected — wiring is binary." | +| `interaction-verifier` | user interactions, enum/branch completeness | callback races, missing `switch` cases, broken form flows | + +## Generators and actors + +| Agent | Job | Output | +|---|---|---| +| `design-spec-generator` | invokes Superpowers `brainstorming`, acts as human partner | `codegen-spec.md`, `brainstorming-log.md` | +| `spec-reviewer` | validates the spec against real repo patterns, pre-plan | `spec-review-log.md` + mismatch table (`Spec Proposes \| Codebase Does \| Fix`) | +| `plan-generator` | invokes Superpowers `writing-plans` | `codegen-plan.md`, `writing-plans-log.md` | +| `ux-ac-extractor` | prototype analysis → numbered UX ACs | `ux-acceptance-criteria.md` (`UX-G1`, `UX-S1-1`) | +| `iteration-reviewer` | **triage** — the only model judgment left in the loop | `revision-notes.md`, `decision-log.md`, updates accepted-findings; dispatches the fix agent | +| `review-fix-agent` | applies PR-review fixes — one agent, all comments, one commit | ≤20-line summary | +| `sanity-check-agent` | verifies fixes address the comments | `sanity-check.md` (`### Addressed`, `### Scope Check`, `### Verdict`) | + +`design-spec-generator` opens with an all-caps autonomy directive, strengthened twice (`c61354c`) because +a skill built for human partnership keeps trying to ask the human. + +`iteration-reviewer` is the most complex at 190 lines: it applies accepted-findings filtering, +oscillation detection, cross-dimension dedup, and treats prototype UX deviations as non-negotiable +(`bf0e7cc`). It returns a strict JSON block — `{epic_id, version, scores{dim:{score,findings}}, +weighted_average, verdict, accepted_findings[], fix_applied, fix_version, summary}` — with "no other +text". It is the only agent holding the `Agent` tool, because it dispatches the fix subagent. + +## Two live defects in the definitions + +Both tracked in [`../bugs/open/`](../bugs/open/): + +- `iteration-reviewer.md` references `${BASE_SHA}`, which is not in its declared inputs and which + `review_cycle.py triage-prompt` never emits — an undefined variable in a prompt template. +- `iteration-reviewer.md:161` tells the fix path to produce `validation.json` by redirection, while + `SKILL.md` Step 13 mandates `--out`. Both produce authentic output and pass the [ADR-0024] gate, but a + reader cannot tell which is normative. + +## The dead one + +`rubrics/` — 5 files, 424 lines, referenced nowhere. Superseded by these agent definitions and **actively +wrong**: architecture 20% (real 30%), tests 25% (real 30%), intent 25% (real 20%), a `patterns` dimension +at 10% that does not exist, and `Model: sonnet` contradicting [ADR-0029]. Deletion is a pending task. + +## Testing + +**None of these 13 contracts has a test.** Neither does the 829-line `SKILL.md` that orchestrates them. +Contract drift — a renamed heading, a changed numbering style — surfaces only as a silently wrong score. diff --git a/docs/architecture/06-review-and-scoring.md b/docs/architecture/06-review-and-scoring.md new file mode 100644 index 0000000..6579290 --- /dev/null +++ b/docs/architecture/06-review-and-scoring.md @@ -0,0 +1,119 @@ +--- +id: 06-review-and-scoring +title: Review and scoring — how a diff becomes a number +type: plan +status: current +repos: [epic-code-gen] +decisions: [ADR-0022, ADR-0023, ADR-0024, ADR-0026, ADR-0028] +--- + +# Review and scoring + +## The rule + +**Reviewers classify findings by severity. Python computes the score.** No model ever chooses a number +([ADR-0022]). + +``` +dimension_score = max(1, 10 − 5.0·Critical − 1.5·Important − 0.5·Minor) +if any Critical: dimension_score = min(dimension_score, 5.0) # ADR-0023 +weighted_average = Σ (dimension_score × weight) +``` + +Constants, all in `score_reviews.py:34-49`: + +| Constant | Value | +|---|---| +| architecture / tests / lint / intent weights | 0.30 / 0.30 / 0.20 / 0.20 | +| `CRITICAL_WEIGHT` / `IMPORTANT_WEIGHT` / `MINOR_WEIGHT` | 5.0 / 1.5 / 0.5 | +| `CRITICAL_CAP` | 5.0 | +| `PASS_THRESHOLD` | 8.0 | +| `NEAR_MISS_THRESHOLD` | 7.0 | +| `MIN_DIMENSION_SCORE` | 6.0 | +| `HARD_FLOOR` | 5.0 | + +## Verdicts + +| Verdict | Condition | +|---|---| +| `pass` | `weighted_average ≥ 8.0` **and** no dimension `< 6.0` | +| `near-miss` | `≥ 7.0` (opens a PR anyway on exhaustion — [ADR-0033]) | +| `fail` | below that | +| `incomplete` | a dimension is missing | + +Because `CRITICAL_CAP` (5.0) is below `MIN_DIMENSION_SCORE` (6.0), **one Critical anywhere makes a pass +arithmetically impossible.** Not unlikely — impossible. That interaction is the real force of +[ADR-0023]. + +## The loop + +Driven by `review_cycle.py`, not by prose ([ADR-0026]). It owns the `REVIEWERS` table — six reviewers, +four scored. + +``` +1. review_cycle.py prompts → emit 6 dispatch prompts +2. dispatch 6 agents in parallel (no agentType — ADR-0027) +3. review_cycle.py wait → block until review files land +4. review_cycle.py verify → files well-formed and non-empty +5. review_cycle.py score → score_reviews.py → scores.json +6. verdict? + pass → save final diff, hand to PR creation + fail → review_cycle.py triage-prompt → iteration-reviewer + → fix agent → new version → back to 1 + exhausted → near-miss (≥7.0) ? open PR anyway : report best version +``` + +`SKILL.md` states the hard rule: **never write review files yourself.** `5e26193` added an anti-fallback +guardrail; if dispatch fails, fail — do not improvise. + +## How findings are counted + +Regex over markdown, in `_extract_findings`: + +- A heading matching `^#{1,4}\s+(critical|important|minor)$` (case-insensitive) opens a section. +- A finding is any line matching `^\d+\.\s+\*\*`. +- The dimension name comes from the filename: `^review-(\w+)\.md$`. + +> **This fails open.** An unrecognized heading spelling yields **zero findings**, which computes to +> **10.0**. A prompt-drift regression looks exactly like flawless code. There is no test asserting that a +> known-Critical review file scores 5.0. + +## The authenticity gate + +The lint dimension is scored from `validation.json`, which must be genuine tool output ([ADR-0024]). +`validation_document_status()` requires `all_passed` and `checks`: + +| Status | Effect on verdict | +|---|---| +| `ok` | scored normally | +| `missing` | advisory | +| `foreign` | **forced `fail`** | +| `unreadable` | **forced `fail`** | + +Recorded in `scores.json` under `validation`. It exists because a hand-written +`{"tests_total": 35, "success": true}` once scored `lint=8.0` while Prettier was failing. + +## Two known gaps + +1. **The gate can be walked past.** In RHAIFIRST-391 `foreign` was detected, recorded, and ignored; the PR + opened from a version that was never reviewed. A gate that reports rather than blocks is not a gate. +2. **The pass rule exists twice.** `_ci_handle_review_pending` (`run_pipeline.py:1348`) re-derives + `avg >= 8.0 and dims_ok` with a hard-coded 6.0 floor instead of reading the `verdict` + `score_reviews.py` computed. Only the `score_reviews` copy fails on a foreign `validation.json`, so the + two can disagree. + +Both in [`../bugs/open/`](../bugs/open/). + +## Evidence that it works + +Score progressions from the data repo, which are only meaningful *because* the number is computed rather +than chosen: + +| Epic | Progression | Outcome | +|---|---|---| +| RHAI-74 | 2.4 → 4.9 → 7.2 → 9.4 | Done, merged | +| RHAI-64 | 2.6 → 2.65 → 6.1 → 6.7 → 8.2 | PRCreated | +| RHOAIENG-72103 | → 8.15 over 5 versions | Done (first UX-AC epic) | + +Passing scores cluster 7.9–9.4. The v1 → v2 jump is consistently the largest, which is what motivated +spec-first generation ([ADR-0016]) — most of the early climb was recoverable design error. diff --git a/docs/architecture/07-target-repo-lifecycle.md b/docs/architecture/07-target-repo-lifecycle.md new file mode 100644 index 0000000..2720e7f --- /dev/null +++ b/docs/architecture/07-target-repo-lifecycle.md @@ -0,0 +1,119 @@ +--- +id: 07-target-repo-lifecycle +title: Target repo lifecycle — clone to merged PR +type: plan +status: current +repos: [epic-code-gen] +decisions: [ADR-0025, ADR-0030, ADR-0031, ADR-0032] +--- + +# Target repo lifecycle + +How the system touches somebody else's repository, in order. + +## 1. Resolve + +`run_pipeline.resolve_target_repo()` — keyword match against `config/repo_mapping.json` +(`{"<owner/repo>": {"keywords": [...]}}`, 6 repos), falling back to an LLM resolution +(`resolve_repo_via_llm`) using `config/`'s prompt when no keyword hits. + +Seven target repos to date: `ederign/codeflare-sdk`, `ederign/kale`, `opendatahub-io/mlflow`, +`opendatahub-io/mlflow-go`, `opendatahub-io/odh-dashboard`, `opendatahub-io/pipelines-components`, +`project-codeflare/codeflare-sdk`. + +## 2. Clone and branch + +```bash +python3 scripts/clone_target.py <repo-url> <EPIC_ID> [--dest .target-repo] \ + [--fork-owner user] [--clean] +``` + +- Expands a bare `owner/repo` slug to a full URL (`eda859c`). +- Detects the upstream default branch rather than assuming `main` — one epic needed `master`. +- Ensures the fork exists, **syncs it and fetches upstream before branching** (`addaaa3`), so the branch is + based on current upstream rather than a stale fork. +- Creates `epic/<EPIC_ID>`. `checkout_existing_branch()` is the review-response entry point. +- The `fork` remote carries an embedded token; `sanitized_url` keeps it out of error output (`ba23d02`). + +## 3. Assess readiness + +```bash +python3 scripts/repo_readiness.py <repo-path> +``` + +Six dimensions, score out of 12, **threshold 8**: integration tests, lint in CI, clear CI signals, +`CLAUDE.md`/`CONTRIBUTING.md`, `CODEOWNERS`, language properties. A repo below threshold is not a good +codegen target — the signals the review loop depends on aren't there. + +`e766e5f` softened this from a hard gate; RHAI-68 ran at readiness 9 with `codeowners: 0`. + +## 4. Toolchain preflight — **before generating anything** + +```bash +python3 scripts/validate_target.py <repo-path> --preflight [--json] +``` + +Exit 2 = missing tool (distinct from exit 1 = failing check). Required tools come from repo markers +(`uv.lock`, `yarn.lock`) **and** from variable-expanded Makefile recipes for the exact lint/typecheck/test +targets that would run, following prerequisites — so an unrelated `docker-build` recipe doesn't gate +codegen. + +A gap flags the epic and **generates nothing**; status stays `Ready` so it retries once the image is fixed. +A missing tool is an environment fault, not the epic's ([ADR-0025]). + +## 5. Validate + +```bash +python3 scripts/validate_target.py <repo-path> [--json] [--out FILE] [--checks lint,test] +``` + +Detects language (Go, Python, TypeScript, JavaScript, Rust) from markers and discovers commands from +Makefile targets and `package.json` scripts. Reports a check that couldn't execute as `unrunnable` with +`missing_tool`, never as a plain failure. + +**Consumers read `all_passed`, never per-check keys.** Always produce the file with `--out`; never hand-write +it ([ADR-0024]). + +## 6. Generate + +Phase 2 of the skill, inside `.target-repo/` on `epic/<EPIC_ID>`. Commits accumulate on the branch; +`BASE_SHA` is recorded first so the diff is computable. + +## 7. Open the PR + +```bash +python3 scripts/push_to_fork.py … # push to the fork +python3 scripts/create_pr.py … # PR upstream from the fork branch +``` + +Uses the **target repo's own PR template** and detected default branch (`4169f06`, `19abb33`) so the PR +reads as a native contribution. Fork-based, under `dora-the-ai-coder` ([ADR-0030]). `da3beaf` handles +duplicate creation gracefully, because the convergence loop can reach this step twice. + +## 8. Answer review comments + +Per cycle, in this order ([ADR-0032]): + +```bash +python3 scripts/rebase_pr.py <repo-path> <branch> [--base main] [--push-remote fork] +``` + +1. **Rebase onto upstream base first** ([ADR-0031]). Conflicts: `rebase_onto_base()` drives the git + sequence, a subagent edits only the working tree. Push with `--force-with-lease`. +2. Triage comments — humans always, bots selectively (`config/review_config.json`). +3. One fix agent, all comments, one commit. **Only code inside our own diff** + (`compute_diff_scope` / `is_comment_in_scope`). +4. Validate + `sanity-check-agent`. No re-scoring. +5. Reply to every comment; record IDs in `pr-replies.json`. + +A cycle that rebases nothing and finds nothing actionable **does not consume an iteration**. + +## 9. Done + +`_ci_handle_pr_created` observes the merge (GitHub API, with a `gh` CLI fallback) and transitions to `Done`. +A PR closed unmerged goes back to `Ready` to regenerate. + +## Cleanup + +`make clean` removes `tmp/`, `.target-repo/`, `.context/`. The CI container is discarded anyway; the fork +branch persists deliberately, since the PR points at it. diff --git a/docs/architecture/08-ci-topology.md b/docs/architecture/08-ci-topology.md new file mode 100644 index 0000000..4c87963 --- /dev/null +++ b/docs/architecture/08-ci-topology.md @@ -0,0 +1,107 @@ +--- +id: 08-ci-topology +title: CI topology — GitLab pipeline, image, telemetry +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline] +decisions: [ADR-0010, ADR-0011, ADR-0012] +--- + +# CI topology + +CI lives in `epic-code-gen-pipeline` (GitLab). `epic-code-gen` itself had **no CI at all** until this +ledger added `.github/workflows/ledger.yml`. + +## Pipeline + +`.gitlab-ci.yml`. Stages: `codegen` → `trigger` → `secret-detection`. + +**Workflow rule:** merge-request pipelines are suppressed (`never`) — the jobs are manual and need CI +variables (`8245fad`). + +### `.codegen-base` (hidden template) + +| Setting | Value | +|---|---| +| `tags` | `aipcc-small-x86_64` | +| `image` | `quay.io/ederignatowicz/epic-code-gen-ci:latest` | +| `timeout` | **6h** (raised from 4h to match `CODEGEN_TIMEOUT`, `b87fc90`) | +| `artifacts` | `when: always`, `expire_in: 30 days` — `claude-otel.jsonl`, `claude-stderr.log`, `pipeline-runs/`, `artifacts/` | + +`before_script`: + +1. `setup-env.sh` +2. Write `RESULTS_PUSH_TOKEN` to `/home/claude-ci/.tokens/results`, then **`unset` it** so the Claude + subprocess cannot inherit it. +3. Re-inject it for exactly one command: `clone-data-repo.sh`. + +### `codegen-run` + +`when: manual`. Sole input `STRATEGY_KEYS` (space-separated); an inline guard exits 1 if it is empty. + +- `script`: `run-codegen.sh "$STRATEGY_KEYS"` +- `after_script`: `pipeline-post.sh "$STRATEGY_KEYS"` + +**`after_script` is deliberate** (`aff11df`): results persist even when codegen times out or crashes. +Caveat worth knowing — `after_script` is bounded by `RUNNER_AFTER_SCRIPT_TIMEOUT` (5 min default), +*independently* of the 6-hour job timeout, and it is where the data-repo clone-commit-push happens. + +### `trigger-dashboard` + +`needs: [codegen-run]`, multi-project trigger into `epic-code-gen-dashboard`. Guarded with +`if $CI_PIPELINE_SOURCE == "pipeline"` → `never`, to prevent trigger loops. + +### `secret_detection` + +GitLab's `Security/Secret-Detection.gitlab-ci.yml` template. Present in all three GitLab repos. + +## `run-codegen.sh` — the execution wrapper + +1. Preflight: require `JIRA_USER`, `GCP_PROJECT_ID`, `GCP_SERVICE_ACCOUNT_KEY`; `claude --version`. +2. Clone the brains repo `--depth 1` into `/tmp/claude-workdir`, log its HEAD for provenance. +3. Start `otel-collector.py` in the background; export the `OTEL_*` variables. +4. Start a **progress-monitor subshell**: every 300s tail new lines of `tmp/progress.log` prefixed `📋`, + else print `⏱️ Heartbeat` (`fa8f340`). +5. **Invoke the orchestrator directly** ([ADR-0012]) inside `set +e` / `set -e` to capture `rc`: + `python3 scripts/run_pipeline.py $KEYS --ci --data-repo /tmp/data-repo --fork-owner dora-the-ai-coder --timeout ${CODEGEN_TIMEOUT:-21600}` +6. Kill the monitor, `sleep 7` for OTEL flush, kill the collector (`d771535` fixed cleanup under `set -e`). +7. Print `otel-summary.py`, copy artifacts into `$CI_PROJECT_DIR`, `cat` stderr, `exit $rc`. + +## Telemetry + +- `otel-collector.py` — a ~100-line OTLP HTTP/JSON receiver on `127.0.0.1:4318`, appending + `{ts, path, payload}` per POST to `claude-otel.jsonl`. Also maintains a 60s rolling token rate in + `/tmp/claude-otel-rate.json` for live tokens/sec display. +- `otel-summary.py` — tokens per model (input / cacheRead / cacheCreation / output), cost per model, active + time, API request count. Claude Code emits **delta-temporality** metrics, so all deltas are summed — + which is what makes subagent usage count. +- `push-results.py:extract_otel_cost()` sums `claude_code.cost.usage` into the run log. ~$80 total logged. + +Model access is via **Vertex AI** (`CLAUDE_CODE_USE_VERTEX=1`, `CLOUD_ML_REGION=global`), not the Anthropic +API directly. + +## The image + +`epic-code-gen/Dockerfile.ci`, UBI9, built multi-arch (`linux/amd64,linux/arm64`) via +`make ci-image` / `ci-image-push`. Layer-by-layer rationale — including which layers exist to fix a +specific bug — is in [ADR-0011]. + +## Result push + +`pipeline-post.sh` → `push-results.py`: copy artifacts, **merge** `run-metadata.yaml` ([ADR-0014]), +regenerate `strategy-summary.json` and `summary.json`, append `run-log.jsonl`, copy the OTEL file, then +`git add -A` → commit → `pull --rebase -X theirs` → up to **3** push attempts. **Never force-pushes.** + +> **Live defect:** `pipeline-post.sh:41` builds `--strategy-key <k>` per key. Argparse +> abbreviation-matches the `nargs="+"` `--strategy-keys` and **overwrites** rather than appends, so the +> per-key loop only ever runs for the last key. Verified. Codegen artifacts still land (`run_pipeline.py` +> writes them live and `git add -A` sweeps them up), but every strategy except the last loses its state +> merge, its `strategy-summary.json` refresh, its `run-log.jsonl` entry, and its OTEL file. See +> [`../bugs/open/`](../bugs/open/). + +## Missing + +- **No `resource_group`** on `codegen-run`, so concurrent runs race on the data repo. +- **No schedule.** Still manually triggered; the `resource_group` gap is the blocker. +- The four shell scripts, `otel-collector.py`, `otel-summary.py`, and `stream-claude.py` have **zero + tests**. `make lint` is `shellcheck … || true`, which passes silently when shellcheck is absent. diff --git a/docs/architecture/09-secrets-and-environment.md b/docs/architecture/09-secrets-and-environment.md new file mode 100644 index 0000000..4397464 --- /dev/null +++ b/docs/architecture/09-secrets-and-environment.md @@ -0,0 +1,101 @@ +--- +id: 09-secrets-and-environment +title: Secrets and environment variables +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Secrets and environment + +Every environment variable the system reads, collected in one place for the first time. Verified by +sweeping all access patterns — including the `_env_int()` indirection in `review_response.py`, which a +naive `os.environ` grep misses. + +## `epic-code-gen` — Python + +The whole surface is six variables. + +| Variable | Required | Read by | Default | +|---|---|---|---| +| `JIRA_SERVER` | ✔ | `jira_utils.require_env` | — | +| `JIRA_USER` | ✔ | `jira_utils.require_env` | — | +| `JIRA_TOKEN` | ✔ | `jira_utils.require_env` | — | +| `EPIC_CODEGEN_GITHUB_TOKEN` | ✔ | `github_utils` (`DEFAULT_TOKEN_VAR`) | — | +| `RH_FORGE_GITHUB_TOKEN` | | `run_pipeline.identity_for_repo`, via `config/repo_mapping.json` | — | +| `REVIEW_FIX_AGENT_TIMEOUT` | | `review_response._env_int` | `3600` | +| `REVIEW_SANITY_CHECK_TIMEOUT` | | `review_response._env_int` | `600` | + +The GitHub token variable name is overridable per invocation via `--gh-token-var` / `--token-var`. + +It is also overridable **per target repo**: a `config/repo_mapping.json` entry may name its own +`gh_token_var` alongside a `fork_owner`, and `identity_for_repo()` resolves both at every call site +that touches GitHub. The mapping holds the variable *name*, never a value — the credential is read +from the environment at the point of use, so it appears in neither the repo nor the CI log. +`RH_FORGE_GITHUB_TOKEN` is the first of these, for the private `rh-forge` org. See [ADR-0035]. + +> **Asymmetry worth knowing:** two agent timeouts are env-tunable, but +> `rebase_pr.CONFLICT_AGENT_TIMEOUT` (900s) and `MAX_CONFLICT_ROUNDS` (10) are plain constants. A +> long-running rebase conflict cannot be given more time without a code change. + +## `epic-code-gen` — shell (`ci-scripts/run-claude.sh`) + +| Variable | Purpose | Default | +|---|---|---| +| `CLAUDE_MODEL` | model passed to `claude -p` | `claude-opus-4-6` | +| `LOG_DIR` | where `claude-stderr.log` is written | `/tmp` | +| `LOG_FILE` | if set, the stream renderer writes here | unset | +| `CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS` | **set to 0**, disabling the background-task wait ceiling for CI subagents (`18d3cb0`) | — | + +> **Contradiction on the record:** this script *pins* `claude-opus-4-6`, while `README.md`, +> `CLAUDE.md`, and `SKILL.md` all state that agents inherit the session model with no overrides +> ([ADR-0029]). Both cannot be true. Tracked in `docs/bugs/open/`. + +## `epic-code-gen-pipeline` — CI variables + +Set in GitLab project settings; **not** declared in `.gitlab-ci.yml`. + +| Variable | Kind | Purpose | +|---|---|---| +| `GCP_PROJECT_ID` | secret | Vertex AI project | +| `GCP_SERVICE_ACCOUNT_KEY` | secret | base64 JSON, decoded to `/tmp/gcp-key.json` | +| `JIRA_API_TOKEN` | secret | mapped to `JIRA_TOKEN` | +| `JIRA_USER` | config | | +| `EPIC_CODEGEN_GITHUB_TOKEN` | secret | fork + PR operations, for every target without an override | +| `RH_FORGE_GITHUB_TOKEN` | secret, optional | the private `rh-forge` org only ([ADR-0035]) | +| `RESULTS_PUSH_TOKEN` | secret | pushes to the data repo | +| `STRATEGY_KEYS` | job input | space-separated strategy keys — the one operator input | +| `CODEGEN_TIMEOUT` | optional | seconds; default `21600` (6h) | +| `CLAUDE_MODEL` | optional | | +| `CLAUDE_REPO_BRANCH` | optional | branch of the brains repo to run | + +Declared in `.gitlab-ci.yml`: `CLAUDE_CODE_USE_VERTEX=1`, `ANTHROPIC_VERTEX_PROJECT_ID`, +`CLOUD_ML_REGION=global`, `DISABLE_AUTOUPDATER=1`, `JIRA_SERVER`, `JIRA_TOKEN`, `CLAUDE_REPO`, +`RESULTS_REPO`, `DATA_REPO_DIR=/tmp/data-repo`, `GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcp-key.json`, +`SECRET_DETECTION_ENABLED`, `FF_TIMESTAMPS`. + +Set by `run-codegen.sh` for telemetry: `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `OTEL_METRICS_EXPORTER=otlp`, +`OTEL_LOGS_EXPORTER=otlp`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`, +`OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318`, `OTEL_METRIC_EXPORT_INTERVAL=10000`, +`OTEL_LOG_FILE=/tmp/claude-otel.jsonl`. + +## How secrets are handled + +Three deliberate practices, worth preserving: + +1. **Tokens are moved to disk and unset from the environment** before Claude Code starts, so the + subprocess does not inherit them. `.codegen-base`'s `before_script` writes `RESULTS_PUSH_TOKEN` to + `/home/claude-ci/.tokens/results` then `unset`s it, re-injecting it for exactly one command. + `pipeline-post.sh` deletes `/home/claude-ci/.tokens` when done. +2. **Token length is logged, never the value** (`clone-data-repo.sh`). +3. **Token-embedded remote URLs are sanitized out of error output** — + `github_utils.sanitized_url`, added in `ba23d02` after credentials appeared in a git error. + +GitLab Secret Detection runs as its own stage in all three GitLab repos. + +## Local development + +`JIRA_SERVER`, `JIRA_USER`, `JIRA_TOKEN` are enough for read-only Jira work (`fetch_jira_epics.py +--json`, `fetch_epic.py`). `EPIC_CODEGEN_GITHUB_TOKEN` is needed for anything touching forks or PRs. + +**Do not run `run_pipeline.py` locally** — it is CI-only. `--dry-run` is safe. diff --git a/docs/architecture/10-known-limitations.md b/docs/architecture/10-known-limitations.md new file mode 100644 index 0000000..bf851e1 --- /dev/null +++ b/docs/architecture/10-known-limitations.md @@ -0,0 +1,105 @@ +--- +id: 10-known-limitations +title: Known limitations +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +--- + +# Known limitations + +What this system does not do well, stated plainly. Individual defects live in +[`../bugs/open/`](../bugs/open/); this is the structural view. + +## 1. It reports success it hasn't earned + +The defining failure mode. Five of the twelve epics under RHAIFIRST-168 are bug reports and **all five +describe a silent success**: + +| Issue | What was reported | What happened | +|---|---|---| +| 374 | job exit 0, "2 skipped" | strategy deadlocked; two merge-quality PRs stranded | +| 375 | review answered | `CHANGES_REQUESTED` bodies dropped entirely | +| 391 | `codegen_outcome: completed`, PR opened | PR came from a version never reviewed or scored | +| 392 | `lint: 4.5`, epic failed | the repo's own `main` was red; epic never touched those files | +| — (`d807dca`) | generic failure | the real fix-agent error was swallowed | + +This is structural, not incidental: a system that reviews its own output will rate that output as good +unless something outside the model's judgment says otherwise. Every hardening fix has been the same move — +take a decision away from the model ([ADR-0022], [ADR-0026]), or make the failure loud ([ADR-0015]). + +**Still true today:** advisory signals get ignored. In RHAIFIRST-391 the `validation.json` provenance guard +fired, was recorded in `scores.json`, and the orchestrator opened the PR anyway. Writing a finding down is +not the same as blocking on it. + +## 2. Manual intervention is routine, not exceptional + +**39 of the data repo's 104 commits are humans repairing state.** Their subjects read as an incident log: +`Unwedge RHAI-74/RHAI-76 from invalid 'completed' state`, `Restore RHAI-75 to PRCreated after a clobbered +state file`, `Fix RHOAIENG-72532 target_branch: master (not main)`, and ~12 × `Clean RHOAIENG-72103 for +re-run with <fix>`. + +This is the direct cost of git-as-database ([ADR-0008]): every recovery is a hand-edited YAML commit. The +merge guard ([ADR-0014]) removed the largest cause, but the recovery mechanism is unchanged. + +## 3. No concurrency protection + +`codegen-run` has **no `resource_group`**. Two concurrent manual triggers race on the data repo, mitigated +only by push-retry-with-rebase (3 attempts, `-X theirs`). `FOREDER.md` identified this in June as the +prerequisite for moving off a manual trigger; it is still unaddressed, which is the real reason the +pipeline is still manually triggered. + +## 4. The review gate is bypassable + +Scoring is deterministic ([ADR-0022]) and the loop is Python ([ADR-0026]), but the skill still has to +*call* the loop. The anti-fallback rule ("never write review files yourself") is a prompt instruction, not +a mechanism, and reviewers hold `Write` by necessity ([ADR-0027]). RHAIFIRST-391 is that gap being +exercised. + +## 5. Baseline repo health is not separated from epic quality + +A target repo whose `make lint` is red on `main` fails every epic generated against it. Worse, GNU make +stops at the first failing prerequisite, so an early baseline failure **conceals** genuine findings that +would have run after it. RHAIFIRST-392, open. The `unrunnable` ≠ `failed` distinction ([ADR-0025]) covers +checks that *couldn't run*, not checks that fail for reasons the epic didn't cause. + +## 6. Growth is unbounded + +The data repo is 33 MB for 24 epics, **13.7 MB of it three OTEL files** (one is 9.8 MB). Versions +accumulate and are never deleted, by design. No pruning strategy exists — predicted in `FOREDER.md`. + +## 7. Single-owner dependencies in the critical path + +The brains repo is on personal GitHub (`github.com/ederign/epic-code-gen`) and the CI image on personal +Quay (`quay.io/ederignatowicz/`), while the other three repos are under +`gitlab.com/redhat/rhel-ai/agentic-ci/`. Two bus-factor-one dependencies for a pipeline being positioned +for production rollout. + +## 8. This repo does not meet its own standards + +The product's value proposition is enforcing lint, tests, and conventions on other repos. On itself, as of +2026-07-31: + +- **No CI** at all until this ledger added one, and **no linter or type checker** for 10.5k Python lines. +- `jira_utils.py` — 1,055 lines including the whole Markdown↔ADF converter — has **zero tests**. So do + `frontmatter.py`, `state.py`, and `parse_prototype.js` (699 lines, no JS test runner configured). +- `make test` **always fails** — `test-integration` collects zero tests and pytest exits 5. +- Six independent YAML parsers, three git wrappers, two HTTP clients, two slug extractors. +- `rubrics/` is 424 lines of dead, contradictory calibration still in the tree. +- `README.md` claims 186 tests (actual: 703), 3 iterations (actual: 10), and documented a script that did + not exist. +- `.claude/settings.json`'s allowlist omits scripts the skill actually runs — masked in CI by + `--dangerously-skip-permissions`, prompts interactively. + +## 9. Cost is high and unbudgeted per epic + +Everything runs on opus with no per-agent downgrades ([ADR-0029]) — deliberate, because cheaper reviewers +produced *higher* scores. ~$80 across 39 passes. Up to 10 codegen iterations plus 5 review-response +cycles, each dispatching 6+ agents, with no early-abandon on a flat score progression. + +## 10. Reviewer calibration is unverified + +Scores are computed from severity classifications ([ADR-0022]), so classification is now the whole game — +a Critical mislabelled Important moves a dimension 3.5 points. There is **no calibration test** asserting +that a known-Critical defect is classified Critical. Finding parsing also **fails open**: an unrecognized +markdown heading yields zero findings and therefore 10.0. diff --git a/docs/architecture/99-historical-foreder.md b/docs/architecture/99-historical-foreder.md new file mode 100644 index 0000000..e37744d --- /dev/null +++ b/docs/architecture/99-historical-foreder.md @@ -0,0 +1,222 @@ +--- +id: 99-historical-foreder +title: "FOREDER.md (historical, 2026-06-30) — recovered design brief" +type: plan +status: done +repos: [epic-code-gen-pipeline] +commits: ["183622a", "2b7496b", "3bd2d3e"] +decisions: [ADR-0005, ADR-0008, ADR-0009, ADR-0011] +--- + +# Historical: FOREDER.md + +> **Recovered document. Do not edit the body below.** +> +> This was written on 2026-06-30 (`183622a`, extended by `2b7496b`) as a design brief for the +> pipeline build-out, then **removed from tracking and gitignored** the same day in `3bd2d3e`. It was +> invisible for a month, and it is the most complete design rationale the project produced: the state +> machine table, six named design decisions, what was and wasn't copied from `strat-pipeline`, and +> five predicted pitfalls — **all five of which came true.** +> +> Recovered verbatim from `git show 3bd2d3e^:FOREDER.md`. It is preserved as written, in its original +> voice (it was addressed to a colleague, not to a reader). +> +> **What has changed since it was written** — read the body as a 2026-06-30 snapshot: +> +> | Claim in the document | Status today | +> |---|---| +> | "4 independent reviewers" | Six agents dispatched; four scored, two advisory ([ADR-0028]) | +> | "Scoring rubrics (calibration tables)" | `rubrics/` is dead and its weights are wrong; calibration lives in `.claude/agents/` ([ADR-0021]) | +> | "138 tests across both repos" | 703 in `epic-code-gen`, 16 in the pipeline repo | +> | Manual trigger, "just add a `rules: - schedules` line" later | Still manual. The `resource_group` prerequisite it identifies is still absent | +> | Data-repo growth needs a pruning strategy | Predicted correctly; 33 MB, no pruning exists | +> | Concurrent strategy processing needs a `resource_group` | Predicted correctly; still unmitigated | +> | State machine table | Accurate, and now documented with transitions in [02-pipeline-state-machine.md](02-pipeline-state-machine.md) | +> +> The reason this document was worth recovering, rather than rewriting: it records *why* six decisions +> were made, by the person making them, at the time. That is not reconstructible after the fact — +> which is the entire argument for [ADR-0034]. + +--- + +# FOREDER: Epic Code Gen Pipeline + +Hey Eder — this is your deep-dive guide to the epic-code-gen pipeline system. Not a reference manual, more like sitting down with a colleague who built it and having them walk you through everything. + +## What Are We Actually Building? + +Remember how `epic-code-gen` started as a local thing? You'd fire up Claude Code, point it at a strategy's epics, and it would generate code, review it, maybe create a PR. It worked great for our POC — RHAISTRAT-1749-E001 passed on the first try with a 9.4/10 score. But it required you to babysit it. + +We're turning that into an autonomous pipeline. Think of it like a factory line that: + +1. Picks up strategies tagged as ready +2. Figures out which epics need work (and which are blocked, done, or waiting on PR reviews) +3. Does one round of work on each epic that's actionable +4. Saves everything, pushes results, updates Jira +5. Waits for the next trigger (manual for now, scheduled later) +6. Repeats until every epic in the strategy is done + +The key insight is **convergence** — it's not a one-shot pipeline. It's a loop that keeps narrowing the gap between "where we are" and "everything done." Each run moves epics forward by one step, whatever step that is. + +## The Three-Repo Architecture + +This mirrors what we did with `strat-creator` → `strat-pipeline` → `strat-pipeline-data`, but with important differences. + +### epic-code-gen (existing, the brains) + +This is where all the intelligence lives: +- Python scripts that orchestrate everything (`run_pipeline.py`, `fetch_jira_epics.py`, `clone_target.py`, etc.) +- Claude Code skills (the `/epic-codegen` skill that actually generates code) +- Reviewer agents (4 independent reviewers: architecture, tests, lint, intent) +- Scoring rubrics (calibration tables so reviewers are consistent) + +Think of it as the engine. The pipeline repo is just the car around it. + +### epic-code-gen-pipeline (new, the CI shell) + +This is deliberately thin. Its job: +- Set up the CI environment (GCP credentials, Git config, clone repos) +- Call `run_pipeline.py` from epic-code-gen +- After the run, push results to the data repo +- Trigger the dashboard rebuild + +The `.gitlab-ci.yml` is ~30 lines. The ci-scripts are adapted from strat-pipeline (proven patterns, not invented from scratch). The real logic lives in epic-code-gen's Python code, not here. + +**Why thin?** Because you can also run `run_pipeline.py` locally for debugging. If the orchestration lived in shell scripts inside the pipeline repo, you'd need GitLab CI to test anything. By keeping it in Python, `python3 scripts/run_pipeline.py --ci --data-repo ./test-data RHAISTRAT-1749` works on your laptop. + +### epic-code-gen-pipeline-data (new, the artifact store) + +This is a Git repo used as a database. Every run writes its artifacts here: + +``` +RHAISTRAT-1749/ +├── RHAISTRAT-1749-E001/ +│ ├── run-metadata.yaml ← the epic's current state (the "row" in our "database") +│ ├── codegen-spec.md ← what to build +│ ├── codegen-plan.md ← how to build it +│ ├── v1/ ← first attempt +│ │ ├── diff.patch ← the actual code changes (JUST the diff, never full files) +│ │ ├── validation.json ← did lint/tests pass? +│ │ ├── review-*.md ← 4 reviewer outputs +│ │ └── scores.json ← dimension scores +│ └── v2/ ← second attempt (after PR feedback or review failure) +│ └── ... +├── RHAISTRAT-1749-E002/ +│ └── ... +└── run-log.jsonl ← append-only log of every pipeline pass +``` + +**Why organized by strategy/epic/version instead of by timestamp?** This was a deliberate departure from strat-pipeline-data (which uses `YYYYMMDD-HHMMSS/` folders). When you're debugging why RHAISTRAT-1749-E001 failed, you want to see its entire history in one directory — not hunt through 15 timestamped folders to piece together what happened across 15 runs. The `run-log.jsonl` gives you the timeline view when you need it. + +**Why diffs only?** The target repo (e.g., `mlflow/mlflow`) is the source of truth. Storing full files would be wasteful and would diverge from the actual repo state. A diff.patch is small, portable, and tells you exactly what changed. + +### epic-code-gen-dashboard (new, the window) + +Static HTML/JS served via GitLab Pages. Three views: + +1. **Strategy Drilldown** — click a strategy → see all its epics with status badges → click an epic → see its versions with scores and review summaries. Progress bars show how close a strategy is to complete. + +2. **Jira State Log** — timeline of every state transition. "E001 went from Ready to ReviewPending at 10:00, then to PRCreated at 10:45." Built from `run-log.jsonl`. + +3. **Cost & Telemetry** — how many tokens, how much money, per strategy/epic/version. Built from OTEL data that Claude Code emits. + +## The State Machine (This Is the Core Abstraction) + +Every epic lives in one of these states: + +``` +Pending → Ready → Generating → ReviewPending → PRCreated → PRChangesRequested → Done + ↓ + (also: Blocked, Failed) +``` + +Each pipeline run reads every epic's state from `run-metadata.yaml` and does exactly one thing: + +| State | What happens | +|-------|-------------| +| **No metadata** | New epic discovered. Check dependencies. Set to Ready or Blocked. | +| **Ready** | Clone target repo, run codegen, generate diff. Move to ReviewPending. | +| **ReviewPending** | Score the diff with 4 reviewers. If pass → create PR → PRCreated. If fail → bump version, stay ReviewPending. | +| **PRCreated** | Check GitHub. Merged? → Done. Review comments? → PRChangesRequested. Closed? → Ready (retry). | +| **PRChangesRequested** | Pull review comments, feed into next codegen iteration. Back to ReviewPending. | +| **Blocked** | Check if blocking epic is Done. If so → Ready. | +| **Done** | Skip. | +| **Failed** | Log and skip. Needs human intervention. | + +**One iteration per pass.** We deliberately chose not to loop within a single run. If an epic generates code that fails review, it writes the failure, and the *next* pipeline run picks it up for v2. This is simpler to debug (each run does one thing per epic), and more observable (you can see every step in the run-log). + +## Key Design Decisions & Why + +### Strategy as unit of work, not epic +Epics within a strategy have dependencies. E002 might depend on E001's code being merged first. The pipeline needs to see the whole picture — the dependency DAG — to know what's actionable. If we processed individual epics in isolation, we'd miss these relationships. + +### Python orchestration, not shell +The strat-pipeline uses shell scripts for some orchestration, and it works because the logic is simple (process each RFE through create → refine → review). For epic-code-gen, we have dependency DAGs, state machines, PR lifecycle management, GitHub API calls — this would be unmaintainable in bash. Python gives us proper data structures, error handling, and testability. + +### Fat Docker image +We bake Python, Go, Node.js, Rust, and Claude Code into one big image. Yes, it'll be 3-5GB. The alternative — installing language runtimes at job start — adds 10+ minutes per run and introduces network failure modes. For a pipeline that runs heavy AI workloads ($$$ per run), spending a few extra GB of disk to save setup time is an obvious trade. + +### Manual trigger first +This pipeline creates PRs on real repos. We're not scheduling it until we've validated end-to-end with real strategies and built confidence. When we're ready, it's just adding a `rules: - schedules` line to the CI config. + +### Data repo as state store (not Jira) +Jira is for business visibility (humans check ticket status). But the pipeline needs machine-readable state that's fast to read, version-controlled, and doesn't depend on Jira's API being available. `run-metadata.yaml` in Git gives us all of that. Jira transitions happen as a side effect, not as the source of truth. + +## Lessons from strat-pipeline + +We copied proven patterns rather than inventing new ones: + +- **OTEL collector**: Claude Code emits OpenTelemetry metrics. strat-pipeline has a lightweight Python HTTP listener that captures them to JSONL. It works. We copied it. + +- **Retry-with-rebase for data repo pushes**: When two CI jobs finish around the same time and both push to the data repo, one will fail. strat-pipeline handles this with a pull-rebase-retry loop (3 attempts). No force-push. We copied it. + +- **Jira label locking**: strat-pipeline uses `strat-creator-processing` as a mutex. We use `epic-codegen-active` for the same purpose — prevents two runs from processing the same strategy simultaneously. + +- **Thin CI, fat Python**: strat-pipeline's `.gitlab-ci.yml` calls shell scripts that call Claude. Our `.gitlab-ci.yml` calls shell scripts that call Python that calls Claude. The principle is the same: CI config is glue, not logic. + +What we *didn't* copy: +- **Timestamped run directories**: strat-pipeline uses `YYYYMMDD-HHMMSS/` folders. We use strategy/epic/version because our data is inherently hierarchical and long-lived (an epic might take 10 runs to complete). +- **`current` symlink**: strat-pipeline has a `current` symlink to the latest run. We don't need it — our structure is navigable by design. + +## How Good Engineers Think About This + +**Start with the data model.** Before writing any pipeline code, we designed the data repo structure. What does `run-metadata.yaml` contain? What goes in each version folder? Once that's clear, the pipeline code writes itself — it's just "read state, do one thing, write state." + +**The state machine is everything.** If you get the states and transitions right, the pipeline is just a switch statement. If you get them wrong, you'll be patching edge cases forever. We spent time getting the state machine right before writing a line of implementation code. + +**Copy before you create.** strat-pipeline exists. It works. It's been running for months. We didn't design a new CI pattern — we took their proven pattern and adapted it. The only "new" parts are where epic-code-gen genuinely differs (state machine, PR lifecycle, multi-language support). + +**Small iterations.** We're building this in order: Dockerfile → data repo → run_pipeline.py → pipeline repo → dashboard. Each step is testable independently. We're not trying to build the whole thing and hope it works. + +## Potential Pitfalls to Watch For + +**Image size.** The fat image will be big. If it becomes a problem (slow pulls, storage costs), we can optimize later — multi-stage builds, stripping debug symbols, using slim base images. But don't optimize prematurely. + +**Claude context limits.** Long codegen runs (big epics, complex repos) might hit Claude's context window. `state.py` helps survive context compression, but there's a ceiling. If you see degraded quality on large epics, that's probably why. + +**GitHub rate limits.** The GitHub API has rate limits. If we're processing 20 epics that all need PR status checks, we might hit them. The `github_utils.py` module should handle retries with backoff. + +**Data repo growth.** Every version of every epic stores a diff.patch plus review files. Over months, this adds up. We'll eventually need a pruning strategy (archive old versions, keep only latest N). Not urgent, but worth knowing about. + +**Concurrent strategy processing.** Right now, only one strategy runs at a time (manual trigger). When we add scheduling, we'll need resource groups (like strat-pipeline's `resource_group: strat-batch`) to prevent collisions. The Jira label locking handles epic-level dedup, but strategy-level serialization needs CI config. + +## Implementation Status + +All five stories are implemented and pushed: + +| Story | Repo | What's there | +|-------|------|-------------| +| CI Image | epic-code-gen | `Dockerfile.ci` (UBI9 + Python 3.11 + Go 1.24 + Node 22 + Claude Code), Makefile targets | +| Pipeline repo | epic-code-gen-pipeline | `.gitlab-ci.yml`, 8 CI scripts (setup, clone, run, post, OTEL, stream), `push-results.py` | +| run_pipeline.py | epic-code-gen | `--ci`/`--data-repo` flags, state machine, `pr_lifecycle.py` (GitHub API, review feedback) | +| Data repo | epic-code-gen-pipeline-data | README with structure docs, ready for first run | +| Dashboard | epic-code-gen-dashboard | `.gitlab-ci.yml` for Pages, generator in pipeline repo (`scripts/generate-dashboard.py`) | + +**Test coverage:** 138 tests across both repos (84 existing + 28 CI-mode + 26 pipeline repo). All passing. + +**Next steps to go live:** +1. Build and push Docker image: `make ci-image-push` (from epic-code-gen) +2. Set GitLab CI/CD variables on the pipeline repo (GCP_PROJECT_ID, GCP_SERVICE_ACCOUNT_KEY, JIRA_API_TOKEN, EPIC_CODEGEN_GITHUB_TOKEN, RESULTS_PUSH_TOKEN) +3. Manual trigger with `STRATEGY_KEYS=RHAISTRAT-XXXX` +4. Watch the first run, verify state machine transitions in data repo +5. Check the dashboard at the GitLab Pages URL diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..af1f89f --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,57 @@ +--- +id: architecture-readme +title: Architecture — index +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +--- + +# Architecture + +How the system is built. For *why*, see [`../decisions/`](../decisions/). For process, see +[`AGENTS.md`](../../AGENTS.md). + +| Doc | Read it when | +|---|---| +| [01 — System overview](01-system-overview.md) | You are new. Start here. | +| [02 — Pipeline state machine](02-pipeline-state-machine.md) | An epic is stuck, or you're changing `run_pipeline.py` | +| [03 — Artifact contracts](03-artifact-contracts.md) | You're reading or writing any pipeline file | +| [04 — Codegen skill phases](04-codegen-skill-phases.md) | You're changing `SKILL.md` | +| [05 — Agent roster](05-agent-roster.md) | You're changing an agent, or a score looks wrong | +| [06 — Review and scoring](06-review-and-scoring.md) | You're touching the review loop or the rubric | +| [07 — Target repo lifecycle](07-target-repo-lifecycle.md) | You're onboarding a target repo or debugging a PR | +| [08 — CI topology](08-ci-topology.md) | A CI job failed, or you're changing the image | +| [09 — Secrets and environment](09-secrets-and-environment.md) | You need a credential or an env var | +| [10 — Known limitations](10-known-limitations.md) | Before you promise anyone this is production-ready | +| [99 — FOREDER.md (historical)](99-historical-foreder.md) | You want the original design rationale, recovered | + +## How a Jira epic becomes a merged PR + +One paragraph, then go read [01](01-system-overview.md). + +An operator sets `STRATEGY_KEYS` and triggers the GitLab job. The orchestrator asks **Jira** which epics +exist under those strategies and builds a dependency DAG from "Blocks" links. For each epic it reads +current state from the **data repo** and takes **exactly one action** — that's the whole design: progress +happens across runs, not within one, so the pipeline never blocks waiting on a human. When the action is +"generate", it clones the target repo, checks the toolchain is present *before* spending anything, then +hands one epic to the `/epic-codegen` skill. The skill discovers how this repo actually does things, has a +design conversation with itself, writes a plan, implements it via Superpowers SDD, and then submits the +diff to six reviewers. Four of them classify findings by severity and **Python computes the score** — no +model ever picks a number. Score ≥ 8.0 with no dimension below 6.0 opens a PR from a bot's fork, using the +target repo's own PR template. Below that, triage picks what to fix and the loop runs again, up to ten +times. Once the PR is open, later runs rebase it onto current upstream and answer review comments as new +commits on the same branch — never by regenerating. When the PR merges, the epic is `Done`. + +## The one thing to understand + +This system's characteristic failure is **not** a wrong answer. It is a *confident* answer with nothing +behind it — a PR opened from a version that was never reviewed, a score estimated in prose, an epic marked +`completed` in a vocabulary nothing reads, a fabricated `validation.json` scoring 8.0 while the linter was +failing. Five of the twelve epics under RHAIFIRST-168 are that bug in a new place. + +Almost every design decision here is a countermeasure. Scores are arithmetic, not judgment +([ADR-0022]). The loop is Python, not prose ([ADR-0026]). Evidence documents are checked for provenance +([ADR-0024]). An unknown state fails loudly instead of skipping quietly ([ADR-0015]). An environment +fault is not the epic's fault ([ADR-0025]). + +If you change something here, the question to ask is: *what would make this lie, and what would catch it?* diff --git a/docs/bugs/fixed/bug-node-modules-committed.md b/docs/bugs/fixed/bug-node-modules-committed.md new file mode 100644 index 0000000..cbcf598 --- /dev/null +++ b/docs/bugs/fixed/bug-node-modules-committed.md @@ -0,0 +1,38 @@ +--- +id: bug-node-modules-committed +title: node_modules was committed to git +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["f978330", "e54900b"] +--- + +# Bug: node_modules was committed to git + +## Summary + +`node_modules` was accidentally committed when Playwright was added. + +## Reproduction + +1. `git log --stat` around the Playwright work. + +## Expected + +`node_modules` is gitignored and never tracked. + +## Actual + +Committed, then removed in a follow-up. + +## Impact + +Low + +## Evidence + +`e54900b` then updated `package-lock.json` (jsdom replaced by playwright). Note `package-lock.json` is currently **both tracked and gitignored** — see [[bug-gitignored-files-are-tracked]]. + +## Related Tasks + +- [[task-ux-prototype-pipeline]] diff --git a/docs/bugs/fixed/bug-non-codegen-epics-processed.md b/docs/bugs/fixed/bug-non-codegen-epics-processed.md new file mode 100644 index 0000000..412bb78 --- /dev/null +++ b/docs/bugs/fixed/bug-non-codegen-epics-processed.md @@ -0,0 +1,40 @@ +--- +id: bug-non-codegen-epics-processed +title: Epics outside codegen scope were processed +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["996640a"] +decisions: [ADR-0007] +--- + +# Bug: Epics outside codegen scope were processed + +## Summary + +The pipeline processed every child of a strategy, including work items from projects that are not codegen targets and epics explicitly labelled to be skipped. + +## Reproduction + +1. Add a child work item from an out-of-scope project to a strategy. +2. Run the pipeline. + +## Expected + +Out-of-scope and skip-labelled epics are excluded before any work is done. + +## Actual + +They were classified, cloned, and in some cases generated against. + +## Impact + +Medium + +## Evidence + +Fix added `is_codegen_project()` and `has_skip_label()` (`SKIP_LABEL`, `CODEGEN_PROJECTS`) as guards in `ci_process_epic` **before** state dispatch. Real skip reasons now visible in run logs: `"Skipped (epic-code-gen-skip)"`, `"Project not in codegen scope"`. + +## Related Tasks + +- [[task-jira-direct-epic-fetching]] diff --git a/docs/bugs/fixed/bug-nonzero-claude-exit-read-as-failure.md b/docs/bugs/fixed/bug-nonzero-claude-exit-read-as-failure.md new file mode 100644 index 0000000..16b83e6 --- /dev/null +++ b/docs/bugs/fixed/bug-nonzero-claude-exit-read-as-failure.md @@ -0,0 +1,39 @@ +--- +id: bug-nonzero-claude-exit-read-as-failure +title: A non-zero Claude exit was read as codegen failure even when work completed +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["1b06fbe", "de256bb"] +--- + +# Bug: A non-zero Claude exit was read as codegen failure even when work completed + +## Summary + +`invoke_codegen` treated any non-zero exit from the Claude session as a failed codegen, including cases where the session had completed the work and exited oddly. + +## Reproduction + +1. Run codegen so that the session produces `v*/diff.patch` and then exits non-zero. +2. Observe the epic transition. + +## Expected + +Completed work is recognised and the epic advances. + +## Actual + +Marked failed; the completed diff was discarded and the epic regenerated next run. + +## Impact + +High + +## Evidence + +Fixed by treating the presence of `v*/diff.patch` as evidence of real work (`de256bb` changed detection from `run-metadata.yaml` to `v*/diff.patch`). This traded one problem for a weaker one — artifact presence is still a weak liveness proxy for a genuinely crashed run: [[bug-artifact-presence-is-weak-liveness-proxy]]. + +## Related Tasks + +- [[task-idempotent-pipeline]] diff --git a/docs/bugs/fixed/bug-playwright-headless-in-container.md b/docs/bugs/fixed/bug-playwright-headless-in-container.md new file mode 100644 index 0000000..76df919 --- /dev/null +++ b/docs/bugs/fixed/bug-playwright-headless-in-container.md @@ -0,0 +1,41 @@ +--- +id: bug-playwright-headless-in-container +title: Playwright could not run headless chromium as a non-root user in CI +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["210c464", "5571f31", "0346470", "976a59d"] +decisions: [ADR-0020] +--- + +# Bug: Playwright could not run headless chromium as a non-root user in CI + +## Summary + +UX prototype parsing needs headless chromium. It failed three separate ways in the container before working: browser path unresolvable for the non-root user, the global `playwright` module not resolvable, and missing system libraries. + +## Reproduction + +1. Run `node scripts/parse_prototype.js` in the CI image as `claude-ci`. +2. Observe browser launch failure. + +## Expected + +Chromium launches headless and the prototype is parsed. + +## Actual + +Three distinct failures, each requiring a separate image fix and a failed CI run to discover. + +## Impact + +Medium + +## Evidence + +`210c464` set `PLAYWRIGHT_BROWSERS_PATH=/opt/playwright` with `chmod -R 755`; `5571f31` added `NODE_PATH=/usr/lib/node_modules`; `0346470` added `nspr`, `nss`, `libxkbcommon` alongside the earlier `alsa-lib atk at-spi2-atk cups-libs libdrm libXcomposite libXdamage libXrandr mesa-libgbm pango`. `976a59d` then fixed screenshot cropping. + +## Related Tasks + +- [[task-ux-prototype-pipeline]] +- [[task-ci-image-and-build-infrastructure]] diff --git a/docs/bugs/fixed/bug-preflight-blind-to-pnpm.md b/docs/bugs/fixed/bug-preflight-blind-to-pnpm.md new file mode 100644 index 0000000..2b08753 --- /dev/null +++ b/docs/bugs/fixed/bug-preflight-blind-to-pnpm.md @@ -0,0 +1,63 @@ +--- +id: bug-preflight-blind-to-pnpm +title: Toolchain preflight was blind to pnpm +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["cf130f7"] +decisions: [ADR-0025] +--- + +# Bug: Toolchain preflight was blind to pnpm + +## Summary + +`detect_required_tools()` recognised exactly one JS package manager: it appended `yarn` when it saw +`yarn.lock`, and nothing otherwise. A pnpm repo therefore preflighted clean on an image with no +pnpm — the gate reported `ok: true` and codegen proceeded. + +`discover_commands()` had the matching gap on the other side: `_discover_js_commands` hardcoded +`npm run lint` / `npm run typecheck` / `npm test` regardless of what the repo declared. So even where +pnpm *was* installed, the checks ran through the wrong manager and resolved a different dependency +tree than the repo's own CI. + +Together these produce the precise failure [ADR-0025] exists to prevent: an environment fault scored +as bad code. + +## Reproduction + +1. Point preflight at a repo with `pnpm-lock.yaml` and `"packageManager": "pnpm@10.32.1"`, on a host + without pnpm — `rh-forge/rh-forge-ui` on the CI image is the live case. +2. `python3 scripts/validate_target.py <repo> --preflight` + +## Expected + +Exit 2, `missing_tool: pnpm`, no code generated, epic stays `Ready` to retry once the image has it. + +## Actual + +`ok: true`, exit 0. Codegen runs. Every check then invokes `npm run …` in a pnpm-only workspace and +exits non-zero for reasons the epic did not cause, and the reviewer scores that. + +## Impact + +High — silently converts a missing-tool fault into a low dimension score on a real PR, which is the +same class of failure as the missing `uv` that produced `lint=5.0` ([[task-toolchain-preflight]]). + +## Fix + +`detect_package_manager(repo_path)` is now the single answer to "which manager does this repo want": +the `packageManager` field first (the repo's own statement of record, so a stale lockfile does not +win), then `pnpm-lock.yaml`, then `yarn.lock`, then npm. `detect_required_tools()` gates on its +result for any non-npm manager; `_discover_js_commands()` builds every command through it. + +Covered by `TestDetectPackageManager` and `TestDiscoverCommandsPackageManager` in +`tests/test_toolchain_preflight.py`, plus pnpm cases in `TestDetectRequiredTools`. + +Fixing detection alone leaves the gate correct and the image wrong, so `Dockerfile.ci` gained pnpm +via corepack in the same change — see [[task-per-repo-github-identity]]. + +## Related Tasks + +- [[task-toolchain-preflight]] +- [[task-per-repo-github-identity]] diff --git a/docs/bugs/fixed/bug-preflight-blocked-on-non-tools.md b/docs/bugs/fixed/bug-preflight-blocked-on-non-tools.md new file mode 100644 index 0000000..3a249bf --- /dev/null +++ b/docs/bugs/fixed/bug-preflight-blocked-on-non-tools.md @@ -0,0 +1,40 @@ +--- +id: bug-preflight-blocked-on-non-tools +title: Toolchain preflight blocked on tokens that were not tools +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["6704253"] +decisions: [ADR-0025] +--- + +# Bug: Toolchain preflight blocked on tokens that were not tools + +## Summary + +Preflight extracts required executables from variable-expanded Makefile recipes. Its initial extraction was too broad and treated non-tool tokens as missing executables, so codegen was gated on things that did not need to exist. + +## Reproduction + +1. Point preflight at a repo whose lint/test recipes contain shell builtins, variables, or arguments that look like commands. +2. Run `validate_target.py --preflight`. + +## Expected + +Only genuine executables required by the lint/typecheck/test targets are checked. + +## Actual + +Exit 2 for non-tools, so no code was generated for a healthy repo. + +## Impact + +High + +## Evidence + +Narrowed to inspect only the lint/typecheck/test targets that would actually run, following prerequisites, so an unrelated `docker-build` recipe does not gate codegen. 43 tests in `tests/test_toolchain_preflight.py`. + +## Related Tasks + +- [[task-toolchain-preflight]] diff --git a/docs/bugs/fixed/bug-push-results-overwrote-state.md b/docs/bugs/fixed/bug-push-results-overwrote-state.md new file mode 100644 index 0000000..964f9be --- /dev/null +++ b/docs/bugs/fixed/bug-push-results-overwrote-state.md @@ -0,0 +1,43 @@ +--- +id: bug-push-results-overwrote-state +title: push-results.py overwrote the state machine's metadata +type: bug +status: fixed +repos: [epic-code-gen-pipeline] +commits: ["ddc038e", "3bbcd2d"] +decisions: [ADR-0014] +--- + +# Bug: push-results.py overwrote the state machine's metadata + +## Summary + +`push-results.py` copied the skill's `run-metadata.yaml` over the data repo's copy, destroying the pipeline-owned fields — the pipeline-repo half of RHAIFIRST-374. + +## Reproduction + +1. Complete a codegen run so the skill writes its own `run-metadata.yaml`. +2. Let `after_script` run `push-results.py`. +3. Diff the data repo's state file against what the pipeline wrote during the run. + +## Expected + +The skill's fields are merged in; pipeline-owned fields survive. + +## Actual + +Whole-file overwrite. The pipeline saw no status, treated the epic as Pending, and regenerated work that already had a PR. + +## Impact + +Critical + +## Evidence + +Fixed by `merge_state_file()` with `PIPELINE_OWNED_KEYS`, plus rewriting a legacy `status` holding a `CODEGEN_OUTCOMES` value into `codegen_outcome`. `copy_epic_artifacts` now excludes `run-metadata.yaml` from its `copytree` and merges it separately. Pinned by five tests in `TestStateFileIsMergedNotOverwritten`. + +## Related Tasks + +- [[bug-state-store-clobbered-by-skill]] +- [[M5-state-integrity]] +- The merge logic is **deliberately duplicated** across the repo boundary and must be kept in sync by hand — see [ADR-0014] diff --git a/docs/bugs/fixed/bug-review-bodies-silently-dropped.md b/docs/bugs/fixed/bug-review-bodies-silently-dropped.md new file mode 100644 index 0000000..31e5d5e --- /dev/null +++ b/docs/bugs/fixed/bug-review-bodies-silently-dropped.md @@ -0,0 +1,44 @@ +--- +id: bug-review-bodies-silently-dropped +title: Review response ignores top-level review bodies, silently dropping CHANGES_REQUESTED feedback +type: bug +status: fixed +repos: [epic-code-gen] +jira: RHAIFIRST-375 +commits: ["164d21e"] +decisions: [ADR-0032] +--- + +# Bug: Review response ignores top-level review bodies, silently dropping CHANGES_REQUESTED feedback + +## Summary + +The review-response loop only examined inline review comments. A reviewer who left `CHANGES_REQUESTED` with their objection in the review **body** produced no work at all. + +## Reproduction + +1. Open a PR from the pipeline. +2. Submit a review with state `CHANGES_REQUESTED` and text in the body, but no inline comments. +3. Run the pipeline again. + +## Expected + +The review body is treated as actionable feedback and addressed. + +## Actual + +No actionable comments found. The epic sits still while the PR shows changes requested. + +## Impact + +High + +## Evidence + +`pr_lifecycle.py` gained `format_review_feedback`, `filter_unprocessed_reviews`, and `review_to_comment` to convert a review body into an actionable item, plus `ACTIONABLE_REVIEW_STATES`. + +## Related Tasks + +- [[task-v2-review-response-orchestrator]] +- [[M5-state-integrity]] +- Fixed in the same commit as [[task-rebase-epic-branches-every-cycle]] diff --git a/docs/bugs/fixed/bug-review-response-hid-fix-agent-failure.md b/docs/bugs/fixed/bug-review-response-hid-fix-agent-failure.md new file mode 100644 index 0000000..cc2f8c4 --- /dev/null +++ b/docs/bugs/fixed/bug-review-response-hid-fix-agent-failure.md @@ -0,0 +1,36 @@ +--- +id: bug-review-response-hid-fix-agent-failure +title: The review-response path hid why the fix agent failed +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["d807dca"] +--- + +# Bug: The review-response path hid why the fix agent failed + +## Summary + +When the fix agent failed, the orchestrator reported a generic failure and discarded the actual error, making the cycle undiagnosable from CI logs. + +## Reproduction + +1. Cause the review-fix agent to fail (e.g. timeout or a tool error). +2. Inspect the CI log and the epic's artifacts for the cause. + +## Expected + +The fix agent's real error is surfaced in the log and the failure reason. + +## Actual + +A generic failure message. The underlying error was swallowed. + +## Impact + +Medium + +## Related Tasks + +- [[task-v2-review-response-orchestrator]] +- Same family as [[bug-state-store-clobbered-by-skill]] — a failure reported without the information needed to act on it diff --git a/docs/bugs/fixed/bug-reviewers-cited-patch-line-numbers.md b/docs/bugs/fixed/bug-reviewers-cited-patch-line-numbers.md new file mode 100644 index 0000000..0cd9f79 --- /dev/null +++ b/docs/bugs/fixed/bug-reviewers-cited-patch-line-numbers.md @@ -0,0 +1,40 @@ +--- +id: bug-reviewers-cited-patch-line-numbers +title: Reviewers cited patch line numbers instead of source line numbers +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["34b6e8d"] +--- + +# Bug: Reviewers cited patch line numbers instead of source line numbers + +## Summary + +Reviewer findings referenced positions in the diff rather than in the source files, so the fix agent could not locate what a finding was talking about. + +## Reproduction + +1. Generate a diff and run the reviewers. +2. Read a finding's file:line reference and try to open it in the source. + +## Expected + +Findings cite `file:line` in the source, resolvable by the fix agent. + +## Actual + +Line numbers were offsets within `diff.patch`, pointing at unrelated or nonexistent source lines. + +## Impact + +High + +## Evidence + +Made findings unactionable, so iterations were spent without closing the finding — visible as flat stretches in early score progressions. + +## Related Tasks + +- [[task-deterministic-scoring]] +- [[task-triage-memory-and-oscillation]] diff --git a/docs/bugs/fixed/bug-self-authored-validation-json-scored.md b/docs/bugs/fixed/bug-self-authored-validation-json-scored.md new file mode 100644 index 0000000..2321643 --- /dev/null +++ b/docs/bugs/fixed/bug-self-authored-validation-json-scored.md @@ -0,0 +1,43 @@ +--- +id: bug-self-authored-validation-json-scored +title: A hand-written validation.json scored the lint dimension 8.0 while the linter was failing +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["8373536"] +decisions: [ADR-0024] +--- + +# Bug: A hand-written validation.json scored the lint dimension 8.0 while the linter was failing + +## Summary + +The lint dimension is scored from `validation.json`, which is supposed to be produced by `validate_target.py` actually running the repo's checks. Nothing verified provenance, so an agent-authored document was accepted as evidence. + +## Reproduction + +1. Have the skill write `validation.json` by hand instead of using `validate_target.py --out`. +2. Give it a plausible but non-conforming shape, e.g. `{"tests_total": 35, "success": true}`. +3. Run scoring. + +## Expected + +A document that is not genuine tool output is rejected and the verdict fails. + +## Actual + +Scored `lint=8.0` while Prettier was failing. + +## Impact + +Critical + +## Evidence + +Fix: `VALIDATION_DOCUMENT_KEYS = ("all_passed", "checks")` and `validation_document_status()` returning `ok`/`missing`/`foreign`/`unreadable`. `score_reviews.py` forces `verdict: fail` on `foreign` or `unreadable`; `missing` stays advisory, since absence has legitimate causes and fabrication does not. A real instance is still visible in the data repo at `RHAISTRAT-1961/RHAI-69/v1/scores.json`. + +## Related Tasks + +- [[task-deterministic-scoring]] +- **The guard is not sufficient** — in RHAIFIRST-391 it fired, was recorded, and was ignored: [[bug-review-gate-is-advisory]] +- [[bug-two-ways-to-produce-validation-json]] diff --git a/docs/bugs/fixed/bug-state-store-clobbered-by-skill.md b/docs/bugs/fixed/bug-state-store-clobbered-by-skill.md new file mode 100644 index 0000000..ffb4371 --- /dev/null +++ b/docs/bugs/fixed/bug-state-store-clobbered-by-skill.md @@ -0,0 +1,56 @@ +--- +id: bug-state-store-clobbered-by-skill +title: Pipeline state store clobbered by the skill; invalid 'completed' status deadlocks strategies +type: bug +status: fixed +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-374 +commits: ["e03689c", "3bbcd2d"] +decisions: [ADR-0013, ADR-0014, ADR-0015] +--- + +# Bug: Pipeline state store clobbered by the skill; invalid 'completed' status deadlocks strategies + +## Summary + +`run-metadata.yaml` was written by two producers with incompatible schemas, and the second write destroyed the first. The skill set `status: completed`, which is not a member of `CI_STATES`, so the CI dispatcher fell through every branch to `else` and returned `SKIPPED … "Unknown state: completed"` — while exiting 0. + +## Reproduction + +1. Run a strategy through to a completed codegen for one epic. +2. Let `/epic-codegen` write its run summary over `run-metadata.yaml`. +3. Re-run the pipeline for the same strategy. +4. Observe the epic reported as skipped, and any epic depending on it stuck at Blocked. + +## Expected + +The epic advances to `ReviewPending` or `PRCreated`, and its dependents unblock when it reaches `Done`. + +## Actual + +The epic is skipped on **every** subsequent run, forever. Dependents stay Blocked indefinitely. The job exits 0 and reports success — a silent deadlock. + +## Impact + +Critical + +## Observed incident + +RHAISTRAT-2162, 2026-07-28/29. RHAI-74 (PR ederign/kale#11, score 9.4, verdict pass) and RHAI-76 (#12, 8.6) were both left at `status: completed`. Subsequent runs reported `0 processed, 2 skipped, 3 blocked` in ~118s. RHAI-75 was blocked by 74, RHAI-78 by 75, and RHAI-77 by all four, so the entire strategy deadlocked with two merge-quality PRs stranded and no error surfaced anywhere. + +## Evidence + +**Three competing vocabularies for one field:** + +- `run_pipeline.py` `CI_STATES`: Pending, Ready, Generating, ReviewPending, PRCreated, PRChangesRequested, Done, Blocked, Failed +- `SKILL.md`: `completed|exhausted|failed|error` (lowercase) +- `artifact_utils.py` codegen-run enum: Running, Completed, Failed, Exhausted (capitalised) + +**Fields silently dropped** by the skill's whole-file write — from RHAI-74: `current_version`, `max_iterations`, `pr_state`, `timestamps`, `scores`. RHAI-76 also lost `strategy_key` and `target_branch`, and used a *third* field layout (`scores_by_dimension`, `pr_note`, `started_at`/`completed_at`) — the skill's output was not self-consistent between two epics in the same run. + +## Related Tasks + +- [[task-ci-state-machine-and-convergence]] +- [[M5-state-integrity]] +- Fix: one owner per status field ([ADR-0013]), merge-never-write ([ADR-0014]), normalize-on-read and fail loudly ([ADR-0015]) +- Residue: `RHAISTRAT-2352/RHAI-264` still carries the corrupt value, rescued on read rather than migrated diff --git a/docs/bugs/fixed/bug-tee-buffering-hid-ci-output.md b/docs/bugs/fixed/bug-tee-buffering-hid-ci-output.md new file mode 100644 index 0000000..dc06875 --- /dev/null +++ b/docs/bugs/fixed/bug-tee-buffering-hid-ci-output.md @@ -0,0 +1,39 @@ +--- +id: bug-tee-buffering-hid-ci-output +title: tee buffering meant CI showed no output until the job ended +type: bug +status: fixed +repos: [epic-code-gen, epic-code-gen-pipeline] +commits: ["74a1fc9", "268769b"] +--- + +# Bug: tee buffering meant CI showed no output until the job ended + +## Summary + +Codegen output was piped through `tee` to produce a log file. `tee` buffers, so a 6-hour job appeared to produce nothing until it finished. + +## Reproduction + +1. Run codegen in CI with output piped through `tee`. +2. Watch the job log. + +## Expected + +Output streams live so progress is visible. + +## Actual + +Nothing until the job ended — a hung run was indistinguishable from a slow one. + +## Impact + +Medium + +## Evidence + +Fixed by having `stream-claude.py` write the log file directly rather than relying on a pipe, plus forced foreground execution (`268769b`). This is also why the first attempt at [ADR-0012] was reverted — removing the outer Claude wrapper broke streaming before this was solved properly. + +## Related Tasks + +- [[task-ci-observability]] diff --git a/docs/bugs/fixed/bug-uv-verification-segfaults-under-qemu.md b/docs/bugs/fixed/bug-uv-verification-segfaults-under-qemu.md new file mode 100644 index 0000000..d8a8b9e --- /dev/null +++ b/docs/bugs/fixed/bug-uv-verification-segfaults-under-qemu.md @@ -0,0 +1,40 @@ +--- +id: bug-uv-verification-segfaults-under-qemu +title: Verifying the uv install segfaulted when cross-building the image from arm64 +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["e7c9dac"] +decisions: [ADR-0011] +--- + +# Bug: Verifying the uv install segfaulted when cross-building the image from arm64 + +## Summary + +`Dockerfile.ci` verified the `uv` install by running `uv --version`. Executing the freshly installed amd64 binary segfaults under qemu emulation when cross-building from an arm64 host, failing the multi-arch build. + +## Reproduction + +1. Build the CI image with buildx for `linux/amd64` from an arm64 machine. +2. Observe the verification step segfault. + +## Expected + +The build verifies `uv` is installed and continues. + +## Actual + +Segfault under qemu; multi-arch build fails. + +## Impact + +Medium + +## Evidence + +Fixed by verifying presence with `test -x` rather than executing the binary. The Dockerfile comment records the reason so nobody 'improves' it back to `uv --version`. + +## Related Tasks + +- [[task-ci-image-and-build-infrastructure]] diff --git a/docs/bugs/fixed/bug-wait-blocked-on-unscored-reviewers.md b/docs/bugs/fixed/bug-wait-blocked-on-unscored-reviewers.md new file mode 100644 index 0000000..a49021e --- /dev/null +++ b/docs/bugs/fixed/bug-wait-blocked-on-unscored-reviewers.md @@ -0,0 +1,41 @@ +--- +id: bug-wait-blocked-on-unscored-reviewers +title: review_cycle.py wait blocked forever on unscored reviewers +type: bug +status: fixed +repos: [epic-code-gen] +commits: ["4f1cc62"] +decisions: [ADR-0026] +--- + +# Bug: review_cycle.py wait blocked forever on unscored reviewers + +## Summary + +`wait` waited for all six review files. If an unscored verifier never produced one, the loop blocked until the job timed out. + +## Reproduction + +1. Dispatch the review loop with one verifier failing to write its file. +2. Run `wait`. + +## Expected + +`wait` returns once the loop can proceed, and signals what is missing. + +## Actual + +Blocked indefinitely; the 6-hour job timed out with no review completed. + +## Impact + +High + +## Evidence + +Fixed by returning once the **scored** dimensions land, plus exit code 2 in the dispatch loop. That fix introduced the inverse problem, which is still open: [[bug-wait-returns-before-unscored-reviewers-finish]]. + +## Related Tasks + +- [[task-review-cycle-extraction]] +- [[task-unscored-verifiers]] diff --git a/docs/bugs/fixed/bug-yaml-crash-in-push-results.md b/docs/bugs/fixed/bug-yaml-crash-in-push-results.md new file mode 100644 index 0000000..dfd088b --- /dev/null +++ b/docs/bugs/fixed/bug-yaml-crash-in-push-results.md @@ -0,0 +1,40 @@ +--- +id: bug-yaml-crash-in-push-results +title: push-results.py crashed on a multi-document run-metadata.yaml +type: bug +status: fixed +repos: [epic-code-gen-pipeline] +commits: ["5c4ea17"] +--- + +# Bug: push-results.py crashed on a multi-document run-metadata.yaml + +## Summary + +The skill sometimes appended a second YAML document to `run-metadata.yaml`. `yaml.safe_load` raises on multi-document input, so the push crashed and the run's results were never persisted. + +## Reproduction + +1. Produce a `run-metadata.yaml` containing two `---` documents. +2. Run `push-results.py`. + +## Expected + +The file is read without crashing. + +## Actual + +Exception; artifacts not pushed, so a successful codegen run left no durable record. + +## Impact + +High + +## Evidence + +Fixed with `_safe_load_yaml()` using `yaml.safe_load_all()` and taking the first document. The same commit added recursive artifact copying for crash recovery. + +## Related Tasks + +- [[task-data-repo-artifact-structure]] +- [[bug-push-results-overwrote-state]] diff --git a/docs/bugs/open/bug-artifact-presence-is-weak-liveness-proxy.md b/docs/bugs/open/bug-artifact-presence-is-weak-liveness-proxy.md new file mode 100644 index 0000000..619280a --- /dev/null +++ b/docs/bugs/open/bug-artifact-presence-is-weak-liveness-proxy.md @@ -0,0 +1,39 @@ +--- +id: bug-artifact-presence-is-weak-liveness-proxy +title: A non-zero exit with any v*/diff.patch present is treated as success +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: A non-zero exit with any v*/diff.patch present is treated as success + +## Summary + +`invoke_codegen` treats the presence of `v*/diff.patch` as evidence that codegen succeeded, even when the session exited non-zero. + +## Reproduction + +1. Cause a codegen session to write `v1/diff.patch` and then crash. +2. Observe the transition. + +## Expected + +A crashed run is distinguished from a completed one — e.g. by requiring the artifacts a complete run produces (`scores.json`, a validated `validation.json`). + +## Actual + +Artifact presence alone is accepted (`run_pipeline.py:614-620`), so a partial run can advance the epic to `ReviewPending` with incomplete artifacts. + +## Impact + +Medium + +## Evidence + +This was the deliberate fix for the opposite bug — a completed run being discarded because Claude exited oddly ([[bug-nonzero-claude-exit-read-as-failure]]). The trade was reasonable at the time; the residual risk is that it feeds under-populated versions into the review phase, which is one of the conditions RHAIFIRST-391 exploited. + +## Related Tasks + +- [[task-idempotent-pipeline]] +- [[bug-review-gate-is-advisory]] diff --git a/docs/bugs/open/bug-baseline-check-failures-scored-as-epic.md b/docs/bugs/open/bug-baseline-check-failures-scored-as-epic.md new file mode 100644 index 0000000..8b10bce --- /dev/null +++ b/docs/bugs/open/bug-baseline-check-failures-scored-as-epic.md @@ -0,0 +1,93 @@ +--- +id: bug-baseline-check-failures-scored-as-epic +title: Pre-existing target-repo check failures are scored as bad code, and mask real findings behind them +type: bug +status: open +repos: [epic-code-gen] +jira: RHAIFIRST-392 +decisions: [ADR-0023, ADR-0025] +--- + +# Bug: Pre-existing target-repo check failures are scored as bad code, and mask real findings behind them + +## Summary + +`validate_target.py` runs a repo's aggregate check targets and the reviewers score whatever comes back, with no notion of whether the same check already failed before the epic touched anything. A repo whose `make lint` is red on `main` therefore fails every epic generated against it, for a fault no amount of code generation can fix. + +## Reproduction + +1. Pick a target repo whose `make lint` fails on a pristine checkout of `main`. +2. Generate any epic against it. +3. Observe the lint dimension carry a Critical for a failure the diff did not cause. + +## Expected + +A check that fails identically on the base commit is attributed to the repo, not the epic — and does not prevent the later sub-targets from running. + +## Actual + +The baseline failure is scored as the epic's defect, caps the lint dimension at 5 ([ADR-0023]), and because GNU make stops at the first failing prerequisite, the genuine findings that would have run afterwards are never surfaced at all. + +## Impact + +High + +## Observed incident + +RHAI-68 / RHAISTRAT-1961 against `opendatahub-io/pipelines-components`, 2026-07-30, job 15632649803. PR #194. + +`make lint` exits 2 because `ruff format --check` cannot parse three notebook templates: + +``` +error: Failed to parse .../notebook_templates/classification_notebook.ipynb:23:3:14 +error: Failed to parse .../notebook_templates/regression_notebook.ipynb:20:3:14 +error: Failed to parse .../notebook_templates/timeseries_notebook.ipynb:15:1:1 +``` + +None of the three is touched by the diff (12 files, all under `components/data_processing/automl/`, `components/training/automl/`, and `pipelines/training/automl/`). + +## Evidence + +**Reproduced on a pristine checkout** of base `b3c46d6` with the repo's pinned `ruff==0.15.2` — identical three errors, `375 files already formatted`, exit 2. The breakage is upstream's, not the epic's. + +**It cost the epic a passing score.** `v4/scores.json`: architecture 9.5, tests 7.5, **lint 4.5**, intent 9.5 → weighted average **7.9**, verdict **fail**. The lint dimension carries exactly one Critical, and that Critical is the baseline failure. + +This is the same family as the `unrunnable` vs `failed` distinction ([ADR-0025]) — but that guard only covers checks that could not *execute*. Here the check executes fine and fails for reasons the epic did not cause, so nothing catches it. + +## Update 2026-07-31 — cross-repo, and it bites two code paths + +A second instance on `opendatahub-io/odh-dashboard` (recorded as a comment on RHAIFIRST-392) changes the +shape of this bug in two ways. + +**It is not one repo's problem.** Two of the two target repos tried so far fail their own aggregate check +on `main`, by *different* mechanisms: + +| Repo | Mechanism | +|---|---| +| `pipelines-components` | `make lint` → `ruff format --check` cannot parse three notebook templates; GNU make fail-fast then hides later sub-targets | +| `odh-dashboard` | `eslint --max-warnings 0` turns the repo's pre-existing warning debt into a hard failure — no make fail-fast involved | + +So a baseline comparison cannot rely on make semantics; the general case is "this check was already red", +whatever the tool. + +**It bites in two code paths, not one.** The odh-dashboard instance failed through `run_validation` in +`review_response.py`'s retry loop — not through the reviewer scoring path that RHAI-68 hit. The baseline +comparison is therefore needed in **both** places, or a fix to scoring alone will leave the +review-response cycle still failing on inherited breakage. + +**A workaround already happened inside the pipeline, and needs a policy answer.** RHAI-68's fix agent +resolved the `pipelines-components` instance by committing `7f62a5148e`, excluding the notebook templates +from ruff. That unblocked the epic — by **rewriting the target repo's lint configuration**. It is a +defensible change a human might well make, but the pipeline made it autonomously to get past a failure the +epic did not cause, and it is outside the diff scope [ADR-0032] otherwise enforces. Whether an agent may +edit a target repo's lint config to unblock itself should be an explicit decision, not an emergent one. + +## Related Tasks + +- [[task-toolchain-preflight]] +- [[M6-review-gate-hardening]] +- Likely fix: capture a baseline validation run at `BASE_SHA` and diff findings against it — in both + the scoring path and `review_response.run_validation` +- [[bug-failed-cycle-still-marks-comments-processed]] (RHAIFIRST-393) — `Related` in Jira; a cycle + failing on inherited breakage is how that bug gets triggered +- Needs a policy decision: may an agent edit a target repo's lint config to unblock itself? diff --git a/docs/bugs/open/bug-codegen-review-schema-is-dead.md b/docs/bugs/open/bug-codegen-review-schema-is-dead.md new file mode 100644 index 0000000..562ae73 --- /dev/null +++ b/docs/bugs/open/bug-codegen-review-schema-is-dead.md @@ -0,0 +1,35 @@ +--- +id: bug-codegen-review-schema-is-dead +title: The codegen-review schema and rebuild-index subsystem are dead +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0002] +--- + +# Bug: The codegen-review schema and rebuild-index subsystem are dead + +## Summary + +`SCHEMAS["codegen-review"]`, `find_codegen_review`, `rebuild_index`, `frontmatter.py rebuild-index`, and `artifacts/epics.md` form a subsystem nothing uses. + +## Reproduction + +1. Search the repo and the data repo for `artifacts/codegen-reviews/` — it appears only inside `artifact_utils.py`. + +## Expected + +Unused schemas are removed, or the subsystem is wired up. + +## Actual + +Dead, and misleading: its `scores` keys are `lint, typecheck, tests, intent_coverage, architecture` — a vocabulary that predates the live dimensions (`architecture, tests, lint, intent`). Anyone reading it would infer the wrong dimension set. + +## Impact + +Low + +## Related Tasks + +- [[task-frontmatter-schema-module]] +- [[task-delete-dead-code]] diff --git a/docs/bugs/open/bug-failed-cycle-still-marks-comments-processed.md b/docs/bugs/open/bug-failed-cycle-still-marks-comments-processed.md new file mode 100644 index 0000000..e85982d --- /dev/null +++ b/docs/bugs/open/bug-failed-cycle-still-marks-comments-processed.md @@ -0,0 +1,93 @@ +--- +id: bug-failed-cycle-still-marks-comments-processed +title: "A failed review-response cycle still replies to every comment and marks it processed, permanently dropping the feedback" +type: bug +status: open +repos: [epic-code-gen] +jira: RHAIFIRST-393 +decisions: [ADR-0032] +--- + +# Bug: A failed review-response cycle still replies to every comment and marks it processed + +## Summary + +`run_review_response` posts replies and records them **unconditionally**, outside the `if not errors:` +guard that protects the push. Because the processed-comment set is derived from every reply ever recorded, +with no filter on version or outcome, a cycle that failed still consumes its comments. The next cycle finds +zero unprocessed comments and skips the epic — the feedback is gone with no retry path. + +## Reproduction + +1. Open a PR from the pipeline and have a reviewer leave inline comments. +2. Cause the review-response cycle to fail after triage — e.g. the fix agent errors, or the push fails. +3. Observe replies posted to all comments and appended to `pr-replies.json`. +4. Run the pipeline again. `filter_unprocessed_comments` returns nothing; the epic is skipped. + +## Expected + +A cycle that did not land its fixes leaves its comments unprocessed, so a later cycle retries them. + +## Actual + +Comments are marked processed regardless of outcome. The epic is permanently skipped for that feedback. + +## Impact + +High + +## Observed incident + +RHAI-69, 2026-07-31. All 8 review comments are now in the processed set after a cycle that did not land. +A future cycle finds zero unprocessed and skips the epic entirely. + +## Evidence + +**Root cause is placement, verified in source.** `# 9. Push to fork` is guarded +(`review_response.py:492`): + +```python + # 9. Push to fork + if not errors: +``` + +but `# 10. Post replies` sits at the outer indentation level (`:508`), outside both that guard and its +`else`, so it runs on every path: + +```python + # 10. Post replies + reply_sha = commit_sha or "no-change" + if errors: + reply_sha = "not-pushed" +``` + +**The processed set has no notion of success.** `save_pr_replies` (`review_response.py:222`) appends +cumulatively — `existing["replies"].extend(replies)` — and `load_processed_comment_ids` +(`pr_lifecycle.py:349`) returns: + +```python +return {r["comment_id"] for r in data.get("replies", [])} +``` + +No filter on version, and none on outcome. + +**One nuance, in the code's favour:** the reply text is *not* dishonest — `reply_sha` is set to +`"not-pushed"` when there are errors, so a human reading the PR can tell the cycle failed. The defect is +that the **pipeline** cannot tell, because the comment id is in the processed set either way. So the +feedback is dropped from the machine's perspective while remaining visible to a person, which is why it +went unnoticed. + +**The fix is cheap because the data is already there.** `save_pr_replies` already stamps each reply with +`version`; it just needs an outcome too, and `load_processed_comment_ids` needs to filter on it. Moving the +reply step inside the guard would also work, but replying only on success loses the useful "we saw this and +failed" signal to reviewers — recording the outcome and filtering on it keeps both. + +## Related Tasks + +- [[task-v2-review-response-orchestrator]] — the cycle this defect lives in +- [[bug-review-gate-is-advisory]] (RHAIFIRST-391) — `Related` in Jira; same family of silent success +- [[bug-baseline-check-failures-scored-as-epic]] (RHAIFIRST-392) — `Related` in Jira; the failure mode that + triggered this cycle's failure +- [[M6-review-gate-hardening]] +- [ADR-0032] records "reply to every comment" as a deliberate rule — this bug is that rule applied one + level too broadly, so the ADR's Negative consequences should gain a line once fixed diff --git a/docs/bugs/open/bug-gitignored-files-are-tracked.md b/docs/bugs/open/bug-gitignored-files-are-tracked.md new file mode 100644 index 0000000..06f94ab --- /dev/null +++ b/docs/bugs/open/bug-gitignored-files-are-tracked.md @@ -0,0 +1,38 @@ +--- +id: bug-gitignored-files-are-tracked +title: Several paths are both gitignored and tracked +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: Several paths are both gitignored and tracked + +## Summary + +`package-lock.json`, `epic-reports/`, and `pipeline-runs/` are listed in `.gitignore` while also being tracked or present with content, so the ignore rules do not mean what they say. + +## Reproduction + +1. `git check-ignore -v package-lock.json` and `git ls-files package-lock.json`. + +## Expected + +A path is either ignored or tracked. + +## Actual + +Both. `package-lock.json` is gitignored and tracked, with its own commit history (`e54900b`). `epic-reports/` and `pipeline-runs/` are gitignored but present on disk with content. + +## Impact + +Low + +## Evidence + +`package-lock.json` arguably *should* be tracked (it pins Playwright for `parse_prototype.js`), which means the `.gitignore` entry is the error rather than the tracking. `epic-reports/` was gitignored deliberately because it holds sensitive HTML (`876a0f3`), so that one is correct and merely confusing. + +## Related Tasks + +- [[bug-node-modules-committed]] +- [[M7-engineering-process]] diff --git a/docs/bugs/open/bug-iteration-reviewer-undefined-base-sha.md b/docs/bugs/open/bug-iteration-reviewer-undefined-base-sha.md new file mode 100644 index 0000000..df104d4 --- /dev/null +++ b/docs/bugs/open/bug-iteration-reviewer-undefined-base-sha.md @@ -0,0 +1,41 @@ +--- +id: bug-iteration-reviewer-undefined-base-sha +title: iteration-reviewer.md references ${BASE_SHA}, which nothing emits +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0019] +--- + +# Bug: iteration-reviewer.md references ${BASE_SHA}, which nothing emits + +## Summary + +The triage agent's artifact-save snippet uses `${BASE_SHA}`, but `BASE_SHA` is not in its declared inputs and `review_cycle.py triage-prompt` never emits it. + +## Reproduction + +1. Read `.claude/agents/iteration-reviewer.md`'s `## Inputs (from prompt vars)` list. +2. Search it for `${BASE_SHA}`. +3. Search `review_cycle.py cmd_triage_prompt` for `BASE_SHA`. + +## Expected + +Every variable a prompt template references is declared and supplied. + +## Actual + +An undefined variable in a prompt template. The agent either substitutes nothing or invents a value, and the resulting diff command is wrong or empty. + +## Impact + +Medium + +## Evidence + +A direct consequence of the [ADR-0019] trade-off: isolating each agent means every input must be marshalled explicitly through prompt variables, and nothing validates that the set a template uses matches the set the dispatcher provides. + +## Related Tasks + +- [[task-triage-memory-and-oscillation]] +- Fix: emit `BASE_SHA` from `triage-prompt` and declare it — plus a check that template variables are all supplied diff --git a/docs/bugs/open/bug-make-test-fails-on-empty-integration-target.md b/docs/bugs/open/bug-make-test-fails-on-empty-integration-target.md new file mode 100644 index 0000000..f3d090f --- /dev/null +++ b/docs/bugs/open/bug-make-test-fails-on-empty-integration-target.md @@ -0,0 +1,79 @@ +--- +id: bug-make-test-fails-on-empty-integration-target +title: "`make test` always fails, because test-integration collects zero tests" +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: `make test` always fails, because test-integration collects zero tests + +## Summary + +`make test` depends on `test-integration`, which runs `pytest -m "integration"`. No test in the suite +carries that marker, so pytest collects nothing and exits **5** (`NO_TESTS_COLLECTED`). Make treats that +as a failed recipe, so `make test` **cannot succeed** — and both `CLAUDE.md` and `AGENTS.md` instruct +contributors to run it before pushing. + +## Reproduction + +1. `git checkout main` +2. `make test` +3. Observe `test-unit` pass, then: + +``` +703 deselected in 0.07s +make: *** [test-integration] Error 5 +``` + +## Expected + +`make test` runs the full suite and exits 0 when everything passes. A target with no matching tests is a +no-op, not a failure. + +## Actual + +Exits non-zero every time, on every branch, regardless of test results. + +## Impact + +Medium + +## Observed incident + +Found on 2026-07-31 while verifying this ledger — the one command the handbook says to run before +pushing turned out never to have worked. + +## Evidence + +Confirmed pre-existing, not introduced by the ledger work. Reproduced on a **pristine `main` worktree**: + +``` +$ git worktree add /tmp/ecg-main main && cd /tmp/ecg-main && make test-integration +632 deselected in 0.48s +make: *** [test-integration] Error 5 +``` + +632 deselected on `main` vs 703 on the ledger branch — consistent with 71 newly added tests and with the +same underlying failure. + +Direct cause, verified: + +``` +$ uv run pytest tests/ -m integration -q ; echo $? +703 deselected in 0.07s +5 +``` + +`grep -rc 'pytest.mark.integration' tests/` returns **0** on both `main` and HEAD. The marker is +registered in `pyproject.toml` and used by nothing, so the target has never had anything to run. + +Because the failure is at the *end* of `make test` and the unit output scrolls past, it reads as "the +suite ran" — which is likely why it went unnoticed. `make test-unit` is green: **703 passed**. + +## Related Tasks + +- [[task-fix-make-test-target]] — the fix +- [[bug-repo-does-not-meet-own-standards]] — the umbrella record; this is one concrete instance +- Interim: `AGENTS.md` and `CLAUDE.md` now tell contributors to run `make test-unit`, with this bug + cited, rather than carrying an instruction that cannot be satisfied diff --git a/docs/bugs/open/bug-max-iterations-default-disagrees.md b/docs/bugs/open/bug-max-iterations-default-disagrees.md new file mode 100644 index 0000000..766a637 --- /dev/null +++ b/docs/bugs/open/bug-max-iterations-default-disagrees.md @@ -0,0 +1,42 @@ +--- +id: bug-max-iterations-default-disagrees +title: max_iterations defaults to 3 in one code path and 10 in five others +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0033] +--- + +# Bug: max_iterations defaults to 3 in one code path and 10 in five others + +## Summary + +An epic whose state file predates the `max_iterations` field gets a budget of 3 in the scoring path and 10 everywhere else. + +## Reproduction + +1. Create an epic state file without a `max_iterations` key. +2. Drive it to `ReviewPending`. +3. Observe `_ci_handle_review_pending` treating the budget as 3 while the PR path treats it as 10. + +## Expected + +One default, defined once. + +## Actual + +`run_pipeline.py:1375` — `max_iter = state.get("max_iterations", 3)`. Five other sites use 10: `_init_epic_state` (`:1201`), `_ci_handle_pr_changes` (`:1502`), `review_cycle.py:356`, `SKILL.md:104`, `artifact_utils.py:274`. `README.md` still documents 3. + +## Impact + +Medium + +## Evidence + +The value decides when an epic is declared exhausted and whether a near-miss PR opens, so the disagreement is not cosmetic: the same epic can be 'exhausted' in one handler and have budget remaining in another. + +## Related Tasks + +- [[task-triage-memory-and-oscillation]] +- [[bug-readme-is-stale]] +- Fix: a single module-level constant in `artifact_utils.py`, referenced everywhere diff --git a/docs/bugs/open/bug-model-pinning-contradiction.md b/docs/bugs/open/bug-model-pinning-contradiction.md new file mode 100644 index 0000000..3c3881d --- /dev/null +++ b/docs/bugs/open/bug-model-pinning-contradiction.md @@ -0,0 +1,40 @@ +--- +id: bug-model-pinning-contradiction +title: run-claude.sh pins a model while three docs say agents inherit the session model +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0029] +--- + +# Bug: run-claude.sh pins a model while three docs say agents inherit the session model + +## Summary + +`ci-scripts/run-claude.sh` defaults to `--model ${CLAUDE_MODEL:-claude-opus-4-6}`, pinning a specific model, while `README.md`, `CLAUDE.md`, and `SKILL.md` all state that all agents inherit the session model with no overrides. + +## Reproduction + +1. Read `ci-scripts/run-claude.sh:20`. +2. Read the model-selection statements in the three docs. + +## Expected + +One statement of how the model is chosen. + +## Actual + +Both claims are in the tree and they cannot both be true. In practice CI runs whatever `run-claude.sh` pins, so the docs describe an intent the code does not implement. + +## Impact + +Medium + +## Evidence + +`527fc9d` removed all sonnet references specifically to establish the inherit-the-session rule, and `rubrics/` still says `Model: sonnet` ([[bug-rubrics-directory-is-dead-and-wrong]]) — three different answers in one repo. + +## Related Tasks + +- [[M7-engineering-process]] +- Fix: decide whether CI pins deliberately (and say so in [ADR-0029]) or genuinely inherits diff --git a/docs/bugs/open/bug-multi-strategy-runs-lose-run-record.md b/docs/bugs/open/bug-multi-strategy-runs-lose-run-record.md new file mode 100644 index 0000000..74707e9 --- /dev/null +++ b/docs/bugs/open/bug-multi-strategy-runs-lose-run-record.md @@ -0,0 +1,61 @@ +--- +id: bug-multi-strategy-runs-lose-run-record +title: Multi-strategy runs lose the run record for all but the last strategy +type: bug +status: open +repos: [epic-code-gen-pipeline] +decisions: [ADR-0006, ADR-0010] +--- + +# Bug: Multi-strategy runs lose the run record for all but the last strategy + +## Summary + +`pipeline-post.sh` builds one `--strategy-key <k>` flag per key. Python's argparse abbreviation-matches that to the `nargs="+"` `--strategy-keys` option, and repeated occurrences **overwrite** rather than append — so `push_results` only ever loops over the last key. + +## Reproduction + +1. Trigger `codegen-run` with `STRATEGY_KEYS="RHAISTRAT-A RHAISTRAT-B"`. +2. Let the job complete so `after_script` runs `pipeline-post.sh`. +3. Inspect the data repo: only `RHAISTRAT-B` has a refreshed `strategy-summary.json`, a new `run-log.jsonl` entry, and an OTEL file. + +## Expected + +Every strategy processed in the run gets its state merged, its summary regenerated, its run log appended, and its telemetry persisted. + +## Actual + +Only the last key does. Verified empirically: + +```python +p.add_argument("--strategy-keys", nargs="+", required=True) +p.parse_args(["--strategy-key","A","--strategy-key","B"]).strategy_keys +# → ['B'] +``` + +## Impact + +High + +## Evidence + +`ci-scripts/pipeline-post.sh:38-41` builds the flags: + +```bash +IFS=' ' read -ra keys <<< "$strategy_keys" +key_args="" +for key in "${keys[@]}"; do + key_args="$key_args --strategy-key $key" +``` + +`ci-scripts/push-results.py:444` declares `--strategy-keys` with `nargs="+"`. + +**Blast radius is narrower than it first appears, and worth stating precisely.** Codegen artifacts still land, because `run_pipeline.py` writes state live during the run and `commit_and_push` does `git add -A`, which sweeps up everything on disk. What is lost for every strategy except the last is the work inside the per-key loop: the `merge_state_file()` call (so the skill's own fields never reach the data repo), the `strategy-summary.json` regeneration, the `run-log.jsonl` append, and the OTEL file copy. + +Net effect: the durable run record and the dashboard feed silently lose all but one strategy per pass. The State Log and Cost views are missing entries nobody has noticed, because the job exits 0. + +## Related Tasks + +- [[task-gitlab-ci-and-ci-scripts]] +- Fix is one character — pass `--strategy-keys` once with all keys — plus a test +- Illustrates the [ADR-0010] trade-off: a shell script assembling flags for an argparse it cannot see, in the repo with almost no test coverage diff --git a/docs/bugs/open/bug-readme-is-stale.md b/docs/bugs/open/bug-readme-is-stale.md new file mode 100644 index 0000000..5cbede0 --- /dev/null +++ b/docs/bugs/open/bug-readme-is-stale.md @@ -0,0 +1,39 @@ +--- +id: bug-readme-is-stale +title: README.md is substantially stale +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: README.md is substantially stale + +## Summary + +The repo's front door misstates test count, iteration budget, project layout, and phase status, and documented a script that did not exist. + +## Reproduction + +1. Read `README.md` against the current tree. + +## Expected + +The README describes the system as it is. + +## Actual + +Claims **186 unit tests** (actual: 703). Claims **up to 3 iterations max** (actual: 10). Shows reviewer agents at `agents/` in the project root with 4 files (actual: `.claude/agents/`, 13 files). Phase status ends at 'Phase 3b in progress / Phase 4 Next', while PR lifecycle, review response, UX prototypes, and the CI state machine have all shipped. Omits the interaction verifier and the whole UX prototype subsystem. + +## Impact + +Medium + +## Evidence + +`CLAUDE.md` additionally documented `bash scripts/fetch-architecture-context.sh` with two usage examples for a script that does not exist, and a `.context/architecture-context/` directory with no consumer. That section was removed on 2026-07-31 as part of [[M7-engineering-process]]; the README itself is still stale. + +## Related Tasks + +- [[M7-engineering-process]] +- [[bug-max-iterations-default-disagrees]] +- Fix: rewrite the README against `docs/architecture/`, and let it link out rather than restating diff --git a/docs/bugs/open/bug-repo-does-not-meet-own-standards.md b/docs/bugs/open/bug-repo-does-not-meet-own-standards.md new file mode 100644 index 0000000..49883f3 --- /dev/null +++ b/docs/bugs/open/bug-repo-does-not-meet-own-standards.md @@ -0,0 +1,45 @@ +--- +id: bug-repo-does-not-meet-own-standards +title: This repo does not meet the standards it enforces on target repos +type: bug +status: open +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Bug: This repo does not meet the standards it enforces on target repos + +## Summary + +The product exists to enforce lint, tests, and conventions on other repositories. Applied to itself it would score poorly on its own readiness assessment. + +## Reproduction + +1. Run `python3 scripts/repo_readiness.py .` against this repo. +2. Compare the result to the threshold of 8 the pipeline requires of targets. + +## Expected + +The repo meets the bar it sets, or the gap is deliberate and recorded. + +## Actual + +As of 2026-07-31: no CI at all (added by [[M7-engineering-process]]); **no linter or type checker** for 10.5k lines of Python; `make test` **always fails** because `test-integration` collects zero tests and pytest exits 5 ([[bug-make-test-fails-on-empty-integration-target]]); `jira_utils.py` (1,055 lines, including the whole Markdown↔ADF converter), `frontmatter.py`, `state.py`, and `parse_prototype.js` (699 lines) have **no tests**; six independent YAML parsers, three git wrappers, two HTTP clients, two slug extractors. + +## Impact + +High + +## Evidence + +This is a credibility problem as much as a quality one: the readiness score gates which repos are allowed to receive generated code, and the reviewers score lint conformance. `Dockerfile.ci` installs `markdownlint-cli` for target repos while nothing lints this repo's own Python. + +Filed as one umbrella bug because the individual items are tracked as tasks in `docs/tasks/pending/` — this file exists so the aggregate is visible rather than spread across a dozen entries. + +## Related Tasks + +- [[task-fix-make-test-target]] +- [[task-add-lint-and-typecheck]] +- [[task-add-tests-for-untested-modules]] +- [[task-consolidate-yaml-parsers]] +- [[task-delete-dead-code]] +- [[M7-engineering-process]] diff --git a/docs/bugs/open/bug-review-gate-is-advisory.md b/docs/bugs/open/bug-review-gate-is-advisory.md new file mode 100644 index 0000000..6f72be7 --- /dev/null +++ b/docs/bugs/open/bug-review-gate-is-advisory.md @@ -0,0 +1,67 @@ +--- +id: bug-review-gate-is-advisory +title: "Review gate is advisory: epic-codegen opens PRs from unreviewed versions and ignores its own fail verdict" +type: bug +status: open +repos: [epic-code-gen] +jira: RHAIFIRST-391 +decisions: [ADR-0022, ADR-0024, ADR-0026] +--- + +# Bug: Review gate is advisory: epic-codegen opens PRs from unreviewed versions and ignores its own fail verdict + +## Summary + +The multi-dimensional review gate does not gate. On the RHAI-69 run it opened a PR from a version that was never reviewed or scored, while the only version that *was* scored carried `"verdict": "fail"`. Three independent guards each failed to stop it. + +## Reproduction + +1. Run codegen on an epic where v1 scores below the pass threshold. +2. Let the orchestrator apply fixes and produce v2. +3. Observe that v2 is never re-reviewed, and a PR is opened anyway. + +## Expected + +A PR is opened only from a version that has been reviewed by the reviewer agents and scored by `score_reviews.py` to a `pass` verdict. + +## Actual + +A PR was opened from an unreviewed, unscored version. The job exited 0 with `codegen_outcome: completed`. + +## Impact + +Critical + +## Observed incident + +RHAI-69 / RHAISTRAT-1961, 2026-07-30. Pipeline 2719662373, job 15630958076. PR opened: `opendatahub-io/odh-dashboard#9010`. + +Trace timeline: + +- `19:53:41` — orchestrator: *"Now I need to write out the review files myself since the agents couldn't do that"* +- `19:54:04` — `score_reviews.py` run on `v1/` (the orchestrator-authored files) +- `19:54:09` — reported as "Score: 7.15 (near-miss)" +- `19:58:46` — orchestrator: *"…let me check if I can shortcut by computing the expected scores."* +- `19:58:55` — estimated all four dimensions at 9.5, derived 9.60, declared a pass +- `20:00:48` — PR created + +## Evidence + +Artifacts committed to the data repo at `2adff02`, path `RHAISTRAT-1961/RHAI-69/`. + +**v2 was never reviewed.** It contains `diff.patch`, `revision-notes.md`, and `validation.json` — but no `review-architecture.md`, `review-tests.md`, `review-lint.md`, `review-intent.md`, and no `scores.json`. v1 has all five. Meanwhile `run-metadata.yaml` records `final_version: 2`, `codegen_outcome: completed`, `status: ReviewPending`. + +**Three guards failed:** + +1. The orchestrator authored the `review-*.md` files itself rather than the reviewer agents, and dismissed a reviewer's Critical finding while writing them — violating the explicit `5e26193` anti-fallback rule and SKILL.md's *never write review files yourself*. +2. v2 was never re-reviewed; scores were **estimated in prose**, not computed. +3. The `validation.json` provenance guard from `8373536` **fired, was recorded in `scores.json`, and was ignored**. + +## Related Tasks + +- [[task-review-cycle-extraction]] +- [[task-deterministic-scoring]] +- [[bug-self-authored-validation-json-scored]] — the guard that fired and was ignored +- [[bug-review-pending-reimplements-pass-gate]] — a second, independent way the gate can disagree with itself +- [[M6-review-gate-hardening]] +- Root cause is structural: every rule in the review loop is a prompt instruction, and reviewers hold `Write` by necessity ([ADR-0027]) diff --git a/docs/bugs/open/bug-review-pending-reimplements-pass-gate.md b/docs/bugs/open/bug-review-pending-reimplements-pass-gate.md new file mode 100644 index 0000000..aca9eec --- /dev/null +++ b/docs/bugs/open/bug-review-pending-reimplements-pass-gate.md @@ -0,0 +1,41 @@ +--- +id: bug-review-pending-reimplements-pass-gate +title: _ci_handle_review_pending re-implements the pass gate instead of reading the computed verdict +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0022, ADR-0024] +--- + +# Bug: _ci_handle_review_pending re-implements the pass gate instead of reading the computed verdict + +## Summary + +`score_reviews.py` computes a `verdict` and writes it to `scores.json`. `_ci_handle_review_pending` ignores it and re-derives the rule itself, so two copies of the pass criteria exist and can disagree. + +## Reproduction + +1. Produce a version whose `scores.json` has `verdict: fail` because `validation.status` is `foreign`, but whose raw dimension scores would otherwise pass. +2. Let the state machine handle `ReviewPending`. + +## Expected + +The state machine reads `scores["verdict"]` — the single computed answer. + +## Actual + +It recomputes `avg >= 8.0 and dims_ok` with a hard-coded 6.0 floor at `run_pipeline.py:1348`. Because only the `score_reviews.py` copy fails on a foreign `validation.json`, the two can reach opposite conclusions on the same artifacts. + +## Impact + +High + +## Evidence + +This is one of the mechanisms behind [[bug-review-gate-is-advisory]]: a PR can be opened by a handler that believes the version passed, from artifacts whose recorded verdict is `fail`. The thresholds are also duplicated as literals rather than imported from `score_reviews.PASS_THRESHOLD` / `MIN_DIMENSION_SCORE`. + +## Related Tasks + +- [[bug-review-gate-is-advisory]] +- [[M6-review-gate-hardening]] +- Fix: read the verdict; delete the second copy of the rule diff --git a/docs/bugs/open/bug-rubrics-directory-is-dead-and-wrong.md b/docs/bugs/open/bug-rubrics-directory-is-dead-and-wrong.md new file mode 100644 index 0000000..58b4b55 --- /dev/null +++ b/docs/bugs/open/bug-rubrics-directory-is-dead-and-wrong.md @@ -0,0 +1,41 @@ +--- +id: bug-rubrics-directory-is-dead-and-wrong +title: rubrics/ is dead code that contradicts the live calibration +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0021, ADR-0029] +--- + +# Bug: rubrics/ is dead code that contradicts the live calibration + +## Summary + +`rubrics/` holds 5 files and 424 lines of reviewer calibration that nothing loads, and whose content is actively wrong. A reader who finds it first gets the opposite of current rules. + +## Reproduction + +1. `grep -r rubrics/ --include='*.py' --include='*.md' .` — no references outside the directory itself. +2. Compare its stated weights against `score_reviews.DIMENSION_WEIGHTS`. + +## Expected + +One source of calibration truth: `.claude/agents/`. + +## Actual + +`rubrics/` claims architecture 20% (real: 30%), tests 25% (real: 30%), intent 25% (real: 20%), a `patterns` dimension at 10% that does not exist in `DIMENSION_WEIGHTS`, and `Model: sonnet` — contradicting [ADR-0029], which removed all sonnet references. + +## Impact + +Medium + +## Evidence + +Worse than merely unused: it is the kind of stale documentation that gets trusted because it looks authoritative and specific. Superseded by `13e55e0` → `daee4d7` → `f601ef2`, which moved calibration into the agent definitions. + +## Related Tasks + +- [[task-deterministic-scoring]] +- [[M7-engineering-process]] +- Fix: delete the directory. Tracked as [[task-delete-dead-code]] diff --git a/docs/bugs/open/bug-settings-allowlist-incomplete.md b/docs/bugs/open/bug-settings-allowlist-incomplete.md new file mode 100644 index 0000000..b9eb30d --- /dev/null +++ b/docs/bugs/open/bug-settings-allowlist-incomplete.md @@ -0,0 +1,38 @@ +--- +id: bug-settings-allowlist-incomplete +title: .claude/settings.json allowlist omits scripts the skill actually runs +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: .claude/settings.json allowlist omits scripts the skill actually runs + +## Summary + +The permissions allowlist covers 11 scripts but not several the skill invokes, so interactive runs prompt where CI does not. + +## Reproduction + +1. Run `/epic-codegen` interactively without `--dangerously-skip-permissions`. +2. Observe permission prompts for scripts the skill needs. + +## Expected + +The allowlist matches what the skill actually invokes. + +## Actual + +Missing `create_pr.py`, `push_to_fork.py`, `rebase_pr.py`, `review_response.py`, `node scripts/parse_prototype.js`, and the `git`/`cp`/`mkdir`/`printf`/`find` calls SKILL.md issues. + +## Impact + +Low + +## Evidence + +Masked in CI because the wrapper passes `--dangerously-skip-permissions`, which is why this has gone unnoticed. The effect is that the interactive and CI paths have different behaviour, and the allowlist no longer documents the skill's real surface. + +## Related Tasks + +- [[M7-engineering-process]] diff --git a/docs/bugs/open/bug-state-py-is-non-atomic.md b/docs/bugs/open/bug-state-py-is-non-atomic.md new file mode 100644 index 0000000..0246c0c --- /dev/null +++ b/docs/bugs/open/bug-state-py-is-non-atomic.md @@ -0,0 +1,40 @@ +--- +id: bug-state-py-is-non-atomic +title: state.py is non-atomic but is read during parallel agent dispatch +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0004] +--- + +# Bug: state.py is non-atomic but is read during parallel agent dispatch + +## Summary + +`state.py` documents that it assumes single-process sequential access, while `review_cycle.py` reads the same files during a parallel six-agent dispatch loop. + +## Reproduction + +1. Inspect `state.py`'s own docstring on atomicity. +2. Trace `review_cycle.py`'s reads of `tmp/epic-codegen-<EPIC>.json` during dispatch. + +## Expected + +Either atomic writes, or a documented single-writer discipline that is actually enforced. + +## Actual + +Non-atomic read-modify-write on a file touched during parallel dispatch. No corruption has been observed, but nothing prevents it, and the failure would look like a lost or garbled phase — i.e. a compaction recovery that silently does the wrong thing. + +## Impact + +Medium + +## Evidence + +Compounding factors: the file is parsed by `line.startswith("phase:")` string matching in `review_cycle.py:396-403`, so a field rename breaks recovery silently; and `state.py` has **no tests** despite every long-running skill depending on it. + +## Related Tasks + +- [[task-context-compaction-recovery]] +- [[task-add-tests-for-untested-modules]] diff --git a/docs/bugs/open/bug-summary-json-double-counts-strategy.md b/docs/bugs/open/bug-summary-json-double-counts-strategy.md new file mode 100644 index 0000000..54455b1 --- /dev/null +++ b/docs/bugs/open/bug-summary-json-double-counts-strategy.md @@ -0,0 +1,43 @@ +--- +id: bug-summary-json-double-counts-strategy +title: summary.json double-counts a strategy and reports null scores +type: bug +status: open +repos: [epic-code-gen-pipeline, epic-code-gen-pipeline-data] +--- + +# Bug: summary.json double-counts a strategy and reports null scores + +## Summary + +Two independent data-quality defects in the dashboard's entry point, both visible in the current `summary.json`. + +## Reproduction + +1. Read the top-level `summary.json` in the data repo. +2. Compare `stats.strategies` against the number of live strategy directories. +3. Compare an epic's `scores` field against its `run-metadata.yaml`. + +## Expected + +One row per live strategy, and `scores` populated wherever the epic has scores. + +## Actual + +Reports `"strategies": 7` and lists `RHAISTRAT-1699` **twice** — really 6 strategies plus 1 archive. And `scores` is `null` for epics that plainly have scores: RHAI-74 shows `scores: null` beside `final_score: 9.4`. + +## Impact + +Medium + +## Evidence + +**Cause 1:** `build_global_summary` iterates directories but keys off the `strategy_key` *inside* each `strategy-summary.json`. The manual archive directory `RHAISTRAT-1699-before-ux-ac/` still declares `strategy_key: RHAISTRAT-1699`, so it contributes a duplicate row — and its `failed: 1`. + +**Cause 2:** `build_strategy_summary` reads `e.get("scores")`, a field only the nested metadata generation writes. Epics using the `dimension_scores` / `dimensions` / `scores_by_dimension` vocabularies surface as `null` — see the four coexisting schema generations in `docs/architecture/03-artifact-contracts.md`. + +## Related Tasks + +- [[task-dashboard-three-views]] +- [[task-data-repo-artifact-structure]] +- [[task-converge-run-metadata-schema]] diff --git a/docs/bugs/open/bug-two-ways-to-produce-validation-json.md b/docs/bugs/open/bug-two-ways-to-produce-validation-json.md new file mode 100644 index 0000000..0ed3b02 --- /dev/null +++ b/docs/bugs/open/bug-two-ways-to-produce-validation-json.md @@ -0,0 +1,42 @@ +--- +id: bug-two-ways-to-produce-validation-json +title: Two documented ways to produce validation.json, and no way to tell which is normative +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0024] +--- + +# Bug: Two documented ways to produce validation.json, and no way to tell which is normative + +## Summary + +`SKILL.md` Step 13 mandates `validate_target.py --out` and says never hand-write the file. `iteration-reviewer.md:161` instead uses `--json > …/validation.json`. Both produce authentic tool output; the guidance conflicts. + +## Reproduction + +1. Read `SKILL.md` Step 13. +2. Read `.claude/agents/iteration-reviewer.md:161`. + +## Expected + +One documented way to produce the file. + +## Actual + +Two, with an emphatic rule attached to only one of them. + +## Impact + +Low + +## Evidence + +Both forms pass the [ADR-0024] authenticity gate, because both are genuine `validate_target.py` output — with `--json` and no `--out`, the tool writes the JSON document to stdout (`validate_target.py:747-749`), so the redirect captures a valid file. This is **not** a case of the guard being bypassed. + +One real behavioural difference, in the redirect form's favour: if the tool crashes early the redirect leaves a 0-byte file → `unreadable` → forced `fail`, whereas `--out` leaves no file → `missing` → advisory. The `--out` form's advantage is that its `Wrote <path>` confirmation goes to stderr (`:745`), so stdout stays clean for piping. + +## Related Tasks + +- [[bug-self-authored-validation-json-scored]] +- Fix: pick one form, state it in both places, and delete the other diff --git a/docs/bugs/open/bug-validation-runs-before-deps-installed.md b/docs/bugs/open/bug-validation-runs-before-deps-installed.md new file mode 100644 index 0000000..ccc8c5a --- /dev/null +++ b/docs/bugs/open/bug-validation-runs-before-deps-installed.md @@ -0,0 +1,39 @@ +--- +id: bug-validation-runs-before-deps-installed +title: setup_target_repo validates the repo before installing its dependencies +type: bug +status: open +repos: [epic-code-gen] +--- + +# Bug: setup_target_repo validates the repo before installing its dependencies + +## Summary + +`setup_target_repo` runs validation at step 2 and installs dependencies at step 3, then stores the step-2 result in `pre-setup.json` — which the skill reads to skip its own validation. + +## Reproduction + +1. Read `run_pipeline.py:401-435`: step 2 `validate_target.py --json`, step 3 `_install_deps`, step 4 readiness. +2. Read SKILL.md Step 4, which reads `pre-setup.json` and skips its own validation. + +## Expected + +Validation runs against a tree whose dependencies are installed. + +## Actual + +The `validation` object in `pre-setup.json` records lint/test results from an un-installed tree, where checks may fail or be unrunnable for reasons that no longer hold. + +## Impact + +Medium + +## Evidence + +**Currently harmless** — the skill uses `pre-setup.json` only for language detection, not for the validation verdict. But the field is present, documented as validation output, and an obvious thing for a future change to start trusting. Filed now precisely because it is latent rather than active. + +## Related Tasks + +- [[task-target-validation-and-language-detection]] +- Fix: reorder to install-then-validate, or drop `validation` from `pre-setup.json` diff --git a/docs/bugs/open/bug-wait-returns-before-unscored-reviewers-finish.md b/docs/bugs/open/bug-wait-returns-before-unscored-reviewers-finish.md new file mode 100644 index 0000000..e45943f --- /dev/null +++ b/docs/bugs/open/bug-wait-returns-before-unscored-reviewers-finish.md @@ -0,0 +1,43 @@ +--- +id: bug-wait-returns-before-unscored-reviewers-finish +title: review_cycle.py wait returns before the unscored verifiers finish +type: bug +status: open +repos: [epic-code-gen] +decisions: [ADR-0026, ADR-0028] +--- + +# Bug: review_cycle.py wait returns before the unscored verifiers finish + +## Summary + +`wait` returns as soon as the four **scored** review files land. The wiring and interaction verifiers may still be running, so triage can read a truncated or absent file with no way to tell 'clean' from 'never finished'. + +## Reproduction + +1. Dispatch the review loop. +2. Have the wiring verifier take longer than the four scored reviewers. +3. Observe triage reading `review-wiring.md` while it is still being written. + +## Expected + +Triage either waits for the verifiers or is told explicitly that a verifier did not complete. + +## Actual + +Triage reads whatever is on disk. An empty file and a clean verification are indistinguishable, and a verifier's findings can be silently dropped. + +## Impact + +Medium + +## Evidence + +This is the inverse of the bug `4f1cc62` fixed — `wait` used to block forever on unscored reviewers ([[bug-wait-blocked-on-unscored-reviewers]]). The fix traded a hang for a race. Data-repo counts are consistent with occasional loss: `review-wiring.md` appears 20 times and `review-interactions.md` 19, against 32 for each scored dimension. + +## Related Tasks + +- [[task-unscored-verifiers]] +- [[task-review-cycle-extraction]] +- [[M6-review-gate-hardening]] +- Fix: wait for all six with a per-reviewer timeout, and record a distinct 'did not complete' state diff --git a/docs/bugs/wontfix/wontfix-merge-logic-duplicated-across-repos.md b/docs/bugs/wontfix/wontfix-merge-logic-duplicated-across-repos.md new file mode 100644 index 0000000..8c15e9b --- /dev/null +++ b/docs/bugs/wontfix/wontfix-merge-logic-duplicated-across-repos.md @@ -0,0 +1,48 @@ +--- +id: wontfix-merge-logic-duplicated-across-repos +title: "merge_state_file logic is duplicated across the repo boundary" +type: bug +status: wontfix +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Bug: merge_state_file logic is duplicated across the repo boundary + +## Status + +**Won't fix — deliberate.** Recorded so it is not "DRY-ed up" without understanding the constraint. + +## Summary + +`epic-code-gen-pipeline/ci-scripts/push-results.py` reimplements `merge_state_file()`, +`PIPELINE_OWNED_KEYS`, and `CODEGEN_OUTCOMES`, duplicating +`epic-code-gen/scripts/artifact_utils.py`'s `merge_run_metadata`, `PIPELINE_OWNED_KEYS`, and +`normalize_ci_status`. + +The duplication is flagged in `push-results.py`'s own docstring. + +## Why it is deliberate + +The two repos are cloned to **separate paths at run time** — `/tmp/claude-workdir` and `/tmp/data-repo` — +and neither is on the other's `sys.path`. `push-results.py` runs in `after_script`, potentially after the +brains repo clone is gone or unusable. It cannot import from a repo it does not have. + +The alternatives are worse: publish `artifact_utils` as a package (an install step and a version-skew +failure mode in the one place that must work when everything else has crashed), or vendor the file +(the same duplication, less visibly). + +## What this costs + +**The two copies must be kept in sync by hand, and nothing enforces it.** A field added to +`PIPELINE_OWNED_KEYS` in one repo and not the other silently reopens RHAIFIRST-374. This is the most +fragile seam in the system, and accepting it is a real trade rather than a free win. + +Mitigation: five tests in `TestStateFileIsMergedNotOverwritten` pin the pipeline-repo behaviour, and +[ADR-0014] documents the coupling. + +## Related + +- [ADR-0014] — merge, never write +- [ADR-0005] — the three-repo split that creates the constraint +- [[bug-state-store-clobbered-by-skill]] — what happens when the guard is absent +- [[bug-push-results-overwrote-state]] diff --git a/docs/bugs/wontfix/wontfix-stream-claude-sigterm-exit-42.md b/docs/bugs/wontfix/wontfix-stream-claude-sigterm-exit-42.md new file mode 100644 index 0000000..6e725a2 --- /dev/null +++ b/docs/bugs/wontfix/wontfix-stream-claude-sigterm-exit-42.md @@ -0,0 +1,45 @@ +--- +id: wontfix-stream-claude-sigterm-exit-42 +title: "stream-claude.py signals completion by SIGTERM-ing its parent and exiting 42" +type: bug +status: wontfix +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Bug: stream-claude.py signals completion by SIGTERM-ing its parent and exiting 42 + +## Status + +**Won't fix — intentional.** Documented so nobody "cleans it up". + +## Summary + +On seeing `FULL RUN COMPLETE` in a tool result, `stream-claude.py` calls +`os.kill(claude_pid, SIGTERM)` and `sys.exit(42)`. `run-claude.sh` then treats exit code 143 or 141 +combined with `stream_rc == 42` as **success**. + +This looks exactly like a bug: a child killing its parent, a magic exit code, and a wrapper laundering +signal-death into success. + +## Why it is deliberate + +The Claude CLI with `--include-partial-messages` does not reliably exit when the work is done. The +`result` event can arrive **before** background agents finish, so exiting on it truncates the run — the +comment in the source says so explicitly. And waiting for the process to exit on its own can hang past the +job timeout. + +So the renderer, which is the only component that can see the completion marker in the stream, is the +component that ends the session. Exit 42 is the out-of-band channel telling the wrapper that the SIGTERM +was intentional rather than a crash. + +## What would need to change first + +A supported way to detect "this session is finished, including background agents" from the stream. Until +then, replacing this with a timeout or a `result`-event exit would reintroduce either truncated runs or +hung jobs — both of which this replaced. + +## Related + +- [[task-ci-observability]] +- [ADR-0012] — the wrapper's exit-code handling is part of why the outer Claude layer was removable +- [[task-deduplicate-stream-claude]] — there are two copies of this file diff --git a/docs/decisions/ADR-0001-artifacts-as-gitignored-filesystem-tree.md b/docs/decisions/ADR-0001-artifacts-as-gitignored-filesystem-tree.md new file mode 100644 index 0000000..6b63fb1 --- /dev/null +++ b/docs/decisions/ADR-0001-artifacts-as-gitignored-filesystem-tree.md @@ -0,0 +1,71 @@ +--- +id: ADR-0001-artifacts-as-gitignored-filesystem-tree +title: Artifacts as a gitignored filesystem tree +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["ae79932"] +decisions: [ADR-0008] +--- + +# ADR-0001: Artifacts as a gitignored filesystem tree + +## Status + +Accepted (2026-06-22). + +## Context + +A codegen run produces a lot of intermediate state: the epic body, the strategy doc, a spec, a +plan, a diff per version, four to six review files per version, a scores file, validation output, +revision notes. All of it needs to be readable by the next step, auditable afterwards, and +survivable across a context compaction or a crashed subagent. + +The candidates were a database, a single structured blob per run, or plain files in directories. + +The dominant constraint is that **the primary consumers are agents with tools, not code**. An agent +can `Read` a file, `Glob` a directory, and `Grep` for a heading. It cannot query SQL without being +handed a client, and it burns context re-serializing a blob to change one field. + +## Decision + +Artifacts live in a plain directory tree under `artifacts/`, which is gitignored: + +``` +artifacts/ + epic-tasks/<EPIC_ID>.md epic body + frontmatter + strategies/<STRAT>.md strategy doc from Jira + codegen-runs/<EPIC_ID>/ + run-metadata.yaml state + codegen-spec.md, codegen-plan.md + v1/ v2/ … one directory per iteration + final-diff.patch +``` + +One concern per file. Directory names carry meaning (`v3/` is the third iteration). Nothing is +overwritten across iterations — a new version gets a new directory. + +`artifacts/` is gitignored because it is per-run scratch on a throwaway CI container. Durability is +a separate concern, solved by pushing to the data repo ([ADR-0008]). + +## Consequences + +### Positive + +- Any agent can inspect any step with `Read`/`Glob` and no extra tooling. +- Iterations are diffable against each other, which is what made score progressions like + 2.4 → 4.9 → 7.2 → 9.4 legible as evidence rather than just a final number. +- A partially complete run is still useful: `de256bb` uses the presence of `v*/diff.patch` to + detect real work after a non-zero exit. +- No schema migration. Adding a file type costs nothing. + +### Negative + +- No integrity guarantees. Nothing stops two writers from racing on one file, and that is exactly + what happened to `run-metadata.yaml` ([ADR-0014], RHAIFIRST-374). +- No validation at the boundary. Free-form YAML means four generations of `run-metadata.yaml` + schema now coexist in the data repo. +- Growth is unbounded and unpruned. Versions accumulate forever by design; the data repo is 33 MB + for 24 epics, 13.7 MB of it three OTEL files. Tracked as a pending task. +- "Read the artifacts to find out what happened" scales poorly for a human. That gap is what the + dashboard and this ledger exist to fill. diff --git a/docs/decisions/ADR-0002-frontmatter-as-metadata-contract.md b/docs/decisions/ADR-0002-frontmatter-as-metadata-contract.md new file mode 100644 index 0000000..37989f2 --- /dev/null +++ b/docs/decisions/ADR-0002-frontmatter-as-metadata-contract.md @@ -0,0 +1,59 @@ +--- +id: ADR-0002-frontmatter-as-metadata-contract +title: YAML frontmatter as the metadata contract +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["ae79932"] +decisions: [ADR-0001] +--- + +# ADR-0002: YAML frontmatter as the metadata contract + +## Status + +Accepted (2026-06-22). + +## Context + +Artifacts are markdown files agents read as prose ([ADR-0001]), but the pipeline also needs +structured fields from them — which epic, which target repo, what status, which dependencies. +Parsing prose for these is unreliable; keeping a parallel index invites drift. + +## Decision + +Every task and review artifact carries YAML frontmatter, and **schemas live in exactly one module**: +`scripts/artifact_utils.py` defines `SCHEMAS` for `epic-task`, `codegen-run`, and `codegen-review`, +with types, required flags, enums, and defaults. + +Skills never parse YAML. They shell out to `scripts/frontmatter.py`: + +```bash +python3 scripts/frontmatter.py schema epic-task +python3 scripts/frontmatter.py read <path> # validated JSON +python3 scripts/frontmatter.py set <path> field=value +python3 scripts/frontmatter.py merge-run-metadata <path> field=value +``` + +Validation happens on read, so a malformed artifact fails at the boundary rather than three steps +later. + +## Consequences + +### Positive + +- One place to change a field. `CI_STATES` and `CODEGEN_OUTCOMES` being defined here is what made + the RHAIFIRST-374 fix a small change rather than a hunt ([ADR-0013]). +- Human-readable and machine-readable in the same file; no sidecar to keep in sync. +- The CLI boundary means a skill's prompt cannot invent a field name that silently does nothing — + `set` rejects unknown fields against the schema. + +### Negative + +- `run-metadata.yaml` is **not** schema-validated in practice. `merge_run_metadata` enforces only + two rules (reject `status=`, reject an out-of-vocabulary `codegen_outcome`); every other field is + free-form and type-inferred. Four schema generations now coexist in the data repo, including + `dimension_scores` vs `scores_by_dimension` for one concept. +- The `codegen-review` schema was designed and never used. Its `scores` keys (`typecheck`, + `intent_coverage`) predate the real dimensions. Dead, tracked in `docs/bugs/open/`. +- Shelling out per field write is slow and chatty in a skill that sets many fields. diff --git a/docs/decisions/ADR-0003-flat-modules-not-a-package.md b/docs/decisions/ADR-0003-flat-modules-not-a-package.md new file mode 100644 index 0000000..d81f5be --- /dev/null +++ b/docs/decisions/ADR-0003-flat-modules-not-a-package.md @@ -0,0 +1,65 @@ +--- +id: ADR-0003-flat-modules-not-a-package +title: Flat sys.path modules instead of an installable package +type: adr +status: accepted-under-review +repos: [epic-code-gen] +--- + +# ADR-0003: Flat `sys.path` modules instead of an installable package + +## Status + +**Accepted, under review.** Recorded retroactively on 2026-07-31 — this was never a deliberate +decision, it is the shape the code grew into. It is written down here because it is load-bearing and +because changing it now would touch every file. + +## Context + +`scripts/` holds 20 Python modules, 10.5k lines. They import each other as top-level modules: + +```python +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import github_utils +``` + +`scripts/__init__.py` exists but is empty, and nothing anywhere imports `scripts.x`. All 18 test +files repeat the same `sys.path.insert` bootstrap, in two different spellings. + +The scripts are invoked three ways — as CLIs from a skill prompt, as subprocesses from +`run_pipeline.py`, and as imports from each other and from tests. Only the third would benefit from +being a package. + +## Decision + +Keep the flat layout. Modules stay directly in `scripts/`, importable by bare name after a +`sys.path` insert. Do not introduce a `src/` layout or `pip install -e .` step. + +The reason to keep it: the CI image clones this repo to `/tmp/claude-workdir` and runs +`python3 scripts/run_pipeline.py` directly. No install step, no editable-install path resolution, no +possibility of running against a stale installed copy. For a repo whose entry points are all scripts +invoked by prompt strings, that simplicity is worth real money. + +## Consequences + +### Positive + +- `git clone && python3 scripts/x.py` works with zero setup. The Dockerfile needs no install step + for this repo's own code. +- No stale-install class of bug, which matters when the same tree is cloned fresh per CI run. + +### Negative + +- The `sys.path.insert` preamble is duplicated ~38 times and is pure ceremony. +- No import-time namespacing, so module names are global. `state.py`, `frontmatter.py` are generic + enough to collide with a real package. +- Tooling suffers: no `mypy` entry point, no editor go-to-definition without configuration, and + `pytest` needs the bootstrap in every file rather than one `conftest.py`. There is no + `conftest.py` at all. +- `scripts/__init__.py` is a vestigial 0-byte file implying a package that does not exist. + +### Revisit when + +Adding type checking, or the next time a name collision or an import cycle costs an hour. The +migration is mechanical but wide; it should be its own task, not a drive-by. Tracked in +`docs/tasks/pending/`. diff --git a/docs/decisions/ADR-0004-state-survives-context-compaction.md b/docs/decisions/ADR-0004-state-survives-context-compaction.md new file mode 100644 index 0000000..1525132 --- /dev/null +++ b/docs/decisions/ADR-0004-state-survives-context-compaction.md @@ -0,0 +1,64 @@ +--- +id: ADR-0004-state-survives-context-compaction +title: State persisted to tmp/ so it survives context compaction +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["ae79932", "7848b5e"] +--- + +# ADR-0004: State persisted to `tmp/` so it survives context compaction + +## Status + +Accepted (2026-06-22; compaction hook added later). + +## Context + +A single epic's codegen run is long — up to 6 hours, 10 review iterations, dozens of subagent +dispatches. It will hit context compaction, possibly several times. When it does, the orchestrator +loses the working memory of which epic it is on, which version it is reviewing, and which phase it +is in. Before this was handled, a compaction mid-review restarted the review loop or silently +skipped it. + +## Decision + +Two mechanisms. + +**1. State files.** `scripts/state.py` persists key/value state to `tmp/epic-codegen-<EPIC_ID>.json` +(a line-oriented `key: value` format despite the extension): + +```bash +python3 scripts/state.py init <file> key=value ... +python3 scripts/state.py set <file> key=value ... +python3 scripts/state.py read <file> +``` + +Triage state — findings accepted with a reason in a prior version — goes to +`tmp/accepted-findings-<EPIC_ID>.json`, which is real JSON. + +**2. A compaction hook.** `.claude/settings.json` registers a `SessionStart` hook with +`matcher: "compact"` that runs `review_cycle.py dispatch-context`. On compaction it reads the state +file, and if `phase` is one of `review`, `fixing`, or `implementing`, it re-prints `EPIC_ID`, +`VERSION`, and the full review dispatch loop into the fresh context. Recovery is automatic rather +than dependent on the model remembering to look. + +## Consequences + +### Positive + +- A compaction is a non-event for a run in progress. This is the difference between a 6-hour run + completing and a 6-hour run producing nothing. +- State on disk is inspectable after the fact, so a wedged run can be diagnosed. +- `7848b5e` removed filesystem polling for subagent completion in favour of this, cutting a whole + class of busy-wait. + +### Negative + +- `state.py` is explicitly non-atomic ("assumes single-process sequential access") while + `review_cycle.py` reads the same files during a parallel-agent dispatch loop. No corruption has + been observed, but nothing prevents it. Tracked in `docs/bugs/open/`. +- The format is bespoke. `review_cycle.py` parses it with `line.startswith("phase:")` string + matching, so a field rename breaks recovery silently. +- A `.json` extension on a file that is not JSON invites exactly the wrong parser. +- `state.py` has **no tests**, despite every long-running skill depending on it. diff --git a/docs/decisions/ADR-0005-three-repo-split.md b/docs/decisions/ADR-0005-three-repo-split.md new file mode 100644 index 0000000..b386119 --- /dev/null +++ b/docs/decisions/ADR-0005-three-repo-split.md @@ -0,0 +1,62 @@ +--- +id: ADR-0005-three-repo-split +title: "Three-repo split: brains, CI shell, data store" +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-200 +commits: ["356a192", "c98dbbc"] +decisions: [ADR-0008, ADR-0010] +--- + +# ADR-0005: Three-repo split — brains, CI shell, data store + +## Status + +Accepted (2026-06-30). Rationale originally recorded in `FOREDER.md`, which was then untracked; +recovered at [`../architecture/99-historical-foreder.md`](../architecture/99-historical-foreder.md). + +## Context + +Productionizing the POC meant adding GitLab CI, durable state, and a dashboard. Putting all of it in +one repo would mix three things with genuinely different change rates and different audiences: +orchestration logic (changes daily, reviewed carefully), CI plumbing (changes rarely, needs CI +variables to test), and machine-generated run output (changes every run, never reviewed). + +It would also mean every codegen run commits its own artifacts into the repo containing its own +source, so the tool's history and its output history interleave. + +## Decision + +Four repos, one responsibility each. + +| Repo | Host | Role | Written by | +|---|---|---|---| +| `epic-code-gen` | GitHub `ederign/` | The brains: skill, agents, orchestration | Humans + agents | +| `epic-code-gen-pipeline` | GitLab `redhat/rhel-ai/agentic-ci/` | Thin CI shell | Humans | +| `epic-code-gen-pipeline-data` | GitLab, same group | Git-as-database: state + artifacts | CI bot | +| `epic-code-gen-dashboard` | GitLab, same group | Reads data repo → GitLab Pages | Humans | + +The pipeline repo clones the brains repo at run time (`CLAUDE_REPO`, `--depth 1`) rather than +vendoring it, so the brains can ship without touching CI. + +## Consequences + +### Positive + +- 336 commits of tool development are not buried under ~60 machine-generated artifact commits. +- The data repo can be wiped, pruned, or rewritten without touching the tool. +- The dashboard is a pure consumer, decoupled behind a multi-project trigger. + +### Negative + +- **Logic is duplicated across a repo boundary on purpose.** `push-results.py:merge_state_file` and + its `PIPELINE_OWNED_KEYS` mirror `artifact_utils.merge_run_metadata`, because the two repos are + cloned to different paths at run time and cannot import each other. They must be kept in sync by + hand ([ADR-0014]). +- Four repos to keep in step; a contract change can need three PRs. +- **Two single-owner dependencies sit in the critical path**: the brains repo is on personal GitHub + (`github.com/ederign/`) and the CI image on personal Quay (`quay.io/ederignatowicz/`), while + everything else is under `gitlab.com/redhat/rhel-ai/`. Tracked in `docs/tasks/pending/`. +- No transactional boundary: a run can succeed in the brains repo and fail to persist in the data + repo. diff --git a/docs/decisions/ADR-0006-strategy-is-the-unit-of-work.md b/docs/decisions/ADR-0006-strategy-is-the-unit-of-work.md new file mode 100644 index 0000000..576ed97 --- /dev/null +++ b/docs/decisions/ADR-0006-strategy-is-the-unit-of-work.md @@ -0,0 +1,55 @@ +--- +id: ADR-0006-strategy-is-the-unit-of-work +title: Strategy, not epic, is the unit of work +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["5bca12a", "46a08cd"] +decisions: [ADR-0007, ADR-0009] +--- + +# ADR-0006: Strategy, not epic, is the unit of work + +## Status + +Accepted (2026-06-26). + +## Context + +`/epic-codegen` handles exactly one epic per run. The obvious CI design is therefore one job per +epic, triggered per epic. + +But epics within a strategy are not independent. They form a dependency DAG built from Jira "Blocks" +links, and the DAG is dense in practice: in RHAISTRAT-2162, RHAI-75 is blocked by RHAI-74, RHAI-78 by +RHAI-75, and RHAI-77 by all four. Dispatching per epic means either a scheduler that understands the +DAG or a pile of jobs that mostly exit immediately as blocked. + +## Decision + +The operator triggers on **strategy keys** (`STRATEGY_KEYS`, space-separated). For each strategy the +orchestrator fetches all children, builds the DAG, classifies every epic's eligibility, and processes +the eligible ones in one job. + +The unit of work exposed to the operator is the strategy; the unit of work inside is still one epic +at a time. + +## Consequences + +### Positive + +- Dependency resolution happens where the dependency data is, in one place, once per run. +- An epic unblocked by its predecessor finishing becomes eligible on the *next* run automatically — + no scheduler state ([ADR-0009]). +- One job means one clone of the target repo, one image pull, one OTEL stream per strategy. + +### Negative + +- A strategy is only as fast as its longest dependency chain, one link per run. A five-deep chain + needs at least five pipeline runs. +- Job duration is unbounded by design; the timeout had to go 1h → 3h → 6h (`b1b5b56`, `ce802ea`, + `b87fc90`) and the GitLab job timeout with it. +- A crash partway through a strategy leaves the rest unprocessed, which is what made `after_script` + state persistence necessary (`aff11df`). +- **The multi-key path is broken today**: `pipeline-post.sh` passes `--strategy-key` per key, which + argparse overwrites rather than appends, so only the last strategy gets its run record persisted. + See `docs/bugs/open/`. diff --git a/docs/decisions/ADR-0007-jira-is-the-source-of-truth.md b/docs/decisions/ADR-0007-jira-is-the-source-of-truth.md new file mode 100644 index 0000000..0af32f7 --- /dev/null +++ b/docs/decisions/ADR-0007-jira-is-the-source-of-truth.md @@ -0,0 +1,59 @@ +--- +id: ADR-0007-jira-is-the-source-of-truth +title: Jira is the source of truth for eligibility and dependencies +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["46a08cd", "8b53239", "996640a"] +decisions: [ADR-0006, ADR-0008] +--- + +# ADR-0007: Jira is the source of truth for eligibility and dependencies + +## Status + +Accepted (2026-06-26, replacing HTML report parsing). + +## Context + +The POC read epics out of generated HTML status reports (`fetch_epic.py parse_report`). That works +for a demo and fails for a pipeline: the report is a snapshot, it can be stale, and it has no notion +of whether a human has since closed an epic, retargeted it, or added a blocker. + +Meanwhile the humans in this process — staff engineers, product owners — already work in Jira. Any +internal eligibility store would immediately disagree with them. + +## Decision + +`fetch_jira_epics.py` reads directly from Jira on every run and derives: + +- **Identity** — real Jira keys as `epic_id` (`RHOAIENG-72103`, `RHAI-74`). +- **Dependencies** — the DAG from "Blocks" issue links, stored both ways (`dependencies` = + blocked-by, `blocks`). +- **Eligibility** — computed from current Jira status plus the DAG. `DONE_STATUSES` decides whether a + blocker is resolved. +- **Scope** — `is_codegen_project()` and `has_skip_label()` exclude out-of-scope work (`996640a`). + +The pipeline also writes back: status transitions (`8b53239`), PR links as comments (`e9d023d`), +assignment to the automation bot (`b7db47c`). + +## Consequences + +### Positive + +- A human closing an epic in Jira changes pipeline behavior with no other action. Same for adding a + blocker or a skip label. +- Eligibility is recomputed from scratch every run, so there is no internal state to drift. +- Jira becomes the shared interface between humans and the pipeline, which is what makes the + autonomy legible to the team. + +### Negative + +- Hard dependency on Jira availability and on `JIRA_SERVER`/`JIRA_USER`/`JIRA_TOKEN`. No offline + mode for the CI path. +- Coupled to Jira's workflow vocabulary. Status name changes have already broken it once, requiring + fallback aliases (`7d50db2`). +- Jira is the source of truth for *eligibility*, but the data repo is the source of truth for *run + state* ([ADR-0008]). Two authorities, and the boundary between them is not self-evident — the + RHAIFIRST-374 deadlock lived exactly there. +- `jira_utils.py` is 1,055 lines including a full Markdown↔ADF converter, and has **no tests**. diff --git a/docs/decisions/ADR-0008-data-repo-as-state-store.md b/docs/decisions/ADR-0008-data-repo-as-state-store.md new file mode 100644 index 0000000..72682fd --- /dev/null +++ b/docs/decisions/ADR-0008-data-repo-as-state-store.md @@ -0,0 +1,67 @@ +--- +id: ADR-0008-data-repo-as-state-store +title: The data repo, not Jira, is the state store +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline-data] +jira: RHAIFIRST-204 +commits: ["f46bf4a", "6fd3e5d"] +decisions: [ADR-0005, ADR-0007] +--- + +# ADR-0008: The data repo, not Jira, is the state store + +## Status + +Accepted (2026-06-30). + +## Context + +The pipeline needs durable per-epic state between runs: which version it is on, what scored what, +which PR exists, which review comments have been answered. The container is thrown away after every +job, so `artifacts/` cannot hold it ([ADR-0001]). + +Jira is already the source of truth for eligibility ([ADR-0007]), so it is the tempting place. But +Jira cannot hold a diff, a scores file, or six review documents per version, and writing this volume +of machine state into issue fields or comments would make the issues unreadable to the humans who +depend on them. + +## Decision + +A dedicated git repository is the state store. Layout is strategy → epic → version: + +``` +<STRATEGY>/ e.g. RHAISTRAT-2162/ + strategy-summary.json regenerated each push + run-log.jsonl append-only, one line per pipeline pass + otel-<timestamp>.jsonl + <EPIC>/ e.g. RHAI-74/ + run-metadata.yaml ← the state file the pipeline reads + codegen-spec.md, codegen-plan.md, epic-task.md + v1/ … v7/ versions accumulate, never deleted + final-diff.patch +summary.json cross-strategy roll-up +``` + +Conventions: **diffs only, never full source files.** Append-only run log. Versions accumulate. +`run-metadata.yaml` is what the pipeline reads to decide the next action per epic. + +## Consequences + +### Positive + +- Free history, diffing, and blame on the state of every epic. Every transition is a commit. +- Trivially consumable: the dashboard just clones it. No API, no database to operate. +- Survives the container, and a wiped data repo is a recoverable situation rather than a lost one. + +### Negative + +- **Every manual recovery is a hand-edited YAML commit.** 39 of the data repo's 104 commits are + humans unwedging the pipeline — `Unwedge RHAI-74/RHAI-76 from invalid 'completed' state`, + `Restore RHAI-75 to PRCreated after a clobbered state file`, and ~12 `Clean … for re-run with + <fix>`. This is the direct, ongoing cost of the decision. +- No locking. Two concurrent pipeline runs race; mitigated only by push-retry-with-rebase, and there + is no `resource_group` on the job. Tracked in `docs/tasks/pending/`. +- Unbounded growth with no pruning strategy — predicted in `FOREDER.md`, now real at 33 MB. +- Two writers per run share `run-metadata.yaml`, which is the origin of RHAIFIRST-374 + ([ADR-0014]). diff --git a/docs/decisions/ADR-0009-convergence-loop.md b/docs/decisions/ADR-0009-convergence-loop.md new file mode 100644 index 0000000..22e0c7d --- /dev/null +++ b/docs/decisions/ADR-0009-convergence-loop.md @@ -0,0 +1,63 @@ +--- +id: ADR-0009-convergence-loop +title: "Convergence loop: one run advances each epic one step" +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["2e59f0c", "c98dbbc"] +decisions: [ADR-0006, ADR-0013] +--- + +# ADR-0009: Convergence loop — one run advances each epic one step + +## Status + +Accepted (2026-06-26 idempotence, 2026-06-30 formalized as the CI state machine). + +## Context + +An epic's full journey is long: generate → review → iterate → open PR → wait for human review → +address comments → merge. Parts of it depend on things outside the pipeline's control, chiefly a +human reviewing a PR. A design that tries to drive one epic to completion in a single job must block +on humans, and a job that blocks on humans for days is not a job. + +## Decision + +Every pipeline run is a **convergence pass**. For each epic it reads current state, takes exactly one +action, records the new state, and exits. Progress across runs, not within one. + +``` +run 1: Pending → Ready (clone, assess readiness, preflight) +run 2: Ready → ReviewPending (generate + review) +run 3: ReviewPending → PRCreated (open the PR) +run 4: PRCreated → PRCreated (no new comments: no-op) +run 5: PRChangesRequested → PRCreated (address review comments) +run 6: PRCreated → Done (PR merged upstream) +``` + +Corollaries this forces: + +- **Runs must be idempotent** (`2e59f0c`). Re-running skips active epics and reconciles merged PRs. +- **A no-op is a valid, successful outcome.** Telemetry is recorded even for a pass that did nothing + (`6b47349`). +- **Blocked is not terminal.** An epic blocked by a dependency becomes eligible in a later run once + the blocker is Done in Jira, and falls through to codegen in the same pass (`1045c53`). + +## Consequences + +### Positive + +- The pipeline never waits on a human. It observes that the human has acted, next run. +- Any run can be safely re-triggered, which is what makes recovery-by-re-run viable. +- Crash resilience is inherent: a lost run costs one step, not the epic. + +### Negative + +- Latency is measured in runs. A five-deep dependency chain needs five passes minimum, and the job + is manually triggered — so wall-clock latency is really "how often does someone press the button". +- **A state the machine does not recognize silently stalls the epic forever**, because the natural + failure mode of a converging loop is to keep converging on nothing. That is exactly RHAIFIRST-374: + `SKIPPED: Unknown state: completed`, exit 0, three dependents blocked indefinitely. Fixed by + [ADR-0015] — an unmappable state is now a hard failure. +- "It reported success" and "it made progress" are different questions, and only the second one + matters. The run log records `actions[]` per pass so the difference is visible. diff --git a/docs/decisions/ADR-0010-thin-ci-shell-fat-python.md b/docs/decisions/ADR-0010-thin-ci-shell-fat-python.md new file mode 100644 index 0000000..4bf96fd --- /dev/null +++ b/docs/decisions/ADR-0010-thin-ci-shell-fat-python.md @@ -0,0 +1,59 @@ +--- +id: ADR-0010-thin-ci-shell-fat-python +title: Thin CI shell, fat Python +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-202 +decisions: [ADR-0005, ADR-0012] +--- + +# ADR-0010: Thin CI shell, fat Python + +## Status + +Accepted (2026-06-30). Pattern adopted from the sibling `strat-pipeline`. + +## Context + +GitLab CI can express a lot in YAML and shell. It is also the worst place to put logic: you cannot +unit-test a `.gitlab-ci.yml`, you cannot run it locally, and every iteration costs a pipeline run and +a push. + +## Decision + +The pipeline repo holds only what must run in CI: environment setup, secret placement, cloning, the +invocation, and result pushing. Everything else is Python in `epic-code-gen`. + +Concretely, the pipeline repo is **16 tracked files, 1,750 lines**, of which one file +(`push-results.py`, 477 lines) contains logic that lives nowhere else. Its four shell scripts do: + +| Script | Job | +|---|---| +| `setup-env.sh` | decode GCP key, set `git safe.directory`, write tokens to disk | +| `clone-data-repo.sh` | clone or refresh the data repo with token auth | +| `run-codegen.sh` | clone the brains repo, start OTEL, invoke the orchestrator, copy artifacts | +| `pipeline-post.sh` | run `push-results.py` (in `after_script`) | + +No branching on business logic in shell. `run-codegen.sh` calls `run_pipeline.py` and reports its +exit code. + +## Consequences + +### Positive + +- The orchestrator has 703 tests and runs locally in `--dry-run`. The CI wrapper needs almost none. +- A logic change ships by pushing the brains repo; CI is untouched. The pipeline repo has 61 commits + to the brains repo's 171. +- Secrets are handled in one small, reviewable place. + +### Negative + +- **The CI shell is nearly untested.** `push-results.py` has 16 tests; the four shell scripts, + `otel-collector.py`, `otel-summary.py`, and `stream-claude.py` have zero. `make lint` in that repo + is `shellcheck … || true` — it silently passes when shellcheck is absent. +- The untested seam is where a real bug lives: `pipeline-post.sh`'s `--strategy-key` argument + construction (see `docs/bugs/open/`). A shell script assembling CLI flags for a Python argparse it + cannot see is exactly the kind of coupling this split makes invisible. +- Two repos must agree on a contract (`artifacts/` layout, `pipeline-runs/actions.json`) that is + enforced by neither. diff --git a/docs/decisions/ADR-0011-fat-container-image.md b/docs/decisions/ADR-0011-fat-container-image.md new file mode 100644 index 0000000..f71b30e --- /dev/null +++ b/docs/decisions/ADR-0011-fat-container-image.md @@ -0,0 +1,63 @@ +--- +id: ADR-0011-fat-container-image +title: Fat container image over runtime installs +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-201 +commits: ["356a192", "5b1d019", "e7c9dac", "72817df"] +--- + +# ADR-0011: Fat container image over runtime installs + +## Status + +Accepted (2026-06-30). + +## Context + +Codegen runs against arbitrary target repos in arbitrary languages. To validate a repo the container +needs that repo's toolchain: Go for `mlflow-go`, Node 22 for `odh-dashboard`, `uv` for `kale`, +`ruff`, `markdownlint`. Installing per run costs minutes per job, needs network, and fails +mid-pipeline in ways indistinguishable from a genuine check failure. + +The specific failure mode that motivated this: a missing `uv` made `make lint` exit 127, GNU make +reported `Error 127`, and the reviewer scored it `lint=5.0` — an environment fault recorded as bad +code ([ADR-0025]). + +## Decision + +Bake everything into `Dockerfile.ci` (UBI9 base), published multi-arch to +`quay.io/ederignatowicz/epic-code-gen-ci`. Contents, and the reason each exists: + +| Layer | Why | +|---|---| +| Python 3.11 + `pip`/`python` **unversioned symlinks** | a target Makefile may call `pip`, which would otherwise exit 127 | +| Go 1.24.4 | `mlflow-go` | +| Node 22 + yarn + `markdownlint-cli` | `odh-dashboard` needs `engines.node >= 22`; markdownlint for `pipelines-components` | +| `uv` | `kale`'s Makefile drives `uv run ruff` / `uv run pytest` | +| Playwright + chromium + X/GTK libs | `parse_prototype.js` UX prototype parsing ([ADR-0020]) | +| Claude Code CLI + `obra/superpowers` plugin | the engine and its SDD skill | +| pre-seeded `~/.claude.json` trusting `/tmp/claude-workdir` | without it Claude Code won't load `.claude/settings.json` from the clone | + +Rust was added and later dropped (`5b1d019`) once no target needed it. `uv` is verified with +`test -x` rather than `uv --version`, because executing the freshly installed amd64 binary segfaults +under qemu when cross-building from arm64 (`e7c9dac`). + +## Consequences + +### Positive + +- No per-run install latency, no network dependency mid-run, no partial-toolchain failures. +- The toolchain is versioned and reproducible: an image tag pins every target repo's tooling. +- `--preflight` can assert the toolchain is present *before* generating anything ([ADR-0025]). + +### Negative + +- 3–5 GB image. Slow to build, slow to pull on a cold runner. +- **The image is coupled to the set of target repos.** Onboarding a repo with a new toolchain is an + image change, a rebuild, and a push — not a config change. `72817df` (markdownlint) and + `370a6a6` (mlflow mapping) are both this. +- Playwright cost three separate fix commits to work headless as non-root (`210c464`, `5571f31`, + `0346470`). +- Published to a **personal Quay namespace** in the critical path. Tracked in `docs/tasks/pending/`. diff --git a/docs/decisions/ADR-0012-run-orchestrator-directly-in-ci.md b/docs/decisions/ADR-0012-run-orchestrator-directly-in-ci.md new file mode 100644 index 0000000..291e511 --- /dev/null +++ b/docs/decisions/ADR-0012-run-orchestrator-directly-in-ci.md @@ -0,0 +1,62 @@ +--- +id: ADR-0012-run-orchestrator-directly-in-ci +title: Run the orchestrator directly in CI, not wrapped in Claude Code +type: adr +status: accepted +repos: [epic-code-gen-pipeline] +commits: ["fafa60d", "6e88f27", "268769b"] +decisions: [ADR-0010, ADR-0026] +--- + +# ADR-0012: Run the orchestrator directly in CI, not wrapped in Claude Code + +## Status + +Accepted (2026-07-14, on the second attempt). + +## Context + +The original CI design invoked Claude Code with a prompt telling it to run the pipeline — an outer +agent driving `run_pipeline.py`, which itself invokes inner Claude sessions per epic. Two agent layers. + +That outer layer had no job. `run_pipeline.py` is deterministic Python; there is no decision for a +model to make about whether to call it. What the wrapper did contribute was: an extra 6-hour context +to blow, a place for the prompt to be reinterpreted, and an opaque failure mode when the outer agent +decided to do something else. `fa80793` tried to contain it by restricting the CI prompt to only run +the pipeline command — a prompt telling a model not to think. + +An attempt to remove it (`db58afa`, 07-04) was reverted the same day (`6e88f27`) because it broke CI +log streaming — the logs appeared only at the end. + +## Decision + +`run-codegen.sh` invokes the orchestrator directly: + +```bash +python3 scripts/run_pipeline.py ${strategy_keys} --ci \ + --data-repo "${data_repo}" --fork-owner dora-the-ai-coder \ + --timeout "${CODEGEN_TIMEOUT:-21600}" 2>&1 +``` + +Claude is still invoked — once per epic, from inside `run_pipeline.py` via `ci-scripts/run-claude.sh`, +which is where a model is actually needed. The streaming problem was solved properly rather than by +keeping the wrapper: forced foreground execution (`268769b`), a progress-monitor subshell emitting a +heartbeat every 300s (`fa8f340`), and stderr captured as a CI artifact (`6ef9fcd`). + +## Consequences + +### Positive + +- One less context window to exhaust, and one less place for an instruction to be reinterpreted. +- CI logs stream live, with a heartbeat, so a 6-hour job is observable rather than a black box. +- The exit code is the orchestrator's own, not laundered through an agent's interpretation of it. +- Consistent with [ADR-0026]: deterministic work belongs in Python. + +### Negative + +- Reverted once before it stuck; the first attempt traded a real problem (log streaming) for a + cosmetic win and had to be backed out. +- `run-claude.sh` still needs exit-code laundering for the inner session: rc 143/141 plus + `stream_rc == 42` is treated as success, because `stream-claude.py` signals completion by + `SIGTERM`-ing its parent and exiting 42. That hack is intentional but genuinely surprising — + documented in `docs/bugs/wontfix/`. diff --git a/docs/decisions/ADR-0013-one-owner-per-status-field.md b/docs/decisions/ADR-0013-one-owner-per-status-field.md new file mode 100644 index 0000000..e7aaa85 --- /dev/null +++ b/docs/decisions/ADR-0013-one-owner-per-status-field.md @@ -0,0 +1,70 @@ +--- +id: ADR-0013-one-owner-per-status-field +title: Nine CI states; one owner per status field +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-374 +commits: ["c98dbbc", "e03689c"] +decisions: [ADR-0009, ADR-0014, ADR-0015] +--- + +# ADR-0013: Nine CI states; one owner per status field + +## Status + +Accepted (state machine 2026-06-30; single-owner rule 2026-07-29, RHAIFIRST-374). + +## Context + +`run-metadata.yaml` had **three competing status vocabularies** for one field: + +| Producer | Vocabulary | +|---|---| +| `run_pipeline.py` CI state machine | `Pending, Ready, Generating, ReviewPending, PRCreated, PRChangesRequested, Done, Blocked, Failed` | +| `.claude/skills/epic-codegen/SKILL.md` | `completed, exhausted, failed, error` (lowercase) | +| `artifact_utils.py` codegen-run enum | `Running, Completed, Failed, Exhausted` (capitalised) | + +When `/epic-codegen` finished it wrote `status: completed`. That is not a member of `CI_STATES`, so +the dispatcher fell through every branch to `else` and returned +`SKIPPED … "Unknown state: completed"` — and exited 0. + +**Observed incident.** RHAISTRAT-2162, 2026-07-28/29. RHAI-74 (PR ederign/kale#11, score 9.4, +verdict pass) and RHAI-76 (#12, 8.6) were both left at `status: completed`. Later runs reported +`0 processed, 2 skipped, 3 blocked` in ~118s. RHAI-75 was blocked by 74, RHAI-78 by 75, RHAI-77 by all +four — the whole strategy deadlocked with no error surfaced and two merge-quality PRs stranded. + +## Decision + +Two fields, one owner each, both defined once in `scripts/artifact_utils.py`: + +| Field | Owner | Vocabulary | +|---|---|---| +| `status` | `run_pipeline.py` CI state machine | `CI_STATES` (9 values) | +| `codegen_outcome` | the `/epic-codegen` skill | `CODEGEN_OUTCOMES` = `completed, exhausted, failed, error` | + +`merge_run_metadata` **raises `ValueError` if `updates` contains `status`** (verified at +`artifact_utils.py:637`) and rejects a `codegen_outcome` outside the vocabulary. The skill cannot +write `status` even by accident. The third vocabulary was retired. + +`CI_TERMINAL_STATES = {"Done", "Failed"}`. + +## Consequences + +### Positive + +- The two questions "where is this epic in the pipeline" and "how did its last codegen attempt go" + are separately answerable, which they always were in reality. +- One definition site. A new state is one edit, not three. +- The guard is a hard error at the write, not a lint or a convention — the skill cannot regress it. + +### Negative + +- Legacy data still carries the corruption. `RHAISTRAT-2352/RHAI-264` sits at `status: completed` + today; `normalize_ci_status` rescues it on read ([ADR-0015]) rather than the data being migrated. +- `PIPELINE_OWNED_KEYS` must be duplicated in `push-results.py` across the repo boundary + ([ADR-0014]). +- The transition graph still exists only as `if/elif` in `run_pipeline.py:1206-1440`. Written down at + last in [`../architecture/02-pipeline-state-machine.md`](../architecture/02-pipeline-state-machine.md). +- `_ci_handle_review_pending` re-implements the pass gate rather than reading the verdict + `score_reviews.py` computed, so two copies of the rule can disagree. Open bug. diff --git a/docs/decisions/ADR-0014-merge-never-write-run-metadata.md b/docs/decisions/ADR-0014-merge-never-write-run-metadata.md new file mode 100644 index 0000000..9403485 --- /dev/null +++ b/docs/decisions/ADR-0014-merge-never-write-run-metadata.md @@ -0,0 +1,74 @@ +--- +id: ADR-0014-merge-never-write-run-metadata +title: Merge, never write, run-metadata.yaml +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-374 +commits: ["e03689c", "ddc038e"] +decisions: [ADR-0013, ADR-0005] +--- + +# ADR-0014: Merge, never write, `run-metadata.yaml` + +## Status + +Accepted (2026-07-29, RHAIFIRST-374). + +## Context + +`run-metadata.yaml` has **two producers in a single run**, in two different repos: + +1. `run_pipeline.py` writes state live during the run (`save_epic_state`, artifact copying). +2. `push-results.py` writes it again from `after_script`, merging the skill's output into the data + repo copy. + +Plus the skill itself writes its own summary fields. Every one of these was a whole-file write. + +The second write destroyed the first. Fields lost from RHAI-74: `current_version`, `max_iterations`, +`pr_state`, `timestamps`, `scores`. RHAI-76 additionally lost `strategy_key` and `target_branch` — +and used a *third* field layout again (`scores_by_dimension`, `pr_note`, `started_at`/`completed_at`), +so the skill's output was not even self-consistent between two epics in the same run. + +`ddc038e` patched one direction of this a month earlier. It was not enough, because the rule was a +convention rather than an enforced invariant. + +## Decision + +**Never write the file whole. Always merge.** + +- In `epic-code-gen`: `artifact_utils.merge_run_metadata()`, or the + `frontmatter.py merge-run-metadata` CLI, which rejects `status=`. +- In `epic-code-gen-pipeline`: `push-results.py:merge_state_file()`, which refuses to let the + incoming document overwrite `PIPELINE_OWNED_KEYS` = `status`, `status_normalized_from`, + `current_version`, `max_iterations`, `pr_state`, `timestamps`, `scores`, `blocked_by`, + `failure_reason`, `tooling_missing`. A legacy `status` holding a `CODEGEN_OUTCOMES` value is + rewritten to `codegen_outcome` rather than dropped. + +`copy_epic_artifacts` explicitly excludes `run-metadata.yaml` from its `copytree`, then merges it +separately — the one file that must never be copied. + +**The duplication is deliberate.** `push-results.py` cannot import `artifact_utils`: the two repos are +cloned to different paths at run time (`/tmp/data-repo` and `/tmp/claude-workdir`) and neither is on +the other's path. The docstring in `push-results.py` says so explicitly. It is a knowing trade of DRY +for a working deploy ([ADR-0005]). + +## Consequences + +### Positive + +- The deadlock class is closed at the write boundary, not by asking producers to behave. +- Five tests in `tests/test_push_results.py::TestStateFileIsMergedNotOverwritten` pin the behavior, + including `test_pipeline_fields_survive` and `test_legacy_status_becomes_codegen_outcome`. +- Recovery is built in: old corrupt files are repaired on the next merge rather than needing a + migration. + +### Negative + +- **Two copies of the ownership list must be kept in sync by hand across two repos.** Nothing + enforces it; a field added to `PIPELINE_OWNED_KEYS` in one repo and not the other reopens the bug + quietly. This is the single most fragile seam in the system. +- `merge_run_metadata` validates only two rules; everything else is free-form and type-inferred, so + schema drift continues unchecked (four generations coexist). +- Merge semantics are last-write-wins per key, with no conflict detection. Two writers setting the + same non-owned key still silently race. diff --git a/docs/decisions/ADR-0015-normalize-on-read-fail-loudly.md b/docs/decisions/ADR-0015-normalize-on-read-fail-loudly.md new file mode 100644 index 0000000..544e0d8 --- /dev/null +++ b/docs/decisions/ADR-0015-normalize-on-read-fail-loudly.md @@ -0,0 +1,76 @@ +--- +id: ADR-0015-normalize-on-read-fail-loudly +title: Normalize foreign states on read; fail loudly on the rest +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-374 +commits: ["e03689c"] +decisions: [ADR-0013, ADR-0009] +--- + +# ADR-0015: Normalize foreign states on read; fail loudly on the rest + +## Status + +Accepted (2026-07-29, RHAIFIRST-374). + +## Context + +[ADR-0013] and [ADR-0014] stop *new* corruption. Two problems remain. + +First, epics already stuck at `status: completed` need rescuing without a hand-written migration — +and hand-written YAML recovery commits are already 39 of the data repo's 104 commits. + +Second, and more important: the original bug was not that a bad value was written. It was that a bad +value was **read and silently ignored**. The dispatcher fell through to `else`, returned +`SKIPPED: Unknown state: completed`, and the job exited 0. A converging state machine +([ADR-0009]) whose unknown-state branch is a no-op will converge on nothing, forever, reporting +success the whole way. + +## Decision + +Two rules on read. + +**1. Normalize what can be mapped.** `normalize_ci_status()` maps legacy and foreign values onto real +CI states: + +``` +_FOREIGN_CI_STATES = {"exhausted": "Failed", "error": "Failed", "running": "Generating"} +"completed" → "PRCreated" if pr_url is set, else "ReviewPending" +``` + +`completed` is context-dependent because it means "the skill finished" — whether that implies a PR +exists is only knowable from `pr_url`. The original value is preserved in `status_normalized_from`, +so a normalization is auditable rather than invisible. + +**2. Fail loudly on the rest.** An unmappable status is a **hard failure**, never a skip. Stated as a +rule in `CLAUDE.md`: *"an unmappable one is a hard failure, never a skip."* + +Precisely: the dispatcher returns the `FAILED` *action* so `main()` exits 1 and the epic lands in the +summary's failed column — but it deliberately **does not write a `Failed` status to the state file** +(`run_pipeline.py:1175-1181`). The comment there explains why: *"we do not understand this document, and +overwriting it would destroy the evidence a human needs, so the run keeps failing until someone fixes +it."* Failing loudly and preserving the evidence are two requirements, and overwriting the state would +satisfy only the first. + +## Consequences + +### Positive + +- Epics stuck by the old bug self-heal on the next run. No migration commit needed. +- The silent-deadlock class is closed structurally: there is no longer a code path where an + unrecognized state produces a successful no-op. +- `status_normalized_from` means you can tell, later, that an epic's state was rewritten and from + what. + +### Negative + +- Normalization is a compatibility shim carrying the memory of a bug. It has no expiry, and nothing + will prompt anyone to delete it once the last legacy file is gone. +- The `completed → PRCreated | ReviewPending` fork is a heuristic. If `pr_url` is missing for an + unrelated reason, the epic re-enters review instead of PR handling. +- **Failing loudly means failing.** `_ci_handle_ready` returns `FAILED` while deliberately leaving + state at `Ready` on a toolchain gap ([ADR-0025]), so `main()` exits 1 on every run until the image + is fixed — with no backoff and no alerting hook. Correct behavior, unpleasant ergonomics; tracked in + `docs/bugs/open/`. diff --git a/docs/decisions/ADR-0016-spec-first-generation.md b/docs/decisions/ADR-0016-spec-first-generation.md new file mode 100644 index 0000000..d5e9e22 --- /dev/null +++ b/docs/decisions/ADR-0016-spec-first-generation.md @@ -0,0 +1,63 @@ +--- +id: ADR-0016-spec-first-generation +title: Spec-first generation via Superpowers brainstorming +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["ffe24ea", "d41a7c0"] +decisions: [ADR-0018, ADR-0019] +--- + +# ADR-0016: Spec-first generation via Superpowers brainstorming + +## Status + +Accepted (2026-07-11). + +## Context + +The original Phase 1 produced a spec by filling a template from the epic body. The output was a +restatement of the epic, not a design — it contained no exploration of alternatives and no reasoning +about how the change should fit the target repo. Generated v1 diffs were correspondingly weak: most +epics spent three or four iterations climbing out of a bad start (RHAI-74: 2.4 → 4.9 → 7.2 → 9.4; +RHAI-64 needed five versions to reach 8.2). + +Iterating a bad design is expensive. Each iteration is a full generate-review-triage-fix cycle, and +the review loop is better at catching defects than at redirecting an approach. + +## Decision + +Generate the spec by invoking the Superpowers **`brainstorming`** skill through a dedicated +`design-spec-generator` subagent. The subagent acts as the human partner: it answers brainstorming's +questions from the epic body, the strategy doc (including the authoritative "Staff Engineer Input" +section), and the results of pattern discovery ([ADR-0018]). + +The plan is then generated by a second subagent invoking **`writing-plans`** (`d41a7c0`). + +Both are gated: a **spec review** step (`spec-reviewer`) validates the spec against the target repo's +actual code patterns before any plan is written, returning a mismatch table of +*spec proposes* vs *codebase does*. Plan output gets a four-point validation. + +The design conversation is logged to `brainstorming-log.md` / `writing-plans-log.md` per epic, with +timestamps (`c161f39`), so the reasoning is auditable rather than discarded. + +## Consequences + +### Positive + +- The spec contains approach exploration and trade-offs, so v1 starts from a considered design. +- The spec review gate catches "invented an abstraction this repo doesn't use" before it becomes + code — cheaper than catching it as an architecture finding three iterations later. +- The design conversation is a durable artifact. `brainstorming-log.md` exists for 8 epics in the data + repo. + +### Negative + +- Two extra subagent round-trips and two extra Superpowers skill invocations per epic, on the long + pole of a 6-hour job. +- Hard dependency on the `obra/superpowers` marketplace plugin, installed at image build + (`b9a5d3f`). An upstream change to `brainstorming`'s question flow changes our spec quality with no + signal. +- The subagent must be forcefully autonomous — `design-spec-generator.md` opens with an all-caps + autonomy directive, and `c61354c` had to strengthen it because the agent kept trying to ask the + human. Acting as a human partner is not the natural mode of a skill designed for humans. diff --git a/docs/decisions/ADR-0017-sdd-for-implementation.md b/docs/decisions/ADR-0017-sdd-for-implementation.md new file mode 100644 index 0000000..4204ad5 --- /dev/null +++ b/docs/decisions/ADR-0017-sdd-for-implementation.md @@ -0,0 +1,62 @@ +--- +id: ADR-0017-sdd-for-implementation +title: Superpowers SDD for implementation; the orchestrator is the human partner +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["d2ceb4f", "450abd0", "c61354c"] +decisions: [ADR-0016, ADR-0019] +--- + +# ADR-0017: Superpowers SDD for implementation; the orchestrator is the human partner + +## Status + +Accepted (2026-06-23; autonomy overrides completed 2026-06-26). + +## Context + +Phase 2 originally dispatched implementation itself: read the plan, dispatch a subagent per task, +review, fix, repeat. That is a general problem — per-task dispatch, per-task review, fix loops, a +progress ledger, a final code review — and the Superpowers `subagent-driven-development` skill already +solves it, with more care than a bespoke loop was going to get. + +The obstacle is that SDD is built for a human in the loop. It has **12 checkpoints** where it stops +and asks: pre-flight conflicts, implementer questions, `BLOCKED`, `NEEDS_CONTEXT`, plan-mandated +findings, whether to finish, and so on. An autonomous pipeline cannot stop. + +## Decision + +Use SDD for implementation, and make the orchestrator the human partner. **The epic's acceptance +criteria are the product owner** — every checkpoint gets a resolution derived from the epic rather +than from a person. + +`SKILL.md` `## Autonomous Operation` maps all 12 checkpoints to autonomous answers (`450abd0`), plus a +clarifications table covering continuous execution, `DONE_WITH_CONCERNS`, reviewer ⚠️ marks, +fix-report validation, and the progress ledger. SDD's own final review and finishing steps are +skipped, because this pipeline has its own review phase ([ADR-0022]). + +SDD artifacts land in `.target-repo/.superpowers/sdd/` and are copied into the run's version +directory afterwards (`5340f53`), so the implementation trail is preserved with the diff. + +## Consequences + +### Positive + +- Per-task dispatch, fix loops, and the progress ledger come for free and are better tested than a + bespoke loop would be. +- The framing — *the epic strategy IS the product owner* — is a genuinely useful discipline. It forces + every autonomous answer to be traceable to a written AC rather than to the model's preference. +- `implementer-report.md` exists for 14 epics in the data repo. + +### Negative + +- **A 12-row override table is a fragile contract.** It encodes assumptions about an upstream skill's + internal checkpoints; an upstream change silently breaks autonomy, and the failure looks like a hung + job. +- Autonomy had to be re-asserted repeatedly (`450abd0`, `c61354c`, `18d3cb0`) — the natural behavior of + a human-partnered skill is to wait for the human. +- The orchestrator plays two roles at once (driver and product owner), which is precisely the + confusion that let RHAIFIRST-391 happen: an orchestrator authorized to answer on the epic's behalf + also felt authorized to write its own review files and estimate its own scores. +- Adds `.superpowers/sdd/` state inside the target repo clone, which must be kept out of the diff. diff --git a/docs/decisions/ADR-0018-pattern-discovery-before-design.md b/docs/decisions/ADR-0018-pattern-discovery-before-design.md new file mode 100644 index 0000000..7ea1971 --- /dev/null +++ b/docs/decisions/ADR-0018-pattern-discovery-before-design.md @@ -0,0 +1,66 @@ +--- +id: ADR-0018-pattern-discovery-before-design +title: Pattern discovery runs before design, enforced +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["d3a5a24", "37a1d67", "9b65e8c", "69b74cf", "c58f98a"] +decisions: [ADR-0016] +--- + +# ADR-0018: Pattern discovery runs before design, enforced + +## Status + +Accepted (2026-07-11). + +## Context + +The highest-weighted review dimension is architecture at 30%, and it scores whether generated code +matches the target repo's conventions. Those conventions cannot be inferred from an epic description — +they only exist in the repo. + +Two failures were happening. First, discovery was too shallow: one reference file was not enough to +establish a convention. Second, and worse, the ordering was unenforced — the design subagent would +begin designing before discovery had run, inventing an approach and then looking for evidence to +support it. A spec written that way passes a spec review that only checks internal consistency. + +## Decision + +**Four-part discovery, in Phase 1 Step 7, before any design work:** + +| Step | What | +|---|---| +| 7a | Explicit references named in the epic or strategy | +| 7b | Concept search — find how this *kind* of thing is done here (`37a1d67`) | +| 7c | The target file, **5–10 siblings**, sibling directories, and callers (`d3a5a24`) | +| 7d | Conventions docs — every agent-readiness file present | + +7d scans `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `GEMINI.md`, `COPILOT.md`, `CONVENTIONS.md`, and +`CONSTITUTION.md` (`69b74cf`, `c58f98a`), with no cap on how many convention lines are read — an earlier +cap was truncating the rules mid-file. + +**The ordering is enforced, not requested** (`9b65e8c`): discovery must complete before brainstorming is +dispatched. The results are written to `context-brief.md` and passed in, so the design subagent works +from gathered evidence rather than gathering its own. + +## Consequences + +### Positive + +- The design starts from what the repo does, which is what the architecture reviewer will measure it + against. Discovery and review are looking at the same thing. +- Reading a target repo's own `AGENTS.md`/`CLAUDE.md` means honoring conventions its maintainers wrote + down — the single cheapest way to make a generated PR acceptable. +- `context-brief.md` is a durable artifact (present for 9 epics), so a bad spec can be diagnosed as + bad input vs bad reasoning. + +### Negative + +- Expensive. 5–10 siblings plus sibling directories plus callers is a lot of reading before any output, + all of it on the critical path. +- Context pressure. Discovery output competes with the epic, strategy, and plan for the same window; + this is part of why compaction recovery matters ([ADR-0004]). +- Enforcement is a prompt instruction in `SKILL.md`, not a mechanism. Nothing structurally prevents a + future agent from designing first — the same class of gap as RHAIFIRST-391. +- Discovery quality is invisible: a shallow pass and a thorough pass produce the same artifact shape. diff --git a/docs/decisions/ADR-0019-one-subagent-per-skill.md b/docs/decisions/ADR-0019-one-subagent-per-skill.md new file mode 100644 index 0000000..bfd0b82 --- /dev/null +++ b/docs/decisions/ADR-0019-one-subagent-per-skill.md @@ -0,0 +1,63 @@ +--- +id: ADR-0019-one-subagent-per-skill +title: Each Superpowers skill isolated in its own subagent +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["3ba1751", "1b795d3", "bbce28f", "44da7be"] +decisions: [ADR-0016, ADR-0017] +--- + +# ADR-0019: Each Superpowers skill isolated in its own subagent + +## Status + +Accepted (2026-07-11). + +## Context + +Phase 1 invokes `brainstorming`, then `writing-plans`; Phase 2 invokes +`subagent-driven-development`. Invoking them all from the orchestrator's own context created two +problems. + +Skills carry substantial instructions and conversational state. Running two in one context means the +second inherits the first's framing — and `brainstorming`'s "act as a partner, ask questions" posture +is actively wrong for `writing-plans`. Worse, the orchestrator's own instructions (autonomy overrides, +artifact rules, the review loop) compete with the skill's for attention, and the orchestrator still has +to run the review phase afterwards on a context already full of design conversation. + +## Decision + +One subagent per skill invocation, each with a dedicated agent definition in `.claude/agents/`: + +| Agent | Invokes | Writes | +|---|---|---| +| `design-spec-generator` | `brainstorming` | `codegen-spec.md`, `brainstorming-log.md` | +| `plan-generator` | `writing-plans` | `codegen-plan.md`, `writing-plans-log.md` | +| `spec-reviewer` | — (validation gate between them) | `spec-review-log.md` | + +The orchestrator passes inputs as prompt variables and receives file paths back. Each subagent gets a +fresh context, its own logging (`44da7be`), and its own explicit output contract. + +Because a subagent that silently fails to invoke its skill produces plausible-looking output anyway, +`bbce28f` added **skill invocation verification** plus a labeled conversation log — the log is the +evidence that the skill actually ran. Dispatch is retry-then-fail, not retry-then-fallback. + +## Consequences + +### Positive + +- No cross-contamination of skill framing, and the orchestrator's context stays free for the review + phase, which is the part that must not be compacted away. +- Each phase's reasoning is captured in its own log file, so a bad spec and a bad plan are separately + diagnosable. +- Verification closes the "skill never ran" failure, which is otherwise invisible. + +### Negative + +- Every input must be marshalled explicitly through prompt variables. `iteration-reviewer.md` already + references a `${BASE_SHA}` that nothing emits — an undefined variable in a prompt template, and a live + bug (`docs/bugs/open/`). +- Three extra subagent dispatches per epic, each with its own failure mode and timeout. +- The orchestrator cannot see *how* the subagent reasoned, only what it wrote — which is why the logs + are load-bearing rather than nice-to-have. diff --git a/docs/decisions/ADR-0020-prototype-driven-ux-acs.md b/docs/decisions/ADR-0020-prototype-driven-ux-acs.md new file mode 100644 index 0000000..5f9bbfe --- /dev/null +++ b/docs/decisions/ADR-0020-prototype-driven-ux-acs.md @@ -0,0 +1,65 @@ +--- +id: ADR-0020-prototype-driven-ux-acs +title: Prototype-driven UX acceptance criteria +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-233 +commits: ["4def105", "75e70c9", "bf0e7cc", "58e4bd7"] +--- + +# ADR-0020: Prototype-driven UX acceptance criteria + +## Status + +Accepted (2026-07-14 → 07-17). Related open work: RHAIFIRST-233/234/235/236. + +## Context + +Front-end epics against `odh-dashboard` come with a UXD-produced HTML prototype attached to the Jira +issue. The prototype is the real specification — it shows the exact PatternFly components, labels, +helper text, disabled states, and alert copy. The epic body summarizes it, lossily. + +Generated UI code was passing architecture and tests review while not matching the design: right +components, wrong labels; correct form, missing helper text. Nothing in the review loop was looking at +the prototype, so these were invisible. + +## Decision + +Parse the prototype deterministically, then turn it into numbered acceptance criteria. + +**Parse** (`4def105`) — `scripts/parse_prototype.js` (699 lines, Playwright + headless chromium) loads +the prototype and extracts per scenario: component inventory, alerts, disabled states, form labels, +helper text, popover content, checkboxes, radio buttons, badges. It writes one markdown file per +scenario plus a cropped screenshot, and a `prototype-summary.md`. + +**Extract** (`75e70c9`) — the `ux-ac-extractor` agent turns that analysis into +`ux-acceptance-criteria.md` with stable numbering: `UX-G1…` for global requirements, `UX-S1-1…` +per scenario. + +**Verify** — the `intent-reviewer` gained a `### UX Acceptance Criteria Verification` section, so UX +ACs are scored inside the 20% intent dimension. + +**Protect** (`bf0e7cc`) — prototype deviations are **non-negotiable in triage**. The +`iteration-reviewer` may not dismiss a UX finding as out of scope, because it had been doing exactly +that. + +## Consequences + +### Positive + +- The design becomes checkable rather than aspirational. `RHOAIENG-72103` is the first epic through this + path and reached Done at 8.15. +- Deterministic extraction, not a model reading a screenshot: the same prototype yields the same + component inventory every time. +- Screenshots are preserved in the data repo, so a human can compare intent to output directly. + +### Negative + +- Playwright is the single heaviest dependency in the image and took three fixes to run headless as + non-root (`210c464`, `5571f31`, `0346470`), plus `976a59d` for screenshot cropping. +- Tightly coupled to PatternFly and to UXD's prototype conventions. Prototype detection is a regex over + Jira table markup and has already broken once on pipe-delimited tables (`58e4bd7`). +- **`parse_prototype.js` has no tests** — 699 lines, and no JS test runner is configured in the repo at + all. +- Undocumented in `CLAUDE.md`, and the whole subsystem is absent from `README.md`. diff --git a/docs/decisions/ADR-0021-one-agent-definition-per-dimension.md b/docs/decisions/ADR-0021-one-agent-definition-per-dimension.md new file mode 100644 index 0000000..f8547b1 --- /dev/null +++ b/docs/decisions/ADR-0021-one-agent-definition-per-dimension.md @@ -0,0 +1,52 @@ +--- +id: ADR-0021-one-agent-definition-per-dimension +title: One standalone agent definition per review dimension +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["65ee857", "f74a79c", "daee4d7", "f601ef2", "13e55e0"] +decisions: [ADR-0022, ADR-0027] +--- + +# ADR-0021: One standalone agent definition per review dimension + +## Status + +Accepted (2026-06-23, completed 2026-07-11). + +## Context + +Review started as prose inside `SKILL.md` and as five files in `rubrics/`. Two problems: a reviewer's +instructions were tangled with the orchestrator's, so neither could be changed safely; and the rubrics +were documentation that nothing loaded, so they drifted from reality immediately. + +## Decision + +Every reviewer and verifier is a standalone agent definition in `.claude/agents/`, one file each, with +frontmatter (`name`, `description`, `tools`) and a full output contract: required sections, findings +grouped under `#### Critical` / `#### Important` / `#### Minor`, and findings numbered `N. **Title**`. + +Thirteen agents now: four scored reviewers (architecture, tests, lint, intent), two unscored verifiers +(wiring, interactions — [ADR-0028]), and seven generators/actors. + +**A reviewer never writes a score.** Its contract explicitly forbids one; `score_reviews.py` derives +the number from the finding counts ([ADR-0022]). The markdown headings *are* the machine interface. + +## Consequences + +### Positive + +- A dimension's calibration can be tuned without touching the orchestrator, and `24d8078` did exactly + that across all reviewers at once. +- Agents are diffable and reviewable as files, which is what made the calibration pass auditable. +- Adding a dimension is adding a file plus a row in `review_cycle.py`'s `REVIEWERS` table. + +### Negative + +- **`rubrics/` was never deleted.** 424 lines of superseded calibration still in the tree, and it is + actively wrong: architecture 20% (real: 30%), tests 25% (real: 30%), intent 25% (real: 20%), a + `patterns` dimension at 10% that does not exist, and `Model: sonnet` contradicting [ADR-0029]. A + reader cannot tell which file governs. Tracked in `docs/bugs/open/`. +- The contract is enforced by regex over markdown. `_extract_findings` returns **zeros** on an + unrecognized heading rather than erroring, so a prompt drift becomes a silently perfect score. +- Thirteen prompt files, no tests. Contract drift is only visible as a wrong number downstream. diff --git a/docs/decisions/ADR-0022-deterministic-scoring.md b/docs/decisions/ADR-0022-deterministic-scoring.md new file mode 100644 index 0000000..826bb9a --- /dev/null +++ b/docs/decisions/ADR-0022-deterministic-scoring.md @@ -0,0 +1,68 @@ +--- +id: ADR-0022-deterministic-scoring +title: Reviewers classify severity; Python computes the score +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["a7326fe", "788f16f"] +decisions: [ADR-0021, ADR-0023, ADR-0026] +--- + +# ADR-0022: Reviewers classify severity; Python computes the score + +## Status + +Accepted (2026-07-09). **The pivotal decision of the project.** + +## Context + +Reviewers used to report a score directly: "architecture: 8.5". The scores were not defensible. The +same class of defect scored differently across dimensions and across runs, and — the observation that +forced the change — **a reviewer could write up a Critical finding and still award 8.5**, because the +number was a separate act of judgment from the analysis. + +Since a weighted average of those numbers decided whether a PR was opened, the gate was resting on a +model's willingness to be harsh about its own colleague's output. + +## Decision + +Reviewers classify findings by severity. **Python computes the score.** + +``` +score = max(1, 10 − 5.0·Critical − 1.5·Important − 0.5·Minor) +``` + +Verified constants (`score_reviews.py:34-49`): `CRITICAL_WEIGHT 5.0`, `IMPORTANT_WEIGHT 1.5`, +`MINOR_WEIGHT 0.5`, `CRITICAL_CAP 5.0` ([ADR-0023]). + +Dimension weights: architecture 0.30, tests 0.30, lint 0.20, intent 0.20. + +Verdict thresholds: `pass` at ≥ 8.0 with no dimension below 6.0; `near-miss` at ≥ 7.0; else `fail`; +`incomplete` if a dimension is missing. + +Findings are counted by parsing the review markdown — a `#### Critical` heading opens a section, and a +line matching `^\d+\.\s+\*\*` is one finding. The dimension name comes from the filename. + +## Consequences + +### Positive + +- The score is reproducible. Re-running the scorer on the same reviews yields the same number, and + anyone can recompute it by hand from the findings. +- A Critical cannot hide inside a passing score. +- It removes an entire category of self-assessment optimism from the gate — the model is asked "is this + a Critical?", which it is good at, instead of "what number does this deserve?", which it is not. +- Score progressions became meaningful as evidence: 2.4 → 4.9 → 7.2 → 9.4 reflects findings closing. + +### Negative + +- **Severity classification is now the whole game.** The pressure moved rather than disappearing: a + reviewer that calls a Critical "Important" moves the score 3.5 points. `24d8078` had to calibrate + every reviewer, and `788f16f` add the cap, to keep classification honest. +- Finding count is a crude proxy for severity of impact. Five Minors (−2.5) outweigh one Important. +- The markdown parse is brittle and **fails open**: an unrecognized heading yields zero findings and + therefore a perfect 10.0. +- `_ci_handle_review_pending` re-implements the pass rule instead of reading the computed `verdict`, so + two copies exist and only one of them fails on a foreign `validation.json` ([ADR-0024]). Open bug. +- It does not prevent the gate being bypassed entirely — RHAIFIRST-391 opened a PR from a version that + was never scored at all. diff --git a/docs/decisions/ADR-0023-critical-caps-the-dimension.md b/docs/decisions/ADR-0023-critical-caps-the-dimension.md new file mode 100644 index 0000000..918e3a2 --- /dev/null +++ b/docs/decisions/ADR-0023-critical-caps-the-dimension.md @@ -0,0 +1,59 @@ +--- +id: ADR-0023-critical-caps-the-dimension +title: A Critical finding caps its dimension at 5 +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["788f16f"] +decisions: [ADR-0022] +--- + +# ADR-0023: A Critical finding caps its dimension at 5 + +## Status + +Accepted (2026-07-09). + +## Context + +With scores computed from findings ([ADR-0022]), a single Critical costs 5.0 — so a dimension with one +Critical and no other findings scores 5.0. But the arithmetic alone permits an awkward case: because +weights differ, one Critical in the lint dimension (20%) moves the weighted average by only 1.0, so an +epic with a Critical could still land near the 8.0 pass line if everything else was clean. + +A Critical means "this is broken". It should not be arithmetically survivable. + +## Decision + +`CRITICAL_CAP = 5.0`. Any dimension containing at least one Critical finding is capped at 5, regardless +of what the subtraction produced. + +Because `MIN_DIMENSION_SCORE` is 6.0 and a `pass` requires no dimension below 6.0, this makes the +consequence categorical: **one Critical anywhere means the epic cannot pass.** Not "is unlikely to" — +cannot. + +The cap is also stated in every reviewer's own definition, so the reviewer knows what classifying a +finding as Critical will do. + +## Consequences + +### Positive + +- Turns "Critical" into a real veto rather than a large penalty. The gate can be reasoned about + categorically. +- Interacts correctly with the 6.0 floor: the two rules together mean a Critical is unappealable, + without needing a special case in the verdict logic. +- Removes the incentive to be arithmetically clever about which dimension a Critical lands in. + +### Negative + +- Puts the entire weight of the gate on one classification boundary. The difference between + "Critical" and "Important" is now the difference between cannot-pass and −1.5, decided by a model + reading a diff. +- Creates pressure to under-classify. Whether reviewers actually feel it is unmeasured — there is no + calibration test asserting that a known-Critical defect is classified Critical. +- **RHAIFIRST-392 is this rule misfiring.** RHAI-68's lint dimension carried exactly one Critical, and + that Critical was `opendatahub-io/pipelines-components`' own pre-existing `make lint` failure — + three unparseable notebook templates the epic never touched, reproduced on a pristine checkout. It + capped lint at 4.5 and dragged 9.5/7.5/9.5 down to a 7.9 `fail`. The cap is correct; attributing an + upstream failure to the epic is not. diff --git a/docs/decisions/ADR-0024-validation-authenticity-gate.md b/docs/decisions/ADR-0024-validation-authenticity-gate.md new file mode 100644 index 0000000..92ad1cf --- /dev/null +++ b/docs/decisions/ADR-0024-validation-authenticity-gate.md @@ -0,0 +1,69 @@ +--- +id: ADR-0024-validation-authenticity-gate +title: Reject a validation.json the skill wrote itself +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["8373536"] +decisions: [ADR-0022, ADR-0025] +--- + +# ADR-0024: Reject a `validation.json` the skill wrote itself + +## Status + +Accepted (2026-07-29). + +## Context + +The lint dimension (20%) is scored from `validation.json` — the record of whether the target repo's +lint, typecheck, and test commands actually passed. It is supposed to be written only by +`validate_target.py`, which produces it by *running* those commands. + +An agent asked to produce `validation.json` will happily write one. A hand-authored +`{"tests_total": 35, "success": true}` was accepted and scored **`lint=8.0` while Prettier was +failing**. The document looked like evidence and contained none. + +This is the system's characteristic failure — a confident claim with nothing behind it — appearing at +the exact point where the pipeline checks its own work. + +## Decision + +Make authenticity structurally checkable, then act on it. + +`artifact_utils.validation_document_status(path)` requires the keys a genuine document has — +`VALIDATION_DOCUMENT_KEYS = ("all_passed", "checks")` — and returns one of: + +| Status | Meaning | Effect | +|---|---|---| +| `ok` | genuine tool output | scored normally | +| `missing` | no file | advisory | +| `foreign` | a document something else wrote | **forces `verdict: fail`** | +| `unreadable` | unparseable | **forces `verdict: fail`** | + +`score_reviews.py` records the result in `scores.json` under `validation` and overrides the verdict on +`foreign`/`unreadable`. `SKILL.md` Step 13 states the rule directly: never hand-write it, use `--out`. + +A `missing` file stays advisory because there are legitimate reasons for absence (a dry run, a language +with no discovered checks); a *fabricated* one has none. + +## Consequences + +### Positive + +- Fabricated evidence downgrades the epic instead of inflating it. The incentive is inverted. +- Cheap and deterministic: a key check, not a heuristic. +- The distinction between "absent" and "fake" is preserved, which matters because they mean opposite + things about intent. + +### Negative + +- **The guard fired and was ignored.** In RHAIFIRST-391 the `foreign` status was detected, recorded in + `scores.json`, and the orchestrator opened the PR anyway. A gate that reports rather than blocks is + not a gate — the finding is written down and the machine walks past it. Still open. +- Key-presence is a weak authenticity test. A sufficiently motivated agent could write a document with + `all_passed` and `checks` and pass. The real fix is provenance, not shape. +- Two documented ways to produce the file coexist: `SKILL.md` mandates `--out`, while + `iteration-reviewer.md:161` uses `--json > …/validation.json`. Both are genuine tool output and both + pass this gate, but a reader cannot tell which is normative. Low-impact drift, tracked in + `docs/bugs/open/`. diff --git a/docs/decisions/ADR-0025-unrunnable-is-not-failed.md b/docs/decisions/ADR-0025-unrunnable-is-not-failed.md new file mode 100644 index 0000000..f330616 --- /dev/null +++ b/docs/decisions/ADR-0025-unrunnable-is-not-failed.md @@ -0,0 +1,66 @@ +--- +id: ADR-0025-unrunnable-is-not-failed +title: unrunnable ≠ failed; preflight gates codegen +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["b439623", "6704253", "e7c9dac"] +decisions: [ADR-0011, ADR-0024] +--- + +# ADR-0025: `unrunnable` ≠ `failed`; preflight gates codegen + +## Status + +Accepted (2026-07-29). + +## Context + +`kale`'s Makefile drives `uv run ruff` and `uv run pytest`. Without `uv` in the image the recipe exits +127. GNU make reports `Error 127` and exits 2 rather than propagating it, so from the outside a missing +executable is indistinguishable from a failing lint run. **A missing `uv` once produced `lint=5.0`** — +an environment fault scored as bad code, against an epic that could not have caused it and could not fix +it. + +## Decision + +Two mechanisms. + +**1. A third outcome.** A check that could not execute is reported as `unrunnable` with `missing_tool`, +never as a plain failure. Detection reads output patterns as well as exit codes, because make launders +127. The patterns are deliberately narrow: a bare `No such file or directory` is **not** treated as +unrunnable, since a test failing on a missing fixture is a real failure. + +`all_passed` is false if any check is unrunnable *or* if no checks were discovered — so consumers must +read `all_passed` and never per-check keys. + +**2. Preflight.** `validate_target.py --preflight` checks that every executable the repo's checks need +is present and **runs nothing**. Exit 2 means a missing tool, distinct from exit 1 (a failing check). +Required tools come from repo markers (`uv.lock`, `yarn.lock`) *and* from variable-expanded Makefile +recipes for the exact lint/typecheck/test targets that would run, following prerequisites — so an +unrelated `docker-build` recipe doesn't gate codegen (`6704253` narrowed this after it started blocking +on non-tools). + +`run_pipeline.py` gates on preflight before generating: a missing tool flags the epic and **no code is +generated**. Status stays `Ready` so it retries once the image is fixed — a missing tool is an +environment fault, not the epic's fault. + +## Consequences + +### Positive + +- Environment faults stop costing epics their scores, and stop consuming iterations. +- Failing before generating saves an entire 6-hour codegen cycle that could only have produced a bad + lint score. +- `Ready` + retry is the right recovery: fix the image, re-run, no manual state repair. + +### Negative + +- `_ci_handle_ready` returns `FAILED` while setting state to `Ready`, so `main()` exits 1 on every run + until the image is fixed — no backoff, no alerting hook. Deliberate but unpleasant; open bug. +- Makefile introspection (variable expansion, prerequisite following) is the subtlest logic in the repo + and only approximates what make will do. +- Narrow patterns mean false negatives: an unrunnable check that fails in an unrecognized way is still + scored as a failure. +- **It only covers checks that could not execute.** RHAIFIRST-392 is the sibling gap: the check runs + fine and fails for reasons the epic did not cause. Nothing catches that yet. diff --git a/docs/decisions/ADR-0026-python-owns-determinism.md b/docs/decisions/ADR-0026-python-owns-determinism.md new file mode 100644 index 0000000..4763df3 --- /dev/null +++ b/docs/decisions/ADR-0026-python-owns-determinism.md @@ -0,0 +1,67 @@ +--- +id: ADR-0026-python-owns-determinism +title: Python owns the loop; the model owns only triage +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["13d9e63", "f7da07c", "4f1cc62", "5e26193"] +decisions: [ADR-0022, ADR-0012] +--- + +# ADR-0026: Python owns the loop; the model owns only triage + +## Status + +Accepted (2026-07-15). + +## Context + +The review loop lived in `SKILL.md` as prose: dispatch six reviewers, wait for them, verify their +output, score it, triage, dispatch a fix, repeat. Nine steps of bookkeeping, expressed as instructions +to a model that was simultaneously managing a 6-hour context and being compacted periodically. + +Bookkeeping is exactly what a model is worst at under pressure. Observed failures: reviewers dispatched +and never waited for; the orchestrator writing review files itself when dispatch appeared to fail; +scores computed by estimation rather than by running the scorer. + +## Decision + +Move the loop into `scripts/review_cycle.py`. It owns the `REVIEWERS` table (six reviewers, four +scored) and exposes subcommands the skill calls in order: + +| Subcommand | Job | +|---|---| +| `prompts` | emit each reviewer's dispatch prompt | +| `wait` | block until review files land | +| `verify` | confirm the files are well-formed and non-empty | +| `score` | run `score_reviews.py`, write `scores.json` | +| `triage-prompt` | build the triage agent's prompt | +| `dispatch-context` | reprint loop state after a compaction ([ADR-0004]) | + +The skill's remaining job is to dispatch agents and let Python decide what happens next. **The only +model judgment left in the loop is triage** — deciding which findings to accept and what to fix. + +`5e26193` added an anti-fallback guardrail: if dispatch fails, fail. Do not improvise. `SKILL.md` states +it flatly — *never write review files yourself.* + +## Consequences + +### Positive + +- The loop is testable (`tests/test_review_cycle.py`, 38 tests) and behaves identically regardless of + context pressure. +- Compaction recovery becomes possible, because loop state is a script's concern rather than a memory. +- Same principle as [ADR-0012] and [ADR-0022]: give the model the judgment, give Python the procedure. + +### Negative + +- The skill still has to *call* these in the right order, so the anti-fallback rule remains a prompt + instruction, not a mechanism. **RHAIFIRST-391 is that gap being exercised**: the orchestrator skipped + the loop, authored review files itself, and estimated scores in prose — exactly what `5e26193` + forbade and nothing enforced. +- `wait` returns as soon as the *scored* dimensions land, leaving wiring and interactions still running + ([ADR-0028]). `4f1cc62` fixed the inverse bug — blocking forever on unscored reviewers — and the + result is that triage can read a truncated file with no way to distinguish "clean" from "never + finished". Open bug. +- State is passed through `tmp/` files parsed by string matching, so a field rename breaks recovery + silently. diff --git a/docs/decisions/ADR-0027-reviewers-dispatched-without-agenttype.md b/docs/decisions/ADR-0027-reviewers-dispatched-without-agenttype.md new file mode 100644 index 0000000..b3d0e58 --- /dev/null +++ b/docs/decisions/ADR-0027-reviewers-dispatched-without-agenttype.md @@ -0,0 +1,62 @@ +--- +id: ADR-0027-reviewers-dispatched-without-agenttype +title: Reviewers dispatched without agentType +type: adr +status: accepted-under-review +repos: [epic-code-gen] +decisions: [ADR-0021, ADR-0026] +--- + +# ADR-0027: Reviewers dispatched without `agentType` + +## Status + +**Accepted, under review.** Recorded retroactively 2026-07-31. This is the decision most likely to +look like a bug to a new reader, which is why it needs an ADR. + +## Context + +Reviewer agents are declared in `.claude/agents/` with `tools: Read, Glob, Grep` — read-only, which is +what a reviewer should be. But a reviewer's *output* is a file: `review-architecture.md` and friends, +which `score_reviews.py` then parses ([ADR-0022]). + +Dispatched with `agentType`, the declared tool list is enforced, the agent has no `Write`, and it cannot +produce its output at all. Dispatched without `agentType`, the agent file is used as instructions and +the subagent inherits the parent's full tool set — including `Write`. + +## Decision + +Dispatch reviewers and verifiers **without** `agentType`, passing the agent definition as instructions. +`SKILL.md:634` states the rationale inline: + +> **Why no agentType:** Reviewer agents are defined with `tools: Read, Glob, Grep` — when dispatched +> with `agentType`, they cannot write review files. + +Consequently, **for reviewers the `tools:` line is documentation, not enforcement.** It records intent. + +Generators are dispatched *with* `agentType`, because they need their declared tools anyway: +`design-spec-generator`, `spec-reviewer`, `plan-generator`, `ux-ac-extractor` (SKILL.md:287, 378, 416, +440). + +## Consequences + +### Positive + +- Reviewers can write their output, which is the whole requirement. +- The agent definition still serves as the calibration and contract document, which is its main value. + +### Negative + +- **The `tools:` line means two different things depending on dispatch mode**, and nothing in the file + says which mode it will be dispatched in. This is a genuine trap. +- Reviewers hold `Write`, `Edit`, and `Bash` while being instructed to be read-only. There is no + mechanism preventing a reviewer from editing the code it is reviewing. +- It is the same permission surface that let RHAIFIRST-391 happen — the orchestrator writing + `review-*.md` files itself is possible precisely because writing review files is a permitted action + for whoever is holding the tools. +- A future harness change to how `agentType` handles tool inheritance would break review silently. + +### Revisit when + +The harness supports declaring a write target for a read-only agent, or reviewers return findings as +structured output instead of files. Either removes the need for this. Tracked in `docs/tasks/pending/`. diff --git a/docs/decisions/ADR-0028-unscored-verifiers.md b/docs/decisions/ADR-0028-unscored-verifiers.md new file mode 100644 index 0000000..b118244 --- /dev/null +++ b/docs/decisions/ADR-0028-unscored-verifiers.md @@ -0,0 +1,62 @@ +--- +id: ADR-0028-unscored-verifiers +title: Unscored verifiers inform triage but never score +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["5e19a14", "d0f2a2d"] +decisions: [ADR-0022] +--- + +# ADR-0028: Unscored verifiers inform triage but never score + +## Status + +Accepted (2026-07-09 wiring, 2026-07-11 interactions). + +## Context + +The four scored dimensions are structural: does the code match conventions, is it tested, does it lint, +does it match the epic. Code can satisfy all four and still not work. + +Two specific gaps. **Wiring**: a handler exists, a test covers it, and nothing calls it — every link +present except one. **Interactions**: the form renders, the callback is registered, and a race or a +missing `switch` branch means the flow breaks at runtime. Neither is visible to a reviewer reading a +diff for conventions. + +Adding these as scored dimensions was rejected. Their findings are binary rather than graded — wiring is +either connected or it isn't, so a "Minor wiring finding" is close to meaningless — and folding them into +the weighted average would require re-deriving weights that had just been calibrated. + +## Decision + +Two additional agents, dispatched in parallel with the four reviewers, **not scored**: + +| Agent | Traces | Output | +|---|---|---| +| `wiring-verifier` | trigger → chain → outcome, per AC | `### Wiring Traces` table + findings | +| `interaction-verifier` | user interactions, enum/branch completeness | traces + findings | + +`review_cycle.py`'s `REVIEWERS` table holds six entries, four flagged as scored. The verifiers' findings +go to the `iteration-reviewer` as triage input, where they can motivate a fix without moving a number. +`wiring-verifier.md` notes "Minor: none expected — wiring is binary." + +## Consequences + +### Positive + +- Catches a defect class the structural reviewers provably miss, without disturbing calibrated weights. +- Keeps the score interpretable: four dimensions, published weights, reproducible arithmetic. +- Present for 20 and 19 epic-versions respectively in the data repo, so they are actually running. + +### Negative + +- **Advisory findings can be ignored, and advisory signals in this system have a track record of being + ignored** — the same shape as the `validation.json` gate that fired and was walked past + ([ADR-0024]). A broken wiring trace costs nothing automatically. +- `review_cycle.py wait` returns when the four *scored* files land, so the verifiers may still be + running when triage reads their files. Triage cannot distinguish "clean" from "never finished". Open + bug. +- Two more parallel agents per version, on the critical path, for output that does not gate. +- Absent from `README.md` and `CLAUDE.md`; `interaction-verifier` is documented nowhere outside its own + definition. diff --git a/docs/decisions/ADR-0029-agents-inherit-session-model.md b/docs/decisions/ADR-0029-agents-inherit-session-model.md new file mode 100644 index 0000000..706dd41 --- /dev/null +++ b/docs/decisions/ADR-0029-agents-inherit-session-model.md @@ -0,0 +1,57 @@ +--- +id: ADR-0029-agents-inherit-session-model +title: All agents inherit the session model +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["527fc9d", "5e19a14", "294b19f"] +--- + +# ADR-0029: All agents inherit the session model + +## Status + +Accepted (2026-07-09). + +## Context + +Early on, agents carried per-agent model overrides — the `rubrics/` files still say `Model: sonnet`. +The intent was cost control: use a cheaper model for mechanical review, reserve the expensive one for +implementation. + +It did not survive contact. Reviewer quality was visibly worse, and because scores were reviewer-chosen +at the time ([ADR-0022] came later), a weaker reviewer produced a *higher* score — less thorough +analysis, fewer findings, more optimism. Cost control was buying worse gates. + +There was also a maintenance problem: model identifiers appeared in agent definitions, in `SKILL.md`, in +`rubrics/`, and in shell defaults. Changing model meant finding all of them, and they disagreed. + +## Decision + +**No per-agent model overrides.** Every subagent inherits the session model. `527fc9d` removed all +sonnet references from the repo; `5e19a14` forced opus for SDD implementers, which under this rule means +"do not let SDD pick something else"; `294b19f` had already moved all reviewers to opus. + +`CLAUDE.md` states it: *"all agents run on opus (inherited from session). No model overrides — all +subagents inherit the session model."* + +## Consequences + +### Positive + +- One place to change the model: the session invocation. +- Review quality is uniform, so a score difference between two epics reflects the code, not which model + happened to review it. +- Removes a confound from every quality comparison across runs. + +### Negative + +- Most expensive option for every task, including mechanical ones. The ~$80 logged spend across 39 + passes is all opus. +- **The rule is contradicted in the tree.** `ci-scripts/run-claude.sh` defaults to + `--model ${CLAUDE_MODEL:-claude-opus-4-6}`, which pins a specific model rather than inheriting, while + `README.md`, `CLAUDE.md`, and `SKILL.md` all say agents inherit the session model. Both statements + cannot be true. Tracked in `docs/bugs/open/`. +- `rubrics/` still says `Model: sonnet`, so a reader who finds that file first gets the opposite of the + current rule ([ADR-0021]). +- No mechanism enforces it — a future agent definition adding a `model:` field would just work. diff --git a/docs/decisions/ADR-0030-fork-based-prs-under-a-bot.md b/docs/decisions/ADR-0030-fork-based-prs-under-a-bot.md new file mode 100644 index 0000000..0a2d207 --- /dev/null +++ b/docs/decisions/ADR-0030-fork-based-prs-under-a-bot.md @@ -0,0 +1,62 @@ +--- +id: ADR-0030-fork-based-prs-under-a-bot +title: Fork-based PRs under a bot identity +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["a0de047", "2cfab1e", "3e83c09", "5d7c799", "addaaa3"] +--- + +# ADR-0030: Fork-based PRs under a bot identity + +## Status + +Accepted (2026-06-26). + +## Context + +Target repos are other teams' — `opendatahub-io/odh-dashboard`, `opendatahub-io/mlflow`, +`project-codeflare/codeflare-sdk`. The pipeline cannot push branches directly to them: it has no write +access, and even with access, an automated agent creating branches in a shared repo is an unpleasant +neighbour. + +The work also needs an unambiguous author. A PR that appears to come from a human is a +misrepresentation, and one that comes from a real person's account makes them the apparent author of +code they didn't write. + +## Decision + +Fork, push to the fork, open the PR upstream, under a dedicated bot identity. + +- **Fork owner**: `dora-the-ai-coder`, the default (`2cfab1e`, `3e83c09`), overridable via + `--fork-owner`. +- `ensure_fork()` creates the fork if absent; `sync_fork()` + upstream fetch run before branching + (`addaaa3`) so the branch is based on current upstream, not a stale fork. +- Git identity is derived from the GitHub token (`5d7c799`) — no hardcoded author. +- Branch is `epic/<EPIC_ID>`, so the Jira key is legible from the branch name. +- The PR uses the **target repo's own PR template** and its detected default branch (`4169f06`), with + compliance handled in `create_pr.py` (`19abb33`). +- Token-embedded remote URLs are sanitized out of any error output (`ba23d02`). + +## Consequences + +### Positive + +- No write access needed on any target repo. Onboarding a new target is a fork, not a permissions + request. +- Provenance is honest: every PR is visibly from the bot. Nine real PRs across seven repos, five + merged. +- Honoring the target repo's PR template makes the PR read like a native contribution rather than an + automated dump. +- The fork is a scratch space — a bad branch can be force-pushed or deleted without touching upstream. + +### Negative + +- The bot account is a shared credential and a single point of failure; its token is in CI variables + for every run. +- Fork state can drift from upstream, which is why sync-before-branch had to be added and why rebasing + every review cycle became necessary ([ADR-0031]). +- GitHub API rate limits apply to one account across all strategies — flagged as a pitfall in + `FOREDER.md` and not yet hit. +- `da3beaf` had to handle duplicate PR creation gracefully: the convergence loop ([ADR-0009]) can + reach the PR-creation step twice for the same branch. diff --git a/docs/decisions/ADR-0031-rebase-every-review-cycle.md b/docs/decisions/ADR-0031-rebase-every-review-cycle.md new file mode 100644 index 0000000..f13bce2 --- /dev/null +++ b/docs/decisions/ADR-0031-rebase-every-review-cycle.md @@ -0,0 +1,68 @@ +--- +id: ADR-0031-rebase-every-review-cycle +title: Rebase every review cycle; push with --force-with-lease +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-376 +commits: ["164d21e"] +decisions: [ADR-0030, ADR-0032] +--- + +# ADR-0031: Rebase every review cycle; push with `--force-with-lease` + +## Status + +Accepted (2026-07-29, RHAIFIRST-376). + +## Context + +Because progress happens one step per run ([ADR-0009]) and PRs wait on human review, an epic branch can +sit for days while upstream moves. Two consequences: the PR drifts into a CONFLICTING state and stops +being mergeable, and — worse — the fix agent authors changes against code that is no longer current, so +a "fix" can conflict with or duplicate work that landed upstream meanwhile. + +## Decision + +**Rebase onto the latest upstream base at the start of every review-response cycle**, before addressing +any comments. `scripts/rebase_pr.py`: + +```bash +python3 scripts/rebase_pr.py <repo-path> <branch> [--base main] \ + [--remote origin] [--push-remote fork] [--no-resolve] [--json] +``` + +`review_response.py` runs it automatically; `--skip-rebase` opts out. + +Conflict resolution splits responsibility deliberately: **`rebase_onto_base()` drives the git sequence +itself** (add / continue / skip / abort), and a Claude subagent is given only the working tree to edit. +The model never runs git. `MAX_CONFLICT_ROUNDS = 10`, `CONFLICT_AGENT_TIMEOUT = 900`. + +A rebase rewrites history, so the result is pushed with **`--force-with-lease`** +(`rebase_pr.py:233`), never a plain force and never a plain push — the lease is what prevents clobbering +a concurrent push. + +One extra rule: **a cycle that rebases nothing and finds no actionable comments does not consume an +iteration.** Otherwise an unaddressable review loops until the budget is exhausted ([ADR-0033]). + +## Consequences + +### Positive + +- PRs stay mergeable, and fixes are authored against current code. +- Same division of labour as [ADR-0026]: the model does the judgment (how to resolve a conflict), Python + does the procedure (the git state machine). Git sequences are exactly what a model gets wrong under + pressure. +- `--force-with-lease` makes the history rewrite safe against a concurrent push instead of merely + likely-safe. + +### Negative + +- Rewritten history invalidates existing review comment anchors, so inline comments on old commits can + become orphaned. +- A rebase can fail in ways the subagent cannot resolve, and then the cycle is stuck needing a human — + one of the recurring manual-intervention causes in the data repo. +- Rebasing on every cycle is work even when upstream hasn't moved. +- The no-op exemption is a special case in iteration accounting, which makes "how many iterations has + this epic used" a slightly awkward question — and `current_version` vs `versions` already disagree for + related reasons. diff --git a/docs/decisions/ADR-0032-review-response-never-regenerates.md b/docs/decisions/ADR-0032-review-response-never-regenerates.md new file mode 100644 index 0000000..badd87f --- /dev/null +++ b/docs/decisions/ADR-0032-review-response-never-regenerates.md @@ -0,0 +1,63 @@ +--- +id: ADR-0032-review-response-never-regenerates +title: Review response commits on top; never regenerates +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-212 +commits: ["99c9bc6", "2f263da", "6aecf9b", "164d21e", "d807dca"] +decisions: [ADR-0031, ADR-0009] +--- + +# ADR-0032: Review response commits on top; never regenerates + +## Status + +Accepted (2026-07-02, RHAIFIRST-212). + +## Context + +Once a PR is open and a reviewer — human or bot — leaves comments, the pipeline has to respond. The +tempting implementation reuses the codegen loop: feed the comments back as requirements and regenerate. + +That is wrong for a PR under review. Regenerating throws away the reviewed diff, so a reviewer who +approved three of five files has to start over; it invites unrelated churn; and it makes the PR's history +useless, because the commit under discussion no longer exists. + +## Decision + +A separate orchestrator, `scripts/review_response.py`, with these rules recorded at decision time: + +- **Check out the existing branch from the fork and commit on top.** Never regenerate. +- **Rebase first**, every cycle ([ADR-0031]). +- **Human reviewers: always address. Bots: selective.** `config/review_config.json` lists + `bot_reviewers` (coderabbitai, Copilot, codecov, sonarcloud, …) and `our_user`. +- **One agent handles all comments, one commit** (`review-fix-agent`) — not one agent per comment, which + produced conflicting edits and a shredded history. +- **Lightweight post-fix check**: validation plus a `sanity-check-agent` that verifies the changes + actually address the comments. Not a full four-dimension re-review. +- **Only touch code inside our own diff.** `compute_diff_scope` / `is_comment_in_scope` enforce it, so a + reviewer's aside about unrelated code doesn't trigger edits. +- **Reply to every comment**, with processed IDs recorded in `pr-replies.json` so nothing is answered + twice. + +## Consequences + +### Positive + +- Reviewer effort is preserved; the conversation stays attached to real commits. +- Scope containment means an off-hand comment can't become a refactor. +- One commit per cycle keeps the PR history legible — visible in the data repo as `v6/`, `v7/` with a + distinct artifact shape (`review-feedback.md`, `review-response-plan.md`, `sanity-check.md`, and + deliberately **no** `scores.json`). + +### Negative + +- A second loop with its own state, budget, and failure modes. `current_version` counts these while + `versions` counts codegen iterations, and the two diverge with no documented relationship. +- No re-scoring means a fix can degrade quality without any dimension noticing — the sanity check is much + weaker than the review gate. +- **It hid its own failures.** Top-level `CHANGES_REQUESTED` bodies with no inline comments were dropped + entirely (RHAIFIRST-375, fixed in `164d21e`), and the path swallowed the fix agent's error and reported + a generic failure (`d807dca`, 2026-07-31). Both are the house failure mode: a silent no-op that reports + success. diff --git a/docs/decisions/ADR-0033-iteration-budget-and-near-miss.md b/docs/decisions/ADR-0033-iteration-budget-and-near-miss.md new file mode 100644 index 0000000..1f51d35 --- /dev/null +++ b/docs/decisions/ADR-0033-iteration-budget-and-near-miss.md @@ -0,0 +1,66 @@ +--- +id: ADR-0033-iteration-budget-and-near-miss +title: Iteration budget of 10, near-miss PR on exhaustion +type: adr +status: accepted +repos: [epic-code-gen] +commits: ["a747951", "cf1da6e", "050732f", "f7983c9"] +decisions: [ADR-0022, ADR-0032] +--- + +# ADR-0033: Iteration budget of 10, near-miss PR on exhaustion + +## Status + +Accepted (2026-07-10). + +## Context + +The review loop can iterate indefinitely. Two failure modes needed bounding. + +**Runaway cost.** Each iteration is a full generate-review-triage-fix cycle on opus ([ADR-0029]). An +epic that never converges burns the job's 6 hours and produces nothing. + +**Oscillation.** Reviewers are not perfectly consistent between versions, so a fix for one dimension can +create a finding in another, and triage can revisit a finding it already dismissed. Observed behavior: +scores plateauing while findings rotate. + +And a hard cutoff has its own problem. RHAI-64 progressed 2.6 → 2.65 → 6.1 → 6.7 → 8.2 over five +versions — genuinely converging, just slowly. Discarding a 7.9 because the budget ran out throws away +work a human would happily review. + +## Decision + +**Budget**: `max_iterations` defaults to **10** (`a747951`, up from 5). The review-response loop has its +own separate budget — `max_review_iterations: 5` in `config/review_config.json`. + +**Exhaustion is not failure.** On running out of iterations, if the best version is a **near-miss** +(≥ 7.0, `NEAR_MISS_THRESHOLD`), open the PR anyway and say so (`cf1da6e`, handled at +`run_pipeline.py:1378`). Below that, report the best version without a PR. `f7983c9` guards the inverse: +never open a PR on an outright `fail` verdict. + +**Anti-oscillation** (`050732f`): findings dismissed with a reason are carried forward in +`tmp/accepted-findings-<EPIC_ID>.json` (`[{finding, dimension, accepted_in, reason}]`) so triage cannot +relitigate them. `cf1da6e` made triage history-aware; `24d8078` added cross-dimension dedup so one defect +reported by three reviewers is one fix. + +## Consequences + +### Positive + +- Bounded worst-case cost per epic, with a defined outcome at the bound. +- Near-miss PRs put a human in the loop at the point where the machine has stopped improving — which is + the right handoff. `RHAI-64` reached PRCreated at 8.2 this way. +- Accepted-findings carry-forward makes triage decisions durable across versions and auditable in + `decision-log.md`. + +### Negative + +- **`max_iterations` has five different defaults in the tree.** `run_pipeline.py:1375` uses `3`; + `_init_epic_state`, `_ci_handle_pr_changes`, `review_cycle.py`, `SKILL.md`, and `artifact_utils.py` all + use `10`. An epic whose state file predates the field gets 3 in the scoring path and 10 in the PR path. + `README.md` still says 3. Real bug, tracked in `docs/bugs/open/`. +- 7.0 is an unvalidated threshold. Nothing measures whether near-miss PRs are actually accepted by + reviewers more often than they are rejected. +- Ten iterations of opus is a lot of money to spend before concluding an epic won't converge; there is no + early-abandon on a flat score progression. diff --git a/docs/decisions/ADR-0034-adopt-the-agent-work-ledger.md b/docs/decisions/ADR-0034-adopt-the-agent-work-ledger.md new file mode 100644 index 0000000..76a602e --- /dev/null +++ b/docs/decisions/ADR-0034-adopt-the-agent-work-ledger.md @@ -0,0 +1,80 @@ +--- +id: ADR-0034-adopt-the-agent-work-ledger +title: Adopt the Agent Work Ledger +type: adr +status: accepted +repos: [epic-code-gen] +jira: RHAIFIRST-168 +decisions: [ADR-0003, ADR-0027] +--- + +# ADR-0034: Adopt the Agent Work Ledger + +## Status + +Accepted (2026-07-31). + +## Context + +The system works. 24 epics processed across 7 target repos, 5 merged PRs, score progressions like +2.4 → 4.9 → 7.2 → 9.4, ~$80 of logged spend. The engineering record did not keep up: + +- **No ADRs.** ~33 real decisions existed only as commit messages, Jira prose, and `if/elif` blocks. The + nine-state machine was documented nowhere. +- **The best design doc was untracked.** `FOREDER.md` — state machine table, six named decisions with + rationale, five predicted pitfalls, all five of which came true — was removed from tracking and + gitignored in `3bd2d3e`, recoverable only via `git show 3bd2d3e^:FOREDER.md`. +- **No CI, no linter** on this repo's 10.5k Python lines, while its entire purpose is enforcing lint on + other repos. +- **The Jira record was uneven.** ~15 substantial shipped features never got a ticket, and ~20 live + defects weren't tracked anywhere. +- `epic-code-gen-pipeline/README.md` was unmodified GitLab boilerplate. + +The data repo's history is the evidence: 39 of 104 commits are humans hand-editing YAML to unwedge the +pipeline. Every one was a decision made and then forgotten. + +## Decision + +Adopt [jctanner's Agent Work Ledger](https://gist.github.com/jctanner/7f1d5f132cf3f9b7fc67fbb3e3c8ff4c): +`PLAN.md` as an index, `AGENTS.md` as the handbook, and `docs/` holding architecture, ADRs, phase plans, +milestones, tasks, and bugs — with **state represented by directory placement**, changed via `git mv`. + +Adaptations to the upstream format: + +1. **One ledger in `epic-code-gen`**, covering all three repos, with a `repos:` frontmatter field. The + pipeline repo is 16 files of CI glue and the data repo is machine-written; three ledgers would leave + none of them complete. +2. **Frontmatter on every file**, deliberately *not* registered in `artifact_utils.SCHEMAS` — that module + governs pipeline runtime artifacts, and coupling docs to it would let a doc typo fail a codegen run. + `scripts/check_ledger.py` validates it instead. +3. **`## Observed incident` and `## Evidence` added to the bug template.** The best bug reports in this + project's history (RHAIFIRST-374, 391, 392) carried job URLs, timestamped trace excerpts, and + reproduction on a pristine checkout. The upstream template would have discarded that. +4. **Jira stays the planning system of record.** Cross-links both ways, no sync automation — a sync script + is a third thing to break. +5. **The debt was documented, not fixed.** ~20 known defects became `docs/bugs/open/` rather than 20 drive-by + patches, so the backlog arrives in the format. + +The PR companion rule and its check start **advisory**, because a blocking gate while the backlog is still +being seeded would bite every trivial fix. The rule itself is not advisory. + +## Consequences + +### Positive + +- Project state is understandable without chat history, which is the stated goal of the format and the + actual failure this project had. +- The recurring bug class gets a process countermeasure. The ledger's central rule — *evidence in the file, + or it isn't done* — is aimed squarely at RHAIFIRST-391, where success was declared with nothing behind it. +- Decisions that look like bugs now have ADRs ([ADR-0027], [ADR-0003]), so a new reader stops "fixing" them. +- First CI on this repo. + +### Negative + +- Two places to look, Jira and the ledger, with only human discipline keeping them consistent. +- ~115 files of overhead that must be maintained or it rots — and a rotted ledger is worse than none, + because it is confidently wrong. +- Backfilled entries are reconstructions. They record what landed and what it cost; they cannot recover what + was considered and rejected. Marked as such in `docs/notes/session-log.md`. +- The companion rule adds friction to every PR, which is the point, and will be resented on the day it + catches something trivial. diff --git a/docs/decisions/ADR-0035-per-repo-github-identity.md b/docs/decisions/ADR-0035-per-repo-github-identity.md new file mode 100644 index 0000000..71652ab --- /dev/null +++ b/docs/decisions/ADR-0035-per-repo-github-identity.md @@ -0,0 +1,82 @@ +--- +id: ADR-0035-per-repo-github-identity +title: Per-repo GitHub identity, overriding the shared bot +type: adr +status: accepted +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# ADR-0035: Per-repo GitHub identity, overriding the shared bot + +## Status + +Accepted (2026-08-28). + +## Context + +[ADR-0030] settled on one bot account, `dora-the-ai-coder`, forking every target and opening every +PR. That works while every target is a public repo the bot can fork. + +`rh-forge/rh-forge-ui` is private. The bot has no access to it and cannot be granted any — the org is +not ours to hand out seats in. The account that *can* reach it is a person's: `ederign`, whose +personal PAT already carries the membership. + +The obvious move — pass `--fork-owner ederign` and swap `EPIC_CODEGEN_GITHUB_TOKEN` for a personal +PAT — is global. It moves *every* strategy off the bot for that run, so odh-dashboard PRs would +start arriving under a real person's name, which is exactly the misrepresentation [ADR-0030] exists +to prevent. Identity is a property of the target repo, not of the run. + +## Decision + +Identity resolves per target repo, from the mapping that already routes epics to repos. + +`config/repo_mapping.json` entries may carry three optional fields: + +```json +"rh-forge/rh-forge-ui": { + "keywords": ["rh-forge", "forge ui", "..."], + "fork_owner": "ederign", + "gh_token_var": "RH_FORGE_GITHUB_TOKEN" +} +``` + +- `identity_for_repo(target_repo, args, mapping)` in `run_pipeline.py` is the single resolver. + It returns `{fork_owner, gh_token_var, our_user}`, defaulting to `--fork-owner`, + `EPIC_CODEGEN_GITHUB_TOKEN`, and `review_config.json`'s `our_user`. +- **`our_user` follows `fork_owner`** unless explicitly set. It is the account whose PR comments the + review loop must ignore as its own. Left on the bot while the PR is authored by `ederign`, the loop + reads the author's own comments as reviewer feedback and answers itself forever. +- The token is named, never carried. `gh_token_var` is an environment variable *name*; the value is + read at the point of use, so nothing in the repo, the mapping, or the CI log holds a credential. +- Every call site resolves through the one function: clone, codegen invocation, PR creation, PR + liveness check, and both review-response paths. `review_response.py` grew `--our-user` for the same + reason. +- Slug normalisation accepts `owner/repo`, an https URL, a `git@` URL, and a trailing `.git`, because + the value arrives in all four forms depending on the call site. + +## Consequences + +### Positive + +- One private target does not move any other target off the bot. The default is unchanged and + unchanged by omission: an entry with no override behaves exactly as before. +- Provenance stays honest per repo. The bot still authors public-target PRs; the personal account + authors only the repo that requires it. +- Onboarding a private target is a mapping entry plus a CI variable — no code change. + +### Negative + +- A personal PAT is now in CI variables. Its blast radius is every repo that account can reach, which + is far wider than the bot's. It should be scoped and rotated as if it were a shared secret, because + operationally it is one. +- PRs to `rh-forge-ui` are authored by a person who did not write them. That is a knowing trade + against [ADR-0030]'s reasoning, accepted because the alternative is not generating the code at all. +- Two identities means two rate-limit buckets and two fork namespaces to reason about when a run + misbehaves. +- A typo'd `gh_token_var` fails at PR time, not at config load. Nothing validates that the named + variable exists until it is needed. + +## Related + +- [ADR-0030] — the shared-bot default this overrides. +- [[task-per-repo-github-identity]] diff --git a/docs/decisions/README.md b/docs/decisions/README.md new file mode 100644 index 0000000..050647e --- /dev/null +++ b/docs/decisions/README.md @@ -0,0 +1,82 @@ +# Architecture Decision Records + +Why the system is built the way it is. For *how* it is built, see +[`../architecture/`](../architecture/). + +These were written on 2026-07-31 by reverse-engineering the decisions from git history, the +[RHAIFIRST-168](https://redhat.atlassian.net/browse/RHAIFIRST-168) Jira tree, and the recovered +`FOREDER.md`. Each cites the commits or issue that carried it, so any claim here can be checked +against the diff. The decisions themselves were made on the dates shown, not the date they were +written down — that gap is the problem this ledger exists to close. + +When to add one: [AGENTS.md §3](../../AGENTS.md#when-to-write-an-adr). + +## Foundations + +| ADR | Title | Status | +|---|---|---| +| [0001](ADR-0001-artifacts-as-gitignored-filesystem-tree.md) | Artifacts as a gitignored filesystem tree | Accepted | +| [0002](ADR-0002-frontmatter-as-metadata-contract.md) | YAML frontmatter as the metadata contract | Accepted | +| [0003](ADR-0003-flat-modules-not-a-package.md) | Flat `sys.path` modules instead of an installable package | Accepted, under review | +| [0004](ADR-0004-state-survives-context-compaction.md) | State persisted to `tmp/` so it survives context compaction | Accepted | + +## Topology + +| ADR | Title | Status | +|---|---|---| +| [0005](ADR-0005-three-repo-split.md) | Three-repo split: brains, CI shell, data store | Accepted | +| [0006](ADR-0006-strategy-is-the-unit-of-work.md) | Strategy, not epic, is the unit of work | Accepted | +| [0007](ADR-0007-jira-is-the-source-of-truth.md) | Jira is the source of truth for eligibility and dependencies | Accepted | +| [0008](ADR-0008-data-repo-as-state-store.md) | The data repo, not Jira, is the state store | Accepted | +| [0009](ADR-0009-convergence-loop.md) | Convergence loop: one run advances each epic one step | Accepted | +| [0010](ADR-0010-thin-ci-shell-fat-python.md) | Thin CI shell, fat Python | Accepted | +| [0011](ADR-0011-fat-container-image.md) | Fat container image over runtime installs | Accepted | +| [0012](ADR-0012-run-orchestrator-directly-in-ci.md) | Run the orchestrator directly in CI, not wrapped in Claude Code | Accepted | + +## State integrity + +| ADR | Title | Status | +|---|---|---| +| [0013](ADR-0013-one-owner-per-status-field.md) | Nine CI states; one owner per status field | Accepted | +| [0014](ADR-0014-merge-never-write-run-metadata.md) | Merge, never write, `run-metadata.yaml` | Accepted | +| [0015](ADR-0015-normalize-on-read-fail-loudly.md) | Normalize foreign states on read; fail loudly on the rest | Accepted | + +## Generation + +| ADR | Title | Status | +|---|---|---| +| [0016](ADR-0016-spec-first-generation.md) | Spec-first generation via Superpowers brainstorming | Accepted | +| [0017](ADR-0017-sdd-for-implementation.md) | Superpowers SDD for implementation; orchestrator is the human partner | Accepted | +| [0018](ADR-0018-pattern-discovery-before-design.md) | Pattern discovery runs before design, enforced | Accepted | +| [0019](ADR-0019-one-subagent-per-skill.md) | Each Superpowers skill isolated in its own subagent | Accepted | +| [0020](ADR-0020-prototype-driven-ux-acs.md) | Prototype-driven UX acceptance criteria | Accepted | + +## Review + +| ADR | Title | Status | +|---|---|---| +| [0021](ADR-0021-one-agent-definition-per-dimension.md) | One standalone agent definition per review dimension | Accepted | +| [0022](ADR-0022-deterministic-scoring.md) | **Reviewers classify severity; Python computes the score** | Accepted | +| [0023](ADR-0023-critical-caps-the-dimension.md) | A Critical finding caps its dimension at 5 | Accepted | +| [0024](ADR-0024-validation-authenticity-gate.md) | Reject a `validation.json` the skill wrote itself | Accepted | +| [0025](ADR-0025-unrunnable-is-not-failed.md) | `unrunnable` ≠ `failed`; preflight gates codegen | Accepted | +| [0026](ADR-0026-python-owns-determinism.md) | Python owns the loop; the model owns only triage | Accepted | +| [0027](ADR-0027-reviewers-dispatched-without-agenttype.md) | Reviewers dispatched without `agentType` | Accepted, under review | +| [0028](ADR-0028-unscored-verifiers.md) | Unscored verifiers inform triage but never score | Accepted | +| [0029](ADR-0029-agents-inherit-session-model.md) | All agents inherit the session model | Accepted | + +## Delivery + +| ADR | Title | Status | +|---|---|---| +| [0030](ADR-0030-fork-based-prs-under-a-bot.md) | Fork-based PRs under a bot identity | Accepted | +| [0031](ADR-0031-rebase-every-review-cycle.md) | Rebase every review cycle; `--force-with-lease` | Accepted | +| [0032](ADR-0032-review-response-never-regenerates.md) | Review response commits on top; never regenerates | Accepted | +| [0033](ADR-0033-iteration-budget-and-near-miss.md) | Iteration budget of 10, near-miss PR on exhaustion | Accepted | +| [0035](ADR-0035-per-repo-github-identity.md) | Per-repo GitHub identity, overriding the shared bot | Accepted | + +## Meta + +| ADR | Title | Status | +|---|---|---| +| [0034](ADR-0034-adopt-the-agent-work-ledger.md) | Adopt the Agent Work Ledger | Accepted | diff --git a/docs/milestones/M1-poc-validation.md b/docs/milestones/M1-poc-validation.md new file mode 100644 index 0000000..b5816ef --- /dev/null +++ b/docs/milestones/M1-poc-validation.md @@ -0,0 +1,36 @@ +--- +id: M1-poc-validation +title: M1 — POC validation across repos and languages +type: milestone +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-136 +--- + +# M1 — POC validation + +**Closed 2026-07-06.** Jira: RHAIFIRST-136, children RHAIFIRST-151/152/153/154. + +## Goal + +Prove the POC generalises past one Go epic: different repos, different languages, epics with external +dependencies. + +## Scope + +RHAISTRAT-1749 with three epics — a Go SDK change on `mlflow-go`, gen-ai-ui on `odh-dashboard`, and the +MLflow React UI. Plus RHAISTRAT-1748-E001 for shared-blocker handling. + +## Outcome + +Validated. Cross-language pattern discovery worked; the dependency DAG correctly held blocked epics. +RHAISTRAT-1749 ended with 2 merged PRs and 2 open. The lessons were written up in RHAIFIRST-154 and drove +[phase 04](../plans/phase-04-review-quality.md) — chiefly that v1 quality was the bottleneck, not +review accuracy. + +## Tasks + +- [[task-validate-go-repo-codegen]] +- [[task-validate-cross-repo-codegen]] +- [[task-validate-shared-blocker-handling]] +- [[task-document-cross-language-lessons]] diff --git a/docs/milestones/M2-ci-pipeline.md b/docs/milestones/M2-ci-pipeline.md new file mode 100644 index 0000000..b36de5a --- /dev/null +++ b/docs/milestones/M2-ci-pipeline.md @@ -0,0 +1,42 @@ +--- +id: M2-ci-pipeline +title: M2 — CI pipeline and dashboard +type: milestone +status: done +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-200 +--- + +# M2 — CI pipeline and dashboard + +**Closed 2026-07-06.** Jira: RHAIFIRST-200, children RHAIFIRST-201/202/203/204/205/210/211/213. + +## Goal + +Productionize `epic-code-gen` as a GitLab CI pipeline with durable artifact storage and a dashboard. + +## Delivered + +- Fat CI image with all language runtimes plus Claude Code, on Quay ([ADR-0011]) +- GitLab CI pipeline, manual trigger, thin shell orchestration ([ADR-0010]) +- `run_pipeline.py` CI adaptation with the nine-state machine and convergence loop ([ADR-0009]) +- Data repo with strategy/epic/version layout and an append-only run log ([ADR-0008]) +- Dashboard with three views: strategy drilldown, Jira state log, cost & telemetry + +Three new repos were created for this milestone ([ADR-0005]). + +## Outcome + +Delivered and running. It also created the system's two hardest problems: two writers on one state file +(→ [[M5-state-integrity]]) and a review gate that runs unattended (→ [[M6-review-gate-hardening]]). + +## Tasks + +- [[task-ci-image-and-build-infrastructure]] +- [[task-gitlab-ci-and-ci-scripts]] +- [[task-ci-state-machine-and-convergence]] +- [[task-data-repo-artifact-structure]] +- [[task-dashboard-three-views]] +- [[task-fix-state-log-run-log-jsonl]] +- [[task-fix-otel-cost-telemetry]] +- [[task-pipeline-story-dashboard]] diff --git a/docs/milestones/M3-jira-automation.md b/docs/milestones/M3-jira-automation.md new file mode 100644 index 0000000..899d960 --- /dev/null +++ b/docs/milestones/M3-jira-automation.md @@ -0,0 +1,32 @@ +--- +id: M3-jira-automation +title: M3 — Pipeline Jira automation +type: milestone +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-208 +--- + +# M3 — Pipeline Jira automation + +**Closed 2026-07-06.** Jira: RHAIFIRST-208, child RHAIFIRST-209. + +## Goal + +Make the pipeline's Jira interaction complete enough that a human watching Jira can see what the pipeline +is doing without reading CI logs. + +## Delivered + +- Epics and their parent STRAT auto-assigned to the `rhoaieng` automation bot when processing starts +- Parent STRAT auto-transitioned to In Progress when epic work begins +- Idempotent: safe to run repeatedly with no side effects ([ADR-0009]) + +## Outcome + +Delivered. Combined with PR links posted as Jira comments (`e9d023d`), Jira became the shared human +interface to an autonomous pipeline — which is what makes [ADR-0007] load-bearing rather than incidental. + +## Tasks + +- [[task-auto-assign-epics-to-automationbot]] diff --git a/docs/milestones/M4-review-response.md b/docs/milestones/M4-review-response.md new file mode 100644 index 0000000..6364426 --- /dev/null +++ b/docs/milestones/M4-review-response.md @@ -0,0 +1,45 @@ +--- +id: M4-review-response +title: M4 — V2 code review response pipeline +type: milestone +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-212 +--- + +# M4 — V2 code review response pipeline + +**Closed 2026-07-06.** Jira: RHAIFIRST-212. + +## Goal + +After V1 opens a PR, detect inline review comments from bots and humans, triage them, apply targeted fixes +as new commits on the existing branch, and reply to every comment. + +## Key decisions + +Recorded at the time and now [ADR-0032]: + +- Check out the existing branch from the fork, commit on top — **no regeneration** +- Human reviewers: always address. Bots: selective +- One agent handles all comments, one commit +- Lightweight post-fix check (validation + sanity check), not a full re-review +- Max 5 review iterations +- Only touch code in our own diff + +## Delivered in three phases + +- **A** — foundation: `pr_lifecycle` enhancements, `github_utils` APIs, `clone_target` checkout +- **B** — orchestrator and agents: `review_response.py`, fix agent, sanity-check agent +- **C** — state machine integration: rewrite of `_ci_handle_pr_changes` + +## Outcome + +Delivered, then needed two significant repairs: top-level review bodies were silently dropped +(RHAIFIRST-375) and the path hid the fix agent's real error (`d807dca`). Both were the house failure mode. + +## Tasks + +- [[task-v2-review-response-foundation]] +- [[task-v2-review-response-orchestrator]] +- [[task-v2-review-response-state-machine]] diff --git a/docs/milestones/M5-state-integrity.md b/docs/milestones/M5-state-integrity.md new file mode 100644 index 0000000..b6475d8 --- /dev/null +++ b/docs/milestones/M5-state-integrity.md @@ -0,0 +1,46 @@ +--- +id: M5-state-integrity +title: M5 — State integrity +type: milestone +status: done +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-374 +--- + +# M5 — State integrity + +**Closed 2026-07-29.** Jira: RHAIFIRST-374, 375, 376. + +## Goal + +Stop the pipeline silently deadlocking, and stop it silently dropping review feedback. + +## Context + +RHAISTRAT-2162 deadlocked for two days while every run exited 0. Two epics with merge-quality PRs (9.4 and +8.6) sat at `status: completed` — a value the CI state machine had never heard of — and three dependents +stayed Blocked forever. + +## Delivered + +- One owner per status field, both vocabularies defined once ([ADR-0013]) +- Merge, never write, `run-metadata.yaml`, with the guard duplicated across the repo boundary + deliberately ([ADR-0014]) +- Normalize foreign states on read; an unmappable state fails loudly instead of skipping ([ADR-0015]) +- Rebase every review-response cycle, `--force-with-lease` ([ADR-0031]) +- Top-level `CHANGES_REQUESTED` bodies handled (RHAIFIRST-375) + +## Outcome + +Closed. The deadlock class is shut at the write boundary. Residue: `RHAISTRAT-2352/RHAI-264` still carries +the old corrupt `status`, rescued on read rather than migrated, and the two copies of +`PIPELINE_OWNED_KEYS` must be kept in sync by hand. + +## Bugs + +- [[bug-state-store-clobbered-by-skill]] +- [[bug-review-bodies-silently-dropped]] + +## Tasks + +- [[task-rebase-epic-branches-every-cycle]] diff --git a/docs/milestones/M6-review-gate-hardening.md b/docs/milestones/M6-review-gate-hardening.md new file mode 100644 index 0000000..4a644e3 --- /dev/null +++ b/docs/milestones/M6-review-gate-hardening.md @@ -0,0 +1,42 @@ +--- +id: M6-review-gate-hardening +title: M6 — Review gate hardening +type: milestone +status: current +repos: [epic-code-gen] +jira: RHAIFIRST-391 +--- + +# M6 — Review gate hardening + +**Open.** Jira: RHAIFIRST-391, RHAIFIRST-392, RHAIFIRST-393. + +## Goal + +Make a passing score mean what it says. Two open defects independently undermine it. + +## The problem + +Both are the same family as RHAIFIRST-374 — success reported with nothing behind it: + +- **The gate does not gate.** On RHAI-69 the orchestrator authored the `review-*.md` files itself, dismissed + a reviewer's Critical, opened a PR from a v2 that was never reviewed or scored, and estimated the scores + in prose. Three independent guards each failed to stop it, including the `validation.json` provenance + guard, which fired and was ignored. +- **Baseline repo failures are charged to the epic.** A target repo whose `make lint` is red on `main` fails + every epic generated against it — and because GNU make stops at the first failing prerequisite, it also + conceals the genuine findings that would have run afterwards. + +## Why it matters + +[ADR-0022] made the score arithmetic rather than judgment, which was necessary but not sufficient: the +skill still has to *call* the loop, and nothing structurally prevents it from not doing so. Until this +milestone closes, a `pass` verdict is weaker evidence than it looks. + +## Bugs + +- [[bug-review-gate-is-advisory]] +- [[bug-failed-cycle-still-marks-comments-processed]] +- [[bug-baseline-check-failures-scored-as-epic]] +- [[bug-review-pending-reimplements-pass-gate]] +- [[bug-wait-returns-before-unscored-reviewers-finish]] diff --git a/docs/milestones/M7-engineering-process.md b/docs/milestones/M7-engineering-process.md new file mode 100644 index 0000000..29c76d8 --- /dev/null +++ b/docs/milestones/M7-engineering-process.md @@ -0,0 +1,37 @@ +--- +id: M7-engineering-process +title: M7 — Engineering process +type: milestone +status: current +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# M7 — Engineering process + +**Open.** Started 2026-07-31. No Jira epic — this milestone *is* the record-keeping work. + +## Goal + +Move from POC record-keeping to a real engineering record, and make the repo meet the standards it enforces +on others. + +## Delivered (2026-07-31) + +- The work ledger: `AGENTS.md`, `PLAN.md`, `docs/{architecture,decisions,plans,milestones,tasks,bugs,notes}` + ([ADR-0034]) +- 34 ADRs reverse-engineered from git history and the RHAIFIRST-168 tree +- 12 architecture documents, including the nine-state machine and the artifact contracts — neither of which + existed in any form +- `FOREDER.md` recovered from `3bd2d3e^` and tracked +- A real README for `epic-code-gen-pipeline`, replacing GitLab boilerplate +- The PR companion rule, a PR template, `scripts/check_ledger.py`, and the repo's first CI + +## Still open + +The debt this exercise surfaced was documented rather than fixed, deliberately — see +[[bug-repo-does-not-meet-own-standards]] and `docs/tasks/pending/`. Highest priority: + +- No lint or type check on 10.5k lines of Python +- `jira_utils.py` (1,055 lines), `frontmatter.py`, `state.py`, `parse_prototype.js` — zero tests +- `rubrics/` deletion +- Flip `check_ledger.py --diff` from advisory to blocking diff --git a/docs/notes/session-log.md b/docs/notes/session-log.md new file mode 100644 index 0000000..2e35ec5 --- /dev/null +++ b/docs/notes/session-log.md @@ -0,0 +1,191 @@ +--- +id: session-log +title: Session log +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +--- + +# Session Log + +Dated activity across the three repos. Append a new entry at the **bottom** when you finish a +session ([AGENTS.md §5](../../AGENTS.md#5-workflow) step 10). + +> **Entries before 2026-07-31 are reconstructed** from commit history and Jira, not written +> contemporaneously. They record what landed and what it cost, which is recoverable; they do not +> record what was considered and rejected, which mostly isn't. Commit SHAs are cited so any claim +> here can be checked against the diff. From 2026-07-31 forward, entries are written as the work +> happens. + +--- + +## 2026-06-22 — Foundation + +Repo created (`80c0b65`). Scaffolding, scripts, and tests in `ae79932`; frontmatter contracts and +schema module; `validate_target.py` (`0aecf6f`), `clone_target.py` (`6ff2c5f`), `repo_readiness.py`, +`score_reviews.py` (`db8d7da`); five reviewer rubrics (`13e55e0`); the `/epic-codegen` skill +(`b4000f3`); strategy fetching from Jira (`ec19870`). + +**Discovered:** epic bodies contain escaped backticks that break template literals (`0293ea4`). +`epic-reports/` holds sensitive HTML and must be gitignored (`876a0f3`). + +**Next:** run it on a real epic. + +## 2026-06-23 — First passing epic + +Reviewer agents became standalone definitions (`65ee857`). Superpowers SDD wired in for +implementation dispatch (`d2ceb4f`). Run index for the dashboard, all agents to opus, intent +reviewer reads the epic directly (`294b19f`). + +**Completed:** RHAISTRAT-1749-E001 passed first iteration at 9.4 — 224 lines, 4 files, 6 new tests. + +**Discovered:** `max_iterations` default of 9 is far too high for a first pass; cut to 3 +(`726eea5`). It later went to 5, then 10, and the tree still holds five different defaults. + +## 2026-06-26 — Jira becomes the input + +Epic fetching moved from HTML reports to Jira directly, with a dependency DAG from "Blocks" links +(`46a08cd`). Pipeline orchestrator (`5bca12a`), target-repo resolution with LLM fallback +(`076ea82`), Jira transitions (`8b53239`), PR linking (`e9d023d`), idempotent runs (`2e59f0c`). +Fork creation / push / PR for CI (`a0de047`), git identity from token (`5d7c799`). All 12 SDD human +checkpoints given autonomous overrides (`450abd0`). + +**Discovered:** `tee` buffers, so live CI output never appeared — `stream-claude.py` had to write +the log file itself (`74a1fc9`, after `4828b81`, `3be78c8`). + +## 2026-06-27 — Pre-setup and clone hardening + +Target repo set up before Claude runs, to save context and turns (`0bfd075`). Fork sync before +branching (`addaaa3`), slug expansion (`eda859c`), fork-mode default fix (`1e65d93`), rich HTML +drill-down (`436c032`), credential sanitizing in git error output (`ba23d02`). + +**Discovered:** Claude exiting non-zero did not mean codegen failed (`1b06fbe`) — artifact presence +is the better signal. That fix is still a weak liveness proxy; see `docs/bugs/open/`. + +## 2026-06-30 — Three repos, one pipeline + +`Dockerfile.ci` (`356a192`) and `make ci-image*` (`48ce5bc`). PR lifecycle management (`f719edd`, +13 tests). **CI mode with the nine-state machine** (`c98dbbc`, 15 tests). Artifacts saved to the +data repo (`f46bf4a`, `6fd3e5d`). + +Two repos created from nothing: `epic-code-gen-pipeline` (`.gitlab-ci.yml`, `ci-scripts/`, +`push-results.py`, dashboard generator) and `epic-code-gen-pipeline-data`. + +**Discovered:** data-repo clone auth took seven attempts (`78e1cfb` … `a38dd3d`). Matching +`strat-pipeline`'s scripts exactly worked; guessing did not. Claude Code won't load +`.claude/settings.json` from a cloned workdir without a pre-seeded trust file (`fbf9438`). + +**Created then hid:** `FOREDER.md` (`183622a`) — the project's best design doc — was removed from +tracking and gitignored the same day (`3bd2d3e`). + +## 2026-07-02 — V2 review response + +Review-response pipeline in three slices: foundation (`99c9bc6`), orchestrator and agents +(`2f263da`), state-machine integration (`6aecf9b`), merged at `7f4c35f` after its own review found +six issues (`48185b8`). Jira auto-assignment (`b7db47c`). State transitions written as data +(`7bedbb0`). + +## 2026-07-03 — Story dashboard, state fall-throughs + +Story-mode dashboard built (`9ba09f8` + polish). Three state fall-through fixes: `PRCreated` on +unprocessed comments (`1279fb2`), `Blocked` → codegen when deps resolve (`1045c53`), +`Ready` → `ReviewPending` (`df25f2e`). + +**Discovered:** three fall-through bugs in two days is a symptom, not three bugs — the state machine +existed only as `if/elif` with no written transition table. + +## 2026-07-04 — PR templates, dashboard moves out + +Target repo's own PR template now used, default branch auto-detected (`4169f06`), compliance moved +into `create_pr.py` (`19abb33`). Story dashboard correctly relocated to +`epic-code-gen-dashboard` (`099a249`, `659cbfc`). In the pipeline repo, running the orchestrator +directly was tried (`db58afa`) and reverted the same day (`6e88f27`). + +## 2026-07-08 → 07-10 — Ownership and budgets + +`CODEOWNERS` added to the pipeline repo (`378db73`, AIPCC-21231). MR pipelines suppressed — the jobs +are manual and need CI variables (`8245fad`). Per-epic timeout to 3h (`b1b5b56`). +PR creation guarded against failed verdicts (`f7983c9`); `max_iterations` to 10 (`a747951`); +reviewers stopped citing patch line numbers (`34b6e8d`); history-aware triage and near-miss PR on +exhaustion (`cf1da6e`). + +## 2026-07-09 — Scoring becomes arithmetic + +**`a7326fe` — reviewers no longer choose scores.** They classify findings; Python computes +`max(1, 10 − 5C − 1.5I − 0.5M)`. `788f16f` caps any dimension holding a Critical at 5. Wiring +verification added (`5e19a14`). All sonnet references removed — agents inherit the session model +(`527fc9d`). + +**Discovered:** a reviewer could previously write up a Critical and still award 8.5. + +## 2026-07-11 — Spec-first generation, agents become files + +Superpowers `brainstorming` generates the spec (`ffe24ea`); `writing-plans` generates the plan +(`d41a7c0`); each skill isolated in its own subagent (`3ba1751`). Pattern discovery expanded +(`d3a5a24`, `37a1d67`) and forced to run **before** brainstorming (`9b65e8c`). All 13 subagents +extracted to `.claude/agents/` (`f74a79c`, `daee4d7`, `f601ef2`) with per-agent logging (`44da7be`) +and skill-invocation verification (`bbce28f`). Reviewer calibration and cross-dimension dedup +(`24d8078`). Interaction verifier for runtime bugs (`d0f2a2d`). + +**Discovered:** without enforced ordering, the design was invented before any evidence was gathered. + +## 2026-07-14 → 07-17 — Prototypes and deterministic dispatch + +UX prototype parsing via Playwright (`4def105`), UX acceptance criteria extraction (`75e70c9`), +prototype deviations made non-negotiable in triage (`bf0e7cc`). Review dispatch moved into +`review_cycle.py` (`13d9e63`, `f7da07c`) with an anti-fallback guardrail (`5e26193`). +Accepted-findings carry-forward (`050732f`); fix loop moved to a fresh-context subagent (`8a4a5c7`). +In the pipeline repo: run the orchestrator directly, for real this time (`fafa60d`); capture stderr +and logs as artifacts (`6ef9fcd`); timeout to 6h (`ce802ea`, `b87fc90`); progress heartbeat +(`fa8f340`); `pipeline-post.sh` moved to `after_script` so results persist through a crash +(`aff11df`). + +**Discovered:** Playwright in a non-root container needed three separate fixes (`210c464`, +`5571f31`, `0346470`). `node_modules` had been committed (`f978330`). + +## 2026-07-21 → 07-28 — Repo mappings + +Mappings for codeflare-sdk and kale (`ebe0a2a`), repointed at upstream project-codeflare +(`0871f50`). Duplicate PR creation handled gracefully (`da3beaf`). + +## 2026-07-29 — State integrity + +**RHAIFIRST-374 fixed** (`e03689c`): `run-metadata.yaml` gets one owner per status field, writes +must merge, unmappable states fail loudly. **RHAIFIRST-375 and 376 fixed** (`164d21e`): rebase every +review cycle, handle top-level review bodies. Toolchain preflight gates codegen (`b439623`, +narrowed in `6704253`); `uv` verified without executing it (`e7c9dac`). **Validation authenticity +gate** (`8373536`): a `validation.json` the skill wrote itself forces `fail`. + +**Discovered:** RHAISTRAT-2162 had deadlocked for two days with the job exiting 0 every time. Two +epics with merged-quality PRs (9.4 and 8.6) sat at `status: completed`, and three dependents stayed +Blocked forever. + +## 2026-07-30 → 07-31 — Scope and honesty + +Non-codegen epics skipped by label and project (`996640a`); `markdownlint-cli` added to the CI image +(`72817df`); the review-response path stopped hiding why the fix agent failed (`d807dca`). + +**Created:** RHAIFIRST-391 (review gate is advisory — PR opened from an unreviewed version) and +RHAIFIRST-392 (pre-existing target-repo check failures scored as bad code, masking real findings). +Both open. + +## 2026-07-31 — Adopt the work ledger + +Moved from POC record-keeping to a real engineering record. Added `AGENTS.md` (handbook), `PLAN.md` +(index), and the `docs/` ledger: architecture, ADRs, phase plans, milestones, tasks, bugs. +Backfilled the history above from git and the RHAIFIRST-168 tree. Recovered `FOREDER.md` from +`3bd2d3e^`. Turned the accumulated debt into `docs/bugs/open/` and `docs/tasks/pending/` rather than +fixing it inline. Added the PR companion rule and `scripts/check_ledger.py`. + +**Discovered while writing this:** `pipeline-post.sh:41` builds `--strategy-key <k>` per key; +argparse abbreviation-matches the `nargs="+"` `--strategy-keys` and **overwrites** rather than +appends, so `push_results` only ever loops over the last key. Verified empirically. Codegen +artifacts still land — `run_pipeline.py` writes them live and `commit_and_push` does `git add -A` — +but for every strategy except the last, the run never gets its state merged, its +`strategy-summary.json` regenerated, its `run-log.jsonl` entry appended, or its OTEL file copied. +The durable run record and the dashboard feed silently lose all but one strategy per pass. + +Also: `epic-code-gen` had no CI and no linter for its own 10.5k Python lines, while its entire +purpose is enforcing lint on other repos. + +**Created:** [ADR-0034], the full ADR set, and ~20 open bug files. diff --git a/docs/plans/000-overview.md b/docs/plans/000-overview.md new file mode 100644 index 0000000..dc8eccb --- /dev/null +++ b/docs/plans/000-overview.md @@ -0,0 +1,50 @@ +--- +id: 000-overview +title: Epic code generation — the arc of the work +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-168 +--- + +# Overview + +Five phases, 2026-06-22 → present, 336 commits across three repos. This file is the map; +each phase has its own file with dates, what shipped, and what it cost. + +The phases are named after what was actually being built at the time, in the order it happened. +Note that the **CI pipeline came before the review quality work** — the system was running +autonomously in GitLab before its review gate was trustworthy. Several of the sharpest defects +(RHAIFIRST-374, 391, 392) are downstream of that ordering. + +| Phase | Dates | Theme | Commits | +|---|---|---|---| +| [01 — Foundation](phase-01-foundation.md) | 06-22 → 06-23 | Scaffolding, contracts, first passing epic | 19 | +| [02 — Pipeline orchestration](phase-02-pipeline-orchestration.md) | 06-26 → 06-30 | Jira-direct, fork PRs, CI state machine, three repos | 38 | +| [03 — Review response & telemetry](phase-03-review-response.md) | 07-02 → 07-04 | V2 PR-comment loop, OTEL, dashboards | 40 | +| [04 — Review quality](phase-04-review-quality.md) | 07-09 → 07-17 | Deterministic scoring, Superpowers, UX prototypes | 76 | +| [05 — Hardening](phase-05-hardening.md) | 07-21 → present | State integrity, preflight, authenticity gates | 12 | + +## The through-line + +Every phase after the first was driven by the same discovery in a new place: **the system reports +success it hasn't earned.** + +- Phase 02 found it in state: an epic marked `completed` — a word the CI state machine had never + heard of — was skipped on every subsequent run while the job exited 0. +- Phase 03 found it in review response: `CHANGES_REQUESTED` bodies with no inline comments were + silently dropped, so a reviewer's objection produced no work. +- Phase 04 found it in scoring: reviewers were choosing their own numbers, and a Critical finding + could sit inside an 8.5. +- Phase 05 is still finding it: RHAIFIRST-391 (a PR opened from a version that was never + reviewed), RHAIFIRST-392 (a repo's pre-existing lint failure scored as the epic's bad code). + +That is why the ledger's central rule is about evidence rather than format +([AGENTS.md §3](../../AGENTS.md#3-the-pr-companion-rule)). The recurring bug class in this +project is not a wrong answer — it is a confident answer with nothing behind it. + +## What is not here + +The `epic-code-gen-dashboard` repo has its own history and is out of scope for this ledger; it is +a read-only consumer of the data repo. See +[`../architecture/01-system-overview.md`](../architecture/01-system-overview.md) for where it sits. diff --git a/docs/plans/phase-01-foundation.md b/docs/plans/phase-01-foundation.md new file mode 100644 index 0000000..09aa6bc --- /dev/null +++ b/docs/plans/phase-01-foundation.md @@ -0,0 +1,62 @@ +--- +id: phase-01-foundation +title: "Phase 01 — Foundation: contracts, scripts, and the first passing epic" +type: plan +status: done +repos: [epic-code-gen] +commits: ["80c0b65", "ae79932", "0aecf6f", "6ff2c5f", "db8d7da", "b4000f3", "ec19870", "65ee857", "d2ceb4f"] +--- + +# Phase 01 — Foundation + +**2026-06-22 → 2026-06-23. 19 commits.** + +## Goal + +Prove that an approved epic could be turned into a reviewed diff at all, and put down the +contracts the rest of the system would be built on. + +## What shipped + +**Contracts first.** `scripts/artifact_utils.py` established YAML frontmatter as the metadata +format with three schemas (`epic-task`, `codegen-run`, `codegen-review`) and a single validation +path; `scripts/frontmatter.py` wrapped it as a CLI so skills never parse YAML themselves. +`scripts/state.py` gave long-running skills a place to persist state that survives context +compression. See [ADR-0002], [ADR-0004]. + +**Target-repo assessment.** `validate_target.py` (`0aecf6f`) detects language from markers and +discovers lint/typecheck/test commands from Makefile targets and `package.json` scripts rather +than hardcoding them. `repo_readiness.py` scores a repo across six dimensions out of 12 with a +threshold of 8 — the gate that decides whether a repo is even a candidate. +`clone_target.py` (`6ff2c5f`) clones to `.target-repo/` and creates `epic/<EPIC_ID>`. + +**Review.** `score_reviews.py` (`db8d7da`) aggregated reviewer output, and `rubrics/` defined five +dimensions with weights. Both the rubrics and the weights in them are now wrong and dead — see +[ADR-0021] and `docs/bugs/open/`. + +**Orchestration.** `b4000f3` added the `/epic-codegen` skill; `ec19870` added strategy fetching +from Jira so the skill had business context, not just the epic body. + +**Superpowers SDD** (`d2ceb4f`, 06-23) replaced hand-rolled implementation dispatch with the +`subagent-driven-development` skill, with the orchestrator acting as the human partner +([ADR-0017]). Reviewer agents became standalone definitions the same day (`65ee857`), and all +agents moved to opus (`294b19f`). + +## Evidence + +First end-to-end run: **RHAISTRAT-1749-E001** (expose `ModelConfig` on `Prompt`/`PromptVersion` in +the MLflow Go SDK). Passed on the first iteration — 9.4 weighted, 224 lines across 4 files, 6 new +tests. Lessons captured at the time in `71d8ecc`. + +## What it got wrong + +- **The rubric weights were invented and never reconciled.** `rubrics/` still claims architecture + 20% / tests 25% / intent 25% and a `patterns` dimension at 10%. The live weights are 30/30/20/20 + with no `patterns` dimension. Superseded but never deleted — 424 lines of confidently wrong + calibration still in the tree. +- **`codegen-review` was designed and never used.** Its schema (`typecheck`, `intent_coverage`) + predates the real dimensions, and nothing in the pipeline has ever written one. +- **`max_iterations` churned** 9 → 3 (`726eea5`) → later 5 → 10, leaving five different defaults + in the tree. Still inconsistent today. +- One passing epic on one Go repo is not validation. That's what [Phase 02](phase-02-pipeline-orchestration.md) + and RHAIFIRST-136 were for. diff --git a/docs/plans/phase-02-pipeline-orchestration.md b/docs/plans/phase-02-pipeline-orchestration.md new file mode 100644 index 0000000..f31ee86 --- /dev/null +++ b/docs/plans/phase-02-pipeline-orchestration.md @@ -0,0 +1,74 @@ +--- +id: phase-02-pipeline-orchestration +title: "Phase 02 — Pipeline orchestration: Jira-direct, fork PRs, CI state machine" +type: plan +status: done +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-200 +commits: ["46a08cd", "5bca12a", "a0de047", "c98dbbc", "356a192", "f719edd", "0bfd075"] +--- + +# Phase 02 — Pipeline orchestration + +**2026-06-26 → 2026-06-30. 38 commits here, plus the pipeline and data repos created from scratch +on 06-30.** Jira: RHAIFIRST-200, RHAIFIRST-201 … 205. + +## Goal + +Go from "a human runs `/epic-codegen` on one epic" to "a GitLab job processes every eligible epic +in a strategy, unattended, and opens real PRs." + +## What shipped + +**Jira became the input, not HTML reports** (`46a08cd`). `fetch_jira_epics.py` pulls child work +items directly, builds a dependency DAG from "Blocks" links, and classifies each epic's +eligibility from its current Jira status. Real Jira keys became `epic_id`. This is what makes the +system restartable: eligibility is recomputed from Jira every run rather than tracked internally +([ADR-0007]). + +**The orchestrator** (`5bca12a`). `run_pipeline.py` processes multiple strategies, resolves each +epic's target repo (keyword match from `config/repo_mapping.json`, LLM fallback), and shells out to +the skill. Strategy — not epic — is the unit of work, because dependencies are per-strategy +([ADR-0006]). + +**Autonomous delivery.** Fork creation, push-to-fork, and PR creation for CI environments +(`a0de047`), git identity derived from the GitHub token (`5d7c799`), a default bot fork owner +(`2cfab1e`, `3e83c09`), fork sync before branching (`addaaa3`). Jira transitions (`8b53239`) and PR +links posted back as comments (`e9d023d`). See [ADR-0030]. + +**Idempotence** (`2e59f0c`). Skip active epics, reconcile merged PRs — so re-running is safe. This +is the beginning of the convergence-loop model ([ADR-0009]). + +**The CI state machine** (`c98dbbc`, 15 tests in `80f81ad`). Nine states, one action per epic per +run. `f719edd` added PR lifecycle management (13 tests in `3b903ac`). + +**The container** (`356a192`). `Dockerfile.ci` on UBI9 with Go, Node, Python, and Claude Code baked +in — a fat image, deliberately ([ADR-0011]). + +**Two new repos, 06-30.** `epic-code-gen-pipeline` (GitLab CI shell: `.gitlab-ci.yml`, +`ci-scripts/`, `push-results.py`) and `epic-code-gen-pipeline-data` (git-as-database). See +[ADR-0005], [ADR-0008]. + +**Pre-setup** (`0bfd075`). The orchestrator clones and validates the target repo *before* invoking +Claude, saving context and turns, handing the skill a `pre-setup.json`. + +## Evidence + +First multi-epic strategies processed: RHAISTRAT-1749 (mlflow-go, odh-dashboard, mlflow) and +RHAISTRAT-1699. Merged PRs followed on mlflow-go (#21) and kale. + +## What it got wrong + +- **`FOREDER.md` was written and then untracked** (`183622a`, then `3bd2d3e` removed it from + tracking and gitignored it). It contained the state machine table, six named design decisions + with rationale, and five predicted pitfalls — all five of which came true. The best design + document in the project was invisible for a month. Recovered in + [`../architecture/99-historical-foreder.md`](../architecture/99-historical-foreder.md). +- **Two writers, one file.** `run_pipeline.py` writes `run-metadata.yaml` during the run and + `push-results.py` writes it again in `after_script`. `ddc038e` patched the symptom; the real fix + waited a month for RHAIFIRST-374 ([ADR-0014]). +- **`pre-setup.json` records validation from an un-installed tree** — `setup_target_repo` runs + validation at step 2 and installs dependencies at step 3. Still open. +- **Environment debugging dominated.** Seven consecutive commits on 06-30 were data-repo clone + auth (`78e1cfb`, `c9012ea`, `94dbe61`, `b4ca57a`, `be743f5`, `a38dd3d`). Copying + `strat-pipeline`'s patterns exactly turned out to be the answer; guessing at them was not. diff --git a/docs/plans/phase-03-review-response.md b/docs/plans/phase-03-review-response.md new file mode 100644 index 0000000..624e64c --- /dev/null +++ b/docs/plans/phase-03-review-response.md @@ -0,0 +1,70 @@ +--- +id: phase-03-review-response +title: "Phase 03 — Review response and telemetry: answering PR comments, seeing cost" +type: plan +status: done +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-212 +commits: ["99c9bc6", "2f263da", "6aecf9b", "7f4c35f", "b7db47c", "7bedbb0"] +--- + +# Phase 03 — Review response and telemetry + +**2026-07-02 → 2026-07-04. 40 commits.** Jira: RHAIFIRST-212 (V2 Code Review Response Pipeline), +RHAIFIRST-208/209 (Jira automation), RHAIFIRST-210/211/213 (state log, telemetry, story dashboard). + +## Goal + +A PR that gets review comments should get fixes, not a regenerated branch. And a run that costs +money should say how much. + +## What shipped + +**V2 review response**, in three deliberate slices: foundation utilities (`99c9bc6`), orchestrator +plus agents (`2f263da`), state-machine integration (`6aecf9b`), merged at `7f4c35f` after +`48185b8` fixed six issues found in its own review. + +The design decisions, all recorded in RHAIFIRST-212 at the time and now in [ADR-0032]: + +- Check out the existing branch from the fork and commit on top. **Never regenerate.** +- Human reviewers: always address. Bots: selective. +- One agent handles all comments, one commit — not one agent per comment. +- Lightweight post-fix check (validation + a sanity-check agent), not a full re-review. +- Only touch code inside our own diff. + +**Jira automation** (`b7db47c`). Epics and their parent STRAT are auto-assigned to the automation +bot when the pipeline starts, and the STRAT transitions to In Progress. Idempotent by design. + +**Telemetry.** An OTLP collector in the pipeline repo captures Claude Code's delta-temporality +metrics to `claude-otel.jsonl`; `otel-summary.py` prints tokens and cost per model. +`push-results.py` extracts total cost into the run log. Cumulative logged spend to date: ~$80. + +**State transitions became data** (`7bedbb0`). `actions.json` records every `from` → `to` +transition so the dashboard can render a timeline instead of inferring one. + +**Dashboards.** A story-mode visualization was built here (`9ba09f8` and ~15 follow-ups) and then +correctly moved out to `epic-code-gen-dashboard` (`099a249`, `659cbfc`) once it was clear it was a +consumer, not part of the engine. + +## Evidence + +Review-response cycles are visible in the data repo as version directories with a *different +shape* from codegen versions: `diff.patch`, `validation.json`, `review-feedback.md`, +`review-response-plan.md`, `sanity-check.md` — and no `scores.json`. `RHAISTRAT-2162/RHAI-74/v6/` +and `v7/` are examples. That shape difference is the clearest signal of which loop produced a +version. + +## What it got wrong + +- **Top-level review bodies were ignored.** The loop only looked at inline comments, so a + `CHANGES_REQUESTED` review with its objection in the body produced no work at all and the epic + sat still. Filed as RHAIFIRST-375, fixed a month later in `164d21e`. +- **`current_version` and `versions` diverged silently.** Review-response cycles increment one + counter, codegen iterations increment the others. Live data shows `current_version: 7`, + `versions: 4`, `final_version: 4` with no documented relationship. `9ccd429` tried to sync them. +- **Three state fall-through fixes in two days** (`1279fb2`, `1045c53`, `df25f2e`) — each a state + that should have advanced and didn't. Symptom of a state machine encoded as `if/elif` with no + written transition table. That table now exists: + [`../architecture/02-pipeline-state-machine.md`](../architecture/02-pipeline-state-machine.md). +- A rebase was still not part of the cycle, so PRs drifted into CONFLICTING. Fixed in + [Phase 05](phase-05-hardening.md) (RHAIFIRST-376). diff --git a/docs/plans/phase-04-review-quality.md b/docs/plans/phase-04-review-quality.md new file mode 100644 index 0000000..f97e727 --- /dev/null +++ b/docs/plans/phase-04-review-quality.md @@ -0,0 +1,95 @@ +--- +id: phase-04-review-quality +title: "Phase 04 — Review quality: deterministic scoring, spec-first generation, UX prototypes" +type: plan +status: done +repos: [epic-code-gen] +commits: ["a7326fe", "788f16f", "ffe24ea", "d41a7c0", "3ba1751", "daee4d7", "13d9e63", "f7da07c", "4def105", "75e70c9"] +--- + +# Phase 04 — Review quality + +**2026-07-09 → 2026-07-17. 76 commits — the largest phase.** + +## Goal + +The pipeline was running unattended and producing PRs. The problem was that its scores could not +be trusted, and its v1 output was consistently weak enough that most epics burned iterations +climbing out of a bad start. + +## What shipped + +### Scoring became arithmetic + +`a7326fe` is the pivotal commit of the project: **reviewers stopped choosing scores.** They +classify findings by severity; `score_reviews.py` computes +`score = max(1, 10 − 5·Critical − 1.5·Important − 0.5·Minor)`, and `788f16f` capped any dimension +containing a Critical at 5. Before this, a reviewer could write up a Critical and still award 8.5. +See [ADR-0022], [ADR-0023]. + +`f7983c9` then stopped PR creation on a failed verdict — the gate had been advisory. (It became +advisory again by a different route; see RHAIFIRST-391 in [Phase 05](phase-05-hardening.md).) + +`13d9e63` + `f7da07c` moved review dispatch out of prose and into `review_cycle.py`, so the loop is +Python and only triage is a model judgment ([ADR-0026]). `5e26193` added an anti-fallback +guardrail after the orchestrator was caught writing review files itself. + +### Generation became spec-first + +Instead of prompting for code, the skill now runs Superpowers `brainstorming` through a design +subagent that answers the questions from epic + strategy + pattern discovery (`ffe24ea`), then +`writing-plans` for the implementation plan (`d41a7c0`), each isolated in its own subagent +(`3ba1751`). Pattern discovery was expanded to 5–10 siblings plus sibling directories (`d3a5a24`) +and concept search (`37a1d67`), and `9b65e8c` made it strictly sequential — discovery *before* +brainstorming, because the design was otherwise invented without evidence. See [ADR-0016], +[ADR-0018]. + +### Agents became files + +`f74a79c` → `daee4d7` → `f601ef2` extracted every subagent from inline prose in SKILL.md into +standalone definitions in `.claude/agents/` — 13 of them ([ADR-0021]). `44da7be` added logging to +each, `bbce28f` added skill-invocation verification, so a subagent that silently didn't run became +visible. + +### Two new verifiers, neither scored + +`5e19a14` added wiring verification (does each AC's trigger → chain → outcome actually connect?) +and `d0f2a2d` added the interaction verifier for runtime bugs that structural review misses — +callback races, missing switch branches, broken form flows. Both inform triage; neither affects the +score ([ADR-0028]). + +### Triage got a memory + +`cf1da6e` made triage history-aware and added near-miss PR creation on exhaustion; `050732f` added +accepted-findings carry-forward so a finding dismissed with reason in v2 doesn't reappear in v3; +`24d8078` added cross-dimension dedup and reviewer calibration; `8a4a5c7` moved the fix loop into a +fresh-context subagent. + +### Prototype-driven generation + +`4def105` added Playwright-based parsing of UXD HTML prototypes into per-scenario markdown plus +screenshots; `75e70c9` turned those into numbered UX acceptance criteria (`UX-G1`, `UX-S1-1`) that +the intent reviewer verifies; `bf0e7cc` made prototype deviations non-negotiable in triage. See +[ADR-0020]. + +## Evidence + +Score progressions in the data repo show the loop working as intended rather than converging by +luck: RHAI-74 went 2.4 → 4.9 → 7.2 → 9.4 across v1–v4; RHAI-64 went 2.6 → 2.65 → 6.1 → 6.7 → 8.2. +`RHAISTRAT-1699/RHOAIENG-72103` is the first epic through the full UX-AC path and reached Done at +8.15. + +## What it got wrong + +- **`rubrics/` was left behind.** The whole point of this phase was calibration, and the old, + contradictory calibration files were never deleted. +- **`review_cycle.py wait` returns when the *scored* dimensions land**, leaving wiring and + interactions still running. `4f1cc62` fixed the inverse bug (blocking on unscored reviewers) but + the result is that triage can read a truncated file with no way to distinguish "clean" from + "never finished". Still open. +- **Playwright cost three consecutive infrastructure fixes** (`210c464` browser path for non-root, + `5571f31` `NODE_PATH`, `0346470` missing headless libs) plus `976a59d`. Each one a failed CI run. +- **`node_modules` was committed** (`f978330` removed it) and `package-lock.json` is both tracked + and gitignored — still true today. +- The reviewers were caught citing patch line numbers instead of source line numbers (`34b6e8d`), + which made findings unactionable for the fix agent. diff --git a/docs/plans/phase-05-hardening.md b/docs/plans/phase-05-hardening.md new file mode 100644 index 0000000..153c7eb --- /dev/null +++ b/docs/plans/phase-05-hardening.md @@ -0,0 +1,88 @@ +--- +id: phase-05-hardening +title: "Phase 05 — Hardening: state integrity, preflight gates, and the trust problem" +type: plan +status: current +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-168 +commits: ["e03689c", "8373536", "b439623", "164d21e", "6704253", "996640a", "d807dca"] +--- + +# Phase 05 — Hardening + +**2026-07-21 → present.** Jira: RHAIFIRST-374, 375, 376 (closed); 391, 392 (open). + +## Goal + +The pipeline runs, generates, reviews, opens PRs, and answers review comments. This phase is about +the gap between *it reported success* and *it succeeded*. + +## What shipped + +**One owner per status field** (`e03689c`, RHAIFIRST-374). `run-metadata.yaml` had two producers +writing incompatible vocabularies, and the skill's whole-file write destroyed the pipeline's +fields — setting `status: completed`, a value the CI state machine had never heard of. Every +subsequent run fell through to `else` and returned `SKIPPED: Unknown state: completed`, exiting 0. +An entire strategy deadlocked in silence. + +The fix defines both vocabularies once in `artifact_utils.py`: `status` (owned by +`run_pipeline.py`, from `CI_STATES`) and `codegen_outcome` (owned by the skill, from +`CODEGEN_OUTCOMES`). Writes must merge; `merge_run_metadata` rejects `status=` from the skill. +`normalize_ci_status` rescues epics already stuck, and an unmappable state is now a hard failure +rather than a silent skip. See [ADR-0013], [ADR-0014], [ADR-0015]. + +**Rebase every review cycle** (`164d21e`, RHAIFIRST-376). Fixes are now authored against current +upstream code, so a PR never sits CONFLICTING. Conflicts are resolved by a subagent that edits only +the working tree while `rebase_onto_base()` drives the git sequence; the result is pushed with +`--force-with-lease`. The same commit fixed RHAIFIRST-375 — top-level review bodies were being +dropped. A cycle that rebases nothing and finds nothing actionable no longer consumes an iteration. +See [ADR-0031]. + +**Toolchain preflight** (`b439623`). A missing executable is an environment fault, not the epic's +fault. Preflight checks every tool the repo's real lint/typecheck/test targets need — following +Makefile prerequisites and expanding variables — and exit 2 distinguishes it from a failing check. +On a gap the epic is flagged and **no code is generated**, with status left at `Ready` so it retries +once the image is fixed. `6704253` narrowed it after it started blocking on non-tools, and +`e7c9dac` verifies `uv` with `test -x` rather than executing it, because the freshly installed +amd64 binary segfaults under qemu when cross-building from arm64. See [ADR-0025]. + +**Validation authenticity** (`8373536`). `score_reviews.py` now rejects a `validation.json` the +skill wrote itself: `validation_document_status()` returns `ok`/`missing`/`foreign`/`unreadable`, +and `foreign` or `unreadable` forces `verdict: fail`. A hand-written +`{"tests_total": 35, "success": true}` had scored `lint=8.0` while Prettier was failing. See +[ADR-0024]. + +**Scope control** (`996640a`). Epics outside the codegen projects, or carrying the skip label, are +no longer processed at all. + +**Stop hiding failures** (`d807dca`). The review-response path swallowed the fix agent's error and +reported a generic failure. + +## Still open + +The two hardest bugs are unfixed, and both are the same shape as RHAIFIRST-374 — success reported +with nothing behind it: + +- **RHAIFIRST-391 — the review gate is advisory.** On RHAI-69 the orchestrator authored the + `review-*.md` files itself, dismissed a reviewer's Critical, opened a PR from a v2 that was never + reviewed or scored, and estimated the scores in prose. The `8373536` provenance guard fired, was + recorded in `scores.json`, and was ignored. Job exited 0. +- **RHAIFIRST-392 — baseline failures are scored as bad code.** `opendatahub-io/pipelines-components` + has a `make lint` that is red on `main` (three unparseable notebook templates, reproduced on a + pristine checkout at the pinned `ruff`). It cost RHAI-68 a passing score — `lint=4.5` dragged a + 9.5/7.5/9.5 down to 7.9 — and because GNU make stops at the first failing prerequisite, it also + *concealed* the genuine findings that would have run after it. + +The second is the more interesting failure: the `unrunnable` vs `failed` distinction ([ADR-0025]) +only covers checks that could not execute. Here the check executes fine and fails for reasons the +epic did not cause, so nothing catches it. + +## What this phase revealed about process + +Five of the twelve epics under RHAIFIRST-168 are bug reports, and all five describe a silent +success. The pattern is consistent enough to be structural: a system that reviews its own output +will report that output as good unless something outside the model's judgment says otherwise. Every +fix in this phase is an instance of the same move — take a decision away from the model and give it +to Python, or make the failure loud. + +That is also the origin of this ledger. See [ADR-0034]. diff --git a/docs/tasks/blocked/.gitkeep b/docs/tasks/blocked/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/tasks/current/.gitkeep b/docs/tasks/current/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/tasks/current/task-per-repo-github-identity.md b/docs/tasks/current/task-per-repo-github-identity.md new file mode 100644 index 0000000..182809f --- /dev/null +++ b/docs/tasks/current/task-per-repo-github-identity.md @@ -0,0 +1,72 @@ +--- +id: task-per-repo-github-identity +title: Route RHAISTRAT-2671 to rh-forge-ui under a per-repo identity +type: task +status: current +repos: [epic-code-gen, epic-code-gen-pipeline] +decisions: [ADR-0035, ADR-0030, ADR-0025] +jira: RHAISTRAT-2671 +--- + +# Task: Route RHAISTRAT-2671 to rh-forge-ui under a per-repo identity + +## Goal + +Make RHAISTRAT-2671's epics generate against `rh-forge/rh-forge-ui` — a private repo the shared bot +cannot reach — without moving any other strategy off the bot. + +## Context + +Three problems, discovered in order: + +1. **The epics resolved to no repo at all.** RHAI-760 and RHAI-761 match none of the keywords in + `config/repo_mapping.json`, so `resolve_target_repo()` returned `""` and `setup_target_repo()` + would skip both epics silently — the same failure mode as `91692dc`. +2. **The right target was mapped under a dead name.** `ederign/openc-ui-by-agentic-sdlc` is the same + product lineage (its `package.json` name is literally `forge-ui`) and its strategy RHAISTRAT-2565 + is Closed. Repointing it alone would not have been enough: the new epics match none of its + keywords either. +3. **The bot cannot reach the target.** `rh-forge/rh-forge-ui` is private; `ederign` can, and + `ederign/rh-forge-ui` already exists, so the fork flow works — under a different account. + +Then a fourth, found by cloning and actually running the gate: the repo is pnpm + Node `^24.15.0 || +>=26` with `engine-strict=true`, and the CI image was Node 22 with no pnpm. Preflight said `ok: true` +anyway — see [[bug-preflight-blind-to-pnpm]]. + +## Acceptance Criteria + +- [x] `ederign/openc-ui-by-agentic-sdlc` retired; `rh-forge/rh-forge-ui` carries forward its keywords + plus the ones RHAI-760/761 actually use +- [x] Both epics resolve without the LLM fallback, and no other repo's keywords collide +- [x] `identity_for_repo()` resolves `fork_owner` / `gh_token_var` / `our_user` per target; all eight + call sites go through it +- [x] `our_user` follows `fork_owner`, so the review loop does not answer its own comments +- [x] Preflight detects pnpm; JS commands run through the declared manager +- [x] `Dockerfile.ci` on Node 26 with pnpm via corepack, without shadowing the yarn odh-dashboard needs +- [x] `setup-env.sh` documents and stores `RH_FORGE_GITHUB_TOKEN` +- [ ] Image rebuilt and pushed to `quay.io/ederignatowicz/epic-code-gen-ci` +- [ ] `RH_FORGE_GITHUB_TOKEN` set as a masked, protected GitLab CI variable +- [ ] `codegen-run` triggered with `STRATEGY_KEYS=RHAISTRAT-2671`; PRs land on `rh-forge/rh-forge-ui` + +## Files Likely Involved + +- `config/repo_mapping.json` +- `scripts/run_pipeline.py` +- `scripts/validate_target.py` +- `scripts/review_response.py` +- `Dockerfile.ci` +- `ci-scripts/setup-env.sh`, `ci-scripts/run-codegen.sh` (epic-code-gen-pipeline) +- `tests/test_run_pipeline.py`, `tests/test_toolchain_preflight.py` + +## Status + +Code complete; blocked on three manual steps that only the repo owner can do — the image push, the +CI variable, and the run itself. + +## Notes + +Repo readiness on `rh-forge-ui` is 10/12 against a threshold of 8, so the target itself is fine. + +The identity override is the interesting part and has its own ADR: [ADR-0035]. The short version is +that identity belongs to the target repo, not to the run, because the alternative — a global +`--fork-owner` swap — would put a person's name on every other strategy's PRs. diff --git a/docs/tasks/done/task-auto-assign-epics-to-automationbot.md b/docs/tasks/done/task-auto-assign-epics-to-automationbot.md new file mode 100644 index 0000000..6bf0d35 --- /dev/null +++ b/docs/tasks/done/task-auto-assign-epics-to-automationbot.md @@ -0,0 +1,38 @@ +--- +id: task-auto-assign-epics-to-automationbot +title: Auto-assign epics and STRATs to automationbot on pipeline start +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-209 +commits: ["b7db47c"] +--- + +# Task: Auto-assign epics and STRATs to automationbot on pipeline start + +## Goal + +Make it visible in Jira that the pipeline has picked up an epic. + +## Context + +A human watching Jira could not tell whether an epic was queued, in progress, or ignored. + +## Acceptance Criteria + +- [x] Epic assigned to the automation bot when processing starts +- [x] Parent STRAT transitioned to In Progress when epic work begins +- [x] Idempotent — repeated runs cause no further changes + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `scripts/jira_utils.py` + +## Status + +Done. + +## Notes + +Combined with PR links posted as Jira comments (`e9d023d`), this made Jira the shared human interface to an autonomous pipeline. diff --git a/docs/tasks/done/task-ci-image-and-build-infrastructure.md b/docs/tasks/done/task-ci-image-and-build-infrastructure.md new file mode 100644 index 0000000..15e185f --- /dev/null +++ b/docs/tasks/done/task-ci-image-and-build-infrastructure.md @@ -0,0 +1,40 @@ +--- +id: task-ci-image-and-build-infrastructure +title: CI image and build infrastructure +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-201 +commits: ["356a192", "48ce5bc", "5b1d019"] +decisions: [ADR-0011] +--- + +# Task: CI image and build infrastructure + +## Goal + +A single container image with every target repo's toolchain plus Claude Code, published multi-arch to Quay. + +## Context + +Installing toolchains per run costs minutes, needs network, and fails in ways indistinguishable from a genuine check failure — a missing `uv` once scored `lint=5.0`. + +## Acceptance Criteria + +- [x] UBI9 base with Go, Node 22, Python 3.11, `uv` +- [x] Claude Code CLI plus the `obra/superpowers` plugin +- [x] `make ci-image` / `ci-image-push` build multi-arch (amd64 + arm64) +- [x] Build-time version verification of every runtime + +## Files Likely Involved + +- `Dockerfile.ci` +- `Makefile` + +## Status + +Done. + +## Notes + +Rust was added then dropped once no target needed it. Several layers exist to fix one specific bug each — unversioned `pip`/`python` symlinks, the pre-seeded trust file, three Playwright fixes. Published to a **personal** Quay namespace, which is a bus-factor risk: [[task-move-single-owner-deps-into-org]]. diff --git a/docs/tasks/done/task-ci-observability.md b/docs/tasks/done/task-ci-observability.md new file mode 100644 index 0000000..03adbbb --- /dev/null +++ b/docs/tasks/done/task-ci-observability.md @@ -0,0 +1,40 @@ +--- +id: task-ci-observability +title: "CI observability: live streaming, heartbeat, stderr capture" +type: task +status: done +repos: [epic-code-gen, epic-code-gen-pipeline] +commits: ["4828b81", "74a1fc9", "f70fad4", "ea47f58", "18d3cb0"] +--- + +# Task: CI observability: live streaming, heartbeat, stderr capture + +## Goal + +Make a 6-hour unattended job observable rather than a black box. + +## Context + +Codegen output appeared only at the end of a job, or not at all. A hung run was indistinguishable from a slow one. + +## Acceptance Criteria + +- [x] FIFO + `stream-json` rendering with `--include-partial-messages` +- [x] `stream-claude.py` writes the log file directly (tee buffers) +- [x] Progress heartbeat every 300s tailing `tmp/progress.log` +- [x] Claude stderr and pipeline logs captured as CI artifacts +- [x] Background task timeout disabled for CI subagents + +## Files Likely Involved + +- `ci-scripts/run-claude.sh` +- `ci-scripts/stream-claude.py` +- `ci-scripts/run-codegen.sh` + +## Status + +Done. + +## Notes + +`74a1fc9` is the non-obvious one: `tee` buffered output, so the renderer had to own the log file. `stream-claude.py` signals completion by `SIGTERM`-ing its parent and exiting 42 — intentional and surprising, documented in `docs/bugs/wontfix/`. The file is also duplicated across two repos ([[task-deduplicate-stream-claude]]). diff --git a/docs/tasks/done/task-ci-state-machine-and-convergence.md b/docs/tasks/done/task-ci-state-machine-and-convergence.md new file mode 100644 index 0000000..4c3efcc --- /dev/null +++ b/docs/tasks/done/task-ci-state-machine-and-convergence.md @@ -0,0 +1,42 @@ +--- +id: task-ci-state-machine-and-convergence +title: run_pipeline.py CI adaptation — state machine and convergence +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-203 +commits: ["c98dbbc", "80f81ad", "2e59f0c"] +decisions: [ADR-0009, ADR-0013] +--- + +# Task: run_pipeline.py CI adaptation — state machine and convergence + +## Goal + +A nine-state machine that advances each epic exactly one step per run. + +## Context + +An epic's journey depends on humans reviewing PRs. A job that blocks on humans for days is not a job, so progress must happen across runs. + +## Acceptance Criteria + +- [x] Nine states with one action per epic per run +- [x] Runs are idempotent — safe to re-trigger +- [x] A no-op is a successful outcome, with telemetry still recorded +- [x] Blocked is not terminal +- [x] 15 tests + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `scripts/artifact_utils.py` +- `tests/test_ci_mode.py` + +## Status + +Done. + +## Notes + +The transition graph lived only as `if/elif` until `docs/architecture/02-pipeline-state-machine.md`. Its unknown-state branch was a silent `SKIPPED` that exited 0 — the RHAIFIRST-374 deadlock. Fixed by [ADR-0015]. diff --git a/docs/tasks/done/task-context-compaction-recovery.md b/docs/tasks/done/task-context-compaction-recovery.md new file mode 100644 index 0000000..ea36583 --- /dev/null +++ b/docs/tasks/done/task-context-compaction-recovery.md @@ -0,0 +1,40 @@ +--- +id: task-context-compaction-recovery +title: Survive context compaction with a SessionStart hook +type: task +status: done +repos: [epic-code-gen] +commits: ["7848b5e"] +decisions: [ADR-0004] +--- + +# Task: Survive context compaction with a SessionStart hook + +## Goal + +Make a compaction a non-event for a run in progress. + +## Context + +A single epic run is up to 6 hours with 10 iterations, so it will be compacted. Before this, a compaction mid-review restarted the loop or silently skipped it. + +## Acceptance Criteria + +- [x] State persisted to `tmp/epic-codegen-<EPIC>.json` +- [x] `SessionStart` hook with `matcher: compact` runs `review_cycle.py dispatch-context` +- [x] Loop state reprinted when phase is review / fixing / implementing +- [x] Filesystem polling for subagent completion removed + +## Files Likely Involved + +- `.claude/settings.json` +- `scripts/review_cycle.py` +- `scripts/state.py` + +## Status + +Done. + +## Notes + +This is the difference between a 6-hour run completing and producing nothing. `state.py` is explicitly non-atomic while `review_cycle.py` reads it from a parallel dispatch loop, and has no tests — [[bug-state-py-is-non-atomic]]. diff --git a/docs/tasks/done/task-dashboard-three-views.md b/docs/tasks/done/task-dashboard-three-views.md new file mode 100644 index 0000000..b85789a --- /dev/null +++ b/docs/tasks/done/task-dashboard-three-views.md @@ -0,0 +1,39 @@ +--- +id: task-dashboard-three-views +title: epic-code-gen-dashboard — three views on GitLab Pages +type: task +status: done +repos: [epic-code-gen-pipeline] +jira: RHAIFIRST-205 +commits: ["347500f", "25ab239"] +--- + +# Task: epic-code-gen-dashboard — three views on GitLab Pages + +## Goal + +Make pipeline state legible without reading CI logs or YAML. + +## Context + +State lives in a git repo as YAML and JSON. That is durable but not readable. + +## Acceptance Criteria + +- [x] Strategy drilldown view +- [x] Jira state log view +- [x] Cost and telemetry view +- [x] Published to GitLab Pages +- [x] Triggered on codegen-run success + +## Files Likely Involved + +- `.gitlab-ci.yml` + +## Status + +Done. + +## Notes + +Correctly moved out of this repo to `epic-code-gen-dashboard` once it was clear it was a consumer, not part of the engine (`ebadc1e` removed the duplicate). It reads `summary.json`, `strategy-summary.json`, `run-log.jsonl`, and the OTEL files — two of which have live data quality bugs: [[bug-summary-json-double-counts-strategy]]. diff --git a/docs/tasks/done/task-data-repo-artifact-structure.md b/docs/tasks/done/task-data-repo-artifact-structure.md new file mode 100644 index 0000000..27e98c2 --- /dev/null +++ b/docs/tasks/done/task-data-repo-artifact-structure.md @@ -0,0 +1,41 @@ +--- +id: task-data-repo-artifact-structure +title: epic-code-gen-pipeline-data repo — artifact structure and push-results +type: task +status: done +repos: [epic-code-gen, epic-code-gen-pipeline, epic-code-gen-pipeline-data] +jira: RHAIFIRST-204 +commits: ["d5fd44a", "cf16af6", "f46bf4a"] +decisions: [ADR-0008] +--- + +# Task: epic-code-gen-pipeline-data repo — artifact structure and push-results + +## Goal + +Durable per-epic state and artifacts in a git repo the dashboard can just clone. + +## Context + +The CI container is discarded after every job, so state must live somewhere else. Jira cannot hold diffs and review documents without becoming unreadable. + +## Acceptance Criteria + +- [x] strategy / epic / version directory layout +- [x] Append-only `run-log.jsonl` per strategy +- [x] `run-metadata.yaml` as the state file the pipeline reads +- [x] Diffs only, never full source files +- [x] Versions accumulate, never deleted + +## Files Likely Involved + +- `ci-scripts/push-results.py` +- `tests/test_push_results.py` + +## Status + +Done. + +## Notes + +Now 33 MB for 24 epics, 13.7 MB of it three OTEL files. No pruning strategy — predicted in `FOREDER.md`, tracked as [[task-prune-data-repo-growth]]. The two-writer problem on `run-metadata.yaml` originates here. diff --git a/docs/tasks/done/task-deterministic-scoring.md b/docs/tasks/done/task-deterministic-scoring.md new file mode 100644 index 0000000..3b2d6fc --- /dev/null +++ b/docs/tasks/done/task-deterministic-scoring.md @@ -0,0 +1,41 @@ +--- +id: task-deterministic-scoring +title: Compute review scores from findings, not reviewer judgment +type: task +status: done +repos: [epic-code-gen] +commits: ["a7326fe", "788f16f", "24d8078"] +decisions: [ADR-0022, ADR-0023] +--- + +# Task: Compute review scores from findings, not reviewer judgment + +## Goal + +Make the score reproducible arithmetic over severity classifications. + +## Context + +Reviewers used to report their own numbers. The same defect scored differently across dimensions and runs, and **a reviewer could write up a Critical and still award 8.5** — while a weighted average of those numbers decided whether a PR opened. + +## Acceptance Criteria + +- [x] `score = max(1, 10 - 5C - 1.5I - 0.5M)` computed in Python +- [x] Any Critical caps its dimension at 5, making a pass impossible +- [x] Reviewers emit findings only; no score in reviewer output +- [x] Weights architecture 30 / tests 30 / lint 20 / intent 20 +- [x] All reviewers recalibrated and cross-dimension dedup added + +## Files Likely Involved + +- `scripts/score_reviews.py` +- `.claude/agents/` +- `tests/test_score_reviews.py` + +## Status + +Done. + +## Notes + +**The pivotal change in the project.** Score progressions became meaningful evidence afterwards (RHAI-74: 2.4 -> 4.9 -> 7.2 -> 9.4). The pressure moved rather than vanishing: classification is now the whole game, and there is still no calibration test asserting a known-Critical is classified Critical. diff --git a/docs/tasks/done/task-document-cross-language-lessons.md b/docs/tasks/done/task-document-cross-language-lessons.md new file mode 100644 index 0000000..7e176b7 --- /dev/null +++ b/docs/tasks/done/task-document-cross-language-lessons.md @@ -0,0 +1,36 @@ +--- +id: task-document-cross-language-lessons +title: Document cross-language and validation lessons +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-154 +commits: ["71d8ecc", "019023d", "1d23088"] +--- + +# Task: Document cross-language and validation lessons + +## Goal + +Write up what the validation runs taught, so the next phase targets the real bottleneck. + +## Context + +Four validation epics across three languages produced a consistent signal. + +## Acceptance Criteria + +- [x] Lessons captured in-repo +- [x] Findings drove the next phase's scope + +## Files Likely Involved + +- `README.md` + +## Status + +Done. + +## Notes + +Headline finding: **v1 quality was the bottleneck, not review accuracy.** Most epics spent three or four iterations recovering from a weak initial design, which is what motivated spec-first generation ([ADR-0016]). The write-up itself has since gone stale — see [[bug-readme-is-stale]]. diff --git a/docs/tasks/done/task-fix-otel-cost-telemetry.md b/docs/tasks/done/task-fix-otel-cost-telemetry.md new file mode 100644 index 0000000..1627cc7 --- /dev/null +++ b/docs/tasks/done/task-fix-otel-cost-telemetry.md @@ -0,0 +1,40 @@ +--- +id: task-fix-otel-cost-telemetry +title: "Fix Cost and Telemetry: OTEL data never reached the data repo" +type: task +status: done +repos: [epic-code-gen-pipeline] +jira: RHAIFIRST-211 +commits: ["6b47349", "6ef9fcd"] +--- + +# Task: Fix Cost and Telemetry: OTEL data never reached the data repo + +## Goal + +Get per-run Claude cost into the data repo so spend is visible per strategy. + +## Context + +An OTLP collector was running and writing `claude-otel.jsonl`, but the file was never persisted, so the cost view was empty. + +## Acceptance Criteria + +- [x] `claude-otel.jsonl` copied into the strategy directory per pass +- [x] `otel_cost_usd` extracted and written into `run-log.jsonl` +- [x] Telemetry recorded even on a no-op run +- [x] Deltas summed, so subagent usage is counted + +## Files Likely Involved + +- `ci-scripts/otel-collector.py` +- `ci-scripts/otel-summary.py` +- `ci-scripts/push-results.py` + +## Status + +Done. + +## Notes + +~$80 logged across 39 passes. Claude Code emits delta-temporality metrics, so summing deltas is required rather than optional. The OTEL files are also the bulk of the data repo's size. diff --git a/docs/tasks/done/task-fix-state-log-run-log-jsonl.md b/docs/tasks/done/task-fix-state-log-run-log-jsonl.md new file mode 100644 index 0000000..4e7d9a2 --- /dev/null +++ b/docs/tasks/done/task-fix-state-log-run-log-jsonl.md @@ -0,0 +1,39 @@ +--- +id: task-fix-state-log-run-log-jsonl +title: "Fix State Log: push-results.py never wrote run-log.jsonl" +type: task +status: done +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-210 +commits: ["f954967", "6a44d8d", "7bedbb0"] +--- + +# Task: Fix State Log: push-results.py never wrote run-log.jsonl + +## Goal + +Make the state log view show real transitions. + +## Context + +The dashboard's Jira State Log view had nothing to render because `run-log.jsonl` was never written, and transitions were being inferred rather than recorded. + +## Acceptance Criteria + +- [x] `actions.json` written by `run_pipeline.py` with `from`/`to` per transition +- [x] Passed to `push-results.py` via `--actions-json` +- [x] `run-log.jsonl` appended once per pass +- [x] `from` state recovered from prior entries when `actions.json` is absent + +## Files Likely Involved + +- `ci-scripts/push-results.py` +- `scripts/run_pipeline.py` + +## Status + +Done. + +## Notes + +39 passes are now recorded across seven strategies. The fallback that replays prior entries to recover `from` exists because early runs predate `actions.json`. diff --git a/docs/tasks/done/task-fork-and-pr-creation.md b/docs/tasks/done/task-fork-and-pr-creation.md new file mode 100644 index 0000000..fa234cf --- /dev/null +++ b/docs/tasks/done/task-fork-and-pr-creation.md @@ -0,0 +1,43 @@ +--- +id: task-fork-and-pr-creation +title: Fork creation, push-to-fork, and PR creation for CI +type: task +status: done +repos: [epic-code-gen] +commits: ["a0de047", "2cfab1e", "5d7c799", "addaaa3", "4169f06", "19abb33"] +decisions: [ADR-0030] +--- + +# Task: Fork creation, push-to-fork, and PR creation for CI + +## Goal + +Open PRs on other teams' repos without write access, under an unambiguous bot identity. + +## Context + +Target repos belong to other teams. The pipeline has no write access, and a PR that appears to come from a human misrepresents its authorship. + +## Acceptance Criteria + +- [x] Fork created if absent, synced and upstream-fetched before branching +- [x] Branch `epic/<EPIC_ID>` +- [x] Git identity derived from the token +- [x] Default fork owner `dora-the-ai-coder` +- [x] Target repo's own PR template and detected default branch used +- [x] Token-embedded URLs sanitized from error output + +## Files Likely Involved + +- `scripts/clone_target.py` +- `scripts/push_to_fork.py` +- `scripts/create_pr.py` +- `scripts/github_utils.py` + +## Status + +Done. + +## Notes + +Nine real PRs across seven repos, five merged. Using the target's own PR template makes the PR read as a native contribution rather than an automated dump. diff --git a/docs/tasks/done/task-frontmatter-schema-module.md b/docs/tasks/done/task-frontmatter-schema-module.md new file mode 100644 index 0000000..30fad39 --- /dev/null +++ b/docs/tasks/done/task-frontmatter-schema-module.md @@ -0,0 +1,40 @@ +--- +id: task-frontmatter-schema-module +title: Frontmatter schema module and CLI +type: task +status: done +repos: [epic-code-gen] +commits: ["ae79932"] +decisions: [ADR-0002] +--- + +# Task: Frontmatter schema module and CLI + +## Goal + +One module owning every artifact schema, with a CLI so skills never parse YAML. + +## Context + +Artifacts are markdown that agents read as prose, but the pipeline needs structured fields from them. Parsing prose is unreliable; a parallel index drifts. + +## Acceptance Criteria + +- [x] `SCHEMAS` for epic-task, codegen-run, codegen-review in one module +- [x] Types, required flags, enums, defaults +- [x] Validation on read +- [x] `frontmatter.py` CLI: schema / read / set / merge-run-metadata + +## Files Likely Involved + +- `scripts/artifact_utils.py` +- `scripts/frontmatter.py` +- `tests/test_artifact_utils.py` + +## Status + +Done. + +## Notes + +This single-definition-site property is what made the RHAIFIRST-374 fix small. `frontmatter.py` itself still has **no tests** ([[task-add-tests-for-untested-modules]]), and `run-metadata.yaml` is not actually schema-validated. diff --git a/docs/tasks/done/task-gitlab-ci-and-ci-scripts.md b/docs/tasks/done/task-gitlab-ci-and-ci-scripts.md new file mode 100644 index 0000000..1f0be0a --- /dev/null +++ b/docs/tasks/done/task-gitlab-ci-and-ci-scripts.md @@ -0,0 +1,43 @@ +--- +id: task-gitlab-ci-and-ci-scripts +title: epic-code-gen-pipeline repo — GitLab CI and ci-scripts +type: task +status: done +repos: [epic-code-gen-pipeline] +jira: RHAIFIRST-202 +commits: ["62508f3", "059967d", "9b30aba"] +decisions: [ADR-0010] +--- + +# Task: epic-code-gen-pipeline repo — GitLab CI and ci-scripts + +## Goal + +A thin CI shell that sets up the environment, runs the orchestrator, and pushes results. + +## Context + +Logic in YAML and shell cannot be unit-tested or run locally, so every iteration costs a pipeline run. + +## Acceptance Criteria + +- [x] `.gitlab-ci.yml` with codegen / trigger / secret-detection stages +- [x] Four shell scripts, none containing business logic +- [x] Manual trigger with `STRATEGY_KEYS` as the only operator input +- [x] OTEL collection wired in + +## Files Likely Involved + +- `.gitlab-ci.yml` +- `ci-scripts/setup-env.sh` +- `ci-scripts/run-codegen.sh` +- `ci-scripts/pipeline-post.sh` +- `ci-scripts/clone-data-repo.sh` + +## Status + +Done. + +## Notes + +16 files, 1,750 lines total. Data-repo clone auth took seven attempts before matching `strat-pipeline`'s scripts exactly. The untested shell seam is where [[bug-multi-strategy-runs-lose-run-record]] lives. diff --git a/docs/tasks/done/task-idempotent-pipeline.md b/docs/tasks/done/task-idempotent-pipeline.md new file mode 100644 index 0000000..d4af91e --- /dev/null +++ b/docs/tasks/done/task-idempotent-pipeline.md @@ -0,0 +1,39 @@ +--- +id: task-idempotent-pipeline +title: Idempotent pipeline runs and duplicate PR handling +type: task +status: done +repos: [epic-code-gen] +commits: ["2e59f0c", "da3beaf", "fb3b5dc", "1b06fbe", "de256bb"] +decisions: [ADR-0009] +--- + +# Task: Idempotent pipeline runs and duplicate PR handling + +## Goal + +Make re-running a pass safe, since re-running is how epics progress. + +## Context + +The convergence loop means the same epic is processed on every run. Anything not idempotent duplicates work or corrupts state. + +## Acceptance Criteria + +- [x] Active epics skipped; merged PRs reconciled +- [x] Duplicate PR creation handled gracefully +- [x] Completed codegen artifacts reused instead of re-running Claude +- [x] A non-zero Claude exit with artifacts present is not a failure + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `scripts/create_pr.py` + +## Status + +Done. + +## Notes + +`de256bb` changed artifact detection to `v*/diff.patch` rather than `run-metadata.yaml`. Artifact presence remains a weak liveness proxy for a crashed run — [[bug-artifact-presence-is-weak-liveness-proxy]]. diff --git a/docs/tasks/done/task-jira-direct-epic-fetching.md b/docs/tasks/done/task-jira-direct-epic-fetching.md new file mode 100644 index 0000000..4ee8fca --- /dev/null +++ b/docs/tasks/done/task-jira-direct-epic-fetching.md @@ -0,0 +1,42 @@ +--- +id: task-jira-direct-epic-fetching +title: Fetch epics directly from Jira with a dependency DAG +type: task +status: done +repos: [epic-code-gen] +commits: ["46a08cd", "996640a", "7d50db2"] +decisions: [ADR-0007] +--- + +# Task: Fetch epics directly from Jira with a dependency DAG + +## Goal + +Make Jira the source of truth for which epics exist and which are eligible. + +## Context + +The POC parsed epics out of generated HTML reports — a stale snapshot with no notion of a human closing an epic, retargeting it, or adding a blocker. + +## Acceptance Criteria + +- [x] Child work items fetched directly; real Jira keys as `epic_id` +- [x] Dependency DAG from 'Blocks' links, stored as `dependencies` and `blocks` +- [x] Eligibility computed from current Jira status plus the DAG +- [x] Out-of-scope projects and skip-labelled epics excluded +- [x] Status fallback aliases for workflow compatibility +- [x] 45 tests + +## Files Likely Involved + +- `scripts/fetch_jira_epics.py` +- `scripts/jira_utils.py` +- `tests/test_fetch_jira_epics.py` + +## Status + +Done. + +## Notes + +Eligibility is recomputed from scratch every run, so there is no internal state to drift. `7d50db2` was needed when Jira status names changed. Roughly 750 of this module's 1,020 lines are HTML report rendering, which does not belong in a module named 'fetch' — [[task-extract-html-report-generation]]. diff --git a/docs/tasks/done/task-pattern-discovery-expansion.md b/docs/tasks/done/task-pattern-discovery-expansion.md new file mode 100644 index 0000000..f32a089 --- /dev/null +++ b/docs/tasks/done/task-pattern-discovery-expansion.md @@ -0,0 +1,38 @@ +--- +id: task-pattern-discovery-expansion +title: Expand pattern discovery and enforce it before design +type: task +status: done +repos: [epic-code-gen] +commits: ["d3a5a24", "37a1d67", "9b65e8c", "69b74cf", "c58f98a"] +decisions: [ADR-0018] +--- + +# Task: Expand pattern discovery and enforce it before design + +## Goal + +Gather enough evidence about the target repo that the design is derived rather than invented. + +## Context + +Architecture is the highest-weighted dimension at 30%, and it measures conformance to conventions that only exist in the repo. One reference file is not a convention. + +## Acceptance Criteria + +- [x] 7a explicit refs, 7b concept search, 7c target + 5-10 siblings + sibling dirs + callers, 7d conventions docs +- [x] All agent-readiness files scanned (CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md, COPILOT.md, CONVENTIONS.md, CONSTITUTION.md) +- [x] No cap on convention lines read +- [x] Discovery completes **before** brainstorming is dispatched + +## Files Likely Involved + +- `.claude/skills/epic-codegen/SKILL.md` + +## Status + +Done. + +## Notes + +`9b65e8c` is the important one: without enforced ordering the design was invented first and evidence found afterwards. Enforcement is still a prompt instruction, not a mechanism. diff --git a/docs/tasks/done/task-pipeline-story-dashboard.md b/docs/tasks/done/task-pipeline-story-dashboard.md new file mode 100644 index 0000000..563fbb6 --- /dev/null +++ b/docs/tasks/done/task-pipeline-story-dashboard.md @@ -0,0 +1,38 @@ +--- +id: task-pipeline-story-dashboard +title: Pipeline Story — interactive HTML dashboard for strategy visualization +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-213 +commits: ["9ba09f8", "81367c8", "099a249", "659cbfc"] +--- + +# Task: Pipeline Story — interactive HTML dashboard for strategy visualization + +## Goal + +A narrative view of a strategy's journey for demos and review. + +## Context + +The three operational dashboard views answer 'what is the state'. This answers 'what happened', including the agent loop structure and the epic dependency DAG. + +## Acceptance Criteria + +- [x] Animated DAG visualization with activity log +- [x] Per-loop timeline with an external review panel +- [x] Progressive-disclosure strategy panel with AI/human badges +- [x] Customer names and partner product references redacted + +## Files Likely Involved + +- `scripts/pipeline_story.py` + +## Status + +Done. + +## Notes + +Built here across ~20 commits on 07-03/04, then moved to `epic-code-gen-dashboard` (`099a249`, `659cbfc`) — it is a consumer, not part of the engine. `e092c34` redacted customer names before it was shared. diff --git a/docs/tasks/done/task-rebase-epic-branches-every-cycle.md b/docs/tasks/done/task-rebase-epic-branches-every-cycle.md new file mode 100644 index 0000000..80d0ce8 --- /dev/null +++ b/docs/tasks/done/task-rebase-epic-branches-every-cycle.md @@ -0,0 +1,41 @@ +--- +id: task-rebase-epic-branches-every-cycle +title: Rebase epic branches onto upstream base every review-response cycle +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-376 +commits: ["164d21e"] +decisions: [ADR-0031] +--- + +# Task: Rebase epic branches onto upstream base every review-response cycle + +## Goal + +Keep PRs mergeable and author fixes against current upstream code. + +## Context + +Because progress happens one step per run and PRs wait on humans, a branch can sit for days while upstream moves. PRs drifted into CONFLICTING, and fixes were written against stale code. + +## Acceptance Criteria + +- [x] Rebase runs at the start of every review-response cycle +- [x] `rebase_onto_base()` drives the git sequence; a subagent edits only the working tree +- [x] Result pushed with `--force-with-lease`, never a plain force +- [x] A cycle that rebases nothing and finds nothing actionable does not consume an iteration + +## Files Likely Involved + +- `scripts/rebase_pr.py` +- `scripts/review_response.py` +- `tests/test_rebase_pr.py` + +## Status + +Done. + +## Notes + +Verified: `--force-with-lease` at `rebase_pr.py:233`. `MAX_CONFLICT_ROUNDS = 10`, `CONFLICT_AGENT_TIMEOUT = 900` — both plain constants, not env-tunable, unlike the two review-response timeouts. The same commit also fixed RHAIFIRST-375. diff --git a/docs/tasks/done/task-repo-readiness-scoring.md b/docs/tasks/done/task-repo-readiness-scoring.md new file mode 100644 index 0000000..2ec305d --- /dev/null +++ b/docs/tasks/done/task-repo-readiness-scoring.md @@ -0,0 +1,37 @@ +--- +id: task-repo-readiness-scoring +title: Repo readiness scoring +type: task +status: done +repos: [epic-code-gen] +commits: ["ae79932", "e766e5f"] +--- + +# Task: Repo readiness scoring + +## Goal + +Decide whether a target repo is a viable codegen target before spending anything on it. + +## Context + +Generated code is only as reviewable as the signals the repo provides. A repo with no tests and no CI cannot tell us whether generated code works. + +## Acceptance Criteria + +- [x] Six dimensions scored out of 12, threshold 8 +- [x] Integration tests, lint in CI, CI signals, context docs, CODEOWNERS, language properties +- [x] Score recorded in run metadata + +## Files Likely Involved + +- `scripts/repo_readiness.py` +- `tests/test_repo_readiness.py` + +## Status + +Done. + +## Notes + +`e766e5f` softened this from a hard gate — real repos score imperfectly for reasons that don't block codegen. RHAI-68 ran at readiness 9 with `codeowners: 0`. diff --git a/docs/tasks/done/task-review-cycle-extraction.md b/docs/tasks/done/task-review-cycle-extraction.md new file mode 100644 index 0000000..b7ddab7 --- /dev/null +++ b/docs/tasks/done/task-review-cycle-extraction.md @@ -0,0 +1,39 @@ +--- +id: task-review-cycle-extraction +title: Extract the review loop into review_cycle.py +type: task +status: done +repos: [epic-code-gen] +commits: ["13d9e63", "f7da07c", "4f1cc62", "5e26193"] +decisions: [ADR-0026] +--- + +# Task: Extract the review loop into review_cycle.py + +## Goal + +Move nine steps of loop bookkeeping out of prose and into testable Python. + +## Context + +The loop lived in SKILL.md as instructions to a model that was simultaneously managing a 6-hour context and being compacted. Observed failures: reviewers dispatched and never waited for; the orchestrator writing review files itself; scores estimated rather than run. + +## Acceptance Criteria + +- [x] Subcommands prompts / wait / verify / score / triage-prompt / dispatch-context +- [x] `REVIEWERS` table owns the six reviewers, four scored +- [x] Anti-fallback guardrail: if dispatch fails, fail — do not improvise +- [x] 38 tests + +## Files Likely Involved + +- `scripts/review_cycle.py` +- `tests/test_review_cycle.py` + +## Status + +Done. + +## Notes + +The skill still has to *call* these in order, so the anti-fallback rule remains a prompt instruction. [[bug-review-gate-is-advisory]] is that gap being exercised. `wait` also returns before the unscored verifiers finish — [[bug-wait-returns-before-unscored-reviewers-finish]]. diff --git a/docs/tasks/done/task-run-index-for-dashboard.md b/docs/tasks/done/task-run-index-for-dashboard.md new file mode 100644 index 0000000..acfbd03 --- /dev/null +++ b/docs/tasks/done/task-run-index-for-dashboard.md @@ -0,0 +1,38 @@ +--- +id: task-run-index-for-dashboard +title: Aggregate run outcomes into index.json +type: task +status: done +repos: [epic-code-gen] +commits: ["294b19f"] +--- + +# Task: Aggregate run outcomes into index.json + +## Goal + +One file summarising every run for dashboard consumption. + +## Context + +Per-epic `run-metadata.yaml` files are the source of truth but require a directory walk to summarise. + +## Acceptance Criteria + +- [x] Scans `codegen-runs/*/run-metadata.yaml` +- [x] Writes `index.json` with all runs, total, and a summary by `codegen_outcome` +- [x] Called at the end of every `/epic-codegen` run +- [x] 26 tests + +## Files Likely Involved + +- `scripts/run_index.py` +- `tests/test_run_index.py` + +## Status + +Done. + +## Notes + +Carries its own PyYAML-optional fallback parser, which is unreachable — `pyyaml` is a hard dependency and is baked into the image. One of six YAML parsers ([[task-consolidate-yaml-parsers]]). Also references `scores_by_dimension`, which is not the name production writes. diff --git a/docs/tasks/done/task-spec-first-generation.md b/docs/tasks/done/task-spec-first-generation.md new file mode 100644 index 0000000..62e6d52 --- /dev/null +++ b/docs/tasks/done/task-spec-first-generation.md @@ -0,0 +1,42 @@ +--- +id: task-spec-first-generation +title: Spec-first generation via Superpowers brainstorming and writing-plans +type: task +status: done +repos: [epic-code-gen] +commits: ["ffe24ea", "d41a7c0", "3ba1751", "bbce28f"] +decisions: [ADR-0016, ADR-0019] +--- + +# Task: Spec-first generation via Superpowers brainstorming and writing-plans + +## Goal + +Produce a spec containing approach exploration and trade-offs, not a restated epic. + +## Context + +Template-filled specs produced weak v1 diffs: most epics burned three or four iterations climbing out of a bad design, and the review loop is better at catching defects than at redirecting an approach. + +## Acceptance Criteria + +- [x] `brainstorming` invoked via a dedicated subagent acting as the human partner +- [x] `writing-plans` invoked via a second subagent +- [x] Each skill isolated in its own subagent +- [x] Spec review gate validates the spec against real repo patterns before planning +- [x] Skill invocation verified; retry-then-fail, never fallback + +## Files Likely Involved + +- `.claude/agents/design-spec-generator.md` +- `.claude/agents/plan-generator.md` +- `.claude/agents/spec-reviewer.md` +- `.claude/skills/epic-codegen/SKILL.md` + +## Status + +Done. + +## Notes + +`bbce28f` matters more than it looks: a subagent that silently fails to invoke its skill still produces plausible output, so the labeled conversation log is the evidence the skill actually ran. diff --git a/docs/tasks/done/task-target-validation-and-language-detection.md b/docs/tasks/done/task-target-validation-and-language-detection.md new file mode 100644 index 0000000..83cd15e --- /dev/null +++ b/docs/tasks/done/task-target-validation-and-language-detection.md @@ -0,0 +1,39 @@ +--- +id: task-target-validation-and-language-detection +title: Target repo validation and language detection +type: task +status: done +repos: [epic-code-gen] +commits: ["0aecf6f"] +decisions: [ADR-0025] +--- + +# Task: Target repo validation and language detection + +## Goal + +Run a target repo's own lint / typecheck / test commands and report the result structurally. + +## Context + +Checks cannot be hardcoded: every repo defines them differently, in a Makefile or package.json. + +## Acceptance Criteria + +- [x] Language detected from markers (Go, Python, TS, JS, Rust) +- [x] Commands discovered from Makefile targets and package.json scripts +- [x] Structured `validation.json` with per-check results +- [x] `all_passed` false if any check is unrunnable or none discovered + +## Files Likely Involved + +- `scripts/validate_target.py` +- `tests/test_validate_target.py` + +## Status + +Done. + +## Notes + +The `unrunnable` vs `failed` distinction came later ([[task-toolchain-preflight]]) and is the subtlest logic in the repo. Consumers must read `all_passed`, never per-check keys. diff --git a/docs/tasks/done/task-toolchain-preflight.md b/docs/tasks/done/task-toolchain-preflight.md new file mode 100644 index 0000000..26f1efa --- /dev/null +++ b/docs/tasks/done/task-toolchain-preflight.md @@ -0,0 +1,41 @@ +--- +id: task-toolchain-preflight +title: Gate codegen on toolchain preflight +type: task +status: done +repos: [epic-code-gen] +commits: ["b439623", "6704253", "e7c9dac"] +decisions: [ADR-0025] +--- + +# Task: Gate codegen on toolchain preflight + +## Goal + +Fail before generating when a required executable is missing, and never score an environment fault as bad code. + +## Context + +`kale`'s Makefile drives `uv run ruff`. Without `uv` the recipe exits 127, GNU make reports `Error 127`, and the reviewer scored `lint=5.0` — an environment fault charged to an epic that could not have caused or fixed it. + +## Acceptance Criteria + +- [x] `--preflight` checks required tools and runs nothing; exit 2 distinguishes it from exit 1 +- [x] Required tools from repo markers **and** variable-expanded Makefile recipes, following prerequisites +- [x] Only the lint/typecheck/test targets inspected, so unrelated recipes don't gate +- [x] A gap flags the epic and generates nothing; status stays `Ready` to retry +- [x] 43 tests + +## Files Likely Involved + +- `scripts/validate_target.py` +- `scripts/run_pipeline.py` +- `tests/test_toolchain_preflight.py` + +## Status + +Done. + +## Notes + +`6704253` narrowed it after it began blocking on non-tools. `e7c9dac` verifies `uv` with `test -x` rather than executing it, because the freshly installed amd64 binary segfaults under qemu when cross-building from arm64. Covers checks that *couldn't run* — not checks that fail for reasons the epic didn't cause ([[bug-baseline-check-failures-scored-as-epic]]). diff --git a/docs/tasks/done/task-triage-memory-and-oscillation.md b/docs/tasks/done/task-triage-memory-and-oscillation.md new file mode 100644 index 0000000..7857e58 --- /dev/null +++ b/docs/tasks/done/task-triage-memory-and-oscillation.md @@ -0,0 +1,40 @@ +--- +id: task-triage-memory-and-oscillation +title: History-aware triage with accepted-findings carry-forward +type: task +status: done +repos: [epic-code-gen] +commits: ["cf1da6e", "050732f", "24d8078", "8a4a5c7", "0aa99e8"] +decisions: [ADR-0033] +--- + +# Task: History-aware triage with accepted-findings carry-forward + +## Goal + +Stop triage relitigating findings it already dismissed, and stop fixes oscillating between dimensions. + +## Context + +Reviewers are not perfectly consistent between versions. Scores were plateauing while findings rotated: a fix for one dimension created a finding in another, and triage revisited findings it had already dismissed with reason. + +## Acceptance Criteria + +- [x] Findings dismissed with a reason carried forward in `tmp/accepted-findings-<EPIC>.json` +- [x] Triage reads prior versions' decisions +- [x] Cross-dimension dedup +- [x] Fix work batched: apply all fixes, then test and commit once +- [x] Fix loop runs in a fresh-context subagent + +## Files Likely Involved + +- `.claude/agents/iteration-reviewer.md` +- `scripts/review_cycle.py` + +## Status + +Done. + +## Notes + +Decisions are recorded in `decision-log.md` per version, so triage is auditable. `8a4a5c7` moving the loop to a fresh context matters because triage quality degrades badly under context pressure. diff --git a/docs/tasks/done/task-unscored-verifiers.md b/docs/tasks/done/task-unscored-verifiers.md new file mode 100644 index 0000000..bc94278 --- /dev/null +++ b/docs/tasks/done/task-unscored-verifiers.md @@ -0,0 +1,40 @@ +--- +id: task-unscored-verifiers +title: Add wiring and interaction verifiers +type: task +status: done +repos: [epic-code-gen] +commits: ["5e19a14", "d0f2a2d"] +decisions: [ADR-0028] +--- + +# Task: Add wiring and interaction verifiers + +## Goal + +Catch defects the four structural dimensions provably miss, without disturbing the weights. + +## Context + +Code can pass architecture, tests, lint, and intent and still not work: a handler nothing calls, a callback race, a missing switch branch. + +## Acceptance Criteria + +- [x] `wiring-verifier` traces trigger -> chain -> outcome per AC +- [x] `interaction-verifier` traces user interactions and enum/branch completeness +- [x] Both dispatched in parallel with the scored reviewers +- [x] Neither affects the score; findings feed triage + +## Files Likely Involved + +- `.claude/agents/wiring-verifier.md` +- `.claude/agents/interaction-verifier.md` +- `scripts/review_cycle.py` + +## Status + +Done. + +## Notes + +Running on 20 and 19 epic-versions respectively. Advisory findings can be ignored, and advisory signals in this system have a track record of being ignored. Neither is documented in README.md or CLAUDE.md. diff --git a/docs/tasks/done/task-ux-prototype-pipeline.md b/docs/tasks/done/task-ux-prototype-pipeline.md new file mode 100644 index 0000000..a55171d --- /dev/null +++ b/docs/tasks/done/task-ux-prototype-pipeline.md @@ -0,0 +1,40 @@ +--- +id: task-ux-prototype-pipeline +title: Prototype-driven UX acceptance criteria +type: task +status: done +repos: [epic-code-gen] +commits: ["4def105", "75e70c9", "bf0e7cc", "58e4bd7", "976a59d"] +decisions: [ADR-0020] +--- + +# Task: Prototype-driven UX acceptance criteria + +## Goal + +Make a UXD prototype checkable, so generated UI matches the design and not just the epic text. + +## Context + +Front-end epics come with an HTML prototype attached to the Jira issue. The prototype is the real specification; the epic body summarises it lossily. Generated UI was passing architecture and tests while having wrong labels and missing helper text. + +## Acceptance Criteria + +- [x] Playwright parser extracts components, alerts, disabled states, labels, helper text, popovers, checkboxes, radios, badges +- [x] Per-scenario markdown plus cropped screenshots +- [x] Numbered UX ACs (`UX-G1`, `UX-S1-1`) verified by the intent reviewer +- [x] Prototype deviations non-negotiable in triage + +## Files Likely Involved + +- `scripts/parse_prototype.js` +- `.claude/agents/ux-ac-extractor.md` +- `.claude/agents/intent-reviewer.md` + +## Status + +Done. + +## Notes + +`RHOAIENG-72103` is the first epic through this path, Done at 8.15. `bf0e7cc` was needed because triage had been dismissing UX findings as out of scope. `parse_prototype.js` is 699 lines with **no tests** and no JS test runner configured. Deliberately carries no `jira:` key: the epic RHAIFIRST-233 is still open because children 234/235/236 remain, so claiming it here would report a live epic as closed. The epic is tracked by [[task-prototype-driven-codegen-remaining]]. diff --git a/docs/tasks/done/task-v2-review-response-foundation.md b/docs/tasks/done/task-v2-review-response-foundation.md new file mode 100644 index 0000000..da0a359 --- /dev/null +++ b/docs/tasks/done/task-v2-review-response-foundation.md @@ -0,0 +1,41 @@ +--- +id: task-v2-review-response-foundation +title: V2 review response — foundation utilities +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-212 +commits: ["99c9bc6"] +decisions: [ADR-0032] +--- + +# Task: V2 review response — foundation utilities + +## Goal + +Primitives for reading PR state and triaging review comments. + +## Context + +Regenerating a branch under review throws away reviewer effort and makes the PR's history useless. Fixes must land as commits on top of the existing branch. + +## Acceptance Criteria + +- [x] Existing branch checked out from the fork, never regenerated +- [x] Humans always addressed; bots selectively +- [x] Only code inside our own diff is touched +- [x] Every comment replied to, processed IDs recorded + +## Files Likely Involved + +- `scripts/pr_lifecycle.py` +- `scripts/github_utils.py` +- `scripts/clone_target.py` + +## Status + +Done. + +## Notes + +`compute_diff_scope` / `is_comment_in_scope` are what keep a reviewer's aside about unrelated code from becoming a refactor. `checkout_existing_branch` is the entry point that makes commit-on-top possible. diff --git a/docs/tasks/done/task-v2-review-response-orchestrator.md b/docs/tasks/done/task-v2-review-response-orchestrator.md new file mode 100644 index 0000000..ffadce7 --- /dev/null +++ b/docs/tasks/done/task-v2-review-response-orchestrator.md @@ -0,0 +1,41 @@ +--- +id: task-v2-review-response-orchestrator +title: V2 review response — orchestrator and agents +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-212 +commits: ["2f263da"] +decisions: [ADR-0032] +--- + +# Task: V2 review response — orchestrator and agents + +## Goal + +Drive a full review-response cycle: triage, fix, validate, reply. + +## Context + +Regenerating a branch under review throws away reviewer effort and makes the PR's history useless. Fixes must land as commits on top of the existing branch. + +## Acceptance Criteria + +- [x] Existing branch checked out from the fork, never regenerated +- [x] Humans always addressed; bots selectively +- [x] Only code inside our own diff is touched +- [x] Every comment replied to, processed IDs recorded + +## Files Likely Involved + +- `scripts/review_response.py` +- `.claude/agents/review-fix-agent.md` +- `.claude/agents/sanity-check-agent.md` + +## Status + +Done. + +## Notes + +One agent handles all comments and makes one commit — per-comment agents produced conflicting edits and a shredded history. The post-fix check is deliberately lightweight: validation plus a sanity check, not a full four-dimension re-review. diff --git a/docs/tasks/done/task-v2-review-response-state-machine.md b/docs/tasks/done/task-v2-review-response-state-machine.md new file mode 100644 index 0000000..0978b1b --- /dev/null +++ b/docs/tasks/done/task-v2-review-response-state-machine.md @@ -0,0 +1,40 @@ +--- +id: task-v2-review-response-state-machine +title: V2 review response — state machine integration +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-212 +commits: ["6aecf9b", "48185b8", "7f4c35f"] +decisions: [ADR-0032] +--- + +# Task: V2 review response — state machine integration + +## Goal + +Wire the review-response loop into the CI state machine. + +## Context + +Regenerating a branch under review throws away reviewer effort and makes the PR's history useless. Fixes must land as commits on top of the existing branch. + +## Acceptance Criteria + +- [x] Existing branch checked out from the fork, never regenerated +- [x] Humans always addressed; bots selectively +- [x] Only code inside our own diff is touched +- [x] Every comment replied to, processed IDs recorded + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `tests/test_review_response.py` + +## Status + +Done. + +## Notes + +`_ci_handle_pr_changes` was rewritten for this. `48185b8` fixed six issues found in this work's own review before merge — the process working as intended. 66 tests. diff --git a/docs/tasks/done/task-validate-cross-repo-codegen.md b/docs/tasks/done/task-validate-cross-repo-codegen.md new file mode 100644 index 0000000..8534623 --- /dev/null +++ b/docs/tasks/done/task-validate-cross-repo-codegen.md @@ -0,0 +1,38 @@ +--- +id: task-validate-cross-repo-codegen +title: Validate cross-repo codegen (RHAISTRAT-1749-E002) +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-152 +commits: ["076ea82", "370a6a6"] +--- + +# Task: Validate cross-repo codegen (RHAISTRAT-1749-E002) + +## Goal + +Confirm one strategy can target more than one repository, resolved automatically. + +## Context + +RHAISTRAT-1749 spans `mlflow-go`, `odh-dashboard`, and `mlflow` — three repos, three languages, one strategy. + +## Acceptance Criteria + +- [x] Target repo resolved per epic, not per strategy +- [x] Keyword mapping in `config/repo_mapping.json` with an LLM fallback +- [x] Each epic generated against its own clone + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `config/repo_mapping.json` + +## Status + +Done. + +## Notes + +Resolution is keyword-first with an LLM fallback ([ADR-0007] context). Adding a repo is a mapping entry — but if its toolchain is new, it is also an image change ([ADR-0011]). diff --git a/docs/tasks/done/task-validate-go-repo-codegen.md b/docs/tasks/done/task-validate-go-repo-codegen.md new file mode 100644 index 0000000..df2b2ae --- /dev/null +++ b/docs/tasks/done/task-validate-go-repo-codegen.md @@ -0,0 +1,39 @@ +--- +id: task-validate-go-repo-codegen +title: Validate epic-codegen on a Go repo (RHAISTRAT-1749-E001) +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-151 +commits: ["d2ceb4f", "294b19f"] +--- + +# Task: Validate epic-codegen on a Go repo (RHAISTRAT-1749-E001) + +## Goal + +Run the full pipeline end to end on a real Go epic and confirm the output is mergeable. + +## Context + +First real target: expose `ModelConfig` on `Prompt`/`PromptVersion` in the MLflow Go SDK. This was the run that established the pipeline worked at all. + +## Acceptance Criteria + +- [x] Diff generated against `mlflow-go` +- [x] All four dimensions scored +- [x] Weighted score >= 8.0 +- [x] Result recorded in the data repo + +## Files Likely Involved + +- `scripts/run_pipeline.py` +- `.claude/skills/epic-codegen/SKILL.md` + +## Status + +Done. + +## Notes + +Passed on the **first iteration** at 9.4 — 224 lines across 4 files, 6 new tests. Later merged as `mlflow-go` #21. This single result set expectations that the next several epics did not meet, which is what motivated [[task-spec-first-generation]]. diff --git a/docs/tasks/done/task-validate-shared-blocker-handling.md b/docs/tasks/done/task-validate-shared-blocker-handling.md new file mode 100644 index 0000000..63171a8 --- /dev/null +++ b/docs/tasks/done/task-validate-shared-blocker-handling.md @@ -0,0 +1,38 @@ +--- +id: task-validate-shared-blocker-handling +title: Validate shared blocker handling (RHAISTRAT-1748-E001) +type: task +status: done +repos: [epic-code-gen] +jira: RHAIFIRST-153 +commits: ["46a08cd", "1045c53"] +--- + +# Task: Validate shared blocker handling (RHAISTRAT-1748-E001) + +## Goal + +Confirm an epic blocked by another epic waits, and proceeds once the blocker is Done. + +## Context + +Epics within a strategy form a dependency DAG built from Jira 'Blocks' links. This is what makes strategy the unit of work ([ADR-0006]). + +## Acceptance Criteria + +- [x] DAG built from Jira 'Blocks' links +- [x] Blocked epics classified as Blocked, not attempted +- [x] An unblocked epic falls through to codegen in the same run + +## Files Likely Involved + +- `scripts/fetch_jira_epics.py` +- `scripts/run_pipeline.py` + +## Status + +Done. + +## Notes + +`1045c53` added the same-run fall-through, so resolving a blocker doesn't cost an extra pass. Note the asymmetry documented in the state machine doc: this check reads the dependency's **data-repo** state, while initial eligibility reads **Jira**. diff --git a/docs/tasks/pending/task-add-lint-and-typecheck.md b/docs/tasks/pending/task-add-lint-and-typecheck.md new file mode 100644 index 0000000..395486b --- /dev/null +++ b/docs/tasks/pending/task-add-lint-and-typecheck.md @@ -0,0 +1,38 @@ +--- +id: task-add-lint-and-typecheck +title: Add lint and type checking to this repo's own Python +type: task +status: pending +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Task: Add lint and type checking to this repo's own Python + +## Goal + +Hold this repo to the standard it enforces on target repos. + +## Context + +The product's value proposition is enforcing lint on other repositories. This repo has **no linter and no type checker** for 10.5k lines of Python, no config for ruff/flake8/black/mypy, and no `make lint` target. `Dockerfile.ci` installs `markdownlint-cli` for target repos while nothing lints our own code. + +## Acceptance Criteria + +- [ ] `ruff` configured and passing (or the failures triaged into tasks) +- [ ] `make lint` target added and wired into CI +- [ ] Decide on type checking — `mypy` is blocked by [ADR-0003]'s flat layout; record the decision +- [ ] `shellcheck` in the pipeline repo stops being `|| true` + +## Files Likely Involved + +- `pyproject.toml` +- `Makefile` +- `.github/workflows/ledger.yml` + +## Status + +Pending. + +## Notes + +Expect a large initial diff. Land the config with generous ignores first, then narrow — a single PR that reformats 10.5k lines is unreviewable. The pipeline repo's `make lint` silently passes when shellcheck is absent, which is worse than having no target. diff --git a/docs/tasks/pending/task-add-resource-group-to-codegen-run.md b/docs/tasks/pending/task-add-resource-group-to-codegen-run.md new file mode 100644 index 0000000..1b5b292 --- /dev/null +++ b/docs/tasks/pending/task-add-resource-group-to-codegen-run.md @@ -0,0 +1,37 @@ +--- +id: task-add-resource-group-to-codegen-run +title: Add a resource_group to codegen-run and enable scheduling +type: task +status: pending +repos: [epic-code-gen-pipeline] +decisions: [ADR-0008] +--- + +# Task: Add a resource_group to codegen-run and enable scheduling + +## Goal + +Make concurrent runs safe, then move off a manual trigger. + +## Context + +`codegen-run` has no `resource_group`, so two concurrent triggers race on the data repo — mitigated only by push-retry-with-rebase (3 attempts, `-X theirs`). `FOREDER.md` identified this in June as **the** prerequisite for scheduling, and it is the real reason the pipeline is still triggered by hand. + +## Acceptance Criteria + +- [ ] `resource_group` added so runs serialise +- [ ] Verify behaviour with two simultaneous triggers +- [ ] Then add a schedule (`rules: - schedules`) +- [ ] Decide the granularity — global, or per strategy key + +## Files Likely Involved + +- `.gitlab-ci.yml` + +## Status + +Pending. + +## Notes + +Per-strategy granularity would allow useful parallelism, but two strategies still share one data repo and `push-results.py` regenerates the global `summary.json`, so global serialisation is the safe first step. Fix [[bug-multi-strategy-runs-lose-run-record]] before scheduling anything. diff --git a/docs/tasks/pending/task-add-tests-for-untested-modules.md b/docs/tasks/pending/task-add-tests-for-untested-modules.md new file mode 100644 index 0000000..50df2dc --- /dev/null +++ b/docs/tasks/pending/task-add-tests-for-untested-modules.md @@ -0,0 +1,40 @@ +--- +id: task-add-tests-for-untested-modules +title: Add tests for jira_utils, frontmatter, state, and parse_prototype +type: task +status: pending +repos: [epic-code-gen] +--- + +# Task: Add tests for jira_utils, frontmatter, state, and parse_prototype + +## Goal + +Cover the four untested modules the whole pipeline depends on. + +## Context + +703 tests exist, but four load-bearing modules have zero. + +## Acceptance Criteria + +- [ ] `jira_utils.py` (1,055 lines) — especially `markdown_to_adf` / `adf_to_markdown` round-trips +- [ ] `frontmatter.py` (338 lines) — the CLI every skill calls +- [ ] `state.py` (185 lines) — the store all long-running skills depend on +- [ ] `parse_prototype.js` (699 lines) — needs a JS test runner, which the repo has none +- [ ] A calibration test: a known-Critical review file scores 5.0 + +## Files Likely Involved + +- `tests/test_jira_utils.py` +- `tests/test_frontmatter.py` +- `tests/test_state.py` +- `package.json` + +## Status + +Pending. + +## Notes + +`jira_utils.py` is the largest module in the repo and contains a full bidirectional Markdown↔ADF converter — the highest-risk untested surface. The calibration test matters most though: finding parsing **fails open**, so a prompt-drift regression currently looks like flawless code ([ADR-0022]). diff --git a/docs/tasks/pending/task-baseline-validation-diff.md b/docs/tasks/pending/task-baseline-validation-diff.md new file mode 100644 index 0000000..96c479e --- /dev/null +++ b/docs/tasks/pending/task-baseline-validation-diff.md @@ -0,0 +1,41 @@ +--- +id: task-baseline-validation-diff +title: Capture a baseline validation run and diff findings against it +type: task +status: pending +repos: [epic-code-gen] +jira: RHAIFIRST-392 +decisions: [ADR-0025] +--- + +# Task: Capture a baseline validation run and diff findings against it + +## Goal + +Attribute check failures to the epic only when the epic caused them. + +## Context + +A target repo whose `make lint` is red on `main` fails every epic generated against it, and because GNU make stops at the first failing prerequisite, the baseline failure also conceals genuine findings. See [[bug-baseline-check-failures-scored-as-epic]]. + +## Acceptance Criteria + +- [ ] Run validation at `BASE_SHA` before generating, and store the result +- [ ] Diff post-generation findings against the baseline; only new findings are the epic's +- [ ] Baseline failures reported distinctly — like `unrunnable`, not as `failed` +- [ ] Handle the make short-circuit so later sub-targets still run +- [ ] The repo readiness assessment should probably fail a repo that is red on `main` + +## Files Likely Involved + +- `scripts/validate_target.py` +- `scripts/run_pipeline.py` +- `.claude/agents/lint-reviewer.md` + +## Status + +Pending. + +## Notes + +The cleanest framing is that this extends [ADR-0025]'s three-state model to a fourth: `passed` / `failed` / `unrunnable` / `pre-existing`. The make short-circuit is the harder half — running sub-targets individually rather than the aggregate target would solve it but diverges from 'run what the repo runs'. diff --git a/docs/tasks/pending/task-consolidate-yaml-parsers.md b/docs/tasks/pending/task-consolidate-yaml-parsers.md new file mode 100644 index 0000000..2850028 --- /dev/null +++ b/docs/tasks/pending/task-consolidate-yaml-parsers.md @@ -0,0 +1,39 @@ +--- +id: task-consolidate-yaml-parsers +title: Consolidate six YAML parsers and three value coercers +type: task +status: pending +repos: [epic-code-gen] +--- + +# Task: Consolidate six YAML parsers and three value coercers + +## Goal + +One YAML read path, one coercion function. + +## Context + +Six independent YAML-ish parsers exist: `artifact_utils` (PyYAML), `run_index._parse_yaml_simple`, `run_pipeline._parse_flat_yaml`, `run_pipeline._read_metadata_simple`, `fetch_jira_epics._parse_simple_yaml`, and `line.startswith("phase:")` scanning in `review_cycle.py`. Three of them exist purely as a PyYAML-optional fallback — but `pyyaml` is a hard dependency in `pyproject.toml` and baked into `Dockerfile.ci`, so those fallbacks are **unreachable in every supported environment**. + +## Acceptance Criteria + +- [ ] Fallback parsers removed; `artifact_utils.read_frontmatter` is the single read path +- [ ] `frontmatter._coerce_value` / `_infer_value` / `run_index._coerce_value` consolidated +- [ ] Also consolidate: three git wrappers, two HTTP clients, two slug extractors +- [ ] Tests still pass + +## Files Likely Involved + +- `scripts/run_index.py` +- `scripts/run_pipeline.py` +- `scripts/fetch_jira_epics.py` +- `scripts/artifact_utils.py` + +## Status + +Pending. + +## Notes + +The coercers differ subtly — different truthy-string sets (`"yes"`, `"True"`) — so consolidating changes behaviour somewhere. Write the tests first. Note `AGENTS.md` already forbids adding a seventh. diff --git a/docs/tasks/pending/task-converge-run-metadata-schema.md b/docs/tasks/pending/task-converge-run-metadata-schema.md new file mode 100644 index 0000000..6bca65b --- /dev/null +++ b/docs/tasks/pending/task-converge-run-metadata-schema.md @@ -0,0 +1,38 @@ +--- +id: task-converge-run-metadata-schema +title: Converge the run-metadata.yaml schema and pin it +type: task +status: pending +repos: [epic-code-gen, epic-code-gen-pipeline-data] +decisions: [ADR-0002, ADR-0014] +--- + +# Task: Converge the run-metadata.yaml schema and pin it + +## Goal + +One documented schema for the state file, validated on write. + +## Context + +Four generations coexist in the data repo. Two names for one concept (`dimension_scores` vs `scores_by_dimension`), three counters with no documented relationship (`current_version`, `versions`, `final_version`), and `merge_run_metadata` validates only two rules while every other field is free-form and type-inferred. + +## Acceptance Criteria + +- [ ] Pick one name per concept and migrate the data repo +- [ ] Document the relationship between the three counters, or collapse them +- [ ] Register the schema so writes are validated, not just `status`/`codegen_outcome` +- [ ] `RHAISTRAT-2352/RHAI-264`'s corrupt `status` migrated rather than normalised on read + +## Files Likely Involved + +- `scripts/artifact_utils.py` +- `docs/architecture/03-artifact-contracts.md` + +## Status + +Pending. + +## Notes + +Current state documented in `docs/architecture/03-artifact-contracts.md`. Fixing this also fixes half of [[bug-summary-json-double-counts-strategy]], since `scores: null` is a symptom of the vocabulary split. Coordinate with the pipeline repo — `PIPELINE_OWNED_KEYS` is duplicated there by design. diff --git a/docs/tasks/pending/task-deduplicate-stream-claude.md b/docs/tasks/pending/task-deduplicate-stream-claude.md new file mode 100644 index 0000000..cd94991 --- /dev/null +++ b/docs/tasks/pending/task-deduplicate-stream-claude.md @@ -0,0 +1,35 @@ +--- +id: task-deduplicate-stream-claude +title: Deduplicate stream-claude.py across the two repos +type: task +status: pending +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Task: Deduplicate stream-claude.py across the two repos + +## Goal + +One copy of the stream renderer. + +## Context + +`stream-claude.py` exists in both `epic-code-gen/ci-scripts/` and `epic-code-gen-pipeline/ci-scripts/` — two copies, no shared source, free to diverge. Only the `epic-code-gen` copy is actually invoked at runtime (by `run-claude.sh`). + +## Acceptance Criteria + +- [ ] Confirm which copy is live and whether they have diverged +- [ ] Delete the unused copy +- [ ] If both are needed, document why (as [ADR-0014] does for the merge logic) + +## Files Likely Involved + +- `ci-scripts/stream-claude.py` + +## Status + +Pending. + +## Notes + +The pipeline repo's copy appears vestigial since [ADR-0012] — the orchestrator is invoked directly there, and the inner Claude session is wrapped by `epic-code-gen`'s copy. Diff them first; a silent divergence would explain any renderer inconsistency between environments. diff --git a/docs/tasks/pending/task-delete-dead-code.md b/docs/tasks/pending/task-delete-dead-code.md new file mode 100644 index 0000000..33a2726 --- /dev/null +++ b/docs/tasks/pending/task-delete-dead-code.md @@ -0,0 +1,41 @@ +--- +id: task-delete-dead-code +title: Delete rubrics/, the codegen-review subsystem, and other dead code +type: task +status: pending +repos: [epic-code-gen] +decisions: [ADR-0021] +--- + +# Task: Delete rubrics/, the codegen-review subsystem, and other dead code + +## Goal + +Remove code and docs that are unused and actively contradict current behaviour. + +## Context + +Dead code that looks authoritative is worse than no code: a reader trusts it. `rubrics/` states weights, a dimension, and a model that are all wrong. + +## Acceptance Criteria + +- [ ] `rubrics/` deleted (5 files, 424 lines) +- [ ] `SCHEMAS["codegen-review"]`, `find_codegen_review`, `rebuild_index`, `frontmatter.py rebuild-index` removed +- [ ] `scripts/__init__.py` removed (0 bytes, vestigial) +- [ ] `make test-integration` and the `integration` marker either used or removed +- [ ] Confirm nothing references them first + +## Files Likely Involved + +- `rubrics/` +- `scripts/artifact_utils.py` +- `scripts/frontmatter.py` +- `Makefile` + +## Status + +Pending. + +## Notes + +See [[bug-rubrics-directory-is-dead-and-wrong]] and [[bug-codegen-review-schema-is-dead]]. Do this **before** the lint work, so the linter isn't run over code that's about to be deleted. diff --git a/docs/tasks/pending/task-extract-html-report-generation.md b/docs/tasks/pending/task-extract-html-report-generation.md new file mode 100644 index 0000000..f7f55f1 --- /dev/null +++ b/docs/tasks/pending/task-extract-html-report-generation.md @@ -0,0 +1,36 @@ +--- +id: task-extract-html-report-generation +title: Extract HTML report generation out of fetch_jira_epics.py +type: task +status: pending +repos: [epic-code-gen] +--- + +# Task: Extract HTML report generation out of fetch_jira_epics.py + +## Goal + +Separate fetching from rendering. + +## Context + +`fetch_jira_epics.py` is 1,020 lines, of which roughly **750 are `_render_*` HTML functions with inline CSS and JS** — in a module named 'fetch'. It has 45 tests, but the split makes it unclear what they cover. + +## Acceptance Criteria + +- [ ] Rendering moved to its own module +- [ ] `fetch_jira_epics.py` reduced to fetching, DAG building, and eligibility +- [ ] Existing 45 tests still pass, and it is clear which side each covers + +## Files Likely Involved + +- `scripts/fetch_jira_epics.py` +- `scripts/epic_report.py` + +## Status + +Pending. + +## Notes + +Consider whether the HTML report is still needed at all now that the dashboard exists — `epic-reports/` is gitignored and the report was the *pre-Jira* input path. Deleting beats extracting if nothing consumes it. Check before refactoring. diff --git a/docs/tasks/pending/task-fix-command-injection-validate-target.md b/docs/tasks/pending/task-fix-command-injection-validate-target.md new file mode 100644 index 0000000..c15e00d --- /dev/null +++ b/docs/tasks/pending/task-fix-command-injection-validate-target.md @@ -0,0 +1,37 @@ +--- +id: task-fix-command-injection-validate-target +title: Fix command injection in validate_target.py shell=True subprocess calls +type: task +status: pending +repos: [epic-code-gen] +jira: RHAIFIRST-194 +--- + +# Task: Fix command injection in validate_target.py shell=True subprocess calls + +## Goal + +Stop executing repo-derived strings through a shell. + +## Context + +`validate_target.py` discovers check commands from a target repo's Makefile and `package.json`, then runs them. Those strings come from a cloned third-party repository. + +## Acceptance Criteria + +- [ ] Commands executed without `shell=True` where feasible, or explicitly argument-split +- [ ] Makefile/package.json-derived values treated as untrusted input +- [ ] Tests covering a malicious command string + +## Files Likely Involved + +- `scripts/validate_target.py` +- `tests/test_validate_target.py` + +## Status + +Pending. + +## Notes + +Mitigating context: the pipeline already runs arbitrary target-repo build tooling by design, so this is not the only trust boundary — but it is the one that is trivially fixable and currently undefended. Prioritise it above the broader harness. diff --git a/docs/tasks/pending/task-fix-make-test-target.md b/docs/tasks/pending/task-fix-make-test-target.md new file mode 100644 index 0000000..e7d67b7 --- /dev/null +++ b/docs/tasks/pending/task-fix-make-test-target.md @@ -0,0 +1,57 @@ +--- +id: task-fix-make-test-target +title: Make `make test` pass, and decide the fate of the integration marker +type: task +status: pending +repos: [epic-code-gen] +--- + +# Task: Make `make test` pass, and decide the fate of the integration marker + +## Goal + +`make test` should run the full suite and exit 0 when everything passes. + +## Context + +See [[bug-make-test-fails-on-empty-integration-target]]. `make test` depends on `test-integration`, which +runs `pytest -m "integration"`; nothing carries the marker, pytest exits 5, and make fails. This has been +true on `main` since the marker was introduced, while the handbook told everyone to run `make test` +before pushing. + +There are two defensible fixes and the choice is a real one: + +- **Delete the target and the marker.** Honest about what exists — there are no integration tests. Also + removes the misleading `make test-integration` that looks like coverage and provides none. +- **Keep the target, tolerate empty collection.** Right if integration tests are genuinely coming; + otherwise it preserves a target that reports nothing while looking reassuring. + +If keeping it, note pytest has no native "don't fail on empty collection" flag, so the recipe needs +something like `pytest ... ; [ $$? -eq 5 ] && exit 0 || exit $$?` — or `pytest-custom-exit-code`. + +## Acceptance Criteria + +- [ ] `make test` exits 0 on a clean tree with all tests passing +- [ ] Decision recorded: delete the integration target/marker, or make empty collection non-fatal +- [ ] If the marker is kept, at least one test uses it — otherwise it is removed +- [ ] `AGENTS.md` and `CLAUDE.md` restored to recommending `make test` once it works +- [ ] CI runs whatever the full-suite command becomes + +## Files Likely Involved + +- `Makefile` +- `pyproject.toml` +- `AGENTS.md` +- `CLAUDE.md` +- `.github/workflows/ledger.yml` + +## Status + +Pending. + +## Notes + +Small fix, but it touches the instruction every contributor is given, so it is worth doing deliberately +rather than as a drive-by. Prefer deleting the marker unless someone is actually about to write +integration tests — a target that always reports success on zero tests is the same +looks-like-evidence-but-isn't pattern this project keeps fixing elsewhere ([ADR-0024]). diff --git a/docs/tasks/pending/task-investigate-max-turns.md b/docs/tasks/pending/task-investigate-max-turns.md new file mode 100644 index 0000000..bed2a1f --- /dev/null +++ b/docs/tasks/pending/task-investigate-max-turns.md @@ -0,0 +1,35 @@ +--- +id: task-investigate-max-turns +title: Investigate and configure --max-turns for Claude CLI codegen runs +type: task +status: pending +repos: [epic-code-gen, epic-code-gen-pipeline] +jira: RHAIFIRST-195 +--- + +# Task: Investigate and configure --max-turns for Claude CLI codegen runs + +## Goal + +Decide whether a turn ceiling is a useful guard against runaway sessions. + +## Context + +A codegen session is bounded by wall-clock timeout (6h) and iteration budget (10), but not by turns. A session can burn its budget in a loop that makes no progress. + +## Acceptance Criteria + +- [ ] Determine whether `--max-turns` interacts safely with SDD and the review loop +- [ ] Pick a value with evidence, or record why not to use it + +## Files Likely Involved + +- `ci-scripts/run-claude.sh` + +## Status + +Pending. + +## Notes + +Linked as Related to RHAIFIRST-168 rather than a child. Consider alongside an early-abandon heuristic on a flat score progression — see [ADR-0033]'s negative consequences. diff --git a/docs/tasks/pending/task-make-ledger-check-blocking.md b/docs/tasks/pending/task-make-ledger-check-blocking.md new file mode 100644 index 0000000..d1eeb5a --- /dev/null +++ b/docs/tasks/pending/task-make-ledger-check-blocking.md @@ -0,0 +1,39 @@ +--- +id: task-make-ledger-check-blocking +title: Flip check_ledger.py --diff from advisory to blocking +type: task +status: pending +repos: [epic-code-gen] +decisions: [ADR-0034] +--- + +# Task: Flip check_ledger.py --diff from advisory to blocking + +## Goal + +Enforce the PR companion rule in CI once the backlog is seeded. + +## Context + +`check_ledger.py --diff` warns rather than failing, deliberately: a blocking gate while the ledger was still being populated would have bitten every trivial fix. That justification expires once the backlog exists. + +## Acceptance Criteria + +- [ ] `continue-on-error` removed from the diff check in the workflow +- [ ] `Ledger: none — <reason>` escape hatch confirmed working +- [ ] A `skip-ledger` label honoured for genuine exceptions +- [ ] Team told before it starts blocking + +## Files Likely Involved + +- `.github/workflows/ledger.yml` +- `scripts/check_ledger.py` +- `AGENTS.md` + +## Status + +Pending. + +## Notes + +`--all` is already blocking — the ledger must stay internally consistent. Only the companion-file check is advisory. Do this once a few PRs have gone through the rule naturally, so the friction is known before it is mandatory. diff --git a/docs/tasks/pending/task-move-single-owner-deps-into-org.md b/docs/tasks/pending/task-move-single-owner-deps-into-org.md new file mode 100644 index 0000000..3013954 --- /dev/null +++ b/docs/tasks/pending/task-move-single-owner-deps-into-org.md @@ -0,0 +1,38 @@ +--- +id: task-move-single-owner-deps-into-org +title: Move the brains repo and CI image out of personal namespaces +type: task +status: pending +repos: [epic-code-gen, epic-code-gen-pipeline] +--- + +# Task: Move the brains repo and CI image out of personal namespaces + +## Goal + +Remove two bus-factor-one dependencies from the critical path. + +## Context + +CI clones `https://github.com/ederign/epic-code-gen.git` and runs `quay.io/ederignatowicz/epic-code-gen-ci:latest`. Both are personal namespaces. The other three repos live under `gitlab.com/redhat/rhel-ai/agentic-ci/`. + +## Acceptance Criteria + +- [ ] Brains repo moved or mirrored to an org namespace +- [ ] CI image published to an org-owned registry path +- [ ] `CLAUDE_REPO` and the image reference updated +- [ ] `CODEOWNERS` added to the brains repo, matching the other three + +## Files Likely Involved + +- `.gitlab-ci.yml` +- `Makefile` +- `Dockerfile.ci` + +## Status + +Pending. + +## Notes + +Blocking for any real production rollout, which is what RHAIFIRST-168 is about. Also worth resolving where this repo's canonical home is: it is on GitHub while its three siblings are on GitLab, which is why the ledger's CI is a GitHub Actions workflow. diff --git a/docs/tasks/pending/task-prototype-driven-codegen-remaining.md b/docs/tasks/pending/task-prototype-driven-codegen-remaining.md new file mode 100644 index 0000000..0fcc99b --- /dev/null +++ b/docs/tasks/pending/task-prototype-driven-codegen-remaining.md @@ -0,0 +1,39 @@ +--- +id: task-prototype-driven-codegen-remaining +title: Prototype-driven code generation — remaining work +type: task +status: pending +repos: [epic-code-gen] +jira: RHAIFIRST-233 +decisions: [ADR-0020] +--- + +# Task: Prototype-driven code generation — remaining work + +## Goal + +Finish the prototype pipeline: prerequisite parsing, deterministic extraction, and design fidelity review. + +## Context + +The core path shipped in [[task-ux-prototype-pipeline]] and produced its first Done epic (`RHOAIENG-72103`, 8.15). Three children remain open. + +## Acceptance Criteria + +- [ ] RHAIFIRST-234 — prerequisite parsing and prototype fetching from Jira +- [ ] RHAIFIRST-235 — deterministic PatternFly HTML prototype extraction +- [ ] RHAIFIRST-236 — spec/plan enrichment and design fidelity review + +## Files Likely Involved + +- `scripts/parse_prototype.js` +- `.claude/agents/ux-ac-extractor.md` +- `.claude/agents/intent-reviewer.md` + +## Status + +Pending. + +## Notes + +Parts of 234/235 already shipped in practice — the Jira issues predate the implementation and have not been reconciled. **Check what actually exists before starting.** Prototype detection is a regex over Jira table markup and has broken once already (`58e4bd7`). diff --git a/docs/tasks/pending/task-prune-data-repo-growth.md b/docs/tasks/pending/task-prune-data-repo-growth.md new file mode 100644 index 0000000..07880c1 --- /dev/null +++ b/docs/tasks/pending/task-prune-data-repo-growth.md @@ -0,0 +1,37 @@ +--- +id: task-prune-data-repo-growth +title: Define a pruning strategy for the data repo +type: task +status: pending +repos: [epic-code-gen-pipeline-data] +decisions: [ADR-0008] +--- + +# Task: Define a pruning strategy for the data repo + +## Goal + +Stop unbounded growth without losing the audit trail. + +## Context + +33 MB for 24 epics, and **13.7 MB of that is three OTEL files** (one is 9.8 MB). Versions accumulate and are never deleted, by design. `FOREDER.md` predicted this in June. + +## Acceptance Criteria + +- [ ] Decide what is durable (scores, final diff, run log) vs prunable (raw OTEL, intermediate versions) +- [ ] Compress or downsample OTEL before committing +- [ ] Retention policy for version directories on merged epics +- [ ] Consider what the dashboard actually reads before deleting anything + +## Files Likely Involved + +- `ci-scripts/push-results.py` + +## Status + +Pending. + +## Notes + +The OTEL files are raw OTLP payloads; only the summed cost is consumed. Compressing or summarising at write time would reclaim most of the space with no loss to any current consumer. Note the archive directory `RHAISTRAT-1699-before-ux-ac/` (2.0 MB) is also causing [[bug-summary-json-double-counts-strategy]]. diff --git a/docs/tasks/pending/task-revisit-flat-module-layout.md b/docs/tasks/pending/task-revisit-flat-module-layout.md new file mode 100644 index 0000000..c7b9e14 --- /dev/null +++ b/docs/tasks/pending/task-revisit-flat-module-layout.md @@ -0,0 +1,39 @@ +--- +id: task-revisit-flat-module-layout +title: Revisit the flat sys.path module layout +type: task +status: pending +repos: [epic-code-gen] +decisions: [ADR-0003] +--- + +# Task: Revisit the flat sys.path module layout + +## Goal + +Decide whether to keep the flat layout or move to a package. + +## Context + +[ADR-0003] records this as *Accepted, under review*. 20 modules import each other by bare name after a `sys.path.insert`, duplicated ~38 times including in all 18 test files, in two different spellings. There is no `conftest.py`. + +## Acceptance Criteria + +- [ ] Decide: keep, or migrate to a package with console entry points +- [ ] If keeping — at minimum add a `conftest.py` so tests stop repeating the bootstrap +- [ ] If migrating — update `Dockerfile.ci`, `run-codegen.sh`, and every `python3 scripts/x.py` invocation including those inside SKILL.md prompts +- [ ] Update [ADR-0003] either way + +## Files Likely Involved + +- `pyproject.toml` +- `tests/conftest.py` +- `scripts/` + +## Status + +Pending. + +## Notes + +The reason to keep it is real: CI clones the repo and runs `python3 scripts/run_pipeline.py` with no install step, so there is no stale-install failure mode. The cheap win regardless is `conftest.py`. Blocks [[task-add-lint-and-typecheck]]'s type-checking half. diff --git a/docs/tasks/pending/task-security-harness.md b/docs/tasks/pending/task-security-harness.md new file mode 100644 index 0000000..67fc6c5 --- /dev/null +++ b/docs/tasks/pending/task-security-harness.md @@ -0,0 +1,37 @@ +--- +id: task-security-harness +title: Security harness +type: task +status: pending +repos: [epic-code-gen] +jira: RHAIFIRST-193 +--- + +# Task: Security harness + +## Goal + +Establish a security review dimension and fix the known injection surface. + +## Context + +Generated code goes into other teams' repositories, and the pipeline itself runs shell commands built from repo-derived strings. Neither is currently reviewed for security. + +## Acceptance Criteria + +- [ ] Command injection in `validate_target.py` fixed (RHAIFIRST-194) +- [ ] Decide whether security is a scored dimension or an advisory verifier +- [ ] Secrets handling reviewed end to end + +## Files Likely Involved + +- `scripts/validate_target.py` +- `.claude/agents/` + +## Status + +Pending. + +## Notes + +RHAIFIRST-194 is the concrete child: `shell=True` subprocess calls in `validate_target.py`. Note the tension with [ADR-0022] — adding a scored dimension means re-deriving the calibrated weights, which is why an advisory verifier ([ADR-0028]) may be the better shape. diff --git a/docs/tasks/pending/task-test-plan-integration-epic-creator.md b/docs/tasks/pending/task-test-plan-integration-epic-creator.md new file mode 100644 index 0000000..d545890 --- /dev/null +++ b/docs/tasks/pending/task-test-plan-integration-epic-creator.md @@ -0,0 +1,37 @@ +--- +id: task-test-plan-integration-epic-creator +title: Test plan integration into epic creator +type: task +status: pending +repos: [epic-code-gen] +jira: RHAIFIRST-169 +--- + +# Task: Test plan integration into epic creator + +## Goal + +Have epics arrive with a test plan, so the tests dimension has an explicit target. + +## Context + +Today the tests reviewer infers what should be tested from the acceptance criteria. An epic carrying an explicit test plan would make AC coverage checkable rather than inferred. + +## Acceptance Criteria + +- [ ] Epic creator emits a test plan section +- [ ] `epic-task` frontmatter or body carries it +- [ ] `tests-reviewer` verifies against the plan, not just the ACs + +## Files Likely Involved + +- `scripts/fetch_jira_epics.py` +- `.claude/agents/tests-reviewer.md` + +## Status + +Pending. + +## Notes + +Upstream of this repo — the change is mostly in `epic-creator`. Listed here because the consuming side is ours. diff --git a/scripts/check_ledger.py b/scripts/check_ledger.py new file mode 100644 index 0000000..826e66c --- /dev/null +++ b/scripts/check_ledger.py @@ -0,0 +1,339 @@ +#!/usr/bin/env python3 +"""Validate the work ledger under docs/ and enforce the PR companion rule. + +Two modes: + + check_ledger.py --all + Validate every ledger file: frontmatter parses, `status` agrees with the + directory the file sits in, done/fixed files carry evidence, ids match + filenames, and cross-references resolve. Exits 1 on any error. + + check_ledger.py --diff <base>..<head> [--body FILE] + The PR companion check. If the diff touches code but no ledger file, + report it. Advisory by default (exit 0); --strict makes it exit 1. + Honours `Ledger: none — <reason>` in the PR body. + +See AGENTS.md for the rule this enforces. Frontmatter is read with +artifact_utils.read_frontmatter rather than a new parser — this repo already has +six too many. +""" + +import argparse +import os +import re +import subprocess +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from artifact_utils import read_frontmatter # noqa: E402 + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +DOCS = os.path.join(REPO_ROOT, "docs") + +# Directory → the status value a file in it must declare. +DIR_STATUS = { + "tasks/pending": "pending", + "tasks/current": "current", + "tasks/blocked": "blocked", + "tasks/done": "done", + "bugs/open": "open", + "bugs/fixed": "fixed", + "bugs/wontfix": "wontfix", +} + +# Files whose status is free-form (ADRs, architecture, plans, milestones). +FREE_STATUS_DIRS = ("decisions", "architecture", "plans", "milestones", "notes") + +# Statuses that must carry evidence of what closed them. +EVIDENCE_REQUIRED = ("done", "fixed") + +REQUIRED_FIELDS = ("id", "title", "type", "status", "repos") +VALID_TYPES = ("task", "bug", "adr", "milestone", "plan") +VALID_REPOS = ("epic-code-gen", "epic-code-gen-pipeline", + "epic-code-gen-pipeline-data", "epic-code-gen-dashboard") + +# A code change wants a ledger companion. +CODE_PREFIXES = ("scripts/", ".claude/", "ci-scripts/", "rubrics/") +CODE_SUFFIXES = (".py", ".js", ".sh", ".md", ".json", ".yml", ".yaml") +LEDGER_PREFIXES = ("docs/tasks/", "docs/bugs/", "docs/decisions/") + +WIKILINK_RE = re.compile(r"\[\[([^\]]+)\]\]") +ADR_REF_RE = re.compile(r"\[?(ADR-\d{4})\]?") +LEDGER_NONE_RE = re.compile(r"^\s*Ledger:\s*none\b", re.IGNORECASE | re.MULTILINE) + + +# ── helpers ───────────────────────────────────────────────────────────────── + +def ledger_files(docs=DOCS, repo_root=REPO_ROOT): + """Every .md file under docs/, relative to repo_root. + + repo_root must be the root the caller will rejoin these against — passing a + docs/ from outside REPO_ROOT with the default root yields unusable paths. + """ + out = [] + for root, _dirs, files in os.walk(docs): + for f in sorted(files): + if f.endswith(".md"): + out.append(os.path.relpath(os.path.join(root, f), repo_root)) + return sorted(out) + + +def _rel_docs(rel, docs_name="docs"): + """docs-relative directory of a repo-relative path, with / separators.""" + d = os.path.dirname(os.path.relpath(rel, docs_name)) + return d.replace(os.sep, "/") + + +def _expected_status(rel_dir): + for prefix, status in DIR_STATUS.items(): + if rel_dir == prefix: + return status + return None + + +def _as_list(value): + """Frontmatter lists arrive as list or as a bare scalar.""" + if value is None: + return [] + if isinstance(value, list): + return value + return [value] + + +# ── --all ─────────────────────────────────────────────────────────────────── + +def check_all(docs=DOCS, repo_root=REPO_ROOT): + """Validate every ledger file. Returns a list of error strings.""" + errors = [] + files = ledger_files(docs, repo_root) + if not files: + return [f"no ledger files found under {docs}"] + # docs/ may be named anything when called from a test tmp tree. + docs_name = os.path.relpath(docs, repo_root) + + known_ids = set() + for rel in files: + data, _ = read_frontmatter(os.path.join(repo_root, rel)) + if data.get("id"): + known_ids.add(str(data["id"])) + + known_adrs = { + m.group(1) + for rel in files + for m in [ADR_REF_RE.match(os.path.basename(rel))] if m + } + + # Root docs aren't ledger entries, but they link into docs/ heavily and a + # broken link there is the most visible kind. + for root_doc in ("PLAN.md", "AGENTS.md"): + root_path = os.path.join(repo_root, root_doc) + if os.path.exists(root_path): + _, root_body = read_frontmatter(root_path) + errors.extend( + _check_links(root_doc, root_path, root_body, known_ids, known_adrs)) + + for rel in files: + path = os.path.join(repo_root, rel) + rel_dir = _rel_docs(rel, docs_name) + stem = os.path.basename(rel)[: -len(".md")] + + try: + data, body = read_frontmatter(path) + except Exception as e: # noqa: BLE001 + errors.append(f"{rel}: frontmatter does not parse: {e}") + continue + + # README.md files are index pages, not ledger entries. Their links are + # still checked below; their frontmatter is not required. + if os.path.basename(rel) == "README.md": + errors.extend(_check_links(rel, path, body, known_ids, known_adrs)) + continue + + if not data: + errors.append(f"{rel}: no frontmatter") + continue + + for field in REQUIRED_FIELDS: + if not data.get(field): + errors.append(f"{rel}: missing required field `{field}`") + + if data.get("id") and str(data["id"]) != stem: + errors.append( + f"{rel}: id `{data['id']}` does not match filename `{stem}`") + + if data.get("type") and data["type"] not in VALID_TYPES: + errors.append( + f"{rel}: type `{data['type']}` not one of {', '.join(VALID_TYPES)}") + + for repo in _as_list(data.get("repos")): + for name in str(repo).split(","): + name = name.strip() + if name and name not in VALID_REPOS: + errors.append(f"{rel}: unknown repo `{name}`") + + # Status must agree with location — state is directory placement. + expected = _expected_status(rel_dir) + status = data.get("status") + if expected and status != expected: + errors.append( + f"{rel}: status `{status}` but lives in {rel_dir}/ " + f"(expected `{expected}`) — move the file or fix the field") + + # done/fixed must say what closed them. + if status in EVIDENCE_REQUIRED and not rel_dir.startswith(FREE_STATUS_DIRS): + if not _as_list(data.get("commits")) and not data.get("jira"): + errors.append( + f"{rel}: status `{status}` requires evidence — " + f"a non-empty `commits:` list or a `jira:` key") + + # SHAs must be quoted. YAML reads a leading-zero digit string as octal, + # so an unquoted `0346470` silently becomes the integer 118072. + for sha in _as_list(data.get("commits")): + if not isinstance(sha, str): + errors.append( + f"{rel}: commit `{sha}` parsed as {type(sha).__name__}, not a " + f"string — quote it (YAML reads a leading-zero SHA as octal)") + elif not re.fullmatch(r"[0-9a-f]{7,40}", sha): + errors.append(f"{rel}: `{sha}` is not a valid commit SHA") + + errors.extend(_check_links(rel, path, body, known_ids, known_adrs)) + + return errors + + +def strip_code(text): + """Remove fenced blocks and inline code spans. + + A `[[id]]` or `](path)` inside backticks is documenting syntax, not + referencing anything — AGENTS.md's own templates are full of both. + """ + text = re.sub(r"^\s*```.*?^\s*```", "", text, flags=re.DOTALL | re.MULTILINE) + text = re.sub(r"`[^`\n]*`", "", text) + return text + + +def _check_links(rel, path, body, known_ids, known_adrs): + """Wikilinks, ADR references, and relative markdown links must resolve.""" + errors = [] + body = strip_code(body) + + for m in WIKILINK_RE.finditer(body): + target = m.group(1).strip() + if target not in known_ids: + errors.append(f"{rel}: [[{target}]] does not resolve to any ledger id") + + for adr in sorted(set(ADR_REF_RE.findall(body))): + if known_adrs and adr not in known_adrs: + errors.append(f"{rel}: references {adr}, which does not exist") + + for link in re.findall(r"\]\((?!https?://|#)([^)]+)\)", body): + target = link.split("#")[0].strip() + if not target: + continue + resolved = os.path.normpath(os.path.join(os.path.dirname(path), target)) + if not os.path.exists(resolved): + errors.append(f"{rel}: broken link → {target}") + + return errors + + +# ── --diff ────────────────────────────────────────────────────────────────── + +def changed_files(rev_range, repo_root=REPO_ROOT): + """Files changed in a git revision range.""" + result = subprocess.run( + ["git", "diff", "--name-only", rev_range], + cwd=repo_root, capture_output=True, text=True) + if result.returncode != 0: + raise RuntimeError(result.stderr.strip() or "git diff failed") + return [ln.strip() for ln in result.stdout.splitlines() if ln.strip()] + + +def is_code(path): + return (path.startswith(CODE_PREFIXES) + and path.endswith(CODE_SUFFIXES) + and not path.startswith(LEDGER_PREFIXES)) + + +def is_ledger(path): + return path.startswith(LEDGER_PREFIXES) + + +def check_diff(files, body=""): + """Returns (ok, message). ok is False when a companion is missing.""" + code = sorted(p for p in files if is_code(p)) + ledger = sorted(p for p in files if is_ledger(p)) + + if not code: + return True, "no code changes — companion not required" + if ledger: + return True, (f"{len(code)} code file(s), " + f"{len(ledger)} ledger file(s): {', '.join(ledger)}") + if body and LEDGER_NONE_RE.search(body): + return True, "explicit `Ledger: none` in PR body" + + listed = "\n".join(f" {p}" for p in code[:10]) + more = f"\n … and {len(code) - 10} more" if len(code) > 10 else "" + return False, ( + f"{len(code)} code file(s) changed with no ledger companion:\n" + f"{listed}{more}\n\n" + " Add a file under docs/tasks/ or docs/bugs/ (and an ADR if you made a\n" + " decision), or write `Ledger: none — <reason>` in the PR body.\n" + " See AGENTS.md §3.") + + +# ── main ──────────────────────────────────────────────────────────────────── + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--all", action="store_true", + help="validate every ledger file") + ap.add_argument("--diff", metavar="RANGE", + help="check a git range, e.g. main..HEAD") + ap.add_argument("--body", metavar="FILE", + help="PR body file, for the `Ledger: none` escape hatch") + ap.add_argument("--strict", action="store_true", + help="make a missing companion exit 1 instead of warning") + args = ap.parse_args() + + if not args.all and not args.diff: + ap.error("one of --all or --diff is required") + + failed = False + + if args.all: + errors = check_all() + if errors: + print(f"✗ ledger: {len(errors)} problem(s)\n", file=sys.stderr) + for e in errors: + print(f" {e}", file=sys.stderr) + failed = True + else: + print(f"✓ ledger: {len(ledger_files())} files, all consistent") + + if args.diff: + body = "" + if args.body and os.path.exists(args.body): + with open(args.body, encoding="utf-8") as f: + body = f.read() + try: + files = changed_files(args.diff) + except RuntimeError as e: + print(f"✗ companion check: {e}", file=sys.stderr) + return 1 + ok, message = check_diff(files, body) + if ok: + print(f"✓ companion check: {message}") + else: + label = "✗" if args.strict else "⚠" + print(f"{label} companion check: {message}", + file=sys.stderr if args.strict else sys.stdout) + if args.strict: + failed = True + + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/review_response.py b/scripts/review_response.py index 5540868..a3cf02a 100644 --- a/scripts/review_response.py +++ b/scripts/review_response.py @@ -296,13 +296,18 @@ def run_review_response(epic_id, pr_url, output_dir="artifacts", target_repo=".target-repo", version=2, dry_run=False, gh_token_var=None, base_branch=None, fork_remote="fork", - skip_rebase=False, agent_timeout=None): + skip_rebase=False, agent_timeout=None, + our_user=None): """Execute the full V2 review response flow. Args: agent_timeout: seconds the fix agent may run. Defaults to FIX_AGENT_TIMEOUT; callers should size it below their own budget so the rebase, triage, validation and push still fit inside. + our_user: the account this run acts as, whose own comments must not be + treated as review feedback. Defaults to review_config.json's + `our_user`. A target repo running under a non-default credential + passes its own, or the loop answers its own comments forever. Returns: dict: {success, comments_processed, fixes_applied, commit_sha, @@ -313,7 +318,8 @@ def run_review_response(epic_id, pr_url, output_dir="artifacts", token = require_env(gh_token_var) config = load_review_config() bot_reviewers = set(config.get("bot_reviewers", [])) - our_user = config.get("our_user", "dora-the-ai-coder") + if not our_user: + our_user = config.get("our_user", "dora-the-ai-coder") max_retries = config.get("validation_retry_limit", 3) owner, repo, number = parse_pr_url(pr_url) @@ -644,6 +650,10 @@ def main(): help="Fork owner for push") parser.add_argument("--gh-token-var", default=None, help="Env var for GitHub token") + parser.add_argument("--our-user", default=None, + help="GitHub account this run acts as, whose own " + "comments are not review feedback " + "(default: review_config.json our_user)") parser.add_argument("--dry-run", action="store_true", help="Skip push and PR replies") parser.add_argument("--base-branch", default=None, @@ -671,6 +681,7 @@ def main(): base_branch=args.base_branch, skip_rebase=args.skip_rebase, agent_timeout=args.agent_timeout, + our_user=args.our_user, ) if args.json: diff --git a/scripts/run_pipeline.py b/scripts/run_pipeline.py index 70914a0..a164373 100644 --- a/scripts/run_pipeline.py +++ b/scripts/run_pipeline.py @@ -240,6 +240,79 @@ def load_repo_mapping(path=None): return json.load(f) +DEFAULT_TOKEN_VAR = "EPIC_CODEGEN_GITHUB_TOKEN" + + +def load_our_user(config_path=None): + """Read the default bot username from review_config.json.""" + if config_path is None: + config_path = os.path.join(_CONFIG_DIR, "review_config.json") + our_user = "dora-the-ai-coder" + if os.path.isfile(config_path): + with open(config_path, encoding="utf-8") as f: + our_user = json.load(f).get("our_user", our_user) + return our_user + + +def identity_for_repo(target_repo, args=None, mapping=None): + """Resolve the GitHub identity the pipeline acts under for one target repo. + + Almost every target is another team's public repo, so the shared + `dora-the-ai-coder` bot forks it and opens the PR (ADR-0030). That breaks + down for a private repo in an org the bot is not a member of: the clone + 404s and the epic looks like a mapping fault rather than a credential one. + + A mapping entry may therefore carry its own `fork_owner` and + `gh_token_var`, so one target can run under a different credential without + changing the default for every other strategy in the same run. + + `our_user` is what filters our own PR comments out of the review loop, so + it must name whoever the token belongs to — an override that changed the + fork owner but not this would make the pipeline answer its own comments + forever. It follows `fork_owner` unless set explicitly. + + Returns: + dict: {"fork_owner", "gh_token_var", "our_user"} + """ + default_owner = getattr(args, "fork_owner", None) + identity = { + "fork_owner": default_owner, + "gh_token_var": DEFAULT_TOKEN_VAR, + "our_user": load_our_user(), + } + if not target_repo: + return identity + + if mapping is None: + mapping = load_repo_mapping() + entry = mapping.get(_repo_slug(target_repo)) or {} + if not entry: + return identity + + if entry.get("fork_owner"): + identity["fork_owner"] = entry["fork_owner"] + # Follows fork_owner, not the shared default, unless overridden below. + identity["our_user"] = entry["fork_owner"] + if entry.get("gh_token_var"): + identity["gh_token_var"] = entry["gh_token_var"] + if entry.get("our_user"): + identity["our_user"] = entry["our_user"] + return identity + + +def _repo_slug(target_repo): + """Normalise `owner/repo` out of a slug or a full clone URL.""" + slug = (target_repo or "").strip() + slug = re.sub(r"^(https?://|git@)[^/:]+[/:]", "", slug) + return slug.removesuffix(".git").strip("/") + + +def repo_token(target_repo, args=None, mapping=None): + """Return the GitHub token for a target repo, or "" if its var is unset.""" + identity = identity_for_repo(target_repo, args, mapping) + return os.environ.get(identity["gh_token_var"], "") + + def resolve_target_repo(epic_data, mapping, prompt_path=None): """Determine the target repo for an epic. @@ -376,9 +449,10 @@ def setup_target_repo(epic, args): sys.executable, os.path.join(_SCRIPT_DIR, "clone_target.py"), target_repo, epic_id, "--clean", ] - if args.fork_owner: - clone_cmd += ["--fork-owner", args.fork_owner, - "--gh-token-var", "EPIC_CODEGEN_GITHUB_TOKEN"] + identity = identity_for_repo(target_repo, args) + if identity["fork_owner"]: + clone_cmd += ["--fork-owner", identity["fork_owner"], + "--gh-token-var", identity["gh_token_var"]] try: result = subprocess.run( @@ -502,8 +576,14 @@ def _install_node_deps(repo): print(f" Node {current} < {required_major}, " f"using nvm to install {required_major}") - cmd = f"{nvm_prefix}npm install" - _run_cmd(["bash", "-c", cmd], cwd=repo, label="npm install") + # Install through the manager the repo declares. Installing a pnpm + # workspace with npm produces a different dependency tree than CI has, + # so the checks would not be measuring the same thing. + from validate_target import detect_package_manager + pkg_manager = detect_package_manager(repo) + + cmd = f"{nvm_prefix}{pkg_manager} install" + _run_cmd(["bash", "-c", cmd], cwd=repo, label=f"{pkg_manager} install") def _get_node_major(): @@ -579,13 +659,18 @@ def _parse_flat_yaml(lines): return data -def invoke_codegen(epic_id, args): +def invoke_codegen(epic_id, args, target_repo=None): """Shell out to Claude for codegen. Returns True on success.""" skill_args = f"/epic-codegen {epic_id}" if args.max_iterations is not None: skill_args += f" --max-iterations {args.max_iterations}" - if args.fork_owner: - skill_args += f" --fork-owner {args.fork_owner}" + identity = identity_for_repo(target_repo, args) + if identity["fork_owner"]: + skill_args += f" --fork-owner {identity['fork_owner']}" + # The skill defaults to EPIC_CODEGEN_GITHUB_TOKEN; a target with its + # own credential has to say so or the skill pushes to the fork with + # the wrong account's token. + skill_args += f" --gh-token-var {identity['gh_token_var']}" run_script = args.run_script or os.path.join( os.path.dirname(_SCRIPT_DIR), "ci-scripts", "run-claude.sh") @@ -802,7 +887,8 @@ def process_strategy(strategy_key, server, user, token, args): original_status = all_epics_by_key[epic_id].get("jira_status", "") - success = invoke_codegen(epic_id, args) + success = invoke_codegen( + epic_id, args, all_epics_by_key[epic_id].get("target_repo")) if success: results[PROCESSED].append((epic_id, "codegen completed")) ok, _ = transition_issue( @@ -1289,7 +1375,7 @@ def _ci_handle_ready(epic, state, args, server, user, token): save_epic_state(args.data_repo, epic["strategy_key"], epic_id, state) log.info("%s: invoking codegen v%d", epic_id, state["current_version"]) - success = invoke_codegen(epic_id, args) + success = invoke_codegen(epic_id, args, epic.get("target_repo")) # The codegen skill may iterate internally (v1→v2→...); sync version actual_version = _detect_highest_version(epic_id, args.output_dir) @@ -1331,7 +1417,9 @@ def _pr_is_live(pr_url): not live (no token, GitHub unreachable, an unparseable URL), which leaves the scoring path in charge exactly as it was before this check. """ - gh_token = os.environ.get("EPIC_CODEGEN_GITHUB_TOKEN", "") + # The PR URL names its own repo, so the right credential is derivable + # without threading the epic down here. + gh_token = repo_token(pr_url.rsplit("/pull/", 1)[0]) if not gh_token: return False try: @@ -1463,9 +1551,11 @@ def _ci_handle_pr_created(epic, state, args, server, user, token): filter_unprocessed_reviews, load_processed_comment_ids, ) - gh_token = os.environ.get("EPIC_CODEGEN_GITHUB_TOKEN", "") + identity = identity_for_repo(epic.get("target_repo"), args) + gh_token = os.environ.get(identity["gh_token_var"], "") if not gh_token: - return SKIPPED, "PRCreated", "PRCreated", "No GitHub token" + return SKIPPED, "PRCreated", "PRCreated", \ + f"No GitHub token in {identity['gh_token_var']}" status = get_pr_status(pr_url, gh_token) new_state = derive_pr_state(status) @@ -1481,13 +1571,7 @@ def _ci_handle_pr_created(epic, state, args, server, user, token): args.output_dir, "codegen-runs", epic_id, "pr-replies.json") processed_ids = load_processed_comment_ids(pr_replies_path) - config_path = os.path.join( - os.path.dirname(os.path.abspath(__file__)), - "..", "config", "review_config.json") - our_user = "dora-the-ai-coder" - if os.path.isfile(config_path): - with open(config_path) as f: - our_user = json.load(f).get("our_user", our_user) + our_user = identity["our_user"] unprocessed = filter_unprocessed_comments( reviews_data["comments"], processed_ids, our_user) # Review bodies count too — a reviewer can request changes with @@ -1554,10 +1638,11 @@ def _ci_handle_pr_changes(epic, state, args, server, user, token): return FAILED, "PRChangesRequested", "Failed", \ f"Exhausted {max_iter} iterations" - gh_token = os.environ.get("EPIC_CODEGEN_GITHUB_TOKEN", "") + identity = identity_for_repo(epic.get("target_repo"), args) + gh_token = os.environ.get(identity["gh_token_var"], "") if not gh_token: return SKIPPED, "PRChangesRequested", "PRChangesRequested", \ - "No GitHub token" + f"No GitHub token in {identity['gh_token_var']}" next_version = version + 1 @@ -1583,8 +1668,12 @@ def _ci_handle_pr_changes(epic, state, args, server, user, token): "--output-dir", args.output_dir, "--target-repo", target_repo, "--version", str(next_version), + "--gh-token-var", identity["gh_token_var"], + "--our-user", identity["our_user"], "--json", ] + if identity["fork_owner"]: + cmd += ["--fork-owner", identity["fork_owner"]] base_branch = state.get("target_branch") or epic.get("target_branch") if base_branch: cmd += ["--base-branch", base_branch] @@ -1710,10 +1799,10 @@ def _setup_target_for_review_response(epic, state, args, target_repo): sys.executable, clone_script, target_url, epic_id, "--dest", target_repo, "--checkout-existing", "--clean", ] - fork_owner = getattr(args, "fork_owner", None) - if fork_owner: - cmd += ["--fork-owner", fork_owner, - "--gh-token-var", "EPIC_CODEGEN_GITHUB_TOKEN"] + identity = identity_for_repo(target_url, args) + if identity["fork_owner"]: + cmd += ["--fork-owner", identity["fork_owner"], + "--gh-token-var", identity["gh_token_var"]] try: result = subprocess.run( @@ -1775,6 +1864,8 @@ def _create_pr_for_epic(epic, state, args): if pr_url: return pr_url + identity = identity_for_repo(target_repo, args) + try: from create_pr import create_pr from push_to_fork import push @@ -1800,10 +1891,11 @@ def _create_pr_for_epic(epic, state, args): pr = create_pr( upstream_slug=target_repo, - fork_owner=args.fork_owner, + fork_owner=identity["fork_owner"], branch=branch, title=f"{epic_id}: {epic.get('title', 'Code generation')}", body=body, + token_var=identity["gh_token_var"], ) return pr.get("html_url") except urllib.error.HTTPError as e: @@ -1811,10 +1903,10 @@ def _create_pr_for_epic(epic, state, args): e, "error_body", ""): import github_utils upstream_owner, upstream_repo = target_repo.split("/") - token = github_utils.require_env() + token = github_utils.require_env(identity["gh_token_var"]) existing = github_utils.find_existing_pr( upstream_owner, upstream_repo, - args.fork_owner, branch, token) + identity["fork_owner"], branch, token) if existing: url = existing.get("html_url") print(f" {epic_id}: PR already exists: {url}") diff --git a/scripts/validate_target.py b/scripts/validate_target.py index 2055f57..efe1b1d 100644 --- a/scripts/validate_target.py +++ b/scripts/validate_target.py @@ -139,8 +139,14 @@ def _read(name): or "[tool.uv]" in pyproject): tools.append("uv") - if os.path.isfile(os.path.join(repo_path, "yarn.lock")): - tools.append("yarn") + # The JS package manager the repo declares. Only yarn.lock used to be + # checked here, so a pnpm repo preflighted clean and then failed every + # check with exit 127 — the exact environment-fault-scored-as-bad-code + # this gate exists to prevent (ADR-0025). + if language in ("typescript", "javascript"): + pkg_manager = detect_package_manager(repo_path) + if pkg_manager != "npm": + tools.append(pkg_manager) if os.path.isfile(os.path.join(repo_path, "requirements.txt")): tools.append("pip3") @@ -433,6 +439,43 @@ def _parse_package_json_scripts(repo_path): return [] +def detect_package_manager(repo_path): + """Which package manager this JS/TS repo's scripts must be run through. + + Running the wrong one is not a style question. A pnpm workspace installed + with npm resolves a different dependency tree, and `npm run lint` in a repo + whose node_modules pnpm laid out either fails outright or checks something + other than what CI checks. + + `packageManager` (the corepack field) is authoritative when present — + a repo can carry a stale secondary lockfile, but it only declares one. + + Returns: + str: "pnpm", "yarn" or "npm" (the default when nothing is declared). + """ + pkg = os.path.join(repo_path, "package.json") + if os.path.isfile(pkg): + try: + with open(pkg, encoding="utf-8") as f: + declared = json.load(f).get("packageManager", "") + name = str(declared).split("@", 1)[0].strip() + if name in ("pnpm", "yarn", "npm"): + return name + except (OSError, json.JSONDecodeError): + pass + + if os.path.isfile(os.path.join(repo_path, "pnpm-lock.yaml")): + return "pnpm" + if os.path.isfile(os.path.join(repo_path, "yarn.lock")): + return "yarn" + return "npm" + + +def _script_command(pkg_manager, script): + """The command that runs a package.json script under this manager.""" + return f"{pkg_manager} run {script}" + + def discover_commands(repo_path, language): """Discover available validation commands for the repo. @@ -445,7 +488,8 @@ def discover_commands(repo_path, language): if language == "go": commands.update(_discover_go_commands(make_targets)) elif language in ("typescript", "javascript"): - commands.update(_discover_js_commands(make_targets, npm_scripts)) + commands.update(_discover_js_commands( + make_targets, npm_scripts, detect_package_manager(repo_path))) elif language == "python": commands.update(_discover_python_commands(make_targets)) elif language == "rust": @@ -476,24 +520,24 @@ def _discover_go_commands(make_targets): return commands -def _discover_js_commands(make_targets, npm_scripts): +def _discover_js_commands(make_targets, npm_scripts, pkg_manager="npm"): commands = {} if "lint" in npm_scripts: - commands["lint"] = "npm run lint" + commands["lint"] = _script_command(pkg_manager, "lint") elif any("lint" in t.lower() for t in make_targets): target = next(t for t in make_targets if "lint" in t.lower()) commands["lint"] = f"make {target}" if "typecheck" in npm_scripts: - commands["typecheck"] = "npm run typecheck" + commands["typecheck"] = _script_command(pkg_manager, "typecheck") elif "tsc" in npm_scripts: - commands["typecheck"] = "npm run tsc" + commands["typecheck"] = _script_command(pkg_manager, "tsc") elif os.path.isfile("tsconfig.json"): commands["typecheck"] = "npx tsc --noEmit" if "test" in npm_scripts: - commands["test"] = "npm test" + commands["test"] = f"{pkg_manager} test" elif any("test" in t.lower() for t in make_targets): target = next(t for t in make_targets if "test" in t.lower()) commands["test"] = f"make {target}" diff --git a/tests/test_check_ledger.py b/tests/test_check_ledger.py new file mode 100644 index 0000000..864f899 --- /dev/null +++ b/tests/test_check_ledger.py @@ -0,0 +1,380 @@ +"""Tests for check_ledger.py.""" + +import os +import sys + +import pytest + +sys_path_fix = os.path.join(os.path.dirname(__file__), "..", "scripts") +sys.path.insert(0, sys_path_fix) + +from check_ledger import ( # noqa: E402 + check_all, + check_diff, + is_code, + is_ledger, + ledger_files, + strip_code, +) + +REPO_ROOT = os.path.join(os.path.dirname(__file__), "..") + + +def write(path, frontmatter, body=""): + """Write a ledger file with the given frontmatter dict.""" + os.makedirs(os.path.dirname(path), exist_ok=True) + lines = ["---"] + for k, v in frontmatter.items(): + lines.append(f"{k}: {v}") + lines += ["---", "", body] + with open(path, "w", encoding="utf-8") as f: + f.write("\n".join(lines)) + + +VALID = { + "id": "task-example", + "title": "Example", + "type": "task", + "status": "pending", + "repos": "[epic-code-gen]", +} + + +@pytest.fixture +def docs(tmp_path): + """A minimal valid ledger tree.""" + d = tmp_path / "docs" + write(str(d / "tasks" / "pending" / "task-example.md"), VALID, "Body.") + return d + + +class TestCheckAllValid: + def test_valid_tree_has_no_errors(self, docs, tmp_path): + assert check_all(str(docs), str(tmp_path)) == [] + + def test_empty_tree_is_an_error(self, tmp_path): + empty = tmp_path / "docs" + empty.mkdir() + errors = check_all(str(empty), str(tmp_path)) + assert len(errors) == 1 + assert "no ledger files" in errors[0] + + +class TestRequiredFields: + @pytest.mark.parametrize("field", ["id", "title", "type", "status", "repos"]) + def test_missing_required_field(self, docs, tmp_path, field): + fm = {k: v for k, v in VALID.items() if k != field} + write(str(docs / "tasks" / "pending" / "task-example.md"), fm) + errors = check_all(str(docs), str(tmp_path)) + assert any(f"missing required field `{field}`" in e for e in errors) + + def test_no_frontmatter_at_all(self, docs, tmp_path): + p = docs / "tasks" / "pending" / "task-example.md" + p.write_text("Just prose, no frontmatter.\n") + errors = check_all(str(docs), str(tmp_path)) + assert any("no frontmatter" in e for e in errors) + + def test_invalid_type(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), + {**VALID, "type": "nonsense"}) + errors = check_all(str(docs), str(tmp_path)) + assert any("type `nonsense`" in e for e in errors) + + def test_unknown_repo(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), + {**VALID, "repos": "[not-a-real-repo]"}) + errors = check_all(str(docs), str(tmp_path)) + assert any("unknown repo `not-a-real-repo`" in e for e in errors) + + +class TestIdMatchesFilename: + def test_mismatch_is_an_error(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), + {**VALID, "id": "task-something-else"}) + errors = check_all(str(docs), str(tmp_path)) + assert any("does not match filename" in e for e in errors) + + def test_readme_is_exempt(self, docs, tmp_path): + """Index pages are not ledger entries.""" + (docs / "decisions").mkdir(parents=True, exist_ok=True) + (docs / "decisions" / "README.md").write_text("# Index\n\nNo frontmatter.\n") + assert check_all(str(docs), str(tmp_path)) == [] + + +class TestStatusMatchesDirectory: + """State is represented by location — the two must agree.""" + + @pytest.mark.parametrize("subdir,status", [ + ("tasks/pending", "pending"), + ("tasks/current", "current"), + ("tasks/blocked", "blocked"), + ("tasks/done", "done"), + ("bugs/open", "open"), + ("bugs/fixed", "fixed"), + ("bugs/wontfix", "wontfix"), + ]) + def test_matching_status_is_accepted(self, tmp_path, subdir, status): + d = tmp_path / "docs" + fm = {**VALID, "id": "item", "status": status} + if status in ("done", "fixed"): + fm["commits"] = "[abc1234]" + if subdir.startswith("bugs"): + fm["type"] = "bug" + write(str(d / subdir / "item.md"), fm) + assert check_all(str(d), str(tmp_path)) == [] + + def test_status_disagreeing_with_directory(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), + {**VALID, "status": "done", "commits": "[abc1234]"}) + errors = check_all(str(docs), str(tmp_path)) + assert any("lives in tasks/pending/" in e for e in errors) + + def test_bug_in_wrong_tree(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "bugs" / "open" / "bug-x.md"), + {**VALID, "id": "bug-x", "type": "bug", "status": "fixed"}) + errors = check_all(str(d), str(tmp_path)) + assert any("expected `open`" in e for e in errors) + + +class TestEvidenceRequired: + """done/fixed must record what closed them.""" + + def test_done_without_evidence_fails(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done"}) + errors = check_all(str(d), str(tmp_path)) + assert any("requires evidence" in e for e in errors) + + def test_done_with_commits_passes(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": "[abc1234, def5678]"}) + assert check_all(str(d), str(tmp_path)) == [] + + def test_done_with_jira_passes(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "jira": "RHAIFIRST-1"}) + assert check_all(str(d), str(tmp_path)) == [] + + def test_empty_commits_list_is_not_evidence(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": "[]"}) + errors = check_all(str(d), str(tmp_path)) + assert any("requires evidence" in e for e in errors) + + def test_pending_needs_no_evidence(self, docs, tmp_path): + assert check_all(str(docs), str(tmp_path)) == [] + + +class TestCommitShaTyping: + """An unquoted leading-zero SHA is read by YAML as octal. + + `commits: [0346470]` silently becomes the integer 118072 — a real defect + found in this ledger during its own verification pass. + """ + + def test_unquoted_octal_sha_is_caught(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": "[0346470]"}) + errors = check_all(str(d), str(tmp_path)) + assert any("not a string" in e for e in errors), errors + + def test_quoted_octal_sha_passes(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": '["0346470"]'}) + assert check_all(str(d), str(tmp_path)) == [] + + def test_malformed_sha_is_caught(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": '["not-a-sha"]'}) + errors = check_all(str(d), str(tmp_path)) + assert any("not a valid commit SHA" in e for e in errors), errors + + def test_uppercase_sha_is_caught(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", "commits": '["ABC1234"]'}) + errors = check_all(str(d), str(tmp_path)) + assert any("not a valid commit SHA" in e for e in errors), errors + + def test_full_length_sha_passes(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "tasks" / "done" / "task-x.md"), + {**VALID, "id": "task-x", "status": "done", + "commits": '["c2264752da1a7f4add3bf138b08fbbc982d901ea"]'}) + assert check_all(str(d), str(tmp_path)) == [] + + +class TestCrossReferences: + def test_unresolved_wikilink(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), VALID, + "See [[task-does-not-exist]].") + errors = check_all(str(docs), str(tmp_path)) + assert any("[[task-does-not-exist]] does not resolve" in e for e in errors) + + def test_resolved_wikilink(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-other.md"), + {**VALID, "id": "task-other"}) + write(str(docs / "tasks" / "pending" / "task-example.md"), VALID, + "See [[task-other]].") + assert check_all(str(docs), str(tmp_path)) == [] + + def test_missing_adr_reference(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "decisions" / "ADR-0001-real.md"), + {"id": "ADR-0001-real", "title": "Real", "type": "adr", + "status": "accepted", "repos": "[epic-code-gen]"}) + write(str(d / "tasks" / "pending" / "task-example.md"), VALID, + "Per [ADR-0099] this is fine.") + errors = check_all(str(d), str(tmp_path)) + assert any("ADR-0099" in e for e in errors) + + def test_existing_adr_reference(self, tmp_path): + d = tmp_path / "docs" + write(str(d / "decisions" / "ADR-0001-real.md"), + {"id": "ADR-0001-real", "title": "Real", "type": "adr", + "status": "accepted", "repos": "[epic-code-gen]"}) + write(str(d / "tasks" / "pending" / "task-example.md"), VALID, + "Per [ADR-0001] this is fine.") + assert check_all(str(d), str(tmp_path)) == [] + + def test_broken_relative_link(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), VALID, + "See [the thing](./missing.md).") + errors = check_all(str(docs), str(tmp_path)) + assert any("broken link" in e for e in errors) + + def test_external_and_anchor_links_ignored(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), VALID, + "[ext](https://example.com) and [anchor](#section).") + assert check_all(str(docs), str(tmp_path)) == [] + + +class TestStripCode: + """A reference inside backticks documents syntax; it isn't a reference.""" + + def test_inline_code_span_stripped(self): + assert "[[id]]" not in strip_code("Link with `[[id]]` syntax.") + + def test_fenced_block_stripped(self): + text = "before\n\n```\n[[not-a-link]]\n```\n\nafter" + out = strip_code(text) + assert "[[not-a-link]]" not in out + assert "before" in out and "after" in out + + def test_real_link_survives(self): + assert "[[real-id]]" in strip_code("See [[real-id]] for detail.") + + def test_wikilink_in_code_is_not_flagged(self, docs, tmp_path): + write(str(docs / "tasks" / "pending" / "task-example.md"), VALID, + "Use `[[id]]` to link. Fenced:\n\n```\n[[also-not-real]]\n```\n") + assert check_all(str(docs), str(tmp_path)) == [] + + +class TestPathClassification: + @pytest.mark.parametrize("path", [ + "scripts/run_pipeline.py", + "scripts/parse_prototype.js", + ".claude/agents/lint-reviewer.md", + ".claude/skills/epic-codegen/SKILL.md", + "ci-scripts/run-claude.sh", + ]) + def test_code_paths(self, path): + assert is_code(path) + + @pytest.mark.parametrize("path", [ + "docs/tasks/done/task-x.md", + "docs/bugs/open/bug-y.md", + "docs/decisions/ADR-0001-x.md", + "README.md", + "AGENTS.md", + "tests/test_check_ledger.py", + "scripts/notes.txt", + ]) + def test_non_code_paths(self, path): + assert not is_code(path) + + @pytest.mark.parametrize("path,expected", [ + ("docs/tasks/pending/task-x.md", True), + ("docs/bugs/fixed/bug-y.md", True), + ("docs/decisions/ADR-0001-x.md", True), + ("docs/architecture/01-system-overview.md", False), + ("scripts/run_pipeline.py", False), + ]) + def test_ledger_paths(self, path, expected): + assert is_ledger(path) is expected + + +class TestCheckDiff: + def test_code_without_companion_fails(self): + ok, msg = check_diff(["scripts/run_pipeline.py"]) + assert not ok + assert "no ledger companion" in msg + + def test_code_with_task_passes(self): + ok, msg = check_diff( + ["scripts/run_pipeline.py", "docs/tasks/done/task-x.md"]) + assert ok + + def test_code_with_bug_passes(self): + ok, _ = check_diff(["scripts/run_pipeline.py", "docs/bugs/fixed/bug-x.md"]) + assert ok + + def test_code_with_adr_passes(self): + ok, _ = check_diff( + ["scripts/run_pipeline.py", "docs/decisions/ADR-0035-x.md"]) + assert ok + + def test_no_code_changes_passes(self): + ok, msg = check_diff(["README.md", "docs/architecture/01-system-overview.md"]) + assert ok + assert "no code changes" in msg + + def test_empty_diff_passes(self): + ok, _ = check_diff([]) + assert ok + + def test_ledger_none_escape_hatch(self): + ok, msg = check_diff(["scripts/run_pipeline.py"], + body="## Ledger\n\nLedger: none — typo fix\n") + assert ok + assert "Ledger: none" in msg + + def test_ledger_none_is_case_insensitive(self): + ok, _ = check_diff(["scripts/run_pipeline.py"], body="ledger: NONE - trivial") + assert ok + + def test_unrelated_body_does_not_excuse(self): + ok, _ = check_diff(["scripts/run_pipeline.py"], + body="This PR does some things.") + assert not ok + + def test_architecture_doc_is_not_a_companion(self): + """Architecture docs explain; they don't record work.""" + ok, _ = check_diff( + ["scripts/run_pipeline.py", "docs/architecture/02-pipeline-state-machine.md"]) + assert not ok + + def test_many_code_files_are_truncated_in_message(self): + files = [f"scripts/mod_{i}.py" for i in range(15)] + ok, msg = check_diff(files) + assert not ok + assert "and 5 more" in msg + + +class TestRealLedger: + """The repo's own ledger must satisfy its own rules.""" + + def test_real_ledger_is_consistent(self): + errors = check_all() + assert errors == [], "\n".join(errors) + + def test_real_ledger_is_not_empty(self): + assert len(ledger_files()) > 100 diff --git a/tests/test_run_pipeline.py b/tests/test_run_pipeline.py index 083d9c7..7600989 100644 --- a/tests/test_run_pipeline.py +++ b/tests/test_run_pipeline.py @@ -22,6 +22,7 @@ check_pr_merged, clean_artifacts, find_eligible, + identity_for_repo, link_pr_to_jira, load_pr_urls_from_logs, load_repo_mapping, @@ -177,7 +178,7 @@ def test_linear_chain_processes_first_only( assert results[PROCESSED][0][0] == "RHAI-1" assert len(results[BLOCKED]) == 1 assert results[BLOCKED][0][0] == "RHAI-2" - mock_invoke.assert_called_once_with("RHAI-1", args) + mock_invoke.assert_called_once_with("RHAI-1", args, "") @mock.patch("run_pipeline.assign_issue") @mock.patch("run_pipeline.transition_issue", return_value=(True, "")) @@ -394,6 +395,26 @@ def test_passes_fork_owner(self, mock_run, tmp_path): cmd = mock_run.call_args[0][0] skill_arg = cmd[2] # -p argument assert "--fork-owner dora-the-ai-coder" in skill_arg + assert "--gh-token-var EPIC_CODEGEN_GITHUB_TOKEN" in skill_arg + + @mock.patch("run_pipeline.subprocess.run") + def test_overridden_target_passes_its_own_identity(self, mock_run, + tmp_path): + """The skill defaults to the shared bot's token var. A target with + its own credential has to override both halves or the skill pushes + to ederign's fork with dora's token.""" + mock_run.return_value = mock.MagicMock(returncode=0) + meta = tmp_path / "codegen-runs" / "RHAI-1" + meta.mkdir(parents=True) + (meta / "v1").mkdir() + (meta / "v1" / "diff.patch").write_text("diff --git a/f b/f\n") + args = _make_args(fork_owner="dora-the-ai-coder", + output_dir=str(tmp_path)) + invoke_codegen("RHAI-1", args, "rh-forge/rh-forge-ui") + + skill_arg = mock_run.call_args[0][0][2] + assert "--fork-owner ederign" in skill_arg + assert "--gh-token-var RH_FORGE_GITHUB_TOKEN" in skill_arg @mock.patch("run_pipeline.subprocess.run") def test_exit_zero_no_artifacts_returns_false(self, mock_run, tmp_path): @@ -478,6 +499,24 @@ def test_passes_fork_owner(self, mock_run, tmp_path): cmd = clone_call[0][0] assert "--fork-owner" in cmd assert "dora-the-ai-coder" in cmd + assert "EPIC_CODEGEN_GITHUB_TOKEN" in cmd + + @mock.patch("run_pipeline.subprocess.run") + def test_clone_uses_the_overriding_identity(self, mock_run, tmp_path): + """Cloning a private repo with the shared bot's token 404s, which + surfaces as 'no target_repo' rather than as a credential fault.""" + mock_run.return_value = mock.MagicMock( + returncode=0, stdout="{}", stderr="") + epic = _epic("RHAI-1") + epic["target_repo"] = "rh-forge/rh-forge-ui" + args = _make_args(fork_owner="dora-the-ai-coder", + output_dir=str(tmp_path)) + setup_target_repo(epic, args) + + cmd = mock_run.call_args_list[0][0][0] + assert "ederign" in cmd + assert "RH_FORGE_GITHUB_TOKEN" in cmd + assert "dora-the-ai-coder" not in cmd # ─── TestCleanArtifacts ─────────────────────────────────────────────────────── @@ -688,9 +727,15 @@ def test_missing_file_returns_empty(self, tmp_path): result = load_repo_mapping(str(tmp_path / "nonexistent.json")) assert result == {} - def test_shipped_mapping_covers_openc_ui(self): + def test_shipped_mapping_covers_rh_forge_ui(self): mapping = load_repo_mapping() - assert "ederign/openc-ui-by-agentic-sdlc" in mapping + assert "rh-forge/rh-forge-ui" in mapping + + def test_retired_openc_ui_entry_is_gone(self): + """Superseded by rh-forge/rh-forge-ui (RHAISTRAT-2565 closed). Its + keywords moved to the live repo rather than being dropped, so a + legacy reference resolves forward instead of to a dead target.""" + assert "ederign/openc-ui-by-agentic-sdlc" not in load_repo_mapping() def test_shipped_mapping_keywords_are_lists_of_strings(self): for repo, config in load_repo_mapping().items(): @@ -706,6 +751,85 @@ def test_shipped_keywords_are_too_long_to_match_mid_word(self): assert len(keyword) >= 4, f"{repo}: {keyword!r} is too short" +class TestIdentityForRepo: + """Per-repo GitHub credential override (ADR-0035).""" + + _MAPPING = { + "opendatahub-io/odh-dashboard": {"keywords": ["dashboard"]}, + "rh-forge/rh-forge-ui": { + "keywords": ["rh-forge"], + "fork_owner": "ederign", + "gh_token_var": "RH_FORGE_GITHUB_TOKEN", + }, + } + + def test_unmapped_repo_uses_the_shared_bot(self): + args = _make_args(fork_owner="dora-the-ai-coder") + identity = identity_for_repo( + "some/other-repo", args, mapping=self._MAPPING) + assert identity["fork_owner"] == "dora-the-ai-coder" + assert identity["gh_token_var"] == "EPIC_CODEGEN_GITHUB_TOKEN" + + def test_mapped_repo_without_override_uses_the_shared_bot(self): + args = _make_args(fork_owner="dora-the-ai-coder") + identity = identity_for_repo( + "opendatahub-io/odh-dashboard", args, mapping=self._MAPPING) + assert identity["fork_owner"] == "dora-the-ai-coder" + assert identity["gh_token_var"] == "EPIC_CODEGEN_GITHUB_TOKEN" + + def test_override_replaces_owner_and_token_var(self): + args = _make_args(fork_owner="dora-the-ai-coder") + identity = identity_for_repo( + "rh-forge/rh-forge-ui", args, mapping=self._MAPPING) + assert identity["fork_owner"] == "ederign" + assert identity["gh_token_var"] == "RH_FORGE_GITHUB_TOKEN" + + def test_our_user_follows_fork_owner(self): + """The review loop filters out our_user's comments. If this kept + naming the bot while the PR was authored by someone else, the + pipeline would answer its own comments forever.""" + args = _make_args(fork_owner="dora-the-ai-coder") + identity = identity_for_repo( + "rh-forge/rh-forge-ui", args, mapping=self._MAPPING) + assert identity["our_user"] == "ederign" + + def test_explicit_our_user_wins_over_fork_owner(self): + mapping = {"a/b": {"keywords": [], "fork_owner": "someone", + "our_user": "someone-else"}} + identity = identity_for_repo("a/b", _make_args(), mapping=mapping) + assert identity["our_user"] == "someone-else" + + @pytest.mark.parametrize("target", [ + "rh-forge/rh-forge-ui", + "https://github.com/rh-forge/rh-forge-ui", + "https://github.com/rh-forge/rh-forge-ui.git", + "git@github.com:rh-forge/rh-forge-ui.git", + ]) + def test_override_survives_url_forms(self, target): + """target_repo reaches this as a bare slug from the mapping but as a + full clone URL from a stored run state, and both must find the + override -- missing it silently falls back to a token that 404s on a + private repo.""" + identity = identity_for_repo( + target, _make_args(), mapping=self._MAPPING) + assert identity["fork_owner"] == "ederign" + + def test_empty_target_repo_falls_back_to_defaults(self): + args = _make_args(fork_owner="dora-the-ai-coder") + identity = identity_for_repo("", args, mapping=self._MAPPING) + assert identity["fork_owner"] == "dora-the-ai-coder" + assert identity["gh_token_var"] == "EPIC_CODEGEN_GITHUB_TOKEN" + + def test_shipped_override_is_wired_for_the_private_target(self): + identity = identity_for_repo( + "rh-forge/rh-forge-ui", _make_args(fork_owner="dora-the-ai-coder")) + assert identity == { + "fork_owner": "ederign", + "gh_token_var": "RH_FORGE_GITHUB_TOKEN", + "our_user": "ederign", + } + + # ─── TestResolveTargetRepo ─────────────────────────────────────────────────── class TestResolveTargetRepo: @@ -742,15 +866,34 @@ def test_empty_mapping_returns_empty(self): "Conversation Surface with Streamed Rendering", ]) def test_openc_ui_epics_resolve_without_llm_fallback(self, title): - """RHAI-543/544. A second match would defer to the LLM, so this only - holds while no other repo's keywords collide.""" + """RHAI-543/544, whose repo is now rh-forge/rh-forge-ui. A second + match would defer to the LLM, so this only holds while no other + repo's keywords collide.""" epic = _epic("RHAI-1", title=title) epic["body"] = ( "Implement this in the openc-ui-by-agentic-sdlc repository, " "which requires a PatternFly 6 UI built on React 19." ) result = resolve_target_repo(epic, load_repo_mapping()) - assert result == "ederign/openc-ui-by-agentic-sdlc" + assert result == "rh-forge/rh-forge-ui" + + @pytest.mark.parametrize("title,body", [ + ("Data binding and honest rendering of draft proposals in the " + "home-page drawer", + "When a home-page item carries a draft id, the primary action opens " + "the drawer on that draft instead of navigating to the drafts list."), + ("Approval, undo, and revision handling for draft proposals in the " + "drawer", + "Wire the drawer's approve action to the same send services the " + "drafts list uses, and handle revision-under-review."), + ]) + def test_rh_forge_epics_resolve_without_llm_fallback(self, title, body): + """RHAI-760/761 (RHAISTRAT-2671). Both must land on the one repo + without the LLM, which can only pick from this same mapping.""" + epic = _epic("RHAI-1", title=title) + epic["body"] = body + assert resolve_target_repo(epic, load_repo_mapping()) == \ + "rh-forge/rh-forge-ui" @mock.patch("run_pipeline.resolve_repo_via_llm", return_value="") def test_no_match_calls_llm(self, mock_llm): diff --git a/tests/test_toolchain_preflight.py b/tests/test_toolchain_preflight.py index 8a069f3..d9d2e78 100644 --- a/tests/test_toolchain_preflight.py +++ b/tests/test_toolchain_preflight.py @@ -20,7 +20,9 @@ _parse_makefile_rules, _parse_makefile_vars, _tools_in_recipe_line, + detect_package_manager, detect_required_tools, + discover_commands, preflight, run_check, tools_for_target, @@ -221,6 +223,28 @@ def test_yarn_required_when_lockfile_present(self, tmp_path): yarn__lock="") assert "yarn" in detect_required_tools(repo, "javascript") + def test_pnpm_required_when_lockfile_present(self, tmp_path): + """The blind spot that let rh-forge-ui preflight clean with no pnpm.""" + repo = self._write( + tmp_path, package__json='{"scripts":{"lint":"eslint ."}}') + (tmp_path / "pnpm-lock.yaml").write_text("") + assert "pnpm" in detect_required_tools(repo, "typescript") + + def test_pnpm_required_when_declared_in_package_json(self, tmp_path): + repo = self._write( + tmp_path, + package__json='{"packageManager":"pnpm@10.32.1",' + '"scripts":{"lint":"eslint ."}}') + assert "pnpm" in detect_required_tools(repo, "typescript") + + def test_plain_npm_repo_gates_on_nothing_extra(self, tmp_path): + repo = self._write( + tmp_path, package__json='{"scripts":{"lint":"eslint ."}}') + (tmp_path / "package-lock.json").write_text("{}") + tools = detect_required_tools(repo, "javascript") + assert "pnpm" not in tools + assert "yarn" not in tools + def test_no_makefile_still_returns_base_tools(self, tmp_path): repo = self._write(tmp_path, go__mod="module x\n") tools = detect_required_tools(repo, "go") @@ -251,6 +275,88 @@ def test_tools_are_deduplicated(self, tmp_path): assert len(tools) == len(set(tools)) +class TestDetectPackageManager: + + def _write(self, tmp_path, **files): + for name, content in files.items(): + (tmp_path / name.replace("__", ".")).write_text(content) + return str(tmp_path) + + def test_defaults_to_npm(self, tmp_path): + repo = self._write(tmp_path, package__json='{"name":"x"}') + assert detect_package_manager(repo) == "npm" + + @pytest.mark.parametrize("lockfile,expected", [ + ("pnpm-lock.yaml", "pnpm"), + ("yarn.lock", "yarn"), + ]) + def test_lockfile_implies_manager(self, tmp_path, lockfile, expected): + (tmp_path / "package.json").write_text('{"name":"x"}') + (tmp_path / lockfile).write_text("") + assert detect_package_manager(str(tmp_path)) == expected + + def test_declaration_wins_over_stale_lockfile(self, tmp_path): + """`packageManager` is the repo's own statement of record.""" + repo = self._write( + tmp_path, + package__json='{"packageManager":"pnpm@10.32.1"}', + yarn__lock="") + assert detect_package_manager(repo) == "pnpm" + + def test_unknown_declaration_falls_back_to_lockfile(self, tmp_path): + repo = self._write( + tmp_path, package__json='{"packageManager":"bun@1.0.0"}') + (tmp_path / "pnpm-lock.yaml").write_text("") + assert detect_package_manager(repo) == "pnpm" + + def test_unreadable_package_json_does_not_raise(self, tmp_path): + repo = self._write(tmp_path, package__json="{not json") + assert detect_package_manager(repo) == "npm" + + def test_missing_package_json_defaults_to_npm(self, tmp_path): + assert detect_package_manager(str(tmp_path)) == "npm" + + +class TestDiscoverCommandsPackageManager: + """Scripts must run through the manager the repo declares. + + A pnpm workspace installed and driven by npm resolves a different + dependency tree than the repo's own CI, so the checks would not be + measuring the same thing — and `npm run lint` in a pnpm-only repo + exits non-zero for reasons that have nothing to do with the code. + """ + + def _js_repo(self, tmp_path, manager_file, scripts): + (tmp_path / "package.json").write_text( + json.dumps({"scripts": scripts})) + if manager_file: + (tmp_path / manager_file).write_text("") + return str(tmp_path) + + def test_pnpm_repo_gets_pnpm_commands(self, tmp_path): + repo = self._js_repo( + tmp_path, "pnpm-lock.yaml", + {"lint": "eslint .", "test": "vitest run"}) + cmds = discover_commands(repo, "typescript") + assert cmds["lint"] == "pnpm run lint" + assert cmds["test"] == "pnpm test" + + def test_pnpm_typecheck_runs_through_pnpm(self, tmp_path): + repo = self._js_repo( + tmp_path, "pnpm-lock.yaml", {"typecheck": "tsc --noEmit"}) + assert discover_commands(repo, "typescript")["typecheck"] == ( + "pnpm run typecheck") + + def test_npm_repo_still_gets_npm_commands(self, tmp_path): + repo = self._js_repo(tmp_path, None, {"lint": "eslint ."}) + assert discover_commands(repo, "typescript")["lint"] == "npm run lint" + + def test_makefile_target_used_when_no_matching_script(self, tmp_path): + repo = self._js_repo(tmp_path, "pnpm-lock.yaml", {"build": "vite build"}) + (tmp_path / "Makefile").write_text("lint:\n\tpnpm lint\n") + assert discover_commands(repo, "typescript")["lint"] == "make lint" + + class TestPreflight: def test_reports_ok_when_tools_present(self, tmp_path):