Review: skill-based prompts and an ensemble pipeline (benchmark candidate, do not merge) by fatihacet · Pull Request #157 · kaicontext/kai-cli

fatihacet · 2026-09-30T15:43:05Z

Do not merge yet. This is the branch the benchmark candidate runs are built from. It stays open while we iterate.

What it changes

The review pipeline (run 15, ad09116 and 66f353b)

Stage	Production (v0.36.0)	This branch
Quick pass	Haiku draft + GLM check; its comment was deleted once the grounded review posted	Retired from CI: an explicit `--fast` refuses (`KAI_REVIEW_QUICK_PASS=1` runs it)
Finders	1 agent	2 agents in parallel: the main model, plus `finder2` (default `openai/gpt-5.5`, low effort, no session of its own)
Sweep	1	2 over the same hunks: the sweep model, plus `sweep2` (default `z-ai/glm-5.2`); issues merged without repeats
Fact-check	6 repository lookups	8
Selection	none: everything the check supported was published	New `rank` stage (default `anthropic/claude-opus-5.5`, medium). Reads the confirmed defects together, publishes the ones a maintainer would want fixed, most important first, 3–6 by change size, never none
Published prose	findings + "Could not verify" + "Limitations"	findings only; the rest stays in the challenge record
Locations	`path:function` issues were held (posted, anchored nowhere, not counted)	`path:function` is resolved to the function's line

The new stages are review-profile stages (finder2, sweep2, rank) with env overrides (KAI_FINDER2_MODEL, KAI_SWEEP2_MODEL, KAI_RANK_MODEL and their _EFFORT twins). off turns a stage off.

The prompts (run 14, f5fde0d…c5b1106)

The review, quick-pass, sweep and fact-check prompts are composed from markdown files in cmd/kai/reviewskill/, compiled into the binary:

one shared defect definition and catalog, instead of four copies that had drifted;
language and risk packs chosen from the diff (Go, Java, Python, TypeScript, Ruby, CI; auth, money, data, external);
the sweep's line checklist;
a known limitation counts as a defect, and unbounded work (no deadline or size limit) counts as reachable.

TestPromptsCarryNoBenchmarkCases still guards against benchmark cases.

The build (e92a8c0)

build-kai-ci.yml gains candidate_kai_cli_ref: it builds kai-cli from a branch or commit instead of its newest tag, and refuses any pin but none. That is how acetz's per-org review_image (kai-server#389) runs this branch while production stays on v0.36.0.

Results (Martian Code Review Bench, 50 PRs, F1 under the Opus / Sonnet / GPT-5.2 judges)

Run	Build	F1	Opus P / R	Opus TP / FP	Scored findings per PR
13	production v0.36.0, GLM-5.2	45.5 / 45.4 / 40.9	39.8 / 53.2	92 / 139	5.3
14	this branch at `e92a8c0` (prompts only), GLM-5.2	44.0 / 42.1 / 41.4	38.1 / 52.0	90 / 146	5.3
15	this branch at `66f353b` (pipeline), Sonnet 5.5 main and sweep, GPT-5.5 fact-check	51.1 / 47.1 / 43.3	48.2 / 54.3	94 / 101	3.7

Reports: kai-server#390, #391, #392; history in kai-server#330.

The prompt restructure alone moved nothing. The pipeline change took the Opus judge from 45.5 to 51.1, almost entirely through precision.

make the published findings shorter, since the Sonnet and GPT extractors split each finding's paragraph and remedy into extra candidates;
measure what the selection step holds back before touching its cap.

The review prompts were Go strings that had grown by accretion: a 17k-character reviewer prompt of shouted rule blocks and incident stories, and a fast pass, a sweep and a fact-check that each restated the definition of a defect in their own words. They are now composed from markdown files in cmd/kai/reviewskill/, compiled into the binary: - procedure.md: how a grounded review proceeds, as steps (orient, trace outward, walk the catalog, ground each suspicion, check what you could not see, look for decisions, write). - defect.md: what counts as a defect and what does not, one cause per issue, and decisions. One definition, shared by the reviewer, the fast pass and the fact-check, so they judge by the same bar. - catalog.md: eight categories of defect to check, the hardest five each with a finding next to something that is not one. Examples come from code outside any benchmark. - report.md, fast.md, sweep.md, challenge.md: each stage's role and output. Codas and output formats are unchanged, so the parsers are untouched. - packs/: language (Go, Java/Kotlin, Python, TypeScript/JavaScript, Ruby/Rails) and risk (auth, money, data, external calls) checklists, added to the reviewer's and the sweep's prompt only when the change touches them (rcPacksFor; logged as "review skill packs: …"). Every rule the old prompts enforced is still pinned by a test, anchored on its new wording; new tests check the composition order, that the codas stay last and complete, that the catalog shows examples, that the skill does not shout, and that packs follow the diff. review-regressions gains record=false: an ad-hoc comparison of two refs that reports to the job summary and leaves the nightly dashboard alone.

…s a defect The first corpus comparison (3 runs each) caught 11/24 against main's 14/24. The logs show where: - The sweep proposed 76 candidates against main's 184. It was reading the whole catalog, findings and non-findings, where main gave it a compact line-level checklist. It gets that checklist back, and the packs. - The fact-check refuted a traced defect because a test comment accepted it as a tradeoff. "Intended behaviour" now means a product change the author chose, not a wrong answer on a realistic input; and the concrete kinds of defect to support come back from main's wording. - Workflow files had no checks of their own. A ci pack covers script injection through ${{ }}, concurrency between workflows, set -e and quoting; workflows are matched by path, shell scripts by extension. - The reviewer filed traced defects as prose "limitations". The report format now says a traced defect goes in ISSUES. - A risk pack needs two hits: one "session" is not an auth change.

…n is no defence In iteration 2 the fact-check refuted a git grep with no deadline as a hypothetical trigger, because the same unbounded pattern already existed beside it. The defect definition now treats a missing deadline or size limit on input the code does not control like a race: reachable today unless a bound is shown. New code that repeats a defect from its neighbour is a defect in the new lines.

To trial a review change on one org before release, the org's review_image (kai-server org_review_config) needs an image built from the branch. The new candidate_kai_cli_ref input builds kai-cli from that ref instead of its newest tag. It refuses any pin but none, so no production default is made from a branch tip, and it is passed through env and checked as a ref name.

…on step The grounded review proposed from one agent and one sweep on one model, and published everything its same-model fact-check found true. Two benchmark runs of one configuration matched ~90 known defects each but only 68 in common, so each run saw a different slice; and the check published true-but-minor findings next to real ones. - A second finder agent runs beside the first on another model family (finder2, default openai/gpt-5.5, low effort), with the same prompt and tools and no session of its own. Its issues join the draft; if the main finder ends without a review, the second's stands in. - A second sweep reads the same hunks on another model (sweep2, default z-ai/glm-5.2); both sweeps' issues are merged without repeats. - The fact-check gets 8 repository lookups instead of 6. - A selection step (rank, default anthropic/claude-opus-5.5, medium effort) reads the confirmed defects together and publishes the ones a maintainer would want fixed, most important first, 3 to 6 by the size of the change and never none; the rest are kept in the record as not published. - The published review drops "Could not verify" and "Limitations": on the pull request every sentence reads as a claim, so only defects go there. - The quick pass is retired from CI: its comment was deleted when the grounded review posted. An explicit --fast now refuses in a second (KAI_REVIEW_QUICK_PASS=1 runs it); a local run without flags is unchanged. The new stages are review-profile stages (finder2, sweep2, rank) with environment overrides (KAI_FINDER2_MODEL, KAI_SWEEP2_MODEL, KAI_RANK_MODEL and their _EFFORT twins); "off" turns one off.

The smoke run of the ensemble published two real defects the main finder wrote as cmd/kai/ship.go:shipIsMergeSubject instead of path:line. Grounding held both (no line), so they were posted as text, anchored nowhere and not counted. rcLocateNamedIssue rewrites path:symbol to the symbol's definition line in the reviewed commit, else its first appearance; anything it cannot find is left as it was.

fatihacet · 2026-09-30T15:46:15Z

@kaicontext review this!

kaicontext · 2026-09-30T15:46:37Z

On it. The review will appear on this pull request in a few minutes. 👀

kaicontext

Kai review · 🔵 Your call, then merge (4/5)

Next: merge when your checks are green — I found nothing to fix.

What I made of it

Read the whole thing — it does what it says, and nothing jumped out at me. ✅

0 confirmed findings, 12 refuted. Intent verified; readiness 4/5.

Important files changed

File	Change
`.github/workflows/build-kai-ci.yml`	modified · +25 −0
`.github/workflows/review-regressions.yml`	modified · +26 −1
`cmd/kai/review_commit.go`	modified · +29 −69
`cmd/kai/review_commit_challenge.go`	modified · +1 −21
`cmd/kai/review_commit_duplicates_test.go`	modified · +1 −1
`cmd/kai/review_commit_ensemble.go`	modified · +390 −0
`cmd/kai/review_commit_ensemble_test.go`	modified · +128 −0
`cmd/kai/review_commit_fast.go`	modified · +1 −50

_{+28 more changed files in the full analysis.}

What I opened — 18 files, 24 turns, 2m27s

20 of the 36 changed files don't appear below: cmd/kai/review_commit_fast.go, cmd/kai/review_commit_fast_test.go, cmd/kai/review_commit_ground_test.go, cmd/kai/review_commit_noise_test.go, cmd/kai/review_commit_rules_test.go, cmd/kai/review_commit_skill_test.go, cmd/kai/review_commit_sweep.go, cmd/kai/reviewskill/catalog.md, cmd/kai/reviewskill/challenge.md, cmd/kai/reviewskill/defect.md, and 10 more.

.github/workflows/build-kai-ci.yml
.github/workflows/review-regressions.yml
cmd/kai/review_commit.go
cmd/kai/review_commit_batches.go
cmd/kai/review_commit_challenge.go
cmd/kai/review_commit_duplicates_test.go
cmd/kai/review_commit_ensemble.go
cmd/kai/review_commit_ensemble_test.go
cmd/kai/review_commit_ground.go
cmd/kai/review_commit_profile.go
cmd/kai/review_commit_repo.go
cmd/kai/review_commit_skill.go
cmd/kai/review_commit_speculation_test.go
cmd/kai/review_commit_verdicts.go
cmd/kai/reviewskill/fast.md
cmd/kai/reviewskill/procedure.md
cmd/kai/reviewskill/report.md
cmd/kai/reviewskill/sweep.md

Full read-through

Scope

review-regressions.yml concurrency group, Report step condition, ad-hoc Report step condition
build-kai-ci.yml candidate_kai_cli_ref input validation regex
GitHub Actions type: boolean input handling in expression context
cmd/kai/review_commit.go (flag guard, prose filtering, ensemble wiring)
cmd/kai/review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcStartSecondFinder, rcWithSecondFinder, rcMergeSweeps)
cmd/kai/review_commit_skill.go (rcPacksFor, rcReviewSystemFor, rcComposeStage, pack selection)
cmd/kai/review_commit_ground.go (rcLocateNamedIssue, rcNamedLocationRe)
cmd/kai/review_commit_profile.go (ensemble stage effort routing)
cmd/kai/review_commit_sweep.go (rcRunSweepWith, rcMergeSweeps, second sweep)
cmd/kai/review_commit_batches.go (rcFinalize)
cmd/kai/review_commit_challenge.go (rcChallengeSystemHead, rcAssembleReview)
.github/workflows/build-kai-ci.yml (candidate guard)
.github/workflows/review-regressions.yml (ad-hoc report)
cmd/kai/reviewskill/*.md (all skill files)
test files: review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_noise_test.go, review_commit_speculation_test.go, review_commit_fast_test.go, review_commit_duplicates_test.go, review_commit_ground_test.go
cmd/kai/review_commit_skill.go (rcReviewSystemFor, rcComposeStage, rcPacksFor, rcLangOf)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcMergeSweeps, rcStartSweep)
cmd/kai/review_commit.go (runReviewCommit, rcRunReviewAgent, quick-pass retirement, rcDefectsOnlyProse call)
cmd/kai/review_commit_profile.go (rcStageEffort routing for ensemble stages)
cmd/kai/review_commit_rules_test.go, review_commit_skill_test.go, review_commit_ensemble_test.go, review_commit_noise_test.go, review_commit_speculation_test.go, review_commit_fast_test.go, review_commit_duplicates_test.go, review_commit_ground_test.go (rewritten/added tests)
.github/workflows/build-kai-ci.yml (candidate_kai_cli_ref guard)
.github/workflows/review-regressions.yml (record=false ad-hoc report, concurrency group)
cmd/kai/reviewskill/.md and packs/.md (embedded skill files)

No proposed defect was confirmed by this check within the reviewed scope.

Limitations

Could not verify GitHub Actions runtime behavior for type: boolean inputs in expression context — no source in this repository establishes whether inputs.record is a boolean or a string in expression comparisons
Did not review the ensemble Go code (rcRankSupported, rcApplyRank, rcStartSecondFinder, etc.) or the skill markdown files for defects — only the four workflow YAML issues were in scope
Did not review kai-engine's agent.ApplyEffort contract or provider concurrency safety (sibling repos)
kai-engine's agent.ApplyEffort internals could not be verified from this repo (whether re-setting ReasoningEffort on a copied opts struct requires re-applying ApplyEffort)
Provider concurrency safety for parallel prov.Send could not be verified from this repo
kai-server side that consumes candidate_kai_cli_ref is in a sibling repo and was not examined
Existence of default model ids (openai/gpt-5.5, z-ai/glm-5.2, anthropic/claude-opus-5.5) in each provider's catalog was not confirmed
kai-engine's agent.ApplyEffort internals not verified (whether re-setting ReasoningEffort on a copied opts struct requires re-applying ApplyEffort) — lives in a sibling repo not in the sources
provider.Provider concurrency safety for parallel prov.Send not verified — same sibling repo; the pre-existing parallel sweep chunks imply it is safe
kai-server side consuming candidate_kai_cli_ref not reviewed
Specific model ids (openai/gpt-5.5, z-ai/glm-5.2, anthropic/claude-opus-5.5) existence in provider catalogs not confirmed

_{+1535 −270 · 36 files · reaches 24 · the full analysis}
_{💬 Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic".}

kaicontext

Kai review · 🟡 Small fixes first (3/5)

Next: fix 2 findings and say yes or no to 2 decisions, then push — I review again on every push.

	What	Where
🐞	`rcDefectsOnlyProse` breaks its scan on the first non-line-start occurrence of `## Could not verify`/`## Limitations`, so a finding body that quotes that heading text leaves a later, real, line-start section unstripped…	`cmd/kai/review_commit_ensemble.go:374`
🐞	`rcMergeSweeps` compares `b.Failed < a.Failed` and sets `out.Failed = b.Failed`, replacing the larger failure count with the smaller instead of summing failures; this loses the count of failed chunks from the sweep with…	`cmd/kai/review_commit_sweep.go:506`
🤔	The rank step re-marks fact-checked confirmed defects as refuted to hold them off the PR, capped 3–6 by change size with a "never none" floor of one; every org running this candidate branch gets its published review f…	—
🤔	Running a second finder and a second sweep under the review's own deadline means every candidate review now waits for the slower of two agents on two model families before challenging; this is a deliberate latency/cover…	—

_{🐞 fix before merge · 🤔 correct as written, but someone should say yes · the ones with a location are also comments on those lines.}

What I made of it

Read through this one. 2 things worth your eyes before it merges, plus 2 decisions to say yes to.

2 confirmed findings, 14 refuted. Intent verified; readiness 3/5.

Important files changed

File	Change
`.github/workflows/build-kai-ci.yml`	modified · +25 −0
`.github/workflows/review-regressions.yml`	modified · +26 −1
`cmd/kai/review_commit.go`	modified · +29 −69
`cmd/kai/review_commit_challenge.go`	modified · +1 −21
`cmd/kai/review_commit_duplicates_test.go`	modified · +1 −1
`cmd/kai/review_commit_ensemble.go`	modified · +390 −0
`cmd/kai/review_commit_ensemble_test.go`	modified · +128 −0
`cmd/kai/review_commit_fast.go`	modified · +1 −50

_{+28 more changed files in the full analysis.}

What I opened — 25 files, 36 turns, 3m33s

13 of the 36 changed files don't appear below: cmd/kai/reviewskill/catalog.md, cmd/kai/reviewskill/challenge.md, cmd/kai/reviewskill/defect.md, cmd/kai/reviewskill/fast.md, cmd/kai/reviewskill/packs/auth.md, cmd/kai/reviewskill/packs/data.md, cmd/kai/reviewskill/packs/external.md, cmd/kai/reviewskill/packs/go.md, cmd/kai/reviewskill/packs/java.md, cmd/kai/reviewskill/packs/money.md, and 3 more.

.github/workflows/build-kai-ci.yml
.github/workflows/review-regressions.yml
api/agent/agent.go
cmd/kai/review_commit.go
cmd/kai/review_commit_batches.go
cmd/kai/review_commit_challenge.go
cmd/kai/review_commit_duplicates_test.go
cmd/kai/review_commit_ensemble.go
cmd/kai/review_commit_ensemble_test.go
cmd/kai/review_commit_fast.go
cmd/kai/review_commit_fast_test.go
cmd/kai/review_commit_ground.go
cmd/kai/review_commit_ground_test.go
cmd/kai/review_commit_noise_test.go
cmd/kai/review_commit_profile.go
cmd/kai/review_commit_repo.go
cmd/kai/review_commit_rules_test.go
cmd/kai/review_commit_skill.go
cmd/kai/review_commit_skill_test.go
cmd/kai/review_commit_speculation_test.go
cmd/kai/review_commit_sweep.go
cmd/kai/reviewskill/packs/ci.md
cmd/kai/reviewskill/procedure.md
cmd/kai/reviewskill/report.md
cmd/kai/reviewskill/sweep.md

Full read-through

Scope

cmd/kai/review_commit_ensemble.go (new ensemble: rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcStartSecondFinder, rcWithSecondFinder, rcMergeSweeps)
cmd/kai/review_commit_ground.go (rcLocateNamedIssue and rcNamedLocationRe)
cmd/kai/review_commit_skill.go (new: prompt composition from reviewskill/*.md, rcPacksFor)
cmd/kai/review_commit.go (runReviewCommit changes: quick-pass refusal, rcLocateNamedIssue call site, rcRankSupported call site, rcDefectsOnlyProse call site)
cmd/kai/review_commit_sweep.go (rcRunSweepWith, rcMergeSweeps, second sweep)
cmd/kai/review_commit_profile.go (ensemble stages, rcStageEffort)
cmd/kai/review_commit_repo.go (rcMaxLookups 6 to 8)
.github/workflows/build-kai-ci.yml (candidate_kai_cli_ref)
.github/workflows/review-regressions.yml (record input, ad-hoc report)
cmd/kai/reviewskill/*.md (new prompt files)
cmd/kai/review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_noise_test.go, review_commit_fast_test.go, review_commit_speculation_test.go, review_commit_ground_test.go, review_commit_duplicates_test.go (test updates)
cmd/kai/review_commit_ground.go (rcLocateNamedIssue, rcNamedLocationRe)
cmd/kai/review_commit_ensemble.go (full file)
cmd/kai/review_commit_skill.go (full file)
cmd/kai/reviewskill/ (fast.md, procedure.md, report.md, sweep.md, challenge.md, defect.md, catalog.md, packs/)
cmd/kai/review_commit.go (rcRunReviewAgent, runReviewCommit)
cmd/kai/review_commit_sweep.go (rcStartSweep, rcMergeSweeps)
cmd/kai/review_commit_fast.go (rcFastReviewSystem)
cmd/kai/review_commit_profile.go (stages, effort)
cmd/kai/review_commit_batches.go (rcFinalize)
cmd/kai/review_commit_challenge.go (rcIssueKey, rcAssembleReview)
.github/workflows/review-regressions.yml (record, concurrency)
All test files: review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_fast_test.go, review_commit_noise_test.go, review_commit_speculation_test.go, review_commit_duplicates_test.go, review_commit_ground_test.go
cmd/kai/review_commit_ground.go (rcLocateNamedIssue, rcIssueLocation, rcResolvePath, rcFileLines)
cmd/kai/review_commit.go (rcRunReviewAgent, runReviewCommit, quick-pass refusal)
cmd/kai/review_commit_sweep.go (rcStartSweep, rcMergeSweeps, rcRunSweepWith)
cmd/kai/reviewskill/*.md (procedure, defect, catalog, fast, sweep, challenge, report, packs/ci)
cmd/kai/review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_noise_test.go, review_commit_speculation_test.go, review_commit_fast_test.go, review_commit_duplicates_test.go, review_commit_ground_test.go
.github/workflows/build-kai-ci.yml (candidate_kai_cli_ref), review-regressions.yml (record=false ad-hoc comparison)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcMergeSweeps, rcStartSweep), cmd/kai/review_commit_ensemble.go (rcDefectsOnlyProse, rcRankSupported, rcApplyRank, rcStartSecondFinder), cmd/kai/review_commit_ground.go (rcLocateNamedIssue), cmd/kai/review_commit_skill.go (prompt composition, rcPacksFor), cmd/kai/review_commit.go (runReviewCommit, rcRunReviewAgent), cmd/kai/reviewskill/*.md, .github/workflows/build-kai-ci.yml, .github/workflows/review-regressions.yml, all new/modified test files

Findings

cmd/kai/review_commit_ensemble.go:374-389 — `rcDefectsOnlyProse` breaks its scan on the first non-line-start occurrence of `## Could not verify`/`## Limitations`, so a finding body that quotes that heading text leaves a later, real, line-start section unstripped and published on the PR; the loop should `continue` past a non-line-start match, not `break`.

rcDefectsOnlyProse uses strings.Index to find each heading, but when the heading text appears mid-line inside a finding body (not at a line start), the guard i > 0 && prose[i-1] != '\n' triggers a break that exits the inner loop. A later, legitimate, line-start section with the same heading is then never stripped, and ships to the PR despite the 'findings only' contract. Concrete trigger: a finding whose body quotes '## Limitations' or '## Could not verify' as part of its prose (model-generated free text copied verbatim by rcAssembleReview).

Remedy: Instead of break when the match is not at a line start, advance past the non-line-start occurrence and continue searching: replace the guard with a skip that searches from i+1 and continues the loop, so a later line-start heading is still found and stripped.

cmd/kai/review_commit_sweep.go:506 — `rcMergeSweeps` compares `b.Failed < a.Failed` and sets `out.Failed = b.Failed`, replacing the larger failure count with the smaller instead of summing failures; this loses the count of failed chunks from the sweep with more failures.

rcMergeSweeps sets out := a (inheriting a.Failed), then overwrites out.Failed = b.Failed only when b.Failed < a.Failed, replacing the larger failure count with the smaller. A merge of sweep a (3 failed chunks) and sweep b (1 failed chunk) reports 1 failed chunk, silently losing 2. The Failed field tracks per-chunk failures (res.Failed++ on error) and is printed in the sweep log line.

Remedy: Change the Failed merge to out.Failed = a.Failed + b.Failed (sum the failures from both sweeps), or keep the maximum: if a.Failed > b.Failed { out.Failed = a.Failed } else { out.Failed = b.Failed }.

Limitations

Cannot read the kai-engine sibling repo: agent.Options struct definition, agent.Run concurrency contract on shared Options, provider.Provider/Response, message types, finding types. The data-race concern about two concurrent agent.Run calls sharing one opts value could not be settled.
Cannot read the kai-server repo that owns the CI invocation of review-commit (whether it passes --fast or --deep, whether it sets KAI_REVIEW_QUICK_PASS). The correctness of the quick-pass retirement from CI rests on that contract.
Could not read the kai-engine sibling repo (agent.Options, agent.Run, provider.Provider, message, finding types) — the concurrency contract for two simultaneous agent.Run calls sharing one opts value lives there
Could not read the kai-server repo that owns the CI invocation of review-commit (whether it passes --fast or --deep, and whether it sets KAI_REVIEW_QUICK_PASS)
Did not verify the model identifiers (openai/gpt-5.5, z-ai/glm-5.2, anthropic/claude-opus-5.5) resolve to real provider endpoints
Did not execute any code or tests
kai-engine sibling repo not readable: agent.Options struct definition, agent.Run's read/write contract on Options, provider.Provider/Request/Response, message types, finding types — the concurrency safety of two simultaneous agent.Run calls sharing one opts value cannot be confirmed from this repo
kai-server repo not readable: whether CI sets KAI_REVIEW_QUICK_PASS=1 or relies on the refusal's exit being tolerated, and how review-commit is invoked in production
rcChallengeSystemTail referenced by TestPromptsCarryNoBenchmarkCases but not opened in full — only confirmed it exists via grep
Cannot read the kai-engine sibling repo (agent.Options, agent.Run concurrency contract) or the kai-server repo (CI invocation contract for review-commit). Did not run tests or execute code; settled behavior from source text alone.

Decisions (need your call)

The rank step re-marks fact-checked confirmed defects as refuted to hold them off the PR, capped 3–6 by change size with a "never none" floor of one; every org running this candidate branch gets its published review filtered by a third model's judgement. The author frames this as a non-production benchmark candidate, but it's the behaviour any org pointed at the candidate image now receives, and the cap is worth a human's explicit yes before this leaves candidate status.
Running a second finder and a second sweep under the review's own deadline means every candidate review now waits for the slower of two agents on two model families before challenging; this is a deliberate latency/coverage trade (visible in the benchmark's precision gain) that affects review wall-time for any org on the candidate image.

_{+1535 −270 · 36 files · reaches 24 · the full analysis}
_{💬 Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic".}

kaicontext · 2026-09-30T15:54:13Z

+// defects: the list of claims the check could not settle and the reviewer's
+// limitations. Both stay in the challenge record; on the pull request, every
+// sentence reads as a claim about the code, so only defects belong there.
+func rcDefectsOnlyProse(prose string) string {


🐞 rcDefectsOnlyProse breaks its scan on the first non-line-start occurrence of ## Could not verify/## Limitations, so a finding body that quotes that heading text leaves a later, real, line-start section unstripped and published on the PR; the loop should continue past a non-line-start match, not break.

kaicontext · 2026-09-30T15:54:13Z

+				r := rcRunSweepWith(ctx, prov, m2, rcStageEffort(rcStageSweep2), system, intent, order, patches)
+				fmt.Fprintf(os.Stderr, "  second sweep (%s): %d chunk(s) read in %s, %d failed, %d defect(s) proposed\n",
+					m2, r.Chunks, time.Since(s).Round(time.Second), r.Failed, len(r.Issues))
+				second <- r


🐞 rcMergeSweeps compares b.Failed < a.Failed and sets out.Failed = b.Failed, replacing the larger failure count with the smaller instead of summing failures; this loses the count of failed chunks from the sweep with more failures.

…eview up Run 15's GPT-5.5 second finder and fact-check cost about $2.12 of a $2.80 review. Tried in its place on the same change: Kimi K2.6 took about a minute a turn; Gemini 3.5 Flash made two tool calls and proposed nothing, and as the fact-check broke the answer protocol and supported every allegation. GPT-5.6 Sol ($2/$10 per Mtok against GPT-5.5's $5/$30) proposed a real issue in 47s and checked cleanly. The review now waits at most 5 minutes past the main finder for the second one, then cancels it and goes on: extra coverage must not hold every review to its hard deadline.

kaicontext

Kai review · 🟡 Small fixes first (3/5)

Next: fix 2 findings and say yes or no to 1 decision, then push — I review again on every push.

	What	Where
🐞	`rcMergeSweeps` replaces `out.Failed` with `b.Failed` only when `a.Failed > 0 && b.Failed < a.Failed`, so when the second sweep has more failures than the first (`b.Failed > a.Failed`), the merged `Failed` count silentl…	`cmd/kai/review_commit_sweep.go:569`
🐞	`rcMergeSweeps` sets `out.Sources = b.Sources` only when `len(out.Sources) == 0`, but when both sweeps produce sources the second sweep's distinct sources are dropped, since the function never unions the two source maps…	`cmd/kai/review_commit_sweep.go:571`
🤔	The ensemble defaults ship `openai/gpt-5.6-sol`, `z-ai/glm-5.2`, and `anthropic/claude-opus-5.5` as per-review model defaults; every org running this candidate branch pays the per-review cost of those calls, and if any…	—

_{🐞 fix before merge · 🤔 correct as written, but someone should say yes · the ones with a location are also comments on those lines.}

What I made of it

Read through this one. 2 things worth your eyes before it merges, plus 1 decision to say yes to.

2 confirmed findings, 26 refuted, 1 unresolved. 1 item could not be verified and is withheld. Intent verified; readiness 3/5.

Important files changed

File	Change
`.github/workflows/build-kai-ci.yml`	modified · +25 −0
`.github/workflows/review-regressions.yml`	modified · +26 −1
`cmd/kai/review_commit.go`	modified · +29 −69
`cmd/kai/review_commit_challenge.go`	modified · +1 −21
`cmd/kai/review_commit_duplicates_test.go`	modified · +1 −1
`cmd/kai/review_commit_ensemble.go`	modified · +415 −0
`cmd/kai/review_commit_ensemble_test.go`	modified · +145 −0
`cmd/kai/review_commit_fast.go`	modified · +1 −50

_{+28 more changed files in the full analysis.}

What I opened — 27 files, 28 turns, 3m24s

11 of the 36 changed files don't appear below: cmd/kai/review_commit_speculation_test.go, cmd/kai/reviewskill/packs/auth.md, cmd/kai/reviewskill/packs/ci.md, cmd/kai/reviewskill/packs/data.md, cmd/kai/reviewskill/packs/external.md, cmd/kai/reviewskill/packs/go.md, cmd/kai/reviewskill/packs/java.md, cmd/kai/reviewskill/packs/money.md, cmd/kai/reviewskill/packs/python.md, cmd/kai/reviewskill/packs/ruby.md, and 1 more.

.github/workflows/build-kai-ci.yml
.github/workflows/review-regressions.yml
cmd/kai/review_commit.go
cmd/kai/review_commit_batches.go
cmd/kai/review_commit_challenge.go
cmd/kai/review_commit_duplicates_test.go
cmd/kai/review_commit_ensemble.go
cmd/kai/review_commit_ensemble_test.go
cmd/kai/review_commit_fast.go
cmd/kai/review_commit_fast_test.go
cmd/kai/review_commit_ground.go
cmd/kai/review_commit_ground_test.go
cmd/kai/review_commit_noise_test.go
cmd/kai/review_commit_profile.go
cmd/kai/review_commit_repo.go
cmd/kai/review_commit_rules_test.go
cmd/kai/review_commit_skill.go
cmd/kai/review_commit_skill_test.go
cmd/kai/review_commit_sweep.go
cmd/kai/review_commit_verdicts.go
cmd/kai/reviewskill/catalog.md
cmd/kai/reviewskill/challenge.md
cmd/kai/reviewskill/defect.md
cmd/kai/reviewskill/fast.md
cmd/kai/reviewskill/procedure.md
cmd/kai/reviewskill/report.md
cmd/kai/reviewskill/sweep.md

Full read-through

Scope

Reviewed the full ensemble + review-skill change at commit 66f353b: review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcSecondFinder, rcWithSecondFinder, rcMergeSweeps), review_commit_ground.go (rcLocateNamedIssue), review_commit_skill.go (rcSkillFile, rcPacksFor, rcReviewSystemFor, rcSweepSystemFor), review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcStartSweep, rcMergeSweeps), review_commit.go (runReviewCommit integration, rcDefectsOnlyProse call site), review_commit_challenge.go (rcAssembleReview, rcFinalize, rcChallengeResult), review_commit_profile.go (rcStageFinder2/Sweep2/Rank, rcStageEffort), review_commit_repo.go (rcMaxLookups), review_commit_fast.go, all test files, all reviewskill/.md files, packs/.md files, build-kai-ci.yml, review-regressions.yml
Traced the integration path: rcRankSupported → rcApplyRank → rcFinalize → rcAssembleReview → res.Review → raw, and rcDefectsOnlyProse against the assembler's section order
Verified GitHub Actions concurrency group expression evaluation for all trigger paths (schedule, workflow_dispatch with record true/false, pin_pr set/empty)
Reviewed all files in the ensemble + review-skill change: review_commit_ensemble.go, review_commit_ground.go, review_commit_skill.go, review_commit_sweep.go, review_commit_profile.go, review_commit_repo.go, review_commit.go, review_commit_challenge.go, review_commit_fast.go, all test files, all reviewskill/.md and packs/.md files, both GitHub workflows, and scripts/review_regressions_report.py. Traced the rcRankSupported to rcApplyRank to rcFinalize integration, rcDefectsOnlyProse to published-prose path, rcPacksFor pack selection, rcMergeSweeps, rcLocateNamedIssue, and the build-kai-ci candidate_ref flow.
cmd/kai/review_commit_ensemble.go (rcEnsembleModel, rcStartSecondFinder, rcAwaitSecondFinder, rcWithSecondFinder, rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcRankCap)
cmd/kai/review_commit.go (runReviewCommit, rcRunReviewAgent — ensemble integration, rcLocateNamedIssue call site, rcDefectsOnlyProse call site)
cmd/kai/review_commit_ground.go (rcLocateNamedIssue)
cmd/kai/review_commit_profile.go (rcStageFinder2, rcStageSweep2, rcStageRank, rcStageModel, rcStageEffort, rcProfileStages)
cmd/kai/review_commit_repo.go (rcMaxLookups)
cmd/kai/review_commit_skill.go (rcReviewSystemFor, rcSweepSystemFor, rcPacksFor, rcLangOf, rcComposeStage)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcMergeSweeps, rcStartSweep)
.github/workflows/build-kai-ci.yml (candidate_kai_cli_ref)
.github/workflows/review-regressions.yml (record toggle, concurrency group)
cmd/kai/reviewskill/.md and packs/.md (all skill files)
all test files: review_commit_ensemble_test.go, review_commit_rules_test.go, review_commit_skill_test.go, review_commit_duplicates_test.go, review_commit_noise_test.go, review_commit_fast_test.go, review_commit_speculation_test.go, review_commit_ground_test.go
cmd/kai/review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcRankCap)
cmd/kai/review_commit.go (rcRunReviewAgent integration: rcRankSupported call site, rcWithSecondFinder, rcDefectsOnlyProse call site)
cmd/kai/review_commit_ground.go (rcLocateNamedIssue, rcNamedLocationRe)
cmd/kai/review_commit_challenge.go (rcAssembleReview, rcChallengeResult, rcAllegationResult, rcFinalize)
cmd/kai/review_commit_batches.go (rcFinalize)
cmd/kai/review_commit_skill.go (rcReviewSystemFor, rcSweepSystemFor, rcComposeStage, rcPacksFor)
cmd/kai/review_commit_profile.go (rcStageEffort, rcStageFinder2/Sweep2/Rank)
cmd/kai/review_commit_ensemble_test.go (all ensemble tests)
cmd/kai/review_commit_rules_test.go, review_commit_skill_test.go, review_commit_noise_test.go, review_commit_fast_test.go, review_commit_ground_test.go, review_commit_duplicates_test.go, review_commit_speculation_test.go
cmd/kai/reviewskill/.md and reviewskill/packs/.md
.github/workflows/build-kai-ci.yml and review-regressions.yml
cmd/kai/review_commit_skill.go (rcComposeStage, rcReviewSystemFor, rcFastReviewSystem, rcSweepSystemFor, rcPacksFor, rcLangOf, rcWords, rcPackText, rcStageFile, rcSkillFile)
cmd/kai/reviewskill/fast.md (file structure: head, shared marker, tail)
cmd/kai/reviewskill/defect.md
cmd/kai/reviewskill/catalog.md
cmd/kai/reviewskill/procedure.md
cmd/kai/reviewskill/report.md
cmd/kai/reviewskill/sweep.md
cmd/kai/reviewskill/challenge.md
cmd/kai/review_commit_skill_test.go (TestReviewSkillFilesCompose, TestReviewSkillDoesNotShout, TestPacksFollowWhatTheChangeTouches)
cmd/kai/review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcWithSecondFinder, rcAwaitSecondFinder, rcStartSecondFinder, rcRankCap, rcRankSystem)
cmd/kai/review_commit_ensemble_test.go (all tests)
cmd/kai/review_commit_profile.go (rcStageFinder2, rcStageSweep2, rcStageRank, rcStageEffort, rcProfileStages)
cmd/kai/review_commit.go (rcRankSupported call site, rcDefectsOnlyProse call site, rcLocateNamedIssue call site, quick pass retirement)
cmd/kai/review_commit_challenge.go (rcChallengeSystemHead, rcChallengeSystemTail, rcFinalize, rcAllegationResult, rcStatusSupported, rcStatusRefuted, rcAssembleReview)
cmd/kai/review_commit_batches.go (rcFinalize, rcAssembleReview)
cmd/kai/review_commit_verdicts.go (rcFinalize caller)
cmd/kai/review_commit_rules_test.go (TestReviewSystemPrompt_KeepsTheHardWonRules, TestReviewCatalogShowsFindingsNextToNonFindings, TestPromptsCarryNoBenchmarkCases)
cmd/kai/review_commit_noise_test.go, review_commit_fast_test.go, review_commit_duplicates_test.go, review_commit_speculation_test.go, review_commit_ground_test.go
.github/workflows/review-regressions.yml (record toggle, concurrency group, ad-hoc report step)
cmd/kai/review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcStartSecondFinder, rcAwaitSecondFinder, rcWithSecondFinder, rcMergeSweeps)
cmd/kai/review_commit_skill.go (rcPacksFor, rcLangOf, rcReviewSystemFor, rcSweepSystemFor, rcComposeStage)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcStartSweep)
cmd/kai/review_commit_profile.go (rcStageFinder2, rcStageSweep2, rcStageRank, rcStageEffort)
cmd/kai/review_commit.go (rcRunReviewAgent integration, rcDefectsOnlyProse call site)
cmd/kai/reviewskill/.md and reviewskill/packs/.md (embedded skill files)
all test files: review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_noise_test.go, review_commit_duplicates_test.go, review_commit_fast_test.go, review_commit_ground_test.go, review_commit_speculation_test.go
cmd/kai/review_commit_skill.go (rcPacksFor, top closure, rcLangOf, pack selection)
cmd/kai/review_commit_skill_test.go (TestPacksFollowWhatTheChangeTouches)
All other files in the change were examined in the draft review and found to have no reachable defects: review_commit_ensemble.go, review_commit_ground.go, review_commit_sweep.go, review_commit_profile.go, review_commit_repo.go, review_commit_fast.go, the two workflow files, and every reviewskill/.md and reviewskill/packs/.md file
cmd/kai/review_commit_ensemble.go (rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcStartSecondFinder, rcAwaitSecondFinder, rcWithSecondFinder, rcRankCap)
cmd/kai/review_commit_batches.go (rcFinalize, rcAssembleReview integration)
cmd/kai/review_commit_challenge.go (rcAssembleReview, rcAllegationResult, rcChallengeResult, rcChallengeMaxTurns)
cmd/kai/review_commit_skill.go (skill composition, rcPacksFor)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcMergeSweeps, second sweep)
cmd/kai/review_commit.go (rcDefectsOnlyProse call site, rcRankSupported call site, quick pass retirement)
reviewskill/.md and reviewskill/packs/.md (embedded prompt files)
All test files: review_commit_ensemble_test.go, review_commit_rules_test.go, review_commit_skill_test.go, review_commit_noise_test.go, review_commit_fast_test.go, review_commit_ground_test.go, review_commit_duplicates_test.go, review_commit_speculation_test.go
cmd/kai/review_commit_ensemble.go (full file, rcRankSupported, rcApplyRank, rcParseRank, rcDefectsOnlyProse, rcStartSecondFinder, rcAwaitSecondFinder, rcWithSecondFinder)
cmd/kai/review_commit_skill.go (full file, all composition functions)
cmd/kai/review_commit_sweep.go (rcRunSweep, rcRunSweepWith, rcStartSweep, rcMergeSweeps)
cmd/kai/review_commit_challenge.go (rcValidateChallenge, rcAssembleReview, rcAllegationResult, rcChallengeResult, rcFinalize)
cmd/kai/review_commit_profile.go (rcStageEffort, rcEnsembleEffort integration)
cmd/kai/review_commit.go (runReviewCommit integration, rcDefectsOnlyProse call site)
cmd/kai/review_commit_verdicts.go (rcDemoteVerdicts, rcFinalize)
All reviewskill/.md and reviewskill/packs/.md files
All test files: review_commit_ensemble_test.go, review_commit_skill_test.go, review_commit_rules_test.go, review_commit_noise_test.go, review_commit_duplicates_test.go, review_commit_fast_test.go, review_commit_ground_test.go, review_commit_speculation_test.go

Findings

cmd/kai/review_commit_sweep.go:569 — `rcMergeSweeps` replaces `out.Failed` with `b.Failed` only when `a.Failed > 0 && b.Failed < a.Failed`, so when the second sweep has more failures than the first (`b.Failed > a.Failed`), the merged `Failed` count silently drops the second sweep's failures, misreporting the number of failed chunks.

rcMergeSweeps only takes the second sweep's Failed count when the first sweep had failures AND the second had fewer. When the second sweep has more chunk failures than the first, or when the first had zero failures, the merged result keeps the first sweep's lower count, silently dropping the second sweep's failures from the reported total.

Remedy: Take the sum or the maximum of the two Failed counts, e.g. out.Failed = a.Failed + b.Failed, or at minimum if b.Failed > out.Failed { out.Failed = b.Failed } to report the worst case.

cmd/kai/review_commit_sweep.go:571 — `rcMergeSweeps` sets `out.Sources = b.Sources` only when `len(out.Sources) == 0`, but when both sweeps produce sources the second sweep's distinct sources are dropped, since the function never unions the two source maps (issues from the second sweep that are not in the first lose their chunk sources for grounding).

rcMergeSweeps keeps only the first sweep's Sources when both sweeps have Sources, dropping the second sweep's chunk sources. Issues proposed only by the second sweep then lose the diff-text sources the challenge gate needs to ground them, because those sources were in the dropped set.

Remedy: Union the two source slices: out.Sources = append(a.Sources, b.Sources...) instead of the conditional replacement.

Could not verify

The check could neither confirm nor rule these out, for the reason given. They are not findings and no fix is proposed; worth a look:

cmd/kai/review_commit_ensemble.go:165 — rcRankSupported returns early when len(supported) < 2, but a change with exactly one confirmed defect still needs the cap applied; though the cap defaults to ≥3, a review with one confirmed defect is left unranked — the doc comment at line 156 says "A review with a confirmed defect always publishes at least one," which this path satisfies by skipping ranking, but the early return means rcFinalize is never called for the single-defect case, potentially leaving un-finalized allegations. — the check for this allegation's batch did not complete (invalid challenge JSON: invalid character 'N' looking for beginning of value)

Limitations

Could not verify the external model provider endpoints (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) exist or accept the request shapes — these are external services outside the repository
Did not execute any code; all analysis is from reading the sources
Did not exhaustively read every packs/*.md file's content for correctness, only confirmed their existence and that the composition mechanism includes them
Could not read external model provider endpoints (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) to confirm they exist. Could not run the test suite. The review-regressions.sh script was not read in full, only the Python report script.
Could not confirm the external model endpoints named in the ensemble defaults (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) exist or accept the request shapes.
Did not execute any code; all analysis is from reading the sources.
Did not exhaustively trace rcAssembleReview's section ordering beyond the fragments visible in the challenge/batches files.
Could not read the model endpoint identifiers (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) — external providers, not in the repo.
Could not run the Go tests or the GitHub Actions workflows to confirm runtime behavior.
The review_commit_ensemble.go file was read at the current HEAD (2af5607), which may differ from the reviewed commit 66f353b by the time of this review; line numbers cited from SOURCE 3 use the file's own line numbers at that commit.
Could not trace the definition of changed in rcRunReviewAgent's parameter list beyond the diff and the call site, but the function signature and rcRankCap's parameter naming establish that len(changed) is a file count, not a line count.
Could not verify the external model identifiers in rcEnsembleDefaults (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) — external providers not in the repository
Did not run any code; all analysis is from reading the sources
Did not examine the reviewskill/packs/*.md file contents beyond confirming they exist (10 files) and that tests assert they reach the prompts
Did not trace rcAssembleReview's full implementation to confirm section ordering empirically — relied on the test and the function's call from rcFinalize
Could not verify the external model provider identifiers (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) exist or accept the request shapes used
Did not run the test suite; all test-behavior claims are reasoned from source text
Could not read the full rcAssembleReview implementation to confirm its exact section ordering relative to rcDefectsOnlyProse, but traced rcFinalize's call to it via grep results
Could not confirm the external model endpoints named in the ensemble defaults (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) exist or accept the request shapes — these are external providers outside the repository
Did not independently re-verify the draft's self-retracted allegations about the rank JSON regex or the regression workflow concurrency group — those were retracted by the draft itself and the retraction reasoning is sound
Could not confirm whether the model identifiers openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5 are valid provider endpoints reachable by orgs running this branch — these are external services outside the repository.
Did not exhaustively verify every string that the old tests checked for in the old prompts is still present in the new skill-file-composed prompts beyond the specific assertions in the updated test files.
Did not run the test suite; all test-pass/fail claims are based on reading the test code and the code under test.
Could not verify the external model identifiers (openai/gpt-5.6-sol, z-ai/glm-5.2, anthropic/claude-opus-5.5) exist or accept the request shapes — external providers not in the repository
Did not run the test suite — no code execution in this review
Did not examine rcSweepIssues, rcSweepChunks, rcSweepPatches, or rcDraftWithSweep implementations in detail
Did not examine the packs/*.md content for correctness beyond confirming they exist and are embedded

Decisions (need your call)

The ensemble defaults ship openai/gpt-5.6-sol, z-ai/glm-5.2, and anthropic/claude-opus-5.5 as per-review model defaults; every org running this candidate branch pays the per-review cost of those calls, and if any provider is unreachable the relevant stage degrades silently to the single-finder path.

_{+1577 −270 · 36 files · reaches 24 · the full analysis}
_{💬 Reply to any of my comments and I'll answer, or say @kaicontext anywhere on this PR — a question, or "take another look at the retry logic".}

kaicontext · 2026-09-30T17:18:42Z

+	if a.Failed > 0 && b.Failed < a.Failed {
+		out.Failed = b.Failed
+	}
+	return out


🐞 rcMergeSweeps replaces out.Failed with b.Failed only when a.Failed > 0 && b.Failed < a.Failed, so when the second sweep has more failures than the first (b.Failed > a.Failed), the merged Failed count silently drops the second sweep's failures, misreporting the number of failed chunks.

kaicontext · 2026-09-30T17:18:42Z

+		out.Failed = b.Failed
+	}
+	return out
+}


🐞 rcMergeSweeps sets out.Sources = b.Sources only when len(out.Sources) == 0, but when both sweeps produce sources the second sweep's distinct sources are dropped, since the function never unions the two source maps (issues from the second sweep that are not in the first lose their chunk sources for grounding).

fatihacet added 6 commits September 30, 2026 07:41

fatihacet marked this pull request as ready for review September 30, 2026 15:45

kaicontext Bot reviewed Sep 30, 2026

View reviewed changes

Navigation Menu

Review: skill-based prompts and an ensemble pipeline (benchmark candidate, do not merge) - #157

Review: skill-based prompts and an ensemble pipeline (benchmark candidate, do not merge)#157
fatihacet wants to merge 7 commits into
mainfrom
feat/review-skills

fatihacet commented Sep 30, 2026

Uh oh!

fatihacet commented Sep 30, 2026

Uh oh!

kaicontext Bot commented Sep 30, 2026

Uh oh!

kaicontext Bot left a comment

Uh oh!

kaicontext Bot left a comment

Uh oh!

kaicontext Bot Sep 30, 2026

Uh oh!

kaicontext Bot Sep 30, 2026

Uh oh!

kaicontext Bot left a comment

Uh oh!

kaicontext Bot Sep 30, 2026

Uh oh!

kaicontext Bot Sep 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fatihacet commented Sep 30, 2026

What it changes

Results (Martian Code Review Bench, 50 PRs, F1 under the Opus / Sonnet / GPT-5.2 judges)

Uh oh!

fatihacet commented Sep 30, 2026

Uh oh!

kaicontext Bot commented Sep 30, 2026

Uh oh!

kaicontext Bot left a comment

Choose a reason for hiding this comment

Kai review · 🔵 Your call, then merge (4/5)

Scope

Limitations

Uh oh!

kaicontext Bot left a comment

Choose a reason for hiding this comment

Kai review · 🟡 Small fixes first (3/5)

Scope

Findings

cmd/kai/review_commit_sweep.go:506 — rcMergeSweeps compares b.Failed < a.Failed and sets out.Failed = b.Failed, replacing the larger failure count with the smaller instead of summing failures; this loses the count of failed chunks from the sweep with more failures.

Limitations

Decisions (need your call)

Uh oh!

kaicontext Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

kaicontext Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

kaicontext Bot left a comment

Choose a reason for hiding this comment

Kai review · 🟡 Small fixes first (3/5)

Scope

Findings

Could not verify

Limitations

Decisions (need your call)

Uh oh!

kaicontext Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

kaicontext Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cmd/kai/review_commit_sweep.go:506 — `rcMergeSweeps` compares `b.Failed < a.Failed` and sets `out.Failed = b.Failed`, replacing the larger failure count with the smaller instead of summing failures; this loses the count of failed chunks from the sweep with more failures.