Let a pr-review run target a candidate BCQuality revision

[Wenjie Fan (gggdttt)]

Why

Prepare-BCQualityRoot.ps1 resolves the BCQuality repository and ref solely from the pinned engine's own bcquality.config.yaml. A run can therefore only ever exercise whatever BCQuality content that engine pin happens to point at, which makes it impossible to answer the question the offline eval gate needs answered: does this proposed BCQuality change move the score?

There are two independent pins involved, and it is worth being explicit about which one this touches:

BC-Bench .github/actions/install-agent-harnesses/action.yml
  └─ ref: 533dd39d…                          <- pin 1: the engine. BC-Bench owns it.
       └─ BC-ALAgents/agents/ALReviewAgent/bcquality.config.yaml
            └─ ref: 18218091…                <- pin 2: BCQuality. Transitive.
                 └─ read at run time by this script.

The gate needs pin 2. Pin 1 stays exactly where it is: it is the reproducibility anchor and must not move per run. An earlier attempt at this went after pin 1 instead and was withdrawn.

What changed

Prepare-BCQualityRoot.ps1 takes two new optional parameters, -CandidateRepo and -CandidateRef, which redirect the fetch. Omitting both reproduces today's behaviour exactly.

The script also now reports the baseline coordinates, the coordinates it actually used, and the resolved commit:

root=<path>
baseline-repo=… baseline-ref=…
resolved-repo=… resolved-ref=… resolved-commit=…

A ref is mutable — refs/pull/N/head moves on every force-push — so recording the commit is what makes a baseline run and a candidate run comparable after the fact. root= is still emitted first and is unchanged, so the existing caller keeps working.

A candidate cannot widen its own filter

The filter still runs from the engine's configuration, not the candidate's:

& $filter -BCQualityRoot $Root -Config $cfg   # $cfg is the engine's, always

So a candidate revision cannot edit its own enabled-layers or knowledge.allow to smuggle in content the filter would otherwise strip. This is a security property of the ordering, not an implementation detail, and it is asserted by a test.

Verification

Both modes were run end to end against a real engine checkout.

Mode	Result
Baseline (no candidate arguments)	Identical to today. Filter removed 27 files. resolved-* equals baseline-*.
Candidate (-CandidateRef refs/pull/152/head)	Fetch succeeded, filter removed 51 files, and resolved-commit=eaef922a8a1cfbe17fa14d670e089966dc27b2d1 matches gh pr view 152 --json headRefOid exactly.

The candidate run also demonstrates the filter property above rather than merely asserting it. BCQuality PR 152 adds an article under community/knowledge/events/, and the engine baseline enables only the microsoft layer — the article was correctly removed with reason=layer-disabled.

ruff format, ruff check, ty check and pytest are all clean (877 passed, 2 skipped). The one ty diagnostic that appears is present on an unmodified origin/main as well.

Scope

Nothing wires the new parameters up yet — this change is inert until a caller passes them. That caller change belongs in agent.py alongside the experiment-configuration work in #851 and lands after it, so this PR deliberately touches neither agent.py nor action.yml and does not conflict with #851.

[Wenjie Fan (gggdttt)]

Closing this. It is unnecessary, and it works against a design decision this repository has already written down.

The capability it adds already exists. --engine-path resolves the engine root (agent.py L161), that root is handed to Prepare-BCQualityRoot.ps1 as -EngineRoot (L91), and the script reads the BCQuality repo and ref from that checkout's own Get-BCQualityConfig.ps1 (L22-23). Pointing a run at a candidate BCQuality revision is therefore just: check out BC-ALAgents, set ref in its bcquality.config.yaml, and pass --engine-path at it. No code change is involved.

More importantly, docs/code-review.md L44 already prescribes the experiment workflow -- push the BCQuality change through a BC-ALAgents branch, move the action pin to that immutable commit, and run BC-Bench from the corresponding commit -- explicitly so that the reproducible dependency chain BC-Bench -> BC-ALAgents -> BCQuality holds. The value of that chain is that the BCQuality revision a run executed is always exactly the one the engine pin declares. -CandidateRef introduces a path where those two disagree, which is the property the documented workflow exists to protect.

My reasoning error is worth recording. The two-pin analysis behind this PR was accurate as a description of the mechanism, but I concluded from it that pin 2 needed an override. Pin 2 is already controllable, because the engine checkout that declares it is controllable. I mistook a layered mechanism for a missing switch.

The underlying goal is unaffected. What an offline eval gate for BCQuality pull requests actually lacks is orchestration -- deriving the engine branch from a BCQuality PR, running the baseline and candidate arms, and reporting the delta. That belongs in BC-ALAgentsInternal and needs no change here.