Bump the BC-ALAgents engine pin to 1.38.6 - #855
Open
Wenjie Fan (gggdttt) wants to merge 1 commit into
Open
Conversation
Moves the pinned engine from e9d9249 (1.36.6) to 1dbb15f (1.38.6). The BCQuality revision the engine declares is unchanged (182180913e6c90b62e6c2a65f13371e53c7adb5e, content version 1.6), so the knowledge corpus under evaluation is identical and the pin chain BC-Bench -> BC-ALAgents -> BCQuality stays consistent. The bump picks up BC-ALAgents #64, which deletes the .good.al / .bad.al samples belonging to a knowledge article that the BCQuality filter removed. Under this repository's filter configuration (enabled-layers: microsoft, knowledge.allow: microsoft/knowledge/**) the community layer is filtered out, which at the pinned BCQuality revision left 52 orphaned .al files in the clone the Copilot CLI uses as its working directory, 26 of them .bad.al. A .bad.al without its article is an anti-pattern sample stripped of the prose that identifies it as one. Also picks up BC-ALAgents #63, which forwards PATHEXT to the Copilot child environment. That fixes a Windows-only failure and does not affect this repository, where code-review evaluates on ubuntu-latest. Minor version bump per CONTRIBUTING.md: a tooling update that may affect results.
Sun Haoran (haoranpb)
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the pinned review engine from
e9d9249(1.36.6) to1dbb15f(1.38.6).Knowledge corpus is unchanged
Both engine commits declare the same BCQuality revision in
agents/ALReviewAgent/bcquality.config.yaml—182180913e6c90b62e6c2a65f13371e53c7adb5e, content version1.6. The knowledge under evaluation is therefore identical, and theBC-Bench -> BC-ALAgents -> BCQualitychain stays consistent: the revision a run executes is still exactly the one its engine pin declares.What the bump picks up
BC-ALAgents #64 — orphaned knowledge samples are now removed with their article.
Invoke-BCQualityFilter.ps1deletes knowledge articles that fall outsideenabled-layers/knowledge.allow, but previously left the sibling<slug>.good.al/<slug>.bad.alsamples behind. This repository evaluates withenabled-layers: microsoftandknowledge.allow: microsoft/knowledge/**, so the community layer is filtered out. Measured at the pinned BCQuality revision, that left 52 orphaned.alfiles in the clone the Copilot CLI uses as its working directory, 26 of them.bad.al— anti-pattern samples stripped of the prose that identifies them as anti-patterns.BC-ALAgents #63 —
PATHEXTis forwarded to the Copilot child environment.New-CopilotChildEnvironmentclears the child environment and re-adds an allow-list that omittedPATHEXT, so on Windows the child could not resolvegittogit.exeand every git call returned empty output instead of an error. This does not affect this repository:EvaluationCategory.CODE_REVIEW.runnerisubuntu-latest, wherePATHEXTdoes not exist. Included because it comes with the range, not as a fix for anything here.Version
0.10.0 -> 0.11.0. PerCONTRIBUTING.md, a tooling update that may affect results is a minor bump.Validation
uv lock --check— consistentuv run ruff format --check/ruff check— cleanuv run ty check . --ignore=unresolved-import --exclude "notebooks/"— 1 diagnostic, pre-existing onmain(src/bcbench/redteam.py:150, unusedty: ignore); verified by re-running with the change stasheduv run pytest -q -m "not e2e"— 874 passed, 2 skippeduv run pytest -q -m e2eagainst a cleanmicrosoft/BCQualitymaincheckout — 1 passedgit diff --stat— 4 files, 1 line each, no formatter or line-ending noise