feat(router): make search policy auditable and fail closed on capability conflict - #46
Conversation
…ity conflict `requiresSearch` was already authoritative over the TaskType default, but three gaps let the routing decision misrepresent what actually happened. 1. Search + vision silently lost a capability. `resolveRoute` tested the search policy before the vision branch, so a visual task carrying images with `requiresSearch: true` routed to Perplexity — which has no multimodal transport — and the images were dropped with no signal to the caller. That combination now throws `UnsupportedCapabilityCombinationError` (code `UNSUPPORTED_CAPABILITY_COMBINATION`) from route resolution, before request identity, budget reservation, circuit permit, or provider dispatch, so neither capability can disappear and neither budget nor circuit state is touched. 2. The Perplexity config could contradict its own decision. `disableSearch: requiresSearch === false && !isSearchTask(type)` was unreachable on a resolved search route and, off-route, produced a search-provider config with web search turned off. A Perplexity config is only ever built for a search route, so `disableSearch` is now always false, and dispatch asserts both that the config has search on and that `decision.searchRequired` equals `provider === PERPLEXITY` in both directions. 3. A decision could not be audited. `RoutingResolution` now carries `searchRequired` and `searchPolicySource` (EXPLICIT | TASK_DEFAULT), so the call log proves whether search was chosen by the caller or by the TaskType default instead of leaving it to be inferred from model names. The rule itself now has exactly one implementation: `resolveSearchPolicy()` returns both the answer and its authority; `requiresSearchProvider()` is a boolean view of it and `isSearchTask()` supplies only the TaskType default. Behaviour that did not change: TaskType defaults when the flag is omitted, the model matrices, vision model selection for a given image count, budget reservation and reconciliation ordering, circuit classification, the bounded fallback chain, image safety validation, and the deprecated 1.x provider subpath exports. Consensus stays an execution modifier — it never pulls a task onto the search plane. Tests: 161 (was 100). Adds the full routing matrix as a table-driven suite, audit-evidence assertions, provider-dispatch proofs with injected clients, capability-conflict proofs, config/decision consistency proofs, and non-regression guards for budget, circuits, image safety, and vision model selection. Version 1.2.0: additive public API (SearchPolicySource, SearchPolicyResolution, resolveSearchPolicy, UnsupportedCapabilityCombinationError, two new RoutingResolution fields) plus a fail-closed correction, within the 1.x line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018YFsaV8KzRmmAkBUHxfE9z
Removing the `!isSearchTask(task.type)` term from the `disableSearch` predicate left the import in `perplexity-matrix.ts` serving nothing but the backward-compatibility re-export. Collapse it to `export … from` and say in the comment why nothing in this module calls it any more: the search decision is settled before a Perplexity config is ever resolved. No behaviour or export-surface change — `isSearchTask` remains importable from both `./matrices/perplexity-matrix.js` and the package root. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018YFsaV8KzRmmAkBUHxfE9z
Campaign 7-ROUTER — explicit search policy + router integrityBaseline:
Verified defects
Implementation
Routing matrix (
|
| Case | requiresSearch |
Expected | Actual |
|---|---|---|---|
A/B/C STRATEGIC_REASONING |
false / true / omitted | NON_SEARCH / SEARCH / NON_SEARCH | ✅ |
D/E/F COMPETITOR_RESEARCH |
true / false / omitted | SEARCH / NON_SEARCH / SEARCH | ✅ |
G/H FACT_VERIFICATION |
false / omitted | NON_SEARCH / SEARCH | ✅ |
I/J CONTENT_GENERATION |
false / true | NON_SEARCH / SEARCH | ✅ |
K/L SCREENSHOT_ANALYSIS + images |
false / omitted | VISION | ✅ |
M SCREENSHOT_ANALYSIS + images |
true | fail closed | ✅ throws |
N/O EXTRACTION, CLASSIFICATION |
true | SEARCH | ✅ |
P/Q MARKET_RESEARCH |
false / omitted | NON_SEARCH / SEARCH | ✅ |
Every non-fail-closed case also asserts searchPolicySource (EXPLICIT for a boolean, TASK_DEFAULT for omitted) and that searchRequired matches the plane actually selected.
Non-regression proved, not assumed
Budget reserved before dispatch on both planes and reconciled to actual cost; failures charged to the circuit of the provider the policy selected (search failure opens Perplexity only, general failure opens OpenRouter only); the capability refusal reserves nothing and opens nothing; image safety validation still precedes every routing outcome; consensus stays an execution modifier and never pulls a task onto the search plane; vision model selection is byte-identical for every (image count × complexity) pair; TaskDescriptor still exposes capability fields only — application-supplied provider/model keys are stripped and the router keeps its choice.
Tests: 100 → 161. The package export map is unchanged: no 1.x provider subpath removed, none added.
Consumer compatibility (read-only; neither repo modified)
The real policy modules from each consumer — Website-Bot@3eb3536 src/intelligence/improve-llm-policy.ts and SEO-Bot@9691264 src/services/improve-llm-policy.ts — were copied unmodified into disposable ephemeral projects, compiled against the packed 1.2.0 tarball under each consumer's own strict compilerOptions, and executed:
- Website-Bot — PASS.
assertWebsiteImprovePolicy()holds. All 5 operations stay off the search plane;VISUAL_PATTERN_ANALYSISandVISUAL_DELTA_ANALYSISstill resolve to vision. - SEO-Bot — PASS.
assertSeoImprovePolicy()holds. The 4 reasoning operations stay non-search;FRESH_WEB_EVIDENCE→perplexity/sonar-pro. The inlinerequiresSearch: truehelpers (COMPETITOR_RESEARCH,CITATION_CHECK,MARKET_RESEARCH) stay on search, andanalyzeScreenshot(LAYOUT_VALIDATION+ images, flag omitted) stays on vision withsearchPolicySource = TASK_DEFAULT.
Neither consumer combines vision with requiresSearch: true, so the new fail-closed path is unreachable from their current call sites.
Version
1.1.3 → 1.2.0. Additive public API (SearchPolicySource, SearchPolicyResolution, resolveSearchPolicy, UnsupportedCapabilityCombinationError, two RoutingResolution fields) plus a fail-closed correction — a minor bump within 1.x. Both consumers pin 1.1.2 exactly, so neither auto-adopts; adoption is each repo's own decision.
Not proven here — please note before tagging
- Real-provider smoke was not run. No
PERPLEXITY_API_KEY/OPENROUTER_API_KEYis reachable from the execution sandbox. Status isREAL_PROVIDER_SMOKE_BLOCKED, not PASS. - Registry state for
1.2.0was not verified from the sandbox (noread:packagesscope). Confirm1.2.0is unpublished before pushing av1.2.0tag — published versions are immutable.
npm run verify:package — the isolated tarball install and export smoke I could not run locally — passed in CI on Node 20.19.0, 22.23.1, and 24.18.0.
Generated by Claude Code
The governed agent toolchain writes phase locks, memory receipts, L4 phase/release receipts, and PR handoffs under `.l9/`. They are machine- and session-local — they carry session IDs and absolute paths — so they are not source and must not be committed. Every other repo in the constellation already ignores this tree (Cursor-Governance and Website-Bot ignore `.l9/` wholesale; l9-ci-core ignores `.l9/runtime/` and `.l9/autonomy/`). LLM-Router was the outlier, which left the files showing up as untracked after any governed run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018YFsaV8KzRmmAkBUHxfE9z
|



Summary
Type of Change
Governance Checklist
setup_workspace_symlinks.sh, symlinks resolve (§2)ls -la .cursor/rules .cursor/skills .cursor/commandsall resolve@v1, never@mainor bare SHABreaking Change
If checked, describe the impact and migration path:
Rollback Plan
Related Issues
Closes #
Commits
Test plan
make pr-check(local changed-files gate) PASS before openGenerated by Claude Code