diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 5042a29..804f44d 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -7,7 +7,8 @@ ```text validated execution task -> effective image set merged into task - -> pure route resolution + -> capability resolution + fail-closed validation + -> pure route resolution (single decision) -> request identity and timestamp -> atomic process-local budget reservation -> provider-family-safe downgrade @@ -19,9 +20,20 @@ validated execution task Route resolution is pure. Request IDs and timestamps are added afterward and do not participate in routing equivalence. -## Search policy authority +## Capability authority -The application declares the capability; the router selects the provider. That split is enforced by a single resolver, `resolveSearchPolicy()` in `src/matrices/search-policy.ts`: +The application declares the capability; the router selects the provider. One authority chain turns a `TaskDescriptor` into one decision that both routing and dispatch consume: + +```text +TaskDescriptor + -> resolveCapabilities (search, source, vision, images) + -> validateCapabilities (fail closed on unsupported combinations) + -> resolve provider/model + -> reserve budget + -> dispatch EXACT resolved capability +``` + +`resolveSearchPolicy()` in `src/matrices/search-policy.ts` is the single implementation of the search rule: ```text typeof task.requiresSearch === 'boolean' @@ -30,21 +42,32 @@ otherwise -> { required: isSearchTask(task.type), source: TASK_DEFAULT } ``` -There is exactly one implementation of this rule. `requiresSearchProvider()` is a boolean view of it and `isSearchTask()` supplies only the `TaskType` default. `resolveRoute()` consumes the resolution and copies `searchRequired` and `searchPolicySource` onto every `RoutingResolution`, so a decision is auditable without inferring intent from model names. +`requiresSearchProvider()` is a boolean view of it and `isSearchTask()` supplies only the `TaskType` default. `resolveCapabilities()` composes the search policy with the canonical vision-task inventory (`VISION_TASKS`, owned by `search-policy.ts`) and the effective image set. `resolveRoute()` consumes the validated capabilities and copies `searchRequired`, `searchPolicySource`, and `visionRequired` onto every `RoutingResolution`; `dispatchProvider()` branches on the same decision fields and asserts the provider contract each branch requires. Dispatch never re-derives a plane from the raw task. + +Fail-closed validation refuses every combination the provider plane would silently drop, before any budget reservation or provider dispatch: + +| Combination | Error code | +| --- | --- | +| Vision task without images | `VISION_INPUT_REQUIRED` | +| Search + vision together | `UNSUPPORTED_CAPABILITY_COMBINATION` | +| Images on a non-vision task | `IMAGES_NOT_SUPPORTED_FOR_TASK` | +| `recency` / `domainFilter` without search | `SEARCH_MODIFIER_WITHOUT_SEARCH` | +| `consensus` on a non-search route | `CONSENSUS_REQUIRES_SEARCH` | -Two invariants keep the audit honest: +Three invariants keep the audit honest: -- A resolved Perplexity config always has `disableSearch: false`. A search decision can never dispatch a config with web search turned off. +- A resolved Perplexity config always has `disableSearch: false`, and `resolvePerplexityConfig()` refuses non-search tasks. A search decision can never dispatch a config with web search turned off. - Before dispatch, `decision.searchRequired` must equal `decision.provider === Provider.PERPLEXITY`. A disagreement in either direction is a hard error, not a downgrade. +- Failed routed calls are auditable: `RoutingDecision` records `outcome`, `failureKind`, and `errorCode` on failure, without ever logging prompts, keys, or image contents. -Search and vision have no combined provider contract. A vision `TaskType` carrying images with `requiresSearch: true` throws `UnsupportedCapabilityCombinationError` from route resolution — before request identity, budget reservation, circuit permit, or provider dispatch — so neither capability is silently discarded. Because the throw precedes reservation and permit acquisition, it cannot affect budget state or provider circuit health. +Because every refusal precedes reservation and permit acquisition, none of them can affect budget state or provider circuit health. ## Module ownership ```text src/types.ts public legacy contracts src/schemas.ts runtime validation for public legacy input -src/matrices/* deterministic model and search resolution +src/matrices/* deterministic model resolution plus capability authority src/pricing.ts canonical OpenRouter price table src/budget/* process-local admission and spend accounting src/circuit-breaker.ts process-local provider health control diff --git a/README.md b/README.md index 602814b..33d4595 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,7 @@ await router.execute( ### Auditing a routing decision -Every `RoutingDecision` — from `route()` and from `getCallLog()` — reports whether search was selected and on whose authority, alongside `taskType`, `complexity`, `provider`, `model`, `reason`, `estimatedCost`, `taskId`, `clientId`, `timestamp`, downgrade state, and (after execution) `actualCost` and `latencyMs`. No credentials or prompts are recorded. +Every `RoutingDecision` — from `route()` and from `getCallLog()` — reports whether search was selected and on whose authority (`searchRequired`, `searchPolicySource`), whether the route is vision-backed (`visionRequired`), and, after execution, the call outcome: `actualCost`, `latencyMs`, and for failed routed calls `outcome: 'FAILED'` with `failureKind` and `errorCode`. The same decision object drives dispatch, so the audit can never disagree with what actually executed. No credentials, prompts, or image contents are recorded. ```ts const decision = router.route({ clientId: 'tenant-a', type: TaskType.MARKET_RESEARCH, complexity: TaskComplexity.HIGH, requiresSearch: false }); @@ -127,12 +127,24 @@ decision.provider; // Provider.OPENROUTER ### Unsupported capability combinations -No provider in this router serves search and vision together. A visual task that supplies images *and* sets `requiresSearch: true` is refused with `UnsupportedCapabilityCombinationError` (code `UNSUPPORTED_CAPABILITY_COMBINATION`) before any budget reservation, circuit permit, or provider dispatch — rather than silently dropping the images or silently skipping the search. Split such work into a vision task and a search task. +The router fails closed on every capability combination the provider plane cannot execute faithfully — nothing is silently dropped: + +| Combination | Error code | +| --- | --- | +| Visual task without images | `VISION_INPUT_REQUIRED` | +| Search and vision together | `UNSUPPORTED_CAPABILITY_COMBINATION` | +| Images on a non-visual task | `IMAGES_NOT_SUPPORTED_FOR_TASK` | +| `recency` / `domainFilter` without search | `SEARCH_MODIFIER_WITHOUT_SEARCH` | +| `consensus` on a non-search route | `CONSENSUS_REQUIRES_SEARCH` | + +All of these throw `UnsupportedCapabilityCombinationError` before any budget reservation, circuit permit, or provider dispatch, so an invalid request never half-executes and never affects budget state or circuit health. Split search+vision work into a vision task and a search task. ## Vision execution Images supplied through execution options are merged into the validated task before routing. This ensures model selection and budget estimation use the same image count that reaches the provider. +A vision task without images fails closed with `VISION_INPUT_REQUIRED` instead of silently degrading to a text-only call, and images attached to a non-visual task fail with `IMAGES_NOT_SUPPORTED_FOR_TASK` instead of being ignored. + ```ts const result = await router.execute( { @@ -157,7 +169,7 @@ Only HTTPS public URLs and bounded `data:image/*;base64` payloads are accepted. For eligible high-complexity Perplexity tasks, `{ consensus: true }` executes the configured variations in parallel. The returned content is selected from the successful responses, while token and cost fields represent the aggregate successful consensus execution so budget reconciliation does not undercount spend. -Consensus is an execution modifier, not search-policy authority. It applies only to a route that already resolved to the search plane; on a general or vision route it is inert and never pulls the task onto a search provider. +Consensus is an execution modifier, not search-policy authority. It applies only to a route that already resolved to the search plane; requesting it on a general or vision route throws `CONSENSUS_REQUIRES_SEARCH` before budget reservation instead of being silently ignored. ## Budget semantics diff --git a/src/index.ts b/src/index.ts index 0e860b8..c21e8e7 100644 --- a/src/index.ts +++ b/src/index.ts @@ -10,7 +10,7 @@ import { import { CircuitBreaker, CircuitOpenError, type CircuitPermit } from './circuit-breaker.js'; import { resolveGeneralConfig, getFallbackChain } from './matrices/general-matrix.js'; import { resolvePerplexityConfig } from './matrices/perplexity-matrix.js'; -import { resolveSearchPolicy, UnsupportedCapabilityCombinationError } from './matrices/search-policy.js'; +import { resolveAndValidateCapabilities, UnsupportedCapabilityCombinationError } from './matrices/search-policy.js'; import { classifyProviderError, isCircuitFailure } from './provider-errors.js'; import { OpenRouterClient, validateImageUrl, type OpenRouterClientLike } from './providers/openrouter.js'; import { PerplexityClient, type PerplexityClientLike } from './providers/perplexity.js'; @@ -19,18 +19,16 @@ import { GeneralModel, Provider, SonarModel, - TaskType, type BudgetConfig, type LLMResponse, type RouterConfig, type RoutingDecision, type RoutingResolution, type TaskDescriptor, + type TaskType, } from './types.js'; import { generateFullSiteQAPlan, resolveVisionConfig, VIEWPORTS, type FullSiteQAConfig, type VisualQATask } from './vision/index.js'; -const VISION_TASKS = new Set([TaskType.VISUAL_QA, TaskType.SCREENSHOT_ANALYSIS, TaskType.LAYOUT_VALIDATION]); - export interface RouterDependencies { clock?: () => Date; idFactory?: () => string; @@ -41,26 +39,26 @@ export interface RouterDependencies { } export function resolveRoute(task: TaskDescriptor): RoutingResolution { - const policy = resolveSearchPolicy(task); - const audit = { taskType: task.type, complexity: task.complexity, searchRequired: policy.required, searchPolicySource: policy.source }; - const imageCount = task.images?.length ?? 0; - - // Fail closed before either capability can be silently discarded. The search - // plane has no multimodal transport, so a visual task carrying images cannot - // also be answered by web search. - if (policy.required && VISION_TASKS.has(task.type) && imageCount > 0) { - throw new UnsupportedCapabilityCombinationError( - `Task[${task.type}] supplied ${imageCount} image(s) and requires search, but no provider in this router serves search and vision together. Split the work into a vision task and a search task.`, - { taskType: task.type, searchRequired: true, imageCount }, - ); - } + // One capability decision for the whole call: routing and dispatch both + // consume this resolution, so a request can never be interpreted one way + // at routing time and another way at dispatch time. Validation refuses + // every combination the provider plane would silently drop. + const capabilities = resolveAndValidateCapabilities(task); + const audit = { + taskType: task.type, + complexity: task.complexity, + searchRequired: capabilities.searchRequired, + searchPolicySource: capabilities.searchPolicySource, + visionRequired: capabilities.visionRequired, + }; - if (policy.required) { + if (capabilities.searchRequired) { const config = resolvePerplexityConfig(task); return { ...audit, provider: Provider.PERPLEXITY, model: config.model, estimatedCost: config.estimatedCostPerCall, reason: config.resolutionReason }; } - if (VISION_TASKS.has(task.type)) { - const config = resolveVisionConfig(task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION, task.complexity, task.images?.length ?? 1); + if (capabilities.visionRequired) { + // Validation guarantees at least one image on a vision route. + const config = resolveVisionConfig(task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION, task.complexity, task.images!.length); return { ...audit, provider: Provider.OPENROUTER, model: config.model, estimatedCost: config.estimatedCostPerCall, reason: config.resolutionReason }; } const config = resolveGeneralConfig(task); @@ -125,6 +123,16 @@ export class L9LLMRouter { const images = task.images; if (images) for (const image of images) validateImageUrl(image); const decision = this.route(task); + // Consensus is a search execution modifier, not hidden routing authority: + // a non-search route would silently ignore it, so refuse the combination + // before any budget reservation. + if (options?.consensus && !decision.searchRequired) { + throw new UnsupportedCapabilityCombinationError( + 'Consensus requires a search-backed route', + undefined, + 'CONSENSUS_REQUIRES_SEARCH', + ); + } const governedMemory = await hydrateRouterPrompt(this.memory, decision.clientId, task.type, userPrompt); const effectiveSystemPrompt = governedMemory ? `${systemPrompt}${governedMemory}` : systemPrompt; @@ -149,6 +157,7 @@ export class L9LLMRouter { reservationId = undefined; decision.actualCost = response.cost; decision.latencyMs = response.latencyMs; + decision.outcome = 'SUCCESS'; this.callLog.push(decision); return response; } catch (error) { @@ -160,6 +169,14 @@ export class L9LLMRouter { if (isCircuitFailure(error, decision.provider)) this.circuitBreaker.recordFailure(permit, this.clock()); else this.circuitBreaker.release(permit, this.clock()); } + // Failed routed calls are auditable too: record the classified failure + // on the decision before rethrowing. Prompt, keys, and image contents + // never enter the call log. + const classified = classifyProviderError(error, decision.provider); + decision.outcome = 'FAILED'; + decision.failureKind = classified.kind; + decision.errorCode = classified.code ?? (error instanceof Error ? error.name : undefined); + this.callLog.push(decision); if (providerCompleted) throw error; throw this.toExecutionError(error, task, decision); } @@ -173,14 +190,16 @@ export class L9LLMRouter { images: string[] | undefined, options: { images?: string[]; assistantContext?: string; consensus?: boolean; signal?: AbortSignal } | undefined, ): Promise { - // The audited decision and the plane about to be dispatched must agree in - // both directions: a search decision may not execute on the general plane, - // and a non-search decision may not execute web search. Perplexity is the - // router's only search-capable provider. + // Dispatch consumes the resolved decision — it never re-derives the plane + // from the raw task. The audited decision and the plane about to be + // dispatched must agree in both directions: a search decision may not + // execute on the general plane, and a non-search decision may not execute + // web search. Perplexity is the router's only search-capable provider. if (decision.searchRequired !== (decision.provider === Provider.PERPLEXITY)) { throw new Error(`Routing decision searchRequired=${decision.searchRequired} disagrees with provider ${decision.provider}`); } - if (decision.provider === Provider.PERPLEXITY) { + if (decision.searchRequired) { + if (decision.provider !== Provider.PERPLEXITY) throw new Error('Search decision resolved a non-Perplexity provider'); const config = resolvePerplexityConfig(task); if (!Object.values(SonarModel).includes(decision.model as SonarModel)) throw new Error('Perplexity route resolved a non-Sonar model'); // A search route may never dispatch a config that turns search off. @@ -199,7 +218,9 @@ export class L9LLMRouter { } return this.perplexity.complete(config, effectiveSystemPrompt, userPrompt, options?.assistantContext, options?.signal); } - if (VISION_TASKS.has(task.type) && images?.length) { + if (decision.visionRequired) { + if (decision.provider !== Provider.OPENROUTER) throw new Error('Vision decision resolved a non-OpenRouter provider'); + if (!images || images.length === 0) throw new Error('Vision route dispatched without images'); const config = resolveVisionConfig(task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION, task.complexity, images.length); config.model = decision.model as GeneralModel; return this.openrouter.completeWithVision(config, effectiveSystemPrompt, userPrompt, images, options?.signal); @@ -274,8 +295,13 @@ export { isSearchTask, requiresSearchProvider, resolveSearchPolicy, + resolveCapabilities, + resolveAndValidateCapabilities, + validateCapabilities, + VISION_TASKS, UnsupportedCapabilityCombinationError, } from './matrices/search-policy.js'; +export type { ResolvedCapabilities, CapabilityConflictCode } from './matrices/search-policy.js'; export { hydrateRouterPrompt } from './memory.js'; export type { RouterMemoryConfig } from './memory.js'; diff --git a/src/matrices/perplexity-matrix.ts b/src/matrices/perplexity-matrix.ts index fc6280f..6d361f2 100644 --- a/src/matrices/perplexity-matrix.ts +++ b/src/matrices/perplexity-matrix.ts @@ -11,10 +11,9 @@ import { type TaskDescriptor, } from '../types.js'; // Re-exported for backward compatibility: `isSearchTask` historically lived in -// this module. Its canonical home is now ./search-policy.ts, and nothing here -// calls it any more — the search decision is made before a Perplexity config -// is ever resolved. +// this module. Its canonical home is now ./search-policy.ts. export { isSearchTask } from './search-policy.js'; +import { resolveSearchPolicy } from './search-policy.js'; function selectSonarModel(complexity: TaskComplexity, rank: number): SonarModel { if (complexity === TaskComplexity.CRITICAL) return SonarModel.SONAR_DEEP_RESEARCH; @@ -40,6 +39,12 @@ function selectReasoningEffort(model: SonarModel, complexity: TaskComplexity): ' } export function resolvePerplexityConfig(task: TaskDescriptor): PerplexityConfig { + // Provider config and routing authority must agree: a Perplexity config is + // only ever produced for a route that resolved to search. A non-search task + // reaching this resolver is a contract violation, not a configurable state. + if (!resolveSearchPolicy(task).required) { + throw new Error('resolvePerplexityConfig called for a non-search task'); + } const rank = complexityRank(task.complexity); const model = selectSonarModel(task.complexity, rank); const searchContextSize = selectSearchContextSize(rank); diff --git a/src/matrices/search-policy.ts b/src/matrices/search-policy.ts index f5cc68f..c1f91cf 100644 --- a/src/matrices/search-policy.ts +++ b/src/matrices/search-policy.ts @@ -15,6 +15,19 @@ const DEFAULT_SEARCH_TASKS = new Set([ TaskType.LINK_PROSPECTING, ]); +/** + * Task types whose *default* capability implies a vision-backed provider. + * + * This is the canonical vision-task inventory for the whole router: routing + * and dispatch both consume it through {@link resolveCapabilities}, so no + * other module may re-derive vision from a raw `TaskType`. + */ +export const VISION_TASKS = new Set([ + TaskType.VISUAL_QA, + TaskType.SCREENSHOT_ANALYSIS, + TaskType.LAYOUT_VALIDATION, +]); + /** * Backward-compatible task-type default. * @@ -62,28 +75,123 @@ export function requiresSearchProvider(task: TaskDescriptor): boolean { } /** - * Fail-closed error for a task that asks for two capabilities the router has no - * provider contract able to satisfy together. + * The single internal capability authority. + * + * Routing and dispatch both consume this resolution, so a request can never + * be interpreted one way at routing time and another way at dispatch time. + */ +export interface ResolvedCapabilities { + /** Whether the resolved route must carry web-search capability (explicit flag or task-type default). */ + searchRequired: boolean; + /** Where `searchRequired` came from: an explicit `TaskDescriptor.requiresSearch` boolean, or the legacy per-TaskType default. */ + searchPolicySource: SearchPolicySource; + /** Whether the task type implies a vision-backed provider. */ + visionRequired: boolean; + /** Whether the descriptor actually carries at least one image. */ + imagesProvided: boolean; +} + +export function resolveCapabilities(task: TaskDescriptor): ResolvedCapabilities { + const policy = resolveSearchPolicy(task); + return { + searchRequired: policy.required, + searchPolicySource: policy.source, + visionRequired: VISION_TASKS.has(task.type), + imagesProvided: Array.isArray(task.images) && task.images.length > 0, + }; +} + +/** + * Capability conflicts the current provider plane cannot execute faithfully. + * + * Every code names a combination that would otherwise degrade silently: + * + * UNSUPPORTED_CAPABILITY_COMBINATION — search + vision together (legacy code) + * VISION_INPUT_REQUIRED — vision task without images + * IMAGES_NOT_SUPPORTED_FOR_TASK — images on a non-vision task + * SEARCH_MODIFIER_WITHOUT_SEARCH — recency/domainFilter without search + * CONSENSUS_REQUIRES_SEARCH — consensus on a non-search route + */ +export type CapabilityConflictCode = + | 'UNSUPPORTED_CAPABILITY_COMBINATION' + | 'VISION_INPUT_REQUIRED' + | 'IMAGES_NOT_SUPPORTED_FOR_TASK' + | 'SEARCH_MODIFIER_WITHOUT_SEARCH' + | 'CONSENSUS_REQUIRES_SEARCH'; + +/** + * Fail-closed error for a task that asks for capabilities the router has no + * provider contract able to satisfy faithfully. * - * Raised today for `vision task type + images + requiresSearch === true`: the - * search plane (Perplexity Sonar) has no multimodal transport in this router, - * so honouring one capability necessarily discards the other. Dropping either - * silently would make the routing audit a lie, so the request is rejected - * before any reservation, circuit permit, or provider dispatch. + * Raised before any budget reservation, circuit permit, or provider dispatch, + * so an invalid request can never half-execute. This is a caller-side contract + * error, not a provider failure: it must never count against provider circuit + * health. * - * This is a caller-side contract error, not a provider failure: it must never - * count against provider circuit health. + * The original `(message, requested)` constructor shape from #46 is preserved; + * newer capability codes pass only `code` and omit `requested`. */ export class UnsupportedCapabilityCombinationError extends Error { - public readonly code = 'UNSUPPORTED_CAPABILITY_COMBINATION'; + public readonly code: CapabilityConflictCode; constructor( message: string, - public readonly requested: Readonly<{ taskType: TaskType; searchRequired: boolean; imageCount: number }>, + public readonly requested?: Readonly<{ taskType: TaskType; searchRequired: boolean; imageCount: number }>, + code: CapabilityConflictCode = 'UNSUPPORTED_CAPABILITY_COMBINATION', ) { super(message); this.name = 'UnsupportedCapabilityCombinationError'; + this.code = code; } toJSON(): Record { - return { name: this.name, code: this.code, message: this.message, requested: this.requested }; + return { + name: this.name, + code: this.code, + message: this.message, + ...(this.requested === undefined ? {} : { requested: this.requested }), + }; + } +} + +/** + * Refuses capability combinations the execution plane would silently drop. + * + * vision without images -> VISION_INPUT_REQUIRED + * search + vision -> UNSUPPORTED_CAPABILITY_COMBINATION (legacy shape) + * images on non-vision -> IMAGES_NOT_SUPPORTED_FOR_TASK + * + * Called inside route resolution before any provider/model selection, so an + * invalid request can never reserve budget or reach dispatch. + */ +export function validateCapabilities(capabilities: ResolvedCapabilities, task: TaskDescriptor): void { + if (capabilities.visionRequired && !capabilities.imagesProvided) { + throw new UnsupportedCapabilityCombinationError('Visual task requires at least one image', undefined, 'VISION_INPUT_REQUIRED'); + } + if (capabilities.searchRequired && capabilities.visionRequired) { + const imageCount = task.images?.length ?? 0; + throw new UnsupportedCapabilityCombinationError( + `Task[${task.type}] supplied ${imageCount} image(s) and requires search, but no provider in this router serves search and vision together. Split the work into a vision task and a search task.`, + { taskType: task.type, searchRequired: true, imageCount }, + ); + } + if (capabilities.imagesProvided && !capabilities.visionRequired) { + throw new UnsupportedCapabilityCombinationError(`Task ${task.type} does not consume images`, undefined, 'IMAGES_NOT_SUPPORTED_FOR_TASK'); + } +} + +/** + * Resolves capabilities and refuses every combination the provider plane + * cannot honor, including search-only modifiers on a non-search route. + * + * `recency` and `domainFilter` are search execution modifiers: a general + * route ignores them, so declaring them without search capability is an + * explicit contract error instead of silently ignored policy. + */ +export function resolveAndValidateCapabilities(task: TaskDescriptor): ResolvedCapabilities { + const capabilities = resolveCapabilities(task); + validateCapabilities(capabilities, task); + const hasSearchModifiers = task.recency !== undefined || (task.domainFilter?.length ?? 0) > 0; + if (!capabilities.searchRequired && hasSearchModifiers) { + throw new UnsupportedCapabilityCombinationError('Search modifiers require search capability', undefined, 'SEARCH_MODIFIER_WITHOUT_SEARCH'); } + return capabilities; } diff --git a/src/types.ts b/src/types.ts index b5154d7..f21fa7f 100644 --- a/src/types.ts +++ b/src/types.ts @@ -207,6 +207,8 @@ export interface RoutingResolution { searchRequired: boolean; /** Whether `searchRequired` came from the caller or from the TaskType default. */ searchPolicySource: SearchPolicySource; + /** Whether the task type implies a vision-backed provider. Dispatch consumes this instead of re-deriving vision from the task. */ + visionRequired: boolean; } export interface RoutingDecision extends RoutingResolution { @@ -217,6 +219,12 @@ export interface RoutingDecision extends RoutingResolution { timestamp: string; downgraded?: boolean; downgradedFrom?: GeneralModel | SonarModel; + /** Terminal state of the routed call. Set to SUCCESS on provider completion; FAILED when the call fails after route resolution. */ + outcome?: 'SUCCESS' | 'FAILED'; + /** Classification of a failed routed call. Never carries prompts, keys, or image contents. */ + failureKind?: ProviderFailureKind; + /** Provider error code, or the error name for local policy failures. */ + errorCode?: string; } export interface CircuitBreakerState { diff --git a/tests/capability-integrity.test.ts b/tests/capability-integrity.test.ts new file mode 100644 index 0000000..fc668bf --- /dev/null +++ b/tests/capability-integrity.test.ts @@ -0,0 +1,220 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { + GeneralModel, + Provider, + SearchPolicySource, + TaskComplexity, + TaskType, + type LLMResponse, + type RoutingDecision, +} from '../src/types.js'; +import { + BudgetReservationError, + L9LLMRouter, + UnsupportedCapabilityCombinationError, + resolveCapabilities, + resolveRoute, + type CapabilityConflictCode, +} from '../src/index.js'; +import { ProviderRequestError } from '../src/provider-errors.js'; + +const response: LLMResponse = { + content: 'ok', model: GeneralModel.GPT4O_MINI, provider: Provider.OPENROUTER, + inputTokens: 1, outputTokens: 1, totalTokens: 2, cost: 0.1, latencyMs: 5, cached: false, +}; + +function harness() { + const openrouterClient = { + complete: async () => response, + completeWithFallback: async () => response, + completeWithVision: async () => response, + }; + const perplexityClient = { + complete: async () => ({ ...response, model: GeneralModel.GPT4O_MINI, provider: Provider.PERPLEXITY }), + completeWithConsensus: async () => ({ + best: { ...response, model: GeneralModel.GPT4O_MINI, provider: Provider.PERPLEXITY }, + all: [], consensusScore: 1, + aggregate: { inputTokens: 1, outputTokens: 1, totalTokens: 2, cost: 0.1, latencyMs: 5, citations: [] }, + }), + }; + const router = new L9LLMRouter( + { perplexityApiKey: 'p', openrouterApiKey: 'o' }, + { openrouterClient, perplexityClient, idFactory: () => 'task-1', clock: () => new Date('2026-01-01T00:00:00Z') }, + ); + router.initClient('c'); + return router; +} + +async function caughtCode(promise: Promise): Promise { + try { + await promise; + return undefined; + } catch (error) { + expect(error).toBeInstanceOf(UnsupportedCapabilityCombinationError); + return (error as UnsupportedCapabilityCombinationError).code; + } +} + +describe('capability resolver is the single internal authority', () => { + it('resolves search policy source and vision from one place', () => { + const caps = resolveCapabilities({ type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, images: ['https://cdn.example.com/a.png'] }); + expect(caps).toEqual({ searchRequired: false, searchPolicySource: SearchPolicySource.TASK_DEFAULT, visionRequired: true, imagesProvided: true }); + + const explicit = resolveCapabilities({ type: TaskType.COMPETITOR_RESEARCH, complexity: TaskComplexity.MEDIUM, requiresSearch: false }); + expect(explicit).toEqual({ searchRequired: false, searchPolicySource: SearchPolicySource.EXPLICIT, visionRequired: false, imagesProvided: false }); + }); + + it('exposes capability evidence on every routing resolution', () => { + const vision = resolveRoute({ type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, images: ['https://cdn.example.com/a.png'] }); + expect(vision).toMatchObject({ searchRequired: false, searchPolicySource: SearchPolicySource.TASK_DEFAULT, visionRequired: true }); + + const search = resolveRoute({ type: TaskType.MARKET_RESEARCH, complexity: TaskComplexity.MEDIUM }); + expect(search).toMatchObject({ searchRequired: true, searchPolicySource: SearchPolicySource.TASK_DEFAULT, visionRequired: false }); + + const general = resolveRoute({ type: TaskType.CONTENT_GENERATION, complexity: TaskComplexity.MEDIUM }); + expect(general).toMatchObject({ searchRequired: false, searchPolicySource: SearchPolicySource.TASK_DEFAULT, visionRequired: false }); + }); + + it('keeps the vision-task inventory inside the search-policy module', () => { + // The double-interpretation bug lived because dispatch re-derived vision + // from a private set in index.ts. The canonical set must live only in the + // capabilities module; index.ts may re-export it but never re-derive a + // plane from it. + expect(readFileSync('src/index.ts', 'utf8')).not.toContain('VISION_TASKS.has'); + expect(readFileSync('src/matrices/search-policy.ts', 'utf8')).toContain('export const VISION_TASKS'); + }); +}); + +describe('fail-closed capability validation happens before any provider action', () => { + it('refuses a vision task without images (VISION_INPUT_REQUIRED) before budget reservation', async () => { + const router = harness(); + const code = await caughtCode(router.execute( + { clientId: 'c', type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM }, + 's', 'u', + )); + expect(code).toBe('VISION_INPUT_REQUIRED'); + expect(router.getClientBudgetReport('c')).toMatchObject({ monthSpend: 0, reservedSpend: 0, activeReservations: 0 }); + expect(router.getCircuitState(Provider.OPENROUTER).failureCount).toBe(0); + }); + + it('refuses images on a non-vision task (IMAGES_NOT_SUPPORTED_FOR_TASK)', async () => { + const router = harness(); + const code = await caughtCode(router.execute( + { clientId: 'c', type: TaskType.CONTENT_GENERATION, complexity: TaskComplexity.MEDIUM }, + 's', 'u', { images: ['https://cdn.example.com/a.png'] }, + )); + expect(code).toBe('IMAGES_NOT_SUPPORTED_FOR_TASK'); + }); + + it('refuses search modifiers without search (SEARCH_MODIFIER_WITHOUT_SEARCH)', async () => { + const router = harness(); + const code = await caughtCode(router.execute( + { clientId: 'c', type: TaskType.STRATEGIC_REASONING, complexity: TaskComplexity.MEDIUM, requiresSearch: false, domainFilter: ['example.com'] }, + 's', 'u', + )); + expect(code).toBe('SEARCH_MODIFIER_WITHOUT_SEARCH'); + expect(router.getClientBudgetReport('c')).toMatchObject({ monthSpend: 0, reservedSpend: 0, activeReservations: 0 }); + }); + + it('refuses consensus without a search-backed route (CONSENSUS_REQUIRES_SEARCH)', async () => { + const router = harness(); + const code = await caughtCode(router.execute( + { clientId: 'c', type: TaskType.STRATEGIC_REASONING, complexity: TaskComplexity.MEDIUM, requiresSearch: false }, + 's', 'u', { consensus: true }, + )); + expect(code).toBe('CONSENSUS_REQUIRES_SEARCH'); + expect(router.getClientBudgetReport('c')).toMatchObject({ monthSpend: 0, reservedSpend: 0, activeReservations: 0 }); + }); + + it('still refuses search + vision with the legacy code and requested payload', () => { + const failed = (() => { + try { + resolveRoute({ type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, requiresSearch: true, images: ['https://cdn.example.com/a.png'] }); + return undefined; + } catch (error) { + return error as UnsupportedCapabilityCombinationError; + } + })(); + expect(failed).toBeInstanceOf(UnsupportedCapabilityCombinationError); + expect(failed?.code).toBe('UNSUPPORTED_CAPABILITY_COMBINATION'); + expect(failed?.requested).toEqual({ taskType: TaskType.SCREENSHOT_ANALYSIS, searchRequired: true, imageCount: 1 }); + }); +}); + +describe('failed routed calls are auditable', () => { + it('records FAILED entries with classification for provider failures', async () => { + const down = () => { throw new ProviderRequestError('gateway down', { provider: Provider.OPENROUTER, kind: 'server', retryable: true, status: 503, code: 'ECONNRESET' }); }; + const router = new L9LLMRouter( + { perplexityApiKey: 'p', openrouterApiKey: 'o' }, + { openrouterClient: { complete: down, completeWithFallback: down, completeWithVision: down }, perplexityClient: { complete: async () => response, completeWithConsensus: async () => ({ best: response, all: [], consensusScore: 1, aggregate: { inputTokens: 0, outputTokens: 0, totalTokens: 0, cost: 0, latencyMs: 0, citations: [] } }) }, idFactory: () => 'task-1' }, + ); + router.initClient('c'); + await expect(router.execute( + { clientId: 'c', type: TaskType.STRATEGIC_REASONING, complexity: TaskComplexity.MEDIUM, requiresSearch: false }, + 'system', 'user', + )).rejects.toThrow(); + const logged = router.getCallLog()[0] as RoutingDecision; + expect(logged.outcome).toBe('FAILED'); + expect(logged.failureKind).toBe('server'); + expect(logged.errorCode).toBe('ECONNRESET'); + expect(logged.searchRequired).toBe(false); + }); + + it('records FAILED entries for local policy failures with the error name', async () => { + // A budget refusal is a local policy failure that happens after route + // resolution and before provider dispatch — it must still be auditable. + const exhaustedStore = { + initClient: async () => undefined, + reserveTask: async () => { throw new BudgetReservationError('budget exhausted'); }, + reconcile: async () => undefined, + release: async () => undefined, + recordSpend: async () => undefined, + resetDaily: async () => undefined, + resetWeekly: async () => undefined, + resetMonthly: async () => undefined, + resetGlobalMonthly: async () => undefined, + checkSurgeAllowance: async () => false, + getClientBudgetReport: async () => undefined, + getAllBudgetReports: async () => [], + getGlobalSpend: async () => undefined, + }; + const router = new L9LLMRouter( + { perplexityApiKey: 'p', openrouterApiKey: 'o' }, + { budgetStore: exhaustedStore, idFactory: () => 'task-1' }, + ); + await expect(router.execute( + { clientId: 'c', type: TaskType.STRATEGIC_REASONING, complexity: TaskComplexity.MEDIUM, requiresSearch: true }, + 's', 'u', + )).rejects.toThrow(); + const logged = router.getCallLog()[0] as RoutingDecision; + expect(logged.outcome).toBe('FAILED'); + expect(['local', 'unknown']).toContain(logged.failureKind); + expect(logged.errorCode).toBeDefined(); + }); + + it('records SUCCESS entries on the existing happy path', async () => { + const router = harness(); + await router.execute({ clientId: 'c', type: TaskType.STRATEGIC_REASONING, complexity: TaskComplexity.MEDIUM, requiresSearch: false }, 's', 'u'); + expect(router.getCallLog()[0]).toMatchObject({ outcome: 'SUCCESS' }); + }); + + it('never leaks prompts, keys, or image contents into the audit', async () => { + const down = () => { throw new ProviderRequestError('boom', { provider: Provider.OPENROUTER, kind: 'server', retryable: true }); }; + const router = new L9LLMRouter( + { perplexityApiKey: 'pplx-secret', openrouterApiKey: 'or-secret' }, + { openrouterClient: { complete: down, completeWithFallback: down, completeWithVision: down }, perplexityClient: { complete: async () => response, completeWithConsensus: async () => ({ best: response, all: [], consensusScore: 1, aggregate: { inputTokens: 0, outputTokens: 0, totalTokens: 0, cost: 0, latencyMs: 0, citations: [] } }) } }, + ); + router.initClient('c'); + await expect(router.execute( + { clientId: 'c', type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, requiresSearch: false }, + 'system prompt with secret-sauce', 'user prompt with secret-sauce', + { images: ['https://cdn.example.com/private-shot.png'] }, + )).rejects.toThrow(); + const serialized = JSON.stringify(router.getCallLog()); + expect(serialized).not.toContain('pplx-secret'); + expect(serialized).not.toContain('or-secret'); + expect(serialized).not.toContain('secret-sauce'); + expect(serialized).not.toContain('private-shot.png'); + }); +}); diff --git a/tests/routing-matrix.test.ts b/tests/routing-matrix.test.ts index 36faa36..ca8bd5e 100644 --- a/tests/routing-matrix.test.ts +++ b/tests/routing-matrix.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from 'vitest'; import { Provider, + RecencyFilter, SearchPolicySource, SonarModel, TaskComplexity, @@ -8,7 +9,7 @@ import { type GeneralModel, type TaskDescriptor, } from '../src/types.js'; -import { L9LLMRouter, resolveRoute, UnsupportedCapabilityCombinationError } from '../src/index.js'; +import { L9LLMRouter, resolveRoute, UnsupportedCapabilityCombinationError, type CapabilityConflictCode } from '../src/index.js'; import { resolveGeneralConfig } from '../src/matrices/general-matrix.js'; import { resolveVisionConfig } from '../src/vision/index.js'; @@ -25,6 +26,8 @@ interface MatrixCase { task: TaskDescriptor; expected: Plane; expectedSource: SearchPolicySource; + /** Required for FAIL_CLOSED rows: the exact machine-readable conflict code. */ + expectedCode?: CapabilityConflictCode; } const task = (over: Partial & Pick): TaskDescriptor => ({ @@ -46,7 +49,14 @@ const MATRIX: MatrixCase[] = [ { id: 'J CONTENT_GENERATION requiresSearch=true', task: task({ type: TaskType.CONTENT_GENERATION, requiresSearch: true }), expected: 'SEARCH', expectedSource: SearchPolicySource.EXPLICIT }, { id: 'K SCREENSHOT_ANALYSIS+imgs requiresSearch=false', task: task({ type: TaskType.SCREENSHOT_ANALYSIS, images: IMAGES, requiresSearch: false }), expected: 'VISION', expectedSource: SearchPolicySource.EXPLICIT }, { id: 'L SCREENSHOT_ANALYSIS+imgs requiresSearch=undefined', task: task({ type: TaskType.SCREENSHOT_ANALYSIS, images: IMAGES }), expected: 'VISION', expectedSource: SearchPolicySource.TASK_DEFAULT }, - { id: 'M SCREENSHOT_ANALYSIS+imgs requiresSearch=true', task: task({ type: TaskType.SCREENSHOT_ANALYSIS, images: IMAGES, requiresSearch: true }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.EXPLICIT }, + { id: 'M SCREENSHOT_ANALYSIS+imgs requiresSearch=true', task: task({ type: TaskType.SCREENSHOT_ANALYSIS, images: IMAGES, requiresSearch: true }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.EXPLICIT, expectedCode: 'UNSUPPORTED_CAPABILITY_COMBINATION' }, + // Capability-integrity rows from the contract: combinations the execution + // plane cannot honor must fail closed with a machine-readable code. + { id: 'R SCREENSHOT_ANALYSIS images=[]', task: task({ type: TaskType.SCREENSHOT_ANALYSIS, images: [] }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.TASK_DEFAULT, expectedCode: 'VISION_INPUT_REQUIRED' }, + { id: 'S SCREENSHOT_ANALYSIS images=undefined', task: task({ type: TaskType.SCREENSHOT_ANALYSIS }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.TASK_DEFAULT, expectedCode: 'VISION_INPUT_REQUIRED' }, + { id: 'T CONTENT_GENERATION images=[image]', task: task({ type: TaskType.CONTENT_GENERATION, images: IMAGES }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.TASK_DEFAULT, expectedCode: 'IMAGES_NOT_SUPPORTED_FOR_TASK' }, + { id: 'U STRATEGIC_REASONING requiresSearch=false + domainFilter', task: task({ type: TaskType.STRATEGIC_REASONING, requiresSearch: false, domainFilter: ['example.com'] }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.EXPLICIT, expectedCode: 'SEARCH_MODIFIER_WITHOUT_SEARCH' }, + { id: 'V STRATEGIC_REASONING requiresSearch=undefined + recency', task: task({ type: TaskType.STRATEGIC_REASONING, recency: RecencyFilter.WEEK }), expected: 'FAIL_CLOSED', expectedSource: SearchPolicySource.TASK_DEFAULT, expectedCode: 'SEARCH_MODIFIER_WITHOUT_SEARCH' }, // Extra coverage required by §5: explicit true lifts otherwise-general task types. { id: 'N EXTRACTION requiresSearch=true', task: task({ type: TaskType.EXTRACTION, requiresSearch: true }), expected: 'SEARCH', expectedSource: SearchPolicySource.EXPLICIT }, { id: 'O CLASSIFICATION requiresSearch=true', task: task({ type: TaskType.CLASSIFICATION, requiresSearch: true }), expected: 'SEARCH', expectedSource: SearchPolicySource.EXPLICIT }, @@ -76,9 +86,18 @@ function planeOf(descriptor: TaskDescriptor): Plane { const VISION_TYPES = new Set([TaskType.VISUAL_QA, TaskType.SCREENSHOT_ANALYSIS, TaskType.LAYOUT_VALIDATION]); describe('§16 routing matrix — explicit requiresSearch is authoritative', () => { - it.each(MATRIX)('$id -> $expected', ({ task: descriptor, expected }) => { + it.each(MATRIX)('$id -> $expected', ({ task: descriptor, expected, expectedCode }) => { if (expected === 'FAIL_CLOSED') { - expect(() => resolveRoute(descriptor)).toThrow(UnsupportedCapabilityCombinationError); + const failed = (() => { + try { + resolveRoute(descriptor); + return undefined; + } catch (error) { + return error as UnsupportedCapabilityCombinationError; + } + })(); + expect(failed, 'expected a fail-closed throw').toBeInstanceOf(UnsupportedCapabilityCombinationError); + expect(failed?.code).toBe(expectedCode); return; } expect(planeOf(descriptor)).toBe(expected); diff --git a/tests/search-policy-dispatch.test.ts b/tests/search-policy-dispatch.test.ts index 644f8b6..aa86def 100644 --- a/tests/search-policy-dispatch.test.ts +++ b/tests/search-policy-dispatch.test.ts @@ -11,7 +11,7 @@ import { type PerplexityConfig, type VisionConfig, } from '../src/types.js'; -import { L9LLMRouter, UnsupportedCapabilityCombinationError } from '../src/index.js'; +import { L9LLMRouter, UnsupportedCapabilityCombinationError, requiresSearchProvider } from '../src/index.js'; import { ProviderRequestError } from '../src/provider-errors.js'; import { resolvePerplexityConfig } from '../src/matrices/perplexity-matrix.js'; import { buildRequestBody } from '../src/providers/perplexity.js'; @@ -151,25 +151,39 @@ describe('§6 search + vision fails closed instead of losing a capability', () = it('leaves vision model selection for a given image count exactly as it was', () => { const { router } = harness(); // Regression guard: the conflict check must not perturb the image count the - // vision matrix sees, including the empty-array edge case. - for (const images of [undefined, [], ['https://cdn.example.com/a.png'], ['https://cdn.example.com/a.png', 'https://cdn.example.com/b.png']]) { + // vision matrix sees on valid vision routes — and a vision task without + // images must fail closed instead of resolving to a phantom vision route. + for (const images of [undefined, []] as (string[] | undefined)[]) { + const failed = (() => { + try { + router.route({ clientId: 'c', type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, images }); + return undefined; + } catch (error) { + return error as UnsupportedCapabilityCombinationError; + } + })(); + expect(failed).toBeInstanceOf(UnsupportedCapabilityCombinationError); + expect(failed?.code).toBe('VISION_INPUT_REQUIRED'); + } + for (const images of [['https://cdn.example.com/a.png'], ['https://cdn.example.com/a.png', 'https://cdn.example.com/b.png']]) { for (const complexity of Object.values(TaskComplexity)) { const decision = router.route({ clientId: 'c', type: TaskType.SCREENSHOT_ANALYSIS, complexity, images }); - const expected = resolveVisionConfig(TaskType.SCREENSHOT_ANALYSIS, complexity, images?.length ?? 1); + const expected = resolveVisionConfig(TaskType.SCREENSHOT_ANALYSIS, complexity, images.length); expect({ model: decision.model, cost: decision.estimatedCost }).toEqual({ model: expected.model, cost: expected.estimatedCostPerCall }); } } }); - it('a visual TaskType with no images and explicit search is a plain search request', async () => { + it('a visual TaskType with no images fails closed even when search is requested', async () => { const { router, calls } = harness(); - await router.execute( + await expect(router.execute( { clientId: 'c', type: TaskType.SCREENSHOT_ANALYSIS, complexity: TaskComplexity.MEDIUM, requiresSearch: true }, 's', 'u', - ); - // Nothing visual was supplied, so nothing visual is discarded. - expect(calls.search).toHaveLength(1); + )).rejects.toMatchObject({ name: 'UnsupportedCapabilityCombinationError', code: 'VISION_INPUT_REQUIRED' }); + // No capability was silently chosen, and no provider was touched. + expect(calls.search).toHaveLength(0); expect(calls.vision).toHaveLength(0); + expect(calls.general).toHaveLength(0); }); }); @@ -178,7 +192,14 @@ describe('§7 Perplexity config agrees with the routing decision', () => { for (const type of Object.values(TaskType)) { for (const complexity of Object.values(TaskComplexity)) { for (const requiresSearch of [true, false, undefined]) { - expect(resolvePerplexityConfig({ type, complexity, requiresSearch, clientId: 'c' }).disableSearch).toBe(false); + const task = { type, complexity, requiresSearch, clientId: 'c' }; + if (requiresSearchProvider(task)) { + expect(resolvePerplexityConfig(task).disableSearch).toBe(false); + } else { + // A non-search task reaching the Perplexity resolver is a contract + // violation, not a configurable state. + expect(() => resolvePerplexityConfig(task)).toThrow(/non-search task/); + } } } } @@ -204,17 +225,15 @@ describe('§7 Perplexity config agrees with the routing decision', () => { }); describe('§14 consensus is an execution modifier, not search-policy authority', () => { - it('does not let consensus=true pull a non-search task onto the search plane', async () => { + it('rejects consensus=true on a non-search route instead of silently ignoring it', async () => { const { router, calls } = harness(); - const result = await router.execute( + await expect(router.execute( { clientId: 'c', type: TaskType.COMPETITOR_RESEARCH, complexity: TaskComplexity.HIGH, requiresSearch: false }, 's', 'u', { consensus: true }, - ); + )).rejects.toMatchObject({ name: 'UnsupportedCapabilityCombinationError', code: 'CONSENSUS_REQUIRES_SEARCH' }); expect(calls.consensus).toHaveLength(0); expect(calls.search).toHaveLength(0); - expect(calls.general).toHaveLength(1); - expect(result.provider).toBe(Provider.OPENROUTER); - expect(router.getCallLog()[0]).toMatchObject({ searchRequired: false, searchPolicySource: SearchPolicySource.EXPLICIT }); + expect(calls.general).toHaveLength(0); }); it('still applies consensus on an actually-selected search route', async () => {