diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f5ae057..c95f8fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -48,7 +48,7 @@ jobs: shell: bash run: | set +e - node --test test/*.test.js 2>&1 | tee /tmp/win-test.log + node --test --import ./test/_setup.js test/*.test.js 2>&1 | tee /tmp/win-test.log ec=${PIPESTATUS[0]} echo "===== FAILING TESTS (name + file) =====" grep -nE '^not ok ' /tmp/win-test.log | head -80 diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f583f0a..123fc3e 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -280,6 +280,22 @@ fails safe to the stock ID on no gateway / unreachable `/v1/models` / no family resolved `tier→model` mapping for verification. The `MODELS` export shape is unchanged: this is a resolution-time layer, not a table edit. +**Typed proposers via TypeSafe System One (`src/jev.js`).** Two of the substrate's proposer +judgments are not text-generation tasks at all: `route`'s complexity band is a classification +(cheap/mid/premium), and preflight's assumption gate is four independent yes/no readings (one +per rubric dimension). When `TYPESAFE_API_KEY` is set (same `FORGE_LLM=1` opt-in), those two +faculties ask Jev instead of a text model — one batched `POST /v1/systemone` returning typed +`choice`/`noul` answers with probability distributions and confidence in ~150ms, versus seconds +of text plus JSON parsing. The module reuses the adjudicate contract verbatim: opt-in, fail-safe +(null → text-LLM fallback → deterministic rubric; a null never moves a verdict), zero-dependency +(the `llm.js` spawned-child pattern, key in child env as `_FORGE_JEV_KEY`), and secret-refusing +on the outgoing state. Jev answers are validated against the questions asked — a choice naming +an option we never offered is garble and fails safe. The reconciles are untouched: `BAND_FLOOR` +still floors the routing band, the assumption gate still bounds completeness to ±band, and +clarifying free-text questions stay with the deterministic rubric, because a System One model +judges but does not author prose. Provenance records which proposer answered +(`llm.provider: "jev"` in `forge route --json`, `assumption.provenance.provider` in preflight). + **Intent cards (`src/intent.js`).** Prompt → intent by the same exemplar k-NN math as model routing — a labeled bank (English + Hinglish rows) under overlap similarity with a confidence gate, NOT a keyword DFA. Note `intentGrams` ≠ `contentGrams`: route.js stops @@ -581,6 +597,7 @@ from the tree it describes. ```mermaid %%{init: {'theme':'base','themeVariables':{'primaryColor':'#201a15','primaryTextColor':'#f2ede7','primaryBorderColor':'#372c22','lineColor':'#f26430','secondaryColor':'#272019','tertiaryColor':'#171310','edgeLabelBackground':'#201a15','clusterBkg':'#171310','clusterBorder':'#4a3b2e','fontFamily':'ui-sans-serif, system-ui, sans-serif','fontSize':'14px'},'flowchart':{'curve':'basis','padding':10,'nodeSpacing':36,'rankSpacing':44}}}%% flowchart LR + test["test
105 files"] test["test
106 files"] src["src
97 files"] landing["landing
61 files"] @@ -588,11 +605,12 @@ flowchart LR global["global
3 files"] bench["bench
2 files"] scripts["scripts
2 files"] + _remember[".remember
1 file"] docs["docs
1 file"] + test -- 201 --> src examples["examples
1 file"] test -- 206 --> src bench -- 7 --> src - examples -- 4 --> src test -- 2 --> scripts scripts --> src src --> global diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b6f09e..12d9f11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,43 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **TypeSafe System One (Jev) as the fast proposer.** Where forge's LLM layer asked a text + model for a judgment that is really a classification or a yes/no — `route`'s complexity band + and preflight's assumption gate — it can now ask Jev instead: typed `choice`/`noul` answers + with real probability distributions and confidence in ~150ms, rather than seconds of text + generation followed by JSON parsing. The new `src/jev.js` client follows the existing + proposer contract exactly: opt-in (`FORGE_LLM=1` plus `TYPESAFE_API_KEY`, overridable via + `TYPESAFE_BASE_URL`), fail-safe (any error → null → text-LLM fallback → deterministic + rubric, and a null never changes a verdict), zero-dependency (one raw HTTPS POST through + the child-process-fetch pattern, the key travelling via child env — never argv, never + logged), and secret-refusing on the way out. Routing keeps its `BAND_FLOOR` reconcile and + gains `llm.provider: "jev"` plus confidence in `forge route --json`; the assumption gate + scores all four rubric dimensions in one batched call (free-text clarifying questions stay + with the deterministic rubric — a System One model judges, it does not author prose). + `test/_setup.js` now scrubs `TYPESAFE_*` so the suite stays hermetic with the key exported. + +### Fixed + +- **The test suite is hermetic.** It inherited the developer's environment, so it was green + in CI and red on any machine where forge was actually installed and enabled — the two + things a maintainer does. An exported `FORGE_LLM=1` both flipped the "llm off by default" + assertion in `test/substrate.test.js` and made the faculties fire real model calls, and a + real `~/.forge` reached `doctor()`'s machine-scoped install check through + `test/doctor.test.js`. Wall time was 593s with two failures. A new `test/_setup.js`, + preloaded via `--import` into every test process, scrubs `FORGE_*`/provider env by prefix, + sandboxes `$HOME` to a throwaway tmpdir, and forces the keyless HTTP runner instead of + shelling out to a real `claude` binary: **0 failures in ~40s**. `test/hermetic.test.js` + pins the scrub list against `envVarsRead()` so the two cannot drift, and fails loudly if + anyone drops the `--import` wiring. Two assertions were wrong rather than merely leaky and + were corrected: `doctor` asserted a global `failed === 0` to prove a local property about + `na` rows, and a comment in `substrate` claimed no runner reaches the real CLI — the + opposite of the truth, and the reason that file spent 85s on live calls. + +### Documentation + +- `CLAUDE.md`: Biome 2.5.2 → 2.5.5 (matching the pin), "600+ tests" → "1000+", and the lint + command `npx biome check` → `npm run check` — the documented command fails outright, since + the npx package is `@biomejs/biome`, not `biome`. - **OpenClaw is a first-class emit target — the compiler's tenth tool.** Instructions need no new file: OpenClaw appends the execution folder's `AGENTS.md` after its configured agent-workspace files as project context, so the canonical source reaches it the same way @@ -198,7 +235,6 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). every PreToolUse hook and recompiled the same trigger-glob RegExp each time; compiled globs are now cached in a module-level map bounded by the distinct globs in the lesson set. - ## [0.27.4] - 2026-08-04 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 0cf48d5..bb8f20b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,14 +3,14 @@ ## Stack - Node.js >=20, pure ESM (`"type": "module"`), zero runtime dependencies. -- Linter/formatter: Biome 2.5.2 (dev dependency). +- Linter/formatter: Biome 2.5.5 (dev dependency). - Types: TypeScript via JSDoc annotations — no `.ts` files, checked by `tsc`. ## Commands - Install: `npm ci` -- Test: `npm test` (node:test, 600+ tests) -- Lint + format: `npx biome check` (or `npm run check`) +- Test: `npm test` (node:test, 1000+ tests) +- Lint + format: `npm run check` (the npx package is `@biomejs/biome`, not `biome`) - Typecheck: `npm run typecheck` - Build pages: `npm run pages:build` diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 133ff44..8c1bec9 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -1452,6 +1452,17 @@ exposes `llm.provenance` per faculty (`llm-cleared` / `llm-tightened` / `llm-rai conservative tighten-/raise-only mode. Each faculty pairs a pure `*LLM` proposer with a `reconcile` step — extend by adding both, never by trusting the model's answer directly. +**TypeSafe System One (Jev) is the preferred proposer when configured.** Where the judgment +is already a classification or a yes/no — `route`'s complexity band (a `choice` over +cheap/mid/premium) and preflight's assumption gate (one batched `noul` per rubric dimension) — +`src/jev.js` asks Jev instead of a text model: typed answers with real probability +distributions and confidence in ~150ms, rather than seconds of generation followed by JSON +parsing. Set `TYPESAFE_API_KEY` (plus the same `FORGE_LLM=1` opt-in) and the two proposers +prefer it automatically; the text-LLM runner remains the fallback on any failure, and the +deterministic rubrics still judge. Free-text clarifying questions stay with the rubric — a +System One model judges, it does not author prose. `forge route --json` shows which proposer +answered under `llm.provider` (`jev` / `text`) with Jev's confidence. + ### Support a new tool Add an emitter module in `src/emit/.js` (mirror an existing one like @@ -1478,6 +1489,8 @@ code reads but this table misses fails CI on the forge repo): | `OPENROUTER_API_KEY` | OpenRouter provider | | `OPENAI_API_KEY` | OpenAI provider (OpenAI-compatible chat/completions); low-configuration auto-detect fallback after Anthropic | | `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Google Gemini provider via its OpenAI-compatible endpoint; low-configuration auto-detect fallback after Anthropic | +| `TYPESAFE_API_KEY` | TypeSafe System One (Jev) — with `FORGE_LLM=1`, the route/assumption proposers prefer typed ~150ms judgments over a text round-trip; unset = text-LLM proposer only | +| `TYPESAFE_BASE_URL` | override the Jev endpoint (default `https://api.typesafe.ai`) — staging/self-hosted | | `FORGE_LLM` | `1` enables the LLM proposer layer (off = fully deterministic) | | `FORGE_LLM_AMBIENT` | `1` lets the ambient hook use the proposer too | | `FORGE_LLM_HTTP` | `1` forces direct HTTP (Anthropic Messages or OpenAI-compatible, per the resolved provider) instead of the `claude` CLI; automatic when the CLI is absent | diff --git a/mintlify/concepts/model-routing.mdx b/mintlify/concepts/model-routing.mdx index 44771cd..f9d7dd1 100644 --- a/mintlify/concepts/model-routing.mdx +++ b/mintlify/concepts/model-routing.mdx @@ -24,6 +24,22 @@ under an overlap-similarity metric with a confidence gate — not a keyword look genuinely needs it. +## Optional typed proposer — TypeSafe Jev + +The rubric is the judge; an optional **proposer** layer can refine it. With `FORGE_LLM=1` +plus `TYPESAFE_API_KEY` set, `route` asks TypeSafe's System One model (Jev) for the +complexity band as a typed `choice` with probabilities and confidence in ~150ms — instead +of a multi-second text-LLM round-trip. The same applies to preflight's assumption gate, +scored as one batched `noul` per rubric dimension. + + + Fail-safe by construction: any Jev error falls back to the text-LLM proposer, then to + the deterministic rubric — and a miss never changes a verdict. Without the key, behavior + is byte-identical. `forge route --json` shows which proposer answered under + `llm.provider` (`jev` / `text`), with Jev's confidence. `TYPESAFE_BASE_URL` overrides + the endpoint for staging or self-hosted deployments. + + ## Intent, then tier Routing shares its math with intent detection (`src/intent.js`): a prompt maps to an diff --git a/mintlify/concepts/pre-action-gate.mdx b/mintlify/concepts/pre-action-gate.mdx index 4fa92f7..551b28e 100644 --- a/mintlify/concepts/pre-action-gate.mdx +++ b/mintlify/concepts/pre-action-gate.mdx @@ -48,6 +48,15 @@ flowchart TD +## Optional model proposers + +Every phase above is a deterministic rubric by default. `FORGE_LLM=1` adds a thin +**proposer** layer that can refine — never decide — the gate. With `TYPESAFE_API_KEY` +also set, the route and assumption proposers prefer TypeSafe's System One (Jev): typed +`choice` / `noul` answers with probabilities in ~150ms instead of a text round-trip. +Any failure falls back to the deterministic path, so the flags are safe to leave off or +on. + ## Blast radius **Blast radius** — the set of files an edit is predicted to impact, read from the code diff --git a/package.json b/package.json index 70a052c..c9280e9 100644 --- a/package.json +++ b/package.json @@ -52,14 +52,14 @@ "scripts" ], "scripts": { - "test": "node --test test/*.test.js", + "test": "node --test --import ./test/_setup.js test/*.test.js", "bench": "node bench/bench.mjs", "lint": "biome lint .", "format": "biome format --write .", "check": "biome check .", "check:fix": "biome check --write .", "typecheck": "tsc -p tsconfig.json", - "coverage": "node --test --experimental-test-coverage test/*.test.js", + "coverage": "node --test --experimental-test-coverage --import ./test/_setup.js test/*.test.js", "bump": "node scripts/bump.mjs", "forge": "node src/cli.js", "pages:build": "node scripts/build-pages.mjs", diff --git a/src/docs_check.js b/src/docs_check.js index 34597e0..09e39e3 100644 --- a/src/docs_check.js +++ b/src/docs_check.js @@ -20,6 +20,7 @@ const DOC_FILES = ["README.md", "docs/GUIDE.md", "ARCHITECTURE.md", "ROADMAP.md" // values injected by host tools rather than set by users. const INTERNAL_ENV = new Set([ "_FORGE_LLM_KEY", + "_FORGE_JEV_KEY", "FORGE_EMBED_KEY", // Test-only override of the settings.json path `forge init` targets — plumbing for // exercising merge/remove/exit-code behavior without touching the real ~/.claude. @@ -31,7 +32,8 @@ const INTERNAL_ENV = new Set([ // Prefixes that mark an env var as OURS to document. A doc may freely mention other // tools' vars (GITHUB_TOKEN, PATH) — those aren't claims about forge's own surface. -const ENV_PREFIX_RE = /\b((?:FORGE|ANTHROPIC|LITELLM|OPENROUTER|ENABLE_CORTEX)_[A-Z0-9_]+)\b/g; +const ENV_PREFIX_RE = + /\b((?:FORGE|ANTHROPIC|LITELLM|OPENROUTER|ENABLE_CORTEX|TYPESAFE)_[A-Z0-9_]+)\b/g; function readDoc(root, rel) { const p = join(root, rel); diff --git a/src/jev.js b/src/jev.js new file mode 100644 index 0000000..61f0c42 --- /dev/null +++ b/src/jev.js @@ -0,0 +1,139 @@ +// forge jev — TypeSafe System One (Jev) client. Jev is not a text LLM: it takes a `state` +// plus typed `questions` (choice / score / noul) and returns typed answers with probabilities +// and confidence in ~150ms — the fast proposer for faculties whose judgment is already a +// classification, rating, or yes/no (route's complexity band, preflight's assumption gate). +// +// Same design contract as src/adjudicate.js, restated for a typed API: +// - OPT-IN. Off unless the LLM layer is enabled (llmEnabled) AND TYPESAFE_API_KEY is set. +// Without the key every function here returns null and behavior is byte-identical. +// - FAIL-SAFE. Any error/timeout/non-2xx/garble/secret → null. A null NEVER changes a +// verdict; callers fall back to the text-LLM proposer, then to the deterministic rubric. +// - ZERO-DEP. One raw HTTPS POST via the child-process-fetch pattern (src/llm.js) — no SDK. +// The key travels via the child's env (_FORGE_JEV_KEY) — never in argv, never logged. +// API contract: https://docs.typesafe.ai/api.md +import { spawnSync } from "node:child_process"; +import { llmEnabled } from "./adjudicate.js"; +import { hasSecret } from "./secrets.js"; +import { clamp01 } from "./util.js"; + +// POST {baseUrl}/v1/systemone, bearer auth. Body arrives on stdin; the key never does. +const HTTP_CHILD = `let raw="";process.stdin.on("data",(d)=>{raw+=d;});process.stdin.on("end",async()=>{try{const{url,payload}=JSON.parse(raw);const key=process.env._FORGE_JEV_KEY||"";const res=await fetch(url,{method:"POST",headers:{"content-type":"application/json",authorization:"Bearer "+key},body:JSON.stringify(payload)});if(!res.ok){process.stderr.write("jev: http "+res.status);process.exit(1);}process.stdout.write(JSON.stringify(await res.json()));}catch(e){process.stderr.write("jev: "+(e.message||e));process.exit(1);}});`; + +/** The TypeSafe API key from the environment ("" when unset — every call then fail-safes). */ +export function jevKey() { + return process.env.TYPESAFE_API_KEY || ""; +} + +/** Base URL, overridable for self-hosted/staging endpoints. */ +export function jevBaseUrl() { + return (process.env.TYPESAFE_BASE_URL || "https://api.typesafe.ai").replace(/\/+$/, ""); +} + +/** + * Is the Jev proposer layer active for this call? Same opt-in as the text-LLM layer + * (`llmEnabled`) plus the key. One switch governs both proposers; Jev is simply preferred + * when its credentials exist. + * @param {{llm?:boolean}} [opts] + */ +export function jevEnabled(opts = {}) { + return llmEnabled(opts) && Boolean(jevKey()); +} + +/** A Choice question: pick one option from `criteria` (option → rubric description). */ +export const choice = (instructions, criteria) => ({ type: "choice", instructions, criteria }); + +/** A Noul question: yes/no as a probability 0–1. `criteria` (optional) says what each means. */ +export const noul = (instructions, criteria) => + criteria ? { type: "noul", instructions, criteria } : { type: "noul", instructions }; + +/** A Score question: rate along ordered `criteria` levels (probability-weighted value). */ +export const score = (instructions, criteria) => ({ type: "score", instructions, criteria }); + +const isUnit = (v) => typeof v === "number" && Number.isFinite(v) && v >= 0 && v <= 1; + +/** + * Validate one raw answer against its question; returns a clean answer or null. + * Typed output guarantees the interface, not the values — a Choice naming an option we + * never offered, or a Noul outside [0,1], is garble and fails safe like any other. + */ +function validateAnswer(question, answer) { + if (!answer || typeof answer !== "object" || answer.type !== question.type) return null; + if (question.type === "noul") { + const n = Number(answer.noul); + if (!Number.isFinite(n)) return null; + return { type: "noul", noul: clamp01(n) }; + } + if (question.type === "choice") { + const pick = String(answer.choice ?? ""); + if (!Object.hasOwn(question.criteria, pick)) return null; + const out = { type: "choice", choice: pick }; + if (isUnit(answer.confidence)) out.confidence = answer.confidence; + if (answer.probabilities && typeof answer.probabilities === "object") { + const probabilities = {}; + for (const [option, p] of Object.entries(answer.probabilities)) { + const n = Number(p); + if (Object.hasOwn(question.criteria, option) && Number.isFinite(n)) + probabilities[option] = clamp01(n); + } + out.probabilities = probabilities; + } + return out; + } + if (question.type === "score") { + const n = Number(answer.score); + if (!Number.isFinite(n)) return null; + const out = { type: "score", score: n }; + if (isUnit(answer.confidence)) out.confidence = answer.confidence; + if (answer.legend && typeof answer.legend === "object") out.legend = answer.legend; + return out; + } + return null; +} + +/** Synchronous HTTPS round-trip through the child (forge faculties are synchronous). */ +function httpCall(payload, timeoutMs) { + const input = JSON.stringify({ url: `${jevBaseUrl()}/v1/systemone`, payload }); + const r = spawnSync(process.execPath, ["-e", HTTP_CHILD], { + input, + encoding: "utf8", + timeout: timeoutMs, + maxBuffer: 4 * 1024 * 1024, + env: { ...process.env, _FORGE_JEV_KEY: jevKey() }, + stdio: ["pipe", "pipe", "pipe"], + }); + if (r.error || r.status !== 0 || !r.stdout) { + throw new Error(r.stderr?.trim() || r.error?.message || "jev call failed"); + } + return JSON.parse(r.stdout); +} + +/** + * Evaluate `state` against `questions` (one API call — questions run in parallel server-side). + * Returns `{ model, answers, usage }` with validated answers only, or null on ANY failure. + * `call` is the injectable transport for tests: (payload) => raw API response object. + * @param {{state?: string|object|Array, questions?: Record, model?: string, + * timeoutMs?: number, call?: (payload: object) => object}} spec + */ +export function systemOne({ state, questions, model = "jev-latest", timeoutMs = 5000, call } = {}) { + try { + if (!jevKey()) return null; + if (!questions || typeof questions !== "object" || !Object.keys(questions).length) return null; + const stateText = typeof state === "string" ? state : JSON.stringify(state); + if (hasSecret(stateText)) return null; // never send a secret to the model + const payload = { state, model, questions }; + const raw = call ? call(payload) : httpCall(payload, timeoutMs); + if (!raw || typeof raw !== "object" || !raw.answers || typeof raw.answers !== "object") + return null; + const answers = {}; + for (const [id, q] of Object.entries(questions)) { + const a = validateAnswer(q, raw.answers[id]); + if (a) answers[id] = a; + } + if (!Object.keys(answers).length) return null; + return { model: raw.model ?? model, answers, usage: raw.usage ?? null }; + } catch (err) { + if (process.env.FORGE_DEBUG === "1") + process.stderr.write(`forge jev: ${err?.message ?? err}\n`); + return null; + } +} diff --git a/src/preflight.js b/src/preflight.js index c25116c..098976b 100644 --- a/src/preflight.js +++ b/src/preflight.js @@ -5,6 +5,7 @@ import { existsSync } from "node:fs"; import { join } from "node:path"; import { adjudicate, asText, asUnit, buildRunner, llmEnabled } from "./adjudicate.js"; import { build as buildAtlas, has, load as loadAtlas } from "./atlas.js"; +import { jevEnabled, noul, systemOne } from "./jev.js"; import { sigmoid } from "./predictor.js"; import { CODE_EXT } from "./util.js"; @@ -300,6 +301,49 @@ export function assessTaskLLM(task, { run = buildRunner() } = {}) { }); } +/** The assumption gate as batched Jev nouls — one per rubric dimension, one API call. */ +export function buildAssumptionNouls(task) { + const questions = {}; + for (const d of DIMENSIONS) { + questions[d.key] = noul( + `A coding agent received this task. Is "${d.key}" — ${d.description} — specified concretely enough to start coding?`, + { + true: `The ${d.description} is concrete and actionable`, + false: `The ${d.description} is missing, vague, or left to be assumed`, + }, + ); + } + return { state: String(task).slice(0, 1200), questions }; +} + +/** + * Ask Jev (TypeSafe System One) for an assumption reading: one batched call scoring every + * rubric dimension as a yes/no probability. `completeness` is their mean; a dimension is + * `missing` when its noul lands below 0.5 (more-likely-unspecified — the noul's own + * semantics, no new threshold constant). Free-text clarifying questions are beyond a + * System One model, so `questions` stays empty and the deterministic rubric's questions + * stand. Returns null when off/unavailable. + * @param {string} task + * @param {object} [opts] + * @param {boolean} [opts.llm] + * @param {(payload:object)=>object} [opts.call] injectable Jev transport (tests) + */ +export function assessTaskJev(task, { llm, call } = {}) { + if (!jevEnabled({ llm })) return null; + const { state, questions } = buildAssumptionNouls(task); + const res = systemOne({ state, questions, call }); + if (!res) return null; + const values = []; + for (const k of DIM_KEYS) { + const v = res.answers?.[k]?.noul; + if (typeof v !== "number") return null; + values.push(v); + } + const completeness = values.reduce((s, v) => s + v, 0) / values.length; + const missing = DIM_KEYS.filter((_, i) => values[i] < 0.5); + return { completeness, missing, questions: [], provider: "jev" }; +} + /** * Verify-don't-trust reconcile for M2. The model may only move completeness within ±band of the * deterministic score, so a clearly-specified or clearly-vague task can never be flipped — only a @@ -311,7 +355,7 @@ export function assessTaskLLM(task, { run = buildRunner() } = {}) { * behaviour). Extra questions survive only if they map to a rubric-flagged dimension or (via * `grounded`) reference a real repo entity. * @param {object} det - assessTask() result - * @param {{completeness:number, missing:string[], questions:string[]}|null} proposal + * @param {{completeness:number, missing:string[], questions:string[], provider?:string}|null} proposal * @param {object} [opts] * @param {number} [opts.askThreshold] * @param {number} [opts.band] @@ -362,7 +406,11 @@ export function reconcileAssumption( shouldAsk && !questions.length ? ["What exactly should this produce, and how will we know it is correct?"] : questions, - provenance: { path, detCompleteness: det.completeness }, + provenance: { + path, + detCompleteness: det.completeness, + ...(proposal.provider ? { provider: proposal.provider } : {}), + }, }; } @@ -415,6 +463,7 @@ export function clarifyBlock(result, { threshold = 0.5 } = {}) { * @param {string} [opts.model] * @param {number} [opts.timeoutMs] * @param {(p:string)=>string} [opts.run] + * @param {(payload:object)=>object} [opts.jevCall] injectable Jev transport (tests) * @param {boolean} [opts.bidirectional] * @param {number} [opts.band] */ @@ -428,6 +477,7 @@ export function preflightRepo( model, timeoutMs, run, + jevCall, bidirectional = true, band, } = {}, @@ -448,9 +498,11 @@ export function preflightRepo( ...gap, assumption: { ...det, provenance: { path: "deterministic" } }, }; - const proposal = assessTaskLLM(text, { - run: run || buildRunner({ model, timeoutMs }), - }); + const proposal = + assessTaskJev(text, { llm, call: jevCall }) ?? + assessTaskLLM(text, { + run: run || buildRunner({ model, timeoutMs }), + }); const grounded = (q) => { const { symbols, files } = referencedEntities(q); return symbols.some(hasSymbol) || files.some((f) => existsSync(join(root, f))); diff --git a/src/route.js b/src/route.js index d2a0737..46c7f50 100644 --- a/src/route.js +++ b/src/route.js @@ -8,6 +8,7 @@ import { adjudicate, asText, buildRunner, llmEnabled } from "./adjudicate.js"; import { matchingLessons } from "./cortex.js"; import { gitChurn, grepFanout } from "./cortex_features.js"; import { recordRoute } from "./cost_report.js"; +import { choice, jevEnabled, systemOne } from "./jev.js"; import { mergedLessons } from "./ledger_read.js"; import { setOverlap } from "./math.js"; import { MODELS } from "./model_tiers.js"; @@ -389,6 +390,51 @@ export function complexityLLM(task, { run = buildRunner() } = {}) { return adjudicate({ prompt: buildComplexityPrompt(task), parse: parseComplexityProposal, run }); } +/** The complexity judgment as a Jev Choice — the same three bands the text proposer uses. */ +export function buildComplexityChoice(task) { + return { + state: String(task).slice(0, 1200), + questions: { + band: choice( + "Judge the intrinsic complexity of this coding task for model selection (not how to do the task).", + { + cheap: "Trivial or boilerplate: a typo, rename, formatting, or a one-line helper", + mid: "A data structure, class, or library-level change with a few moving parts", + premium: "Algorithmic, systems, concurrency, architectural, or multi-module work", + }, + ), + }, + }; +} + +/** + * Ask Jev (TypeSafe System One) for a complexity band. Same proposal contract as + * complexityLLM — the band still floors at BAND_FLOOR and the deterministic rubric still + * judges — but the answer is typed and carries the probability distribution Jev computed, + * in ~150ms instead of a text round-trip. Returns null when off/unavailable. + * @param {string} task + * @param {object} [opts] + * @param {boolean} [opts.llm] + * @param {(payload:object)=>object} [opts.call] injectable Jev transport (tests) + */ +export function complexityJev(task, { llm, call } = {}) { + if (!jevEnabled({ llm })) return null; + const { state, questions } = buildComplexityChoice(task); + const res = systemOne({ state, questions, call }); + const ans = res?.answers?.band; + if (!ans) return null; + const band = ans.choice.toLowerCase(); + if (!(band in BAND_FLOOR)) return null; + return { + band, + score: BAND_FLOOR[band], + reason: ans.confidence != null ? `jev confidence ${ans.confidence.toFixed(2)}` : "jev choice", + provider: "jev", + confidence: ans.confidence ?? null, + probabilities: ans.probabilities ?? null, + }; +} + /** * Repo wrapper: gather the real signals for a task and route it. `run` is injectable for tests. * @param {string} root @@ -398,6 +444,7 @@ export function complexityLLM(task, { run = buildRunner() } = {}) { * @param {string} [opts.model] * @param {number} [opts.timeoutMs] * @param {(p:string)=>string} [opts.run] + * @param {(payload:object)=>object} [opts.jevCall] injectable Jev transport (tests) * @param {boolean} [opts.bidirectional] * @param {number} [opts.routingBand] * @param {number} [opts.signalFloor] @@ -411,6 +458,7 @@ export function routeTask( model, timeoutMs, run, + jevCall, bidirectional = true, routingBand = 0.2, signalFloor = 0.4, @@ -453,9 +501,12 @@ export function routeTask( // below the rubric, and never below `signalFloor` when the rubric confidently matched an // algorithmic/architectural exemplar, so a "distributed rate-limiter" can't be talked down // to the cheap tier. - // With `bidirectional:false` it stays raise-only. Fail-safe: a null proposal is ignored. + // Jev (typed, ~150ms) is the preferred proposer when its key is configured; the text-LLM + // runner is the fallback, and a null from either is ignored (fail-safe). + // With `bidirectional:false` it stays raise-only. const proposal = llmEnabled({ llm }) - ? complexityLLM(task, { run: run || buildRunner({ model, timeoutMs }) }) + ? (complexityJev(task, { llm, call: jevCall }) ?? + complexityLLM(task, { run: run || buildRunner({ model, timeoutMs }) })) : null; const strongSignal = rubric.strongTopicSignal; let score = detScore; @@ -481,7 +532,13 @@ export function routeTask( signals, rubric, llm: proposal - ? { band: proposal.band, reason: proposal.reason, direction: path.replace("llm-", "") } + ? { + band: proposal.band, + reason: proposal.reason, + direction: path.replace("llm-", ""), + provider: proposal.provider ?? "text", + ...(proposal.confidence != null ? { confidence: proposal.confidence } : {}), + } : null, provenance: { path }, ...recommended, diff --git a/test/_setup.js b/test/_setup.js new file mode 100644 index 0000000..31ff7f8 --- /dev/null +++ b/test/_setup.js @@ -0,0 +1,51 @@ +// The suite's hermetic boundary. Preloaded into EVERY test process via +// `node --test --import ./test/_setup.js`, so it runs before any test module body. +// +// Three ambient things made this suite non-hermetic — CI green, developer machine red: +// 1. Exported FORGE_*/provider env. An exported FORGE_LLM=1 flipped the assertion at +// test/substrate.test.js:105 AND made that file fire real model calls: 552s vs <1s. +// 2. The real $HOME. src reads ~/.forge (src/doctor.js:374), ~/.claude/settings.json +// (src/doctor.js:93), ~/.claude/projects (src/cost_report.js:219) and +// ~/.local/state/forgekit (src/recall.js:22); git reads ~/.gitconfig. +// 3. A real `claude` binary on PATH, which src/adjudicate.js shells out to. +// +// A test that needs one of these BACK just sets it in its own file's top-level body: +// node --test gives every FILE its own process, so a per-file assignment runs after this +// and wins. That is already the convention at test/doctor.test.js:20 and +// test/recall.test.js:11 — no opt-in machinery required. +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// A PREFIX DENYLIST, not an allowlist. An allowlist would have to enumerate everything git, +// node, bash and Windows need (SYSTEMROOT, PATHEXT, COMSPEC, SSH_AUTH_SOCK, TMP...) and rots +// on contact — and this suite spawns git/node/bash/npx and runs on windows-latest. The +// prefix rule is a superset of src/docs_check.js envVarsRead(): it also covers provider keys +// forge does not read yet, and any FORGE_NEWTHING added next week, with zero edits. That +// zero-maintenance property is the point; test/hermetic.test.js pins it against +// envVarsRead() so the two can never drift. +const SCRUB = + /^(_?FORGE_|CLAUDE_|ANTHROPIC_|OPENAI_|OPENROUTER_|GEMINI_|GOOGLE_|LITELLM_|ENABLE_CORTEX_|TYPESAFE_|XDG_)/; +// Not prefix-matchable. FORCE_COLOR is the dangerous one: it OUTRANKS NO_COLOR in +// src/fmt.js supportsColor(), so an exported FORCE_COLOR=1 defeats the explicit NO_COLOR=1 +// that test/radar.test.js passes to its spawned CLI. +const SCRUB_EXACT = ["CLAUDECODE", "FORCE_COLOR", "NO_COLOR", "COLORTERM"]; + +for (const key of Object.keys(process.env)) if (SCRUB.test(key)) delete process.env[key]; +for (const key of SCRUB_EXACT) delete process.env[key]; + +// An empty home, not a missing one. os.homedir() honours $HOME (POSIX) / $USERPROFILE +// (Windows) and is not cached, so every ~/.forge, ~/.claude and ~/.gitconfig read lands in +// throwaway space. This is also what makes `git init` deterministic: no inherited +// init.defaultBranch, and no commit.gpgsign, which would otherwise block the suite on a +// passphrase prompt. +const home = mkdtempSync(join(tmpdir(), "forge-test-home-")); +process.env.HOME = home; +process.env.USERPROFILE = home; + +// The last uninjected model call. Faculties that build their own runner (src/substrate.js, +// src/route.js, src/anchor.js, src/preflight.js) reach the real `claude` binary when it is +// on PATH. FORGE_LLM_HTTP=1 forces src/adjudicate.js down the HTTP branch instead, where the +// now-keyless provider resolution returns null and the runner throws synchronously — the +// exact fail-safe path CI already takes. No subprocess, no socket, no timeout. +process.env.FORGE_LLM_HTTP = "1"; diff --git a/test/doctor.test.js b/test/doctor.test.js index 11a860c..6787910 100644 --- a/test/doctor.test.js +++ b/test/doctor.test.js @@ -168,7 +168,14 @@ test("doctor: a missing atlas is UNAVAILABLE, not ACTIVE; a fresh one is ACTIVE const atlasRow = r.results.find((x) => x.label === "atlas"); assert.equal(atlasRow.status, "na", "not built is neither ok nor a failure"); assert.equal(r.health.atlas, "UNAVAILABLE"); - assert.equal(r.failed, 0, "na never counts toward failed totals"); + // Say what this means locally. Asserting `r.failed === 0` proved the same point only when + // NO check anywhere failed, which made it machine-dependent: a stale ~/.forge on the + // developer's box (a legitimate `fail` from the machine-scoped checkInstall) broke it. + assert.equal( + r.failed, + r.results.filter((x) => x.status === "fail").length, + "na never counts toward failed totals", + ); const built = fixture(); writeFileSync(join(built, "a.js"), "export const one = 1;\n"); diff --git a/test/hermetic.test.js b/test/hermetic.test.js new file mode 100644 index 0000000..41a00fd --- /dev/null +++ b/test/hermetic.test.js @@ -0,0 +1,58 @@ +// The suite's hermetic contract, pinned. This file fails if test/_setup.js stops running +// (someone drops --import from package.json), stops covering the env surface (someone adds a +// process.env read under a new prefix), or stops sandboxing $HOME. +// +// Why this exists: both historical failures here — test/substrate.test.js:105 and +// test/doctor.test.js — were invisible in CI (clean env, no ~/.forge) and only fired on a +// machine where forge was actually installed and enabled. CI-green/local-red is the exact +// failure mode a hermetic boundary prevents, and only a test can notice the boundary is gone. +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { homedir, userInfo } from "node:os"; +import { test } from "node:test"; +import { envVarsRead } from "../src/docs_check.js"; + +// userInfo().homedir reads the passwd DB and IGNORES $HOME — the only oracle for "the real +// home" that survives its own sandbox. +test("_setup ran: $HOME is a sandbox, not the developer's home", () => { + assert.notEqual(homedir(), userInfo().homedir, "test/_setup.js did not run — is --import wired?"); +}); + +// Regex false positives from comments, not real reads (src/commit_gate.js, src/consensus.js, +// src/docs_check.js, src/docs_impact.js all document `process.env.X` in prose). TERM is +// deliberately kept: only TERM=dumb is meaningful in src/fmt.js and it forces colour off, +// which already matches a non-TTY test process. +const NOT_SCRUBBED = new Set(["TERM", "TOKEN", "X"]); + +test("every env var src reads is scrubbed (the denylist cannot drift from envVarsRead)", () => { + const leaked = [...envVarsRead()].filter( + // FORGE_LLM_HTTP is set BY _setup on purpose: it forces the keyless HTTP runner. + (v) => !NOT_SCRUBBED.has(v) && v !== "FORGE_LLM_HTTP" && process.env[v] !== undefined, + ); + assert.deepEqual(leaked, [], `not scrubbed by test/_setup.js: ${leaked.join(", ")}`); +}); + +test("canary: a hostile env cannot reach a test process", () => { + const canary = + "import{homedir,userInfo}from'node:os';" + + "if(process.env.FORGE_LLM||process.env.ANTHROPIC_API_KEY)throw new Error('env leaked');" + + "if(homedir()===userInfo().homedir)throw new Error('HOME leaked');"; + const r = spawnSync( + process.execPath, + // A file:// URL, not a path: --import resolves module specifiers, and on Windows a + // plain absolute path (D:\…) parses as the URL scheme "d:" (ERR_UNSUPPORTED_ESM_URL_SCHEME). + ["--import", new URL("./_setup.js", import.meta.url).href, "-e", canary], + { + encoding: "utf8", + env: { + ...process.env, + HOME: userInfo().homedir, + USERPROFILE: userInfo().homedir, + FORGE_LLM: "1", + ANTHROPIC_API_KEY: ["sk", "ant", "api03", "HOSTILECANARYVALUE"].join("-"), + FORCE_COLOR: "1", + }, + }, + ); + assert.equal(r.status, 0, r.stderr); +}); diff --git a/test/jev.test.js b/test/jev.test.js new file mode 100644 index 0000000..76ee323 --- /dev/null +++ b/test/jev.test.js @@ -0,0 +1,275 @@ +// Jev (TypeSafe System One) client + proposer wiring. No network: the transport is +// injected everywhere, and the API key is set in-file (per the hermetic convention — +// every test file gets its own process, so this assignment runs after _setup's scrub). +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; + +process.env.TYPESAFE_API_KEY = "test-key-not-real"; + +import { choice, jevEnabled, noul, score, systemOne } from "../src/jev.js"; +import { assessTaskJev, buildAssumptionNouls, preflightRepo } from "../src/preflight.js"; +import { complexityJev, routeTask } from "../src/route.js"; + +const fixture = () => mkdtempSync(join(tmpdir(), "forge-jev-")); + +test("builders emit the exact API question shapes", () => { + assert.deepEqual(choice("Pick one", { a: "A thing", b: null }), { + type: "choice", + instructions: "Pick one", + criteria: { a: "A thing", b: null }, + }); + assert.deepEqual(noul("Is it so?"), { type: "noul", instructions: "Is it so?" }); + assert.deepEqual(noul("Is it so?", { true: "Yes means this", false: "No means that" }), { + type: "noul", + instructions: "Is it so?", + criteria: { true: "Yes means this", false: "No means that" }, + }); + assert.deepEqual(score("Rate it", ["Low", "High"]), { + type: "score", + instructions: "Rate it", + criteria: ["Low", "High"], + }); +}); + +test("jevEnabled: both the LLM opt-in AND the key are required", () => { + assert.equal(jevEnabled({ llm: true }), true); + assert.equal(jevEnabled({ llm: false }), false); + assert.equal(jevEnabled(), false, "FORGE_LLM is scrubbed in tests — off by default"); + const key = process.env.TYPESAFE_API_KEY; + delete process.env.TYPESAFE_API_KEY; + assert.equal(jevEnabled({ llm: true }), false, "no key — every call fail-safes"); + process.env.TYPESAFE_API_KEY = key; +}); + +test("systemOne: a valid mixed response is validated and returned under the same ids", () => { + const res = systemOne({ + state: "My VPS has been down for 3 hours!", + questions: { + department: choice("Which team?", { billing: "Money", technical: "Outages" }), + is_urgent: noul("Urgent?"), + }, + call: () => ({ + model: "jev-1.13.0", + answers: { + department: { + type: "choice", + choice: "technical", + probabilities: { technical: 0.88, billing: 0.12 }, + confidence: 0.81, + }, + is_urgent: { type: "noul", noul: 0.97 }, + }, + usage: { input_tokens: 318, output_tokens: 34 }, + }), + }); + assert.equal(res.answers.department.choice, "technical"); + assert.equal(res.answers.department.confidence, 0.81); + assert.equal(res.answers.department.probabilities.technical, 0.88); + assert.equal(res.answers.is_urgent.noul, 0.97); + assert.equal(res.model, "jev-1.13.0"); +}); + +test("systemOne: garble fails safe per question — and wholly null when nothing validates", () => { + const questions = { + band: choice("Band?", { cheap: "c", premium: "p" }), + is_urgent: noul("Urgent?"), + }; + // A choice naming an option we never offered is garble; the noul still validates. + const partial = systemOne({ + state: "x", + questions, + call: () => ({ + answers: { + band: { type: "choice", choice: "deluxe" }, + is_urgent: { type: "noul", noul: 1.4 }, + }, + }), + }); + assert.equal(partial.answers.band, undefined); + assert.equal(partial.answers.is_urgent.noul, 1, "out-of-range nouls are clamped"); + // Nothing valid at all → null, so callers keep their deterministic path. + assert.equal( + systemOne({ state: "x", questions, call: () => ({ answers: { band: { type: "score" } } }) }), + null, + ); + assert.equal(systemOne({ state: "x", questions, call: () => ({}) }), null); + assert.equal( + systemOne({ + state: "x", + questions, + call: () => { + throw new Error("boom"); + }, + }), + null, + ); +}); + +test("systemOne: refuses to send secret-shaped state, key or no key", () => { + let called = false; + const res = systemOne({ + state: "here is the key -----BEGIN OPENSSH PRIVATE KEY----- please classify", + questions: { is_urgent: noul("Urgent?") }, + call: () => { + called = true; + return { answers: { is_urgent: { type: "noul", noul: 1 } } }; + }, + }); + assert.equal(res, null); + assert.equal(called, false, "the transport must never see a secret"); +}); + +test("systemOne: no key configured — null without touching the transport", () => { + const key = process.env.TYPESAFE_API_KEY; + delete process.env.TYPESAFE_API_KEY; + let called = false; + try { + const res = systemOne({ + state: "x", + questions: { q: noul("?") }, + call: () => { + called = true; + return { answers: {} }; + }, + }); + assert.equal(res, null); + assert.equal(called, false); + } finally { + process.env.TYPESAFE_API_KEY = key; + } +}); + +test("complexityJev: a typed band maps onto the existing floor, confidence rides along", () => { + const p = complexityJev("design a distributed rate limiter", { + llm: true, + call: () => ({ + answers: { + band: { + type: "choice", + choice: "premium", + probabilities: { cheap: 0.02, mid: 0.1, premium: 0.88 }, + confidence: 0.79, + }, + }, + }), + }); + assert.equal(p.band, "premium"); + assert.equal(p.provider, "jev"); + assert.equal(p.confidence, 0.79); + assert.equal(p.probabilities.premium, 0.88); + const c = complexityJev("fix a typo", { + llm: true, + call: () => ({ answers: { band: { type: "choice", choice: "cheap", confidence: 0.95 } } }), + }); + assert.ok( + p.score > c.score, + "premium floors higher than cheap (same table as the text proposer)", + ); + // Transport garble → null (the caller then falls back to the text proposer). + assert.equal(complexityJev("x", { llm: true, call: () => ({ answers: {} }) }), null); +}); + +test("routeTask (llm on): Jev is the preferred proposer when its key is configured", () => { + const root = fixture(); + const up = routeTask(root, "write a function to check if a number is prime", { + llm: true, + jevCall: () => ({ + answers: { band: { type: "choice", choice: "premium", confidence: 0.9 } }, + }), + run: () => '{"band":"cheap","reason":"should not be used"}', + }); + assert.ok(["opus", "fable"].includes(up.key), `raised to ${up.key}`); + assert.equal(up.provenance.path, "llm-raised"); + assert.equal(up.llm.provider, "jev"); + assert.equal(up.llm.confidence, 0.9); +}); + +test("routeTask (llm on): a Jev miss falls back to the text proposer, then to deterministic", () => { + const root = fixture(); + const viaText = routeTask(root, "write a function to check if a number is prime", { + llm: true, + jevCall: () => { + throw new Error("jev down"); + }, + run: () => '{"band":"premium","reason":"text fallback"}', + }); + assert.equal(viaText.llm.provider, "text"); + assert.equal(viaText.provenance.path, "llm-raised"); + const det = routeTask(root, "fix a typo", { + llm: true, + jevCall: () => { + throw new Error("jev down"); + }, + run: () => "not json", + }); + assert.equal(det.llm, null); + assert.equal(det.provenance.path, "deterministic"); +}); + +test("assessTaskJev: batched dimension nouls become completeness + missing, no invented questions", () => { + const { state, questions } = buildAssumptionNouls("fix the bug"); + assert.deepEqual(Object.keys(questions).sort(), [ + "constraints", + "inputs_outputs", + "success_criteria", + "target_scope", + ]); + assert.ok(typeof state === "string" && state.includes("fix the bug")); + const p = assessTaskJev("fix the bug", { + llm: true, + call: () => ({ + answers: { + inputs_outputs: { type: "noul", noul: 0.2 }, + target_scope: { type: "noul", noul: 0.9 }, + success_criteria: { type: "noul", noul: 0.3 }, + constraints: { type: "noul", noul: 0.8 }, + }, + }), + }); + assert.ok( + Math.abs(p.completeness - 0.55) < 1e-9, + `mean of the four nouls, got ${p.completeness}`, + ); + assert.deepEqual(p.missing.sort(), ["inputs_outputs", "success_criteria"]); + assert.deepEqual(p.questions, [], "System One judges; it does not author clarifying questions"); + assert.equal(p.provider, "jev"); + // A dimension the API dropped → the whole reading fails safe. + assert.equal( + assessTaskJev("x", { + llm: true, + call: () => ({ answers: { inputs_outputs: { type: "noul", noul: 0.5 } } }), + }), + null, + ); +}); + +test("preflightRepo (llm on): the Jev reading flows through reconcileAssumption with provenance", () => { + const root = fixture(); + const r = preflightRepo(root, "fix the login bug", { + allowBuild: false, + llm: true, + jevCall: () => ({ + answers: { + inputs_outputs: { type: "noul", noul: 0.9 }, + target_scope: { type: "noul", noul: 0.9 }, + success_criteria: { type: "noul", noul: 0.9 }, + constraints: { type: "noul", noul: 0.9 }, + }, + }), + }); + assert.equal(r.assumption.provenance.provider, "jev"); + // Verify-don't-trust: the Jev reading (0.9) is bounded to within ±band of the + // deterministic completeness — it lifts, but can never flip the gate on its own. + const det = preflightRepo(root, "fix the login bug", { allowBuild: false, llm: false }); + assert.ok( + r.assumption.completeness > det.assumption.completeness, + `lifts past deterministic ${det.assumption.completeness}, got ${r.assumption.completeness}`, + ); + assert.ok( + r.assumption.completeness <= det.assumption.completeness + 0.25 + 1e-9, + "but never beyond the reconcile band", + ); +}); diff --git a/test/substrate.test.js b/test/substrate.test.js index 4fe7877..4f1c8fa 100644 --- a/test/substrate.test.js +++ b/test/substrate.test.js @@ -110,8 +110,10 @@ test("substrateCheck (llm off by default): provenance is deterministic across fa test("substrateCheck (llm on, explicit): opt-in flag threads through and stays fail-safe", () => { const root = repo(); - // No `run` injection reaches the real CLI here, but the substrate must not throw and must - // still return a coherent contract regardless of whether the CLI exists. + // substrateCheck has no `run` seam — the faculties build their own runner, so with a real + // `claude` on PATH this DID shell out for real (85s of live model calls). test/_setup.js + // sets FORGE_LLM_HTTP=1, forcing the keyless HTTP branch that throws synchronously. Either + // way the substrate must not throw and must still return a coherent contract. const r = substrateCheck(root, "Update computeTax in math.js", { llm: true }); assert.equal(r.llm.enabled, true); assert.ok(r.route.model.id, "still returns a routed model");