diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e7126f77..4b3eba85 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -279,19 +279,37 @@ runs `bump.mjs auto`: it releases only when a `feat`/`fix`/`perf`/breaking commi when none exist, and exits `3` (a clean skip, not a failure) otherwise — so releases cut themselves without a chore/docs merge spamming the registry. -**Custom-gateway model remap (`src/gateway_model_map.js`).** The tier table (`model_tiers.json`) -pins public Anthropic IDs, but a self-hosted LiteLLM/proxy gateway serves its own model names, so a -stock ID sent verbatim 404s. When a non-default gateway base URL is configured, the module fetches -`GET /v1/models` **once per process** (a spawned-node child with the key in env, never argv — the -`llm.js` pattern) and scores each advertised id against every tier's family: the family word -(haiku/sonnet/opus/fable) is a hard gate, the `setOverlap` coefficient of the tier's name tokens -picks the best match, ties break toward the id closest to the canonical name. `resolveModel` -(providers) and `buildRunner` (adjudicate) consult it only when the resolved id is a _stock_ ID — -an explicit `.forge/providers.json` alias or `ANTHROPIC_MODEL` override is never touched — and it -fails safe to the stock ID on no gateway / unreachable `/v1/models` / no family match, so direct -`api.anthropic.com` users are byte-identical. `forge doctor`'s **gateway models** row prints the -resolved `tier→model` mapping for verification. The `MODELS` export shape is unchanged: this is a -resolution-time layer, not a table edit. +**Runtime model resolution (`src/model_catalog.js`, `src/http_cache.js`).** A tier names a +model _family_; `model_tiers.json` keeps a snapshot of ids and prices only as data of last +resort (its `pricingVerified` date still drives `forge doctor`'s staleness warning). The concrete +id is resolved where one is needed — `buildRunner` (adjudicate, on the runner's first call, so +building a runner stays free on the hook path), `emitGatewayConfig`, `estimateSpendFromLogs`, +`forge route`, `forge models` — by `resolveTierModel`: the newest model of the family in the +active provider's live catalog (the Anthropic Models API with `ANTHROPIC_API_KEY`, following +`has_more`/`last_id` → `after_id`; a custom gateway's `/v1/models`; OpenRouter's list), where +"of the family" is a whole-token match of the family word on id or display name and "newest" +is the catalog's `created_at` (then the parsed version, the `YYYYMMDD` stamp, the plainest id). +`resolveTierPrice` / `resolveModelPrice` price an id from OpenRouter's public catalog +(per-token strings → per million, ids matched by canonical token set), then the snapshot row, +the router registry, and the family's tier — an id nothing prices is reported, never billed at a +guessed rate. Each step runs only when the previous is unavailable; an explicit +`.forge/providers.json` id or `ANTHROPIC_MODEL` override is never replaced. Fetches go through a +small private HTTP cache under `.forge/cache/` (self-gitignored, and listed in +`.forge/.gitignore`): freshness comes only from the response (`Cache-Control: max-age`, +`Expires`, `Age`, `Date`), everything else is revalidated with `If-None-Match` / +`If-Modified-Since`, and a failed request serves the stored copy as stale — the snapshot is older +still. The transport is a spawned-node child (the `llm.js` pattern: synchronous, headers on +stdin, never argv), 3 s timeout, never throws; `FORGE_NO_CATALOG_FETCH=1` turns it off. + +**Custom-gateway model remap (`src/gateway_model_map.js`).** A self-hosted LiteLLM/proxy gateway +serves its own model names, so a snapshot ID sent verbatim 404s. When a non-default gateway base +URL is configured, the module reads its `GET /v1/models` through the same catalog fetcher (once +per process) and maps each tier onto the newest advertised model of its family — the same +`newestInFamily` rule; the `setOverlap` score against the tier's name tokens is still reported +with each pick. It fails safe to the snapshot ID on an unreachable `/v1/models` / no family +match. `forge doctor`'s **gateway models** row prints the resolved `tier→model` mapping for +verification. The `MODELS` export shape is unchanged: resolution is a layer over the table, not +an edit of it. **Typed proposers via TypeSafe System One (`src/jev.js`).** Two of the substrate's proposer judgments are not text-generation tasks at all: `route`'s complexity band is a classification @@ -620,8 +638,8 @@ from the tree it describes. ```mermaid %%{init: {'theme':'base','themeVariables':{'primaryColor':'#201a15','primaryTextColor':'#f2ede7','primaryBorderColor':'#372c22','lineColor':'#f26430','secondaryColor':'#272019','tertiaryColor':'#171310','edgeLabelBackground':'#201a15','clusterBkg':'#171310','clusterBorder':'#4a3b2e','fontFamily':'ui-sans-serif, system-ui, sans-serif','fontSize':'14px'},'flowchart':{'curve':'basis','padding':10,'nodeSpacing':36,'rankSpacing':44}}}%% flowchart LR - test["test
117 files"] - src["src
109 files"] + test["test
121 files"] + src["src
111 files"] landing["landing
61 files"] research["research
37 files"] global["global
5 files"] @@ -629,7 +647,7 @@ flowchart LR scripts["scripts
2 files"] docs["docs
1 file"] examples["examples
1 file"] - test -- 240 --> src + test -- 244 --> src bench -- 8 --> src examples -- 4 --> src test -- 2 --> bench diff --git a/CHANGELOG.md b/CHANGELOG.md index 074fe0d6..764eb9d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,75 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **Model tiers resolve to the newest live model instead of pinned ids.** A tier now names a + model family (haiku / sonnet / opus / fable), and wherever a concrete id or price is needed + (the LLM runner, `forge route gateway`, the cost estimate, `forge route`, `forge models`) it + is resolved at that moment: + - **Model id:** the newest model of the family in the active provider's live catalog: the + Anthropic Models API (`GET /v1/models` with `ANTHROPIC_API_KEY`, all pages), a custom + gateway's `/v1/models`, or OpenRouter's list. Family membership is the family word as a + whole token of the id or display name, and "newest" is the catalog's own `created_at`. + No model id is written in the code. Direct-API users were never resolved before: the + gateway remap skipped `api.anthropic.com`. + - **Price:** OpenRouter's public catalog, converted from USD per token to per million and + matched to the resolved id by its tokens (`claude-opus-4-8` ↔ `anthropic/claude-opus-4.8`). + - **Fallback:** each step runs only when the previous one is unavailable (no key, offline, + timeout, non-2xx, no family match). Next comes the last cached catalog response, then the + snapshot in `src/model_tiers.json`, which keeps its `pricingVerified` date and the + `forge doctor` staleness warning. Explicit ids in `.forge/providers.json` and + `ANTHROPIC_MODEL` still win. + - **Caching:** catalogs are cached under `.forge/cache/`, which ignores itself in git and is + listed in `.forge/.gitignore`. Freshness comes only from the response (`Cache-Control`, + `Expires`, `Age`); otherwise the next use revalidates with `If-None-Match` / + `If-Modified-Since`. Lookups time out after 3 s and never throw. The per-prompt hooks never + resolve anything, and the runner resolves on its first call, not when it is built. + `FORGE_NO_CATALOG_FETCH=1` keeps every lookup offline. + - New exports from `model-tiers`: `resolveTierModel`, `resolveTierPrice`, + `resolveModelPrice`, `resolveTiers` and `describeResolution`. The existing exports are + unchanged. +- **`forge models`** prints what every tier resolves to right now: family, model id, created + date, price, and where the id and the price came from (`--json` for the full resolution). + +### Security + +- **A catalog can no longer write anything but a model id into the generated gateway config.** + Ids and display names come from a live catalog, and `forge route gateway` writes a file the + user feeds to LiteLLM as routing config. Two layers now stand between them: + - **At the boundary:** a catalog row whose id is not id-shaped (whitespace, control + characters, over 200 characters) is dropped in `normalizeCatalogPage`, so it is never + resolved, written to config, or passed to a model call. A display name is kept as one + printable line. + - **At the emitter:** every catalog-sourced value is a quoted YAML scalar with control + characters escaped, and comments are collapsed to one line. + + Without this, a display name carrying a newline could add a second entry for a tier alias, + and LiteLLM's `simple-shuffle` would then send a share of that tier's prompts to the spliced + model. `test/route.test.js` pins it with a crafted catalog. + +### Changed + +- **`forge route` shows the resolved model id** and where it came from, under the + recommendation. The price reads `live price` when OpenRouter lists that id. `--json` carries + the same resolution as `resolved` and `price` beside the unchanged `model` row, and the MCP + `route_task` names the resolved id too, so no surface reports a different model than another. +- **A catalog held in memory expires when the response says it does.** One lookup per catalog + per process keeps `forge models` from asking once per tier, but the entry now carries the + response's own freshness, so a long-running dashboard or MCP server picks up a new model + instead of holding its first answer until restart. +- **`forge route gateway` / `forge config gateway`** write the resolved ids. They drop the + hard-coded "Models verified 2026-07-05" line: each tier alias carries an `# id:` comment + naming its source. The passthrough list keeps the snapshot id when it differs, so a + client pinned to it still works. An OpenRouter provider now gets real OpenRouter ids + behind `openrouter/`. +- **The custom-gateway remap picks the newest model of each family** the gateway advertises + (its creation time, then the version), not the one closest to the snapshot's version. The + gateway fetch shares the new catalog code; the old separate fetch child is gone. +- **The session-log cost estimate no longer bills unknown models at $3/$15.** Each logged + model is priced from the live catalog, else the snapshot row, the router registry, or its + family's tier. A model nothing prices is listed as unpriced and left out of the total. + ## [1.1.2] - 2026-09-22 ### Added diff --git a/README.md b/README.md index 80436499..eaf387f4 100644 --- a/README.md +++ b/README.md @@ -430,6 +430,7 @@ and output live in [`docs/GUIDE.md`](docs/GUIDE.md). | | `forge stack` | detect this repo's real stack (languages, frameworks, test commands) from its manifests | | | `forge integrations` | opt-in third-party MCP servers (e.g. context7) — add records the managed set and writes only with --yes (--adopt claims a same-name entry); remove reverses it | | | `forge cost` | real per-day spend via ccusage + measured stage factors (--stages) | +| | `forge models` | each tier's model family resolved to a concrete model — newest in the provider's live catalog (else the shipped snapshot), with its price and where both came from | | **Labs (experimental)** | `forge taste` | enable one UI-taste tool for this repo (no arg = list) | | | `forge uicheck` | deterministic UI checks — contrast · fingerprint · design · visual | | | `forge imagine` | consequence simulation — predicted breaks + the minimal dry-run test suite for a task | diff --git a/bench/impact_cases.mjs b/bench/impact_cases.mjs index b2202f7b..8fea2114 100644 --- a/bench/impact_cases.mjs +++ b/bench/impact_cases.mjs @@ -60,7 +60,7 @@ // (test/dash.test.js:69 mentions the name only inside an assertion message — a string, // not a reference — so it is NOT labeled as a dependent.) // -// contentHash (src/util.js) — 10 files. The widest fan-out in the set, and the case that +// contentHash (src/util.js) — 11 files. The widest fan-out in the set, and the case that // used to carry a documented FALSE NEGATIVE: src/atlas.js binds it to an alias, // `const hash = contentHash;` at :187, with no call parentheses, and the old import regex // captured module paths rather than named bindings, so no edge reached atlas.js. That is @@ -73,6 +73,7 @@ // - src/cost_report.js imports it (:14); routeRef() calls it (:212) // - src/diagnose.js imports it (:15); failureSignature() calls it (:57) // - src/embed.js imports it (:35) and calls it (:202) +// - src/http_cache.js imports it (:18); cacheFile() calls it (:124) // - src/ledger.js imports it (:18) and calls it (:136, :141, :947, :962, :963) // - src/ledger_store.js imports it (:43) and calls it (:414, :598) // - src/reuse.js imports it (:15) and calls it (:94, :116, :400) @@ -132,6 +133,7 @@ export const IMPACT_CASES = [ "src/cost_report.js", "src/diagnose.js", "src/embed.js", + "src/http_cache.js", "src/ledger.js", "src/ledger_store.js", "src/reuse.js", diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 172f6cff..9f919473 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -32,7 +32,7 @@ Every command is real and wired. Grouped by what it does: | **Substrate** | `forge substrate` · `forge preflight` · `forge impact` · `forge scope` · `forge context` · `forge route` · `forge verify` · `forge precommit` | | **Memory** | `forge cortex` · `forge recall` · `forge remember` · `forge brain` · `forge ledger` · `forge handoff` · `forge decide` · `forge know` | | **Quality** | `forge scan` · `forge spec` · `forge harden` · `forge radar` | -| **Config** | `forge brand` · `forge atlas` · `forge stack` · `forge integrations` · `forge cost` | +| **Config** | `forge brand` · `forge atlas` · `forge stack` · `forge integrations` · `forge cost` · `forge models` | | **Labs (experimental)** | `forge taste` · `forge uicheck` · `forge imagine` · `forge lean` · `forge anchor` · `forge diagnose` · `forge dash` · `forge report` · `forge deja` · `forge reuse` · `forge rank` · `forge collide` | @@ -199,22 +199,77 @@ churn, past-mistake density, ambiguity). Whichever facet detects difficulty sets ```console $ forge route "write an is_prime function" - → Haiku 4.5 (simple, $1/$5 per M tok) + → Haiku 4.5 (simple, $1/$5 per M tok, current effective) + model: claude-haiku-4-5-20251001 — shipped snapshot, pricing verified 2026-09-22 (no ANTHROPIC_API_KEY for the Models API) lint, formatting, docs, stubs, trivial well-defined edits driven by: similar to "check if a number is prime" (sim 1.00, complexity 0.08) $ forge route "design and implement a distributed rate limiter with sliding windows across 3 services" - → Opus 4.8 (complex, $5/$25 per M tok) + → Opus 4.8 (complex, $5/$25 per M tok, current effective) + model: claude-opus-4-8 — shipped snapshot, pricing verified 2026-09-22 (no ANTHROPIC_API_KEY for the Models API) architecture, cross-module refactor, novel algorithms, multi-layer debugging complexity 0.73 · driven by: similar to "implement a rate limiter with a token bucket" (sim 0.43, complexity 0.78) ``` +The recommendation is a **tier**, and a tier names a model **family** (haiku / sonnet / opus / +fable). The `model:` line is that family resolved to a concrete id at the moment you ask — +`forge models` (below) explains how. With `ANTHROPIC_API_KEY` set it reads +`newest opus in the api.anthropic.com catalog, created …`, and the price says `live price` +when OpenRouter lists that id. + Unseen phrasings route by resemblance — "two threads deadlock when the queue is full" lands in the concurrency neighborhood without any keyword list needing the literal token "race condition". To tune routing, add labeled rows to `EXEMPLARS` (data, not weights). `ANTHROPIC_MODEL` / `FORGE_MODEL` override the tier choice entirely. Run `forge route gateway` to emit a LiteLLM config so the routing happens automatically. +Its tier aliases point at the same resolved ids, and each alias carries an `# id:` comment +saying where its id came from (the live catalog, or the shipped snapshot and why). + +### `forge models` — what each tier resolves to + +Forge ships no model id it depends on. `src/model_tiers.json` names each tier's **family** +and keeps a snapshot of ids and prices (with its `pricingVerified` date) as data of last +resort. Wherever a concrete id or price is actually needed — the LLM runner, the gateway +config, the cost estimate, `forge route`, `forge models` — it is resolved at that moment: + +1. **Model id** — the newest model of the family in the active provider's **live catalog**: + the Anthropic Models API (`GET /v1/models`, all pages; needs `ANTHROPIC_API_KEY`), a custom + gateway's `/v1/models`, or OpenRouter's list for an OpenRouter provider. "Of the family" + means the family word is a whole token of the id or display name; "newest" is the + catalog's own `created_at` (then the parsed version, for catalogs without dates). No model + id is listed anywhere in the code. +2. **Price** — OpenRouter's public catalog (`GET https://openrouter.ai/api/v1/models`, no key), + converted from USD per token to per million and matched to the resolved id by its tokens + (`claude-opus-4-8` ↔ `anthropic/claude-opus-4.8`). +3. **Fallback**, each step only when the previous one is unavailable (no key, offline, timeout, + non-2xx, no family match): the last cached catalog response, then the shipped snapshot. + An id configured explicitly (`.forge/providers.json`, a gateway alias) is used as-is, and + `ANTHROPIC_MODEL` / `FORGE_MODEL` still pin every call. + +```console +$ forge models + provider anthropic (Anthropic (direct)) + + tier family model created $/M tok id from price from + simple haiku claude-haiku-4-5-20251001 — $1/$5 snapshot snapshot + medium sonnet claude-sonnet-5 — $2/$10 snapshot snapshot + complex opus claude-opus-4-8 — $5/$25 snapshot snapshot + extreme fable claude-fable-5 — $10/$50 snapshot snapshot + + haiku shipped snapshot, pricing verified 2026-09-22 (no ANTHROPIC_API_KEY for the Models API) + … +``` + +With a key, `id from` reads `catalog`, `created` is the catalog's date, and the provenance +line names the catalog (`newest opus in the api.anthropic.com catalog, created …`). `--json` +prints the full resolution. Catalog responses are cached under `.forge/cache/` (git-ignored), +and **freshness comes from the response itself**: a cached response is reused without a +request only while its `Cache-Control: max-age` / `Expires` says it is fresh; otherwise the +next use revalidates it with `If-None-Match` / `If-Modified-Since`, and when that request +fails the cached copy is used (reported as the last cached copy). Forge has no TTL of its own. +Lookups time out after 3 s and never throw, and the per-prompt hooks never resolve anything. +`FORGE_NO_CATALOG_FETCH=1` keeps every lookup offline (the cache, then the snapshot). **`forge route calibrate`** is the _advisory → gated promotion_ (overview §4): it fits an affine correction of the rubric's score toward a held-out split of a labelled fixture and @@ -273,17 +328,17 @@ Corporate gateway environments work out of the box: with `ANTHROPIC_BASE_URL` + `ANTHROPIC_AUTH_TOKEN` set (LiteLLM-style gateways), detection classifies the gateway, auth uses the token as a Bearer credential, and `ANTHROPIC_MODEL` pins the model. -**Custom gateways that rename models.** The tier table ships public Anthropic IDs +**Custom gateways that rename models.** The tier snapshot carries public Anthropic IDs (`claude-haiku-4-5-…`, `claude-sonnet-5`, …), but a self-hosted gateway often serves its own names (`bedrock-claude-haiku`, `prod-sonnet-5`). When a non-default gateway base URL is -set, Forge asks it once per process (`GET /v1/models`) and scores each advertised model -against every tier's family — the family word (haiku/sonnet/opus/fable) gates the match, the -overlap score picks the best id — then remaps each tier onto a real gateway model. It is a -silent, low-configuration fallback: no gateway, an unreachable `/v1/models`, or no family match and -the stock IDs are used unchanged; direct `api.anthropic.com` sessions never probe. An explicit -model in `.forge/providers.json` (or `ANTHROPIC_MODEL`) always wins over the remap. `forge -doctor` prints the resolved `tier→model` mapping under **gateway models** so you can verify it -and pin explicit IDs if a family scored wrong. +set, Forge asks it (`GET /v1/models`) and maps each tier onto the **newest** model of that +tier's family the gateway advertises — the rule `forge models` applies to the Anthropic +Models API: the family word (haiku/sonnet/opus/fable) gates the match, then the gateway's +creation time, the parsed version and the plainest id decide. It is a silent, +low-configuration fallback: an unreachable `/v1/models` or no family match and the snapshot +IDs are used unchanged. An explicit model in `.forge/providers.json` (or `ANTHROPIC_MODEL`) +always wins over the remap. `forge doctor` prints the resolved `tier→model` mapping under +**gateway models** so you can verify it and pin explicit IDs if a family matched wrong. ### `forge impact ` — what will this edit break? @@ -1495,7 +1550,7 @@ Create `global/crew/.md` with frontmatter. It installs into `~/.claude/age | how often it asks | `source/substrate.json` → `defaults.askThreshold` (0.6) | | blast-radius sensitivity | `source/substrate.json` → `defaults.impactThreshold` (0.1) | | a routing outcome | `src/route.js` → add a labeled row to `EXEMPLARS` (data, not weights); constants in `RUBRIC` | -| model tiers / prices | `src/model_tiers.js` | +| model tiers / prices | `src/model_tiers.json` (tier → family, plus the snapshot of last resort); runtime resolution in `src/model_catalog.js` — `forge models` shows it | | an assumption question | `src/preflight.js` → `DIMENSIONS[]` | | the verify checklist | `src/substrate.js` → `verificationChecklist()` | | when the ambient hook speaks | `src/substrate.js` → `substrateContext()` | @@ -1604,6 +1659,7 @@ code reads but this table misses fails CI on the forge repo): | `FORCE_COLOR` | forces CLI color on even when piped, e.g. in CI (`0` forces off) — takes precedence over `NO_COLOR` | | `TERM` / `COLORTERM` | `TERM=dumb` disables color; `COLORTERM=truecolor`/`24bit` upgrades to the brand palette's 24-bit hues | | `FORGE_NO_UPDATE_CHECK` | `1` silences the `forge doctor` update notice | +| `FORGE_NO_CATALOG_FETCH` | `1` never fetches a model catalog (Anthropic Models API, gateway `/v1/models`, OpenRouter prices): ids and prices come from the `.forge/cache/` copy, else the shipped snapshot — for air-gapped machines | | `FORGE_DEBUG` | `1` writes fail-safe error details to stderr instead of swallowing them | --- diff --git a/mintlify/cli/config.mdx b/mintlify/cli/config.mdx index 7100ba7c..52b462f4 100644 --- a/mintlify/cli/config.mdx +++ b/mintlify/cli/config.mdx @@ -1,6 +1,6 @@ --- title: "Config commands" -description: "Providers, cost, dashboards, brand, atlas, stack, and opt-in MCP integrations: config, cost, dash, brand, atlas, stack, report, tools, integrations." +description: "Providers, cost, model resolution, dashboards, brand, atlas, stack, and opt-in MCP integrations: config, cost, models, dash, brand, atlas, stack, report, tools, integrations." --- The Config group covers providers, observability, the code graph, and stack detection. @@ -29,6 +29,25 @@ forge cost --stages # measured per-stage cost factors default. +## `forge models` + +Each tier's model family resolved to a concrete model right now: the newest model of the +family in the active provider's live catalog (the Anthropic Models API with +`ANTHROPIC_API_KEY`, a gateway's `/v1/models`, or OpenRouter's list), priced from +OpenRouter's public catalog — else the cached copy, else the shipped snapshot. Every row says +where its id and price came from. + +```bash +forge models # tier, family, resolved id, created, price, source +forge models --json # the full resolution +``` + + + Catalogs are cached under `.forge/cache/` and refreshed by the response's own + `Cache-Control` / `Expires` / `ETag` — forge has no TTL of its own. + `FORGE_NO_CATALOG_FETCH=1` keeps every lookup offline. + + ## `forge dash` Local dashboard over the ledger, metrics, and blast radius. Read-mostly: the only writes diff --git a/mintlify/concepts/model-routing.mdx b/mintlify/concepts/model-routing.mdx index f9d7dd1d..8f0e9dc2 100644 --- a/mintlify/concepts/model-routing.mdx +++ b/mintlify/concepts/model-routing.mdx @@ -1,10 +1,10 @@ --- title: "Model routing" -description: "A deterministic, diffable rubric picks the cheapest capable model tier before dispatch — with a fail-safe remap for self-hosted gateways." +description: "A deterministic, diffable rubric picks the cheapest capable model tier before dispatch — each tier a model family, resolved to the newest live model with a fail-safe fallback." --- Forge recommends the cheapest capable model for a task **before** dispatch, from a -deterministic rubric you can read in the repo (`src/model_tiers.json`). Unlike a gateway +deterministic rubric you can read in the repo (`src/route.js`). Unlike a gateway that decides inside the proxy at request time, the routing decision is visible and diffable in git. @@ -47,37 +47,47 @@ intent by the same exemplar k-NN estimator. Note the two use different stop-sets treats generic verbs (`fix` / `add` / `build`) as complexity noise, but those verbs are exactly the intent signal. -## The tier table +## Tiers name families; ids are resolved -The tier table (`src/model_tiers.json`) pins public Anthropic model IDs by family -(haiku / sonnet / opus / fable). Doc prices are reconciled against this file by the docs -check, so prose and the table can't drift. - -## Self-hosted gateway remap - -A self-hosted LiteLLM or proxy gateway serves its own model names, so a stock ID sent -verbatim would 404. When a non-default gateway base URL is configured, Forge -(`src/gateway_model_map.js`) fetches `GET /v1/models` **once per process** and scores -each advertised id against every tier's family: +Each tier names a model **family** (haiku / sonnet / opus / fable). The concrete id and price +are resolved where they are needed — the LLM runner, the gateway config, the cost estimate, +`forge route`, `forge models` — never on the per-prompt hook path: - - The family word (haiku / sonnet / opus / fable) must match — it is a hard gate. + + The active provider's own model list — the Anthropic Models API (`GET /v1/models`, with + `ANTHROPIC_API_KEY`), a custom gateway's `/v1/models`, or OpenRouter's list. The family + word must be a whole token of the id or display name; "newest" is the catalog's + `created_at`. No model id is listed in the code. - - Among the family, the `setOverlap` coefficient of the tier's name tokens picks the - best match. + + OpenRouter's public catalog, per-token prices converted to per million, matched to the + resolved id by its tokens (`claude-opus-4-8` ↔ `anthropic/claude-opus-4.8`). - - Ties break toward the id closest to the canonical name. + + Only when the previous step is unavailable: the last cached catalog response, then the + shipped snapshot in `src/model_tiers.json` (with its `pricingVerified` date). Doc prices + are reconciled against that snapshot by the docs check. +`forge models` prints what every tier resolves to and where each id and price came from. +Catalogs are cached under `.forge/cache/`, refreshed only by what the response's own +`Cache-Control` / `Expires` / `ETag` allow; `FORGE_NO_CATALOG_FETCH=1` keeps it offline. + +## Self-hosted gateway remap + +A self-hosted LiteLLM or proxy gateway serves its own model names, so a snapshot ID sent +verbatim would 404. When a non-default gateway base URL is configured, Forge +(`src/gateway_model_map.js`) reads its `GET /v1/models` and maps each tier onto the newest +advertised model of its family — the same rule as above: the family word is a hard gate, +then the gateway's creation time, the parsed version, and the id closest to the canonical +name decide. + - The remap consults the gateway **only** when the resolved id is a _stock_ ID — an - explicit `.forge/providers.json` alias or an `ANTHROPIC_MODEL` override is never - touched. It fails safe to the stock ID on no gateway, an unreachable `/v1/models`, or - no family match, so direct `api.anthropic.com` users are byte-identical. + An explicit `.forge/providers.json` alias or an `ANTHROPIC_MODEL` override is never + touched. The remap fails safe to the snapshot ID on an unreachable `/v1/models` or no + family match. `forge doctor`'s **gateway models** row prints the resolved `tier → model` mapping for diff --git a/src/adjudicate.js b/src/adjudicate.js index fb543de1..b84ed5f9 100644 --- a/src/adjudicate.js +++ b/src/adjudicate.js @@ -11,9 +11,8 @@ // - ZERO-DEP. Access is a `claude -p` CLI shell-out; the runner is injectable so the pure // prompt/parse/verify logic is fully testable without the CLI or the network. import { execFileSync, spawnSync } from "node:child_process"; -import { gatewayModelId } from "./gateway_model_map.js"; import { buildHttpRunner as httpRunner } from "./llm.js"; -import { MODELS } from "./model_tiers.js"; +import { MODELS, resolveTierModel } from "./model_tiers.js"; import { envModelOverride } from "./providers.js"; import { hasSecret } from "./secrets.js"; @@ -45,19 +44,36 @@ function hasClaude() { return _claudeAvail; } +/** + * The concrete model id a runner sends for `model`. An ANTHROPIC_MODEL/FORGE_MODEL override is + * honored verbatim, and so is a literal id; a tier key resolves to the newest model of its family + * in the catalog the environment reaches (a custom gateway's /v1/models, else the Anthropic Models + * API with ANTHROPIC_API_KEY), falling back to the shipped snapshot id (model_tiers). + * @param {string} model tier key (haiku/sonnet/opus/fable) or a literal model id + * @param {{root?: string|null, fetchImpl?: Function, env?: Record}} [opts] + * test seams for the catalog lookup + * @returns {string} + */ +export function runnerModel(model, { root = process.cwd(), fetchImpl, env } = {}) { + const override = envModelOverride(); + if (override) return override; + if (!MODELS[model]) return model; + return resolveTierModel(model, { root, fetchImpl, env })?.id ?? MODELS[model].id; +} + /** Build an injectable LLM runner. Tries direct HTTP when `claude` CLI is unavailable - * or when FORGE_LLM_HTTP=1. Falls back to `claude -p` otherwise. */ + * or when FORGE_LLM_HTTP=1. Falls back to `claude -p` otherwise. The model id is resolved on + * the FIRST call, not here: building a runner stays free on the hook path, and the catalog + * lookup rides along with the (much slower) model call it precedes. */ export function buildRunner({ model = "haiku", timeoutMs = 20000 } = {}) { - const override = envModelOverride(); - const stock = override || MODELS[model]?.id || model; - // A forced override is honored verbatim; otherwise remap the tier's stock id onto a custom - // gateway's real model when one is advertised (no-op for direct Anthropic — see gateway_model_map). - const resolvedModel = override ? stock : gatewayModelId(model, stock); + /** @type {string|null} */ + let resolved = null; + const modelId = () => (resolved ??= runnerModel(model)); if (process.env.FORGE_LLM_HTTP === "1" || !hasClaude()) { - return httpRunner({ model: resolvedModel, timeoutMs }); + return (prompt) => httpRunner({ model: modelId(), timeoutMs })(prompt); } return (prompt) => - execFileSync("claude", ["-p", "--model", resolvedModel], { + execFileSync("claude", ["-p", "--model", modelId()], { input: prompt, encoding: "utf8", timeout: timeoutMs, diff --git a/src/cli.js b/src/cli.js index 27d6fb5b..625a0f90 100755 --- a/src/cli.js +++ b/src/cli.js @@ -1527,7 +1527,7 @@ HANDLERS.cost = async (argv) => { console.log(out.trim()); } catch { const { estimateSpendFromLogs } = await import("./cost_report.js"); - const est = estimateSpendFromLogs(); + const est = estimateSpendFromLogs({ root: process.cwd() }); if (est && est.totalCost > 0) { console.log( ` $${est.totalCost.toFixed(2)} estimated from Claude session logs (${est.sessions} session(s))`, @@ -1535,9 +1535,16 @@ HANDLERS.cost = async (argv) => { if (est.byModel.length) { for (const m of est.byModel) console.log( - ` ${m.model.padEnd(30)} $${m.cost.toFixed(4)} (${m.inTokens} in / ${m.outTokens} out)`, + ` ${m.model.padEnd(30)} ${m.priced ? `$${m.cost.toFixed(4)}` : "unpriced"} (${m.inTokens} in / ${m.outTokens} out)${m.priceSource ? ` · price: ${m.priceSource}` : ""}`, ); } + if (est.unpriced?.length) + console.log( + paint( + ` not in the total — no catalog or snapshot price for: ${est.unpriced.join(", ")}`, + "dim", + ), + ); console.log(paint("\n install ccusage for precise tracking: npm i -g ccusage", "dim")); } else { console.log( @@ -1553,6 +1560,61 @@ HANDLERS.cost = async (argv) => { ); return; }; +HANDLERS.models = async (argv) => { + // What each tier resolves to RIGHT NOW: family → newest model in the active provider's live + // catalog (else the shipped snapshot), priced from OpenRouter's catalog (else the snapshot). + const { describeResolution, PRICING_VERIFIED, resolveTiers } = await import("./model_tiers.js"); + const { activeProvider, envModelOverride } = await import("./providers.js"); + const root = process.cwd(); + const provider = activeProvider(root); + const tiers = resolveTiers({ root, provider }); + const override = envModelOverride(); + if (argv.includes("--json")) + return console.log( + JSON.stringify( + { provider: provider.name, override, pricingVerified: PRICING_VERIFIED, tiers }, + null, + 2, + ), + ); + heading(`${BRAND.brand} models — each tier's family, resolved to a concrete model\n`); + console.log(` provider ${provider.name} (${provider.label || provider.name})`); + if (override) + console.log( + ` override ${override} — ANTHROPIC_MODEL/FORGE_MODEL pins every call; the tiers below apply without it`, + ); + const priceText = (p) => (p ? `$${p.inCost}/$${p.outCost}` : "—"); + const priceFrom = (p) => + !p + ? "unpriced" + : p.source === "catalog" + ? "catalog" + : `snapshot${p.basis === "family" ? " (tier)" : ""}`; + const width = Math.max(28, ...tiers.map((t) => (t.model?.id ?? "").length + 2)); + console.log( + `\n ${"tier".padEnd(8)} ${"family".padEnd(7)} ${"model".padEnd(width)} ${"created".padEnd(11)} ${"$/M tok".padEnd(10)} ${"id from".padEnd(9)} price from`, + ); + for (const t of tiers) { + console.log( + ` ${t.class.padEnd(8)} ${t.family.padEnd(7)} ${(t.model?.id ?? "—").padEnd(width)} ${(t.model?.createdAt?.slice(0, 10) ?? "—").padEnd(11)} ${priceText(t.price).padEnd(10)} ${(t.model?.source ?? "—").padEnd(9)} ${priceFrom(t.price)}`, + ); + } + console.log(""); + for (const t of tiers) console.log(` ${t.family.padEnd(7)} ${describeResolution(t.model)}`); + const live = tiers.find((t) => t.price?.source === "catalog")?.price; + console.log( + live + ? `\n prices: live from ${live.catalog}${live.cache === "stale" ? " (last cached copy — catalog unreachable)" : ""}` + : `\n prices: shipped snapshot, verified ${PRICING_VERIFIED} (OpenRouter's catalog unavailable or unlisted)`, + ); + console.log( + paint( + " cache: .forge/cache/ — reused while the response's own Cache-Control/Expires says fresh, else revalidated (ETag); FORGE_NO_CATALOG_FETCH=1 stays offline", + "dim", + ), + ); + return; +}; HANDLERS.spec = async (argv) => { const s = await import("./speclock.js"); const sub = argv[1] || "check"; @@ -2018,18 +2080,29 @@ HANDLERS.route = async (argv) => { } const rec = r.routeTask(process.cwd(), task); r.meterRoute(process.cwd(), task, rec); + // The recommendation is a tier (a model family); its concrete id and price are resolved here, + // in the command — routeTask stays network-free because the hooks run it. BOTH output modes + // resolve, so a script reading --json never sees a different model than the text prints. + const { describeResolution, resolveTierModel, resolveTierPrice } = await import( + "./model_tiers.js" + ); + const { activeProvider } = await import("./providers.js"); + const opts = { root: process.cwd(), provider: activeProvider(process.cwd()) }; + const resolved = resolveTierModel(rec.key, opts); + const price = resolveTierPrice(rec.key, { ...opts, resolved }); if (json) { - console.log(JSON.stringify(rec, null, 2)); + // `model` stays the snapshot row (its shape is public); `resolved` is what would be called. + console.log(JSON.stringify({ ...rec, resolved, price }, null, 2)); } else { heading(`${BRAND.brand} route — cheapest capable model\n`); - const { priceOf } = await import("./model_tiers.js"); - const price = priceOf(rec.key) || { - inCost: rec.model.inCost, - outCost: rec.model.outCost, - }; + const name = + resolved?.source === "catalog" && resolved.displayName + ? resolved.displayName + : rec.model.name; console.log( - ` → ${paint(rec.model.name, "accent")} (${rec.tier}, $${price.inCost}/$${price.outCost} per M tok, current effective)`, + ` → ${paint(name, "accent")} (${rec.tier}, ${price ? `${price.inCost}/${price.outCost} per M tok, ${price.source === "catalog" ? "live price" : "current effective"}` : "price unknown"})`, ); + if (resolved) console.log(` model: ${resolved.id} — ${describeResolution(resolved)}`); console.log(` ${rec.model.use}`); console.log( ` complexity ${bar(rec.score, 8)} ${rec.score.toFixed(2)}${rec.reasons.length ? ` · driven by: ${rec.reasons.join(", ")}` : ""}`, diff --git a/src/commands.js b/src/commands.js index 8bd433f0..111f9a7f 100644 --- a/src/commands.js +++ b/src/commands.js @@ -83,6 +83,13 @@ export const COMMANDS = { remember: "add a durable fact to this repo's portable memory (forge brain)", brain: "show / rebuild the portable project memory index", cost: "real per-day spend via ccusage + measured stage factors (--stages)", + models: { + summary: + "each tier's model family resolved to a concrete model — newest in the provider's live catalog (else the shipped snapshot), with its price and where both came from", + usage: "forge models [--json]", + flags: [{ flag: "--json", desc: "machine-readable resolution (id, created, price, sources)" }], + examples: ["forge models", "forge models --json"], + }, spec: "spec-as-contract — init (OpenSpec) / lock / check drift", cortex: "self-correcting project memory — status / why ", deja: "anti-repetition — have you done this task before? ranks prior solved/verified sessions", @@ -209,7 +216,7 @@ export const GROUPS = { ], Memory: ["cortex", "recall", "remember", "brain", "ledger", "handoff", "decide", "know"], Quality: ["scan", "spec", "harden", "radar"], - Config: ["brand", "atlas", "stack", "integrations", "cost"], + Config: ["brand", "atlas", "stack", "integrations", "cost", "models"], "Labs (experimental)": [ "taste", "uicheck", diff --git a/src/cortex_mcp.js b/src/cortex_mcp.js index 6b85a4a8..0961c2f8 100644 --- a/src/cortex_mcp.js +++ b/src/cortex_mcp.js @@ -52,7 +52,17 @@ async function callTool(name, args = {}) { } if (name === "route_task") { const rec = routeTask(root, String(args.task ?? "")); - return `Recommended: ${rec.model.name} (${rec.tier}). complexity ${rec.score.toFixed(2)}${rec.reasons.length ? ` — ${rec.reasons.join(", ")}` : ""}.`; + // routeTask names a tier; the concrete id is resolved here, as `forge route` does, so an + // agent is told the model it would actually call rather than the shipped snapshot. + // Best-effort: the resolver falls back to the snapshot when no catalog answers. + let model = rec.model.name; + try { + const { describeResolution, resolveTierModel } = await import("./model_tiers.js"); + const { activeProvider } = await import("./providers.js"); + const resolved = resolveTierModel(rec.key, { root, provider: activeProvider(root) }); + if (resolved) model = `${resolved.id} (${describeResolution(resolved)})`; + } catch {} + return `Recommended: ${model} (${rec.tier}). complexity ${rec.score.toFixed(2)}${rec.reasons.length ? ` — ${rec.reasons.join(", ")}` : ""}.`; } if (name === "assumption_gate") diff --git a/src/cost_report.js b/src/cost_report.js index 832d73b4..6bcb275d 100644 --- a/src/cost_report.js +++ b/src/cost_report.js @@ -10,7 +10,8 @@ import { existsSync, readdirSync, readFileSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; import { read, record } from "./metrics.js"; -import { MODELS } from "./model_tiers.js"; +import { fetchPriceCatalog } from "./model_catalog.js"; +import { MODELS, resolveModelPrice } from "./model_tiers.js"; import { contentHash } from "./util.js"; /** Saving weight per cache-hit tier — must stay consistent with reuse.js savedEstimate @@ -272,20 +273,21 @@ function usageTokens(usage) { /** * Fallback spend estimation from Claude's native JSONL session logs when ccusage * is unavailable. Scans ~/.claude/projects/ for session files and computes cost - * from token counts x model_tiers pricing — uncached input and output at the model's - * rates, cache writes and reads at CACHE_PRICE_RATIO of its input rate. Claude Code - * writes one API response on several lines (one per content block), each repeating the - * same message id and usage, and a resumed session re-logs its history into a new file, + * from token counts x each logged model's price — uncached input and output at the model's + * rates, cache writes and reads at CACHE_PRICE_RATIO of its input rate. A model is priced by + * model_tiers.resolveModelPrice: OpenRouter's live catalog for that exact id, else the shipped + * snapshot (its own row, the router registry's row, then its family's tier). A model nothing + * prices is reported in `unpriced` and left out of the total — never billed at a guessed rate. + * Claude Code writes one API response on several lines (one per content block), each repeating + * the same message id and usage, and a resumed session re-logs its history into a new file, * so each message id is counted once across every file. Best-effort, never throws. + * @param {{root?: string|null, fetchImpl?: Function, date?: string}} [opts] catalog cache root + * (default cwd) and test seams, passed to resolveModelPrice */ -export function estimateSpendFromLogs() { +export function estimateSpendFromLogs({ root = process.cwd(), fetchImpl, date } = {}) { try { const projectsDir = join(homedir(), ".claude", "projects"); if (!existsSync(projectsDir)) return null; - const pricingPerM = {}; - for (const [, m] of Object.entries(MODELS)) { - pricingPerM[m.id] = { inCost: m.inCost, outCost: m.outCost }; - } const byModel = {}; const seen = new Set(); let sessions = 0; @@ -330,19 +332,30 @@ export function estimateSpendFromLogs() { } let totalCost = 0; const modelBreakdown = []; + const unpriced = []; + // One price-catalog lookup for the whole estimate, however many models the logs name. + const priceCatalog = Object.keys(byModel).length + ? fetchPriceCatalog({ root, fetchImpl }) + : null; for (const [model, u] of Object.entries(byModel)) { - const pricing = pricingPerM[model] || { inCost: 3, outCost: 15 }; - const cost = - (u.inTokens * pricing.inCost + - u.outTokens * pricing.outCost + - u.cacheWrite5mTokens * pricing.inCost * CACHE_PRICE_RATIO.write5m + - u.cacheWrite1hTokens * pricing.inCost * CACHE_PRICE_RATIO.write1h + - u.cacheReadTokens * pricing.inCost * CACHE_PRICE_RATIO.read) / - 1_000_000; - totalCost += cost; + const pricing = resolveModelPrice(model, { root, date, priceCatalog }); + const cost = pricing + ? (u.inTokens * pricing.inCost + + u.outTokens * pricing.outCost + + u.cacheWrite5mTokens * pricing.inCost * CACHE_PRICE_RATIO.write5m + + u.cacheWrite1hTokens * pricing.inCost * CACHE_PRICE_RATIO.write1h + + u.cacheReadTokens * pricing.inCost * CACHE_PRICE_RATIO.read) / + 1_000_000 + : 0; + if (pricing) totalCost += cost; + else unpriced.push(model); modelBreakdown.push({ model, cost, + priced: Boolean(pricing), + priceSource: pricing + ? `${pricing.source}${pricing.basis ? `:${pricing.basis}` : ""}` + : null, inTokens: u.inTokens, outTokens: u.outTokens, cacheWriteTokens: u.cacheWrite5mTokens + u.cacheWrite1hTokens, @@ -350,7 +363,7 @@ export function estimateSpendFromLogs() { }); } modelBreakdown.sort((a, b) => b.cost - a.cost); - return { totalCost, sessions, byModel: modelBreakdown }; + return { totalCost, sessions, byModel: modelBreakdown, unpriced }; } catch { return null; } diff --git a/src/dash.js b/src/dash.js index dcfadd6b..2d323556 100644 --- a/src/dash.js +++ b/src/dash.js @@ -121,7 +121,7 @@ export function dashData(root, { nowDay = epochDay() } = {}) { } catch {} let spend = null; try { - spend = estimateSpendFromLogs(); + spend = estimateSpendFromLogs({ root }); } catch {} // First-run signal for the empty-state copy: a truly untouched .forge/ has no // ledger claims AND no metrics events. `metrics.recent` is capped but only ever @@ -543,7 +543,7 @@ export function serve(root, { port = 4242, host = "127.0.0.1" } = {}) { }), ); if (url.pathname === "/api/spend") { - const spend = estimateSpendFromLogs(); + const spend = estimateSpendFromLogs({ root }); return sendJson(res, 200, spend || { totalCost: 0, sessions: 0, byModel: [] }); } if (url.pathname === "/api/impact") { diff --git a/src/gateway_model_map.js b/src/gateway_model_map.js index 23692635..4e0f6189 100644 --- a/src/gateway_model_map.js +++ b/src/gateway_model_map.js @@ -1,30 +1,35 @@ // forge gateway model map — remap complexity tiers onto a CUSTOM gateway's real model IDs. // -// The problem: model_tiers.json pins public Anthropic IDs (claude-haiku-4-5-20251001, …). +// The problem: model_tiers.json's snapshot carries public Anthropic IDs (claude-haiku-4-5-…). // A self-hosted LiteLLM/proxy gateway rarely exposes those exact names — it advertises its // OWN ids (e.g. "bedrock-claude-haiku", "prod-sonnet", "claude-3-5-sonnet-v2"). Sending a // stock id straight to such a gateway 404s. So we ask the gateway what it actually serves -// (GET /v1/models, once per process) and SCORE each advertised id against every tier's family -// — the same DATA-is-a-table / DECISION-is-a-formula rule the rest of forge follows: the tier -// families are data, the pick is a graded overlap score (src/math.js setOverlap), inspectable -// and testable. +// (GET /v1/models, once per process, through src/model_catalog.js) and pick, per tier, the +// NEWEST advertised model of that tier's family — the same generic rule resolveTierModel applies +// to the Anthropic Models API: family word as a whole token, then catalog creation time, then the +// parsed version. The overlap score (src/math.js setOverlap) is still reported so a pick can be +// inspected. // // Contract (zero breaking change): -// - Only engages for a NON-default gateway base URL. Direct api.anthropic.com → no-op, no net. +// - Only engages for a NON-default gateway base URL. Direct api.anthropic.com → no-op here +// (direct users resolve through resolveTierModel, which needs ANTHROPIC_API_KEY). // - FAIL-SAFE. No gateway, unreachable, unparseable, or no family match → returns the stock // id unchanged. Callers are byte-identical to before when there is nothing to remap. // - The MODELS export shape is untouched; nothing here mutates model_tiers. -import { spawnSync } from "node:child_process"; import { setOverlap } from "./math.js"; -import { MODELS, TIER_ORDER } from "./model_tiers.js"; - -const ANTHROPIC_DEFAULT = "https://api.anthropic.com"; - -// GET {base}/v1/models in a spawned node child so this module stays synchronous like every -// other forge faculty (embed.js / llm.js pattern). The auth key travels via the child's env -// (_FORGE_LLM_KEY) — never in argv, never logged. Accepts both OpenAI-shaped ({data:[{id}]}) -// and Anthropic-shaped ({data:[{id}]}) catalogs; both key the list under data[].id. -const FETCH_CHILD = `let raw="";process.stdin.on("data",(d)=>{raw+=d;});process.stdin.on("end",async()=>{try{const{url,timeoutMs}=JSON.parse(raw);const key=process.env._FORGE_LLM_KEY||"";const headers={"anthropic-version":"2023-06-01"};if(key.startsWith("Bearer ")){headers.authorization=key;}else if(key){headers["x-api-key"]=key;headers.authorization="Bearer "+key;}const ac=new AbortController();const timer=setTimeout(()=>ac.abort(),timeoutMs||5000);let res;try{res=await fetch(url,{headers,signal:ac.signal});}finally{clearTimeout(timer);}if(!res.ok){process.stderr.write("http "+res.status);process.exit(1);}const data=await res.json();const rows=Array.isArray(data)?data:Array.isArray(data&&data.data)?data.data:[];const ids=rows.map((m)=>(typeof m==="string"?m:m&&m.id)).filter((x)=>typeof x==="string"&&x);process.stdout.write(JSON.stringify(ids));}catch(e){process.stderr.write(String((e&&e.message)||e));process.exit(1);}});`; +import { + CATALOG_TIMEOUT_MS, + envGatewayBase, + fetchCatalog, + gatewayKey, + gatewaySource, + newestInFamily, + tokenize, + versionOf, +} from "./model_catalog.js"; +import { familyOfTier, MODELS, TIER_ORDER } from "./model_tiers.js"; + +export { versionOf }; // Process-lifetime cache: base URL -> string[] (advertised ids) | null (fetched, none usable). // "Once per process" is the whole point — the ambient LLM path must not re-probe on every call. @@ -38,40 +43,17 @@ export function _resetGatewayCache() { /** * The active gateway base URL to remap against, or null when there is nothing to remap. * Mirrors llm.js resolution (LITELLM_BASE_URL wins, then ANTHROPIC_BASE_URL). The default - * Anthropic endpoint returns null so direct-API users never trigger a probe or a remap. + * Anthropic endpoint returns null so direct-API users never trigger a gateway probe or remap. * @returns {string|null} */ export function gatewayBase() { - const url = (process.env.LITELLM_BASE_URL || process.env.ANTHROPIC_BASE_URL || "").replace( - /\/+$/, - "", - ); - if (!url) return null; - if (url.toLowerCase() === ANTHROPIC_DEFAULT) return null; // direct Anthropic — stock ids are correct - return url; + return envGatewayBase(process.env); } -function apiKey() { - return ( - process.env.ANTHROPIC_API_KEY || - process.env.ANTHROPIC_AUTH_TOKEN || - process.env.LITELLM_API_KEY || - "" - ); -} - -function spawnFetch(base, timeoutMs) { - const r = spawnSync(process.execPath, ["-e", FETCH_CHILD], { - input: JSON.stringify({ url: `${base}/v1/models`, timeoutMs }), - encoding: "utf8", - timeout: timeoutMs + 1000, - maxBuffer: 4 * 1024 * 1024, - env: { ...process.env, _FORGE_LLM_KEY: apiKey() }, - stdio: ["pipe", "pipe", "pipe"], - }); - if (r.error || r.status !== 0 || !r.stdout) return null; - const ids = JSON.parse(r.stdout); - return Array.isArray(ids) ? ids : null; +/** GET {base}/v1/models through the shared catalog fetcher (memory only, never persisted here). */ +function defaultFetch(base, timeoutMs) { + const cat = fetchCatalog(gatewaySource(base, gatewayKey(process.env)), { timeoutMs }); + return cat ? cat.models.map((m) => m.id) : null; } /** @@ -80,12 +62,12 @@ function spawnFetch(base, timeoutMs) { * @param {{timeoutMs?: number, fetchImpl?: (base:string)=>string[]}} [opts] fetchImpl is injectable for tests * @returns {string[]|null} advertised ids, or null on any failure */ -export function fetchModelIds(base, { timeoutMs = 5000, fetchImpl } = {}) { +export function fetchModelIds(base, { timeoutMs = CATALOG_TIMEOUT_MS, fetchImpl } = {}) { if (!base) return null; if (_catalogCache.has(base)) return _catalogCache.get(base); let ids = null; try { - ids = fetchImpl ? fetchImpl(base) : spawnFetch(base, timeoutMs); + ids = fetchImpl ? fetchImpl(base) : defaultFetch(base, timeoutMs); } catch { ids = null; } @@ -97,116 +79,54 @@ export function fetchModelIds(base, { timeoutMs = 5000, fetchImpl } = {}) { return result; } -// A version part is a short number; a date stamp (20250929) is not. A run of them is ONE token -// ("claude-3-5-sonnet" → "3.5"), because as separate "3" and "5" tokens the 5 of Sonnet 3.5 -// matched the 5 of Sonnet 5 and the gateway map picked a two-generation-old model. -const isVersionPart = (t) => /^\d{1,3}$/.test(t); - -/** Tokens of a model id or name, with consecutive version numbers collapsed into one token. */ -function tokenize(s) { - const parts = String(s) - .toLowerCase() - .split(/[^a-z0-9]+/) - .filter(Boolean); - const out = new Set(); - for (let i = 0; i < parts.length; ) { - if (!isVersionPart(parts[i])) { - out.add(parts[i++]); - continue; - } - const run = []; - while (i < parts.length && isVersionPart(parts[i])) run.push(parts[i++]); - out.add(run.join(".").replace(/(?:\.0)+$/, "")); - } - return out; -} - -/** The first version in an id ("claude-sonnet-4-5-20250929" → [4,5]), or null. */ -export function versionOf(modelId) { - const parts = String(modelId) - .toLowerCase() - .split(/[^a-z0-9]+/) - .filter(Boolean); - for (let i = 0; i < parts.length; i++) { - if (!isVersionPart(parts[i])) continue; - const run = []; - while (i < parts.length && isVersionPart(parts[i])) run.push(Number(parts[i++])); - while (run.length > 1 && run[run.length - 1] === 0) run.pop(); - return run; - } - return null; -} - -/** Newer first; an id with no version ranks last. */ -function compareVersions(a, b) { - if (!a && !b) return 0; - if (!a) return 1; - if (!b) return -1; - for (let i = 0; i < Math.max(a.length, b.length); i++) { - const d = (b[i] ?? 0) - (a[i] ?? 0); - if (d) return d; - } - return 0; -} - /** Reference token set for a tier: the family key plus its marketing-name tokens (e.g. haiku → {haiku,"4.5"}). */ export function familyTokens(tier) { return tokenize(`${tier} ${MODELS[tier]?.name ?? ""}`); } /** - * Score how well a gateway model id belongs to a tier family, in [0,1]. - * The family word itself (haiku/sonnet/opus/fable) is a HARD gate — absent it, the id is not a - * candidate for that tier (score 0), so an unrelated model can never be mis-assigned. Present it, - * the score is the overlap coefficient of the tier's reference tokens with the id's tokens, which - * rewards a version match ("claude-sonnet-5" scores 1.0 for sonnet; "prod-sonnet" scores lower). + * Score how well a gateway model id matches a tier's snapshot model, in [0,1] — reported with + * each pick so it can be inspected. The family word itself (haiku/sonnet/opus/fable) is a HARD + * gate — absent it, the id is not a candidate for that tier (score 0), so an unrelated model can + * never be mis-assigned. Present it, the score is the overlap coefficient of the tier's reference + * tokens with the id's tokens ("claude-sonnet-5" scores 1.0 for sonnet; "prod-sonnet" lower). * @param {string} modelId * @param {string} tier * @returns {number} */ export function familyScore(modelId, tier) { const toks = tokenize(modelId); - if (!toks.has(tier)) return 0; // family word MUST be present + if (!toks.has(familyOfTier(tier))) return 0; // family word MUST be present return setOverlap(familyTokens(tier), toks); } -// Deterministic tie-break among equal-scoring candidates: the newest version of the family first -// (Sonnet 4.5 over Sonnet 3.5 when neither is the pinned Sonnet 5), then the id closest to the -// canonical name (fewest tokens — less vendor/deployment noise), then lexicographic for stability. -function tieBreak(a, b) { - const byVersion = compareVersions(versionOf(a), versionOf(b)); - if (byVersion) return byVersion; - const na = tokenize(a).size; - const nb = tokenize(b).size; - if (na !== nb) return na - nb; - return a < b ? -1 : a > b ? 1 : 0; -} - /** - * Pure: given a gateway's advertised ids, pick the best id per tier by family score. - * @param {string[]} ids + * Pure: given a gateway's advertised ids, pick per tier the NEWEST id of that tier's family + * (catalog order rules in model_catalog.newestInFamily: version, snapshot date, then the id + * closest to the canonical name). Unrelated ids are never assigned. + * @param {Array} ids * @returns {Record} only tiers with a family match appear */ export function buildGatewayMap(ids = []) { - const list = [...new Set((ids || []).filter((x) => typeof x === "string" && x))]; + const rows = []; + const seen = new Set(); + for (const x of ids || []) { + const row = typeof x === "string" ? { id: x } : x; + if (!row || typeof row.id !== "string" || !row.id || seen.has(row.id)) continue; + seen.add(row.id); + rows.push(row); + } /** @type {Record} */ const map = {}; for (const tier of TIER_ORDER) { - let best = null; - for (const id of list) { - const score = familyScore(id, tier); - if (score <= 0) continue; - if (!best || score > best.score || (score === best.score && tieBreak(id, best.id) < 0)) { - best = { id, score }; - } - } - if (best) map[tier] = best; + const pick = newestInFamily(rows, familyOfTier(tier)); + if (pick) map[tier] = { id: pick.id, score: familyScore(pick.id, tier) }; } return map; } /** - * The tier→gateway-model mapping for the active gateway. Fetches /v1/models (cached) and scores. + * The tier→gateway-model mapping for the active gateway. Fetches /v1/models (cached) and picks. * @param {{base?: string, fetchImpl?: (base:string)=>string[], timeoutMs?: number}} [opts] * @returns {{active:boolean, base:(string|null), reachable?:boolean, catalog?:string[], models:Record}} */ @@ -226,7 +146,7 @@ export function gatewayModelMap({ base, fetchImpl, timeoutMs } = {}) { /** * Resolve a tier to a gateway model id, or return `fallbackId` unchanged (silent fallback). - * This is the one function callers reach for: it never throws and never blocks a direct-API user. + * It never throws and never blocks a direct-API user. * @param {string} tier * @param {string} fallbackId the stock id to use when there is nothing to remap * @param {{base?: string, fetchImpl?: (base:string)=>string[], timeoutMs?: number}} [opts] diff --git a/src/gitignore.js b/src/gitignore.js index 0f9b87be..86c3f0f6 100644 --- a/src/gitignore.js +++ b/src/gitignore.js @@ -82,10 +82,11 @@ export function removeGitignoreBlock(root) { } // Per-session hook logs (raw prompts and shell commands) under .forge/sessions/ are local -// runtime state that must never reach git. A repo may deliberately commit OTHER .forge/ +// runtime state that must never reach git, and .forge/cache/ holds derived HTTP responses +// (model catalogs) that are re-fetched on demand. A repo may deliberately commit OTHER .forge/ // content (the ledger, decisions.md), so rather than rewrite the user's root .gitignore // the tool owns a nested .forge/.gitignore listing only its private runtime dirs. -export const FORGE_PRIVATE_DIRS = ["sessions/"]; +export const FORGE_PRIVATE_DIRS = ["sessions/", "cache/"]; /** * Ensure `/.forge/.gitignore` ignores every FORGE_PRIVATE_DIRS entry. Appends only diff --git a/src/http_cache.js b/src/http_cache.js new file mode 100644 index 00000000..da112d39 --- /dev/null +++ b/src/http_cache.js @@ -0,0 +1,232 @@ +// forge http cache — a small PRIVATE HTTP cache for the JSON catalogs forge reads (model lists, +// model prices). The one rule: freshness comes from the RESPONSE, never from a constant in this +// code. A response is reused without a request only while its own `Cache-Control: max-age` / +// `Expires` says it is fresh (RFC 9111 §4.2); otherwise the next use sends a CONDITIONAL request +// (`If-None-Match` / `If-Modified-Since`) and a `304` refreshes the stored copy. A response with +// no caching headers therefore revalidates on every use — forge never invents a TTL. +// +// When the request fails (offline, timeout, non-2xx, unparseable body) the stored copy is served +// and marked `stale`: the caller's next fallback (the shipped snapshot) is older data still, so a +// stale catalog beats it. `no-store` responses are used once and never written. +// +// Transport: a spawned node child running global fetch, so this module stays SYNCHRONOUS like +// every other forge faculty (the embed.js / llm.js pattern). Request headers — including any +// credential — travel on the child's stdin, never argv, and are never logged or persisted. +import { spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { contentHash, readJsonSafe } from "./util.js"; + +/** Response headers the cache reads. The child forwards only these; nothing else is stored. */ +export const CACHE_HEADERS = ["cache-control", "expires", "date", "age", "etag", "last-modified"]; + +// GET one URL: stdin {url, headers, timeoutMs, keep} → stdout {status, headers, body}. Any HTTP +// status is returned (a 304 is an answer, not a failure); only a network error/timeout exits 1. +const GET_CHILD = `let raw="";process.stdin.on("data",(d)=>{raw+=d;});process.stdin.on("end",async()=>{try{const{url,headers,timeoutMs,keep}=JSON.parse(raw);const ac=new AbortController();const timer=setTimeout(()=>ac.abort(),timeoutMs||3000);try{const res=await fetch(url,{headers,signal:ac.signal});const h={};for(const k of keep){const v=res.headers.get(k);if(v!=null)h[k]=v;}const body=res.status===304?"":await res.text();process.stdout.write(JSON.stringify({status:res.status,headers:h,body}));}finally{clearTimeout(timer);}}catch(e){process.stderr.write(String((e&&e.message)||e));process.exit(1);}});`; + +/** + * The default transport: one synchronous GET. `FORGE_NO_CATALOG_FETCH=1` turns it off (air-gapped + * machines, and the test suite's hermetic boundary) — callers then see "unavailable" and fall back + * to the cached copy or the snapshot. Never throws. + * @param {{url:string, headers?:Record, timeoutMs?:number}} req + * @returns {{status:number, headers:Record, body:string}|null} null on network failure + */ +export function httpGet({ url, headers = {}, timeoutMs = 3000 }) { + if (process.env.FORGE_NO_CATALOG_FETCH === "1") return null; + try { + const r = spawnSync(process.execPath, ["-e", GET_CHILD], { + input: JSON.stringify({ url, headers, timeoutMs, keep: CACHE_HEADERS }), + encoding: "utf8", + timeout: timeoutMs + 2000, + maxBuffer: 32 * 1024 * 1024, + stdio: ["pipe", "pipe", "pipe"], + }); + if (r.error || r.status !== 0 || !r.stdout) return null; + const res = JSON.parse(r.stdout); + return Number.isInteger(res?.status) ? res : null; + } catch { + return null; + } +} + +/** `Cache-Control` → directive map (`max-age=60, no-cache` → {"max-age":"60","no-cache":true}). */ +export function parseCacheControl(value) { + /** @type {Record} */ + const out = {}; + for (const part of String(value ?? "").split(",")) { + const [name, ...rest] = part.trim().split("="); + if (!name) continue; + const arg = rest.join("=").trim().replace(/^"|"$/g, ""); + out[name.toLowerCase()] = arg === "" ? true : arg; + } + return out; +} + +/** Lower-case header names; drop non-string values. */ +function lowerHeaders(h) { + /** @type {Record} */ + const out = {}; + for (const [k, v] of Object.entries(h ?? {})) if (typeof v === "string") out[k.toLowerCase()] = v; + return out; +} + +const pickCacheHeaders = (h) => + Object.fromEntries(CACHE_HEADERS.filter((k) => k in h).map((k) => [k, h[k]])); + +/** + * Freshness lifetime in seconds, from the response alone (RFC 9111 §4.2.1, private cache): + * `no-store`/`no-cache` → 0; `max-age` wins over `Expires`; `Expires` counts from the response's + * `Date` (or when it was received). No explicit freshness → 0, i.e. revalidate on next use — no + * heuristic lifetime, no default TTL. + * @param {Record} headers stored response headers (lower-case) + * @param {number} receivedAt epoch ms the response arrived + */ +export function freshnessLifetime(headers, receivedAt) { + const cc = parseCacheControl(headers["cache-control"]); + if (cc["no-store"] || cc["no-cache"]) return 0; + if (cc["max-age"] !== undefined) { + const maxAge = Number(cc["max-age"]); + return Number.isFinite(maxAge) ? Math.max(0, maxAge) : 0; + } + if (headers.expires) { + const expires = Date.parse(headers.expires); + if (!Number.isFinite(expires)) return 0; // an invalid Expires means "already expired" + const date = Date.parse(headers.date ?? ""); + return Math.max(0, (expires - (Number.isFinite(date) ? date : receivedAt)) / 1000); + } + return 0; +} + +/** Current age in seconds (RFC 9111 §4.2.3): the larger of the `Age` header and the apparent + * age at receipt, plus the time the copy has been resident here. */ +export function currentAge(headers, receivedAt, now) { + const ageHeader = Number(headers.age); + const ageValue = Number.isFinite(ageHeader) && ageHeader > 0 ? ageHeader : 0; + const date = Date.parse(headers.date ?? ""); + const apparent = Number.isFinite(date) ? Math.max(0, (receivedAt - date) / 1000) : 0; + return Math.max(apparent, ageValue) + Math.max(0, (now - receivedAt) / 1000); +} + +/** Is a stored record still fresh at `now`? */ +export function isFresh(record, now) { + return ( + freshnessLifetime(record.headers, record.receivedAt) > + currentAge(record.headers, record.receivedAt, now) + ); +} + +/** The on-disk file for a URL: readable host prefix + a hash of the full URL. */ +export function cacheFile(dir, url) { + let host = "url"; + try { + host = new URL(url).host.replace(/[^a-z0-9.-]+/gi, "_"); + } catch {} + return join(dir, `${host}-${contentHash(url).slice(0, 12)}.json`); +} + +function readRecord(file, url) { + const rec = file ? readJsonSafe(file) : null; + return rec && rec.url === url && rec.headers && Number.isFinite(rec.receivedAt) && "value" in rec + ? rec + : null; +} + +// The cache directory ignores itself, like the ledger's derived `.state-cache.json`: a nested +// `.gitignore` of `*` keeps every cached catalog (and the ignore file) out of git whether or not +// `forge init` has written `.forge/.gitignore` yet. +function writeRecord(dir, file, record) { + try { + mkdirSync(dir, { recursive: true }); + const ignore = join(dir, ".gitignore"); + if (!existsSync(ignore)) + writeFileSync( + ignore, + "# forge HTTP cache — derived, re-fetched on demand, never committed\n*\n", + ); + writeFileSync(file, JSON.stringify(record)); + } catch {} // a read-only checkout simply revalidates on every use +} + +/** + * GET a JSON document through the cache. Order of preference: + * 1. a stored copy that is still fresh by its own headers — no request; + * 2. the network — conditional when a validator is stored (`304` → the stored copy, refreshed); + * 3. the stored copy, marked stale, when the request fails or the body is unusable. + * `transform` maps the parsed body to the value that is stored and returned (e.g. a normalized + * catalog page); a transform that returns null/undefined marks the response unusable. + * @param {string} url + * @param {{headers?:Record, dir?:string|null, fetchImpl?:(req:{url:string,headers:Record,timeoutMs:number})=>({status:number,headers?:Record,body?:string}|null), + * timeoutMs?:number, now?:number, transform?:(json:any)=>any}} [opts] + * `dir` null → memory only (nothing persisted). `fetchImpl` is the injectable transport. + * @returns {{value:any, cache:"fresh"|"revalidated"|"network"|"stale", url:string, + * freshUntil:number}|null} `freshUntil` is the epoch ms this answer stops being fresh by its + * own headers — for a caller holding it in memory. A stale copy is already expired. + */ +export function cachedGetJson( + url, + { + headers = {}, + dir = null, + fetchImpl = httpGet, + timeoutMs = 3000, + now = Date.now(), + transform = (x) => x, + } = {}, +) { + // When this answer stops being fresh, in epoch ms. A caller that holds a response in memory + // (a long-running server) uses it so its copy expires when the response says it does, rather + // than living for the life of the process. + const expiry = (h, receivedAt) => + receivedAt + Math.max(0, freshnessLifetime(h, receivedAt) * 1000); + const file = dir ? cacheFile(dir, url) : null; + const stored = readRecord(file, url); + if (stored && isFresh(stored, now)) + return { + value: stored.value, + cache: "fresh", + url, + freshUntil: expiry(stored.headers, stored.receivedAt), + }; + + const reqHeaders = { ...headers }; + if (stored?.headers.etag) reqHeaders["if-none-match"] = stored.headers.etag; + if (stored?.headers["last-modified"]) + reqHeaders["if-modified-since"] = stored.headers["last-modified"]; + let res = null; + try { + res = fetchImpl({ url, headers: reqHeaders, timeoutMs }); + } catch { + res = null; + } + + if (res && res.status === 304 && stored) { + // RFC 9111 §4.3.4: the 304's headers update the stored ones. A 304 without Date/Age must not + // inherit the OLD Date, or the refreshed copy would look as old as the original response. + const fresh = lowerHeaders(res.headers); + const merged = { ...stored.headers, ...pickCacheHeaders(fresh) }; + for (const k of ["date", "age"]) if (!(k in fresh)) delete merged[k]; + const record = { ...stored, receivedAt: now, headers: merged }; + if (file && !parseCacheControl(merged["cache-control"])["no-store"]) + writeRecord(dir, file, record); + return { value: stored.value, cache: "revalidated", url, freshUntil: expiry(merged, now) }; + } + + if (res && res.status >= 200 && res.status < 300) { + let value = null; + try { + value = transform(JSON.parse(String(res.body ?? ""))); + } catch { + value = null; + } + if (value != null) { + const h = pickCacheHeaders(lowerHeaders(res.headers)); + if (file && !parseCacheControl(h["cache-control"])["no-store"]) + writeRecord(dir, file, { v: 1, url, receivedAt: now, headers: h, value }); + return { value, cache: "network", url, freshUntil: expiry(h, now) }; + } + } + + // A stale copy beats nothing, but it is already expired: a caller holding it in memory must + // try again on its next use rather than keep it. + return stored ? { value: stored.value, cache: "stale", url, freshUntil: now } : null; +} diff --git a/src/llm.js b/src/llm.js index 1f3731dc..c691197e 100644 --- a/src/llm.js +++ b/src/llm.js @@ -7,6 +7,7 @@ // config: the Anthropic Messages API (default), and the OpenAI-compatible chat/completions // API that OpenAI, Google Gemini, OpenRouter, and LiteLLM all expose. import { spawnSync } from "node:child_process"; +import { resolveTierModel } from "./model_tiers.js"; // Anthropic Messages API — POST {baseUrl}/v1/messages, x-api-key / bearer auth. const HTTP_CHILD_ANTHROPIC = `let raw="";process.stdin.on("data",(d)=>{raw+=d;});process.stdin.on("end",async()=>{try{const{url,model,prompt,maxTokens}=JSON.parse(raw);const key=process.env._FORGE_LLM_KEY||"";const headers={"content-type":"application/json","anthropic-version":"2023-06-01"};if(key.startsWith("Bearer "))headers.authorization=key;else if(key)headers["x-api-key"]=key;const body=JSON.stringify({model,max_tokens:maxTokens||1024,messages:[{role:"user",content:prompt}]});const res=await fetch(url,{method:"POST",headers,body});if(!res.ok){process.stderr.write("llm: http "+res.status);process.exit(1);}const data=await res.json();const text=(data.content||[]).filter(b=>b.type==="text").map(b=>b.text).join("");process.stdout.write(text);}catch(e){process.stderr.write("llm: "+(e.message||e));process.exit(1);}});`; @@ -77,10 +78,11 @@ export function resolveHttpProvider() { /** * Build an HTTP-based LLM runner. Same contract as adjudicate.buildRunner: * returns (prompt) => string. Selects the Anthropic or OpenAI-compatible wire format - * from the resolved provider. + * from the resolved provider. With no `model`, the haiku tier is resolved at call time (live + * catalog, else the shipped snapshot) — there is no model id pinned here. * @param {{model?: string, timeoutMs?: number}} [opts] */ -export function buildHttpRunner({ model = "claude-haiku-4-5-20251001", timeoutMs = 20000 } = {}) { +export function buildHttpRunner({ model, timeoutMs = 20000 } = {}) { return (prompt) => { const provider = resolveHttpProvider(); if (!provider) @@ -88,7 +90,11 @@ export function buildHttpRunner({ model = "claude-haiku-4-5-20251001", timeoutMs "no LLM provider configured — set ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, OPENAI_API_KEY, or GEMINI_API_KEY", ); const child = provider.format === "openai" ? HTTP_CHILD_OPENAI : HTTP_CHILD_ANTHROPIC; - const chosenModel = provider.model || provider.defaultModel || model; + const chosenModel = + provider.model || + provider.defaultModel || + model || + resolveTierModel("haiku", { root: process.cwd() })?.id; const input = JSON.stringify({ url: `${provider.baseUrl}${provider.path}`, model: chosenModel, diff --git a/src/model_catalog.js b/src/model_catalog.js new file mode 100644 index 00000000..e340e7cd --- /dev/null +++ b/src/model_catalog.js @@ -0,0 +1,484 @@ +// forge model catalog — turn a model FAMILY (haiku/sonnet/opus/fable) into a concrete model id, +// and a model id into a price, from LIVE catalogs instead of ids and prices pinned in code. +// +// ids ← the active provider's own `GET /v1/models`: the Anthropic Models API (paginated +// `has_more`/`last_id` → `after_id`), a custom gateway's list, or OpenRouter's. +// prices ← OpenRouter's public `GET /api/v1/models` (no key): USD per TOKEN as strings. +// +// Everything that decides is a generic rule over the catalog's own data — no model id, version or +// date is named here: +// - family membership is a whole-token match of the family word in the id or display name; +// - "newest" is the catalog's `created_at` (Anthropic) / `created` (OpenAI-style, OpenRouter), +// then the parsed version, then a YYYYMMDD snapshot stamp, then the least-decorated id; +// - an id matches a price row when their canonical token sets agree (vendor namespace, snapshot +// date and separators ignored: `claude-opus-4-8` ↔ `anthropic/claude-opus-4.8`). +// Fetching goes through src/http_cache.js (freshness from the response's own headers). Every +// function here is total: an unavailable catalog is `null`, never a throw. +import { join } from "node:path"; +import { cachedGetJson, httpGet } from "./http_cache.js"; + +export const ANTHROPIC_API = "https://api.anthropic.com"; +export const ANTHROPIC_VERSION = "2023-06-01"; +export const OPENROUTER_API = "https://openrouter.ai/api/v1"; +/** Short on purpose: a catalog lookup precedes real work and must never stall it. */ +export const CATALOG_TIMEOUT_MS = 3000; +// The Models API's largest page — one request lists the whole catalog in practice. +const ANTHROPIC_PAGE_LIMIT = 1000; +// A pagination guard, not a data limit: a server that repeats `has_more` forever stops here. +const MAX_PAGES = 50; + +// --------------------------------------------------------------------------- +// Tokens and versions — shared with gateway_model_map.js. +// --------------------------------------------------------------------------- + +// A version part is a short number; a date stamp (20250929) is not. A run of them is ONE token +// ("claude-3-5-sonnet" → "3.5"), because as separate "3" and "5" tokens the 5 of Sonnet 3.5 +// matched the 5 of Sonnet 5 and the gateway map picked a two-generation-old model. +const isVersionPart = (t) => /^\d{1,3}$/.test(t); +const isDateStamp = (t) => /^(19|20)\d{6}$/.test(t); +const words = (s) => + String(s ?? "") + .toLowerCase() + .split(/[^a-z0-9]+/) + .filter(Boolean); + +/** Tokens of a model id or name, with consecutive version numbers collapsed into one token. */ +export function tokenize(s) { + const parts = words(s); + const out = new Set(); + for (let i = 0; i < parts.length; ) { + if (!isVersionPart(parts[i])) { + out.add(parts[i++]); + continue; + } + const run = []; + while (i < parts.length && isVersionPart(parts[i])) run.push(parts[i++]); + out.add(run.join(".").replace(/(?:\.0)+$/, "")); + } + return out; +} + +/** The first version in an id ("claude-sonnet-4-5-20250929" → [4,5]), or null. */ +export function versionOf(modelId) { + const parts = words(modelId); + for (let i = 0; i < parts.length; i++) { + if (!isVersionPart(parts[i])) continue; + const run = []; + while (i < parts.length && isVersionPart(parts[i])) run.push(Number(parts[i++])); + while (run.length > 1 && run[run.length - 1] === 0) run.pop(); + return run; + } + return null; +} + +/** Newer first; an id with no version ranks last. */ +export function compareVersions(a, b) { + if (!a && !b) return 0; + if (!a) return 1; + if (!b) return -1; + for (let i = 0; i < Math.max(a.length, b.length); i++) { + const d = (b[i] ?? 0) - (a[i] ?? 0); + if (d) return d; + } + return 0; +} + +/** The YYYYMMDD snapshot stamp in an id ("claude-3-5-sonnet-20241022" → 20241022), or 0. */ +export function dateStampOf(modelId) { + const hit = words(modelId).find(isDateStamp); + return hit ? Number(hit) : 0; +} + +/** Epoch ms from a catalog timestamp: RFC 3339 string, or unix seconds/ms number. null if absent + * or unknown: the Models API sets `created_at` to the epoch when a release date is unknown, so an + * epoch value (≤ 0) means "no date", never "the oldest model". */ +export function createdMs(v) { + if (typeof v === "number" && Number.isFinite(v) && v > 0) return v < 1e12 ? v * 1000 : v; + if (typeof v === "string" && v) { + const t = Date.parse(v); + return Number.isFinite(t) && t > 0 ? t : null; + } + return null; +} + +/** + * USD per token (OpenRouter's `"0.000003"` strings) → USD per million tokens (3). Rounded to 12 + * significant digits so binary float noise (2.9999999999999996) never reaches a price. Negative + * (OpenRouter's "-1" = dynamic/variable pricing), empty or non-numeric → null. + */ +export function perMillion(perToken) { + if (perToken == null || perToken === "") return null; + const n = typeof perToken === "number" ? perToken : Number(String(perToken).trim()); + if (!Number.isFinite(n) || n < 0) return null; + return Number((n * 1e6).toPrecision(12)); +} + +// --------------------------------------------------------------------------- +// Catalog pages → normalized rows. +// --------------------------------------------------------------------------- + +/** + * @typedef {{id:string, displayName?:string, createdAt?:string, inCost?:number, outCost?:number}} CatalogModel + */ + +/** What a model id may look like: vendor namespaces, versions, snapshot dates, `:variant` + * suffixes — but no whitespace, no control characters, and nothing that could end a YAML + * scalar or a shell word. Catalogs are network data; ids reach generated config and model + * calls. */ +export const SAFE_MODEL_ID = /^[A-Za-z0-9][A-Za-z0-9._:@+/-]{0,199}$/; + +/** One printable line: control characters and line separators collapse to a space. */ +export function printableLine(s) { + let out = ""; + for (const ch of String(s)) { + const n = ch.charCodeAt(0); + out += (n < 32 && n !== 9) || n === 127 || n === 0x2028 || n === 0x2029 ? " " : ch; + } + return out.replace(/\s+/g, " ").trim().slice(0, 200); +} + +/** + * Normalize one catalog page — Anthropic (`data[]{id,display_name,created_at}` + `has_more`/ + * `last_id`), OpenAI-style (`data[]{id,created}`), OpenRouter (`data[]{id,name,created,pricing}`) + * or a bare id array — into rows plus the cursor of the next page. null when it is not a catalog. + * @param {any} json + * @returns {{models: CatalogModel[], next: string|null}|null} + */ +export function normalizeCatalogPage(json) { + const rows = Array.isArray(json) ? json : Array.isArray(json?.data) ? json.data : null; + if (!rows) return null; + /** @type {CatalogModel[]} */ + const models = []; + for (const r of rows) { + const id = typeof r === "string" ? r : typeof r?.id === "string" ? r.id : ""; + // A resolved id is written into generated config and passed to a model call, so it must + // look like a model id: no whitespace, no control characters, nothing exotic. A row that + // fails this is dropped here, at the boundary, rather than sanitized at each consumer. + if (!id || !SAFE_MODEL_ID.test(id)) continue; + /** @type {CatalogModel} */ + const m = { id }; + const name = r?.display_name ?? r?.name; + // A display name is shown to a person, never used as an id: keep it to one printable line. + if (typeof name === "string" && name) m.displayName = printableLine(name); + const created = createdMs(r?.created_at ?? r?.created); + if (created != null) m.createdAt = new Date(created).toISOString(); + const inCost = perMillion(r?.pricing?.prompt); + const outCost = perMillion(r?.pricing?.completion); + if (inCost != null && outCost != null) Object.assign(m, { inCost, outCost }); + models.push(m); + } + const next = json?.has_more === true && typeof json?.last_id === "string" ? json.last_id : null; + return { models, next }; +} + +// --------------------------------------------------------------------------- +// Family resolution. +// --------------------------------------------------------------------------- + +/** Vendor namespace of an id ("anthropic/claude-opus-4.8" → "anthropic"), or "". */ +export const namespaceOf = (id) => (String(id).includes("/") ? String(id).split("/")[0] : ""); +const bareId = (id) => String(id).split("/").pop() ?? ""; + +/** True when `family` is a whole token of the model's id or display name (word-boundary match). */ +export function inFamily(model, family) { + const f = String(family ?? "").toLowerCase(); + if (!f) return false; + return words(model?.id).includes(f) || words(model?.displayName).includes(f); +} + +// Newest first: catalog creation time when BOTH rows are dated; otherwise the parsed version +// decides first (a new model listed with an unknown release date must not sink below every dated +// one), then a dated row beats an undated one of the same version, then the snapshot date stamp, +// then the least-decorated id (fewest tokens), then lexicographic. +function newerFirst(a, b) { + const ca = createdMs(a.createdAt); + const cb = createdMs(b.createdAt); + if (ca != null && cb != null && ca !== cb) return cb > ca ? 1 : -1; + const byVersion = compareVersions(versionOf(a.id), versionOf(b.id)); + if (byVersion) return byVersion; + if ((ca == null) !== (cb == null)) return ca == null ? 1 : -1; + const byStamp = dateStampOf(b.id) - dateStampOf(a.id); + if (byStamp) return byStamp; + const bySize = tokenize(a.id).size - tokenize(b.id).size; + if (bySize) return bySize; + return a.id < b.id ? -1 : a.id > b.id ? 1 : 0; +} + +/** + * The newest catalog model of a family, or null. `namespace` restricts a multi-vendor catalog + * (OpenRouter) to the vendor the provider is configured for. A `:variant` row + * ("…-sonnet:thinking") is skipped when its base id is also listed. + * @param {CatalogModel[]} models + * @param {string} family + * @param {{namespace?: string}} [opts] + * @returns {CatalogModel|null} + */ +export function newestInFamily(models, family, { namespace = "" } = {}) { + const list = (models ?? []).filter((m) => m && typeof m.id === "string" && m.id); + const ids = new Set(list.map((m) => m.id)); + let best = null; + for (const m of list) { + if (namespace && namespaceOf(m.id) !== namespace) continue; + if (m.id.includes(":") && ids.has(m.id.split(":")[0])) continue; + if (!inFamily(m, family)) continue; + if (!best || newerFirst(m, best) < 0) best = m; + } + return best; +} + +/** Order-insensitive identity of a model id across catalogs: namespace, snapshot date stamps and + * separators dropped, version runs collapsed (`claude-haiku-4-5-20251001` ≡ `anthropic/claude-haiku-4.5`). */ +export function canonicalKey(modelId) { + return [...tokenize(bareId(String(modelId).toLowerCase()))] + .filter((t) => !isDateStamp(t)) + .sort() + .join(" "); +} + +/** + * The catalog row that IS `modelId` under canonicalKey, or null. An exact id wins; then a row in + * the same namespace; then the least-decorated row. + * @param {string} modelId + * @param {CatalogModel[]} models + * @returns {CatalogModel|null} + */ +export function matchCatalogModel(modelId, models) { + if (!modelId) return null; + const exact = (models ?? []).find((m) => m.id === modelId); + if (exact) return exact; + const key = canonicalKey(modelId); + if (!key) return null; + const ns = namespaceOf(modelId); + const hits = (models ?? []).filter((m) => canonicalKey(m.id) === key); + hits.sort( + (a, b) => + Number(namespaceOf(b.id) === ns) - Number(namespaceOf(a.id) === ns) || + a.id.length - b.id.length || + (a.id < b.id ? -1 : 1), + ); + return hits[0] ?? null; +} + +// --------------------------------------------------------------------------- +// Which catalog, fetched how. +// --------------------------------------------------------------------------- + +/** + * @typedef {{kind:"anthropic"|"gateway"|"openrouter", url:string, headers:Record}} CatalogSource + * @typedef {{kind:null, reason:string}} NoCatalog + */ + +const trimUrl = (u) => String(u ?? "").replace(/\/+$/, ""); + +/** The Anthropic Models API, authenticated with an API key (x-api-key). */ +export function anthropicSource(apiKey) { + return { + kind: /** @type {const} */ ("anthropic"), + url: `${ANTHROPIC_API}/v1/models?limit=${ANTHROPIC_PAGE_LIMIT}`, + headers: { "x-api-key": apiKey, "anthropic-version": ANTHROPIC_VERSION }, + }; +} + +/** A custom gateway's `/v1/models`. Gateways differ in the auth header they read, so a raw key goes + * out as both x-api-key and a Bearer token; a value already prefixed "Bearer " is sent verbatim. */ +export function gatewaySource(base, key = "") { + /** @type {Record} */ + const headers = { "anthropic-version": ANTHROPIC_VERSION }; + if (key.startsWith("Bearer ")) headers.authorization = key; + else if (key) { + headers["x-api-key"] = key; + headers.authorization = `Bearer ${key}`; + } + return { kind: /** @type {const} */ ("gateway"), url: `${trimUrl(base)}/v1/models`, headers }; +} + +/** OpenRouter's public model list — ids for an OpenRouter provider, and every price. */ +export function openRouterSource(base = OPENROUTER_API) { + return { kind: /** @type {const} */ ("openrouter"), url: `${trimUrl(base)}/models`, headers: {} }; +} + +/** The credential a gateway reads, in the same order llm.js sends one. */ +export function gatewayKey(env = process.env, envKey = "") { + return ( + (envKey && env[envKey]) || + env.ANTHROPIC_API_KEY || + env.ANTHROPIC_AUTH_TOKEN || + env.LITELLM_API_KEY || + "" + ); +} + +/** The non-default gateway base the environment points at (LITELLM_BASE_URL wins), or null. */ +export function envGatewayBase(env = process.env) { + const url = trimUrl(env.LITELLM_BASE_URL || env.ANTHROPIC_BASE_URL || ""); + if (!url || url.toLowerCase() === ANTHROPIC_API) return null; + return url; +} + +/** + * Which live catalog lists the models the active provider serves. + * - no provider: derived from the environment exactly as llm.js reaches a model — a custom + * gateway base, else the Anthropic Models API when ANTHROPIC_API_KEY is set; + * - an OpenRouter provider: OpenRouter's catalog; + * - an Anthropic-format provider: its base (direct → Models API; anything else → gateway); + * - an OpenAI-format vendor (OpenAI, Gemini): none — those tiers are configured ids, not families. + * @param {{provider?: any, env?: Record}} [opts] + * @returns {CatalogSource|NoCatalog} + */ +export function catalogSource({ provider = null, env = process.env } = {}) { + if (!provider) { + const gw = envGatewayBase(env); + if (gw) return gatewaySource(gw, gatewayKey(env)); + if (env.ANTHROPIC_API_KEY) return anthropicSource(env.ANTHROPIC_API_KEY); + return { kind: null, reason: "no ANTHROPIC_API_KEY for the Models API" }; + } + if (provider.type === "openrouter") return openRouterSource(provider.baseUrl || OPENROUTER_API); + if (provider.format === "openai") + return { kind: null, reason: `${provider.name ?? provider.type} serves its own model ids` }; + const base = trimUrl(provider.baseUrl || ANTHROPIC_API); + if (base.toLowerCase() === ANTHROPIC_API) { + return env.ANTHROPIC_API_KEY + ? anthropicSource(env.ANTHROPIC_API_KEY) + : { kind: null, reason: "no ANTHROPIC_API_KEY for the Models API" }; + } + return gatewaySource(base, gatewayKey(env, provider.envKey)); +} + +// Process-lifetime memo for the DEFAULT transport only: one CLI run asks for the same catalog +// once per tier, and a revalidation per ask would multiply requests. An injected fetchImpl (tests) +// always goes through, so every step of the HTTP cache stays observable. +const _memo = new Map(); + +/** Clear the per-process catalog memo (tests only). */ +export function _resetCatalogMemo() { + _memo.clear(); +} + +const CACHE_RANK = { fresh: 0, revalidated: 1, network: 2, stale: 3 }; + +/** + * Fetch a whole catalog (following `has_more`/`last_id` → `after_id`) through the HTTP cache. + * @param {CatalogSource} source + * @param {{root?: string|null, fetchImpl?: Function, timeoutMs?: number, now?: number}} [opts] + * root → persist under `/.forge/cache/`; null → memory only. + * @returns {{models: CatalogModel[], url: string, cache: "fresh"|"revalidated"|"network"|"stale", + * pages: number, freshUntil: number}|null} + * null when the catalog is unavailable (no response and nothing cached, or a page missing) + */ +export function fetchCatalog( + source, + { root = null, fetchImpl, timeoutMs = CATALOG_TIMEOUT_MS, now } = {}, +) { + if (!source?.url) return null; + const transport = typeof fetchImpl === "function" ? fetchImpl : httpGet; + const at = now ?? Date.now(); + // One lookup per catalog per process keeps `forge models` from asking once per tier. The entry + // expires when the RESPONSE says it does (freshUntil), so a long-running dashboard or MCP + // server picks up a new model instead of holding its first answer until restart. An + // unavailable catalog is remembered until the same moment, so a broken network is not retried + // on every call either. + const memoKey = transport === httpGet ? `${root ?? ""}\n${source.url}` : null; + if (memoKey) { + const hit = _memo.get(memoKey); + if (hit && at < hit.until) return hit.result; + } + const dir = root ? join(root, ".forge", "cache") : null; + let result = null; + try { + result = fetchPages(source, { dir, transport, timeoutMs, now: at }); + } catch { + result = null; + } + // No freshness at all (freshUntil === at) still memoizes for this call only. + if (memoKey) _memo.set(memoKey, { result, until: result?.freshUntil ?? at }); + return result; +} + +function fetchPages(source, { dir, transport, timeoutMs, now }) { + const models = []; + const seen = new Set(); + const visited = new Set(); + /** @type {"fresh"|"revalidated"|"network"|"stale"} */ + let cache = "fresh"; + let pages = 0; + // The catalog is only as fresh as its least fresh page. + let freshUntil = Number.POSITIVE_INFINITY; + for (let url = source.url; url && !visited.has(url) && pages < MAX_PAGES; ) { + visited.add(url); + const page = cachedGetJson(url, { + headers: source.headers, + dir, + fetchImpl: transport, + timeoutMs, + now, + transform: normalizeCatalogPage, + }); + if (!page) return null; // a missing page makes the catalog incomplete — i.e. unavailable + pages++; + if (CACHE_RANK[page.cache] > CACHE_RANK[cache]) cache = page.cache; + freshUntil = Math.min(freshUntil, page.freshUntil ?? now); + for (const m of page.value.models) { + if (seen.has(m.id)) continue; + seen.add(m.id); + models.push(m); + } + url = page.value.next ? withQuery(source.url, "after_id", page.value.next) : null; + } + return { + models, + url: source.url, + cache, + pages, + freshUntil: Number.isFinite(freshUntil) ? freshUntil : now, + }; +} + +function withQuery(url, key, value) { + const u = new URL(url); + u.searchParams.set(key, value); + return u.toString(); +} + +/** A short human label for a catalog URL ("api.anthropic.com"). */ +export function catalogHost(url) { + try { + return new URL(url).host; + } catch { + return String(url ?? ""); + } +} + +/** + * OpenRouter's price catalog, fetched once — pass the result as `priceCatalog` to price many ids + * without asking again (null = tried, unavailable). + * @param {{root?: string|null, fetchImpl?: Function, timeoutMs?: number, now?: number}} [opts] + */ +export function fetchPriceCatalog(opts = {}) { + return fetchCatalog(openRouterSource(), opts); +} + +/** + * The live price of a model id from OpenRouter's public catalog, or null (unavailable / unlisted / + * unpriced). Per-million USD. + * @param {string} modelId + * @param {{root?: string|null, fetchImpl?: Function, timeoutMs?: number, now?: number, + * priceCatalog?: ReturnType}} [opts] priceCatalog: a prefetched catalog + * @returns {{inCost:number, outCost:number, matchedId:string, catalog:string, cache:string}|null} + */ +export function catalogPrice(modelId, opts = {}) { + if (!modelId) return null; + const cat = opts.priceCatalog !== undefined ? opts.priceCatalog : fetchPriceCatalog(opts); + if (!cat) return null; + const hit = matchCatalogModel( + modelId, + cat.models.filter((m) => m.inCost != null && m.outCost != null), + ); + if (!hit || hit.inCost == null || hit.outCost == null) return null; + return { + inCost: hit.inCost, + outCost: hit.outCost, + matchedId: hit.id, + catalog: cat.url, + cache: cat.cache, + }; +} diff --git a/src/model_tiers.js b/src/model_tiers.js index b192a47b..e0f1630e 100644 --- a/src/model_tiers.js +++ b/src/model_tiers.js @@ -1,8 +1,27 @@ // forge model tiers — the routing target table. Cheapest capable model per complexity tier. // Costs are per-million tokens (input/output). The premise: a prime-number finder does not -// need Fable 5. Size the model to the task. Data lives in model_tiers.json so rotation is -// a config change, not a code change. +// need Fable 5. Size the model to the task. +// +// A tier names a model FAMILY (its key: haiku/sonnet/opus/fable). The concrete id and price are +// RESOLVED at the point of use (resolveTierModel / resolveTierPrice): the newest model of that +// family in the active provider's live catalog, priced from OpenRouter's public catalog (see +// src/model_catalog.js). model_tiers.json is the shipped SNAPSHOT — the data of last resort when +// no catalog is reachable — and keeps its `pricingVerified` date so `forge doctor` can say when +// the snapshot itself went stale. Nothing on the hook hot path resolves; only the places that +// need a concrete id or price do (the LLM runner, the gateway config, the cost estimate, +// `forge route`, `forge models`). import { readFileSync } from "node:fs"; +import { + canonicalKey, + catalogHost, + catalogPrice, + catalogSource, + fetchCatalog, + inFamily, + namespaceOf, + newestInFamily, +} from "./model_catalog.js"; +import { loadRegistry } from "./router/registry.js"; const data = JSON.parse(readFileSync(new URL("./model_tiers.json", import.meta.url), "utf8")); @@ -51,3 +70,227 @@ export function allPricePairs(models = MODELS) { } return pairs; } + +// --------------------------------------------------------------------------- +// Runtime resolution: family → newest concrete id, id → live price. +// --------------------------------------------------------------------------- + +/** The family word a tier stands for (its key, unless the table names one explicitly). */ +export const familyOfTier = (tier) => MODELS[tier]?.family ?? tier; + +/** The tier whose family a model id belongs to (whole-token match), or null. */ +export function tierOfModel(modelId) { + return TIER_ORDER.find((t) => inFamily({ id: modelId }, familyOfTier(t))) ?? null; +} + +/** True when `id` (vendor namespace ignored) is the shipped snapshot's id — for `tier` when + * given, else for any tier: a family placeholder a catalog may replace, as opposed to an explicit + * id someone configured on purpose. + * @param {string|null|undefined} id + * @param {string} [tier] */ +export function isSnapshotId(id, tier) { + if (!id) return false; + const bare = String(id).split("/").pop(); + const rows = tier ? [MODELS[tier]].filter(Boolean) : Object.values(MODELS); + return rows.some((m) => m.id === bare); +} + +/** + * @typedef {object} ResolveOpts + * @property {string|null} [root] project root; catalogs persist under `/.forge/cache/` (null = memory only) + * @property {any} [provider] the active provider (providers.js); omitted → derived from env like llm.js + * @property {Record} [env] environment to read keys/base URLs from (default process.env) + * @property {Function} [fetchImpl] injectable transport `({url, headers, timeoutMs}) => {status, headers, body}|null` + * @property {number} [timeoutMs] per-request timeout + * @property {number} [now] clock for cache freshness (epoch ms) + */ + +/** + * @typedef {object} ResolvedModel + * @property {string} id concrete model id to send + * @property {string} family the family word the tier names + * @property {"catalog"|"snapshot"|"config"} source where the id came from + * @property {string} [createdAt] catalog creation time (ISO), catalog source only + * @property {string} [displayName] + * @property {string} [catalog] catalog URL, catalog source only + * @property {string} [cache] fresh | revalidated | network | stale + * @property {string} [reason] why the catalog was not used (snapshot source) + */ + +/** + * Resolve a tier to a concrete model id. Chain, each step only when the previous is unavailable: + * 1. an explicit, non-snapshot id configured for the tier (a gateway alias, another vendor's + * model) is honored verbatim — `source: "config"`; + * 2. the newest model of the tier's family in the provider's live catalog (fresh cache, or a + * revalidated/refetched response, or the last cached copy when the request fails) — + * `source: "catalog"`; + * 3. the shipped snapshot id (model_tiers.json, or the provider's configured snapshot id) — + * `source: "snapshot"`, with the reason the catalog was not used. + * Never throws; network calls are short and only happen here, never on import. + * @param {string} tier + * @param {ResolveOpts} [opts] + * @returns {ResolvedModel|null} null for an unknown tier + */ +export function resolveTierModel(tier, opts = {}) { + const m = MODELS[tier]; + if (!m) return null; + const family = familyOfTier(tier); + const configured = opts.provider?.models?.[tier] ?? null; + if (configured && !isSnapshotId(configured, tier)) + return { id: configured, family, source: "config" }; + const snapshotId = configured ?? m.id; + /** @type {ResolvedModel} */ + const snapshot = { id: snapshotId, family, source: "snapshot", displayName: m.name }; + try { + const src = /** @type {any} */ (catalogSource(opts)); + if (!src.kind) return { ...snapshot, reason: src.reason }; + const host = catalogHost(src.url); + const cat = fetchCatalog(src, opts); + if (!cat) return { ...snapshot, reason: `${host} catalog unavailable` }; + const pick = newestInFamily(cat.models, family, { namespace: namespaceOf(snapshotId) }); + if (!pick) return { ...snapshot, reason: `no ${family} model in the ${host} catalog` }; + return { + id: pick.id, + family, + source: "catalog", + ...(pick.createdAt ? { createdAt: pick.createdAt } : {}), + ...(pick.displayName ? { displayName: pick.displayName } : {}), + catalog: cat.url, + cache: cat.cache, + }; + } catch { + return snapshot; + } +} + +/** + * @typedef {object} ResolvedPrice + * @property {number} inCost USD per million input tokens + * @property {number} outCost USD per million output tokens + * @property {"catalog"|"snapshot"} source + * @property {"exact"|"registry"|"family"} [basis] snapshot only: the id's own row, the registry's row, or the family tier's + * @property {string} [matchedId] the catalog/registry row that priced it + * @property {string} [catalog] + * @property {string} [cache] + */ + +// Snapshot prices for an id, most specific first: the tier table's own row (with its dated +// windows), then the universal router's registry (data/models.json + .forge/models.json), then the +// tier of the id's family. The registry is read lazily and once. +let _registry = null; +function registryModels(root) { + if (!_registry || _registry.root !== root) + _registry = { root, models: loadRegistry(root ?? null).models }; + return _registry.models; +} + +/** + * @param {string} modelId + * @param {{date?: string, root?: string|null}} [opts] + * @returns {ResolvedPrice|null} + */ +function snapshotPrice(modelId, { date, root } = {}) { + const key = canonicalKey(modelId); + const own = TIER_ORDER.find((t) => canonicalKey(MODELS[t].id) === key); + if (own) + return { ...priceOf(own, date), source: "snapshot", basis: "exact", matchedId: MODELS[own].id }; + for (const r of registryModels(root)) { + if (r.price_in == null || r.price_out == null) continue; + const ids = [r.id, r.run_model_id, ...Object.values(r.providers ?? {})].filter(Boolean); + if (ids.some((id) => canonicalKey(id) === key)) + return { + inCost: r.price_in, + outCost: r.price_out, + source: "snapshot", + basis: "registry", + matchedId: r.id, + }; + } + const tier = tierOfModel(modelId); + if (tier) + return { + ...priceOf(tier, date), + source: "snapshot", + basis: "family", + matchedId: MODELS[tier].id, + }; + return null; +} + +/** + * Price an arbitrary model id (per million tokens): the live OpenRouter catalog, else the + * snapshot (exact row → registry → the family's tier). null when nothing prices it — callers + * report it as unpriced rather than guessing a number. + * @param {string} modelId + * @param {ResolveOpts & {date?: string, priceCatalog?: any}} [opts] priceCatalog: a catalog from + * model_catalog.fetchPriceCatalog, to price many ids with one lookup + * @returns {ResolvedPrice|null} + */ +export function resolveModelPrice(modelId, opts = {}) { + if (!modelId) return null; + try { + const live = catalogPrice(modelId, opts); + if (live) return { ...live, source: "catalog" }; + } catch {} + try { + return snapshotPrice(modelId, opts); + } catch { + return null; + } +} + +/** + * The price of a tier's RESOLVED model: live when OpenRouter lists that id, else the tier's + * snapshot price (its dated window for `opts.date`) — but only for a model of the tier's own + * family: a configured id from another vendor or a gateway alias is not priced as a Claude tier + * (null = unknown). Pass `resolved` to reuse a resolution. + * @param {string} tier + * @param {ResolveOpts & {date?: string, resolved?: ResolvedModel|null}} [opts] + * @returns {ResolvedPrice|null} + */ +export function resolveTierPrice(tier, opts = {}) { + if (!MODELS[tier]) return null; + const resolved = opts.resolved ?? resolveTierModel(tier, opts); + if (!resolved) return null; + try { + const live = catalogPrice(resolved.id, opts); + if (live) return { ...live, source: "catalog" }; + } catch {} + if (!inFamily(resolved, familyOfTier(tier))) return null; + const exact = canonicalKey(resolved.id) === canonicalKey(MODELS[tier].id); + return { + ...priceOf(tier, opts.date), + source: "snapshot", + basis: exact ? "exact" : "family", + matchedId: MODELS[tier].id, + }; +} + +/** + * Every tier resolved at once — the data behind `forge models`. + * @param {ResolveOpts & {date?: string}} [opts] + */ +export function resolveTiers(opts = {}) { + return TIER_ORDER.map((tier) => { + const model = resolveTierModel(tier, opts); + const price = resolveTierPrice(tier, { ...opts, resolved: model }); + return { tier, class: MODELS[tier].tier, family: familyOfTier(tier), model, price }; + }); +} + +/** + * One line saying where a resolved id came from — shown by `forge route`, `forge models` and the + * gateway config, so a user can always see WHY a tier maps to the id it does. + * @param {ResolvedModel|null} r + * @returns {string} + */ +export function describeResolution(r) { + if (!r) return ""; + if (r.source === "catalog") { + const created = r.createdAt ? `, created ${r.createdAt.slice(0, 10)}` : ""; + const stale = r.cache === "stale" ? ", last cached copy (catalog unreachable)" : ""; + return `newest ${r.family} in the ${catalogHost(r.catalog)} catalog${created}${stale}`; + } + if (r.source === "config") return "configured for this provider"; + return `shipped snapshot, pricing verified ${PRICING_VERIFIED}${r.reason ? ` (${r.reason})` : ""}`; +} diff --git a/src/providers.js b/src/providers.js index 328103b9..0d5e170c 100644 --- a/src/providers.js +++ b/src/providers.js @@ -6,8 +6,7 @@ import { execFileSync } from "node:child_process"; import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; -import { gatewayModelId } from "./gateway_model_map.js"; -import { MODELS } from "./model_tiers.js"; +import { isSnapshotId, MODELS, resolveTierModel } from "./model_tiers.js"; const PROVIDERS_FILE = "providers.json"; const PROVIDERS_DIR = ".forge"; @@ -171,22 +170,19 @@ export function envModelOverride() { return process.env.ANTHROPIC_MODEL?.trim() || process.env.FORGE_MODEL?.trim() || null; } -/** True when `id` is a stock Anthropic public id from model_tiers (i.e. the passthrough default, - * not a deliberately-configured gateway alias). Only stock ids are candidates for a gateway remap. */ -function isStockId(id) { - return id != null && Object.values(MODELS).some((m) => m.id === id); -} - /** Resolve a tier key (haiku/sonnet/opus/fable) to the active provider's model ID. - * When the provider is a custom gateway and the resolved id is a stock Anthropic public id the - * gateway may not serve, remap it onto a real advertised model (gateway_model_map). Explicit - * aliases and direct-Anthropic setups are returned untouched — silent, zero-breaking fallback. */ + * A configured id that is only the shipped snapshot's placeholder (a stock id, bare or + * namespaced) resolves through the provider's live catalog to the newest model of the tier's + * family (model_tiers.resolveTierModel — a custom gateway's /v1/models, the Anthropic Models API, + * or OpenRouter's list), falling back to that snapshot id. Explicit aliases (a gateway's + * forge-simple, another vendor's model) are returned untouched. */ export function resolveModel(root, tierKey) { const override = envModelOverride(); if (override) return override; const provider = activeProvider(root); const configured = provider.models?.[tierKey] ?? MODELS[tierKey]?.id ?? null; - return isStockId(configured) ? gatewayModelId(tierKey, configured) : configured; + if (!isSnapshotId(configured, tierKey)) return configured; + return resolveTierModel(tierKey, { root, provider })?.id ?? configured; } /** Switch the active provider. Returns the new active config. */ diff --git a/src/route.js b/src/route.js index 649af2c0..e1232e06 100644 --- a/src/route.js +++ b/src/route.js @@ -11,7 +11,8 @@ import { recordRoute, routeRef } from "./cost_report.js"; import { choice, jevEnabled, systemOne } from "./jev.js"; import { mergedLessons } from "./ledger_read.js"; import { setOverlap } from "./math.js"; -import { MODELS } from "./model_tiers.js"; +import { printableLine } from "./model_catalog.js"; +import { describeResolution, MODELS, resolveTierModel } from "./model_tiers.js"; import { preflightRepo, referencedEntities } from "./preflight.js"; import { promotionGate } from "./promote.js"; import { activeProvider, envModelOverride } from "./providers.js"; @@ -710,11 +711,23 @@ export function meterRoute(root, task, rec) { } catch {} } +// The tiers the gateway exposes as request aliases (forge-); fable is never a routing default. +const GATEWAY_TIERS = ["haiku", "sonnet", "opus"]; +const ANTHROPIC_UPSTREAM = { + name: "anthropic", + type: "anthropic", + baseUrl: "https://api.anthropic.com", +}; + /** Emit a LiteLLM config exposing the complexity tiers as aliases (request the one `forge route` picks). - * Provider-aware: uses the active provider's model IDs for the passthrough entries - * and the correct LiteLLM model prefix (anthropic/ for direct, openrouter/ for OR). - * Returns `{ ok: false, reason }` for hosted gateways the user cannot configure. */ -export function emitGatewayConfig(root = process.cwd()) { + * Provider-aware: each tier's model id is RESOLVED (model_tiers.resolveTierModel) — the newest + * model of the tier's family in the upstream's live catalog (OpenRouter's for an OpenRouter + * provider, the Anthropic Models API otherwise), else the shipped snapshot — behind the LiteLLM + * prefix for that upstream (anthropic/ or openrouter/). Each alias says where its id came from. + * Returns `{ ok: false, reason }` for hosted gateways the user cannot configure. + * @param {string} [root] + * @param {{fetchImpl?: Function, env?: Record}} [opts] catalog test seams */ +export function emitGatewayConfig(root = process.cwd(), { fetchImpl, env } = {}) { const prov = activeProvider(root); if (prov._autoDetected && prov._source === "LITELLM_BASE_URL") { return { @@ -725,7 +738,51 @@ export function emitGatewayConfig(root = process.cwd()) { `Use standard model names (the gateway handles routing).`, }; } - const prefix = prov.type === "openrouter" ? "openrouter/" : "anthropic/"; + const openrouter = prov.type === "openrouter"; + const prefix = openrouter ? "openrouter/" : "anthropic/"; + // The config's upstream: OpenRouter for an OpenRouter provider; otherwise LiteLLM's anthropic/ + // provider, i.e. the Anthropic API itself (even when the active provider is this gateway). + const upstream = openrouter ? prov : ANTHROPIC_UPSTREAM; + const resolved = GATEWAY_TIERS.map((tier) => ({ + m: MODELS[tier], + r: resolveTierModel(tier, { root, provider: upstream, fetchImpl, env }), + })); + const bare = (id) => String(id).split("/").pop(); + // Ids and display names come from a LIVE catalog (a gateway's or OpenRouter's /v1/models) and + // land in a file the user feeds to LiteLLM as routing config. A display name carrying a + // newline would otherwise splice in a SECOND entry for a tier alias, and LiteLLM's + // simple-shuffle would then send a share of that tier's prompts to the spliced model. So every + // catalog-sourced value is a double-quoted YAML scalar with control characters escaped, and + // every comment is collapsed to one line. + const yamlStr = (s) => { + let out = '"'; + for (const ch of String(s)) { + const n = ch.charCodeAt(0); + if (ch === "\\" || ch === '"') out += `\\${ch}`; + else if (n < 32 || n === 127) out += `\\x${n.toString(16).padStart(2, "0")}`; + else out += ch; + } + return `${out}"`; + }; + // One printable line for a comment: the same rule the catalog applies to display names. + const comment = printableLine; + const aliases = resolved.map(({ m, r }) => + [ + ` - model_name: forge-${m.tier.padEnd(8)} # ${comment(r.displayName ?? m.name)} — ${comment(m.use)}`, + ` # id: ${comment(describeResolution(r))}`, + ` litellm_params: { model: ${yamlStr(prefix + r.id)} }`, + ].join("\n"), + ); + // Passthrough: each resolved id, plus the snapshot id when it differs and the upstream is + // Anthropic — a client still pinned to the older id keeps working through the gateway. + const passIds = []; + for (const { m, r } of resolved) + for (const id of [r.id, openrouter ? null : m.id]) + if (id && !passIds.includes(id)) passIds.push(id); + const passthrough = passIds.map( + (id) => + ` - model_name: ${yamlStr(bare(id))}\n litellm_params: { model: ${yamlStr(prefix + id)} }`, + ); const path = join(root, "litellm.config.yaml"); const body = `# Forge Preflight — LiteLLM routing config (complexity tier -> model). # HOW ROUTING WORKS: LiteLLM routes by the REQUESTED model name; it cannot infer task @@ -735,22 +792,13 @@ export function emitGatewayConfig(root = process.cwd()) { # pip install "litellm[proxy]==" # supply-chain: pin exact, no floating tag # litellm --config litellm.config.yaml # then export ANTHROPIC_BASE_URL=http://localhost:4000 # Provider: ${prov.label || prov.name} (${prov.type}) -# Models verified 2026-07-05; re-verify via dev-radar. +# Model ids were resolved when this file was written (each "# id:" line says from where); +# re-run 'forge route gateway' to pick up newer models, 'forge models' to preview them. model_list: # Tier aliases — request one of these (per 'forge route') to pick a model by complexity. - - model_name: forge-simple # ${MODELS.haiku.name} — ${MODELS.haiku.use} - litellm_params: { model: ${prefix}${MODELS.haiku.id} } - - model_name: forge-medium # ${MODELS.sonnet.name} — default - litellm_params: { model: ${prefix}${MODELS.sonnet.id} } - - model_name: forge-complex # ${MODELS.opus.name} - litellm_params: { model: ${prefix}${MODELS.opus.id} } +${aliases.join("\n")} # Passthrough — a normal claude-* request still works when pointed at the gateway. - - model_name: ${MODELS.haiku.id} - litellm_params: { model: ${prefix}${MODELS.haiku.id} } - - model_name: ${MODELS.sonnet.id} - litellm_params: { model: ${prefix}${MODELS.sonnet.id} } - - model_name: ${MODELS.opus.id} - litellm_params: { model: ${prefix}${MODELS.opus.id} } +${passthrough.join("\n")} ${prov.envKey ? `litellm_settings:\n drop_params: true\n set_verbose: false` : ""} router_settings: routing_strategy: simple-shuffle diff --git a/test/_catalog_stub.js b/test/_catalog_stub.js new file mode 100644 index 00000000..7a17adfd --- /dev/null +++ b/test/_catalog_stub.js @@ -0,0 +1,49 @@ +// Shared stubs for the model-catalog tests: a scripted, synchronous transport in place of the +// real network (the suite is hermetic — no test may open a socket), plus catalog-shaped bodies. + +/** + * A scripted transport: the first route whose key is a substring of the URL answers; unmatched + * URLs get `null` (a network failure). Every request is recorded, headers included. + * @param {Record} routes url-substring → response | (req) => response + */ +export function stubTransport(routes) { + const calls = []; + const fetchImpl = (req) => { + calls.push(req); + for (const [match, respond] of Object.entries(routes)) { + if (req.url.includes(match)) return typeof respond === "function" ? respond(req) : respond; + } + return null; + }; + return { fetchImpl, calls }; +} + +/** A 200 JSON response with the given response headers. */ +export const ok = (body, headers = {}) => ({ status: 200, headers, body: JSON.stringify(body) }); + +/** An Anthropic Models API page: rows are [id, created_at, display_name?]. */ +export function anthropicPage(rows, { hasMore = false } = {}) { + return { + data: rows.map(([id, created_at, display_name]) => ({ + type: "model", + id, + display_name: display_name ?? id, + created_at, + })), + has_more: hasMore, + first_id: rows[0]?.[0] ?? null, + last_id: rows.at(-1)?.[0] ?? null, + }; +} + +/** An OpenRouter /api/v1/models body: rows are [id, promptPerToken, completionPerToken, created?]. */ +export function openRouterBody(rows) { + return { + data: rows.map(([id, prompt, completion, created]) => ({ + id, + name: id, + created: created ?? 1_700_000_000, + pricing: { prompt, completion }, + })), + }; +} diff --git a/test/_setup.js b/test/_setup.js index 31ff7f8e..fea92309 100644 --- a/test/_setup.js +++ b/test/_setup.js @@ -49,3 +49,9 @@ process.env.USERPROFILE = home; // now-keyless provider resolution returns null and the runner throws synchronously — the // exact fail-safe path CI already takes. No subprocess, no socket, no timeout. process.env.FORGE_LLM_HTTP = "1"; + +// The model catalogs (src/model_catalog.js) need NO key for OpenRouter's public price list, so a +// scrubbed env alone does not keep `forge route`/`forge cost`/`forge models` off the network. +// FORGE_NO_CATALOG_FETCH=1 turns the default transport off: every catalog lookup falls back to +// the shipped snapshot. Tests that exercise the live path inject a stub fetchImpl instead. +process.env.FORGE_NO_CATALOG_FETCH = "1"; diff --git a/test/adjudicate.test.js b/test/adjudicate.test.js index e3c6b4b9..228e5a65 100644 --- a/test/adjudicate.test.js +++ b/test/adjudicate.test.js @@ -7,7 +7,10 @@ import { buildRunner, extractJson, llmEnabled, + runnerModel, } from "../src/adjudicate.js"; +import { MODELS } from "../src/model_tiers.js"; +import { anthropicPage, ok, stubTransport } from "./_catalog_stub.js"; const parseScore = (o) => { const score = asUnit(o.score); @@ -165,3 +168,31 @@ test("buildRunner: FORGE_LLM_HTTP=1 with no provider → runner throws descripti } } }); + +test("runnerModel: a tier resolves through the catalog; overrides and literal ids pass verbatim", () => { + const t = stubTransport({ + "api.anthropic.com": ok( + anthropicPage([ + ["claude-haiku-9", "2027-01-01T00:00:00Z"], + ["claude-haiku-4-5-20251001", "2025-10-01T00:00:00Z"], + ]), + ), + }); + const env = { ANTHROPIC_API_KEY: "sk-test" }; + assert.equal(runnerModel("haiku", { root: null, fetchImpl: t.fetchImpl, env }), "claude-haiku-9"); + assert.equal( + runnerModel("claude-some-literal-id", { root: null, fetchImpl: t.fetchImpl, env }), + "claude-some-literal-id", + ); + assert.equal( + runnerModel("haiku", { root: null, fetchImpl: () => null, env }), + MODELS.haiku.id, + "snapshot fallback", + ); + process.env.ANTHROPIC_MODEL = "pinned-model"; + try { + assert.equal(runnerModel("haiku", { root: null, fetchImpl: t.fetchImpl, env }), "pinned-model"); + } finally { + delete process.env.ANTHROPIC_MODEL; + } +}); diff --git a/test/cost_report.test.js b/test/cost_report.test.js index e08de594..f41bba2a 100644 --- a/test/cost_report.test.js +++ b/test/cost_report.test.js @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; -import { homedir, tmpdir } from "node:os"; +import { EOL, homedir, tmpdir } from "node:os"; import { join } from "node:path"; import { test } from "node:test"; import { @@ -15,6 +15,7 @@ import { } from "../src/cost_report.js"; import { read as readMetrics } from "../src/metrics.js"; import { substrateCheck } from "../src/substrate.js"; +import { ok, openRouterBody, stubTransport } from "./_catalog_stub.js"; const tmp = () => mkdtempSync(join(tmpdir(), "forge-cost-")); @@ -254,6 +255,39 @@ test("estimateSpendFromLogs: prices cache tokens and counts a repeated response assert.ok(Math.abs(opus.cost - 0.23755) < 1e-9, "was $0.038 before (input+output only)"); }); +test("estimateSpendFromLogs: live catalog price by id, family fallback, unknown models unpriced", () => { + const dir = join(homedir(), ".claude", "projects", "cost-pricing-fixture"); + mkdirSync(dir, { recursive: true }); + const line = (id, model, input, output) => + JSON.stringify({ + message: { id, model, usage: { input_tokens: input, output_tokens: output } }, + }); + writeFileSync( + join(dir, "s.jsonl"), + [ + line("p1", "claude-3-opus-20240229", 1_000_000, 0), + line("p2", "claude-opus-9-20300101", 1_000_000, 0), + line("p3", "", 1_000_000, 0), + "", + ].join(EOL), + ); + const t = stubTransport({ + "openrouter.ai": ok(openRouterBody([["anthropic/claude-3-opus", "0.000015", "0.000075"]])), + }); + const est = estimateSpendFromLogs({ root: null, fetchImpl: t.fetchImpl }); + const by = Object.fromEntries(est.byModel.map((m) => [m.model, m])); + assert.equal(by["claude-3-opus-20240229"].cost, 15, "OpenRouter's live $15/M input"); + assert.equal(by["claude-3-opus-20240229"].priceSource, "catalog"); + assert.equal( + by["claude-opus-9-20300101"].priceSource, + "snapshot:family", + "an unknown Opus is priced as the Opus tier", + ); + assert.equal(by[""].priced, false); + assert.deepEqual(est.unpriced, [""], "never billed at a guessed $3/$15"); + assert.equal(t.calls.length, 1, "one catalog request per estimate, not one per model"); +}); + test("renderCostReport: measured factors print as percentages with event counts", () => { const root = tmp(); seed(root, [ diff --git a/test/gateway_model_map.test.js b/test/gateway_model_map.test.js index 82dcb73e..a3ce4753 100644 --- a/test/gateway_model_map.test.js +++ b/test/gateway_model_map.test.js @@ -64,6 +64,17 @@ test("buildGatewayMap maps each tier to the best family-matching advertised id", assert.ok(!map.fable, "no fable model advertised → tier omitted (caller keeps the stock id)"); }); +test("buildGatewayMap takes the NEWEST family member, not the snapshot's version", () => { + // The snapshot pins Sonnet 5; a gateway that already serves a Sonnet 6 gets Sonnet 6. + assert.equal(buildGatewayMap(["claude-sonnet-5", "prod-sonnet-6"]).sonnet.id, "prod-sonnet-6"); + // A gateway that reports creation times is ordered by them (Anthropic-shaped rows). + const map = buildGatewayMap([ + { id: "team-opus-a", createdAt: "2026-01-01T00:00:00Z" }, + { id: "team-opus-b", createdAt: "2026-03-01T00:00:00Z" }, + ]); + assert.equal(map.opus.id, "team-opus-b"); +}); + test("buildGatewayMap breaks ties toward the id closest to the canonical name", () => { // Both contain "sonnet" + "5" → equal overlap score; the shorter, less-noisy id wins. const map = buildGatewayMap(["vendor-region-prod-sonnet-5-preview", "claude-sonnet-5"]); diff --git a/test/hermetic.test.js b/test/hermetic.test.js index eeda88d7..7e61bd9c 100644 --- a/test/hermetic.test.js +++ b/test/hermetic.test.js @@ -27,14 +27,17 @@ const NOT_SCRUBBED = new Set(["TERM", "TOKEN", "X"]); // test above), so a src read of them sees that home, never the developer's. They count as a // leak again the moment one holds the real home. const SANDBOXED = new Set(["HOME", "USERPROFILE"]); +// Set (not scrubbed) by _setup: each forces a network-free path, pinned by the test below. +const SET_BY_SETUP = new Set(["FORGE_LLM_HTTP", "FORGE_NO_CATALOG_FETCH"]); test("every env var src reads is scrubbed (the denylist cannot drift from envVarsRead)", () => { const realHome = userInfo().homedir; const leaked = [...envVarsRead()].filter( - // FORGE_LLM_HTTP is set BY _setup on purpose: it forces the keyless HTTP runner. + // FORGE_LLM_HTTP and FORGE_NO_CATALOG_FETCH are set BY _setup on purpose: they force the + // keyless HTTP runner and keep the model catalogs off the network. (v) => !NOT_SCRUBBED.has(v) && - v !== "FORGE_LLM_HTTP" && + !SET_BY_SETUP.has(v) && !(SANDBOXED.has(v) && process.env[v] !== realHome) && process.env[v] !== undefined, ); diff --git a/test/http_cache.test.js b/test/http_cache.test.js new file mode 100644 index 00000000..f82d27bc --- /dev/null +++ b/test/http_cache.test.js @@ -0,0 +1,248 @@ +// The private HTTP cache behind the model catalogs. Freshness must come from the RESPONSE (its +// Cache-Control / Expires / validators), never from a TTL in forge's code — so every test here +// scripts the headers and moves an injected clock, and no test touches the network. +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { existsSync, mkdtempSync, readdirSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { + cachedGetJson, + currentAge, + freshnessLifetime, + httpGet, + parseCacheControl, +} from "../src/http_cache.js"; + +const URL_ = "https://catalog.example/v1/models"; +const tmpDir = () => join(mkdtempSync(join(tmpdir(), "forge-httpcache-")), ".forge", "cache"); +const T0 = Date.parse("2026-09-22T12:00:00Z"); + +/** A transport that answers from a queue and records what it was asked. */ +function scripted(...responses) { + const calls = []; + return { + calls, + fetchImpl: (req) => { + calls.push(req); + const next = responses.shift(); + if (next instanceof Error) throw next; + return next ?? null; + }, + }; +} +const body = (v) => JSON.stringify(v); + +test("parseCacheControl reads directives with and without arguments", () => { + assert.deepEqual(parseCacheControl('max-age=60, No-Cache, private="x"'), { + "max-age": "60", + "no-cache": true, + private: "x", + }); + assert.deepEqual(parseCacheControl(undefined), {}); +}); + +test("freshness comes only from the response: max-age, then Expires − Date, else zero", () => { + assert.equal(freshnessLifetime({ "cache-control": "max-age=300" }, T0), 300); + assert.equal( + freshnessLifetime({ "cache-control": "max-age=300, no-cache" }, T0), + 0, + "no-cache forbids reuse without revalidation", + ); + assert.equal( + freshnessLifetime( + { expires: "Tue, 22 Sep 2026 13:00:00 GMT", date: "Tue, 22 Sep 2026 12:00:00 GMT" }, + T0, + ), + 3600, + ); + assert.equal(freshnessLifetime({ expires: "not a date" }, T0), 0, "invalid Expires = expired"); + assert.equal(freshnessLifetime({ etag: '"v1"' }, T0), 0, "no caching headers → no invented TTL"); + // Age: an upstream cache already held it 50 s, and it has sat here 10 s more. + assert.equal(currentAge({ age: "50" }, T0, T0 + 10_000), 60); +}); + +test("a fresh response is reused without any request; expiry triggers a conditional GET", () => { + const dir = tmpDir(); + const t = scripted( + { status: 200, headers: { "Cache-Control": "max-age=60", ETag: '"v1"' }, body: body([1]) }, + { status: 304, headers: { "cache-control": "max-age=60" }, body: "" }, + ); + const first = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 }); + // freshUntil is when this answer stops being fresh — max-age from when it arrived. + assert.deepEqual(first, { value: [1], cache: "network", url: URL_, freshUntil: T0 + 60_000 }); + + const reused = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + 59_000 }); + assert.equal(reused.cache, "fresh"); + assert.equal(reused.freshUntil, T0 + 60_000, "a reused copy keeps the original expiry"); + assert.equal(t.calls.length, 1, "inside max-age: no request at all"); + + const later = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + 61_000 }); + assert.deepEqual(later, { + value: [1], + cache: "revalidated", + url: URL_, + freshUntil: T0 + 61_000 + 60_000, + }); + assert.equal(t.calls.length, 2); + assert.equal(t.calls[1].headers["if-none-match"], '"v1"', "revalidation is conditional"); + + // The 304 refreshed the stored copy's clock: fresh again for another max-age. + assert.equal( + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + 100_000 }).cache, + "fresh", + ); + assert.equal(t.calls.length, 2); +}); + +test("no caching headers → revalidate on every use (Last-Modified validator)", () => { + const dir = tmpDir(); + const lm = "Mon, 21 Sep 2026 00:00:00 GMT"; + const t = scripted( + { status: 200, headers: { "last-modified": lm }, body: body({ a: 1 }) }, + { status: 304, headers: {}, body: "" }, + { status: 200, headers: {}, body: body({ a: 2 }) }, + ); + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 }); + const second = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + 1 }); + assert.equal(second.cache, "revalidated", "the very next use asks again"); + assert.equal(t.calls[1].headers["if-modified-since"], lm); + const third = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + 2 }); + assert.deepEqual(third.value, { a: 2 }, "a changed resource replaces the stored copy"); +}); + +test("a failed request serves the stored copy as stale; nothing stored → null", () => { + const dir = tmpDir(); + const t = scripted( + { status: 200, headers: { etag: '"v1"' }, body: body(["cached"]) }, + null, // offline / timeout + { status: 503, headers: {}, body: "busy" }, + { status: 200, headers: {}, body: "not json" }, + new Error("socket hang up"), + ); + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 }); + for (let i = 1; i <= 4; i++) { + const r = cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 + i }); + // A stale copy is already expired: freshUntil is now, so a caller holding it retries. + assert.deepEqual( + r, + { value: ["cached"], cache: "stale", url: URL_, freshUntil: T0 + i }, + `failure #${i}`, + ); + } + assert.equal(cachedGetJson(URL_, { dir: tmpDir(), fetchImpl: () => null, now: T0 }), null); +}); + +test("a 304 without Date does not inherit the old Date (the copy would look old again)", () => { + const dir = tmpDir(); + const t = scripted( + { + status: 200, + headers: { + "cache-control": "max-age=60", + etag: '"v1"', + date: "Tue, 22 Sep 2026 12:00:00 GMT", + }, + body: body([1]), + }, + { status: 304, headers: { "cache-control": "max-age=60" }, body: "" }, + ); + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 }); + const hourLater = T0 + 3_600_000; + assert.equal( + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: hourLater }).cache, + "revalidated", + ); + assert.equal( + cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: hourLater + 30_000 }).cache, + "fresh", + ); +}); + +test("no-store is used once and never written; the cache dir ignores itself in git", () => { + const dir = tmpDir(); + const t = scripted({ status: 200, headers: { "cache-control": "no-store" }, body: body([1]) }); + assert.equal(cachedGetJson(URL_, { dir, fetchImpl: t.fetchImpl, now: T0 }).cache, "network"); + assert.equal(existsSync(dir), false, "nothing persisted for no-store"); + + const root = mkdtempSync(join(tmpdir(), "forge-httpcache-git-")); + execFileSync("git", ["init", "-q"], { cwd: root }); + const cacheDir = join(root, ".forge", "cache"); + cachedGetJson(URL_, { + dir: cacheDir, + fetchImpl: () => ({ status: 200, headers: { etag: '"x"' }, body: body([2]) }), + now: T0, + }); + const files = readdirSync(cacheDir); + assert.ok(files.includes(".gitignore")); + assert.match(readFileSync(join(cacheDir, ".gitignore"), "utf8"), /^\*$/m); + const status = execFileSync("git", ["status", "--porcelain", "--untracked-files=all"], { + cwd: root, + encoding: "utf8", + }); + assert.equal(status.trim(), "", "cached catalogs never show up as untracked files"); + const stored = files.find((f) => f.endsWith(".json")); + assert.match(stored, /^catalog\.example-/, "file name carries the host"); + assert.doesNotMatch(readFileSync(join(cacheDir, stored), "utf8"), /x-api-key|authorization/i); +}); + +test("FORGE_NO_CATALOG_FETCH=1 turns the real transport off without spawning anything", () => { + const prev = process.env.FORGE_NO_CATALOG_FETCH; + process.env.FORGE_NO_CATALOG_FETCH = "1"; + try { + const started = Date.now(); + assert.equal(httpGet({ url: "https://unreachable.invalid/v1/models" }), null); + assert.ok(Date.now() - started < 500, "no child process, no DNS, no timeout"); + } finally { + if (prev === undefined) delete process.env.FORGE_NO_CATALOG_FETCH; + else process.env.FORGE_NO_CATALOG_FETCH = prev; + } +}); + +// The one place the REAL transport runs: against a loopback server in a child process (never an +// external host). A child, because httpGet is synchronous — a server in this process could not +// answer while spawnSync blocks the event loop. +const LOOPBACK_SERVER = `const http=require("http");const s=http.createServer((q,r)=>{if(q.url.startsWith("/slow"))return;if(q.headers["if-none-match"]==='"v1"'){r.writeHead(304,{etag:'"v1"',"cache-control":"max-age=5"});return r.end();}r.writeHead(200,{"content-type":"application/json",etag:'"v1"',"cache-control":"max-age=5","x-unrelated":"dropped"});r.end(JSON.stringify({data:[{id:"m1"}],key:q.headers["x-api-key"]||null}));});s.listen(0,"127.0.0.1",()=>process.stdout.write(String(s.address().port)));`; + +test("httpGet: the real child transport — headers out, status/validators back, 304, timeout", async () => { + const { spawn } = await import("node:child_process"); + const server = spawn(process.execPath, ["-e", LOOPBACK_SERVER], { + stdio: ["ignore", "pipe", "inherit"], + }); + const prev = process.env.FORGE_NO_CATALOG_FETCH; + try { + const port = await new Promise((resolve, reject) => { + server.once("error", reject); + server.once("exit", (code) => reject(new Error(`loopback server exited (${code})`))); + server.stdout.once("data", (d) => resolve(Number(String(d).trim()))); + }); + delete process.env.FORGE_NO_CATALOG_FETCH; + const base = `http://127.0.0.1:${port}`; + const res = httpGet({ + url: `${base}/v1/models`, + headers: { "x-api-key": "k1" }, + timeoutMs: 5000, + }); + assert.equal(res.status, 200); + assert.deepEqual( + JSON.parse(res.body), + { data: [{ id: "m1" }], key: "k1" }, + "headers reach the server", + ); + assert.equal(res.headers.etag, '"v1"'); + assert.equal(res.headers["cache-control"], "max-age=5"); + assert.equal(res.headers["x-unrelated"], undefined, "only caching headers come back"); + + const notModified = httpGet({ url: `${base}/v1/models`, headers: { "if-none-match": '"v1"' } }); + assert.equal(notModified.status, 304, "a 304 is an answer, not a failure"); + + const started = Date.now(); + assert.equal(httpGet({ url: `${base}/slow`, timeoutMs: 300 }), null, "timeout → null"); + assert.ok(Date.now() - started < 4000, "the short timeout is honoured"); + } finally { + if (prev === undefined) delete process.env.FORGE_NO_CATALOG_FETCH; + else process.env.FORGE_NO_CATALOG_FETCH = prev; + server.kill(); + } +}); diff --git a/test/model_catalog.test.js b/test/model_catalog.test.js new file mode 100644 index 00000000..fcd6dd27 --- /dev/null +++ b/test/model_catalog.test.js @@ -0,0 +1,275 @@ +// The generic rules behind runtime model resolution: family membership, "newest", catalog +// normalization, pagination, cross-catalog id matching and per-token → per-million prices. +// Every catalog here is a stub — nothing is fetched. +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { + anthropicSource, + canonicalKey, + catalogSource, + fetchCatalog, + inFamily, + matchCatalogModel, + newestInFamily, + normalizeCatalogPage, + openRouterSource, + perMillion, +} from "../src/model_catalog.js"; +import { anthropicPage, ok, stubTransport } from "./_catalog_stub.js"; + +const tmpRoot = () => mkdtempSync(join(tmpdir(), "forge-catalog-")); + +test("perMillion converts OpenRouter's per-token strings without float noise", () => { + assert.equal(perMillion("0.000003"), 3); + assert.equal(perMillion("0.000015"), 15); + assert.equal(perMillion("0.0000008"), 0.8); + assert.equal(perMillion("0.00000125"), 1.25); + assert.equal(perMillion("3e-6"), 3); + assert.equal(perMillion(0.000075), 75); + assert.equal(perMillion("0"), 0, "a free model is priced at 0, not unpriced"); + for (const bad of ["-1", "", null, undefined, "n/a"]) + assert.equal(perMillion(bad), null, `${bad}`); +}); + +test("normalizeCatalogPage reads Anthropic, OpenAI-style, OpenRouter and bare-array shapes", () => { + const a = normalizeCatalogPage( + anthropicPage([["claude-x-1", "2026-01-02T00:00:00Z", "Claude X 1"]], { hasMore: true }), + ); + assert.deepEqual(a, { + models: [ + { id: "claude-x-1", displayName: "Claude X 1", createdAt: "2026-01-02T00:00:00.000Z" }, + ], + next: "claude-x-1", + }); + const oai = normalizeCatalogPage({ object: "list", data: [{ id: "m", created: 1_767_225_600 }] }); + assert.equal(oai.models[0].createdAt, "2026-01-01T00:00:00.000Z", "unix seconds"); + assert.equal(oai.next, null); + const or = normalizeCatalogPage({ + data: [{ id: "v/m", name: "V: M", pricing: { prompt: "0.000001", completion: "0.000005" } }], + }); + assert.deepEqual(or.models[0], { id: "v/m", displayName: "V: M", inCost: 1, outCost: 5 }); + assert.deepEqual(normalizeCatalogPage(["a", "", 7, "b"]).models, [{ id: "a" }, { id: "b" }]); + assert.equal(normalizeCatalogPage({ error: "nope" }), null, "not a catalog"); +}); + +test("family membership is a whole-token match on id or display name — no id list", () => { + assert.ok(inFamily({ id: "claude-opus-4-8" }, "opus")); + assert.ok(inFamily({ id: "anthropic/claude-3.5-sonnet" }, "sonnet")); + assert.ok(inFamily({ id: "vendor-model-9", displayName: "Claude Haiku 9" }, "haiku")); + assert.ok( + inFamily({ id: "any-new-vendor-name-opus-v12" }, "opus"), + "a never-seen id still matches", + ); + assert.ok(!inFamily({ id: "octopus-large" }, "opus"), "substring is not membership"); + assert.ok(!inFamily({ id: "claude-opus4" }, "opus"), "the family word must stand alone"); + assert.ok(!inFamily({ id: "claude-mythos-5-1" }, "fable")); +}); + +test("newest is the catalog's created_at — not list order, not the version number", () => { + const models = [ + { id: "claude-opus-4-8", createdAt: "2026-05-01T00:00:00Z" }, + { id: "claude-opus-5", createdAt: "2026-08-01T00:00:00Z" }, + { id: "claude-opus-4-9-preview", createdAt: "2026-07-01T00:00:00Z" }, + { id: "claude-sonnet-9", createdAt: "2027-01-01T00:00:00Z" }, + ]; + assert.equal(newestInFamily(models, "opus").id, "claude-opus-5"); + // A later point release of an older generation IS newer by the catalog's own clock. + const patch = [...models, { id: "claude-opus-4-8-1", createdAt: "2026-09-01T00:00:00Z" }]; + assert.equal(newestInFamily(patch, "opus").id, "claude-opus-4-8-1"); + assert.equal(newestInFamily(models, "haiku"), null); +}); + +test("an epoch created_at means 'release date unknown', not 'oldest model'", () => { + // The Models API docs: created_at "may be set to an epoch value if the release date is + // unknown". A new model listed that way must still win on version against older dated ones. + const models = [ + { id: "claude-opus-5", createdAt: "2026-07-24T00:00:00Z" }, + { id: "claude-opus-5-5", createdAt: "1970-01-01T00:00:00Z" }, + ]; + assert.equal(newestInFamily(models, "opus").id, "claude-opus-5-5"); + // Same version: the dated row beats the undated one. + const sameVersion = [ + { id: "claude-opus-5", createdAt: "1970-01-01T00:00:00Z" }, + { id: "claude-opus-5-20260724", createdAt: "2026-07-24T00:00:00Z" }, + ]; + assert.equal(newestInFamily(sameVersion, "opus").id, "claude-opus-5-20260724"); +}); + +test("undated catalogs (gateways) fall back to version, then snapshot date, then the plainest id", () => { + assert.equal( + newestInFamily([{ id: "claude-3-5-sonnet-20241022" }, { id: "claude-sonnet-4-5" }], "sonnet") + .id, + "claude-sonnet-4-5", + ); + assert.equal( + newestInFamily( + [{ id: "claude-3-5-sonnet-20240620" }, { id: "claude-3-5-sonnet-20241022" }], + "sonnet", + ).id, + "claude-3-5-sonnet-20241022", + "same version: the later snapshot stamp wins", + ); + assert.equal( + newestInFamily([{ id: "vendor-prod-sonnet-5-preview" }, { id: "claude-sonnet-5" }], "sonnet") + .id, + "claude-sonnet-5", + ); +}); + +test("a namespace restricts a multi-vendor catalog; a :variant yields to its base id", () => { + const models = [ + { id: "anthropic/claude-sonnet-5", createdAt: "2026-06-01T00:00:00Z" }, + { id: "anthropic/claude-sonnet-5:thinking", createdAt: "2026-06-02T00:00:00Z" }, + { id: "othervendor/sonnet-remix", createdAt: "2026-09-01T00:00:00Z" }, + ]; + assert.equal( + newestInFamily(models, "sonnet", { namespace: "anthropic" }).id, + "anthropic/claude-sonnet-5", + ); + assert.equal(newestInFamily(models, "sonnet").id, "othervendor/sonnet-remix"); +}); + +test("ids match across catalogs by canonical tokens (separators, namespace, snapshot date)", () => { + assert.equal(canonicalKey("claude-opus-4-8"), canonicalKey("anthropic/claude-opus-4.8")); + assert.equal( + canonicalKey("claude-haiku-4-5-20251001"), + canonicalKey("anthropic/claude-haiku-4.5"), + ); + assert.equal( + canonicalKey("claude-3-5-sonnet-20241022"), + canonicalKey("anthropic/claude-3.5-sonnet"), + ); + assert.notEqual(canonicalKey("claude-sonnet-4-5"), canonicalKey("claude-sonnet-4")); + assert.notEqual(canonicalKey("claude-3.7-sonnet"), canonicalKey("claude-3.7-sonnet:thinking")); + const rows = [ + { id: "anthropic/claude-3.7-sonnet:thinking" }, + { id: "anthropic/claude-3.7-sonnet" }, + { id: "anthropic/claude-opus-4.8" }, + ]; + assert.equal( + matchCatalogModel("claude-3-7-sonnet-20250219", rows).id, + "anthropic/claude-3.7-sonnet", + ); + assert.equal(matchCatalogModel("claude-opus-4-8", rows).id, "anthropic/claude-opus-4.8"); + assert.equal(matchCatalogModel("claude-opus-5", rows), null, "no guess for an unlisted id"); +}); + +test("catalogSource: env-derived like llm.js, provider-aware when given one", () => { + assert.equal(catalogSource({ env: {} }).kind, null, "no key → no catalog"); + const direct = catalogSource({ env: { ANTHROPIC_API_KEY: "sk-a" } }); + assert.equal(direct.kind, "anthropic"); + assert.equal(direct.url, "https://api.anthropic.com/v1/models?limit=1000"); + assert.deepEqual(direct.headers, { "x-api-key": "sk-a", "anthropic-version": "2023-06-01" }); + assert.equal( + catalogSource({ env: { ANTHROPIC_AUTH_TOKEN: "t" } }).kind, + null, + "the Models API is only asked with an API key", + ); + const gw = catalogSource({ + env: { ANTHROPIC_BASE_URL: "http://gw:4000/", LITELLM_API_KEY: "k" }, + }); + assert.equal(gw.kind, "gateway"); + assert.equal(gw.url, "http://gw:4000/v1/models"); + assert.equal(gw.headers.authorization, "Bearer k"); + assert.equal( + catalogSource({ + env: { ANTHROPIC_BASE_URL: "https://api.anthropic.com/", ANTHROPIC_API_KEY: "a" }, + }).kind, + "anthropic", + "the default base URL is direct Anthropic, not a gateway", + ); + const or = catalogSource({ + provider: { type: "openrouter", baseUrl: "https://openrouter.ai/api/v1" }, + env: {}, + }); + assert.deepEqual(or, openRouterSource()); + assert.equal( + catalogSource({ provider: { name: "openai", format: "openai" }, env: {} }).kind, + null, + ); + assert.equal( + catalogSource({ provider: { type: "litellm", baseUrl: "http://localhost:4000" }, env: {} }) + .kind, + "gateway", + ); +}); + +test("fetchCatalog follows has_more/last_id with after_id and persists each page", () => { + const root = tmpRoot(); + const page1 = anthropicPage( + [ + ["claude-a-2", "2026-02-01T00:00:00Z"], + ["claude-a-1", "2026-01-01T00:00:00Z"], + ], + { hasMore: true }, + ); + const page2 = anthropicPage([["claude-b-1", "2025-01-01T00:00:00Z"]]); + const { fetchImpl, calls } = stubTransport({ + after_id: (req) => (req.url.includes("after_id=claude-a-1") ? ok(page2) : null), + "/v1/models": ok(page1), + }); + const cat = fetchCatalog(anthropicSource("sk"), { root, fetchImpl }); + assert.deepEqual( + cat.models.map((m) => m.id), + ["claude-a-2", "claude-a-1", "claude-b-1"], + ); + assert.equal(cat.pages, 2); + assert.match(calls[1].url, /limit=1000&after_id=claude-a-1$/); + assert.equal(calls[0].headers["x-api-key"], "sk"); + // Both pages were persisted: offline, the whole catalog still comes back (marked stale). + const offline = fetchCatalog(anthropicSource("sk"), { root, fetchImpl: () => null }); + assert.equal(offline.models.length, 3); + assert.equal(offline.cache, "stale"); + + // A page that cannot be fetched (and was never cached) makes the whole catalog unavailable. + const broken = stubTransport({ after_id: null, "/v1/models": ok(page1) }); + assert.equal( + fetchCatalog(anthropicSource("sk"), { root: tmpRoot(), fetchImpl: broken.fetchImpl }), + null, + ); + + // A server that repeats the same cursor forever is stopped, not followed. + const loop = stubTransport({ "/v1/models": ok(page1) }); + const looped = fetchCatalog(anthropicSource("sk"), { root: null, fetchImpl: loop.fetchImpl }); + assert.equal(looped.models.length, 2); + assert.ok(loop.calls.length <= 2); +}); + +test("normalizeCatalogPage drops rows whose id is not a plausible model id", () => { + // Catalog ids reach generated routing config and model calls, so an id that is not + // id-shaped (whitespace, control characters) is dropped at the boundary, not sanitized later. + const page = anthropicPage([ + ["claude-opus-5", "2026-08-01T00:00:00Z", "Claude Opus 5"], + ["claude opus with spaces", "2026-08-02T00:00:00Z", "spaces"], + ["anthropic/claude-opus-5:batch", "2026-08-01T00:00:00Z", "a :variant id is fine"], + ]); + page.data.push({ type: "model", id: `x${String.fromCharCode(10)}y`, created_at: "2026-08-03" }); + const { models } = /** @type {{models: any[]}} */ (normalizeCatalogPage(page)); + assert.deepEqual( + models.map((m) => m.id), + ["claude-opus-5", "anthropic/claude-opus-5:batch"], + ); + // A display name is shown to a person, so it is kept — as one printable line. + const noisy = `Two${String.fromCharCode(10)}lines${String.fromCharCode(7)}here`; + const named = normalizeCatalogPage(anthropicPage([["m-1", "2026-08-01T00:00:00Z", noisy]])); + assert.equal(named?.models[0].displayName, "Two lines here"); +}); + +test("fetchCatalog reports when its answer expires (the memo uses it, no invented TTL)", () => { + const T = 1_800_000_000_000; + const withMaxAge = stubTransport({ + "api.anthropic.com": ok(anthropicPage([["claude-opus-5", "2026-08-01T00:00:00Z"]]), { + "cache-control": "max-age=120", + }), + }); + const a = fetchCatalog(anthropicSource("sk-a"), { fetchImpl: withMaxAge.fetchImpl, now: T }); + assert.equal(a?.freshUntil, T + 120_000, "expiry comes from the response's own max-age"); + const noHeaders = stubTransport({ + "api.anthropic.com": ok(anthropicPage([["claude-opus-5", "2026-08-01T00:00:00Z"]])), + }); + const b = fetchCatalog(anthropicSource("sk-a"), { fetchImpl: noHeaders.fetchImpl, now: T }); + assert.equal(b?.freshUntil, T, "no freshness stated → revalidate on the next use"); +}); diff --git a/test/model_tiers.test.js b/test/model_tiers.test.js index 76bffc55..7779c5be 100644 --- a/test/model_tiers.test.js +++ b/test/model_tiers.test.js @@ -1,6 +1,20 @@ import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { test } from "node:test"; -import { allPricePairs, priceOf } from "../src/model_tiers.js"; +import { + allPricePairs, + describeResolution, + MODELS, + priceOf, + resolveModelPrice, + resolveTierModel, + resolveTierPrice, + resolveTiers, + TIER_ORDER, +} from "../src/model_tiers.js"; +import { anthropicPage, ok, openRouterBody, stubTransport } from "./_catalog_stub.js"; // A synthetic table, so the window logic stays tested whether or not a real model currently // carries a schedule. @@ -53,3 +67,245 @@ test("allPricePairs includes both scheduled and flat prices", () => { assert.ok(has(real, 1, 5), "haiku flat price present"); assert.ok(has(real, 2, 10), "sonnet price present"); }); + +// --------------------------------------------------------------------------- +// Runtime resolution — a tier names a FAMILY; the id and price come from live catalogs. +// Every catalog below is a stub transport: no test touches the network. +// --------------------------------------------------------------------------- + +const KEY_ENV = { ANTHROPIC_API_KEY: "sk-ant-test" }; +const tmpRoot = () => mkdtempSync(join(tmpdir(), "forge-tiers-")); +const ANTHROPIC_TODAY = anthropicPage([ + ["claude-sonnet-5", "2026-06-01T00:00:00Z", "Claude Sonnet 5"], + ["claude-opus-4-8", "2026-05-01T00:00:00Z", "Claude Opus 4.8"], + ["claude-haiku-4-5-20251001", "2025-10-01T00:00:00Z", "Claude Haiku 4.5"], + ["claude-3-opus-20240229", "2024-02-29T00:00:00Z", "Claude 3 Opus"], +]); + +test("resolveTierModel: the newest family member in the live catalog, no code or data change", () => { + const today = stubTransport({ "api.anthropic.com": ok(ANTHROPIC_TODAY) }); + const before = resolveTierModel("opus", { + env: KEY_ENV, + root: tmpRoot(), + fetchImpl: today.fetchImpl, + }); + assert.equal(before.id, "claude-opus-4-8"); + assert.equal(before.source, "catalog"); + assert.equal(before.createdAt, "2026-05-01T00:00:00.000Z"); + + // Anthropic ships a new Opus. Same code, same model_tiers.json — the catalog alone moves the tier. + const release = anthropicPage([ + ["claude-opus-5", "2026-08-01T00:00:00Z", "Claude Opus 5"], + ...ANTHROPIC_TODAY.data.map((m) => [m.id, m.created_at, m.display_name]), + ]); + const tomorrow = stubTransport({ "api.anthropic.com": ok(release) }); + const after = resolveTierModel("opus", { + env: KEY_ENV, + root: tmpRoot(), + fetchImpl: tomorrow.fetchImpl, + }); + assert.equal(after.id, "claude-opus-5"); + assert.equal(after.displayName, "Claude Opus 5"); + assert.equal(MODELS.opus.id, "claude-opus-4-8", "the snapshot is untouched"); + // The other families are unaffected by an Opus release. + const sonnet = resolveTierModel("sonnet", { env: KEY_ENV, fetchImpl: tomorrow.fetchImpl }); + assert.equal(sonnet.id, "claude-sonnet-5"); + // The request is the documented Models API call. + assert.equal(today.calls[0].url, "https://api.anthropic.com/v1/models?limit=1000"); + assert.equal(today.calls[0].headers["x-api-key"], "sk-ant-test"); + assert.equal(today.calls[0].headers["anthropic-version"], "2023-06-01"); +}); + +test("resolveTierModel fallback chain: every step, each only when the previous is unavailable", () => { + // 1. no key → the snapshot, and it says why (no request made). + const none = stubTransport({}); + const noKey = resolveTierModel("haiku", { env: {}, fetchImpl: none.fetchImpl }); + assert.deepEqual( + [noKey.id, noKey.source, noKey.reason], + [MODELS.haiku.id, "snapshot", "no ANTHROPIC_API_KEY for the Models API"], + ); + assert.equal(none.calls.length, 0); + + // 2. offline / timeout / non-2xx with nothing cached → the snapshot. + for (const failure of [ + null, + { status: 500, headers: {}, body: "" }, + { status: 401, body: "{}" }, + ]) { + const t = stubTransport({ "api.anthropic.com": failure }); + const r = resolveTierModel("haiku", { env: KEY_ENV, root: tmpRoot(), fetchImpl: t.fetchImpl }); + assert.equal(r.source, "snapshot"); + assert.equal(r.reason, "api.anthropic.com catalog unavailable"); + } + + // 3. reachable, but no model of the family → the snapshot. + const noFable = stubTransport({ "api.anthropic.com": ok(ANTHROPIC_TODAY) }); + const fable = resolveTierModel("fable", { env: KEY_ENV, fetchImpl: noFable.fetchImpl }); + assert.equal(fable.id, MODELS.fable.id); + assert.equal(fable.reason, "no fable model in the api.anthropic.com catalog"); + + // 4. a cached catalog outlives a failed request: offline next time still gets the live answer. + const root = tmpRoot(); + const online = stubTransport({ "api.anthropic.com": ok(ANTHROPIC_TODAY, { etag: '"c1"' }) }); + resolveTierModel("opus", { env: KEY_ENV, root, fetchImpl: online.fetchImpl }); + const offline = resolveTierModel("opus", { env: KEY_ENV, root, fetchImpl: () => null }); + assert.deepEqual( + [offline.id, offline.source, offline.cache], + ["claude-opus-4-8", "catalog", "stale"], + ); + + // 5. a transport that throws is contained. + const boom = () => { + throw new Error("kaboom"); + }; + assert.equal(resolveTierModel("opus", { env: KEY_ENV, fetchImpl: boom }).source, "snapshot"); + assert.equal(resolveTierModel("no-such-tier"), null); +}); + +test("resolveTierModel revalidates a header-less catalog with If-None-Match on the next use", () => { + const root = tmpRoot(); + let n = 0; + const t = stubTransport({ + "api.anthropic.com": (req) => { + if (n++ === 0) return ok(ANTHROPIC_TODAY, { etag: '"cat-1"' }); + return req.headers["if-none-match"] === '"cat-1"' ? { status: 304, headers: {} } : null; + }, + }); + resolveTierModel("sonnet", { env: KEY_ENV, root, fetchImpl: t.fetchImpl }); + const again = resolveTierModel("sonnet", { env: KEY_ENV, root, fetchImpl: t.fetchImpl }); + assert.equal(t.calls.length, 2, "no TTL: the next use asks again"); + assert.equal(t.calls[1].headers["if-none-match"], '"cat-1"'); + assert.deepEqual([again.id, again.cache], ["claude-sonnet-5", "revalidated"]); + // With max-age on the response, the next use inside it makes no request at all. + const root2 = tmpRoot(); + const cached = stubTransport({ + "api.anthropic.com": ok(ANTHROPIC_TODAY, { "cache-control": "max-age=600" }), + }); + resolveTierModel("sonnet", { env: KEY_ENV, root: root2, fetchImpl: cached.fetchImpl }); + const hit = resolveTierModel("haiku", { env: KEY_ENV, root: root2, fetchImpl: cached.fetchImpl }); + assert.equal(cached.calls.length, 1); + assert.equal(hit.cache, "fresh"); +}); + +test("resolveTierModel honours explicit provider ids and namespaces OpenRouter picks", () => { + // An explicit alias is configuration, not a family placeholder: never resolved away. + const gw = { type: "litellm", baseUrl: "http://gw:4000", models: { haiku: "forge-simple" } }; + const t = stubTransport({}); + assert.deepEqual(resolveTierModel("haiku", { provider: gw, env: {}, fetchImpl: t.fetchImpl }), { + id: "forge-simple", + family: "haiku", + source: "config", + }); + assert.equal(t.calls.length, 0); + // OpenRouter: its own catalog, restricted to the vendor namespace the provider is set up for. + const orProvider = { + type: "openrouter", + baseUrl: "https://openrouter.ai/api/v1", + models: { opus: `anthropic/${MODELS.opus.id}` }, + }; + const or = stubTransport({ + "openrouter.ai": ok({ + data: [ + { id: "anthropic/claude-opus-4.8", created: 1_777_000_000 }, + { id: "anthropic/claude-opus-5", created: 1_785_000_000 }, + { id: "someone-else/opus-finetune", created: 1_790_000_000 }, + ], + }), + }); + const r = resolveTierModel("opus", { provider: orProvider, env: {}, fetchImpl: or.fetchImpl }); + assert.equal(r.id, "anthropic/claude-opus-5"); + assert.equal(r.source, "catalog"); +}); + +test("resolveTierPrice: live OpenRouter per-token price → per million, else the tier snapshot", () => { + const catalogs = stubTransport({ + "api.anthropic.com": ok( + anthropicPage([["claude-opus-5", "2026-08-01T00:00:00Z", "Claude Opus 5"]]), + ), + "openrouter.ai": ok( + openRouterBody([ + ["anthropic/claude-opus-5", "0.000005", "0.000025"], + ["anthropic/claude-sonnet-5", "0.0000021", "0.0000105"], + ]), + ), + }); + const opts = { env: KEY_ENV, fetchImpl: catalogs.fetchImpl }; + const opus = resolveTierPrice("opus", opts); + assert.deepEqual( + [opus.inCost, opus.outCost, opus.source, opus.matchedId], + [5, 25, "catalog", "anthropic/claude-opus-5"], + ); + // Sonnet: no Sonnet in this Anthropic catalog → the snapshot id, which OpenRouter prices live. + const sonnet = resolveTierPrice("sonnet", opts); + assert.deepEqual([sonnet.inCost, sonnet.outCost, sonnet.source], [2.1, 10.5, "catalog"]); + // Haiku: nobody lists it → the snapshot's own row. + assert.deepEqual(resolveTierPrice("haiku", { ...opts, date: "2026-09-22" }), { + inCost: 1, + outCost: 5, + source: "snapshot", + basis: "exact", + matchedId: MODELS.haiku.id, + }); + // A resolved id the snapshot does not know, priced offline → its tier's price ("family"). + const offlineOpus = resolveTierPrice("opus", { + ...opts, + resolved: { id: "claude-opus-5", family: "opus", source: "catalog" }, + fetchImpl: () => null, + }); + assert.deepEqual([offlineOpus.source, offlineOpus.basis], ["snapshot", "family"]); +}); + +test("resolveModelPrice (cost report): catalog → snapshot row → registry → family → unpriced", () => { + const or = stubTransport({ + "openrouter.ai": ok(openRouterBody([["anthropic/claude-3-opus", "0.000015", "0.000075"]])), + }); + const live = resolveModelPrice("claude-3-opus-20240229", { fetchImpl: or.fetchImpl }); + assert.deepEqual([live.inCost, live.outCost, live.source], [15, 75, "catalog"]); + const offline = { fetchImpl: () => null, date: "2026-09-22" }; + const exact = resolveModelPrice(MODELS.opus.id, offline); + assert.deepEqual([exact.inCost, exact.basis], [MODELS.opus.inCost, "exact"]); + const registry = resolveModelPrice("claude-sonnet-4-5-20250929", offline); + assert.deepEqual([registry.inCost, registry.outCost, registry.basis], [3, 15, "registry"]); + const family = resolveModelPrice("claude-opus-9-20300101", offline); + assert.deepEqual([family.inCost, family.basis], [MODELS.opus.inCost, "family"]); + assert.equal(resolveModelPrice("", offline), null, "no guessed $3/$15"); + assert.equal(resolveModelPrice("gpt-4o", offline), null); +}); + +test("resolveTiers + describeResolution: every tier, with where its id came from", () => { + const rows = resolveTiers({ env: {}, fetchImpl: () => null }); + assert.deepEqual( + rows.map((r) => r.tier), + TIER_ORDER, + ); + for (const r of rows) { + assert.equal(r.model.source, "snapshot"); + assert.equal(r.price.source, "snapshot"); + assert.match(describeResolution(r.model), /shipped snapshot, pricing verified \d{4}-\d\d-\d\d/); + } + assert.equal( + describeResolution({ + id: "x", + family: "opus", + source: "catalog", + catalog: "https://api.anthropic.com/v1/models?limit=1000", + createdAt: "2026-08-01T00:00:00.000Z", + cache: "stale", + }), + "newest opus in the api.anthropic.com catalog, created 2026-08-01, last cached copy (catalog unreachable)", + ); +}); + +test("resolveTierPrice never prices another vendor's configured model as a Claude tier", () => { + const openai = { name: "openai", format: "openai", models: { haiku: "gpt-5-nano" } }; + assert.equal( + resolveTierPrice("haiku", { provider: openai, env: {}, fetchImpl: () => null }), + null, + "unknown, not the Haiku snapshot price", + ); + const or = stubTransport({ + "openrouter.ai": ok(openRouterBody([["openai/gpt-5-nano", "0.00000005", "0.0000004"]])), + }); + const live = resolveTierPrice("haiku", { provider: openai, env: {}, fetchImpl: or.fetchImpl }); + assert.deepEqual([live.inCost, live.outCost, live.source], [0.05, 0.4, "catalog"]); +}); diff --git a/test/models_cli.test.js b/test/models_cli.test.js new file mode 100644 index 00000000..63e0bd9e --- /dev/null +++ b/test/models_cli.test.js @@ -0,0 +1,57 @@ +// `forge models` and the resolved-id line of `forge route`, end to end through the real CLI. +// The spawned CLI inherits test/_setup.js's FORGE_NO_CATALOG_FETCH=1 and scrubbed env, so it +// never reaches a network: every tier resolves to the shipped snapshot and SAYS so. +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { MODELS, TIER_ORDER } from "../src/model_tiers.js"; + +const CLI = fileURLToPath(new URL("../src/cli.js", import.meta.url)); +const run = (args) => + spawnSync(process.execPath, [CLI, ...args], { + cwd: mkdtempSync(join(tmpdir(), "forge-models-cli-")), + encoding: "utf8", + env: { ...process.env, FORGE_NO_HINT: "1", NO_COLOR: "1" }, + }); + +test("forge models --json: every tier, its resolved id, price and where both came from", () => { + const r = run(["models", "--json"]); + assert.equal(r.status, 0, r.stderr); + const out = JSON.parse(r.stdout); + assert.deepEqual( + out.tiers.map((t) => t.tier), + TIER_ORDER, + ); + for (const t of out.tiers) { + assert.equal(t.family, t.tier); + assert.equal(t.model.id, MODELS[t.tier].id); + assert.equal(t.model.source, "snapshot"); + assert.equal(t.model.reason, "no ANTHROPIC_API_KEY for the Models API"); + assert.equal(t.price.source, "snapshot"); + assert.equal(t.price.inCost, MODELS[t.tier].inCost); + } + assert.match(out.pricingVerified, /^\d{4}-\d\d-\d\d$/); +}); + +test("forge models: a readable table plus one provenance line per family", () => { + const r = run(["models"]); + assert.equal(r.status, 0, r.stderr); + for (const tier of TIER_ORDER) { + assert.ok(r.stdout.includes(MODELS[tier].id), `${tier} id shown`); + assert.match(r.stdout, new RegExp(`${tier}\\s+shipped snapshot, pricing verified`)); + } + assert.match(r.stdout, /prices: shipped snapshot, verified/); +}); + +test("forge route shows the concrete model the tier resolves to, and where it came from", () => { + const r = run(["route", "write an is_prime function"]); + assert.equal(r.status, 0, r.stderr); + assert.ok( + r.stdout.includes(`model: ${MODELS.haiku.id} — shipped snapshot, pricing verified`), + r.stdout, + ); +}); diff --git a/test/route.test.js b/test/route.test.js index c00b20db..9e90071d 100644 --- a/test/route.test.js +++ b/test/route.test.js @@ -19,6 +19,7 @@ import { routeTask, rubricComplexity, } from "../src/route.js"; +import { anthropicPage, ok, stubTransport } from "./_catalog_stub.js"; test("contentGrams: stopwords dropped, unigrams+bigrams kept", () => { const g = contentGrams("Implement a rate limiter with the token bucket"); @@ -166,11 +167,91 @@ test("emitGatewayConfig writes a LiteLLM config that never pins @latest", () => assert.match(yaml, /forge-simple/, "tier aliases present"); assert.match( yaml, - /model_name: claude-haiku/, + /model_name: "claude-haiku/, "passthrough for real model names so plain claude-* traffic works", ); }); +test("emitGatewayConfig resolves each tier from the live catalog and says so — no pinned date", () => { + const root = mkdtempSync(join(tmpdir(), "forge-route-")); + const offline = readFileSync(emitGatewayConfig(root), "utf8"); + assert.doesNotMatch(offline, /Models verified/, "no hard-coded verification date"); + assert.match(offline, /# id: shipped snapshot, pricing verified/); + + const t = stubTransport({ + "api.anthropic.com": ok( + anthropicPage([ + ["claude-opus-5", "2026-08-01T00:00:00Z", "Claude Opus 5"], + ["claude-opus-4-8", "2026-05-01T00:00:00Z", "Claude Opus 4.8"], + ["claude-sonnet-5", "2026-06-01T00:00:00Z", "Claude Sonnet 5"], + ["claude-haiku-4-5-20251001", "2025-10-01T00:00:00Z", "Claude Haiku 4.5"], + ]), + ), + }); + const yaml = readFileSync( + emitGatewayConfig(root, { fetchImpl: t.fetchImpl, env: { ANTHROPIC_API_KEY: "sk-test" } }), + "utf8", + ); + const lines = yaml.split(/\r?\n/).map((l) => l.trim()); + const alias = lines.findIndex((l) => l.startsWith("- model_name: forge-complex")); + assert.match(lines[alias], /Claude Opus 5/, "the catalog's display name"); + assert.equal( + lines[alias + 1], + "# id: newest opus in the api.anthropic.com catalog, created 2026-08-01", + ); + assert.equal(lines[alias + 2], 'litellm_params: { model: "anthropic/claude-opus-5" }'); + assert.ok(lines.includes('- model_name: "claude-opus-5"'), "the resolved id passes through"); + assert.ok( + lines.includes('- model_name: "claude-opus-4-8"'), + "a client pinned to the snapshot id still passes", + ); + assert.doesNotMatch(yaml, /@latest|:latest/); +}); + +test("emitGatewayConfig: a catalog string can never splice a second entry into the config", () => { + // The generated file is fed to LiteLLM as routing config, and ids/display names come from a + // live catalog. A name carrying a newline must not add an entry: with simple-shuffle, a second + // `forge-complex` would send a share of that tier's prompts to the spliced model. + const root = mkdtempSync(join(tmpdir(), "forge-route-")); + const evil = "\n - model_name: forge-complex\n litellm_params: { model: anthropic/pwned }\n#"; + const t = stubTransport({ + "api.anthropic.com": ok( + anthropicPage([ + [`claude-opus-9${evil}`, "2026-08-01T00:00:00Z", `Opus 9"${evil}`], + ["claude-sonnet-5", "2026-06-01T00:00:00Z", "Claude Sonnet 5"], + ["claude-haiku-4-5-20251001", "2025-10-01T00:00:00Z", "Claude Haiku 4.5"], + ]), + ), + }); + const yaml = readFileSync( + emitGatewayConfig(root, { fetchImpl: t.fetchImpl, env: { ANTHROPIC_API_KEY: "sk-test" } }), + "utf8", + ); + const lines = yaml.split(/\r?\n/); + const entries = lines.filter((l) => /^\s*- model_name:/.test(l)); + assert.equal( + entries.filter((l) => l.includes("forge-complex")).length, + 1, + `one complex alias only:\n${yaml}`, + ); + // Two layers hold: the id never passed SAFE_MODEL_ID, so the tier fell back to the snapshot + // and the hostile text is nowhere in the file… + assert.ok(!yaml.includes("pwned"), `the spliced text never reaches the file:\n${yaml}`); + assert.match( + lines[lines.findIndex((l) => l.includes("forge-complex")) + 2], + /model: "anthropic\/claude-opus-4-8"/, + "the complex tier falls back to the snapshot id", + ); + // …and every emitted value is a quoted scalar, so even a value that passed the id check + // could not end its own line. + assert.ok( + entries.every((l) => /^\s*- model_name: (forge-\w+\s|")/.test(l)), + `every entry name is a plain alias or a quoted id:\n${entries.join("\n")}`, + ); + for (const l of lines.filter((l) => l.includes("litellm_params"))) + assert.match(l, /litellm_params: \{ model: "[^"]*" \}$/, l); +}); + test("complexityLLM: parses a band into a score floor, rejects junk", () => { const cheap = complexityLLM("x", { run: () => '{"band":"cheap","reason":"trivial"}' }); assert.equal(cheap.band, "cheap");