Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/852-model-retired.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A provider model its provider has stopped serving is recorded as retired with the date: Settings and the repository page say so, the job's model line names the reason, and an apply that would restart a workspace onto that model is refused with its name.
12 changes: 12 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1792,6 +1792,18 @@ settings form disables the checkbox of a provider whose allowed flag is off.
existing repository keeps the unrestricted behaviour until an owner sets a
policy.

A provider can stop offering a model it used to serve. The gateway already
treated a 404 whose body names the model as retryable so a job could finish on
another model; it now also writes `provider_models.retired_at` — the first date
that 404 was seen, never moved by a later one — and marks the attempt in the
job's `model_json` with `retired: true`. Settings shows the model with its
retirement date, the repository page marks a retired default and any per-bot
picker that points at it, and the job's model line names the reason beside the
status. An apply whose selection still holds a retired model is refused with
that model's name, so a restart cannot move a workspace onto a model its provider
no longer serves; saving the selection itself still works. Only a 404 matching the
retired-model pattern writes the column — a 429, a 5xx and any other 404 never do.

### Per-job cost, evidence and metrics

The bridge's model proxy normally uses the per-repository instance token. While
Expand Down
6 changes: 6 additions & 0 deletions docs/quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,12 @@ writes one of those rows rather than none: the provider's `x-litellm-response-co
figure as a known USD row when the response carried one, and a `cost_known = 0`
row with reason `no_usage`, its audit entry and its job event when it did not.

An attempt that never reached a model is not billed: a `404` whose body says the
model does not exist writes no ledger row, so a model its provider has retired
costs nothing, and the fallback to the next selected model is priced on its own
answer. The retirement itself is recorded on the provider model
(`provider_models.retired_at`, #852), not in the ledger.

For streaming requests Fledge adds `stream_options.include_usage = true` and
parses each SSE `data:` line. It keeps the last `usage` object, including nested
`completion_tokens_details` / `prompt_tokens_details`, and meters after the
Expand Down
10 changes: 10 additions & 0 deletions src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4378,6 +4378,16 @@ export async function buildApp(options: BuildOptions): Promise<BuiltApp> {

app.post('/repos/:id/models/apply', async (request, reply) => {
const { viewer, repo } = ownedRepo(request, String((request.params as Record<string, string>).id), true, true)
// #852: a retired model is still in the selection, so applying would restart
// the workspace onto a model its provider no longer serves. Refused by name
// rather than deleted: the maintainer picks a live model and nothing is lost.
const retiredSelection = store.listRepoModelSelection(repo.id).find((model) => model.retired_at !== null && model.retired_at !== undefined)
if (retiredSelection !== undefined) {
const t = translator(request.locale ?? 'zh')
const on = String(retiredSelection.retired_at).slice(0, 10)
const message = t(`模型“${retiredSelection.label}”已于 ${on} 被提供方下架,请先选择一个可用的模型再应用。`, `The model "${retiredSelection.label}" was retired by its provider on ${on}. Choose a live model before applying.`)
return reply.redirect(`/repos/${repo.id}?error=${encodeURIComponent(message)}#repo-settings`)
}
try {
const result = await instances.applyModels(repo.id)
if (!result.ok) return reply.redirect(`/repos/${repo.id}?error=${encodeURIComponent(result.error ?? '模型应用失败。')}#repo-settings`)
Expand Down
25 changes: 24 additions & 1 deletion src/db.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ const { DatabaseSync } = require('node:sqlite') as { DatabaseSync: new (path: st
// restore check (#652) has to prove a fixture written by an older build still
// migrates to exactly this number, and a literal in two places would let the
// check and the migration disagree.
export const SCHEMA_VERSION = 44
export const SCHEMA_VERSION = 45

export type UserStatus = 'onboarding' | 'pending' | 'approved' | 'rejected'
export type UserRole = 'user' | 'admin' | 'superadmin'
Expand Down Expand Up @@ -383,6 +383,8 @@ export interface ProviderModelRow {
provider_id: string
model: string
label: string
/** #852: the first day a provider answered "this model does not exist"; null while live. */
retired_at: string | null
created_at: string
}

Expand Down Expand Up @@ -1493,6 +1495,7 @@ CREATE TABLE IF NOT EXISTS provider_models (
provider_id TEXT NOT NULL REFERENCES model_providers(id) ON DELETE CASCADE,
model TEXT NOT NULL,
label TEXT NOT NULL,
retired_at TEXT,
created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS provider_models_provider_idx ON provider_models(provider_id, created_at ASC);
Expand Down Expand Up @@ -2416,6 +2419,7 @@ export class Store {
this.migrateColumns()
this.migrateRepoColumns()
this.migrateProviderColumns()
this.migrateProviderModelColumns()
this.migrateWebhookColumns()
this.migrateArtifactColumns()
this.migrateManagementColumns()
Expand Down Expand Up @@ -2839,6 +2843,14 @@ export class Store {
this.run("UPDATE jobs SET base_branch = (SELECT upstream_default_branch FROM repos WHERE repos.id = jobs.repo_id) WHERE base_branch IS NULL")
}

// #852: a provider can stop offering a model it used to serve, and the row must
// carry that fact so the console can say it. Additive: a row written before the
// column keeps NULL, which is exactly the "live" it was always shown as.
private migrateProviderModelColumns(): void {
const columns = new Set(this.all<{ name: string }>('PRAGMA table_info(provider_models)').map((row) => row.name))
if (!columns.has('retired_at')) this.run('ALTER TABLE provider_models ADD COLUMN retired_at TEXT')
}

private migrateProviderColumns(): void {
const columns = new Set(this.all<{ name: string }>('PRAGMA table_info(model_providers)').map((row) => row.name))
const additions: Array<[string, string]> = [
Expand Down Expand Up @@ -4870,6 +4882,17 @@ export class Store {
)
}

// #852: the first date wins. A second 404 for the same model must not move it,
// so the write is decided by the WHERE clause rather than by the caller.
markProviderModelRetired(modelId: string, at: string): boolean {
return this.transaction(() => {
const live = this.get<{ id: string }>('SELECT id FROM provider_models WHERE id = ? AND retired_at IS NULL', modelId)
if (live === undefined) return false
this.run('UPDATE provider_models SET retired_at = ? WHERE id = ?', at, modelId)
return true
})
}

createProviderModel(input: { id: string; providerId: string; model: string; label: string }): ProviderModelRow {
this.run('INSERT INTO provider_models (id, provider_id, model, label, created_at) VALUES (?, ?, ?, ?, ?)', input.id, input.providerId, input.model, input.label, nowIso())
return this.get<ProviderModelRow>('SELECT * FROM provider_models WHERE id = ?', input.id)!
Expand Down
10 changes: 8 additions & 2 deletions src/gateway/gateway.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ import { log } from '../lib/log.js'
import { decryptSecret } from '../lib/secrets.js'
import { headerCost, quotaView } from './quota.js'
import { fallbackPriceTable, parsePriceTable, requestCost, type PriceTable } from '../metrics/cost.js'
import { modelOrder, retryableModelFailure } from './modelFallback.js'
import { modelOrder, retiredModelFailure, retryableModelFailure } from './modelFallback.js'
import { nowIso } from '../lib/time.js'
import { parseProviderAllowlist, providerPolicyRefusal } from './modelUrl.js'
import { parseModelRun } from '../lib/modelRun.js'
import type { ModelAttempt, ModelFallback } from '../lib/modelRun.js'
Expand Down Expand Up @@ -318,7 +319,12 @@ export function registerGateway(app: FastifyInstance, deps: GatewayDeps): void {
}
const text = await readAnswer(answer.res)
const retryable = retryableModelFailure(answer.status, text)
attempts.push({ ...attemptName, status: answer.status, retryable })
// #852: a retired model is the one 404 that is not about the URL. The row is
// written once with the first date, so the console can say when it happened;
// a 429, a 5xx or any other 404 never writes it.
const retired = retiredModelFailure(answer.status, text)
if (retired && candidate !== undefined) store.markProviderModelRetired(candidate.id, nowIso())
attempts.push({ ...attemptName, status: answer.status, retryable, ...(retired ? { retired: true } : {}) })
lastFailure = { status: answer.status, text, headers: { ...forwardHeaders(answer.headers), ...quotaHeaders() } }
// Auth, an exhausted quota, a payment problem and a policy refusal are the
// repository's own answer: a second model must not mask them.
Expand Down
10 changes: 9 additions & 1 deletion src/gateway/modelFallback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ export function modelOrder(
const QUOTA_WORDS = /insufficient_quota|quota|billing|payment|budget|额度|余额/i
const POLICY_WORDS = /content[_ ]?policy|policy[_ ]?violation|safety|refus/i

// #852: the body that makes a 404 retryable is also the body that says the model
// is gone for good, so the two decisions share one pattern and cannot drift.
const RETIRED_MODEL_WORDS = /model[^\n]{0,60}(not found|not exist|unavailable|decommission|retired)/i

export function retiredModelFailure(status: number, body: string): boolean {
return status === 404 && RETIRED_MODEL_WORDS.test(body)
}

export function retryableModelFailure(status: number, body: string): boolean {
// Auth is the repository's credential, not the model's health: another model
// behind the same key is no more likely to be accepted.
Expand All @@ -35,7 +43,7 @@ export function retryableModelFailure(status: number, body: string): boolean {
if (status === 408) return true
// A retired model is the one "unavailable" a 404 can mean; any other 404 is a
// wrong URL, which another model will not repair.
if (status === 404) return /model[^\n]{0,60}(not found|not exist|unavailable|decommission|retired)/i.test(body)
if (status === 404) return retiredModelFailure(status, body)
if (status === 500 || status === 502 || status === 503 || status === 504) return true
return false
}
3 changes: 3 additions & 0 deletions src/lib/modelRun.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ export interface ModelAttempt {
label: string
status: number
retryable: boolean
/** #852: the provider answered that this model does not exist, so it is retired. */
retired?: boolean
}

// The first fallback the job made, kept apart from `attempts` because those are
Expand Down Expand Up @@ -40,6 +42,7 @@ function attempt(value: unknown): ModelAttempt | null {
label: typeof row.label === 'string' && row.label !== '' ? row.label : modelId,
status,
retryable: row.retryable === true,
...(row.retired === true ? { retired: true } : {}),
}
}

Expand Down
Loading
Loading