Skip to content

Commit b76eebf

Browse files
committed
fix(codebuddy): 11158 归 model_not_found;耗尽无选号时明细落候选渠道
11158「套餐不允许」与 11102「无此模型」对当前凭证同义:重试无意义、换号 才有用、与账号健康无关。此前落 400 其他 → INVALID → invalid_request, 归因面板显示「请求无效」。两者共用 MODEL_BLOCKED_CODES,列表探活同源。 _pick 首轮返回 None 时 provider 还停在占位 '-',归因面板看不出哪个渠道 挂的(实测 deepseek-v4.1-flash-sg)。改为落收窄后的首个候选渠道。
1 parent bd48203 commit b76eebf

8 files changed

Lines changed: 95 additions & 9 deletions

File tree

‎README.en.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -512,7 +512,7 @@ Four tabs (channel / model / user / credential) share one table (channel / reque
512512

513513
**Two latency scopes**: successful requests only. Average = `latency_sum ÷ successes` (hourly rollup); percentile from per-request detail (90 days). `NULL` `ttfb_ms` ranks independently in each column. No samples → `—`.
514514

515-
**Failure attribution**: panel below group table, `GET /api/stats/errors` (operator+) aggregates `ok=0` detail by `(channel, model, error_type)` → failure count, **unrescued count**, last failure time (descending, `limit` ≤ 5000). **Failure count = per-attempt**: every upstream rejection lands row, even when retry on another credential succeeds; failure row whose `request_id` also has success row for **same user** counts as rescued; `unrescued = failures − rescued` = **failure attempts never rescued** (a request failing twice counts twice; not number of lost requests). Hard credential failure (DEAD, e.g. 401) also lands a `credential_unavailable` row; an upstream account restriction (CodeBuddy 11140) lands `account_restricted`, cools the account for 2h and stops rotating within that channel, escalating to a whole-channel pause on a second hit in the window; counted as rescued if retry succeeds elsewhere; terminal row not duplicated once rotation exhausted. Click row → detail with *failed only* (`GET /api/stats/events?ok=0`). Admin/operator only. `/metrics` `upstream_errors_total` still in-process counter; not merged.
515+
**Failure attribution**: panel below group table, `GET /api/stats/errors` (operator+) aggregates `ok=0` detail by `(channel, model, error_type)` → failure count, **unrescued count**, last failure time (descending, `limit` ≤ 5000). **Failure count = per-attempt**: every upstream rejection lands row, even when retry on another credential succeeds; failure row whose `request_id` also has success row for **same user** counts as rescued; `unrescued = failures − rescued` = **failure attempts never rescued** (a request failing twice counts twice; not number of lost requests). Hard credential failure (DEAD, e.g. 401) also lands a `credential_unavailable` row; an upstream account restriction (CodeBuddy 11140) lands `account_restricted`, cools the account for 2h and stops rotating within that channel, escalating to a whole-channel pause on a second hit in the window; when the upstream says this credential cannot use that model (11102 no such model / 11158 plan not allowed) the row is `model_not_found` and writes an (account, model) negative cache instead of cooling the account; when rotation exhausts without ever picking a credential (all candidates cooling / empty pool) the row records the **candidate channel** rather than `-`; counted as rescued if retry succeeds elsewhere; terminal row not duplicated once rotation exhausted. Click row → detail with *failed only* (`GET /api/stats/events?ok=0`). Admin/operator only. `/metrics` `upstream_errors_total` still in-process counter; not merged.
516516

517517
**Row drill-down**: clicking group row filters detail by that row (channel / model / credential; user dimension uses scope), stacking conditions. `GET /api/stats/events` accepts `provider` / `model` / `credential_id` (AND) + `ok=0`.
518518

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -521,7 +521,7 @@ tabs 切四维(渠道 / 模型 / 用户 / 凭证),共用一表(渠道 /
521521

522522
**延迟两类口径**:只统计成功请求。平均 = `latency_sum ÷ 成功数`(小时汇总);分位从逐请求明细算(仅 90 天)。`ttfb_ms` = `NULL` 的两列各自独立排名。无样本 → `—`。
523523

524-
**失败归因**:`GET /api/stats/errors`(operator+)按 `(渠道, 模型, 错误类型)` 聚合窗口内 `ok=0` → 失败次数 / **未救回数** / 最近失败时刻(降序,`limit` ≤ 5000)。**失败数 = 尝试口径**:换号重试每次上游拒绝各落一行;同用户同 `request_id` 另有成功行 = 已救回。`未救回 = 失败数 − 救回数`(同请求多次失败算多次,≠ 受损请求数)。凭证 401 硬禁用(DEAD)也落 `credential_unavailable`;账号被上游风控限制(CodeBuddy 11140)落 `account_restricted`,账号级 2h 冷却、当前请求不再换号,同渠道窗口内**第二个不同凭证**也中招时整渠道退避 24h;换号成功算已救回;轮换耗尽不重复落收尾行。点行钻取 +「只看失败」(`GET /api/stats/events?ok=0`)。仅 admin / operator。`/metrics` 的 `upstream_errors_total` 取进程内计数,不合并。
524+
**失败归因**:`GET /api/stats/errors`(operator+)按 `(渠道, 模型, 错误类型)` 聚合窗口内 `ok=0` → 失败次数 / **未救回数** / 最近失败时刻(降序,`limit` ≤ 5000)。**失败数 = 尝试口径**:换号重试每次上游拒绝各落一行;同用户同 `request_id` 另有成功行 = 已救回。`未救回 = 失败数 − 救回数`(同请求多次失败算多次,≠ 受损请求数)。凭证 401 硬禁用(DEAD)也落 `credential_unavailable`;账号被上游风控限制(CodeBuddy 11140)落 `account_restricted`,账号级 2h 冷却、当前请求不再换号,同渠道窗口内**第二个不同凭证**也中招时整渠道退避 24h;上游明确说该号用不了这个模型(11102 无此模型 / 11158 套餐不允许)落 `model_not_found`,写 (账号, 模型) 负缓存而非冷却账号;一次都没选中凭证就耗尽时(候选全在冷却 / 池子为空)明细落**候选渠道**而不是 `-`;换号成功算已救回;轮换耗尽不重复落收尾行。点行钻取 +「只看失败」(`GET /api/stats/events?ok=0`)。仅 admin / operator。`/metrics` 的 `upstream_errors_total` 取进程内计数,不合并。
525525

526526
**点行钻取**:点分组表任一行 → 明细按该行过滤(渠道 / 模型 / 凭证;用户维度走作用域参数),条件可叠加。`GET /api/stats/events` 接受 `provider` / `model` / `credential_id`(AND)+ `ok=0`。
527527

‎TECHNICAL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -652,6 +652,7 @@ PRAGMA foreign_keys = ON; -- 仅 api_keys 之外无外键(users 与 api_k
652652
`ErrKind.BLOCKED`(上游明确说没有这个模型)统计落 `error_type=model_not_found`,**不再并进 `invalid_request`**。后者是「请求无效」(`ErrKind.INVALID` / `ErrKind.REQUEST`),归因面板显示「请求无效」,读起来像客户端参数写错了——实际是上游把该模型下架了(实测 2026-10-26 officeAce 404 `ModelArts.81009` 被标成「请求无效」)。
653653

654654
- 受控枚举 `CONTROLLED_ERROR_TYPES` 增 `model_not_found`(前端 `USAGE_ERROR_LABEL` 同步:「上游无此模型」)。
655+
- **套餐不允许也归这里**(2026-10-29):CodeBuddy 11158 `model [...] is not allowed for user type [personal_free_user]` 与 11102「该后端无此模型」对**当前凭证**是同一件事——重试无意义、换号才有用、与账号健康无关。此前 11158 落进 `400 其他 → ErrKind.INVALID`,归因面板显示「请求无效」(实测 10:36 / 10:37 国际站免费号请求 `gpt-6-luna`、`gemini-3.8-flash`)。现在两者共用 `codebuddy.events.MODEL_BLOCKED_CODES`,列表探活(`client.filter_available_models`)也读同一个集合,不再各写一份。
655656
- 可用性视图(`GET /api/models/availability`)的计入口径增 `model_not_found`:它**仍属**模型级失败,只是分类更准。
656657
- **收尾行不变**:轮换耗尽时那一行仍按 `_exhaust_error_type` 落 `model_unavailable` / `no_healthy_credential`——「上游没这个模型」(尝试级归因)与「这个模型现在全都不可用」(收尾行)是两件事。
657658

‎src/engine/executor.py‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -398,6 +398,21 @@ def _skip_provider(self, provider_id: str, tried: set[str]) -> None:
398398
for candidate in self._deps.credentials.candidates([provider_id]):
399399
tried.add(candidate.credential_id)
400400

401+
def _attribution_provider(self, target: ModelTarget, observed: str) -> str:
402+
"""失败明细里的渠道名:真打过就用实际渠道,一次都没选中就用候选渠道。
403+
404+
`_pick` **首轮**返回 None(候选全在冷却 / 一个凭证都没有)时,渠道名
405+
还停在占位 `"-"`:归因面板只剩一个横杠,看不出是哪个渠道挂的(实测
406+
2026-10-29 10:18 `deepseek-v4.1-flash-sg`——唯一持有它的
407+
codebuddy_intl 两个号刚因 ConnectTimeout 进 SOFT 冷却,明细落成
408+
`provider="-"`)。收窄后的候选按声明顺序取第一个:那是「本该服务它」
409+
的渠道,也是运维该去查的那个。
410+
411+
候选必然非空:`_select` 在无注册上游时先抛 `NoProviderForModel`,
412+
走到 `_pick` 返回 None 的分支时注册集已校验过。
413+
"""
414+
return observed if observed != "-" else self._registered(target)[0]
415+
401416
def _upstream_model(self, provider_id: str, model: str) -> str:
402417
"""归一模型名 → 该上游注册的原始 id(大小写变体映射,未知则原样)。"""
403418
if self._deps.upstream_model_name is None:
@@ -681,7 +696,8 @@ def terminal(message: str, code: str) -> bytes:
681696
"invalid_request")
682697
if not attributed:
683698
self._deps.record(
684-
username=state.username, provider=state.provider,
699+
username=state.username,
700+
provider=self._attribution_provider(target, state.provider),
685701
credential_id=state.credential_id, model=target.model, ok=False,
686702
error_type=self._exhaust_error_type(target),
687703
latency_ms=_elapsed_ms(state.started))
@@ -1243,7 +1259,8 @@ async def _complete_model(self, request: ChatRequest, target: ModelTarget,
12431259
target.model, last_error, self._suggestions(target.model)))
12441260
if not attributed:
12451261
self._deps.record(
1246-
username=username, provider=last_provider,
1262+
username=username,
1263+
provider=self._attribution_provider(target, last_provider),
12471264
credential_id=last_credential,
12481265
model=target.model, ok=False,
12491266
error_type=self._exhaust_error_type(target),

‎src/provider/codebuddy/client.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
from ..lazy_clients import LazyClients
3333
from . import events as cb_events
3434
from .credential import CodeBuddyCredential, parse_credential
35-
from .events import UpstreamProtocolViolation
35+
from .events import MODEL_BLOCKED_CODES, UpstreamProtocolViolation
3636
from .headers import (
3737
CN_ENDPOINT,
3838
EP_CHAT,
@@ -69,7 +69,9 @@
6969
MODEL_ACCESS_CACHE_MAX_ENTRIES = 256
7070
MODEL_ACCESS_PROBE_TIMEOUT = 5.0
7171
MODEL_ACCESS_PROBE_CONCURRENCY = 4
72-
MODEL_ACCESS_DENIED_CODES = frozenset({11102, 11158})
72+
# 判活与错误分类共用同一个业务码集合(`events.MODEL_BLOCKED_CODES`):两边
73+
# 各写一份曾让 11158 在探活里算「不可用」、在请求里算「请求无效」。
74+
MODEL_ACCESS_DENIED_CODES = MODEL_BLOCKED_CODES
7375
_MODEL_ACCESS_CACHE: dict[tuple[str, str], tuple[float, dict[str, bool | None]]] = {}
7476
_PROBE_PROMPT = "hi"
7577

‎src/provider/codebuddy/events.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,16 @@ def _usage(raw: dict[str, Any]) -> Usage:
4343
# saturated」。上游明确要求「switch models」,说明同凭证其他模型可用,
4444
# 落账号级 SOFT 会让整张凭证 60s 内全部模型一起不可用。
4545
_MODEL_SATURATED_CODE = 14003
46-
_MODEL_BLOCKED_CODE = 11102 # 该后端无此模型 → (账号, 模型) 负缓存
46+
# 「这个号用不了这个模型」→ (账号, 模型) 负缓存。两个业务码同义:
47+
# 11102 该后端无此模型(上游根本没部署)
48+
# 11158 套餐不允许:`model [gpt-6-luna] is not allowed for user type
49+
# [personal_free_user]`(国际站免费号实测 2026-10-29 10:36 / 10:37)
50+
# 两者对**当前凭证**的结论完全一样——重试多少次都失败,换号才有意义,且与
51+
# 账号健康无关(不能落账号级冷却)。此前只认 11102,11158 掉进 `400 其他`
52+
# 被归成 `invalid_request`,归因面板显示「请求无效」,读起来像客户端参数
53+
# 写错了(与 2026-10-26 officeAce 81009 同一类误判)。
54+
# 列表探活(client.filter_available_models)共用本集合,口径不会再漂。
55+
MODEL_BLOCKED_CODES = frozenset({11102, 11158})
4756
_CREDITS_EXHAUSTED_CODE = 14018 # 积分耗尽 → 冷却到次日签到
4857
_PLAN_EXHAUSTED_CODE = 1005 # 权益/套餐耗尽
4958
# 请求级错误(不是账号的问题):请求体坏 / 上下文超限 / 图片无效 / 渠道风控。
@@ -77,7 +86,7 @@ def classify_status(status: int, body: bytes = b"") -> ErrKind:
7786
7887
* 402 / 14018 → 余额不足(冷却到次日 04:00,等签到恢复)
7988
* 1005 → 权益耗尽(12h 长冷却)
80-
* 11102「该后端无此模型」→ (账号, 模型) 负缓存(400/404 形态)
89+
* 11102/11158「该号用不了这个模型」→ (账号, 模型) 负缓存(400/404 形态)
8190
* 11140「账号被风控限制」→ 账号级长冷却,**当前请求不再换号**
8291
* 401/403 → session 失效(硬禁用)
8392
* 404 → 软冷却,不累计错误数
@@ -90,7 +99,7 @@ def classify_status(status: int, body: bytes = b"") -> ErrKind:
9099
return ErrKind.CREDIT
91100
if _PLAN_EXHAUSTED_CODE in codes:
92101
return ErrKind.PLAN
93-
if status in (400, 404) and _MODEL_BLOCKED_CODE in codes:
102+
if status in (400, 404) and codes & MODEL_BLOCKED_CODES:
94103
return ErrKind.BLOCKED
95104
if status >= 400 and _is_account_restricted(body):
96105
# 必须拦在 401/403 → DEAD 之前:11140 是上游按账号/IP 拉黑,不是凭据

‎tests/engine/test_executor_stats.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,58 @@ async def test_complete_all_model_cooled_records_model_unavailable(repo):
290290
{"messages": [{"role": "user", "content": "hi"}]}))
291291
assert collector.events[-1]["error_type"] == "model_unavailable"
292292

293+
async def test_stream_exhaustion_without_any_pick_records_candidate_provider(repo):
294+
"""一次都没选中凭证(候选全在冷却 / 池子为空)时,明细要落**候选渠道**。
295+
296+
`_pick` 首轮返回 None 时渠道名还停在占位 `"-"`:归因面板只剩一个横杠,
297+
看不出是哪个渠道挂的(实测 2026-10-29 10:18 `deepseek-v4.1-flash-sg`
298+
落成 `provider="-"`,而它只在 codebuddy_intl 上有)。
299+
"""
300+
credentials, _db = repo
301+
collector = RecordingCollector()
302+
executor = _exec_with_stats(repo, [GOOD_EVENTS], collector)
303+
304+
from src.compat.openai.request import parse_chat_request
305+
306+
async for _ in executor.stream(parse_chat_request(
307+
{"messages": [{"role": "user", "content": "hi"}], "stream": True})):
308+
pass
309+
event = collector.events[-1]
310+
assert event["ok"] is False and event["provider"] == "codebuddy"
311+
assert event["credential_id"] is None
312+
313+
async def test_complete_exhaustion_without_any_pick_records_candidate_provider(repo):
314+
"""非流式同口径:没有选中凭证时明细仍能指出是哪个渠道。"""
315+
credentials, _db = repo
316+
collector = RecordingCollector()
317+
executor = _exec_with_stats(repo, [GOOD_EVENTS], collector)
318+
319+
from src.compat.openai.request import parse_chat_request
320+
321+
with pytest.raises(NoHealthyCredential):
322+
await executor.complete(parse_chat_request(
323+
{"messages": [{"role": "user", "content": "hi"}]}))
324+
assert collector.events[-1]["provider"] == "codebuddy"
325+
326+
async def test_exhaustion_after_a_tried_credential_keeps_actual_provider(repo):
327+
"""真打过上游时明细记**实际**渠道,不被候选渠道顶掉。"""
328+
credentials, _db = repo
329+
credentials.add(provider="codebuddy", credential_data={"bearer_token": "t"})
330+
collector = RecordingCollector()
331+
332+
class Soft(Exception):
333+
def kind(self):
334+
return ErrKind.SOFT
335+
336+
executor = _exec_with_stats(repo, [[Soft()]], collector, max_rotate=1)
337+
from src.compat.openai.request import parse_chat_request
338+
339+
with pytest.raises(NoHealthyCredential):
340+
await executor.complete(parse_chat_request(
341+
{"messages": [{"role": "user", "content": "hi"}]}))
342+
assert collector.events[-1]["provider"] == "codebuddy"
343+
assert collector.events[-1]["credential_id"] is not None
344+
293345
async def test_complete_records_success_and_failure(repo):
294346
credentials, _db = repo
295347
credentials.add(provider="codebuddy", credential_data={"bearer_token": "t"})

‎tests/provider/codebuddy/test_client.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,10 @@ def test_classify_body_markers():
4444
# 「该后端无此模型」:(账号, 模型) 负缓存
4545
(400, b'{"code": 11102}', ErrKind.BLOCKED),
4646
(404, b'{"code": 11102}', ErrKind.BLOCKED),
47+
# 「套餐不允许」:对**当前凭证**与 11102 等价(重试无意义、换号才有用)
48+
(400, b'{"code": 11158, "msg": "model [gpt-6-luna] is not allowed for user '
49+
b'type [personal_free_user]"}', ErrKind.BLOCKED),
50+
(404, b'{"code": 11158}', ErrKind.BLOCKED),
4751
# 请求级错误:不罚号,仍换号(请求体坏 / 上下文超限 / 渠道风控 / 图片无效)
4852
(400, b'{"code": 11101}', ErrKind.REQUEST),
4953
(400, b'{"code": 500, "msg": "Unmarshal chat params failed"}', ErrKind.REQUEST),
@@ -55,6 +59,7 @@ def test_classify_body_markers():
5559
# 裸数字不误伤:111020 不是 11102(要求带 "code": 键值形态)
5660
(400, b'{"trace": 1110201}', ErrKind.INVALID),
5761
(400, b'{"code": 111020}', ErrKind.INVALID),
62+
(400, b'{"code": 111580}', ErrKind.INVALID),
5863
# 嵌套信封:业务码在内层,按集合成员判断不漏
5964
(400, b'{"code": 0, "data": {"code": 11102}}', ErrKind.BLOCKED),
6065
(404, b"not json at all", ErrKind.SOFT),

0 commit comments

Comments
 (0)