Skip to content

fix(agent): handle mid-stream quota_limit and upstream_waf_challenge with auto account rotation in runOpenAIAgentTurn #166

Description

@Yash12212

Problem Description:
When running multi-turn OpenAI Agent requests with tool-calling enabled via handleOpenAIAgentStream(), sendChatRequest() handles account failover only during connection establishment / preflight chunk 1.

However, if an upstream account hits its hourly quota limit mid-stream (code: 'quota_limit') or triggers WAF validation (code: 'upstream_waf_challenge'), assertNoUpstreamFailure() in openai-agent-runtime.js throws an UpstreamResponseError. Because collectOpenAIAgentAttempt() inside runOpenAIAgentTurn() is not caught for account-level rotation, the error bubbles directly to chat.js, returning 502 Qwen 上游返回业务错误 to the client instead of auto-rotating to another healthy account in the pool.

Stack Trace:

[ERROR] [AGENT] ❌ OpenAI Agent 回合处理失败
UpstreamResponseError: Qwen 上游返回业务错误
    at assertNoUpstreamFailure (src/utils/upstream-error.js:41:11)
    at src/utils/openai-agent-runtime.js:138:5
    at consumeFrames (src/utils/sse.js:153:19)
    at async collectOpenAIAgentAttempt (src/utils/openai-agent-runtime.js:134:24)
    at async runOpenAIAgentTurn (src/utils/openai-agent-runtime.js:423:21)
    ...
  code: 'quota_limit',
  publicMessage: 'Qwen 上游返回业务错误'

Proposed Solution / Diff:
In src/utils/openai-agent-runtime.js, catch UpstreamResponseError for quota_limit / upstream_waf_challenge, record the cooldown against the exhausted account, and invoke requestSender without account binding to rotate to the next available account in the pool:

--- a/src/utils/openai-agent-runtime.js
+++ b/src/utils/openai-agent-runtime.js
@@ -7,7 +7,8 @@ const {
 } = require('./tool-prompt.js')
 const { consumeSSEStream, createUpstreamResponseFilter } = require('./sse.js')
 const { createUpstreamDeltaNormalizer } = require('./chat-helpers.js')
-const { assertNoUpstreamFailure } = require('./upstream-error.js')
+const { assertNoUpstreamFailure, UpstreamResponseError } = require('./upstream-error.js')
+const accountManager = require('./account.js')
 const {
   parseAgentControlText,
   createAgentControlStreamParser,
@@ -420,10 +421,43 @@ const runOpenAIAgentTurn = async (initialResponse, options = {}) => {
 
   for (let attemptNumber = 1; attemptNumber <= maxAttempts; attemptNumber++) {
     attemptsMade = attemptNumber
-    const attempt = await collectOpenAIAgentAttempt(currentResponse, {
-      ...options,
-      attempt_number: attemptNumber
-    })
+    let attempt
+    try {
+      attempt = await collectOpenAIAgentAttempt(currentResponse, {
+        ...options,
+        attempt_number: attemptNumber
+      })
+    } catch (streamErr) {
+      if (streamErr instanceof UpstreamResponseError && (streamErr.code === 'quota_limit' || streamErr.code === 'upstream_waf_challenge')) {
+        const activeAccount = options.currentAccount?.email || upstreamContext.currentAccount?.email
+        if (activeAccount) {
+          if (streamErr.code === 'quota_limit') {
+            logger.warn(`Agent attempt ${attemptNumber} 遭遇配额耗尽 (${activeAccount}),已置入冷却并自动轮换账户重试`, 'AGENT')
+            accountManager.recordAccountQuotaLimit(activeAccount)
+          } else {
+            logger.warn(`Agent attempt ${attemptNumber} 遭遇 WAF 验证 (${activeAccount}),已置入冷却并自动轮换账户重试`, 'AGENT')
+            accountManager.recordAccountFailure(activeAccount, 'WAF_CHALLENGE')
+          }
+        }
+        if (typeof requestSender === 'function' && attemptNumber < maxAttempts) {
+          const rotateResponse = await requestSender(retryBaseBody, {
+            chatId: null,
+            parentId: null,
+            currentAccount: null,
+            agentRetry: true
+          })
+          if (rotateResponse?.status && rotateResponse.response) {
+            currentResponse = rotateResponse.response
+            upstreamContext = mergePresent(upstreamContext, {
+              chatId: rotateResponse.chatId,
+              currentAccount: rotateResponse.currentAccount
+            })
+            continue
+          }
+        }
+      }
+      throw streamErr
+    }

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions