Skip to content

Probe current model's reasoning ladder first in ACP-native discovery - #3008

Open
bradhallett wants to merge 1 commit into
get-bb:mainfrom
bradhallett:fix/acp-reasoning-probe-priority
Open

Probe current model's reasoning ladder first in ACP-native discovery#3008
bradhallett wants to merge 1 commit into
get-bb:mainfrom
bradhallett:fix/acp-reasoning-probe-priority

Conversation

@bradhallett

Copy link
Copy Markdown
Contributor

Problem

When an ACP agent exposes a large model catalog, most models show only a Medium reasoning effort in the model picker, even though the agent advertises a full ladder (low/medium/high/…) per model.

Root cause: discoverAcpNativeReasoningByModel probes models serially for their thought_level options under a fixed 5s budget (ACP_NATIVE_REASONING_DISCOVERY_TIMEOUT_MS = 5_000). With a 374-model catalog (current omp), the probe reaches only the first ~10–50 models before the deadline; every unprobed model falls back to the medium-only ACP_NATIVE_REASONING_EFFORTS entry. The session's currently selected model is frequently deep in the list (e.g. zai/* sorts last), so the model the user is actually on is the least likely to have real reasoning levels.

Change

  1. Probe the current model first. modelsToProbe is now seeded with the model config option's currentValue (already captured at session/new) before the reasoningProbePriorityModelIds ordering. The active model always gets its real ladder within budget.
  2. Env-overridable discovery deadlineacpNativeReasoningDiscoveryTimeoutMs() reads ACP_NATIVE_REASONING_DISCOVERY_TIMEOUT_MS (default 5_000 unchanged), mirroring the sessionBusyRetryTimeoutMs() pattern. Operators with small catalogs can raise it; tests can shrink it.

Non-current model ordering is unchanged. No behavior change for clients that don't hit the deadline.

Tests

  • New: "probes the current model's reasoning ladder first under a tight discovery deadline" — 500ms deadline, 8-model fixture with 150ms probe latency, current model at position 8. Asserts the current model reports its real [low, medium, high] ladder while an unprobed neighbor keeps the medium-only fallback.
    • Kill 1 (revert priority seeding): current model never probed → real-ladder assertion fails.
    • Kill 2 (revert env helper to fixed 5s): all probes complete → unprobed-neighbor fallback assertion fails.
  • Updated: the Cursor probe-order test now expects currentValue ("default") first — the seeding change legitimately reorders it.
  • Fixture knobs: FAKE_ACP_INITIAL_MODEL (set the model option's currentValue) and FAKE_ACP_SET_CONFIG_MODEL_DELAY_MS (per-probe latency), documented in the fixture header.

A follow-up PR will make probing fully lazy (capture reasoning levels from session/set_config_option responses on model switch) so every model a user touches gets real levels without eager discovery.

  • Suite: 312/312 · typecheck · oxfmt · version guard PASS (SDK 0.4.41)

…very deadline env-overridable

Reasoning discovery probed ACP models serially under a fixed 5s budget,
so with large catalogs (e.g. omp's 374 models) the active model was
never reached and fell back to the medium-only effort list. Seed the
probe order with the model config option's currentValue, ahead of
reasoningProbePriorityModelIds, and read the deadline from
ACP_NATIVE_REASONING_DISCOVERY_TIMEOUT_MS (default 5s).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant