Repository navigation
[Docs] Add decision-model capability matrix - #58
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9882073ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| |---|---|---|---|---|---| | ||
| | TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options | `choice`, `noul`; service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** | | ||
| | CLM-v0.1-8B (`--model clm`; default serving label `clm-latest`) | Text | `choice`, `noul` | Tool decisions, browser, rails, `decide`/`probe`, MCP | HTTP `/v1/systemone` to a compatible CLM service; real encoder is external | Adapter fixtures include stub responses. A historical ticket-router report records real Qwen3-8B through upstream `clm-serve` on an RTX 4090, but its Agents source SHA is not pinned in the report and the serving engine is not Omni native CLM. It reports 18/90 correct (20%) for the normal tool-loop framing and a 4/12 fit-probe verdict of “not a decision-model task”; the 90 decisions repeat the same 30-ticket set and are not 90 independent tickets. Treat this as poor, provenance-incomplete historical evidence, not current or broad validation. See [CLM report](../evals/ticket_router/CLM.md). | | ||
| | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. | |
There was a problem hiding this comment.
The matrix omits MCP from the in-process LAYA row even though s1a/mcp_server.py explicitly accepts model="laya" in the decide tool. Readers using this matrix to choose an invocation path will incorrectly conclude that only served LAYA is available through MCP; add MCP to this row's registered fronts.
Useful? React with 👍 / 👎.
| | CLM-v0.1-8B (`--model clm`; default serving label `clm-latest`) | Text | `choice`, `noul` | Tool decisions, browser, rails, `decide`/`probe`, MCP | HTTP `/v1/systemone` to a compatible CLM service; real encoder is external | Adapter fixtures include stub responses. A historical ticket-router report records real Qwen3-8B through upstream `clm-serve` on an RTX 4090, but its Agents source SHA is not pinned in the report and the serving engine is not Omni native CLM. It reports 18/90 correct (20%) for the normal tool-loop framing and a 4/12 fit-probe verdict of “not a decision-model task”; the 90 decisions repeat the same 30-ticket set and are not 90 independent tickets. Treat this as poor, provenance-incomplete historical evidence, not current or broad validation. See [CLM report](../evals/ticket_router/CLM.md). | | ||
| | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. | | ||
| | LAYA (`--model laya-served`; served checkpoint/revision is reported by the endpoint) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`, MCP | Agents HTTP client can connect directly to a Laya worker or through System1-Omni `omni-jev` at `/v1/systemone` | Historical direct-worker and Omni-front MPS ticket-router measurements are recorded below (90 decisions/configuration, 63 correct each); historical CPU served request replay is separate and is not task-success evidence. Other hardware/checkpoints unverified. | | ||
| | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front only; not in tool/rails/CLI/MCP registrations | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | |
There was a problem hiding this comment.
The claim that OmniJev is not registered in the CLI is false for its supported browser context: BROWSER_MODEL_NAMES includes omnijev, so commands such as s1a run flights --model omnijev are valid CLI usage. Only the standalone decide/probe parsers exclude it, so naming those commands instead of all CLI avoids contradicting the browser registration and the documented run evidence below.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c2cc43b92
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. | | ||
| | LAYA (`--model laya-served`; served checkpoint/revision is reported by the endpoint) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`, MCP | Agents HTTP client can connect directly to a Laya worker or through System1-Omni `omni-jev` at `/v1/systemone` | Historical direct-worker and Omni-front MPS ticket-router measurements are recorded below (90 decisions/configuration, 63 correct each); historical CPU served request replay is separate and is not task-success evidence. Other hardware/checkpoints unverified. | | ||
| | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front only; not in tool/rails/CLI/MCP registrations | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | | ||
| | Cua-S1 Nano (`cua-ai/cua-s1-nano-0.1/text`, `--model cua`, default variant) | Text | `choice` only | Tool/browser paths where registered; no screenshot input | In-process `cua_s1.nano.NanoScorer` | No real Nano inference artifact found in this checkout. **CPU/GPU/MPS unverified here.** | |
There was a problem hiding this comment.
For users choosing an invocation front from this matrix, this row suggests Cua Nano is limited to tool and browser paths, but the checked revision also includes cua in DECIDE_MODEL_NAMES for both s1a decide and s1a probe, and in the MCP decide tool's model enumeration. List those registrations explicitly, as the Jev and CLM rows do, so readers do not incorrectly conclude that Nano is unavailable through the standalone CLI or MCP.
Useful? React with 👍 / 👎.
| | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front only; not in tool/rails/CLI/MCP registrations | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | | ||
| | Cua-S1 Nano (`cua-ai/cua-s1-nano-0.1/text`, `--model cua`, default variant) | Text | `choice` only | Tool/browser paths where registered; no screenshot input | In-process `cua_s1.nano.NanoScorer` | No real Nano inference artifact found in this checkout. **CPU/GPU/MPS unverified here.** | | ||
| | Cua-S1 4B 0.2 text adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/text`, `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=text`) | Text | `choice` only | Tool decisions and text-only client paths; desktop option grounding is implemented | In-process Transformers/PEFT scorer | Contract tests use a fake scorer; real checkpoint/device validation **not present in this checkout**. | | ||
| | Cua-S1 4B 0.2 multimodal adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/multimodal`, variant `4b`) | One current screenshot plus text task/context | `choice` only; maximum 26 options in text and multimodal modes; multimodal requires exactly one screenshot | Screenshot-dependent desktop decision adapter; this does not itself complete the Omni capture/execute bridge | In-process Transformers/PEFT scorer with screenshot passed as a temporary file | Contract tests use a fake scorer; separate historical external evidence reports a bounded MPS fixture run (see below). Current-main reproduction and CUDA validation remain unverified. | |
There was a problem hiding this comment.
When CUA_S1_VARIANT=4b and CUA_S1_MODALITY=multimodal, the browser front is wired for this adapter: BROWSER_MODEL_NAMES includes cua, CuaFourBModel sets supports_images, and BrowserDecisionModel captures and attaches a screenshot for any image-capable backend. Describing the scenario only as a desktop adapter hides an implemented browser path from readers of the capability matrix; include the browser front while retaining the 26-option limitation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 75c0c85711
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | LAYA (`--model laya-served`; served checkpoint/revision is reported by the endpoint) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`, MCP | Agents HTTP client can connect directly to a Laya worker or through System1-Omni `omni-jev` at `/v1/systemone` | Historical direct-worker and Omni-front MPS ticket-router measurements are recorded below (90 decisions/configuration, 63 correct each); historical CPU served request replay is separate and is not task-success evidence. Other hardware/checkpoints unverified. | | ||
| | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front via `s1a run` and MCP `run_agent`; direct `build_model()` factory path. Not registered in tool/rails, CLI `decide`/`probe`, or MCP `decide` | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | | ||
| | Cua-S1 Nano (`cua-ai/cua-s1-nano-0.1/text`, `--model cua`, default variant) | Text | `choice` only | Tool/browser paths where registered; no screenshot input | In-process `cua_s1.nano.NanoScorer` | No real Nano inference artifact found in this checkout. **CPU/GPU/MPS unverified here.** | | ||
| | Cua-S1 4B 0.2 text adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/text`, `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=text`) | Text | `choice` only | Tool decisions and text-only client paths; desktop option grounding is implemented | In-process Transformers/PEFT scorer | Contract tests use a fake scorer; real checkpoint/device validation **not present in this checkout**. | |
There was a problem hiding this comment.
With CUA_S1_VARIANT=4b and CUA_S1_MODALITY=text, the factory still builds this adapter under the registered cua name, so it is available through the browser front, CLI decide/probe, and MCP decide in addition to tool agents. Describing these only as unspecified “text-only client paths” makes the front matrix imply a narrower implementation than the code provides; enumerate those fronts explicitly, as for the other variants.
Useful? React with 👍 / 👎.
2 participants
Why
Issue #54 asks for an Agents capability matrix that separates implemented model paths from real validation and explicitly marks untested combinations.
How
Add
docs/capabilities.md, grounded in current registrations and contract tests and exact historical run artifacts/revisions. Link System1-Omni tracker #83 and its supported-models page. Distinguish current implementation from bounded historical validation and mark unverified combinations.What
docs/decision-models.md.Verification
mainis90246d6b317964674412c43d0dee966a01ca515c; current PR head is75c0c85711e573008a5ce17702f3fe9190dc9099, with latest main in its ancestry. GitHub reports exactly two changed files:docs/capabilities.mdanddocs/decision-models.md.git diff --checkpassed; local relative Markdown links resolve. The external System1-Omni tracker and supported-models pages were fetched and checked.37946905355passed all four test jobs; browser job was skipped. The previous corrected head0c2cc43b9271904cd3ab19609744ffe1a9ede566also passed run37945857495.Demo / evidence
N/A: documentation-only change, no runtime behavior or new measured quality/performance claim. The matrix labels historical runs and separates adapter registration from application completion.
Related: #54.
Note
Low Risk
Documentation-only; no runtime or API changes beyond aligning OmniJev error documentation with existing code.
Overview
Adds
docs/capabilities.md, a decision-model capability matrix pinned to a specific Agents revision that splits registered/contract-tested paths from real run validation, marks unverified hardware and checkpoint combinations, links System1-Omni tracker #83, and catalogs historical artifacts (LAYA, OmniJev, Cua-S1, Open-Jev, CLM) with explicit limits on what each row proves.docs/decision-models.mdnow points readers at the matrix and documents OmniJev missing-screenshot failures asMODEL_CALL_FAILED(not config error), consistent with current adapter behavior.Reviewed by Cursor Bugbot for commit 75c0c85. Bugbot is set up for automated code reviews on this repo. Configure here.