Repository navigation
[Docs] Add decision-model capability matrix #58
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # Decision-model capability matrix | ||
|
|
||
| Status checked against System1-Agents `90246d6b317964674412c43d0dee966a01ca515c` (2026-10-09). This page records what the code registers separately from what the repository has actually validated. A registration or contract test establishes adapter behavior, not model quality, an application success rate, or support on every device accepted by a loader. | ||
|
|
||
| The capability rows below reflect registrations and contract tests at the checked revision; validation rows are separately tied to their recorded Agents and model revisions. | ||
|
|
||
| For issue trackers and serving support, see [System1-Omni model tracker #83](https://github.com/ThinkFlowLab/system1-omni/issues/83) and its [supported models and hardware](https://github.com/ThinkFlowLab/system1-omni/blob/main/docs/supported-models.md). Omni owns model loading and accelerator execution; Agents owns the decision adapter and agent/application behavior. Omni support alone does not validate an Agents combination. | ||
|
|
||
| ## Implemented paths | ||
|
|
||
| | Model / checkpoint | Input modality | Question types implemented | Scenario / front | Serving path | Hardware validation in this repository | | ||
| |---|---|---|---|---|---| | ||
| | TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options | `choice`, `noul`; service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** | | ||
| | CLM-v0.1-8B (`--model clm`; default serving label `clm-latest`) | Text | `choice`, `noul` | Tool decisions, browser, rails, `decide`/`probe`, MCP | HTTP `/v1/systemone` to a compatible CLM service; real encoder is external | Adapter fixtures include stub responses. A historical ticket-router report records real Qwen3-8B through upstream `clm-serve` on an RTX 4090, but its Agents source SHA is not pinned in the report and the serving engine is not Omni native CLM. It reports 18/90 correct (20%) for the normal tool-loop framing and a 4/12 fit-probe verdict of “not a decision-model task”; the 90 decisions repeat the same 30-ticket set and are not 90 independent tickets. Treat this as poor, provenance-incomplete historical evidence, not current or broad validation. See [CLM report](../evals/ticket_router/CLM.md). | | ||
| | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. | | ||
| | LAYA (`--model laya-served`; served checkpoint/revision is reported by the endpoint) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`, MCP | Agents HTTP client can connect directly to a Laya worker or through System1-Omni `omni-jev` at `/v1/systemone` | Historical direct-worker and Omni-front MPS ticket-router measurements are recorded below (90 decisions/configuration, 63 correct each); historical CPU served request replay is separate and is not task-success evidence. Other hardware/checkpoints unverified. | | ||
| | OmniJev (`--model omnijev`; default `tinnel123/OmniJev-0.8B@v1.1` on `Qwen/Qwen3.5-0.8B`; configurable 2B/4B) | Screenshot plus text state/context | `choice`, `noul` | Browser front via `s1a run` and MCP `run_agent`; direct `build_model()` factory path. Not registered in tool/rails, CLI `decide`/`probe`, or MCP `decide` | In-process OmniJev clone via `MSO1.system_one`; no System1-Omni served vision path | Historical A100 80 GB 4B runs, Windows 11 CPU 0.8B runs, and A40 46 GB 4B runs are detailed below. They do not establish broad task success; current-main reproduction unverified. | | ||
| | Cua-S1 Nano (`cua-ai/cua-s1-nano-0.1/text`, `--model cua`, default variant) | Text | `choice` only | Tool/browser paths where registered; no screenshot input | In-process `cua_s1.nano.NanoScorer` | No real Nano inference artifact found in this checkout. **CPU/GPU/MPS unverified here.** | | ||
|
There was a problem hiding this comment.
For users choosing an invocation front from this matrix, this row suggests Cua Nano is limited to tool and browser paths, but the checked revision also includes Useful? React with 👍 / 👎. |
||
| | Cua-S1 4B 0.2 text adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/text`, `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=text`) | Text | `choice` only | Tool decisions and text-only client paths; desktop option grounding is implemented | In-process Transformers/PEFT scorer | Contract tests use a fake scorer; real checkpoint/device validation **not present in this checkout**. | | ||
|
There was a problem hiding this comment.
With Useful? React with 👍 / 👎. |
||
| | Cua-S1 4B 0.2 multimodal adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/multimodal`, variant `4b`) | One current screenshot plus text task/context | `choice` only; maximum 26 options in text and multimodal modes; multimodal requires exactly one screenshot | Screenshot-dependent desktop decision adapter; this does not itself complete the Omni capture/execute bridge | In-process Transformers/PEFT scorer with screenshot passed as a temporary file | Contract tests use a fake scorer; separate historical external evidence reports a bounded MPS fixture run (see below). Current-main reproduction and CUDA validation remain unverified. | | ||
|
There was a problem hiding this comment.
When Useful? React with 👍 / 👎. |
||
|
|
||
| `random` is a seeded uniform-random choice baseline; `rule` is a hand-written one-hot choice policy. Neither is a model checkpoint. Both implement choice-only selection. `llm` refers to the separate chat model, not a decision-model backend. The `score` answer shape in Omni is not evidence that an Agents client can issue a score question. | ||
|
|
||
| ## Real validation evidence | ||
|
|
||
| These rows cite exact artifacts found in this checkout or clearly identify external historical evidence. Model/task outcome and engine/device performance are separate claims. Older or forked Agents revisions are not fresh validation of the checked `main` revision. Incomplete provenance is called out explicitly. The OmniJev implementation and contract tests are in the checked main tree (`s1a/decision_models/factory.py`, `s1a/browser/browse.py`, `tests/test_decision_models_omnijev.py`); the following run rows name the historical Agents commits that actually produced them. | ||
|
|
||
| | Agents revision and artifact | Model / checkpoint and serving path | Input / question / scenario | Hardware actually identified | What the artifact establishes and does not establish | | ||
| |---|---|---|---|---| | ||
| | Agents historical `served-laya` branch (report does not give a complete Agents SHA), [`evals/ticket_router/SERVED_LAYA.md`](../evals/ticket_router/SERVED_LAYA.md) and [`served_laya_records.json`](../evals/ticket_router/served_laya_records.json) | LAYA English `convaiinnovations/laya@55cf4c4`; in-process C-in, direct worker C-direct, and Omni `omni-jev` front C-front | Ticket router, 30-ticket set shuffled across three seeds; 90 decisions/configuration | Apple M1 Pro, 16 GB, macOS 26.1; MPS for served C-direct/C-front and in-process MPS for C-in | Records 63/90 correct for each configuration. Direct worker used compiled fp16 MPS; C-front routed through Omni; all (seed,ticket) pairs agreed across configs. Historical report, incomplete Agents source SHA; one application and checkpoint only, not broad quality evidence. | | ||
| | Replay shaping at Agents `9f87fed`; separate live browser runs at `70227de` and `69ad3e5`; [`laya-browser-shaping/README.md`](results/laya-browser-shaping/README.md) and [`served-system1-omni/README.md`](results/laya-browser-shaping/served-system1-omni/README.md) | LAYA text worker through Omni `omni-jev`; Omni `ae86032cba2466f45f42c2ebdfadbcfa8c30eb9f`; worker checkpoint routing was not confirmed for every replay tick | Standalone `urllib` replay of 12 recorded Google Flights browser ticks, 45 questions; pre-shaped with Agents `laya_state()`/`laya_browser_question()`, sending requested 1536/1024 and default windows; separate live browser runs | Replay: Linux x86_64, 8 CPU cores, no GPU. Live run: Windows 11 CPU | The replay uses pre-shaped bodies and requested window fields, but it is not execution through Agents `ServedLayaModel`. The older worker ignored the requested window values and matched its default-window answers; per-tick checkpoint routing is uncertain. Separate live in-process browser attempts ended in early `DONE` and did not complete the task. These artifacts show input-shape and serving-request behavior, not a successful browser application. | | ||
| | Agents `e64eb58e7fd94806edfa6206aa08e66f01bdae78`, external [PR review evidence release](https://github.com/QianCyrus/system1-agents/releases/tag/pr-review-evidence-20261007) | Served LAYA English `convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`; Omni `99865743d27316fbe81362dc0f0e6de6fda86284`, `frontend.laya_mps` direct HTTP worker | Desktop document fixture; four decisions in one task: Save, Save, type, Save; independent file check | macOS 26.2 arm64; MPS; physical chip not reported | One historical fixture task succeeded on the final Save, after two ineffective Save choices; fresh file and exact content were independently checked. External archive also pins Laya and both repository revisions. One task only; not broad desktop capability. | | ||
| | Agents `222e656` baseline and historical PR #37 head `c197fe6`, [`laya-upgrade/README.md`](results/laya-upgrade/README.md); raw `records/` were attached to PR #37 as `pr37-evidence.zip` | LAYA English and multilingual (`LAYA_SUBFOLDER=multilingual`), in-process; Laya 0.3.5 vs 0.3.20 | 30 ticket-router tickets, choice/noul and six-question requests; adapter equivalence/package check | Apple M1 Pro, 16 GB; macOS 26.1; CPU and MPS | README reports identical outputs across three runs per compared CPU/MPS cell for those revisions and checkpoint revision `7b928d8`. Typed/custom checkpoints, CUDA and other Apple chips are out of scope. Raw records are external evidence and not in this checkout. | | ||
| | Forked Agents historical commit `8955622` documenting source/test revision `cf49ee6635abc202a2beb5404ac02e54ebd0abbc`, [visual evidence report](https://github.com/QianCyrus/system1-agents/blob/8955622/evals/desktop/visual-evidence.md) | Cua-S1 4B 0.2 multimodal adapter `cua-ai/cua-s1-4b-0.2@16818868b0cc7813808aae4e87b417657046ab79`; base `Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` | Screenshot-based desktop Save/Cancel fixture; four model trials, plus fixed correct/wrong controls and independent file checks | macOS 26.2, arm64, MPS/bfloat16; physical chip not specified | Historical external/fork evidence reports four of four fixture tasks saved successfully with independent file checks, and a wrong-point control failed as intended. Checkpoint/source revisions are pinned. This is a bounded synthetic fixture result, not broad desktop quality or validation of current `main`. | | ||
| | Agents `1d01e69a5f8137d289eb53a1d2432f0fd38bfe53`; [`docs/results/omnijev-flights/README.md`](https://github.com/ThinkFlowLab/system1-agents/blob/main/docs/results/omnijev-flights/README.md), `2026-09-29__*` run records | OmniJev 4B release v1.1, rank-32 LoRA on Qwen3.5-4B; OmniJev clone `14dbec4`; in-process `MSO1.system_one`, full prompt. The run artifact does not record exact Hugging Face adapter/base snapshot SHAs. | Google Flights, one-way Zurich to London, Nov. 1, 2026; 10 counted runs at this source revision. | NVIDIA A100 80 GB PCIe Linux | 1/10 completed; one other reached the results page with Google reporting an error, and eight did not. Historical task evidence only; no current-main or general quality claim. See exact run folders and denominators in the linked report. | | ||
| | Agents `e2bf4c5971cba128195846a1e2aba5663c82c2f0`; same [run report](https://github.com/ThinkFlowLab/system1-agents/blob/main/docs/results/omnijev-flights/README.md), `2026-09-30__*` records | OmniJev 4B release v1.1, rank-32 LoRA on Qwen3.5-4B; clone `14dbec4`; in-process `MSO1.system_one`, full prompt. Exact Hugging Face adapter/base snapshot SHAs are not recorded. | Same Google Flights task; 10 counted runs after the history fix. | NVIDIA A100 80 GB PCIe Linux | 0/10 completed; 3 more reached the results page but did not emit DONE before budget, and 7 did not reach results. Historical task evidence only; not current-main validation. | | ||
| | Agents `6d9c015b36d02a16828b7eed661e8207a875a876`; [CPU run artifacts](https://github.com/ThinkFlowLab/system1-agents/blob/main/docs/results/omnijev-flights/README.md#recorded-runs-on-cpu-2026-10-06-at-6d9c015) | OmniJev-0.8B v1.1 on Qwen3.5-0.8B; clone `14dbec4`, full prompt. Exact Hugging Face adapter/base snapshot SHAs are not recorded. | Two Google Flights attempts on the consent/form pages; first made two decisions (136 s, 128 s) before the task deadline; second ended BLOCKED after blank page/WAIT. | Windows 11 CPU, no GPU | Two bounded failed attempts; no task completion and no CPU performance generalization. | | ||
| | Agents `0829ce0dd71350db78a29105e61d162c3f4280d4`, [`docs/results/omnijev-flights/README.md`](https://github.com/ThinkFlowLab/system1-agents/blob/main/docs/results/omnijev-flights/README.md), `2026-10-07-a40/run1`–`run3`, `versions.txt`, `answer.json`, `decision_ticks.json`, `calls.jsonl` | OmniJev 4B v1.1 on Qwen3.5-4B; OmniJev `14dbec4f71e194852c8d7b88ab36ef639493f400`; in-process `MSO1.system_one`, full prompt. The artifact does not record exact Hugging Face adapter/base snapshot SHAs. | Google Flights, Zurich to London, Nov. 8, 2026; 3 runs; 25 browser decisions each; all ended at form budget and none reached results | NVIDIA A40 46 GB, Linux 6.8, driver 615.71.09 | 0/3 completed; per-run median decision 1.8/2.0/1.8 s. This records decision latency and failed task attempts only; no successful task or general performance claim. | ||
| | Agents `22e685b` and Omni `4a79980`, [`2026-10-08-a40-native/README.md`](results/decision-timeout/2026-10-08-a40-native/README.md), `versions.txt`, and per-run `answer.json`, `decision_ticks.json`, `calls.jsonl` | Open-Jev-9B `ZefanCai/Open-Jev-9B@47e9668` on `Qwen/Qwen3.5-9B@c202236`; generic Agents Jev HTTP adapter → Omni native Open-Jev worker | Text candidate scoring from Google Flights browser task; 5 s default, 30 s, and 1 s timeout runs | NVIDIA A40 46 GB, driver 615.71.09, Ubuntu 24.04 | 5 s default timed out before a decision; 30 s answered two decisions then model returned `BLOCKED`; 1 s timed out cleanly. This is request-path/timeout evidence, **not** successful flight search. Open-Jev-27B-v1.1 native load did not fit this A40; see the same artifact folder. The Jev client is generic and does not itself guarantee either Open-Jev checkpoint. | | ||
|
|
||
| OmniJev has current-main registration and adapter tests; its real run artifacts are historical and do not validate current main. The A100, CPU, and A40 artifacts identify OmniJev release labels and repository clone commits but omit exact Hugging Face adapter/base snapshot hashes; those checkpoint file revisions remain unpinned. The matrix intentionally leaves other combinations unverified when evidence is only an adapter test, a server request replay, an upstream Omni result, or a device-capable loader without an exact corresponding run artifact. The historical external desktop reports are bounded fixture evidence and do not establish broad capability or current-main behavior. A hardware label in Omni documentation is not automatically an Agents validation row. | ||
|
|
||
| ## Updating the matrix | ||
|
|
||
| For an implementation update, cite the current registration and contract tests, then list modality, supported question types, front/scenario, and serving path. For a validation update, add the exact Agents and Omni revisions, model and checkpoint revision, input/question scope, scenario and result, worker/frontend, device, and linked run artifacts. Label historical, external, replayed, stubbed, failed, and incomplete runs as such. Do not infer the served checkpoint from an adapter name, infer device validation from code support, or turn one run into a rate or performance claim. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The matrix omits MCP from the in-process LAYA row even though
s1a/mcp_server.pyexplicitly acceptsmodel="laya"in thedecidetool. Readers using this matrix to choose an invocation path will incorrectly conclude that only served LAYA is available through MCP; add MCP to this row's registered fronts.Useful? React with 👍 / 👎.