docs(local-llm): record the dense-27B retrial and the control-run rule - #232
docs(local-llm): record the dense-27B retrial and the control-run rule#232JacobPEvans-personal wants to merge 2 commits into
Conversation
Adds Qwen3.8-27B-4bit to the Mac Studio model registry: trialled as the serving default and reverted the same day on throughput, reproducing the same dense-vs-MoE penalty the retired Qwen3.6-27B showed a generation earlier. Adds the control-run rule to the mlx-benchmarks methodology page. A stated regime can match on paper and still differ in practice, so the incumbent must be re-measured alongside the candidate with the same harness.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This PR adds a new model entry (Qwen3.8-27B-4bit) to the Mac Studio model registry table, documenting that it was trialled as the serving default and reverted the same day on throughput grounds, with comparison figures against the 35B MoE. It also adds a new section to the MLX benchmarks doc emphasizing that a matching stated regime alone is insufficient, recommending re-measuring the incumbent alongside the candidate in the same session and matching probe concurrency to the endpoint. The changes are documentation-only, spanning the model comparison table and the benchmarking methodology/warnings text.
No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 15 functions depend on the 15 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 15 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 15 function(s) in the blast radius were not formally verified this run
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the local LLM documentation for the Mac Studio homelab by adding details on the trial and reversion of the Qwen3.8-27B-4bit model due to throughput constraints. It also updates the MLX benchmarks documentation to emphasize the importance of concurrent control runs to avoid harness effects. The review feedback suggests minor formatting improvements, specifically removing spaces in parameter size abbreviations (e.g., changing '27 B' to '27B') and percentage values (e.g., changing '20 %' to '20%') for consistency.
| | **`gpt-oss-120b-MXFP4-Q8`** | 63.3 GB (On-demand) | On-demand Reasoning | **Best for constrained prose and SPL query authoring.** Loaded on demand and idle-unloaded after 900s rather than held resident. Highly capable on complex reasoning, but weak at code review (confident false positives). **Held back from agent work by a serving-layer gap, not by the model:** it emits semantically correct calls in its own channel markup, the server ships no parser for that format, so `tool_calls` returns empty and the markup leaks into `content` (verified 2026-07-26/27). The missing parser landed as a declarative patch with a per-model toggle, and remains correct and worth having — the leakage is real and measured. The throughput argument once attached to it does not survive: see the re-measured figures below. See [Backends & tool calling](/local-llm/backends). Requires high `max_tokens` for JSON. `reasoning_effort=low` by default. | | ||
| | **`Qwen3.6-35B-A3B-4bit`** | 20.4 GB (Swap tier) | Swap-tier structured fallback | **The uniform-4bit structured-output baseline.** Passes strict JSON port-allocation tests but was superseded as the agent default by the resident agentic brain above (uniform 4-bit degrades multi-turn tool calling earlier). Runs with thinking off by default. | | ||
| | **`Qwen3.6-27B-4bit`** | 16.1 GB | Retired (2026-07-07) | **Retired after evals.** Decoded at 23–27 tok/s (4× slower than the 35B MoE), produced low-effort code reviews, and filled no unique capability niche. Removed from the swap tier. | | ||
| | **`Qwen3.8-27B-4bit`** | 16.1 GB (Swap tier) | Trialled as default, reverted (2026-08-14) | **A newer generation does not beat the architecture.** Promoted to serving default on release, then reverted the same day on throughput: 37.1 cumulative / 27.7 decode tok/s against the 35B MoE's **138.8 / 106.5**, both measured minutes apart on the same host with the same harness. That is the same ~4× dense-vs-MoE penalty the retired `Qwen3.6-27B` showed one generation earlier — 27 B dense parameters activate per token against roughly 3 B for an A3B MoE. Kept cached in the swap tier for a quality-led re-trial; the throughput case is settled. | |
There was a problem hiding this comment.
For consistency with the rest of the document (e.g., Qwen3.8-27B-4bit, 35B MoE, 30B-class), please use 27B and 3B instead of 27 B and 3 B (without the space).
| cumulative against the incumbent's recorded **115.2** — a regime that looked | ||
| identical on every published field. Re-measuring the incumbent **with the same | ||
| harness, in the same session** put it at **138.8**, not 115.2: the harness ran | ||
| hot, and the naive comparison had understated the real gap by roughly 20 %. It |
There was a problem hiding this comment.
In standard English typography, percentage values are typically written without a space before the percent sign (e.g., 20% instead of 20 %). Let's remove the space for consistency and readability.
hot, and the naive comparison had understated the real gap by roughly 20%. It
…ree pages An audit found this defect described in the present tense on three public pages months after the fix landed. A disabled model's physical id is no longer aliased onto the resident; it returns HTTP 404. overview.mdx and choosing-a-model.mdx are corrected in place. verifying-the-instrument.mdx keeps Mechanism A in full, with a note that it is fixed, because the shape generalises to any proxy that collapses a catalogue and it fails in the worst possible way - a 200 with the requested name echoed back. Stale versions of this claim are not harmless: the same wrong assumption, in the private runbooks, contributed to an agent-fabric outage on 2026-08-14.
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This PR revises the local-LLM and MLX-benchmarks docs to reframe the "alias grafting on collapse" defect as fixed (disabled model ids now return HTTP 404) while retaining its description as a failure pattern worth recognizing. It adds a new Qwen3.8-27B-4bit entry to the Mac Studio model registry documenting a trialled-then-reverted default with throughput figures, and expands the benchmarks guide with guidance on re-measuring the incumbent alongside candidates and matching probe concurrency. Changes are confined to documentation/prose across four .mdx files.
No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 54 functions depend on the 54 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 54 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 54 function(s) in the blast radius were not formally verified this run
Two edits, both from measurements taken 2026-08-14.
local-llm/mac-studio.mdx— addsQwen3.8-27B-4bitto the model registry: promoted to serving default on release, reverted the same day at 37.1 cumulative / 27.7 decode tok/s against the 35B MoE's 138.8 / 106.5, measured minutes apart on the same host with the same harness. Same ~4x dense-vs-MoE penalty the already-retiredQwen3.6-27Bshowed a generation earlier. Stays cached in the swap tier for a quality-led re-trial.tools/mlx-benchmarks.mdx— adds the control-run rule. The page already warns against comparing figures from different regimes; this adds the case where the regime matches on paper and the harness still differs, which understated the gap above by roughly 20%. Also records matching probe concurrency to the endpoint's, since a wider probe collects HTTP 429s and averages them into the aggregate.