Skip to content

feat: JEV-9B vision on Core ML + Mario demo (JEV-9B and a 9M pixel student with lookahead) - #39

Merged
Alex-Wengg merged 20 commits into
mainfrom
feat/jev-9b-mario
Oct 12, 2026
Merged

feat: JEV-9B vision on Core ML + Mario demo (JEV-9B and a 9M pixel student with lookahead)#39
Alex-Wengg merged 20 commits into
mainfrom
feat/jev-9b-mario

Conversation

@Alex-Wengg

@Alex-Wengg Alex-Wengg commented Oct 12, 2026

Copy link
Copy Markdown
Member

Adds JEV-9B (Qwen3.5-9B + vision LoRA, HF FluidInference/jev-9b-coreml) as a Core ML runtime (JevManager, JevModelStore, JevCheck), and JevMarioDemo, where it plays Super Mario Bros.

  • JEV-9B brain: three yes/no questions about close-ups ahead of Mario (enemy, pipe or step, gap); a yes means jump. It clears 1-1.
  • Student brain: a 9.0M-parameter MobileNetV3 trained on a beam-search solver's routes for all 32 levels plus DAgger. All 185 ops run on the ANE, about 0.5 ms per decision.
  • Lookahead (on by default): before each move the app saves the emulator state and simulates candidate moves, each followed by the student's next 6. It plays the first one that survives and gains ground.
32 levels × 3 starts Flags Levels cleared at least once Cleared every time
Student alone 53/96 23 12
Student + lookahead 78/96 30 21
  • Play all 23 plays the 23 levels that clear from the start in the app, verified one by one.
  • Uncapped max speed: 1.9–5.5× real time with lookahead.
  • Monitor and log: demo.sh opens a Ghostty window with the ANE busy bar and the decision log.

Notes for reviewers:

  • The emulator bridge uses gym-super-mario-bros through uv. It bundles the NES ROM, so it is for local use only; no ROM, frames or video are in this PR.
  • The student .mlpackage is in a private HF repo for now, so run the student brain with MARIO_STUDENT_MODEL=<path>. Training code is in model-lab models/mario-tiny.
  • With lookahead, the result is the student plus emulator simulation, not the network alone.

🤖 Generated with Claude Code

Alex-Wengg and others added 20 commits October 12, 2026 16:03
@Alex-Wengg @claude
feat: JEV-9B with vision on Core ML (JevManager) + Mario demo
50e7007 
JevManager runs autotrust/JEV-9B's System 1 (Qwen3.5-9B + merged
decision LoRA) from FluidInference/jev-9b-coreml @a718625d: fp32 vision
tower per patch bucket, 8 w8 decoder parts chained on the GPU, and the
adapted lm_head rows + per-slot bias + per-kind temperature on the host.
Noul / score / choice questions over text and images; the vision tower
runs once per image and each question is one decoder pass. Reuses the
clef-vision image preprocessor, vision host math and Qwen BPE tokenizer.

JevCheck: parity against model-lab fixtures (identical token ids and
M-RoPE positions on 16 text + image records, max probability drift
4.7e-4) and latency (0.95 s for one image + 3 questions, M5 Pro).

JevMarioDemo: the model plays Super Mario Bros. 1-1 through three yes/no
questions on close-ups ahead of Mario (jump if any fires); reaches the
flag in 70 decisions, same as the Python reference. The emulator runs in
nes_bridge.py via uv + gym-super-mario-bros, which the user installs
(it bundles the ROM); nothing ROM-related is in the repo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: Tiny (0.46M-param pixel student) brain alongside JEV-9B
aa8e9eb 
Picker between JEV-9B (three questions) and the mario-tiny Core ML student
(TINY_MARIO_MODEL), which picks run / jump / hop / wait from raw frames in
~0.45 ms; reaches the flag on 1-1 in 32 decisions at 60 fps. Line-buffered
log; final outcome printed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: pixel-student brain (~9M params), lazy JEV download, le…
8013d5b 
…vel choice

The second brain loads any mario-tiny Core ML student (MARIO_STUDENT_MODEL)
and labels it with the parameter count from the package metadata; the
first 4 outputs are the move logits (multi-head students append survival
logits). JEV-9B is downloaded only when picked. MARIO_LEVEL selects the
level in nes_bridge.py. The 9M MobileNetV3 + RAM-aux student reaches the
flag on 1-1 (32 decisions, 0.65 ms median) and 6-1 (28 decisions).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: show the student's Core ML op placement (MLComputePlan)
f2ae591 
Activity meters (macmon ane_power) read 0 W for the Neural Engine on
macOS 27, so the panel and log state where Core ML runs the model's ops,
e.g. 'Neural Engine · 185/185 ops'.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
demo.sh / bars.py: no clipped load bars
f482e7b 
The tmux session is created detached at 160 columns; a narrower Ghostty
window attached to it showed the right edge cut off. window-size latest
makes it follow the client, Ghostty opens at 150x44, and bars.py sizes
the bar from the right-hand text width.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
demo.sh: MONITOR=asitop runs sudo asitop in the top pane (ANE power)
a91d1a2 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
bars.py --ane: Neural Engine power via sudo powermetrics; demo.sh MON…
cad09b6 
…ITOR=ane

asitop 0.0.24 crashes on M5 Pro (no E0-Cluster in powermetrics) and
macmon's ane_power reads 0 W on macOS 27.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: T layout, ANE busy bar, max-speed mode
9dd12cb 
- demo.sh centres the game window across the top and places the demo
  terminal (bars | decision log, side by side) full width underneath;
  fixes a set -e abort when MONITOR is unset.
- bars.py --log: ANE row = model milliseconds per second from the
  decision log (all 185 ops run on the Neural Engine per MLComputePlan;
  no hardware counter reads the ANE on macOS 27), plus ms and rate.
- Max speed toggle (MARIO_TURBO=1): no 60 fps pacing; live game fps and
  x real time readout. 1-1 start to flag ~7 s incl. load (~40 s at 60 fps).
- Probability bars draw instantly (ProgressView animation lagged).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: level playlist (MARIO_PLAYLIST=level[:idle],...)
9753dc4 
The bridge switches level and idle start on reset; the app plays the list
in order, auto-advances, and shows a flag / death chip per level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: level picker grid and resizable window
28b3c0f 
32-level toggle grid (reliable levels dotted, finished ones green/red),
Play selected restarts at any time (the previous game is cancelled and
awaited; the frame loop now checks cancellation in max-speed mode too);
the game view scales with the window (contentMinSize).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: one level menu + Play instead of the level grid
2eb2872 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: nothing plays until Play; one level per press
94fb19d 
Launch only loads the model and the emulator; Play runs the level in the
menu once (no playlist auto-advance); Max speed starts off.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: stable-retro bridge, solver-style student, batched max …
9411800 
…speed (24-32x real time)

- retro_bridge.py: stable-retro (any level, the gym ROM), 'run' plays a
  whole move per request (holding through game-controlled transitions)
  and returns the final frame + the one 4 frames earlier.
- The student's moves come from the package metadata: 4-move models keep
  the land-then-decide loop; 11-move solver-trained models decide every
  8-frame macro. Max speed = one batched call and one redraw per macro.
- Measured: 1-1 at 1,936 fps (32x real time, 0.5 s), 6-1 at 1,438 fps
  (24x), both to the flag; normal speed ~52 fps.
- MARIO_TURBO / MARIO_AUTOPLAY env for benchmarks (off by default).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: fast mode capped at 10x real time (MARIO_SPEED_CAP)
16038bb 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: fast mode cap 5x by default
72714fa 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: student lookahead on emulator save states (78/96, 30/32…
cb91e11 
… levels)

Before each move the app saves the emulator state and simulates candidates
in the student's order (move + the student's next 6 moves, Core ML on the
ANE); it plays the first that survives and gains ground (x, area or pipe),
else the longest-lasting. A stalled continuation also tries the move
repeated (swim strokes). Mirrors model-lab mario-tiny student_retro.py
--lookahead 6 and matches its outcomes on 1-1, 4-2, 5-2, 7-1.

Bridge: save / load (no image), area + transition in the reply, frames
padded only for the two sent back; Swift reads the pipe in chunks instead
of one read per byte (5-2 uncapped 1.3x -> 1.9x real time). Toggle in the
UI, MARIO_LOOKAHEAD=0 starts it off.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: Play all 23 (levels the student + lookahead clears from…
3911030 
… the start), max speed uncapped

The playlist is the 23 levels that reach the flag in the app at idle 0
(deterministic emulator; verified one by one with r10). MARIO_PLAYLIST
overrides it. Results show as a wrapping grid with a flag count. Max speed
no longer caps at 5x (MARIO_SPEED_CAP sets a cap).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: compact student panel (2-column move bars), demo.sh lay…
6a8f75d 
…out fixes

Move probabilities in a 2-column grid, tighter spacing and a one-line
move / latency row: the window drops from 688 to 606 pt tall, leaving the
monitor more room. demo.sh restarts the tmux session every launch (a stale
session skipped opening Ghostty), waits for the new Ghostty process, and
re-applies the T layout until it holds (Ghostty re-places its window).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: swift-format line lengths
db24ce3 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg @claude
JevMarioDemo: @preconcurrency import CoreML (Swift 6.1 CI sendability)
6a8e6ee 
macos-15 CI rejects MLModel / MLComputePlan crossing into the @mainactor
policy; same import as the other FluidUse Core ML managers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Alex-Wengg
Alex-Wengg merged commit 49c423e into main Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Alex-Wengg