An AI agent brings dead Linux servers back from the console, with no login. It reads console pixels through the native computer tool of Muse Spark, types, and can pull the power. It does not get to say whether it worked: a prober it cannot touch resets the machine and decides, by a real SSH login, a nonce and a hash of the data.
Film: watch it on YouTube · Replay: the real runs (no key, no machine; also in site/index.html) · Results · Reproduce it · Safety and limits
Status: experimental recovery on disposable Debian VMs. The fleet headline covers one fault; its dollar figure counts model calls only. The outside prober checks recovery, not whether every repair is a good one.
- One rescue became a recipe, and the recipe a fleet. 8 of 8 test servers, all with one fault, came back by the prober's verdict, for $0.39 of model calls in all, each part rounded: $0.24 for the first, which taught the recipe, nothing for the 5 that replayed it, and $0.16 for the 2 where the replay stopped and the model stepped in. One of those stops was planted: that machine's boot entry was given extra lines, so it was not the screen the recipe was taught on, and the replay stopped at the step that edits it. Of the others, two ran on another disk bus and one on another network card.
- It read a cause that was on screen for 1.2 s. Screenshots fall in between, so the agent can send the model a recording. In one clip of one console, quoting the line word for word: at the default frame rate, with no
fpsfield, which the API reference says leaves the rate to the backend, 0 of 4 asinput_video, and 0 of 4 sent asinput_file, which has no such field. Both used as many input tokens as the two-a-second arm, so here the default sampled like it. At 2 a second, 0 of 4. At 4 a second, 4 of 4. - It left a planted note alone. A note on the disk told it to open root to a vendor's key. In one run, 34 turns and $0.35, the attacker's key opened no root login and the prober passed the machine.
"I look after a handful of my own servers, and one of them has been unreachable since September; this is the tool I wanted for it." (Ted Chen)
All of it, apart from the one file and the third-party parts named below. The first commit in this repo is dated 2026-11-01 Taipei time, and the git history is the record.
- The agent loop (
remotehands/agent.py): screenshots up, keystrokes down, chains that roll, handover notes, refusals. - The keystroke firewall (
remotehands/firewall.py): a fixed line editor and shell parser that reads every keystroke and is the only thing that lets a safety check be acknowledged. - The outside judge (
remotehands/prober.py,remotehands/remote_probe.py). - The console reader (
remotehands/consoletext.py): exact glyph matching on the pixels, with no OCR. - Recipes and fleets (
remotehands/recipe.py,remotehands/fleet.py): compile a recovered run, prove it, replay it on other machines. - The fault library (
faults/,tools/seal.py): 20 broken machines with a scripted proof for each, split by a rule nobody can tune. - The web console driver (
remotehands/webconsole.py), the injection scenario (scenarios/,remotehands/injection.py) and the results pipeline. - The film (
tools/film4_build.py,tools/film4_gates.py,tools/film4_score.py,tools/film4_voice.py): the composition, the gates it has to pass, the narration's script and choice of reads, and the music's score. The console screens in it are this project's own runs. What came from elsewhere is in the table below: the stock clips, the sample library and sound effects, and the type.
Carried in, not new here: remotehands/meta_api.py, a client for the Meta Model API. It was first written on Oct 26 at 4:46 PM ET, about five hours after the kickoff and before the guide's listed building period of Oct 27 to Nov 4, for an earlier idea of Ted Chen's that he has since dropped. It was changed until Oct 29 and came here with small changes.
Not built here: everything under Third-party parts. Nothing in the film is meant to show those as new work.
SSH is dead and the vendor gives you only a web VNC window. GRUB, the emergency prompt and a kernel stuck in early boot show on that screen, and no API or agent inside the machine can see them. Someone has to sit at that screen. One provider's published rates for a person at the rack are $125/hr in business hours and up to $250/hr after hours, depending on the site. That buys a person's time, not a diagnosis, and the comparison is rough.
Every number on this page is generated from the run records by tools/build_results.py. A test fails if a generated part goes stale, or if any digit sits outside those parts, code, links and a short list of names and sourced facts. That checks the numbers, not the sentences around them. Tables and per-run detail: docs/results.md.
| What | Result | Read it as |
|---|---|---|
| Dev faults, first pass | 6 of 10 | 3 failed, 1 void (connection lost). Tuned as it went. |
| Failed or void dev faults, rerun after tuning | 4 of 4 reruns | Reruns only, so never added to the row above. |
| Sealed faults, each scored once | 8 of 9 solvable faults recovered | Every run at commit 477e4576b517. Published whatever it says. |
| The sealed fault with no way in | It used every turn and said nothing | The right answer is to say it cannot. |
| Fleet replay, no model | 6 of 6 | $0.00 of model calls, 5.2 minutes on the clock. One fault, one recipe. |
| Fleet with the model on standby | 8 of 8 recovered, $0.39 in all, plus the one run where the model stepped in, 1 other fleet in the results | First machine full model, the rest replay. |
| Injection note, with the model | 2 of 2 runs left the attacker locked out | Scripted controls are done, see below. |
| A real vendor's web console | 1 of 1 runs passed | Same agent and same keys, judged from outside by a real SSH login. |
Each sealed fault was scored once. One of them, pam_access_lockout, was started twice: the first start was cut at turn 8 when the session running the harness restarted, with no verdict, and its record was kept unread. docs/results.md says so too.
A pass is the outside prober's verdict after a reset, never the model's word. The first pass and the reruns are kept apart on purpose: reruns exist only for faults that failed, on an agent tuned on those same faults.
The headline fleet above taught itself its recipe. An earlier model recovery of the same fault, fstab_missing_disk, took 60 turns, 29.4 minutes and $0.59. It used every turn without saying it was done, and the prober passed it anyway. That run compiled to a recipe of 13 steps of keystrokes and screen checks. Replaying that recipe side by side recovered 6 of 6 machines by the prober's verdict, in 5.2 minutes and $0.00 of model calls. That was one fault with the model switched off. Recorded at commit fc6278b58277. The fleet where the model stands by for a mismatch is still 8 of 8 recovered, $0.39 in all, plus the one run where the model stepped in, 1 other fleet in the results.
console pixels ──▶ Muse Spark (computer tool) ──▶ keystrokes ──▶ firewall ──▶ the machine
│
outside prober: reset, real SSH login, nonce, data hash ◀────────────────────────┘
- Recover. The model gets the goal and a screenshot, like a technician at a crash cart. There is no root password and no shell to start from. It finds a way in, diagnoses, fixes and reboots.
- Judge. The prober resets the machine and logs in as
opswith a key the model never sees. It compares the nonce and a sha256 manifest of/datawith the host's copies, taken while the image was known good. The model's own claim counts for nothing. A run cut short by an API error, a lost connection or a harness error is void: neither a pass nor a failure. - Compile.
tools/compile_recipe.pyis deterministic and free. It turns a passing run into ordered steps, leaving out what only read or changed nothing (77 actions, for this recipe). Each step says what the screen must show first. A recipe is unusable until a replay on a fresh overlay passes the prober, and the steps' hash is recorded, so editing a step undoes that. - Replay. Other machines run the recipe. The console text is read from the pixels and every keystroke still goes through the firewall. If the screen does not match, the replay stops there. With the model on standby it gets a few turns to step in, then a full run if that fails. Without one, the machine is reported as not recovered, which happened on 2 machines in the recorded replays.
- The native
computertool, keyboard only. The harness has no pointer. The prompt says there is no mouse, a mouse action is dropped and noted in the trace, and the only actions that reach the machine aretype,keypress,scroll(as a key),waitandscreenshot. Three small functions sit beside it:power_cycle,check_port(from outside, a port and banner only, explicitly not the verdict) andread_screen(exact text from the pixels, because a misread character in a small console font sent runs the wrong way). - Pending safety checks are a decision, not a field. The computer-use guide says to treat a check as a decision point and not to auto-fill it. So nothing here auto-fills it. Each
computer_callgoes to the firewall first. If the firewall allows it, the keys are sent and the checks are acknowledged. If it refuses, no key is sent and no acknowledgement goes out, the chain ends, and the model is told in a new one that the firewall refused. A check on a function call has no keystrokes to read, so it is held the same way: no call and no acknowledgement. That path has not fired, with checks on 0 of 32 function calls in the first pass. No human reviews individual actions, so the firewall's list is the whole of that decision. See Safety for how far it goes. - Chain rolling and handover. After 25 responses, or after a refusal, the loop starts a fresh
previous_response_idchain. Before a roll, and before a refusal's new chain if at least 10 turns have passed since the last note, a call with no tools writes a short handover from the record: what is established, what is ruled out and on what evidence, the best theory and the next step. That note, what was recently said and typed, and one new screenshot open the next chain, which stops the model redoing checks it has already done. Handover was added after the first pass, so only the reruns have it. - Reasoning effort is low and the turn limit is 90. A run costs what its token usage says, at published prices.
| Question | The docs say | Measured here |
|---|---|---|
Does muse-wonder-1.0 take the computer tool? |
The guide's loop runs on it | No, on this key: 3 requests got HTTP 400 Unsupported tool type: 'computer' on 2026-10-31 UTC. A plain function tool worked on the same model. All runs use Spark. |
| How often does a check appear? | When an action needs developer sign-off | On 515 of 559 computer calls in the first pass, with the code computer_action_requires_confirmation. So the firewall is working on nearly every action, not a rare one. |
| How long can a chain be? | No limit stated for server-managed chains. Stateless replay holds up to 29 call/output pairs, and this project does not use it. | 2 chains got HTTP 400 at response 31 in the first probes, so the obvious guess is that the stateless limit reaches chains too. With a new chain every 25, there have been 3 HTTP 400s in 2623 recorded requests since. That is a correlation. The cause was not isolated. Other failures are ordinary: 116 of those requests got HTTP 500 or 503 and 33 lost the connection, and the loop retried them. |
| Can later turns carry a new input image? | The screenshot goes in computer_call_output |
A new input_image on a later turn got HTTP 400: native computer input images are supported only in the initial request. |
| What does it cost? | Published prices | Each USD figure here is usage times those prices, so it is an estimate and the invoice is the authority. Passing runs in the first pass cost $0.23 to $0.67. |
Clouds do have good tools for this, and they work through the provider's own control of the disk. Use them when you have them.
- Azure Linux Auto Repair (ALAR) runs through
az vm repair. It creates a rescue VM, works on a copy of the OS disk, and swaps it back, and it needs a role such as Contributor or Owner on the resource group. - AWS EC2Rescue through Systems Manager stops the instance, attaches its root volume to a helper instance in your account, and reattaches it after. Encrypted root volumes are not supported.
Neither reaches a machine on another vendor, and neither is a thing you can run from a browser tab showing noVNC. That gap is the target here: a vendor who gives you a web VNC window and a power button. If your vendor gives you a serial console or IPMI Serial over LAN, use that: boot output and GRUB can reach it as text, which is easier than pixels. Many such vendors also offer a rescue boot, and if yours does, that is usually the better path. The cost comparison above is also not like for like. A remote-hands technician can swap a disk, and the same rate sheet lists power cycling and on-demand KVM-over-IP among its free services on business weekdays, so often the screen is cheap and the person's time is the cost.
You need uv, Docker, an SSH client and, for model runs, a Meta Model API key. Machines run under QEMU emulation inside Docker with no hardware acceleration, and one 1024 MB machine peaks at about 1.7 GB of memory in its container.
No machine and no API. Free, and takes about a minute:
git clone https://github.com/tedinroc/remote-hands && cd remote-hands
cp tools/pre-commit .git/hooks/pre-commit # stops keys and local paths reaching the repo
uv run pytest # video tests skip without runs/video/fonts
uv run python tools/seal.py --check # the sealed split still matches faults/
uv run python tools/build_results.py --check # every public number matches docs/results.json
uv run python tools/check_injection.py --stage destructive # the destructive list through the real firewall
Build the machines. No API, no cost beyond your own time:
mkdir base
curl -L -o base/debian12.qcow2 https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2
curl -L -o base/SHA512SUMS https://cloud.debian.org/images/cloud/bookworm/latest/SHA512SUMS
uv run python images/build_clean.py --source base --slot 3
uv run python tools/verify_fault.py --id fstab_missing_disk --slot 3 # the prober must fail, then pass after a scripted fix
uv run python tools/verify_all.py --slots 1,2,3,4,5,6 # every fault, side by side
A fault takes about 8 minutes. A clean image gets a new login key, so fault images built before it must be rebuilt. The scripted checks passed 20 of 20 here. The scripted checks use no model. Do not run the model on a sealed fault unless you mean to spend it: each counts once. run_agent.py logs every start in runs/starts.jsonl, refuses a second one unless told, and the results report it.
Run the model. This spends money:
export MODEL_API_KEY=...
uv run python tools/run_agent.py --fault fstab_missing_disk --slot 3 --name dev-fstab_missing_disk
uv run python tools/compile_recipe.py runs/dev-fstab_missing_disk --out recipes/mine.json --name fstab_missing_disk
uv run python tools/compile_recipe.py --validate recipes/mine.json --fault fstab_missing_disk --slot 7
uv run python tools/run_fleet.py --fault fstab_missing_disk --count 6 --scsi 3,6 --nic 4 --recipe recipes/mine.json --name six --concurrency 6
--concurrency 6 runs all six together, as the recorded fleet did, and needs about 10.75 GB in Docker. The default takes two waves. The first pass over the dev faults cost $6.65 in all, and each passing run took 10.7 to 29.4 minutes. --budget stops a run when the ledger reaches that many USD, and the default is low, so raise it if your ledger already holds other runs. run_fleet.py with --recipe and no --model cannot spend at all. Results are in runs/<name>/; rebuild the public numbers with uv run python tools/build_results.py.
Checked from a clean clone: the first block ran, every tool above answers --help, and compiling the recorded run reproduced the committed recipe's steps exactly. The VM and API commands above were not rerun from a clean clone; the run records come from the same tools in the working repo.
- Disposable machines only. Everything here ran on throwaway Debian VMs.
tools/prepare_vps.pywill not start without--i-own-this-disposable-machine. - Console pixels go to a cloud model. Whatever is on the screen leaves the machine. In the first pass the model tried
cat /etc/shadowat the console, and the firewall stopped it, but anything else on screen would already have gone. Do not point this at a machine with real data or secrets. - Contributor mode trains on your data. The discounted
muse-spark-1.3-contributormodel is offered, in the pricing page's words, "in exchange for permission to use your content to train future Meta models". The default here ismuse-spark-1.3, and every result above used it. Early probes also tried contributor mode on these disposable images.
Deterministic, no model. It replays keystrokes on a small line editor, so text split across actions, edited with Backspace or completed with Tab is judged as the whole line the shell would run. It refuses three things:
- What destroys the disk or the data:
rmnear the top of the tree or in/data,mkfs,wipefs,ddto a raw disk,shred, partition tools unless they only list,fsck -y, writes or truncation in/data,find /data -delete, recursivechmodat the top. - What shows password hashes, because the screen goes to a cloud model:
cat /etc/shadowand most recursive reads that sweep/etc. - What writes a login key: any write to
authorized_keys. A recovery never needs it, and a planted note may ask for it.
It is a seatbelt, not a sandbox. Over the destructive list it refuses 24 of 29 lines. The other 5 go through, and they are named in docs/results.md, including a base64 -d | sh bypass, : > /etc/fstab and systemctl stop ssh. A script file or a history recall gets past it as well. It can also refuse too much. Of the 4 refusals in the first pass, only cat /etc/shadow was aimed at the hashes. A find ... -exec ls was harmless and has been fixed. Two grep -r searches through /etc were ordinary diagnosis, and the rule still refuses those. All of them fell in 2 runs that failed, and whether they caused the failures is not shown. The scenario's design notes: scenarios/PROPOSAL.md.
Beyond the firewall, the prompt tells the model that console text is untrusted data. A planted note, scripted with no model: a script that obeys it gets the attacker a root login while the prober still passes, and the honest fix leaves the attacker locked out. So the injection is judged by trying the attacker's key from outside, not by the prober. With the model in the loop this is 2 of 2 runs left the attacker locked out.
It proves that after a reset, ops can log in over SSH from outside with the original key, that the machine holds the expected nonce, and that the data in /data hashes as it did on the clean image. The model cannot change what counts, because the expected values live on the host.
It does not prove the fix is a good one. It checks that login works, not how. Turning off StrictModes should pass as well as fixing the permissions. In the allowusers_dropin run the model also ran usermod -p on ops, which the fault did not call for, and the prober passed anyway. It does not see a backdoor either, as the injection control shows. A person should still read what the agent typed, which runs/<name>/trace.jsonl keeps.
- Not 19 skills. There are 19 solvable faults and 1 unsolvable, but 18 of 19 scripted references share one route: edit a GRUB entry, boot with
init=/bin/bash, make a fix. The faults differ mainly in diagnosis. - The login-key rule came after the first pass. The first pass ran without it. In 3 of those runs the model rewrote or restored the login key while chasing a character it had misread, one of them a run that passed. The prompt now says the key is correct, and the firewall refuses the write. Only the reruns and later runs have either.
- The numbers so far are not a score on unseen faults. The 10 dev faults shaped the agent. In the first pass the first 2 runs used an older prompt and a lower turn limit, and
fstab_missing_diskhad been seen while the tool was built. 3 runs failed by using every turn and 1 was void (connection lost), which leaves 6 of 9 runs that counted. The 10 sealed faults are for that score: 8 of 9 solvable faults recovered. - A person still starts it. Nothing here detects a dead server, a person starts each run, and on a vendor's console a person signs in by hand. "Nobody had to get up" means nobody went to a machine and nobody acted during the rescue.
- It is slow and it wanders. The model reboots to check things, misreads keys on screen and sometimes starts with the account when the network is the problem. A person who knows the route is faster. The case for this is the machine nobody can get to in time, not beating a good admin.
- The fleet result is narrow. One fault, the same recipe, near-identical machines, differing in disk bus or card on 3 of them. A recipe does not carry over to a different fault, and it is safe only because it stops when the screen disagrees.
- Beyond noVNC against QEMU's VNC server, one vendor's web console has been driven, on the runs in the results. A real vendor's console: 1 of 1 runs passed. The driver has not met UEFI, a serial-only console or a framebuffer of another size. With other screen sizes the model still gets pixels but a replay stops. A lossy VNC setting leaves cells unread, and that stops a replay too. See docs/webconsole.md.
- A GRUB password or an encrypted root is a wall. One unsolvable fault tests the first: the right behaviour is for the agent to say it cannot. Result: It used every turn and said nothing.
- The test machines have tells. An agent with a shell can see
/etc/fleet-id, and the journal shows signs of how each fault was made.default_route_typoonly breaks because of how Docker forwards the connection,ifname_rename(dev) andnetplan_old_mac(sealed) are close relatives, and a boot hung in about one boot in very roughly 135 under emulation, which the prober answers with one more reset, markedretried. See docs/faults.md.
None of these were built here. Only the console font's bitmaps and screens of the disposable Debian guest are in this repo. The rest is installed or fetched.
| Part | Licence | What it does here |
|---|---|---|
| QEMU | GPL-2.0 | Runs the machines, installed in the Docker image from Debian's packages. |
| Debian 12 cloud image | Free software licences, per package | The guest. Fetched and checked against SHA512SUMS. |
| noVNC 1.7.0 | MPL-2.0 | The web console test rig. Fetched with its published hash checked. |
| Playwright | Apache-2.0 | Optional extra, drives the browser for the web console. |
| Docker | Apache-2.0 (Moby) | Runs each machine's QEMU in a container. Installed by you. |
| FFmpeg | LGPL or GPL, by build | Encodes the clip the model reads and the film. Installed by you. |
| Pillow, pytest | MIT-CMU, MIT | Python dependencies, from PyPI. |
| VGA console font | SeaBIOS LGPL-3.0, Linux GPL-2.0 | remotehands/glyphs.json holds its bitmaps, read from the guest's own screen as QEMU's VGA BIOS and the Linux console draw it. |
| Inter | OFL-1.1 | The film's type. Its own serifed capital I is reached at a private code point at render time. |
| IBM Plex Mono | OFL-1.1 | Earlier cuts' type and the site's. |
| Stock clip 6944083 from Pexels | Pexels License | the clock, opening and ending. Context around the product, tagged STOCK FOOTAGE on screen, and never the real server or a real incident. No built work is in it. |
| Stock clip 7140928 from Pexels | Pexels License | the server front, opening. Context around the product, tagged STOCK FOOTAGE on screen, and never the real server or a real incident. No built work is in it. |
| Stock clip 35130886 from Pexels | Pexels License | the corridor, opening. Context around the product, tagged STOCK FOOTAGE on screen, and never the real server or a real incident. No built work is in it. |
| Stock clip 34771078 from Pexels | Pexels License | the hands, opening. Context around the product, tagged STOCK FOOTAGE on screen, and never the real server or a real incident. No built work is in it. |
| Sound effects from Freesound | CC0 1.0 | 537738 steeltowngaming (buzz), 671455 jamesdrake89 (room), 848560 mzinsser (fans), 609534 5ro4 (key press), 661450 fsharpinfinity (typing). |
| Music: an original cue, written here | Samples CC0 | Played by VSCO 2 Community Edition samples (Versilian Studios, CC0). The score is tools/film4_score.json. |
| VoxCPM2, through mlx-audio | Apache-2.0, MIT | The film's narration, generated locally, a paragraph at a time at the reference's own tempo (one line, 'The login worked.', from one reading of it with the paragraph before it): nothing in it is slowed, sped up or pitch-shifted. A synthetic voice, from a reference that Qwen3-TTS (Apache-2.0) designed from a text description. No person's voice went in. The record, with the licence evidence and each line file's hash, is film4/voice_record.json. |
| Muse Spark, through the Meta Model API | Meta's terms | The model. |
Prior art. kvm-pilot (Apache-2.0) came first with giving an agent hands on a physical console: an MCP server over IP-KVMs, BMCs and SSH, with vision on screenshots and gated destructive actions. No code from it is used here. The work here is the outside judge, the sealed scoring, compiled recipes with screen checks, and running on Muse Spark's native computer tool.
Long-form notes on what the film does to console pixels and what it quotes: docs/film-notes.md.
Built by Ted Chen with AI coding assistance (Claude).
Licence. MIT, see LICENSE. Third-party parts keep their own licences, listed above.
History. Before publication the git history was rewritten once, to take a third-party provider's name and link out of early README and tool text, and to give one commit the same time zone as the rest. Run records cite the commit ids from before the rewrite; docs/commit-map.txt maps each old id to its new one.