web-agent

A mobile-first web chat plus Siri and Matrix bridges for a persistent pi coding-agent session — one agent, three surfaces, one memory.

Talk to the same always-on agent from your phone in three ways: dictate to Siri and hear the answer spoken back, send a message in a Matrix room, or open a chat-style web dashboard with live streaming, image attach/edit and tappable choices. Everything lands in one long-lived session, so context follows you across surfaces.

 Siri (Shortcuts/SSH) ─┐                      ┌─ spoken answer, or Matrix fallback
 Matrix room ──────────┼──▶ FIFO ──▶ daemon ──┼─▶ pi --mode rpc  (persistent session)
 web dashboard ────────┘         ▲   │        └─ Matrix post / SSE stream
                                 │   └─ HTTP + SSE control plane :8383
                          Matrix listener (long-poll sync, event-driven)

Features

• Persistent session — one pi --mode rpc process with a fixed session id; full memory across restarts, shared by every surface. Restarts resume the active transcript (recorded in $AGENT_SESSION_DIR/active-session), not whichever namesake file pi's id lookup happens to pick first.
• Siri — dictate a command over SSH; short answers are spoken back, long ones fall back to Matrix.
• Matrix — an event-driven listener (server-held long-poll sync, not busy polling) forwards room messages to the agent; answers post back to the room.
• Web chat — mobile-first, light/dark:
  • live token streaming with tool-call rows (Claude Code style) and markdown — one row per call, described in plain language (Running ls -la) with the exact command and result behind a toggle, identical live and after a reload
  • Stop (clear_queue + abort) and send-while-busy = steer
  • model switcher, status dot, reconnect with snapshot re-render
  • stays live on a phone: an iPhone web app that is locked or sent to the background loses its stream and often resumes with a socket that looks open but delivers nothing. The daemon pings every 15s as a real event; the page reconnects after 40s without one, whenever it comes back into view, and when the network returns — and every reconnect resyncs the transcript, the status and any open question, even in the middle of a run (it used to sit on "working…" until the next prompt)
  • session history sidebar — list, view, archive and create sessions; new sessions can be named at creation; beside the session name in the header, the pencil renames the live one and the archive icon archives it (and starts a fresh one)
  • resume a past session from its banner ("▶ Resume"): it becomes the live session for web, Siri and Matrix, and an archived one moves back out of Archived. Any run it interrupts is answered rather than dropped
  • delete a past session from the same banner ("Delete"), so what is about to go is on screen first — a soft delete into trash/, purged after 30 days
  • pictures: attach from camera/roll (client-side downscale); the daemon auto-switches to a vision model when the session model is text-only
  • image editing: sent photos are saved to disk with their paths named in the prompt, so the agent edits them with ffmpeg; results render inline
  • interactive choices: the agent can end a reply with a choose block that renders as tappable buttons
  • failures are loud: a run that ends in a provider/runtime error (stopReason: "error") renders as an error block in the transcript — live and after a reload — is logged as status: error in the task log, and is spoken/posted as an explicit failure instead of a blank answer
• Session history sidebar — every transcript pi writes stays on disk; the dashboard lists them (previous + archived), opens any of them read-only, and can archive the running session or start a new one without restarting the daemon:
  • GET /sessions — list sessions/ and archive/ with the session's name and first-prompt title, message count and timestamps; the active file is flagged. The sidebar leads with the name (the thing you search for) and shows the first prompt on a dimmer line underneath
  • GET /session?file=&dir= — one transcript as a messages snapshot (thinking stripped, tool results capped), rendered by the same snapshot renderer as the live view; basename + dir allowlist, realpath containment, image blobs dropped
  • POST /opensession {file, dir} — resume an existing transcript in place: switches pi onto that file and tells every tab to refresh. Same allowlist/containment as /session; the session keeps its own messages and name. A run in flight is stopped, and a Siri/Matrix caller waiting on it is told the run was interrupted instead of timing out. Resuming from archive/ moves the file back into sessions/ (so it is a normal live session again and re-archiving works) — 409 if that name is already taken there, and the file is moved back if the switch fails. An empty or non-JSON file is refused with 422. (Needed because a restart resumes the oldest file with the session id — docs/spec.md §17.5.)
  • POST /newsession / POST /archive — switch pi onto a fresh header-only session file (id preserved, so --session-id resume keeps working) and optionally move the old transcript to archive/. A run in flight is aborted first; a session_switched SSE event tells every open tab to refresh. RPC new_session was deliberately not used: it mints a random session id and would orphan the session on the next restart.
  • GET /update, POST /update/check, POST /update — the self-update check and the Update button (see Updating from the dashboard)
  • GET /version — what the running daemon is: DAEMON_VERSION plus the commit and timestamp install.sh wrote to $AGENT_SESSION_DIR/build.json (absent is fine — a hand-copied daemon reports its version alone). The dashboard renders it in the build badge; /state stays an untouched pi pass-through.
  • POST /deletesession {file, dir} — soft-delete a past transcript: it moves to $AGENT_SESSION_DIR/trash/ with a timestamp suffix, and the janitor purges trashed files older than AGENT_TRASH_DAYS (default 30 days). The live session is refused with 409 (switch away first), and so is a delete that would race a switch. Until the purge it is recoverable with a plain mv; after it, the nightly restic snapshot is the only copy. Attachments are NOT deleted — files under attachments/in|out|web-N… are keyed by request id, not by session, so a transcript and the pictures it referred to are independent.

Requirements

• pi v0.85+ on the host
• Python 3.10+ (stdlib only — no pip dependencies)
• a Matrix account + room for the bridge (optional; skip it and use web + Siri)
• WireGuard or another private network for the dashboard (it speaks plain HTTP by design — no TLS inside the tunnel)

Install

git clone https://github.com/kamilakis/web-agent.git
cd web-agent
./install.sh

install.sh copies bin/* to ~/.local/bin, the chat to ~/.local/share/agent-session/web/, and the units to ~/.config/systemd/user/, then enables both services and restarts them so the new code is what runs (NO_RESTART=1 ./install.sh to restart later yourself).

Post-install

1. Matrix bridge (optional): create a config dir with three plain-text files — token, homeserver_url, room_id — and point both services at it:

   mkdir -p ~/.config/web-agent/matrix
   printf '%s' 'https://matrix.example.org' > ~/.config/web-agent/matrix/homeserver_url
   printf '%s' '!roomid:example.org'         > ~/.config/web-agent/matrix/room_id
   printf '%s' 'syt_yourtoken'               > ~/.config/web-agent/matrix/token
   # in both unit files: Environment=AGENT_MATRIX_CONFIG=%h/.config/web-agent/matrix
   systemctl --user daemon-reload && systemctl --user restart agent-matrix-listener.service

2. Siri (optional): a Shortcuts automation — Dictate Text → Run Script Over SSH (ssh <host> agent-task "Dictated Text", full path required) → Speak Text ← Shell Script Result.

   ⚠ Put personal settings in a drop-in, not in the unit. install.sh copies the generic systemd/*.service files over the installed units, so Environment= lines added to a unit by hand are lost the next time it runs (learned the hard way: a re-install dropped AGENT_WEB_HOST back to 127.0.0.1 and emptied AGENT_MATRIX_SENDERS, i.e. the dashboard went unreachable and the Matrix allowlist opened up). Use:

   mkdir -p ~/.config/systemd/user/agent-session.service.d
   cat > ~/.config/systemd/user/agent-session.service.d/local.conf <<'EOF'
   [Service]
   Environment=AGENT_WEB_HOST=192.0.2.10
   Environment=AGENT_MATRIX_CONFIG=%h/.config/web-agent/matrix
   EOF
   systemctl --user daemon-reload && systemctl --user restart agent-session

3. chat: open http://<host>:8383 from a device on the private network. AGENT_WEB_HOST (default 127.0.0.1) controls the bind address — set it to your VPN IP to reach the dashboard remotely.

Always on

agent-session is a systemd user service: enabled, and Linger=yes so it runs at boot and keeps running with nobody logged in.

git clone … && cd web-agent && ./install.sh      # installs, enables, writes build.json

systemctl --user status agent-session            # is it up?
curl -s http://<host>:8383/version               # which build is running?
curl -s http://<host>:8383/state                 # is the chat path answering?

/state is the readiness check that matters: it round-trips get_state to pi, so a 200 means the agent can actually answer. A 504 means the daemon is up but pi is not — restart the service.

systemctl --user restart agent-session           # interrupts a run in flight
journalctl --user -u agent-session -b            # the systemd view
tail -f ~/.local/share/agent-session/daemon.log  # the daemon's own log

A restart is cheap: the daemon resumes the active transcript (see AGENT_SESSION_DIR/active-session), so the conversation continues.

Two settings exist because of one incident — at boot on 2026-09-21 11:07:26 the daemon died with FileNotFoundError: 'pi'. pi lives in ~/.local/bin, and a user unit started before any login has imported the environment gets systemd's default PATH, which does not include it; the service only came up because systemd retried six seconds later. So:

• the daemon finds its helpers next to its own binary (find_bin()), with AGENT_PI_BIN as an override, and at startup extends PATH for pi and everything it runs: its own dir and ~/.local/bin first, the standard system dirs appended, nothing already there removed. (An earlier fix pinned PATH in the unit, which dropped every other directory the environment carried.) Add dirs of your own, such as Go's, in ~/.config/environment.d/50-path.conf: PATH=/usr/local/go/bin:$HOME/go/bin:${PATH} — the user manager reads it at boot too;
• Restart=always with StartLimitIntervalSec=0 — an always-on chat service keeps retrying rather than sitting dead after a burst of failures.

tests/service.test.sh runs the daemon with a PATH stripped of every place pi could live, including one case with nothing installed at all.

Updating from the dashboard

Once ./install.sh has run from a git checkout, later updates need no shell. The daemon checks that checkout for new commits on master every 15 minutes (git fetch only; nothing moves). When there are some, a bar under the header says so — Update available · 2 commits · smaller text…, with the commit list in its tooltip — and offers Update and Later.

Update starts agent-update.service, a one-shot unit of its own, which:

1. refuses, touching nothing, unless the checkout is clean, on master, and can fast-forward (the bar says so up front, with no button, when it can't);
2. fast-forwards and runs tests/run-all.sh — a failure stops it there, with the checkout moved back and the old build still running;
3. runs ./install.sh, which restarts the daemon (a reply in flight is stopped; the button says so when one is);
4. waits for the new build to answer (health.json, written once pi answers get_state), and if it does not within 90s, rolls back: the previous commit is checked out and installed again.

The bar shows each stage while it runs. When the new build is up, open pages reload themselves — except one with a half-written message or an open question, which gets a Reload button instead — and say Updated to …. Everything the updater did is in $AGENT_SESSION_DIR/update.log, its outcome in update.json.

It is its own unit because install.sh restarts agent-session, and systemd stops everything in that unit along with it: an updater started as the daemon's child would kill itself halfway through. The remote is only ever read; a failed test or rollback only moves the local branch back to where it was.

The tests need node and curl on the unit's PATH. If node lives somewhere unusual, add its dir in ~/.config/environment.d/, or skip the tests with a drop-in for agent-update.service (Environment=AGENT_UPDATE_SKIP_TESTS=1).

Var	Default	Meaning
AGENT_UPDATE_CHECK	900	seconds between update checks; 0 turns them off
AGENT_UPDATE_BRANCH	master	the branch to follow
AGENT_UPDATE_SKIP_TESTS	0	(agent-update.service) 1 installs without running the tests
AGENT_UPDATE_HEALTH_WAIT	90	(agent-update.service) seconds for the new build to answer before rolling back

Usage line

The dashboard shows one dim line above the composer: what the current model has cost, and — where the provider can be asked — what is left on the account.

deepseek-flash · $4.40 left · today $0.51

Tap it for the full picture: balance and top-up, today's turns and tokens (input/output/cached), the session total, and which helper supplied the balance.

Two sources, deliberately kept apart:

Cost, from the transcript — works for every provider. pi writes a usage block with a cost on every assistant message, so the daemon sums those from the live transcript (GET /usage). No API key, no per-provider code, and it keeps working the moment you switch model or provider. Each message also names its own model, so a session that spanned a switch is summed correctly.

Balance, from a helper — provider-specific and optional. If AGENT_USAGE_CMD is set (it reports on AGENT_USAGE_PROVIDER, default AGENT_PROVIDER, and is not run for any other provider), or an agent-usage-<provider> / <provider>-usage executable is installed, the daemon runs it on a slow clock (AGENT_USAGE_INTERVAL) and shows what it returns. The contract is JSON:

{"currency": "USD", "total": "4.40", "topped_up": "4.40",
 "spent": 5.51, "topup_total": 9.91}

total may be a decimal string (as DeepSeek returns it) or a number of minor units with an exponent. error instead of those keys is fine — the line says balance unavailable and keeps the cost. Nothing is invented: a provider with no helper simply has no balance, because only the provider knows one.

DeepSeek is the worked example on this box: api.deepseek.com/user/balance is all an API key can reach — the token-usage charts need a browser session — so the line shows credit left plus spend since the top-up, anchored on topup_total (see live-stats/bin/deepseek-usage, which owns both the helper and that anchor).

Approval gate

A pi extension asks before anything: source in ~/assistant/pi-extensions/approval-gate/, symlinked into ~/.pi/agent/extensions/ so pi discovers it (and so its changes are under version control). Read-only commands run without a prompt:

Runs unattended	ls, cat, head, wc, grep, rg, find (without -exec/-delete), sed -n, awk (without system(/>/`
Asks first	anything that writes, deletes, escalates, executes code or leaves the box: rm mv cp mkdir touch chmod tee, any >/>> redirect, sed -i, find -exec, xargs, sudo systemctl kill reboot, git commit|push|reset|clean|apply, python node sh bash, make npm pip, ssh scp rsync, docker, and anything not on the allowlist

Every segment of a compound command must be a read (ls && rm x asks), and a redirection anywhere makes it a write (echo hi > f asks). A wrong ask costs one tap; a wrong allow changes something nobody agreed to, so unknown commands always ask. The reasoning is shown in the dialog (why ask: …).

Approval choices: Allow once, Allow for this session (in-memory, resets on restart), Deny, and Deny, but instead… — the last one hands your typed instruction back to the agent as the block reason, so it can correct course. /gate on|off|status|forget controls it; /gate forget clears the session allow-list.

In the dashboard a question appears as a modal with a button per option, so it reaches you wherever you are — phones included — instead of waiting on a tty. Questions that overlap (parallel tool calls) queue behind each other, and one asked while the page was closed or its connection dropped is picked up on reload or reconnect (GET /ui). The dialog's Cancel picks the gate's own Deny option, so a refusal is reported to the agent as a denial, not as nobody answered. Nothing hangs forever: an unanswered question is cancelled after AGENT_UI_TIMEOUT (default 90s), or after AGENT_UI_NOUI_GRACE (15s) when no dashboard is connected at all — which is what lets Siri and Matrix turns fail closed rather than sit there. The dialog counts that down, and a question that runs out of time is reported to the agent as nobody answered, not as a denial, so it never narrates a decision you did not make.

Var	Default	Meaning
AGENT_UI_TIMEOUT	90	seconds an extension question waits for an answer
AGENT_UI_NOUI_GRACE	15	seconds to wait for a dashboard to appear before cancelling

tests/ui-relay.test.sh covers the round trip (select, confirm, input, cancel, timeout, no-UI); classify.test.mjs next to the extension holds the 123-case classification table.

Tests

No dependencies, no network, and nothing touches the live agent — every suite starts a throwaway daemon on its own port with a fake pi on PATH:

bash tests/run-all.sh          # everything
node tests/ui.test.js          # one suite

Suite	Covers
describeTool.test.js	the plain-language tool descriptions (§8.3)
toolRows.test.js	tool rows: parallel calls, errors, live vs reloaded
ui.test.js	the page under a stub DOM: error surfacing, resume, two tabs
opensession.test.sh	resume against the daemon: T1–T8, T13
delete.test.sh	delete against the daemon: D1–D6, D8
version.test.sh	GET /version, and that /state stays a pure pass-through
usage.test.sh	GET /usage: transcript totals, balance helpers, and their failure modes
ui-relay.test.sh	extension dialogs reach the dashboard and the answer reaches pi
errors.test.sh	a failed run is surfaced, never a silent empty answer
hardening.test.sh	stopped runs, slow SSE clients, dialog timeouts, usage/archive/vision edge cases
install.test.sh	install.sh restarts what it installed (stubbed systemctl)
update.test.sh	the daemon's update check and POST /update, against a local bare "origin"
agent-update.test.sh	bin/agent-update: fast-forward, test, install, rollback, refusals
resume.test.sh	a restart resumes the recorded transcript, not the oldest namesake

See tests/README.md for the fake pi's controls and the known gaps.

Configuration

All knobs are env vars (set them in a systemd drop-in, not the unit files — see Post-install). Defaults are generic — nothing personal is baked in.

Var	Default	Meaning
AGENT_PROVIDER / AGENT_MODEL	deepseek / deepseek-v4-pro	pi provider and model
AGENT_VISION_MODEL	deepseek/deepseek-v4-flash-vision-exp	model auto-selected for image turns
AGENT_TRASH_DAYS	30	days a deleted transcript stays in trash/ before the janitor purges it
AGENT_USAGE_CMD	unset	command that prints the current provider's account balance as JSON (see Usage line above)
AGENT_USAGE_PROVIDER	AGENT_PROVIDER	the provider AGENT_USAGE_CMD reports on; on any other provider no balance is shown
AGENT_USAGE_REFRESH	30	seconds between transcript re-reads for the usage line
AGENT_USAGE_INTERVAL	900	seconds between balance API calls
AGENT_SESSION_ID	siri-agent	session id = memory key; change to wipe
AGENT_SESSION_NAME	siri-agent	display name for new sessions (the web UI can name them per-session)
AGENT_TASK_WAIT	15	seconds Siri holds the SSH call open
AGENT_SETTLE_GRACE	30	seconds before an unanswered answer falls back to Matrix
AGENT_WEB_HOST / AGENT_WEB_PORT	127.0.0.1 / 8383	dashboard bind address/port
AGENT_WEB_TOKEN	unset	optional bearer token (Authorization: Bearer, or ?token= for EventSource)
AGENT_WEB_ENABLED	1	0 disables the HTTP server
AGENT_ATTACH_DIR	~/.local/share/agent-session/attachments	image editing scratch space, served by /media
AGENT_MATRIX_CONFIG	~/.config/web-agent/matrix	dir with token / homeserver_url / room_id
AGENT_MATRIX_SENDERS	(empty)	allowlist of Matrix senders; empty = anyone except the bot
AGENT_MATRIX_STATE_DIR	~/.local/state/agent-matrix-listener	Matrix sync-token storage
AGENT_UI_TIMEOUT / AGENT_UI_NOUI_GRACE	90 / 15	extension-question timeouts (see Approval gate)
AGENT_SSE_PING	15	seconds between keep-alive events on the live stream (the page reconnects after 40s without one)
AGENT_PI_BIN	(found)	the pi binary; default: next to the daemon, then PATH, then ~/.local/bin

The dashboard also shows a build badge (top right): its own UI_VERSION and, from GET /version, the running daemon's DAEMON_VERSION and the commit install.sh recorded. The two numbers move together — they identify one deploy, not one file — so a mismatch really does mean the page and the daemon came from different installs. Tap the badge for the full string.

Security notes

• The chat intentionally speaks plain HTTP — put it on a private network (WireGuard/Tailscale) and bind AGENT_WEB_HOST to that interface only.
• AGENT_WEB_TOKEN adds a bearer-token check on every route (including ?token= for EventSource, which cannot send headers).
• /media serves only the attachments tree; realpath containment defeats path traversal and symlink escapes. Request bodies are size-capped (413). /session reads only sessions/ and archive/ by basename, with the same realpath containment.
• The Matrix listener only forwards text messages from AGENT_MATRIX_SENDERS (never the bot's own messages, never edits), and never replays history: the first sync advances until the stream token stops moving.
• The daemon holds no credentials; Matrix credentials live in a plain config dir you control.

How it works

Three writers feed one FIFO that the daemon drains into a single pi session. The daemon derives each run's source (siri / matrix / web) from pi's own agent_start events — never from a guess — so answers always return to the surface that asked: Siri gets a result file (spoken) with a Matrix grace fallback, Matrix gets a room post, and the web dashboard streams everything over SSE. See docs/spec.md for the full design, including the run-state model, the vision-model auto-switch for image turns, and the interactive-choice convention (docs/choose-convention.md).

docs/ also contains the design → external-LLM-review → build trail: spec-review.md (12 findings, all resolved), review-qwen72b.md (an independent Qwen2.5-72B review of the revised spec), and spec.md §17 (failed runs are visible — the 2026-09-25 silent-failure incident and its fix).

Acknowledgments

web-agent is a thin layer around pi — the agent, the tools, the session persistence and the whole interaction model are pi's. Its --mode rpc JSONL protocol turned out to be a complete embedding API: streaming deltas, per-tool progress, steering, abort, model switching and even headless UI dialogs are first-class protocol events, and the docs match the source line for line. This project is essentially a renderer and a few buttons on top of it. Thank you.

License

MIT