Live views: logs --watch and batch watch

CHANGELOG.md

@@ -17,6 +17,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman
 - **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API.
 - **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`.
 - **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`).
+- **Live views.** `spicrawl logs --watch` is a live, scrollable request log (new rows on top, filter by status, tier and errors, search, Enter for the `logs get` page) and `spicrawl batch watch <id>` is a dashboard for a batch job (progress bar, done/failed/total, ETA, credits, a per-item table, failed-only filter, item detail). Both are full-screen only when stdin and stdout are terminals and none of `--json`, `-q`, `CI` or `TERM=dumb` is set; otherwise `logs --watch` streams new rows as plain lines (JSONL with `--json`) and `batch watch` is `batch wait`. `batch watch` ends with the final job and `batch wait`'s exit codes (0, 4, 11); `NO_COLOR` is honoured; the terminal is restored on q, Ctrl-C (exit 130), errors and panics. New dependencies, all Charm: `charm.land/bubbletea/v2`, `charm.land/lipgloss/v2`, `charm.land/bubbles/v2`. The release binary grows by about 2 MB (10.2 to 12.3 MB for linux/amd64, stripped; about +20%).
 - **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names.
 - **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary.
 

DESIGN.md

@@ -170,6 +170,7 @@ spicrawl batch get <id>
 spicrawl batch results <id> [--status S] [--all] [-o FILE]   JSONL
 spicrawl batch content <id> <seq> [-o FILE]                   one item's document
 spicrawl batch wait <id> [--max-wait 30m]                     exit 11 when it gives up, 4 if failed/cancelled
+spicrawl batch watch <id> [--poll 2s] [--max-wait 30m]        live dashboard on a TTY; else exactly `batch wait`
 spicrawl batch cancel <id> [--yes]                           --yes required without a TTY
 spicrawl batch retry|close <id>
 spicrawl batch append <id> <file|->
@@ -182,6 +183,8 @@ spicrawl browser url [--engine browser|js] [--country CC] [--ttl S]
 
 spicrawl logs [--errors] [--status S] [--limit N] [--all] [--all-projects]
                                                 GET /v1/requests (--all-projects: all_projects=true, read scope)
+spicrawl logs --watch [--poll 2s] [--errors] [--status S] [--limit N] [--all-projects]
+                                                live view: full-screen on a TTY, else plain lines (JSONL with --json)
 spicrawl logs get <request-id>
 spicrawl usage [--from D] [--to D] [--group-by day]           GET /v1/usage
 spicrawl usage summary | reconciliation

README.md

@@ -180,7 +180,7 @@ The key is passed as an argument, so other users on the machine can see it in th
 | Command | What it does |
 |---|---|
 | `spicrawl scrape <url\|->` | Fetch one page, or many from stdin, as Markdown, HTML, text, JSON, PDF or extracted data |
-| `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` |
+| `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `watch`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` |
 | `spicrawl sessions` | Keep one rendering tier and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` |
 | `spicrawl browser` | Cloud browser over a DevTools Protocol (CDP) WebSocket (`browser url`). Coming soon |
 | `spicrawl init` | Set up AI agent clients: MCP server config and the agent skill |
@@ -190,7 +190,7 @@ The key is passed as an argument, so other users on the machine can see it in th
 | `spicrawl logout` | Remove the saved API key |
 | `spicrawl auth` | `auth status`: which key and base URL are in use, where they come from, and whether the key works |
 | `spicrawl config` | Read and write the config file: `get`, `set`, `unset`, `path` |
-| `spicrawl logs` | Recent API requests, newest first; `logs get <request-id>` for one |
+| `spicrawl logs` | Recent API requests, newest first; `logs get <request-id>` for one; `logs --watch` for a live view |
 | `spicrawl usage` | Billed usage over a time window; `usage summary`, `usage reconciliation` |
 | `spicrawl status` | Check the API is reachable, ready and accepts your key, and that detected agent clients have the MCP server; prints a repair command for each failed check |
 | `spicrawl docs` | Print a Spicrawl docs page as Markdown (`--list` for the index); no API key needed |
@@ -209,6 +209,17 @@ On a terminal `spicrawl --help` lists the commands grouped by purpose, with exam
 
 All of it is on stderr and stdout is byte-for-byte unchanged. It switches itself off, and the CLI prints exactly what it printed before, when stderr is not a terminal (a pipe, a file, an agent or CI log), with `--json` or `-q/--quiet`, with `NO_COLOR` set (or `CLICOLOR=0`), with `TERM=dumb`, or when `CI` is set.
 
+### Live views
+
+`spicrawl logs --watch` and `spicrawl batch watch <id>` open a full-screen view that updates as the API answers. They need a terminal for stdin and stdout; the colours follow your terminal's theme and `NO_COLOR` turns them off (the view stays usable: the selected row is marked with `>`).
+
+- `logs --watch` polls the request log (`--poll`, default `2s`). New requests appear at the top and the selection stays on its row. Arrows, `PgUp`/`PgDn`, `g`/`G` move; `Enter` opens a request (what `logs get` prints); `/` searches; `s`, `t` and `e` filter by status, tier and errors only; `c` clears the filters; `p` pauses; `r` refreshes; `q` or `Esc` quits. `--errors`, `--status`, `--limit` and `--all-projects` work as in `logs`. The filters apply to the requests loaded so far (the newest `--limit`, 200 by default, and everything since), so start with `--errors` or `--status` to narrow the query itself.
+- `batch watch <id>` shows a progress bar with done, failed and total, the ETA and credits, and a table of the finished items (status, HTTP status, tier, duration, URL), newest first. `Enter` opens an item, `f` shows only failed ones, and `s`, `t`, `/` filter as above. When the job finishes the view closes, the final job is printed and the exit code is `batch wait`'s: `0` completed, `4` failed or cancelled, `11` if `--max-wait` elapsed first. `q` leaves the job running (exit `0`); `Ctrl-C` exits `130`.
+
+A failed poll never closes a view: the error shows on the status bar and the poll is retried with backoff (a `429` waits for its `Retry-After`). A wrong key or an unknown job id fails at once with its usual exit code, before the screen opens. Resizing is followed, and the views fit 80x24; below 40x10 they ask for a bigger terminal.
+
+When stdin or stdout is not a terminal, or with `--json`, `-q/--quiet`, `CI` or `TERM=dumb`, there is no full-screen view: `logs --watch` streams new requests as plain lines, oldest first (JSONL with `--json`, which is the default when stdout is piped), and `batch watch` is exactly `batch wait`.
+
 ### How to convert a web page to Markdown from the command line
 
 ```sh

go.mod

@@ -3,13 +3,34 @@ module github.com/OfficialSpicrawl/cli
 go 1.27.1
 
 require (
+	charm.land/bubbles/v2 v2.2.1
+	charm.land/bubbletea/v2 v2.1.0
+	charm.land/lipgloss/v2 v2.0.6
+	github.com/charmbracelet/colorprofile v0.4.3
+	github.com/charmbracelet/x/ansi v0.11.9
+	github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f
+	github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169
 	github.com/mattn/go-isatty v0.0.20
 	github.com/spf13/cobra v1.10.2
 	github.com/spf13/pflag v1.0.9
 	golang.org/x/term v0.6.0
 )
 
 require (
+	github.com/aymanbagabas/go-udiff v0.4.1 // indirect
+	github.com/charmbracelet/harmonica v0.2.0 // indirect
+	github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4 // indirect
+	github.com/charmbracelet/x/term v0.2.2 // indirect
+	github.com/charmbracelet/x/termios v0.1.1 // indirect
+	github.com/charmbracelet/x/windows v0.2.2 // indirect
+	github.com/clipperhouse/displaywidth v0.11.0 // indirect
+	github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
-	golang.org/x/sys v0.6.0 // indirect
+	github.com/lucasb-eyer/go-colorful v1.4.1 // indirect
+	github.com/mattn/go-runewidth v0.0.30 // indirect
+	github.com/muesli/cancelreader v0.2.2 // indirect
+	github.com/rivo/uniseg v0.4.7 // indirect
+	github.com/xo/terminfo v1.0.0 // indirect
+	golang.org/x/sync v0.22.0 // indirect
+	golang.org/x/sys v0.47.0 // indirect
 )

go.sum

@@ -1,16 +1,61 @@
+charm.land/bubbles/v2 v2.2.1 h1:Fq1+qm5hV6GkvzLQDhCBpXXE5tLgvh1PRriCLwSvIQU=
+charm.land/bubbles/v2 v2.2.1/go.mod h1:wdMgn+sje1KNXdwFizIWjbf328fIUBxqEmJ/vYPo8yc=
+charm.land/bubbletea/v2 v2.1.0 h1:A9fj9h5LFrvQhQHFNggp8K8rvY6CUfgn9dMbb6p2HIw=
+charm.land/bubbletea/v2 v2.1.0/go.mod h1:xWHeyxIX9/wiFU2kgPW04ojOZyTzrWf+8lGjUCaftiU=
+charm.land/lipgloss/v2 v2.0.6 h1:EaGKeuA8FvF+v2BT5VmZd2LoYLaMZJXA5n34th8nCIQ=
+charm.land/lipgloss/v2 v2.0.6/go.mod h1:ipDDJNSGa1hlwDtSfW1s2/xR8Vdhbut4PXh2zEKZd0Q=
+github.com/aymanbagabas/go-udiff v0.4.1 h1:OEIrQ8maEeDBXQDoGCbbTTXYJMYRCRO1fnodZ12Gv5o=
+github.com/aymanbagabas/go-udiff v0.4.1/go.mod h1:0L9PGwj20lrtmEMeyw4WKJ/TMyDtvAoK9bf2u/mNo3w=
+github.com/charmbracelet/colorprofile v0.4.3 h1:QPa1IWkYI+AOB+fE+mg/5/4HRMZcaXex9t5KX76i20Q=
+github.com/charmbracelet/colorprofile v0.4.3/go.mod h1:/zT4BhpD5aGFpqQQqw7a+VtHCzu+zrQtt1zhMt9mR4Q=
+github.com/charmbracelet/harmonica v0.2.0 h1:8NxJWRWg/bzKqqEaaeFNipOu77YR5t8aSwG4pgaUBiQ=
+github.com/charmbracelet/harmonica v0.2.0/go.mod h1:KSri/1RMQOZLbw7AHqgcBycp8pgJnQMYYT8QZRqZ1Ao=
+github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4 h1:hISLzUbwcvJbGoNR6A3L/TpguRRnACBmZOee8LcHLJw=
+github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4/go.mod h1:aJDVk+WOVM1K04rdc/71PMPoUGfjxjhvXfP3Pjz1Zio=
+github.com/charmbracelet/x/ansi v0.11.9 h1:kIrqLBxtTVbLyyHO2Evoi93EjQo1mylWmpNEsfPxXAM=
+github.com/charmbracelet/x/ansi v0.11.9/go.mod h1:DG/hpOnaI2QJ/KMMqLL2h0KVhluuJLazf8973P+Vy5w=
+github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f h1:8CnFOYzrMArVN42jYaGvnBo3mxdONgt09fly+9B96GY=
+github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f/go.mod h1:V8n/g3qVKNxr2FR37Y+otCsMySvZr601T0C7coEP0bw=
+github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169 h1:jP7g4cGLDeRdqlUbQ9U3pQLm4MdOPiMknRqPId46XyY=
+github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169/go.mod h1:aRoQwQWmN9LBG2xi3sVByMFt2fdkPCagd0GAJ1qwOfw=
+github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
+github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
+github.com/charmbracelet/x/termios v0.1.1 h1:o3Q2bT8eqzGnGPOYheoYS8eEleT5ZVNYNy8JawjaNZY=
+github.com/charmbracelet/x/termios v0.1.1/go.mod h1:rB7fnv1TgOPOyyKRJ9o+AsTU/vK5WHJ2ivHeut/Pcwo=
+github.com/charmbracelet/x/windows v0.2.2 h1:IofanmuvaxnKHuV04sC0eBy/smG6kIKrWG2/jYn2GuM=
+github.com/charmbracelet/x/windows v0.2.2/go.mod h1:/8XtdKZzedat74NQFn0NGlGL4soHB0YQZrETF96h75k=
+github.com/clipperhouse/displaywidth v0.11.0 h1:lBc6kY44VFw+TDx4I8opi/EtL9m20WSEFgwIwO+UVM8=
+github.com/clipperhouse/displaywidth v0.11.0/go.mod h1:bkrFNkf81G8HyVqmKGxsPufD3JhNl3dSqnGhOoSD/o0=
+github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
+github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/lucasb-eyer/go-colorful v1.4.1 h1:1EO+WB73+EH8EVbzlrG3KLAfEypQWVHIBqlTf+2hNss=
+github.com/lucasb-eyer/go-colorful v1.4.1/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/mattn/go-runewidth v0.0.30 h1:+KUuiDA4fF0R1p5FeueHefjDm+GIM+kWfFnDjybOPgk=
+github.com/mattn/go-runewidth v0.0.30/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8=
+github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
+github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
 github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
 github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
 github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/xo/terminfo v1.0.0 h1:2ZpYzqWzyyytjk3TP6aJVDhkMAkc99/1xKQdA3TDTBY=
+github.com/xo/terminfo v1.0.0/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
-golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
+golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
+golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
+golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw=
 golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=