cli: keep stderr to the problem document in JSON mode

CHANGELOG.md

@@ -7,6 +7,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman
 - **Live progress on a terminal.** A spinner with a label and elapsed time covers every wait over about 300 ms (`scrape`, `scrape -`, `batch submit`/`wait`, and any other API call), and `scrape`, `scrape -` and `batch wait` end with a `✓`/`✗` summary line (page, status, credits, duration) on stderr. API errors keep their `ERR::` code and hint in a short block. Everything is on stderr, only when stderr is a terminal, and off with `--json`, `-q`, `NO_COLOR`, `TERM=dumb` or `CI`: stdout and the plain output are unchanged. New dependency: `golang.org/x/term`, for the terminal width.
 - **Retries are safe.** A connection that drops after the request was sent now exits 12 ("may have run and been billed"), not 10 ("nothing was sent, safe to retry"). A gateway 5xx without an API problem document is retried only for GET/HEAD, and is no longer mistaken for the target's status under `--original-status`. A 301/302/303 on a POST is an error naming the Location instead of a silent GET. `Retry-After` waits are capped at 60 s and announced on stderr.
 - **Ctrl-C exits 130** with "interrupted", including while `scrape -` waits on stdin. A mistyped subcommand (`spicrawl batch wiat`) is a usage error with a suggestion, not help and exit 0.
+- **JSON mode keeps stderr clean** (**behaviour change**). With `--json`, or whenever stdout is not a terminal, stderr is now the problem document of a failure and nothing else; it is empty on success. Warnings and status lines ("wrote N bytes", "more results: --cursor ...", the `--engine` deprecation notice, the `ws://` warning) used to print there as prose, sometimes before the problem document, so `cmd --json 2>err.json; jq . err.json` failed. They still print in human mode. The API's `warnings` field is still in the JSON on stdout.
 - **Keys:** a trailing CR/LF or space on the key (flag, env or file) is trimmed. The config file is written via a private temp file, short keys are fully masked, and commands that get the key and base URL from flags work without `$HOME`.
 - **scrape:** `--body null` is a usage error, not a panic (also `batch submit`). Big integers survive `--json` (**key order now follows the API**, as `--jsonl` already did). Stdout write errors stop the run instead of exiting 0. `-o` writes atomically and **a target error page no longer overwrites the file**. `-o -` means stdout. Screenshot labels never overwrite each other. `--body` merges deeply. A BOM on stdin is stripped. Human output no longer HTML-escapes `<>&`. The multi-URL summary counts correctly. `--impersonate` help says it is on by default.
 - **Empty results and `--extract` order** (**behaviour change**, human mode only). `batch list` ("no batch jobs") and `batch results` ("no finished items yet") now print that message on stderr, as `sessions list` and `logs` already did, so stdout is empty and `spicrawl batch list | wc -l` is 0; `-q` silences it. `scrape --extract` (and the file `-o` writes for it) lists the extracted fields in the API's order instead of sorted alphabetically. `--json` and `--jsonl` are unchanged.
@@ -20,7 +21,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman
 
 **Set up from one prompt.** `spicrawl init --all --yes --api-key <key>` then `spicrawl status --json` is enough for an agent to set Spicrawl up with no questions asked; the README has the prompt to paste ("Set up with one prompt").
 
-- **init:** `--all` connects every detected client (conflicts with `--client`). `--api-key` is checked with the API and saved as `login` does, before any client file is written; a rejected key writes nothing. The key is then written into each client's MCP config as is, in the user-level files (`--global` is implied; `--global=false` picks the project files, which are added to the project's `.gitignore` when it has one; `--use-env=true` keeps the `${SPICRAWL_API_KEY}` reference), so the agents work with nothing exported. A key already saved in the config file (by `login` or an earlier `--api-key`) is embedded the same way when `init` runs without `--api-key`, with no new check; a key that is only in `$SPICRAWL_API_KEY` keeps today's `${SPICRAWL_API_KEY}` references. The key is never printed, only masked, and the error output masks it too. After the result list (stdout, unchanged) init prints a report on stderr: `connected:`, `skipped:` with the reason, `api key:` and the `next:` steps. `$SPICRAWL_API_KEY` is used as before and is not checked or saved.
+- **init:** `--all` connects every detected client (conflicts with `--client`). `--api-key` is checked with the API and saved as `login` does, before any client file is written; a rejected key writes nothing. The key is then written into each client's MCP config as is, in the user-level files (`--global` is implied; `--global=false` picks the project files, which are added to the project's `.gitignore` when it has one; `--use-env=true` keeps the `${SPICRAWL_API_KEY}` reference), so the agents work with nothing exported. A key already saved in the config file (by `login` or an earlier `--api-key`) is embedded the same way when `init` runs without `--api-key`, with no new check; a key that is only in `$SPICRAWL_API_KEY` keeps today's `${SPICRAWL_API_KEY}` references. The key is never printed, only masked, and the error output masks it too. After the result list (stdout, unchanged) init prints a report on stderr in human mode (like every message, it is silent in JSON mode, where the list has the `action` and `reason` of each file): `connected:`, `skipped:` with the reason, `api key:` and the `next:` steps. `$SPICRAWL_API_KEY` is used as before and is not checked or saved.
 - **status** (**behaviour change**): it prints a repair command for each failed check (`reachable`, `ready`, `key`, `workers.<kind>`, `mcp`), each marked `unattended_safe: true|false`; `--json` has them in a `repairs` array of `{check, command, unattended_safe, why}`, plus an `mcp` object (`detected`, `connected`, `needs_env`). It checks that each agent client found in this directory or your home directory has a Spicrawl MCP server that can authenticate: an entry that reads `${SPICRAWL_API_KEY}` does not count while that variable is unset in the environment `status` runs in (`mcp.needs_env`, with a repair that rewrites the entry using the saved key), and a project-level entry shadows a user-level one. **It now exits 1 when any check fails** (the API reports not ready, a worker kind has none ready, an MCP server is missing), where a not-ready API used to exit 0. Key and network failures still exit 3 and 10.
 
 ## 0.1.3 — 2026-10-05

DESIGN.md

@@ -16,6 +16,13 @@ never invent a field the API does not accept (the API rejects unknown fields).
    `TERM=dumb` or `CI`. Switched off, the output is the plain text it always was.
    It never prints an engine or browser name; it says what the request asked for
    ("rendering JavaScript"), not which tier the API picked.
+   In JSON mode stderr is stricter still: the problem document of a failure and
+   nothing else, or empty. `output.Printer.Warn` and `Info` (warnings, "wrote N
+   bytes", "more results: --cursor", deprecation notices) write nothing there,
+   so `cmd --json 2>err.json; jq . err.json` always parses. The API's
+   `warnings` field is still in the JSON on stdout. A command that must tell an
+   agent something puts it in its JSON result, not on stderr.
+   `internal/cmd/json_stderr_test.go` runs every golden case against this.
    An empty result is not data either: in human mode an empty list
    (`sessions list`, `logs`, `batch list`, `batch results`) prints nothing on
    stdout and its one-line message ("no sessions") on stderr, so
@@ -53,8 +60,9 @@ never invent a field the API does not accept (the API rejects unknown fields).
    retried POST could be billed twice. Commands that report per-item errors
    use `cmd.ExitCode(err)` to stay in step with the process exit code.
 5. **Errors are the API's problem document.** In JSON mode a failure writes
-   the RFC 7807 body to stderr unchanged; in human mode `error:`, `hint:`,
-   `target status:`, `retryable:` and `request id:` lines.
+   the RFC 7807 body to stderr unchanged, as the only thing on stderr; in
+   human mode `error:`, `hint:`, `target status:`, `retryable:` and
+   `request id:` lines.
 6. **`-` reads stdin.** Commands that take URLs accept `-` for one per line.
 7. **Streams are JSONL.** Anything that yields many records supports
    `--jsonl` (one compact JSON object per line) so output can be piped.
@@ -63,15 +71,16 @@ never invent a field the API does not accept (the API rejects unknown fields).
    stdout, JSON output carries `target_error`, and stderr gets one line.
 9. **Secrets are printed once.** `browser url` writes the URL (which carries
    the key) once on stdout; its connection hints on stderr use
-   `$SPICRAWL_BROWSER_URL` and are suppressed by `-q`. A `ws://` URL warns on
-   stderr, always, that the key travels unencrypted.
+   `$SPICRAWL_BROWSER_URL` and are suppressed by `-q`. In human mode a `ws://`
+   URL warns on stderr, always, that the key travels unencrypted; in JSON mode
+   stderr stays empty and the `url` field shows the scheme.
 10. **Engine routing is internal.** A customer sees three rendering tiers,
     `fetch`, `js` and `browser`, and never an engine or browser name. The API
     already reports tiers. `sessions create` and `browser url` take a tier with
     `--engine`; the engine names older CLIs took (which the API still honours)
-    are sent as typed, with a deprecation notice on stderr that does not repeat
-    them. `scrape` and `batch submit` keep `--engine` and the display flag next
-    to it hidden: the API ignores both and says so (`ENGINE_IGNORED`). In human output any
+    are sent as typed, with a deprecation notice on stderr (human mode only)
+    that does not repeat them. `scrape` and `batch submit` keep `--engine` and
+    the display flag next to it hidden: the API ignores both and says so (`ENGINE_IGNORED`). In human output any
     engine name the API sends is shown as its tier (`output.Tier`,
     `output.PublicText`). JSON and `--jsonl` are the API's data and stay exactly
     as sent. `internal/cmd/engine_names_test.go` locks the help, the human
@@ -106,7 +115,7 @@ cmd/spicrawl/main.go  entry point (go install github.com/OfficialSpicrawl/cli/cm
 internal/api        HTTP client, Problem, DoWithRetry
 internal/config     key/base resolution, config file
 internal/exitcode   exit codes
-internal/output     Printer (JSON/human/JSONL, stderr info)
+internal/output     Printer (JSON/human/JSONL, stderr info and warnings in human mode only)
 internal/ui         stderr spinner, summary line, error block, colour (off unless a terminal)
 internal/cmd        one file per command group; each registers in init()
 ```

README.md

@@ -151,7 +151,7 @@ Set up Spicrawl for yourself. My API key is <YOUR_API_KEY>. Only if your shell r
 
 What the two commands do, with no questions asked:
 
-- `init --all --yes --api-key <key>` connects every detected client (`--all`; it conflicts with `--client`) without confirming (`--yes`). The key is checked with the API and saved to the config file as `spicrawl login` does, before any client file is written; a rejected key exits 3 and writes nothing. Then the key is written into each client's MCP config as is, in your **user-level** files (`~/.claude.json`, `~/.cursor/mcp.json`, ... with mode 0600; `--global` is implied), so the agents authenticate right after a restart with nothing to export. A key already saved in the config file (by `login` or an earlier `--api-key`) is embedded the same way when you run `init` without `--api-key`; a key that is only in `$SPICRAWL_API_KEY` stays a reference. Pass `--use-env=true` to keep the `${SPICRAWL_API_KEY}` reference instead, or `--global=false` to write the project files: those get the same key, a warning, and a line in the project's `.gitignore` when it has one that does not ignore them yet. The key is never printed, only masked (`spicrawl_live_ab12…cd34`), and API errors are masked too. The result list stays on stdout (JSON when it is not a terminal); a short report follows on stderr:
+- `init --all --yes --api-key <key>` connects every detected client (`--all`; it conflicts with `--client`) without confirming (`--yes`). The key is checked with the API and saved to the config file as `spicrawl login` does, before any client file is written; a rejected key exits 3 and writes nothing. Then the key is written into each client's MCP config as is, in your **user-level** files (`~/.claude.json`, `~/.cursor/mcp.json`, ... with mode 0600; `--global` is implied), so the agents authenticate right after a restart with nothing to export. A key already saved in the config file (by `login` or an earlier `--api-key`) is embedded the same way when you run `init` without `--api-key`; a key that is only in `$SPICRAWL_API_KEY` stays a reference. Pass `--use-env=true` to keep the `${SPICRAWL_API_KEY}` reference instead, or `--global=false` to write the project files: those get the same key, a warning, and a line in the project's `.gitignore` when it has one that does not ignore them yet. The key is never printed, only masked (`spicrawl_live_ab12…cd34`), and API errors are masked too. The result list is on stdout (JSON when it is not a terminal). In human mode (a terminal, or `NO_COLOR`/`CI`) a short report follows on stderr; in JSON mode, which is what an agent gets, stderr stays empty and `connected`/`skipped` are the `action` of each `mcp` entry in the list:
 
   ```text
   connected: Claude Code, Cursor

internal/cmd/agents_golden_test.go

@@ -2,6 +2,7 @@ package cmd
 
 import (
 	"net/http"
+	"strings"
 	"testing"
 )
 
@@ -47,14 +48,15 @@ func init() {
 	c11.stderr = true
 
 	// One prompt, no questions: --all --yes --api-key against a fake API that
-	// accepts or rejects the key. The report on stderr is machine-readable here.
+	// accepts or rejects the key. In JSON mode stderr is empty (or the problem
+	// document), so the report a person reads is golden in the human variants.
 	setupClients := func(t *testing.T, e *goldenEnv) {
 		goldenWrite(t, e.Dir, ".claude/settings.json", `{}`)
 		goldenWrite(t, e.Dir, ".cursor/rules/x.mdc", `x`)
 	}
 	oneShot := func(name string, key int, noKey bool, args ...string) goldenCase {
 		return goldenCase{
-			name: name, args: append([]string{"init", "--all", "--yes", "--dir", "{dir}"}, args...), noKey: noKey, stderr: true,
+			name: name, args: append([]string{"init", "--all", "--yes", "--dir", "{dir}"}, args...), noKey: noKey,
 			env: env, setup: setupClients,
 			server: goldenServe(map[string]http.HandlerFunc{
 				"GET /skill.md": func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(gAgentSkill)) },
@@ -89,7 +91,7 @@ func init() {
 	}
 	savedKey.files = []string{"~/.claude.json", "~/.cursor/mcp.json"}
 	rejected := oneShot("agents/init_api_key_rejected_exit3", 401, true, "--api-key", goldenFlagKey)
-	rejected.files = nil
+	rejected.files, rejected.stderr = nil, true
 	noKeyAtAll := oneShot("agents/init_all_yes_no_key", 200, true)
 	noKeyAtAll.files = nil
 	envKey := oneShot("agents/init_all_yes_env_key", 200, false)
@@ -98,6 +100,13 @@ func init() {
 	both.stderr = true
 
 	goldenAdd(allKey, projectKey, useEnv, savedKey, rejected, noKeyAtAll, envKey, both)
+	// What a person reads: the table on stdout and the report (connected,
+	// skipped, key, next steps) on stderr.
+	for _, c := range []goldenCase{allKey, projectKey, useEnv, savedKey, rejected, noKeyAtAll, envKey} {
+		c.name = "agents_human/" + strings.TrimPrefix(c.name, "agents/")
+		c.human, c.stderr, c.files = true, true, nil
+		goldenAdd(c)
+	}
 	goldenAdd(c, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11,
 		agent("agents_human/mcp_print_all", true, nil, nil, "mcp", "install", "--client", "all", "--dir", "{dir}", "--print"),
 		agent("agents_human/mcp_install_all", true, nil, nil, "mcp", "install", "--client", "all", "--dir", "{dir}"),

internal/cmd/agents_test.go

@@ -255,18 +255,23 @@ func TestMCPInstallCodexAppendsToGlobal(t *testing.T) {
 }
 
 func TestMCPInstallClaudeGlobalEmbedsKeyAndWarns(t *testing.T) {
+	// The warning is for a person; JSON mode keeps stderr empty and says
+	// `secret: true` in the result.
+	dirH, _ := agentTestEnv(t)
+	_, humanErr, code := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "claude", "--global", "--dir", dirH)
+	if code != 0 || !strings.Contains(humanErr, "API key in plain text") || strings.Contains(humanErr, ".gitignore") {
+		t.Errorf("human: exit %d, stderr %q", code, humanErr)
+	}
+
 	dir, home := agentTestEnv(t)
 	path := filepath.Join(home, ".claude.json")
 	agentWrite(t, path, `{"numStartups": 3, "projects": {"/x": {}}}`)
 	rs, errOut, code := agentRun(t, "mcp", "install", "--client", "claude", "--global", "--dir", dir)
 	if code != 0 {
 		t.Fatalf("exit %d: %s", code, errOut)
 	}
-	if !rs[0].Secret || !strings.Contains(errOut, "API key in plain text") {
-		t.Errorf("no secret warning: %+v %s", rs, errOut)
-	}
-	if strings.Contains(errOut, ".gitignore") {
-		t.Errorf("gitignore advice for a home file: %s", errOut)
+	if !rs[0].Secret || errOut != "" {
+		t.Errorf("JSON mode: %+v, stderr %q", rs, errOut)
 	}
 	m := agentReadJSON(t, path)
 	if m["numStartups"] != float64(3) {
@@ -280,7 +285,7 @@ func TestMCPInstallClaudeGlobalEmbedsKeyAndWarns(t *testing.T) {
 
 func TestMCPInstallLiteralInProjectSuggestsGitignore(t *testing.T) {
 	dir, _ := agentTestEnv(t)
-	_, errOut, code := agentRun(t, "mcp", "install", "--client", "cursor", "--use-env=false", "--dir", dir)
+	_, errOut, code := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "cursor", "--use-env=false", "--dir", dir)
 	if code != 0 {
 		t.Fatalf("exit %d: %s", code, errOut)
 	}
@@ -378,15 +383,23 @@ func TestSkillInstallUpdatesExistingCopyInPlace(t *testing.T) {
 }
 
 func TestSkillInstallFallsBackToEmbedded(t *testing.T) {
-	dir, _ := agentTestEnv(t)
+	dirH, _ := agentTestEnv(t)
 	srv, _ := newRecordingServer(t, func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(500) })
 	t.Setenv(agentEnvSkillURL, srv.URL)
+	// The notice is for a person; JSON mode keeps stderr empty.
+	_, humanErr, code := runCLIHuman(t, agentTestBase, "skill", "install", "--client", "codex", "--global", "--dir", dirH)
+	if code != 0 || !strings.Contains(humanErr, "built into this CLI") {
+		t.Errorf("human: exit %d, stderr %q", code, humanErr)
+	}
+
+	dir, _ := agentTestEnv(t)
+	t.Setenv(agentEnvSkillURL, srv.URL)
 	rs, errOut, code := agentRun(t, "skill", "install", "--client", "codex", "--global", "--dir", dir)
 	if code != 0 {
 		t.Fatalf("exit %d: %s", code, errOut)
 	}
-	if !strings.Contains(errOut, "built into this CLI") {
-		t.Errorf("stderr = %s", errOut)
+	if errOut != "" {
+		t.Errorf("JSON mode wrote prose to stderr: %q", errOut)
 	}
 	b, _ := os.ReadFile(rs[0].File)
 	if !strings.Contains(string(b), "name: spicrawl\n") || !strings.Contains(string(b), "http://127.0.0.1:1/v1/scrape") {
@@ -838,7 +851,8 @@ func TestMCPInstallSecretTightensExistingFile(t *testing.T) {
 	if err := os.Chmod(path, 0o644); err != nil {
 		t.Fatal(err)
 	}
-	_, errOut, code := agentRun(t, "mcp", "install", "--client", "claude", "--global", "--dir", dir)
+	// A person is told on stderr; JSON mode keeps it empty and says so in the result's reason.
+	_, errOut, code := runCLIHuman(t, agentTestBase, "mcp", "install", "--client", "claude", "--global", "--dir", dir)
 	if code != 0 {
 		t.Fatalf("exit %d: %s", code, errOut)
 	}

internal/cmd/batch_results.go

@@ -207,7 +207,7 @@ var batchContentCmd = &cobra.Command{
 	Short: "Print one finished item's document (GET /v1/batch/{id}/tasks/{seq}/content)",
 	Long: `Fetch a single item's payload by its seq (0-based submission order, "seq" on a
 result line) without paging the whole result set. The API caps it at 512 KiB;
-a cut payload is reported on stderr.
+a cut payload is reported on stderr in human mode.
 
 In JSON mode the payload is printed as the API returned it (a JSON string holding
 the document). In human mode, and always with -o, the document itself is written.