Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman

- **init:** `--all` connects every detected client (conflicts with `--client`). `--api-key` is checked with the API and saved as `login` does, before any client file is written; a rejected key writes nothing. The key is then written into each client's MCP config as is, in the user-level files (`--global` is implied; `--global=false` picks the project files, which are added to the project's `.gitignore` when it has one; `--use-env=true` keeps the `${SPICRAWL_API_KEY}` reference), so the agents work with nothing exported. A key already saved in the config file (by `login` or an earlier `--api-key`) is embedded the same way when `init` runs without `--api-key`, with no new check; a key that is only in `$SPICRAWL_API_KEY` keeps today's `${SPICRAWL_API_KEY}` references. The key is never printed, only masked, and the error output masks it too. After the result list (stdout, unchanged) init prints a report on stderr in human mode (like every message, it is silent in JSON mode, where the list has the `action` and `reason` of each file): `connected:`, `skipped:` with the reason, `api key:` and the `next:` steps. `$SPICRAWL_API_KEY` is used as before and is not checked or saved.
- **status** (**behaviour change**): it prints a repair command for each failed check (`reachable`, `ready`, `key`, `workers.<kind>`, `mcp`), each marked `unattended_safe: true|false`; `--json` has them in a `repairs` array of `{check, command, unattended_safe, why}`, plus an `mcp` object (`detected`, `connected`, `needs_env`). It checks that each agent client found in this directory or your home directory has a Spicrawl MCP server that can authenticate: an entry that reads `${SPICRAWL_API_KEY}` does not count while that variable is unset in the environment `status` runs in (`mcp.needs_env`, with a repair that rewrites the entry using the saved key), and a project-level entry shadows a user-level one. **It now exits 1 when any check fails** (the API reports not ready, a worker kind has none ready, an MCP server is missing), where a not-ready API used to exit 0. Key and network failures still exit 3 and 10.
- **init, fixes from QA.** (1) The plan is made and the key checked before anything is saved: a plan that is refused (a customised `spicrawl` entry without `--force`) saves and writes nothing, where it used to leave the new key saved. The key is saved after the changes are confirmed (or with `--yes`) and applied; the plan lists it as `save key`. (2) An existing `spicrawl` entry that differs from the one to write only in its `Authorization: Bearer ...` token (same URL and shape, in `.mcp.json`, `.cursor/mcp.json`, VS Code's `mcp.json` and Codex's `config.toml`) is replaced without `--force`, so a new `--api-key` or a switch from a reference to the key works on a re-run; the old file is kept as `.bak`. (3) With `--global=false`, the `.bak` copy of each project file that holds the key is added to `.gitignore` as well (it can hold the previous key), and a key file that git already tracks is flagged: a warning for a person, and `tracked by git: git rm --cached <file>` in the result's `reason`. `git` is only asked when it is installed.
- **status, fixes from QA.** An MCP entry with a key written into it that is not the key in use (compared by fingerprint, neither printed) is reported as `mcp.stale` and counts as a failed `mcp` check. The repair for `needs_env` and `stale` entries is now `init --client <ids> --yes` (plus `--global=false` for a project entry), without `--force`.
- **README, npm launcher.** The one-prompt text names the command that proves it works: `run npx -y @spicrawl/cli@latest scrape https://news.ycombinator.com`. The `status` text says which output has the repairs where. `npm/bin/spicrawl.js` read `SPICRAWL_BINARY` twice; once now.

## 0.1.3 — 2026-10-05

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ For other clients (Gemini CLI, OpenCode, Factory Droid, Devin and more) use the
Paste this into a coding agent that runs shell commands on your computer (Claude Code, Cursor, Codex, VS Code with Copilot agent mode), and put your key in place of `<YOUR_API_KEY>`. Create a key at [app.spicrawl.com/dashboard/keys](https://app.spicrawl.com/dashboard/keys).

```text
Set up Spicrawl for yourself. My API key is <YOUR_API_KEY>. Only if your shell runs on my computer, run: npx -y @spicrawl/cli@latest init --all --yes --api-key <YOUR_API_KEY> — it finds every coding agent installed here, adds the Spicrawl MCP server and installs the Spicrawl skill. Check it: npx -y @spicrawl/cli@latest status --json. If a check failed, run the repairs it lists in order; repairs with unattended_safe: false are mine, so show me the command instead of running it. Prove it works: scrape https://news.ycombinator.com with Spicrawl and tell me today's top story. Report which agents connected and anything you could not finish. Never repeat my API key back to me. If you have no shell on my computer (a chat app or cloud sandbox), stop and send me to https://docs.spicrawl.com/agents/quickstart instead. New MCP tools appear only in a session started after the install.
Set up Spicrawl for yourself. My API key is <YOUR_API_KEY>. Only if your shell runs on my computer, run: npx -y @spicrawl/cli@latest init --all --yes --api-key <YOUR_API_KEY> — it finds every coding agent installed here, adds the Spicrawl MCP server and installs the Spicrawl skill. Check it: npx -y @spicrawl/cli@latest status --json. If a check failed, run the repairs it lists in order; repairs with unattended_safe: false are mine, so show me the command instead of running it. Prove it works: run npx -y @spicrawl/cli@latest scrape https://news.ycombinator.com and tell me today's top story. Report which agents connected and anything you could not finish. Never repeat my API key back to me. If you have no shell on my computer (a chat app or cloud sandbox), stop and send me to https://docs.spicrawl.com/agents/quickstart instead. New MCP tools appear only in a session started after the install.
```

What the two commands do, with no questions asked:
Expand All @@ -171,7 +171,7 @@ What the two commands do, with no questions asked:
]
```

`unattended_safe: false` means the user must act (supply a key) or decide; the command holds a placeholder, never a key. The checks are `reachable`, `ready`, `key`, `workers.<kind>` and `mcp`: each agent client found in this directory or your home directory needs a Spicrawl MCP server that can authenticate (skipped when no client is found). A server whose key is `${SPICRAWL_API_KEY}` does not count while that variable is unset where `status` runs (`mcp.needs_env`); its repair rewrites the entry with the saved key, or asks for a key if none is saved. A project-level entry shadows a user-level one. Without `--json`, `status` prints the same repairs as `check:`, `command:`, `unattended_safe:` and `why:` lines.
`unattended_safe: false` means the user must act (supply a key) or decide; the command holds a placeholder, never a key. The checks are `reachable`, `ready`, `key`, `workers.<kind>` and `mcp`: each agent client found in this directory or your home directory needs a Spicrawl MCP server that can authenticate (skipped when no client is found). A server whose key is `${SPICRAWL_API_KEY}` does not count while that variable is unset where `status` runs (`mcp.needs_env`), and neither does one with another key written into it than the one in use (`mcp.stale`, compared by fingerprint; neither key is printed). The repair, `init --client <ids> --yes`, rewrites the entry with the saved key, or asks for a key if none is saved. A project-level entry shadows a user-level one. On a terminal, `status` draws a checklist with the repairs under it; with `NO_COLOR`, `TERM=dumb` or `CI` set it prints them as `check:`, `command:`, `unattended_safe:` and `why:` lines. Piped (as an agent runs it), it prints the JSON above.

The key is passed as an argument, so other users on the machine can see it in the process list while the command runs, and it ends up in plain text in the client configs. To avoid both, export `SPICRAWL_API_KEY` and drop `--api-key`: as long as no key is saved in the config file, `init` then writes `$SPICRAWL_API_KEY` references and does not check or save the key (run `spicrawl login` to save it, after which `init` embeds it). Claude Code rewrites `~/.claude.json` itself, so if a running session overwrites the entry, `status` reports the `mcp` check as failed and its repair writes it again (with the saved key, as a literal).

Expand Down
44 changes: 41 additions & 3 deletions internal/cmd/agents.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,18 @@ package cmd

import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/url"
"os"
"os/exec"
"path/filepath"
"reflect"
"strings"
"time"

"github.com/OfficialSpicrawl/cli/internal/config"
"github.com/OfficialSpicrawl/cli/internal/output"
Expand Down Expand Up @@ -521,14 +524,49 @@ func agentSecretGuidance(p *output.Printer, rs []*agentResult, s agentScope) {
} else {
p.Warn("%s now contains your API key in plain text; its permissions are 0600 (owner only)", r.File)
}
if rel, err := filepath.Rel(s.Dir, r.File); err == nil && !strings.HasPrefix(rel, "..") {
if ignored, _ := os.ReadFile(filepath.Join(s.Dir, ".gitignore")); !gitignoreHas(ignored, filepath.ToSlash(rel)) {
p.Warn("keep it out of version control: echo %s >> .gitignore", filepath.ToSlash(rel))
// A project file also leaves a .bak that can hold the previous key.
names := []string{agentProjectRel(s, r.File)}
if r.Backup != "" {
names = append(names, agentProjectRel(s, r.Backup))
}
ignored, _ := os.ReadFile(filepath.Join(s.Dir, ".gitignore"))
for _, rel := range names {
if rel == "" {
continue
}
if !gitignoreHas(ignored, rel) {
p.Warn("keep it out of version control: echo %s >> .gitignore", rel)
}
if gitTracked(s.Dir, rel) {
p.Warn("%s is tracked by git, so the key in it will be committed: run git rm --cached %s, and treat the key as exposed if it was ever committed", rel, rel)
}
}
}
}

// agentProjectRel is file relative to the project directory, slash-separated,
// or "" when it is outside it (a user-level file).
func agentProjectRel(s agentScope, file string) string {
rel, err := filepath.Rel(s.Dir, file)
if err != nil || strings.HasPrefix(rel, "..") {
return ""
}
return filepath.ToSlash(rel)
}

// gitTracked reports whether git tracks rel in dir. It asks git when git is
// installed and dir is a repository, and says no otherwise: a best-effort
// warning, not a dependency.
func gitTracked(dir, rel string) bool {
git, err := exec.LookPath("git")
if err != nil {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return exec.CommandContext(ctx, git, "-C", dir, "ls-files", "--error-unmatch", "--", rel).Run() == nil
}

func agentDisplayPath(p string, s agentScope) string {
if rel, err := filepath.Rel(s.Dir, p); err == nil && !strings.HasPrefix(rel, "..") {
return rel
Expand Down
33 changes: 23 additions & 10 deletions internal/cmd/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -146,30 +146,43 @@ func authRunLogin(cmd *cobra.Command, _ []string) error {
return err
}

// authSaveKey checks key against the API and saves it to the config file (and
// --base-url, when given) exactly as "login" does; "init --api-key" shares it.
// Nothing is saved when the check fails. It returns the settings the key was
// checked against.
func authSaveKey(p *output.Printer, key string) (config.Resolved, error) {
// authCheck resolves the settings for key and checks the key against the API.
// Nothing is saved.
func authCheck(p *output.Printer, key string) (config.Resolved, error) {
r, err := config.Resolve(key, flagBaseURL)
if err != nil {
return r, err
}
sp := spin(p, "Checking the key with %s", r.BaseURL)
err = authCheckKey(rootCmd.Context(), r.BaseURL, key)
sp.Stop()
if err != nil {
return r, err
}
return r, err
}

// authStore saves key to the config file, and --base-url when given, as
// "login" does.
func authStore(key string) error {
f, err := config.Load()
if err != nil {
return r, err
return err
}
f.APIKey = key
if flagBaseURL != "" {
f.BaseURL = strings.TrimRight(flagBaseURL, "/")
}
return r, config.Save(f)
return config.Save(f)
}

// authSaveKey checks key against the API and saves it, exactly as "login" does.
// Nothing is saved when the check fails. It returns the settings the key was
// checked against. "init --api-key" checks first and saves only after its plan
// is confirmed and applied.
func authSaveKey(p *output.Printer, key string) (config.Resolved, error) {
r, err := authCheck(p, key)
if err != nil {
return r, err
}
return r, authStore(key)
}

// authPromptKey asks for the key on the terminal. Without golang.org/x/term
Expand Down
33 changes: 24 additions & 9 deletions internal/cmd/initcmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -139,12 +139,17 @@ func runInit(cmd *cobra.Command, _ []string) error {
}
keyLine += ")"
}
var saveKey func() error // set when --api-key is to be saved once the changes are applied
saveNote := "" // the same, for the plan
if keySource == "flag" {
// Check it and save it before writing any client file: a bad key must not leave a half-done setup.
r, err := authSaveKey(p, key)
// Check it now, so a rejected key stops the run before anything is planned
// or written; it is saved only after the plan was confirmed and applied.
r, err := authCheck(p, key)
if err != nil {
return err
}
saveKey = func() error { return authStore(key) }
saveNote = config.MaskKey(key) + " to " + r.Path
keyLine = config.MaskKey(key) + " (checked, saved to " + r.Path
if embed {
keyLine += "; written into the client configs"
Expand Down Expand Up @@ -190,7 +195,7 @@ func runInit(cmd *cobra.Command, _ []string) error {
if pending > 0 && !initFlags.yes {
// JSON mode keeps stderr for the problem document, unless a person is about to be asked.
if !p.JSON || initStdinIsTerminal() {
initPrintPlan(p.Err, errUI(p), rs, s)
initPrintPlan(p.Err, errUI(p), rs, s, saveNote)
}
if !initStdinIsTerminal() {
if p.JSON {
Expand All @@ -216,6 +221,11 @@ func runInit(cmd *cobra.Command, _ []string) error {
if err := agentApply(rs); err != nil {
return err
}
if saveKey != nil {
if err := saveKey(); err != nil {
return err
}
}
if err := agentReport(p, rs, s); err != nil {
return err
}
Expand Down Expand Up @@ -304,13 +314,15 @@ func initPlanGitignore(s agentScope, rs []*agentResult) (*agentResult, error) {
if r.Kind != "mcp" || !r.Secret || r.Action == agentActionSkipped {
continue
}
rel, err := filepath.Rel(s.Dir, r.File)
if err != nil || strings.HasPrefix(rel, "..") {
rel := agentProjectRel(s, r.File)
if rel == "" {
continue
}
rel = filepath.ToSlash(rel)
if !gitignoreHas(old, rel) && !slices.Contains(add, rel) {
add = append(add, rel)
// The .bak copy of a config that held a key holds it too.
for _, name := range []string{rel, rel + ".bak"} {
if !gitignoreHas(old, name) && !slices.Contains(add, name) {
add = append(add, name)
}
}
}
if len(add) == 0 {
Expand All @@ -336,7 +348,7 @@ func gitignoreHas(content []byte, rel string) bool {
return false
}

func initPrintPlan(w io.Writer, u *ui.UI, rs []*agentResult, s agentScope) {
func initPrintPlan(w io.Writer, u *ui.UI, rs []*agentResult, s agentScope, saveNote string) {
fmt.Fprintln(w, u.Bold("plan:"))
for _, r := range rs {
verb := map[string]string{
Expand All @@ -357,6 +369,9 @@ func initPrintPlan(w io.Writer, u *ui.UI, rs []*agentResult, s agentScope) {
}
fmt.Fprintln(w, line)
}
if saveNote != "" {
fmt.Fprintf(w, " %s %-7s %-6s %s\n", u.Green("save "), "", "key", saveNote)
}
}

func initConfirm(w io.Writer, r io.Reader) (bool, error) {
Expand Down
Loading
Loading