diff --git a/.agents/skills/mcp-qa/SKILL.md b/.agents/skills/mcp-qa/SKILL.md index b255701b9c..7ccd5d7bc9 100644 --- a/.agents/skills/mcp-qa/SKILL.md +++ b/.agents/skills/mcp-qa/SKILL.md @@ -71,7 +71,7 @@ dispatch without changing real Sentry data. For output-format changes, also inspect the raw MCP tool result when possible, not only the LLM's final answer. The final answer can add model-specific text that is not part of the tool response. Review raw tool output against -`docs/contributing/tool-responses.md`: it should be user-facing, structured, +`design/mcp-tool-responses.md` and `docs/contributing/tool-responses.md`: it should be user-facing, structured, and free of raw API JSON, internal implementation IDs, empty placeholders, and unrelated instructions. diff --git a/.agents/skills/verify-toolkit/SKILL.md b/.agents/skills/verify-toolkit/SKILL.md new file mode 100644 index 0000000000..940c96abae --- /dev/null +++ b/.agents/skills/verify-toolkit/SKILL.md @@ -0,0 +1,127 @@ +--- +name: verify-toolkit +description: Prove Sentry MCP and Sentry CLI behavior in this repo by running the real stdio server and the `sentry` CLI from source, offline, and capturing MCP and terminal evidence. Use to verify a change to MCP tools, tool responses, skills, scopes, stdio auth, or CLI auth, or to check a claim from `map/`. For QA against a live Sentry org or a real agent client, use the `mcp-qa` skill instead. +--- + +# Verify Toolkit + +The primary surface is the Sentry MCP server over stdio, driven as an MCP +client drives it: `initialize`, `tools/list`, `tools/call`. The secondary +surface is the `sentry` CLI in a terminal. Two servers are available: + +- **mock**: the eval server (`packages/mcp-server-evals/src/bin/start-mock-stdio.ts`). + Real stdio transport and tool handlers, every skill, experimental mode, and + the Sentry API answered by the MSW fixtures in `packages/mcp-server-mocks`. + Use it for tool calls and responses. +- **cli**: `sentry mcp` from CLI source, with a fake token, an empty config + dir, and an unreachable Sentry host (`http://127.0.0.1:9`). Use it for what + the real entry point lists: skills, scopes, auth, and server metadata. Its + tool calls fail, because no Sentry answers. + +Nothing here reaches a live Sentry org. Live data, real logins, real agent +clients, and the Cloudflare `/mcp` server are `mcp-qa` territory +(`.agents/skills/mcp-qa/SKILL.md`). + +## Launch + +From the repo root, once per checkout or after changing MCP or CLI code: + +```bash +pnpm install --frozen-lockfile +env -u SENTRY_AUTH_TOKEN pnpm run build +(cd packages/cli && node script/ensure-generated.mjs) +``` + +Ready means: `pnpm run build` exits 0, and `ensure-generated` exits 0 and +leaves `packages/cli/src/generated/command-api.json` as valid JSON. Each +drive starts its own server process and stops it when it finishes, so there +is no long-running server and nothing to tear down. Runs are isolated: each +gets its own temporary home and CLI config dir, and two runs can go side by side. + +Set a run directory for evidence, outside the checkout: + +```bash +export VERIFY_RUN="${TMPDIR:-/tmp}/verify-toolkit/$(date +%Y%m%d-%H%M%S)" +``` + +## Doctor + +```bash +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/doctor-mock" +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/doctor-cli" +``` + +Both print `connected: Sentry MCP (...), N tools` and exit 0. If either +fails, read `server-stderr.txt` in its directory, then rerun Launch. + +## Drive + +`scripts/mcp-call.mjs` starts a server, lists tools, runs each `--call`, and +writes `initialize.json`, `tools-list.json`, `call--.json` (request +and raw `CallToolResult`), and `server-stderr.txt` to `--out`. Listing first +makes the MCP SDK validate every structured result against the tool's +`outputSchema`, as a real client does; a mismatch fails the run. + +```bash +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out DIR \ + [--server mock|cli] [--no-token] [--server-arg ARG]... \ + [--call TOOL 'JSON_ARGS']... +``` + +- `--server-arg` passes one argument to `sentry mcp`, such as + `--server-arg --skills --server-arg inspect` or `--server-arg acme/`. +- `--no-token` starts `sentry mcp` with no token and no login. +- Mock fixtures use the org `sentry-mcp-evals`, project `cloudflare-mcp`, + issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`. + +`scripts/check-results.mjs` checks evidence against map claims, prints one +`PASS` or `FAIL` line per claim, appends them to `checks.txt` in the same +directory, and exits 1 on any failure: + +```bash +node .agents/skills/verify-toolkit/scripts/check-results.mjs responses DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs surface DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs subset WIDE_DIR DIR +``` + +`scripts/sentry-offline.mjs` runs one CLI command with the same offline env, +and writes `command.txt`, `stdout.txt`, `stderr.txt`, and `exit-code.txt`: + +```bash +node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out DIR [--token] -- auth status +``` + +`--token` sets `SENTRY_AUTH_TOKEN` to a fake token. The script prints +`PRINTED THE TOKEN` if that token appears in the output. + +## Evidence + +Everything goes under `$VERIFY_RUN`, one directory per drive, named for the +claim it proves. Proof standards: + +- Drive the real entry point: MCP JSON-RPC over stdio, or the `sentry` + command. Never call handlers or internal helpers. +- Keep the raw MCP result, not only an agent's answer. +- Keep the action and the result: the request sits next to the result in + each `call-*.json`, and the command next to its output for the CLI. +- The only mock is the Sentry API, an external HTTP boundary. Do not mock + the server, the transport, or the CLI. +- Offline `sentry mcp` makes network attempts, such as a scope check at + startup. They go to `127.0.0.1:9` and fail; check `server-stderr.txt` and + do not assume a skipped network call. +- Cite the claim ID (`#`) with each artifact you report. + +## Cleanup + +Each script stops the process it started and removes its temporary config +dir. If you stop a script early, kill only the PID of the server it started. +Never kill by process name: a developer's own `sentry mcp` may be running. +Do not delete `$VERIFY_RUN`; it is the evidence. + +## Map + +`map/README.md` lists the map files and the entry contract. Cite a claim as +`#`, such as `mcp-tool-responses#structured-text-match`. The goals +behind the claims are in `design/`. diff --git a/.agents/skills/verify-toolkit/map/README.md b/.agents/skills/verify-toolkit/map/README.md new file mode 100644 index 0000000000..c094c63586 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/README.md @@ -0,0 +1,59 @@ +# Toolkit verification map + +This directory is the maintained source for verifying Sentry MCP and Sentry +CLI behavior. Read this index, then use the matching map file as the recipe. +The goals and rules that these claims prove are in `design/`. + +## Baseline preconditions + +- Launch from `../SKILL.md` has run on this checkout since the last change to + MCP or CLI code. +- `VERIFY_RUN` names a fresh directory outside the checkout. +- The Doctor commands in `../SKILL.md` pass. +- Never drive a server, login, or config dir that this run did not create. + +## Driving conventions + +- Run every command from the repo root. +- Treat every command as literal. Keep tool names, arguments, and flags. +- `mcp-call.mjs` is the MCP client; `sentry-offline.mjs` is the terminal. +- One `--out` directory per bullet, under `$VERIFY_RUN`. + +## Proof and skip reporting + +- MCP proof is the `call-*.json` or `tools-list.json` file, plus the + `checks.txt` line for the claim. +- CLI proof is `command.txt`, `stdout.txt`, `stderr.txt`, and + `exit-code.txt`. +- Record the claim ID and the entry point with every artifact. +- Report an unreachable path with the command you ran and the unmet + precondition. Do not report a claim as verified through a different entry + point. + +## Claim IDs + +- A claim ID is lowercase kebab-case and unique in its file. Cite it as + `#`, where `` is the file name without `.md`. +- Tests, evals, and PRs cite claims by this ID. +- Do not rename or reuse an ID. When a behavior changes meaning, retire its + ID and add a new one. Search the repo for citations first. + +## Map entry contract + +Each map file starts with an H1 title and one paragraph. Then it has exactly +five H2 sections, in this order: `Claims`, `How to get to it`, +`Driving it with ` (starting with `Preconditions:`), `Gotchas`, and +`Gaps` (`None.` when empty). Each claim names one proof: `drive`, `test` with +the exact command, or `eval` with the exact command. Keep source structure +out of the map. + +## Map files + +- `mcp-tool-responses.md` covers what a tool result holds: structured and + markdown results, suggested actions, side quests, and tool errors. +- `mcp-tool-surface.md` covers what `tools/list` and `initialize` give a + client: every callable tool, safety hints, titles, one page, and + instructions. +- `mcp-access.md` covers who can start `sentry mcp` and what a session + reaches: login, skills, and org scope. +- `cli.md` covers CLI behavior with no login and with a token. diff --git a/.agents/skills/verify-toolkit/map/cli.md b/.agents/skills/verify-toolkit/map/cli.md new file mode 100644 index 0000000000..8962de92a1 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/cli.md @@ -0,0 +1,35 @@ +# Sentry CLI + +How the `sentry` command behaves in a terminal with no login and with an +environment token: commands that need a login stop with one clear message, +and `auth status` never prints the token in full. The goals are in +`design/sentry-cli.md`. + +## Claims + +- `no-login-exit` A command that needs a login, with none, exits 10 and says to run `sentry auth login`. Proof: `drive`. +- `token-masked` `sentry auth status` with `SENTRY_AUTH_TOKEN` reports the env token as active and never prints it in full. Proof: `drive`. + +## How to get to it + +- `sentry ` in a terminal, here from CLI source through `sentry-offline.mjs`. +- `SENTRY_AUTH_TOKEN` in the environment, or a login from `sentry auth login`. + +## Driving it with sentry-offline.mjs + +Preconditions: + +- Launch from `../SKILL.md` passes, and `$VERIFY_RUN` is set. + +- **`no-login-exit`: list issues with no login.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-no-login" -- issue list acme/`. It prints `exit 10`. `stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.` +- **`token-masked`: check status with a token.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-token" --token -- auth status`. It prints `exit 0` and no `PRINTED THE TOKEN`. `stdout.txt` contains `Authenticated via SENTRY_AUTH_TOKEN`, and its `Token` row shows a shortened token, such as `sntryu_v...line`. + +## Gotchas + +- The first CLI run on a fresh checkout generates files and prints that work. Run Launch's `ensure-generated` step first, or the output mixes with the proof. +- A short token prints as `****` and a long one as its first 8 and last 4 characters. Both pass; only the full token fails. +- `auth status` also reports `Could not verify credentials` offline. That line is expected and is not part of the claim. + +## Gaps + +- `json-output` List and view commands print JSON with `--json`. Proving it needs a Sentry host that answers. diff --git a/.agents/skills/verify-toolkit/map/mcp-access.md b/.agents/skills/verify-toolkit/map/mcp-access.md new file mode 100644 index 0000000000..e441183086 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-access.md @@ -0,0 +1,41 @@ +# MCP access + +Who can start `sentry mcp`, and what a session reaches: it refuses to start +with no login, skills narrow the tool list, `inspect` changes no Sentry data, +and an org scope removes the org choice from every tool. The goals are in +`design/mcp-access.md`. + +## Claims + +- `no-login-exits` With no token and no login, `sentry mcp` exits before the MCP handshake and says to run `sentry auth login`. Proof: `drive`. +- `skills-narrow` `--skills inspect` lists a strict subset of the default session's tools. Proof: `drive`. +- `inspect-read-only` An `inspect` session lists no tool that changes data, except feedback and onboarding status. Proof: `drive`. +- `scope-hides-org` An `/` scope lists no `find_organizations`, and no tool takes `organizationSlug`. Proof: `drive`. +- `remote-scoped-url` A remote token minted for one org's URL gets 403 on another org's URL. Proof: `test` with `pnpm --filter @sentry/mcp-cloudflare exec vitest run src/server/lib/mcp-handler.test.ts -t "returns 403 when the token is org-scoped but the MCP URL uses a different organization"`. + +## How to get to it + +- `sentry mcp`, with `SENTRY_AUTH_TOKEN` or a CLI login. +- `--skills`, `--disable-skills`, `--all-skills`, `MCP_SKILLS`, and `MCP_DISABLE_SKILLS`. +- An `/` or `/` target, or `--org` and `--project`. +- Remote HTTP at `/mcp`, `/mcp/`, and `/mcp//` with OAuth. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. + +- **`no-login-exits`: start with nothing.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --no-token --out "$VERIFY_RUN/access-no-login"`. It exits 1 with `mcp-call failed: MCP error -32000: Connection closed`, and `server-stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.` +- **`skills-narrow`, `inspect-read-only`: compare skills.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/access-default"` and `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg --skills --server-arg inspect --out "$VERIFY_RUN/access-inspect"`. Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs subset "$VERIFY_RUN/access-default" "$VERIFY_RUN/access-inspect"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only "$VERIFY_RUN/access-inspect"`. Both print `PASS`, and the subset line names removed tools such as `update_issue`. +- **`scope-hides-org`: scope to an org.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg acme/ --out "$VERIFY_RUN/access-scoped"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped "$VERIFY_RUN/access-scoped"`. It prints `PASS`. + +## Gotchas + +- The scope check at startup cannot reach the offline host, so `sentry mcp acme/` starts as if the org exists. That is the documented behavior for a network failure, not proof that `acme` exists. +- `--no-token` also needs the empty home and config dir that the script makes: the CLI reads a token from `~/.sentryclirc`. The script also sets the token variables to empty, because `pnpm run cli` loads `packages/cli/.env.local` for any variable that is not set. Never point `HOME` or `SENTRY_CONFIG_DIR` at a real login for this drive. + +## Gaps + +- `scope-not-found-exits` A scope that Sentry answers 404 for stops `sentry mcp` with exit code 23 and names the missing part. The offline drive has no Sentry host that answers. +- `token-host-scoped` An org auth token is sent only to the host it was issued for. The offline drive has no second host to send it to. diff --git a/.agents/skills/verify-toolkit/map/mcp-tool-responses.md b/.agents/skills/verify-toolkit/map/mcp-tool-responses.md new file mode 100644 index 0000000000..036fc9e6a3 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-tool-responses.md @@ -0,0 +1,57 @@ +# MCP tool responses + +What a `tools/call` result gives an agent: structured content that is the +answer, markdown for tools without an output schema, follow-up calls the +session can make, side quests that name a fix, and tool errors that are not +dressed up as success. The goals are in `design/mcp-tool-responses.md`. + +## Claims + +- `structured-text-match` A tool with an output schema returns structured content, and its text is that payload as compact JSON. Proof: `drive`. +- `schema-valid` Every structured result matches the tool's advertised output schema. Proof: `drive` (the MCP client rejects a mismatch). +- `suggested-actions-callable` Suggested actions are at most three, each a complete `tool_call` with arguments, naming a tool in this session's `tools/list`. Proof: `drive`. +- `side-quest-fix` Each side quest has a tip that names the fix and an https docs link. Proof: `drive`. +- `markdown-only` A tool with no output schema returns markdown with a `#` title and no structured content. Proof: `drive`. +- `no-placeholders` Markdown results contain no `undefined`, `[object Object]`, or `NaN`. Proof: `drive`. +- `tool-error` A call with missing input returns `isError` with an input error message and no structured content. Proof: `drive`. +- `schema-guard` Every handler result in the mcp-core suite matches its output schema. Proof: `test` with `pnpm --filter @sentry/mcp-core exec vitest run src/server.metadata.test.ts -t "validates handler results against outputSchema in every test"`. + +## How to get to it + +- `tools/call` from any MCP client over stdio, here through `mcp-call.mjs` against the mock server. +- The same result reaches remote HTTP clients; this map does not drive that transport. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. +- The mock server's fixtures include org `sentry-mcp-evals`, issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`. + +- **`structured-text-match`, `schema-valid`, `suggested-actions-callable`, `side-quest-fix`, `markdown-only`, `no-placeholders`, `tool-error`: call four tools.** Run: + + ```bash + node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/mcp-tool-responses" \ + --call get_issue_details '{"organizationSlug":"sentry-mcp-evals","issueId":"CLOUDFLARE-MCP-41"}' \ + --call find_projects '{"organizationSlug":"sentry-mcp-evals"}' \ + --call get_trace_details '{"organizationSlug":"sentry-mcp-evals","traceId":"a4d1aae7216b47ff8117cf4e09ce9d0a"}' \ + --call get_issue_details '{}' + ``` + + It exits 0 (so the SDK accepted both structured results against their schemas, `schema-valid`) and prints `ok, structuredContent` for calls 1 and 2, `ok, content only` for call 3, and `isError, content only` for call 4. + +- **Same claims: check the evidence.** Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs responses "$VERIFY_RUN/mcp-tool-responses"`. Every line is `PASS`: `structured-text-match` and `suggested-actions-callable` for calls 1 and 2, `side-quest-fix` for call 1, `markdown-only` and `no-placeholders` for call 3, and `tool-error` for call 4. +- **`tool-error`: read the message.** `call-4-get_issue_details.json` has `"isError": true`, and its text starts with `**Input Error**` and names `issueId` or `issueUrl`. + +## Gotchas + +- The mock server grants every skill and runs in experimental mode, so its `tools/list` is not what a user's session lists. Use `mcp-tool-surface.md` and `mcp-access.md` for that. +- A fixture the mock server lacks becomes an HTTP 500 tool error whose text includes the mock's own stack trace. That is a fixture gap, not a product result: use the fixture IDs above. +- `find_projects` ignores its `query` against the mock; do not use it to prove filtering. +- `check-results.mjs` appends to `checks.txt`. Use a fresh `--out` per run. + +## Gaps + +- `untrusted-contained` Upstream text cannot add headings, list items, or links to a markdown result. No mock fixture carries markdown in an upstream value. +- `feedback-only-for-bugs` A feedback suggestion appears only for likely Sentry MCP bugs. The mock server registers no feedback tool, because it has no DSN, so no result can carry the suggestion. +- `optional-detail-fails` A failed optional detail, such as a suspect commit, does not stop the main result. No fixture makes only the optional request fail. diff --git a/.agents/skills/verify-toolkit/map/mcp-tool-surface.md b/.agents/skills/verify-toolkit/map/mcp-tool-surface.md new file mode 100644 index 0000000000..45ca41ab41 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-tool-surface.md @@ -0,0 +1,39 @@ +# MCP tool surface + +What a client gets from `initialize` and `tools/list` on the real `sentry mcp` +entry point: every tool the session can call, listed directly, each with +safety hints and a unique title, in one page, with short server +instructions. The goals are in `design/mcp-tools.md`. + +## Claims + +- `lists-all` A default session lists its tools directly, with no `search_sentry_tools` or `execute_sentry_tool`. Proof: `drive`. +- `hints-set` Every listed tool sets `readOnlyHint`, `destructiveHint`, and `openWorldHint` to a boolean. Proof: `drive`. +- `unique-titles` Every listed tool has a title, and no two share one. Proof: `drive`. +- `one-page` `tools/list` returns every tool with no `nextCursor`. Proof: `drive`. +- `instructions-short` Server instructions are at most 1,500 characters. Proof: `drive`. +- `metadata-lint` Server identity, tool names, hints, titles, description lengths, and `_meta` keys pass the metadata linter in stable and experimental mode. Proof: `test` with `pnpm --filter @sentry/mcp-core exec vitest run src/server.metadata.test.ts`. + +## How to get to it + +- `sentry mcp` over stdio, from an MCP client's server config. +- `npx @sentry/mcp-server`, which runs the same command. +- Remote HTTP at `https://mcp.sentry.dev/mcp`; this map does not drive it. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. + +- **`lists-all`, `hints-set`, `unique-titles`, `one-page`, `instructions-short`: list tools.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/mcp-tool-surface"`. It prints `connected: Sentry MCP (cli), N tools` with N above 25. Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs surface "$VERIFY_RUN/mcp-tool-surface"`. Every line is `PASS`. +- **`instructions-short`: read them.** `initialize.json` has `instructions` that name only tools present in `tools-list.json`. + +## Gotchas + +- The offline host is self-hosted from the CLI's view, so the default session leaves out `seer` tools such as `analyze_issue_with_seer`. Pass `--server-arg --all-skills` to list them. +- The mock server's list is not the user's list: it grants every skill in experimental mode. Drive `--server cli` for this file. + +## Gaps + +- `gateway-cap` The fallback gateway surface lists at most 25 tools. It is a code constant that is off, so no entry point can turn it on. diff --git a/.agents/skills/verify-toolkit/scripts/check-results.mjs b/.agents/skills/verify-toolkit/scripts/check-results.mjs new file mode 100755 index 0000000000..8189e55f43 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/check-results.mjs @@ -0,0 +1,202 @@ +#!/usr/bin/env node +// Check mcp-call.mjs evidence against map claims. Prints one PASS/FAIL line +// per claim, appends them to /checks.txt, and exits 1 on any FAIL. +// check-results.mjs responses DIR mcp-tool-responses claims, per call +// check-results.mjs surface DIR mcp-tool-surface claims +// check-results.mjs read-only DIR mcp-access#inspect-read-only +// check-results.mjs scoped DIR mcp-access#scope-hides-org +// check-results.mjs subset WIDE DIR mcp-access#skills-narrow (DIR lists fewer tools) +import { appendFileSync, readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; + +/** Tools every skill keeps, though they send data to Sentry (design/mcp-access.md). */ +const EVERY_SKILL_WRITES = new Set([ + "submit_feedback_to_sentry_team", + "onboarding_status_update", +]); +const GATEWAY_TOOLS = ["search_sentry_tools", "execute_sentry_tool"]; +const PLACEHOLDERS = ["undefined", "[object Object]", "NaN"]; +const MAX_INSTRUCTIONS = 1500; +const MAX_SUGGESTED_ACTIONS = 3; + +const [mode, ...dirs] = process.argv.slice(2); +const dir = dirs.at(-1); +if (!dir) { + console.error( + "usage: check-results.mjs responses|surface|read-only|scoped DIR | subset WIDE DIR", + ); + process.exit(2); +} +const readJson = (d, name) => JSON.parse(readFileSync(join(d, name), "utf8")); +const listed = (d) => readJson(d, "tools-list.json"); +const toolNames = (d) => listed(d).tools.map((tool) => tool.name); + +const lines = []; +function check(claim, problems, detail = "") { + const ok = problems.length === 0; + lines.push( + `${ok ? "PASS" : "FAIL"} ${claim}${detail ? ` (${detail})` : ""}${ok ? "" : `: ${problems.join("; ")}`}`, + ); +} + +function responses() { + const { tools } = listed(dir); + const byName = new Map(tools.map((tool) => [tool.name, tool])); + const calls = readdirSync(dir) + .filter((file) => /^call-\d+-.+\.json$/.test(file)) + .sort(); + for (const file of calls) { + const { request, result } = readJson(dir, file); + const text = result.content + .filter((item) => item.type === "text") + .map((item) => item.text) + .join("\n"); + const declaresSchema = Boolean(byName.get(request.name)?.outputSchema); + if (result.isError) { + check( + "mcp-tool-responses#tool-error", + result.structuredContent ? ["error result has structuredContent"] : [], + file, + ); + } else if (declaresSchema) { + const sc = result.structuredContent; + check( + "mcp-tool-responses#structured-text-match", + !sc + ? ["no structuredContent"] + : text === JSON.stringify(sc) + ? [] + : ["text is not the compact JSON of structuredContent"], + file, + ); + const actions = sc?.suggestedActions ?? []; + const problems = + actions.length > MAX_SUGGESTED_ACTIONS + ? [`${actions.length} actions`] + : []; + for (const action of actions) { + if (action.type !== "tool_call") problems.push(`type ${action.type}`); + if (!byName.has(action.toolName)) + problems.push(`${action.toolName} is not in tools/list`); + if (!action.arguments || Object.keys(action.arguments).length === 0) + problems.push(`${action.toolName} has no arguments`); + } + check( + "mcp-tool-responses#suggested-actions-callable", + problems, + `${file}, ${actions.length} actions`, + ); + if (Array.isArray(sc?.sideQuests)) { + check( + "mcp-tool-responses#side-quest-fix", + sc.sideQuests + .filter( + (quest) => !quest.tip || !quest.docs?.startsWith("https://"), + ) + .map((quest) => `${quest.id} has no tip or docs link`), + `${file}, ${sc.sideQuests.length} side quests`, + ); + } + } else { + const problems = []; + if (result.structuredContent) + problems.push("markdown tool returned structuredContent"); + if (!text.startsWith("# ")) + problems.push("text does not start with a # title"); + check("mcp-tool-responses#markdown-only", problems, file); + const found = PLACEHOLDERS.filter((token) => + new RegExp( + `(^|[^\\w])${token.replace(/[[\]]/g, "\\$&")}([^\\w]|$)`, + ).test(text), + ); + check( + "mcp-tool-responses#no-placeholders", + found.map((token) => `contains ${token}`), + file, + ); + } + } +} + +function surface() { + const { tools, nextCursor } = listed(dir); + const { instructions = "" } = readJson(dir, "initialize.json"); + check( + "mcp-tool-surface#lists-all", + GATEWAY_TOOLS.filter((name) => tools.some((tool) => tool.name === name)), + `${tools.length} tools`, + ); + check( + "mcp-tool-surface#hints-set", + tools.flatMap((tool) => + ["readOnlyHint", "destructiveHint", "openWorldHint"] + .filter((hint) => typeof tool.annotations?.[hint] !== "boolean") + .map((hint) => `${tool.name} has no ${hint}`), + ), + ); + const titles = tools.map((tool) => tool.title); + check( + "mcp-tool-surface#unique-titles", + tools + .filter((tool, i) => !tool.title || titles.indexOf(tool.title) !== i) + .map((tool) => `${tool.name}: ${tool.title ?? "no title"}`), + ); + check( + "mcp-tool-surface#one-page", + nextCursor ? [`nextCursor ${nextCursor}`] : [], + ); + check( + "mcp-tool-surface#instructions-short", + instructions.length > MAX_INSTRUCTIONS + ? [`${instructions.length} characters`] + : [], + `${instructions.length} characters`, + ); +} + +function readOnly() { + const writes = listed(dir).tools.filter( + (tool) => + !tool.annotations?.readOnlyHint && !EVERY_SKILL_WRITES.has(tool.name), + ); + check( + "mcp-access#inspect-read-only", + writes.map((tool) => `${tool.name} is not read-only`), + ); +} + +function scoped() { + const { tools } = listed(dir); + const problems = tools + .filter((tool) => tool.name === "find_organizations") + .map(() => "find_organizations is listed"); + for (const tool of tools) { + if (tool.inputSchema?.properties?.organizationSlug) + problems.push(`${tool.name} takes organizationSlug`); + } + check("mcp-access#scope-hides-org", problems); +} + +function subset() { + const wide = new Set(toolNames(dirs[0])); + const narrow = toolNames(dir); + const extra = narrow.filter((name) => !wide.has(name)); + const removed = [...wide].filter((name) => !narrow.includes(name)); + const problems = extra.map((name) => `${name} is not in ${dirs[0]}`); + if (removed.length === 0) problems.push("nothing was removed"); + check( + "mcp-access#skills-narrow", + problems, + `${narrow.length} of ${wide.size} tools; removed ${removed.join(", ")}`, + ); +} + +const modes = { responses, surface, "read-only": readOnly, scoped, subset }; +if (!modes[mode]) { + console.error(`unknown mode: ${mode}`); + process.exit(2); +} +modes[mode](); +console.log(lines.join("\n")); +appendFileSync(join(dir, "checks.txt"), `${lines.join("\n")}\n`); +process.exit(lines.some((line) => line.startsWith("FAIL")) ? 1 : 0); diff --git a/.agents/skills/verify-toolkit/scripts/mcp-call.mjs b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs new file mode 100755 index 0000000000..62d7790dad --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs @@ -0,0 +1,134 @@ +#!/usr/bin/env node +// Drive a Sentry MCP stdio server the way an MCP client does, with no LLM. +// Writes initialize.json, tools-list.json, one call--.json per call, +// and server-stderr.txt to --out. See ../SKILL.md ("Drive"). +import { mkdirSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { offlineEnv } from "./offline-env.mjs"; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), "../../../.."); +const usage = + "usage: mcp-call.mjs --out DIR [--server mock|cli] [--no-token] [--server-arg ARG]... [--call TOOL JSON_ARGS]..."; + +const opts = { server: "mock", token: true, serverArgs: [], calls: [] }; +const argv = process.argv.slice(2); +for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === "--out") opts.out = argv[++i]; + else if (arg === "--server") opts.server = argv[++i]; + else if (arg === "--no-token") opts.token = false; + else if (arg === "--server-arg") opts.serverArgs.push(argv[++i]); + else if (arg === "--call") + opts.calls.push([argv[++i], JSON.parse(argv[++i])]); + else { + console.error(`unknown argument: ${arg}\n${usage}`); + process.exit(2); + } +} +if (!opts.out || !["mock", "cli"].includes(opts.server)) { + console.error(usage); + process.exit(2); +} + +// The SDK is a dependency of the test client, not of the repo root. +const requireFromClient = createRequire( + join(repo, "packages/mcp-test-client/package.json"), +); +const sdk = (path) => + import( + pathToFileURL( + requireFromClient.resolve(`@modelcontextprotocol/sdk/${path}`), + ).href + ); +const { Client } = await sdk("client/index.js"); +const { StdioClientTransport } = await sdk("client/stdio.js"); + +// Both servers get the offline env. The mock server answers the Sentry API +// from MSW fixtures and ignores it; `sentry mcp` gets a fake token (unless +// --no-token), an empty config dir, and an unreachable Sentry host. +const { env, cleanup } = offlineEnv({ token: opts.token }); +// The mock server answers for its default host, sentry.io. +if (opts.server === "mock") delete env.SENTRY_HOST; +const [command, args] = + opts.server === "mock" + ? [ + process.execPath, + [ + "--experimental-strip-types", + "--import", + join(repo, "scripts/strip-types-resolve.mjs"), + join(repo, "packages/mcp-server-evals/src/bin/start-mock-stdio.ts"), + ...opts.serverArgs, + ], + ] + : [ + "pnpm", + [ + "--silent", + "--dir", + join(repo, "packages/cli"), + "run", + "cli", + "mcp", + ...opts.serverArgs, + ], + ]; + +mkdirSync(opts.out, { recursive: true }); +const write = (name, value) => + writeFileSync(join(opts.out, name), `${JSON.stringify(value, null, 2)}\n`); + +const transport = new StdioClientTransport({ + command, + args, + env, + cwd: repo, + stderr: "pipe", +}); +let stderr = ""; +transport.stderr?.on("data", (chunk) => { + stderr += chunk; +}); + +const client = new Client({ name: "verify-toolkit", version: "0.0.0" }); +let failed = false; +try { + await client.connect(transport); + write("initialize.json", { + server: opts.server, + command: [command, ...args].join(" "), + serverVersion: client.getServerVersion(), + capabilities: client.getServerCapabilities(), + instructions: client.getInstructions(), + }); + // Listing first also makes the SDK validate each call's structuredContent + // against the tool's outputSchema, as a real client does. + const { tools, nextCursor } = await client.listTools(); + write("tools-list.json", { tools, nextCursor: nextCursor ?? null }); + console.log( + `connected: ${client.getServerVersion()?.name} (${opts.server}), ${tools.length} tools`, + ); + for (const [n, [name, toolArgs]] of opts.calls.entries()) { + const file = `call-${n + 1}-${name}.json`; + const result = await client.callTool({ name, arguments: toolArgs }); + write(file, { request: { name, arguments: toolArgs }, result }); + const shape = result.structuredContent + ? "structuredContent" + : "content only"; + console.log( + `tools/call ${name}: ${result.isError ? "isError" : "ok"}, ${shape} -> ${file}`, + ); + } +} catch (error) { + failed = true; + console.error( + `mcp-call failed: ${error instanceof Error ? error.message : String(error)}`, + ); +} finally { + await client.close().catch(() => {}); + cleanup(); + writeFileSync(join(opts.out, "server-stderr.txt"), stderr); +} +process.exit(failed ? 1 : 0); diff --git a/.agents/skills/verify-toolkit/scripts/offline-env.mjs b/.agents/skills/verify-toolkit/scripts/offline-env.mjs new file mode 100755 index 0000000000..3c0cdba9e5 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/offline-env.mjs @@ -0,0 +1,42 @@ +// Environment for driving `sentry` from source with no live Sentry account. +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +/** Any request to this host fails fast: nothing listens on the discard port. */ +const OFFLINE_SENTRY_URL = "http://127.0.0.1:9"; +export const OFFLINE_TOKEN = "sntryu_verify_toolkit_offline"; + +/** + * Copy of the caller's env with no real credentials, an empty home and CLI + * config dir, an unreachable Sentry host, and telemetry off. The empty home + * keeps the CLI from reading a token in `~/.sentryclirc`. Call `cleanup()` + * when done. + */ +export function offlineEnv({ token }) { + const home = mkdtempSync(join(tmpdir(), "verify-toolkit-home-")); + const configDir = join(home, ".sentry"); + const env = { ...process.env }; + // Set, not deleted: `pnpm run cli` loads `packages/cli/.env.local` for + // unset variables only, and the CLI treats an empty token as unset. + for (const name of [ + "SENTRY_AUTH_TOKEN", + "SENTRY_ACCESS_TOKEN", + "SENTRY_TOKEN", + ]) { + env[name] = ""; + } + Object.assign(env, { + HOME: home, + USERPROFILE: home, + SENTRY_CONFIG_DIR: configDir, + SENTRY_HOST: OFFLINE_SENTRY_URL, + SENTRY_URL: OFFLINE_SENTRY_URL, + SENTRY_CLI_NO_TELEMETRY: "1", + }); + if (token) env.SENTRY_AUTH_TOKEN = OFFLINE_TOKEN; + return { + env, + cleanup: () => rmSync(home, { recursive: true, force: true }), + }; +} diff --git a/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs b/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs new file mode 100755 index 0000000000..148d0740e1 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs @@ -0,0 +1,46 @@ +#!/usr/bin/env node +// Run the `sentry` CLI from source with no live Sentry account, and record +// command.txt, stdout.txt, stderr.txt, and exit-code.txt in --out. +// Usage: sentry-offline.mjs --out DIR [--token] -- ... +import { spawnSync } from "node:child_process"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { OFFLINE_TOKEN, offlineEnv } from "./offline-env.mjs"; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), "../../../.."); +const argv = process.argv.slice(2); +const split = argv.indexOf("--"); +const own = split === -1 ? argv : argv.slice(0, split); +const cliArgs = split === -1 ? [] : argv.slice(split + 1); +const outIndex = own.indexOf("--out"); +const out = outIndex === -1 ? undefined : own[outIndex + 1]; +if (!out || cliArgs.length === 0) { + console.error( + "usage: sentry-offline.mjs --out DIR [--token] -- ...", + ); + process.exit(2); +} + +const { env, cleanup } = offlineEnv({ token: own.includes("--token") }); +const run = spawnSync( + "pnpm", + ["--silent", "--dir", join(repo, "packages/cli"), "run", "cli", ...cliArgs], + { + cwd: repo, + env, + encoding: "utf8", + }, +); +cleanup(); + +mkdirSync(out, { recursive: true }); +writeFileSync(join(out, "command.txt"), `sentry ${cliArgs.join(" ")}\n`); +writeFileSync(join(out, "stdout.txt"), run.stdout); +writeFileSync(join(out, "stderr.txt"), run.stderr); +writeFileSync(join(out, "exit-code.txt"), `${run.status}\n`); +const leaked = `${run.stdout}${run.stderr}`.includes(OFFLINE_TOKEN); +console.log( + `sentry ${cliArgs.join(" ")}: exit ${run.status}${leaked ? ", PRINTED THE TOKEN" : ""}`, +); +process.exit(run.status ?? 1); diff --git a/AGENTS.md b/AGENTS.md index 3e27c63a5b..dc9b6823ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,6 +41,7 @@ toolkit/ ## Documentation Map +- design/README.md — Design docs: each system's goals and rules. Read the owning file before changing a system. - docs/README.md — Toolkit documentation index - docs/cli/README.md — CLI guides and the published CLI site - docs/mcp/README.md — MCP guides across its packages @@ -50,7 +51,7 @@ toolkit/ **Read before MCP tool changes:** - docs/contributing/adding-tools.md — Tool implementation guide -- docs/contributing/tool-responses.md — Tool output policy and QA review checklist +- docs/contributing/tool-responses.md — Conventions every tool result follows (goals and rules: design/mcp-tool-responses.md) - docs/testing/overview.md — Testing requirements and snapshot policy - docs/contributing/common-patterns.md — Error handling, Zod schemas, shared formatting patterns - docs/contributing/error-handling.md — Error types and propagation @@ -134,7 +135,9 @@ pnpm run --filter @sentry/mcp-core generate-definitions For MCP tool QA, use the `mcp-qa` skill at `.agents/skills/mcp-qa/SKILL.md`: stdio-first local CLI and real agent clients; Cloudflare HTTP or `/mcp` only -for transport, OAuth, routing, or hosted-server compatibility. +for transport, OAuth, routing, or hosted-server compatibility. To prove MCP or +CLI behavior offline, with no Sentry account, use the `verify-toolkit` skill at +`.agents/skills/verify-toolkit/SKILL.md` and its claim map. ## Task Management diff --git a/design/README.md b/design/README.md new file mode 100644 index 0000000000..09750a9fa5 --- /dev/null +++ b/design/README.md @@ -0,0 +1,38 @@ +# Design + +Each file states the intent of one major system: the goals it must meet, +where users reach it, and the rules users rely on. Teammates read it to learn +why the system works the way it does. Agents read it before they change the +system. The `create-design-docs` skill writes these files, and the +`maintain-design-docs` skill checks the product against them. + +## File contract + +Each file has an H1 title and one paragraph that says what the user gets. +Then it has these H2 sections, in this order: + +1. `Intent`: the goals that the system must meet, and why. Each goal + decides between real options, such as "runs work with no person + present". +2. `Entry points`: each place where a user reaches the system, by its + visible name. +3. `Rules`: limits that users rely on, such as who can act, and what the + system is not. +4. `Gaps`: only when the product does not meet a goal or a rule yet. Say + how it falls short, and which goal or rule it breaks. + +These files state intent. The code shows what the product does, not what it +should do. Do not change a goal or a rule only to match the code. Name only +user actions, visible names, required access, and outcomes that a user can +see. Leave out source paths, symbols, schemas, history, and rollout notes. + +Do not list behaviors. When the repo has a verification skill, its map owns +each behavior as a claim with a proof. Name a behavior only by its claim ID, +such as `automations#run-now`, and do not restate it. + +## Systems + +- [MCP tools](./mcp-tools.md): the Sentry tools an agent can list and call, and how it finds the right one. +- [MCP tool responses](./mcp-tool-responses.md): what a tool result gives an agent to answer a Sentry question. +- [MCP access](./mcp-access.md): who can use Sentry MCP, over which transport, and what a session can reach. +- [Sentry CLI](./sentry-cli.md): the `sentry` command for people and agents in a terminal. diff --git a/design/mcp-access.md b/design/mcp-access.md new file mode 100644 index 0000000000..c9c360273e --- /dev/null +++ b/design/mcp-access.md @@ -0,0 +1,52 @@ +# MCP access + +A person or team connects Sentry MCP once, with their own Sentry account, and +each session reaches only the Sentry data and tools that they chose. + +## Intent + +- Sentry decides what a user can do. Sentry MCP never widens that, and each + session can narrow it further with skills and an org or project scope. +- Skills are how a user chooses what a session can do. A user who picks + only read access gets a session that cannot change their issues, projects, + or configuration. +- One login serves the CLI and the stdio server, so a developer signs in once + with `sentry auth login`. +- An MCP client that passes a token in its server config gets that account, + and a stored login never silently replaces it. +- On remote HTTP, the MCP client never holds the user's Sentry credentials. +- Stale or invalid access fails closed: the user signs in again instead of a + session continuing on a credential that Sentry rejects. + +## Entry points + +- Stdio: `sentry mcp` (also published as `@sentry/mcp-server`), with a CLI + login or `SENTRY_AUTH_TOKEN`. +- Remote HTTP: `https://mcp.sentry.dev/mcp`, `/mcp/`, and + `/mcp//`, with Sentry OAuth and a skill choice on the consent + screen. +- Remote HTTP with a `Sentry-Bearer` token, which the caller owns. +- Skills: `inspect`, `seer`, `triage`, and `project-management`, chosen with + `--skills`, `--disable-skills`, `--all-skills`, `MCP_SKILLS`, or the consent + screen. +- Scope: an `/` or `/` target on `sentry mcp`, or a scoped + remote URL. + +## Rules + +- A session lists and runs only the tools of its granted skills. +- `inspect` never changes issues, projects, teams, DSNs, alerts, or + monitors. Those tools need `triage` or `project-management`. Every skill + can still send feedback to the Sentry team and report onboarding status. +- A scoped session works only on its org or project, and a token minted for a + scoped URL does not work on a broader one. +- Refreshing access never widens it. +- An org auth token is sent only to the Sentry host it was issued for. +- Without a login or a token, `sentry mcp` exits with a message that says to + run `sentry auth login`. It does not start a session that fails every call. +- A scope that Sentry says does not exist stops `sentry mcp` at startup, with + a message that names the missing part. + +## Gaps + +None. diff --git a/design/mcp-tool-responses.md b/design/mcp-tool-responses.md new file mode 100644 index 0000000000..cd5a0e5da4 --- /dev/null +++ b/design/mcp-tool-responses.md @@ -0,0 +1,58 @@ +# MCP tool responses + +A Sentry MCP tool result gives an agent what it needs to answer a real Sentry +question, in Sentry's own terms, without reading upstream API payloads. + +## Intent + +- A result is product output for the agent and the user. It has the fields + someone needs to inspect, compare, explain, or act on the resource. +- A result models what users need, not what the Sentry API returns. Upstream + field names, internal IDs, and empty placeholders stay out. +- A typed result is one answer. When a tool returns structured content, that + content is the answer, and the text is the same answer for clients that only + read text. No product data exists in only one of the two. +- A result never carries secrets, credentials, or tokens, and upstream text + cannot rewrite the result around it. +- A result is a fact about this call, not an instruction. It may point at a + next tool call that the result supports, but it never tells the agent how + to behave. + +## Entry points + +- The result of any `tools/call`: text content, plus structured content for + tools that declare an output schema. +- Suggested actions: follow-up tool calls that a client can offer. +- Response notes and side quests: short tips at the end of a result, such as + a pagination cursor or source maps that failed to apply. +- Feedback suggestions that point at `submit_feedback_to_sentry_team`. + +## Rules + +- A tool that declares an output schema returns structured content that + matches it on every successful call, and the text is that same payload as + compact JSON. +- A markdown tool returns markdown only. It does not attach a sparse + structured side channel, such as suggested actions alone. +- Structured content is a stable contract. It holds only the fields that + callers should depend on, and never passes through upstream fields as-is. +- A suggested action is complete enough to call as-is, names a tool the + session can call, and comes from this result. A result suggests at most + three. +- A failed call is a tool error (`isError`), not a success-shaped result. An + empty result is a success. A missing resource is an input error that points + at the tool that can find it. +- Optional details, such as a suspect commit, never stop the main result from + loading when they fail. +- Long lists and long text are capped, and the result says how much it left + out. +- A side quest appears only when Sentry's own data shows the problem + unambiguously, and it names the fix. +- A feedback suggestion appears only when the result likely shows a Sentry + MCP bug or gap. Input, permission, auth, rate limit, and configuration + errors, and empty results that are expected, never get one. + +## Gaps + +- `search_attributes` returns no structured content yet. This breaks "a typed + result is one answer" for the one listed tool that has no output schema. diff --git a/design/mcp-tools.md b/design/mcp-tools.md new file mode 100644 index 0000000000..79cf5834d3 --- /dev/null +++ b/design/mcp-tools.md @@ -0,0 +1,56 @@ +# MCP tools + +Sentry MCP gives an agent tools to read and change Sentry data from the +harness the user already works in, such as Claude Code, Codex, or Cursor. + +## Intent + +- The agent finds the right Sentry tool through its harness's own tool + search. Sentry MCP lists every tool the session can call, and does not hide + long-tail tools behind a search tool of its own, because tier 1 harnesses + now load tool schemas on demand. +- A change works in every tier 1 harness (Codex, Claude Code, Cursor). Tier 2 + harnesses (OpenCode, Pi, VS Code with Copilot) do not break, and a known + tier 2 gap is written down in the supported harnesses guide before it ships. +- A session lists only the tools it can call. Skills, the org or project + scope, and the server mode decide that before a tool is listed or run, so + the agent never picks a tool that fails for access reasons. +- Each tool definition stays small, because every name, description, and + schema costs context in every session that lists it. +- Tool descriptions and parameter descriptions carry the guidance on when to + pick a tool. Results answer the current call, and do not steer the agent + like a system prompt. + +## Entry points + +- `tools/list` and `tools/call` from an MCP client, over stdio (`sentry mcp`, + `@sentry/mcp-server`) or remote HTTP (`https://mcp.sentry.dev/mcp`). +- Server instructions, sent on `initialize`. +- `get_sentry_resource` for a pasted Sentry URL, or an issue, event, trace, + or replay ID. +- `--experimental` (stdio) or experimental mode (remote) for tool variants + that are not stable yet. +- `search_sentry_tools` and `execute_sentry_tool`, only on the fallback + gateway surface. + +## Rules + +- Every tool says whether it only reads, whether it can destroy data, and + whether it reaches outside Sentry. Harness filters and confirmation prompts + depend on these hints, so none is ever left unset. +- Every tool has a unique display title. +- `tools/list` returns one page, so a client that ignores paging still sees + every tool. +- Server instructions stay under 1,500 characters, and name a tool only when + the session has it. +- The fallback gateway surface lists about 20 tools, and never more than 25. +- Tools are not a raw proxy for the Sentry API. A tool exists for a question + or task a user brings, and its result follows + [MCP tool responses](./mcp-tool-responses.md). + +## Gaps + +- VS Code with Copilot loads every tool schema up front for models other + than Claude newer than 4.5 and GPT-5.4 or later, so a full listing costs + those sessions context. This breaks "the agent finds the right Sentry tool + through its harness's own tool search" for a tier 2 harness. diff --git a/design/sentry-cli.md b/design/sentry-cli.md new file mode 100644 index 0000000000..e7c94929de --- /dev/null +++ b/design/sentry-cli.md @@ -0,0 +1,38 @@ +# Sentry CLI + +The `sentry` command lets a developer or a coding agent work with Sentry from +a terminal, with as little setup as the current project allows. + +## Intent + +- Zero config. The CLI finds the org and project from the arguments, then + the saved defaults, then DSNs in the project, so most commands run with no + target. +- Commands follow `gh` conventions, so a user who knows `gh` can guess them. +- Agents are first-class users. Every list and view command has JSON output + and predictable behavior, and the CLI ships its own agent skill. +- The CLI is fast to start and to repeat, as a native binary with cached API + responses. +- One login serves every command and `sentry mcp`. + +## Entry points + +- `sentry `, such as `sentry issue list`, `sentry issue view`, + `sentry auth login`, and `sentry mcp`. +- Targets: `/`, `/`, a project name, or a Sentry URL. +- `--json` on list and view commands, and `-w`/`--web` on view commands. +- The npm package `sentry`, release binaries, and the docs at + . + +## Rules + +- An explicit target always wins over saved defaults and DSN detection. +- A command that needs a login and has none exits non-zero with a message + that says to run `sentry auth login`. +- The CLI never prints a token in full. +- Telemetry goes only to the Sentry CLI's own project, and + `SENTRY_CLI_NO_TELEMETRY=1` or `DO_NOT_TRACK=1` turns it off. + +## Gaps + +None. diff --git a/docs/README.md b/docs/README.md index 28d5ea2bb7..0dbae7f6d3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,8 @@ directories below; the MCP index explains which guides apply to that product. ## Shared Contributor Guides +- [Design docs](../design/README.md) — the goals and rules of each major + system. Read the owning file before you change a system. - [Coding guidelines](contributing/coding-guidelines.md) — repository-wide style and the separate CLI and MCP patterns. - [Documentation style](contributing/documentation-style-guide.md) — how to diff --git a/docs/contributing/tool-responses.md b/docs/contributing/tool-responses.md index 8e8bfa2b2d..6c47e210ef 100644 --- a/docs/contributing/tool-responses.md +++ b/docs/contributing/tool-responses.md @@ -1,30 +1,17 @@ # Tool Responses -Tool responses are product UX for agents and users. A response should help an -assistant answer a real Sentry question without forcing it to understand -upstream API shapes, internal implementation details, or placeholder values. +How to build an MCP tool result that meets +[MCP tool responses](../../design/mcp-tool-responses.md), which owns the goals +and rules. This guide covers conventions that apply to every tool. It does not +describe each tool's response: a tool's own output belongs in its description, +its `outputSchema`, and its tests. -Use this policy when adding a tool, changing formatted handler output, updating +Use it when adding a tool, changing formatted handler output, updating snapshots, or QAing MCP behavior. For the full tool implementation flow, see [adding-tools.md](adding-tools.md). For snapshot requirements, see [../testing/overview.md](../testing/overview.md). For end-to-end MCP validation, see [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md). -## Response Contract - -Every formatted tool response should optimize for: - -- **Usefulness**: Include the fields someone needs to inspect, compare, explain, - or act on the resource. -- **Legibility**: Use Sentry product terms and stable markdown sections. -- **Actionability**: Include IDs, URLs, pagination cursors, and follow-up tool - hints only when they help navigation or the next tool call. -- **Safety**: Do not expose secrets, credentials, raw tokens, or unrelated - internal details. Follow [../operations/security.md](../operations/security.md) - for security boundaries. -- **Agent fit**: The output should be easy for an LLM to quote, summarize, and - reason over without post-processing raw API payloads. - ## Markdown Shape Use a predictable markdown structure: @@ -87,6 +74,8 @@ Do not leak implementation details into ordinary tool output: `conditionResult` instead of `result`. - Empty sections created only because the API has that property. - Long opaque payloads that require the model to reverse-engineer meaning. +- Request headers, cookies, and bodies, and other upstream data that can carry + credentials. - Broad assistant instructions such as `IMPORTANT`, `MUST`, `CRITICAL`, or output that tries to override behavior beyond the current result. @@ -121,6 +110,11 @@ Translate upstream API shapes into user-facing terms: strict JSON clients such as Codex reject that lone surrogate, so the server replaces it with U+FFFD in every tool response. Formatters do not need surrogate-aware slicing. +- Treat optional enrichment, such as a suspect commit, as best effort: when its + request fails, keep the main result, and report only unexpected server or + validation failures to Sentry. +- Return an empty result as a success. Return a missing named resource as a + user input error that names the tool that can find it. - Prefer shared helpers for dates, actors, IDs, and unknown values. If a helper emits noisy placeholders for a domain-specific field, filter before calling it. - Render untrusted values with `@sentry/toolkit-core/markdown`, chosen by what @@ -135,91 +129,6 @@ When changing Sentry API endpoint usage, validate the upstream behavior in should model what Sentry returns, but tool responses should model what users need. -### Issue Details - -`get_issue_details` declares `outputSchema` and returns `structuredContent` -for every event type, including types it has no dedicated mapping for, which -keep their identity, message, tags, and contexts. - -It includes the issue's suspect commit when available as `suspectCommit`, with -its SHA, message, author, and source, or `null` without one. Commit lookup is -optional: failures do not prevent issue details from loading, but unexpected -server or response-validation failures are reported to Sentry. - -For metric issues, `event.metricAlert` comes from the occurrence's -`evidenceData`, which the event API returns camelCased -(`dataSources[].queryObj.snubaQuery`, `detectorId`). It has the Metric Monitor -ID, dataset, aggregate, query, interval, environment, evaluated window, value, -and thresholds. `suggestedActions` replace the trace action with -`get_metric_monitor_details` and a pre-filled `search_errors`, `search_logs`, -`search_traces`, or `search_metrics` call for the monitor's query. - -The event API's legacy `context` field (extra data) can be `null`. Issue details -must still load; `event.extra` is `null` for null or empty values while the -event's error, stacktrace, and structured `contexts` data are preserved. - -`event` maps the entries into fields: the exception chain (the raised -exception first), up to 20 thread summaries with a `threadCount`, the crashed -thread when there is no exception, request method and URL, CSP report, N+1 -evidence and spans, metric alert, evidence, user (including email and IP), tags, -extra data, and contexts. Frames use the shared shape in -`tools/support/structured-stacktrace.ts`: most recent call last, with only the -fields that have a value (`inApp` only when true, `absPath` only when it differs -from `filename`). Every frame gets its `contextLine`. Only the most relevant -frame (the last in-app frame of the raised exception or the thread) gets -`preContext`, `postContext` (3 lines each) and `vars`, with each variable cut -to 200 characters of JSON. Request headers, cookies and body, frame addresses, -symbols and source links, tag search queries, and external issue link ids stay -out. An event with more than one thread suggests `get_event_stacktrace` for a -thread that is neither crashed nor current. - -### Event Stacktrace - -`get_event_stacktrace` returns one thread under `thread`, using the same frame -and thread shapes as `get_issue_details`, with `selection.status` saying how the -`thread` selector resolved. When the selector matches no thread or several, -`thread` is null, `selection.message` says why, and `availableThreads` lists -every thread summary. Only the thread's most relevant frame (its last in-app -frame) gets pre/post context and variables. Held locks, -registers and the raw (unsymbolicated) stacktrace stay out. - -### Profile Details - -`get_profile_details` returns one profile summary under `profile`, -discriminated by `profileType`, with sample analysis beside it in `samples`. -For `transaction` and `continuous` profiles, `samples` has frame, sample, and -stack counts, per-thread sample counts, and the ten most frequent frames (only -application frames when `focusOnUserCode` is true). `legacy` speedscope -profiles, such as Android transaction profiles, return `samples: null` because -Sentry does not return sample-format data for them; when the profile names a -transaction and `get_profile` is callable in the session, `suggestedActions` -points at the aggregated flamegraph. Each summary includes `profileUrl`; -transaction and legacy profiles include `traceUrl` when the profile has a trace -ID. - -## Selected Event Package Versions - -Use `get_sentry_resource` with `resourceType: "event"` (or an event URL) and -`packageNames: ["example-package", "@example/client"]` to inspect dependency -versions recorded with that event. The catalog tool `get_issue_details` accepts -the same selection with an explicit `eventId`. - -- Selection is optional: omitting it preserves the ordinary response. -- Accept 1–10 exact, case-sensitive names, each up to 256 characters. Duplicate - names produce one entry; package names are not normalized across ecosystems. -- Use the existing response's `packages` map without extra API calls. Include - `event.packageVersions` in `structuredContent`: a `metadataAvailable` flag and - selected `packages` entries with `name`, `status`, `version`, and - `truncated`. Status is `recorded`, `not_listed`, or `version_not_recorded`; - unavailable metadata has an empty selection. -- Missing, null, and empty maps mean metadata is unavailable. In a populated map, - distinguish a name not listed from a null or blank version. Neither proves a - dependency was absent from the application. -- Truncate versions after 256 characters with an explicit marker. Do not return - unrelated packages. -- Package selection requires an exact event; issue/latest-event and other resource - lookups reject it. - ## Structured Content MCP tools may expose `structuredContent` alongside generated text `content`. @@ -430,8 +339,12 @@ they do not replace a full-response snapshot. For output-format changes: - Run the normal quality gate from [quality-checks.md](quality-checks.md). +- Capture the raw result offline with the `verify-toolkit` skill + ([.agents/skills/verify-toolkit/SKILL.md](../../.agents/skills/verify-toolkit/SKILL.md)); + its `mcp-tool-responses` map checks the rules every result follows. - Use the stdio MCP QA path in - [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md). + [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md) + against prod data. - Inspect the raw MCP tool result when possible, not only the LLM's final answer. The test client's final answer can add model-specific phrasing that is not part of the tool response. @@ -441,42 +354,3 @@ For output-format changes: If the raw tool result is clean but the agent final answer adds unrelated content, treat that as a client/agent prompt issue rather than a tool response formatting issue. - -### Documentation Search - -`search_docs` returns `query`, `guide` (null when omitted), and `results`, each -with the `path` to pass to `get_doc`, the docs `url`, a 0–1 `relevance` score, -and a matching `snippet`. Snippets are search chunks, not full pages, and only -the first three results carry one; later results have `snippet: null`. When -`get_doc` is callable, `suggestedActions` fetches the full page of the first -result with a path. -No matches return `results: []`. A search failure is a tool error, including -an error reported inside a successful search response. - -`get_doc` returns a user input error for a missing page and points the agent to -`search_docs`. docs.sentry.io serves missing pages with status 200 and an -`X-Sentry-Docs-Not-Found` header, so the tool checks that header as well as -status 404. - -### Cron Monitor Environment Deletion - -`delete_monitor_environment` schedules irreversible deletion of one exact -environment from a project-scoped cron monitor. The structured response -identifies the project, monitor, and environment; `success` means the deletion -was accepted or the target was already absent. Deletion completes in the -background, preserving the monitor and its other environments. - -### Transaction Profiles - -`get_profile` returns the aggregated flamegraph for one transaction as -structured content. Frames appear once in `analysis.frames` with their -call-tree `occurrences` (not a sample count), p75/p95/p99 durations, and -insights; `topFunctions`, hot path `stack` arrays, and hot path `leaf` values -refer to them by index, so shared parent frames are not repeated per hot path. -A stack reads from root to leaf, and its last frame is the primary bottleneck. -With `focusOnUserCode`, library frames are omitted from stacks, and `leaf` -still names the sampled leaf frame. `userCodeMs` and `libraryMs` split the main -thread's sampled time by whether each sample's leaf frame is application code, -so they add up to `estimatedTotalMs`. Missing data is a successful result: -`hasCurrentData` and `hasBaselineData` report which period had profiles, and -`analysis` or `comparison` is `null`. diff --git a/docs/operations/embedded-agents.md b/docs/operations/embedded-agents.md index 2087d94f92..4255dea537 100644 --- a/docs/operations/embedded-agents.md +++ b/docs/operations/embedded-agents.md @@ -304,4 +304,4 @@ Should output: - [Security](security.md) - Authentication and token management - [Testing](../testing/overview.md) - Testing MCP server functionality - [Common Patterns](../contributing/common-patterns.md) - Error handling and shared formatting patterns -- [Tool Responses](../contributing/tool-responses.md) - Tool output policy and QA review checklist +- [Tool Responses](../contributing/tool-responses.md) - Conventions every tool result follows