Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/mcp-qa/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ dispatch without changing real Sentry data.
For output-format changes, also inspect the raw MCP tool result when possible,
not only the LLM's final answer. The final answer can add model-specific text
that is not part of the tool response. Review raw tool output against
`docs/contributing/tool-responses.md`: it should be user-facing, structured,
`design/mcp-tool-responses.md` and `docs/contributing/tool-responses.md`: it should be user-facing, structured,
and free of raw API JSON, internal implementation IDs, empty placeholders, and
unrelated instructions.

Expand Down
127 changes: 127 additions & 0 deletions .agents/skills/verify-toolkit/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
---
name: verify-toolkit
description: Prove Sentry MCP and Sentry CLI behavior in this repo by running the real stdio server and the `sentry` CLI from source, offline, and capturing MCP and terminal evidence. Use to verify a change to MCP tools, tool responses, skills, scopes, stdio auth, or CLI auth, or to check a claim from `map/`. For QA against a live Sentry org or a real agent client, use the `mcp-qa` skill instead.
---

# Verify Toolkit

The primary surface is the Sentry MCP server over stdio, driven as an MCP
client drives it: `initialize`, `tools/list`, `tools/call`. The secondary
surface is the `sentry` CLI in a terminal. Two servers are available:

- **mock**: the eval server (`packages/mcp-server-evals/src/bin/start-mock-stdio.ts`).
Real stdio transport and tool handlers, every skill, experimental mode, and
the Sentry API answered by the MSW fixtures in `packages/mcp-server-mocks`.
Use it for tool calls and responses.
- **cli**: `sentry mcp` from CLI source, with a fake token, an empty config
dir, and an unreachable Sentry host (`http://127.0.0.1:9`). Use it for what
the real entry point lists: skills, scopes, auth, and server metadata. Its
tool calls fail, because no Sentry answers.

Nothing here reaches a live Sentry org. Live data, real logins, real agent
clients, and the Cloudflare `/mcp` server are `mcp-qa` territory
(`.agents/skills/mcp-qa/SKILL.md`).

## Launch

From the repo root, once per checkout or after changing MCP or CLI code:

```bash
pnpm install --frozen-lockfile
env -u SENTRY_AUTH_TOKEN pnpm run build
(cd packages/cli && node script/ensure-generated.mjs)
```

Ready means: `pnpm run build` exits 0, and `ensure-generated` exits 0 and
leaves `packages/cli/src/generated/command-api.json` as valid JSON. Each
drive starts its own server process and stops it when it finishes, so there
is no long-running server and nothing to tear down. Runs are isolated: each
gets its own temporary home and CLI config dir, and two runs can go side by side.

Set a run directory for evidence, outside the checkout:

```bash
export VERIFY_RUN="${TMPDIR:-/tmp}/verify-toolkit/$(date +%Y%m%d-%H%M%S)"
```

## Doctor

```bash
node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/doctor-mock"
node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/doctor-cli"
```

Both print `connected: Sentry MCP (...), N tools` and exit 0. If either
fails, read `server-stderr.txt` in its directory, then rerun Launch.

## Drive

`scripts/mcp-call.mjs` starts a server, lists tools, runs each `--call`, and
writes `initialize.json`, `tools-list.json`, `call-<n>-<tool>.json` (request
and raw `CallToolResult`), and `server-stderr.txt` to `--out`. Listing first
makes the MCP SDK validate every structured result against the tool's
`outputSchema`, as a real client does; a mismatch fails the run.

```bash
node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out DIR \
[--server mock|cli] [--no-token] [--server-arg ARG]... \
[--call TOOL 'JSON_ARGS']...
```

- `--server-arg` passes one argument to `sentry mcp`, such as
`--server-arg --skills --server-arg inspect` or `--server-arg acme/`.
- `--no-token` starts `sentry mcp` with no token and no login.
- Mock fixtures use the org `sentry-mcp-evals`, project `cloudflare-mcp`,
issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`.

`scripts/check-results.mjs` checks evidence against map claims, prints one
`PASS` or `FAIL` line per claim, appends them to `checks.txt` in the same
directory, and exits 1 on any failure:

```bash
node .agents/skills/verify-toolkit/scripts/check-results.mjs responses DIR
node .agents/skills/verify-toolkit/scripts/check-results.mjs surface DIR
node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only DIR
node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped DIR
node .agents/skills/verify-toolkit/scripts/check-results.mjs subset WIDE_DIR DIR
```

`scripts/sentry-offline.mjs` runs one CLI command with the same offline env,
and writes `command.txt`, `stdout.txt`, `stderr.txt`, and `exit-code.txt`:

```bash
node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out DIR [--token] -- auth status
```

`--token` sets `SENTRY_AUTH_TOKEN` to a fake token. The script prints
`PRINTED THE TOKEN` if that token appears in the output.

## Evidence

Everything goes under `$VERIFY_RUN`, one directory per drive, named for the
claim it proves. Proof standards:

- Drive the real entry point: MCP JSON-RPC over stdio, or the `sentry`
command. Never call handlers or internal helpers.
- Keep the raw MCP result, not only an agent's answer.
- Keep the action and the result: the request sits next to the result in
each `call-*.json`, and the command next to its output for the CLI.
- The only mock is the Sentry API, an external HTTP boundary. Do not mock
the server, the transport, or the CLI.
- Offline `sentry mcp` makes network attempts, such as a scope check at
startup. They go to `127.0.0.1:9` and fail; check `server-stderr.txt` and
do not assume a skipped network call.
- Cite the claim ID (`<file>#<id>`) with each artifact you report.

## Cleanup

Each script stops the process it started and removes its temporary config
dir. If you stop a script early, kill only the PID of the server it started.
Never kill by process name: a developer's own `sentry mcp` may be running.
Do not delete `$VERIFY_RUN`; it is the evidence.

## Map

`map/README.md` lists the map files and the entry contract. Cite a claim as
`<file>#<id>`, such as `mcp-tool-responses#structured-text-match`. The goals
behind the claims are in `design/`.
59 changes: 59 additions & 0 deletions .agents/skills/verify-toolkit/map/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Toolkit verification map

This directory is the maintained source for verifying Sentry MCP and Sentry
CLI behavior. Read this index, then use the matching map file as the recipe.
The goals and rules that these claims prove are in `design/`.

## Baseline preconditions

- Launch from `../SKILL.md` has run on this checkout since the last change to
MCP or CLI code.
- `VERIFY_RUN` names a fresh directory outside the checkout.
- The Doctor commands in `../SKILL.md` pass.
- Never drive a server, login, or config dir that this run did not create.

## Driving conventions

- Run every command from the repo root.
- Treat every command as literal. Keep tool names, arguments, and flags.
- `mcp-call.mjs` is the MCP client; `sentry-offline.mjs` is the terminal.
- One `--out` directory per bullet, under `$VERIFY_RUN`.

## Proof and skip reporting

- MCP proof is the `call-*.json` or `tools-list.json` file, plus the
`checks.txt` line for the claim.
- CLI proof is `command.txt`, `stdout.txt`, `stderr.txt`, and
`exit-code.txt`.
- Record the claim ID and the entry point with every artifact.
- Report an unreachable path with the command you ran and the unmet
precondition. Do not report a claim as verified through a different entry
point.

## Claim IDs

- A claim ID is lowercase kebab-case and unique in its file. Cite it as
`<file>#<id>`, where `<file>` is the file name without `.md`.
- Tests, evals, and PRs cite claims by this ID.
- Do not rename or reuse an ID. When a behavior changes meaning, retire its
ID and add a new one. Search the repo for citations first.

## Map entry contract

Each map file starts with an H1 title and one paragraph. Then it has exactly
five H2 sections, in this order: `Claims`, `How to get to it`,
`Driving it with <harness>` (starting with `Preconditions:`), `Gotchas`, and
`Gaps` (`None.` when empty). Each claim names one proof: `drive`, `test` with
the exact command, or `eval` with the exact command. Keep source structure
out of the map.

## Map files

- `mcp-tool-responses.md` covers what a tool result holds: structured and
markdown results, suggested actions, side quests, and tool errors.
- `mcp-tool-surface.md` covers what `tools/list` and `initialize` give a
client: every callable tool, safety hints, titles, one page, and
instructions.
- `mcp-access.md` covers who can start `sentry mcp` and what a session
reaches: login, skills, and org scope.
- `cli.md` covers CLI behavior with no login and with a token.
35 changes: 35 additions & 0 deletions .agents/skills/verify-toolkit/map/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Sentry CLI

How the `sentry` command behaves in a terminal with no login and with an
environment token: commands that need a login stop with one clear message,
and `auth status` never prints the token in full. The goals are in
`design/sentry-cli.md`.

## Claims

- `no-login-exit` A command that needs a login, with none, exits 10 and says to run `sentry auth login`. Proof: `drive`.
- `token-masked` `sentry auth status` with `SENTRY_AUTH_TOKEN` reports the env token as active and never prints it in full. Proof: `drive`.

## How to get to it

- `sentry <group> <command>` in a terminal, here from CLI source through `sentry-offline.mjs`.
- `SENTRY_AUTH_TOKEN` in the environment, or a login from `sentry auth login`.

## Driving it with sentry-offline.mjs

Preconditions:

- Launch from `../SKILL.md` passes, and `$VERIFY_RUN` is set.

- **`no-login-exit`: list issues with no login.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-no-login" -- issue list acme/`. It prints `exit 10`. `stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.`
- **`token-masked`: check status with a token.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-token" --token -- auth status`. It prints `exit 0` and no `PRINTED THE TOKEN`. `stdout.txt` contains `Authenticated via SENTRY_AUTH_TOKEN`, and its `Token` row shows a shortened token, such as `sntryu_v...line`.

## Gotchas

- The first CLI run on a fresh checkout generates files and prints that work. Run Launch's `ensure-generated` step first, or the output mixes with the proof.
- A short token prints as `****` and a long one as its first 8 and last 4 characters. Both pass; only the full token fails.
- `auth status` also reports `Could not verify credentials` offline. That line is expected and is not part of the claim.

## Gaps

- `json-output` List and view commands print JSON with `--json`. Proving it needs a Sentry host that answers.
41 changes: 41 additions & 0 deletions .agents/skills/verify-toolkit/map/mcp-access.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# MCP access

Who can start `sentry mcp`, and what a session reaches: it refuses to start
with no login, skills narrow the tool list, `inspect` changes no Sentry data,
and an org scope removes the org choice from every tool. The goals are in
`design/mcp-access.md`.

## Claims

- `no-login-exits` With no token and no login, `sentry mcp` exits before the MCP handshake and says to run `sentry auth login`. Proof: `drive`.
- `skills-narrow` `--skills inspect` lists a strict subset of the default session's tools. Proof: `drive`.
- `inspect-read-only` An `inspect` session lists no tool that changes data, except feedback and onboarding status. Proof: `drive`.
- `scope-hides-org` An `<org>/` scope lists no `find_organizations`, and no tool takes `organizationSlug`. Proof: `drive`.
- `remote-scoped-url` A remote token minted for one org's URL gets 403 on another org's URL. Proof: `test` with `pnpm --filter @sentry/mcp-cloudflare exec vitest run src/server/lib/mcp-handler.test.ts -t "returns 403 when the token is org-scoped but the MCP URL uses a different organization"`.

## How to get to it

- `sentry mcp`, with `SENTRY_AUTH_TOKEN` or a CLI login.
- `--skills`, `--disable-skills`, `--all-skills`, `MCP_SKILLS`, and `MCP_DISABLE_SKILLS`.
- An `<org>/` or `<org>/<project>` target, or `--org` and `--project`.
- Remote HTTP at `/mcp`, `/mcp/<org>`, and `/mcp/<org>/<project>` with OAuth.

## Driving it with mcp-call.mjs

Preconditions:

- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set.

- **`no-login-exits`: start with nothing.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --no-token --out "$VERIFY_RUN/access-no-login"`. It exits 1 with `mcp-call failed: MCP error -32000: Connection closed`, and `server-stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.`
- **`skills-narrow`, `inspect-read-only`: compare skills.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/access-default"` and `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg --skills --server-arg inspect --out "$VERIFY_RUN/access-inspect"`. Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs subset "$VERIFY_RUN/access-default" "$VERIFY_RUN/access-inspect"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only "$VERIFY_RUN/access-inspect"`. Both print `PASS`, and the subset line names removed tools such as `update_issue`.
- **`scope-hides-org`: scope to an org.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg acme/ --out "$VERIFY_RUN/access-scoped"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped "$VERIFY_RUN/access-scoped"`. It prints `PASS`.

## Gotchas

- The scope check at startup cannot reach the offline host, so `sentry mcp acme/` starts as if the org exists. That is the documented behavior for a network failure, not proof that `acme` exists.
- `--no-token` also needs the empty home and config dir that the script makes: the CLI reads a token from `~/.sentryclirc`. The script also sets the token variables to empty, because `pnpm run cli` loads `packages/cli/.env.local` for any variable that is not set. Never point `HOME` or `SENTRY_CONFIG_DIR` at a real login for this drive.

## Gaps

- `scope-not-found-exits` A scope that Sentry answers 404 for stops `sentry mcp` with exit code 23 and names the missing part. The offline drive has no Sentry host that answers.
- `token-host-scoped` An org auth token is sent only to the host it was issued for. The offline drive has no second host to send it to.
57 changes: 57 additions & 0 deletions .agents/skills/verify-toolkit/map/mcp-tool-responses.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# MCP tool responses

What a `tools/call` result gives an agent: structured content that is the
answer, markdown for tools without an output schema, follow-up calls the
session can make, side quests that name a fix, and tool errors that are not
dressed up as success. The goals are in `design/mcp-tool-responses.md`.

## Claims

- `structured-text-match` A tool with an output schema returns structured content, and its text is that payload as compact JSON. Proof: `drive`.
- `schema-valid` Every structured result matches the tool's advertised output schema. Proof: `drive` (the MCP client rejects a mismatch).
- `suggested-actions-callable` Suggested actions are at most three, each a complete `tool_call` with arguments, naming a tool in this session's `tools/list`. Proof: `drive`.
- `side-quest-fix` Each side quest has a tip that names the fix and an https docs link. Proof: `drive`.
- `markdown-only` A tool with no output schema returns markdown with a `#` title and no structured content. Proof: `drive`.
- `no-placeholders` Markdown results contain no `undefined`, `[object Object]`, or `NaN`. Proof: `drive`.
- `tool-error` A call with missing input returns `isError` with an input error message and no structured content. Proof: `drive`.
- `schema-guard` Every handler result in the mcp-core suite matches its output schema. Proof: `test` with `pnpm --filter @sentry/mcp-core exec vitest run src/server.metadata.test.ts -t "validates handler results against outputSchema in every test"`.

## How to get to it

- `tools/call` from any MCP client over stdio, here through `mcp-call.mjs` against the mock server.
- The same result reaches remote HTTP clients; this map does not drive that transport.

## Driving it with mcp-call.mjs

Preconditions:

- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set.
- The mock server's fixtures include org `sentry-mcp-evals`, issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`.

- **`structured-text-match`, `schema-valid`, `suggested-actions-callable`, `side-quest-fix`, `markdown-only`, `no-placeholders`, `tool-error`: call four tools.** Run:

```bash
node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/mcp-tool-responses" \
--call get_issue_details '{"organizationSlug":"sentry-mcp-evals","issueId":"CLOUDFLARE-MCP-41"}' \
--call find_projects '{"organizationSlug":"sentry-mcp-evals"}' \
--call get_trace_details '{"organizationSlug":"sentry-mcp-evals","traceId":"a4d1aae7216b47ff8117cf4e09ce9d0a"}' \
--call get_issue_details '{}'
```

It exits 0 (so the SDK accepted both structured results against their schemas, `schema-valid`) and prints `ok, structuredContent` for calls 1 and 2, `ok, content only` for call 3, and `isError, content only` for call 4.

- **Same claims: check the evidence.** Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs responses "$VERIFY_RUN/mcp-tool-responses"`. Every line is `PASS`: `structured-text-match` and `suggested-actions-callable` for calls 1 and 2, `side-quest-fix` for call 1, `markdown-only` and `no-placeholders` for call 3, and `tool-error` for call 4.
- **`tool-error`: read the message.** `call-4-get_issue_details.json` has `"isError": true`, and its text starts with `**Input Error**` and names `issueId` or `issueUrl`.

## Gotchas

- The mock server grants every skill and runs in experimental mode, so its `tools/list` is not what a user's session lists. Use `mcp-tool-surface.md` and `mcp-access.md` for that.
- A fixture the mock server lacks becomes an HTTP 500 tool error whose text includes the mock's own stack trace. That is a fixture gap, not a product result: use the fixture IDs above.
- `find_projects` ignores its `query` against the mock; do not use it to prove filtering.
- `check-results.mjs` appends to `checks.txt`. Use a fresh `--out` per run.

## Gaps

- `untrusted-contained` Upstream text cannot add headings, list items, or links to a markdown result. No mock fixture carries markdown in an upstream value.
- `feedback-only-for-bugs` A feedback suggestion appears only for likely Sentry MCP bugs. The mock server registers no feedback tool, because it has no DSN, so no result can carry the suggestion.
- `optional-detail-fails` A failed optional detail, such as a suspect commit, does not stop the main result. No fixture makes only the optional request fail.
Loading
Loading