From c304c40d0693a0cd3a4f2675796aefc32aa59f40 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Mon, 16 Nov 2026 22:45:49 +0000 Subject: [PATCH 1/4] docs: Add design docs and cut per-tool shapes from tool responses tool-responses.md had grown into a catalog of individual tool outputs (issue details, stacktraces, profiles, package versions, docs search, cron environment deletion). A tool's own output belongs in its description, outputSchema, and tests, so those sections are gone. The few general rules they carried (optional enrichment is best effort, empty results succeed, missing resources are input errors, request headers and bodies stay out) moved into the shared conventions. Add design/ with one file per major system: MCP tools, MCP tool responses, MCP access, and the Sentry CLI. Each states the goals and the rules users rely on, and the response goals now live in design/mcp-tool-responses.md. AGENTS.md and the docs index link it. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com> --- AGENTS.md | 3 +- design/README.md | 38 +++++++ design/mcp-access.md | 52 ++++++++++ design/mcp-tool-responses.md | 58 +++++++++++ design/mcp-tools.md | 56 ++++++++++ design/sentry-cli.md | 38 +++++++ docs/README.md | 2 + docs/contributing/tool-responses.md | 156 +++------------------------- docs/operations/embedded-agents.md | 2 +- 9 files changed, 260 insertions(+), 145 deletions(-) create mode 100644 design/README.md create mode 100644 design/mcp-access.md create mode 100644 design/mcp-tool-responses.md create mode 100644 design/mcp-tools.md create mode 100644 design/sentry-cli.md diff --git a/AGENTS.md b/AGENTS.md index 3e27c63a5b..f44a0e687a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,6 +41,7 @@ toolkit/ ## Documentation Map +- design/README.md — Design docs: each system's goals and rules. Read the owning file before changing a system. - docs/README.md — Toolkit documentation index - docs/cli/README.md — CLI guides and the published CLI site - docs/mcp/README.md — MCP guides across its packages @@ -50,7 +51,7 @@ toolkit/ **Read before MCP tool changes:** - docs/contributing/adding-tools.md — Tool implementation guide -- docs/contributing/tool-responses.md — Tool output policy and QA review checklist +- docs/contributing/tool-responses.md — Conventions every tool result follows (goals and rules: design/mcp-tool-responses.md) - docs/testing/overview.md — Testing requirements and snapshot policy - docs/contributing/common-patterns.md — Error handling, Zod schemas, shared formatting patterns - docs/contributing/error-handling.md — Error types and propagation diff --git a/design/README.md b/design/README.md new file mode 100644 index 0000000000..09750a9fa5 --- /dev/null +++ b/design/README.md @@ -0,0 +1,38 @@ +# Design + +Each file states the intent of one major system: the goals it must meet, +where users reach it, and the rules users rely on. Teammates read it to learn +why the system works the way it does. Agents read it before they change the +system. The `create-design-docs` skill writes these files, and the +`maintain-design-docs` skill checks the product against them. + +## File contract + +Each file has an H1 title and one paragraph that says what the user gets. +Then it has these H2 sections, in this order: + +1. `Intent`: the goals that the system must meet, and why. Each goal + decides between real options, such as "runs work with no person + present". +2. `Entry points`: each place where a user reaches the system, by its + visible name. +3. `Rules`: limits that users rely on, such as who can act, and what the + system is not. +4. `Gaps`: only when the product does not meet a goal or a rule yet. Say + how it falls short, and which goal or rule it breaks. + +These files state intent. The code shows what the product does, not what it +should do. Do not change a goal or a rule only to match the code. Name only +user actions, visible names, required access, and outcomes that a user can +see. Leave out source paths, symbols, schemas, history, and rollout notes. + +Do not list behaviors. When the repo has a verification skill, its map owns +each behavior as a claim with a proof. Name a behavior only by its claim ID, +such as `automations#run-now`, and do not restate it. + +## Systems + +- [MCP tools](./mcp-tools.md): the Sentry tools an agent can list and call, and how it finds the right one. +- [MCP tool responses](./mcp-tool-responses.md): what a tool result gives an agent to answer a Sentry question. +- [MCP access](./mcp-access.md): who can use Sentry MCP, over which transport, and what a session can reach. +- [Sentry CLI](./sentry-cli.md): the `sentry` command for people and agents in a terminal. diff --git a/design/mcp-access.md b/design/mcp-access.md new file mode 100644 index 0000000000..c9c360273e --- /dev/null +++ b/design/mcp-access.md @@ -0,0 +1,52 @@ +# MCP access + +A person or team connects Sentry MCP once, with their own Sentry account, and +each session reaches only the Sentry data and tools that they chose. + +## Intent + +- Sentry decides what a user can do. Sentry MCP never widens that, and each + session can narrow it further with skills and an org or project scope. +- Skills are how a user chooses what a session can do. A user who picks + only read access gets a session that cannot change their issues, projects, + or configuration. +- One login serves the CLI and the stdio server, so a developer signs in once + with `sentry auth login`. +- An MCP client that passes a token in its server config gets that account, + and a stored login never silently replaces it. +- On remote HTTP, the MCP client never holds the user's Sentry credentials. +- Stale or invalid access fails closed: the user signs in again instead of a + session continuing on a credential that Sentry rejects. + +## Entry points + +- Stdio: `sentry mcp` (also published as `@sentry/mcp-server`), with a CLI + login or `SENTRY_AUTH_TOKEN`. +- Remote HTTP: `https://mcp.sentry.dev/mcp`, `/mcp/`, and + `/mcp//`, with Sentry OAuth and a skill choice on the consent + screen. +- Remote HTTP with a `Sentry-Bearer` token, which the caller owns. +- Skills: `inspect`, `seer`, `triage`, and `project-management`, chosen with + `--skills`, `--disable-skills`, `--all-skills`, `MCP_SKILLS`, or the consent + screen. +- Scope: an `/` or `/` target on `sentry mcp`, or a scoped + remote URL. + +## Rules + +- A session lists and runs only the tools of its granted skills. +- `inspect` never changes issues, projects, teams, DSNs, alerts, or + monitors. Those tools need `triage` or `project-management`. Every skill + can still send feedback to the Sentry team and report onboarding status. +- A scoped session works only on its org or project, and a token minted for a + scoped URL does not work on a broader one. +- Refreshing access never widens it. +- An org auth token is sent only to the Sentry host it was issued for. +- Without a login or a token, `sentry mcp` exits with a message that says to + run `sentry auth login`. It does not start a session that fails every call. +- A scope that Sentry says does not exist stops `sentry mcp` at startup, with + a message that names the missing part. + +## Gaps + +None. diff --git a/design/mcp-tool-responses.md b/design/mcp-tool-responses.md new file mode 100644 index 0000000000..cd5a0e5da4 --- /dev/null +++ b/design/mcp-tool-responses.md @@ -0,0 +1,58 @@ +# MCP tool responses + +A Sentry MCP tool result gives an agent what it needs to answer a real Sentry +question, in Sentry's own terms, without reading upstream API payloads. + +## Intent + +- A result is product output for the agent and the user. It has the fields + someone needs to inspect, compare, explain, or act on the resource. +- A result models what users need, not what the Sentry API returns. Upstream + field names, internal IDs, and empty placeholders stay out. +- A typed result is one answer. When a tool returns structured content, that + content is the answer, and the text is the same answer for clients that only + read text. No product data exists in only one of the two. +- A result never carries secrets, credentials, or tokens, and upstream text + cannot rewrite the result around it. +- A result is a fact about this call, not an instruction. It may point at a + next tool call that the result supports, but it never tells the agent how + to behave. + +## Entry points + +- The result of any `tools/call`: text content, plus structured content for + tools that declare an output schema. +- Suggested actions: follow-up tool calls that a client can offer. +- Response notes and side quests: short tips at the end of a result, such as + a pagination cursor or source maps that failed to apply. +- Feedback suggestions that point at `submit_feedback_to_sentry_team`. + +## Rules + +- A tool that declares an output schema returns structured content that + matches it on every successful call, and the text is that same payload as + compact JSON. +- A markdown tool returns markdown only. It does not attach a sparse + structured side channel, such as suggested actions alone. +- Structured content is a stable contract. It holds only the fields that + callers should depend on, and never passes through upstream fields as-is. +- A suggested action is complete enough to call as-is, names a tool the + session can call, and comes from this result. A result suggests at most + three. +- A failed call is a tool error (`isError`), not a success-shaped result. An + empty result is a success. A missing resource is an input error that points + at the tool that can find it. +- Optional details, such as a suspect commit, never stop the main result from + loading when they fail. +- Long lists and long text are capped, and the result says how much it left + out. +- A side quest appears only when Sentry's own data shows the problem + unambiguously, and it names the fix. +- A feedback suggestion appears only when the result likely shows a Sentry + MCP bug or gap. Input, permission, auth, rate limit, and configuration + errors, and empty results that are expected, never get one. + +## Gaps + +- `search_attributes` returns no structured content yet. This breaks "a typed + result is one answer" for the one listed tool that has no output schema. diff --git a/design/mcp-tools.md b/design/mcp-tools.md new file mode 100644 index 0000000000..79cf5834d3 --- /dev/null +++ b/design/mcp-tools.md @@ -0,0 +1,56 @@ +# MCP tools + +Sentry MCP gives an agent tools to read and change Sentry data from the +harness the user already works in, such as Claude Code, Codex, or Cursor. + +## Intent + +- The agent finds the right Sentry tool through its harness's own tool + search. Sentry MCP lists every tool the session can call, and does not hide + long-tail tools behind a search tool of its own, because tier 1 harnesses + now load tool schemas on demand. +- A change works in every tier 1 harness (Codex, Claude Code, Cursor). Tier 2 + harnesses (OpenCode, Pi, VS Code with Copilot) do not break, and a known + tier 2 gap is written down in the supported harnesses guide before it ships. +- A session lists only the tools it can call. Skills, the org or project + scope, and the server mode decide that before a tool is listed or run, so + the agent never picks a tool that fails for access reasons. +- Each tool definition stays small, because every name, description, and + schema costs context in every session that lists it. +- Tool descriptions and parameter descriptions carry the guidance on when to + pick a tool. Results answer the current call, and do not steer the agent + like a system prompt. + +## Entry points + +- `tools/list` and `tools/call` from an MCP client, over stdio (`sentry mcp`, + `@sentry/mcp-server`) or remote HTTP (`https://mcp.sentry.dev/mcp`). +- Server instructions, sent on `initialize`. +- `get_sentry_resource` for a pasted Sentry URL, or an issue, event, trace, + or replay ID. +- `--experimental` (stdio) or experimental mode (remote) for tool variants + that are not stable yet. +- `search_sentry_tools` and `execute_sentry_tool`, only on the fallback + gateway surface. + +## Rules + +- Every tool says whether it only reads, whether it can destroy data, and + whether it reaches outside Sentry. Harness filters and confirmation prompts + depend on these hints, so none is ever left unset. +- Every tool has a unique display title. +- `tools/list` returns one page, so a client that ignores paging still sees + every tool. +- Server instructions stay under 1,500 characters, and name a tool only when + the session has it. +- The fallback gateway surface lists about 20 tools, and never more than 25. +- Tools are not a raw proxy for the Sentry API. A tool exists for a question + or task a user brings, and its result follows + [MCP tool responses](./mcp-tool-responses.md). + +## Gaps + +- VS Code with Copilot loads every tool schema up front for models other + than Claude newer than 4.5 and GPT-5.4 or later, so a full listing costs + those sessions context. This breaks "the agent finds the right Sentry tool + through its harness's own tool search" for a tier 2 harness. diff --git a/design/sentry-cli.md b/design/sentry-cli.md new file mode 100644 index 0000000000..e7c94929de --- /dev/null +++ b/design/sentry-cli.md @@ -0,0 +1,38 @@ +# Sentry CLI + +The `sentry` command lets a developer or a coding agent work with Sentry from +a terminal, with as little setup as the current project allows. + +## Intent + +- Zero config. The CLI finds the org and project from the arguments, then + the saved defaults, then DSNs in the project, so most commands run with no + target. +- Commands follow `gh` conventions, so a user who knows `gh` can guess them. +- Agents are first-class users. Every list and view command has JSON output + and predictable behavior, and the CLI ships its own agent skill. +- The CLI is fast to start and to repeat, as a native binary with cached API + responses. +- One login serves every command and `sentry mcp`. + +## Entry points + +- `sentry `, such as `sentry issue list`, `sentry issue view`, + `sentry auth login`, and `sentry mcp`. +- Targets: `/`, `/`, a project name, or a Sentry URL. +- `--json` on list and view commands, and `-w`/`--web` on view commands. +- The npm package `sentry`, release binaries, and the docs at + . + +## Rules + +- An explicit target always wins over saved defaults and DSN detection. +- A command that needs a login and has none exits non-zero with a message + that says to run `sentry auth login`. +- The CLI never prints a token in full. +- Telemetry goes only to the Sentry CLI's own project, and + `SENTRY_CLI_NO_TELEMETRY=1` or `DO_NOT_TRACK=1` turns it off. + +## Gaps + +None. diff --git a/docs/README.md b/docs/README.md index 28d5ea2bb7..0dbae7f6d3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,6 +20,8 @@ directories below; the MCP index explains which guides apply to that product. ## Shared Contributor Guides +- [Design docs](../design/README.md) — the goals and rules of each major + system. Read the owning file before you change a system. - [Coding guidelines](contributing/coding-guidelines.md) — repository-wide style and the separate CLI and MCP patterns. - [Documentation style](contributing/documentation-style-guide.md) — how to diff --git a/docs/contributing/tool-responses.md b/docs/contributing/tool-responses.md index 8e8bfa2b2d..9a5a248c88 100644 --- a/docs/contributing/tool-responses.md +++ b/docs/contributing/tool-responses.md @@ -1,30 +1,17 @@ # Tool Responses -Tool responses are product UX for agents and users. A response should help an -assistant answer a real Sentry question without forcing it to understand -upstream API shapes, internal implementation details, or placeholder values. +How to build an MCP tool result that meets +[MCP tool responses](../../design/mcp-tool-responses.md), which owns the goals +and rules. This guide covers conventions that apply to every tool. It does not +describe each tool's response: a tool's own output belongs in its description, +its `outputSchema`, and its tests. -Use this policy when adding a tool, changing formatted handler output, updating +Use it when adding a tool, changing formatted handler output, updating snapshots, or QAing MCP behavior. For the full tool implementation flow, see [adding-tools.md](adding-tools.md). For snapshot requirements, see [../testing/overview.md](../testing/overview.md). For end-to-end MCP validation, see [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md). -## Response Contract - -Every formatted tool response should optimize for: - -- **Usefulness**: Include the fields someone needs to inspect, compare, explain, - or act on the resource. -- **Legibility**: Use Sentry product terms and stable markdown sections. -- **Actionability**: Include IDs, URLs, pagination cursors, and follow-up tool - hints only when they help navigation or the next tool call. -- **Safety**: Do not expose secrets, credentials, raw tokens, or unrelated - internal details. Follow [../operations/security.md](../operations/security.md) - for security boundaries. -- **Agent fit**: The output should be easy for an LLM to quote, summarize, and - reason over without post-processing raw API payloads. - ## Markdown Shape Use a predictable markdown structure: @@ -87,6 +74,8 @@ Do not leak implementation details into ordinary tool output: `conditionResult` instead of `result`. - Empty sections created only because the API has that property. - Long opaque payloads that require the model to reverse-engineer meaning. +- Request headers, cookies, and bodies, and other upstream data that can carry + credentials. - Broad assistant instructions such as `IMPORTANT`, `MUST`, `CRITICAL`, or output that tries to override behavior beyond the current result. @@ -121,6 +110,11 @@ Translate upstream API shapes into user-facing terms: strict JSON clients such as Codex reject that lone surrogate, so the server replaces it with U+FFFD in every tool response. Formatters do not need surrogate-aware slicing. +- Treat optional enrichment, such as a suspect commit, as best effort: when its + request fails, keep the main result, and report only unexpected server or + validation failures to Sentry. +- Return an empty result as a success. Return a missing named resource as a + user input error that names the tool that can find it. - Prefer shared helpers for dates, actors, IDs, and unknown values. If a helper emits noisy placeholders for a domain-specific field, filter before calling it. - Render untrusted values with `@sentry/toolkit-core/markdown`, chosen by what @@ -135,91 +129,6 @@ When changing Sentry API endpoint usage, validate the upstream behavior in should model what Sentry returns, but tool responses should model what users need. -### Issue Details - -`get_issue_details` declares `outputSchema` and returns `structuredContent` -for every event type, including types it has no dedicated mapping for, which -keep their identity, message, tags, and contexts. - -It includes the issue's suspect commit when available as `suspectCommit`, with -its SHA, message, author, and source, or `null` without one. Commit lookup is -optional: failures do not prevent issue details from loading, but unexpected -server or response-validation failures are reported to Sentry. - -For metric issues, `event.metricAlert` comes from the occurrence's -`evidenceData`, which the event API returns camelCased -(`dataSources[].queryObj.snubaQuery`, `detectorId`). It has the Metric Monitor -ID, dataset, aggregate, query, interval, environment, evaluated window, value, -and thresholds. `suggestedActions` replace the trace action with -`get_metric_monitor_details` and a pre-filled `search_errors`, `search_logs`, -`search_traces`, or `search_metrics` call for the monitor's query. - -The event API's legacy `context` field (extra data) can be `null`. Issue details -must still load; `event.extra` is `null` for null or empty values while the -event's error, stacktrace, and structured `contexts` data are preserved. - -`event` maps the entries into fields: the exception chain (the raised -exception first), up to 20 thread summaries with a `threadCount`, the crashed -thread when there is no exception, request method and URL, CSP report, N+1 -evidence and spans, metric alert, evidence, user (including email and IP), tags, -extra data, and contexts. Frames use the shared shape in -`tools/support/structured-stacktrace.ts`: most recent call last, with only the -fields that have a value (`inApp` only when true, `absPath` only when it differs -from `filename`). Every frame gets its `contextLine`. Only the most relevant -frame (the last in-app frame of the raised exception or the thread) gets -`preContext`, `postContext` (3 lines each) and `vars`, with each variable cut -to 200 characters of JSON. Request headers, cookies and body, frame addresses, -symbols and source links, tag search queries, and external issue link ids stay -out. An event with more than one thread suggests `get_event_stacktrace` for a -thread that is neither crashed nor current. - -### Event Stacktrace - -`get_event_stacktrace` returns one thread under `thread`, using the same frame -and thread shapes as `get_issue_details`, with `selection.status` saying how the -`thread` selector resolved. When the selector matches no thread or several, -`thread` is null, `selection.message` says why, and `availableThreads` lists -every thread summary. Only the thread's most relevant frame (its last in-app -frame) gets pre/post context and variables. Held locks, -registers and the raw (unsymbolicated) stacktrace stay out. - -### Profile Details - -`get_profile_details` returns one profile summary under `profile`, -discriminated by `profileType`, with sample analysis beside it in `samples`. -For `transaction` and `continuous` profiles, `samples` has frame, sample, and -stack counts, per-thread sample counts, and the ten most frequent frames (only -application frames when `focusOnUserCode` is true). `legacy` speedscope -profiles, such as Android transaction profiles, return `samples: null` because -Sentry does not return sample-format data for them; when the profile names a -transaction and `get_profile` is callable in the session, `suggestedActions` -points at the aggregated flamegraph. Each summary includes `profileUrl`; -transaction and legacy profiles include `traceUrl` when the profile has a trace -ID. - -## Selected Event Package Versions - -Use `get_sentry_resource` with `resourceType: "event"` (or an event URL) and -`packageNames: ["example-package", "@example/client"]` to inspect dependency -versions recorded with that event. The catalog tool `get_issue_details` accepts -the same selection with an explicit `eventId`. - -- Selection is optional: omitting it preserves the ordinary response. -- Accept 1–10 exact, case-sensitive names, each up to 256 characters. Duplicate - names produce one entry; package names are not normalized across ecosystems. -- Use the existing response's `packages` map without extra API calls. Include - `event.packageVersions` in `structuredContent`: a `metadataAvailable` flag and - selected `packages` entries with `name`, `status`, `version`, and - `truncated`. Status is `recorded`, `not_listed`, or `version_not_recorded`; - unavailable metadata has an empty selection. -- Missing, null, and empty maps mean metadata is unavailable. In a populated map, - distinguish a name not listed from a null or blank version. Neither proves a - dependency was absent from the application. -- Truncate versions after 256 characters with an explicit marker. Do not return - unrelated packages. -- Package selection requires an exact event; issue/latest-event and other resource - lookups reject it. - ## Structured Content MCP tools may expose `structuredContent` alongside generated text `content`. @@ -441,42 +350,3 @@ For output-format changes: If the raw tool result is clean but the agent final answer adds unrelated content, treat that as a client/agent prompt issue rather than a tool response formatting issue. - -### Documentation Search - -`search_docs` returns `query`, `guide` (null when omitted), and `results`, each -with the `path` to pass to `get_doc`, the docs `url`, a 0–1 `relevance` score, -and a matching `snippet`. Snippets are search chunks, not full pages, and only -the first three results carry one; later results have `snippet: null`. When -`get_doc` is callable, `suggestedActions` fetches the full page of the first -result with a path. -No matches return `results: []`. A search failure is a tool error, including -an error reported inside a successful search response. - -`get_doc` returns a user input error for a missing page and points the agent to -`search_docs`. docs.sentry.io serves missing pages with status 200 and an -`X-Sentry-Docs-Not-Found` header, so the tool checks that header as well as -status 404. - -### Cron Monitor Environment Deletion - -`delete_monitor_environment` schedules irreversible deletion of one exact -environment from a project-scoped cron monitor. The structured response -identifies the project, monitor, and environment; `success` means the deletion -was accepted or the target was already absent. Deletion completes in the -background, preserving the monitor and its other environments. - -### Transaction Profiles - -`get_profile` returns the aggregated flamegraph for one transaction as -structured content. Frames appear once in `analysis.frames` with their -call-tree `occurrences` (not a sample count), p75/p95/p99 durations, and -insights; `topFunctions`, hot path `stack` arrays, and hot path `leaf` values -refer to them by index, so shared parent frames are not repeated per hot path. -A stack reads from root to leaf, and its last frame is the primary bottleneck. -With `focusOnUserCode`, library frames are omitted from stacks, and `leaf` -still names the sampled leaf frame. `userCodeMs` and `libraryMs` split the main -thread's sampled time by whether each sample's leaf frame is application code, -so they add up to `estimatedTotalMs`. Missing data is a successful result: -`hasCurrentData` and `hasBaselineData` report which period had profiles, and -`analysis` or `comparison` is `null`. diff --git a/docs/operations/embedded-agents.md b/docs/operations/embedded-agents.md index 2087d94f92..4255dea537 100644 --- a/docs/operations/embedded-agents.md +++ b/docs/operations/embedded-agents.md @@ -304,4 +304,4 @@ Should output: - [Security](security.md) - Authentication and token management - [Testing](../testing/overview.md) - Testing MCP server functionality - [Common Patterns](../contributing/common-patterns.md) - Error handling and shared formatting patterns -- [Tool Responses](../contributing/tool-responses.md) - Tool output policy and QA review checklist +- [Tool Responses](../contributing/tool-responses.md) - Conventions every tool result follows From 04b022228fd56bcc9ee5177cf3e72aa28fe57b74 Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Mon, 16 Nov 2026 22:50:32 +0000 Subject: [PATCH 2/4] feat(skills): Add verify-toolkit skill with an offline claim map Add a project verification skill that drives the real stdio MCP server and the sentry CLI from source with no Sentry account: the eval mock server for tool calls, and `sentry mcp` with a fake token, an empty config dir, and an unreachable host for what the real entry point lists. Its helpers capture raw MCP results and CLI output and check them against claims in four map files: tool responses, tool surface, MCP access, and the CLI. tool-responses.md, mcp-qa, and AGENTS.md point at it for offline proof; mcp-qa stays the path for live data and real agent clients. Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com> --- .agents/skills/mcp-qa/SKILL.md | 2 +- .agents/skills/verify-toolkit/SKILL.md | 127 +++++++++++ .agents/skills/verify-toolkit/map/README.md | 59 +++++ .agents/skills/verify-toolkit/map/cli.md | 35 +++ .../skills/verify-toolkit/map/mcp-access.md | 41 ++++ .../verify-toolkit/map/mcp-tool-responses.md | 57 +++++ .../verify-toolkit/map/mcp-tool-surface.md | 39 ++++ .../verify-toolkit/scripts/check-results.mjs | 202 ++++++++++++++++++ .../verify-toolkit/scripts/mcp-call.mjs | 132 ++++++++++++ .../verify-toolkit/scripts/offline-env.mjs | 35 +++ .../verify-toolkit/scripts/sentry-offline.mjs | 46 ++++ AGENTS.md | 4 +- docs/contributing/tool-responses.md | 6 +- 13 files changed, 782 insertions(+), 3 deletions(-) create mode 100644 .agents/skills/verify-toolkit/SKILL.md create mode 100644 .agents/skills/verify-toolkit/map/README.md create mode 100644 .agents/skills/verify-toolkit/map/cli.md create mode 100644 .agents/skills/verify-toolkit/map/mcp-access.md create mode 100644 .agents/skills/verify-toolkit/map/mcp-tool-responses.md create mode 100644 .agents/skills/verify-toolkit/map/mcp-tool-surface.md create mode 100755 .agents/skills/verify-toolkit/scripts/check-results.mjs create mode 100755 .agents/skills/verify-toolkit/scripts/mcp-call.mjs create mode 100755 .agents/skills/verify-toolkit/scripts/offline-env.mjs create mode 100755 .agents/skills/verify-toolkit/scripts/sentry-offline.mjs diff --git a/.agents/skills/mcp-qa/SKILL.md b/.agents/skills/mcp-qa/SKILL.md index b255701b9c..7ccd5d7bc9 100644 --- a/.agents/skills/mcp-qa/SKILL.md +++ b/.agents/skills/mcp-qa/SKILL.md @@ -71,7 +71,7 @@ dispatch without changing real Sentry data. For output-format changes, also inspect the raw MCP tool result when possible, not only the LLM's final answer. The final answer can add model-specific text that is not part of the tool response. Review raw tool output against -`docs/contributing/tool-responses.md`: it should be user-facing, structured, +`design/mcp-tool-responses.md` and `docs/contributing/tool-responses.md`: it should be user-facing, structured, and free of raw API JSON, internal implementation IDs, empty placeholders, and unrelated instructions. diff --git a/.agents/skills/verify-toolkit/SKILL.md b/.agents/skills/verify-toolkit/SKILL.md new file mode 100644 index 0000000000..7742a3f1d2 --- /dev/null +++ b/.agents/skills/verify-toolkit/SKILL.md @@ -0,0 +1,127 @@ +--- +name: verify-toolkit +description: Prove Sentry MCP and Sentry CLI behavior in this repo by running the real stdio server and the `sentry` CLI from source, offline, and capturing MCP and terminal evidence. Use to verify a change to MCP tools, tool responses, skills, scopes, stdio auth, or CLI auth, or to check a claim from `map/`. For QA against a live Sentry org or a real agent client, use the `mcp-qa` skill instead. +--- + +# Verify Toolkit + +The primary surface is the Sentry MCP server over stdio, driven as an MCP +client drives it: `initialize`, `tools/list`, `tools/call`. The secondary +surface is the `sentry` CLI in a terminal. Two servers are available: + +- **mock**: the eval server (`packages/mcp-server-evals/src/bin/start-mock-stdio.ts`). + Real stdio transport and tool handlers, every skill, experimental mode, and + the Sentry API answered by the MSW fixtures in `packages/mcp-server-mocks`. + Use it for tool calls and responses. +- **cli**: `sentry mcp` from CLI source, with a fake token, an empty config + dir, and an unreachable Sentry host (`http://127.0.0.1:9`). Use it for what + the real entry point lists: skills, scopes, auth, and server metadata. Its + tool calls fail, because no Sentry answers. + +Nothing here reaches a live Sentry org. Live data, real logins, real agent +clients, and the Cloudflare `/mcp` server are `mcp-qa` territory +(`.agents/skills/mcp-qa/SKILL.md`). + +## Launch + +From the repo root, once per checkout or after changing MCP or CLI code: + +```bash +pnpm install --frozen-lockfile +env -u SENTRY_AUTH_TOKEN pnpm run build +(cd packages/cli && node script/ensure-generated.mjs) +``` + +Ready means: `pnpm run build` exits 0, and `ensure-generated` exits 0 and +leaves `packages/cli/src/generated/command-api.json` as valid JSON. Each +drive starts its own server process and stops it when it finishes, so there +is no long-running server and nothing to tear down. Runs are isolated: each +gets its own temporary CLI config dir, and two runs can go side by side. + +Set a run directory for evidence, outside the checkout: + +```bash +export VERIFY_RUN="${TMPDIR:-/tmp}/verify-toolkit/$(date +%Y%m%d-%H%M%S)" +``` + +## Doctor + +```bash +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/doctor-mock" +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/doctor-cli" +``` + +Both print `connected: Sentry MCP (...), N tools` and exit 0. If either +fails, read `server-stderr.txt` in its directory, then rerun Launch. + +## Drive + +`scripts/mcp-call.mjs` starts a server, lists tools, runs each `--call`, and +writes `initialize.json`, `tools-list.json`, `call--.json` (request +and raw `CallToolResult`), and `server-stderr.txt` to `--out`. Listing first +makes the MCP SDK validate every structured result against the tool's +`outputSchema`, as a real client does; a mismatch fails the run. + +```bash +node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out DIR \ + [--server mock|cli] [--no-token] [--server-arg ARG]... \ + [--call TOOL 'JSON_ARGS']... +``` + +- `--server-arg` passes one argument to `sentry mcp`, such as + `--server-arg --skills --server-arg inspect` or `--server-arg acme/`. +- `--no-token` starts `sentry mcp` with no token and no login. +- Mock fixtures use the org `sentry-mcp-evals`, project `cloudflare-mcp`, + issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`. + +`scripts/check-results.mjs` checks evidence against map claims, prints one +`PASS` or `FAIL` line per claim, appends them to `checks.txt` in the same +directory, and exits 1 on any failure: + +```bash +node .agents/skills/verify-toolkit/scripts/check-results.mjs responses DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs surface DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped DIR +node .agents/skills/verify-toolkit/scripts/check-results.mjs subset WIDE_DIR DIR +``` + +`scripts/sentry-offline.mjs` runs one CLI command with the same offline env, +and writes `command.txt`, `stdout.txt`, `stderr.txt`, and `exit-code.txt`: + +```bash +node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out DIR [--token] -- auth status +``` + +`--token` sets `SENTRY_AUTH_TOKEN` to a fake token. The script prints +`PRINTED THE TOKEN` if that token appears in the output. + +## Evidence + +Everything goes under `$VERIFY_RUN`, one directory per drive, named for the +claim it proves. Proof standards: + +- Drive the real entry point: MCP JSON-RPC over stdio, or the `sentry` + command. Never call handlers or internal helpers. +- Keep the raw MCP result, not only an agent's answer. +- Keep the action and the result: the request sits next to the result in + each `call-*.json`, and the command next to its output for the CLI. +- The only mock is the Sentry API, an external HTTP boundary. Do not mock + the server, the transport, or the CLI. +- Offline `sentry mcp` makes network attempts, such as a scope check at + startup. They go to `127.0.0.1:9` and fail; check `server-stderr.txt` and + do not assume a skipped network call. +- Cite the claim ID (`#`) with each artifact you report. + +## Cleanup + +Each script stops the process it started and removes its temporary config +dir. If you stop a script early, kill only the PID of the server it started. +Never kill by process name: a developer's own `sentry mcp` may be running. +Do not delete `$VERIFY_RUN`; it is the evidence. + +## Map + +`map/README.md` lists the map files and the entry contract. Cite a claim as +`#`, such as `mcp-tool-responses#structured-text-match`. The goals +behind the claims are in `design/`. diff --git a/.agents/skills/verify-toolkit/map/README.md b/.agents/skills/verify-toolkit/map/README.md new file mode 100644 index 0000000000..c094c63586 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/README.md @@ -0,0 +1,59 @@ +# Toolkit verification map + +This directory is the maintained source for verifying Sentry MCP and Sentry +CLI behavior. Read this index, then use the matching map file as the recipe. +The goals and rules that these claims prove are in `design/`. + +## Baseline preconditions + +- Launch from `../SKILL.md` has run on this checkout since the last change to + MCP or CLI code. +- `VERIFY_RUN` names a fresh directory outside the checkout. +- The Doctor commands in `../SKILL.md` pass. +- Never drive a server, login, or config dir that this run did not create. + +## Driving conventions + +- Run every command from the repo root. +- Treat every command as literal. Keep tool names, arguments, and flags. +- `mcp-call.mjs` is the MCP client; `sentry-offline.mjs` is the terminal. +- One `--out` directory per bullet, under `$VERIFY_RUN`. + +## Proof and skip reporting + +- MCP proof is the `call-*.json` or `tools-list.json` file, plus the + `checks.txt` line for the claim. +- CLI proof is `command.txt`, `stdout.txt`, `stderr.txt`, and + `exit-code.txt`. +- Record the claim ID and the entry point with every artifact. +- Report an unreachable path with the command you ran and the unmet + precondition. Do not report a claim as verified through a different entry + point. + +## Claim IDs + +- A claim ID is lowercase kebab-case and unique in its file. Cite it as + `#`, where `` is the file name without `.md`. +- Tests, evals, and PRs cite claims by this ID. +- Do not rename or reuse an ID. When a behavior changes meaning, retire its + ID and add a new one. Search the repo for citations first. + +## Map entry contract + +Each map file starts with an H1 title and one paragraph. Then it has exactly +five H2 sections, in this order: `Claims`, `How to get to it`, +`Driving it with ` (starting with `Preconditions:`), `Gotchas`, and +`Gaps` (`None.` when empty). Each claim names one proof: `drive`, `test` with +the exact command, or `eval` with the exact command. Keep source structure +out of the map. + +## Map files + +- `mcp-tool-responses.md` covers what a tool result holds: structured and + markdown results, suggested actions, side quests, and tool errors. +- `mcp-tool-surface.md` covers what `tools/list` and `initialize` give a + client: every callable tool, safety hints, titles, one page, and + instructions. +- `mcp-access.md` covers who can start `sentry mcp` and what a session + reaches: login, skills, and org scope. +- `cli.md` covers CLI behavior with no login and with a token. diff --git a/.agents/skills/verify-toolkit/map/cli.md b/.agents/skills/verify-toolkit/map/cli.md new file mode 100644 index 0000000000..8962de92a1 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/cli.md @@ -0,0 +1,35 @@ +# Sentry CLI + +How the `sentry` command behaves in a terminal with no login and with an +environment token: commands that need a login stop with one clear message, +and `auth status` never prints the token in full. The goals are in +`design/sentry-cli.md`. + +## Claims + +- `no-login-exit` A command that needs a login, with none, exits 10 and says to run `sentry auth login`. Proof: `drive`. +- `token-masked` `sentry auth status` with `SENTRY_AUTH_TOKEN` reports the env token as active and never prints it in full. Proof: `drive`. + +## How to get to it + +- `sentry ` in a terminal, here from CLI source through `sentry-offline.mjs`. +- `SENTRY_AUTH_TOKEN` in the environment, or a login from `sentry auth login`. + +## Driving it with sentry-offline.mjs + +Preconditions: + +- Launch from `../SKILL.md` passes, and `$VERIFY_RUN` is set. + +- **`no-login-exit`: list issues with no login.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-no-login" -- issue list acme/`. It prints `exit 10`. `stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.` +- **`token-masked`: check status with a token.** Run `node .agents/skills/verify-toolkit/scripts/sentry-offline.mjs --out "$VERIFY_RUN/cli-token" --token -- auth status`. It prints `exit 0` and no `PRINTED THE TOKEN`. `stdout.txt` contains `Authenticated via SENTRY_AUTH_TOKEN`, and its `Token` row shows a shortened token, such as `sntryu_v...line`. + +## Gotchas + +- The first CLI run on a fresh checkout generates files and prints that work. Run Launch's `ensure-generated` step first, or the output mixes with the proof. +- A short token prints as `****` and a long one as its first 8 and last 4 characters. Both pass; only the full token fails. +- `auth status` also reports `Could not verify credentials` offline. That line is expected and is not part of the claim. + +## Gaps + +- `json-output` List and view commands print JSON with `--json`. Proving it needs a Sentry host that answers. diff --git a/.agents/skills/verify-toolkit/map/mcp-access.md b/.agents/skills/verify-toolkit/map/mcp-access.md new file mode 100644 index 0000000000..f1f6188374 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-access.md @@ -0,0 +1,41 @@ +# MCP access + +Who can start `sentry mcp`, and what a session reaches: it refuses to start +with no login, skills narrow the tool list, `inspect` changes no Sentry data, +and an org scope removes the org choice from every tool. The goals are in +`design/mcp-access.md`. + +## Claims + +- `no-login-exits` With no token and no login, `sentry mcp` exits before the MCP handshake and says to run `sentry auth login`. Proof: `drive`. +- `skills-narrow` `--skills inspect` lists a strict subset of the default session's tools. Proof: `drive`. +- `inspect-read-only` An `inspect` session lists no tool that changes data, except feedback and onboarding status. Proof: `drive`. +- `scope-hides-org` An `/` scope lists no `find_organizations`, and no tool takes `organizationSlug`. Proof: `drive`. +- `remote-scoped-url` A remote token minted for one org's URL gets 403 on another org's URL. Proof: `test` with `pnpm --filter @sentry/mcp-cloudflare exec vitest run src/server/lib/mcp-handler.test.ts -t "returns 403 when the token is org-scoped but the MCP URL uses a different organization"`. + +## How to get to it + +- `sentry mcp`, with `SENTRY_AUTH_TOKEN` or a CLI login. +- `--skills`, `--disable-skills`, `--all-skills`, `MCP_SKILLS`, and `MCP_DISABLE_SKILLS`. +- An `/` or `/` target, or `--org` and `--project`. +- Remote HTTP at `/mcp`, `/mcp/`, and `/mcp//` with OAuth. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. + +- **`no-login-exits`: start with nothing.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --no-token --out "$VERIFY_RUN/access-no-login"`. It exits 1 with `mcp-call failed: MCP error -32000: Connection closed`, and `server-stderr.txt` contains `Not authenticated. Run 'sentry auth login' first.` +- **`skills-narrow`, `inspect-read-only`: compare skills.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/access-default"` and `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg --skills --server-arg inspect --out "$VERIFY_RUN/access-inspect"`. Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs subset "$VERIFY_RUN/access-default" "$VERIFY_RUN/access-inspect"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs read-only "$VERIFY_RUN/access-inspect"`. Both print `PASS`, and the subset line names removed tools such as `update_issue`. +- **`scope-hides-org`: scope to an org.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --server-arg acme/ --out "$VERIFY_RUN/access-scoped"` and `node .agents/skills/verify-toolkit/scripts/check-results.mjs scoped "$VERIFY_RUN/access-scoped"`. It prints `PASS`. + +## Gotchas + +- The scope check at startup cannot reach the offline host, so `sentry mcp acme/` starts as if the org exists. That is the documented behavior for a network failure, not proof that `acme` exists. +- `--no-token` also needs the empty config dir that the script makes. Never point `SENTRY_CONFIG_DIR` at a real login for this drive. + +## Gaps + +- `scope-not-found-exits` A scope that Sentry answers 404 for stops `sentry mcp` with exit code 23 and names the missing part. The offline drive has no Sentry host that answers. +- `token-host-scoped` An org auth token is sent only to the host it was issued for. The offline drive has no second host to send it to. diff --git a/.agents/skills/verify-toolkit/map/mcp-tool-responses.md b/.agents/skills/verify-toolkit/map/mcp-tool-responses.md new file mode 100644 index 0000000000..036fc9e6a3 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-tool-responses.md @@ -0,0 +1,57 @@ +# MCP tool responses + +What a `tools/call` result gives an agent: structured content that is the +answer, markdown for tools without an output schema, follow-up calls the +session can make, side quests that name a fix, and tool errors that are not +dressed up as success. The goals are in `design/mcp-tool-responses.md`. + +## Claims + +- `structured-text-match` A tool with an output schema returns structured content, and its text is that payload as compact JSON. Proof: `drive`. +- `schema-valid` Every structured result matches the tool's advertised output schema. Proof: `drive` (the MCP client rejects a mismatch). +- `suggested-actions-callable` Suggested actions are at most three, each a complete `tool_call` with arguments, naming a tool in this session's `tools/list`. Proof: `drive`. +- `side-quest-fix` Each side quest has a tip that names the fix and an https docs link. Proof: `drive`. +- `markdown-only` A tool with no output schema returns markdown with a `#` title and no structured content. Proof: `drive`. +- `no-placeholders` Markdown results contain no `undefined`, `[object Object]`, or `NaN`. Proof: `drive`. +- `tool-error` A call with missing input returns `isError` with an input error message and no structured content. Proof: `drive`. +- `schema-guard` Every handler result in the mcp-core suite matches its output schema. Proof: `test` with `pnpm --filter @sentry/mcp-core exec vitest run src/server.metadata.test.ts -t "validates handler results against outputSchema in every test"`. + +## How to get to it + +- `tools/call` from any MCP client over stdio, here through `mcp-call.mjs` against the mock server. +- The same result reaches remote HTTP clients; this map does not drive that transport. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. +- The mock server's fixtures include org `sentry-mcp-evals`, issue `CLOUDFLARE-MCP-41`, and trace `a4d1aae7216b47ff8117cf4e09ce9d0a`. + +- **`structured-text-match`, `schema-valid`, `suggested-actions-callable`, `side-quest-fix`, `markdown-only`, `no-placeholders`, `tool-error`: call four tools.** Run: + + ```bash + node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --out "$VERIFY_RUN/mcp-tool-responses" \ + --call get_issue_details '{"organizationSlug":"sentry-mcp-evals","issueId":"CLOUDFLARE-MCP-41"}' \ + --call find_projects '{"organizationSlug":"sentry-mcp-evals"}' \ + --call get_trace_details '{"organizationSlug":"sentry-mcp-evals","traceId":"a4d1aae7216b47ff8117cf4e09ce9d0a"}' \ + --call get_issue_details '{}' + ``` + + It exits 0 (so the SDK accepted both structured results against their schemas, `schema-valid`) and prints `ok, structuredContent` for calls 1 and 2, `ok, content only` for call 3, and `isError, content only` for call 4. + +- **Same claims: check the evidence.** Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs responses "$VERIFY_RUN/mcp-tool-responses"`. Every line is `PASS`: `structured-text-match` and `suggested-actions-callable` for calls 1 and 2, `side-quest-fix` for call 1, `markdown-only` and `no-placeholders` for call 3, and `tool-error` for call 4. +- **`tool-error`: read the message.** `call-4-get_issue_details.json` has `"isError": true`, and its text starts with `**Input Error**` and names `issueId` or `issueUrl`. + +## Gotchas + +- The mock server grants every skill and runs in experimental mode, so its `tools/list` is not what a user's session lists. Use `mcp-tool-surface.md` and `mcp-access.md` for that. +- A fixture the mock server lacks becomes an HTTP 500 tool error whose text includes the mock's own stack trace. That is a fixture gap, not a product result: use the fixture IDs above. +- `find_projects` ignores its `query` against the mock; do not use it to prove filtering. +- `check-results.mjs` appends to `checks.txt`. Use a fresh `--out` per run. + +## Gaps + +- `untrusted-contained` Upstream text cannot add headings, list items, or links to a markdown result. No mock fixture carries markdown in an upstream value. +- `feedback-only-for-bugs` A feedback suggestion appears only for likely Sentry MCP bugs. The mock server registers no feedback tool, because it has no DSN, so no result can carry the suggestion. +- `optional-detail-fails` A failed optional detail, such as a suspect commit, does not stop the main result. No fixture makes only the optional request fail. diff --git a/.agents/skills/verify-toolkit/map/mcp-tool-surface.md b/.agents/skills/verify-toolkit/map/mcp-tool-surface.md new file mode 100644 index 0000000000..45ca41ab41 --- /dev/null +++ b/.agents/skills/verify-toolkit/map/mcp-tool-surface.md @@ -0,0 +1,39 @@ +# MCP tool surface + +What a client gets from `initialize` and `tools/list` on the real `sentry mcp` +entry point: every tool the session can call, listed directly, each with +safety hints and a unique title, in one page, with short server +instructions. The goals are in `design/mcp-tools.md`. + +## Claims + +- `lists-all` A default session lists its tools directly, with no `search_sentry_tools` or `execute_sentry_tool`. Proof: `drive`. +- `hints-set` Every listed tool sets `readOnlyHint`, `destructiveHint`, and `openWorldHint` to a boolean. Proof: `drive`. +- `unique-titles` Every listed tool has a title, and no two share one. Proof: `drive`. +- `one-page` `tools/list` returns every tool with no `nextCursor`. Proof: `drive`. +- `instructions-short` Server instructions are at most 1,500 characters. Proof: `drive`. +- `metadata-lint` Server identity, tool names, hints, titles, description lengths, and `_meta` keys pass the metadata linter in stable and experimental mode. Proof: `test` with `pnpm --filter @sentry/mcp-core exec vitest run src/server.metadata.test.ts`. + +## How to get to it + +- `sentry mcp` over stdio, from an MCP client's server config. +- `npx @sentry/mcp-server`, which runs the same command. +- Remote HTTP at `https://mcp.sentry.dev/mcp`; this map does not drive it. + +## Driving it with mcp-call.mjs + +Preconditions: + +- Launch and Doctor from `../SKILL.md` pass, and `$VERIFY_RUN` is set. + +- **`lists-all`, `hints-set`, `unique-titles`, `one-page`, `instructions-short`: list tools.** Run `node .agents/skills/verify-toolkit/scripts/mcp-call.mjs --server cli --out "$VERIFY_RUN/mcp-tool-surface"`. It prints `connected: Sentry MCP (cli), N tools` with N above 25. Run `node .agents/skills/verify-toolkit/scripts/check-results.mjs surface "$VERIFY_RUN/mcp-tool-surface"`. Every line is `PASS`. +- **`instructions-short`: read them.** `initialize.json` has `instructions` that name only tools present in `tools-list.json`. + +## Gotchas + +- The offline host is self-hosted from the CLI's view, so the default session leaves out `seer` tools such as `analyze_issue_with_seer`. Pass `--server-arg --all-skills` to list them. +- The mock server's list is not the user's list: it grants every skill in experimental mode. Drive `--server cli` for this file. + +## Gaps + +- `gateway-cap` The fallback gateway surface lists at most 25 tools. It is a code constant that is off, so no entry point can turn it on. diff --git a/.agents/skills/verify-toolkit/scripts/check-results.mjs b/.agents/skills/verify-toolkit/scripts/check-results.mjs new file mode 100755 index 0000000000..8189e55f43 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/check-results.mjs @@ -0,0 +1,202 @@ +#!/usr/bin/env node +// Check mcp-call.mjs evidence against map claims. Prints one PASS/FAIL line +// per claim, appends them to /checks.txt, and exits 1 on any FAIL. +// check-results.mjs responses DIR mcp-tool-responses claims, per call +// check-results.mjs surface DIR mcp-tool-surface claims +// check-results.mjs read-only DIR mcp-access#inspect-read-only +// check-results.mjs scoped DIR mcp-access#scope-hides-org +// check-results.mjs subset WIDE DIR mcp-access#skills-narrow (DIR lists fewer tools) +import { appendFileSync, readFileSync, readdirSync } from "node:fs"; +import { join } from "node:path"; + +/** Tools every skill keeps, though they send data to Sentry (design/mcp-access.md). */ +const EVERY_SKILL_WRITES = new Set([ + "submit_feedback_to_sentry_team", + "onboarding_status_update", +]); +const GATEWAY_TOOLS = ["search_sentry_tools", "execute_sentry_tool"]; +const PLACEHOLDERS = ["undefined", "[object Object]", "NaN"]; +const MAX_INSTRUCTIONS = 1500; +const MAX_SUGGESTED_ACTIONS = 3; + +const [mode, ...dirs] = process.argv.slice(2); +const dir = dirs.at(-1); +if (!dir) { + console.error( + "usage: check-results.mjs responses|surface|read-only|scoped DIR | subset WIDE DIR", + ); + process.exit(2); +} +const readJson = (d, name) => JSON.parse(readFileSync(join(d, name), "utf8")); +const listed = (d) => readJson(d, "tools-list.json"); +const toolNames = (d) => listed(d).tools.map((tool) => tool.name); + +const lines = []; +function check(claim, problems, detail = "") { + const ok = problems.length === 0; + lines.push( + `${ok ? "PASS" : "FAIL"} ${claim}${detail ? ` (${detail})` : ""}${ok ? "" : `: ${problems.join("; ")}`}`, + ); +} + +function responses() { + const { tools } = listed(dir); + const byName = new Map(tools.map((tool) => [tool.name, tool])); + const calls = readdirSync(dir) + .filter((file) => /^call-\d+-.+\.json$/.test(file)) + .sort(); + for (const file of calls) { + const { request, result } = readJson(dir, file); + const text = result.content + .filter((item) => item.type === "text") + .map((item) => item.text) + .join("\n"); + const declaresSchema = Boolean(byName.get(request.name)?.outputSchema); + if (result.isError) { + check( + "mcp-tool-responses#tool-error", + result.structuredContent ? ["error result has structuredContent"] : [], + file, + ); + } else if (declaresSchema) { + const sc = result.structuredContent; + check( + "mcp-tool-responses#structured-text-match", + !sc + ? ["no structuredContent"] + : text === JSON.stringify(sc) + ? [] + : ["text is not the compact JSON of structuredContent"], + file, + ); + const actions = sc?.suggestedActions ?? []; + const problems = + actions.length > MAX_SUGGESTED_ACTIONS + ? [`${actions.length} actions`] + : []; + for (const action of actions) { + if (action.type !== "tool_call") problems.push(`type ${action.type}`); + if (!byName.has(action.toolName)) + problems.push(`${action.toolName} is not in tools/list`); + if (!action.arguments || Object.keys(action.arguments).length === 0) + problems.push(`${action.toolName} has no arguments`); + } + check( + "mcp-tool-responses#suggested-actions-callable", + problems, + `${file}, ${actions.length} actions`, + ); + if (Array.isArray(sc?.sideQuests)) { + check( + "mcp-tool-responses#side-quest-fix", + sc.sideQuests + .filter( + (quest) => !quest.tip || !quest.docs?.startsWith("https://"), + ) + .map((quest) => `${quest.id} has no tip or docs link`), + `${file}, ${sc.sideQuests.length} side quests`, + ); + } + } else { + const problems = []; + if (result.structuredContent) + problems.push("markdown tool returned structuredContent"); + if (!text.startsWith("# ")) + problems.push("text does not start with a # title"); + check("mcp-tool-responses#markdown-only", problems, file); + const found = PLACEHOLDERS.filter((token) => + new RegExp( + `(^|[^\\w])${token.replace(/[[\]]/g, "\\$&")}([^\\w]|$)`, + ).test(text), + ); + check( + "mcp-tool-responses#no-placeholders", + found.map((token) => `contains ${token}`), + file, + ); + } + } +} + +function surface() { + const { tools, nextCursor } = listed(dir); + const { instructions = "" } = readJson(dir, "initialize.json"); + check( + "mcp-tool-surface#lists-all", + GATEWAY_TOOLS.filter((name) => tools.some((tool) => tool.name === name)), + `${tools.length} tools`, + ); + check( + "mcp-tool-surface#hints-set", + tools.flatMap((tool) => + ["readOnlyHint", "destructiveHint", "openWorldHint"] + .filter((hint) => typeof tool.annotations?.[hint] !== "boolean") + .map((hint) => `${tool.name} has no ${hint}`), + ), + ); + const titles = tools.map((tool) => tool.title); + check( + "mcp-tool-surface#unique-titles", + tools + .filter((tool, i) => !tool.title || titles.indexOf(tool.title) !== i) + .map((tool) => `${tool.name}: ${tool.title ?? "no title"}`), + ); + check( + "mcp-tool-surface#one-page", + nextCursor ? [`nextCursor ${nextCursor}`] : [], + ); + check( + "mcp-tool-surface#instructions-short", + instructions.length > MAX_INSTRUCTIONS + ? [`${instructions.length} characters`] + : [], + `${instructions.length} characters`, + ); +} + +function readOnly() { + const writes = listed(dir).tools.filter( + (tool) => + !tool.annotations?.readOnlyHint && !EVERY_SKILL_WRITES.has(tool.name), + ); + check( + "mcp-access#inspect-read-only", + writes.map((tool) => `${tool.name} is not read-only`), + ); +} + +function scoped() { + const { tools } = listed(dir); + const problems = tools + .filter((tool) => tool.name === "find_organizations") + .map(() => "find_organizations is listed"); + for (const tool of tools) { + if (tool.inputSchema?.properties?.organizationSlug) + problems.push(`${tool.name} takes organizationSlug`); + } + check("mcp-access#scope-hides-org", problems); +} + +function subset() { + const wide = new Set(toolNames(dirs[0])); + const narrow = toolNames(dir); + const extra = narrow.filter((name) => !wide.has(name)); + const removed = [...wide].filter((name) => !narrow.includes(name)); + const problems = extra.map((name) => `${name} is not in ${dirs[0]}`); + if (removed.length === 0) problems.push("nothing was removed"); + check( + "mcp-access#skills-narrow", + problems, + `${narrow.length} of ${wide.size} tools; removed ${removed.join(", ")}`, + ); +} + +const modes = { responses, surface, "read-only": readOnly, scoped, subset }; +if (!modes[mode]) { + console.error(`unknown mode: ${mode}`); + process.exit(2); +} +modes[mode](); +console.log(lines.join("\n")); +appendFileSync(join(dir, "checks.txt"), `${lines.join("\n")}\n`); +process.exit(lines.some((line) => line.startsWith("FAIL")) ? 1 : 0); diff --git a/.agents/skills/verify-toolkit/scripts/mcp-call.mjs b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs new file mode 100755 index 0000000000..c61f173233 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs @@ -0,0 +1,132 @@ +#!/usr/bin/env node +// Drive a Sentry MCP stdio server the way an MCP client does, with no LLM. +// Writes initialize.json, tools-list.json, one call--.json per call, +// and server-stderr.txt to --out. See ../SKILL.md ("Drive"). +import { mkdirSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { offlineEnv } from "./offline-env.mjs"; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), "../../../.."); +const usage = + "usage: mcp-call.mjs --out DIR [--server mock|cli] [--no-token] [--server-arg ARG]... [--call TOOL JSON_ARGS]..."; + +const opts = { server: "mock", token: true, serverArgs: [], calls: [] }; +const argv = process.argv.slice(2); +for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === "--out") opts.out = argv[++i]; + else if (arg === "--server") opts.server = argv[++i]; + else if (arg === "--no-token") opts.token = false; + else if (arg === "--server-arg") opts.serverArgs.push(argv[++i]); + else if (arg === "--call") + opts.calls.push([argv[++i], JSON.parse(argv[++i])]); + else { + console.error(`unknown argument: ${arg}\n${usage}`); + process.exit(2); + } +} +if (!opts.out || !["mock", "cli"].includes(opts.server)) { + console.error(usage); + process.exit(2); +} + +// The SDK is a dependency of the test client, not of the repo root. +const requireFromClient = createRequire( + join(repo, "packages/mcp-test-client/package.json"), +); +const sdk = (path) => + import( + pathToFileURL( + requireFromClient.resolve(`@modelcontextprotocol/sdk/${path}`), + ).href + ); +const { Client } = await sdk("client/index.js"); +const { StdioClientTransport } = await sdk("client/stdio.js"); + +// Both servers get the offline env. The mock server answers the Sentry API +// from MSW fixtures and ignores it; `sentry mcp` gets a fake token (unless +// --no-token), an empty config dir, and an unreachable Sentry host. +const { env, cleanup } = offlineEnv({ token: opts.token }); +const [command, args] = + opts.server === "mock" + ? [ + process.execPath, + [ + "--experimental-strip-types", + "--import", + join(repo, "scripts/strip-types-resolve.mjs"), + join(repo, "packages/mcp-server-evals/src/bin/start-mock-stdio.ts"), + ...opts.serverArgs, + ], + ] + : [ + "pnpm", + [ + "--silent", + "--dir", + join(repo, "packages/cli"), + "run", + "cli", + "mcp", + ...opts.serverArgs, + ], + ]; + +mkdirSync(opts.out, { recursive: true }); +const write = (name, value) => + writeFileSync(join(opts.out, name), `${JSON.stringify(value, null, 2)}\n`); + +const transport = new StdioClientTransport({ + command, + args, + env, + cwd: repo, + stderr: "pipe", +}); +let stderr = ""; +transport.stderr?.on("data", (chunk) => { + stderr += chunk; +}); + +const client = new Client({ name: "verify-toolkit", version: "0.0.0" }); +let failed = false; +try { + await client.connect(transport); + write("initialize.json", { + server: opts.server, + command: [command, ...args].join(" "), + serverVersion: client.getServerVersion(), + capabilities: client.getServerCapabilities(), + instructions: client.getInstructions(), + }); + // Listing first also makes the SDK validate each call's structuredContent + // against the tool's outputSchema, as a real client does. + const { tools, nextCursor } = await client.listTools(); + write("tools-list.json", { tools, nextCursor: nextCursor ?? null }); + console.log( + `connected: ${client.getServerVersion()?.name} (${opts.server}), ${tools.length} tools`, + ); + for (const [n, [name, toolArgs]] of opts.calls.entries()) { + const file = `call-${n + 1}-${name}.json`; + const result = await client.callTool({ name, arguments: toolArgs }); + write(file, { request: { name, arguments: toolArgs }, result }); + const shape = result.structuredContent + ? "structuredContent" + : "content only"; + console.log( + `tools/call ${name}: ${result.isError ? "isError" : "ok"}, ${shape} -> ${file}`, + ); + } +} catch (error) { + failed = true; + console.error( + `mcp-call failed: ${error instanceof Error ? error.message : String(error)}`, + ); +} finally { + await client.close().catch(() => {}); + cleanup(); + writeFileSync(join(opts.out, "server-stderr.txt"), stderr); +} +process.exit(failed ? 1 : 0); diff --git a/.agents/skills/verify-toolkit/scripts/offline-env.mjs b/.agents/skills/verify-toolkit/scripts/offline-env.mjs new file mode 100755 index 0000000000..45db8fb4c2 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/offline-env.mjs @@ -0,0 +1,35 @@ +// Environment for driving `sentry` from source with no live Sentry account. +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +/** Any request to this host fails fast: nothing listens on the discard port. */ +const OFFLINE_SENTRY_URL = "http://127.0.0.1:9"; +export const OFFLINE_TOKEN = "sntryu_verify_toolkit_offline"; + +/** + * Copy of the caller's env with no real credentials, an empty CLI config dir, + * an unreachable Sentry host, and telemetry off. Call `cleanup()` when done. + */ +export function offlineEnv({ token }) { + const configDir = mkdtempSync(join(tmpdir(), "verify-toolkit-config-")); + const env = { ...process.env }; + for (const name of [ + "SENTRY_AUTH_TOKEN", + "SENTRY_ACCESS_TOKEN", + "SENTRY_TOKEN", + "SENTRY_HOST", + ]) { + delete env[name]; + } + Object.assign(env, { + SENTRY_CONFIG_DIR: configDir, + SENTRY_URL: OFFLINE_SENTRY_URL, + SENTRY_CLI_NO_TELEMETRY: "1", + }); + if (token) env.SENTRY_AUTH_TOKEN = OFFLINE_TOKEN; + return { + env, + cleanup: () => rmSync(configDir, { recursive: true, force: true }), + }; +} diff --git a/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs b/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs new file mode 100755 index 0000000000..148d0740e1 --- /dev/null +++ b/.agents/skills/verify-toolkit/scripts/sentry-offline.mjs @@ -0,0 +1,46 @@ +#!/usr/bin/env node +// Run the `sentry` CLI from source with no live Sentry account, and record +// command.txt, stdout.txt, stderr.txt, and exit-code.txt in --out. +// Usage: sentry-offline.mjs --out DIR [--token] -- ... +import { spawnSync } from "node:child_process"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { OFFLINE_TOKEN, offlineEnv } from "./offline-env.mjs"; + +const repo = resolve(dirname(fileURLToPath(import.meta.url)), "../../../.."); +const argv = process.argv.slice(2); +const split = argv.indexOf("--"); +const own = split === -1 ? argv : argv.slice(0, split); +const cliArgs = split === -1 ? [] : argv.slice(split + 1); +const outIndex = own.indexOf("--out"); +const out = outIndex === -1 ? undefined : own[outIndex + 1]; +if (!out || cliArgs.length === 0) { + console.error( + "usage: sentry-offline.mjs --out DIR [--token] -- ...", + ); + process.exit(2); +} + +const { env, cleanup } = offlineEnv({ token: own.includes("--token") }); +const run = spawnSync( + "pnpm", + ["--silent", "--dir", join(repo, "packages/cli"), "run", "cli", ...cliArgs], + { + cwd: repo, + env, + encoding: "utf8", + }, +); +cleanup(); + +mkdirSync(out, { recursive: true }); +writeFileSync(join(out, "command.txt"), `sentry ${cliArgs.join(" ")}\n`); +writeFileSync(join(out, "stdout.txt"), run.stdout); +writeFileSync(join(out, "stderr.txt"), run.stderr); +writeFileSync(join(out, "exit-code.txt"), `${run.status}\n`); +const leaked = `${run.stdout}${run.stderr}`.includes(OFFLINE_TOKEN); +console.log( + `sentry ${cliArgs.join(" ")}: exit ${run.status}${leaked ? ", PRINTED THE TOKEN" : ""}`, +); +process.exit(run.status ?? 1); diff --git a/AGENTS.md b/AGENTS.md index f44a0e687a..dc9b6823ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -135,7 +135,9 @@ pnpm run --filter @sentry/mcp-core generate-definitions For MCP tool QA, use the `mcp-qa` skill at `.agents/skills/mcp-qa/SKILL.md`: stdio-first local CLI and real agent clients; Cloudflare HTTP or `/mcp` only -for transport, OAuth, routing, or hosted-server compatibility. +for transport, OAuth, routing, or hosted-server compatibility. To prove MCP or +CLI behavior offline, with no Sentry account, use the `verify-toolkit` skill at +`.agents/skills/verify-toolkit/SKILL.md` and its claim map. ## Task Management diff --git a/docs/contributing/tool-responses.md b/docs/contributing/tool-responses.md index 9a5a248c88..6c47e210ef 100644 --- a/docs/contributing/tool-responses.md +++ b/docs/contributing/tool-responses.md @@ -339,8 +339,12 @@ they do not replace a full-response snapshot. For output-format changes: - Run the normal quality gate from [quality-checks.md](quality-checks.md). +- Capture the raw result offline with the `verify-toolkit` skill + ([.agents/skills/verify-toolkit/SKILL.md](../../.agents/skills/verify-toolkit/SKILL.md)); + its `mcp-tool-responses` map checks the rules every result follows. - Use the stdio MCP QA path in - [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md). + [.agents/skills/mcp-qa/SKILL.md](../../.agents/skills/mcp-qa/SKILL.md) + against prod data. - Inspect the raw MCP tool result when possible, not only the LLM's final answer. The test client's final answer can add model-specific phrasing that is not part of the tool response. From 965a213c956b0f7f780360f069325c674e9aa04a Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Mon, 16 Nov 2026 23:01:53 +0000 Subject: [PATCH 3/4] fix(skills): Isolate verify-toolkit from ~/.sentryclirc The CLI reads a token from ~/.sentryclirc when no env token is set, so the offline env now also points HOME at an empty temp dir. Without it the no-login drive could start authenticated. --- .agents/skills/verify-toolkit/SKILL.md | 2 +- .agents/skills/verify-toolkit/map/mcp-access.md | 2 +- .../skills/verify-toolkit/scripts/offline-env.mjs | 13 +++++++++---- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/.agents/skills/verify-toolkit/SKILL.md b/.agents/skills/verify-toolkit/SKILL.md index 7742a3f1d2..940c96abae 100644 --- a/.agents/skills/verify-toolkit/SKILL.md +++ b/.agents/skills/verify-toolkit/SKILL.md @@ -36,7 +36,7 @@ Ready means: `pnpm run build` exits 0, and `ensure-generated` exits 0 and leaves `packages/cli/src/generated/command-api.json` as valid JSON. Each drive starts its own server process and stops it when it finishes, so there is no long-running server and nothing to tear down. Runs are isolated: each -gets its own temporary CLI config dir, and two runs can go side by side. +gets its own temporary home and CLI config dir, and two runs can go side by side. Set a run directory for evidence, outside the checkout: diff --git a/.agents/skills/verify-toolkit/map/mcp-access.md b/.agents/skills/verify-toolkit/map/mcp-access.md index f1f6188374..b4c4a69011 100644 --- a/.agents/skills/verify-toolkit/map/mcp-access.md +++ b/.agents/skills/verify-toolkit/map/mcp-access.md @@ -33,7 +33,7 @@ Preconditions: ## Gotchas - The scope check at startup cannot reach the offline host, so `sentry mcp acme/` starts as if the org exists. That is the documented behavior for a network failure, not proof that `acme` exists. -- `--no-token` also needs the empty config dir that the script makes. Never point `SENTRY_CONFIG_DIR` at a real login for this drive. +- `--no-token` also needs the empty home and config dir that the script makes: the CLI reads a token from `~/.sentryclirc`. Never point `HOME` or `SENTRY_CONFIG_DIR` at a real login for this drive. ## Gaps diff --git a/.agents/skills/verify-toolkit/scripts/offline-env.mjs b/.agents/skills/verify-toolkit/scripts/offline-env.mjs index 45db8fb4c2..854f34cfd7 100755 --- a/.agents/skills/verify-toolkit/scripts/offline-env.mjs +++ b/.agents/skills/verify-toolkit/scripts/offline-env.mjs @@ -8,11 +8,14 @@ const OFFLINE_SENTRY_URL = "http://127.0.0.1:9"; export const OFFLINE_TOKEN = "sntryu_verify_toolkit_offline"; /** - * Copy of the caller's env with no real credentials, an empty CLI config dir, - * an unreachable Sentry host, and telemetry off. Call `cleanup()` when done. + * Copy of the caller's env with no real credentials, an empty home and CLI + * config dir, an unreachable Sentry host, and telemetry off. The empty home + * keeps the CLI from reading a token in `~/.sentryclirc`. Call `cleanup()` + * when done. */ export function offlineEnv({ token }) { - const configDir = mkdtempSync(join(tmpdir(), "verify-toolkit-config-")); + const home = mkdtempSync(join(tmpdir(), "verify-toolkit-home-")); + const configDir = join(home, ".sentry"); const env = { ...process.env }; for (const name of [ "SENTRY_AUTH_TOKEN", @@ -23,6 +26,8 @@ export function offlineEnv({ token }) { delete env[name]; } Object.assign(env, { + HOME: home, + USERPROFILE: home, SENTRY_CONFIG_DIR: configDir, SENTRY_URL: OFFLINE_SENTRY_URL, SENTRY_CLI_NO_TELEMETRY: "1", @@ -30,6 +35,6 @@ export function offlineEnv({ token }) { if (token) env.SENTRY_AUTH_TOKEN = OFFLINE_TOKEN; return { env, - cleanup: () => rmSync(configDir, { recursive: true, force: true }), + cleanup: () => rmSync(home, { recursive: true, force: true }), }; } From 274fda14a989c85975b3824f42f877664199c52a Mon Sep 17 00:00:00 2001 From: "sentry-junior[bot]" <264270552+sentry-junior[bot]@users.noreply.github.com> Date: Mon, 16 Nov 2026 23:19:10 +0000 Subject: [PATCH 4/4] fix(skills): Keep packages/cli/.env.local out of verify-toolkit drives pnpm run cli loads packages/cli/.env.local for unset variables, so a deleted token or host could come back from it. The offline env now sets the token variables to empty and SENTRY_HOST to the offline host; the mock server drops SENTRY_HOST because it answers for sentry.io. --- .agents/skills/verify-toolkit/map/mcp-access.md | 2 +- .agents/skills/verify-toolkit/scripts/mcp-call.mjs | 2 ++ .agents/skills/verify-toolkit/scripts/offline-env.mjs | 6 ++++-- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/.agents/skills/verify-toolkit/map/mcp-access.md b/.agents/skills/verify-toolkit/map/mcp-access.md index b4c4a69011..e441183086 100644 --- a/.agents/skills/verify-toolkit/map/mcp-access.md +++ b/.agents/skills/verify-toolkit/map/mcp-access.md @@ -33,7 +33,7 @@ Preconditions: ## Gotchas - The scope check at startup cannot reach the offline host, so `sentry mcp acme/` starts as if the org exists. That is the documented behavior for a network failure, not proof that `acme` exists. -- `--no-token` also needs the empty home and config dir that the script makes: the CLI reads a token from `~/.sentryclirc`. Never point `HOME` or `SENTRY_CONFIG_DIR` at a real login for this drive. +- `--no-token` also needs the empty home and config dir that the script makes: the CLI reads a token from `~/.sentryclirc`. The script also sets the token variables to empty, because `pnpm run cli` loads `packages/cli/.env.local` for any variable that is not set. Never point `HOME` or `SENTRY_CONFIG_DIR` at a real login for this drive. ## Gaps diff --git a/.agents/skills/verify-toolkit/scripts/mcp-call.mjs b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs index c61f173233..62d7790dad 100755 --- a/.agents/skills/verify-toolkit/scripts/mcp-call.mjs +++ b/.agents/skills/verify-toolkit/scripts/mcp-call.mjs @@ -49,6 +49,8 @@ const { StdioClientTransport } = await sdk("client/stdio.js"); // from MSW fixtures and ignores it; `sentry mcp` gets a fake token (unless // --no-token), an empty config dir, and an unreachable Sentry host. const { env, cleanup } = offlineEnv({ token: opts.token }); +// The mock server answers for its default host, sentry.io. +if (opts.server === "mock") delete env.SENTRY_HOST; const [command, args] = opts.server === "mock" ? [ diff --git a/.agents/skills/verify-toolkit/scripts/offline-env.mjs b/.agents/skills/verify-toolkit/scripts/offline-env.mjs index 854f34cfd7..3c0cdba9e5 100755 --- a/.agents/skills/verify-toolkit/scripts/offline-env.mjs +++ b/.agents/skills/verify-toolkit/scripts/offline-env.mjs @@ -17,18 +17,20 @@ export function offlineEnv({ token }) { const home = mkdtempSync(join(tmpdir(), "verify-toolkit-home-")); const configDir = join(home, ".sentry"); const env = { ...process.env }; + // Set, not deleted: `pnpm run cli` loads `packages/cli/.env.local` for + // unset variables only, and the CLI treats an empty token as unset. for (const name of [ "SENTRY_AUTH_TOKEN", "SENTRY_ACCESS_TOKEN", "SENTRY_TOKEN", - "SENTRY_HOST", ]) { - delete env[name]; + env[name] = ""; } Object.assign(env, { HOME: home, USERPROFILE: home, SENTRY_CONFIG_DIR: configDir, + SENTRY_HOST: OFFLINE_SENTRY_URL, SENTRY_URL: OFFLINE_SENTRY_URL, SENTRY_CLI_NO_TELEMETRY: "1", });