Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -431,6 +431,7 @@
]
},
"flaky-tests/reference/mcp-reference/search-test",
"flaky-tests/reference/mcp-reference/list-test-collections",
"flaky-tests/reference/mcp-reference/fix-flaky-test",
"flaky-tests/reference/mcp-reference/set-up-test-uploads",
"flaky-tests/reference/mcp-reference/investigate-ci-failure"
Expand Down
9 changes: 9 additions & 0 deletions flaky-tests/get-started/migrate-to-test-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,15 @@ It is one switch for every collection event, not one per event type. What you ar

Leave it off until your consumer is ready and nothing reaches it in the meantime. See [Webhooks](../webhooks/) for the payloads.

## AI agents and the MCP server

The [Trunk MCP server](../reference/mcp-reference/index) reads the same places the Trunk app shows you:

- **While you are migrating**, [`search-test`](../reference/mcp-reference/search-test) and [`investigate-ci-failure`](../reference/mcp-reference/investigate-ci-failure) read both your repositories and your collections. A test can appear once in each, under two different IDs.
- **Once you have fully migrated**, they read only your collections, and [`fix-flaky-test`](../reference/mcp-reference/fix-flaky-test) is not available yet, because it works only with repository test IDs.

[`list-test-collections`](../reference/mcp-reference/list-test-collections) lists the collections a repository uploads to, and `search-test` can search just one of them.

## Dashboards start fresh

Collections don't backfill. A collection's metrics begin at its first upload, so a correctly configured collection can still look empty for a day, and comparisons against a repository's history aren't meaningful until the collection has accumulated its own.
Expand Down
5 changes: 5 additions & 0 deletions flaky-tests/reference/mcp-reference/fix-flaky-test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ The `fix-flaky-test` tool retrieves insights and historical failure analysis abo

**Return Type:** Structured analysis data with fix recommendations. Structure: metadata, summary, facts

<Note>
`fix-flaky-test` does not yet support [test collections](/flaky-tests/test-collections). It works with the test IDs [`search-test`](./search-test) returns for the repository, not for a test collection. If your organization has fully moved to test collections, the tool says so instead of returning an investigation.
</Note>

## Parameters

### Required Parameters
Expand Down Expand Up @@ -75,4 +79,5 @@ Retrieve the investigation for test <testCaseId> with investigationId <investiga
| `Please provide either \`investigationId\` or \`testCaseId\`.` | Neither `investigationId` nor `testCaseId` was provided | Provide one of `investigationId` or `testCaseId` |
| `This investigation was skipped before producing a completed summary.` | The repository or test reached its monthly [investigation limit](../../agents/autofix-flaky-tests#investigation-limits), or investigations are disabled for the repository | Start the investigation by hand from the test's detail page, which is exempt from both limits, or revisit the prerequisites in [Autofix Flaky Tests](../../agents/autofix-flaky-tests) |
| `This investigation failed before producing a completed summary. Please contact Trunk support.` | Investigation error | This feature is still in Beta, please contact support |
| `... cannot investigate a test in a test collection yet` | Your organization has fully moved to [test collections](/flaky-tests/test-collections) | Use [`search-test`](./search-test) to see the test's status and quarantine state in its collection |
| Repository authorization error | Insufficient permissions or invalid repo name | Verify repository name format and your access permissions |
1 change: 1 addition & 0 deletions flaky-tests/reference/mcp-reference/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ The Trunk MCP server is available at `https://mcp.trunk.io/mcp` and exposes the
| Tool | Capability |
|---|---|
| [`search-test`](./search-test) | Experimental: Lookup the id of a test case |
| [`list-test-collections`](./list-test-collections) | List the test collections a repository uploads to |
| [`fix-flaky-test`](./fix-flaky-test) | Experimental: Retrieve insights around a failing/flaky test |
| [`investigate-ci-failure`](./investigate-ci-failure) | Experimental: Retrieve failing test logs from a CI run |
| [`setup-trunk-uploads`](./set-up-test-uploads) | Create a setup plan to upload test results |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The `investigate-ci-failure` tool investigates a failing CI run by fetching stru

## Prerequisites

- Your repository must be set up to [upload test results to Trunk](../../get-started/index)
- Your repository must be set up to [upload test results to Trunk](../../get-started/index), with or without a [test collection](/flaky-tests/test-collections)
- For best results, [enable quarantining](../../quarantining/) so known-flaky tests are filtered out automatically

## Parameters
Expand Down Expand Up @@ -48,7 +48,7 @@ Investigate the CI failure at https://github.com/trunk-io/trunk/actions/runs/123

## What the tool does

- Looks up test result uploads Trunk has received for that run
- Looks up test result uploads Trunk has received for that run, including uploads to [test collections](/flaky-tests/test-collections). An upload recorded for both the repository and a collection while you [migrate](/flaky-tests/get-started/migrate-to-test-collections) is only read once
- Parses the test runs to extract test names, error messages, stdout and stderr
- Filters out quarantined (known-flaky) tests so you only see real failures
- Returns structured failure details you can act on
Expand All @@ -62,3 +62,6 @@ Investigate the CI failure at https://github.com/trunk-io/trunk/actions/runs/123
| `Invalid workflow URL` | Malformed or incorrect workflow URL | Verify the URL follows the pattern `https://github.com/{owner}/{repo}/actions/runs/{runId}` |
| `No test results were uploaded for this CI run` | No test run uploads were uploaded from the provided workflow | Check that the workflow run URL is correct and that it is uploading test results. Compilation and build failures will not upload test results |
| `No test uploads found for this repository` | Repo hasn't configured Trunk test result uploads | Follow setup instructions to [upload test results](../../get-started/index) |
| `No test uploads from this repository reached its Trunk test collections in the last 45 days` | Your organization uses test collections, and the repository has not uploaded to one recently | Check the job passes a [Collection ID](/flaky-tests/test-collections#upload-test-results-to-a-collection) |
| `No test results were found for this CI run in the stores Trunk could read` | Part of Trunk's test result storage was unavailable, so the run's results could not be ruled out | Try again shortly, or pull the CI logs from the workflow URL |
| `Trunk could not read ... just now` (after a report) | Part of Trunk's test result storage was unavailable, so some of the run's results may be missing | Try again shortly |
55 changes: 55 additions & 0 deletions flaky-tests/reference/mcp-reference/list-test-collections.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: "List Test Collections"
description: "MCP tool reference: list-test-collections"
og:title: "Trunk MCP tool: List Test Collections"
---
## Overview

The `list-test-collections` tool lists the [test collections](/flaky-tests/test-collections) a repository uploads test results to. A repository's tests can be split across several collections, by language, suite, or product area, and each holds different tests. Use this tool to find a collection's ID, then pass it to [`search-test`](./search-test) to search only that collection.

**Return Type:** Each collection's Collection ID, name, and first and last upload time, most recently uploaded first.

## Parameters

### Required Parameters

| Parameter | Type | Description |
| ---------- | ------ | --------------------------------------------------------------- |
| `repoName` | string | Repository name in `owner/repo` format (e.g., `trunk-io/trunk`) |

### Optional Parameters

| Parameter | Type | Description |
| --------- | ------ | ---------------------------------------------------------------------- |
| `orgSlug` | string | The name of your organization in the Trunk app |

<Note>
Only collections the repository has uploaded to in the last 45 days are listed. A collection that has not received an upload from the repository in that time is not shown.
</Note>

## Getting Parameter Values

If your AI assistant doesn't have direct access to Git information, use these commands:

**Get repository name:**

```bash
git remote -v
```

Look for the repository name in the output (e.g., `trunk-io/trunk` from `git@github.com:trunk-io/trunk.git`)

## Usage Examples

### List collections

```
Which Trunk test collections does this repository upload to?
```

## Error Handling

| Error | Cause | Resolution |
| ------------------------------ | --------------------------------------------- | --------------------------------------------------------- |
| `{repoName} has uploaded no test results to a Trunk test collection in the last 45 days.` | The repository uploads without a Collection ID, or has not uploaded recently | See [Upload test results to a collection](/flaky-tests/test-collections#upload-test-results-to-a-collection) |
| Repository authorization error | Insufficient permissions or invalid repo name | Verify repository name format and your access permissions |
32 changes: 26 additions & 6 deletions flaky-tests/reference/mcp-reference/search-test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,9 @@ og:title: "Trunk MCP tool: Search Test"
---
## Overview

The `search-test` tool looks up the ID of a test case given its name.
The `search-test` tool looks up a test by part of its name, class, or file. It searches the repository's tests and, if your organization uses [test collections](/flaky-tests/test-collections), the repository's tests in every collection.


**Return Type:** Metadata about the test, including its ID.
**Return Type:** Metadata about each matching test, including its ID and status. Results from test collections are listed in their own section, with the test's ID, its Collection ID, and whether it is quarantined.

## Parameters

Expand All @@ -17,14 +16,27 @@ The `search-test` tool looks up the ID of a test case given its name.
| Parameter | Type | Description |
| ---------- | ------ | --------------------------------------------------------------- |
| `repoName` | string | Repository name in `owner/repo` format (e.g., `trunk-io/trunk`) |
| `testNameSearch` | string | Search string for the test name. Does not include filepaths |
| `testNameSearch` | string | Case-insensitive part of the test's name, class, or file |

### Optional Parameters

| Parameter | Type | Description |
| --------- | ------ | ---------------------------------------------------------------------- |
| `orgSlug` | string | The name of your organization in the Trunk app |
| `limit` | number | Limit for test results to return, up to 20 |
| `testCollectionShortId` | string | The [Collection ID](/flaky-tests/test-collections#the-collection-id) of one test collection to search. Get it from [`list-test-collections`](./list-test-collections). Omit to search every collection |
| `limit` | number | Maximum number of results to return, up to 20, from the repository and again from its test collections. Defaults to 5 |

## Which tests are searched

What the tool searches follows your organization's [migration to test collections](/flaky-tests/get-started/migrate-to-test-collections), and matches what the Trunk app shows you:

| Organization | Searched |
| --- | --- |
| Not using test collections | The repository's tests |
| Migrating to test collections | The repository's tests and its tests in every collection. The same test can appear in both lists |
| Fully on test collections | The repository's tests in every collection |

Passing `testCollectionShortId` searches only that collection.

## Getting Parameter Values

Expand All @@ -46,9 +58,17 @@ Look for the repository name in the output (e.g., `trunk-io/trunk` from `git@git
What's the test case ID for the test "clear all filters button appears in empty state and clears filters"
```

### Search one test collection

```
Find the "checkout" tests in the backend-unit test collection
```

## Error Handling

| Error | Cause | Resolution |
| ------------------------------ | --------------------------------------------- | --------------------------------------------------------- |
| `No tests matched {searchString} in repo {repoName}` | No results found | Check your search string and try again |
| `No tests matched {searchString} in {repoName}` | No results found | Check your search string and try again |
| `No tests matched {searchString} in {repoName} in test collection {collectionId}` | No results in that collection, or no collection with that ID in your organization | Check the Collection ID with [`list-test-collections`](./list-test-collections) |
| `Trunk could not read ... just now` | One of the places Trunk stores test results was unavailable, so results from it may be missing | Try again shortly |
| Repository authorization error | Insufficient permissions or invalid repo name | Verify repository name format and your access permissions |
Loading