diff --git a/assets/flaky-tests/get-started/collection-overview-dark.png b/assets/flaky-tests/get-started/collection-overview-dark.png new file mode 100644 index 00000000..887897cf Binary files /dev/null and b/assets/flaky-tests/get-started/collection-overview-dark.png differ diff --git a/assets/flaky-tests/get-started/collection-overview-light.png b/assets/flaky-tests/get-started/collection-overview-light.png new file mode 100644 index 00000000..eb92ce22 Binary files /dev/null and b/assets/flaky-tests/get-started/collection-overview-light.png differ diff --git a/assets/flaky-tests/get-started/test-collections-and-repositories.svg b/assets/flaky-tests/get-started/test-collections-and-repositories.svg new file mode 100644 index 00000000..36b93e5f --- /dev/null +++ b/assets/flaky-tests/get-started/test-collections-and-repositories.svg @@ -0,0 +1,67 @@ + + Collections and repositories are many-to-many + Three uploads from two repositories land in two collections. The api repository splits its unit tests and end-to-end tests across two collections, and the Unit Tests collection receives uploads from both the api and web repositories. + + + + + REPOSITORIES + COLLECTIONS + + + api + unit tests + + + api + end-to-end tests + + + web + unit tests + + + Unit Tests + --test-collection-id Ab3Kd9Zq + + + E2E Tests + --test-collection-id Qz7Mp2Ln + + + + + + + + + + + One repository can split across collections — the api repository above sends its unit + tests to one collection and its end-to-end tests to another. One collection can span repositories. + diff --git a/assets/flaky-tests/migration-legacy-view-button-dark.png b/assets/flaky-tests/migration-legacy-view-button-dark.png new file mode 100644 index 00000000..7cf009e6 Binary files /dev/null and b/assets/flaky-tests/migration-legacy-view-button-dark.png differ diff --git a/assets/flaky-tests/migration-legacy-view-button-light.png b/assets/flaky-tests/migration-legacy-view-button-light.png new file mode 100644 index 00000000..6d660ae1 Binary files /dev/null and b/assets/flaky-tests/migration-legacy-view-button-light.png differ diff --git a/assets/flaky-tests/migration-test-collections-button-dark.png b/assets/flaky-tests/migration-test-collections-button-dark.png new file mode 100644 index 00000000..5bbddf58 Binary files /dev/null and b/assets/flaky-tests/migration-test-collections-button-dark.png differ diff --git a/assets/flaky-tests/migration-test-collections-button-light.png b/assets/flaky-tests/migration-test-collections-button-light.png new file mode 100644 index 00000000..9f02e639 Binary files /dev/null and b/assets/flaky-tests/migration-test-collections-button-light.png differ diff --git a/flaky-tests/get-started/test-collections.mdx b/flaky-tests/get-started/test-collections.mdx index 0137bcb5..f4a357a4 100644 --- a/flaky-tests/get-started/test-collections.mdx +++ b/flaky-tests/get-started/test-collections.mdx @@ -1,75 +1,116 @@ --- title: "Test Collections" -description: "Organize your flaky tests into named collections to track and analyze specific subsets of your test suite." +description: "Group tests into named collections, each with its own flake detection, quarantining, and ticketing settings." +og:title: "Organizing tests with collections in Trunk Flaky Tests" hidden: true --- -Test Collections let you group tests from any repository into named sets. Use collections to focus on a subset of your test suite, such as tests owned by a specific team, tests covering a critical service, or any grouping that matters to your workflow. +A test collection is a named group of tests with its own settings. Each collection has independent flake detection, quarantining, and ticketing, so you can hold fast unit tests and fragile end-to-end tests to different standards. -Each collection has its own view of tests, uploads, and settings, separate from the full test suite view. +You create a collection in the Trunk app, then point a CI job at it by passing the collection's ID to the uploader. + + + The Overview tab of a collection named trunk2-pr-e2e, showing its Collection ID, flaky and broken test counts, PRs impacted, three daily charts, and the top ten most unstable tests. + The Overview tab of a collection named trunk2-pr-e2e, showing its Collection ID, flaky and broken test counts, PRs impacted, three daily charts, and the top ten most unstable tests. + + +## Collections and repositories + +Collections and repositories are many-to-many. A collection can include tests from multiple repositories, and a repository can be broken into multiple collections. This is decided by which collection you specify at upload time. + + + Three repositories uploading into two collections: the api repository sends its unit tests to one collection and its end-to-end tests to another, while the unit test collection also receives uploads from the web repository. + + +## The Collection ID + +Every collection has an ID: eight alphanumeric characters, unique across all of Trunk. It appears in the collection's URL, on the collection's page header, and in the **ID** column of the collections list, each with a button to copy it. + +``` +https://app.trunk.io//flaky-tests/collections/ +``` + +The ID is what CI needs. Nothing else about a collection is used to route uploads. ## Create a collection -Only organization admins can create collections. +Any member of your organization can create a collection. -1. Navigate to **Flaky Tests** → **Collections** in the Trunk web app. +1. In Flaky Tests, open **Collections**. 2. Click **Create Collection**. -3. Enter a **Name** and optional **Description**. +3. Enter a **Collection name**, and a description if you want one. 4. Click **Create collection**. -After creation, you land on the collection detail page. The **Tests** and **Uploads** tabs are disabled until you upload test results to the collection. +You land on the new collection. Its **Tests** and **Uploads** tabs stay disabled until test results arrive. -## Upload tests to a collection +## What a new collection starts with -To populate a collection with test data, include the collection's short ID in your uploader configuration. The collection short ID appears in the URL when viewing the collection: +A new collection starts with a basic set of flake-detection monitors, so detection starts working as soon as test results arrive. -``` -https://app.trunk.io//flaky-tests/collections/ -``` +You can change them, and everything else about the collection, from its **Monitors** and **Settings** tabs. + +## Upload test results to a collection -Pass the short ID when uploading results using the Trunk CLI: +Pass the collection ID to the uploader alongside your organization slug. Both are required: ```bash -trunk flakytests upload --test-collection-short-id ... +./trunk-analytics-cli upload \ + --junit-paths "test_output.xml" \ + --org-url-slug \ + --test-collection-id \ + --token $TRUNK_API_TOKEN ``` -See the [Uploader reference](../reference/cli-reference) for full upload options. +You can set the ID as the `TRUNK_TEST_COLLECTION_ID` environment variable instead, or pass it as the `test-collection-id` input to the [uploader action](./ci-providers/github-actions). + +Everything else about your setup is unchanged. Adding the collection ID to a working upload step is the whole change. See [**Test Frameworks**](./frameworks/) and [**CI Providers**](./ci-providers/) for the setup itself, and the [**CLI reference**](../reference/cli-reference#test-collections) for the full flag list. + + +A CI job uploads to exactly one collection. To send results to more than one collection, add an upload step per collection. + + +## Collection tabs -## View collection tests and uploads +| Tab | What it holds | +| --- | --- | +| **Overview** | Setup instructions before results arrive; the collection's dashboard afterwards. | +| **Tests** | Every test in the collection, with health status, failure rates, and labels. | +| **Uploads** | Upload history for the collection. | +| **Quarantining** | Quarantine analytics for the collection. | +| **Monitors** | The collection's [flake detection monitors](../detection/). | +| **Settings** | The collection's name and description, [quarantining](../quarantining/), upload handling, and [ticketing](../management/ticketing/automatic-ticketing). | -Once tests are uploaded to a collection, the **Tests** and **Uploads** tabs become active on the collection detail page. +## Finish setting up a collection -* **Tests** tab: Shows all tests associated with this collection, with their flaky status, failure rates, and labels. -* **Uploads** tab: Shows the history of test uploads sent to this collection. -* **Overview** tab: Shows setup instructions and the upload configuration for this collection. -* **Tests** tab: Shows all tests associated with this collection, with their flaky status, failure rates, and labels. -* **Uploads** tab: Shows the history of test uploads sent to this collection. +Each collection has a setup checklist, on its **Overview** tab before results arrive and on its **Settings** tab permanently. It tracks four steps: -## Edit a collection +1. **Send your first upload** +2. **Ingest your test results** +3. **Review flake detection** +4. **Review quarantining** -Only organization admins can edit collection settings. +The first two complete on their own once uploads are flowing. The rest can be configured in the app. + +## Permissions + +| Action | Admin | Member | +| --- | --- | --- | +| View collections | Yes | Yes | +| Create a collection | Yes | Yes | +| Edit a collection's name or description | Yes | No | +| Change quarantining settings | Yes | No | +| Delete a collection | Yes | No | -1. Navigate to the collection detail page. -2. Click the **Settings** tab. -3. Update the **Name** or **Description**. -4. Click **Save changes**. ## Delete a collection -Only organization admins can delete collections. +Only organization admins can delete a collection. -1. Navigate to the collection's **Settings** tab. +1. Open the collection's **Settings** tab. 2. Click **Delete collection**. -3. Confirm deletion in the dialog. +3. Confirm in the dialog. -Deleting a collection removes it from the **Collections** list. Test data uploaded to the collection is not deleted from your overall test suite. - -## Permissions +The collection disappears from the collections list. Test results already uploaded to it are not deleted. -| Action | Admin | Member | -| ------------------------ | ----- | ------ | -| View collections | Yes | Yes | -| Create collection | Yes | No | -| Edit collection settings | Yes | No | -| Delete collection | Yes | No | +## Questions -Members can browse existing collections and view tests and uploads, but cannot create, edit, or delete collections. +Ask us in [Slack](https://slack.trunk.io) or email [support@trunk.io](mailto:support@trunk.io). diff --git a/flaky-tests/migrate-to-test-collections.mdx b/flaky-tests/migrate-to-test-collections.mdx new file mode 100644 index 00000000..0fb7f4e7 --- /dev/null +++ b/flaky-tests/migrate-to-test-collections.mdx @@ -0,0 +1,125 @@ +--- +title: "Migrate to Test Collections" +description: "Move an existing organization from repository-based Flaky Tests to test collections, one CI job at a time." +og:title: "Migrating to test collections in Trunk" +hidden: true +--- +[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you will need to gradually migrate to using collections, and both views stay available while you do. + +Nothing breaks while you migrate. A CI job that doesn't pass a collection ID keeps uploading exactly as it does today. + +## What changes + +| | Repository-based | Collections | +| --- | --- | --- | +| Where results land | Determined by your git remote | The collection ID you pass to the uploader | +| Flake detection | Monitors per repository | Monitors per collection | +| Quarantining | Settings per repository | Settings per collection | +| Ticketing | Integration per repository | Integrations per organization, selected per collection | +| Dashboards | Repository overview | Collection overview | +| History | Stays where it is | Starts at the collection's first upload | + +## Both views, while you migrate + +Flaky Tests opens on your test collections. To reach the repository view, click **Legacy view** on the collections list; to come back, click **Test collections** on the repository overview. Trunk remembers whichever you chose and opens there next time. + + + The Repositories header in Flaky Tests, with a Test collections button on the right. + The Repositories header in Flaky Tests, with a Test collections button on the right. + + + + The Collections list header in Flaky Tests, with a Legacy view button on the right, a collections search box, and a Create Collection button. + The Collections list header in Flaky Tests, with a Legacy view button on the right, a collections search box, and a Create Collection button. + + +## Nothing carries over from your repositories + +Monitors, quarantine overrides, ticketing configuration, and infrastructure-failure thresholds are all per-collection. A repository's configuration does not transfer to a collection, so anything you have tuned on a repository has to be set up again on the collection you want it on. + +That is often what you want. You may wish to create different monitors for each collection, such as more sensitive flakiness detection for unit tests than for end-to-end tests. + +## Migrate a CI job + + + +Collections usually map to a team, a service, or a test suite. Group tests you want to configure and review together. + +See [Create a collection](./get-started/test-collections#create-a-collection). + + + +Add `--test-collection-id ` to the job's existing upload step, alongside the `--org-url-slug` it already passes. Start with one job rather than all of them. + + + +Open the collection's **Uploads** tab. An upload appears as soon as Trunk accepts it, and fills in once the results are processed — so a bundle that contained no test results visibly stops at the first stage. The collection's **Tests** tab unlocks once test cases have been ingested. + + + +Open the collection's **Monitors** tab and check the seeded defaults against what you run on the repository today. Adjust thresholds and branch patterns here rather than assuming the repository's carried over. + + + +Quarantining is configured per collection, and enabling it replaces the repository's quarantining for every upload routed to that collection. Your repository's **Always Quarantine** and **Never Quarantine** overrides do not follow. + +Overrides can only be set once quarantining is enabled on the collection, so re-apply the ones that matter as soon as you turn it on. + + + +Move the rest of your CI jobs when you're ready. Uploads without a collection ID keep going to the repository view, so a partly-migrated organization is a normal state to sit in. + + + + +Enabling quarantining on a collection takes effect immediately, with no overrides in place. Between enabling it and re-applying your overrides, that collection's tests are governed by its own settings and nothing else — a test you had pinned to **Never Quarantine** on the repository is no longer pinned. + + +## Links in CI output + +Once a job passes a collection ID, the links the CLI prints at the end of a run point at the collection. To keep the repository links while your team is still moving over, pass `--hide-test-collection-links` or set `TRUNK_HIDE_TEST_COLLECTION_LINKS`. + +## Running both views at once + +While a repository and a collection both have monitors covering the same tests, they detect independently. That's the point — it's how you compare the two before committing — but it means: + +- A test can be flagged in both views, at different times, according to each one's thresholds. +- If you use webhooks, one detection can produce two events — see below. + +## Webhooks while you migrate + +Collection webhook events are **off by default while you are migrating**, and you switch them on from your organization's webhooks settings once your consumers are ready. The control appears only while you have both views, since that is the only time it changes anything — once you have fully migrated, collection events are simply sent. + +It is one switch for every collection event, not one per event type. What you are asserting by turning it on is that your consumer handles collection payloads at all: + +- A collection event carries a `test_collection` object. Repository events don't, and that is the only supported way to tell the two apart. +- **You cannot deduplicate on `test_case.id`.** Collection and repository events don't share a test-case ID space, so the same underlying test arrives under two unrelated IDs. `repository.id` is the only handle common to both. +- `repository` is omitted rather than guessed when an upload carries no repository. + +Leave it off until your consumer is ready and nothing reaches it in the meantime. See [Webhooks](./webhooks/) for the payloads. + +## Dashboards start fresh + +Collections don't backfill. A collection's metrics begin at its first upload, so a correctly configured collection can still look empty for a day, and comparisons against a repository's history aren't meaningful until the collection has accumulated its own. + +Your repository history stays where it is and isn't affected by migrating. + +## Frequently asked questions + + + +Collections don't backfill history — metrics start at the collection's first upload. If uploads are arriving and the dashboard is still empty, check the **Uploads** tab to confirm the results were processed and not just accepted. + + + +Quarantining has to be enabled on the collection, and a test has to have been detected by one of the collection's monitors. Check both on the collection's **Settings** tab and **Monitors** tab. + + + +Yes. Collections and repositories are many-to-many. A repository is part of a test's identity, but that test can be uploaded separately to multiple collections. + + + +## Questions + +Migrations turn up things docs don't cover. Ask us in [Slack](https://slack.trunk.io) or email [support@trunk.io](mailto:support@trunk.io). diff --git a/flaky-tests/reference/cli-reference.mdx b/flaky-tests/reference/cli-reference.mdx index 66903ede..e1ccbf1b 100644 --- a/flaky-tests/reference/cli-reference.mdx +++ b/flaky-tests/reference/cli-reference.mdx @@ -146,6 +146,27 @@ Variant names are displayed in brackets next to test names in your dashboard: +## Test Collections + +[Test collections](/flaky-tests/get-started/test-collections) group tests so they can be configured and reviewed together, with their own flake detection and quarantining. To upload into one, pass its ID with `--test-collection-id`: + +```sh Upload to a test collection +./trunk-analytics-cli upload --junit-paths "test_output.xml" \ + --org-url-slug \ + --test-collection-id \ + --token $TRUNK_API_TOKEN +``` + +The collection ID is eight alphanumeric characters, and you copy it from the collection in the Trunk app. `--org-url-slug` is still required — the collection ID is an addition to it, not a replacement. + +You can set the ID as the `TRUNK_TEST_COLLECTION_ID` environment variable instead, and the [uploader action](/flaky-tests/get-started/ci-providers/github-actions) takes it as the `test-collection-id` input. + + +Each invocation uploads to a single collection. To send the same results to more than one collection, run the upload once per collection. + + +When a collection ID is passed, the links printed at the end of a run point at the collection rather than the repository. Pass `--hide-test-collection-links` to keep the repository links. + ## Running and Quarantining Tests You can also execute tests and upload results to Trunk in a single step using the `test` command to **wrap** your test command. @@ -257,6 +278,8 @@ The `upload` and `test` commands accept the following options: | `--use-bazel-target-for-codeowners` | When uploading a Bazel BEP file, use the Bazel target as a fallback path to associate test cases to codeowners. **Optional**. | | `--allow-empty-test-results` | Don't fail commands if test results are empty or missing. Use it when you sometimes skip all tests for certain CI jobs. Defaults to `true`. | | `--variant ` | Upload tests to a specific variant group. **Optional**. | +| `--test-collection-id ` | Upload to a [test collection](/flaky-tests/get-started/test-collections), in addition to (not instead of) `--org-url-slug`. Can also be set with the `TRUNK_TEST_COLLECTION_ID` environment variable. **Optional**. | +| `--hide-test-collection-links` | Print repository links at the end of a run instead of test collection links. Only has an effect when a collection ID is passed. Can also be set with the `TRUNK_HIDE_TEST_COLLECTION_LINKS=true` environment variable. **Optional**. | | `--test-process-exit-code` `` | Specify the exit code of the test previously run. This is used by the upload command to identify errors that happen outside of the context of the test execution (such as build errors). | | `--dry-run` | Write the test results bundle to a local `./bundle_upload` directory instead of uploading it to Trunk. [Quarantining](../quarantining/#quarantining-without-uploading) still runs and still determines the exit code. Can also be set with the `TRUNK_DRY_RUN=true` environment variable. **Optional**. |