Skip to content

docs(flaky-tests): add the collection id to every setup instruction - #323

Draft
TylerJang27 wants to merge 3 commits into
mainfrom
test-collections/setup-instructions
Draft

docs(flaky-tests): add the collection id to every setup instruction#323
TylerJang27 wants to merge 3 commits into
mainfrom
test-collections/setup-instructions

Conversation

@TylerJang27

@TylerJang27 TylerJang27 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Important

Do not land until the default test_collection_status enum is flipped. These pages are already live, so merging earlier would tell every reader to pass a collection ID before collections are the default — including organizations for which they are not yet the right instruction.

Overview

PR 2 of the Test Collections docs stack: every framework and CI-provider setup page gains the collection ID beside the org identifier it already passes.

Details

Stacked on #322 (test-collections/concept-and-migration-pages) — review that one first.

52 examples across 33 pages. Each insertion matches the page's own placeholder style rather than normalising them:

Style Pages Becomes
--org-url-slug <TRUNK_ORG_URL_SLUG> 35 --test-collection-id <TRUNK_TEST_COLLECTION_ID>
--org-url-slug $TRUNK_ORG_URL_SLUG 4 --test-collection-id $TRUNK_TEST_COLLECTION_ID
org-slug: (uploader action) 7 test-collection-id:
${bamboo.TRUNK_ORG_URL_SLUG} 3 ${bamboo.TRUNK_TEST_COLLECTION_ID}
$(TRUNK_ORG_URL_SLUG) (Azure) 3 $(TRUNK_TEST_COLLECTION_ID)

Considerations

  • The collection ID is additive, never a swap. Every example keeps its org identifier — the CLI requires --org-url-slug regardless, and the uploader action marks org-slug required: true. This is the same invariant trunk-io/trunk2#5830 fixes in the in-product snippet generators, so the app and the docs now produce matching commands.
  • The prose lives in one string, not 33 copies. 23 of these pages share one exact sentence about where to find the org slug and token; that sentence now covers the collection ID and links to the concept page. The remaining pages carry a self-describing placeholder and already link onward to the CLI reference — adding a paragraph to each was the version most likely to drift.
  • trunk-analytics-cli test was verified, not assumed. Several pages document the test subcommand rather than upload. Both accept --test-collection-id, confirmed against the shipped binary — had test not taken it, this sweep would have broken those pages in exactly the way #5830 describes.
  • A bug worth naming, because it nearly shipped. The first pass added a shell line-continuation \ to the uploader action's YAML lines, wherever org-slug: happened to be the last key in its with: block. YAML has no continuations, so those blocks became invalid — and it was invisible in a unified diff, which shows the added line but not the modified one above it. The rule now applies continuations only to shell flags. Verified afterwards that no fenced block ends on a dangling \, and no YAML line carries one.
  • Placeholder rather than a literal, even where the neighbour is literal. The GitHub Actions page shows org-slug: my-trunk-org-slug, but a collection ID is eight opaque characters that must be copied from the app, so a literal example would read as a value to use. The same blocks already use <TEST OUTPUT PATH>, so the style is consistent locally.
  • The org slug is a variable, not a secret. github-actions.mdx stored it as secrets.TRUNK_ORG_URL_SLUG; only the API token is a credential, so it is now vars.. Every remaining secrets.TRUNK_* reference in the product docs is the API token.
  • quarantining/index.mdx and multiple-repositories.mdx are in scope after all, rather than deferred to PR 3, so the sweep is complete in one place. That added ten examples, including an --org-url-slug=[org] equals-style pair the original regex could not match.
  • Scope correction from the plan. It expected a settings-path sentence on seven monitor pages; only pass-on-retry-monitor.mdx has one. That one now names the collection Monitors tab alongside the repository path. Nothing else on the detection pages needed changing.
  • These pages are already live. Unlike docs(flaky-tests): add the test collections concept and migration pages #322, nothing here can be hidden, so this is the PR where collections first become customer-visible — before the hidden pages are unhidden. Worth a deliberate decision on whether it lands before or after PRs 3–4.

Testing

  • Verified no fenced code block ends on a dangling continuation and no YAML line carries a \, across every page under flaky-tests/get-started/.
  • Spot-checked one page per placeholder style, plus the multi-command pages (github-actions.mdx at 9 occurrences, xctest.mdx at 4) for indentation and continuation handling.
  • Not previewed with mint dev. Worth doing before this leaves draft — the risk here is a code block that renders wrong rather than prose that reads wrong.

Follow-up Work

PRs 3–5 of the stack: the straggler copy (quarantining, ticketing, webhooks, PR comments, the API reference), the dashboard replacement, and the flip that unhides #322's pages.

References

🤖 Generated with Claude Code

A common pattern is to run one monitor scoped to stable branches (like `main`) with a shorter recovery period for fast feedback, and a second monitor scoped to `release/*` branches with a longer recovery period for builds where intermittent failures are more expensive to re-investigate.

To add a monitor, navigate to **Settings** in the Trunk web app, open the repository, open the **Monitors** tab, and click **Add monitor**. Each monitor you create appears as a separate row in the monitors table and can be individually enabled, disabled, or deleted.
To add a monitor, open the **Monitors** tab — on a [test collection](/flaky-tests/get-started/test-collections), or on a repository under **Settings** and click **Add monitor**. Each monitor you create appears as a separate row in the monitors table and can be individually enabled, disabled, or deleted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally incorrect. repository monitors are under the monitors tab, not the settings tab.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and it was wrong before this stack touched it — repository monitors are at /flaky-tests/repo/{repoId}/monitor, a Monitors tab, with nothing monitor-related under Settings. My rewrite preserved the error while adding the collection scope beside it.

Now reads: "To add a monitor, open the Monitors tab — on a test collection or on a repository — and click Add monitor." The path is the same for both scopes, so it needs no fork.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to update the other rspec instructions (the ruby gem) to set the test collection env var. I think you missed that one

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caught a bigger gap than the one page — thank you. The whole env-var lane was missed, because the gem takes no CLI flags and my sweep keyed on --org-url-slug.

That's 11 occurrences: the gem example on nine CI-provider pages (buildkite, circleci, droneci, github-actions, gitlab, jenkins, semaphoreci, travisci, bitbucket, azure), plus rspec/index.mdx and otherci.mdx — neither of which the sweep touched at all, since neither carries the flag.

Confirmed the gem reads it before adding it: TRUNK_TEST_COLLECTION_ID is documented in rspec-trunk-flaky-tests/lib/trunk_spec_helper.rb. Each invocation now sets it in that page's own style — inline beside the other two assignments, or as a continued line where the example is already split — and rspec/index.mdx's "as long as you have set…" note names it too.

For otherci.mdx I put it where the slug and token are established, calling out the two ways it differs: it isn't a secret, and it's per job rather than per organization.

One judgement call worth a look: on github-actions.mdx the gem line reads secrets.TRUNK_ORG_URL_SLUG, and I used vars.TRUNK_TEST_COLLECTION_ID rather than secrets. — a collection ID isn't secret, and the uploader-action examples elsewhere use vars.. Say the word if you'd rather it match its neighbour.

TylerJang27 and others added 3 commits September 9, 2026 18:40
The framework and CI-provider pages each carry their own upload command,
and none mentioned a collection — so a reader following the RSpec page
verbatim landed an upload nowhere they had been told to look. Collections
are the default for new organizations, so these are now wrong by default
rather than merely incomplete.

52 examples across 33 pages gain `--test-collection-id` (or the
uploader action's `test-collection-id` input) **beside** the org
identifier they already pass, matching each page's own placeholder style
— `<TRUNK_ORG_URL_SLUG>`, `$TRUNK_ORG_URL_SLUG`, Bamboo's
`${bamboo.…}`, Azure's `$(…)`.

The 23 pages that share one sentence explaining where to find the org
slug and token now explain the collection ID in the same place, so the
prose lives in one string rather than 23 copies.

The `test` subcommand was checked as well as `upload` — both accept the
flag, so the pages documenting `trunk-analytics-cli test` are safe.

Scope correction: the plan expected a settings-path sentence on seven
monitor pages. Only `pass-on-retry-monitor.mdx` has one; it now names the
collection Monitors tab alongside the repository path.

These pages are already live, so this is where collections first become
customer-visible — ahead of the hidden pages being unhidden.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review fixes.

**The env-var lane was missed entirely.** The RSpec gem takes no CLI
flags — it reads `TRUNK_ORG_URL_SLUG` and `TRUNK_API_TOKEN` from the
environment — so the flag sweep skipped every one of its invocations.
That is 11 occurrences: the gem example on each of nine CI-provider
pages, plus `rspec/index.mdx` and `otherci.mdx`, which the sweep never
touched at all because neither carries an `--org-url-slug` flag.

The gem does read `TRUNK_TEST_COLLECTION_ID` (confirmed in
`rspec-trunk-flaky-tests/lib/trunk_spec_helper.rb`), so each invocation
now sets it in that page's own style — inline beside the other two
assignments, or as a continued line where the example is already split.

`otherci.mdx` gets it where the slug and token are established, with the
two things that differ: it is not a secret, and it is per job rather than
per organization.

The monitors path was wrong before this stack touched it — repository
monitors live under the **Monitors** tab, not **Settings** — so the
sentence now names the tab alone and covers both scopes.

`quarantining/index.mdx` and `multiple-repositories.mdx` also carry
upload commands, and are deliberately left to the next PR in the stack,
which rewrites both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wo pages

`github-actions.mdx` stored the org slug as `secrets.TRUNK_ORG_URL_SLUG`.
Only the API token is a credential, so it is now `vars.` — which also
makes the collection ID beside it consistent. Audited the rest: every
remaining `secrets.TRUNK_*` reference is the API token.

`quarantining/index.mdx` and `multiple-repositories.mdx` were being left
to the next PR; they get the collection ID here instead, so the sweep is
complete in one place. Ten examples across the two, including the
`--org-url-slug=[org]` equals-style pair the original sweep's regex could
not match — it required a space separator.

Every page carrying an org identifier now carries a collection ID.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@TylerJang27
TylerJang27 changed the base branch from test-collections/concept-and-migration-pages to main September 9, 2026 18:40
@trunk-io

trunk-io Bot commented Sep 9, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@TylerJang27
TylerJang27 force-pushed the test-collections/setup-instructions branch from 3efd41f to 8938898 Compare September 9, 2026 18:41
@TylerJang27
TylerJang27 added this pull request to stack #327 September 9, 2026 18:45
@mintlify

mintlify Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
trunk 🟢 Ready View Preview Sep 9, 2026, 6:58 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant