docs(flaky-tests): add the collection id to every setup instruction - #323
docs(flaky-tests): add the collection id to every setup instruction#323TylerJang27 wants to merge 3 commits into
Conversation
| A common pattern is to run one monitor scoped to stable branches (like `main`) with a shorter recovery period for fast feedback, and a second monitor scoped to `release/*` branches with a longer recovery period for builds where intermittent failures are more expensive to re-investigate. | ||
|
|
||
| To add a monitor, navigate to **Settings** in the Trunk web app, open the repository, open the **Monitors** tab, and click **Add monitor**. Each monitor you create appears as a separate row in the monitors table and can be individually enabled, disabled, or deleted. | ||
| To add a monitor, open the **Monitors** tab — on a [test collection](/flaky-tests/get-started/test-collections), or on a repository under **Settings** — and click **Add monitor**. Each monitor you create appears as a separate row in the monitors table and can be individually enabled, disabled, or deleted. |
There was a problem hiding this comment.
This was originally incorrect. repository monitors are under the monitors tab, not the settings tab.
There was a problem hiding this comment.
You're right, and it was wrong before this stack touched it — repository monitors are at /flaky-tests/repo/{repoId}/monitor, a Monitors tab, with nothing monitor-related under Settings. My rewrite preserved the error while adding the collection scope beside it.
Now reads: "To add a monitor, open the Monitors tab — on a test collection or on a repository — and click Add monitor." The path is the same for both scopes, so it needs no fork.
There was a problem hiding this comment.
We need to update the other rspec instructions (the ruby gem) to set the test collection env var. I think you missed that one
There was a problem hiding this comment.
Caught a bigger gap than the one page — thank you. The whole env-var lane was missed, because the gem takes no CLI flags and my sweep keyed on --org-url-slug.
That's 11 occurrences: the gem example on nine CI-provider pages (buildkite, circleci, droneci, github-actions, gitlab, jenkins, semaphoreci, travisci, bitbucket, azure), plus rspec/index.mdx and otherci.mdx — neither of which the sweep touched at all, since neither carries the flag.
Confirmed the gem reads it before adding it: TRUNK_TEST_COLLECTION_ID is documented in rspec-trunk-flaky-tests/lib/trunk_spec_helper.rb. Each invocation now sets it in that page's own style — inline beside the other two assignments, or as a continued line where the example is already split — and rspec/index.mdx's "as long as you have set…" note names it too.
For otherci.mdx I put it where the slug and token are established, calling out the two ways it differs: it isn't a secret, and it's per job rather than per organization.
One judgement call worth a look: on github-actions.mdx the gem line reads secrets.TRUNK_ORG_URL_SLUG, and I used vars.TRUNK_TEST_COLLECTION_ID rather than secrets. — a collection ID isn't secret, and the uploader-action examples elsewhere use vars.. Say the word if you'd rather it match its neighbour.
The framework and CI-provider pages each carry their own upload command,
and none mentioned a collection — so a reader following the RSpec page
verbatim landed an upload nowhere they had been told to look. Collections
are the default for new organizations, so these are now wrong by default
rather than merely incomplete.
52 examples across 33 pages gain `--test-collection-id` (or the
uploader action's `test-collection-id` input) **beside** the org
identifier they already pass, matching each page's own placeholder style
— `<TRUNK_ORG_URL_SLUG>`, `$TRUNK_ORG_URL_SLUG`, Bamboo's
`${bamboo.…}`, Azure's `$(…)`.
The 23 pages that share one sentence explaining where to find the org
slug and token now explain the collection ID in the same place, so the
prose lives in one string rather than 23 copies.
The `test` subcommand was checked as well as `upload` — both accept the
flag, so the pages documenting `trunk-analytics-cli test` are safe.
Scope correction: the plan expected a settings-path sentence on seven
monitor pages. Only `pass-on-retry-monitor.mdx` has one; it now names the
collection Monitors tab alongside the repository path.
These pages are already live, so this is where collections first become
customer-visible — ahead of the hidden pages being unhidden.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review fixes. **The env-var lane was missed entirely.** The RSpec gem takes no CLI flags — it reads `TRUNK_ORG_URL_SLUG` and `TRUNK_API_TOKEN` from the environment — so the flag sweep skipped every one of its invocations. That is 11 occurrences: the gem example on each of nine CI-provider pages, plus `rspec/index.mdx` and `otherci.mdx`, which the sweep never touched at all because neither carries an `--org-url-slug` flag. The gem does read `TRUNK_TEST_COLLECTION_ID` (confirmed in `rspec-trunk-flaky-tests/lib/trunk_spec_helper.rb`), so each invocation now sets it in that page's own style — inline beside the other two assignments, or as a continued line where the example is already split. `otherci.mdx` gets it where the slug and token are established, with the two things that differ: it is not a secret, and it is per job rather than per organization. The monitors path was wrong before this stack touched it — repository monitors live under the **Monitors** tab, not **Settings** — so the sentence now names the tab alone and covers both scopes. `quarantining/index.mdx` and `multiple-repositories.mdx` also carry upload commands, and are deliberately left to the next PR in the stack, which rewrites both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wo pages `github-actions.mdx` stored the org slug as `secrets.TRUNK_ORG_URL_SLUG`. Only the API token is a credential, so it is now `vars.` — which also makes the collection ID beside it consistent. Audited the rest: every remaining `secrets.TRUNK_*` reference is the API token. `quarantining/index.mdx` and `multiple-repositories.mdx` were being left to the next PR; they get the collection ID here instead, so the sweep is complete in one place. Ten examples across the two, including the `--org-url-slug=[org]` equals-style pair the original sweep's regex could not match — it required a space separator. Every page carrying an org identifier now carries a collection ID. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
3efd41f to
8938898
Compare
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Important
Do not land until the default
test_collection_statusenum is flipped. These pages are already live, so merging earlier would tell every reader to pass a collection ID before collections are the default — including organizations for which they are not yet the right instruction.Overview
PR 2 of the Test Collections docs stack: every framework and CI-provider setup page gains the collection ID beside the org identifier it already passes.
Details
Stacked on #322 (
test-collections/concept-and-migration-pages) — review that one first.52 examples across 33 pages. Each insertion matches the page's own placeholder style rather than normalising them:
--org-url-slug <TRUNK_ORG_URL_SLUG>--test-collection-id <TRUNK_TEST_COLLECTION_ID>--org-url-slug $TRUNK_ORG_URL_SLUG--test-collection-id $TRUNK_TEST_COLLECTION_IDorg-slug:(uploader action)test-collection-id:${bamboo.TRUNK_ORG_URL_SLUG}${bamboo.TRUNK_TEST_COLLECTION_ID}$(TRUNK_ORG_URL_SLUG)(Azure)$(TRUNK_TEST_COLLECTION_ID)Considerations
--org-url-slugregardless, and the uploader action marksorg-slugrequired: true. This is the same invariant trunk-io/trunk2#5830 fixes in the in-product snippet generators, so the app and the docs now produce matching commands.trunk-analytics-cli testwas verified, not assumed. Several pages document thetestsubcommand rather thanupload. Both accept--test-collection-id, confirmed against the shipped binary — hadtestnot taken it, this sweep would have broken those pages in exactly the way #5830 describes.\to the uploader action's YAML lines, whereverorg-slug:happened to be the last key in itswith:block. YAML has no continuations, so those blocks became invalid — and it was invisible in a unified diff, which shows the added line but not the modified one above it. The rule now applies continuations only to shell flags. Verified afterwards that no fenced block ends on a dangling\, and no YAML line carries one.org-slug: my-trunk-org-slug, but a collection ID is eight opaque characters that must be copied from the app, so a literal example would read as a value to use. The same blocks already use<TEST OUTPUT PATH>, so the style is consistent locally.github-actions.mdxstored it assecrets.TRUNK_ORG_URL_SLUG; only the API token is a credential, so it is nowvars.. Every remainingsecrets.TRUNK_*reference in the product docs is the API token.quarantining/index.mdxandmultiple-repositories.mdxare in scope after all, rather than deferred to PR 3, so the sweep is complete in one place. That added ten examples, including an--org-url-slug=[org]equals-style pair the original regex could not match.pass-on-retry-monitor.mdxhas one. That one now names the collection Monitors tab alongside the repository path. Nothing else on the detection pages needed changing.Testing
\, across every page underflaky-tests/get-started/.github-actions.mdxat 9 occurrences,xctest.mdxat 4) for indentation and continuation handling.mint dev. Worth doing before this leaves draft — the risk here is a code block that renders wrong rather than prose that reads wrong.Follow-up Work
PRs 3–5 of the stack: the straggler copy (quarantining, ticketing, webhooks, PR comments, the API reference), the dashboard replacement, and the flip that unhides #322's pages.
References
🤖 Generated with Claude Code