Skip to content

docs(flaky-tests): replace the dashboard's landing page for collections - #325

Draft
TylerJang27 wants to merge 2 commits into
test-collections/straggler-copyfrom
test-collections/dashboard
Draft

docs(flaky-tests): replace the dashboard's landing page for collections#325
TylerJang27 wants to merge 2 commits into
test-collections/straggler-copyfrom
test-collections/dashboard

Conversation

@TylerJang27

Copy link
Copy Markdown
Contributor

Important

Do not land until the default test_collection_status enum is flipped. This page is live, and it would otherwise tell every reader that /<org>/flaky-tests lands on collections before it does.

Overview

PR 4 of the Test Collections docs stack: dashboard.mdx documents the collections list and the collection dashboard, replacing the repositories overview it opened on.

Details

Stacked on #324#323#322. Review those first.

The page opened with "When you navigate to /<your-org>/flaky-tests, you land on a repositories overview" — false for any organization past the migration, since that URL serves collections. This is a replacement of the two landing sections, not an addition beside them.

Everything below the fold — test cases overview, test case details, code owners, the Summary and Test History tabs — is already scope-agnostic and is untouched.

Considerations

  • Two columns have no repository equivalent, and both earn explanation: the Collection ID (the value CI passes, so it belongs on the identity column rather than buried in prose) and Setup. Setup is documented as a count and explicitly not a percentage — the denominator grows as steps ship, so 3/4 and 3/5 are honestly different where 75% and 60% invite a meaningless comparison.
  • Two deliberate absences are stated outright, so they get read as decisions rather than filed as bugs: there is no total row across collections, and side-by-side comparison of two collections is not available yet.
  • The repository view keeps a subsection. It stays reachable behind Legacy view during a migration, so the quarantine status icons and the inactive-repositories behaviour are retained rather than deleted, with a pointer to the migration guide.
  • Every label was taken from the source, not the screenshot. The three table titles are exact (Top 10 Most Unstable Tests, Top 10 Longest-Running Tests, Top 10 Longest-Manually-Quarantined Tests, from overview-tables-section.tsx), and the status-changes chart documents all three series — Newly Broken included, which the screenshot's legend omits because that window had no broken transitions. Documenting only what the screenshot showed would have left a series out.
  • Reuses docs(flaky-tests): add the test collections concept and migration pages #322's collection overview screenshot rather than adding a second copy of the same view. One screenshot is still needed: the collections list.
  • A pre-existing run of back-to-back <Info> callouts further down the page (the framework caveats) is left alone — confirmed it pre-dates the stack.

Testing

  • Every link and asset path in the diff resolves against the file tree.
  • Confirmed the key-metrics-* images dropped from this page are still referenced by overview.mdx, so nothing is orphaned.
  • Table titles, chart series, metric-card labels and the range toggle checked against the components rather than written from the screenshots.
  • Not previewed with mint dev.

Follow-up Work

References

🤖 Generated with Claude Code

`/<org>/flaky-tests` lands on test collections, so the page opened on a
premise that is false for any organization past the migration — it
described a repositories overview at a URL that no longer serves one.

The collections list is documented in its place, column by column,
including the two columns repositories have no equivalent for: the
Collection ID that CI passes, and the Setup count. Two deliberate
absences are stated rather than left to be filed as bugs — there is no
total row, because a flaky count summed across unrelated collections is
not actionable, and side-by-side comparison is not available yet.

The collection dashboard replaces "Key repository metrics": the range
toggle, the three impact cards, the three daily charts — including that
Test Status Changes counts transitions rather than totals — and the
filter bar with its three top-ten tables, whose repository filter is the
one place a collection exceeds repository parity.

The repository view keeps a subsection, since it stays reachable behind
**Legacy view** during a migration, and it retains the quarantine icons
and inactive-repository behaviour.

Everything below the fold is scope-agnostic already — test cases, test
details, the Summary and Test History tabs — and is untouched. Reuses the
collection overview screenshot from the concept page rather than adding a
second copy; the collections list still needs one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@TylerJang27
TylerJang27 force-pushed the test-collections/dashboard branch from 62a5566 to 722ddd0 Compare September 9, 2026 18:41
Comment thread flaky-tests/dashboard.mdx Outdated
## Collections overview

When you navigate to `/<your-org>/flaky-tests`, you land on a repositories overview showing all monitored repositories at a glance.
When you navigate to `/<your-org>/flaky-tests`, you land on your [test collections](./get-started/test-collections). The page exists to answer two questions — which collection is unhealthy, and which is not finished being set up — so every row carries both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When you navigate to `/<your-org>/flaky-tests`, you land on your [test collections](./get-started/test-collections). The page exists to answer two questions — which collection is unhealthy, and which is not finished being set up — so every row carries both.
When you navigate to `/<your-org>/flaky-tests`, you land on your [test collections](./get-started/test-collections). The page exists to answer two questions — what collections have flaky tests, and which are not finished being set up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread flaky-tests/dashboard.mdx Outdated
Comment on lines +30 to +32
<Info>
**Setup** is a count, never a percentage. The number of steps grows as features ship, so `3/4` and `3/5` are honestly different where 75% and 60% would invite a comparison that means nothing. It is the same number the collection's own setup checklist shows.
</Info>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this block

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread flaky-tests/dashboard.mdx Outdated
**Setup** is a count, never a percentage. The number of steps grows as features ship, so `3/4` and `3/5` are honestly different where 75% and 60% would invite a comparison that means nothing. It is the same number the collection's own setup checklist shows.
</Info>

There is deliberately no total row: a flaky count summed across unrelated collections is not a number anyone acts on, and the page's job is to point at one collection. Comparing two collections side by side is not available yet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this block too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@TylerJang27
TylerJang27 added this pull request to stack #327 September 9, 2026 18:45
Applies the suggested opening, and drops the two blocks called out —
the Setup-is-a-count callout and the paragraph about there being no
total row.

The screenshot filled a gap the prose had wrong: quarantine shield icons
appear on **collection** rows too, not only repository ones. They were
documented under Repositories alone. The icon table moves up to the
collections list where it belongs, and the repositories subsection now
refers back to it rather than restating it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@TylerJang27

Copy link
Copy Markdown
Contributor Author

All three applied in 6e81b91, and the collections-list screenshot is in — so this PR has no TODOs left.

The screenshot caught a real error in the prose. Quarantine shield icons appear on collection rows too, not only repository ones — trunk2-pr-e2e and trunk2-pr-ts-unit both carry one in your shot. I had documented them under Repositories alone. Confirmed in collections-overview-table.tsx (same Shield / ShieldCheck treatment), so the icon table has moved up to the collections list where it belongs, and the repositories subsection now refers back to it instead of restating it.

The shot is a good one for this page for a reason worth noting: it happens to show Setup at 0/4, 2/4, 3/4 and 4/4 across five collections, and a collection with REPOS 0, so the columns demonstrate themselves rather than needing the callout that just came out.

Both files landed as assets/flaky-tests/collections-list-{light,dark}.png, matched on dimensions and verified by luminance (247.6 vs 30.3).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant