Skip to content

Three e2e tests pass only on retry, and CI is configured not to notice #217

Description

@adamjohnwright

Found while auditing for settings that accept a failure quietly, after
--legacy-peer-deps turned out to be doing exactly that (#215).

playwright.config.ts sets retries: 2 in CI. A test that fails and then passes
is reported as flaky and the build goes green. The last Tests run on main
reported four:

Test
e2e/download-feedback.spec.ts:212 "says it is working during the wait"
e2e/back-button.spec.ts:40 "leaves the pathway browser without stepping through tabs first"
e2e/interactor-threshold.spec.ts:437 "says so, and offers the way to it" — fixed in #216
e2e/pathway-browser.spec.ts:171 "the Expression tab renders the Expression Atlas heatmap"

Why this matters more than it looks

A flaky test is not a test that is occasionally wrong; it is a test telling you
something and being silenced. The one I fixed had a real race in it — the badges
are added to the graph and the handler that hides them runs afterwards, so
reading once could catch the moment in between. That is a fact about the
application's timing, discovered only because the retry was looked at.

The other three may be the same kind of thing or may be genuine network
non-determinism against third-party services. They are not mine to guess at.

Not a proposal to remove retries

These suites reach real services over the network, and some non-determinism is
honest. The problem is that flakiness is invisible: it is in the log and nowhere
else, so nobody is accountable for a number that can only grow.

Options, roughly in order of cost:

  1. Report it. Surface the flaky list in the job summary so it is seen without
    reading the log.
  2. Cap it. Fail the build if more than N tests are flaky, so the number
    cannot grow silently.
  3. Fix them. test(interactors): stop racing the handler that hides the badges #216 is one; the pattern there — poll rather than read once —
    probably applies to at least one of the others.

How to see it yourself

gh run view <run-id> --log | grep -iE "flaky|retry #"

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions