You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found while auditing for settings that accept a failure quietly, after --legacy-peer-deps turned out to be doing exactly that (#215).
playwright.config.ts sets retries: 2 in CI. A test that fails and then passes
is reported as flaky and the build goes green. The last Tests run on main
reported four:
Test
e2e/download-feedback.spec.ts:212
"says it is working during the wait"
e2e/back-button.spec.ts:40
"leaves the pathway browser without stepping through tabs first"
e2e/interactor-threshold.spec.ts:437
"says so, and offers the way to it" — fixed in #216
e2e/pathway-browser.spec.ts:171
"the Expression tab renders the Expression Atlas heatmap"
Why this matters more than it looks
A flaky test is not a test that is occasionally wrong; it is a test telling you
something and being silenced. The one I fixed had a real race in it — the badges
are added to the graph and the handler that hides them runs afterwards, so
reading once could catch the moment in between. That is a fact about the
application's timing, discovered only because the retry was looked at.
The other three may be the same kind of thing or may be genuine network
non-determinism against third-party services. They are not mine to guess at.
Not a proposal to remove retries
These suites reach real services over the network, and some non-determinism is
honest. The problem is that flakiness is invisible: it is in the log and nowhere
else, so nobody is accountable for a number that can only grow.
Options, roughly in order of cost:
Report it. Surface the flaky list in the job summary so it is seen without
reading the log.
Cap it. Fail the build if more than N tests are flaky, so the number
cannot grow silently.
Found while auditing for settings that accept a failure quietly, after
--legacy-peer-depsturned out to be doing exactly that (#215).playwright.config.tssetsretries: 2in CI. A test that fails and then passesis reported as flaky and the build goes green. The last Tests run on main
reported four:
e2e/download-feedback.spec.ts:212e2e/back-button.spec.ts:40e2e/interactor-threshold.spec.ts:437e2e/pathway-browser.spec.ts:171Why this matters more than it looks
A flaky test is not a test that is occasionally wrong; it is a test telling you
something and being silenced. The one I fixed had a real race in it — the badges
are added to the graph and the handler that hides them runs afterwards, so
reading once could catch the moment in between. That is a fact about the
application's timing, discovered only because the retry was looked at.
The other three may be the same kind of thing or may be genuine network
non-determinism against third-party services. They are not mine to guess at.
Not a proposal to remove retries
These suites reach real services over the network, and some non-determinism is
honest. The problem is that flakiness is invisible: it is in the log and nowhere
else, so nobody is accountable for a number that can only grow.
Options, roughly in order of cost:
reading the log.
cannot grow silently.
probably applies to at least one of the others.
How to see it yourself