Arjun Nair
When a test fails intermittently, the first impulse is often a retry. If it passes next time, the issue seems fixed: CI is green and the PR merges. But the flakiness is only hidden. Whatever made the test non-deterministic remains, and it usually gets worse.
Flaky tests are symptoms of deeper problems. Flakiness matters not because it is automatically dangerous, but because it usually points to a small set of underlying issues, all of which create real trouble at scale. Identifying the issue changes the remedy.
Three Causes Behind Most Flakiness
The first cause is timing. The test expects an element to exist or an operation to finish before it has. It passes when the app is quick and CI load is light, then fails under a busy runner, slow network, or longer animation. Use explicit waits, not sleeps: waitForSelector, waitForLoadState, and waitForResponse in Playwright assert state instead of guessing timing. A test using await page.waitForTimeout(2000) instead of a specific condition is primed for future flakiness.
The second cause is ordering and shared state. A test may pass alone but fail after a particular test because teardown left shared state behind: database records, session cookies, localStorage keys, or feature flags that were not reset. Integration tests using a real database without isolation are common offenders. The remedy is genuine isolation, not merely randomizing order and hoping.
The third cause is subtle selector drift. The element is found, but its state differs from what the test expects. A transition may still be running, a lazy-loaded component may still be rendering, or a dynamic class may have appeared without the selector accounting for it. Unlike hard selector failures that immediately produce ElementNotFound, these look behavioral but are structural. They occur inconsistently because the race is sometimes won and sometimes lost.
Why Retries Worsen Flakiness
Retrying a flaky test does three things, and none helps. It hides non-determinism in the codebase or test, slows CI as retry rates rise, and teaches engineers to dismiss flakiness because it eventually passes.
The teams with the worst flakiness usually added retries early and never reviewed them. Once the retry rate hurts, dozens of tests may retry twice, while enough genuine failures have been hidden that no one knows what is wrong. Diagnosis becomes far harder after a year of concealed flake data.
Retries have one valid use: covering environmental noise outside your control, such as network instability in some CI environments. Even then, log every retry with enough context to review the pattern. If a test retries more than once a week, it is not environmental noise. It is a test or application problem.
Diagnosing the Category
The quickest way to classify a flaky test is to compare its failure message with CI runner load at that moment. Timing failures in Playwright often show an error such as Timeout 30000ms exceeded while waiting for.... If a timeout is always present and failures increase with CI queue time, timing is the likely cause.
To find shared-state problems, run the suspect test in strict isolation: use a fresh process, fresh database fixture, and cleared browser context. If it stays deterministic alone but flakes in the full suite, shared state is responsible.
Subtle structural drift appears in messages about element state: element is not visible, element is not enabled, or expected to be visible, got hidden. These are not hard failures: the element exists and the selector found it, but its state differs from expectations. Assert the state before acting: await expect(button).toBeVisible() before await button.click(), rather than retrying past the issue.
When Flakiness Indicates an Application Bug
Sometimes a flaky test reveals a real race in application code. A form may submit twice, a component may render differently based on API response order, or an animation may block input in some render cycles but not others. These are application bugs the test is detecting non-deterministically.
Here, making the test less sensitive is the wrong fix. Retries or longer timeouts may make it deterministic while allowing the race to reach production. Investigate and correct the application behavior instead. The flaky test is working; we need to understand its message.
We saw this clearly with early-access teams using TestSprighte. A test that intermittently failed during checkout exposed a double-submission bug triggered by specific network timing. The test was right and was the only automated signal to catch the issue before users did. Suppressing it with retries would have been a mistake.
Setting a Flakiness Budget
A practical team approach is to set an explicit budget: a maximum number of retried runs per week for each test. Exceed it, and quarantine the test from the main suite while scheduling a focused fix sprint. It still runs separately and its failures remain tracked, but it no longer blocks deploys. This creates pressure to investigate instead of allowing masked retries to accumulate.
Quarantine works because it exposes flakiness's cost. A list of 15 quarantined tests prompts a management discussion; a CI run passing after 15 silent retries does not. Making flakiness visible is the first step toward proper diagnosis, and diagnosis is required to fix it.