Retries and flaky tests

A retry re-runs a test automatically when it fails, before the failure counts. A test that fails and then passes on a retry is marked flaky: it passed this time, but it can’t be trusted until you find out why it failed first.

Use retries when a failure might not be your app’s fault, for example a slow third-party widget, a timing-sensitive animation or a shared staging database. Don’t use them to hide real bugs: every flaky pass is recorded and shows up in Triage and Analytics.

How a retry works

1. A run finishes as failed or error.
2. If the test still has retries left, Plainwright queues a new run straight away. It has the same batch, environment, browser and device, attempt goes up by one, retry_of_id points at the failed run, and Triggered by reads retry.
3. The failed run waits. It sends no alert, and the CI gate reports it as pending instead of failed (see CI webhook and gates).
4. If a retry passes, it is marked flaky and the test counts as passed. If every attempt fails, the last attempt is the result: it is the one that alerts and fails the gate.

The run page links each attempt to the next, so you can compare the failed attempt with the one that passed.

Set the number of retries

Retries are set per project and can be overridden per test. The maximum is 5.

Setting	Where	Values	Default	Applies to
Project default	Settings → General → Runs → Retries on failure	0–5	0	Web UI (ui) and api tests that don’t set their own value
Per test	Test → Settings → Reliability → Retries on failure	Project default, Don’t retry, or 1–5	Project default	Any test kind

The project default only applies to ui and api tests. Every other kind (conversation, red team, evals, MCP, A2A, quality) is retried only when you set retries on that test.

Through the API, set retries on a test, or default_retries on the project. null on a test means “use the project default”.

bash

# One test: retry up to 2 times (editor role)
curl -fsS -X PUT "$PLAINWRIGHT_URL/api/test-cases/$TEST_ID" \
  -H "Authorization: Bearer $PLAINWRIGHT_TOKEN" -H "Content-Type: application/json" \
  -d '{"retries": 2}'

# Project default for ui/api tests (admin role)
curl -fsS -X PUT "$PLAINWRIGHT_URL/api/projects/$PROJECT_ID" \
  -H "Authorization: Bearer $PLAINWRIGHT_TOKEN" -H "Content-Type: application/json" \
  -d '{"default_retries": 1}'

PowerShell

$h = @{ Authorization = "Bearer $env:PLAINWRIGHT_TOKEN" }
# One test: retry up to 2 times (editor role)
Invoke-RestMethod -Method Put -Uri "$env:PLAINWRIGHT_URL/api/test-cases/$env:TEST_ID" `
  -Headers $h -ContentType 'application/json' -Body '{"retries": 2}'
# Project default for ui/api tests (admin role)
Invoke-RestMethod -Method Put -Uri "$env:PLAINWRIGHT_URL/api/projects/$env:PROJECT_ID" `
  -Headers $h -ContentType 'application/json' -Body '{"default_retries": 1}'

See API overview for getting a token.

When a failure is not retried

Plainwright skips the retry, and the failure stands, when:

• the test is quarantined;
• the run was blocked by a CAPTCHA or bot wall (blocked_reason is set), because a retry would hit the same wall (see One-time codes and CAPTCHAs);
• the run could not start at all, for example an unknown test kind, a test that was deleted, or a usage quota that has run out;
• the test has used all its attempts (attempt is already greater than the retry count);
• the run already has a retry (re-executing the same job never queues a second one).

If the job queue is unavailable when the retry should be scheduled, the original failure stands and alerts as normal.

Retries cost a full run

Each retry is a complete new run, with its own AI calls and token spend. On a test with 3 retries, a consistently failing test costs four runs. Keep retries low (1 is usually enough) and fix or quarantine tests that keep needing them.

How flaky tests are detected

A run is flagged flaky (is_flaky: true) when it passes after an earlier attempt in its retry chain failed or errored.

Over a time window, a test counts as flaky when any of these happened:

Signal	Meaning
Passed on retry	At least one run passed after a failed attempt.
Mixed trials	A multi-trial run (see Trials, thresholds and cost) had both passing and failing trials.
Pass/fail flips	Its final outcomes flipped between passed and failed at least twice. Retried attempts don’t count here, only each chain’s final result.

The same rule is used by the Triage page’s Flaky tab, the Analytics page and the /api/analytics/flakiness endpoint. A human verdict override counts as the run’s result.

Open it from a project: Triage → Flaky, then choose a window of 7, 14 or 30 days. Each row shows the signals, the pass rate of the final outcomes and the last run. Use Quarantine to take a test out of gates and alerts while you investigate.

Read it from the API

curl -fsS "$PLAINWRIGHT_URL/api/reliability/project/$PROJECT_ID?days=14" \
  -H "Authorization: Bearer $PLAINWRIGHT_TOKEN"

Response (trimmed)

{
  "days": 14,
  "flaky": [
    {
      "test_case_id": "a4b07fe9-…",
      "name": "Apply a coupon code",
      "module_name": "Checkout",
      "kind": "ui",
      "quarantined": false,
      "retries": null,
      "runs": 6,
      "failures": 2,
      "flaky_runs": 2,
      "mixed_trials": 0,
      "flips": 0,
      "pass_rate": 1.0,
      "last_status": "passed",
      "last_run_id": "6a54a645-…",
      "last_flaky_at": "2026-10-20T15:02:11"
    }
  ],
  "quarantined": [],
  "counts": { "flaky": 2, "quarantined": 1, "tests": 14, "retried_runs": 3 }
}

days is clamped to 1–90. retried_runs counts runs that were retried in the window.

Retries, trials and alerts

Retries and trials solve different problems:

	Retries	Trials
When they run	Only after a failure	Always, every run
Result	The last attempt wins; a pass on retry is flaky	A pass rate (or mean score) judged against the threshold
Good for	Occasional infrastructure noise	Non-deterministic systems, such as chatbots and LLM features
Set with	retries (0–5)	trials (1–10)

Alerts follow the final attempt: a failure that will be retried never alerts on its own, and a flaky pass only alerts if you turn on Also alert on flaky passes in Notifications.

Next steps

Quarantine and re-runKeep a flaky test running without failing CI, and re-run only what failed.

Triage and issuesSee failures grouped by cause and file them to your tracker.

Root cause and run chatAsk the AI why a run failed, and whether it's the app or the test.

Writing good testsWording that makes tests stable in the first place.