GitHub Actions

This page gives you a GitHub Actions job that runs after your deploy, triggers every active test in a Plainwright project through the CI webhook, waits for the batch to finish, and fails the workflow when the gate fails. Quarantined tests never fail it, and the server takes retries and human overrides into account.

Set it up

In Plainwright, open the project’s Settings → General → Run from CI (you need to be an admin) and copy the webhook secret. The project ID is the UUID at the end of the webhook URL.

In GitHub, open the repository’s Settings → Secrets and variables → Actions and add three repository secrets:

Secret	Example
`PLAINWRIGHT_URL`	`https://plainwright.example.com` (a trailing slash is fine)
`PLAINWRIGHT_PROJECT_ID`	`74a76eba-773d-42a7-8f4a-165e386b0e8f`
`PLAINWRIGHT_WEBHOOK_SECRET`	the secret from step 1

Optional: to test a staging or preview environment instead of the project’s base URL, add a repository variable (not a secret) named PLAINWRIGHT_ENVIRONMENT_ID with the environment’s ID.
Add one of the workflows below as .github/workflows/plainwright.yml, and put your deploy steps where the comment says.

Short and easy to adapt. curl and jq are preinstalled on ubuntu-latest.

name: Deploy and test with Plainwright

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy-and-test:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4

      # … your deploy steps go here …

      - name: Run Plainwright tests and wait for the gate
        env:
          PLAINWRIGHT_URL: ${{ secrets.PLAINWRIGHT_URL }}
          PLAINWRIGHT_PROJECT_ID: ${{ secrets.PLAINWRIGHT_PROJECT_ID }}
          PLAINWRIGHT_WEBHOOK_SECRET: ${{ secrets.PLAINWRIGHT_WEBHOOK_SECRET }}
          PLAINWRIGHT_ENVIRONMENT_ID: ${{ vars.PLAINWRIGHT_ENVIRONMENT_ID }}
        run: |
          set -euo pipefail
          BASE="${PLAINWRIGHT_URL%/}"
          AUTH="X-Webhook-Secret: $PLAINWRIGHT_WEBHOOK_SECRET"
          BODY=$(jq -n --arg by "github · $GITHUB_REPOSITORY@${GITHUB_SHA::7}" \
            --arg env "${PLAINWRIGHT_ENVIRONMENT_ID:-}" \
            '{triggered_by: $by} + (if $env == "" then {} else {environment_id: $env} end)')

          RESP=$(curl -fsS -X POST "$BASE/api/runs/webhook/$PLAINWRIGHT_PROJECT_ID" \
            -H "$AUTH" -H "Content-Type: application/json" -d "$BODY")
          COUNT=$(jq '.run_ids | length' <<<"$RESP")
          if [ "$COUNT" -eq 0 ]; then echo "No active tests, nothing to gate."; exit 0; fi
          STATUS_URL=$(jq -r .status_url <<<"$RESP")
          echo "Queued $COUNT runs · batch $(jq -r .batch_id <<<"$RESP")"

          GATE=pending
          for i in $(seq 1 120); do                       # 120 × 15 s = 30 minutes
            REPORT=$(curl -fsS "$BASE$STATUS_URL" -H "$AUTH")
            GATE=$(jq -r .gate <<<"$REPORT")
            [ "$GATE" != "pending" ] && break
            jq -r '.summary | "… \(.passed) passed · \(.failed) failed · \(.pending) running"' <<<"$REPORT"
            sleep 15
          done

          {
            echo "### Plainwright gate: ${GATE^^}"
            echo
            echo "| Result | Kind | Test |"
            echo "| --- | --- | --- |"
            jq -r '.runs[] | "| \(.gate | ascii_upcase)\(if .quarantined then " (quarantined)" else "" end) | \(.kind) | [\(.name)](\(.url)) |"' <<<"$REPORT"
          } >> "$GITHUB_STEP_SUMMARY"

          case "$GATE" in
            pass) echo "Quality gate passed." ;;
            fail) echo "::error title=Plainwright::Quality gate failed"; exit 1 ;;
            *)    echo "::error title=Plainwright::Timed out waiting for the gate"; exit 1 ;;
          esac

The workflow that ships in the repository as .github/workflows/test.yml. It uses only Python’s standard library, prints an aligned table in the log and the job summary, tolerates brief network errors while polling, and sets step outputs. It also accepts the old TESTAI_* secret names.

name: Deploy & Plainwright quality gate

# After your deploy step, trigger every active test in a Plainwright project and wait for the
# batch verdict. The job fails when the gate fails (quarantined tests never fail it; flaky
# retries and human overrides are taken into account by the server).
#
# Repository secrets:
#   PLAINWRIGHT_URL             e.g. https://plainwright.example.com   (no trailing slash needed)
#   PLAINWRIGHT_PROJECT_ID      Project → Settings → General
#   PLAINWRIGHT_WEBHOOK_SECRET  Project → Settings → General (CI webhook secret)
# Optional repository variable:
#   PLAINWRIGHT_ENVIRONMENT_ID  run against a named environment (e.g. staging) instead of the project URL
# The old TESTAI_* secret names still work.

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:

jobs:
  plainwright:
    name: Plainwright tests
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4

      - name: Deploy application
        run: |
          # Your deploy step here
          echo "Deploying..."

      - name: Run tests and wait for the gate
        id: gate
        env:
          PLAINWRIGHT_URL: ${{ secrets.PLAINWRIGHT_URL || secrets.TESTAI_URL }}
          PLAINWRIGHT_PROJECT_ID: ${{ secrets.PLAINWRIGHT_PROJECT_ID || secrets.TESTAI_PROJECT_ID }}
          PLAINWRIGHT_WEBHOOK_SECRET: ${{ secrets.PLAINWRIGHT_WEBHOOK_SECRET || secrets.TESTAI_WEBHOOK_SECRET }}
          PLAINWRIGHT_ENVIRONMENT_ID: ${{ vars.PLAINWRIGHT_ENVIRONMENT_ID }}
          PLAINWRIGHT_TIMEOUT_S: "1800"   # give up waiting after 30 minutes
          PLAINWRIGHT_POLL_S: "15"
        run: |
          python3 - <<'PY'
          import json, os, sys, time, urllib.error, urllib.request

          BASE = (os.environ.get("PLAINWRIGHT_URL") or "").strip().rstrip("/")
          PROJECT = (os.environ.get("PLAINWRIGHT_PROJECT_ID") or "").strip()
          SECRET = (os.environ.get("PLAINWRIGHT_WEBHOOK_SECRET") or "").strip()
          ENV_ID = (os.environ.get("PLAINWRIGHT_ENVIRONMENT_ID") or "").strip()
          TIMEOUT_S = int(os.environ.get("PLAINWRIGHT_TIMEOUT_S") or 1800)
          POLL_S = max(5, int(os.environ.get("PLAINWRIGHT_POLL_S") or 15))


          def fail(message, code=1):
              print(f"::error title=Plainwright::{message}")
              sys.exit(code)


          def output(**values):
              path = os.environ.get("GITHUB_OUTPUT")
              if path:
                  with open(path, "a", encoding="utf-8") as fh:
                      for k, v in values.items():
                          fh.write(f"{k}={v}\n")


          def call(method, path, body=None):
              data = json.dumps(body).encode() if body is not None else None
              req = urllib.request.Request(BASE + path, data=data, method=method, headers={
                  "X-Webhook-Secret": SECRET, "Content-Type": "application/json",
                  "Accept": "application/json", "User-Agent": "plainwright-ci"})
              with urllib.request.urlopen(req, timeout=30) as resp:
                  return json.loads(resp.read().decode("utf-8") or "{}")


          def detail(err):
              try:
                  return json.loads(err.read().decode("utf-8")).get("detail") or ""
              except Exception:
                  return ""


          RESULT = {"pass": "PASS", "fail": "FAIL", "pending": "PENDING"}


          def score_text(r):
              if r.get("score") is None:
                  return "-"
              t = r.get("threshold")
              return f"{r['score']:.2f}" + (f" / {t:.2f}" if t is not None else "")


          def result_text(r):
              text = RESULT.get(r.get("gate"), str(r.get("gate")))
              if r.get("quarantined"):
                  text += " (quarantined)"
              if r.get("is_flaky"):
                  text += " (flaky)"
              return text


          def summarize(report):
              runs = report.get("runs") or []
              s = report.get("summary") or {}
              headers = ["Test", "Kind", "Result", "Score / threshold", "Attempt"]
              rows = [[r.get("name") or r.get("test_case_id") or "?", r.get("kind") or "", result_text(r),
                       score_text(r), str(r.get("attempt") or 1)] for r in runs]
              widths = [max(len(h), *(len(row[i]) for row in rows)) if rows else len(h) for i, h in enumerate(headers)]
              line = lambda cells: "  ".join(c.ljust(w) for c, w in zip(cells, widths))
              print()
              print(line(headers))
              print(line(["-" * w for w in widths]))
              for row in rows:
                  print(line(row))
              totals = (f"{s.get('total', 0)} tests · {s.get('passed', 0)} passed · {s.get('failed', 0)} failed"
                        f" · {s.get('quarantined_failed', 0)} quarantined failures · {s.get('pending', 0)} pending")
              print()
              print(f"Gate: {str(report.get('gate')).upper()} — {totals}")
              for r in runs:
                  if r.get("gate") == "fail" and r.get("error_message"):
                      print(f"  {r.get('name')}: {r['error_message']}")
                  if r.get("gate") == "fail" and r.get("url"):
                      print(f"    {r['url']}")
              path = os.environ.get("GITHUB_STEP_SUMMARY")
              if path:
                  esc = lambda v: str(v).replace("|", "\\|").replace("\n", " ")
                  md = [f"### Plainwright gate: {str(report.get('gate')).upper()}", "", totals, "",
                        "| " + " | ".join(headers) + " |", "|" + "---|" * len(headers)]
                  for r, row in zip(runs, rows):
                      name = f"[{esc(row[0])}]({r['url']})" if str(r.get("url") or "").startswith("http") else esc(row[0])
                      md.append("| " + " | ".join([name] + [esc(c) for c in row[1:]]) + " |")
                  with open(path, "a", encoding="utf-8") as fh:
                      fh.write("\n".join(md) + "\n")


          if not (BASE and PROJECT and SECRET):
              fail("Set the PLAINWRIGHT_URL, PLAINWRIGHT_PROJECT_ID and PLAINWRIGHT_WEBHOOK_SECRET repository secrets.", 2)

          repo, sha = os.environ.get("GITHUB_REPOSITORY", ""), os.environ.get("GITHUB_SHA", "")[:7]
          body = {"triggered_by": "github" + (f" · {repo}@{sha}" if repo else ""),
                  "ref": os.environ.get("GITHUB_REF", ""), "sha": os.environ.get("GITHUB_SHA", "")}
          if ENV_ID:
              body["environment_id"] = ENV_ID
          try:
              trigger = call("POST", f"/api/runs/webhook/{PROJECT}", body)
          except urllib.error.HTTPError as e:
              fail(f"Triggering tests failed: HTTP {e.code} {detail(e)}".strip())
          except (urllib.error.URLError, OSError, ValueError) as e:
              fail(f"Could not reach Plainwright at {BASE}: {e}")

          batch_id = trigger.get("batch_id") or ""
          status_url = trigger.get("status_url") or (f"/api/batches/{batch_id}" if batch_id else "")
          run_ids = trigger.get("run_ids") or []
          print(f"Queued {len(run_ids)} test run(s) · batch {batch_id or '-'}")
          output(batch_id=batch_id)
          if not run_ids:
              print("No active tests in this project, nothing to gate.")
              output(gate="pass")
              sys.exit(0)
          if not status_url:
              fail("The server did not return a status_url (is this an older Plainwright?)")

          report, errors, done = None, 0, False
          for poll in range(max(1, TIMEOUT_S // POLL_S)):
              if poll:
                  time.sleep(POLL_S)
              try:
                  report, errors = call("GET", status_url), 0
              except urllib.error.HTTPError as e:
                  if e.code in (401, 403, 404):
                      fail(f"Reading the batch status failed: HTTP {e.code} {detail(e)}".strip())
                  errors += 1
              except (urllib.error.URLError, OSError, ValueError):
                  errors += 1
              if errors >= 5:
                  fail("Plainwright stopped answering while waiting for results.")
              if report and report.get("status") == "done":
                  done = True
                  break
              if report:
                  s = report.get("summary") or {}
                  print(f"… {s.get('passed', 0)} passed · {s.get('failed', 0)} failed · "
                        f"{s.get('pending', 0)} still running", flush=True)

          if not done:
              if report:
                  summarize(report)
              output(gate="timeout")
              fail(f"Timed out after {TIMEOUT_S}s waiting for batch {batch_id}.")

          summarize(report)
          gate = report.get("gate")
          s = report.get("summary") or {}
          output(gate=gate, passed=s.get("passed", 0), failed=s.get("failed", 0))
          if gate == "fail":
              fail(f"Quality gate failed: {s.get('failed', 0)} of {s.get('total', 0)} tests failed.")
          print("Quality gate passed.")
          PY

What you get

The job fails when the gate fails, and also when the server can’t be reached, the secret is wrong (403) or the batch takes longer than the timeout.
A job summary (the workflow run’s Summary page) with one row per test: result, kind and a link to the run in Plainwright. Run links are built from the server’s APP_BASE_URL, so set it to your public URL.
Step outputs (full workflow): batch_id, gate (pass, fail or timeout), passed and failed. Read them in later steps as steps.gate.outputs.gate.
Triggered by on each run reads github · <owner>/<repo>@<short sha>, so you can find the commit from the run page.

Customize

Change	How
Wait longer	Full workflow: `PLAINWRIGHT_TIMEOUT_S` (default `1800`). Compact: the `seq 1 120` loop count. Keep the job’s `timeout-minutes` above it.
Poll less often	Full workflow: `PLAINWRIGHT_POLL_S` (default `15`, minimum `5`). Compact: `sleep 15`.
Another browser or device	Add `browser` (`firefox`, `webkit`) or `device` (`iphone`, `pixel`, `ipad`) to the JSON body. See Browsers and devices.
Run on pull requests	Add `pull_request:` under `on:`. Pull requests from forks don’t get repository secrets, so the step can’t trigger tests for them.
Run after a separate deploy workflow	Use `on: workflow_run` with `workflows: ["Deploy"]` and `types: [completed]`, or `on: deployment_status`.
Re-run only the failures	Rerun failed needs a user token rather than the webhook secret: `POST /api/reliability/project/{id}/rerun-failed?batch_id=<batch>`. See Quarantine and re-run.

Troubleshooting

Symptom	Fix
`HTTP 403 Invalid secret`	The secret is wrong or was rotated in Plainwright. Copy it again from Run from CI.
`HTTP 404 Project not found`	`PLAINWRIGHT_PROJECT_ID` is wrong.
`Could not reach Plainwright`, or `curl: (6)` / `curl: (7)`	The URL isn’t reachable from the runner. Check DNS, HTTPS and firewalls.
`HTTP 503`	The server’s job queue (Redis) is down. Check the stack on the server.
The gate stays `pending` until the timeout	No worker is picking up runs, or the tests are slow. Check that workers are running (Scaling workers) and raise the timeout.
Every UI test fails with the same error	Open the cause in Triage. A wrong environment URL or an expired AI provider key often shows up this way.

Next steps

CI webhook and gatesThe webhook, batch status and gate rules in detail.

Retries and flaky testsStop one flaky test from blocking deploys.

LLM evalsFail a deploy when an AI feature's score drops.