Skip to content
Plainwright
Open source · Self-hosted · Apache-2.0

Plainwright

Write tests in plain English — for web apps and AI products

QuickstartGitHub

docker compose up --build

What you can test

Section titled “What you can test”

Web UI, no selectors

An AI agent drives Chromium, Firefox or WebKit through Playwright MCP and reads the live page on every step. A renamed button doesn’t break the test. UI tests

APIs

Describe what an endpoint should do. The agent sends real HTTP requests and checks the responses, with your OpenAPI spec as optional context. API tests

Site quality

Accessibility (axe-core), performance, broken-link and SEO checks run as scripted browser checks, with no AI model and no token cost. Quality checks

Chatbots and assistants

A simulated user with a persona and a goal talks to your bot through its chat widget, HTTP API or A2A. A judge scores every turn. Conversation tests

Red teaming

Seven attack packs, from prompt injection and jailbreaks to PII leaks and tool misuse. Each attack is judged resisted or breached. Red teaming

LLM and RAG evals

Score answers over a CSV or JSONL dataset for correctness, relevance, safety, faithfulness and context quality. The CI gate fails when a score drops. LLM evals

MCP servers and A2A agents

A tool-contract check for any MCP server (Streamable HTTP or SSE), then an agent that exercises the tools. A2A agents get agent-card validation, then a conversation over A2A JSON-RPC. MCP servers

Reliability and triage

Retries, flaky-test detection, quarantine and re-running only the failures. Similar failures are grouped and filed to Jira, Linear or GitHub, by hand or automatically. Triage and issues

Your models, your servers

Use Anthropic, OpenAI, Gemini, OpenRouter, Groq or a local Ollama model. Self-host the stack with Docker Compose and gate deploys from CI. Self-hosting

How it works

Section titled “How it works”

  1. Describe the test. Write what should happen, the way you’d tell a colleague: “Check that a returning customer can pay with a saved card.” For AI products, describe a persona and a goal, or point at a dataset.

  2. An agent runs it against the real thing. A worker picks up the run. For a web test, the agent reads the page’s accessibility snapshot and clicks and types like a person. For an AI product, a simulated user or attacker holds the conversation, or each dataset row is sent to your endpoint.

  3. You get evidence, not just a verdict. Each run reports pass/fail or a score, with the steps, screenshots, transcript, console and network logs, and a plain-English explanation when something breaks.

  4. Wire it into your workflow. Trigger runs from CI with a webhook and wait for the gate, schedule them with cron, and send alerts to Slack, Discord, email or any webhook.

How a Plainwright run flows from a plain-English test to a verdictPlain-English testUI · API · AI productAPI + queueFastAPI · RedisWorkerAI agent · judgeReal targetbrowser · API · agentVerdict + evidencesteps · screenshots · scoresWhere it landsdashboard · CI gate · alertsrunjobactobserveHow a Plainwright run flows from a plain-English test to a verdictPlain-English testUI · API · AI productAPI + queueFastAPI · RedisWorkerAI agent · judgeReal targetbrowser · API · agentVerdict + evidencesteps · screenshotsWhere it landsdashboard · CI · alerts
A test is queued, a worker runs the AI agent against the real target, and the verdict flows to the dashboard, CI and alerts.

Start here

Section titled “Start here”
QuickstartRun Plainwright with Docker Compose and pass your first test.
ConceptsProjects, test kinds, runs, providers and judges in five minutes.
REST APIEvery endpoint, generated from the OpenAPI schema.
How it compares18 AI testing tools side by side, with a source for every cell.
BenchmarksThe public 23-task suite, the metrics and how to reproduce a run.
Security modelWhat protects your credentials, and what to set before going live.