Introduction

Plainwright is an open-source, AI-native testing platform that you host yourself. You describe a test the way you would explain it to a colleague, for example “Add a todo and check the counter says 1 item left”. An AI agent then runs it against the real thing: a web app in a headless browser, an HTTP API, a chatbot, a RAG pipeline, an MCP server or an A2A agent. You get back a verdict or a score, with the evidence behind it: steps, screenshots, transcripts, logs and a plain-English explanation of anything that broke.

There are no selectors and no recorded scripts. On every step the agent reads the live page (its accessibility snapshot) and decides what to click or type next, so a renamed button, a new cookie banner or a redirect doesn’t break the test.

A test is queued, a worker runs the AI agent against the real target, and the verdict flows to the dashboard, CI and alerts.

What you can test

Every test has a kind. The kind decides which engine runs it and how it is judged.

Group	Kind	Use it to check
Web app	Web UI (ui)	User flows in a real browser: sign-up, checkout, uploads, settings
Web app	API (api)	REST endpoints, by sending real HTTP requests
Web app	Site quality (quality)	Accessibility, performance, broken links and SEO, with no AI model involved
AI product	Conversation (conversation)	That a chatbot or assistant reaches the user’s goal and meets your criteria
AI product	Red team (redteam)	That an assistant resists prompt injection, jailbreaks, leaks and misuse
AI product	LLM eval (eval)	Answer quality over a dataset: correctness, relevance, safety, a rubric, latency
AI product	RAG eval (rag)	Grounding and retrieval quality: faithfulness and context precision and recall
Protocols	MCP server (mcp)	An MCP server’s tool contract, then real tool calls by an agent
Protocols	A2A agent (a2a)	An agent’s card, then a conversation over A2A JSON-RPC

Test kinds explains each one and when to pick it.

Around the tests

Evidence you can act on

Steps with screenshots, a guaranteed final screenshot, console and network logs, transcripts, per-row eval results, an AI explanation of failures, root-cause analysis and a chat about the run.

Reliable signals

Retries, flaky-test detection, quarantine, repeated trials with a pass rate, failure clustering, and a human verdict override when the AI gets it wrong.

Real-world flows

An encrypted secrets vault (the AI only ever sees {{secrets.NAME}}), saved logins, one-time codes from TOTP or an email inbox, test files to upload, and environments for staging and production.

Your workflow

A CI webhook with a pass/fail gate, cron schedules, alerts to Slack, Discord, email or any webhook, and issues filed to Jira, Linear or GitHub.

What makes it different

• Plain English, no selectors. A test is a description, not code. The agent works out the steps on the live page each time it runs.
• Web apps and AI products in one place. The same projects, runs, CI gate and alerts cover a checkout flow, a support bot and a RAG endpoint.
• Bring your own model. Anthropic, OpenAI, Google Gemini, OpenRouter, Groq or a local Ollama model. Every provider drives the same tool-calling agent. See AI providers.
• Self-hosted and open source. One docker compose up runs the whole stack on your machine or server, and your database, keys and screenshots stay there. The only data that leaves is what the agent sends to the AI provider you choose: page snapshots, API requests and replies, transcripts, and the final screenshot when the judge checks assertions. The model works with {{secrets.NAME}} placeholders rather than the values in your vault, and a local Ollama model keeps everything on your machine. Plainwright is licensed under Apache-2.0; the source is on GitHub.

Things to know before you start

Runs call a paid model

Most kinds call your AI provider on every run: the agent’s tool calls, the simulated user’s messages and the judge’s scores all use tokens. Each run page shows its token count, and the cost caps in Configuration bound the expensive kinds. Site quality tests use no model at all.

• The agent is not deterministic. The same test can take a slightly different path each time. Write outcomes rather than click-by-click scripts (Writing good tests), and use retries or trials where a single run isn’t enough.
• Every model must support tool calling. The engine is entirely tool-driven. The Test connection button on a provider checks this before you rely on it.
• CAPTCHAs are detected, never solved. A run that hits one is marked blocked, with advice on test keys and allowlisting. See One-time codes and CAPTCHAs.
• Browsers run headless in containers (Chromium, Firefox and WebKit, with desktop, iPhone, Pixel and iPad viewports).
• Set a strong SECRET_KEY first. It signs logins and encrypts every stored key and secret. The Quickstart shows how to generate one.

Next steps

QuickstartFrom git clone to a passing test in about five minutes.

InstallationRequirements, ports, first-run notes and local development.

ConceptsProjects, modules, tests, runs, providers and judges.

Tour of a projectWhere everything lives in the app.