Changelog

Notable changes in each release. Plainwright follows semantic versioning: breaking changes bump the major version.

2.0.0 Latest

October 2026. The first open-source release. TestAI is now Plainwright, and it tests AI products as well as web apps.

Upgrading from 1.x

Migrations are automatic. The schema is now managed with Alembic. On first start the backend recognises a 1.x database, marks it as the baseline, and upgrades it in place. Existing projects, tests and runs are kept; old runs read as Web UI runs. You no longer run manual ALTER statements.
Database names are unchanged. The internal database and user stay testai / testai_db, so existing volumes keep working.
SECRET_KEY must match on the backend, workers and scheduler, because provider keys, secrets and integration configs are encrypted with a key derived from it.
Replace a default or short SECRET_KEY. Owners and admins now see a warning while the key is missing, a publicly known default or shorter than 32 characters, and with PLAINWRIGHT_ENV=production the backend refuses to start. Change it with python -m app.cli rotate-secret-key, which re-encrypts stored data, never by editing .env alone. Rotate SECRET_KEY
Email addresses must be confirmed for invitations. The security migration lower-cases stored addresses and marks accounts that signed in with Google or GitHub as confirmed. Password-only accounts start unconfirmed: they confirm through the emailed link (needs SMTP), a password reset, or python -m app.cli verify-email <email>. Confirming email addresses
Pick a current model. The provider model lists only offer current, tool-capable models. Saved providers on a retired model (such as gpt-4-turbo, o1-mini, Gemini 1.5 and 2.0, or Groq’s Llama models without enterprise access) should be switched to a listed model and tested again. AI providers
CI secrets are renamed to PLAINWRIGHT_URL, PLAINWRIGHT_PROJECT_ID and PLAINWRIGHT_WEBHOOK_SECRET. The bundled workflow still accepts the old TESTAI_* names.

Breaking changes

GET /api/projects and GET /api/projects/{id} return webhook_secret: null and a masked notify_webhook to viewers and editors. Only owners and admins see the credentials. CI that read the secret through the API as a non-admin needs an admin account (or a stored secret).
Alert webhooks, API-test targets, eval targets, MCP servers, A2A agents, tracker hosts, IMAP hosts and the URLs the browser agent opens are refused when they point at cloud-metadata addresses, loopback (localhost), the stack’s own services (including frontend and nginx) or the Docker host on the platform’s own ports. Set ALLOW_INTERNAL_TARGETS=true if you really need an internal target. Outbound requests
Ollama and custom OpenAI-compatible endpoints go through the same guard; host.docker.internal stays allowed for Ollama on any port that isn’t one of the platform’s.
Google and GitHub sign-in no longer put a token in the callback URL. The callback carries a one-time code (#code=…) that the app trades with POST /api/oauth/exchange, only in the browser that started the sign-in. A Google or GitHub sign-in links to an existing account only when that account’s address is confirmed.
Changing or resetting a password, POST /api/auth/logout-all and removal from your last project revoke every older token. POST /api/auth/change-password returns a fresh access_token.
Sign-in, registration and password-reset requests are rate-limited (429).
The browser services moved to their own browser network, and the frontend’s port 3000 is published on 127.0.0.1 only.
The server’s fallback model is claude-haiku-5-5 (was claude-haiku-4-5), and a project without a judge model judges with the cheap model of its run provider instead of the run model.
Filing an issue returns the tracker’s own error status (400, 404 or 502) instead of 501.

Test AI products

Conversation tests: a simulated user with a persona and a goal talks to your chatbot through its website chat widget, an HTTP API or an A2A agent; a judge scores every reply and the whole conversation against your success criteria. Conversation tests
Red teaming with seven attack packs: prompt injection, jailbreak, system-prompt leak, PII leak, off-topic, harmful content and tool misuse. Each attack is judged resisted or breached, and a leaked protected secret is a breach without a judge call. Red teaming
LLM evals over CSV, TSV, JSONL or JSON datasets with correctness, relevance, safety, custom rubric and latency metrics, and RAG evals with faithfulness, answer relevance, context precision and context recall. LLM evals, RAG evals
MCP server tests: an automatic tool-contract check, then an agent that exercises the tools, over Streamable HTTP or SSE. MCP servers
A2A agent tests: agent-card validation, then a conversation over A2A JSON-RPC. A2A agents
Trials and scores: run a test up to 10 times and judge the pass rate or mean score against a threshold. Judging defaults to a cheap model of the run’s provider, or pick a judge model per project. Trials, thresholds and cost
Cost estimates: every run records an estimated cost in US dollars, pricing each model call at its list price; Analytics totals it by model. Estimate the cost

Test web apps

API tests: an agent sends real HTTP requests, with an OpenAPI spec as optional context. API tests
Site quality checks: accessibility (axe-core), performance, broken links and SEO, with no AI model and no token cost. Site quality checks
Plain-English assertions judged against the final page and screenshot, and visual regression with baselines per browser and device, diffs, ignore regions and one-click approval. Assertions and visual regression
Secrets vault: encrypted project secrets the AI only ever sees as {{secrets.NAME}} placeholders, scrubbed from everything stored or shown. Saved logins: sign in once and start other tests already signed in. Secrets and saved logins
One-time codes: authenticator (TOTP) secrets and a connected email inbox for emailed codes and magic links. CAPTCHA detection marks blocked runs with advice instead of retrying them. One-time codes and CAPTCHAs, Email inbox (IMAP)
Templates: 26 starter tests across every test kind.
JavaScript dialogs: the agent answers alert, confirm and prompt dialogs with browser_handle_dialog. UI tests
Crawler Stop cancels a crawl on the server before its AI call. Crawler

Reliability and triage

Automatic retries (per project and per test), flaky-test detection and quarantine. Retries and flaky tests
Rerun failed, for the whole project or one batch. Quarantine and re-run
Triage: failures grouped by cause, new versus known, and one-click issues in Jira, Linear and GitHub, de-duplicated per cause, or filed automatically for every new cause. Triage and issues
AI root cause with a category and a suggested fix, and a chat about any run, both on demand. Root cause and run chat
Human verdict overrides, used by gates, Triage and Analytics. Verdict overrides

Integrations and CI

CI gate: the webhook returns a batch, and GET /api/batches/{id} reports pass, fail or pending using the latest attempt, overrides, score thresholds and quarantine. A ready-made GitHub Actions workflow prints a summary table. CI webhook and gates
Webhook secret rotation from Settings → General → Run from CI.
Email alerts alongside Slack, Discord and webhook alerts, with quiet rules for quarantined tests, pending retries and flaky passes. Discord webhook URLs now work as pasted. Alerts
Imports from Jira, Linear, GitHub, Figma and Notion into suggested tests. Jira, Linear and GitHub, Figma and Notion imports

Google and GitHub sign-in and password reset by email. The buttons only appear when the server is configured for them. Sign-in and email
Email confirmation, Sign out other sessions in Settings → Security, and a warning banner for owners and admins when the server is misconfigured (GET /api/system/warnings).
A guided setup for new accounts: add a provider, create a project, pick templates, and watch the first run live.
A redesigned app: calm, light and dark themes, a command menu (Ctrl+K or ⌘K), a project board canvas, drawers for details, a run page with tabs per kind (transcript, results, evidence, visual, quality, server, agent card, chat), and layouts that work down to phone width.
Analytics adds score trends per kind and metric, reliability counts and usage by model. Analytics

Self-hosting and operations

Parallel workers (WORKER_REPLICAS, default 2). Scaling workers
S3, R2 and MinIO storage for screenshots and evidence. Object storage
Monitoring: Prometheus metrics, JSON logs and Sentry. Monitoring
Optional billing with Stripe and usage quotas, off by default. Billing
Deploy guides for Render (with a blueprint), Railway, Fly.io and a VPS.
Security: a production mode that refuses a weak SECRET_KEY, a rotate-secret-key command, revocable sessions, rate limits, hardened nginx headers, an outbound-request guard with a DNS check that also covers the browser’s navigations and uploads, secrets that only reach the project’s own hosts and are never returned by the API, and /metrics access rules (METRICS_TOKEN). Security model
Apache License 2.0, with contributions signed off under the Developer Certificate of Origin. License

Quality

The release ships with about 1,700 backend tests and 800 frontend tests. Unit tests never call a real LLM, network service or Docker.

1.x (TestAI)

The releases before the rename, as TestAI:

Plain-English Web UI tests driven by an AI agent through Playwright MCP, reading the live page instead of selectors, with automatic recovery from stale and ambiguous elements.
Six AI providers: Anthropic, OpenAI, Gemini, OpenRouter, Groq and Ollama (with live model discovery and model downloads), each checked for tool calling.
Chromium, Firefox and WebKit, with desktop, iPhone, Pixel and iPad devices.
Environments, per-project test files, cron scheduling with a stuck-run reaper, and the CI webhook.
Run evidence: steps, a guaranteed final screenshot, per-action screenshots, console and network logs, a live run view, and an AI explanation of failures.
Projects, modules and tests on a board; the AI Crawler (URL, PDF or OpenAPI to suggested tests).
Members with viewer, editor and admin roles; email invitations; Slack, Discord and webhook alerts; analytics for pass rate, flakiness and token usage.

Next steps

RoadmapWhat's coming next.

Upgrades, migrations and backupsUpgrade a running server safely.