Benchmarks

The repository ships a benchmark harness, bench/, that measures Plainwright with different AI models on the same public practice sites, and can put other AI testing tools next to it. This page describes the method, the suite and how to reproduce a run.

Results: not yet measured

No benchmark has been run against a real AI model yet, so there are no published numbers. The harness is built and unit-tested against a fake API, and it has been checked end to end against the real backend without any model. Results will be published here together with the raw results.jsonl, the suite hash and the date of the prices used.

What it measures

For each tool and model, over every trial of every task:

Metric	Definition
Pass rate	Expected-pass runs that passed, divided by all expected-pass runs, with a Wilson 95 % interval.
False-pass rate	Control runs reported as passed, divided by all control runs. A false pass is the most expensive mistake a testing tool can make.
Flaky tasks	Tasks with two or more runs whose outcomes mix pass and not-pass, out of tasks with two or more runs.
Errors	Runs that ended in error or hit the bench’s own timeout. They stay in the pass-rate denominator.
Median and p90 seconds	The app’s duration_ms (execution only, queue wait excluded), summed over attempts. Percentiles interpolate between the closest ranks.
Median tokens	Input plus output tokens per AI run (ui and api tasks).
$ per test	The run’s tokens priced with the bench’s price list (median over AI runs; the total covers every run). The server’s own estimate (cost_usd) is recorded next to it.
Self-healing	From each run’s evidence timeline: failed browser actions, how many a later action recovered, stale-reference versus ambiguous-target failures, and stuck-detector stops.
Per tag	The pass rate of each tag (login, iframes, dialogs …), and the false-pass rate of control.

The verdict is the run’s effective status, so a human verdict override wins over the raw status, as it does in the app.

Not measured: authoring time, maintenance as an app changes, test generation (the Crawler), CI integration, visual regression, mobile devices, and evals of AI products. The cost leaves out prompt-caching discounts and the hardware behind local models.

The suite

23 tasks on four public practice sites: 17 browser (UI) tasks, 3 API tasks and 3 site-quality tasks. Three of them (two browser, one API) are controls. Every task was checked against the live sites on 21 October 2026. The tasks are YAML files in bench/suite/, each with an id, kind, base_url, the plain-English natural_language, the expected outcome, a difficulty and tags.

Browser tasks (15)

Every one is expected to pass.

Task	Site	Difficulty	What it checks
saucedemo-login	saucedemo.com	easy	The standard user logs in
saucedemo-locked-out	saucedemo.com	easy	The locked-out user sees an error
saucedemo-sort-price	saucedemo.com	medium	Sort products by price
saucedemo-checkout	saucedemo.com	hard	Buy a backpack end to end
internet-form-auth	the-internet	easy	Log in and out of the secure area
internet-checkboxes	the-internet	easy	Toggle checkboxes
internet-dropdown	the-internet	easy	Choose a dropdown option
internet-dynamic-loading	the-internet	medium	Wait for content rendered after a delay
internet-dynamic-controls	the-internet	medium	Enable a disabled input
internet-iframe	the-internet	medium	Read text inside an iframe editor
internet-file-upload	the-internet	medium	Upload a file
internet-tables	the-internet	medium	Read a data table
internet-js-confirm	the-internet	medium	Accept a JavaScript confirm dialog
todomvc-add-complete	demo.playwright.dev	medium	Add, complete and filter todos
practice-login	practicetestautomation.com	easy	Log in

API and site-quality tasks (5)

Every one is expected to pass.

Task	Kind	Site	What it checks
api-jsonplaceholder-get-post	api	jsonplaceholder	Read a post
api-jsonplaceholder-create-post	api	jsonplaceholder	Create a post
quality-internet-links	quality	the-internet	No broken links on the home page
quality-practice-seo	quality	practicetestautomation.com	SEO basics
quality-todomvc-performance	quality	demo.playwright.dev	Page performance within generous budgets

Controls (3)

Each is expected to fail.

Task	Kind	Site	The deliberate mistake
control-saucedemo-wrong-price	ui	saucedemo.com	The backpack’s price is wrong
control-internet-wrong-password	ui	the-internet	Log in with a wrong password
control-api-wrong-user-name	api	jsonplaceholder	User 1 has the wrong name

• Controls carry a deliberately wrong expectation (the backpack costs $29.99, not $19.99; the password is wrong). A correct agent must report failed; a passed is a false pass.
• internet-js-confirm needs the agent to answer a JavaScript confirm dialog. Plainwright gives the agent browser_handle_dialog from 2.0.0 (see UI tests); the task stays in the suite so the number shows whether that works, and how other tools handle dialogs.
• Quality tasks are scripted browser checks with no AI model in the check. They measure the determinism and speed of the non-AI path and aren’t compared across tools.

Fairness rules

1. Same tasks, same words. Every tool gets the YAML natural_language verbatim. Only tool-specific glue is allowed (imports, the opening goto). Don’t tune wording for one tool.
2. Same model when comparing tools. Rank tools against each other only on the same model. If a tool can’t use that model, report it separately.
3. Same machine, network and time window, with the same headless Chromium at a desktop size. Interleave tools across trials if you can. Timings from different machines aren’t comparable.
4. First attempts. No automatic retries, unless every tool retries the same number of times. The records keep attempts and first_status either way.
5. No replay caches. Turn off any cache that replays earlier actions without the model, or trials 2 and 3 measure the cache instead of the agent.
6. Same order, shuffled. The task order is shuffled per trial from --seed, and every model gets the same order.
7. Warm sites. Before each model’s runs, the bench sends one plain request to every target site (no model involved), so a sleeping host doesn’t charge its cold start to one task.
8. Concurrency 1 for timing claims. Parallel runs share CPU and the provider’s rate limits.
9. Everything counts. Errors and timeouts stay in the denominator. Publish the raw results.jsonl with the report and never drop runs.
10. Pinned suite. Results carry the suite hash. Compare runs only when the hashes match.

How the bench drives Plainwright

Everything goes through the public REST API with your own account.

• Project. The bench creates or reuses a dedicated project, Plainwright Bench, turns its alerts off, sets its automatic retries to --retries (default 0) and records the suite name and hash in its settings.
• Environments and files. Each target site gets an environment, and every run is triggered against its task’s environment. The fixture file is uploaded to the project’s files.
• Tests are created with POST /api/crawler/create, which skips the planned-steps preview call that creating a UI test makes. A test that has drifted from its YAML is deleted and recreated.
• Model switching. The worker picks a run’s model when the run starts, from the project’s AI model for runs. So the bench waits until the project has no pending or running runs, switches the model, checks the switch, and only then triggers that model’s runs. It also checks that each run’s ai_model matches and flags any mismatch.
• Polling and resume. Each run is polled until it finishes, following retries to the last attempt. Every record is written to results.jsonl as soon as its run ends, and --resume skips records that are already there.
• Cost cap. The bench refuses to start when the estimate is over --max-cost, and stops triggering new runs once measured spend plus the runs in flight would pass it.

Reproduce a run

You need a running Plainwright (see the Quickstart), the backend’s Python virtual environment for the bench scripts, and an AI provider with a tool-capable model. Run the commands from the repository root.

1. See the plan and the cost estimate. A dry run calls nothing, not even the app:

   PowerShell

   backend\.venv\Scripts\python bench\run_bench.py --dry-run --models claude-haiku-5-5 --trials 3

   bash

   backend/.venv/bin/python bench/run_bench.py --dry-run --models claude-haiku-5-5 --trials 3

2. Add the model in the app. Open AI providers → Add provider, pick the model, give it a label such as Claude Haiku, and click Save and test. The bench only uses providers you created.

3. Run it. The bench asks before it spends anything, and --max-cost is a hard cap in US dollars:

   PowerShell

   $env:PLAINWRIGHT_EMAIL = "you@example.com"; $env:PLAINWRIGHT_PASSWORD = "your-password"
   backend\.venv\Scripts\python bench\run_bench.py --models "Claude Haiku" --trials 3 --max-cost 2

   bash

   export PLAINWRIGHT_EMAIL=you@example.com PLAINWRIGHT_PASSWORD='your-password'
   backend/.venv/bin/python bench/run_bench.py --models "Claude Haiku" --trials 3 --max-cost 2

4. Write the report. The run prints its directory when it ends:

   PowerShell

   backend\.venv\Scripts\python bench\report.py bench\results\<UTC timestamp>

   bash

   backend/.venv/bin/python bench/report.py bench/results/<UTC timestamp>

The bash commands use the Linux and macOS path of the virtual environment; in Git Bash on Windows it’s backend/.venv/Scripts/python. The remaining examples use the bash form.

Flag	What it does
--models "A,B"	AI-provider labels, ids or model ids. default means no project override: your default provider, else the server key.
--trials N	Trials per task per model, 1–10 (default 3).
--only / --skip	Keep or drop tasks by tag or id, such as --only login,iframes or --skip control. Each task is also tagged with its kind.
--retries N	The bench project’s automatic retries (default 0, so first attempts are measured).
--concurrency N	Runs in flight at once (default 1, for the cleanest timings).
--resume --out <dir>	Continue an interrupted run.
--browser, --device, --seed	Browser, device preset and shuffle seed.
--base-url	The Plainwright instance (default http://localhost:8080, or PLAINWRIGHT_URL).
--yes	Skip the confirmation prompt, for scripts.

Exit codes: 0 finished, 2 usage or configuration error, 3 stopped by --max-cost, 4 API error.

Compare another tool

bench/adapters/ exports the suite for Shortest, TesterArmy e2e and Auto Playwright and explains how to run each one fairly (same model, no replay cache). Append their runs to a results.jsonl in the same format, then merge them into one report:

backend/.venv/bin/python bench/report.py bench/results/<run> --results other-tool.jsonl --suite bench/suite

The adapters follow each tool’s README and haven’t been run against the real tools yet, so check the commands against the version you install.

Cost

The dry run estimates one model × 3 trials = 69 runs, 60 of which call a model. It assumes a quarter of expected-pass attempts fail and every control fails, prices the tokens with bench/pricing.json, and adds a ×1.25 safety margin.

Model	Provider	Estimate for 69 runs	About $ per run
openai/gpt-oss-20b	Groq	$0.41	$0.006
claude-haiku-5-5	Anthropic	$0.56	$0.008
gpt-6-luna	OpenAI	$0.56	$0.008
gpt-4o-mini	OpenAI	$0.82	$0.012
openai/gpt-oss-120b	Groq	$0.82	$0.012
gemini-3.5-flash-lite	Gemini	$1.83	$0.027
gemini-3.8-flash	Gemini	$4.20	$0.061
claude-haiku-4-5	Anthropic	$5.60	$0.081
claude-sonnet-5-5	Anthropic	$11.20	$0.162
gpt-6-sol	OpenAI	$11.20	$0.162
claude-opus-5-5	Anthropic	$22.40	$0.325
Any Ollama model	Local	$0	$0

That is about 3.9 million input and 118,000 output tokens per model. With --retries 1, claude-haiku-5-5 goes to about $0.73.

• Prices were read from each vendor’s own pricing page on 21 October 2026; each row in pricing.json names its source. Groq’s Llama models are listed with their last public price but marked unverified, because Groq now offers them to enterprise customers only. gpt-4o-mini shuts down on 14 January 2027, so results on it can’t be repeated after that.
• Token counts are priors, not measurements: 45,000, 90,000 and 180,000 input tokens for easy, medium and hard UI tasks. After the first real run, results.json reports the measured medians; copy them into pricing.json and the next estimate will be close.
• Failure explanations are counted in each run’s own tokens since 2.0.0, so the bench adds nothing for them. Against an older server, set the overhead in pricing.json back as its note says.

Output

A run directory, bench/results/<UTC timestamp>/ (git-ignored), holds:

• results.jsonl: one line per task × model × trial with the verdict (status, effective_status, first_status, outcome, correct, false_pass), attempts and flakiness, duration_ms and wall time, tokens, cost_usd_est and the server’s cost_usd_reported, the self-healing signals and the run’s link.
• meta.json: the suite hash, models, trials, seed, concurrency, retries, estimate and spend.
• results.json, RESULTS.md and charts/*.svg, written by report.py: a summary per tool and model and seven bar charts (pass rate, false-pass rate, flaky rate, median and p90 seconds, median tokens and median cost) in light and dark.

The harness has its own tests, which use a fake Plainwright API and never touch the network or a model:

backend/.venv/bin/python -m pytest -q bench/tests

Next steps

How Plainwright comparesFeatures of 18 products side by side, with sources.

AI providersModels, prices and which one a run uses.

Writing good testsThe same advice the suite's wording follows.

Trials, thresholds and costHow runs are priced in the app.