Writing good tests

A plain-English test is a brief for an AI agent. A good brief says where to start, what to do, what data to use and exactly what proves success. This page collects the patterns that make UI tests and API tests pass for the right reasons, fail for the right reasons, and cost as little as possible.

The short version

1. One user goal per test. “Sign up”, “Apply a coupon”, “Change the password”. Not all three.
2. Say what proves success. Name the exact text, number, page or state you expect.
3. Use the words on the screen. Refer to buttons and fields by their visible labels, in quotes.
4. Give the data. Which product, which coupon, which account. Use secrets for credentials.
5. Put end-state checks in assertions. They are judged against the final screenshot and page text.
6. Keep it under about 15 actions. A run stops at 30 tool calls.

Be specific about the result

The agent decides the verdict from your words. If you don’t say what success looks like, it guesses.

Vague

Test the checkout.

The agent has to invent the flow, the product, the payment method and the pass condition. Two runs may test two different things.

Specific

Search for "Classic T-shirt" and open the first result.
Choose size M and click "Add to cart".
Open the cart and click "Checkout".
Fill in the shipping form with a realistic US address and continue.
Verify the order summary lists one Classic T-shirt (M) and a shipping cost.
Do not submit any payment.

Every step is something the agent can do and every check is something it can read on the page.

Good verbs for checks are verify, check that and expect, followed by something observable: “verify the page heading is ‘Order confirmed’”, “check that the cart badge shows 2”, “expect an error saying the email is already registered”.

Refer to what the user sees

The agent reads the page’s accessibility tree, which is built from labels, roles and visible text. Describe elements the same way:

• Good: click “Save changes”, the “Email” field, the user menu in the top right, the second row of the Orders table.
• Avoid: CSS classes, XPath, element ids, or “the blue button” (the agent doesn’t see colour; it reads text).

If the visible text is ambiguous, say which one: “the ‘Delete’ button in the row for invoice INV-42”. When a page truly has no usable labels, a unique CSS selector in quotes works as a last resort, because the agent’s tools accept one.

Give the data, and make it unique

Tests that create things should not collide with earlier runs.

Sign up with a new email address of the form qa+<current timestamp>@example.com
and the password {{secrets.SIGNUP_PASSWORD}}.

• Credentials: store them as secrets and write {{secrets.NAME}}. The model types the placeholder and never sees the value.
• Fixed inputs (coupon codes, product names, search terms): write them in quotes.
• Generated inputs: ask for a unique value, as above, so reruns don’t fail on “already exists”.
• Files to upload: upload them to the project’s Files and refer to them by name.

Use assertions for the end state

Steps describe the journey; assertions describe the destination. After a passing run, an AI judge checks each assertion against the final screenshot and the final page text, and any failed assertion fails the run. This is also the place for anything visual, because the judge sees the screenshot while the agent only reads text.

Assertions
  The page heading is "Order confirmed"
  An order number is shown
  No error message is visible

See Assertions and visual regression.

Keep tests short and independent

• One flow per test keeps failures easy to read and runs fast. A run stops at 30 tool calls, and each step resends the conversation, so long tests cost more per step.
• Don’t chain tests. Each run starts in a fresh, signed-out browser. If many tests need to be signed in, use a saved login instead of logging in at the top of every test.
• Group related tests in a module so you can run them together from the board.

Handle waiting and dynamic pages

The agent can wait for text to appear or disappear. Tell it what to wait for when something is slow:

Click "Generate report" and wait until the text "Report ready" appears (it can take up to 30 seconds).
Then verify the report table has at least one row.

Pages that change after a click are handled automatically: the agent re-reads the page and retries when an element it was about to use is gone. You don’t need to write “refresh the page” or “try again”.

Things that are hard for the agent: content drawn on a <canvas>, drag-and-drop with precise coordinates, and elements inside cross-origin iframes such as embedded payment widgets. Test those flows with a provider’s test mode, or check the outcome instead of the interaction.

Don’t do real damage

The agent does exactly what you ask on a real site. Point tests at staging with an environment, use test accounts, and say what not to do:

Proceed to the payment step and verify the card form is shown. Do not submit any payment.

Choose a model that can drive a browser

Every model you use must support tool (function) calling. Test connection in AI settings checks this.

• Small, fast models are fine for short, specific tests, but they sometimes describe an action instead of calling the tool. If runs stop early or report steps that never happened, try a more capable model.
• Some open models on Groq or OpenRouter occasionally produce an invalid tool call. Re-running usually works.
• Reasoning models are supported; their hidden reasoning is stripped before the result is read.

See AI providers.

Keep runs cheap

Habit	Why it saves tokens
Specific instructions	Fewer exploratory steps. Every step resends the conversation.
Short tests	Cost grows with the number of steps.
Saved logins	Skips the sign-in steps in every test.
Assertions instead of extra steps	One judge call at the end instead of more agent steps.
API tests for endpoints	No browser and much smaller tool results than page snapshots.
Quality checks for a11y, perf, links and SEO	No AI model at all.
Trials only where behaviour varies	trials: 5 costs five runs.

Plainwright already trims context for you: only the two most recent tool outputs are kept in full and older large ones are replaced with a stub, because the agent only needs the current page.

When a test fails

Open the run and read, in order: the explanation, the failed step, then the Evidence screenshots around it. These messages mean something specific:

Message	Meaning	What to do
Stopped after 5 consecutive failed actions	The agent couldn’t find or use an element, even after re-reading the page.	Check the element exists and is visible in that state; describe it by its visible label; try a stronger model.
Max tool calls reached (status error)	The agent used all 30 tool calls without finishing.	Split the test, or make the steps more specific.
Assertion failed: …	The flow passed but an end-state check didn’t.	Read the judge’s reasoning under the assertion.
Visual change: X% of pixels differ	The final screen differs from the approved baseline.	Approve the change, or add an ignore region.
Could not reach or drive the browser automation server	The Playwright MCP service is down or still starting.	Check the playwright-mcp containers. See Troubleshooting.
The agent reported a pass without sending any request	An API test claimed success without calling the API.	Make the request explicit: “Send GET /health …”.

If the verdict itself is wrong, a human can override it. For intermittent failures, turn on retries.

Ask the run

On a finished run, Chat answers questions such as “why did step 3 fail?” using the run’s steps, errors and evidence, and Analyze root cause suggests a fix. Both are one AI call each, only when you click. See Root cause and run chat.

Next steps

UI testsHow the browser agent works and what it can do.

Assertions and visual regressionPlain-English end-state checks and screenshot baselines.

Crawler, templates and importsStart from drafted tests instead of a blank page.

Retries and flaky testsDeal with tests that fail only sometimes.