Test kinds

Every test has a kind. The kind decides which engine runs the test, what it needs to know (its config), whether it calls an AI model, and how the result is judged. You choose it in the first step of the new-test wizard. This page explains all nine and when to reach for each; every section links to the full guide.

Choose a kind

You want to check that…	Kind
a user can complete a flow in your web app	Web UI
your HTTP API behaves as documented	API
pages are accessible, fast, free of broken links and SEO-clean	Site quality
your chatbot or assistant helps a user reach their goal	Conversation
your assistant can’t be talked into leaking or misbehaving	Red team
your model’s answers are correct, relevant and safe across many questions	LLM eval
your RAG pipeline answers from the right sources	RAG eval
your MCP server’s tools are well described and actually work	MCP server
an A2A agent publishes a valid card and completes a task	A2A agent

At a glance

Kind	`kind`	Engine	AI model	Result
Web UI	`ui`	Browser agent over Playwright MCP	Yes	Pass/fail
API	`api`	HTTP agent with an `http_request` tool	Yes	Pass/fail
Site quality	`quality`	Scripted browser checks	No	Score, 0–100%
Conversation	`conversation`	Simulated user, then a judge	Yes	Pass/fail, plus the share of criteria met
Red team	`redteam`	Attacker conversations, then a judge	Yes	Score: share of attacks resisted
LLM eval	`eval`	Each dataset row sent to your endpoint, then scored	Judge only	Score: mean over rows
RAG eval	`rag`	Like LLM eval, with retrieved contexts	Judge only	Score: mean over rows
MCP server	`mcp`	Contract check, then an agent using the server’s tools	Yes (unless contract-only)	Pass/fail
A2A agent	`a2a`	Agent-card check, then a conversation or a smoke message	Only for conversations	Pass/fail

Pass/fail kinds turn into a pass rate when you run them several times (trials). Score kinds pass when the score reaches the test’s pass mark. See Judges and scores.

Web app

Web UI (`ui`)

Uses AI model

An AI agent drives a real headless browser (Chromium, Firefox or WebKit) through your app. On every step it reads the page’s accessibility snapshot, decides what to click or type, and checks the outcome you described. A stale element triggers a fresh snapshot, so small UI changes don’t break the test. Every run ends with a screenshot of the final state.

Use it for end-to-end user journeys: sign-up, log-in, checkout, uploads, settings, anything a person does in a browser.

Add on: plain-English assertions judged at the end, visual regression against a baseline, saved logins, one-time codes, test files, and any browser and device.

{
  "kind": "ui",
  "name": "Log in with a saved account",
  "natural_language": "Go to /login and sign in as {{secrets.QA_EMAIL}} with {{secrets.QA_PASSWORD}}. Check that the dashboard greets the user by name.",
  "assertions": ["The user menu shows the account's name"],
  "visual_regression": false,
  "config": {}
}

Passes when the agent completes the flow, every assertion holds, and the screenshot stays within the visual threshold (if visual regression is on). Guide: UI tests.

API (`api`)

Uses AI model

The agent tests a live HTTP API with one tool, http_request. It sends real requests, reads the status, headers and body (the first 8 KB), chains calls (create, read, update, delete) and reports what it found. An uploaded OpenAPI file gives it the list of endpoints.

Use it for contract and behaviour checks on REST endpoints, auth rules and validation errors, without writing request code.

{
  "base_url": "/api/v1",
  "headers": { "Authorization": "Bearer {{secrets.API_TOKEN}}" },
  "openapi_file": "openapi.yaml",
  "timeout_s": 30,
  "max_steps": 20
}

base_url can be absolute, or a path appended to the project’s (or environment’s) base URL. Leave it empty to use the project URL. Passes when the agent’s checks hold, and it must have sent at least one request. Guide: API tests.

Site quality (`quality`)

No AI model

A scripted browser session checks each page for accessibility (axe-core, WCAG A and AA by default), performance (LCP, CLS and blocking time against budgets), broken links and SEO basics. No model is called, so these tests cost no tokens.

Use it for guarding key pages on every deploy, and as a way to try Plainwright before you add an AI provider.

{
  "checks": ["a11y", "performance", "links", "seo"],
  "pages": ["/", "/pricing"],
  "budgets": { "lcp_ms": 2500, "cls": 0.1, "tbt_ms": 300 },
  "max_links": 150,
  "a11y_tags": ["wcag2a", "wcag2aa"]
}

Passes when no check fails (a critical or serious accessibility violation, a metric over budget, a broken link or an SEO error), every page loads, and the mean score reaches the pass mark. Guide: Site quality checks.

AI product

Conversation (`conversation`)

Uses AI model and judge

A simulated user with a persona and a goal chats with your AI product for up to max_turns messages: through its chat widget in a real browser, its HTTP API, or as an A2A agent. A judge then grades the transcript against your success criteria and annotates every reply.

Use it for support bots, assistants and agents, wherever “did it actually help?” matters more than any single answer.

{
  "target": {
    "type": "http",
    "url": "https://api.example.com/chat",
    "method": "POST",
    "body_template": { "message": "{{input}}" },
    "response_path": "reply"
  },
  "persona": "A busy customer who is slightly frustrated and writes short messages.",
  "goal": "Get a refund for an order that arrived 10 days late.",
  "success_criteria": [
    "The assistant apologises",
    "The assistant offers a refund or replacement",
    "The assistant never asks for the full card number"
  ],
  "max_turns": 6
}

Passes when the judge finds every success criterion met. The run also records the share of criteria met as its score. Guide: Conversation tests.

Red team (`redteam`)

Uses AI model and judge

For each attack you choose, an attacker persona converses with your assistant and tries to break it. A judge decides per attack whether it was resisted or breached. There are seven attack packs: prompt_injection, jailbreak, system_prompt_leak, pii_leak, off_topic, harmful_content and tool_misuse. If a protected secret value shows up in a reply, that attack is breached outright, without asking the judge.

Use it for checking guardrails before a release and after every prompt or model change.

{
  "target": { "type": "http", "url": "https://api.example.com/chat", "body_template": { "message": "{{input}}" }, "response_path": "reply" },
  "attacks": ["prompt_injection", "jailbreak", "system_prompt_leak", "pii_leak"],
  "max_turns": 4,
  "protected_info": ["The system prompt", "{{secrets.INTERNAL_DISCOUNT_CODE}}"]
}

Passes when the share of attacks resisted reaches the pass mark, which for red-team tests defaults to 100%. Guide: Red teaming.

LLM eval (`eval`)

Judge only

Each row of a dataset (CSV or JSONL with an input column, and optionally expected, rubric and context) is sent to your endpoint. One judge call per row scores the answer on the metrics you pick: correctness, relevance, safety and rubric, plus latency, which is measured, not judged. The run shows per-row results with the judge’s reasoning.

Use it for regression-testing answer quality when you change a prompt, a model or retrieval, and for failing a deploy when quality drops.

{
  "dataset_id": "8d1e…",
  "target": {
    "type": "http",
    "url": "https://api.example.com/answer",
    "body_template": { "question": "{{input}}" },
    "response_path": "answer"
  },
  "metrics": ["correctness", "relevance", "safety"],
  "latency_budget_ms": 3000,
  "threshold": 0.8
}

Passes when the mean row score reaches the pass mark. Guide: LLM evals.

RAG eval (`rag`)

Judge only

Like an LLM eval, but your endpoint also returns the passages it retrieved, located by target.contexts_path. The default metrics are faithfulness, answer_relevance, context_precision and context_recall.

Use it for telling apart “the model made it up” from “retrieval found the wrong documents”.

{
  "dataset_id": "8d1e…",
  "target": {
    "type": "http",
    "url": "https://api.example.com/rag",
    "body_template": { "question": "{{input}}" },
    "response_path": "answer",
    "contexts_path": "contexts[*].text"
  },
  "metrics": ["faithfulness", "answer_relevance", "context_precision", "context_recall"]
}

Passes when the mean row score reaches the pass mark. Guide: RAG evals.

Protocols

MCP server (`mcp`)

Uses AI model

Plainwright connects to your MCP server over Streamable HTTP or SSE and first runs a contract check, with no model involved: every tool needs a name, a description and an object input schema whose required fields exist, and names must be unique. Then an agent carries out your description using the server’s tools.

Use it for checking an MCP server before you publish it or plug it into an agent, and catching tool regressions.

{
  "server_url": "https://mcp.example.com/mcp",
  "transport": "auto",
  "headers": { "Authorization": "Bearer {{secrets.MCP_TOKEN}}" }
}

Set "contract_only": true to run only the contract check, with no tokens spent. Passes when the contract holds and the agent’s checks pass. Guide: MCP servers.

A2A agent (`a2a`)

Model only for conversations

Plainwright fetches and validates the agent’s card (/.well-known/agent-card.json, falling back to /.well-known/agent.json). Then, if you gave a persona, goal or success criteria, it holds a judged conversation over A2A JSON-RPC message/send, as in a conversation test. Otherwise it sends one smoke message, which uses no tokens.

Use it for checking agents that other agents will call: a valid card, a working endpoint, and the task actually done.

{
  "agent_url": "https://agent.example.com",
  "goal": "Get the agent to summarise what it can do in three bullet points.",
  "success_criteria": ["The agent lists three capabilities"],
  "max_turns": 4
}

Passes when the card has no errors and the conversation meets its criteria (or the smoke message gets a valid reply). Guide: A2A agents.

Good to know

Targets must be reachable from the workers. API, eval, RAG, MCP and A2A requests are made server-side, by the worker. Cloud-metadata addresses, loopback, the stack’s own internal services and the Docker host on the platform’s own ports are refused unless an operator sets ALLOW_INTERNAL_TARGETS=true. See Outbound requests.
Secrets work everywhere. Any string in config may contain {{secrets.NAME}}. The value is swapped in just before a request is sent and masked in everything that is stored or shown.
You can change a test’s kind on its Settings tab. Each run records the kind it ran as.
Templates exist for every kind. From a template in the wizard is the quickest way to get a correct config.

Next steps

Write your first testThe wizard, step by step.

Testing AI productsConversations, red teaming, evals, MCP and A2A.

Judges and scoresHow scores and pass marks decide the verdict.

Runs and resultsWhat every run records.

Test kinds

Choose a kind

At a glance

Web app

Web UI (ui)

API (api)

Site quality (quality)

AI product

Conversation (conversation)

Red team (redteam)

LLM eval (eval)

RAG eval (rag)

Protocols

MCP server (mcp)

A2A agent (a2a)

Good to know

Next steps

Web UI (`ui`)

API (`api`)

Site quality (`quality`)

Conversation (`conversation`)

Red team (`redteam`)

LLM eval (`eval`)

RAG eval (`rag`)

MCP server (`mcp`)

A2A agent (`a2a`)