Skip to content
Plainwright

Glossary

Short definitions of the terms used in the app and in these docs, in alphabetical order. Names in code are the values you’ll see in the API and in test configuration.

A

Section titled “A”
  • A2A (Agent2Agent). An open protocol for agents to call each other over JSON-RPC. An a2a test validates an agent’s card and then talks to it. See A2A agents.
  • Accessibility snapshot. A text outline of the page (roles, names, states) that the browser agent reads on every step instead of using CSS selectors. Each element in it has a ref.
  • Agent. The AI model working in a loop with tools: browser tools for Web UI tests, http_request for API tests, the server’s own tools for MCP tests.
  • Agent card. The JSON document an A2A agent publishes at /.well-known/agent-card.json (or /.well-known/agent.json) describing its name, skills and endpoint.
  • Assertion. A plain-English statement checked by the judge at the end of a Web UI run, such as “The receipt shows an order number”. See Assertions and visual regression.
  • Attack. One red-team strategy, such as prompt_injection or pii_leak. There are seven. See Red teaming.
  • Attempt. The number of a run within its retries. The first run is attempt 1; an automatic retry is attempt 2, and so on.

B

Section titled “B”
  • Base URL. Where a project’s tests start. Environments add more base URLs.
  • Batch. Runs started together: a module, Run all, the CI webhook or Rerun failed. They share a batch_id, and the CI gate is computed per batch.
  • Blocked. A run that stopped at a CAPTCHA (blocked_reason: "captcha"). Plainwright detects CAPTCHAs; it never solves them.

C

Section titled “C”
  • CI gate. The verdict for a batch: pass, fail or pending. CI polls it after calling the webhook. See CI webhook and gates.
  • Config. A test’s kind-specific settings (config): target URL, persona, dataset, metrics, pages and so on.
  • Contexts. The passages a RAG system retrieved for a question, read from its response with contexts_path and used by the RAG metrics.
  • Contract check. The rule-based part of an MCP test: every tool needs a name, a description and a valid input schema.
  • Crawler. Reads a website, a PDF or an OpenAPI spec and suggests tests. See Crawler, templates and imports.

D

Section titled “D”
  • Dataset. Rows of questions (input) with optional expected, rubric and context columns, used by LLM and RAG evals. See Datasets.
  • Default (environment). The project’s own base URL, used when no other environment is selected.
  • Default provider. The AI provider your projects use unless a project picks another.

E

Section titled “E”
  • Effective status. The status that counts: a human override if there is one, otherwise the engine’s status. See Judges and scores.
  • Engine. The code that runs one kind of test.
  • Environment. A named base URL for a project, such as Staging. See Environments.
  • Error. A run status meaning the test couldn’t run properly (configuration, provider or infrastructure problem), so there is no real verdict. Compare failed.
  • Eval. Short for LLM eval (eval): scoring an AI system’s answers over a dataset. See LLM evals.
  • Evidence. What a run collected to back its verdict: screenshots after each action, console and network logs, requests, transcripts.

F

Section titled “F”
  • Failed. A run status meaning the test ran and the product didn’t meet its expectations.
  • Failure signature. A fingerprint of a failure’s error, used to group similar failures in Triage.
  • Flaky. A run that passed after an earlier attempt failed, or a test whose results flip between pass and fail. See Retries and flaky tests.

J

Section titled “J”
  • Judge. An LLM call that grades evidence against criteria or metrics and returns scores with reasoning.
  • Judge model. The model used as the judge: the project’s Judge model setting, or else a cheap model from the run model’s provider, such as claude-haiku-5-5.

K

Section titled “K”
  • Kind. The type of a test: ui, api, quality, conversation, redteam, eval, rag, mcp or a2a. See Test kinds.

L

Section titled “L”
  • Live view. The run page while a run is in progress, streaming browser frames, actions, conversation turns or eval progress.

M

Section titled “M”
  • MCP (Model Context Protocol). A protocol for exposing tools to AI models. Plainwright drives its browsers through Playwright MCP, and mcp tests check your own MCP servers. See MCP servers.
  • Metric. One dimension a judge scores, such as correctness or faithfulness. See Metrics reference.
  • Module. A group of tests inside a project, shown as a column on the board.

O

Section titled “O”
  • Override. A person’s verdict on a finished run (passed or failed), with a note. It changes the effective status, never the original status. See Verdict overrides.
  • Owner. The user who created a project. Runs use the owner’s default AI provider unless the project picks another.

P

Section titled “P”
  • Pass mark. The score a scored run needs to pass; also called the threshold. 80% by default, 100% for red-team tests.
  • Pass rate. The share of trials that passed, for a pass/fail test run with several trials.
  • Pending. A run status: queued and waiting for a worker.
  • Persona. Who the simulated user is in a conversation test, such as “a busy customer who writes short messages”.
  • Placeholder. {{secrets.NAME}} in a test or config. The AI only ever sees the placeholder; the real value is swapped in at the last moment.
  • Planned steps. A preview of the steps a Web UI test will likely take, drafted when the test is saved. Runs don’t follow it rigidly.
  • Playwright MCP. The open-source server (@playwright/mcp) that runs headless browsers and exposes them as tools. Each browser kind runs in its own container.
  • Project. One app or AI product: its base URL, modules, tests, runs and settings.
  • Provider. A configured AI model and its key: Anthropic, OpenAI, Gemini, OpenRouter, Groq or Ollama. See AI providers.

Q

Section titled “Q”
  • Quality. Short for site quality (quality): accessibility, performance, link and SEO checks, with no AI model. See Site quality checks.
  • Quarantine. A test that still runs but never fails the CI gate or sends alerts. See Quarantine and re-run.

R

Section titled “R”
  • RAG (retrieval-augmented generation). Answering from retrieved documents. A rag test scores both the answer and the retrieval. See RAG evals.
  • Red team. Adversarial testing: an attacker persona tries to make your assistant leak, misbehave or break its rules.
  • Ref. An element’s id in the accessibility snapshot, such as e24, which the browser agent uses to click or type.
  • Retry. An automatic new attempt after a failed run, in the same batch. See Retries and flaky tests.
  • Role. What a project member may do: viewer, editor or admin, plus the owner. See Team and roles.
  • Root cause. An on-demand AI analysis of a failed run, with a category and a suggested fix. See Root cause and run chat.
  • Run. One execution of one test, with its status, evidence and score.
  • Run chat. Questions about a run, answered from its evidence.
  • Run model. The model that drives the agent and plays simulated users for a run.

S

Section titled “S”
  • Saved login. A browser session (cookies and local storage) captured by a setup test and reused by other Web UI tests, so they start signed in. See Secrets and saved logins.
  • Schedule. A cron expression (UTC) that runs a test automatically. See Scheduling.
  • Score. A number from 0 to 1 that a scored run produces, shown as a percentage.
  • Secret. An encrypted value in a project’s vault, referenced as {{secrets.NAME}}. Names start with a capital letter and use only capitals, digits and underscores, such as QA_PASSWORD.
  • SECRET_KEY. The server’s master key. It signs sign-in tokens and is the key from which the encryption key for stored provider keys, secrets and saved logins is derived. Set it once per install; see Security model.
  • Simulated user. The model playing a user (or an attacker) in conversation and red-team tests.
  • Status. The engine’s verdict for a run: pending, running, passed, failed or error.
  • Success criteria. What must be true at the end of a conversation test, each graded by the judge.
  • System default model. The server-wide fallback: Anthropic’s claude-haiku-5-5 with the server’s ANTHROPIC_API_KEY, used when a project and its owner have no provider.

T

Section titled “T”
  • Target. What a conversation, red-team or eval test talks to: a website chat widget (browser), an HTTP endpoint (http) or an A2A agent (a2a).
  • Template. A ready-made test you fill in with a few values. There are 26, across every kind.
  • Test. A plain-English description plus settings, in a module. Also called a test case.
  • Threshold. See pass mark.
  • Trial. One repetition of a test within a single run. A run can have 1 to 10 trials.
  • Triage. The project tab that groups similar failures, flags new ones and lists flaky and quarantined tests. See Triage and issues.
  • Trigger. What started a run (triggered_by): manual, module_run, project_run, ci_webhook, schedule, retry, rerun-failed or auth capture.

V

Section titled “V”
  • Visual baseline. The approved screenshot a Web UI test’s final screenshot is compared with, one per test, browser and device.
  • Visual regression. Failing a run when its final screenshot differs from the baseline by more than the threshold (0.5% of pixels by default).

W

Section titled “W”
  • Webhook secret. The project secret CI sends in the X-Webhook-Secret header to start a batch. Admins can see and rotate it.
  • Worker. The background process that executes runs, one at a time. Add workers to run more tests in parallel. See Scaling workers.