Projects, modules and tests

Everything in Plainwright sits in a four-level hierarchy. A project is one app or AI product. It is split into modules, the columns on its board. A module holds tests, and every time a test executes it produces a run.

• DirectoryAcme Storefront project, starting at shop.example.com

  • DirectoryCheckout module

    • DirectoryPay with a saved card Web UI test

      • run 7d9ef98c passed
      • run 860f9b4b failed
    • Users API CRUD API test
    • Homepage quality site quality test
  • DirectoryAssistant module

    • Refund request is handled politely conversation test
    • Resists prompt injection and leaks red-team test

Deleting something deletes everything below it: removing a module removes its tests and their runs, and deleting a project removes all of it, including its secrets, datasets and settings.

Projects

A project is the unit of ownership, sharing and configuration. Create one from Projects → New project with a name and a base URL, the address where tests start (usually staging or production).

A project holds:

• its modules, tests and runs;
• environments: extra base URLs such as Staging or a preview deploy;
• secrets and saved logins, test files and datasets;
• integrations, notifications and a CI webhook with its secret;
• members and their roles.

Project settings that change how tests run

These live in Settings → General and apply to every test in the project unless a test sets its own.

Setting	Default	Effect
Base URL	required	Where Web UI, API and site quality tests start, unless an environment is selected.
Retries on failure	No retries (0)	Automatic re-runs of failed Web UI and API runs, 0–5. See Retries and flaky tests.
Pass mark	80%	The score a scored test needs to pass. See Judges and scores.
AI model for runs	The owner’s default provider	Which AI provider drives the agent. An admin can pick one of their own providers or one of the owner’s.
Judge model	The run provider’s cheap model	Which model scores conversations, red-team attacks, evals and assertions. See Judges and scores.

Modules

A module is a group of related tests, shown as a column on the board: Authentication, Checkout, Support bot. It has a name, an optional description and a colour. Modules are how you run part of a project: the ▷ button in a column header queues every active test in that module as one batch.

There is no rule for how to split a project. Grouping by feature or user journey works well, because a failing column then tells you which part of the product is in trouble.

Tests

A test is a plain-English description plus the settings that tell Plainwright how to run and judge it. Its kind decides the engine: a browser agent, an HTTP agent, a simulated user, a dataset scorer and so on. See Test kinds.

Field	Default	Meaning
name	required	The title shown on the board, in run lists and in alerts.
kind	ui	One of ui, api, conversation, redteam, eval, rag, mcp, a2a, quality.
natural_language	required	The description. For Web UI, API and MCP tests the agent acts on it; for the other kinds it is a note shown on reports (the API still expects the field, so send "" if you have no note).
config	{}	Kind-specific settings: the target URL, persona, dataset, metrics, pages and so on. At most 64 KB. {{secrets.NAME}} placeholders are allowed in any string.
assertions	[]	Plain-English checks judged at the end of a Web UI run (up to 20).
visual_regression	false	Compare the final screenshot with an approved baseline.
trials	1	Run the test 1–10 times per run and judge the pass rate or mean score.
retries	project default	Automatic re-runs of a failed run, 0–5.
quarantined	false	Runs, but never fails the CI gate or sends alerts.
auth_state_id	none	A saved login to start from.
schedule_cron	none	A cron expression (UTC) for scheduled runs.
is_active	true	Paused tests are skipped by module runs, Run all, the CI webhook and the scheduler. You can still run them by hand.
source, source_ref	manual	Where the test came from: manual, template, crawler, jira, linear, github, figma or notion, with a reference such as the issue URL.

Here is a Web UI test as the API returns it (GET /api/test-cases/{id}, trimmed):

GET /api/test-cases/842fa703-…

{
  "id": "842fa703-0708-44c5-820a-8eb03c9c21f2",
  "module_id": "290b01b3-16bc-40cb-95b8-32028ce481cc",
  "name": "Pay with a saved card",
  "kind": "ui",
  "natural_language": "Open the cart, go to checkout, pay with the saved Visa card and check that the receipt shows the order number.",
  "config": {},
  "assertions": ["The receipt shows an order number", "No error banner is visible"],
  "visual_regression": false,
  "trials": 1,
  "retries": null,
  "quarantined": false,
  "auth_state_id": null,
  "schedule_cron": null,
  "is_active": true,
  "source": "manual",
  "last_run_status": "failed",
  "last_run_effective_status": "failed",
  "run_count": 3
}

Where tests come from

• The wizard: Write your first test.
• Templates: 26 ready-made tests across every kind, in five categories (Web app, API, AI product, Protocols, Quality).
• The crawler: suggestions from a website, a PDF or an OpenAPI spec.
• Imports: drafts from Jira, Linear and GitHub issues, Figma files and Notion pages.

See Crawler, templates and imports.

Runs

A run is one execution of one test, with its own status, evidence, score and token count. Runs started together (a module, Run all, the CI webhook, Rerun failed) share a batch, which is what the CI gate checks. See Runs and results.

Who can do what

Every project has one owner, the person who created it, plus any members they invite.

Role	Can
Viewer	See the project, its tests, runs and evidence. Secrets show names only.
Editor	Everything a viewer can, plus create, edit, delete and run tests and modules; manage environments, secrets, saved logins and datasets; override verdicts.
Admin	Everything an editor can, plus change project settings, integrations and notifications, see and rotate the CI webhook secret, and manage members.
Owner	Everything, including deleting the project.

Note

Runs use the owner’s default AI provider unless the project picks one in Settings → General. An editor who clicks Run doesn’t spend their own key. See AI providers.

Invite people from Settings → Members or the Team page. See Team and roles.

Next steps

Test kindsThe nine kinds and when to use each.

Runs and resultsThe life of a run and what it records.

EnvironmentsOne suite, many base URLs.

GlossaryEvery term in one place.