Architecture

Plainwright is a small set of services: a web API, a job queue, workers that run tests, a scheduler, headless browsers behind Playwright MCP, and Postgres. This page explains what each one does, how a test run moves through them, and how the code inside a worker is layered. Read it before you operate a deployment or change the engine.

The Docker Compose services and the path of one run: the API queues it, a worker runs the engine against the real target, and results land in Postgres while live events stream back.

Services

Docker Compose (docker-compose.yml) starts these services:

Service	Built from	Role	Scale
nginx	nginx:alpine	The public entry point on port 8080. Serves the frontend at /, proxies /api/ and /screenshots/ to the backend, streams server-sent events unbuffered, and hides /metrics.	1
frontend	frontend/	The React app, built with Vite and served as static files. It calls the API at the relative path /api.	1
backend	backend/Dockerfile	FastAPI: the REST API, authentication and roles, the live-event stream, the CI webhook. Runs database migrations when it starts.	1 or more
worker	backend/Dockerfile.worker	RQ workers on the test_runs queue. Each one runs one test at a time.	WORKER_REPLICAS, default 2
scheduler	backend/Dockerfile	Every 60 seconds, queues tests whose cron schedule is due, and marks runs whose worker died as errored.	Exactly 1
postgres	postgres:15-alpine	All data: users, projects, tests, runs, results, secrets (encrypted).	1
redis	redis:7-alpine	The job queue, and pub/sub for live run events.	1
playwright-mcp	mcr.microsoft.com/playwright	Headless Chromium behind the Playwright MCP server.	1
playwright-mcp-firefox, playwright-mcp-webkit	same image	The same server with Firefox and WebKit.	1 each

Two volumes hold state: postgres_data, and screenshots (shared by backend, workers and nginx), unless you move artifacts to S3 or R2 (see Object storage). The host folder ./test-files holds uploaded project files; the backend writes it and the browsers read it.

Only nginx should be public

The browsers, Postgres, Redis, the frontend and the backend’s own port are meant for the internal network; Docker Compose publishes them on 127.0.0.1 only, or not at all. The MCP browser servers in particular will open any URL they are told to. Publish only nginx (behind HTTPS); see Network exposure.

How a run flows

The numbers match the diagram above.

1. Trigger. A person clicks Run, a pipeline calls the CI webhook, the scheduler finds a due cron, or a failed run is retried. Every path goes through one function, queueing.enqueue_run, which creates a TestRun with status pending, copies the test’s kind, attaches the batch ID and the chosen environment, browser and device.

2. Enqueue. The run is committed to Postgres first, then a run_test_job job is pushed onto the Redis queue test_runs with the job ID run-<run_id> and a timeout of 300 seconds per trial. If Redis is down, the API answers 503.

3. Pick up. A free worker takes the job and loads the run, test, project and owner. It resolves the run model (the project’s provider, else the owner’s default, else the server’s ANTHROPIC_API_KEY with claude-haiku-5-5) and the judge model (the project’s judge, else the cheap model of the run model’s provider), builds the secret resolver, starts the usage meter that prices every model call, and checks the billing quota (when billing is on) before spending anything.

4. Execute. For each trial (1–10), the worker runs the engine for the test’s kind. Engines call LLM providers and act on the target: through Playwright MCP for browser tests, or directly over HTTP, MCP or A2A. For Web UI tests, plain-English assertions are then judged, the visual diff is computed, and every failed run is checked for a CAPTCHA wall.

5. Results. Trials are aggregated into one result (a pass rate or a mean score against the threshold), everything is scrubbed of secret values and written to Postgres, with screenshots in storage. A failed or errored run gets a failure signature for Triage; a failed run also gets an AI explanation, and an errored one a plain-language error message. Then the retry rule may queue another attempt, and alerts go out.

6. Live events. Throughout, the worker publishes events (steps, screenshot frames, conversation turns, eval rows, trial progress, done) to the Redis channel runevents:<run_id>. The backend relays them to the browser over GET /api/runs/{run_id}/stream. Events are not stored; the finished run in Postgres is the record.

Inside a worker

Inside a worker: the runner picks the engine for the test kind; engines combine shared building blocks; toolsets and targets are the only layer that talks to the outside world.

The runner (runner_service.run_test_job) is the same for every kind: models, secrets, quota, trials, aggregation, persistence, retries and alerts. It looks the engine up in a registry by the test’s kind.

Engines (services/engines/) each turn one kind of test into an EngineResult: a status, steps, and optionally a transcript, scores, judge verdict, screenshots, evidence, eval rows or a quality report. Engines never write to the database themselves.

Kind	Engine does	Built on
ui	An AI agent reads the page and clicks and types until the test is done	Agent loop + BrowserToolset (+ TOTP and email-inbox tools)
api	An agent sends real HTTP requests and checks the responses	Agent loop + HttpToolset (http_request)
conversation	A simulated user with a persona and goal talks to the target; one judge call scores it	Simulated user + target (HTTP, A2A, or a chat widget driven through the browser) + judge
redteam	An attacker conversation per attack pack; each is judged resisted or breached	Simulated user + target + judge
eval, rag	Each dataset row is sent to your endpoint and scored, four rows at a time	HttpTarget or A2ATarget + judge metrics (one call per row)
mcp	A tool-contract check, then an agent that exercises the server’s tools	Streamable HTTP or SSE client + agent loop + RemoteMCPToolset
a2a	Agent-card validation, then a conversation (or one smoke message)	A2A client + the conversation engine
quality	Accessibility, performance, broken-link and SEO checks	Scripted Playwright MCP session, no LLM

The agent loop (ai_service.run_agent) is one loop with three provider adapters: Anthropic’s tool-use API, the OpenAI-compatible API (OpenAI, OpenRouter, Groq, Ollama) and Gemini. It exposes a toolset’s tools as function tools, runs each tool call, and feeds the result back until the model returns its final JSON verdict. The loop also:

• keeps only the most recent large tool outputs in full and stubs older ones, so token use stays flat over long tests;
• strips <think> blocks from reasoning models before parsing;
• recovers from the two commonest browser errors: a stale element reference triggers a fresh page snapshot, and an ambiguous element gets a hint to pick a unique target;
• stops after 30 tool calls by default (an error), and detects an agent that is stuck;
• swaps {{secrets.NAME}} placeholders for real values only inside tool arguments, at the moment a tool runs, and scrubs values out of every result.

Toolsets are the only code that touches the outside world. BrowserToolset talks to Playwright MCP over a long-lived SSE session (one browser session per run, with a fresh profile), exposes its browser_* tools, captures evidence, and always takes a final screenshot. HttpToolset and the targets make guarded HTTP calls. RemoteMCPToolset wraps a server under test and prefixes its tools with target__.

Why the MCP bridge runs in the worker

Every provider, including Anthropic, drives the browser through Plainwright’s own MCP client rather than a provider-side MCP connector, because a provider’s cloud can’t reach the internal playwright-mcp host. The client uses MCP’s SSE transport because a Streamable HTTP session expires during the model’s thinking time between tool calls.

Data model

Everything hangs off a project:

User ─┬─ AIConfig (provider keys, encrypted)
      └─ Project ─┬─ Module (board column) ── TestCase ── TestRun ─┬─ EvalResult (one per dataset row)
                  │                                                 ├─ RunChatMessage
                  │                                                 └─ retry_of → TestRun
                  ├─ ProjectMember (viewer · editor · admin)
                  ├─ Environment · Secret · AuthState (saved login) · Dataset
                  ├─ Integration (Jira, Linear, GitHub, Figma, Notion, IMAP; encrypted)
                  └─ Issue (filed per failure signature)

• A test has no project ID of its own: the chain is TestRun → TestCase → Module → Project.
• Kind-specific settings live in JSON columns (TestCase.config, TestRun.extras, Project.settings), so new features rarely need a schema change.
• A run’s effective status is its human override’s status when there is one, else its status. Gates, Triage and Analytics all use it.
• IDs are UUIDs. Secrets, provider keys, saved logins and integration configs are encrypted with a key derived from SECRET_KEY, which must be the same on the backend, workers and scheduler.

The schema is managed with Alembic (backend/migrations/). The backend upgrades the database when it starts, under a Postgres advisory lock so several backends can start at once; workers and the scheduler never migrate. See Upgrades, migrations and backups.

Outbound safety

Tests point Plainwright at URLs you supply, so every server-side request to a user-supplied URL (API tests, conversation and eval targets, MCP servers, A2A agents, link checks, alert webhooks, tracker and IMAP hosts, Ollama and custom model endpoints) goes through one guard. It refuses non-HTTP schemes, cloud-metadata and loopback addresses, the stack’s own services by name, container name and address, and the Docker host on the platform’s own ports, including host names that resolve to any of these, unless the server sets ALLOW_INTERNAL_TARGETS=true.

The browser agent’s navigations pass the same guard before they reach Playwright MCP, uploads are confined to the run’s project folder, and the model can only call the tools it was given. What a page loads by itself isn’t filtered, so the browser services sit on their own browser network with only the backend and workers. Details are in Outbound requests.

Observability

The backend serves Prometheus metrics at /metrics on its own port (nginx returns 404 for it), optionally protected by METRICS_TOKEN. Logs can be JSON (LOG_FORMAT=json), and errors go to Sentry when SENTRY_DSN is set. See Monitoring.

Where the code lives

Path	Contents
backend/app/main.py	The FastAPI app and router mounts
backend/app/api/	Thin routers, one file per area
backend/app/core/	Settings, models (database.py), auth and roles, encryption, migrations, observability
backend/app/services/runner_service.py, queueing.py	The job and the single enqueue path
backend/app/services/engines/	One engine module per kind family, registered by kind
backend/app/services/ai_service.py	Agent loop, toolsets, the MCP client
backend/app/services/llm.py, judge.py, targets.py, secrets.py	Provider calls, judging, HTTP targets, the secret resolver
backend/app/services/scheduler.py	The scheduler process
frontend/src/	Pages, feature components, the UI kit and API clients

The full layout and how to run each part locally are in Contributing.

Next steps

Scaling workersRun more tests at once.

ConfigurationEvery environment variable the services read.

Security modelRoles, secrets, encryption and outbound guards.

Test kindsWhat each engine tests, from the user's side.