Security model

Plainwright runs an AI agent that signs in to your apps, sends requests to your APIs and talks to your AI products, so it holds real credentials and reaches into your network. This page explains what protects those credentials, what each part of the system is trusted with, and what you need to configure before other people can reach your install. It describes the code as it is; the hardening checklist at the end is the short version.

Trust boundaries

Party	Trusted with	Not trusted with
People using the app	What their project role allows (see Roles).	Other projects; secret values, which are write-only.
The AI model provider	Test descriptions, page snapshots, screenshots, API responses and conversation transcripts sent during runs.	Vault secret values: the model only ever sees {{secrets.NAME}} placeholders.
Systems under test	Nothing. Pages, API responses and chatbot replies are untrusted input and may contain prompt injection.	—
The operator	Everything: the server, .env, the database.	—

If test data must not leave your network, use a local model through Ollama.

SECRET_KEY and production mode

SECRET_KEY is the install’s master key. It signs every sign-in token, and the key that encrypts stored credentials is derived from it. Anyone who has it can sign in as any user and, with a database copy, decrypt every stored provider key and secret.

The backend checks the key at start, before it touches the database. A key that is empty, a publicly known default (such as supersecretkey123, the Compose fallback) or shorter than 32 characters is treated according to PLAINWRIGHT_ENV:

• production (or prod): the backend refuses to start, with “Refusing to start with PLAINWRIGHT_ENV=production: SECRET_KEY is …”.
• development (the default): the backend starts, logs a CRITICAL line starting SECURITY: at every start, and reports the problem to project owners and admins through GET /api/system/warnings, which the app shows as a banner. Everyone else gets an empty list, so ordinary members and new sign-ups never learn that the install is misconfigured.

To replace a weak key on an install that already holds data, use Rotate SECRET_KEY: python -m app.cli rotate-secret-key re-encrypts every stored value under the new key in one transaction, so nothing has to be re-entered. Changing the variable by hand makes stored values unreadable instead.

Keep SECRET_KEY out of reach

Keep it out of the repository and out of logs, use the same value on the backend, workers and scheduler, and keep a copy with your backups: a database dump can’t be decrypted without the key that was current when it was taken.

Accounts and sign-in

• Passwords are hashed with bcrypt. At registration, on change and on reset they must be at least 8 characters, at most 72 bytes (bcrypt reads no more), and not the email address or the part before the @.
• Email addresses are stored in lower case and compared case-insensitively everywhere.
• Sessions are JWTs signed with SECRET_KEY (HS256), valid for ACCESS_TOKEN_EXPIRE_MINUTES (7 days by default). Only tokens of type access sign in; OAuth state, login codes and confirmation links carry their own types and can’t be used as sessions. The browser keeps the token in localStorage and sends it as Authorization: Bearer …; no cookies are used for the API, so CORS never allows credentials.
• Revocation. Every token carries the account’s token version. Changing or resetting the password, POST /api/auth/logout-all, and being removed from the last project you belonged to all bump it, which revokes every token issued before. Rotating SECRET_KEY signs everyone out.
• Throttling. Sign-in, registration, password-reset requests and confirmation emails are rate-limited; see Rate limits below.
• Email confirmation. Anyone can register any address, so an address only counts once its owner proves they receive mail there: through the link emailed at registration (valid 72 hours), a Google or GitHub sign-in that verified it, or a completed password reset. Pending project invitations only attach to confirmed addresses, and an invitation to an unconfirmed account stays pending until it’s confirmed. See Confirming email addresses.
• Google and GitHub sign-in use the authorization-code flow. A signed, 10-minute state is tied to an HttpOnly, SameSite=Lax nonce cookie. The callback hands the browser a one-time login code, valid for 2 minutes, never a session token, and POST /api/oauth/exchange swaps it for a session only together with the same browser’s cookie. That blocks login CSRF, including a code or token planted in someone else’s browser. Only provider-verified emails sign in, and they link to an existing account only when that account’s address is confirmed. Redirects after sign-in accept same-site relative paths only. See Sign-in and email.
• Password reset tokens are 32 random bytes, stored only as a SHA-256 hash, single-use and short-lived, and never logged. The request form answers identically whether or not the account exists. Completing a reset revokes the account’s other sessions and confirms its address.
• Registration is open. Anyone who can reach the sign-in page can create an account. A new account sees only its own projects and the projects it’s invited to, but it can run tests and spend the server-wide ANTHROPIC_API_KEY if you set one. To keep an install private, restrict who can reach it: a VPN, an IP allowlist at your proxy, or an identity-aware proxy. HTTP basic auth at the proxy doesn’t combine well, because the API already uses the Authorization header. If CI must reach the server, allow /api/runs/webhook/ and /api/batches/ from your CI.

Rate limits

Endpoint	Limit
Sign-in (failed attempts)	10 per account and client address, 30 per account, 100 per client address, each per 15 minutes
Registration	20 per client address per hour
Password-reset requests	1 a minute and 5 an hour per account; 20 an hour per client address
Confirmation emails (on request)	5 per user per hour

Counters live in Redis, so every backend replica shares them; while Redis is unreachable, each backend counts on its own. A blocked request gets HTTP 429, “Too many attempts — wait a few minutes and try again.”, with a Retry-After header. A successful sign-in clears that account’s counter for that client address. An unknown email costs the same bcrypt time as a wrong password, so timing doesn’t reveal which accounts exist.

Per-address limits behind a proxy

The limits per client address use the address the backend sees. uvicorn only takes the real client address from X-Forwarded-For when the request comes from an address listed in FORWARDED_ALLOW_IPS (default 127.0.0.1). Behind the bundled nginx, the backend therefore sees nginx’s own address for everyone, and the per-address limits act install-wide: 100 failed sign-ins from anyone pause password sign-in for everybody for 15 minutes. With FORWARDED_ALLOW_IPS=*, as in the Render blueprint, the left-most X-Forwarded-For entry counts, which clients can set themselves. Rate-limit by IP at the edge proxy as well, where the real address is known (example in the checklist).

Roles and project access

Every project has an owner and members with a role. Each API call checks the caller’s role on the project behind the object, and answers 404 to someone with no access at all, so project ids can’t be probed.

Role	Can
Viewer	See the project, its tests, runs, results and evidence. Sees secret names only.
Editor	Everything a viewer can, plus create and edit modules, tests, environments, datasets, secrets and saved logins; run tests; override verdicts; run AI root cause and run chat; import tests.
Admin	Everything an editor can, plus project settings (including retries, pass marks and the judge model), notifications and test messages, the CI webhook secret (view and rotate), members and roles, and integrations.
Owner	Everything, including deleting the project. Billing usage counts against the owner.

AI provider configurations belong to a user, not a project. See Team and roles.

What’s encrypted, hashed or plain

Encrypted values use Fernet (AES-128-CBC with an HMAC) with a key derived from SECRET_KEY (app/core/crypto.py), and are stored with an enc: prefix.

Data	Protection	Shown back to users
AI provider API keys	Encrypted	Masked: first 6 and last 4 characters
Vault secrets (passwords, API keys, text, TOTP seeds)	Encrypted	Never. Editors see •••• plus the last 4 characters of api_key and text values of 8+ characters.
Saved logins (captured browser sessions)	Encrypted	A summary only: cookie count, domains, expiry
Integration credentials (Jira, Linear, GitHub, Figma, Notion, IMAP)	Encrypted (the whole config)	Masked
User passwords	bcrypt hash	—
Password-reset tokens	SHA-256 hash	—
CI webhook secret	Plain text in the database	Admins and the owner only
Alert webhook URL (Slack, Discord, custom)	Plain text in the database	Full URL to admins; origin only to others
Test configurations, run results, transcripts	Plain text; vault values scrubbed	To project members
.env (SMTP password, S3 keys, Stripe and OAuth secrets, SECRET_KEY)	Plain text on the server’s disk	—

Provider keys that older releases stored in plain text are still read. The rotation command encrypts them, as does the upgrade’s database migration. The database itself isn’t encrypted by Plainwright, so use encrypted disks or your provider’s encryption at rest, and protect backups like the live database.

Secrets during a test run

Tests refer to credentials as {{secrets.NAME}} placeholders, never as values (see Secrets and saved logins).

1. The AI agent plans and types with the placeholder. The model never receives the value.
2. Just before a tool runs (typing into a field, sending an HTTP request), the worker swaps in the real value. A totp secret resolves to the current one-time code.
3. Placeholders follow the destination. In a URL the agent opens (browser_navigate) or an HTTP request it sends, a placeholder is only filled in when the request goes to one of the project’s own hosts: the project URL, its environments, and endpoints configured in the test. For any other host the literal placeholder is sent, so a prompt-injected page can’t make the agent carry a secret to another site in one step.
4. Tools that execute code in the page (browser_evaluate, browser_run_code) get the literal placeholder, never the value. The UI-test agent isn’t offered them at all, and it refuses any tool name it didn’t declare to the model, even when a provider lets one through.
5. Everything that’s stored, shown or sent back to the model is scrubbed. The raw value and its common encodings (JSON and JavaScript string escapes, URL encodings, HTML entities), and a value cut off at a truncation point, are replaced with ••••. That covers steps, evidence, transcripts, the live view and failure explanations.

Values shorter than 4 characters aren’t scrubbed, because masking them would mangle unrelated text. Screenshots are images and are not redacted: a page that displays a secret shows it in the screenshot.

Outbound requests (SSRF guard)

Several features make the server send requests to URLs that users type in: API tests, eval and conversation targets over HTTP, MCP servers and A2A agents under test, alert webhooks, the broken-link checker, the IMAP inbox, the Jira/GitHub integrations, and the URLs the AI agent tells the browser to open. Every one of them passes check_url_allowed (app/services/targets.py), and requests the server sends itself re-check every redirect hop.

Refused by default:

• Schemes other than http and https.
• Cloud-metadata and link-local addresses (169.254.0.0/16 including 169.254.169.254, 100.100.100.200, fd00:ec2::254, fe80::/10) and metadata host names such as metadata.google.internal.
• Loopback (127.0.0.0/8, ::1, localhost, *.localhost), 0.0.0.0, multicast and reserved ranges.
• The platform’s own services: postgres, redis, backend, worker, scheduler, frontend, nginx and the playwright-mcp* services, by service name, by container name (such as plainwright-redis-1) and, with DNS checks on, by the addresses they resolve to.
• The Docker host (host.docker.internal or the default gateway) on the platform’s own ports: 3000, 5432, 5433, 6379, 8000, 8080 and 8931. Other ports on the host stay reachable, for apps you run locally.

Addresses are checked in every spelling: decimal, hex and octal IPv4, and IPv4-mapped and other embedded IPv6 forms. With SSRF_DNS_CHECK=true (the default), host names are resolved too, and refused when any address they resolve to is refused (lookups time out after 2 s and are cached for 60 s).

Not refused: private ranges such as 10.0.0.0/8 and 192.168.0.0/16, and other host names on your network. That’s deliberate, so you can test internal staging services, but it means a project editor can make the worker send HTTP requests to anything the worker can reach on your network. The DNS check also can’t stop DNS rebinding between the check and the request.

ALLOW_INTERNAL_TARGETS=true turns the guard off (apart from the scheme check). Use it only on a single-user install where you need to test a service on the compose network.

The browser and your network

The AI agent’s navigations go through the guard above, and it can only upload files from the current run’s project folder (/test-files/<project_id>/), never another project’s. What the page does next isn’t checked: a page the browser opens can redirect, load resources and run scripts that reach any address the playwright-mcp containers can reach.

The default docker-compose.yml limits that with a separate browser network. The three browser services are only on that network, together with the backend and the workers, so a page can’t reach Postgres, Redis, the scheduler, the frontend or nginx by name. It can still reach the internet, your private network, the Docker host’s interfaces, and the backend and workers on the browser network (the API still needs a sign-in token, and /metrics doesn’t answer the browser services). Each run gets a fresh, isolated browser profile (--isolated), so cookies and logins don’t carry over between runs. If editors shouldn’t be able to reach other internal systems through the browser, give the browser services a network that only reaches what you want tested.

Artifacts and uploaded files

• Screenshots and evidence are served at /screenshots/<key> and /api/files/<key> without sign-in, because <img> tags can’t send a bearer token. Keys are random 128-bit names, so the URLs act as unguessable capability links; anyone who has one can open it.
• /api/files serves files with X-Content-Type-Options: nosniff and a sandboxing Content-Security-Policy, and anything that isn’t an image, video, PDF or plain data is sent as a download, so a stored HTML file can never run on the app’s origin. Keys are validated against path traversal.
• With S3 storage and no public base URL, links redirect to presigned URLs that expire after an hour. Keep the bucket private.
• Project test files are limited to 25 MB each, with sanitized file names. They live in one folder per project, and browser uploads are confined to the run’s own project folder.

Network exposure

Only the reverse proxy should be reachable from outside. The rest have no authentication of their own or expose internals:

Port	Service	Why it must stay private
8931	Playwright MCP (Chromium)	No authentication, and host checks are off (--allowed-hosts '*'). Anyone who reaches it can drive a browser inside your network and read every project’s test files.
6379	Redis	No password. Anyone who reaches it can read and inject queued jobs.
5432 / 5433	Postgres	All data.
8000	Backend	Bypasses the proxy’s headers and limits.
3000	Frontend	Harmless static files, but it bypasses the proxy’s security headers.

The default docker-compose.yml publishes the browser service, Redis, Postgres, the backend and the frontend on 127.0.0.1 only, and doesn’t publish the Firefox and WebKit browser services at all. Only nginx (8080) listens on every interface; Deploy to a VPS binds it to loopback behind a TLS proxy. On a platform with private networking, don’t give these services public URLs.

Headers and transport

The bundled nginx removes its version banner and adds, on every response: X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy: strict-origin-when-cross-origin, a Permissions-Policy that disables camera, microphone, geolocation and payment, and a Content-Security-Policy limited to frame-ancestors 'self'; base-uri 'self'; object-src 'none'. There’s no script or style policy. TLS terminates at your own proxy in front of nginx; enable HSTS there once HTTPS works (see Deploy to a VPS).

Webhooks and machine access

• CI webhook (POST /api/runs/webhook/{project_id}) and batch status (GET /api/batches/{id}) accept the project’s secret in X-Webhook-Secret, compared in constant time. Admins can rotate it in the project settings; the old secret stops working immediately.
• Stripe webhooks are accepted only with a valid Stripe-Signature no older than 5 minutes.
• Prometheus metrics (/metrics) need Authorization: Bearer <METRICS_TOKEN> when a token is set. Without one, production mode answers 404 and development mode answers only loopback and private-network clients. nginx never proxies /metrics. See Monitoring.

Logs and error reports

• Provider keys are masked in API responses and never logged in full. Password-reset links are never logged.
• Sentry, when enabled, sends no personal data, request bodies or cookies, and replaces the Authorization, Cookie, X-Webhook-Secret, Stripe-Signature, X-API-Key and Proxy-Authorization headers with [Filtered].
• In development mode, the backend logs a CRITICAL warning at start while SECRET_KEY is missing, a known default or too short.

Known limitations

• Open registration with no setting to turn it off.
• Per-address sign-in limits see the proxy’s address unless the backend trusts its forwarded headers (see Rate limits).
• The SSRF guard allows private addresses and can’t stop DNS rebinding. Inside the browser, only the agent’s own navigations are checked, not what the page loads; the browser network keeps pages away from the database, Redis and the platform UI, but not from the backend and workers.
• Artifact URLs are capability links without sign-in.
• The CI webhook secret and alert webhook URLs are stored unencrypted.

Hardening checklist

• SECRET_KEY is a random value of 32+ characters, identical on backend, workers and scheduler, stored outside the repository and backed up.
• PLAINWRIGHT_ENV=production, so a missing or weak key stops the backend instead of starting it.
• No warning banner for owners and admins (GET /api/system/warnings returns an empty list).
• Only the TLS proxy is public. Ports 8931, 6379, 5432/5433, 8000 and 3000 aren’t reachable from outside (remember that Docker-published ports bypass host firewalls like ufw).
• HTTPS in front of nginx, with HSTS once it works.
• APP_BASE_URL and CORS_ORIGINS are the real public origin.
• Access is restricted to the people who should have it (VPN, allowlist or identity-aware proxy), or you accept open registration.
• SMTP or a Google/GitHub sign-in is configured, so people can confirm their addresses and reset passwords.
• Sign-in is rate-limited by IP at the edge proxy (example below).
• ALLOW_INTERNAL_TARGETS=false (the default) and SSRF_DNS_CHECK=true, unless this is a trusted single-user install.
• The browser services’ network reaches only what should be tested.
• METRICS_TOKEN is set if anything scrapes /metrics from outside the host, or metrics are off.
• The artifact bucket is private and has a lifecycle rule.
• Postgres backups run regularly and are protected like the live database, together with the SECRET_KEY that was current when each was taken.
• Leave ANTHROPIC_API_KEY empty if you don’t want every account to spend it.

A per-IP rate limit for the sign-in, registration and password-reset endpoints, with nginx as the TLS proxy on the host. Unlike the backend, the edge proxy sees each client’s real address:

# In the http block, e.g. /etc/nginx/conf.d/plainwright-ratelimit.conf
limit_req_zone $binary_remote_addr zone=pw_auth:10m rate=10r/m;

# In the Plainwright server block, next to "location /"
location ~ ^/api/auth/(login|register|password/forgot)$ {
    limit_req zone=pw_auth burst=5 nodelay;
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Caddy needs a rate-limit plugin for the same effect.

Reporting a vulnerability

Please report security problems privately, not in a public issue: use GitHub’s private vulnerability reporting for the repository. Include what you found, how to reproduce it, and the version or commit. You’ll get an acknowledgement, and a fix and advisory will be coordinated with you before details are published.

Next steps

• Configuration: every setting mentioned on this page.
• Upgrades, migrations and backups: rotate SECRET_KEY.
• Deploy to a VPS: a production setup with private ports and HTTPS.
• Secrets and saved logins: using the vault in tests.