FAQ

About the project

Is Plainwright free and open source?

Yes. Plainwright is released under the Apache License 2.0 (see LICENSE and License). You can self-host it, change it and use it commercially. Billing support exists for people who run it as a paid service, and it’s off by default.

Why do the database and some internal names say “testai”?

TestAI was the project’s earlier name. The database, its user and a few internal names stay testai so existing installs keep their data when they upgrade. Everything people see says Plainwright.

Running it

What do I need to run it?

Docker with Compose v2.24 or newer, and about 8 GB of RAM and 4 CPU cores for a few parallel browser tests. Then docker compose up -d --build and open http://localhost:8080. See Docker Compose. For hosted platforms, see Render, Railway and Fly.io.

Why is the app on port 8080 and not 80?

Port 80 is often taken, on Windows commonly by IIS. The app uses relative URLs, so you can publish nginx on any port. On a server, put a TLS proxy on 443 in front of it; see Deploy to a VPS.

What does it cost to run?

Two things: the machines, and your AI provider’s tokens. Plainwright adds no fees of its own. Site quality checks (accessibility, performance, links, SEO) run without an AI model, and a local Ollama model costs nothing per token. Server-wide caps (MAX_TRIALS, MAX_EVAL_ROWS_PER_RUN, MAX_CONVERSATION_TURNS, JUDGE_MAX_TOKENS) bound what one run can spend; see Trials, thresholds and cost. Analytics shows token use by model and by run.

How many tests can run at the same time?

As many as you have workers: each worker runs one test at a time. Docker Compose starts two; set WORKER_REPLICAS for more, and give the browser services memory to match. See Scaling workers.

Do I need the Firefox and WebKit services?

Only if tests choose those browsers. The default compose file starts all three browser services, and the worker waits for them to start. To run Chromium only, remove playwright-mcp-firefox and playwright-mcp-webkit from your compose setup together with the worker’s depends_on entries for them; runs that ask for Firefox or WebKit then fail with a browser-service error.

Can I use my own Postgres or Redis?

Yes, Postgres 15 or newer and any Redis that doesn’t evict keys. In Docker Compose, DATABASE_URL and REDIS_URL are set by the compose file and win over .env, so override them for the backend, worker and scheduler in a docker-compose.override.yml. The worker image’s default command also names redis://redis:6379, so override its command too:

docker-compose.override.yml

x-external: &external
  DATABASE_URL: postgresql://plainwright:secret@db.internal:5432/plainwright
  REDIS_URL: redis://cache.internal:6379
services:
  backend:
    environment: *external
  scheduler:
    environment: *external
  worker:
    environment: *external
    command: ["rq", "worker", "--url", "redis://cache.internal:6379", "test_runs"]

Use the postgresql:// scheme, not postgres://. You can then remove the bundled postgres and redis services and the depends_on entries that point at them.

Is there a Kubernetes or Helm setup?

Not in the repository. The images are ordinary Docker images, so the topology from Railway or Fly.io carries over: separate deployments for the backend, workers and one scheduler; a private browser service; S3 storage; and a proxy that serves the frontend and /api on one origin. Contributions are welcome.

How do I upgrade?

Back up, git pull, then docker compose up -d --build. The backend migrates the database when it starts. See Upgrades, migrations and backups.

How do I change SECRET_KEY?

With the rotation command, never by editing the value: SECRET_KEY also encrypts every stored provider key and secret, so a new value on its own makes them unreadable. python -m app.cli rotate-secret-key --new-key-env NEW_SECRET_KEY re-encrypts them under the new key in one transaction; then you put the new key in .env and restart the backend, workers and scheduler together. Everyone has to sign in again; nothing has to be re-entered. The full procedure is in Rotate SECRET_KEY.

What does PLAINWRIGHT_ENV do?

It defaults to development, where the backend starts even with a missing, publicly known or short SECRET_KEY but logs a CRITICAL warning and shows owners and admins a banner. Set PLAINWRIGHT_ENV=production on any install other people use: the backend then refuses to start with such a key, and /metrics answers only requests with METRICS_TOKEN. See Production mode.

AI models and data

Which AI models can I use?

Anthropic (Claude), OpenAI, Google Gemini, OpenRouter, Groq and Ollama for local models. The model must support tool calling, because the agent does everything through tools; Test connection in AI providers checks this before you rely on a model. See AI providers.

Which model does a run use?

The first one that’s set, in this order:

1. The model chosen in the project’s settings.
2. The default AI provider of the project’s owner, also for runs a teammate starts.
3. The server-wide fallback: Anthropic claude-haiku-5-5 with ANTHROPIC_API_KEY, if it’s set.

Scored tests use the project’s judge model when one is chosen. Otherwise they use the cheap model of the same provider (for example claude-haiku-5-5 or gpt-6-luna) with the same key; see AI providers.

Does my test data leave my network?

Yes, to the AI provider you choose: test descriptions, page snapshots, screenshots (for models that accept images), API responses and conversation transcripts are sent to the model so it can run and judge the test. Vault secret values never are; the model only sees {{secrets.NAME}} placeholders. To keep everything local, use an Ollama model.

Plainwright has no usage telemetry. Apart from the AI provider and the systems you test, it only talks to services you configure (SMTP, S3, Sentry, Stripe, Google or GitHub sign-in, Jira, Linear, GitHub, Figma, Notion). The browser services download @playwright/mcp and their browser from the internet when they start, and the web app loads its fonts from Google Fonts.

Can it run fully offline?

Not out of the box: the browser services install from npm and download their browser at every start, the web app loads fonts from Google Fonts, and a hosted AI provider needs the internet. With a local Ollama model and browser images prepared in advance, the rest of the stack doesn’t need outside access.

Where are screenshots stored, and who can see them?

On the server’s screenshots volume, or in your S3-compatible bucket with object storage. Artifact URLs don’t require sign-in (images are loaded by <img> tags), but they use random 128-bit names, so only people who have a link can open it. Apart from replaced visual baselines, Plainwright doesn’t delete old artifacts; use a bucket lifecycle rule (see Object storage).

Accounts and teams

Can I stop strangers from signing up?

There’s no setting for it: anyone who can reach the sign-in page can register, and new accounts only see projects they own or are invited to. Registration is throttled, but not closed. Keep the install reachable only by the people who should use it, with a VPN, an IP allowlist or an identity-aware proxy. See Security model.

Can several people share one install?

Yes. Invite people to a project as viewers, editors or admins in Project → Settings → Members. Someone whose account address is confirmed is added straight away. Otherwise the invitation waits until the person confirms the address: with the emailed confirmation link, by signing in with Google or GitHub, or by completing a password reset. See Confirming email addresses and Team and roles.

An invited person doesn’t get access

Their address isn’t confirmed yet, and pending invitations only attach to confirmed addresses. Ask them to click the confirmation link in their email, or to sign in with Google or GitHub if it’s enabled.

Without SMTP or a sign-in provider there’s no way for them to confirm it themselves. If you know the person really owns the address, an operator can confirm it from the backend container, and their pending invitations attach straight away:

docker compose exec backend python -m app.cli verify-email person@example.com
# Email marked as verified.

The address is matched case-insensitively. The command prints “No account with that email.” if the person hasn’t registered yet; ask them to sign up first.

How do I reset someone’s password when email isn’t set up?

Password-reset links need SMTP, and they’re never shown or logged. Without SMTP, the person can sign in with Google or GitHub if you’ve enabled them (that links to an account whose address is confirmed), or an operator can set a new password from the backend container. This also signs out the person’s other sessions:

docker compose exec backend python -c "
from sqlalchemy import func
from app.core.database import SessionLocal, User
from app.core.auth import hash_password, bump_token_version
db = SessionLocal()
user = db.query(User).filter(func.lower(User.email) == 'person@example.com').first()
user.hashed_password = hash_password('a-new-temporary-password')
bump_token_version(user)
db.commit()
print('updated')
"

Caution

The new password ends up in your shell history. Use a temporary one of at least 8 characters, and ask the person to change it in Settings → Security straight away.

Is billing required?

No. Billing is off by default and self-hosted installs have no plan limits. It exists for people who offer Plainwright as a paid service; see Billing.

Next steps

• Troubleshooting: fixes for specific errors.
• Configuration: every setting.
• Security model: what’s protected and how.