Skip to content
Plainwright

Monitoring

A production install has three signals, each off or minimal by default and each turned on with one setting: structured logs on stdout, error reports to Sentry, and Prometheus metrics from the backend. All three work the same on Docker Compose and hosted platforms.

Signal Setting Where it comes from
Logs LOG_FORMAT=json Every service’s stdout
Errors SENTRY_DSN Backend, worker and scheduler
Metrics METRICS_ENABLED (on by default), METRICS_TOKEN GET /metrics on the backend
Liveness — GET /health on the backend

Logs

Section titled “Logs”

With the default LOG_FORMAT=text, logs are human-readable lines. Set LOG_FORMAT=json for a log shipper (Loki, Datadog, CloudWatch, Better Stack and the like): every line becomes one JSON object, uvicorn’s and RQ’s own logs included.

{"ts": "2026-10-21T09:15:02.113Z", "level": "info", "logger": "uvicorn.access", "service": "backend", "msg": "172.18.0.9:41234 - \"GET /api/projects HTTP/1.0\" 200"}
Field Meaning
ts UTC timestamp, millisecond precision
level debug, info, warning, error, critical
logger Python logger name, e.g. uvicorn.access, rq.worker, plainwright.storage
service backend, worker or scheduler
msg The message
exc, stack Traceback, when there is one

Any extra fields a log call adds appear as top-level keys. Chatty HTTP and storage client libraries are kept at warning.

Alert on any critical line. In development mode the backend logs one at every start while SECRET_KEY is missing, a known default or too short, starting with SECURITY:; see Configuration warnings.

Terminal window
docker compose logs -f --since 10m backend worker

Errors with Sentry

Section titled “Errors with Sentry”

Set SENTRY_DSN on the backend, workers and scheduler. Each process reports with its service name as the server_name and a service tag, so you can filter by service.

Setting Default Meaning
SENTRY_DSN (empty) Your project’s DSN. Empty turns Sentry off.
SENTRY_ENVIRONMENT production The environment tag.
SENTRY_TRACES_SAMPLE_RATE 0 Share of requests sent as performance traces (0 to 1). 0 sends errors only.

What gets sent: unhandled exceptions in API requests, and every log record at error level or above, with the preceding info records as breadcrumbs. What doesn’t: no personal data, no request bodies, no cookies, and the Authorization, Cookie, X-Webhook-Secret, Stripe-Signature, X-API-Key and Proxy-Authorization headers are replaced with [Filtered].

Errors inside test runs reach Sentry through the worker’s logging, because the runner logs every exception it catches.

Metrics with Prometheus

Section titled “Metrics with Prometheus”

The backend serves Prometheus metrics at GET /metrics. It’s on by default, and the gauges are computed from the database and Redis at scrape time, so they’re correct with any number of backend and worker replicas. Scrape one backend; every backend reports the same numbers.

The bundled nginx answers /metrics with 404, so it’s never reachable through the public URL. Scrape the backend directly (backend:8000/metrics in Compose, or 127.0.0.1:8000/metrics on the host). Who the backend answers depends on two settings:

METRICS_TOKEN set METRICS_TOKEN empty
Development mode Only requests with Authorization: Bearer <token> Only clients on loopback or a private network, such as the compose network and the host
PLAINWRIGHT_ENV=production Only requests with Authorization: Bearer <token> Nobody: /metrics answers 404

METRICS_ENABLED=false turns the endpoint off everywhere.

prometheus.yml
scrape_configs:
  - job_name: plainwright
    metrics_path: /metrics
    scrape_interval: 30s
    authorization:
      credentials: "<METRICS_TOKEN>"     # required in production; omit in development without a token
    static_configs:
      - targets: ["backend:8000"]
Terminal window
# Quick check from the Docker host:
curl -s http://127.0.0.1:8000/metrics | grep '^plainwright_'

Metric reference

Section titled “Metric reference”
Metric Type Labels Meaning
plainwright_http_requests_total counter method, route, status API requests. route is the route template, e.g. /api/runs/{run_id}, so cardinality stays low.
plainwright_http_request_duration_seconds histogram method, route API latency.
plainwright_http_requests_in_flight gauge — Requests being handled now.
plainwright_runs_last_24h gauge status, kind Runs created in the last 24 hours.
plainwright_runs_in_progress gauge status Runs pending or running right now.
plainwright_run_duration_seconds_last_24h gauge kind, stat avg, p50, p95 and max duration of runs in the last 24 hours.
plainwright_flaky_runs_last_24h gauge — Runs in the last 24 hours that failed, then passed on retry.
plainwright_tokens_last_24h gauge direction AI tokens used by runs in the last 24 hours, input and output.
plainwright_cost_usd_last_24h gauge — Estimated AI spend in the last 24 hours, where the cost is known.
plainwright_queue_up gauge — 1 when Redis answered the scrape.
plainwright_queue_depth gauge queue Jobs waiting in the test_runs queue.
plainwright_queue_workers gauge queue Workers listening on the queue.
plainwright_users, plainwright_projects gauge — Totals.
plainwright_test_cases gauge kind Tests by kind.
plainwright_build_info gauge service, storage Always 1; storage is local or s3.

The standard Python process, platform and garbage-collector metrics are included as well.

Alert rules

Section titled “Alert rules”
plainwright-alerts.yml
groups:
  - name: plainwright
    rules:
      - alert: PlainwrightQueueDown
        expr: plainwright_queue_up == 0
        for: 2m
        annotations:
          summary: The backend can't reach Redis; no runs can be queued.
      - alert: PlainwrightNoWorkers
        expr: plainwright_queue_workers{queue="test_runs"} == 0
        for: 5m
        annotations:
          summary: No worker is listening; queued runs won't start.
      - alert: PlainwrightQueueBacklog
        expr: plainwright_queue_depth{queue="test_runs"} > 20
        for: 10m
        annotations:
          summary: Runs are waiting; add workers.
      - alert: PlainwrightRunErrors
        expr: sum(plainwright_runs_last_24h{status="error"}) > 10
        annotations:
          summary: Many runs ended in error (browser service, provider keys or quota), not in a test failure.

A run with status error means the test couldn’t run at all (a browser service down, a rejected API key, a quota). A failed run is a real test failure, so alert on errors and let failures flow to your alerts and CI gate.

Configuration warnings

Section titled “Configuration warnings”

The backend checks its own configuration at start. Today that’s SECRET_KEY: empty, a publicly known default, or shorter than 32 characters. With PLAINWRIGHT_ENV=production such a key stops the backend from starting. In development mode the backend starts, and the problem is reported in three places:

  • a critical log line starting SECURITY:, at every start;

  • a bar under the app’s header for project owners and admins. Dismissing it hides it for that browser session only, so it comes back until the problem is fixed;

  • GET /api/system/warnings, for anyone signed in. Owners and admins get the list, everyone else an empty one:

    {
      "warnings": [
        {
          "code": "default_secret_key",
          "severity": "critical",
          "message": "SECRET_KEY is a publicly known default value. Anyone who knows it can forge sign-ins …",
          "docs": "docs/security.md#rotate-the-secret-key"
        }
      ]
    }

The codes are default_secret_key (missing or publicly known) and weak_secret_key (too short). Fix either with Rotate SECRET_KEY.

Health checks

Section titled “Health checks”

GET /health on the backend answers {"status": "ok"} without touching the database. Docker Compose uses it for the backend’s health check, and Render uses it as the web service’s health check path. The browser services have their own Compose health checks, which pass once the MCP server answers on port 8931.

The public nginx sends every path that isn’t /api/ or /screenshots/ to the frontend, so https://<your-domain>/health returns the web app, not the backend. For an external uptime check through the public URL, request a public API route instead, such as GET https://<your-domain>/api/oauth/providers, which returns a small JSON object.

Next steps

Section titled “Next steps”