How Plainwright compares

This page puts Plainwright next to 17 other products that test web apps or AI products with AI: open-source test tools, browser-agent frameworks, LLM evaluation tools and commercial QA platforms. It covers 23 features in five groups, from licensing and pricing to AI-product testing and CI.

Every cell was checked on 21 October 2026 against the product’s own documentation, pricing page or repository. Each mark links to the page that supports it, and hovering a mark shows its note. The same notes, readable on any screen, are under Product details. The data lives in docs/comparison.json, the same file that produces docs/comparison.md and the table in the README, and this page is built from it.

Legend

Yes.Yes: documented and available
Partial.Partial: exists, with notable limits (see the note)
No.No: documented as absent
Not documented.Not documented: no evidence found, which is not the same as missing

Not documented means we found no evidence either way. It is not a claim that the feature is missing. In the Pricing model column, the mark means free to use (you may still pay your own model provider), free tier or open source plus paid plans, or paid only, and the text says how the product charges: free, usage, seat or sales.

Products

Open-source AI test tools: Auto Playwright, e2e by TesterArmy, Shortest (antiwork)
AI browser-agent frameworks (building blocks, not test platforms): Browser Use, Stagehand (Browserbase)
AI / LLM evaluation tools: DeepEval (Confident AI), LangWatch Scenario, promptfoo
Commercial AI QA platforms: Applitools, Katalon, mabl, Momentic, QA Wolf, Testim (Tricentis), testRigor, Testsigma, Octomind (likely defunct)

Browser Use and Stagehand are building blocks for writing your own browser agents, not test platforms, so platform features such as CI gates, triage or team roles aren’t expected from them.

License, hosting and pricing

License, hosting and pricing: Open-source license, Self-hostable, Pricing model, for each product. Each mark links to its source.
Product	Open-source license	Self-hostable	Pricing model
Plainwright	Yes. Apache-2.0. Source: LICENSE.	Yes. docker compose stack: Postgres, Redis, Playwright MCP per browser, API, worker, scheduler. Source: docker-compose.yml.	Free.free (Apache-2.0, self-hosted); you pay your LLM provider; optional Stripe billing off by default.
Open-source AI test tools
Auto Playwright	Yes. MIT. Abstracta wrote about it; repo is by lucgagan, not Abstracta. Source: github.com.	Yes. npm library inside your Playwright tests. Source: github.com.	Free.free (OSS); you pay OpenAI usage.
e2e by TesterArmy	Yes. Apache-2.0 (hosted TesterArmy platform is proprietary). Source: github.com.	Yes. Framework runs locally or in your CI; hosted browsers optional. Source: github.com.	Free tier or open source, plus paid plans.free OSS; hosted TesterArmy is usage-tiered: Free, $99, $299/mo by test runs; Enterprise sales.
Shortest (antiwork)	Yes. MIT. Source: github.com.	Yes. Library you run locally or in your CI. Source: github.com.	Free.free (OSS); you pay Anthropic API usage.
AI browser-agent frameworks (building blocks, not test platforms)
Browser Use	Yes. MIT (library); Browser Use Cloud is a separate paid hosted service. Source: github.com.	Yes. Library runs locally with your own browser and model; Cloud is hosted-only. Source: github.com.	Free tier or open source, plus paid plans.free (MIT) plus your LLM; Cloud pay-as-you-go from $0.02/browser-hour, $1 signup credit.
Stagehand (Browserbase)	Yes. MIT. Source: github.com.	Yes. Runs locally on your Chrome with your own model keys; Browserbase optional. Source: docs.stagehand.dev.	Free tier or open source, plus paid plans.free (MIT) plus your LLM; Browserbase optional: Free 1 browser-hour, Developer $20/mo, Startup $99/mo.
AI / LLM evaluation tools
DeepEval (Confident AI)	Yes. DeepEval Apache-2.0; Confident AI platform proprietary. Source: github.com.	Partial. Framework runs locally; Confident AI on-prem only on Enterprise. Source: confident-ai.com.	Free tier or open source, plus paid plans.free OSS; Confident AI Free, Starter $200/mo, Team $2,000/mo, Enterprise custom.
LangWatch Scenario	Yes. Apache-2.0 (GitHub license metadata). Source: github.com.	Yes. Library runs locally; LangWatch platform self-hosts via docker compose. Source: langwatch.ai.	Free tier or open source, plus paid plans.free OSS; LangWatch Developer free, Growth EUR 29/core-seat/mo + events; Enterprise custom.
promptfoo	Yes. MIT; Enterprise and On-Prem editions are commercial. Source: github.com.	Yes. Runs locally or self-hosted; On-Premise edition for enterprises. Source: promptfoo.dev.	Free tier or open source, plus paid plans.free Community (10k red-team probes/month); Enterprise and On-Prem by sales.
Commercial AI QA platforms
Applitools	No. Proprietary. Source: applitools.com.	Partial. SaaS; private cloud (Professional add-on, Enterprise) and custom on-prem (Enterprise add-on). Source: applitools.com.	Paid.usage: Starter $667/mo (annual, 1,000 page checkpoints); Professional/Enterprise quote; free trial.
Katalon	No. Proprietary; Studio has a free tier but is closed source. Source: katalon.com.	Partial. Studio and Runtime Engine run on your machines; private cloud/on-prem platform Enterprise-only. Source: katalon.com.	Free tier or open source, plus paid plans.seat: Free local tier; Professional $2,199/seat/yr list ($1,000 promo); Enterprise quote; CI/cloud add-ons.
mabl	No. Proprietary. Source: mabl.com.	Partial. SaaS; local and CI runs via CLI are free. Source: mabl.com.	Paid.sales: custom quote; 500 cloud-run credits/month baseline; 14-day trial.
Momentic	No. Proprietary; tests are YAML files in your repository. Source: momentic.ai.	Partial. Enterprise runs execute inside your network; result viewer self-hostable; control plane is SaaS. Source: momentic.ai.	Free tier or open source, plus paid plans.usage credits: Free 2,000/mo; Pay-as-you-go $125/mo; Enterprise test-based; no seat fees.
QA Wolf	No. Proprietary platform; generated tests are open-source Playwright/Appium code; CLI is open source. Source: qawolf.com.	Partial. SaaS platform; CLI runs your flows locally or in CI. Source: docs.qawolf.com.	Free tier or open source, plus paid plans.usage: $1/web run, $2 mobile/desktop; no seats; managed service by sales.
Testim (Tricentis)	No. Proprietary (Tricentis). Source: testim.io.	Partial. SaaS authoring; tests run locally, on any Selenium grid, or a VPC grid option. Source: testim.io.	Free tier or open source, plus paid plans.free Community plan after trial; Essentials/Pro/Mobile by sales quote; unlimited runs, parallel-based.
testRigor	No. Proprietary. Free tier makes tests public, not open-source software. Source: testrigor.com.	Not documented. On-prem not mentioned on site pages checked. Source: testrigor.com.	Free tier or open source, plus paid plans.free Public plan (tests public); Private Complete trial; Enterprise custom; no list prices.
Testsigma	No. Cloud platform proprietary; legacy Apache-2.0 Community Edition repo, last release v3.0.1 (Aug 2023). Source: github.com.	Partial. On-prem docker-compose deployment on Enterprise; old OSS self-host docs now redirect away. Source: testsigma.com.	Paid.sales: Pro and Enterprise by custom quote; no public prices or free tier.
Octomind (likely defunct)	No. Proprietary SaaS; Debugtopus and CLI helper tools were open source (now archived). Source: github.com.	Partial. SaaS; Debugtopus ran tests locally; private-location worker only tunneled. Source: raw.githubusercontent.com.	Not documented.Pricing page offline; domain had no DNS records on 2026-10-21.

Testing web apps

Testing web apps: Plain-English UI tests, No selectors / self-healing, API tests, Accessibility / performance / SEO checks, Visual regression, Natural-language assertions, Multi-browser (Chromium/Firefox/WebKit), for each product. Each mark links to its source.
Product	Plain-English UI tests	No selectors / self-healing	API tests	Accessibility / performance / SEO checks	Visual regression	Natural-language assertions	Multi-browser (Chromium/Firefox/WebKit)
Plainwright	Yes. LLM drives a real browser through Playwright MCP from a plain-English test. Source: backend/app/services/engines/ui.py.	Yes. Reads live snapshots; stale-ref and ambiguous-element auto-recovery; stuck detection. Source: backend/app/services/ai_service.py.	Yes. Plain-English API tests via an http_request tool; OpenAPI input; secret placeholders. Source: backend/app/services/engines/api.py.	Yes. axe accessibility, performance budgets (LCP/CLS/TBT), broken links, SEO; no LLM needed. Source: backend/app/services/engines/quality.py.	Yes. Pixel diff against approved baseline per test/browser/device; ignore regions; threshold. Source: backend/app/services/visual.py.	Yes. Plain-English end-of-run assertions graded by a judge on screenshot plus page text. Source: backend/app/services/assertions.py.	Yes. Chromium, Firefox and WebKit MCP services; emulated phone/tablet viewport presets. Source: docker-compose.yml.
Open-source AI test tools
Auto Playwright	Partial. Plain-text auto() steps embedded in Playwright code, not standalone tests. Source: github.com.	Partial. auto() steps need no selectors; no healing or caching documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Not documented; README says it does not use screenshots. Source: github.com.	Yes. auto() assertions are questions returning true/false. Source: github.com.	Yes. 'Every browser that Playwright supports.' Source: github.com.
e2e by TesterArmy	Yes. agent.act('natural-language goal') inside TypeScript tests; hosted product takes plain-language steps. Source: github.com.	Yes. Verified agent steps replay without model calls; agent takes over when app changes. Source: e2e.tester.army.	Yes. Code-based: fetch plus value/schema matchers in same suite; no model needed. Source: e2e.tester.army.	Not documented. No accessibility, performance or SEO checks in docs index. Source: e2e.tester.army.	Yes. toHaveScreenshot compares screen or element to stored PNG on web, iOS, Android. Source: e2e.tester.army.	Yes. agent.assert('plain-English condition'). Source: github.com.	Yes. Chromium, Firefox and WebKit via Playwright; iOS/Android simulators too. Source: github.com.
Shortest (antiwork)	Yes. Tests are plain-English strings, e.g. shortest('Login to the app...'). Source: github.com.	Partial. AI executes steps each run without selectors; no explicit self-healing mechanism documented. Source: github.com.	Yes. 'Test API endpoints using natural language' via APIRequest. Source: github.com.	Not documented. No accessibility, performance or SEO checks found in README. Source: github.com.	Not documented. Not mentioned in README. Source: github.com.	Yes. Expectations written in natural language; optional code callbacks for extra assertions. Source: github.com.	Not documented. Built on Playwright; only Chromium referenced. Source: github.com.
AI browser-agent frameworks (building blocks, not test platforms)
Browser Use	Partial. Agent runs natural-language browser tasks; no test runner or suites; separate QA skill scores sites. Source: github.com.	Partial. LLM chooses elements from live page state each step; no test-maintenance or healing features. Source: github.com.	Not documented. Browser agent only; HTTP calls possible via your own custom Python tools. Source: docs.browser-use.com.	Not documented. Not in core library; separate Vibetest-Use example (needs browser-use <0.5) flags broken links/a11y. Source: docs.browser-use.com.	Not documented. Screenshots feed the agent; no baseline comparison documented. Source: github.com.	Partial. Built-in LLM judge grades the run against optional ground_truth text; no per-step assertions. Source: github.com.	Not documented. Chromium-family only over CDP (chromium, chrome, msedge channels); Firefox/WebKit not documented. Source: docs.browser-use.com.
Stagehand (Browserbase)	Partial. act()/extract()/observe() take natural-language instructions inside code; no test runner. Source: docs.stagehand.dev.	Yes. act/observe/extract self-heal when the site changes; cached actions fall back to inference. Source: github.com.	Not documented. Browser automation SDK; API testing not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Screenshots via page API; no baseline comparison documented. Source: github.com.	Not documented. No assertion API; extract() returns schema-typed data you assert in your own code. Source: docs.stagehand.dev.	Not documented. Local Chrome or any Chromium over CDP; Firefox/WebKit not documented. Source: docs.stagehand.dev.
AI / LLM evaluation tools
DeepEval (Confident AI)	Not documented. Not a browser-testing tool. Source: github.com.	Not documented. Not a browser-testing tool. Source: github.com.	Not documented. No REST API testing documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Yes. G-Eval: LLM-as-judge on custom natural-language criteria. Source: deepeval.com.	Not documented. Not a browser-testing tool. Source: github.com.
LangWatch Scenario	Not documented. Not a browser-testing tool. Source: github.com.	Not documented. Not a browser-testing tool. Source: github.com.	Partial. HTTP adapters test agents deployed as HTTP services; not general REST testing. Source: scenario.langwatch.ai.	Not documented. Source: github.com.	Not documented. Source: github.com.	Yes. JudgeAgent evaluates natural-language success criteria each turn. Source: github.com.	Not documented. Not a browser-testing tool. Source: github.com.
promptfoo	No. Browser provider runs scripted steps with selectors, not plain-English goals. Source: promptfoo.dev.	No. Browser provider steps use CSS/role/XPath selectors. Source: promptfoo.dev.	Partial. HTTP provider targets LLM/agent endpoints; not general REST API testing. Source: promptfoo.dev.	Not documented. Source: promptfoo.dev.	Not documented. Source: promptfoo.dev.	Yes. Model-graded assertions such as llm-rubric. Source: promptfoo.dev.	No. Browser provider implements Chromium only. Source: promptfoo.dev.
Commercial AI QA platforms
Applitools	Yes. Autonomous: plain-English steps plus LLM-generated steps from stated intent (Professional+). Source: applitools.com.	Yes. Execution Cloud heals changed locators from recorded element data; enabled via support. Source: applitools.com.	Yes. API steps from cURL, raw HTTP or plain English; validate status, headers, body. Source: applitools.com.	Partial. Contrast Advisor checks WCAG contrast ratios; performance and SEO not documented. Source: applitools.com.	Yes. Core product: Visual AI baselines, match levels, ignore regions, grouped auto-maintenance. Source: applitools.com.	Partial. Plain-English 'Verify that...' assertions on page text and values; LLM judging not documented. Source: applitools.com.	Partial. Ultrafast Grid renders Chrome, Firefox, Safari, Edge; Autonomous user actions always run on Chrome. Source: applitools.com.
Katalon	Partial. AI agents generate Studio scripts from natural language; output is keyword/Groovy code. Source: katalon.com.	Yes. Fallback locators (XPath, CSS, attributes, image), then LLM-based AI healing; fixes need approval. Source: docs.katalon.com.	Yes. Web, mobile, API and desktop testing; AI generates API tests from OpenAPI (beta). Source: docs.katalon.com.	Partial. axe-core accessibility via custom keyword, 'not officially supported'; performance and SEO not documented. Source: docs.katalon.com.	Yes. TestOps Visual Testing: pixel, layout (AI) and content comparison against baselines. Source: docs.katalon.com.	Not documented. No LLM-judged natural-language assertions found; verifications are keywords or code. Source: docs.katalon.com.	Yes. Chrome, Firefox, Edge and Safari locally and on TestCloud. Source: docs.katalon.com.
mabl	Partial. Agents author low-code tests from requirements, flows or exploration. Source: mabl.com.	Yes. AI auto-healing keeps tests current as the app changes. Source: mabl.com.	Yes. API testing; import Postman collections. Source: mabl.com.	Partial. Accessibility and performance testing included; SEO not documented. Source: mabl.com.	Partial. AI visual assertions; pixel-baseline diffing not confirmed in sources checked. Source: help.mabl.com.	Yes. Visual (formerly GenAI) assertions from plain-language criteria; cloud runs only. Source: help.mabl.com.	Yes. Chrome, Edge, Firefox, Safari (WebKit) in cloud. Source: help.mabl.com.
Momentic	Yes. AI act steps pursue natural-language goals; NL element descriptions. Source: momentic.ai.	Yes. NL element targets; locator auto-heal, permanent healing, quarantine. Source: momentic.ai.	Partial. request and graphqlRequest steps inside tests; not a standalone API suite. Source: momentic.ai.	Not documented. No accessibility/performance/SEO checks in docs index. Source: momentic.ai.	Yes. visualDiff against golden files (viewport or element). Source: momentic.ai.	Yes. AI assert (DOM, accessibility tree, screenshot) and assertVisually. Source: momentic.ai.	No. Chromium-family only (Chromium, Chrome, Chrome for Testing). Source: momentic.ai.
QA Wolf	Yes. Request flows in natural language; agents write Playwright/Appium code. Source: qawolf.com.	Partial. Output is selector-based code; agents diagnose and repair broken tests. Source: qawolf.com.	Not documented. No dedicated API-testing docs found. Source: docs.qawolf.com.	Partial. axe-core accessibility and Lighthouse performance guides; SEO not documented. Source: docs.qawolf.com.	Yes. toHaveScreenshot baselines on web and native mobile. Source: docs.qawolf.com.	Yes. Semantic assertions (toSatisfy) and judge-model assertions on AI output. Source: docs.qawolf.com.	Yes. Chrome, Firefox and WebKit runs; real iOS devices, Android, Windows. Source: qawolf.com.
Testim (Tricentis)	Partial. Recorder-based low-code; Agentic Test Automation builds tests from natural language for Salesforce only. Source: docs.tricentis.com.	Yes. AI/ML smart locators self-heal to keep tests working as the app changes. Source: testim.io.	Yes. Add API action and Validate API steps (header, body, status); Pro plan. Source: docs.tricentis.com.	Partial. axe-core page/element accessibility validations (Pro; Chrome/Edge); performance and SEO not documented. Source: docs.tricentis.com.	Partial. Element, viewport and full-page visual validation via Applitools Eyes integration; Pro plan. Source: docs.tricentis.com.	Partial. Copilot turns a description into JavaScript custom-step code; no LLM-judged assertions found. Source: tricentis.com.	Yes. Chrome, Firefox, Safari, Edge on Testim grid or Selenium/third-party grids; local Chrome only. Source: docs.tricentis.com.
testRigor	Yes. 'free-flowing plain English to build test automation'. Source: testrigor.com.	Yes. Identifies elements as seen on screen, no XPath; AI self-healing page. Source: testrigor.com.	Yes. Invoke APIs, validate return codes, store results. Source: testrigor.com.	Partial. Accessibility via Deque axe per page; performance/SEO not mentioned. Source: testrigor.com.	Partial. Image comparison steps mentioned; baseline workflow not detailed. Source: testrigor.com.	Yes. 'check that ... using ai' sentiment, semantic and vision checks. Source: testrigor.com.	Yes. Cross-browser and cross-platform with a single test. Source: testrigor.com.
Testsigma	Yes. Steps are plain-English NLP actions; Generator agent turns prompts and requirements into tests. Source: testsigma.com.	Yes. Locator-based elements; Healer agent and GenAI autoheal repair locators mid-run for approval. Source: testsigma.com.	Yes. REST and SOAP API steps with response verifications; tests generated from Swagger/WSDL. Source: testsigma.com.	Partial. Per-step WCAG accessibility scans (Enterprise plan); performance and SEO not documented. Source: testsigma.com.	Yes. Screenshot baselines or Figma frames per step; comparison modes and ignore regions. Source: testsigma.com.	Partial. 'Verify' steps are plain-English NLP templates bound to elements; no LLM-judged assertions found. Source: testsigma.com.	Yes. 800+ browser/OS combinations in cloud lab; Chrome, Firefox, Edge capabilities documented. Source: testsigma.com.
Octomind (likely defunct)	Yes. Prompt the AI agent in natural language to generate a test. Source: raw.githubusercontent.com.	Partial. Generated deterministic Playwright code; AI auto-fix (beta) proposed fixes for review. Source: raw.githubusercontent.com.	Not documented. Not documented in archived docs. Source: github.com.	Not documented. Not documented in archived docs. Source: github.com.	Not documented. Not documented in archived docs. Source: github.com.	Not documented. Assertions were visual-editor types (contain text, style). Source: raw.githubusercontent.com.	Yes. Scheduled runs on Chromium, Firefox or Safari. Source: raw.githubusercontent.com.

Testing AI products

Testing AI products: Conversation / agent-to-agent tests, Red teaming, LLM eval suites, RAG evals, MCP server tests, A2A agent tests, for each product. Each mark links to its source.
Product	Conversation / agent-to-agent tests	Red teaming	LLM eval suites	RAG evals	MCP server tests	A2A agent tests
Plainwright	Yes. Simulated persona chats via browser UI, HTTP API or A2A; per-turn judge; repeated trials. Source: backend/app/services/engines/conversation.py.	Yes. Seven attack packs: injection, jailbreak, prompt leak, PII, off-topic, harmful, tool misuse. Source: backend/app/services/redteam_library.py.	Yes. CSV/JSONL datasets against an API; correctness, relevance, safety, rubric, latency metrics. Source: backend/app/services/engines/evals.py.	Yes. Faithfulness, answer relevance, context precision, context recall. Source: backend/app/services/judge.py.	Yes. Tests MCP servers over Streamable HTTP and SSE transports. Source: backend/app/services/engines/mcp.py.	Yes. Fetches the A2A agent card and runs judged conversations against the agent. Source: backend/app/services/engines/a2a.py.
Open-source AI test tools
Auto Playwright	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.
e2e by TesterArmy	Not documented. Personas run UI flows; no chatbot conversation testing documented. Source: e2e.tester.army.	Not documented. Source: e2e.tester.army.	Not documented. Source: e2e.tester.army.	Not documented. Source: e2e.tester.army.	Not documented. Ships an MCP server for coding agents; testing MCP servers not documented. Source: e2e.tester.army.	Not documented. Source: e2e.tester.army.
Shortest (antiwork)	Not documented. No chatbot/agent conversation testing documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.
AI browser-agent frameworks (building blocks, not test platforms)
Browser Use	Not documented. Building block; not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Ships an MCP server that drives browsers; testing MCP servers not documented. Source: docs.browser-use.com.	Not documented. Source: github.com.
Stagehand (Browserbase)	Not documented. Building block; not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Evals CLI benchmarks Stagehand primitives and models on web tasks, not your LLM app. Source: docs.stagehand.dev.	Not documented. Source: github.com.	Not documented. MCP adapter and WebMCP expose or invoke tools; testing MCP servers not documented. Source: docs.stagehand.dev.	Not documented. Source: github.com.
AI / LLM evaluation tools
DeepEval (Confident AI)	Yes. Conversation simulator plus multi-turn metrics (completeness, role adherence). Source: deepeval.com.	Partial. Via sister Apache-2.0 package DeepTeam; platform module is Enterprise add-on. Source: trydeepteam.com.	Yes. Core feature: many metrics, datasets, agent trajectory evals. Source: github.com.	Yes. Faithfulness, contextual precision/recall/relevancy, RAGAS. Source: deepeval.com.	Partial. MCP metrics score an agent's MCP usage; no direct MCP server test harness. Source: deepeval.com.	Not documented. No A2A integration found. Source: github.com.
LangWatch Scenario	Yes. UserSimulatorAgent plus JudgeAgent; scripted or autopilot; voice agents supported. Source: github.com.	Yes. RedTeamAgent with Crescendo and GOAT multi-turn attacks; JSON reports. Source: github.com.	Partial. Evaluators attach to scenarios; dataset experiments live in LangWatch platform. Source: langwatch.ai.	Partial. LangWatch evaluators such as faithfulness on RAG agents. Source: langwatch.ai.	Not documented. Not in Scenario docs index. Source: scenario.langwatch.ai.	Not documented. Not in Scenario docs index. Source: scenario.langwatch.ai.
promptfoo	Yes. Simulated-user provider runs multi-turn persona conversations; graded by assertions. Source: promptfoo.dev.	Yes. Plugins and strategies for vulnerability scanning of LLM apps and agents. Source: promptfoo.dev.	Yes. Core feature: declarative eval configs across providers. Source: github.com.	Yes. RAG guide; context-faithfulness, context-recall and related metrics. Source: promptfoo.dev.	Yes. MCP provider plus MCP security-testing guide. Source: promptfoo.dev.	Partial. A2A provider supports HTTP+JSON binding only; JSON-RPC and gRPC not yet. Source: promptfoo.dev.
Commercial AI QA platforms
Applitools	Not documented. No AI/chatbot application testing found. Source: applitools.com.	Not documented. Source: applitools.com.	Not documented. Source: applitools.com.	Not documented. Source: applitools.com.	Not documented. Applitools MCP server is for authoring with AI clients; testing MCP servers not documented. Source: applitools.com.	Not documented. Source: applitools.com.
Katalon	Not documented. No AI/chatbot application testing found in docs index. Source: docs.katalon.com.	Not documented. Source: docs.katalon.com.	Not documented. Source: docs.katalon.com.	Not documented. Source: docs.katalon.com.	Not documented. Katalon MCP servers connect AI clients for authoring; testing MCP servers not documented. Source: katalon.com.	Not documented. Source: docs.katalon.com.
mabl	Partial. Intent-based assertions on chatbot/LLM output; no multi-turn simulator documented. Source: mabl.com.	Partial. Validates guardrails and content policies; no attack library documented. Source: mabl.com.	Partial. Criteria-based checks of LLM output inside tests; no dataset eval suites. Source: mabl.com.	Not documented. Not mentioned. Source: mabl.com.	Not documented. mabl MCP server is for authoring; testing MCP servers not documented. Source: mabl.com.	Not documented. Source: mabl.com.
Momentic	Not documented. Source: momentic.ai.	Not documented. Source: momentic.ai.	Not documented. Source: momentic.ai.	Not documented. Source: momentic.ai.	Not documented. Mo consumes MCP servers for context; testing them not documented. Source: momentic.ai.	Not documented. Source: momentic.ai.
QA Wolf	Partial. Non-deterministic AI and voice-agent assertions; no persona simulator documented. Source: docs.qawolf.com.	Not documented. Source: docs.qawolf.com.	Partial. Judge-model verdicts on AI output inside flows; no dataset eval suites. Source: docs.qawolf.com.	Not documented. Source: docs.qawolf.com.	Partial. Homepage lists validating MCP tool calls; no dedicated docs found. Source: qawolf.com.	Not documented. Source: docs.qawolf.com.
Testim (Tricentis)	Not documented. No AI/chatbot application testing found. Source: tricentis.com.	Not documented. Source: tricentis.com.	Not documented. Source: tricentis.com.	Not documented. Source: tricentis.com.	Not documented. Source: tricentis.com.	Not documented. Source: tricentis.com.
testRigor	Partial. AI checks on chatbot replies in UI; no multi-turn simulator. Source: testrigor.com.	Not documented. Not mentioned. Source: testrigor.com.	Not documented. Only UI-level AI validations; no dataset eval suites. Source: testrigor.com.	Not documented. Not mentioned. Source: testrigor.com.	Not documented. Offers MCP server for authoring; testing MCP servers not documented. Source: testrigor.com.	Not documented. Source: testrigor.com.
Testsigma	Not documented. No chatbot or AI-app conversation testing found; agents target conventional apps. Source: testsigma.com.	Not documented. Source: testsigma.com.	Not documented. Source: testsigma.com.	Not documented. Source: testsigma.com.	Not documented. Arcus MCP server lets AI clients draft tests; testing MCP servers not documented. Source: testsigma.com.	Not documented. Source: testsigma.com.
Octomind (likely defunct)	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Had an MCP server for authoring; MCP testing not documented. Source: github.com.	Not documented. Source: github.com.

Models

Models: Bring your own LLM / key, Local LLM (Ollama), for each product. Each mark links to its source.
Product	Bring your own LLM / key	Local LLM (Ollama)
Plainwright	Yes. Anthropic, OpenAI, Gemini, OpenRouter, Groq, Ollama; project, user, system key priority. Source: backend/app/services/llm.py.	Yes. No key needed; live model discovery and pull from host Ollama. Source: backend/app/api/ai_configs.py.
Open-source AI test tools
Auto Playwright	Partial. Your own OpenAI or Azure OpenAI key (openaiBaseUrl); no other providers documented. Source: github.com.	Not documented. Source: github.com.
e2e by TesterArmy	Yes. Any AI SDK provider, gateway, API key or existing subscription. Source: e2e.tester.army.	Yes. OpenAI-compatible local server; docs reference Ollama. Source: e2e.tester.army.
Shortest (antiwork)	Partial. Your own Anthropic key; config shows provider 'anthropic' only. Source: github.com.	Not documented. No local-model support documented. Source: github.com.
AI browser-agent frameworks (building blocks, not test platforms)
Browser Use	Yes. OpenAI, Anthropic, Gemini, Groq, OpenRouter, DeepSeek, LiteLLM, any OpenAI-compatible provider. Source: docs.browser-use.com.	Yes. ChatOllama runs local models through Ollama. Source: docs.browser-use.com.
Stagehand (Browserbase)	Yes. OpenAI, Anthropic, Google with own keys; custom LLM callback for any provider. Source: docs.stagehand.dev.	Partial. Local/offline models via bring-your-own LLM callback; Ollama not named. Source: docs.stagehand.dev.
AI / LLM evaluation tools
DeepEval (Confident AI)	Yes. Judge metrics run on any LLM, including custom models. Source: github.com.	Yes. Ollama model integration. Source: deepeval.com.
LangWatch Scenario	Yes. LiteLLM (Python) / AI SDK (TypeScript) model strings for simulator and judge. Source: github.com.	Partial. Via LiteLLM/OpenAI-compatible endpoints; Ollama documented for LangWatch platform. Source: langwatch.ai.
promptfoo	Yes. Many providers: OpenAI, Anthropic, Bedrock, Azure, Python, gateways. Source: promptfoo.dev.	Yes. Ollama provider. Source: promptfoo.dev.
Commercial AI QA platforms
Applitools	Not documented. Autonomous uses proprietary AI including LLMs; bringing your own key not documented. Source: applitools.com.	Not documented. Source: applitools.com.
Katalon	Yes. StudioAssist: OpenAI, Azure OpenAI, Gemini, Bedrock or OpenAI-compatible key; platform BYO AI Enterprise-only. Source: docs.katalon.com.	Not documented. OpenAI-compatible base URL is configurable; local models or Ollama not documented. Source: docs.katalon.com.
mabl	Not documented. Source: mabl.com.	Not documented. Source: mabl.com.
Momentic	Not documented. Momentic routes to its hosted providers; BYO key not documented. Source: momentic.ai.	Not documented. Source: momentic.ai.
QA Wolf	Not documented. AI included ('no AI credits or token bills'); BYO key not documented. Source: qawolf.com.	Not documented. Source: docs.qawolf.com.
Testim (Tricentis)	Not documented. Copilot uses OpenAI via Tricentis; bringing your own key not documented. Source: help.testim.io.	Not documented. Source: help.testim.io.
testRigor	Not documented. Underlying LLM not stated. Source: testrigor.com.	Not documented. Source: testrigor.com.
Testsigma	Yes. BYOK for Azure OpenAI, OpenAI, Gemini and Vertex AI; per-feature model mapping. Source: testsigma.com.	Not documented. BYOK lists only cloud providers (Azure OpenAI, OpenAI, Gemini, Vertex AI). Source: testsigma.com.
Octomind (likely defunct)	Not documented. Source: github.com.	Not documented. Source: github.com.

Workflow

Workflow: CI gate, Failure triage + issue filing, AI root cause, Scheduling / monitoring, Team roles (RBAC), for each product. Each mark links to its source.
Product	CI gate	Failure triage + issue filing	AI root cause	Scheduling / monitoring	Team roles (RBAC)
Plainwright	Yes. CI webhook plus batch quality gate honoring thresholds, retries and quarantine. Source: backend/app/services/gates.py.	Yes. Clusters similar failures (new vs known); files Jira, Linear or GitHub issues. Source: backend/app/services/triage.py.	Yes. On-demand root cause: category, suggested fix, confidence; grounded 'why did this fail' chat. Source: backend/app/services/root_cause.py.	Yes. Cron per test via scheduler service; Slack-compatible webhook and email alerts. Source: backend/app/services/scheduler.py.	Yes. Project members with viewer, editor and admin roles; email invites. Source: backend/app/core/database.py.
Open-source AI test tools
Auto Playwright	Not documented. No CI docs; runs inside Playwright Test, which is CI-capable. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Library; no team features. Source: github.com.
e2e by TesterArmy	Yes. CI guides (GitHub Actions, EAS, Bitrise, Codemagic); PR comment reporter. Source: e2e.tester.army.	Not documented. No issue-tracker filing documented. Source: e2e.tester.army.	Not documented. Trace/debug pages; no AI root-cause feature documented. Source: e2e.tester.army.	Partial. Hosted TesterArmy schedules hourly/daily/weekly with Slack alerts; OSS relies on CI. Source: tester.army.	Not documented. Hosted plans list unlimited members; roles not documented. Source: tester.army.
Shortest (antiwork)	Yes. Runs headless in CI/CD; example GitHub workflow linked. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Not documented; would rely on your CI scheduler. Source: github.com.	Not documented. Library; no team features documented. Source: github.com.
AI browser-agent frameworks (building blocks, not test platforms)
Browser Use	Not documented. Can run headless in CI, but no pass/fail gate or test reporter documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. No scheduler in the library. Source: github.com.	Not documented. Library has no team features; Cloud project roles not documented. Source: github.com.
Stagehand (Browserbase)	Not documented. No CI gate or test reporter documented. Source: github.com.	Not documented. Source: github.com.	Not documented. Source: github.com.	Not documented. No scheduler; deployment guide invokes Stagehand via Vercel cron. Source: docs.stagehand.dev.	Not documented. Library; no team features. Source: github.com.
AI / LLM evaluation tools
DeepEval (Confident AI)	Yes. deepeval test run in any CI/CD (pytest-style). Source: deepeval.com.	Not documented. Source: confident-ai.com.	Not documented. Source: confident-ai.com.	Partial. Confident AI: online evals and real-time alerting from Starter; scheduled dataset runs. Source: confident-ai.com.	Partial. Custom RBAC and SSO on Confident AI Team tier and above. Source: confident-ai.com.
LangWatch Scenario	Yes. pytest/vitest tests; GitHub Actions CI/CD guide. Source: scenario.langwatch.ai.	Not documented. Source: github.com.	Partial. Red-team report lists findings with prioritized fixes; no general failure RCA. Source: scenario.langwatch.ai.	Partial. LangWatch platform alerts, automations and online-eval monitors. Source: langwatch.ai.	Partial. LangWatch platform RBAC; SSO/RBAC emphasized on Enterprise. Source: langwatch.ai.
promptfoo	Yes. GitHub Action and CI/CD integration guides. Source: promptfoo.dev.	Not documented. Source: promptfoo.dev.	Not documented. Source: promptfoo.dev.	Partial. Continuous monitoring listed for Enterprise only. Source: promptfoo.dev.	Partial. Enterprise: team access control, SSO, permission profiles. Source: promptfoo.dev.
Commercial AI QA platforms
Applitools	Yes. Trigger Autonomous plans from Jenkins or GitHub Actions via API; Eyes SDKs integrate with CI. Source: applitools.com.	Yes. Auto-maintenance groups similar diffs for batch resolution; create or link Jira issues. Source: applitools.com.	Partial. RCA shows DOM/CSS differences behind visual diffs; no natural-language failure explanation found. Source: applitools.com.	Yes. Autonomous plans run once, daily, weekly or monthly across browsers and devices. Source: applitools.com.	Yes. Per-team access rights, including View only and Team admin. Source: applitools.com.
Katalon	Yes. Runtime Engine add-on runs tests in Jenkins, GitHub Actions, Azure DevOps, GitLab, Docker. Source: katalon.com.	Yes. Submit failures to Jira; TestOps AI drafts defects from failure evidence for review. Source: katalon.com.	Yes. AI failure analysis adds the failure reason to HTML, PDF and email reports. Source: docs.katalon.com.	Yes. Scheduled test runs in TestOps; scheduled suites in Cloud Studio (beta). Source: docs.katalon.com.	Yes. Roles and permissions in platform admin; RBAC and audit logs listed for Enterprise. Source: docs.katalon.com.
mabl	Yes. GitHub Actions, CircleCI, Jenkins, Bitbucket, GitLab integrations. Source: mabl.com.	Yes. Create Jira issues from runs; Auto TFA pre-fills (Advanced AI add-on). Source: help.mabl.com.	Yes. Agent failure analysis with suggested failure reason. Source: help.mabl.com.	Yes. Plan triggers: timer or schedule (cloud runs); auto re-run on failure. Source: help.mabl.com.	Yes. Owner, Editor and Viewer workspace roles. Source: help.mabl.com.
Momentic	Yes. GitHub Actions with sharding; PR check and deploy-gate setups. Source: momentic.ai.	Yes. Failure classification and triage; Mo files bugs to Linear. Source: momentic.ai.	Yes. Classification and triage explain failures and decide whether to heal permanently. Source: momentic.ai.	Yes. Scheduled nightly regression suites; Slack notifications. Source: momentic.ai.	Yes. Team members and roles; RBAC, SAML SSO, SCIM on Enterprise. Source: momentic.ai.
QA Wolf	Yes. CI SDK waits for QA Wolf's verdict; GitHub status checks on PRs. Source: docs.qawolf.com.	Yes. Failures reproduced by agent; bugs filed and synced to Jira or Linear. Source: docs.qawolf.com.	Yes. Agents investigate failures and fix outdated tests; humans in managed tier. Source: qawolf.com.	Yes. Triggers run flows on deploy or on a schedule. Source: docs.qawolf.com.	Partial. SSO (SAML/OIDC) documented; no per-seat limits; role model not found. Source: docs.qawolf.com.
Testim (Tricentis)	Yes. Testim CLI with generated CI commands and JUnit XML report for CI display. Source: docs.tricentis.com.	Yes. One-button bug reports to Jira, Trello or GitHub; error aggregation across runs. Source: tricentis.com.	Partial. RCA with highlighted screenshots, console/network logs and failure suggestions; AI cause analysis not described. Source: testim.io.	Yes. Scheduler runs labeled tests or suites; email, Slack and webhook notifications. Source: docs.tricentis.com.	Partial. Project owners and teammates only; teammates can view, edit and run all tests. Source: docs.tricentis.com.
testRigor	Yes. GitHub Actions, GitLab, Jenkins, Azure DevOps, CircleCI integrations. Source: testrigor.com.	Partial. Jira, Zephyr, TestRail integrations listed; auto-filing not described. Source: testrigor.com.	Not documented. Source: testrigor.com.	Yes. Scheduled suite runs via monitoring, weekly to every four hours. Source: testrigor.com.	Partial. Teams and company admins; Enterprise plans only (guide may be dated). Source: testrigor.com.
Testsigma	Yes. GitHub Actions, Jenkins, or shell script that triggers a plan, waits, saves JUnit. Source: testsigma.com.	Yes. Bug Reporter files analyzed failures to Jira, Linear, Bugzilla, Trello and others. Source: testsigma.com.	Yes. Analyzer agent: error type, root cause, visual evidence, fix suggestions with Apply Fix. Source: testsigma.com.	Yes. Scheduled test plans via UI or REST API; parallel runs. Source: testsigma.com.	Yes. Six per-project roles from Account Administrator to Read Only; audit logs. Source: testsigma.com.
Octomind (likely defunct)	Yes. GitHub and Azure DevOps native; Jenkins and Vercel via API. Source: raw.githubusercontent.com.	Not documented. Test-management sync (Xray, Zephyr) only; no issue filing found. Source: github.com.	Partial. Auto-fix analyzed failure root cause; RCA playbook 'under active development'. Source: raw.githubusercontent.com.	Yes. Scheduled runs with browser and screen-size selection. Source: raw.githubusercontent.com.	Not documented. Not documented in archived docs. Source: github.com.

Notes

Octomind appears to be defunct: on 21 October 2026 its domain had no DNS records and its repositories were archived in July 2026. Its cells come from its archived documentation.
Browser Use and Stagehand are frameworks you build agents with. A not documented or no in their workflow columns says what they are, not that they are lacking.
Plainwright’s own cells cite files in this repository (such as docker-compose.yml) and were checked against the code on 21 October 2026.
Partial always comes with a note that says what the limit is: an Enterprise-only plan, a feature that needs a paid add-on, or one that covers only part of the column.
The landing page’s Compare section shows a trimmed copy of the same data with eight headline columns. A grouped column there shows ✓ only when every feature in it is ✓.

Product details

Every product’s full row, with the note and source behind each mark.

Plainwright · This project

Apache-2.0, self-hosted AI-native testing: plain-English browser/API tests plus LLM evals, red teaming, MCP and A2A tests.

License, hosting and pricing

Yes. Open-source license. Apache-2.0. LICENSE
Yes. Self-hostable. docker compose stack: Postgres, Redis, Playwright MCP per browser, API, worker, scheduler. docker-compose.yml
Yes. Pricing model. free (Apache-2.0, self-hosted); you pay your LLM provider; optional Stripe billing off by default. backend/app/core/config.py

Testing web apps

Yes. Plain-English UI tests. LLM drives a real browser through Playwright MCP from a plain-English test. backend/app/services/engines/ui.py
Yes. No selectors / self-healing. Reads live snapshots; stale-ref and ambiguous-element auto-recovery; stuck detection. backend/app/services/ai_service.py
Yes. API tests. Plain-English API tests via an http_request tool; OpenAPI input; secret placeholders. backend/app/services/engines/api.py
Yes. Accessibility / performance / SEO checks. axe accessibility, performance budgets (LCP/CLS/TBT), broken links, SEO; no LLM needed. backend/app/services/engines/quality.py
Yes. Visual regression. Pixel diff against approved baseline per test/browser/device; ignore regions; threshold. backend/app/services/visual.py
Yes. Natural-language assertions. Plain-English end-of-run assertions graded by a judge on screenshot plus page text. backend/app/services/assertions.py
Yes. Multi-browser (Chromium/Firefox/WebKit). Chromium, Firefox and WebKit MCP services; emulated phone/tablet viewport presets. docker-compose.yml

Testing AI products

Yes. Conversation / agent-to-agent tests. Simulated persona chats via browser UI, HTTP API or A2A; per-turn judge; repeated trials. backend/app/services/engines/conversation.py
Yes. Red teaming. Seven attack packs: injection, jailbreak, prompt leak, PII, off-topic, harmful, tool misuse. backend/app/services/redteam_library.py
Yes. LLM eval suites. CSV/JSONL datasets against an API; correctness, relevance, safety, rubric, latency metrics. backend/app/services/engines/evals.py
Yes. RAG evals. Faithfulness, answer relevance, context precision, context recall. backend/app/services/judge.py
Yes. MCP server tests. Tests MCP servers over Streamable HTTP and SSE transports. backend/app/services/engines/mcp.py
Yes. A2A agent tests. Fetches the A2A agent card and runs judged conversations against the agent. backend/app/services/engines/a2a.py

Models

Yes. Bring your own LLM / key. Anthropic, OpenAI, Gemini, OpenRouter, Groq, Ollama; project, user, system key priority. backend/app/services/llm.py
Yes. Local LLM (Ollama). No key needed; live model discovery and pull from host Ollama. backend/app/api/ai_configs.py

Workflow

Yes. CI gate. CI webhook plus batch quality gate honoring thresholds, retries and quarantine. backend/app/services/gates.py
Yes. Failure triage + issue filing. Clusters similar failures (new vs known); files Jira, Linear or GitHub issues. backend/app/services/triage.py
Yes. AI root cause. On-demand root cause: category, suggested fix, confidence; grounded 'why did this fail' chat. backend/app/services/root_cause.py
Yes. Scheduling / monitoring. Cron per test via scheduler service; Slack-compatible webhook and email alerts. backend/app/services/scheduler.py
Yes. Team roles (RBAC). Project members with viewer, editor and admin roles; email invites. backend/app/core/database.py

Auto Playwright · Open-source test tool

MIT auto() helper for Playwright (author Luc Gagan) inspired by ZeroStep; OpenAI-driven. Low activity since mid-2025. github.com

License, hosting and pricing

Yes. Open-source license. MIT. Abstracta wrote about it; repo is by lucgagan, not Abstracta. github.com
Yes. Self-hostable. npm library inside your Playwright tests. github.com
Yes. Pricing model. free (OSS); you pay OpenAI usage. github.com

Testing web apps

Partial. Plain-English UI tests. Plain-text auto() steps embedded in Playwright code, not standalone tests. github.com
Partial. No selectors / self-healing. auto() steps need no selectors; no healing or caching documented. github.com
Not documented. API tests. Not documented. github.com
Not documented. Accessibility / performance / SEO checks. Not documented. github.com
Not documented. Visual regression. Not documented; README says it does not use screenshots. github.com
Yes. Natural-language assertions. auto() assertions are questions returning true/false. github.com
Yes. Multi-browser (Chromium/Firefox/WebKit). 'Every browser that Playwright supports.' github.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. Not documented. github.com
Not documented. Red teaming. Not documented. github.com
Not documented. LLM eval suites. Not documented. github.com
Not documented. RAG evals. Not documented. github.com
Not documented. MCP server tests. Not documented. github.com
Not documented. A2A agent tests. Not documented. github.com

Models

Partial. Bring your own LLM / key. Your own OpenAI or Azure OpenAI key (openaiBaseUrl); no other providers documented. github.com
Not documented. Local LLM (Ollama). Not documented. github.com

Workflow

Not documented. CI gate. No CI docs; runs inside Playwright Test, which is CI-capable. github.com
Not documented. Failure triage + issue filing. Not documented. github.com
Not documented. AI root cause. Not documented. github.com
Not documented. Scheduling / monitoring. Not documented. github.com
Not documented. Team roles (RBAC). Library; no team features. github.com

e2e by TesterArmy · Open-source test tool

Apache-2.0 agentic E2E framework for web and mobile; NL goals in TypeScript tests with cached replay. Pre-1.0. github.com

License, hosting and pricing

Yes. Open-source license. Apache-2.0 (hosted TesterArmy platform is proprietary). github.com
Yes. Self-hostable. Framework runs locally or in your CI; hosted browsers optional. github.com
Partial. Pricing model. free OSS; hosted TesterArmy is usage-tiered: Free, $99, $299/mo by test runs; Enterprise sales. tester.army

Testing web apps

Yes. Plain-English UI tests. agent.act('natural-language goal') inside TypeScript tests; hosted product takes plain-language steps. github.com
Yes. No selectors / self-healing. Verified agent steps replay without model calls; agent takes over when app changes. e2e.tester.army
Yes. API tests. Code-based: fetch plus value/schema matchers in same suite; no model needed. e2e.tester.army
Not documented. Accessibility / performance / SEO checks. No accessibility, performance or SEO checks in docs index. e2e.tester.army
Yes. Visual regression. toHaveScreenshot compares screen or element to stored PNG on web, iOS, Android. e2e.tester.army
Yes. Natural-language assertions. agent.assert('plain-English condition'). github.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Chromium, Firefox and WebKit via Playwright; iOS/Android simulators too. github.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. Personas run UI flows; no chatbot conversation testing documented. e2e.tester.army
Not documented. Red teaming. Not documented. e2e.tester.army
Not documented. LLM eval suites. Not documented. e2e.tester.army
Not documented. RAG evals. Not documented. e2e.tester.army
Not documented. MCP server tests. Ships an MCP server for coding agents; testing MCP servers not documented. e2e.tester.army
Not documented. A2A agent tests. Not documented. e2e.tester.army

Models

Yes. Bring your own LLM / key. Any AI SDK provider, gateway, API key or existing subscription. e2e.tester.army
Yes. Local LLM (Ollama). OpenAI-compatible local server; docs reference Ollama. e2e.tester.army

Workflow

Yes. CI gate. CI guides (GitHub Actions, EAS, Bitrise, Codemagic); PR comment reporter. e2e.tester.army
Not documented. Failure triage + issue filing. No issue-tracker filing documented. e2e.tester.army
Not documented. AI root cause. Trace/debug pages; no AI root-cause feature documented. e2e.tester.army
Partial. Scheduling / monitoring. Hosted TesterArmy schedules hourly/daily/weekly with Slack alerts; OSS relies on CI. tester.army
Not documented. Team roles (RBAC). Hosted plans list unlimited members; roles not documented. tester.army

Shortest (antiwork) · Open-source test tool

MIT natural-language E2E framework on Playwright, executed by Anthropic Claude. Last npm release 0.4.9 (Apr 2025). github.com

License, hosting and pricing

Yes. Open-source license. MIT. github.com
Yes. Self-hostable. Library you run locally or in your CI. github.com
Yes. Pricing model. free (OSS); you pay Anthropic API usage. github.com

Testing web apps

Yes. Plain-English UI tests. Tests are plain-English strings, e.g. shortest('Login to the app...'). github.com
Partial. No selectors / self-healing. AI executes steps each run without selectors; no explicit self-healing mechanism documented. github.com
Yes. API tests. 'Test API endpoints using natural language' via APIRequest. github.com
Not documented. Accessibility / performance / SEO checks. No accessibility, performance or SEO checks found in README. github.com
Not documented. Visual regression. Not mentioned in README. github.com
Yes. Natural-language assertions. Expectations written in natural language; optional code callbacks for extra assertions. github.com
Not documented. Multi-browser (Chromium/Firefox/WebKit). Built on Playwright; only Chromium referenced. github.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. No chatbot/agent conversation testing documented. github.com
Not documented. Red teaming. Not documented. github.com
Not documented. LLM eval suites. Not documented. github.com
Not documented. RAG evals. Not documented. github.com
Not documented. MCP server tests. Not documented. github.com
Not documented. A2A agent tests. Not documented. github.com

Models

Partial. Bring your own LLM / key. Your own Anthropic key; config shows provider 'anthropic' only. github.com
Not documented. Local LLM (Ollama). No local-model support documented. github.com

Workflow

Yes. CI gate. Runs headless in CI/CD; example GitHub workflow linked. github.com
Not documented. Failure triage + issue filing. Not documented. github.com
Not documented. AI root cause. Not documented. github.com
Not documented. Scheduling / monitoring. Not documented; would rely on your CI scheduler. github.com
Not documented. Team roles (RBAC). Library; no team features documented. github.com

Browser Use · Agent framework

MIT Python library for LLM-driven browser agents (any model, including Ollama) plus paid Browser Use Cloud; a building block, not a test platform. github.com

License, hosting and pricing

Yes. Open-source license. MIT (library); Browser Use Cloud is a separate paid hosted service. github.com
Yes. Self-hostable. Library runs locally with your own browser and model; Cloud is hosted-only. github.com
Partial. Pricing model. free (MIT) plus your LLM; Cloud pay-as-you-go from $0.02/browser-hour, $1 signup credit. browser-use.com

Testing web apps

Partial. Plain-English UI tests. Agent runs natural-language browser tasks; no test runner or suites; separate QA skill scores sites. github.com
Partial. No selectors / self-healing. LLM chooses elements from live page state each step; no test-maintenance or healing features. github.com
Not documented. API tests. Browser agent only; HTTP calls possible via your own custom Python tools. docs.browser-use.com
Not documented. Accessibility / performance / SEO checks. Not in core library; separate Vibetest-Use example (needs browser-use <0.5) flags broken links/a11y. docs.browser-use.com
Not documented. Visual regression. Screenshots feed the agent; no baseline comparison documented. github.com
Partial. Natural-language assertions. Built-in LLM judge grades the run against optional ground_truth text; no per-step assertions. github.com
Not documented. Multi-browser (Chromium/Firefox/WebKit). Chromium-family only over CDP (chromium, chrome, msedge channels); Firefox/WebKit not documented. docs.browser-use.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. Building block; not documented. github.com
Not documented. Red teaming. Not documented. github.com
Not documented. LLM eval suites. Not documented. github.com
Not documented. RAG evals. Not documented. github.com
Not documented. MCP server tests. Ships an MCP server that drives browsers; testing MCP servers not documented. docs.browser-use.com
Not documented. A2A agent tests. Not documented. github.com

Models

Yes. Bring your own LLM / key. OpenAI, Anthropic, Gemini, Groq, OpenRouter, DeepSeek, LiteLLM, any OpenAI-compatible provider. docs.browser-use.com
Yes. Local LLM (Ollama). ChatOllama runs local models through Ollama. docs.browser-use.com

Workflow

Not documented. CI gate. Can run headless in CI, but no pass/fail gate or test reporter documented. github.com
Not documented. Failure triage + issue filing. Not documented. github.com
Not documented. AI root cause. Not documented. github.com
Not documented. Scheduling / monitoring. No scheduler in the library. github.com
Not documented. Team roles (RBAC). Library has no team features; Cloud project roles not documented. github.com

Stagehand (Browserbase) · Agent framework

MIT TypeScript/Python/Go SDK from Browserbase adding natural-language act/extract/observe to Chromium automation; a building block, not a test platform. github.com

License, hosting and pricing

Yes. Open-source license. MIT. github.com
Yes. Self-hostable. Runs locally on your Chrome with your own model keys; Browserbase optional. docs.stagehand.dev
Partial. Pricing model. free (MIT) plus your LLM; Browserbase optional: Free 1 browser-hour, Developer $20/mo, Startup $99/mo. browserbase.com

Testing web apps

Partial. Plain-English UI tests. act()/extract()/observe() take natural-language instructions inside code; no test runner. docs.stagehand.dev
Yes. No selectors / self-healing. act/observe/extract self-heal when the site changes; cached actions fall back to inference. github.com
Not documented. API tests. Browser automation SDK; API testing not documented. github.com
Not documented. Accessibility / performance / SEO checks. Not documented. github.com
Not documented. Visual regression. Screenshots via page API; no baseline comparison documented. github.com
Not documented. Natural-language assertions. No assertion API; extract() returns schema-typed data you assert in your own code. docs.stagehand.dev
Not documented. Multi-browser (Chromium/Firefox/WebKit). Local Chrome or any Chromium over CDP; Firefox/WebKit not documented. docs.stagehand.dev

Testing AI products

Not documented. Conversation / agent-to-agent tests. Building block; not documented. github.com
Not documented. Red teaming. Not documented. github.com
Not documented. LLM eval suites. Evals CLI benchmarks Stagehand primitives and models on web tasks, not your LLM app. docs.stagehand.dev
Not documented. RAG evals. Not documented. github.com
Not documented. MCP server tests. MCP adapter and WebMCP expose or invoke tools; testing MCP servers not documented. docs.stagehand.dev
Not documented. A2A agent tests. Not documented. github.com

Models

Yes. Bring your own LLM / key. OpenAI, Anthropic, Google with own keys; custom LLM callback for any provider. docs.stagehand.dev
Partial. Local LLM (Ollama). Local/offline models via bring-your-own LLM callback; Ollama not named. docs.stagehand.dev

Workflow

Not documented. CI gate. No CI gate or test reporter documented. github.com
Not documented. Failure triage + issue filing. Not documented. github.com
Not documented. AI root cause. Not documented. github.com
Not documented. Scheduling / monitoring. No scheduler; deployment guide invokes Stagehand via Vercel cron. docs.stagehand.dev
Not documented. Team roles (RBAC). Library; no team features. github.com

DeepEval (Confident AI) · Eval tool

Apache-2.0 pytest-style LLM eval framework (agents, RAG, multi-turn, MCP metrics); Confident AI is the hosted platform. github.com

License, hosting and pricing

Yes. Open-source license. DeepEval Apache-2.0; Confident AI platform proprietary. github.com
Partial. Self-hostable. Framework runs locally; Confident AI on-prem only on Enterprise. confident-ai.com
Partial. Pricing model. free OSS; Confident AI Free, Starter $200/mo, Team $2,000/mo, Enterprise custom. confident-ai.com

Testing web apps

Not documented. Plain-English UI tests. Not a browser-testing tool. github.com
Not documented. No selectors / self-healing. Not a browser-testing tool. github.com
Not documented. API tests. No REST API testing documented. github.com
Not documented. Accessibility / performance / SEO checks. Not documented. github.com
Not documented. Visual regression. Not documented. github.com
Yes. Natural-language assertions. G-Eval: LLM-as-judge on custom natural-language criteria. deepeval.com
Not documented. Multi-browser (Chromium/Firefox/WebKit). Not a browser-testing tool. github.com

Testing AI products

Yes. Conversation / agent-to-agent tests. Conversation simulator plus multi-turn metrics (completeness, role adherence). deepeval.com
Partial. Red teaming. Via sister Apache-2.0 package DeepTeam; platform module is Enterprise add-on. trydeepteam.com
Yes. LLM eval suites. Core feature: many metrics, datasets, agent trajectory evals. github.com
Yes. RAG evals. Faithfulness, contextual precision/recall/relevancy, RAGAS. deepeval.com
Partial. MCP server tests. MCP metrics score an agent's MCP usage; no direct MCP server test harness. deepeval.com
Not documented. A2A agent tests. No A2A integration found. github.com

Models

Yes. Bring your own LLM / key. Judge metrics run on any LLM, including custom models. github.com
Yes. Local LLM (Ollama). Ollama model integration. deepeval.com

Workflow

Yes. CI gate. deepeval test run in any CI/CD (pytest-style). deepeval.com
Not documented. Failure triage + issue filing. Not documented. confident-ai.com
Not documented. AI root cause. Not documented. confident-ai.com
Partial. Scheduling / monitoring. Confident AI: online evals and real-time alerting from Starter; scheduled dataset runs. confident-ai.com
Partial. Team roles (RBAC). Custom RBAC and SSO on Confident AI Team tier and above. confident-ai.com

LangWatch Scenario · Eval tool

Apache-2.0 simulation-based agent testing (user simulator + judge, red teaming, voice) for Python/TypeScript; pairs with LangWatch. github.com

License, hosting and pricing

Yes. Open-source license. Apache-2.0 (GitHub license metadata). github.com
Yes. Self-hostable. Library runs locally; LangWatch platform self-hosts via docker compose. langwatch.ai
Partial. Pricing model. free OSS; LangWatch Developer free, Growth EUR 29/core-seat/mo + events; Enterprise custom. langwatch.ai

Testing web apps

Not documented. Plain-English UI tests. Not a browser-testing tool. github.com
Not documented. No selectors / self-healing. Not a browser-testing tool. github.com
Partial. API tests. HTTP adapters test agents deployed as HTTP services; not general REST testing. scenario.langwatch.ai
Not documented. Accessibility / performance / SEO checks. Not documented. github.com
Not documented. Visual regression. Not documented. github.com
Yes. Natural-language assertions. JudgeAgent evaluates natural-language success criteria each turn. github.com
Not documented. Multi-browser (Chromium/Firefox/WebKit). Not a browser-testing tool. github.com

Testing AI products

Yes. Conversation / agent-to-agent tests. UserSimulatorAgent plus JudgeAgent; scripted or autopilot; voice agents supported. github.com
Yes. Red teaming. RedTeamAgent with Crescendo and GOAT multi-turn attacks; JSON reports. github.com
Partial. LLM eval suites. Evaluators attach to scenarios; dataset experiments live in LangWatch platform. langwatch.ai
Partial. RAG evals. LangWatch evaluators such as faithfulness on RAG agents. langwatch.ai
Not documented. MCP server tests. Not in Scenario docs index. scenario.langwatch.ai
Not documented. A2A agent tests. Not in Scenario docs index. scenario.langwatch.ai

Models

Yes. Bring your own LLM / key. LiteLLM (Python) / AI SDK (TypeScript) model strings for simulator and judge. github.com
Partial. Local LLM (Ollama). Via LiteLLM/OpenAI-compatible endpoints; Ollama documented for LangWatch platform. langwatch.ai

Workflow

Yes. CI gate. pytest/vitest tests; GitHub Actions CI/CD guide. scenario.langwatch.ai
Not documented. Failure triage + issue filing. Not documented. github.com
Partial. AI root cause. Red-team report lists findings with prioritized fixes; no general failure RCA. scenario.langwatch.ai
Partial. Scheduling / monitoring. LangWatch platform alerts, automations and online-eval monitors. langwatch.ai
Partial. Team roles (RBAC). LangWatch platform RBAC; SSO/RBAC emphasized on Enterprise. langwatch.ai

promptfoo · Eval tool

MIT CLI for LLM/agent evals and red teaming; acquisition by OpenAI announced 2026-03-09, stays open source. github.com

License, hosting and pricing

Yes. Open-source license. MIT; Enterprise and On-Prem editions are commercial. github.com
Yes. Self-hostable. Runs locally or self-hosted; On-Premise edition for enterprises. promptfoo.dev
Partial. Pricing model. free Community (10k red-team probes/month); Enterprise and On-Prem by sales. promptfoo.dev

Testing web apps

No. Plain-English UI tests. Browser provider runs scripted steps with selectors, not plain-English goals. promptfoo.dev
No. No selectors / self-healing. Browser provider steps use CSS/role/XPath selectors. promptfoo.dev
Partial. API tests. HTTP provider targets LLM/agent endpoints; not general REST API testing. promptfoo.dev
Not documented. Accessibility / performance / SEO checks. Not documented. promptfoo.dev
Not documented. Visual regression. Not documented. promptfoo.dev
Yes. Natural-language assertions. Model-graded assertions such as llm-rubric. promptfoo.dev
No. Multi-browser (Chromium/Firefox/WebKit). Browser provider implements Chromium only. promptfoo.dev

Testing AI products

Yes. Conversation / agent-to-agent tests. Simulated-user provider runs multi-turn persona conversations; graded by assertions. promptfoo.dev
Yes. Red teaming. Plugins and strategies for vulnerability scanning of LLM apps and agents. promptfoo.dev
Yes. LLM eval suites. Core feature: declarative eval configs across providers. github.com
Yes. RAG evals. RAG guide; context-faithfulness, context-recall and related metrics. promptfoo.dev
Yes. MCP server tests. MCP provider plus MCP security-testing guide. promptfoo.dev
Partial. A2A agent tests. A2A provider supports HTTP+JSON binding only; JSON-RPC and gRPC not yet. promptfoo.dev

Models

Yes. Bring your own LLM / key. Many providers: OpenAI, Anthropic, Bedrock, Azure, Python, gateways. promptfoo.dev
Yes. Local LLM (Ollama). Ollama provider. promptfoo.dev

Workflow

Yes. CI gate. GitHub Action and CI/CD integration guides. promptfoo.dev
Not documented. Failure triage + issue filing. Not documented. promptfoo.dev
Not documented. AI root cause. Not documented. promptfoo.dev
Partial. Scheduling / monitoring. Continuous monitoring listed for Enterprise only. promptfoo.dev
Partial. Team roles (RBAC). Enterprise: team access control, SSO, permission profiles. promptfoo.dev

Applitools · Commercial platform

Visual AI regression testing (Eyes, Ultrafast Grid, Execution Cloud) plus Autonomous plain-English functional and API tests; checkpoint-based pricing. applitools.com

License, hosting and pricing

No. Open-source license. Proprietary. applitools.com
Partial. Self-hostable. SaaS; private cloud (Professional add-on, Enterprise) and custom on-prem (Enterprise add-on). applitools.com
No. Pricing model. usage: Starter $667/mo (annual, 1,000 page checkpoints); Professional/Enterprise quote; free trial. applitools.com

Testing web apps

Yes. Plain-English UI tests. Autonomous: plain-English steps plus LLM-generated steps from stated intent (Professional+). applitools.com
Yes. No selectors / self-healing. Execution Cloud heals changed locators from recorded element data; enabled via support. applitools.com
Yes. API tests. API steps from cURL, raw HTTP or plain English; validate status, headers, body. applitools.com
Partial. Accessibility / performance / SEO checks. Contrast Advisor checks WCAG contrast ratios; performance and SEO not documented. applitools.com
Yes. Visual regression. Core product: Visual AI baselines, match levels, ignore regions, grouped auto-maintenance. applitools.com
Partial. Natural-language assertions. Plain-English 'Verify that...' assertions on page text and values; LLM judging not documented. applitools.com
Partial. Multi-browser (Chromium/Firefox/WebKit). Ultrafast Grid renders Chrome, Firefox, Safari, Edge; Autonomous user actions always run on Chrome. applitools.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. No AI/chatbot application testing found. applitools.com
Not documented. Red teaming. Not documented. applitools.com
Not documented. LLM eval suites. Not documented. applitools.com
Not documented. RAG evals. Not documented. applitools.com
Not documented. MCP server tests. Applitools MCP server is for authoring with AI clients; testing MCP servers not documented. applitools.com
Not documented. A2A agent tests. Not documented. applitools.com

Models

Not documented. Bring your own LLM / key. Autonomous uses proprietary AI including LLMs; bringing your own key not documented. applitools.com
Not documented. Local LLM (Ollama). Not documented. applitools.com

Workflow

Yes. CI gate. Trigger Autonomous plans from Jenkins or GitHub Actions via API; Eyes SDKs integrate with CI. applitools.com
Yes. Failure triage + issue filing. Auto-maintenance groups similar diffs for batch resolution; create or link Jira issues. applitools.com
Partial. AI root cause. RCA shows DOM/CSS differences behind visual diffs; no natural-language failure explanation found. applitools.com
Yes. Scheduling / monitoring. Autonomous plans run once, daily, weekly or monthly across browsers and devices. applitools.com
Yes. Team roles (RBAC). Per-team access rights, including View only and Team admin. applitools.com

Katalon · Commercial platform

Studio IDE (no/low/full code) plus TestOps, TestCloud and TrueTest; StudioAssist AI agents, self-healing and failure analysis; per-seat pricing. katalon.com

License, hosting and pricing

No. Open-source license. Proprietary; Studio has a free tier but is closed source. katalon.com
Partial. Self-hostable. Studio and Runtime Engine run on your machines; private cloud/on-prem platform Enterprise-only. katalon.com
Partial. Pricing model. seat: Free local tier; Professional $2,199/seat/yr list ($1,000 promo); Enterprise quote; CI/cloud add-ons. katalon.com

Testing web apps

Partial. Plain-English UI tests. AI agents generate Studio scripts from natural language; output is keyword/Groovy code. katalon.com
Yes. No selectors / self-healing. Fallback locators (XPath, CSS, attributes, image), then LLM-based AI healing; fixes need approval. docs.katalon.com
Yes. API tests. Web, mobile, API and desktop testing; AI generates API tests from OpenAPI (beta). docs.katalon.com
Partial. Accessibility / performance / SEO checks. axe-core accessibility via custom keyword, 'not officially supported'; performance and SEO not documented. docs.katalon.com
Yes. Visual regression. TestOps Visual Testing: pixel, layout (AI) and content comparison against baselines. docs.katalon.com
Not documented. Natural-language assertions. No LLM-judged natural-language assertions found; verifications are keywords or code. docs.katalon.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Chrome, Firefox, Edge and Safari locally and on TestCloud. docs.katalon.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. No AI/chatbot application testing found in docs index. docs.katalon.com
Not documented. Red teaming. Not documented. docs.katalon.com
Not documented. LLM eval suites. Not documented. docs.katalon.com
Not documented. RAG evals. Not documented. docs.katalon.com
Not documented. MCP server tests. Katalon MCP servers connect AI clients for authoring; testing MCP servers not documented. katalon.com
Not documented. A2A agent tests. Not documented. docs.katalon.com

Models

Yes. Bring your own LLM / key. StudioAssist: OpenAI, Azure OpenAI, Gemini, Bedrock or OpenAI-compatible key; platform BYO AI Enterprise-only. docs.katalon.com
Not documented. Local LLM (Ollama). OpenAI-compatible base URL is configurable; local models or Ollama not documented. docs.katalon.com

Workflow

Yes. CI gate. Runtime Engine add-on runs tests in Jenkins, GitHub Actions, Azure DevOps, GitLab, Docker. katalon.com
Yes. Failure triage + issue filing. Submit failures to Jira; TestOps AI drafts defects from failure evidence for review. katalon.com
Yes. AI root cause. AI failure analysis adds the failure reason to HTML, PDF and email reports. docs.katalon.com
Yes. Scheduling / monitoring. Scheduled test runs in TestOps; scheduled suites in Cloud Studio (beta). docs.katalon.com
Yes. Team roles (RBAC). Roles and permissions in platform admin; RBAC and audit logs listed for Enterprise. docs.katalon.com

mabl · Commercial platform

Low-code, agentic test automation for web, mobile, API, performance and accessibility; quote-based pricing. mabl.com

License, hosting and pricing

No. Open-source license. Proprietary. mabl.com
Partial. Self-hostable. SaaS; local and CI runs via CLI are free. mabl.com
No. Pricing model. sales: custom quote; 500 cloud-run credits/month baseline; 14-day trial. mabl.com

Testing web apps

Partial. Plain-English UI tests. Agents author low-code tests from requirements, flows or exploration. mabl.com
Yes. No selectors / self-healing. AI auto-healing keeps tests current as the app changes. mabl.com
Yes. API tests. API testing; import Postman collections. mabl.com
Partial. Accessibility / performance / SEO checks. Accessibility and performance testing included; SEO not documented. mabl.com
Partial. Visual regression. AI visual assertions; pixel-baseline diffing not confirmed in sources checked. help.mabl.com
Yes. Natural-language assertions. Visual (formerly GenAI) assertions from plain-language criteria; cloud runs only. help.mabl.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Chrome, Edge, Firefox, Safari (WebKit) in cloud. help.mabl.com

Testing AI products

Partial. Conversation / agent-to-agent tests. Intent-based assertions on chatbot/LLM output; no multi-turn simulator documented. mabl.com
Partial. Red teaming. Validates guardrails and content policies; no attack library documented. mabl.com
Partial. LLM eval suites. Criteria-based checks of LLM output inside tests; no dataset eval suites. mabl.com
Not documented. RAG evals. Not mentioned. mabl.com
Not documented. MCP server tests. mabl MCP server is for authoring; testing MCP servers not documented. mabl.com
Not documented. A2A agent tests. Not documented. mabl.com

Models

Not documented. Bring your own LLM / key. Not documented. mabl.com
Not documented. Local LLM (Ollama). Not documented. mabl.com

Workflow

Yes. CI gate. GitHub Actions, CircleCI, Jenkins, Bitbucket, GitLab integrations. mabl.com
Yes. Failure triage + issue filing. Create Jira issues from runs; Auto TFA pre-fills (Advanced AI add-on). help.mabl.com
Yes. AI root cause. Agent failure analysis with suggested failure reason. help.mabl.com
Yes. Scheduling / monitoring. Plan triggers: timer or schedule (cloud runs); auto re-run on failure. help.mabl.com
Yes. Team roles (RBAC). Owner, Editor and Viewer workspace roles. help.mabl.com

Momentic · Commercial platform

AI-native YAML tests in your repo with NL steps/assertions, auto-heal and triage; plus 'Mo' AI QA agent. momentic.ai

License, hosting and pricing

No. Open-source license. Proprietary; tests are YAML files in your repository. momentic.ai
Partial. Self-hostable. Enterprise runs execute inside your network; result viewer self-hostable; control plane is SaaS. momentic.ai
Partial. Pricing model. usage credits: Free 2,000/mo; Pay-as-you-go $125/mo; Enterprise test-based; no seat fees. momentic.ai

Testing web apps

Yes. Plain-English UI tests. AI act steps pursue natural-language goals; NL element descriptions. momentic.ai
Yes. No selectors / self-healing. NL element targets; locator auto-heal, permanent healing, quarantine. momentic.ai
Partial. API tests. request and graphqlRequest steps inside tests; not a standalone API suite. momentic.ai
Not documented. Accessibility / performance / SEO checks. No accessibility/performance/SEO checks in docs index. momentic.ai
Yes. Visual regression. visualDiff against golden files (viewport or element). momentic.ai
Yes. Natural-language assertions. AI assert (DOM, accessibility tree, screenshot) and assertVisually. momentic.ai
No. Multi-browser (Chromium/Firefox/WebKit). Chromium-family only (Chromium, Chrome, Chrome for Testing). momentic.ai

Testing AI products

Not documented. Conversation / agent-to-agent tests. Not documented. momentic.ai
Not documented. Red teaming. Not documented. momentic.ai
Not documented. LLM eval suites. Not documented. momentic.ai
Not documented. RAG evals. Not documented. momentic.ai
Not documented. MCP server tests. Mo consumes MCP servers for context; testing them not documented. momentic.ai
Not documented. A2A agent tests. Not documented. momentic.ai

Models

Not documented. Bring your own LLM / key. Momentic routes to its hosted providers; BYO key not documented. momentic.ai
Not documented. Local LLM (Ollama). Not documented. momentic.ai

Workflow

Yes. CI gate. GitHub Actions with sharding; PR check and deploy-gate setups. momentic.ai
Yes. Failure triage + issue filing. Failure classification and triage; Mo files bugs to Linear. momentic.ai
Yes. AI root cause. Classification and triage explain failures and decide whether to heal permanently. momentic.ai
Yes. Scheduling / monitoring. Scheduled nightly regression suites; Slack notifications. momentic.ai
Yes. Team roles (RBAC). Team members and roles; RBAC, SAML SSO, SCIM on Enterprise. momentic.ai

QA Wolf · Commercial platform

Agents (plus optional managed QA engineers) build and maintain Playwright/Appium suites; pay per test run. qawolf.com

License, hosting and pricing

No. Open-source license. Proprietary platform; generated tests are open-source Playwright/Appium code; CLI is open source. qawolf.com
Partial. Self-hostable. SaaS platform; CLI runs your flows locally or in CI. docs.qawolf.com
Partial. Pricing model. usage: $1/web run, $2 mobile/desktop; no seats; managed service by sales. qawolf.com

Testing web apps

Yes. Plain-English UI tests. Request flows in natural language; agents write Playwright/Appium code. qawolf.com
Partial. No selectors / self-healing. Output is selector-based code; agents diagnose and repair broken tests. qawolf.com
Not documented. API tests. No dedicated API-testing docs found. docs.qawolf.com
Partial. Accessibility / performance / SEO checks. axe-core accessibility and Lighthouse performance guides; SEO not documented. docs.qawolf.com
Yes. Visual regression. toHaveScreenshot baselines on web and native mobile. docs.qawolf.com
Yes. Natural-language assertions. Semantic assertions (toSatisfy) and judge-model assertions on AI output. docs.qawolf.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Chrome, Firefox and WebKit runs; real iOS devices, Android, Windows. qawolf.com

Testing AI products

Partial. Conversation / agent-to-agent tests. Non-deterministic AI and voice-agent assertions; no persona simulator documented. docs.qawolf.com
Not documented. Red teaming. Not documented. docs.qawolf.com
Partial. LLM eval suites. Judge-model verdicts on AI output inside flows; no dataset eval suites. docs.qawolf.com
Not documented. RAG evals. Not documented. docs.qawolf.com
Partial. MCP server tests. Homepage lists validating MCP tool calls; no dedicated docs found. qawolf.com
Not documented. A2A agent tests. Not documented. docs.qawolf.com

Models

Not documented. Bring your own LLM / key. AI included ('no AI credits or token bills'); BYO key not documented. qawolf.com
Not documented. Local LLM (Ollama). Not documented. docs.qawolf.com

Workflow

Yes. CI gate. CI SDK waits for QA Wolf's verdict; GitHub status checks on PRs. docs.qawolf.com
Yes. Failure triage + issue filing. Failures reproduced by agent; bugs filed and synced to Jira or Linear. docs.qawolf.com
Yes. AI root cause. Agents investigate failures and fix outdated tests; humans in managed tier. qawolf.com
Yes. Scheduling / monitoring. Triggers run flows on deploy or on a schedule. docs.qawolf.com
Partial. Team roles (RBAC). SSO (SAML/OIDC) documented; no per-seat limits; role model not found. docs.qawolf.com

Testim (Tricentis) · Commercial platform

Recorder-based low-code web, mobile and Salesforce tests with AI smart locators; Copilot writes custom JS steps; agentic NL authoring for Salesforce. testim.io

License, hosting and pricing

No. Open-source license. Proprietary (Tricentis). testim.io
Partial. Self-hostable. SaaS authoring; tests run locally, on any Selenium grid, or a VPC grid option. testim.io
Partial. Pricing model. free Community plan after trial; Essentials/Pro/Mobile by sales quote; unlimited runs, parallel-based. testim.io

Testing web apps

Partial. Plain-English UI tests. Recorder-based low-code; Agentic Test Automation builds tests from natural language for Salesforce only. docs.tricentis.com
Yes. No selectors / self-healing. AI/ML smart locators self-heal to keep tests working as the app changes. testim.io
Yes. API tests. Add API action and Validate API steps (header, body, status); Pro plan. docs.tricentis.com
Partial. Accessibility / performance / SEO checks. axe-core page/element accessibility validations (Pro; Chrome/Edge); performance and SEO not documented. docs.tricentis.com
Partial. Visual regression. Element, viewport and full-page visual validation via Applitools Eyes integration; Pro plan. docs.tricentis.com
Partial. Natural-language assertions. Copilot turns a description into JavaScript custom-step code; no LLM-judged assertions found. tricentis.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Chrome, Firefox, Safari, Edge on Testim grid or Selenium/third-party grids; local Chrome only. docs.tricentis.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. No AI/chatbot application testing found. tricentis.com
Not documented. Red teaming. Not documented. tricentis.com
Not documented. LLM eval suites. Not documented. tricentis.com
Not documented. RAG evals. Not documented. tricentis.com
Not documented. MCP server tests. Not documented. tricentis.com
Not documented. A2A agent tests. Not documented. tricentis.com

Models

Not documented. Bring your own LLM / key. Copilot uses OpenAI via Tricentis; bringing your own key not documented. help.testim.io
Not documented. Local LLM (Ollama). Not documented. help.testim.io

Workflow

Yes. CI gate. Testim CLI with generated CI commands and JUnit XML report for CI display. docs.tricentis.com
Yes. Failure triage + issue filing. One-button bug reports to Jira, Trello or GitHub; error aggregation across runs. tricentis.com
Partial. AI root cause. RCA with highlighted screenshots, console/network logs and failure suggestions; AI cause analysis not described. testim.io
Yes. Scheduling / monitoring. Scheduler runs labeled tests or suites; email, Slack and webhook notifications. docs.tricentis.com
Partial. Team roles (RBAC). Project owners and teammates only; teammates can view, edit and run all tests. docs.tricentis.com

testRigor · Commercial platform

Plain-English test automation across web, mobile, desktop and API, with AI-based checks and axe accessibility. testrigor.com

License, hosting and pricing

No. Open-source license. Proprietary. Free tier makes tests public, not open-source software. testrigor.com
Not documented. Self-hostable. On-prem not mentioned on site pages checked. testrigor.com
Partial. Pricing model. free Public plan (tests public); Private Complete trial; Enterprise custom; no list prices. testrigor.com

Testing web apps

Yes. Plain-English UI tests. 'free-flowing plain English to build test automation'. testrigor.com
Yes. No selectors / self-healing. Identifies elements as seen on screen, no XPath; AI self-healing page. testrigor.com
Yes. API tests. Invoke APIs, validate return codes, store results. testrigor.com
Partial. Accessibility / performance / SEO checks. Accessibility via Deque axe per page; performance/SEO not mentioned. testrigor.com
Partial. Visual regression. Image comparison steps mentioned; baseline workflow not detailed. testrigor.com
Yes. Natural-language assertions. 'check that ... using ai' sentiment, semantic and vision checks. testrigor.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Cross-browser and cross-platform with a single test. testrigor.com

Testing AI products

Partial. Conversation / agent-to-agent tests. AI checks on chatbot replies in UI; no multi-turn simulator. testrigor.com
Not documented. Red teaming. Not mentioned. testrigor.com
Not documented. LLM eval suites. Only UI-level AI validations; no dataset eval suites. testrigor.com
Not documented. RAG evals. Not mentioned. testrigor.com
Not documented. MCP server tests. Offers MCP server for authoring; testing MCP servers not documented. testrigor.com
Not documented. A2A agent tests. Not documented. testrigor.com

Models

Not documented. Bring your own LLM / key. Underlying LLM not stated. testrigor.com
Not documented. Local LLM (Ollama). Not documented. testrigor.com

Workflow

Yes. CI gate. GitHub Actions, GitLab, Jenkins, Azure DevOps, CircleCI integrations. testrigor.com
Partial. Failure triage + issue filing. Jira, Zephyr, TestRail integrations listed; auto-filing not described. testrigor.com
Not documented. AI root cause. Not documented. testrigor.com
Yes. Scheduling / monitoring. Scheduled suite runs via monitoring, weekly to every four hours. testrigor.com
Partial. Team roles (RBAC). Teams and company admins; Enterprise plans only (guide may be dated). testrigor.com

Testsigma · Commercial platform

Codeless plain-English (NLP) automation for web, mobile, API, desktop and Salesforce with AI agents; Apache-2.0 Community Edition unmaintained since 2023. testsigma.com

License, hosting and pricing

No. Open-source license. Cloud platform proprietary; legacy Apache-2.0 Community Edition repo, last release v3.0.1 (Aug 2023). github.com
Partial. Self-hostable. On-prem docker-compose deployment on Enterprise; old OSS self-host docs now redirect away. testsigma.com
No. Pricing model. sales: Pro and Enterprise by custom quote; no public prices or free tier. testsigma.com

Testing web apps

Yes. Plain-English UI tests. Steps are plain-English NLP actions; Generator agent turns prompts and requirements into tests. testsigma.com
Yes. No selectors / self-healing. Locator-based elements; Healer agent and GenAI autoheal repair locators mid-run for approval. testsigma.com
Yes. API tests. REST and SOAP API steps with response verifications; tests generated from Swagger/WSDL. testsigma.com
Partial. Accessibility / performance / SEO checks. Per-step WCAG accessibility scans (Enterprise plan); performance and SEO not documented. testsigma.com
Yes. Visual regression. Screenshot baselines or Figma frames per step; comparison modes and ignore regions. testsigma.com
Partial. Natural-language assertions. 'Verify' steps are plain-English NLP templates bound to elements; no LLM-judged assertions found. testsigma.com
Yes. Multi-browser (Chromium/Firefox/WebKit). 800+ browser/OS combinations in cloud lab; Chrome, Firefox, Edge capabilities documented. testsigma.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. No chatbot or AI-app conversation testing found; agents target conventional apps. testsigma.com
Not documented. Red teaming. Not documented. testsigma.com
Not documented. LLM eval suites. Not documented. testsigma.com
Not documented. RAG evals. Not documented. testsigma.com
Not documented. MCP server tests. Arcus MCP server lets AI clients draft tests; testing MCP servers not documented. testsigma.com
Not documented. A2A agent tests. Not documented. testsigma.com

Models

Yes. Bring your own LLM / key. BYOK for Azure OpenAI, OpenAI, Gemini and Vertex AI; per-feature model mapping. testsigma.com
Not documented. Local LLM (Ollama). BYOK lists only cloud providers (Azure OpenAI, OpenAI, Gemini, Vertex AI). testsigma.com

Workflow

Yes. CI gate. GitHub Actions, Jenkins, or shell script that triggers a plan, waits, saves JUnit. testsigma.com
Yes. Failure triage + issue filing. Bug Reporter files analyzed failures to Jira, Linear, Bugzilla, Trello and others. testsigma.com
Yes. AI root cause. Analyzer agent: error type, root cause, visual evidence, fix suggestions with Apply Fix. testsigma.com
Yes. Scheduling / monitoring. Scheduled test plans via UI or REST API; parallel runs. testsigma.com
Yes. Team roles (RBAC). Six per-project roles from Account Administrator to Read Only; audit logs. testsigma.com

Octomind (likely defunct) · Commercial platform

AI agents generated and auto-fixed Playwright E2E tests. octomind.dev unresolvable 2026-10-21; all GitHub repos archived July 2026. octomind.dev

License, hosting and pricing

No. Open-source license. Proprietary SaaS; Debugtopus and CLI helper tools were open source (now archived). github.com
Partial. Self-hostable. SaaS; Debugtopus ran tests locally; private-location worker only tunneled. raw.githubusercontent.com
Not documented. Pricing model. Pricing page offline; domain had no DNS records on 2026-10-21. octomind.dev

Testing web apps

Yes. Plain-English UI tests. Prompt the AI agent in natural language to generate a test. raw.githubusercontent.com
Partial. No selectors / self-healing. Generated deterministic Playwright code; AI auto-fix (beta) proposed fixes for review. raw.githubusercontent.com
Not documented. API tests. Not documented in archived docs. github.com
Not documented. Accessibility / performance / SEO checks. Not documented in archived docs. github.com
Not documented. Visual regression. Not documented in archived docs. github.com
Not documented. Natural-language assertions. Assertions were visual-editor types (contain text, style). raw.githubusercontent.com
Yes. Multi-browser (Chromium/Firefox/WebKit). Scheduled runs on Chromium, Firefox or Safari. raw.githubusercontent.com

Testing AI products

Not documented. Conversation / agent-to-agent tests. Not documented. github.com
Not documented. Red teaming. Not documented. github.com
Not documented. LLM eval suites. Not documented. github.com
Not documented. RAG evals. Not documented. github.com
Not documented. MCP server tests. Had an MCP server for authoring; MCP testing not documented. github.com
Not documented. A2A agent tests. Not documented. github.com

Models

Not documented. Bring your own LLM / key. Not documented. github.com
Not documented. Local LLM (Ollama). Not documented. github.com

Workflow

Yes. CI gate. GitHub and Azure DevOps native; Jenkins and Vercel via API. raw.githubusercontent.com
Not documented. Failure triage + issue filing. Test-management sync (Xray, Zephyr) only; no issue filing found. github.com
Partial. AI root cause. Auto-fix analyzed failure root cause; RCA playbook 'under active development'. raw.githubusercontent.com
Yes. Scheduling / monitoring. Scheduled runs with browser and screen-size selection. raw.githubusercontent.com
Not documented. Team roles (RBAC). Not documented in archived docs. github.com

Propose a correction

Corrections are welcome, from vendors too. A change needs a public source: a documentation page, pricing page or repository file that shows the feature.

Edit the product’s cell in docs/comparison.json. Each cell has a value (yes, partial, no or not_documented), a short note, the source URL that supports it, and the checked date. Plainwright’s own cells use repo:<path> sources.
Regenerate the Markdown table and the README summary from the repository root:
Terminal window
```
python scripts/build_comparison.py
```
Refresh the landing page’s copy and check it, from frontend/:
Terminal window
```
UPDATE_COMPARISON=1 npx vitest run src/data/comparison.test.js
```
On PowerShell, set $env:UPDATE_COMPARISON = '1' first, then run npx vitest run src/data/comparison.test.js.
Check this page with npm run build in docs-site/, then open a pull request that says what changed and links the source. See Contributing.

If you’d rather not edit JSON, open an issue with the product, the feature, what’s wrong and a link that shows it.