A2A agents

An A2A test checks an agent that speaks the Agent2Agent (A2A) protocol. Plainwright fetches the agent’s agent card, validates it against the spec, and then either sends one smoke-test message (no model tokens) or runs a full conversation test with a simulated user, judged against your success criteria.

Use it to make sure other agents can discover and call yours: a broken card, a wrong endpoint or a task that never completes all show up here before a partner integration hits them.

How an A2A test runs

1. Find the agent card. Plainwright tries the well-known locations for agent_url (see Card discovery) and keeps the first JSON object it finds.

2. Validate the card. Required fields, types and transports are checked. Errors fail the test (unless strict_card is off); suggestions never do.

3. Find the JSON-RPC endpoint from the card.

4. Talk to the agent. With a persona, goal or success criteria, a simulated user holds a conversation over message/send and the judge grades it. Otherwise one smoke message is sent and the reply must be a valid Message or Task with text.

Create an A2A test

1. On the project board, choose Add test in a module, pick A2A agent and press Continue.

2. Under Configure, enter the Agent URL (the agent’s base URL, or the full URL of its card), any Headers it needs, and optionally a Reply timeout.

3. Under What to test, choose Smoke check (optionally with your own message) or Conversation (persona, goal, first message, max turns and success criteria).

4. Press Continue, then Create & run.

Configuration

Key	Type	Default	What it does
agent_url	string	required	The agent’s base URL, or the URL of its card (ending in .json). Secrets work here.
headers	object	{}	Sent with every request, for example {"Authorization": "Bearer {{secrets.AGENT_TOKEN}}"}.
strict_card	boolean	true	Card errors fail the test. With false they are reported only.
timeout_s	integer	60	Seconds to wait for each request, 1–300.
smoke_message	string	Hello! In one or two sentences, what can you help me with?	The smoke-check message.
persona, goal, success_criteria, opening_message, max_turns, threshold			Setting any of persona, goal or success_criteria turns the test into a conversation. They work exactly as in a conversation test.

Card discovery

For https://agent.example.com/support, Plainwright tries, in order, and keeps the first 200 response that is a JSON object:

1. https://agent.example.com/support/.well-known/agent-card.json (A2A 0.3 and later)
2. https://agent.example.com/support/.well-known/agent.json (A2A 0.2 and earlier)
3. https://agent.example.com/.well-known/agent-card.json
4. https://agent.example.com/.well-known/agent.json

A URL ending in .json is fetched as is. Cards larger than 512 KB are rejected, and the lookup gives up on each location after 20 seconds. If nothing is found, the test fails with every URL tried and what each returned.

Card validation

Both A2A 0.2 and 0.3 field names are accepted. snake_case names such as default_input_modes are read too, with a warning that the spec uses camelCase.

Field	Error when	Warning when
name, description, version	missing or not a non-empty string
url	missing or not an http(s) URL, and no additionalInterfaces URL either	missing, but an additionalInterfaces entry has one
protocolVersion	present but not a string such as "0.3.0"	missing (the card is treated as 0.2)
preferredTransport	not a string, or not JSON-RPC while no JSON-RPC interface is listed
additionalInterfaces	not a list
capabilities	missing or not an object	streaming, pushNotifications or stateTransitionHistory is not true/false
defaultInputModes, defaultOutputModes	missing, empty, or not a list of strings
skills	missing or not a list	the list is empty
each skill	not an object; missing id, name or description; duplicate id; tags not a list of strings	no tags; examples, inputModes or outputModes not a list of strings
provider		no organization
documentationUrl, iconUrl		not an http(s) URL

The run’s scores.card is 1 for a card with no errors and 0 otherwise.

Which endpoint is called

Plainwright posts JSON-RPC to the card’s url when the preferred transport is JSON-RPC (or isn’t stated). Otherwise it uses the first additionalInterfaces entry whose transport is JSON-RPC, and as a last resort the agent URL itself.

If the card advertises a loopback address (localhost, 127.0.0.1, 0.0.0.0 or ::1) but you reached the agent somewhere else, the card’s path is kept and moved onto the host you gave. That makes a card generated with a development default still usable. The endpoint must pass the same outbound checks as the agent URL.

Smoke check

With no persona, goal or criteria, Plainwright sends one message (smoke_message, or the default greeting) and checks the reply. No model tokens are used. The test passes when the result:

• is an A2A Message or Task (anything else fails with neither a Message nor a Task),
• is not a task in the failed state, and
• contains some text.

Conversation over A2A

Give the test a persona, a goal or success criteria and it becomes a conversation test whose target is the agent. The simulated user, turn limit, judge and scoring are the same as in conversation tests, and the card validation still applies: with strict_card on, a card error fails the run even if the conversation passed.

How the messages travel:

Situation	What Plainwright does
Each turn	A JSON-RPC 2.0 message/send with a user message of one text part.
Agent answers with a Task that is submitted or working	Polls tasks/get every second, for up to 60 seconds, until it completes.
Agent answers with a task in input-required or auth-required	Reads the agent’s question and continues the same task on the next turn.
The conversation continues	Every turn reuses the contextId the agent returned, so the agent sees one conversation.
An A2A 0.1 agent answers method not found (-32601)	Switches to the older tasks/send method.
The agent replies with an event stream	Reads the last JSON-RPC result from the stream.

The reply text is taken from the task’s artifacts, then its status message, then the agent’s last message in the history. An interrupted task (input-required) shows its status message first.

Examples

Smoke check

a2a-smoke-test.json

{
  "module_id": "<module-id>",
  "name": "Helpdesk agent card and endpoint",
  "kind": "a2a",
  "natural_language": "The helpdesk agent publishes a valid card and answers over JSON-RPC.",
  "config": {
    "agent_url": "https://helpdesk-agent.example.com",
    "headers": { "Authorization": "Bearer {{secrets.AGENT_TOKEN}}" },
    "smoke_message": "What can you help me with?"
  }
}

Conversation

a2a-conversation-test.json

{
  "module_id": "<module-id>",
  "name": "Helpdesk agent handles a refund",
  "kind": "a2a",
  "natural_language": "The helpdesk agent refunds a duplicate charge and explains the timeline.",
  "config": {
    "agent_url": "https://helpdesk-agent.example.com",
    "headers": { "Authorization": "Bearer {{secrets.AGENT_TOKEN}}" },
    "timeout_s": 90,
    "persona": "A customer who was charged twice and writes short, polite messages.",
    "goal": "Get the duplicate charge for order #4411 refunded.",
    "opening_message": "Hi, I was charged twice for order #4411. Can one of them be refunded?",
    "success_criteria": [
      "The agent confirms the refund",
      "The agent gives a timeline for the refund"
    ],
    "max_turns": 4
  }
}

Create and run either as shown in Work from the API.

Read the result

The Agent card tab shows where the card was found, the JSON-RPC endpoint, the agent’s name, version, protocol version and transport, every validation error and suggestion, and the skills the card advertises.

For a conversation, the Transcript tab shows the judge’s verdict per criterion and every turn, as for any conversation test.

Status	When
Passed	The card has no errors (or strict_card is off) and the smoke reply was valid, or the conversation met every criterion.
Failed	No card was found, the card has errors with strict_card on, the endpoint isn’t allowed, the smoke message couldn’t be sent or its reply was invalid, failed or empty, or the conversation failed its criteria.
Error	No agent URL, an agent URL that isn’t allowed, or the conversation couldn’t run (for example the simulated user’s model or the judge failed).

A2A as a target elsewhere

Conversation and red-team tests accept {"type": "a2a", "agent_url": "…"} as their target. They use the card only to find the endpoint (falling back to the agent URL when there is none) and don’t validate it. Use the A2A test kind when the card itself matters.

Next steps

Conversation testsPersonas, goals, criteria and how the judge scores a transcript.

Red teamingAttack an A2A agent with the seven attack packs.

MCP serversCheck an MCP server's tool contract and exercise its tools.

Trials, thresholds and costRepeat conversations and read a pass rate.