AI agent email tests combine two sources of uncertainty: model behavior and asynchronous delivery. A test can fail because a prompt changed, a mailbox received old mail, a parser chose the wrong message, or delivery simply took longer than the test timeout. Repeating until green hides these distinctions and makes releases harder to trust.
A better architecture makes mailbox ownership, state transitions, and external effects deterministic. Relay supplies temporary inboxes and message access through REST and MCP; your test harness supplies isolation, controlled inputs, assertions, and evidence.
Isolate every run and assign explicit ownership
Give each test run a unique inbox and a run ID recorded by both the harness and agent. Do not share a catch-all inbox across parallel workers unless every consumer can prove ownership of each message. Parallel runs otherwise create races that look like model failures.
Relay provides temporary inboxes on separate Free mail domains and on BYOD domains. Choose the domain policy appropriate to the scenario using the domains documentation; relaymcp.net is the website and does not itself issue mailbox addresses. Register the inbox ID before triggering any external workflow.
An ownership rule should require an exact inbox match and can add a run-specific correlation value in the outbound message or recipient address where the tested system supports it. Do not use subject alone as a lock: retries and external systems can duplicate it. If unexpected mail arrives, quarantine it from the run rather than letting the agent improvise.
Make the agent harness deterministic
Freeze model configuration, prompts, tool schemas, parser versions, and policy rules for a given test report. Where possible, separate model planning from deterministic execution and assert the plan's structure instead of exact prose. Use recorded tool responses for unit tests, controlled MCP interactions for integration tests, and a smaller set of live delivery tests for end-to-end confidence.
A state-based assertion is more stable than checking that the agent 'said' it finished. For example, require selectedMessageId to belong to run.inboxId and require at most one approved external effect.
function assertCompletedRun(run, effects) {
if (run.state !== 'completed') throw new Error('Run did not complete');
if (run.selectedMessageId == null) throw new Error('No message selected');
if (run.inboxId !== run.expectedInboxId) throw new Error('Inbox ownership changed');
if (effects.length !== 1) throw new Error(`Expected one effect, got ${effects.length}`);
if (effects[0].runId !== run.id) throw new Error('Effect ownership mismatch');
}
Control time where your code allows it. Inject clocks for expiration and retry tests instead of waiting through real backoff. Preserve a real timeout in the small live suite so delivery and network failures are still exercised.
Distinguish real delivery from capture
SMTP capture answers whether your application attempted to send a message and what bytes it generated. Real delivery through Relay additionally exercises DNS, transport, mailbox acceptance, and subsequent retrieval. Neither replaces the other.
Use capture for high-volume template, localization, and header tests. Use real temporary inboxes for workflows that depend on actual receipt, such as verification and reply handling. Do not claim deliverability to third-party providers from a successful Relay test; that would require separate evidence and provider-specific analysis.
| Layer | Primary question | Typical oracle |
|---|---|---|
| Unit | Does parsing or policy work? | Fixed fixtures and expected decisions |
| Tool integration | Does the agent call MCP correctly? | Tool arguments and state transitions |
| SMTP capture | Was intended mail generated? | Envelope, headers, and body |
| Live Relay | Was mail received and retrievable? | Message identity and workflow effect |
Poll with a bounded deadline and record elapsed time. Avoid fixed sleeps, which are slow on success and fragile on delay.
Protect data and contain external effects
Use synthetic identities and purpose-built fixtures whenever possible. Real customer mail introduces retention, access, and compliance obligations and can contain secrets unrelated to the test. Redact bodies, addresses, tokens, and authorization headers from CI logs; store a message ID and content hash when you need correlation without content.
Relay API keys must remain server-side secrets in environment variables or a CI secret store. Scope them to the required operation, rotate them, and revoke unused keys through your account. Do not print MCP initialization configuration in debug output.
External side effects need equal containment. Point ticket creation, payments, and account registration to test projects or sandboxes. Require idempotency keys with the run ID so a retried agent cannot create duplicates. For richer content boundaries, apply the same untrusted-mail rules described in the MCP agent guide.
Measure reliability by failure category
A single pass rate is not enough. Track delivery latency, time to message selection, wrong-message selections, extraction ambiguity, tool errors, policy denials, retries, duplicate effects, and cleanup failures. Tag each result with parser version, model configuration, domain type, and test layer so regressions can be localized.
Classify failures before retrying. A transport timeout is different from a wrong OTP, and a policy denial may be a successful safety assertion. Quarantine known flaky tests only with an owner and expiry; otherwise quarantine becomes a permanent hiding place.
Keep local test artifacts for a bounded period and make their cleanup part of the harness. A fresh inbox per run provides isolation without assuming that prior Relay mail has been removed. Review the privacy policy and support guidance for source-data retention separately from artifact handling.
Frequently asked questions
Should every agent test use live email delivery?
No. Use fixtures and SMTP capture for fast, deterministic coverage, then run a focused live suite against Relay for receipt and retrieval behavior. This layered approach reduces flaky tests while preserving end-to-end evidence.
Can parallel tests share one temporary inbox?
They can technically read the same inbox, but ownership becomes difficult and message selection can race. Prefer one inbox per run. If sharing is unavoidable, enforce deterministic partitioning and reject messages without an exact ownership marker.
What is the best assertion for an AI email agent?
Assert durable state and authorized effects rather than conversational wording. Verify inbox and message ownership, allowed tool calls, extraction status, approval records, and exactly the expected external side effects.
Test the system, not just the model output
Reliable agent email testing combines fresh inbox ownership, frozen configurations, layered delivery checks, explicit state assertions, and contained credentials and effects. Metrics should explain where time and failures occur, while redacted artifacts preserve privacy. Build the fast suite around deterministic fixtures and reserve live Relay runs for behavior that truly needs delivery. Explore Relay's documentation to add isolated mailbox coverage without making your CI pipeline depend on shared mail or lucky timing.