Shadow-test a new AI email agent on live threads with Nylas

A developer from Nylas demonstrates how to shadow-test an AI email agent on live threads using Nylas Agent Accounts, ensuring the candidate agent records its proposed actions without sending or modifying customer mailboxes. The approach involves persisting shadow events and runs in a database, comparing candidate outputs against trusted outcomes, and handling webhook redeliveries safely.

You have an email agent that works in a test inbox. It classifies a support request, pulls the relevant context, and drafts a plausible reply. The risky next step is turning it loose on support@yourcompany.com : a new prompt can misunderstand a frustrated customer, a new model can change its format, and a harmless-looking tool change can make the agent write twice. There is a useful stage between “it passed the fixture” and “it can send customer email”: shadow mode . The candidate agent receives the same live inbound messages as production, reads the same canonical thread context, and records what it would have done. It never sends, moves, labels, or creates a draft in the customer mailbox. A person or the existing production workflow still owns the outcome. This post builds that boundary around a Nylas Agent Account. Nylas delivers the event and exposes the message/thread data; your application persists the candidate version, proposed action, and comparison result. That division matters: shadow mode is an application rollout feature, not an email-provider switch. I work on the Nylas CLI, so I use it to inspect the live plumbing and send test events. The implementation below is deliberately database-shaped rather than tied to a queue or model provider. A shadow run consumes real traffic but produces no customer-visible side effect. For each inbound message, it can return a structured proposal such as: { "action": "reply", "reason": "The customer asked for a password-reset link.", "body": "Hi Dana, here is a fresh password-reset link…", "confidence": 0.91 } That record is useful only when you compare it with an outcome you trust: the production agent's action, a human-approved reply, a support label, or an explicit reviewer verdict. A candidate that produces polished prose but would have answered a billing question instead of escalating it has not passed. Shadow mode is not a privacy exemption. The model sees live customer content, so run it only where you already have authority to process that content, minimize retention, and exclude attachments and tools unless the experiment specifically needs them. “No send” is an important safety property; it is not the whole threat model. This is a spoke, not a second webhook tutorial. The durable ingest pattern is in Build a webhook-driven email pipeline for your AI agent https://dev.to/mqasimca/build-a-webhook-driven-email-pipeline-for-your-ai-agent-211m : verify the raw-body signature, acknowledge quickly, persist a deduplicated job, and do model work in a worker. Subscribe to message.created as that guide shows. Nylas delivers webhooks at least once, so a redelivery must not create another shadow run. The top-level notification id is the delivery key; the inner data.object.id is the message key. Keep both, because they answer different questions. Do not create a Nylas draft for the candidate reply. Drafts are visible mailbox state, which means a human can mistake an experiment for a proposed response and send it. Shadow output belongs in your own database beside the rollout decision. This small schema is enough to start: create table shadow events notification id text primary key, grant id text not null, message id text not null, thread id text, received at timestamptz not null default now ; create table shadow runs notification id text not null references shadow events, candidate version text not null, proposed action jsonb, input hash text, input snapshot jsonb, status text not null default 'pending', outcome text not null default 'pending', reviewer verdict text, actual message id text, started at timestamptz, created at timestamptz not null default now , primary key notification id, candidate version ; In the webhook transaction, insert the event and its pending run before enqueueing it. ON CONFLICT DO NOTHING makes a redelivery a no-op: with event as insert into shadow events notification id, grant id, message id, thread id values $1, $2, $3, $4 on conflict notification id do update set notification id = excluded.notification id returning notification id insert into shadow runs notification id, candidate version select notification id, $5 from event on conflict do nothing returning notification id; Enqueue only when that statement returns a row. The first table makes webhook ingest idempotent. The second lets a deliberate backfill run another candidate version without pretending that prompt version A and model version B are the same experiment. This example retains an encrypted, access-controlled input snapshot for 30 days so a reviewer can replay a candidate; delete it after that window and retain only the hash and labels. After deletion, replay is intentionally impossible. The webhook tells you what changed. Fetch the message before you ask the candidate to reason about it, and use thread id to assemble the conversation context you actually need. The Nylas Message API returns a message body; the Messages list supports filtering by thread id . curl --request GET \ --url "https://api.us.nylas.com/v3/grants/<GRANT ID /messages/<MESSAGE ID " \ --header "Authorization: Bearer <NYLAS API KEY " curl --request GET \ --url "https://api.us.nylas.com/v3/grants/<GRANT ID /messages?thread id=<THREAD ID " \ --header "Authorization: Bearer <NYLAS API KEY " nylas email read <message-id nylas email threads show <thread-id Filter out messages the Agent Account sent itself before creating a run. message.created covers sent and received messages, and an experiment that scores its own output as fresh inbound mail is not measuring customer traffic. The worker below is illustrative pseudocode. First claim the already-created run; only the worker that changes pending to running may call the candidate. The important part is the absence of messages/send and draft creation: js async function shadowMessage job, candidate { const claimed = await db.query update shadow runs set status = 'running', started at = now where notification id = $1 and candidate version = $2 and status = 'pending' returning notification id , job.notificationId, candidate.version ; if claimed.rowCount return; const message = await nylas.messages.get job.grantId, job.messageId ; if message.from 0 ?.email === AGENT ADDRESS { await db.query update shadow runs set status = 'ignored' where notification id = $1 and candidate version = $2 , job.notificationId, candidate.version ; return; } const messages = await nylas.messages.list { grantId: job.grantId, threadId: job.threadId, } ; const proposal = await candidate.propose { message, thread: messages.data, tools: , // shadow mode cannot cause external side effects attachments: , // add only after a separate attachment review } ; await db.query update shadow runs set status = 'complete', input hash = $3, input snapshot = $4, proposed action = $5 where notification id = $1 and candidate version = $2 , job.notificationId, candidate.version, hash message, messages.data , JSON.stringify redactForEvaluation message, messages.data , JSON.stringify proposal , ; } On a worker failure, mark the run failed and deliberately reset it to pending before retrying; alert on stale running claims. Do not let a second delivery run beside the first one. The candidate should return a constrained object, not free-form internal reasoning. For a support agent, reply , escalate , ignore , and request information are usually enough. Put the proposed body in the object only when a reviewer needs to evaluate wording; otherwise store an action and a compact rationale. The comparison should match the decision you intend to automate. If the rollout target is routing, grade routing; do not let an eloquent draft hide a wrong category. If the target is reply generation, compare the candidate draft to a human-approved reply and give reviewers a way to mark it safe, wrong, incomplete, or unsafe. Useful initial checks are deliberately boring: | Check | What it catches | |---|---| | Action agreement | Candidate replied when a human escalated, or vice versa. | | Unsafe-action rate | Candidate proposed sending, disclosing, or changing something outside its allowed scope. | | Duplicate-run rate | Redelivered notifications or worker retries created more than one candidate result. | | Time to proposal | A candidate that needs minutes is not ready for a real-time reply loop. | | Reviewer override reason | The missing policy, tool, or context that the next version needs. | Do not compare raw generated text with string equality. Two safe replies can use different wording. Start with a reviewer label, then add task-specific checks. For example, confirm that an escalation preserved the thread ID, that a refund case avoided a promise, or that a support reply cited the current policy. Make the outcome link explicit. When the existing workflow sends or a reviewer decides, write that decision against the same notification and candidate version: update shadow runs set outcome = $3, actual message id = $4, reviewer verdict = $5 where notification id = $1 and candidate version = $2; For a human reply, $4 is the sent message ID; for a routing-only experiment it can be null and $3 is the final queue or escalation action. That gives the review UI a concrete pair: the candidate proposal beside what actually happened. Shadow mode earns trust gradually. A practical rollout has three stages: The category boundary should be narrow enough to explain in one sentence. “Reset-password questions that match a known account and need no account change” is a rollout boundary. “Support email” is not. Nylas can send a test webhook to an endpoint, and the CLI can display a mock payload. Test that the ingest path accepts one notification, records one shadow events row, and never calls the send or drafts path. nylas webhook test payload message.created nylas webhook test send https://agent.example.com/webhooks/nylas Then replay the same payload. Assuming your existing append-only audit log records attempted Nylas operations, this is the smallest database check for the boundary: select select count from shadow runs where notification id = '<NOTIFICATION ID ' and candidate version = '<VERSION ' = 1 as one candidate run, not exists select 1 from audit log where notification id = '<NOTIFICATION ID ' and operation in 'messages.send', 'drafts.create' as no mailbox mutation; Both values must be true. This catches duplicate evaluation before it becomes accidental duplicate behavior when you turn the candidate on. For a brand-new mailbox with no business risk, an ephemeral Agent Account is the faster test. For a production workflow where the difference between a good and bad reply matters, shadow mode is the bridge you want. It lets you observe a candidate on the messy mix of real threads, reply styles, missing information, and edge cases that fixtures never reproduce without making the customer your canary. The durable rule is simple: Nylas carries the message and thread; your application records the experiment; the candidate proposes; a human or the existing workflow acts. Only after those records show that the candidate handles a deliberately narrow category should it earn the right to send. Where to go next: