{"slug": "shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas", "title": "Shadow-test a new AI email agent on live threads with Nylas", "summary": "A developer from Nylas demonstrates how to shadow-test an AI email agent on live threads using Nylas Agent Accounts, ensuring the candidate agent records its proposed actions without sending or modifying customer mailboxes. The approach involves persisting shadow events and runs in a database, comparing candidate outputs against trusted outcomes, and handling webhook redeliveries safely.", "body_md": "You have an email agent that works in a test inbox. It classifies a support request, pulls the relevant context, and drafts a plausible reply. The risky next step is turning it loose on `support@yourcompany.com`\n\n: a new prompt can misunderstand a frustrated customer, a new model can change its format, and a harmless-looking tool change can make the agent write twice.\n\nThere is a useful stage between “it passed the fixture” and “it can send customer email”: **shadow mode**. The candidate agent receives the same live inbound messages as production, reads the same canonical thread context, and records what it *would* have done. It never sends, moves, labels, or creates a draft in the customer mailbox. A person or the existing production workflow still owns the outcome.\n\nThis post builds that boundary around a Nylas Agent Account. Nylas delivers the event and exposes the message/thread data; your application persists the candidate version, proposed action, and comparison result. That division matters: shadow mode is an application rollout feature, not an email-provider switch.\n\nI work on the Nylas CLI, so I use it to inspect the live plumbing and send test events. The implementation below is deliberately database-shaped rather than tied to a queue or model provider.\n\nA shadow run consumes real traffic but produces no customer-visible side effect. For each inbound message, it can return a structured proposal such as:\n\n```\n{\n  \"action\": \"reply\",\n  \"reason\": \"The customer asked for a password-reset link.\",\n  \"body\": \"Hi Dana, here is a fresh password-reset link…\",\n  \"confidence\": 0.91\n}\n```\n\nThat record is useful only when you compare it with an outcome you trust: the production agent's action, a human-approved reply, a support label, or an explicit reviewer verdict. A candidate that produces polished prose but would have answered a billing question instead of escalating it has not passed.\n\nShadow mode is not a privacy exemption. The model sees live customer content, so run it only where you already have authority to process that content, minimize retention, and exclude attachments and tools unless the experiment specifically needs them. “No send” is an important safety property; it is not the whole threat model.\n\nThis is a spoke, not a second webhook tutorial. The durable ingest pattern is in [Build a webhook-driven email pipeline for your AI agent](https://dev.to/mqasimca/build-a-webhook-driven-email-pipeline-for-your-ai-agent-211m): verify the raw-body signature, acknowledge quickly, persist a deduplicated job, and do model work in a worker.\n\nSubscribe to `message.created`\n\nas that guide shows. Nylas delivers webhooks at least once, so a redelivery must not create another shadow run. The top-level notification `id`\n\nis the delivery key; the inner `data.object.id`\n\nis the message key. Keep both, because they answer different questions.\n\nDo not create a Nylas draft for the candidate reply. Drafts are visible mailbox state, which means a human can mistake an experiment for a proposed response and send it. Shadow output belongs in your own database beside the rollout decision.\n\nThis small schema is enough to start:\n\n```\ncreate table shadow_events (\n  notification_id text primary key,\n  grant_id text not null,\n  message_id text not null,\n  thread_id text,\n  received_at timestamptz not null default now()\n);\n\ncreate table shadow_runs (\n  notification_id text not null references shadow_events,\n  candidate_version text not null,\n  proposed_action jsonb,\n  input_hash text,\n  input_snapshot jsonb,\n  status text not null default 'pending',\n  outcome text not null default 'pending',\n  reviewer_verdict text,\n  actual_message_id text,\n  started_at timestamptz,\n  created_at timestamptz not null default now(),\n  primary key (notification_id, candidate_version)\n);\n```\n\nIn the webhook transaction, insert the event and its `pending`\n\nrun before enqueueing it. `ON CONFLICT DO NOTHING`\n\nmakes a redelivery a no-op:\n\n```\nwith event as (\n  insert into shadow_events (notification_id, grant_id, message_id, thread_id)\n  values ($1, $2, $3, $4)\n  on conflict (notification_id) do update\n    set notification_id = excluded.notification_id\n  returning notification_id\n)\ninsert into shadow_runs (notification_id, candidate_version)\nselect notification_id, $5 from event\non conflict do nothing\nreturning notification_id;\n```\n\nEnqueue only when that statement returns a row. The first table makes webhook ingest idempotent. The second lets a deliberate backfill run another candidate version without pretending that prompt version A and model version B are the same experiment. This example retains an encrypted, access-controlled input snapshot for 30 days so a reviewer can replay a candidate; delete it after that window and retain only the hash and labels. After deletion, replay is intentionally impossible.\n\nThe webhook tells you what changed. Fetch the message before you ask the candidate to reason about it, and use `thread_id`\n\nto assemble the conversation context you actually need. The Nylas Message API returns a message body; the Messages list supports filtering by `thread_id`\n\n.\n\n```\ncurl --request GET \\\n  --url \"https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages/<MESSAGE_ID>\" \\\n  --header \"Authorization: Bearer <NYLAS_API_KEY>\"\n\ncurl --request GET \\\n  --url \"https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages?thread_id=<THREAD_ID>\" \\\n  --header \"Authorization: Bearer <NYLAS_API_KEY>\"\nnylas email read <message-id>\nnylas email threads show <thread-id>\n```\n\nFilter out messages the Agent Account sent itself before creating a run. `message.created`\n\ncovers sent and received messages, and an experiment that scores its own output as fresh inbound mail is not measuring customer traffic.\n\nThe worker below is illustrative pseudocode. First claim the already-created run; only the worker that changes `pending`\n\nto `running`\n\nmay call the candidate. The important part is the absence of `messages/send`\n\nand draft creation:\n\n``` js\nasync function shadowMessage(job, candidate) {\n  const claimed = await db.query(`\n    update shadow_runs\n    set status = 'running', started_at = now()\n    where notification_id = $1 and candidate_version = $2 and status = 'pending'\n    returning notification_id\n  `, [job.notificationId, candidate.version]);\n  if (!claimed.rowCount) return;\n\n  const message = await nylas.messages.get(job.grantId, job.messageId);\n  if (message.from[0]?.email === AGENT_ADDRESS) {\n    await db.query(`update shadow_runs set status = 'ignored'\n                    where notification_id = $1 and candidate_version = $2`,\n      [job.notificationId, candidate.version]);\n    return;\n  }\n\n  const messages = await nylas.messages.list({\n    grantId: job.grantId,\n    threadId: job.threadId,\n  });\n  const proposal = await candidate.propose({\n    message,\n    thread: messages.data,\n    tools: [],              // shadow mode cannot cause external side effects\n    attachments: [],        // add only after a separate attachment review\n  });\n\n  await db.query(`\n    update shadow_runs\n    set status = 'complete', input_hash = $3, input_snapshot = $4,\n        proposed_action = $5\n    where notification_id = $1 and candidate_version = $2\n  `, [\n    job.notificationId,\n    candidate.version,\n    hash(message, messages.data),\n    JSON.stringify(redactForEvaluation(message, messages.data)),\n    JSON.stringify(proposal),\n  ]);\n}\n```\n\nOn a worker failure, mark the run `failed`\n\nand deliberately reset it to `pending`\n\nbefore retrying; alert on stale `running`\n\nclaims. Do not let a second delivery run beside the first one.\n\nThe candidate should return a constrained object, not free-form internal reasoning. For a support agent, `reply`\n\n, `escalate`\n\n, `ignore`\n\n, and `request_information`\n\nare usually enough. Put the proposed body in the object only when a reviewer needs to evaluate wording; otherwise store an action and a compact rationale.\n\nThe comparison should match the decision you intend to automate. If the rollout target is routing, grade routing; do not let an eloquent draft hide a wrong category. If the target is reply generation, compare the candidate draft to a human-approved reply and give reviewers a way to mark it safe, wrong, incomplete, or unsafe.\n\nUseful initial checks are deliberately boring:\n\n| Check | What it catches |\n|---|---|\n| Action agreement | Candidate replied when a human escalated, or vice versa. |\n| Unsafe-action rate | Candidate proposed sending, disclosing, or changing something outside its allowed scope. |\n| Duplicate-run rate | Redelivered notifications or worker retries created more than one candidate result. |\n| Time to proposal | A candidate that needs minutes is not ready for a real-time reply loop. |\n| Reviewer override reason | The missing policy, tool, or context that the next version needs. |\n\nDo not compare raw generated text with string equality. Two safe replies can use different wording. Start with a reviewer label, then add task-specific checks. For example, confirm that an escalation preserved the thread ID, that a refund case avoided a promise, or that a support reply cited the current policy.\n\nMake the outcome link explicit. When the existing workflow sends or a reviewer decides, write that decision against the same notification and candidate version:\n\n```\nupdate shadow_runs\nset outcome = $3, actual_message_id = $4, reviewer_verdict = $5\nwhere notification_id = $1 and candidate_version = $2;\n```\n\nFor a human reply, `$4`\n\nis the sent message ID; for a routing-only experiment it can be null and `$3`\n\nis the final queue or escalation action. That gives the review UI a concrete pair: the candidate proposal beside what actually happened.\n\nShadow mode earns trust gradually. A practical rollout has three stages:\n\nThe category boundary should be narrow enough to explain in one sentence. “Reset-password questions that match a known account and need no account change” is a rollout boundary. “Support email” is not.\n\nNylas can send a test webhook to an endpoint, and the CLI can display a mock payload. Test that the ingest path accepts one notification, records one `shadow_events`\n\nrow, and never calls the send or drafts path.\n\n```\nnylas webhook test payload message.created\nnylas webhook test send https://agent.example.com/webhooks/nylas\n```\n\nThen replay the same payload. Assuming your existing append-only audit log records attempted Nylas operations, this is the smallest database check for the boundary:\n\n```\nselect\n  (select count(*) from shadow_runs\n   where notification_id = '<NOTIFICATION_ID>' and candidate_version = '<VERSION>') = 1\n    as one_candidate_run,\n  not exists (\n    select 1 from audit_log\n    where notification_id = '<NOTIFICATION_ID>'\n      and operation in ('messages.send', 'drafts.create')\n  ) as no_mailbox_mutation;\n```\n\nBoth values must be true. This catches duplicate evaluation before it becomes accidental duplicate behavior when you turn the candidate on.\n\nFor a brand-new mailbox with no business risk, an ephemeral Agent Account is the faster test. For a production workflow where the difference between a good and bad reply matters, shadow mode is the bridge you want. It lets you observe a candidate on the messy mix of real threads, reply styles, missing information, and edge cases that fixtures never reproduce without making the customer your canary.\n\nThe durable rule is simple: Nylas carries the message and thread; your application records the experiment; the candidate proposes; a human or the existing workflow acts. Only after those records show that the candidate handles a deliberately narrow category should it earn the right to send.\n\nWhere to go next:", "url": "https://wpnews.pro/news/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas", "canonical_source": "https://dev.to/mqasimca/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas-2mjd", "published_at": "2026-08-29 11:16:27+00:00", "updated_at": "2026-08-29 11:48:55.027546+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops"], "entities": ["Nylas", "Nylas CLI"], "alternates": {"html": "https://wpnews.pro/news/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas", "markdown": "https://wpnews.pro/news/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas.md", "text": "https://wpnews.pro/news/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas.txt", "jsonld": "https://wpnews.pro/news/shadow-test-a-new-ai-email-agent-on-live-threads-with-nylas.jsonld"}}