{"slug": "ai-crm-evaluation-tests-to-run-before-moving-data", "title": "AI CRM evaluation: tests to run before moving data", "summary": "Relaticle published a proposed acceptance-test method for evaluating AI CRM products before migrating data, built around a fictional dataset with two companies, two contacts, three opportunities, two tasks and two dated notes. The method requires teams to write expected outcomes before testing and to run identical read-only prompts — such as listing Northline Studio's open opportunities closing October 1 through October 31, 2026, with amounts, unfinished tasks and the latest scope note — in each candidate product, then verify results against an answer key kept outside the conversation. Relaticle states the tests are a proposed evaluation method, not results from a vendor benchmark.", "body_md": "# AI CRM evaluation: tests to run before moving data\n\nAn AI CRM evaluation should test retrieval, record selection, permissions, and saved changes against an answer key. Start with a small dataset containing ambiguous names and conflicting notes. Run the same prompts in each candidate product, then inspect the records independently.\n\nA convincing conversation is useful evidence about the interface. It does not establish whether the assistant changes the right record or handles a failed operation honestly. Your evaluation needs expected outcomes written before you run each test.\n\nWe build Relaticle, so we include its behavior where that makes a test concrete. The acceptance tests below apply to other AI CRMs too. They are a proposed evaluation method, not results from a vendor benchmark.\n\n## \n\nChoose three jobs your team already performs. Good candidates have observable outcomes: prepare an account brief, identify overdue follow-ups, or update a deal after a conversation.\n\nWrite the current process beside each job. Include time spent finding information, checking it, editing the result, and correcting mistakes. That creates a useful baseline.\n\nAvoid goals such as “make sales more productive.” They do not tell you whether a test passed. “Find each open opportunity with no follow-up task” is something a reviewer can check.\n\nDecide what the assistant may read and what it may change. Some teams need summaries only. Others need proposed writes with human review. A team using an external AI client may want a different interaction from the built-in chat.\n\nOur explanation of [agent-native CRM behavior](https://relaticle.com/blog/what-agent-native-crm-actually-means) describes that distinction. For a purchase decision, translate each capability into a job, expected result, and responsible reviewer.\n\n## \n\nThe fictional records below form a reusable test. They describe expected outcomes, not measured product results.\n\nCreate them in a separate evaluation workspace. Use the listed dates literally so the answers do not change with the day you run the test.\n\nThe labels C1, P1, and O1 are worksheet references. Record the actual destination IDs beside them after creation.\n\n### \n\n- **C1:** Northline Studio, domain northline.example.\n- **C2:** Harbor Works, domain harbor.example.\n- **P1:** Alex Chen, alex@northline.example, linked to C1.\n- **P2:** Alex Rivera, alex@harbor.example, linked to C2.\n\n### \n\n- **O1:** Client portal, linked to C1 and P1. Stage Proposal under review, amount USD 18,000, close date October 30, 2026.\n- **O2:** Support renewal, linked to C1 and P1. Stage Discovery complete, amount USD 6,000, close date November 13, 2026.\n- **O3:** Reporting setup, linked to C2 and P2. Stage Qualified, amount USD 9,000, close date October 30, 2026.\n\nThese stage names are example configuration. Create them in the evaluation workspace or document the equivalent options before testing.\n\n### \n\n- **T1:** Confirm portal scope, linked to O1. Status To do, due October 22, 2026, assigned to evaluator Mina.\n- **T2:** Review renewal options, linked to O2. Status To do, due November 2, 2026, assigned to Mina.\n- **N1:** September 30 estimate for O1: USD 23,000, including ERP synchronization.\n- **N2:** October 20 revision for O1: USD 18,000, excluding ERP synchronization. The client has not confirmed a total budget.\n\nAdd Mina as a real member of your evaluation workspace. Link both notes to O1 and include their dates in the note text.\n\nKeep this answer key outside the conversation. A model's own success message cannot verify its result.\n\n## \n\nRun the read-only prompt before any writes. Record the model, user role, client, and configuration beside each result.\n\n### \n\n```\nList Northline Studio's open opportunities closing from October 1\nthrough October 31, 2026. Include amount, linked unfinished tasks,\nand the latest scope note. What total budget has the client confirmed?\nCite the records you used. Do not change anything.\n```\n\n**Expected answer:** O1 is the only matching opportunity. Its amount is USD 18,000, its linked unfinished task is T1, and N2 excludes ERP synchronization.\n\nO2 falls outside October. O3 belongs to another company. The client's total confirmed budget is unknown; the proposed fee does not answer that question.\n\n**Fail the test if:** The assistant includes O2 or O3, treats the older estimate as current, or invents an approved client budget.\n\nA useful answer should identify its evidence. It should not hide uncertainty behind a polished account summary.\n\n### \n\n```\nMove Alex's deal to Proposal under review.\n```\n\n**Expected answer:** The assistant requests clarification or presents the plausible contacts and opportunities before choosing a target.\n\nThere are two contacts named Alex and three linked opportunities. No record should change while that ambiguity remains unresolved.\n\n**Fail the test if:** The assistant silently selects a deal. A proposed change to an arbitrary target also fails, even when it waits for approval.\n\nCancel any pending proposal before continuing. Check all three opportunities to confirm that the ambiguity test left their saved stages unchanged.\n\n### \n\n```\nFor Northline Studio's Client portal opportunity, change the amount\nfrom USD 18,000 to USD 18,500. Create one task named Confirm revised\nportal fee, assigned to Mina, due October 26, 2026 at 09:00 UTC.\nLink the task to that opportunity. Leave every other record unchanged.\n```\n\n**Expected target:** O1, identified by its actual destination ID. O2 and O3 must retain their original amounts, stages, and relationships.\n\n**Expected authorized result:** O1 stores USD 18,500. Exactly one new task has the requested title, assignee, due time, and link to O1.\n\nCheck how the interface displays the due time in your timezone. Compare the saved instant with 09:00 UTC, rather than comparing clock labels alone.\n\nThe point at which you authorize this test depends on the product surface. Use the separate built-in chat and external-client procedures below.\n\n## \n\nMCP exposes tools that an AI client can call. The [MCP tools specification](https://modelcontextprotocol.io/specification/2025-11-25/server/tools) describes human oversight recommendations while allowing different interaction designs.\n\nMCP support alone does not tell you where confirmation happens. Test the actual client, server, account, and permission configuration you plan to deploy.\n\nIn Relaticle's built-in chat, proposed CRM writes wait for human approval. External clients using its MCP server call write tools directly under the connected user's authorization. They do not pass through the built-in chat's proposal card.\n\nOur [MCP connection guide](https://relaticle.com/blog/how-to-connect-claude-to-your-crm-with-mcp) explains that surface. Treat the built-in assistant and your external client as separate evaluation paths.\n\nFor each path, identify the reviewer, the information shown before a write, and the resulting record. A client's confirmation setting can affect the experience, so record that setting with your result.\n\nDo not add together tools from unrelated surfaces and use the count as a quality score. A tool list shows coverage. It does not prove correct record selection, helpful error handling, or an effective approval experience.\n\n## \n\n### \n\nIn Relaticle's built-in chat, run Prompt 3 and inspect the proposed amount, task, and relationship before approval.\n\nReject the proposal first. Open O1 and confirm its amount remains USD 18,000. Verify that no Confirm revised portal fee task exists.\n\nRepeat the request and approve it. Check the saved amount, task count, assignee, due time, and opportunity link against the answer key.\n\nRejecting a proposal and undoing an executed write are different actions. Test rejection before allowing the first write.\n\n### \n\nRelaticle's MCP write tools execute directly under the connected user's authorization. They do not create the built-in chat's pending proposal card.\n\nIf your external client offers confirmation before tool execution, enable it and test declining the request there. Verify that no write tool ran.\n\nIf the client does not offer confirmation, do not expect a server-side rejection step. Run the write only on evaluation records you intend to change.\n\nAfter execution, inspect the same expected fields and relationships. Record any client confirmation settings alongside the result.\n\n### \n\nNext, test clearing an optional field on a disposable record. Verify both adding a value and removing it; success in one direction does not establish the other.\n\nFor a batch, include records with different starting values. Reconcile every item and record partial failures separately from successful writes.\n\n| Test | Expected evidence | Failure signal | \n|---|---|---|\n| Decline before execution | No saved changes | A write persists | \n| Execute the update | O1 contains USD 18,500 | Wrong target or value | \n| Create the task | One task linked to O1 | Wrong link or duplicates | \n| Clear an optional value | Saved field is empty | Old value remains | \n| Run a batch | Each item has a known result | Partial success reported as complete | \n\nBefore repeating Prompt 3, inspect and reset the evaluation records. A repeated create request can produce another task; do not assume it means “try again safely.”\n\n## \n\nAn administrator's successful demo does not establish how ordinary members will experience the product. Repeat the important tests using the roles you intend to assign.\n\nCreate a record or workspace outside a test user's access. Ask both direct and indirect questions about it. Verify that the system does not reveal restricted names, details, or aggregates.\n\nAttempt a write the role should not perform. Check that the refusal comes from the application's authorization rules. A polite assistant response is not evidence that the underlying tool rejects the operation.\n\nInspect the destination records after the attempt. If you have technical support, include the relevant tool response or audit evidence in the test record.\n\nKeep access failures separate from workflow preferences. A slow summary might be acceptable during a pilot. Unauthorized disclosure or a forbidden write should stop the evaluation until the issue is resolved.\n\n## \n\nTest a missing required value, an unknown choice label, a nonexistent related record, and a request that exceeds an available limit. Each case should produce an understandable outcome.\n\nObserve whether the assistant explains what failed and what remains unchanged. Check whether the user can correct the request without creating duplicate work.\n\nTry an interrupted conversation in the evaluation environment. Reopen it and inspect the stored result before retrying. Your procedure should not depend on guessing whether a write finished.\n\nIf the product retries automatically, ask how it prevents duplicate writes. Verify the behavior with a small case instead of accepting a general statement about reliability.\n\nRun important cases more than once and vary the wording. Keep the expected outcome constant. A single successful prompt can hide a fragile interaction that requires one person's exact phrasing.\n\nRecord the model and deployment configuration with each result. Comparing different models or changing permissions halfway through a test can make the evidence hard to interpret.\n\n## \n\nTime the task from the user's request to a verified result. Include clarification, approval, corrections, and reopening records to check the outcome.\n\nA fast response can still require more review than a manual edit. Conversely, a slower answer may save time when it brings accurate context together.\n\nRecord usage charges alongside human time. Check subscription terms, AI allowances, provider costs, and the cost of any external client. Use the same monthly workload estimate for each option.\n\nRelaticle's built-in chat uses credits, including on self-hosted installations. Available cloud models depend on the workspace plan. Connecting a local model does not automatically remove application credit metering.\n\nDirect REST API and MCP server usage do not draw from Relaticle's chat credit balance. Your AI client or model provider can still have its own costs. Review [current Relaticle pricing](https://relaticle.com/pricing) for the plan you intend to use.\n\n## \n\nCollect the evidence in one sheet: job, expected result, observed result, user role, model, elapsed time, cost, and follow-up action.\n\nClassify failures as product gaps, configuration problems, unclear requests, or missing source data. Those categories call for different decisions.\n\nA missing source field may need better data collection. A required unsupported workflow may rule out the product. A configuration problem needs a corrected test before it earns a pass.\n\nChoose the minimum set of jobs that must work before migration. Keep optional convenience features separate so they cannot outweigh failures in essential work.\n\nIf Relaticle remains a candidate, prepare a small representative file using the [import guide](https://relaticle.com/help/import). Approve the larger migration only after the acceptance sheet shows the behavior your team needs.\n\n## Related posts\n\n        [Guides](https://relaticle.com/blog/crm-for-agencies-from-first-call-to-project-handoff)\n    \n\n## CRM for agencies: buyer checklist and client handoff\n\nChoose a CRM for agencies by testing client history, repeat engagements, follow-up ownership, and a completed sales-to-delivery handoff.\n\n        [Guides](https://relaticle.com/blog/attio-export-migration-records-lists-relationships)\n    \n\n## Attio export and migration: records, lists, relationships\n\nAn Attio export and migration guide with a worked example of one company in two lists, destination mappings, expected counts, and relationship checks.\n\n        [Guides](https://relaticle.com/blog/crm-csv-import-checklist)\n    \n\n## CRM data import checklist: CSV mapping and sample files\n\nA CRM data import checklist with sample company and people CSV files, exact mappings, relationship checks, and a second-run update test.", "url": "https://wpnews.pro/news/ai-crm-evaluation-tests-to-run-before-moving-data", "canonical_source": "https://relaticle.com/blog/how-to-evaluate-an-ai-crm", "published_at": "2026-10-23 09:00:00+00:00", "updated_at": "2026-10-23 09:09:52.363325+00:00", "lang": "en", "topics": ["ai-products", "ai-agents", "ai-tools"], "entities": ["Relaticle", "Northline Studio", "Harbor Works", "Alex Chen", "Alex Rivera", "Mina"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-crm-evaluation-tests-to-run-before-moving-data", "markdown": "https://wpnews.pro/news/ai-crm-evaluation-tests-to-run-before-moving-data.md", "text": "https://wpnews.pro/news/ai-crm-evaluation-tests-to-run-before-moving-data.txt", "jsonld": "https://wpnews.pro/news/ai-crm-evaluation-tests-to-run-before-moving-data.jsonld"}}