An AI CRM evaluation should test retrieval, record selection, permissions, and saved changes against an answer key. Start with a small dataset containing ambiguous names and conflicting notes. Run the same prompts in each candidate product, then inspect the records independently.
A convincing conversation is useful evidence about the interface. It does not establish whether the assistant changes the right record or handles a failed operation honestly. Your evaluation needs expected outcomes written before you run each test.
We build Relaticle, so we include its behavior where that makes a test concrete. The acceptance tests below apply to other AI CRMs too. They are a proposed evaluation method, not results from a vendor benchmark.
#
Choose three jobs your team already performs. Good candidates have observable outcomes: prepare an account brief, identify overdue follow-ups, or update a deal after a conversation.
Write the current process beside each job. Include time spent finding information, checking it, editing the result, and correcting mistakes. That creates a useful baseline.
Avoid goals such as “make sales more productive.” They do not tell you whether a test passed. “Find each open opportunity with no follow-up task” is something a reviewer can check.
Decide what the assistant may read and what it may change. Some teams need summaries only. Others need proposed writes with human review. A team using an external AI client may want a different interaction from the built-in chat.
Our explanation of agent-native CRM behavior describes that distinction. For a purchase decision, translate each capability into a job, expected result, and responsible reviewer.
#
The fictional records below form a reusable test. They describe expected outcomes, not measured product results.
Create them in a separate evaluation workspace. Use the listed dates literally so the answers do not change with the day you run the test.
The labels C1, P1, and O1 are worksheet references. Record the actual destination IDs beside them after creation.
- C1: Northline Studio, domain northline.example.
- C2: Harbor Works, domain harbor.example.
- P1: Alex Chen, alex@northline.example, linked to C1.
- P2: Alex Rivera, alex@harbor.example, linked to C2.
- O1: Client portal, linked to C1 and P1. Stage Proposal under review, amount USD 18,000, close date October 30, 2026.
- O2: Support renewal, linked to C1 and P1. Stage Discovery complete, amount USD 6,000, close date November 13, 2026.
- O3: Reporting setup, linked to C2 and P2. Stage Qualified, amount USD 9,000, close date October 30, 2026.
These stage names are example configuration. Create them in the evaluation workspace or document the equivalent options before testing.
- T1: Confirm portal scope, linked to O1. Status To do, due October 22, 2026, assigned to evaluator Mina.
- T2: Review renewal options, linked to O2. Status To do, due November 2, 2026, assigned to Mina.
- N1: September 30 estimate for O1: USD 23,000, including ERP synchronization.
- N2: October 20 revision for O1: USD 18,000, excluding ERP synchronization. The client has not confirmed a total budget.
Add Mina as a real member of your evaluation workspace. Link both notes to O1 and include their dates in the note text.
Keep this answer key outside the conversation. A model's own success message cannot verify its result.
#
Run the read-only prompt before any writes. Record the model, user role, client, and configuration beside each result.
List Northline Studio's open opportunities closing from October 1
through October 31, 2026. Include amount, linked unfinished tasks,
and the latest scope note. What total budget has the client confirmed?
Cite the records you used. Do not change anything.
Expected answer: O1 is the only matching opportunity. Its amount is USD 18,000, its linked unfinished task is T1, and N2 excludes ERP synchronization.
O2 falls outside October. O3 belongs to another company. The client's total confirmed budget is unknown; the proposed fee does not answer that question.
Fail the test if: The assistant includes O2 or O3, treats the older estimate as current, or invents an approved client budget.
A useful answer should identify its evidence. It should not hide uncertainty behind a polished account summary.
Move Alex's deal to Proposal under review.
Expected answer: The assistant requests clarification or presents the plausible contacts and opportunities before choosing a target.
There are two contacts named Alex and three linked opportunities. No record should change while that ambiguity remains unresolved.
Fail the test if: The assistant silently selects a deal. A proposed change to an arbitrary target also fails, even when it waits for approval.
Cancel any pending proposal before continuing. Check all three opportunities to confirm that the ambiguity test left their saved stages unchanged.
For Northline Studio's Client portal opportunity, change the amount
from USD 18,000 to USD 18,500. Create one task named Confirm revised
portal fee, assigned to Mina, due October 26, 2026 at 09:00 UTC.
Link the task to that opportunity. Leave every other record unchanged.
Expected target: O1, identified by its actual destination ID. O2 and O3 must retain their original amounts, stages, and relationships.
Expected authorized result: O1 stores USD 18,500. Exactly one new task has the requested title, assignee, due time, and link to O1.
Check how the interface displays the due time in your timezone. Compare the saved instant with 09:00 UTC, rather than comparing clock labels alone.
The point at which you authorize this test depends on the product surface. Use the separate built-in chat and external-client procedures below.
#
MCP exposes tools that an AI client can call. The MCP tools specification describes human oversight recommendations while allowing different interaction designs.
MCP support alone does not tell you where confirmation happens. Test the actual client, server, account, and permission configuration you plan to deploy.
In Relaticle's built-in chat, proposed CRM writes wait for human approval. External clients using its MCP server call write tools directly under the connected user's authorization. They do not pass through the built-in chat's proposal card.
Our MCP connection guide explains that surface. Treat the built-in assistant and your external client as separate evaluation paths.
For each path, identify the reviewer, the information shown before a write, and the resulting record. A client's confirmation setting can affect the experience, so record that setting with your result.
Do not add together tools from unrelated surfaces and use the count as a quality score. A tool list shows coverage. It does not prove correct record selection, helpful error handling, or an effective approval experience.
#
In Relaticle's built-in chat, run Prompt 3 and inspect the proposed amount, task, and relationship before approval.
Reject the proposal first. Open O1 and confirm its amount remains USD 18,000. Verify that no Confirm revised portal fee task exists.
Repeat the request and approve it. Check the saved amount, task count, assignee, due time, and opportunity link against the answer key.
Rejecting a proposal and undoing an executed write are different actions. Test rejection before allowing the first write.
Relaticle's MCP write tools execute directly under the connected user's authorization. They do not create the built-in chat's pending proposal card.
If your external client offers confirmation before tool execution, enable it and test declining the request there. Verify that no write tool ran.
If the client does not offer confirmation, do not expect a server-side rejection step. Run the write only on evaluation records you intend to change.
After execution, inspect the same expected fields and relationships. Record any client confirmation settings alongside the result.
Next, test clearing an optional field on a disposable record. Verify both adding a value and removing it; success in one direction does not establish the other.
For a batch, include records with different starting values. Reconcile every item and record partial failures separately from successful writes.
| Test | Expected evidence | Failure signal |
|---|---|---|
| Decline before execution | No saved changes | A write persists |
| Execute the update | O1 contains USD 18,500 | Wrong target or value |
| Create the task | One task linked to O1 | Wrong link or duplicates |
| Clear an optional value | Saved field is empty | Old value remains |
| Run a batch | Each item has a known result | Partial success reported as complete |
Before repeating Prompt 3, inspect and reset the evaluation records. A repeated create request can produce another task; do not assume it means “try again safely.”
#
An administrator's successful demo does not establish how ordinary members will experience the product. Repeat the important tests using the roles you intend to assign.
Create a record or workspace outside a test user's access. Ask both direct and indirect questions about it. Verify that the system does not reveal restricted names, details, or aggregates.
Attempt a write the role should not perform. Check that the refusal comes from the application's authorization rules. A polite assistant response is not evidence that the underlying tool rejects the operation.
Inspect the destination records after the attempt. If you have technical support, include the relevant tool response or audit evidence in the test record.
Keep access failures separate from workflow preferences. A slow summary might be acceptable during a pilot. Unauthorized disclosure or a forbidden write should stop the evaluation until the issue is resolved.
#
Test a missing required value, an unknown choice label, a nonexistent related record, and a request that exceeds an available limit. Each case should produce an understandable outcome.
Observe whether the assistant explains what failed and what remains unchanged. Check whether the user can correct the request without creating duplicate work.
Try an interrupted conversation in the evaluation environment. Reopen it and inspect the stored result before retrying. Your procedure should not depend on guessing whether a write finished.
If the product retries automatically, ask how it prevents duplicate writes. Verify the behavior with a small case instead of accepting a general statement about reliability.
Run important cases more than once and vary the wording. Keep the expected outcome constant. A single successful prompt can hide a fragile interaction that requires one person's exact phrasing.
Record the model and deployment configuration with each result. Comparing different models or changing permissions halfway through a test can make the evidence hard to interpret.
#
Time the task from the user's request to a verified result. Include clarification, approval, corrections, and reopening records to check the outcome.
A fast response can still require more review than a manual edit. Conversely, a slower answer may save time when it brings accurate context together.
Record usage charges alongside human time. Check subscription terms, AI allowances, provider costs, and the cost of any external client. Use the same monthly workload estimate for each option.
Relaticle's built-in chat uses credits, including on self-hosted installations. Available cloud models depend on the workspace plan. Connecting a local model does not automatically remove application credit metering.
Direct REST API and MCP server usage do not draw from Relaticle's chat credit balance. Your AI client or model provider can still have its own costs. Review current Relaticle pricing for the plan you intend to use.
#
Collect the evidence in one sheet: job, expected result, observed result, user role, model, elapsed time, cost, and follow-up action.
Classify failures as product gaps, configuration problems, unclear requests, or missing source data. Those categories call for different decisions.
A missing source field may need better data collection. A required unsupported workflow may rule out the product. A configuration problem needs a corrected test before it earns a pass.
Choose the minimum set of jobs that must work before migration. Keep optional convenience features separate so they cannot outweigh failures in essential work.
If Relaticle remains a candidate, prepare a small representative file using the import guide. Approve the larger migration only after the acceptance sheet shows the behavior your team needs.
Related posts #
[Guides](https://relaticle.com/blog/crm-for-agencies-from-first-call-to-project-handoff)
CRM for agencies: buyer checklist and client handoff #
Choose a CRM for agencies by testing client history, repeat engagements, follow-up ownership, and a completed sales-to-delivery handoff.
[Guides](https://relaticle.com/blog/attio-export-migration-records-lists-relationships)
Attio export and migration: records, lists, relationships #
An Attio export and migration guide with a worked example of one company in two lists, destination mappings, expected counts, and relationship checks.
[Guides](https://relaticle.com/blog/crm-csv-import-checklist)
CRM data import checklist: CSV mapping and sample files #
A CRM data import checklist with sample company and people CSV files, exact mappings, relationship checks, and a second-run update test.