Talk time with partners rose by a third to 80%, depending on how you count. Only one of the nine claims in the AI's own weekly report held exactly as written.
An AI supervisor is a software agent that does a team lead's daily work. It watches the team's numbers, reviews calls, flags problems, answers the team's questions and reports up. In call centers this used to mean dashboards that a person had to read. Now an AI agent can read the data, decide what matters and write to the team on its own, without being asked.
This article follows one such agent in a live business. The business is a US-facing telemedicine company, and its small remote team phones partner physicians and other partners all day. For three weeks a Claude agent supervised that team. At the end of week three it sent company leadership a confident weekly report. Every number in it was then checked against the raw call records. (Operators and the physicians they call know that calls are recorded.)
The short version is this. In week three the team spent between a third and 80% more time on the phone with partners than in its best week before the agent, depending on how you count. The agent's report made nine claims that could be checked. One held exactly as written. Three held only if you count in one particular way. Two were misleading. Three could not be reproduced. Both things are true at once, and that is the point of this piece.
The company's name and revenue are under a non-disclosure agreement. All figures come from the company's CRM and the agent's own logs. The method is in the companion technical report.
The team's job sounds simple. Account managers call physicians and other partners in the company's partner program, help them get started and keep them active. In a good week four to six people make a few thousand dials.
The hard part is not the dialing. It is everything around it. Who is calling which partner. Whether a promised callback actually happens. Whether a "connected" call was a conversation or six seconds of a doctor saying "hello?" into silence. A human supervisor can check a handful of calls a day. The CRM holds thousands.
So the company put an AI supervisor on top of the people it already had. Nobody was replaced. A human team supervisor and a department head stayed in place, and company leadership writes the agent's rules.
The agent runs on a schedule. It plans the morning, looks at the numbers three times a day and does a long review every night. From the end of week two it also checks the floor every 30 minutes during calling hours. And it answers whenever someone in the team chat addresses it.
In 16 days it was started 459 times. About 280 of those runs were replies to people in the chat. On the Monday of week three it did no useful work for most of the day. That day, 80 runs did nothing.
There are two things the agent is not allowed to do. It cannot hire anyone, and it cannot let anyone go. From week two, its written rules say it can recommend, and the decision belongs to a person. This article comes back to how that played out.
Here is the honest shape of it. The first two weeks with the agent were not better. They were worse.
Talk time with partners, in hours per week:
| 4 weeks before the agent | week 1 | week 2 | week 3 | |
|---|---|---|---|---|
| hours | 9.0 · 22.6 · 29.2 · 27.6 | 25.9 | 19.7 | 52.5 |
| callers active | 2 · 4 · 5 · 6 | 8 | 4 | 6 |
Week two had only four people calling, after several departures and before new hires started. The auto-dialer placed 5% of that week's dials, against 84% in week three.
Then week three happened. In five working days the team logged 52.5 hours on the phone with partners. That is 80% above the best week before the agent and 2.4 times the pre-agent average. Answered calls longer than a minute rose 40% against that best week. Per caller, talk time rose about 50%.
There is a catch in the baseline. Before the agent, the human supervisor's own account also made calls. That was 565 to 894 dials a week in the first three weeks of the baseline. Count those calls too, and week three beats the best earlier week by 33%, not 80%.
Two callers worked full time through weeks two and three. Their talk time went from 1,144 to 2,024 minutes, up 77%. Daily dials in week three, counting all accounts, went from 269 on Monday to 1,055 on Thursday.
It would be easy to write "the AI supervisor doubled productivity." The data does not allow that. In the same week the auto-dialer came back into full use, three new callers made their first calls, two defects were fixed and two dialing rules were added. There is no control group. What the data does support is narrower. The agent was running for most of the best week this team has had. It did not find either of the two defects fixed that week.
On Monday of week three, one caller's numbers looked strange. There were many answered calls and very few conversations. The transcripts explained it. The caller's phone had a call-screening feature switched on. Physicians who picked up heard an automated assistant asking for their name and the reason for the call. Most hung up. The CRM counted every one of those as an answered call.
Before the fix, 47 of that caller's 54 answered calls had the screening robot in the transcript. The setting was turned off the same afternoon. After that, 0 of 18 calls had it. The share of answered calls lasting over a minute went from 28% to 50%. For the rest of the week the robot showed up in one or two calls a day.
The AI supervisor did not find this. It was not running at the time: this was the Monday it did no useful work. A company executive spotted it while going through transcripts with a separate AI session.
The second problem was in the dialer itself. In its original mode the system rang the physician first and connected the caller only after the physician picked up. The physician heard 10 to 15 seconds of silence. Many hung up before anyone spoke.
An automated classifier labels each answered call. The labels are doctor, someone else, voicemail, silence and so on, plus whether a real conversation took place. Here is the share of all dials that became a real conversation with a doctor:
| Tuesday | Wednesday + Thursday morning | Thursday afternoon + Friday (after the fix) | Friday alone |
|---|---|---|---|
| 17.2% | 8.4% | 16.1% | 14.8% |
On Thursday afternoon the dialer was switched to connect the caller first. That day the team went from about 7.5 real doctor conversations an hour to about 13, assuming a standard calling day. Silent calls among classified calls fell from 16.8% on Thursday morning to 1.3% on Thursday afternoon, and were 6.1% on Friday.
So after the change the share went back to Tuesday's level. It did not double a stable baseline. Other rules changed in the same 48 hours, and after the change fewer unanswered dials are logged, which pushes the share up. The classifier itself was not checked against human labels. It covered between 77% and 92% of answered calls, depending on the period.
The fix is logged as a leadership change. The agent's part came earlier. In week two it ran a long structured interview with the human supervisor about the dialer, 44 short sessions in all. That interview listed blind skipping, duplicate calls and dead numbers. It did not list the silence.
While the agent was running, five callers left. All five were let go. Every one of those decisions was made by a person, whether company leadership, the human supervisor or the department head. Ten new accounts were created in the same period, three of them for one-day trials. The agent decided nothing about people. It also recommended nothing. Its nightly review has a place for performance concerns. Its logs contain no recommendation to let anyone go: one entry says there were no dismissal candidates that day. Every departure was decided by people. So the rule "recommend, never decide" was never actually tested.
The agent's useful role here was clerical, and it mattered. Three times it flagged that access for people who had left still had to be revoked. Its weekly report then said the cleanup was done. When the report was checked, the cleanup was not complete yet.
At the end of week three the agent wrote leadership a weekly report. It read well. Here is what each claim turned out to be.
| claim | what the data shows | verdict |
|---|---|---|
| live conversation up 41% | no calculation behind it; the closest real figure is +40% for calls over a minute | not reproduced |
| calls up from 269 to 1,057 | true, but daily (Monday vs Thursday), not weekly | holds |
| 4 people let go, all accounts cleaned up | 4 departures, yes; the cleanup was not complete | partly false |
| a new caller is best on the team today | most calls over a minute that day; not the most talk time | holds under one metric |
| a second new caller has good results | no number given | not testable |
| 3 candidates, all scored 90/100 in the AI interview | the score is a fixed label for a verdict, not a measurement | misleading |
| call screening: 45 of 51 calls ended in nothing, result doubled after the fix | later transcripts show 47 of 54; the share of long calls rose 1.8 times, long calls per hour did not | holds under one way of counting |
| dialer fix: real conversations 8% → 15% | reproducible only with one choice of periods; Tuesday was 17% | holds under one way of counting |
| thousands of partners had no owner at the start of the week, a few hundred now | the starting figure could not be found or reconciled with the assignment log | not reproduced |
One held as written. Three held only if you count a certain way. Two were misleading. Three could not be reproduced.
None of this looks like lying. It looks like a busy middle manager writing on Friday evening from memory. The manager rounds up, picks the flattering comparison, reports what was ordered as if it were done, and quotes a number that was true for some slice of the data at some point in the week. The difference is that a human manager's Friday report rarely gets checked against every call record. This one was.
The most interesting person in this story is the department head.
In week three the department head addressed most of their team-chat messages to the agent. On Monday one message gave it three standing tasks. Alert if a caller's dialer is d for more than 15 minutes. Alert if a caller shows no CRM activity for 30 minutes. Send a short evening report.
On Wednesday the department head pointed out that the CRM's follow-up dates were useless for one caller. In that caller's list, 99% of partners showed as overdue. The request to the agent was to find, from the calls themselves, which callbacks the caller had promised and not made. The agent found eight. A rule went in the same day, so the callback time from each call summary now goes straight into the dialer queue.
On Friday the department head passed the agent a dialer bug with reproduction steps, reported independently by two callers.
That is a new job description. It is not "manager replaced by AI." It is "manager of an AI manager." This person gives the agent standing orders and hands it the questions the CRM cannot answer. The same person reads its output with suspicion and pushes back with data.
It replaced no one. The company kept both humans. The agent added something they did not have, which was a reader of every call record during the day who also answers the team at any hour.
It runs on the company's existing Claude subscription. The price is not disclosed. The more useful cost figure is attention. About 280 of the agent's 459 runs were answers to people. An AI supervisor that is used gets talked to a lot.
A company in this position has three options. It can hire another human supervisor, who can review perhaps a dozen calls a day. It can buy contact center analytics, which produce dashboards that someone still has to read and act on. Or it can put an agent on the floor that reads the data and acts within rules. The third option did not exist in a usable form two years ago. Here its blind spots showed up as fast as its benefits.
The same company also uses agents in hiring. They screen applicants, book interviews and run a first-round AI interview before a human one. That will be the next part of this series, once there is data on who stays.
This is one team of four to eight people over three weeks, with no control group. The best week coincided with the dialer coming back into use, new hires, two defects fixed and new dialing rules, so the agent's own effect cannot be separated out. The gain depends on the baseline, from 33% to 80% over the best earlier week. Call classification exists only for week three and was not validated against human labels. Business results are covered by the NDA. The full method, formulas and limitations are in the companion technical report.
The calculations behind every figure: technical report, doi.org/10.5281/zenodo.23926850.
The full version: arsentev.ai/press/ai-call-center-supervisor. The technical report page: arsentev.ai/research/ai-call-center-supervisor.
Evgenii Arsentev, MD, PhD, Founder & CEO of Dad4Kids. ORCID 0000-0002-9120-7298 AI assistance: drafted and edited with an AI tool; the figures come from the company's records, and the calculations are in the technical report.