Every customer conversation carries two kinds of information: the words being said, and how the customer says them. People pick up on the latter without trying, but AI often doesn’t. We can hear the hesitation in a “sure,” the strain in a “that’s fine,” and the relief when a problem finally gets solved. We’re naturally attuned to it. In contrast, AI agents typically work off of transcripts, which don’t include any of these nuances to begin with. They can answer every question correctly and still feel completely tone-deaf if they don’t notice that a customer is confused, frustrated, or ready to move forward.
Our customers have been asking for a way to know how their callers feel, and we wanted our agents to know, too. So we built it into our most recent model launch.
Today, we’re introducing Sentiment in Tango, Simple AI’s conversational awareness model. Tango makes Simple the first voice agent on the market to process sentiment directly from audio, tracking frustration, confusion, and happiness on every call.
Why a Transcript Can’t Capture Sentiment #
When we introduced Tango, we explained how most voice AI works. Models stitch together to transcribe what the customer says, generate a response from that text, then convert it back to speech. Tango moved the decisions that matter most, like end-of-turn and interruption, out of that pipeline by reading them straight from the audio. Sentiment is the clearest example of why that matters.
Much of how we read emotion lives in the parts of speech that don’t make it into a transcript to begin with, like pitch, pacing, volume, and emphasis. Text-based sentiment tools have to infer feeling from word choice alone, which works when the words themselves are emotional, like “this is ridiculous,” but breaks down when the words are neutral and the tone carries the meaning. This is common in customer conversations, where people tend to stay polite even when they aren’t happy. “Thank you” can be gracious, earnest, or deflated, and a transcript records them all the same way. Tango hears the difference.
The same “thank you,” in four tones
A transcript records all four the same way. Tango hears the difference.
Bars show Tango’s score for each signal. Numbers are the level, from 0 to 3.
Why Sentiment Is a Hard Problem #
For people, reading emotion is second nature. For machines, it has been one of the harder problems in speech research. The first reason for this is that there’s no agreed-upon way to divide up emotion. Some research datasets label frustration and confusion, some measure intensity, and some split happiness into separate categories: joy, delight, and satisfaction. Each dataset carves up the emotional space differently, so results from one model rarely translate to another.
The second is data. Labeling emotion from audio is slow and subjective, so many datasets rely on actors reading scripted lines, which keeps them small and doesn’t always sound like a real phone call. Even trained labelers don’t always agree on what they’re hearing.
The third is that the field is still young. Recent progress has come from feeding audio directly into language models and classifiers, which is the same family of approaches that Tango is built on.
How Tango Scores Sentiment #
Three signals, each scored from 0 to 3
Rather than try to label every possible emotion, we focused Tango on three signals that shape a customer conversation and that an agent can actually act on. Confusion tells an agent to slow down and clarify. Frustration tells it to change its approach or escalate to a human agent. Happiness tells it the conversation is going well.
Each signal is scored on a scale from 0 to 3. A 0 means the feeling isn’t present, while 3 is the strongest expression of it: for frustration, think of a caller who has significantly raised their voice. The levels capture how strongly the feeling comes through. Each signal has its own dedicated output in the model, so the three scores move separately. It’s possible for a caller to register as both confused and frustrated at the same time.
Scores that follow the conversation
Tango updates its sentiment scores continuously as the caller speaks, every 80ms. The scores describe the caller’s state over time, rather than in isolated outbursts.
Take a caller who was unhappy with their order. As they explain the problem, frustration rises to a 3, and stays there for as long they continue talking. When the agent finds the order and offers a resolution, you can hear the relief in the caller’s voice, and the score drops back down.
The first thank-you
The caller describes a bad pot roast. Frustration registers at 3 from her first words.
She asks to file a complaint. Frustration holds at 3 while she is speaking.
The agent apologizes and offers to file the claim right away.
The caller thanks the agent. Happiness registers for the first time.
Tango reacts to the conversation as it happens, and doesn’t try to decide whether a later rise is the same confusion or a new one. It reports how the caller sounds at every point in the call, which is exactly what an agent needs to decide what to do next, and what a reviewer needs to see to identify where a call turned.
One model, one pass over the audio
Sentiment isn’t a separate system added on top of Tango. The same model that detects end-of-turn and interruptions produces the sentiment scores, in the same pass over the same audio. There’s no second pipeline to run, no audio to reprocess, and no added latency for the agent. Because Tango already computes sentiment on every call, turning it on is simply a matter of deciding what to do with it.
In development, we found that training these skills together also made Tango better at each of them. Understanding when a caller is frustrated helps the model understand their pacing, and understanding pacing helps it read frustration.
And because Tango is built for real phone audio, sentiment works on the compressed 8kHz telephony that most customer calls still run on.
What This Looks (and Sounds) Like #
The best way to understand sentiment is to hear it. In each example below, listen to the audio, then watch how Tango’s scores move.
From payment to praise
The caller asks to pay by Visa. Happiness starts at 1.
The agent readily agrees to take the card. Happiness rises to 2.
The caller thanks and praises the agent. Happiness peaks at 3.
The agent receives the compliment warmly as the call closes.
Happiness is as useful a signal as frustration. Here, Tango hears a caller’s mood lift, and reacts accordingly.
Reads fine on paper
After the greeting, the caller explains why the wheel-bearing part numbers may not match.
Confusion registers at 1, then climbs to 2 as the caller checks and reads both bearing numbers.
The agent looks up the inner and outer bearings in the catalog.
The agent confirms that both parts are correct. Confusion eases back to 1.
Confusion rises as a customer tries to identify that the parts he received were the right ones for his car. The agent is patient, communicates what he’s looking for, and provides the confirmation.
Frustrated from the first sentence
The caller immediately says this is her second call about the same order.
Frustration jumps to 2 as she explains that she still has not received the right item.
The agent apologizes and acknowledges both the missing order and the repeated call.
A preview of what’s coming: Tango hears frustration building, and the agent slows down to acknowledge it before moving on.
What Sentiment Makes Possible #
A sentiment score on its own is information. The value comes from what agents and teams do with it, and that falls into two broad categories: changing how an agent behaves during a conversation, and understanding customers better across many of them.
During the call: agents that adapt
A good human agent adjusts constantly based on how the customer sounds. Sentiment gives an AI agent the same input, so its next move can depend on how the customer feels, not just what they said.
- Clarifying when confusion rises: Instead of moving on to the next step, the agent can slow down, rephrase, or check that the customer is following.
- Changing course when frustration builds: The agent can acknowledge the frustration, offer a different path, or bring in someone from your team when a person is the right answer.
- Matching the customer’s tone: A brisk, cheerful caller and a hesitant one need different pacing. Sentiment lets the agent tell them apart.
- Recognizing the right moment: When a customer is happy with how the conversation is going, that’s often the natural time to suggest what comes next, whether that’s an upgrade, an add-on, or scheduling the next appointment.
After the call: insight across every conversation
With Tango, every call becomes a record of how the customer felt from start to finish. That means anyone reviewing it can see exactly where the conversation turned, for better or worse, without listening to the whole thing.
Across many calls, the patterns get more useful. If frustration rises every time a particular policy, product, or step in the process comes up, that’s a signal about the customer experience itself, showing the business exactly where to improve.
We use the same signal to improve our own agents, finding the moments where callers get stuck and fixing them together with each customer.
What This Means for Your Business #
Sentiment connects directly to the numbers your team is already measured on.
- Conversion: A customer who’s confused about price, timing, or next steps is less likely to commit. An agent that hears confusion can clear it up while there’s still a sale to make.
- Customer satisfaction: Surveys reach a small share of customers, and only after the call is over. Sentiment gives you a read on every call, as it happens.
- Loyalty and lifetime value: Relationships are built over many conversations. Knowing how each one actually felt to the customer, not just whether the issue was resolved, shows you which relationships are healthy and which need attention.
What’s Next for Sentiment #
Think about the best sales associate you’ve ever had. They knew your name, remembered what you bought last time, and could tell from your voice whether today was a good day. That kind of attention used to be reserved for luxury retail, because it was too costly to give every customer. Sentiment brings us one step closer to offering it on every call.
Want to hear it for yourself? Schedule a demo with our team.