Ticket triage with an LLM: tone moves the priority

A developer's test of two Claude models on 25 short customer support messages found that category and language labels were nearly always correct, but priority labels tracked emotional tone rather than the underlying problem. Sonnet matched expected labels on 21 messages and Haiku on 16, with 12 of the 13 misses involving priority and 10 of those 12 skewing higher due to shouting, anger, quote requests or delayed parcels; the two models disagreed with each other on seven messages. Adding written rules as a skill reportedly improved results.

Two Claude models labelled 25 support messages for us. Category and language came out almost always right; priority tracked the tone instead of the problem. Not consistently. Sonnet and Haiku, two Claude models, each received 25 short customer messages with a request for three labels in JSON: a category from a fixed list, a priority from urgent, high, normal and low, and a language code. Without written rules for choosing the priority, Sonnet matched our expected labels on 21 messages and Haiku on 16. The category and the language were nearly always right. Twelve of the thirteen misses were about priority, and ten of those twelve went up rather than down: shouting, anger, a request for a quote and a delayed parcel were enough to move it. The two models also disagreed with each other on seven messages. Once our rules were added as a skill, both… Read the full report on AISkills402: https://aiskills402.com/blog/llm-ticket-triage-tone-priority https://aiskills402.com/blog/llm-ticket-triage-tone-priority