{"slug": "polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms", "title": "Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms", "summary": "A study of seven large language models found that the models agree with each other on politeness judgments more than they agree with human raters, according to the arXiv paper 2609.29001v1. The paper reports that model-human alignment is associated with explicit linguistic cues, while some rapport-building strategies appear more often in misaligned cases, and that in the three-way categorical task models systematically overproduce Neutral labels and underpredict Impolite labels, a pattern that persisted against expert consensus on a diagnostic subset. The authors conclude that pragmatic evaluations should examine directional patterns of model-human disagreement rather than aggregate agreement metrics alone.", "body_md": "arXiv:2609.29001v1 Announce Type: new \nAbstract: Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.", "url": "https://wpnews.pro/news/polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms", "canonical_source": "https://arxiv.org/abs/2609.29001", "published_at": "2026-09-25 04:00:00+00:00", "updated_at": "2026-09-25 04:00:52.966443+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "ai-safety"], "entities": ["arXiv", "2609.29001v1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms", "markdown": "https://wpnews.pro/news/polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms.md", "text": "https://wpnews.pro/news/polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms.txt", "jsonld": "https://wpnews.pro/news/polite-but-misaligned-evaluating-llm-politeness-judgments-against-human-norms.jsonld"}}