Do LLMs Actually Understand Sarcasm, or Just Pattern-Match It?

A Turkish sarcasm detection project at Acıbadem University revealed that large language models often rely on pattern-matching rather than true pragmatic understanding, raising questions about their ability to comprehend sarcasm across languages.

Member-only story Do LLMs Actually Understand Sarcasm, or Just Pattern-Match It? Lessons from building a Turkish sarcasm detector that cheated and got caught. This post draws on my senior capstone project on Turkish sarcasm detection, developed at Acıbadem University in collaboration with Dedecta. Sarcasm analysis may sound like a classical & simple task to do in NLP. But it’s actually very different and language-dependent compared to other NLP tasks such as sentiment analysis and/or topic classification. While sentiment analysis looks for certain words in a sentence, this method is impossible for sarcasm detection. Let’s understand this with an example: “This was a great idea.” This sentence could mean a real compliment or a heavy negative comment in social media language. The difference between the two is in the context, or in the hints that come from the tone, the shared social context. A person could solve this momentarily. What about an LLM? The Problem With “Understanding” Pragmatic understanding vs. statistical correlation Before asking whether an LLM understands sarcasm, we need to be precise about what sarcasm actually is, because “understanding” a phenomenon you can’t define is just guesswork. Sarcasm is a specialized form of verbal irony: the speaker says the opposite of what they literally mean, typically to mock, criticize, or express contempt. But…